akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Learning slow, loss curve fluctuating with batches

username: Targ3t
cooked: <p>Hi,<br>
I’m trying to implement a bidirectional character level LSTM model.<br>
The inputs consist of one hot encoded sequences and each element of that sequence has a corresponding target class. I’m using CrossEntropyLoss and Adam() as optimizer. In order to be able to use CrossEntropyLoss I’m reshaping the output tensor and removing padded elements using a mask to have the shape of (all_sequence_elements,number_of_output_features). I’m also removing padded elements (0’s) from the target tensor so it ends up being a 1D vector consisting of values corresponding to each input element.<br>
I tried to run the model on a very small dataset but the learning is pretty slow even for lr=0.01 for 1 batch and the loss is very high for multiple batches and fluctuates a lot.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/871a0ae8b0ee256557dc2adf4f2782015a106aef.png" data-download-href="https://discuss.pytorch.org/uploads/default/871a0ae8b0ee256557dc2adf4f2782015a106aef" title="losses"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/871a0ae8b0ee256557dc2adf4f2782015a106aef_2_690x296.png" alt="losses" data-base62-sha1="jhafMuPl5ujXLfWo1Df6EtBjLFJ" width="690" height="296" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/871a0ae8b0ee256557dc2adf4f2782015a106aef_2_690x296.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/871a0ae8b0ee256557dc2adf4f2782015a106aef_2_1035x444.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/871a0ae8b0ee256557dc2adf4f2782015a106aef.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">losses</span><span class="informations">1073×461 31.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Code:</p>
<pre><code class="lang-auto">        optimizer.zero_grad()
        output, (hn, cn) = rnn(input_tensor, (h0, c0)) #output(max_sequence_len,batch_size,out_features*2)
        mask = (target_tensor != 0)
        target = target_tensor[mask] #target(targets_from_all_sequences_combined)
        output = (output[:, :, :hidden_size] + output[:, :, hidden_size:]) # sum of both directions
        output = output.view(-1, n_output_letters)
        output = output[mask.view(-1)]
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
</code></pre>
<p>I made sure that there’s no mistakes in my input and target tensors.<br>
Please let me know if I’m doing something wrong.<br>
Thanks.</p>

username: hs271095
cooked: <p>Did you find a solution to this problem?</p>
How to register hook function for functional form?

username: wwiiiii
cooked: <p>I’m now trying to implement Guided backpropagation from <a href="https://arxiv.org/abs/1412.6806" rel="nofollow noopener">https://arxiv.org/abs/1412.6806</a> , which alters the gradient calculation in ReLU layer.</p>
<p>When given model is implemented in the way containing nn.ReLU() instance as its submodule, I can easily register backward hook function to its every submodule as below.</p>
<pre><code class="lang-auto">model = resnet101(pretrained=True)
def hook(module, grad_input, grad_output):
    if isinstance(module, nn.ReLU):
        # do something
        return changed_grad_input
for i, v in model.named_modules():
    v.register_backward_hook(hook)
</code></pre>
<p>However, if the model applies ReLU within its forward function with functional form, as below,</p>
<pre><code class="lang-auto">class foo(nn.Module):
    def forward(self, x):
        x = poo(x)
        x = F.relu(x)
        return x
</code></pre>
<p>it seems above approach doesn’t register hook function for that.</p>
<p>How can I deal with hook function in this kind of situation?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately I’m not sure if it is easy to do this.<br>
I guess you could always replace <code>F.relu</code> by a custom function that would use the nn.Module version with your hook.<br>
Otherwise, you can add a hook on any tensor you want, but that would require modifying the <code>forward</code> method of your module.</p>
Cross Validation

username: DJ_1992
cooked: <p>Hii,<br>
I would like to do cross validation on my dataset.<br>
Currently I have a binary classification network for medical images and having a 90% accuracy on my validation set. My validation image dataset is small, so i would like to do cross validation.<br>
I am using a custom dataset class to load the dataset and the Folders are arranged in this way:<br>
Train/1stclass<br>
Train/2ndclass<br>
Valid/1stClass<br>
Valid/2ndclass</p>
<p>Should I mix them in one Folder for the Cross Validation?<br>
I am new to pytorch,  Please if anyone can help me with this… I would really appreciate it … Thank You</p>

username: arminarj
cooked: <p>Hi Dear <a class="mention" href="/u/dj_1992">@DJ_1992</a>,<br>
There is a similar answer to your inquiry, Hope it helps. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
<a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035/2">https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035/2?u=arminarj</a></p>

username: DJ_1992
cooked: <aside class="quote no-group quote-modified" data-username="arminarj" data-post="2" data-topic="75245">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arminarj/48/21696_2.png" class="avatar"> arminarj:</div>
<blockquote>
<p><a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035/2" class="inline-onebox">What is the best way to apply k-fold cross validation in CNN? - #2 by ptrblck</a></p>
</blockquote>
</aside>
<p>Thanks for the link <a class="mention" href="/u/arminarj">@arminarj</a>, I tried to make it work in my project…<br>
But I still have some doubts… I am using the NeuralNetClassifier from skorch</p>
<blockquote>
<p>net = NeuralNetClassifier(<br>
model,<br>
criterion= nn.CrossEntropyLoss,<br>
max_epochs=200,<br>
lr=0.001,<br>
optimizer=optim.Adam,<br>
train_split=0.2,<br>
iterator_train__shuffle=True<br>
)<br>
net.fit(images,labels)</p>
</blockquote>
<p>images and labels are tensors… I am confused as to what should be in net.fit()? And also how can I use cvsplit for Cross Validation??<br>
I am getting this error:</p>
<blockquote>
<p>TypeError: ‘float’ object is not callable</p>
</blockquote>
Build PyTorch 1.4.0 from source failed

username: Yinglei_Zhang
cooked: <p>I am building pytorch from source and got a error below.</p>
<pre><code class="lang-auto">../../lib/libc10.so: error: undefined reference to 'std::thread::_State::~_State()', version 'GLIBCXX_3.4.22'
../../lib/libc10.so: error: undefined reference to 'typeinfo for std::thread::_State', version 'GLIBCXX_3.4.22'
../../lib/libc10.so: error: undefined reference to 'std::runtime_error::runtime_error(char const*)', version 'GLIBCXX_3.4.21'
../../lib/libc10.so: error: undefined reference to 'std::thread::_M_start_thread(std::unique_ptr&lt;std::thread::_State, std::default_delete&lt;std::thread::_State&gt; &gt;, void (*)())', version 'GLIBCXX_3.4.22'
../../lib/libgtest.a(gtest-all.cc.o):gtest-all.cc:function testing::internal::GoogleTestFailureException::GoogleTestFailureException(testing::TestPartResult const&amp;): error: undefined reference to 'std::runtime_error::runtime_error(char const*)'
</code></pre>
<p>To see a full build log, you can refer to: <a href="https://gist.github.com/YingleiZhang/fc98c94d493ac39506daacefa0298b38#file-build-with-gcc-7-5-L2-L86" rel="nofollow noopener">https://gist.github.com/YingleiZhang/fc98c94d493ac39506daacefa0298b38#file-build-with-gcc-7-5-L2-L86</a>.<br>
Any help is appreciated.</p>
CNN and Actor Critic

username: Mehdi
cooked: <p>Hello,</p>
<p>When using convolutional neural networks with reinforcement learning, it is best to have the critic and the policy share the convolutional layers or rather create two separate entities of it ?</p>
<p>Thanks !</p>

username: smth
cooked: <p>my thinking is that it’s probably best to share conv layers in this case.</p>

username: CesMak
cooked: <p>In the general sense of Actor-Critic family of algorithms, there is no need to share the network parameters. You could have total separate two networks. That would cost you more memory and compute and most likely take longer.</p>
<p>Sharing the parameters improves the training time however it makes it harder to train because the parameters need more careful tuning. This is because the norm of gradients flowing back from the actor gradients and the critic gradients are at completely different scales which is hard to calibrate.</p>
A littile question about pytorch training parameters

username: jiengup
cooked: <p>I’m learning pytorch and DL, I’m training a classifier on a image dataset. I have splited train dataset and validation dataset and I compelete traning on training set like this:</p>
<pre><code class="lang-auto">model = Classifier().cuda()
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
num_epoch = 30
for epoch in range(num_epoch):
    ...
</code></pre>
<p>After that, I want to merge training set and validation set to get a bigger dataset and train more 30 epochs.<br>
Could I code like this?:</p>
<pre><code class="lang-auto">train_val_x = np.concatenate((train_x, val_x), axis=0)
train_val_y = np.concatenate((train_y, val_y), axis=0)
train_val_set = ImgDataset(train_val_x, train_val_y, train_transform)
train_val_loader = DataLoader(train_val_set, batch_size=batch_size, shuffle=True)
model_best = Classifier().cuda()
loss = nn.CrossEntropyLoss() # 因為是 classification task，所以 loss 使用 CrossEntropyLoss
optimizer = torch.optim.Adam(model_best.parameters(), lr=0.001) # optimizer 使用 Adam
num_epoch = 30
for epoch in range(num_epoch):
    ...
</code></pre>
<p>Or should I use the formmer var <strong>model</strong> instead of <strong>model_best</strong><br>
Actually my question is that in pytorch, <strong>model.Module.parameter()</strong> method can remain my pre-train parameters as static?<br>
Thanks a lot. ^.^</p>
Swp memory increases very fast

username: sidd.suresh97
cooked: <pre><code class="lang-auto">transform = transforms.Compose(
    [transforms.Resize((375,375)), transforms.ToTensor()])
path = #path to folder containing Images
train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.ImageFolder(path, transform=transform),
    batch_size=100,
    num_workers=0,
    shuffle=True)
class DNN:
    def __init__(self):
        self.outputs = []
        self.model = models.resnet50(pretrained=True)
        self.model.avgpool.register_forward_hook(self.hook)
        self.model.eval()
    
    def hook(self,module, input, output):
        self.outputs.append(output)
    # Loop over 
    def step(self, inputs):
        data, label = inputs # ignore label
        _ = self.model(data)
        return 0

    
    def predict(self, dataloader):
        for i, batch in enumerate(dataloader):
            print(i)
            _ = self.step(batch)

a = DNN()
a.predict(train_loader)
</code></pre>
<p>I am trying to get features from the pre-final layer of resnet18 but even after a single batch, the swap memory on my cpu keeps increasing till my laptop runs out of memory.</p>
<p>I am not using a GPU. I’m using a 16gb MacBook pro and have about 25 GB of free space.</p>

username: ptrblck
cooked: <p>I’ve just tested the code on a V100 + CUDA10.2 and after the forward pass with a batch size of 100 approx. 23GB GPU memory will be used, which would explain why your swap is being filled.<br>
You could reduce the batch size to save memory or use e.g. <code>torch.utils.checkpoint</code> to trade compute for memory.</p>
Set weights/bias of linear layer based on condition

username: wthrift
cooked: <p>I’d like to add the ability to ignore padding indexes to nn.Linear, what is the best way to manually set the weights/bias of an nn.Linear to 0 when a certain value is passed into the layer from the input features?</p>
<p>In my specific application, I have an embedding for protein sequences that maps numbers (tokenized amino acids) to vectors, but I need to map those vectors to the size of my model. It seems natural to use nn.Linear to receive the tensor of shape (sequence, embedding_dimension) and output to shape (sequence, model_dimension), but I need to ignore padding. Specifically, the embedded padding token would be this vector (-5, -5, -5,-5, -5, -5,-5, -5, -5) (the embedding dimension is 9), so I want to set the weights/bias of nn.Linear to 0 whenever it encounters a -5.</p>

username: ptrblck
cooked: <p>If you are setting the parameters to zero in the module, all training will be lost.<br>
Maybe the functional API approach would work, where you define your <code>weight</code> as an <code>nn.Parameter</code> and apply it via <code>F.linear</code>.<br>
This would allow you to use a condition and either use your <code>weight</code> or any other dummy tensor.</p>
<p>Also, I’m not sure, if zeroing out the gradients in the module wouldn’t also work for the padding inputs.</p>
How to define Forward method in network?

username: MLCharlotte
cooked: <p>I am trying to build an RNN that trains on YouTube comments. I am learning as I progress, I am struggling to correctly define the forward pass for my network. I have previously done a CNN but this current network is different and it is causing me confusion, I would appreciate any help!</p>
<p>Here is the network:</p>
<blockquote>
<p>class EncoderCNN(nn.Module):<br>
def <strong>init</strong>(self, embed_size):</p>
<pre><code>    super(EncoderCNN, self).__init__()
    resnet = models.resnet152(pretrained=True) 
    layers = list(resnet.children())[:-1]      
    self.resnet = nn.Sequential(*layers)
    
   
    self.linear = nn.Linear(resnet.fc.in_features, embed_size)
    
 
    self.bn = nn.BatchNorm1d(embed_size, momentum=0.01)
    
    def forward(self, images):
   

    
    
    features = self.linear(features)
    return features
</code></pre>
</blockquote>

username: ptrblck
cooked: <p>What causes your confusion and where are you stuck?<br>
Based on the code snippet it looks like you are trying to build a CNN again while you mentioned that you would like to create an RNN?</p>
How to set a fixed steps after which weights are updated, expect updating after every batch

username: prateekgupta891
cooked: <p>Image Size varies a lot. Hence kept the Batch size = 1.<br>
Now i need to propogate back the loss and do the optimizer step after every BACK_STEP no. of batches ( batch_size = 1).</p>
<pre><code class="lang-auto">model.train()
BACK_STEP = 128
for batch_idx, (inputs,labels) in enumerate(train_loader):
    inputs = inputs.to(device)
    labels = labels.to(device)

    if (batch_idx)%BACK_STEP==0:
        optimizer.zero_grad()

    with torch.set_grad_enabled(True):
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)
        #losses +=loss
        #print(labels,outputs,preds,loss)
        loss.backward()
        if (batch_idx+1)%BACK_STEP==0:
            optimizer.step()

    # statistics
    running_loss += loss.item() * inputs.size(0)
    running_corrects += torch.sum(preds == labels.data)
    
scheduler.step()
</code></pre>
<p>Is it the correct way to do it?<br>
Because I think loss needs to be added as well for all images passes and then propogated back.<br>
Any changes you would suggest?</p>

username: ptrblck
cooked: <p>The code looks alright and you don’t necessarily need to accumulate the loss, since the gradients will be automatically accumulated.<br>
<a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">Here</a> is a detailed description for different approaches.</p>
Filter tensors whose elements are zeros

username: muammar
cooked: <p>I have a tensor with the following dimensionality:</p>
<pre><code class="lang-auto">torch.Size([30, 5, 90])
</code></pre>
<p>Some of the tensors in the second dimension are just 90 zeros. For example:</p>
<pre><code class="lang-auto">mytensor[0][0]

tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>How can I drop those <code>(n, 90)</code> tensors whose all elements are zeros? I want to do this because in the forward propagation, even though those tensors are zeros the bias is added and the output is never zero. I would appreciate any help on this.</p>

username: jmandivarapu1
cooked: <p>I am not sure about the efficnt way. But one way to do this is create an empty tensor, Iterate the current tensor then stack the elements only which you want to the empty tenor using ```torch.stack``</p>

username: ptrblck
cooked: <p>This code snippet should work:</p>
<pre><code class="lang-python"># Setup
x = torch.randn(30, 5, 90)
x[:, 0] = 0.
x[:, 3] = 0.

# Filter
idx = (x != 0.).all(2).all(0)
y = x[:, idx]
print(y.shape)
&gt; torch.Size([30, 3, 90])
</code></pre>
How to modify filter gradients using register_hook having separate update function for each filter gradient

username: csailnadi
cooked: <p>I am wondering how I can use <code>register_hook</code> to modify filters gradients for the convolutional layer.</p>
<p>Say my convo layer has 10 filters, 3 channels, 5x5 --&gt; (10, 3, 5, 5).<br>
Each 32x32 filter should have its own gradient e.g G.<br>
I have the corresponding tensor of the size  (10, 3, 5, 5) where 5x5 matrix M is a matrix that I want to multiply filter’s channels gradient e.g MxG and use this as a gradient. Note M is different for each filter.</p>
<p>For fully connected network I simply did<br>
<code>model.layer.weight.register_hook(lambda grad: torch.t(torch.mm(M, torch.t(grad))))</code></p>
<p>How do I do it for convolutional filters?<br>
lambda grad will iterate over (3, 5, 5) tensors. How do I update my gradient in this case…?</p>

username: ptrblck
cooked: <p>If your already have a tensor in the shape <code>[10, 3, 5, 5]</code>, where the <code>5x5</code> tensors correspond to the mask, you can simply multiply the gradient with this tensor. There is no need to loop over each mask.</p>
Memory error Batch norm testing

username: sabyUWO
cooked: <p>Hi,<br>
I am getting a memory error during batch norm in the testing phase(model.eval()), changing the batch size (even as low as 1) doesn’t fix it; the training works fine. Can anyone kindly take a look?. The testing size is 64k. Be safe!</p>
<pre><code class="lang-auto">File "/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/tune/HyperTuneTwoStreamRPv1/DPPIPSSM128.py", line 514, in &lt;module&gt;
    main()
  File "/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/tune/HyperTuneTwoStreamRPv1/DPPIPSSM128.py", line 393, in main
    results = test(model.eval(),tst_loader,device)
  File "/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/tune/HyperTuneTwoStreamRPv1/DPPIPSSM128.py", line 439, in test
    output = net(crop0, crop1).cuda()
  File "/project/6034601/saby2k13/pyDPPI/pyDPPIv2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/tune/HyperTuneTwoStreamRPv1/DPPIPSSM128.py", line 267, in forward
    output2 = self.forward_once(input2)
  File "/home/saby2k13/projects/ctb-ilie/saby2k13/pyDPPI/tune/HyperTuneTwoStreamRPv1/DPPIPSSM128.py", line 214, in forward_once
    output = self.bn2(output)
  File "/project/6034601/saby2k13/pyDPPI/pyDPPIv2/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/project/6034601/saby2k13/pyDPPI/pyDPPIv2/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/project/6034601/saby2k13/pyDPPI/pyDPPIv2/lib/python3.6/site-packages/torch/nn/functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.75 GiB total capacity; 13.97 GiB already allocated; 3.50 MiB free; 733.67 MiB cached)
</code></pre>

username: ptrblck
cooked: <p>Could you check the memory usage on your GPU via <code>nvidia-smi</code> and make sure no other processes are using memory?<br>
Did you change anything else besides calling <code>model.eval()</code>, e.g. the spatial input shapes?</p>
Records in batches of DataLoader

username: sgaur
cooked: <p>How can we know the number of records in each batches of Dataloader</p>

username: jmandivarapu1
cooked: <p>u can enumrate the DataLoader then print <code>data.shape</code></p>

username: sgaur
cooked: <aside class="quote no-group" data-username="jmandivarapu1" data-post="2" data-topic="75289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmandivarapu1/48/3544_2.png" class="avatar"> jmandivarapu1:</div>
<blockquote>
<p>data.shape</p>
</blockquote>
</aside>
<p>Data loader Does not supports Indexing</p>

username: jmandivarapu1
cooked: <p>This might work?</p>
<pre><code class="lang-auto"> for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
           print(data.shape)
</code></pre>
Backprop from scratch - Computational Graph viewpoint

username: iamshnoo
cooked: <p>I am trying to write feed-forward and back propagation from a computational graph point of view in PyTorch as described <a href="http://cs231n.github.io/optimization-2/#staged" rel="nofollow noopener">in this CS231n page</a> for the MultiLabel Soft Margin Loss function of PyTorch. The feed-forward part, I have figured out and it is giving the same results as the loss function of PyTorch. However, for the backprop portion, I am having trouble.</p>
<pre><code class="lang-auto">x = torch.tensor([[ 0.1778,  0.1203, -0.2264]], requires_grad=True) # 1 Row, 3 columns (N=1,C=3) (Batch_size=1, num_classes=3)
y = torch.tensor([[0., 1., 0.]], requires_grad=True)

#logsigmoid(x) = torch.log(1/(1+torch.exp(-x)))
#loss = torch.mean(-(target * logsigmoid(x) + (1 - target) * logsigmoid(-x)),dim=1)


# forward pass
w = -1
l = torch.log(1/(1+torch.exp(-x)))                                                              
g = w * x                                           
a = y * l 
k = g + l                                            
h = k * y * w
b = h + k

### no change below this line
c = a + b                                            #(7)
d = -1 * c                                           #(8)
e = torch.sum(d,dim=1)                               #(9)
f = e / x.size(1)                                    #(10)

print(f) # tensor([0.6690], grad_fn=&lt;DivBackward0&gt;), same as value from actual function
f.backward()
print(x.grad.data) # tensor([[ 0.1814, -0.1567,  0.1479]])
print(y.grad.data) # tensor([[-0.0593, -0.0401,  0.0755]])

# back propagation steps from scratch 
dfde = 1 / x.size(1) 

dfdd = torch.ones(x.size(1)) * dfde

dfdc = w * dfdd

dfda = 1 * dfdc
dfdb = 1 * dfdc

### no change above this line

dfdh = 1 * dfdb
dfdk = 1 * dfdb             # ---
                            #    | ----&gt; df/dk is being repeated. This is a problem
dfdk = y * w * dfdh         # --- 
dfdy = k * w * dfdh

dfdg = 1 * dfdk
dfdl = 1 * dfdk            # ---
                           #    | ----&gt; df/dl is being repeated. This is a problem. 
dfdl = y * dfda            # ---
dfdy = l * dfda            #### df/dy = logsigmoid(x) * (-1) * torch.ones(x.size(1)) * 1/x.size(1) -----&gt; This should be correct. But doesn't match the y.grad.data

dfdw = x * dfdg
dfdx = w * dfdg            #---
                           #   | -----&gt; df/dx is begin repeated. This is a problem.
dfdx = (dl/dx) * dfdl      #--- 
</code></pre>
<p>So, I have mentioned in the comments in the code, where I am currently facing problems. I just don’t know how to go about resolving them.<br>
I just need to find df/dx and df/dy and check whether they are equal to x.grad.data and y.grad.data<br>
The reason being that, I am interested to know what functions df/dx and df/dy actually are.</p>
<p>Can someone please help me out with this particular task?<br>
And if possible, I would also like some general advice on how to overcome such beginner problems in the future.</p>
Glow: Compiling a model with varying size input

username: antimora
cooked: <p>Does the glow’s bundle compiler support input with varying size? Like varying image size input?</p>

username: jfix
cooked: <p>I think this depends on the model. E.g. I’ve seen C2 models where the shapes of the network are inferred based on the input size, so you have to specify that at least when compiling. But other models may have specific shape expectations in the model and so aren’t flexible, you’d hit an error if you try different image sizes that are not compatible.</p>
Reading h5py files into tensors

username: Aliman
cooked: <p>So I have a training set and a test set both in h5py format. I also have a data_load function that loads the files and returns NumPy arrays. The main problem is I don’t need NumPy as I am working with Tensors. I am expecting to have an x&amp;y tensor of size N(batch size) and D_in(input size for each image) and D_out(Output size of each tensor).</p>
<p>The problem:</p>
<p>x&amp;y do not get converted to tensors of dimensions mentioned below.If anything their types remain to be numpy.ndarray. Any help is appreciated.</p>
<pre><code class="lang-auto">
</code></pre>
<pre><code class="lang-auto"># Load the training data

train_dataset =h5py.File(train_file, 'r')


# Separate features(x) and labels(y) for training set
train_set_x_orig =np.array(train_dataset["train_set_x"][:])
train_set_y_orig =np.array(train_dataset["train_set_y"][:])

# Load the test data
test_dataset =h5py.File(test_file,'r') 

# Separate features(x) and labels(y) for training set
test_set_x_orig =np.array(test_dataset["test_set_x"][:]) 
test_set_y_orig =np.array(test_dataset["test_set_y"][:])

classes = np.array(test_dataset["list_classes"][:]) # the list of classes

train_set_y_orig = torch.from_numpy(train_set_y_orig.reshape((1, train_set_y_orig.shape[0])))
test_set_y_orig = torch.from_numpy(test_set_y_orig.reshape((1, test_set_y_orig.shape[0])))

return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes






x = torch.Tensor(N, D_in)
y = torch.Tensor(N, D_out)
train_file="data/train_catvnoncat.h5"
test_file="data/test_catvnoncat.h5"
x,y,_,_,_=load_data(train_file,test_file)
</code></pre>
<pre><code class="lang-auto">type or paste code here
</code></pre>

username: ptrblck
cooked: <p>I’m not sure, if I’m missing something, but it seems you are not converting <code>train_set_x_orig</code> and <code>test_set_x_orig</code> to tensors, but instead just their <code>_y_</code> equivalent? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Importing PyTorch after OpenCV messes with CPU affinity

username: sun_stealer
cooked: <p>This is on the extreme end of weird software issues I ever had to deal with. Looking for help!</p>
<p>Consider this Python script:</p>
<pre><code class="lang-auto">import psutil
import cv2


def run():
    curr_process = psutil.Process()
    curr_process.cpu_affinity([3])
    print('Before', curr_process.cpu_affinity())
    import torch
    print('After', curr_process.cpu_affinity())


import multiprocessing
multiprocessing.Process(target=run).start()
</code></pre>
<p>You would think that importing PyTorch has nothing to do with CPU affinity and the output before and after would be the same, but no. The output of this script on my machine is:</p>
<pre><code class="lang-auto">Before [3]
After [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
</code></pre>
<p>The issue, of course, arose in a big application and it took considerable effort to boil it down to this minimal example. I found that reproducing it requires a key condition: PyTorch and OpenCV should be imported from different processes. One before, and one after the fork. It does not reproduce if I import both in the child process, or in the main process, but it does reproduce if I swap <code>cv2</code> and <code>torch</code> in this script.</p>
<p>First of all, what does importing the library has to do with the CPU affinity? Perhaps this is some kind of optimization attempt by the authors of the library that went rogue?</p>
<p>Then, how do I fix something like this? I could, of course, import everything before fork as a quick hacky solution, but I’d like to get to the bottom of things.</p>
<p>My versions of libraries (both installed from Conda repo):</p>
<ul>
<li><code>opencv=4.1.0=py37h3aa1047_6</code></li>
<li><code>pytorch=1.4.0=py3.7_cuda10.1.243_cudnn7.6.3_0</code></li>
</ul>
<p>Also posted on SO: <a href="https://stackoverflow.com/questions/60963839/importing-opencv-after-importing-pytorch-messes-with-cpu-affinity" rel="nofollow noopener">https://stackoverflow.com/questions/60963839/importing-opencv-after-importing-pytorch-messes-with-cpu-affinity</a></p>

username: ptrblck
cooked: <p>Could you please create an issue on GitHub, please?</p>
Split images based on patient folders for cross validation

username: eyk
cooked: <p>Hi,<br>
I need to split my images to validation and train set in cross validation, in a way that images of each patient be either in train or validation. each patient has about 8-11 images.<br>
root<br>
|___<strong>patien_1</strong> im1<br>
|                      | __ im2 …<br>
|<br>
|___<strong>patient_2</strong> im1<br>
|                       | __ im2 …<br>
|<br>
.<br>
.<br>
.</p>
<p>how I can split images based on patient folders?</p>

username: ptrblck
cooked: <p>If you have all target values for all images, you could create a corresponding group for each patient and use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupShuffleSplit.html">sklearn.model_selection.GroupShuffleSplit</a>.</p>

username: eyk
cooked: <p>this is a binary classification task [0,1]. but how I can make the group of patients?</p>

username: ptrblck
cooked: <p>Each patient would get an index (similar to a target) and you could pass it to the <code>GroupShuffleSplit</code> to get the indices for the train and validation split based on the patients.</p>
<p>E.g. if patient_1 has 1 image and patient_2 2 images, you could pass the groups as: <code>[0, 1, 1]</code>.</p>

username: eyk
cooked: <p>Thanks for explanation!<br>
I have 300 patients, this means that my group range should be 0-299? or I need to make the groups based on target 0-1 ?</p>

username: ptrblck
cooked: <p>Each patient should belong to a unique group, so the range should be <code>[0, 299]</code>.</p>
PyTorch LSTM Issues

username: Daniel_de_Marchi
cooked: <p>Alright. So I have a training set currently in a numpy array of dimension [50573, 322] for the X values and [50573, 126] for Y. To preprocess my data I am calling…</p>
<p>y_train = y_train.astype(float)<br>
y_test = y_test.astype(float)<br>
scaler = StandardScaler()<br>
X_train = scaler.fit_transform(X_train)<br>
X_test = scaler.transform(X_test)</p>
<p>X_train = np.reshape(X_train, (len(X_train), end-1, 1))<br>
X_test = np.reshape(X_test, (len(X_test), end-1, 1))</p>
<p>X_train = Variable(torch.Tensor(torch.from_numpy(X_train).float()))<br>
y_train = Variable(torch.Tensor(torch.from_numpy(y_train).float()))</p>
<p>For my model I am calling…</p>
<p>class LSTMNET(nn.Module):</p>
<pre><code>def __init__(self, input_dim, hidden_dim, batch_size, output_dim=1,
                num_layers=2):
    super(LSTMNET, self).__init__() 
    self.input_dim = input_dim
    self.hidden_dim = hidden_dim
    self.batch_size = batch_size
    self.num_layers = num_layers

    # Define the LSTM layer
    self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)

    # Define the output layer
    self.linear = nn.Linear(self.hidden_dim, output_dim)

def init_hidden(self):
    # This is what we'll initialise our hidden state as
    return (torch.zeros(self.num_layers, self.batch_size, self.hidden_dim),
            torch.zeros(self.num_layers, self.batch_size, self.hidden_dim))

def forward(self, input):
    input = torch.Tensor(input)
    # Forward pass through LSTM layer
    # shape of lstm_out: [input_size, batch_size, hidden_dim]
    # shape of self.hidden: (a, b), where a and b both 
    # have shape (num_layers, batch_size, hidden_dim).
    print(input.shape)
    lstm_out, self.hidden = self.lstm(input.view(len(input), self.batch_size, -1))
    
    # Only take the output from the final timetep
    # Can pass on the entirety of lstm_out to the next layer if it is a seq2seq prediction
    y_pred = self.linear(lstm_out[-1].view(self.batch_size, -1))
    return y_pred.view(-1)
</code></pre>
<p>Then…</p>
<p>lstm = LSTMNET(1, 1, batch_size=batch_size, output_dim=126, num_layers=1)</p>
<p>All I want to do is train this. But something simple like…</p>
<p>predicted = lstm(X_train)</p>
<p>Filled with errors. Can’t figure out how this is supposed to work. Please advise.</p>
How to make mask from given lengths for nn.MultiheadAttention input?

username: hadaev8
cooked: <p><a href="https://pytorch.org/docs/master/nn.html?highlight=multiheadattention#torch.nn.MultiheadAttention" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/nn.html?highlight=multiheadattention#torch.nn.MultiheadAttention</a></p>
<p>nn.MultiheadAttention accept key_padding_mask and attn_mask</p>
<p>its easy to make  key_padding_mask like this</p>
<pre><code>def get_mask_from_lengths(lengths):
	max_len = torch.max(lengths).item()
	ids = torch.arange(0, max_len, out=torch.cuda.LongTensor(max_len))
	mask = (ids &lt; lengths.unsqueeze(1))
	return ~mask
</code></pre>
<p>But this work only on key value like this</p>
<p>                    <a href="https://i.imgur.com/Y9RPraD.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/Y9RPraD.png" width="364" height="250">
          </a>

</p>
<p>Or like this</p>
<p>                    <a href="https://i.imgur.com/GqL0YTr.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/GqL0YTr.png" width="364" height="252">
          </a>

</p>
<p>How to make attn mask of size (batch_size*nheads, lens_0, lens_1) for masking both query and key values?</p>

username: hadaev8
cooked: <p>Since only -inf values in axis case nan values in softmax output, its not obvious for me how to perform query len padding.</p>
Why computing time increase with backpropagation hooks?

username: csailnadi
cooked: <p>I was waiting a ton of time for an epoch to complete, but failed to get to that point.<br>
When I measured the time taken for each iteration it seems like time taken increases all the time.<br>
It started from 0.0104 and after some seconds reached 0.103. Why does time keep increasing? Why is it so slow?</p>
<pre><code class="lang-auto">
            start = time.time()
            optimizer.zero_grad()
            loss.backward()

            with torch.no_grad():
                model.forward(K) #dummy pass
                M1, M2, M3 = get_Ms()

                model.linear1.weight.retain_grad()
                model.linear1.weight.register_hook(lambda grad: torch.t(torch.mm(M1, torch.t(grad))))

                model.linear2.weight.retain_grad()
                model.linear2.weight.register_hook(lambda grad: torch.t(torch.mm(M2, torch.t(grad))))

                model.linear3.weight.retain_grad()
                model.linear3.weight.register_hook(lambda grad: torch.t(torch.mm(M3, torch.t(grad))))

            optimizer.step()

            end = time.time()
            time_taken = end - start
            print("time_taken=",time_taken)
</code></pre>
<p>I wrote this code as a replacement for the code below, which runs just fine (not fast, but reasonable). I expected to use hooks to use native optim backprop to speed up the computation, but instead the time completely exploded. Why so?</p>
<pre><code class="lang-auto">with torch.no_grad():
                    model.forward(K)
                    M1, M2, M3 = get_Ms()

                    model.linear1.weight -= lr * torch.t(torch.mm(M1, torch.t(model.linear1.weight.grad)))
                    model.linear2.weight -= lr * torch.t(torch.mm(M2, torch.t(model.linear2.weight.grad)))
                    model.linear3.weight -= lr * torch.t(torch.mm(M3, torch.t(model.linear3.weight.grad)))

                    model.linear1.bias -= lr * model.linear1.bias.grad
                    model.linear2.bias -= lr * model.linear2.bias.grad
                    model.linear3.bias -= lr * model.linear3.bias.grad
</code></pre>

username: csailnadi
cooked: <p>I found the answer myself. Maybe it will help someone.<br>
After creating a hook it should be removed or it will pile up.<br>
See <a href="https://discuss.pytorch.org/t/why-not-removing-a-register-hook-slows-down-the-training-gradually/63168">here</a></p>
<p>So after <code>optimizer.step()</code><br>
I call <code>h1.remove()</code> where h1 is the returned value of <code>register_hook()</code> function.</p>
How can i run on the second GPU not first one

username: Abdelrahman_Elsayed
cooked: <p>hello,<br>
I have 4 GPU,  how can I train my model in the second GPU not the first cuz the first I am using it to train another model<br>
RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 5.56 GiB total capacity; 43.77 MiB already allocated; 10.19 MiB free; 52.00 MiB reserved in total by PyTorch)</p>
<p>so I need to train in 2 , 3 , 4 GPU not the first is that possible?</p>

username: bluesky314
cooked: <p>If you are using the terminal you can tell it which cuda devices it can access with the device number from nvidia-smi. The first gpu is named indexed with 0. For eg to run on your second gpu: <code>CUDA_VISIBLE_DEVICES=1 python densenet161.py</code></p>

username: Abdelrahman_Elsayed
cooked: <p>nvcc: NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2019 NVIDIA Corporation<br>
Built on Fri_Feb__8_19:08:26_Pacific_Standard_Time_2019<br>
Cuda compilation tools, release 10.1, V10.1.105</p>

username: bluesky314
cooked: <p>This is not relevant.</p>

username: Abdelrahman_Elsayed
cooked: <p>what do u mean with that is not relevant ?</p>

username: bluesky314
cooked: <p>Just run CUDA_VISIBLE_DEVICES=gpu_number python file.py</p>

username: Abdelrahman_Elsayed
cooked: <p>CUDA_VISIBLE_DEVICES=1 can i use it in code cuz i cant in terminal<br>
iam using window 10</p>
<p>CUDA_VISIBLE_DEVICES’ is not recognized as an internal or external command,<br>
operable program or batch file.</p>

username: peterjc123
cooked: <pre><code class="lang-auto">set CUDA_VISIBLE_DEVICES=1
python xxx.py
</code></pre>
What is the purpose of serialization?

username: mfcs
cooked: <p>Hi everyone!!<br>
I didn’t understand yet the purpose of the serialization process.<br>
What reasons we have to execute a serialization process and what is the purpose to serialize a tensor or<br>
a model in PyTorch?</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

username: richard
cooked: <p>Serialization can be used for the following:</p>
<ul>
<li>checkpointing: saving the current weights of the model in the middle of training so that if your current training process gets terminated unexpectedly you don’t have to retrain from the start</li>
<li>loading a trained model for inference: if you have trained a model, you’ll probably want to save the weights so that the model can be used in the future. torch.save and torch.load give a way to do that.</li>
</ul>

username: mfcs
cooked: <p>Ok I understood this and make sense, thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>But what is the advantages?<br>
Couldn’t the model be saved in another format?<br>
I would like to understand the advantages of performing serialization, what kind of representation serialization provides that it is necessary to perform.</p>

username: mfcs
cooked: <p>Is it a way to represent the data inside tensors or weights of a net?</p>
VGGnet and DenseNet training time

username: davidlee
cooked: <p>Hi,</p>
<p>I am wondering why DenseNet training time takes way way more than VGGnet,<br>
even though the number of params for DenseNet is much less than the one of VGGNet.<br>
Params for DenseNet : 0.7M<br>
Params for VGGNet : 17M</p>
<p>Thanks!</p>

username: jmandivarapu1
cooked: <aside class="quote no-group" data-username="davidlee" data-post="1" data-topic="75166">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davidlee/48/12812_2.png" class="avatar"> davidlee:</div>
<blockquote>
<p>I am wondering why DenseNet training time takes way way more than VGGnet,<br>
even though the number of params for DenseNet is much less than the one of VGGNet.<br>
Params for DenseNet : 0.7M<br>
Params for VGGNet : 17M</p>
</blockquote>
</aside>
<p>I know this is a open ended question. But here is some of my thoughts.</p>
<p>Training time depends on lot of things</p>
<ol>
<li>
<p>Hope you are using same setup for both the networks like learning rate and batch sizes and all. Because each net has it’s own recipe to reach the convergence state</p>
</li>
<li>
<p>Sometime may the net you are training is underfitting the data so then it takes more time in this case DenseNet</p>
</li>
<li>
<p>Same as above but overfitting in the case of VGGNET. You can test this by checking their accuracies on the training set.</p>
</li>
</ol>

username: davidlee
cooked: <p>Hi jmandivarapu1,</p>
<p>Thank you for your answer, but<br>
I wasn’t trying to say the convergence or performance aspects.<br>
I just wonder the pure computation time for training between two networks!<br>
I am using exact same environment to those networks.</p>

username: jmandivarapu1
cooked: <p>Then my guess would be. It should take more time for computation of the backward graph for VGG than the dense net. If the rest of the environment variables are the same.</p>
Unable to solve these erors

username: AntuAnant
cooked: <p>My code:</p>
<p>class LSTMModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(LSTMModel, self).<strong>init</strong>()<br>
self.LSTM = nn.LSTM(input_size=input_size, hidden_size=hidden_size,<br>
batch_first=True)</p>
<pre><code>def forward(self ,x, hidden, cell):
    x = x.view(batch_size, sequence_length, input_size)
    out, (hidden, cell)= self.LSTM(x, hidden, cell)
    out= out.view(-1, num_classes)
    return out,hidden,cell
def init_hidden(self):
    return Variable(torch.zeros(num_layers, batch_size, hidden_size))
</code></pre>
<p>model = LSTMModel()<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.Adam(model.parameters(), lr=0.1)</p>
<p>for epoch in range(100):<br>
optimizer.zero_grad()<br>
loss = 0<br>
hidden = model.init_hidden()<br>
cell = model.init_hidden()<br>
print(“Predicted String”)<br>
for ins, label in zip(inputs,labels):<br>
output,hidden,cell = model(ins,hidden,cell)<br>
val, idx = output.max(1)<br>
print(idx2char[idx.data[0]])<br>
loss+=criterion(output, label.unsqueeze(0))</p>
<pre><code>print(", epoch: %d, loss: %1.3f" % (epoch+1, loss.data[0]))

loss.backward()
optimizer.step()
</code></pre>
<p>Predicted String<br>
Traceback (most recent call last):</p>
<p>File “”, line 1, in <br>
runfile(‘E:/Python Deep Learnig Projects/Code/Section_3/seqtest.py’, wdir=‘E:/Python Deep Learnig Projects/Code/Section_3’)</p>
<p>File “C:\Users\admin\Miniconda3\envs\tensorflow\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 827, in runfile<br>
execfile(filename, namespace)</p>
<p>File “C:\Users\admin\Miniconda3\envs\tensorflow\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 110, in execfile<br>
exec(compile(f.read(), filename, ‘exec’), namespace)</p>
<p>File “E:/Python Deep Learnig Projects/Code/Section_3/seqtest.py”, line 62, in <br>
output,hidden,cell = model(ins,hidden,cell)</p>
<p>File “C:\Users\admin\Miniconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)</p>
<p>File “E:/Python Deep Learnig Projects/Code/Section_3/seqtest.py”, line 45, in forward<br>
out, (hidden, cell)= self.LSTM(x, hidden, cell)</p>
<p>File “C:\Users\admin\Miniconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)</p>
<p>TypeError: forward() takes from 2 to 3 positional arguments but 4 were given</p>

username: b02202050
cooked: <aside class="quote no-group" data-username="AntuAnant" data-post="1" data-topic="75237">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/antuanant/48/12756_2.png" class="avatar"> AntuAnant:</div>
<blockquote>
<p>out, (hidden, cell)= self.LSTM(x, hidden, cell)</p>
</blockquote>
</aside>
<p>Modify this line to</p>
<pre data-code-wrap="python"><code class="lang-python">out, (hidden, cell)= self.LSTM(x, (hidden, cell))
</code></pre>