akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>

Himanshu_Singh: <p>in dataloader files, it’s showing error.<br>
Here is the code</p>
<p>class CDataset(Dataset):</p>
<pre><code>def __init__(self, dataset, img_dir, df, transform=None): 
    self.dataset = dataset
    self.img_dir = img_dir
    self.df=df
    self.transform = transforms.Compose(transform) if transform else None
    
def __len__(self): 
    return len(self.dataset)

def return_image_mask(self, file):
    ds=dcmread(self.img_dir+file)
    arr=ds.pixel_array
    pat_id=file[:-4]
    if self.df[self.df['patientId']==pat_id].iloc[0]['Target']==0:
        mask=None
    else:
        mask=np.zeros(arr.shape,dtype='int')
        temp=self.df[self.df['patientId']==pat_id]
        for i in range(len(temp)):
            x=temp.iloc[i]['x']
            y=temp.iloc[i]['y']
            w=temp.iloc[i]['width']
            h=temp.iloc[i]['height']
            cv2.rectangle(mask, (int(x), int(y)), (int(x+w), int(y+h)),255,-1)
    return arr,mask
    
def __getitem__(self, i):
    img,mask = self.return_image_mask(self.dataset[i])
    if self.transform:
        img = self.transform(img)
        if mask is not None:
            mask = self.transform(mask)
    temp = self.df[self.df['patientId'] == self.dataset[i][:-4]]
    return img,mask,temp.iloc[0].Target
</code></pre>
<p>transform = [transforms.ToPILImage(),transforms.Resize(448), transforms.ToTensor()]<br>
train_data=CDataset(train, img_folder,train_df,transform)<br>
test_data=CDataset(test, img_folder,train_df,transform)</p>
<p>train_dl = DataLoader(train_data, batch_size=64)<br>
test_dl = DataLoader(test_data, batch_size=64)</p>

JuanFMontesinos: <p>The dataloader has a function which generates batches of elements from the output of getitem.<br>
As the error says, the outputs of that function are rectricted to those.</p>

Himanshu_Singh: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="110507">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>has a function which generates batches of elements</p>
</blockquote>
</aside>
<p>But I don’t see any restrictions as such</p>

JuanFMontesinos: <p>Default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘NoneType’&gt;</p>
<p>At some point you are either returning a mask, which is a np array, and None.<br>
When the system tries to make a batch it fails (cannot compose arrays with None)</p>
<p>The simplest case is you to write your own collate function to return a list of elements instead of trying to batchify your output “mask”. Or return a mask of -1 values or something meaningful to you</p>
Question related to model.train()

ashwani-bhat: <p>I know model.train() is called at the start of every epoch. But can I call model.train() after every batch. My code is working when I called it after every batch.  I just wanted to know if there are any downsides to it.</p>
<p>Something like this:</p>
<pre><code class="lang-auto">for every epoch:
    for every batch:
         model.train()
         # some lines of training code
</code></pre>

ptrblck: <p>You could call it after each batch, but this would not be necessary, if you haven’t called <code>model.eval()</code> between these calls.<br>
Note that <code>model.train()</code> changes the internal <code>training</code> flag, which then changes the behavior of some modules (e.g. dropout will be disabled during evaluation).</p>

ashwani-bhat: <p>Thanks. Yes I did call model.eval() after each batch, that’s why I asked. Thanks for the solution.</p>
How do I make a residual net with noise not in place?

pytorchkiw: <p>Hi, each layer should be calculated from the last layer as<br>
next_layer = last_layer + Relu( Linear( last_layer ) ) + noise<br>
Each layer has the same input dim as output dim.<br>
This is what I did</p>
<pre><code class="lang-auto">class MyNN(nn.Module)
    def __init__(self, space_dim):
        super(MyNN, self).__init__()
        self.layers = nn.ModuleList()
        for i in range(self.path_length-1):
            linear_layer = nn.Linear(space_dim, space_dim)
            self.layers.append(nn.Sequential(linear_layer, nn.LeakyReLU(0.2, inplace=True)))

    def forward(self, z):
        for i in range(self.path_length-1):
            z += self.layers[i](z)
            noise = (self.noisefactor * torch.normal(0,1,z.size())).detach()
            z += noise
        return z
</code></pre>
<p>I am getting <code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1000, 1]], which is output 0 of AddBackward0,</code>. In the stacktrace this happens when executing a linear layer. I guess that is because I am not properly saving the inbetween values of <code>z</code>. Should I just throw them into some list? Or preallocate a tensor where I save them to?</p>

ptrblck: <p>Could you remove the inplace operation and check, if this would solve the error?</p>
<pre><code class="lang-python">    def forward(self, z):
        for i in range(self.path_length-1):
            z = z + self.layers[i](z)
            noise = (self.noisefactor * torch.normal(0,1,z.size())).detach()
            z = z + noise
        return z
</code></pre>
Torchvision vgg16 pretrained model layer naming

guangye: <p>Hello,</p>
<p>I am curious about the layer naming (key values of state_dict) of the vgg16 pretrained model from torchvision.models module, e.g.: ‘features.0.weight’, ‘features.0.bias’, ‘features.2.weight’, ‘features.2.bias’, etc. The number increases by 2 with each new convolutional or fc layer, and increases by 3 each time encountering a max pooling layer.</p>
<p>Do you have an idea of the underlying logic?</p>
<p>Thank you very much!</p>
<p>Guangye</p>

ptrblck: <p>The numbered indices in these modules names are created by the <code>nn.Sequential</code> module.<br>
Layers without any parameters will still get the index, but won’t be shown in the <code>state_dict</code>.<br>
Here is a small example:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Sigmoid(),
    nn.Linear(1, 1))
print(model.state_dict().keys())
&gt; odict_keys(['0.weight', '0.bias', '2.weight', '2.bias', '5.weight', '5.bias'])
</code></pre>
<p>As you can see, only the layer indices 0, 2, and 5 are shown, as these layers contain parameters.</p>
Applying residual links to a simple FFNN

spring: <p>Hi,<br>
I want to apply the residual link used in ResNet(<a href="https://arxiv.org/abs/1512.03385" class="inline-onebox" rel="noopener nofollow ugc">[1512.03385] Deep Residual Learning for Image Recognition</a>) to a simple model. Can I just add the input x and the output value passed through the linear layer in the element-wise direction as shown below?<br>
Is there anything else I need to do more?</p>
<pre><code>class neuralNet(torch.nn.Module):

    def __init__(self, input_size, hidden_size, num_classes):
        super(neuralNet, self).__init__()
        self.linear1 = torch.nn.Linear(input_size, hidden_size)
        self.relu1 = nn.ReLU()
        self.linear2 = torch.nn.Linear(hidden_size, hidden_size)
        self.relu2 = nn.ReLU()

        self.linear3 = torch.nn.Linear(hidden_size, hidden_size)
        self.relu3 = nn.ReLU()
        self.linear4 = torch.nn.Linear(hidden_size, output_size)
        self.relu4 = nn.ReLU()


    def forward(self, x):
        out = self.linear1(x)
        out = self.relu1(out)
        out = self.linear2(out)
        out = out.add(x)            # H(x) = F(x) + x
        out = self.relu2(out)

        out = self.linear3(out)
        out = self.relu3(out)
        out = self.linear4(out)
        out = out.add(x)            # H(x) = F(x) + x
        out = self.relu4(out)

        return out
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba6fb8d8657ab8b45c6a7b1c527a54e1e41e20d3.png" alt="image" data-base62-sha1="qBiaIBVJOMr6fpMsmSIDF1nWsh5" width="486" height="277"></p>

ptrblck: <p>Seems to be a double post from <a href="https://discuss.pytorch.org/t/how-to-apply-residual-links-to-simple-ffnn/109154/2">here</a> with a potential answer.</p>
Tell Pytorch's Autograd not to include a operation for gradient computation

hrushi: <p>I have a custom layer. Let the layer be called ‘Gaussian’</p>
<pre><code>class Gaussian(nn.Module):
  def __init__():  
    super(Gaussian, self).__init__()
 
 #@torch.no_grad     
  def forward(self, x):
    x  = x + torch.rand(x.shape) 
    x[2:]  = x[2:] + x[1] # Some similar operations are also performed
    return x

cnn_model = nn.Sequential(nn.Conv2d(1, 32, 5), Gaussian(), nn.ReLU())
</code></pre>
<p>If <code>a</code> is the input, I want gradient of <code>a</code> to exclude operations that are present in the <code>Gaussian</code> module, but include the calculations in other layers of the neural network(nn.Conv2d etc).</p>
<p>In the end, my aim is to use the Gaussian module to perform calculations but that calculations should not be included in gradient computation.</p>
<p>I tried to do the following:</p>
<ol>
<li>
<p>Used the <code>@torch.no_grad</code> above the forward method of the <code>Gaussian</code></p>
</li>
<li>
<p>Using detach after every operation in the Gaussian module: <code>x  = x + torch.rand(x.shape) </code> and similarly for other operations</p>
</li>
</ol>
<p>Are the above methods correct?</p>

ptrblck: <p>Both approaches would break the computation graph and the previous layers wouldn’t get a valid gradient.<br>
You could try to adapt <a href="https://discuss.pytorch.org/t/relu-with-leaky-derivative/32818/2">this workflow</a> (or write a custom <code>autograd.Function</code> and define the <code>backward</code> method manually).</p>

hrushi: <p>The modification helped me, thank you</p>
Chose best segmentation model based on dice or cross entropy loss?

LanceVance: <p>Hello everyone,<br>
I am working on a multiclass segmentation problem using a Unet based approach. As loss function I use cross entropy, but for validation purposes dice and IoU are calculated too. Currently, the weights are stored (and overwritten) after each epoch.  Therefore, I end up with the weights of the last epoch, which are not necessarily the best. Now I want to implement that the best model is saved.</p>
<p>My question now is, on what value should the decision for the best model be based? The loss, which is used to train the model, or the dice which is the metric I want to maximize. Is there a universal answer to this, or does it differ from problem to problem?</p>

ptrblck: <p>I’m not sure if there is a universal answer to this, but usually you would use the lowest validation loss or the best validation metric (dice score in your case).<br>
These two values are often correlated, but you might also see e.g. a decreasing loss while the accuracy doesn’t change much, which could be the case if the model increases it’s predictions for already correctly classified samples.</p>
RuntimeError: Trying to create tensor with negative dimension -256

_TV: <p>Hi.</p>
<p>in torch 1.7.0 , I can’t debug this error.</p>
<pre><code class="lang-auto">class DQN(nn.Module):
    
    def __init__(self, num_states, num_actions):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3,8,kernel_size=3,stride=2)
        self.bn1 = nn.BatchNorm2d(8)
        self.conv2 = nn.Conv2d(8, 16, kernel_size=3, stride=2)
        self.bn2 = nn.BatchNorm2d(16)
        self.conv3 = nn.Conv2d(16, 32, kernel_size=3, stride=2)
        self.bn3 = nn.BatchNorm2d(32)

        def conv2d_size_out(size, kernel_size = 3, stride = 1):
            return (size - (kernel_size - 1) - 1) // stride  + 1
        convh = conv2d_size_out(conv2d_size_out(conv2d_size_out(2)))
        convw = conv2d_size_out(conv2d_size_out(conv2d_size_out(num_states)))
        linear_input_size = convw * convh * 32
        self.head = nn.Linear(linear_input_size, num_actions)
        
    
    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))
</code></pre>
<p>input : [[num1,num2,num3,num4,num5],[num6,num7,num8,num9,num10]]</p>

ptrblck: <p><code>convh</code> is negative in your example, which would thus try to create the linear layer with a negative number of features.<br>
Make sure <code>conv2d_size_out</code> yields a positive value, which can later be used.</p>
Find the matrix multiplication of every matrix with another in a batch

iamshnik: <p>I have an input of the shape:</p>
<p>N * F * C * W * W,</p>
<p>which is basically F feature maps for a single example in a batch. I want the matrix product of each feature map with another in a single example. The output dimension should be</p>
<p>N * F * F * C * W * W</p>
<p>using <code>torch.matmul</code> I cannot model the interaction between every pair of feature maps in an example and the output would be N * F * C * W * W.</p>
<p>I can use a loop to do so, but that would be inefficient, is there any other way to do so?</p>

ptrblck: <p>I’m not sure if I misunderstand the use case, but wouldn’t this <code>matmul</code> work?</p>
<pre><code class="lang-python">N, F, C, W = 2, 3, 4, 5
x = torch.randn(N, F, C, W, W)
y = torch.matmul(x.unsqueeze(2), x.unsqueeze(1))
print(y.shape)
&gt; torch.Size([2, 3, 3, 4, 5, 5])
</code></pre>
How to balance data in PyTorch DataLoader

SarahTeoh: <p>Hi, I currently have train data that is imbalanced.<br>
Distribution of the train data:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df9e1216956fa5a3b5a2c08b298ec0b84224143e.png" data-download-href="https://discuss.pytorch.org/uploads/default/df9e1216956fa5a3b5a2c08b298ec0b84224143e" title="Screen Shot 2021-01-16 at 21.49.06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9e1216956fa5a3b5a2c08b298ec0b84224143e_2_690x208.png" alt="Screen Shot 2021-01-16 at 21.49.06" data-base62-sha1="vUd8IY9AIzCbw01t07xLSx4C9K6" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9e1216956fa5a3b5a2c08b298ec0b84224143e_2_690x208.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9e1216956fa5a3b5a2c08b298ec0b84224143e_2_1035x312.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9e1216956fa5a3b5a2c08b298ec0b84224143e_2_1380x416.png 2x" data-dominant-color="121716"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-01-16 at 21.49.06</span><span class="informations">1718×520 47.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to adjust the data so that every range has at least 50 samples.<br>
For example, 0~0.25 has 50 samples, 0.25~0.5 has 50 samples and so on.</p>
<p>How can I do that?<br>
I know PyTorch DataLoader has BatchSampler that can be used to sample an equal number of samples from each class, but the sampler uses class labels while my data is not class label.</p>

ptrblck: <p>You could use a <code>WeightedRandomSampler</code> and try to adapt <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this example</a> for your use case.<br>
Based on the figure it seems you are working on a regression task, so you would need to create bins first before calculating the “class weights” (which would be bin weights in your case).</p>
Convert tensor of floats to tensor of ordinal data

durden: <p>Hi,</p>
<p>Let’s say I have a tensor with shape (32,), e.g.</p>
<pre><code class="lang-auto">X = [ 0.0379, -0.0372,  0.0277, -0.1918, -0.0679,  0.0858,  0.0655,  0.2807,
     -0.1375, -0.0066,  0.1309,  0.0893,  0.0757,  0.1891,  0.2998, -0.1810,
     -0.0809,  0.1543, -0.0852, -0.1351,  0.0900, -0.1985, -0.0525,  0.0582,
      0.0410,  0.3986,  0.3757,  0.2447, -0.3319, -0.2319, -0.0495, -0.0034]
</code></pre>
<p>I’m seeking how to convert X into a (32, 100) tensor where 100 is the dimension of a one-hot encoding.<br>
I know that values range between [-1, 1].<br>
I would like to discretise this interval in 100 bins and place a 1 in the bin the value corresponds to, and 0 elsewhere.<br>
Is it possible to do this efficiently with pytorch?</p>
<p>Thanks</p>

ptrblck: <p>You could use <code>np.digitize</code> to create the indices of the bins and then use the <code>one_hot</code> method as seen here:</p>
<pre><code class="lang-python">x = np.array([ 0.0379, -0.0372,  0.0277, -0.1918, -0.0679,  0.0858,  0.0655,  0.2807,
              -0.1375, -0.0066,  0.1309,  0.0893,  0.0757,  0.1891,  0.2998, -0.1810,
              -0.0809,  0.1543, -0.0852, -0.1351,  0.0900, -0.1985, -0.0525,  0.0582,
              0.0410,  0.3986,  0.3757,  0.2447, -0.3319, -0.2319, -0.0495, -0.0034])

bins = np.linspace(x.min(), x.max(), 100)
y = np.digitize(x, bins) - 1
y = torch.from_numpy(y)
one_hot = torch.nn.functional.one_hot(y, num_classes=100)
</code></pre>
Computing loss along batch dimension

mohit117: <p>I have tensors of BxCHxW where B is the batch. Then are the following codes equivalent?</p>
<pre><code class="lang-auto">l1_loss = torch.nn.L1Loss()
loss = l1_loss(pred,gt)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">l1_loss = torch.nn.L1Loss()

for idx in range(B):
  if idx==0:
    loss = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:])
  else:
    loss = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:]) + loss

loss = loss/B
</code></pre>

ptrblck: <p>Yes, this seems to be the case for random input values:</p>
<pre><code class="lang-python">pred = torch.randn(10, 10, 10, 10)
gt = torch.randn(10, 10, 10, 10)

l1_loss = torch.nn.L1Loss()
loss1 = l1_loss(pred,gt)


for idx in range(pred.size(0)):
  if idx==0:
    loss2 = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:])
  else:
    loss2 = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:]) + loss2

loss2 = loss2/pred.size(0)

print(torch.allclose(loss1, loss2))
&gt; True
</code></pre>
Model on regression task not converging

Pager07: <p>I have a FFN as follows a to do regression task given x</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self,input_dim, hidden_dim, output_dim, num_layers):
        super(MLP,self,).__init__()
        self.num_layers = num_layers
        self.hidden = nn.Linear(input_dim, hidden_dim)
        self.out = nn.Linear(hidden_dim, 1)
        self.norm = nn.LayerNorm(hidden_dim)
        self.dorp50 = nn.Dropout(0.5)
        
    
    def forward(self,x):
        x = self.norm(x)
        x = self.hidden(x)
        x = F.relu(x)
        x = F.dropout(x)
        x = self.norm(x)

        x = self.out(x)
        x  = F.Sigmoid(x)
        return x 
</code></pre>
<p>The input the FFN is (bs,seq_len,d_model) ie. (1,1518,1024).<br>
The output of the FNN is (bs,seq_len,d_model) ie (1,1518,1).<br>
I trying to get a score between 0-1 for every time step.</p>
<p>The target is of shape (1518), a score between [0-1]. Baically a score for every time step.<br>
This is what the distribution of the targets look like<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f030a9a8835e8c565a53d2b02653e63750f42c2.png" alt="image" data-base62-sha1="8ZqB0irMo3sPMxazcpTdy4l2RBU" width="603" height="425"><br>
As you can see that the targets for most of the time stamps are mostly 0.</p>
<p>I am using MSELoss(). The problem is that the FFN, learns to predict all 0 as output and gets away with having a very low loss. Because it’s predicted 0 and we are using sigmoid, this leads to the gradient to be 0. Thus, the ratio of the FFN layer weights to update  is 0. Hence,  the model stops learning.</p>
<p>Is there something I am missing? I am aware that GANs generators use sigmoid to regress pixel value then use MSELoss, how come they don’t suffere from the problem I am suffering?</p>
<p>I am been working to debug this model for the past 2 days by now,I hope someone more knowledgeable than me can shine a light on this.</p>
<p>Thanks</p>

ptrblck: <p>You could try to remove the <code>sigmoid</code>, if you think it might kill the gradients.<br>
If that doesn’t help, a good debugging step is to try to overfit a small dataset first (e.g. just take 10 samples) by playing around with some hyperparameters and make sure your model and training routine is able to do so.<br>
Once this is done, you could scale up the use case again by using more data.<br>
If your model isn’t able to overfit a tiny dataset, then there might be a real bug in the code (e.g. you might have forgotten to zero out the gradients) or the model is just unable to learn the distribution (which I don’t think would be the case given your explanation of the use case).</p>
Using retrain graph=True in optimizing different parameters using different losses

m_alz: <p>I want to optimize N different input parameters, z_i, (all fed into the same network).  (similar to <a href="https://discuss.pytorch.org/t/optimising-different-parameters-with-different-loss-functions-efficiently/56789/4">this</a> thread)<br>
I have defined N different losses L_i and optimizers_i corresponding to each parameter.</p>
<p>for i=1:N:<br>
optimiser[i].zero_grad()<br>
loss[i].backward(retain_graph=True)<br>
optimiser[i].step()</p>
<p>is it necessarily to use “retain_graph=True” every time we call loss[i].backwards()?</p>
Optimising different parameters with different loss functions efficiently

henry_charlesworth: <p>I have a problem and I feel like the way I’m currently solving it is extremely inefficient so I’m hoping that it might be possible to speed this up.</p>
<p>The simplified set up is the following: I have some set of N parameters {x_i} which all go in to some complicated function. From the output of this function, I produce N different losses L_i (each of which can in principle depend on all of the initial parameters), but I want to optimise each parameter x_i with respect to its corresponding loss L_i only. What is the best way to do this?</p>
<p>Currently I am doing:<br>
for i=1:N:<br>
optimiser[i].zero_grad()<br>
loss[i].backward(retain_graph=True)<br>
optimiser[i].step()</p>
<p>but this is extremely slow. Is there a better way of doing this?</p>

albanD: <p>Unfortunately, if you only want the ith loss to effect the ith set of parameters, you cannot really do anything else as the backward of a single loss will “polute” the gradient of all the parameters.</p>

henry_charlesworth: <p>It seems like I should be able to make all of the losses into a tensor and then take the gradient with respect to all parameters (essentially a Jacobian), and then just choose the relevant gradients. But it doesn’t seem that PyTorch has a good way of computing Jacobians from what I’ve read so far.</p>

albanD: <p>Unfortunately, the only thing autograd can compute is jacobian vector products. So if you want to reconstruct a full jacobian, you have to do backward with tensors [1, 0, 0] then [0, 1, 0] and [0, 0, 1].<br>
You can look in the forum actually, there are a few posts discussing how to compute full Jacobian that might help you speed things up a bit <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to profile different components in forward pass?

rsz: <p>hi, i have</p>
<pre><code class="lang-auto">class ToyNetwork(nn.Module):
	def __init__(self, embedding_dim, hidden_dim):
		super(ToyNetwork, self).__init__()
		self.lstm1 = nn.LSTM(embedding_dim, hidden_dim, 70)
		self.lstm2 = nn.LSTM(embedding_dim, hidden_dim, 70)
	
	def forward(self, inputs, hidden1, hidden2):
		start = time.monotonic()
		self.lstm1(inputs, hidden1)
		torch.cuda.synchronize()

		mid = time.monotonic()
		self.lstm2(inputs, hidden2)
		torch.cuda.synchronize()

		print(f"""second network time: {time.monotonic()-mid}, first network time: {mid-start}""")
</code></pre>
<p>second network time: 0.010582592338323593, first network time: 0.02081933245062828</p>
<p>im wondering why there are a lot of differences between time taken by first lstm network vs second lstm network</p>
<p>thank you!</p>

ptrblck: <p>You would have to synchronize the code before starting and stopping all timers, while this is not the case for the first call into the <code>LSTM</code>.<br>
Also, I would recommend to use multiple iterations and calculate the average of the operations, as well as to add warmup iterations.<br>
The <code>torch.utils.benchmark</code> utility does this automatically for you and is the recommended way to profile workloads.</p>
A strange phenomenon in speed when use jit/torchscript. What's the difference between 1.6 and 1.7 in jit/torchscript ?

ccilery: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/51442" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/51442" target="_blank" rel="noopener nofollow ugc">A strange phenomenon in speed when use jit/torchscript. What's the difference between 1.6 and 1.7 in jit/torchscript ?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-02-01" data-time="05:34:34" data-timezone="UTC">05:34AM - 01 Feb 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/ccilery" target="_blank" rel="noopener nofollow ugc">
          <img alt="ccilery" src="https://avatars.githubusercontent.com/u/19500797?v=4" class="onebox-avatar-inline" width="20" height="20">
          ccilery
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">phenomenon
When I run code in pytorch 1.6 and pytorch 1.7.1, I found a strange phenomenon in speed. The speed is stable...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">oncall: jit</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Strange behavior of BatchNorm2d in evaluation mode (train vs eval)

sigma_x: <p>During training I set  all <code>BatchNorm2d</code> layers to <code>track_running_stats=True</code>, <code>affine=True</code>. I use batch size = 1. During evaluation the batch size is also 1, and what I’ve noticed that if the batch normalization layers are set <code>train=True</code> (i.e. they don’t use means and stds to normalize the image), performance is much better. What am I doing wrong here?</p>

tom: <p>Leaving batch norm training enabled and using batch size 1 makes it behave in a similar way to InstanceNorm w.r.t. training and inference (but you have a different setup what to average over). Depending on the task, that can be a reasonable choice. We usually don’t do that because it is problematic run into problems when inputs are very “dull” (i.e. have very little standard deviaton), they are blown up a lot, so the neural network considered as a function may not be not terribly continuous (well, technically continuity is an “on/off” term, but things can get really “steep”).</p>
<p>Best regards</p>
<p>Thomas</p>

sigma_x: <p>Thanks, I guess I don’t fully understand how batchnorm works during training/evaluation in pytorch. During training, batchnorm accumulates the history of means/stds and normalizes the inputs using the current. From the doc: `</p>
<blockquote>
<p>‘by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation’</p>
</blockquote>
<p>Does it mean than only one mean/std  (‘compound’) is used during the evaluation? I noticed Pytorch stores either 64 or 256 means/std.</p>

tom: <p>No it is by channel, but so the various *Norm modules provide all sorts of variations. I would always recommend to experiment with things to be certain they work as expected: set 1 channel mean to 1 and the others to 0 and all std to 0 and see what happens.<br>
Then look at what changes when setting 1 std to 2 etc.</p>
<p>Best regards</p>
<p>Thomas</p>

sigma_x: <p>Won’t it return all NaNs if I set std=0?</p>

User4: <p>No, because they always add an epsilon that is small enough to prevent the std = 0 but still not affects a lot to the result (1e-5 or something).</p>
<p>I guess your problem should be related to the running mean and running variance since you set your batch size as 1, the running stats will be strong affected by the later sample. You can try to reduce the momentum or change to group norm.</p>

sigma_x: <p>Thanks. So for batch size &gt; 1, are mean/std averaged per batch in such case? That is, for each channel, I take x(one feature) and subtract the mean for that channel, after taking the mean average of that channel in the whole batch?</p>

User4: <p>yes, we use batch mean in the training phase and the average for evaluation phase</p>
How to convert variable-sized 3D matrix to 2D?

Sine: <p>Hi, everyone. My output size of TransformerEncoder  is [Batch size, Sequence length, Embedding size], where the sequence length is variable. I want to convert my output size to [Batch size, 1, Embedding size].</p>
<p>I tried to use AvgPool1d, but kernel size is not set easily.<br>
Thanks.</p>

ptrblck: <p>You could use adaptive pooling layers (<code>nn.AdaptiveAvgPool1d</code>), which use a variable kernel size to create the desired output shape.<br>
Alternatively, you could also just calculate the mean via <code>output = output.mean(1, keepdim=True)</code>.</p>

Sine: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Torch-geometric

Yin1: <p>When I set batch size to 512, an error is reported：<br>
RuntimeError: nonzero is not supported for tensors with more than INT_MAX elements, file a support request</p>
<p>But when I set batch size to 256， it will run， why？</p>
<p>Report error code at ：</p>
<pre><code class="lang-auto">def dense_to_sparse(tensor):
    r"""Converts a dense adjacency matrix to a sparse adjacency matrix defined
    by edge indices and edge attributes.

    Args:
        tensor (Tensor): The dense adjacency matrix.
     :rtype: (:class:`LongTensor`, :class:`Tensor`)
    """
    assert tensor.dim() == 2
    index = tensor.nonzero(as_tuple=False).t().contiguous()
    value = tensor[index[0], index[1]]
    return index, value
</code></pre>
<p>Thank you very much for your reply</p>

ptrblck: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and explain your use case there?<br>
Based on the error message it seems there is an internal limitation for <code>nonzero</code> using <code>INT_MAX</code> elements.</p>

suyouooooo: <p>I have met the same error:<br>
File “/home/lib/python3.6/site-packages/torch_geometric/utils/sparse.py”, line 18, in dense_to_sparse<br>
index = tensor.nonzero().t().contiguous()<br>
RuntimeError: nonzero is not supported for tensors with more than INT_MAX elements,   file a support request</p>
<p>When I set batch size to 20, it will run, but when I set batch size to 40 or 30, the error will be reported.</p>
<p>So I wonder if you have solved this problem?</p>
Constrain outputs in a regression problem

ncuxomun: <p>Hi, everyone.</p>
<p>I am attempting to constrain some outputs of my regression network, say x, y, z = model(data), where x, y, z are scalars. The constrain that I want to impose is that when predicting all three dependent variables, the condition “x + y &lt;=1.0” must be honored. Given this description, can I implement this in a forward function?</p>
<p>Thank you!</p>

Dwight_Foster: <p>You could use the torch.clamp() function found <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html#torch.clamp" rel="noopener nofollow ugc">here</a> and set max to 0.5 for each and min to whatever you want. Then x+y could never be greater than 1.</p>

ncuxomun: <p>thanks for replying. What if either x or y can be larger than 0.5?</p>

Dwight_Foster: <p>Ok I found a way. You could do something like this</p>
<pre><code class="lang-auto">ratio = 1/(x+y)
x *= ratio
y *= ratio
</code></pre>
<p>Then the output of x + y will always be 1. The only problem is that if x+y is already less than one this will make it equal to one. To solve that you could just use an if else statement like this</p>
<pre><code class="lang-auto">if x+y &gt; 1:
   ratio = 1/(x+y)
   x *= ratio
   y *= ratio
else:
   continue
</code></pre>

ncuxomun: <p>Thanks, Dwight! I will give your suggestion a try!</p>

Dwight_Foster: <p>Ok let me know if there are any more issues.</p>

InnovArul: <aside class="quote no-group" data-username="ncuxomun" data-post="1" data-topic="110158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ncuxomun/48/30836_2.png" class="avatar"> ncuxomun:</div>
<blockquote>
<p>the condition “x + y &lt;=1.0” must be honored</p>
</blockquote>
</aside>
<p>Do you think <code>softmax</code> might help here?</p>

ncuxomun: <p>Hi. Yes, indeed, it did help. So, what I did was I had one Linear layer with a single output for x, which is then connected to a loss function (x_hat, x) and to another that has to make sure that (x_hat + y, ones(size(x_hat+y)). So, in total I had three loss functions. At last, I had to weigh each respective loss function.</p>
Feed & Concat Tensors of Multi-resolution Image at a Conv-layer

Ahmedest61: <p>Hi</p>
<p>I want to train a 2-conv/3-conv layered model by feeding and transforming every single input image into multiple resolution, e.g., 128,96 and 64,48. At a specific conv-layer (e.g, 3rd conv), I then need the tensor outputs of both the resolution. What can be the easiest way to do it on pytroch?</p>
<p>Thank You.</p>

ptrblck: <p>You could implement the rescaling and the usage of different code paths in the <code>forward</code> method of your model.<br>
To rescale the tensor activations you could use e.g. <code>torch.nn.functional.interpolate</code> or an upsamling layer.</p>

Ahmedest61: <p>Thank You, done that - during training: for 4D tensor (Batch, Channel, Width, Height), I used python indexing in loop, e.g., Imagesbatch[:,:,::ImgScale,::ImgScale] for image down sampling followed with forward pass and tensor concatenation. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
how to shuffle part of labels at runtime

kyy: <p>Hi,</p>
<p>Given a dataset containing N data points, I am interested in shuffling 10% of labels at each epoch (these 10% data points are randomly selected at each epoch). Does anyone know what might be a good way to do this?</p>
<p>Thank you!</p>

ptrblck: <p>I’m not sure, if I understand the question completely.<br>
Would you like to permute only 10% of the samples in each epoch while the rest would be drawn sequentially?<br>
If so, I think you could create the sequential indices, permute only a subset of it, and pass it to <code>torch.utils.data.Subset</code> in each epoch.</p>
How to change only learning rate when loading optimzer

Yolkandwhite: <p>Hi, I’m loading the optimizer like this.</p>
<pre><code class="lang-auto">    optimizer.load_state_dict(ckpoint['optimizer'])

</code></pre>
<p>I want to change only learning rate without changing any other things.</p>
<p>how can I fix it?</p>

ptrblck: <p>You can assign a new learning rate to the desired <code>param_group</code>, e.g.:</p>
<pre><code class="lang-python">optimizer.param_groups[0]['lr'] = 1e-3
</code></pre>
<p>should work.</p>
Calculating error

MasLiang: <p>When I wrote:</p>
<pre><code class="lang-auto">torch.round(torch.tensor(0.5))
</code></pre>
<p>I think it will be ‘1’. But I got ‘0’.<br>
And when I wrote this:</p>
<pre><code class="lang-auto">torch.round(torch.tensor(1.5)
</code></pre>
<p>I got the correct value ‘2’.<br>
What happened to ‘0.5’?</p>

ptrblck: <p>I think the <code>torch.round</code> operation sticks to numpy’s implementation, which explains:</p>
<blockquote>
<p>For values exactly halfway between rounded decimal values, NumPy rounds to the nearest even value. Thus 1.5 and 2.5 round to 2.0, -0.5 and 0.5 round to 0.0, etc. Results may also be surprising due to the inexact representation of decimal fractions in the IEEE floating point standard <a href="https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.around.html#r9">[R9]</a> and errors introduced when scaling by powers of ten.</p>
</blockquote>

MasLiang: <p>I got it. Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
What is "bridge" in forward method in U-Net?

je_ar: <p>hello everyone. I am new to Pytorch, sorry if my question is too simple. I found this code on <a href="https://github.com/jvanvugt/pytorch-unet/blob/master/unet.py" rel="noopener nofollow ugc">Github</a> for implementing U-Net architecture. I don’t understand what is “bridge” for in the forward method. could anybody explain it to me please?</p>
<p>def center_crop(self, layer, target_size):<br>
_, _, layer_height, layer_width = layer.size()<br>
diff_y = (layer_height - target_size[0]) // 2<br>
diff_x = (layer_width - target_size[1]) // 2<br>
return layer[<br>
:, :, diff_y : (diff_y + target_size[0]), diff_x : (diff_x + target_size[1])<br>
]</p>
<pre><code>def forward(self, x, bridge):
    up = self.up(x)
    crop1 = self.center_crop(bridge, up.shape[2:])
    out = torch.cat([up, crop1], 1)
    out = self.conv_block(out)

    return out</code></pre>

ptrblck: <p>The <code>bridge</code> tensor is most likely an activation tensor from the “left” part from the model (the down path), which is then passed to the “right” side of the model (the up path).<br>
The paper shows the architecture in a figure and you would see the connections between the down/up paths.</p>
How to export video classification model to torch script

WeNeedMore: <p>pytorch version : 1.7.1<br>
os : win10 64</p>
<p>Trying to export the video classification by following script</p>
<pre><code class="lang-auto">import os
import numpy as np
import torch
import torchvision

model = torchvision.models.video.r3d_18(pretrained=True, progress=True)
model.eval()

img = torch.zeros((16, 3, 112, 112))
_ = model(img)  # dry run
traced_script_module = torch.jit.trace(model, img)
traced_script_module.save("video_r3d_18.pt")
</code></pre>
<p>The script give me error messages</p>
<p><code>RuntimeError: Expected 5-dimensional input for 5-dimensional weight [64, 3, 3, 7, 7], but got 4-dimensional input of size [3, 16, 112, 112] instead</code></p>
<p>Afte I change it to [64, 3, 3, 7, 7], I can export the model, but the training codes and the doc, both use [3,16,112,112], this is weird, why this happen?</p>
<p>Thanks</p>

ptrblck: <p>The error is raised in the “dry run” section, bot the tracing, since you are passing a 4-dimensional input, while 3D models expect a 5-dimensional input.<br>
You could unsqueeze the “depth” dimension and the code should work:</p>
<pre><code class="lang-python">img = torch.zeros((16, 3, 1, 112, 112))
_ = model(img)  # dry run
</code></pre>
Training isues: validation loss is lower than training loss

lifeblack: <p>Hi I have a regression problem and designed a deep neural network… I run train mynetwork with small dataset an this is my loss versus epoch diagram: red: validation loss, blue: training loss</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d6a2d1ec892f7eaca6b50867d05507c7913584a.png" alt="image" data-base62-sha1="msyrynTeVxd57VDIMP2EBxD89V8" width="650" height="413"><br>
my validation loss is lower than training loss… my other problem is generalization… when I train my network with all of training data, the loss doesnt change… could you mind helping me solve this?</p>

J_Johnson: <p>That’s normal while training. This is because your training loss is averaging every batch loss between the epoch beginning and end. But your validation loss is only calculated after that epoch has already finished. So the validation will always show a slight advantage if the network improved during that epoch.</p>

ptrblck: <p>Usage of <code>dropout</code> layers might also yield a higher training loss, since your model would be a “small” version of the validation model.</p>
Problem with error backpropagation

praveen_kandula: <p>Although the program is running without errors, the result is not intended one. Got stuck for the past two days. Any help will be greatly appreciated.</p>
<pre><code class="lang-auto">        optimizer_G.zero_grad()
        for res_idx in range(2):
            a = someloss
            b = someloss
            c = someloss
            d = someloss
            
            if res_idx is 0:
                loss_G = a + b + c + d
                
            else:
                loss_G += a + b + c + d
        loss_G.backward()
        optimizer_G.step()
</code></pre>
<p>The losses a, b, c, d changes as loop continues. Is this the correct way to define the loss (since loss parameters are overwritten) and updating the parameters?</p>

ptrblck: <p>Could you explain the issue a bit more?<br>
If you are recalculating the <code>a, b, c, d</code> losses through <code>someloss</code>, it’s expected that they will change.<br>
How is <code>someloss</code> defined and what would be the expected behavior?</p>
Is there any way to make the specific class get into the minibatch from dataloader?

hongjunchoi: <p>Hi, I’d like to make the dataloader that samples the minibatch consists of specific class all the time.</p>
<p>For example, if the batch size 32, and the class label is 4(cat, dog, truck, ship),  I want to make the minibatch as 4examples of cat, 12 examples of dog, 4 examples of truck, 12 examples of ship.</p>
<p>Is there any solutions to make dataloader like above?</p>

ptrblck: <p>You could create a custom <code>sampler</code>, which is responsible to create the indices passed to <code>Dataset.__getitem__</code>. This custom sampler could use the <code>target</code> to create the valid indices for each batch such that the desired classes will be used.</p>
Is JIT tracing modules containing dropout or batch normalization submodules a bad idea?

ResidentMario: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html" rel="noopener nofollow ugc">torch.jit documentation</a> states that one of the limitations of tracing is that calls which differ based on whether the model is in <code>train</code> or <code>eval</code> mode will only ever use whatever mode the module was in at trace time. Specifically:</p>
<blockquote>
<ul>
<li>In the returned <a href="https://pytorch.org/docs/stable/generated/torch.jit.ScriptModule.html#torch.jit.ScriptModule" rel="noopener nofollow ugc"><code>ScriptModule</code></a>, operations that have different behaviors in <code>training</code> and <code>eval</code> modes will always behave as if it is in the mode it was in during tracing, no matter which mode the ScriptModule is in.</li>
</ul>
</blockquote>
<p>I know of some commonly used layer types have <code>train</code> / <code>eval</code> behavioral differences; <code>BatchNorm2d</code> and <code>Dropout</code> come to mind. Does this mean that tracing is A Bad Idea for modules making use of these two layers (and others like it?).</p>

ptrblck: <p>If might be a bad idea, if you would like to switch the behavior of the traced model.<br>
In that case you could try to script your model.</p>
GRU output shape seems to differ from documentation

Backup_Backup: <p>While reading the documentation for the GRU I read that: “output of shape (seq_len, batch, num_directions * hidden_size)”</p>
<p>When testing this my results seemed to have a different shape based on whether or not batch_first=True</p>
<p>The test was as follows:</p>
<pre><code>gru_bf = nn.GRU(input_size=512,
             hidden_size=1024,
             num_layers=1,
             batch_first=True,
             bidirectional=False)

gru = nn.GRU(input_size=512,
             hidden_size=1024,
             num_layers=1,
             batch_first=False,
             bidirectional=False)

batch = 1
seq = 100
input_size = 512

in_bf = torch.randn(batch, seq, input_size)
input = torch.randn(seq, batch, input_size)

out_bf,_ = gru_bf(in_bf)
out, _ = gru(input)

print(out_bf.shape)
print(out.shape)
</code></pre>
<p>The output is :</p>
<pre><code>torch.Size([1, 100, 1024])
torch.Size([100, 1, 1024])
</code></pre>
<p>It seems that if batch_first=True the output will have shape [batch, seq_len, hidden_size * n_dir]<br>
else it will have the shape specified by the documentation of [seq_len, batch, hidden_size * n_dir]</p>
<p>Is this indeed what is happening or am I misunderstanding something?<br>
Thanks for the help!</p>

Dwight_Foster: <p>Yes this is correct. batch_first just decides whether the batch should come first in the shape of the output or not. If you set it to True the batch size will be the first dimension. If false the batch_size will be the second dimension.</p>

Backup_Backup: <p>Alright, thank you very much.</p>
How to transform recall@top-k metric to regular recall?

Ivo_99: <p>Could anybody help me out by translating this recall@top-k formula to a “regular” recall?<br>
It concerns a classification task which ranges from 3087 to 1027 classes.</p>
<p><strong>Current code for recall@10,20,30</strong></p>
<pre><code class="lang-auto">def recallTop(y_true, y_pred, rank=[10, 20, 30]):
  outer = []
  for x in range(len(y_pred)):
    pred_value = y_pred[x]
    true_value = y_true[x]
    pred_value = torch.round(pred_value).clone().detach()
    TP = torch.sum(torch.logical_and(true_value == 1, pred_value))   # True positives (predictions)
    inner = []
    for i in rank:
      TP_k = torch.sum(torch.logical_and(pred_value[:, :i] == 1, true_value[:, :i]))  # True positives @top 10, 20, 30
      inner.append(TP_k)
    avg = torch.div(torch.tensor(inner), TP)
    avg[torch.isnan(avg)] = 0
    outer.append(avg.tolist())

  return (np.array(outer)).mean(axis=0)
</code></pre>
<p><strong>Current code in training loop</strong></p>
<pre><code class="lang-auto">train_recall.append(recallTop(y, output))
</code></pre>
<p>Current code in every epochs</p>
<pre><code class="lang-auto">avg_train_recall = (np.array(train_recall)).mean(axis=0)

print("Epoch: {}/{}...".format(e + 1, epochs),
"Train Recall@10, Recall@20, Recall@30", avg_train_recall)
</code></pre>
Is there function to replicate PadLayer

dcl: <p>I am looking for a fucntion that works similar to <a href="https://tensorlayer.readthedocs.io/en/latest/_modules/tensorlayer/layers/padding.html" rel="noopener nofollow ugc">PadLayer of tensorlayer</a>. I can use numpy instead to do this, But would like to know if there is a torch equivalent to this PadLayer</p>

InnovArul: <p><code>torch.nn.functional.pad</code> might do what you want.<br>
<a href="https://pytorch.org/docs/stable/nn.functional.html#pad" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/nn.functional.html#pad</a></p>

dcl: <p>No i cant use that I have already looked at it. But how do i pad a 4d tensor. You can look at the example in padlyer link in the question. More precisely i need reflect pading for 2nd and 3rd dims of 4d tensor</p>

dcl: <p>Archived what is needed.</p>
<p>torch_tensor.permute would bring the required dimensions to end and then apply a nn.functional.pad with reflect option on the 4d tensor for the last two dimensions. Later permute it back to its original form.</p>