akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeEror: mat1 dim 1 must match mat2 dim 0

ssk: <p>HELLO   iam getting this error while i was running this code cell<br>
def update_lr(optimizer, lr):</p>
<pre><code>for param_group in optimizer.param_groups:

    param_group['lr'] = lr
</code></pre>
<h1>Train the model</h1>
<p>total_step = len(train_loader)</p>
<p>curr_lr = learning_rate</p>
<p>for epoch in range(num_epochs):</p>
<pre><code>for i, (images, labels) in enumerate(train_loader):

    images = images.to(device)

    labels = labels.to(device)

    # Forward pass

    outputs = model(images)

    loss = criterion(outputs, labels)

    # Backward and optimize

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()
</code></pre>
<p>how to sove this</p>

ptrblck: <p>This error is most likely raised due to a tensor shape mismatch inside your model, e.g. in a linear layer. Print the activation shapes inside the <code>forward</code> method and make sure the feature size matches the expected <code>in_features</code> from the linear layers.</p>
How to convert the tensorflow code to pytorch?

explore_graphs: <p>Hi everyone,</p>
<p>I am trying to convert the following code in TensorFlow to PyTorch. But I couldn’t get the point of how to write the placeholders and tf.variable_scope() in PyTorch. Is there any equivalent function for those?</p>
<p>class Dense(Layer):<br>
“”“Dense layer.”""<br>
def <strong>init</strong>(self, input_dim, output_dim, placeholders, dropout=0., sparse_inputs=False,<br>
act=nn.ReLU(), bias=False, featureless=False, **kwargs):<br>
super(Dense, self).<strong>init</strong>(**kwargs)</p>
<pre><code>    if dropout:
        self.dropout = placeholders['dropout']
    else:
        self.dropout = 0.

    self.act = act
    self.sparse_inputs = sparse_inputs
    self.featureless = featureless
    self.bias = bias

    # helper variable for sparse dropout
    self.num_features_nonzero = placeholders['num_features_nonzero']

    with tf.variable_scope(self.name + '_vars'):
        self.vars['weights'] = glorot([input_dim, output_dim],
                                      name='weights')
        if self.bias:
            self.vars['bias'] = zeros([output_dim], name='bias')

    if self.logging:
        self._log_vars()

def _call(self, inputs):

    x = inputs

    # # dropout
    if self.sparse_inputs:
        x = SparseDropout(x, 1-self.dropout, self.num_features_nonzero)
    else:
        x = nn.Dropout(x, 1-self.dropout)

    # transform
    output = dot(x, self.vars['weights'], sparse=self.sparse_inputs)
    
    # bias
    if self.bias:
        output += self.vars['bias']

    return self.act(output)</code></pre>

InnovArul: <p>In Pytorch, you do not deal with maintaining different scopes as in TF.<br>
Here, the network layers (<code>torch.nn.*</code>) are included with their own parameters internally.<br>
I strongly suggest you to go through Pytorch tutorials:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" target="_blank" rel="noopener">Neural Networks — PyTorch Tutorials 1.7.1 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


explore_graphs: <p>Oh I see, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> also, there is no placeholders in PyTorch, right?</p>

InnovArul: <p>You are right.<br>
There is no placeholders in pytorch, as it is not building static-graph like TF.</p>
Performance different after loading saved models

amir_veyseh: <p>In my training script, I train a LSTM model and evaluate it after each epoch. After evaluation, I save the model using <code>torch.save</code> (I have tried saving both state_dict or the entire model) and reload it using <code>torch.load</code> (in the same training script, before continuing to the next epoch). The performance of the loaded model is the same as the saved model. However, when I run another script for evaluation which use the same code I used in the training script for loading the saved model, the performance is substantially different (accuracy drops from 70% to 10%). I’ve tried various methods for fixing the randomness using the techniques suggested in this page: <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.7.0 documentation</a></p>
<p>Is there any suggestions about what could be the cause of this huge difference in the performance of the loaded model in the same training script and a separate evaluating script?</p>

Abhilash_Srivastava: <p>This should not happen.</p>
<p>If you share your code, we can have a look.</p>

amir_veyseh: <p>Thanks, You are right. The problem was not due to randomness in PyTorch. I had a simple python bug. I was iterating a set and using the order of its items in my code.</p>
Conv2d vs conv3d

mohit117: <p>Hello,</p>
<p>Suppose I am working with <code>n</code> RGB video frames with convolution kernels <code>k x k</code>. I can channelwise stack all the frames and use pytorch conv2d with kernel <code>3n x k x k</code> or can simply use 3d convolutions with kernels <code>n x 3 x k x k</code>. So which should be used for highest accuracy? Theoretically, in both cases, the neural network should find either configuration comfortable as the parameter complexity and receptive field remains the same. Or is it that conv3d will be advantageous?</p>
<p>Thankyou very much.</p>

InnovArul: <p>I can’t say for sure which is advantageous. It may depend on specific tasks.</p>
<p>Logically, if we talk about video-based tasks, <code>Conv2d</code> may not observe temporal context and each frame will be processed independently (except they might interact during Batchnorm).<br>
<code>Conv3d</code> would make the features from each frame interact and may learn temporal contexts. Empirically, <code>Conv3d</code> is shown to perform better. You may refer to <code>resnet3D</code> or I3D model design.</p>

mohit117: <p>But as i said the frames are concatenated channel wise and thus the 2D kernel will be looking at all the frames and hence all frames interact as will be the case with conv3D.</p>

InnovArul: <p>I see. My bad that I didn’t understand it.</p>
<p>There is still a disadvantage with <code>Conv2d</code> in cases like below:</p>
<p>Let’s say we have 6 frames with RGB channels.<br>
With <code>Conv3d</code>, we can emulate applying a conv kernel for every 3 frames to learn short-range temporal features. i.e., with <code>in_channels=3 &amp;  kernel_size (3,5,5)</code> for example. In this way, there is a possibility to learn motion features in a hierarchical way.</p>
<p>With <code>Conv2d</code>, I am not sure if we can emulate it. As you said, we can apply <code>18 x 5 x 5</code> kernel. But it may not be efficient as learning hierarchically.</p>

mohit117: <p>Thanks for the understanding.</p>
After Update pytorch from 1.1.0 to 1.7.0. DDP Model cannot converge

sunshichen: <p>I trained a 3D unet model by Pytorch 1.1.0 Distributed Data Parallel. The test result is good. But after I update Pytorch to V1.7.0. Same code different result. Anyone can help me with that? Is it a syncbatchnorm problem or what?</p>
<p>Even if I finetune the pretrained model(on 1.1.0). Both losses during training and validation increased a lot using v1.7.0.</p>
<p>Below is the training epoch loss of two version. <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07c2daefd5ccf897e8ef8ce5ebb457f7277f70b7.png" alt="loss" data-base62-sha1="16EOHaiJCjvXSxeH7ltZRI2g8zZ" width="354" height="258"></p>

ptrblck: <p>Are you only seeing the increase in the losses while using DDP or also while training the model on a single GPU?</p>

sunshichen: <p>I only tested on DDP. Because my input data is too large to put in single GPU. So I have to use DDP with syncbatchnorm.<br>
P.S. Could it be a cuda version problem? I used Pytorch1.7+cu101, however my server still using cuda10.</p>

ptrblck: <p>It’s unclear where the difference is coming from and I would recommend to check the loss curves on a single GPU first (by reducing the batch size or slimming down the model).<br>
The next steps depend on the outcome of this run (e.g. using DDP again and removing SyncBN etc.).</p>

sunshichen: <p>The input data is too large that it can only fit into GPU with batchsize=1. This is why I have to choose syncbatchnorm with DDP to simulate larger batchsize. By the way, groupnorm took much more GPU memory(Why I didn’t use it.). Do you have any idea where should I check first except for try single GPU?</p>

ptrblck: <p>Try to eliminate as many moving parts as possible in order to isolate it.</p>

sunshichen: <p>Can you give me more details? Like an example?</p>
<p>P.S.<br>
Even though I trained on 4 GPU(simulate bs=4) still converge much slower than before(2 GPU, simulate bs=2)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c9f7c005aea66a516f35c55b7428ad86220f6dd.png" alt="loss2" data-base62-sha1="45d3TItOGszlNFcDBRotdezqyNn" width="170" height="116"></p>

Hzzone: <p>I have also met this problem when I tried to reproduce GLOW using this code, <a href="https://github.com/rosinality/glow-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rosinality/glow-pytorch: PyTorch implementation of Glow</a> with torch 1.7+cu101. It works ok with DP.</p>
How does nn.conv2D implement the feature map of LeNet-5?

miner_tom: <p>I have been going through the documentation of conv2D as well as a couple of tutorials. Reading the docs from pytorch I see</p>
<pre><code class="lang-auto">torch.nn.Conv2d(in_channels: int, out_channels: int, kernel_size: Union[T, Tuple[T, T]], stride: Union[T, Tuple[T, T]] = 1, padding: Union[T, Tuple[T, T]] = 0, dilation: Union[T, Tuple[T, T]] = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')
</code></pre>
<p>From the LeNet paper of November 1998 I see that the third convolution layer is implemented with 6 input layers and 16 output layers. The 16 output layers are made from a combination of the 6 input layers according to a table, also in the paper:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b19819e9c15ccd9c35b1f7a355220bc3a933576e.png" alt="LenetFeatureMap" data-base62-sha1="pl4o24KuuW8qYm7qF4F0Mo4hzBQ" width="640" height="480"></p>
<p>What I do not see is how pytorch implements this feature map table.</p>
<p>Thank You<br>
Tom</p>

InnovArul: <p><strong>Short answer:</strong> In my understanding, with <code>Conv2d</code>, there is no way of specifying local connections unless the connections are enforced after <code>Conv2d</code> with a hand-designed mask tensor. <code>Conv2d</code> connects all input feature maps to all output feature maps (with the exception of <code>group</code> convolution).</p>
<p>In my understanding of LeNet paper, the local connections in 'LeNet` were hand-designed for two reasons:</p>
<ol>
<li>to keep the computation &amp; memory complexity within bounds.</li>
<li>to break the symmetry between the learned features</li>
</ol>
<p>With the developments of GPU technology over the years, reason 1 is not needed as of today (atleast for small networks like LeNet). For reason 2, we may use other methods such as randomly initializing weights, dropout etc., to break symmetry.</p>
<p>The <code>Conv2d</code> is designed to be simplistic and perform the conv operation with all <code>input</code> feature maps connected to all <code>output</code> feature maps. It is up to the user to enforce any hand-designed connection rules, if needed. after getting the conv output.</p>

miner_tom: <aside class="quote no-group" data-username="miner_tom" data-post="1" data-topic="107096">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miner_tom/48/30088_2.png" class="avatar"> miner_tom:</div>
<blockquote>
<p>What I do not see is how pytorch implements this feature map table.</p>
</blockquote>
</aside>
<p>Thank you, InnovaArul, for your response to my post.</p>
<p>I was not expecting that answer. Your response makes perfect sense, and I am grateful.</p>
<p>Regards<br>
Tom</p>
Torch.onnx.export of a model returning dict

vadimkantorov: <p>Is it supported? Can I specify the mapping of output keys to output_names? Will specifying output_names retrieve the specified output keys from the output dict?</p>
<p>It seems that by chance I get the correct outputs out of onnxruntime session run, probably because of the fact that python3 dicts preserve key order by default, but it’s not very reliable.</p>
<p>Thanks!</p>

liangcheng: <p>Same question here. Does anyone have an answer?</p>
Initial hidden state of GRU when testing the real data

Hou_ZeYu: <p>I try to understand how GRU work in the model.<br>
I already read the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html#torch.nn.GRU" rel="noopener nofollow ugc">GRU</a> document.<br>
I think I known how it work in the forward process.<br>
Here is my example for demo forward follow blow.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33374d5f87a63dd2c695d9e890b06ab53186ffec.png" alt="001" data-base62-sha1="7j4PNDsn9eXfiBwgJxO76ESO0JS" width="623" height="176"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e5065e6c9acbd34a829e8729c6e559fc780c030.png" alt="002" data-base62-sha1="8TfR1otakJiImfkbpnKUWVT4tFu" width="606" height="188"></p>
<p>For each input data, I need to initialize a hidden state.<br>
In training process, I can randomly initialize it.<br>
After finishing training process, if I want to test a real data, how to determine the initial hidden state?<br>
Still by randomly choose? Or I make some mistake about forward of GRU?</p>

Abhilash_Srivastava: <p>Use the same method for both train and test (random initialization in this case).</p>

Hou_ZeYu: <p>The same input data, different initial state, will get the different result, is it normal?</p>

Abhilash_Srivastava: <p>Can you elaborate on what you exactly mean?</p>
<ol>
<li>Did you mean that the same model after training is used with different initial states?</li>
<li>Did you mean training with different initial states?</li>
</ol>
What's the difference between batch and batch_size?

hpf: <p>Hi,There is a problem I don’t quite understand<br>
In Pytorch,the input shape of RNN is 【seq_len, batch, input_size】or  【batch, seq_len, input_size】,<br>
In Keras,   the input shape of RNN is  【batch_size, time_steps, input_dim】,</p>
<p>so,Do  batch_size and batch mean the same？I found some different answers on stackoverflow, but it made me more confused.like this：</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm">
  <header class="source">

      <a href="https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9965155/pinkbanter" target="_blank" rel="noopener nofollow ugc">
    <img alt="PinkBanter" src="https://i.stack.imgur.com/VdWBQ.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm" target="_blank" rel="noopener nofollow ugc">Understanding input shape to PyTorch LSTM</a>
</h4>

<div class="tags">
  <strong>python, pytorch, lstm, tensor</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9965155/pinkbanter" target="_blank" rel="noopener nofollow ugc">
    PinkBanter
  </a>
  on <a href="https://stackoverflow.com/questions/61632584/understanding-input-shape-to-pytorch-lstm" target="_blank" rel="noopener nofollow ugc">10:11AM - 06 May 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I think<br>
【the total data = Batch * Batch_size】<br>
just like，in pytorch，the  batch_size of【Dataloader】:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0067ccf57da1d04841ad962916471b5683278d07.png" alt="image" data-base62-sha1="3Aoi60pYBczHkMYVddQMDoesTl" width="676" height="40"><br>
but I dont know if I am aright.</p>

InnovArul: <p><code>batch</code> in pytorch LSTM documentation == <code>batch_size</code> in Keras RNN documentation</p>

hpf: <p>Thank you for your reply，<br>
I quite agree with you，but,if it is true, Why not write the 【batch_size】 in the LSTM(pytorch)？  because the dataloader is written with batch_ size.and in the meaning of dataloader，it seems that batch and batch_size are different.<br>
in other words，</p>
CCA, CKA for captum

Brando_Miranda: <p>Is CCA or CKA available for captum?</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/captum/issues/572">
  <header class="source">

      <a href="https://github.com/pytorch/captum/issues/572" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/captum</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/captum/issues/572" target="_blank" rel="noopener nofollow ugc">CCA/CKA/OPD support</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-12-24" data-time="20:35:04" data-timezone="UTC">08:35PM - 24 Dec 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/brando90" target="_blank" rel="noopener nofollow ugc">
          <img alt="brando90" src="https://avatars.githubusercontent.com/u/1855278?v=4" class="onebox-avatar-inline" width="20" height="20">
          brando90
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Is CCA or CKA or Orthogonal Procrustes Distance (OPD) available for captum?</p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="quote quote-modified" data-post="1" data-topic="104889">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-a-good-cca-cka-library-for-pytorch-that-works-ideally-with-gpu/104889">What is a good cca, cka library for pytorch that works (ideally with GPU)?</a> 
  </div>
  <blockquote>
    I found: 


but it doesn’t seem to work reliably (or is it documented). 
Is there a standard library pytorch peeps use for cca, cka, neural net comparison analysis that actually works? (ideally on gpu)
  </blockquote>
</aside>
Model doesn't train

wheresmadog: <p>Hi, I’m trying to train a binary classifier where 64x64x3 images are as inputs. But the model only outputs close to 0. I also, found that the parameters before and after training differ, that is, gradient updates anyway.</p>
<p>Is there anything else I should try?</p>
<pre><code class="lang-auto">def fn_fit(self, optimizer, loss_fn, batch_size=16, epochs=10, lr=0.001):
        train_load = DataLoader(self.train, batch_size=batch_size, shuffle=True)
        
        total_step = len(train_load)
        for epoch in range(epochs):
            for step, (x_batch, y_batch) in enumerate(train_load):
                #Initializing
                self.model.train()
                optimizer.zero_grad()

                #Loading GPU Memory
                x_batch, y_batch = x_batch.to(device), y_batch.to(device)

                #Feed Forward
                pred = self.model(x_batch)
                loss = loss_fn(pred, y_batch)

                #Feed Backward
                loss.backward()        

                #Update Parameters
                optimizer.step()
                
                if step%(total_step//10)==0 or step==total_step:
                    self.train_loss.append(loss.item())
                    print(f'Epoch:{epoch+1:2d}/{epochs}, Step:{step:3d}/{total_step}, Training Loss:{loss.item():.4f}')
</code></pre>

ptrblck: <p>Try to overfit a small dataset, e.g. just 10 samples, by playing around with the hyperparameters.<br>
Once your model is able to do so scale up the use case again.</p>

wheresmadog: <p>I resized the input of images, and turns out the model is finally learning. Thank you!</p>
RNN with and without loop output mismatch

csblacknet: <p>For some tasks I have seen two different ways of using RNNS:</p>
<ul>
<li>In the First Method, Input [batch_size, input_size, sequence_size] is directly passed and the outputs, hidden are returned by it.</li>
<li>In Second Method, Each timestamp [input_size, sequence_size] of the Input is passed and the outputs, hidden are received on the other end, this process happens in a loop(for i in range(batch_size)).</li>
</ul>
<p>My doubt is what’s really the difference between these two methods?</p>
<p>To support this, I have written a small piece of code, which might help you understand where I went wrong.</p>
<p>This is my basic RNN Class,</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes, device):
        super(RNN, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.device = device

        #nn.RNN and nn.GRU work same way
        self.rnn = nn.GRU(input_size, hidden_size, num_layers, batch_first = True)
        # input : batch_size, sequence_len, input_size
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self,x,  h0 = None):
        if h0 is None:
          h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(self.device)
        out, ht = self.rnn(x, h0)

        return out, ht

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
rnn = RNN(28, 128, 1 , 10, device)
</code></pre>
<p>Below I try to get outputs and hidden directly without looping through each timestamp</p>
<pre><code class="lang-auto">#Without Loop
inps = torch.randn((32,100,28))
out_wl,ht_wl = rnn(inps)
print("Output shape : ", out_wl.shape) #Output shape :  torch.Size([32, 100, 128])
print("Hidden shape : ", ht_wl.shape) #Hidden shape :  torch.Size([1, 32, 128])
</code></pre>
<p>Here I am trying to send one timestamp(row) at a time and record all of the outputs.</p>
<pre><code class="lang-auto">#With Loop
outs = []
hns = []
hn_temp = torch.zeros(1, 1, 128).to(device)
for inp in inps:
  inp = inp.unsqueeze(0)
  out_temp,hn_temp = rnn(inp, hn_temp)
  outs.append(out_temp)
  hns.append(hn_temp)

outs = torch.stack(outs)
hns = torch.stack(hns)

print(outs.shape) #torch.Size([32, 1, 100, 128])
print(hns.shape) #torch.Size([32, 1, 1, 128])

outs = outs.squeeze(1)
hns = hns.squeeze(1)

print(outs.shape) #torch.Size([32, 100, 128])
print(hns.shape) #torch.Size([32, 1, 128])
</code></pre>
<p>The issue here is when I do</p>
<pre><code class="lang-auto">out_wl == outs
</code></pre>
<p>The output tensor is not True everywhere.</p>
<p>This all boils down to few questions,</p>
<ul>
<li>Both methods I talked about, are they same or not?</li>
<li>If they are same then the resulting output doesn’t match, do mind model has not done any backward pass and the same randn input is used in both with and without loop snippets.</li>
<li>Is last value of outs same as hns, does that mean outs is just a collection of hns for all s</li>
</ul>
Best practices for training on 500GB of instances, all are large

turian: <p>I am curious what are best practices for loading training instances that are 220K floats from disk, over 500GB of instances. I am training on audio. I am maxing out at minibatch size 2048, otherwise GPU memory is exhausted. I am using pytorch-lightning but fine with converting to vanilla torch.</p>
<p>I am concerned that if I switch to multiple GPUs, the training speed with nonetheless be gated by the speed of the harddisk, not how many GPUs I have.</p>
How to convert pytorch image captioning model(CNN+LSTM-encoder decoder) model to tflite?

learn1: <p>I am trying to convert CNN+LSTM (encoder decoder) model mentioned in the following github repo is : <a href="https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning" rel="noopener nofollow ugc">Pytorch image captioning </a></p>
<p>I want to convert this pytorch model to tflite. It has both encoder and decoder checkpoints. As far as i understand both of them have to be converted to tflite (correct me if i am wrong)</p>
<p><strong>approach:</strong> using the example mentioned in onnx2keras library, <a href="https://pypi.org/project/onnx2keras/" rel="noopener nofollow ugc">onnx2keras</a> i was able to convert encoder to tflite. but with decoder i am facing the following issue<br>
<em><strong>Not sure what is the right approach. can anyone suggest better approach and help me achieve a tflite model</strong></em></p>
<blockquote>
<p>File “convert_pytorch_tf.py”, line 63, in <br>
change_ordering=False)<br>
File “/root/anaconda3/envs/pyt2tf/lib/python3.7/site-packages/pytorch2keras/converter.py”, line 53, in pytorch_to_keras<br>
dummy_output = model(*args)<br>
File “/root/anaconda3/envs/pyt2tf/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
TypeError: forward() missing 2 required positional arguments: ‘captions’ and ‘lengths’</p>
</blockquote>
[Available for Hire] [Remote Team]

Canopus: <p>Hi, everyone, Wishing you all Wonderful Holidays &amp; Cheers to a Brighter 2021🎉</p>
<p>We are a team of Data Scientist and specialize in cognitive technologies like AI, natural language processing, and machine learning. We will collaborate with you to discuss your problems and see what innovative solutions we can develop to solve them.</p>
<p>Our client’s cases here, <a href="https://tinyurl.com/y7umn6gg" rel="noopener nofollow ugc">https://tinyurl.com/y7umn6gg</a></p>
<p>Chat soon!</p>
<p>–<br>
Regards<br>
Sandeep | <a href="mailto:sandeep.thakur@canopusinfosystems.com">sandeep.thakur@canopusinfosystems.com</a></p>
NLP model dataset loader

rivalset: <p>I’m trying to train a BERT-style model using PyTorch. I’m looking for recommendations on how to store the pretraining data (i.e. hdf5, parquet, tfexample, apache arrow…) and how to load it for training in PyTorch.</p>
<p>It would be great if there was a dataset loader that already supports multi-threaded loading, training using multiple workers, mixing different datasets without having to write much code.</p>

BramVanroy: <p>I can highly recommend the huggingface datasets, which uses the very ast and efficient Apache arrow data format. <a href="https://github.com/huggingface/datasets" class="inline-onebox" rel="noopener nofollow ugc">GitHub - huggingface/datasets: 🤗 The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools</a></p>
Bi-directional and multi-layer LSTM in Seq2Seq auto-encoders

catosphere: <p>Hello everyone,</p>
<p>I do not have a Pytorch issue to report but I would like to ask for good practices / recommendations on using bi-directional and multi-layer LSTMs for a Seq2Seq auto-encoder please.</p>
<p>Before I give details, when I train my model with default LSTM(num_layers=1,bidirectional=False) for both encoder and decoder I have some decent reconstruction results on the task. I try using either several layers, or bidirectional layer and results turns out pretty bad whereas I hoped it would help the model training. I have several hundreds of thousands of training samples and the model capacity is quite low so I don’t think it comes from over-parameterizing the model but rather I guess I am not using these functionalities well … Here I come to ask your advices about it.</p>
<p><strong>In the default case: (we assume everywhere batch_first=True)</strong></p>
<p>E_output , _ = encoder_RNN ( input of shape = [batch,seq,in_dim] )<br>
E_global_context = E_output[:,-1,:]  of shape = [batch,hid_dim]</p>
<p>D_global_context = global_context.unsqueeze(1).repeat(1seq,1)<br>
D_output , _ = decoder_RNN ( D_global_context ) of shape = [batch,seq,out_dim]</p>
<p><strong>multi-layer encoder_RNN:</strong> the encoder output is the same (independent from num_layers)</p>
<p><strong>bi-directional encoder_RNN:</strong> concatenate the forward and reverse last states</p>
<p>E_output = E_output.view(batch,seq,2,hid_dim)<br>
E_global_context = torch.cat([E_output[:,-1,0,:], E_output[:,0,1,:]],-1)  of shape = [batch,2*hid_dim]</p>
<p><strong>multi-layer decoder_RNN:</strong> the decoder output is the same (independent from num_layers)</p>
<p><strong>bi-directional decoder_RNN:</strong> D_output of shape = [batch,seq,out_dim*2]</p>
<p>Before and after the RNNs is a Linear layer to adapt to the actual input feature size, latent context size and output feature size. I am just wondering if I am correctly treating the outputs of multi-layer / bi-directional RNNs since this causes much worse results than using the default LSTM settings.</p>
<p>Thanks for reading and for your advices !</p>

Abhilash_Srivastava: <p>Overall it looks fine. In this part:</p>
<pre><code class="lang-auto">E_global_context = torch.cat([E_output[:,-1,0,:], E_output[:,0,1,:]],-1)
</code></pre>
<p>I was wondering should it be instead:</p>
<pre><code class="lang-auto">E_global_context = torch.cat([E_output[:,-1,0,:], E_output[:,-1,1,:]],-1)
</code></pre>

catosphere: <p>That’s a good point I have been wondering about too.</p>
<p>According to the LSTM doc:</p>
<p><strong>output</strong> of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the LSTM, for each t.</p>
<p>To my understanding, for the forward pass, we want -1 which is the last step T.<br>
For the backward pass, we want the step 0 which is the last reverse step.</p>
<p>Maybe someone around the forum can confirm whether torch.cat([E_output[:,-1,0,:], E_output[:,0,1,:]],-1) does that or not ?</p>
Help RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

linnabrown: <p>I meet a problem when I calculating the loss function.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 122, in &lt;module&gt;
    g_loss.backward()
  File "/nas/longleaf/home/lehuang/tool/anaconda/envs/deephic/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/nas/longleaf/home/lehuang/tool/anaconda/envs/deephic/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 256, 1, 1]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>

InnovArul: <aside class="quote no-group" data-username="linnabrown" data-post="1" data-topic="107030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/linnabrown/48/32676_2.png" class="avatar"> linnabrown:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 256, 1, 1]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
</blockquote>
</aside>
<p>As given in the error message, there seems to be a variable of size <code> [1, 256, 1, 1]</code> is modified in-place. If you would like to post some code, we might be able to help.</p>
<p>Just to give you an example,</p>
<pre><code class="lang-auto">x = torch.randn(5,6, requires_grad=True)
z = x.sum(-1)
z += z * z
z.sum().backward()
</code></pre>
<p>In the above sample code, <code>z += z * z</code> uses <code>z</code> to perform an operation, at the same time, It modifies <code>z</code> in place. When the gradient is calculated, the old <code>z</code> value that’s needed is already modified, thus resulting in error.</p>
<p>You can enable pytorch’s anomaly detection as below which may help you to locate the line that gives error:</p>
<pre><code class="lang-auto">import torch
with torch.autograd.set_detect_anomaly(True):
    x = torch.randn(5,6, requires_grad=True)
    z = x.sum(-1)
    z += z * z
    z.sum().backward()
</code></pre>
TypeError: can only concatenate tuple (not "int") to tuple

seanbenhur: <p>When I try to build vocab from the my data I get this error u can view my dataset <a href="https://drive.google.com/file/d/1af5l_FYJWPOckUdgCBHUpOj7bnAysnD6/view?usp=sharing" rel="noopener nofollow ugc">here</a><br>
TypeError: can only concatenate tuple (not “int”) to tuple</p>
<blockquote>
<p>QUERY.build_vocab(train_data,max_size=MAX_VOCAB_SIZE)<br>
TypeError                                 Traceback (most recent call last)<br>
 in ()<br>
1<br>
----&gt; 2 QUERY.build_vocab(train_data,max_size=MAX_VOCAB_SIZE)</p>
</blockquote>
<blockquote>
<p>1 frames<br>
/usr/local/lib/python3.6/dist-packages/torchtext/vocab.py in <strong>init</strong>(self, counter, max_size, min_freq, specials, vectors, unk_init, vectors_cache, specials_first)<br>
66             del counter[tok]<br>
67<br>
—&gt; 68         max_size = None if max_size is None else max_size + len(self.itos)<br>
69<br>
70         # sort by frequency, then alphabetically</p>
</blockquote>
<p>TypeError: can only concatenate tuple (not “int”) to tuple</p>
<p>can anyone help!?</p>

Abhilash_Srivastava: <p>Nothing wrong with this part.<br>
Probably something wrong in creating <code>train_data</code>.</p>
Bert bug? please have a look

XiMing_Dong: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19f31b21fab3860739bdd053b5affd5f6ab71b98.png" data-download-href="https://discuss.pytorch.org/uploads/default/19f31b21fab3860739bdd053b5affd5f6ab71b98" title="Screen Shot 2020-12-04 at 2.18.07 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19f31b21fab3860739bdd053b5affd5f6ab71b98_2_690x422.png" alt="Screen Shot 2020-12-04 at 2.18.07 PM" data-base62-sha1="3HyMUFIbpuOwJMG5kElSbX35JuU" width="690" height="422" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19f31b21fab3860739bdd053b5affd5f6ab71b98_2_690x422.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19f31b21fab3860739bdd053b5affd5f6ab71b98_2_1035x633.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/9/19f31b21fab3860739bdd053b5affd5f6ab71b98.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-12-04 at 2.18.07 PM</span><span class="informations">1069×654 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the first epoch is running and the second is a break, I have no ideas about it since I am a fish. plz have a look and help me thanks</p>

ptrblck: <p>Assuming you are using the Huggingface Transformer library with Tensorflow, I would recommend to create an issue in their repository or discussion board to get the best and fastest help. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

XiMing_Dong: <p>How can I do that? I do not see a discussion board about BERT<br>
thanks</p>

XiMing_Dong: <p>How can I do that? I do not see a discussion board about BERT<br>
thanks</p>
<aside class="quote no-group quote-modified" data-username="XiMing_Dong" data-post="3" data-topic="105169">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ximing_dong/48/31953_2.png" class="avatar"> XiMing_Dong:</div>
<blockquote>
<p>unless</p>
</blockquote>
</aside>

ptrblck: <p>You could try the <a href="https://discuss.huggingface.co/">Huggingface forum</a> or their <a href="https://github.com/huggingface">GitHub repository</a>.</p>

XiMing_Dong: <p>thanks and have a good one</p>

Aditya_Sihag: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. I’ve posted a query, and haven’t received any reply, and I really need help with this. Your advice would really help me. Could you please have a look?</p>
<p><a href="https://discuss.pytorch.org/t/portraitnet-model-training-loss-not-decreasing-beyond-initial-point/106988">https://discuss.pytorch.org/t/portraitnet-model-training-loss-not-decreasing-beyond-initial-point/106988</a></p>
Model Quantization

pepsbn: <p>Hi all, is there any official implementation/model for supporting model quantization in Pytorch? I mean sth similar to that of TF (<a href="https://www.tensorflow.org/performance/quantization" rel="nofollow noopener">https://www.tensorflow.org/performance/quantization</a>)</p>
<p>Cheers<br>
-P</p>

Felix_Lessange: <p>Shouldn’t this be possible with 1.0 ? I mean, it seems it was advertised like this. Can’t find it though.</p>

justusschock: <p>This should be possible with <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md" rel="nofollow noopener">glow</a> (haven’t tested it yet)</p>

dingyongchao: <aside class="quote no-group" data-username="Felix_Lessange" data-post="2" data-topic="16567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/felix_lessange/48/8959_2.png" class="avatar"> Felix_Lessange:</div>
<blockquote>
<p>though</p>
</blockquote>
</aside>
<p>I want training quantification such as TF as well! Waiting!</p>

Felix_Lessange: <p>I’m still very interested in this. Current state isn’t workable (and doesn’t support all operators).</p>
<p>Has pytorch dropped the idea of allowing speed optimization inside the framework ?</p>
Malloc, and corrupted size errors

Felix_Lessange: <p>Hi.<br>
I’m using pytorch in multi gpu setting, and also with some multiprocessing for loading and preprocessing and quite often encountering these errors, which I can’t manage to solve.</p>
<p><code>malloc.c:4023: _int_malloc: Assertion (unsigned long) (size) &gt;= (unsigned long) (nb)</code><br>
<code>corrupted size vs. prev_size</code></p>
<p>Theses appear very randomly too. I thought this was related to <a href="https://github.com/pytorch/pytorch/issues/2507" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/2507</a>, but their solution didn’t work for me :</p>
<pre><code class="lang-auto">sudo apt-get install libtcmalloc-minimal4 
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4" 
</code></pre>
<p>It seems that this error pops up really often when working with GroupNormalization. I can’t even do a whole epoch actually. I thought it might be due to the multiprocessed nature of my pipeline (I use torch.multiprocessing.Array to comunicate batches between processes), but the fact that the frequency depends on the architecture used is pretty weird to me.</p>
<p>Have you got any idea what might cause this ?</p>
<p>python                 3.7<br>
torch                    1.0.0</p>

saransh_karira: <p>Hey <a class="mention" href="/u/felix_lessange">@Felix_Lessange</a> ! Did you resolve it? If yes, then how??</p>

Felix_Lessange: <p>Hey. Sorry for late reply. I never saw this in 2020. I’m guessing some silent driver / pytorch / kernel update ?</p>
How to force pytorch to use NCCL?

Hawk: <p>Hi. I’m currently working with facebook’s Detectron2 framework based on pytorch 1.7.0, and I notice the training is using <code>gloo</code> by default, which is possibly prone to causes the <code>runtime error: connection closed by peer</code>. Now I want to give <code>nccl</code> a try, and what should I do to enforce this?</p>

rvarm1: <p>Hi <a class="mention" href="/u/hawk">@Hawk</a>,</p>
<p>You can simply pass in <code>backend="nccl"</code> into <code>dist.init_process_group</code> to enable training with the NCCL backend (assuming that NCCL is installed on your system and pytorch is installed with NCCL support).</p>

ppwwyyxx: <aside class="quote no-group" data-username="Hawk" data-post="1" data-topic="106386">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/22d042/48.png" class="avatar"> Hawk:</div>
<blockquote>
<p>notice the training is using <code>gloo</code> by default</p>
</blockquote>
</aside>
<p>That’s not precise. Detectron2 uses both NCCL and Gloo under the hood.</p>
Iterable.next() never stops running

Soumyadeep: <p>I am trying to create my own custom dataset to train a CNN.My dataset has images and corresponding annotations.<br>
The code is as follows:</p>
<pre><code class="lang-auto">
import ...

annotation=...
 
class Data(Dataset):
#    def _init_(self, annotation , root_dir , transform=None):   
    def __init__(self, root_dir,annotation):   
        self.annotations = annotation
        self.root_dir = ...
        
    def __len__(self):
        return len(self.annotations[0,:])
    
    def __getitem__(self,idx):
        image_name = ..
        image_mat = scipy.io.loadmat(voxel_name) #images are saved as mat files                                                
        image=torch.from_numpy(image_mat)
        annotation = ...
        return image, annotation 
               
trainset = Data(root_dir = "....",annotation = annotation)
dataloader = DataLoader(trainset, batch_size=5,
                        shuffle=True, num_workers=4)


dataiter = iter(dataloader)

</code></pre>
<p>Now the iterable dataiter is created.<br>
However, whenever I do : <code>image,annotation = dataiter.next()</code> , the code just keeps on running , never giving a result .<br>
Can someone please explain this ?<br>
Thanks .</p>

ptrblck: <p>Could you try to get a sample directly with:</p>
<pre><code class="lang-auto">image, annotation = trainset[0]
</code></pre>
<p>Does it work?<br>
If it’s working, could you try to set <code>num_workers=0</code> in your <code>DataLoader</code> and run it again?</p>

Ahmed_Hendawy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  had the same issue. It worked with me when I set the <code>num_workers</code> to zero but I could not understand why increasing the number of workers will make the loading stuck?</p>

ptrblck: <p>If you are using Windows, you would need to if-clause protection as described in the <a href="https://pytorch.org/docs/stable/notes/windows.html#usage-multiprocessing">Windows FAQ</a>. If that’s not the case, other reasons might be insufficient shared memory or any other problem with multiprocessing on your system.<br>
Also, if you are using a Jupyter Notebook, try to run the script directly in a terminal, as it might give you better error messages.</p>