akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
MNIST labels are not of size (batch_size, num_classes)

username: ayaz-amin
cooked: <p>Hi, so I am messing around with some datasets when suddenly I got a size error. This normally does not happen, so I check the shapes of the data and targets.</p>
<p>The shape of the MNIST targets is of size (batch_size), no classes anywhere.</p>
<pre><code class="lang-python">x, y = iter(train_loader).next()
print(y.shape)
</code></pre>
<pre><code class="lang-auto">torch.Size([32])
</code></pre>
<p>Is this a bug with PyTorch, or am I missing something?</p>

username: mailcorahul
cooked: <p>The tensor <code>y</code> will contain the target class index(from 0-9) instead of one hot encoded vector. That’s why the shape is of <code>batch_size</code>.</p>
"double free or corruption (!prev)" when using gpu. fine when using cpu

username: Dan_Erez
cooked: <p>I am training a net. When I train it on the gpu it runs fine other than being slow.<br>
When I train on the GPU  which I have used extensively with pytorch otherwise) the training quits abruptly.</p>
<p>I get the cryptic message : “double free or corruption (!prev)”<br>
I have mapped the issue with breakpoints to being during backward().</p>
<p>As I increase batch size it gets worse. At batch size of 1 it may run for almost 1K iterations before crushing.<br>
At batch size= 4 It usually happens with in the first 5 batches. every so often it will make it through 30 iterations before failing.</p>
<p>Anyone deal with something like this before?</p>

username: Dan_Erez
cooked: <p>looks like it has to do the memory footprint in some way. If I decrease the network size or the input size everything is fine. Even though in general the GPU is no where close to maxing out when looking at msi</p>

username: albanD
cooked: <p>Do you know how to use <code>gdb</code> to get the stack trace of where this happens?<br>
Otherwise, if you have a small code sample that reproduces the error, that would be very helpful.</p>

username: Dan_Erez
cooked: <p>I’m using pycharm in debug mode - which i believe uses dbg somewhere in there.<br>
I finally got things to work. I found that having very long sequences ( 2000+ time-steps ) caused the issue. It may have been due to my loader. I didn’t inspect it in depth since I found a solution.<br>
It doesn’t repeat itself on simple small models and I can’t really share my production architecture/training procedures.</p>
<p>if you sack a bunch of 1d convs so that you get a model in the 10’s of millions of parameters and try to run it on a small batch of a very long sequence you may get similar errors.</p>
<p>Thanks for getting back to me.</p>
How to use spawn function in torch.multiprocessing module

username: Neel_S
cooked: <p>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
<p>This is the error that I get when I want to use mutiple gpus to evaluate different models using multiprocessing.</p>

username: bapi
cooked: <p>Try <code>mp.set_start_method('spawn', force=True)</code> at your main; like the following:</p>
<pre><code class="lang-auto">if __name__ == '__main__':

    mp.set_start_method('spawn', force=True)

    main()
</code></pre>
Questions about Jit RNNs

username: G.M
cooked: <p>I’m recently interested in Touchscript, so I went to look at the custom LSTMs using Touchscript. In particular, I’m looking at this <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">file</a>. As a result, I have a few questions.<br>
<strong>1.</strong> I understand that combining the weights of a linear transformation are for speeding up the code. But why compute it separately for inputs and hiddens, as it is <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py#L108" rel="nofollow noopener">here</a>:<br>
<code>        hx, cx = state         gates = (torch.mm(input, self.weight_ih.t()) + self.bias_ih +                  torch.mm(hx, self.weight_hh.t()) + self.bias_hh)         ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)</code><br>
Can someone explain the benefit in comparison to first concatenating the tensors and compute together?<br>
**2.**Following the previous question, according to my understanding, the code mentioned above is trying to apply linear transformation separately to inputs and hiddens and add them up at last. However, 2 biases are added together, as:<br>
<code>        gates = (torch.mm(input, self.weight_ih.t()) + self.bias_ih +                  torch.mm(hx, self.weight_hh.t()) + self.bias_hh)</code><br>
I think the 2 biases can be replaced by a single one with the exact same effect, so I want to ask what’s the difference between adding 1 bias and 2 biases.<br>
**3.**If combing several operations can speed up the code, then instead of using: `        ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)</p>
<pre><code>    ingate = torch.sigmoid(ingate)
    forgetgate = torch.sigmoid(forgetgate)
    cellgate = torch.tanh(cellgate)
    outgate = torch.sigmoid(outgate)`
</code></pre>
<p>, why not just:<br>
<code>                 gates[:, 0:3*self.hidden_size].sigmoid_()  # Doesn't have to be in-place         ingate, forgetgate, outgate, cellgate = gates.chunk(4, 1)         cellgate=cellgate.tanh()</code><br>
Aren’t the operations in this way more “combined together”, or is it that combing operations doesn’t effect element-wise operations much?</p>
<p>Thanks for your time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
What is the difference between CrossEntropyLoss and HingeEmbeddingLoss

username: stuck
cooked: <p>Hi all,</p>
<p>I was reading the documentation of torch.nn and I look for a loss function that I can use on my dependency parsing task.</p>
<p>On some papers, the authors said the Hinge loss is a plausible one for the task. However, it seems the Cross Entropy is OK to use.</p>
<p>Also, for my implementation, Cross Entropy fits more than the Hinge. What should I do? What are the differences and pros/cons between these two loss function? And also, is the Hinge Embedding Loss is the same one with the regular Hinge loss that mentioned in <a href="https://medium.com/analytics-vidhya/understanding-loss-functions-hinge-loss-a0ff112b40a1" rel="nofollow noopener">here</a>?</p>
<p>Thank you and happy new year to all.</p>
Implemented negative sampling using scatter function

username: Jeeyung_Kim
cooked: <p>I want to implement negative sampling. The tensor  <code>x</code>  is very sparse, containing 1 or 0.<br>
I try to make 100 negative samples, and calculate the loss only using positive and 100 negative samples.</p>
<p>I included “sampling” when I calculate loss, and it made backpropagation impossible…</p>
<p>but I don’t know the way to fix it.</p>
<p>I got this error :<br>
<code>RuntimeError: Function ScatterBackward0 returned an invalid gradient at index 1 - got [5, 100] but expected shape compatible with [5, 16736]</code></p>
<p>I attach my snippet below.<br>
Thank you in advance</p>
<pre><code class="lang-auto">where_zeros = torch.zeros(x.shape).to(device)
pos_score = torch.where(x &gt; 0, recon_x, where_zeros)
false_score = torch.where(x == 0.0, recon_x, where_zeros)
for i in range(x.shape[0]):
    negs = false_score[i].nonzero().view(1,-1)
    # print(negs)
    weight = torch.ones(negs.shape).to(device)
    neg_samples = torch.multinomial(weight, 100)
    if i == 0:
        neg_ids = torch.tensor(neg_samples, dtype=torch.long).unsqueeze(0).to(device)
    else:
        neg_ids = torch.cat((neg_ids, torch.tensor(neg_samples, dtype=torch.long).unsqueeze(0).to(device)), 0)

neg_ids = neg_ids.squeeze()

neg_x = torch.zeros(x.shape).to(device).scatter(1, neg_ids, x)
neg_score = torch.zeros(x.shape).to(device).scatter(1, neg_ids, recon_x)
Recon_loss_pos = F.mse_loss(F.sigmoid(pos_score), x)
Recon_loss_neg = F.mse_loss(F.sigmoid(neg_score), neg_x)
Recon_loss = Recon_loss_neg + Recon_loss_pos
</code></pre>

username: yunqiu
cooked: <p>Same error, have you solved this problem?</p>
How to use `LayerNorm` and `RNNCell`?

username: zuoxingdong
cooked: <p>As the layer normalization is implemented, how could we use it with <code>*Cell</code> module ?</p>

username: G.M
cooked: <p>Unfortunately, you can’t. You have to implement it your self as the layer norm are usually applied before the activation of the gates.<br>
Fortunately, pytorch offers an Touchscript optimized implementation on Github. You can find it <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">here</a>.</p>

username: hadaev8
cooked: <p>Have you xample of use?</p>

username: G.M
cooked: <p>the link has some examples. Should be enough. Ask if u aren’t clear <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Custom LSTM cell implementation

username: MarvinMayson
cooked: <p>I would like to implement a custom version of the typical LSTM cell as it is implemented in Pytorch, say, change one of the activation functions at a gate. For this, I would like to see how the LSTM is implemented in Pytorch at the moment. I can find some code <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" rel="nofollow noopener">here</a>, but unfortunately, I cannot find the exact LSTM computations there etc.</p>
<p>Related posts can for example be found <a href="https://discuss.pytorch.org/t/how-to-modify-lstm/12004/12">here</a>, but all they delivered me is that nobody has found the LSTM cell code in github.</p>
<p>Where is the LSTM cell implemented in the Pytorch Github exactly?</p>

username: kangkang_li
cooked: <pre><code class="lang-auto">class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(LSTMCell, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.weight_ih = Parameter(torch.randn(4 * hidden_size, input_size))
        self.weight_hh = Parameter(torch.randn(4 * hidden_size, hidden_size))
        self.bias_ih = Parameter(torch.randn(4 * hidden_size))
        self.bias_hh = Parameter(torch.randn(4 * hidden_size))

    def forward(self, input, state):
        # type: (Tensor, Tuple[Tensor, Tensor]) -&gt; Tuple[Tensor, Tuple[Tensor, Tensor]]
        hx, cx = state
        gates = (torch.mm(input, self.weight_ih.t()) + self.bias_ih +
                 torch.mm(hx, self.weight_hh.t()) + self.bias_hh)
        ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)

        ingate = torch.sigmoid(ingate)
        forgetgate = torch.sigmoid(forgetgate)
        cellgate = torch.tanh(cellgate)
        outgate = torch.sigmoid(outgate)

        cy = (forgetgate * cx) + (ingate * cellgate)
        hy = outgate * torch.tanh(cy)

        return hy, cy
</code></pre>

username: G.M
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">This</a> is what you r looking for.</p>
Conan Build for Libtorch

username: forwardmeasure
cooked: <p>Hello all,<br>
I have been working on a Conan recipe for libtorch. I am almost there, but keep running into an issue with this error:</p>
<p>( "You found a build system error: AVX512F, AVX512DQ, AVX512VL " "is defined (via e.g. -mavx512f, -mavx512dq, and -mavx512vl) " “but CAFFE2_PERF_WITH_AVX512 is not defined.”)</p>
<p>The weird thing is that I can build libtorch from the command line using CMake, so there’s got to be something related to how things are working within the Conan apparatus. I’ve looked long and hard, but haven’t found what could be wrong, so decided to ask here.</p>
<p>The Conan recipe can be found here: <a href="https://github.com/forwardmeasure/conan/tree/master/recipes/conan-libtorch" rel="nofollow noopener">https://github.com/forwardmeasure/conan/tree/master/recipes/conan-libtorch</a></p>
<p>I am attempting to build V.1.3.1 on an Ubuntu 19.10 machine with Cuda turned off. I’d eventually like to get Cuda working as well, but first things first.</p>
<p>Any help would be appreciated!</p>
<p>Kind regards,<br>
Prashanth</p>

username: forwardmeasure
cooked: <p>Here’s a quick update. I’ve fixed the problem, and have a working installation of Libtorch under Conan. Please feel free to help yourselves to the Conan recipe if you need it!</p>
<p>Prashanth</p>
LayerNorm in LSTMCell

username: Seungyoung_Park
cooked: <p>I found that nn.LayerNorm is available in PyTorch 0.4.0.</p>
<p>Can anyone please tell me how to employ nn.LayerNorm in multi-layered LSTMCell ?</p>
<p>Now, I am using nn.LayerNorm as like below…</p>
<pre><code>self.lstm0 = nn.LSTMCell(input_size, hidden_size)
self.lstm1 = nn.LSTMCell(hidden_size, hidden_size)
self.lstm2 = nn.LSTMCell(hidden_size, hidden_size)
self.ln1 = nn.LayerNorm(hidden_size)
self.ln2 = nn.LayerNorm(hidden_size)
....

h_t0, c_t0 = self.lstm0(input, (h_t0, c_t0))
h_t1, c_t1 = self.lstm1(self.ln1(h_t0), (h_t1, c_t1))
h_t2, c_t2 = self.lstm1(self.ln2(h_t1), (h_t2, c_t2))
</code></pre>
<p>Thanks,<br>
Seungyoung</p>

username: G.M
cooked: <p>You have to implement it yourself. Take a look <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">here</a></p>
Correlation NN or Convolution NN

username: zyc
cooked: <p>Hi,</p>
<p>I learned that <a href="https://discuss.pytorch.org/t/is-the-conv2d-implementation-in-pytorch-correlation-mathematically/37459">Conv2d is actually a cross-correlation operator</a>, hence, does that means our network is actually a Correlation NN instead of Convolution NN?</p>
<p>I can understand the difference between Correlation and Convolution is just a rotation of 180. However, what does it actually means when it comes to CNN? What if I change from convolution with correlation?</p>

username: G.M
cooked: <p>Well, we just call it “Convolution NN”.</p>
<p>According to my understanding(which could be wrong), whether you rotate the weights or not doesn’t matter(assume that after the rotation, it is still possible to use matrix multiplication) because to the NN, it is just weight.</p>
How to type a dynamically assigned TorchScript module to be consumed by a TorchScript function/method?

username: Enamex
cooked: <p>Example:</p>
<pre><code class="lang-auto">@jit.script
class Foo:
    ...
@jit.script
class Quux:
    ...
@jit.script
class Bar:
    def __init__(self, cell):
        # cell may be either a Foo or a Quux creator
        self.cell1 = cell()
        self.cell2 = cell()
    def forward(self, x):
        self.sub(x, self.cell1)
        self.sub(x, self.cell2)
    def sub(self, x: Tensor, cell: ?????):
        ...
</code></pre>
The backpropagation in end-to-end

username: WormWakedEarlyEatenB
cooked: <p>I’m confused about the backpropagation in end-to-end experiments. I replace the layer fc of Resnet50 with a CNN classifier. The input of CNN is a feature-map with size 100x2048, each row of the feature-map is generated by Resnet50 and they are just simply concatenated to form a feature-map, which means that several outputs of Resnet50 form one input of CNN classifier. So is it possible to backpropagate gradients from CNN classifier to Resnet50 when the loss function is based on the prediction of CNN classifier?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you don’t change the setting of the resnet50 from torchvision, then its weights will require gradients and so when you call <code>.backward()</code> on your loss, all the <code>.grad</code> field in the resnet will be populated and you can perform optimizer steps on it.</p>

username: WormWakedEarlyEatenB
cooked: <pre><code class="lang-auto">def finetuneTrain(model, CNNModel, device, train_loader, optimizer, epoch):
    model.train()
    correct = 0
    for (data, target) in train_loader:
        target = target.to(device)
        features = []
        for volume_count in range(len(data)):
            volume = torch.Tensor(data[volume_count])
            volume = volume.to(device)
            feature = model(volume).cpu().detach().numpy()
            tmp = np.zeros((100, feature.shape[1]), dtype=np.float32)
            volume_size = feature.shape[0]
            tmp[:volume_size, :] = feature
            features.append(tmp)

        optimizer.zero_grad()
        features = torch.Tensor(features).to(device)
        output = CNNModel(features)
        loss = F.cross_entropy(output, target)
        loss.backward()
        for name, param in model.named_parameters():
            if 'layer4' in name or 'layer3' in name:
                print(param.grad)
        optimizer.step()
</code></pre>
<p>Here is my training function, but the output is all None, I think that the backpropagation failed.</p>

username: albanD
cooked: <p>You are explicitly calling <code>.detach()</code> on the output of the <code>model</code>. So you explicitly break the history as no gradient will be computed beyond the detach operation. You will need to remove it (and remove the use of numpy Tensors as we cannot differentiate through these operations, you need to use only Tensors).</p>

username: WormWakedEarlyEatenB
cooked: <p>I just followed the suggestion of PyCharm and didn’t understand <code>.detach()</code> well. I’ll try as your suggestion.</p>
Register_hook throwing an error during model.eval()

username: Umair_Javaid
cooked: <p>I have a custom module, which I am adding after VGG-16 feature module</p>
<pre><code class="lang-auto">class Last_layers_mod(nn.Module):
  def __init__(self):
    super(Last_layers_mod, self).__init__()  
    self.gap = nn.AdaptiveAvgPool2d((1,1))

  def forward(self, x):
    if self.train:
      x.register_hook(lambda x: print("gap reg_hook: ", x.size()))
    x = self.gap(x)
    
    x = x.view(-1, self.num_flat_features(x))
    return x

  def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features
</code></pre>
<p>The model works perfectly during training but as soon as the first epoch ends and the model is in model.eval() phase, it throws</p>
<blockquote>
<p>RuntimeError: cannot register a hook on a tensor that doesn’t require gradient</p>
</blockquote>
<p>How to fix this issue?</p>
user_suspended: yes

username: albanD
cooked: <p>You can replace <code>if self.train:</code> by <code>if self.train and x.requires_grad:</code></p>
How detach() affect to the model?

username: oasjd7
cooked: <pre><code class="lang-auto">Network flow:
[Model_1] -&gt; [Model_2] -&gt; [Model_3]

optimizer1 = {Model_1 + Model_2}
optimizer2 = {Model_3}
</code></pre>
<p>When I train with optimizer2,<br>
If I don’t use detach() to input of model_3, this can affect to Model_1 and Model2 ???<br>
<code>Model_3(input.detach()) or Model_3(input)</code></p>

username: JuanFMontesinos
cooked: <p>it depends on were input comes from. If input is the output of another model it will break backprop . If it’s just a tensor nothing happens.</p>

username: Eta_C
cooked: <p>Copied from <a href="https://discuss.pytorch.org/t/what-the-difference-between-detach-detach-and-with-torch-no-grad-in-our-training-phase/64390">this</a> topic.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="64390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/what-the-difference-between-detach-detach-and-with-torch-no-grad-in-our-training-phase/64390/2">What the difference between detach(), detach_(),and with torch.no_grad() in our training phase?</a></div>
<blockquote>
<p><code>detach()</code> operates on a tensor and returns the same tensor, which will be detached from the computation graph at this point, so that the backward pass will stop at this point.<br>
<code>detach_()</code> is the inplace operation of <code>detach()</code> .</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="64390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/what-the-difference-between-detach-detach-and-with-torch-no-grad-in-our-training-phase/64390/4">What the difference between detach(), detach_(),and with torch.no_grad() in our training phase?</a></div>
<blockquote>
<p>They might yield the same result, but in specific edge cases you might get different and unexpected results:</p>
<ul>
<li>An optimizer, which doesn’t hold certain parameters, won’t update them. However other optimizers could still update these parameters, if they are passed.</li>
<li>If you are using weight decay, have updated some parameters in the past and are now freezing them, these parameters could still be updated, if they were passed to the optimizer.</li>
<li><code>detach()</code> could act in the same way as the second point.</li>
</ul>
<p>That being said, I would prefer the explicit way instead of relying implicitly that no side effects can occur. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
MPI backend cannot work!

username: lkywk
cooked: <p>I use the MPI as the backend in distributed pytorch among multiple nodes.</p>
<p>exec code:<br>
mpirun --hostfile hostfile -n 2 python *.py</p>
<p>and I revised the init line which looks like init line in pytorch distributed_test.py file ,</p>
<p>the init command line <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
os.environ[‘MASTER_ADDR’] = '172.31.7.117’<br>
os.environ[‘MASTER_PORT’] = ‘23456’</p>
<pre><code>dist.init_process_group(init_method='env://', backend='mpi')
group = dist.new_group([i for i in range(dist.get_world_size())]) 
</code></pre>
<p>I am sure that openmpi is installed correctly. Pytorch is build from source with MPI support.</p>
<p>I really appreciate any help.</p>
Torchtext textclasification with custom tokenizer

username: yonigottesman
cooked: <p>When I use the IMDB dataset I use TEXT,LABEL fields to specify how to tokenize and preprocess the data. Then i build vocab and i can choose min_freq and max_size.<br>
How can i do all of the above when using YelpReviewPolarity for example?<br>
Looking at the code and tutorial it seems i cannot change the tokenizer.<br>
Also, how can i use BucketIterator with YelpReviewPolarity dataset?</p>

username: travis-harper
cooked: <p>You’re in luck! Torchtext does allow you to use a custom tokenizer:</p>
<pre><code class="lang-auto">from torchtext.data import Field

def custom_tokenizer(text):
    return [token.text for token in nlp.tokenizer(text)]

TEXT = Field(sequential=True, tokenize=custom_tokenizer)
LABEL = Field(sequential=False)
</code></pre>
<p>It looks like the YelpReviewPolarity dataset is formatted in CSV. The easiest way to parse it would be with the <code>TabularDataset</code> class:</p>
<pre><code class="lang-auto">from torchtext.data import TabularDataset

train_td, test_td = TabularDataset.splits(
    path='path-to-yelp-review-polarity-data/', train='train.csv', test='test.csv',
    format='csv', skip_header=False, fields=[('text', TEXT), ('label', LABEL)])

TEXT.build_vocab(train_td)
text_vocab = TEXT.vocab
LABEL.build_vocab(train_td)
label_vocab = LABEL.vocab
</code></pre>
<p>I haven’t used the <code>BucketIterator</code> but it’s not too different from the standard <code>Iterator</code>. I haven’t tested this but try:</p>
<pre><code class="lang-auto">from torchtext.data import BucketIterator

train_iter, test_iter = BucketIterator.splits(
    (train_td, test_td),
    sort_key=lambda x: len(x.text),
    shuffle=True,
    sort_within_batch=False,
    batch_sizes=(batch_size, test_batch_size),
    device=device)
</code></pre>
<p>You could use my authorship attribution training notebook as a reference to see how it all comes together: <a href="https://github.com/travis-harper/authorship-attribution/blob/master/gutenberg-10-author-gru-v5-training.ipynb" rel="nofollow noopener">https://github.com/travis-harper/authorship-attribution/blob/master/gutenberg-10-author-gru-v5-training.ipynb</a></p>
<p>All I’m doing here is feeding words into an RNN to make a categorical prediction. This isn’t the best way to perform authorship attribution, but it does use the torchtext tools you are interested in. I hope this helps!</p>

username: yonigottesman
cooked: <p>Oh ok thanks, I can use this standard way of opening a custom csv files…<br>
I was wondering why the API is different between the IMDB and the TextClassification classes.<br>
The IMDB works with fields and TextClassification doesnt.</p>
Pooling over PackedSequence

username: knowledge_unlimited
cooked: <p>Hi,</p>
<p>I have the following network:</p>
<blockquote>
<p>input = torch.nn.utils.rnn.pack_padded_sequence(input,sequenceLengths,batch_first=True)<br>
output, hidden = GRU(input,hiddenLayer)<br>
output, seqLengths = torch.nn.utils.rnn.pad_packed_sequence(output,batch_first=True)</p>
</blockquote>
<p>How do I do max pooling over output as sequence length is variable? Assuming output is of dimension N x L x hiddenLayerDim. N = batchSize, L=length of longest sequence.</p>
<p>I would like to do max pooling along the length (L) dim (=1).</p>

username: whr94621
cooked: <p>I think you should use <code>nn.functional.max_pool1d</code></p>

username: knowledge_unlimited
cooked: <p>But the sequence length is different across different slices of the batch. The values in the sequence may be negative, so I don’t want to pick up padded zeros as max while simply doing maxpool1d</p>

username: cerisara
cooked: <p>One option would be to pad with float("-Inf") instead of padding with 0, so that the max operation will never return padding.</p>

username: vdw
cooked: <p>You might want to consider creating batches where all sequences within one batch have the same length. No need for padding or packing; <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">see this older post</a>. It’s simply convenient, and I use it all the time.</p>
Training stops without any error in loss.backward() while using GPU

username: timoth-y
cooked: <p>Trying to use Cuda to train model. Already passed model and input to cuda device:</p>
<pre><code class="lang-auto">if use_gpu: 
	self.net.cuda();
criterion = nn.CrossEntropyLoss();
optimizer = optim.Adam(self.net.parameters(), lr=1e-3);
</code></pre>
<pre><code class="lang-auto">inputs, labels = data;
if use_gpu:
	inputs = inputs.cuda();
	labels = labels.cuda();
</code></pre>
<p>But when it executes loss.backward() it immediately crashes without any error.</p>
<pre><code class="lang-auto">outputs = self.net(inputs);
loss = criterion(outputs, labels);
loss.backward();
</code></pre>
<p>I am using pytorch version 1.3.1+cu92 with NVIDIA GeForce RTX 2060<br>
Already tried to install cuda driver.<br>
Any tips?</p>

username: ptrblck
cooked: <p>Could you try to use CUDA10 with your Turing card, as older CUDA versions might not work properly with your GPU.<br>
Also, you should have received a warning regarding this mismatch, but maybe it was hidden by the crash. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: timoth-y
cooked: <p>The problem was in mismatch of CUDA and Pytorch versions. I had Pytorch -v: 1.3.1+cu92.<br>
However, when I tried run training with installed CUDA 9.2 I’ve had the similar problem.<br>
It seems like Turing cards is only supported by CUDA 10.X:</p>
<blockquote>
<p>CUDA 10 is the first version of CUDA to support the new NVIDIA Turing architecture<br>
<a href="https://devblogs.nvidia.com/cuda-10-features-revealed/" class="inline-onebox" rel="noopener nofollow ugc">CUDA 10 Features Revealed: Turing, CUDA Graphs, and More | NVIDIA Technical Blog</a></p>
</blockquote>
<p>With the latest version of CUDA and Pytorch everything works correctly and most importantly fast!<br>
Thanks</p>
Working Example of a Matlab-Libtorch Interface

username: Haydnspass
cooked: <p>Hey,</p>
<p>I wrote some sample code to interface MATLAB - PyTorch/Libtorch by the use of C++ and Libtorch.<br>
I did it because for our application it was not an option to write to binary and load it (because of performance issues).<br>
You can find the sample code here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Haydnspass/PytorchMatlabBind" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/16120273?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Haydnspass/PytorchMatlabBind" target="_blank" rel="nofollow noopener">Haydnspass/PytorchMatlabBind</a></h3>

<p>This repo shows a minimal and still alpha-version example of how to interface Matlab and PyTorch. - Haydnspass/PytorchMatlabBind</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>As I am rather new to C++, the code is by no means clean, but it (should) work.<br>
The CMakeLists.txt builds an arbitrary shared dummy library as well as the actual interface which will be a .mex file for the use in MATLAB.</p>
<p>Cheers,<br>
Lucas</p>

username: Papillon
cooked: <p>Nice work and really useful. I had troubles on Mac OS x when using find_package(Matlab) in Cmake. I changed one line in your CmakeLists.txt and it worked flawless:</p>
<p>find_package(Matlab COMPONENTS MX_LIBRARY ENG_LIBRARY)</p>
<p>It is also possible to specify the Torch path directly if someone meets issues when including the Pytorch library:</p>
<p>find_package(Torch REQUIRED PATHS […YOUR PATH…]/libtorch)</p>
PyTorch 1.3.1 build fails on C++ error

username: marcjoos-phd
cooked: <p>Dear all,</p>
<p>I try to build from source PyTorch 1.3.1. I gave a try both with gcc (from my system (RHEL 7), so gcc 4.8.5) and with a recent Intel compiler (icc 2019.0.5.281 that is more or less the latest release of Intel 2019 compiler) and I run in the same error.</p>
<p>Here is the compilation error I get:</p>
<pre><code class="lang-auto">/usr/include/c++/4.8.2/functional:2450:7: error: ‘std::function&lt;_Res(_ArgTypes ...)&gt;::function(_Functor) [with _Functor = caffe2::OnnxifiOp&lt;Context&gt;::buildBackendAndGraph(caffe2::Workspace*, const std::vector&lt;long unsigned int&gt;&amp;, const string&amp;) [with Context = caffe2::CPUContext; std::string = std::basic_string&lt;char&gt;]::__lambda27; &lt;template-parameter-2-2&gt; = void; _Res = std::shared_ptr&lt;caffe2::onnx::BackendGraphInfo&gt;; _ArgTypes = {}]’, declared using local type ‘caffe2::OnnxifiOp&lt;Context&gt;::buildBackendAndGraph(caffe2::Workspace*, const std::vector&lt;long unsigned int&gt;&amp;, const string&amp;) [with Context = caffe2::CPUContext; std::string = std::basic_string&lt;char&gt;]::__lambda27’, is used but never defined [-fpermissive]
       function&lt;_Res(_ArgTypes...)&gt;::
</code></pre>
<p>I build with the following command line:</p>
<pre><code class="lang-auto">BLAS=MKL python3 setup.py bdist_wheel
</code></pre>
<p>(Well, Actually it is a bit more complicated because for some reason it fails to detect the MKL library, so I launch the following, that crashes, then I go in <code>build/</code> directory and launch the Cmake command line provided by <code>setup.py</code> adding <code>-DINTEL_MKL_DIR=$MKLROOT</code> with <code>$MKLROOT</code> the root directory of my MKL installation, but in the end it does the same.)</p>
<p>Few months ago I was able to build (also from source) PyTorch 1.2.0 without any major issue on the same system (and with gcc compiler from my RHEL 7 system).</p>
<p>Any help would be appreciated!</p>

username: marcjoos-phd
cooked: <p>A quick add: I realised I had an inconsistency (the libc used above is from 4.8.2) so I corrected it (to use the right libc) but it changed nothing.</p>
Save Memory for Multiple Forward Passes?

username: k2lin
cooked: <p>Hi</p>
<p>I am trying to train a network that would be used to run on three different images and thus generate three outputs. The resulting outputs are combined together in some way later to calculate the loss.<br>
The issue is that since the model is quite huge, running it three times might cause OOM errors for GPU.<br>
I was using <code>torch.no_grad()</code> to inference two of the images and only allow the gradient backprop through the last image.<br>
Is there a better way to do so?<br>
I was told it’s better to let the gradient go through all three images.<br>
Thanks.</p>

username: albanD
cooked: <p>If you have very big memory issues, you can use the <a href="https://pytorch.org/docs/stable/checkpoint.html#torch-utils-checkpoint">checkpoint</a> module to reduce the memory usage (at the cost of more compute).</p>

username: k2lin
cooked: <p>Thank you.<br>
Another question, is the issue mentioned in this repo fixed in 1.3?<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/csrhddlam/pytorch-checkpoint" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/16692643?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/csrhddlam/pytorch-checkpoint" target="_blank" rel="nofollow noopener">csrhddlam/pytorch-checkpoint</a></h3>

<p>Contribute to csrhddlam/pytorch-checkpoint development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Essentially, the checkpoint function would be slower on multi-GPUs when using nn.DataParallel.</p>

username: albanD
cooked: <p>I am not familiar with that repo. You might want to double check. I am not sure what the difference between the implementations is.</p>
How to initialize Variables in PyTorch?

username: Kong
cooked: <p>How do I initialize a layer if it is not constructed using nn.Linear for example?</p>
<blockquote>
<p>self.nlc =Variable(torch.randn(10,1,1,1,1).type(torch.FloatTensor), requires_grad=True)</p>
</blockquote>
<p>Is there like a method below?</p>
<blockquote>
<p>self.nlc =Variable(torch.glorot(10,1,1,1,1).type(torch.FloatTensor), requires_grad=True)</p>
</blockquote>

username: ptrblck
cooked: <p><code>Variables</code> are deprecated since version <code>0.4.0</code>, so you can just use tensors not (ans set <code>requires_grad=True</code> in the initialization).<br>
<a href="https://pytorch.org/docs/stable/nn.init.html"><code>torch.nn.init</code></a> provides various methods to initialize your parameters. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Kong
cooked: <p>Sorry, but could you give an example? Do you create an empty tensor then initialize it like below? Is it correct?</p>
<pre><code class="lang-auto">A = torch.empty(5, 7, requires_grad=True).type(torch.cuda.FloatTensor)
torch.nn.init.normal_(A)
</code></pre>

username: ptrblck
cooked: <p>Your example is correct.<br>
Note that you should wrap this tensor in <code>nn.Parameter</code>, if you would like to optimize it inside an <code>nn.Module</code>. <code>nn.Parameters</code> will be automatically registered inside modules, if you use assign them as attributes:</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        A = torch.empty(5, 7, device='cpu')
        self.A = nn.Parameter(A)
        
    def forward(self, x):
        return x * self.A

module = MyModule()
print(dict(module.named_parameters()))
&gt; {'A': Parameter containing:
tensor([[-7.8389e-37,  3.0623e-41, -7.8627e-37,  3.0623e-41,  1.1210e-43,
          0.0000e+00,  8.9683e-44],
        [ 0.0000e+00, -7.8579e-37,  3.0623e-41,  1.4013e-45,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
          0.0000e+00, -7.7193e-37],
        [ 3.0623e-41,  1.8077e-43,  0.0000e+00,  4.7530e-06,  4.5845e-41,
         -7.8459e-38,  3.0623e-41],
        [ 0.0000e+00,  0.0000e+00,  1.3593e-43,  0.0000e+00, -7.9340e-37,
          3.0623e-41, -7.8739e-37]], requires_grad=True)}
</code></pre>
<p>By wrapping them in <code>nn.Parameter</code>, the <code>requires_gradient</code> attribute will be set to <code>True</code> by default.</p>
<p>Let me know, if you have more questions or something is unclear.</p>

username: Kong
cooked: <p>That is much better thank you very much !</p>
How to normalize multidimensional tensor?

username: akashs
cooked: <p>Is it possible to extend/apply the transforms.Normalize to normalize multidimensional tensor in custom pytroch dataset class? I have a tensor with shape <code>(S x C x A x W)</code> and I want to normalize on C dimension, here S: sequence length, C: feature channel, A: feature attributes, W: window length of each sub-sequence data.</p>

username: ptrblck
cooked: <p>You could apply the normalization manually to avoid the dimension check:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 4, 5)
mean = torch.tensor([0.5, 0.5, 0.5])
std = torch.tensor([0.5, 0.5, 0.5])
x.sub_(mean[None, :, None, None]).div_(std[None, :, None, None])
</code></pre>
<p><code>torchvision.transforms.functional.normalize</code> uses a similar code as seen <a href="https://github.com/pytorch/vision/blob/d2c763e14efe57e4bf3ebf916ec243ce8ce3315c/torchvision/transforms/functional.py#L218">here</a>.</p>

username: akashs
cooked: <p>Thanks a lot for the quick reply and concise solution <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"><br>
I have a naive confusion. I think this approach doesn’t guarantee that the data will be in [-1,+1] range. We should have to set the mean and std manually. am I right?<br>
Is there any quick and concise approach(like this one) to normalize the multidimensional tensor to the normal distribution?</p>

username: akashs
cooked: <p>I found another <a href="https://stackoverflow.com/a/47408301/1596703" rel="nofollow noopener">solution</a></p>
<pre><code class="lang-auto">import torch.nn.functional as f
f.normalize(input, p=2, dim=2)
</code></pre>
<p>But I am not sure whether this approach will normalize data on a normal distribution.</p>

username: ptrblck
cooked: <p>My code snippet will standardize the data (also knows as z-scores), so that it will have a zero mean and unit variance. I’m not sure I understand your use case completely. Would you like to pass any distribution to the normalization method and get a normal distribution as the output?</p>
How to increase the CNN's inference time

username: crissallan
cooked: <p>Hello, I was recently using CNN(HRNet) for facial landmarks detection, and i will send those landmarks info to a web server. I met a bottleneck of the model’s efficiency. The inference time for each image on GPU(GTX 2080) is about 80ms. After take the preprocessing and io time into account, the time for getting the result of one image is about 170ms.<br>
Should I use multi-thread to accelerate the process? I mean could I use a thread to do inference process(use GPU infrence more images in a batch), and use another thread to send the result landmarks to server.<br>
Or Should I turn to use Tensor RT? I heard it is a more faster approach.</p>
After the training, the model is saved and then loaded, and the accuracy is inconsistent

username: CF-chen-feng-CF
cooked: <p>The result of the direct test after the model training is correct, but the model is saved as .pt after training, and the model after reloading the .pt is inaccurate.<br>
Libtorch version 1.1.0</p>

username: xiaoliu_xiaoliu
cooked: <aside class="quote no-group quote-modified" data-username="CF-chen-feng-CF" data-post="1" data-topic="59091">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> CF-chen-feng-CF:</div>
<blockquote>
<p>Libtorch版本1.1.0</p>
</blockquote>
</aside>
<p>Same problem happened on my case. Training and Directly Inference get a normal and correct result, however, Training-Save and Load-Inference get a innormal and competely wrong result.<br>
Any HELP?</p>
<p>Here is my flow path of my code, Step 6 output is same but Step 7 not and competely different:<br>
Step 1 Model Initialization<br>
Step 2 Model Training(printf:output result with an all-ones intput at last training step)<br>
Step 3 torch::save(model,“D:\path\to\model.pt”);<br>
Step 4 Initialization model2<br>
Step 5 torch::load(model2,“D:\path\to\model.pt”);<br>
Step 6 std::cout&lt;&lt;model.parameters()[169]&lt;&lt;std::endl;<br>
std::cout&lt;&lt;model2.parameters()[169]&lt;&lt;std::endl;<br>
Step 7 torch::Tensor t1=model-&gt;forward(torch::ones(1,1,224,224).to(torch::kCUDA));<br>
torch::Tensor t2=model2-&gt;forward(torch::ones(1,1,224,224).to(torch::kCUDA));</p>

username: xiaoliu_xiaoliu
cooked: <p>Same problem happened on my case at Libtorch1.1.0<br>
Training and Directly Inference get a normal and correct result, however, Training-Save and Load-Inference get a innormal and competely wrong result.<br>
Any HELP?</p>
<p>Here is my flow path of my code, Step 6 output is same but Step 7 not and competely different:<br>
Step 1 Model Initialization<br>
Step 2 Model Training(printf:output result with an all-ones intput at last training step)<br>
Step 3 torch::save(model,“D:\path\to\model.pt”);<br>
Step 4 Initialization model2<br>
Step 5 torch::load(model2,“D:\path\to\model.pt”);<br>
Step 6 std::cout&lt;&lt;model.parameters()[169]&lt;&lt;std::endl;<br>
std::cout&lt;&lt;model2.parameters()[169]&lt;&lt;std::endl;<br>
Step 7 torch::Tensor t1=model-&gt;forward(torch::ones(1,1,224,224).to(torch::kCUDA));<br>
torch::Tensor t2=model2-&gt;forward(torch::ones(1,1,224,224).to(torch::kCUDA));</p>

username: yf225
cooked: <p><a class="mention" href="/u/cf-chen-feng-cf">@CF-chen-feng-CF</a> <a class="mention" href="/u/xiaoliu_xiaoliu">@xiaoliu_xiaoliu</a> Could you share with us the full code to reproduce this issue?</p>

username: xiaoliu_xiaoliu
cooked: <p>Solved！Just overlook of using .eval() .<a class="mention" href="/u/yf225">@yf225</a></p>
Output of nn.Linear as robust class score

username: manurare
cooked: <p>Hi!. I’m trying to come up with an alternative metric to measure the robustness of my classifier apart from the typical ones as F1, accuracy, recall etc.</p>
<p>The dataset is always the same and the model under evaluation is resnet-18. I want to perform a grid search of different learning rates, optimizers etc and check which configuration gives me the most robust classification. The last layer of the resnet-18 is a nn.linear with two outputs. I was thinking if the cumulative sum of each of the nn.linear outputs it is a good way to represent the classification robustness. In other words, imagine that the nn.linear outputs represent dog and cat prediction values. If I sum all the dog values and all the cat values is the same as saying the higher the dog sum value the strongest my classifier into detecting less dog false positives? I thought this would be interesting if for example you have a binary classification of tumor/not tumor you do not care if you are a bit lax onto allowing tumor false positives but you would not want to predict “not tumor” false positives. I know there are metrics which can more or less measure the robustness but Do you think this method is useful?</p>
Nan loss for each epoch

username: rim_zayani
cooked: <p>Hello,</p>
<p>I’m training a model composed of two fully connected layers with relus.<br>
But I get a loss value of nan for each epoch (I’m using MSE loss).<br>
I printed the weights and bias and I found some nan values.<br>
Do you have any idea how can I resolve this problem?<br>
Here is some examples of input signal I’m using:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce5e2f455626e64f1174aa32d37121c263b5035a.png" alt="sig1" data-base62-sha1="trC9dd7bnJZv2W44kQKzHv9TBiO" width="295" height="154"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88e3ff98b7ad33c3fa06611abf44311dc5caa60f.png" alt="sig2" data-base62-sha1="jwZq3PzVCNia4JbdjFehhmY5FIH" width="295" height="157"><br>
Thanks in advance.</p>

username: ptrblck
cooked: <p>Could you check the input tensors for NaN or Inf values?<br>
These values would create a NaN loss.<br>
Also, how high is your learning rate?<br>
Do you see the NaN loss right in the first batch?</p>

username: rim_zayani
cooked: <p>I checked, the input tensors don’t have nan or inf values. It is composed of sparse signals.<br>
I tried a learning rate between 1e-3 and 1e-5, but I get the same results.<br>
Yes the nan loss begins from the first batch.<br>
I’m using a training dataset of 5000 signals.<br>
I noticed that when I decrease the number of signals to 50, I don’t get nan losses.</p>

username: ptrblck
cooked: <p>Are you normalizing the input data or what value range are you currently dealing with? Based on your information you might create an overflow somewhere.</p>

username: rim_zayani
cooked: <p>No i’m not normalizing the input data because it’s already between 0 and 1.</p>
Elementwise kernel

username: mahmoodn
cooked: <p>Using profiler I see that this kernel is in the top important kernels affecting gpu time.</p>
<pre><code class="lang-auto">void at::native::elementwise_kernel&lt;512, 1, at::native::gpu_kernel_impl&lt;at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1}&gt;(at::TensorIterator&amp;, at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1} const&amp;)::{lambda(int)#2}&gt;(int, at::native::gpu_kernel_impl&lt;at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1}&gt;(at::TensorIterator&amp;, at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1} const&amp;)::{lambda(int)#2})
</code></pre>
<p>Looking at the code I see,</p>
<pre><code class="lang-auto">template&lt;int nt, int vt, typename func_t&gt;
C10_LAUNCH_BOUNDS_2(nt, launch_bound2)
__global__ void elementwise_kernel(int N, func_t f) {
  int tid = threadIdx.x;
  int nv = nt * vt;
  int idx = nv * blockIdx.x + tid;
  #pragma unroll
  for (int i = 0; i &lt; vt; i++) {
    if (idx &lt; N) {
      f(idx);
      idx += nt;
    }
  }
}
</code></pre>
<p>So what is important there is the f(). I think it is <code>add_kernel_cuda</code>. Am I right?</p>