akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Autograd function for a given model architecture

username: Juls
cooked: <p>Dear all,</p>
<p>if we fix the model architecture, is it possible to extract the actual function that computes the gradient of the loss wrt the model’s parameters?</p>
<p>It can be interesting for learning purposes and also enable fast porting of the python code to FPGAs using HLS.</p>
<p>It can be interesting</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check this tool <a href="https://github.com/szagoruyko/pytorchviz/">https://github.com/szagoruyko/pytorchviz/</a> to be able to see the graph that corresponds to the backward. But this won’t tell you what are the actual mathematical operations that are performed I’m afraid.</p>

username: Juls
cooked: <p>Thanks for your reply, Alban.</p>
<p>Do you know if there’s a tool to extract the whole graph to be able to recode the backward pass as a C++ function?</p>

username: albanD
cooked: <p>Hi,</p>
<p>No there is no such tool I’m afraid.<br>
Most of the backward functions are not even exposed as user APIs.</p>

username: Juls
cooked: <p>I see, thank you anyway.</p>
Basic_english and spacy tokenizer

username: gnadaf
cooked: <p>On some of the systems, I need to replace “basic_english” with “spacy”, it works.</p>
<p>What is the difference between these  basic_english and spacy tokenizer?</p>

username: BramVanroy
cooked: <p>Why do you need to replace it with spaCy? What does not work correctly for you? The basic_english tokenizer should work as long as your language is English. It’s in the name. It will do a very basic string normalization and then splitting by whitespace. If you use other parsers, such as <a href="https://spacy.io/api/tokenizer" rel="noopener nofollow ugc">spaCy</a>, that library will be used to tokenize the text.</p>
How do gradients accumulate when using different batch sizes on one GPU and multiple GPUS?

username: feiyuhuahuo
cooked: <p>I have two situations:</p>
<ol>
<li>If I use batch size 8 on one GPU,  the gradients of the first 4 samples and the gradients of the last 4 samples are summed together, right?</li>
<li>Then I am training on two GPUs with Distributed Data Parallel (batch size is 4 per GPU, 8 for total). I read from PyTorch Documents that gradients are averaged when synchronizing. So the gradients of the first 4 samples and the gradients of the last 4 samples are averaged, right?</li>
</ol>
<p>Am I right for these two situations?<br>
Besides, can I sum the gradients when synchronizing just as the way training on one GPU does? Is there a parameter to control this when using Distributed Data Parallel?</p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>Yes they are all summed</li>
<li>You are they are averaged accross GPU only (not within). So for 2 GPU, you will get the sum/2.</li>
</ol>
<p>I don’t think DDP has an option to disable this averaging. You can multiply the final gradients by the number of devices if you need the sum.</p>

username: feiyuhuahuo
cooked: <p>Hi,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b4ce543cfc6682d49a1d96ee7630b32d3a11804.png" alt="image" data-base62-sha1="aK8yC0jIr7IJb5SjuMBriBlzyRe" width="479" height="82"><br>
It’s not easy to manage the gradients of all parameters. So if I need to sum the gradients across 2 GPUs, do you think that adding <code>loss_total*=2</code> below 'loss_total=loss_b + loss_m + loss_c + loss_s` can achieve this effect?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes scaling the loss by 2 will also scale all the gradients by 2 !</p>
Batch wise beam search decoding

username: Hari_Krishnan
cooked: <p>I am using the following code for implementing beam search for text generation.</p>
<pre><code class="lang-auto">def beam_search_decoder(data, k):
	sequences = [[list(), 0.0]]
	# walk over each step in sequence
	for row in data:
		all_candidates = list()
		# expand each current candidate
		for i in range(len(sequences)):
			seq, score = sequences[i]
			for j in range(len(row)):
				candidate = [seq + [j], score - torch.log(row[j])]
				all_candidates.append(candidate)
		# order all candidates by score
		ordered = sorted(all_candidates, key=lambda tup:tup[1])
		# select k best
		sequences = ordered[:k]
	return sequences
</code></pre>
<p>This performs beam search decoding for batch_size 1. I am working with a vocab_size of ~9900, so this function itself is extremely slow. Is there an alternative faster way to do batch-wise beam search ?</p>
What is maximum ammount of train steps in adam?

username: hadaev8
cooked: <p>The number of steps is param of calculations in adam.<br>
Does it mean with some step, updates would be around zero, or im wrong?</p>

username: fadetoblack
cooked: <p>Hi, can you please elaborate your query a little bit? Are you referring to torch.optim.Adam?</p>

username: hadaev8
cooked: <p>Yes, this one<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html#Adam" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html#Adam" target="_blank" rel="noopener nofollow ugc">torch.optim.adam — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
2 values depend on state[‘step’] var.<br>
bias_correction1 change lr at the start of training. Then it always will be 1.<br>
Im not sure about bias_correction2.
Augmentations to only one class

username: SU801T
cooked: <p>Hi,</p>
<p>I’m not sure if I have correctly implemented applying augmentations to only one class during training (binary classes). The code runs however, I’m just unsure if the augmentations is actually happening during the get item function. Would anyone be able to tell me if this would cause any potential issues (for example memory or speed)? :</p>
<pre><code class="lang-auto"> if self.phase == 'train':
            self.transforms = T.Compose([T.RandomResizedCrop(224),
                                    T.ToTensor(),
                                    T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) 
            self.transforms_augs = T.Compose([T.RandomResizedCrop(224),
                                    T.RandomHorizontalFlip(),
                                    T.RandomVerticalFlip(),
                                    T.ToTensor(),
                                    T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) 

        else:
            self.transforms = T.Compose([T.RandomResizedCrop(224),T.ToTensor(),
                                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])


    def __getitem__(self, index):
        if self.phase == 'train':
            path = self.imgs[index]
            label = int(path.split('/')[11])
            data = Image.open(path).convert('RGB')
            if label == 1:
                data = self.transforms_augs(data)
            else:
                data = self.transforms(data)

</code></pre>
<p>Cheers,</p>
<p>T</p>

username: Caruso
cooked: <p>Hi,</p>
<p>for me your augmentation looks good. You could visualize your input data by denormalizing it first, converting it from [B, C, H, W] to [B, H, W, C] using <code>torch.permute()</code>, maybe min-max it to be between 0-255, then convert it to a numpy array and e.g. show it with matplotlib <code>imshow()</code>.</p>
<p>Greetings.</p>

username: SU801T
cooked: <p>Thanks, I’ll give that a try!</p>
<p>T</p>
Error Handling Runs Loop

username: aksg87
cooked: <p>Hello all,</p>
<p>I was curious – for hyperparameter search which sometimes leads to issues like CUDA out of memory due to batch_size and/or image size, related parameters is this code structure an okay practice? If there is a better way of handling this I would love to hear comments!</p>
<pre><code class="lang-python">for run_count, hyperparams in enumerate(HYPERPARAMS, start=1):

    # define model, dataloaders, etc
    try:
        #Call Training Utils     

    except RuntimeError as e:
       # Handle Possible Errors
        with open("run_parameters.txt", "a+") as text_file:
            text_file.write("*** Runtime Error: {} \n\n".format(e))            
        
    finally:
        #Rest For Next Run
        del model
        del optimizer_ft
        del exp_lr_scheduler
        del dataloaders
        gc.collect()
        torch.cuda.empty_cache()</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>If you’re designing this from scratch, I would advise to create new processes to run the inner job to ensure no issue with cleanup of remaining states.<br>
You can check this issue for issues related to GPU OOM failures: <a href="https://github.com/pytorch/pytorch/issues/18853">https://github.com/pytorch/pytorch/issues/18853</a>.<br>
Also be aware that if an assert happens in a cuda kernel (like index out of range), the cuda driver <strong>cannot</strong> recover from it and you will have to restart the process. So your code example won’t help in that case as all the other trainings will fail.</p>

username: aksg87
cooked: <p>Thanks! Makes sense. I appreciate the advice.</p>

username: AAYUSH_LAKHANI
cooked: <p>Hey, did u get any fix for it? I’m a beginner in PyTorch, please explain</p>
Save initial gradient for all upstream layers

username: francescocarzaniga
cooked: <p>I have a model that implements a layer with a custom forwards and backwards function following the extension <a href="https://pytorch.org/docs/master/notes/extending.html" rel="noopener nofollow ugc">guide</a>. However the backwards function is not computed using the gradient of the downstream layer, but the gradient of the output layer.</p>
<p>What’s the cleanest way to pass it to the backwards function of upstream layers?</p>
ONNX link error when building from source (no CUDA)

username: fralik
cooked: <p>I am trying to build PyTorch from the source code and getting a linking error. I followed the official instructions: I use conda environment, do not use CUDA, tried both MS Studio 2019 and ninja (clean builds).</p>
<p>My setup is like this:</p>
<pre><code class="lang-auto">conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses
conda install -c conda-forge libuv=1.39

git clone --recursive https://github.com/pytorch/pytorch
cd pytorch

set DISTUTILS_USE_SDK=1
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
set REL_WITH_DEB_INFO=1
set USE_CUDA=0
set USE_CUDNN=0
set BUILD_TEST=0
python setup.py install
</code></pre>
<p>I am getting linking errors for <code>torch_python.lib</code> (here just one missing reference):</p>
<pre><code class="lang-auto">   Creating library lib\torch_python.lib and object lib\torch_python.exp
shape_type_inference.cpp.obj : error LNK2019: unresolved external symbol "private: static class onnx_torch::GraphProto * __cdecl google::protobuf::Arena::CreateMaybeMessage&lt;class onnx_torch::GraphProto&gt;(class google::protobuf::Arena *)" (??$CreateMaybeMessage@VGraphProto@onnx_torch@@$$V@Arena@protobuf@google@@CAPEAVGraphProto@onnx_torch@@PEAV012@@Z) referenced in function "protected: static class onnx_torch::GraphProto * __cdecl google::protobuf::MessageLite::CreateMaybeMessage&lt;class onnx_torch::GraphProto&gt;(class google::protobuf::Arena *)" (??$CreateMaybeMessage@VGraphProto@onnx_torch@@@MessageLite@protobuf@google@@KAPEAVGraphProto@onnx_torch@@PEAVArena@12@@Z)
</code></pre>
<p>I was checking the project and as I understood some libraries are compiled as dynamic and some as static. ONNX is compiled as static. <code>torch_python.lib</code> doesn’t reference ONNX directly. When I tried to link it manually, then I got name collisions with protobuf library.</p>
<p>Anyone knows the fix?</p>
If the input_dim=2 in LSTM

username: s_roy
cooked: <p>I am new to Pytorch, Kindly help to understand error.</p>
<p>class LSTMClassifier(nn.Module):</p>
<pre><code>def __init__(self, input_dim, seq_size, hidden_dim, label_size, batch_size, bidirectional, num_layers):

   



    super(LSTMClassifier, self).__init__()

    self.input_dim = input_dim

    self.seq_size = seq_size

    self.hidden_dim = hidden_dim

    self.batch_size = batch_size

    self.num_layers = num_layers

    self.bidirectional = bidirectional

    self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers=num_layers, bidirectional=bidirectional, batch_first=True)

    self.hidden = self.init_hidden()

    self.fc = nn.Linear(hidden_dim*seq_size, 128)

    if bidirectional:

        self.fc = nn.Linear(2*hidden_dim*seq_size, 128)  

                  

    self.hidden2label = nn.Linear(128, label_size)



def init_hidden(self):

    first_size = 1

    if self.bidirectional:

        first_size = 2

    h0 = Variable(torch.zeros(first_size*self.num_layers, self.batch_size, self.hidden_dim)).float()

    c0 = Variable(torch.zeros(first_size*self.num_layers, self.batch_size, self.hidden_dim)).float()

    return (h0, c0) 

def forward(self, x):

    lstm_out, self.hidden = self.lstm(x, self.hidden)

    lstm_out = torch.squeeze(lstm_out) #original

    y = self.fc(lstm_out)

    y = F.relu(y)

    y = self.hidden2label(y)

    return y
</code></pre>
<p>…<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)<br>
1674         ret = torch.addmm(bias, input, weight.t())<br>
1675     else:<br>
-&gt; 1676         output = input.matmul(weight.t())<br>
1677         if bias is not None:<br>
1678             output += bias</p>
<p>RuntimeError: size mismatch, m1: [320 x 2], m2: [20 x 128] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:41</p>

username: ptrblck
cooked: <p>The linear layer is raising this error, as the number of input features from the activation doesn’t match the <code>in_features</code> from the layer.<br>
Check the shape of <code>lstm_out</code> and make sure it’s <code>[batch_size, *, in_features]</code>.</p>
<p>PS: I would also be careful with using <code>torch.squeeze</code> without specifying the <code>dim</code> argument, as it could also remove the batch dimension, if you are working with a single sample, which is usually not desired.</p>

username: s_roy
cooked: <p>I changed the line from<br>
<span class="hashtag">#self</span>.fc = nn.Linear(hidden_dim*seq_size, 128) <span class="hashtag">#original</span></p>
<p>to<br>
self.fc = nn.Linear(self.hidden_dim, 128, bias=True)</p>
<p>error resolved… but result is something unexpected…</p>
How to load UCF-101 dataset for Feature Extraction?

username: div_sh
cooked: <p>Hi!<br>
I am trying to extract frames from clips in the UCF 101 dataset. When I load the first fold for the training set using the data loader (frames_per_clip = 32), I get  1490434 datapoints. I was expecting 833 data points (833, 3, 32, H , W). I want to use the R(2+1)D model to extract features from UCF-101, so that I get features of shape [833, 512]. Can someone please help me figure this out?</p>
<p>Thanks!</p>
Tensor Slicing in C++

username: hlahkar
cooked: <p>What will be the equivalent C++ implementation for the below tensor operation:</p>
<p>pe[:, 0::2] = torch.sin(position * div_term)</p>
<p>Assuming pe is a two-dimensional tensor.</p>

username: thomasjo
cooked: <p>Check out the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="noopener nofollow ugc">Tensor Indexing API</a> documentation. It contains tables showing how to translate both getter and setter constructs from Python to C++ using <em>libtorch</em>.</p>

username: hlahkar
cooked: <p>Thanks for the response; will be trying out the same</p>
How to remove the adaptive average pool layer from vgg19?

username: I_am_you
cooked: <p>I have loaded the pretrained model of vgg19. How to remove the adaptive average pool layer which is present before the classifier?</p>

username: Eta_C
cooked: <pre><code class="lang-python">import torch.nn as nn
from torchvision.models import vgg19

net = vgg19(pretrained=True)
net.avgpool = nn.Identity()
</code></pre>
<p>But if you remove the average pooling layer, your output before classifier must have shape (N, 512, 7, 7), or you have to change the <code>in_features</code> of <code>net.classifier[0]</code> (the Linear layer).</p>

username: I_am_you
cooked: <p>Hi <a class="mention" href="/u/eta_c">@Eta_C</a> thanks for your reply. I am currently using pytorch 0.4.0 for my task so Identity function is not supported in this version i am supposing because of which it is unavailable. Is there any other alternative?</p>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">net2 = nn.Sequential()
net2.add_module('features', nn.Sequential(*list(net.named_children())[0][1]))
net2.add_module('classifier', nn.Sequential(*list(net.named_children())[2][1]))
</code></pre>

username: Eta_C
cooked: <p>If you do not have <code>nn.Identity</code>, just define it.</p>
<pre><code class="lang-auto">import torch.nn as nn
from torchvision.models import vgg19

class Identity(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
    def forward(self, x):
        return x

net = vgg19(pretrained=True)
net.avgpool = Identity()
</code></pre>

username: I_am_you
cooked: <p><a class="mention" href="/u/eta_c">@Eta_C</a> thanks for your solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: NumesSanguis
cooked: <p>I don’t know if in earlier versions of PyTorch the following works, but in v1.6 deleting a layer is as simple as:</p>
<pre><code class="lang-python">del net.avgpool
</code></pre>
<p>This both removes the layer from model.modules and model.state_dict.<br>
This is also does not create zombie layers, as an Identity layer would do; Simplifying model loading.</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/delete-layers-in-faster-rcnn/79428/10">here</a> with a warning, that this approach might break your model if the <code>forward</code> method isn’t fixed afterwards.</p>
Torch.conv2d with groups

username: deJQK
cooked: <p>I am trying to use <code>torch.conv2d</code> or <code>torch.nn.functional.conv2d</code> with groups specified, but it reports runtime error. The code is something like this:</p>
<pre><code class="lang-auto">a = torch.rand(4, 3, 8, 8)
b = torch.ones(3, 3, 3)
out = torch.conv2d(a, b, groups=3)
</code></pre>
<p>The error is something like this:</p>
<blockquote>
<p>RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]</p>
</blockquote>
<p>Changing <code>torch.conv2d</code> to <code>torch.nn.functional.conv2d</code> gives the same error.</p>
<p>The following code works:</p>
<pre><code class="lang-auto">a = torch.rand(4, 3, 8, 8)
b = torch.ones(3, 3, 3, 3)
out = torch.conv2d(a, b)
</code></pre>
<p>and the following also works:</p>
<pre><code class="lang-auto">a = torch.rand(4, 3, 8, 8)
b = torch.ones(3, 3, 3, 3)
out = torch.conv2d(a, b, groups=1)
</code></pre>

username: ptrblck
cooked: <p>The <code>weight</code> tensor is expected to have 4 dimensions as <code>[out_channels, in_channels, height, width]</code>, which is the case for the second approach.</p>

username: deJQK
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> . The following code</p>
<pre><code class="lang-auto">a = torch.rand(4, 3, 8, 8)
b = torch.ones(3, 3, 3, 3)
out = torch.conv2d(a, b, groups=3)
</code></pre>
<p>gives this bug:</p>
<blockquote>
<p>RuntimeError: Given groups=3, weight of size 3 3 3 3, expected input[4, 3, 8, 8] to have 9 channels, but got 3 channels instead</p>
</blockquote>

username: ptrblck
cooked: <p>3 groups with a filter using <code>in_channels=3</code> would need 9 channels.<br>
The <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html#grouped-convolutions">grouped conv section</a> or the docs give you more information on the usage of the <code>groups</code> argument.</p>
<p>If you want to process each input channel separately, use:</p>
<pre><code class="lang-python">input = torch.rand(4, 3, 8, 8)
weight = torch.ones(3, 1, 3, 3)
out = torch.conv2d(input, weight, groups=3)
</code></pre>

username: deJQK
cooked: <p>Thanks a lot. It is clear now. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Multiple forward passes, single conditional backward pass

username: Alan_Wang
cooked: <p>Suppose forward passes are performed multiple times through a single model within a for loop, but only a single backward pass is called. Can PyTorch handle this scenario? Do the intermediate values of the forward pass get overwritten with each new forward pass, thus rendering the backward pass incorrect? Or does PyTorch manage all the forward passes in separate computational graphs?</p>
<p>An example of this scenario could look like this, where the model takes as input some random quantity and only the lowest resulting loss is optimized:</p>
<pre><code class="lang-auto">optimizer.zero_grad()

size = 3
losses = torch.zeros(size)
for h in torch.randn(size):
    out = model(x, h)
    single_loss = get_loss(out, target)
    losses[h] = single_loss
                        
sort_losses, _ = torch.sort(losses) # Sort by loss, low to high

loss = sort_losses[0] # Optimize only the lowest loss
loss.backward()
optimizer.step()

</code></pre>
<p>Thanks in advance for any help.</p>

username: ptrblck
cooked: <p>Your approach would calculate the gradients of the used model parameters w.r.t. the “lowest loss”, i.e. <code>sort_losses[0]</code>.<br>
Note however, that all computation graphs (with all intermediate activations) will be stored during the execution of the loop, which will of course increase the memory usage.</p>
<p>Also, if you are using e.g. batchnorm layers, their running stats will be updated in each forward pass, which might not fit your use case.</p>
All input tensors must be on the same device. Received cuda:1 and cuda:0

username: pirate-zhang
cooked: <p>When i tried to test one trained model with torch,it was an error.The error is as follows:<br>
Traceback (most recent call last):<br>
File “run.py”, line 152, in <br>
main()<br>
File “run.py”, line 86, in main<br>
saver.write_test_img(ep, i, model, index_a = index_a, index_b = index_b)<br>
File “/home/zhangwei/LADN/src/saver.py”, line 123, in write_test_img<br>
assembled_images = model.assemble_outputs()<br>
File “/home/zhangwei/LADN/src/model.py”, line 933, in assemble_outputs<br>
row1 = torch.cat((images_a[0:1, ::], images_b1[0:1, ::], images_b2[0:1, ::], images_a4[0:1, ::], images_a3[0:1, ::]),3)<br>
RuntimeError: All input tensors must be on the same device. Received cuda:1 and cuda:0<br>
And my command is</p>
<pre><code class="lang-auto">CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 python3 run.py --backup_gpu 1 --dataroot ../datasets/makeup --name makeup_test --resize_size 576 --crop_size 512 --local_style_dis --n_local 12 --phase test --test_forward --test_random --result_dir ../results --test_size 300 --resume ../models/light.pth --no_extreme
</code></pre>
<p>i’d really appreacite it if someone could help me with it.</p>

username: albanD
cooked: <p>Hi,</p>
<p>As mentioned in the error message, the Tensors you give to the <code>cat</code> operations are on different devices which is not allowed. You need to move all the Tensors to the same device before calling <code>cat</code>.</p>

username: pirate-zhang
cooked: <p>Thanks for you suggestion.But i didn’t call the cat operations actually,did i?I am still a little confused.</p>

username: fadetoblack
cooked: <p>From the error message you posted, it seems cat operation has been used here:</p>
<p>File “/home/zhangwei/LADN/src/model.py”, line 933, in assemble_outputs</p>

username: pirate-zhang
cooked: <p>Thanks.I have found it.</p>

username: pirate-zhang
cooked: <p>Thanks.I have found it! I try to debug it.</p>
Pytorch LSTM returns two tensors, why?

username: dmitryako
cooked: <p>I am new to NLP and using LSTM for prediction and classification. The latest pytorch 1.6 uses<br>
(<a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging</a>) LSTM via:     x, _ = self.lstm(x). In older internet posts,  x, self.hidden_cell = self.lstm(x, self.hidden_cell), which does not work now by producing cryptic<br>
"RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. ".</p>
<p>So, my question: is there any useful info in that hidden_cell from LSTM which could be used for better classification and/or prediction?</p>

username: fadetoblack
cooked: <p>It returns the shape as seen in the docs.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" target="_blank" rel="noopener nofollow ugc">LSTM — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Concatenate two images [Android]

username: sieme97
cooked: <p>The input to the network is 4 channel tensor. For that, I am using bitmap to read two images (one is gray scale and the other is rgb). Any idea on how to concatenate in a tensor form?</p>

username: ptrblck
cooked: <p>Once you’ve loaded each image and transformed it to a tensor in the shape <code>[channels, height, width]</code>, you could concatenate it with <code>torch.cat</code> as seen here:</p>
<pre><code class="lang-python">gray = torch.randn(1, 24, 24)
rgb = torch.randn(3, 24, 24)
x = torch.cat((gray, rgb), dim=0)
print(x.shape)
&gt; torch.Size([4, 24, 24])
</code></pre>
Dictionary trainable in pytorch

username: Gourav_Datta
cooked: <p>I have a dictionary whose values corresponding to some keys are being used in the forward function of my class. How can I make this dictionary trainable?</p>

username: ptrblck
cooked: <p>What should be trainable in this <code>dict</code>? Could you explain your idea and use case a bit more? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why Pytorch loss function gives different results than TF?

username: chrismaliszewski
cooked: <p>I’m trying to figure out why PT binary_cross_entropy gives different results than TF’s binary_cross_entropy.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201012-085602-RC00_336660316" class="site-icon" width="15" height="15">
      <a href="https://colab.research.google.com/drive/1hzFEI05_hKNRV4gE1FFDumd1ygN3LVsQ?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1hzFEI05_hKNRV4gE1FFDumd1ygN3LVsQ?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Anyone?</p>

username: fadetoblack
cooked: <p>This is not a pytorch related issue, however, if you use from_logits=True in tensorflow’s binary_crossentropy function, you will get the same value as pytorch’s function.</p>
<p>If you check the <a href="https://pytorch.org/docs/stable/nn.functional.html#binary-cross-entropy" rel="noopener nofollow ugc">pytorch docs for binary cross entropy</a>, it can be seen in the example that sigmoid is applied to the outputs before passing into the cross entropy loss function.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss = F.binary_cross_entropy(F.sigmoid(input), target)
</code></pre>

username: chrismaliszewski
cooked: <p>The prediction I provided is already calculated with nn.Sigmoid.</p>
<p>I’ll need to read about from_logits. Thank you.</p>
Domain adaptation loss: loss.backward() fails

username: chalulu
cooked: <p>Hi!<br>
I would like to implement this paper: <a href="https://arxiv.org/pdf/1905.12775.pdf" rel="noopener nofollow ugc">DSNE</a>, which is a domain adaptation technique. I would like to reproduce the results for 2 domains: the source domain (MNIST in my case) and the target domain (SVHN in my case).</p>
<p>The idea of this paper is to compute a loss which is the difference between two terms. The firs term is the maximum of the L2 distances between samples from the source domain and samples from the target domain which have the same labels. The second term is the minimum of the L2 distances between samples from the source domain and samples from the target domains which have different labels.</p>
<p>I tried to implement this loss like this:</p>
<pre><code class="lang-auto">class HausdorffLoss(torch.nn.Module):
    def __init__(self):
        super(HausdorffLoss, self).__init__()
    
    def forward(self, features_s, features_t, labels_s, labels_t):
        m = torch.cdist(labels_s.unsqueeze(-1).double(), labels_t.unsqueeze(-1).double(), p=1)
        m = torch.eq(m,0)
        #Compute l2 norm
        dist = torch.cdist(features_s.double(), features_t.double(), p=2)
        #Apply mask
        pos = torch.max(torch.masked_select(dist, m))
        neg = torch.min(torch.masked_select(dist, ~m))
        
        #Compute diff
        loss = torch.relu(pos - neg)
        return loss
</code></pre>
<p>where “features_s” and “features_t” are feature vectors computed by a VGG-16 network for the samples of the source domain and target domain respectively.</p>
<p>However in my training loop the call to loss.backward() throws this error:<br>
<code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>
<p>I am not sure why it is happening. I am still a beginner in pytorch so I would be glad if someone could help me.</p>
<p>Thanks!</p>
Trainset.classes returns a list with '.ipynb_checkpoints'

username: martinoywa
cooked: <p>After creating my trainset as below<br>
<code>trainset = torchvision.datasets.ImageFolder('data/train/', transform=transform)</code><br>
running <code>trainset.classes</code> returns this list <code>['.ipynb_checkpoints', '0', '1']</code>.</p>
<p>Later when I try to train my model I get the error<br>
<code>IndexError: Target 2 is out of bounds</code><br>
which I believe is due to my classes being length 3 instead of 2.</p>
<p>How do I fix this safely?</p>

username: martinoywa
cooked: <p>Was able to fix the issue by deleting the .ipynb_checkpoints folder using <code>shutil.rmtree('data/train/.ipynb_checkpoints')</code></p>
BatchNorm performed unexpectedly

username: sherylwang
cooked: <p>Hi, I’ve tried to train a CNN after merged the BatchNorm layers into Convolution layers. After merge bn, the BactchNorm layer is remained and the parameters are reset to make the output same as the input, the code is as below:</p>
<pre><code class="lang-auto">           child.eval()
           child.running_mean = child.running_mean.new_full(
               child.running_mean.shape, 0)
           child.running_var = child.running_var.new_full(
               child.running_var.shape, 1)
           if child.weight is not None:
               child.weight.data = child.weight.data.new_full(
                   child.weight.shape, 1)
           if child.bias is not None:
               child.bias.data = child.bias.data.new_full(child.bias.shape, 0)
           child.track_running_stats = False
           child.momentum = 0
           child.reset_running_stats()
           child.eps = 0
</code></pre>
<p>“child” is referred to a BatchNorm layer. But unfortunately, the BatchNorm layer still have output different from the input after the reset. It’s weird.<br>
The torch version is 1.6.0, CUDA10.2, run in the docker.</p>

username: ptrblck
cooked: <p>How large is the difference?<br>
Note that you might be running into the limited numerical precision if the max. abs difference is approx. <code>1e-6</code>.<br>
This code snippet shows that resetting the parameters as well as buffers, and calling <code>eval()</code> should work:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
x = torch.randn(2, 3, 24, 24)

# change bn stats
for _ in range(10):
    out = bn(x)

# make sure output is not equal to input
print((out != x).all())
&gt; tensor(True)

# reset
bn.reset_parameters()
bn.reset_running_stats()

print(dict(bn.named_parameters()))
&gt; {'weight': Parameter containing:
tensor([1., 1., 1.], requires_grad=True), 'bias': Parameter containing:
tensor([0., 0., 0.], requires_grad=True)}

print(dict(bn.named_buffers()))
&gt; {'running_mean': tensor([0., 0., 0.]), 'running_var': tensor([1., 1., 1.]), 'num_batches_tracked': tensor(0)}

bn.eval()
# test again
for _ in range(10):
    out = bn(x)

# check for equal output
print(torch.allclose(out, x)) # use allclose for numerical precision
&gt; True
</code></pre>

username: sherylwang
cooked: <p>Thanks for reply! After checking my  code, I find “model.train()” have been called unexpected in the code script.  This should be the reason for the unexpected performance.</p>
How to train two datasets with one dataloader?

username: John1231983
cooked: <p>Hello all, I have a problem as follows:</p>
<ul>
<li>Dataset 1 has 1000 images</li>
<li>Dataset 2 has 10 images.</li>
</ul>
<p>I want to train a model , in which the images in the dataset 2 must be always use in the training. For example, I want to train with batch size of 64, then for each mini-batch, we should load all 10 images from dataset 2 and remaining from dataset 1. I am using customer dataset with distributed sampler. Do we have any solution for that?</p>

username: fadetoblack
cooked: <p>Have you tried using a separate dataloader with batch size 10 for Dataset 2?</p>

username: John1231983
cooked: <aside class="quote no-group" data-username="fadetoblack" data-post="2" data-topic="99281">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fadetoblack/48/21784_2.png" class="avatar"> fadetoblack:</div>
<blockquote>
<p>ed using a separate dataloader with batch size 10 for Dataset 2?</p>
</blockquote>
</aside>
<p>Good idea but how to get them work together</p>

username: fadetoblack
cooked: <p>Create dataloader1 for Dataset 1 with batch size 54.<br>
Create dataloader2 for Dataset 2 with batch size 10<br>
Then do something like:</p>
<pre><code class="lang-auto">for x1,y1 in dataloader1:
   x2, y2 = next(iter(dataloader2))
   # then concatenate x = [x1,x2] and y = [y1,y2]
   # use x,y for fwd pass and backwd pass
   
</code></pre>
<p><a href="https://github.com/pytorch/pytorch/issues/1917" rel="noopener nofollow ugc">Relevant discussion</a></p>
I want to do machine learning with android

username: ZeTyS_TOMe
cooked: <h1>Question about “torch.jit.xxxx”.</h1>
<p>I would like to know how to serialize the models in “pytorch mobie”.</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.resnet18(pretrained=True)
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("app/src/main/assets/model.pt")
</code></pre>
<p>But,the network I want to use is another network, such as <a href="https://github.com/sacmehta/ESPNet" rel="noopener nofollow ugc">ESPNet</a>.<br>
Does anyone know how to make a “.pt” file?</p>

username: pytorchtester0
cooked: <p>Hello,</p>
<p>ESPNet is not saved by PyTorch script (i mean by torch.jit.save) it has been saved with torch.save already so you can directly use it i think.</p>
<p><a href="https://github.com/sacmehta/ESPNet/tree/master/pretrained/encoder" rel="noopener nofollow ugc">https://github.com/sacmehta/ESPNet/tree/master/pretrained/encoder</a></p>
<p>Anyway, I managed to load the encoder this way by previously downloading the pre-trained model from Github.<br>
Regards,</p>
<pre><code class="lang-auto">import torch
model = ESPNet(20,encoderFile="espnet_p_2_q_8.pth", p=2, q=8)
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt") #Will do the same save a load_dict
#Exemple de save du modele ESPNet avec TorchScript
torch.jit.save(traced_script_module, "scriptmodel.pth")
#Load example with ToarchScript
modelLoaded = torch.jit.load("scriptmodel.pth")
</code></pre>
<p>The output is</p>
<pre><code class="lang-auto">Encoder loaded!
</code></pre>
<p>For ESPNet to be noted that it uses the GPU so tu use CPU i copy Model.py from <a href="https://github.com/sacmehta/ESPNet/tree/master/train" rel="noopener nofollow ugc">https://github.com/sacmehta/ESPNet/tree/master/train</a> in my Notebook then i change this instruction</p>
<pre><code class="lang-auto">self.encoder.load_state_dict(torch.load(encoderFile)
</code></pre>
<p>With</p>
<pre><code class="lang-auto">self.encoder.load_state_dict(torch.load(encoderFile,map_location='cpu'))
</code></pre>

username: ZeTyS_TOMe
cooked: <p>Thank you pytorchtester0!!!</p>
<p>I’m Japanese and I’m sorry for my poor English.<br>
I’m going to try it out as soon as possible.</p>
<p>I just have two more question,</p>
<pre><code class="lang-auto">example = torch.rand(1, 3, 224, 224)
</code></pre>
<p>First,<br>
I believe the ESPNet input image is resized to 1024x512.<br>
I don’t have to change?</p>
<p>Second,<br>
I would like to test it with ESPNet_Encoder.<br>
So could you tell me how to use “ESPNet_Encoder” as well?</p>
How to select 3D tensor from a 2D mask?

username: James_Yip
cooked: <p>Hi I have a 3D tensor like (batch_size, seq_len, dim), and some of them in the 2nd dimension are zero-padded. And I have a mask generated by the lengths. I want to select the tensor by the mask. The behavior is like masked_select but returns a 2D tensor.</p>
<pre><code class="lang-auto">a = torch.rand((3, 3, 3))
a[1, 2] = 0
a[2, 2] = 0
a[2, 1] = 0
</code></pre>
<pre><code class="lang-auto">print(a)
tensor([[[0.7910, 0.4829, 0.7381],
         [0.9005, 0.2266, 0.5940],
         [0.8811, 0.8379, 0.9670]],

        [[0.3192, 0.9537, 0.1001],
         [0.5695, 0.0185, 0.2561],
         [0.0000, 0.0000, 0.0000]],

        [[0.8885, 0.0043, 0.3867],
         [0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000]]])
</code></pre>
<pre><code class="lang-auto">b = torch.BoolTensor([[1, 1, 1],
                      [1, 1, 0],
                      [1, 0, 0]])
</code></pre>
<p>Expected output</p>
<pre><code class="lang-auto">tensor([[0.7910, 0.4829, 0.7381],
        [0.9005, 0.2266, 0.5940],
        [0.3192, 0.9537, 0.1001],
        [0.5695, 0.0185, 0.2561],
        [0.8885, 0.0043, 0.3867]])
</code></pre>
<p>Thanks!</p>

username: Caruso
cooked: <p>Hi,</p>
<p>you can use this mask for slicing, so <code>c = a[b]</code> should return your expected output</p>

username: James_Yip
cooked: <p>Thanks for this elegant solution</p>
Bidirectional LSTM Implementation

username: c4n
cooked: <p>Hi,<br>
I notice that when you do bidirectional LSTM in pytorch, it is common to do floor division on hidden dimension for example:</p>
<pre><code>def init_hidden(self):
    return (autograd.Variable(torch.randn(2, 1, self.hidden_dim // 2)),
            autograd.Variable(torch.randn(2, 1, self.hidden_dim // 2)))
</code></pre>
<p><a href="http://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html#sphx-glr-beginner-nlp-advanced-tutorial-py" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html#sphx-glr-beginner-nlp-advanced-tutorial-py</a></p>
<p>I’m not quite sure why do we need to do this extra step when changing from uni-directional to bi-directional LSTM.</p>

username: smth
cooked: <p>this is probably expected by the LSTM module (and in turn the CuDNN RNN API). It’s a convention more than anything from what I understand.</p>

username: holli
cooked: <p>It’s a good convention. In that example they create a custom model and the hidden_dim defines the output size they want from the LSTM. Bidirectional has twice the amount of hidden variables so if you wan’t to keep the final output the same you have to divide the hidden_dim by 2.</p>
<pre><code class="lang-auto">hidden_size = 128
lstm = nn.LSTM(10, hidden_size, num_layers=1, batch_first=True, bidirectional=False)
hidden_vect_1 = (
    Variable(torch.zeros(1, 1, hidden_size)),
    Variable(torch.zeros(1, 1, hidden_size)))
output, hidden = lstm(Variable(torch.rand(1, 5, 10)), hidden_vect_1)
print('Output size:', output.size(), '- Hidden size:', [h.size() for h in hidden])
# Output size: torch.Size([1, 5, 128]) - Hidden size: [torch.Size([1, 1, 128]), torch.Size([1, 1, 128])]
</code></pre>
<pre><code class="lang-auto">lstm = nn.LSTM(10, hidden_size, num_layers=1, batch_first=True, bidirectional=True)
hidden_vect_1 = (
    Variable(torch.zeros(2, 1, hidden_size)),
    Variable(torch.zeros(2, 1, hidden_size)))
output, hidden = lstm(Variable(torch.rand(1, 5, 10)), hidden_vect_1)
print('Output size:', output.size(), '- Hidden size:', [h.size() for h in hidden])
# Output size: torch.Size([1, 5, 256]) - Hidden size: [torch.Size([2, 1, 128]), torch.Size([2, 1, 128])]
# Note that the output is twice as large, and also that the hidden are [2,1,128] instead of [1,1,128]
</code></pre>

username: Sanjayvarma11
cooked: <p>If you use same hidden size in LSTM while using bidirectional then it will produce twice the amount of hidden varibles where you wont even use half of  them.So you are wasting computation.But you can keep same hidden size and apply a fully connected layer ate end to convert ‘2x’ size to ‘x’ size if you want:))</p>
Where is the forward() function of the mask rcnn?

username: HaziqRazali
cooked: <p>This seems really silly but where is the forward() function of the mask rcnn? I have been looking through <a href="https://github.com/pytorch/vision/blob/2831f11abcb9ec7b951b6bbbcb7a85b79ee2fd79/torchvision/models/detection/mask_rcnn.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/2831f11abcb9ec7b951b6bbbcb7a85b79ee2fd79/torchvision/models/detection/mask_rcnn.py</a> but can’t find it at all.</p>
<p>I am trying to modify the mask rcnn’s forward pass <a href="https://pytorch.org/docs/stable/torchvision/models.html#torchvision.models.detection.maskrcnn_resnet50_fpn" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/models.html#torchvision.models.detection.maskrcnn_resnet50_fpn</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>It is inherited from its parent class.<br>
So all the way to this class: <a href="https://github.com/pytorch/vision/blob/2831f11abcb9ec7b951b6bbbcb7a85b79ee2fd79/torchvision/models/detection/generalized_rcnn.py#L15">https://github.com/pytorch/vision/blob/2831f11abcb9ec7b951b6bbbcb7a85b79ee2fd79/torchvision/models/detection/generalized_rcnn.py#L15</a></p>
Problem with Video Classification Model

username: dancing_neuron
cooked: <p>I have built a basic video classification neural network. The idea is to use a CNN to generate feature vector for each frame, combine it for a video, and then feed it into an LSTM. Here is the model -</p>
<pre><code class="lang-auto">
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 33 * 33, 1200)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1,17424)
        #x = F.relu(self.fc1(x))
        x = self.fc1(x)
        return x

class VideoNet(nn.Module):
    def __init__(self):
        super(VideoNet,self).__init__()
        self.convnet = ConvNet()
    
    def forward(self,x):
        no_frames = list(x.shape)[1]
        features = []
        for i in range(no_frames):
            features.append(self.convnet(x[:,i,:,:,:]))
        return torch.stack(features,dim=1)


class VideoLSTM(torch.nn.Module):
    def __init__(self,feature_size, hidden_dim):
        super().__init__()
        self.vidnet = VideoNet()
        self.lstm = nn.LSTM(feature_size, hidden_dim, batch_first=True)
        self.linear = nn.Linear(hidden_dim, 9)
        self.dropout = nn.Dropout(0.2)

    def forward(self, x):
        x = self.vidnet(x)
        x = self.dropout(x)
        lstm_out, (ht, ct) = self.lstm(x)
        return self.linear(ht[-1])



</code></pre>
<p>I tried to do the “Sanity Check” of feeding only one batching to see if the loss goes to zero. It turns out, the loss starts from 2.2 and only reaches around 1.8, even after 100 iterations… I am begin to think there is something fundamentally wrong with the network.   In each iteration, I am ensuring that training functions are called -</p>
<pre><code class="lang-auto">    video_model.train()
    #for batch_idx, (data, target) in enumerate(train_loader):
    optimizer.zero_grad()
    output = video_model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
</code></pre>
<p>Could you please point out where am I going wrong? Thank you so much!</p>
How to process squence of multi-dimensional input in nn.GRU?

username: dlresearcher
cooked: <p>As per the pytorch documentation, nn.GRU expects input in the shape (seq_len, batch, input_size) or (batch, seq_len, input_size) based on whether the batch_first is False or True, where the input_size is an integer. However, if I am dealing with a sequence of multi-dimensional tensors then is it possible to use nn.GRU() without explicitly flattening the input at each time stamp? If yes, then what should be the input_size then? as it only takes integer and not tuple of integers.</p>
<p>For example say, I have a grayscale video of 16 frames with each frame dimension being 128x128. Here, I can easily set seq_len to 16, but how to deal with the shape of each frame at each time stamp? Do I need to flatten it to a single dimension of 128*128?<br>
(I understand passing a video directly to GRU may not be a good idea, but just wanted to know how nn.GRU deals with multidimensional input at each time step)</p>