akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch Profiler

Sutharsan_Mahendren: <p>I checked this tutorial <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler With TensorBoard — PyTorch Tutorials 1.8.1+cu102 documentation</a> on CPU only.<br>
I got the following error</p>
<pre><code>p.step()
</code></pre>
<p>File “torch/profiler/profiler.py”, line 230, in <strong>exit</strong><br>
self._exit_actions()<br>
File “torch/profiler/profiler.py”, line 346, in _exit_actions<br>
self._start_trace()<br>
File “torch/profiler/profiler.py”, line 367, in _start_trace<br>
assert self.profiler is not None<br>
AssertionError</p>
Which copy_ is better?

blackbirdbarber: <p>x = torch.tensor([0,0,0,0])<br>
k=torch.tensor([1,2,3,4])</p>
<p>x.data.copy_(k) <span class="hashtag">#1</span><br>
<span class="hashtag">#or</span><br>
x.copy_(k) <span class="hashtag">#2</span><br>
x</p>

albanD: <p>Hi,</p>
<p>The first one should never be used as <code>.data</code> should not be used.<br>
The new api for <span class="hashtag">#1</span> is:</p>
<pre><code class="lang-auto">with torch.no_grad():
    x.copy_(k)
</code></pre>
<p>And the difference is clearer: use <span class="hashtag">#1</span> (EDIT: the version of <span class="hashtag">#1</span> given just above in my comment, not your original one) if the copy should not be tracked by the autograd (like initializing some weights) and use <span class="hashtag">#2</span> if you want gradients.</p>

blackbirdbarber: <p>I may be tired, but I do not understand. If you said:</p>
<blockquote>
<p>The first one should never be used as  <code>.data</code>  should not be used.</p>
</blockquote>
<p>And later:</p>
<blockquote>
<p>use <span class="hashtag-raw">#1</span> if the copy should not be tracked by the autograd</p>
</blockquote>
<p>This is opposite.</p>

albanD: <p>Sorry, I wasn’t very clear. Use the other equivalent version of <span class="hashtag">#1</span> that I gave you (the one with torch.no_grad()).</p>

pnic: <p>Hi, I have a question regarding this reply.</p>
<p>Why shouldn’t .data.copy_ be used?? Is p.copy_ always preferred than p.data.copy_?</p>

albanD: <p><code>.data</code> should not be used anymore.<br>
It is unsafe and might lead to silently wrong gradients.</p>

pnic: <p>Regardless of the pytorch version we are using?? Can I get more understanding of why it is unstable??</p>

ptrblck: <p><a class="mention" href="/u/alband">@albanD</a> explains it in more detail in <a href="https://discuss.pytorch.org/t/using-tensor-data/79640/4">this post</a>.</p>
Cannot load pytorch1.7 model in pytorch1.6

Feywell: <p>I try to use torch.jit.save() save model in pytorch1.7 machine.<br>
But I can not use torch.jit.load() load model in my local machin(pytorch1.6)</p>
<p>ERROR message:<br>
RuntimeError:</p>
<blockquote>
<p>aten::_convolution(Tensor input, Tensor weight, Tensor? bias, int<span class="chcklst-box fa fa-square-o fa-fw"></span> stride, int<span class="chcklst-box fa fa-square-o fa-fw"></span> padding, int<span class="chcklst-box fa fa-square-o fa-fw"></span> dilation, bool transposed, int<span class="chcklst-box fa fa-square-o fa-fw"></span> output_padding, int groups, bool benchmark, bool deterministic, bool cudnn_enabled) → (Tensor):<br>
Expected at most 12 arguments but found 13 positional arguments.</p>
</blockquote>
<blockquote>
<p>Serialized   File “code/<strong>torch</strong>/torch/nn/modules/conv.py”, line 11<br>
argument_1: Tensor) → Tensor:<br>
_0 = self.bias<br>
input = torch._convolution(argument_1, self.weight, _0, [1, 1], [0, 0], [1, 1], False, [0, 0], 1, False, False, True, True)<br>
~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
return input<br>
class ConvTranspose2d(Module):</p>
</blockquote>

ptrblck: <p>PyTorch is backwards compatible (unless deprecation warning indicate otherwise), which means you should be able older models (e.g. stored in <code>1.6</code>) in newer PyTorch versions (e.g. <code>1.7</code>). However, it’s not forward compatible, so you would need to update your PyTorch version.</p>
Torchvision.utils.save_image - Vectorgraphic suitable for Word and PowerPoint available?

MaxM: <p>Hello together,</p>
<p>I wanted to ask if there is an option so save images with torchvision.utils.save_image using a vectorgraphic filetype that is accepted by Word and PowerPoint.</p>
<p>I know that I can save the images as .eps what is a vectorgraphic, but isn´nt compatible with Word and PP.</p>
<p>On the other hand .svg and .emf are suitable with Word and PP but aren´t supported by torchvision.utils.save_image.</p>
<p>Is there any type of vectorgraphic I missed that is suitable for both sides?</p>
<p>Many thanky in advance!</p>
<p>Best regards,<br>
Max</p>

ptrblck: <p><code>torchvision.utils.save_image</code> uses <code>PIL.Image.save</code> to save the image in <a href="https://github.com/pytorch/vision/blob/093757dbebcc7ab265287d00a365705d305db2a6/torchvision/utils.py#L135-L136">these lines of code</a>. Based on this it seems you can pass a <code>format</code> argument, which should be one of the supported <code>PIL</code> formats as seen <a href="https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html">here</a>.</p>

MaxM: <p>I`ve seen this also. But using .wmf does’nt work for me, altough it should be supported. I get the following error.</p>
<pre><code>raise OSError("WMF save handler not installed")</code></pre>

ptrblck: <p>This error seems to indicate that the WMF library is missing or that <code>PIL</code> wasn’t installed with it, so you might need to either install dependencies or update <code>PIL</code> etc.</p>
Is there a Pytorch function for getting size of a tensor in GPU as if it were a PNG file without converting it to a CPU tensor?

Pablo1: <p>Hello everyone,</p>
<p>Is there a direct way to get the size of a cuda-tensor as if it were a PNG image without converting it to cpu-tensor?</p>
<p>I’ve tried with encode_png but it requires to pass the tensor to cpu, but I want to know the size of a tensor as if it were a PNG file before actually saving it into disk.</p>
<p>Many thanks in advance</p>
Help developing generative model for time series

Spiruel: <p>Hi,</p>
<p>I am looking at using a generator to generate time series data.</p>
<p>Example code I am looking at is the <code>CausalConvGenerator</code> in (<a href="https://github.com/proceduralia/pytorch-GAN-timeseries/blob/8e7d62fed6f4061d13ec9dfd84e07520d4257ed2/models/convolutional_models.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-GAN-timeseries/convolutional_models.py at 8e7d62fed6f4061d13ec9dfd84e07520d4257ed2 · proceduralia/pytorch-GAN-timeseries · GitHub</a>)</p>
<p>The generator currently takes</p>
<pre><code class="lang-auto">Input: (batch_size, seq_len, input_size)
Output: (batch_size, seq_len, outputsize)
</code></pre>
<p>I am looking to modify this to instead take</p>
<pre><code class="lang-auto">Input: (batch_size, noise_dim)
Output: (batch_size, seq_len, outputsize)
</code></pre>
<p>Would someone mind pointing me in the right direction on how to modify the architecture?</p>

ptrblck: <p>The linked model uses <code>nn.Conv1d</code> layers, which is why the 3-dimensional input is expected (it should have the shape <code>[batch_size, channels, seq_len]</code>).<br>
If you want to use only two dimensions, you could change the layer type to e.g. linear layers and would need to experiment with the architecture (i.e. number of features etc.). Alternatively, you could also add an additional dimension (e.g. the channel or temporal dimension) and run some experiments with the current architecture.</p>
How to convert 'c10::List' to std::vector

alanzhai219: <p>when define a new op, some parameters are torch::List. It is not easy to use such data type.<br>
So,<br>
1.How  to convert ‘c10::List’ to std::vector???<br>
2.BTW, could anyone can show me the usage of c10::List or its API?</p>
<p>Thanks</p>

Arsenii: <ol>
<li>c10_list.vec()</li>
<li>c10_list[0]; c10_list[1]</li>
</ol>
libtorch torch::save

rmx564354093: <p><strong>In pytorch when train model over we can save model . such as</strong><br>
torch.save({<br>
‘epoch’: epoch,<br>
‘model_state_dict’: model.state_dict(),<br>
‘optimizer_state_dict’: optimizer.state_dict(),<br>
‘loss’: loss,<br>
…<br>
}, PATH)</p>
<p><strong>and load model and param such as</strong><br>
model.load_state_dict(checkpoint[‘model_state_dict’])<br>
optimizer.load_state_dict(checkpoint[‘optimizer_state_dict’])<br>
epoch = checkpoint[‘epoch’]<br>
loss = checkpoint[‘loss’]</p>
<p>but in libtorch . I don’t know how to achieve it . Please help me?</p>

ptrblck: <p>You could save the objects directly via <code>torch::save</code> as seen <a href="https://github.com/pytorch/pytorch/blob/3a66a1cb99d7e1c1fe7abf8f390e177b9183a436/test/cpp/api/serialize.cpp#L65-L159">here</a>.</p>
C++ PyTorch model inference inside of a running CUDA kernel

rdmelzer: <p>Hi,</p>
<p>I have a CUDA C++ application where I would like to call a PyTorch model on the GPU while I’m already inside of a kernel, running on the device. What library should I use to achieve this? Ideally I’d like to load up a model trained in PyTorch, then do inference inside of a running CUDA kernel. Is this possible?</p>
<p>For example, say I have a reinforcement learning environment implemented in a CUDA kernel. I launch 100 instances of the kernel on the GPU to compute 100 trajectories. The kernel loops over timesteps collecting states from the environment, computing next states. How can I integrate PyTorch model inference within that loop?</p>
<p>Thanks!</p>

ptrblck: <p>You could call other kernels form your current CUDA kernel, but since an entire PyTorch model isn’t a single kernel, I don’t think it’s possible.<br>
I’m also unsure about your use case and why you you wouldn’t be able to execute the model from e.g. <code>libtorch</code> (C++).</p>
Using Automatic Mixed Precision for existing tochvision models

sajaldash: <p>Hi,<br>
I am trying to use AMP with my resnet50 code from <a href="https://github.com/horovod/horovod/blob/master/examples/pytorch/pytorch_imagenet_resnet50.py" rel="noopener nofollow ugc">https://github.com/horovod/horovod/blob/master/examples/pytorch/pytorch_imagenet_resnet50.py</a>. I changed the lines 75 through 86 to the following snippet. As I understand from PyTorch documentation, I am supposed to use autocast in the forward pass. But, since I am using a predefined model (torchvision.models.resnet50()), I don’t have access to that part of the code. So, I am getting the error</p>
<blockquote>
<p>File “pytorch_imagenet_resnet50_mp.py”, line 78, in train<br>
with autocast:<br>
AttributeError: __enter__</p>
</blockquote>
<p>How can I use AMP with these predefined models?</p>
<pre><code>        for i in range(0, len(data), args.batch_size):
            data_batch = data[i:i + args.batch_size]
            target_batch = target[i:i + args.batch_size]
            with autocast:
                output = model(data_batch)
                train_accuracy.update(accuracy(output, target_batch))
                loss = F.cross_entropy(output, target_batch)
                train_loss.update(loss)
                # Average gradients among sub-batches
                loss.div_(math.ceil(float(len(data)) / args.batch_size))
            scaler.scale(loss).backward()
        # Gradient is applied across all ranks
        #optimizer.step()
        scaler.step(optimizer)
</code></pre>

ptrblck: <p>You would have to call the op via:</p>
<pre><code class="lang-python">with autocast():
    ...
</code></pre>
A Python package for seamlessly testing your model

Peng_Yan: <p>I just created a package called torcheck that help you carry out sanity check around your model. It has received several stars, so I wanted to share with the community.</p>
<p>For a general introduction, please check this out: <a href="https://towardsdatascience.com/testing-your-pytorch-models-with-torcheck-cb689ecbc08c" rel="noopener nofollow ugc">Testing Your PyTorch Models with Torcheck</a></p>
<p>The GitHub link is: <a href="https://github.com/pengyan510/torcheck" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pengyan510/torcheck</a></p>
<p>The major benefit is that you no longer need to write additional testing code for your model training. Just add a few lines of code specifying the checks before training, torcheck will then take over and perform the checks simultaneouly while the training happens.</p>
<p>Another benefit is that torcheck allows you to check your model on different levels. Instead of checking the whole model, you can specify checks for a submodule, a linear layer, or even the weight tensor! This enables more customization around the sanity checks.</p>
How can i complete the task with .pth files?

mathwseg: <p>i used per-trained model for my datasets and got files with .pth (epochs) … i need to use the result for image captioning … how can i extract the features from the files i got ?</p>
<p>Does epochs with .pth  that come from training the model are the weights ?</p>

eqy: <p>Yes, you can use the <code>.pth</code> files to load model weights and from there you can use the intermediate activations of the model as features. The ImageNet example: <a href="https://github.com/pytorch/examples/blob/cbb760d5e50a03df667cdc32a61f75ac28e11cbf/imagenet/main.py#L186" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at cbb760d5e50a03df667cdc32a61f75ac28e11cbf · pytorch/examples · GitHub</a><br>
shows an example of loading from checkpoints.</p>

mathwseg: <aside class="quote no-group quote-modified" data-username="eqy" data-post="2" data-topic="123802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>you can use the intermediate activation of the model as features</p>
</blockquote>
</aside>
<p>excuse me what do you mean by using the intermediate activation of the model as features . is there any example for this ?<br>
and the way you shared it is supposed for any model right ?<br>
Appreciating your help</p>

eqy: <p>Yes, the model loading should be generic across models.<br>
I think extracting model features has been thoroughly discussed in the past; example thread:<br>
<a href="https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model">https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model</a></p>

mathwseg: <p>Thanks for sharing the link i will try it but excuse me . what is the difference between save tensor and extract features as i tried</p>
<pre><code>    def forward(self, x):
        x = self.forward_features(x)
        y=x[-1].cpu().numpy()
        with open('res.pkl', 'wb') as df_file:
             pickle.dump(y, file = df_file)
    return x
</code></pre>
<p>but i got the tensor for one image .  Does the code i wrote right ? and if it is. how can i got it for all images.</p>

eqy: <p>I haven’t tried an implementation like that before but if you indeed are seeing values saved to storage then I suspect it’s working fine. You can compute the features for all images simply by passing all of them through the model (you can do this in a batched way with something like a dataloader). The ImageNet example also shows how to iterate through a dataset with a dataloader.</p>

mathwseg: <p>thanks but what is the difference between save tensor and extract features</p>

eqy: <p>It depends on what you actually want to do. If you want to use features computed on the images later, one way to do this is to extract them and save them for another model to use later. You can also just extract them (e.g., by returning them from the forward function) and pass them directly to another model or function.</p>
Is it possible to quantize model have more than one input?

kim-ninh: <p>I’m wondering if it is possible to quantize a model that has 2 inputs (2 tensors). Basically, I having a GAN model in which the generator module requires 2 inputs for inference. Right now, I’m looking at the example with Static Quantization in <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">here</a> but the sample model to be quantized just have 1 input.</p>

Zafar: <p>Yes it is. I am not sure what exactly your model is, but here is an example on how you could make a model with two inputs in the forward</p>
<pre><code class="lang-auto">class SomeAwesomeModel(nn.Module):
    def __init__(self):
        self.quant_x = torch.quantization.QuantStub()
        self.quant_y = torch.quantization.QuantStub()
        self.func = nn.quantized.FloatFunctional()
        self.dequant = torch.quantization.DeQuantStub()
 
   def forward(self, x, y):
        qx = self.quant_x(x)
        qy = self.quant_y(y)
        qz = self.func.add(qx, qy)
        z = self.dequant(qz)
        return z
</code></pre>
<p>The model above expect two floating point tensors as input, quantizes them, and after performing some funciton, returns the dequantized version.</p>

kim-ninh: <p>Thanks a lot <a class="mention" href="/u/zafar">@Zafar</a>, this is exactly what I’m looking for. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

addisonklinke: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Perhaps the need for one <code>QuantStub</code> per model input could be mentioned in the QAT <a href="https://pytorch.org/docs/stable/quantization.html#quantization-aware-training" rel="noopener nofollow ugc">docs</a>? On my first read through this distinction was not clear to me until I started getting errors in my project code</p>
Omitting backprop for random batch samples for specified part of a NN

phiwei: <p>Hi,</p>
<p>sorry if a question like this has been posed before, I could not find anything unfortunately, but was not certain of the correct terminology. I would like to implement the following to work on a multiple-instance-learning (MIL) task:</p>
<p>I have a CNN that consists of three parts: a feature extraction network, a fully connected attention-network that predicts weights for bag instances, which are then used to compute a weighted average of the instance-wise feature vectors, followed by a classifier network for this bag-level representation. This setup benefits from very large batch sizes. I would like to make a forward pass with a very large batch, then update the weights of the attention network and classifier network with gradients from all bag instances/samples, but then only update the feature extractor CNN with a very small percentage of the samples, i.e. 10%.</p>
<p>Is it possible to do this to reduce the GPU memory requirements? If so, could you point me in the right direction of how I could do this? In my naive thinking, I would simply need something like a drop-out in the backward path of a layer between feature extractor CNN and fully connected NNs that is only an identity function in a forward pass.</p>
<p>I am admittedly clueless of how to approach this. Any pointers would be much appreciated!</p>
<p>Thanks!</p>
Gradient of loss w.r.t. a row of model input

phantom90: <p>Hi There,</p>
<p>Could you please help how to calculate gradient of loss w.r.t. <em><strong>a row of model input</strong></em>.</p>
<p>Thanks~</p>

soulitzer: <p>Could you clarify what you mean by row of model input. Are you talking about a subtensor of a parameter, a row of your training data? Do you mean to say that normally your model accepts a batch of inputs, but you’d like to compute the gradients wrt to the parameters, but using a single sample instead?</p>
Character-level GRU multi-class tutorial?

Parkz: <p>I am working on a character-level multiclass problem and want to know if there are any good sites that explain how to code GRUs for this sort of thing, or if anyone can share the code for it. Here is my code for a LSTM version:<br>
<span class="hashtag">#basic</span> model, need to modify to situation<br>
class NLP_model(nn.Module):<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, hidden_dim, num_classes):<br>
super(NLP_model, self).<strong>init</strong>()<br>
self.char_embedding = nn.Embedding(vocab_size, embedding_dim)<br>
self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers = 5, bidirectional= True)<br>
self.fc = nn.Linear(hidden_dim*2,num_classes)<br>
<span class="hashtag">#self</span>.att = nn.MultiheadAttention(embed_dim, num_heads, …)</p>
<pre><code>def forward(self, x):
    x = self.char_embedding(x)
    output, hidden = self.lstm(x)
    hidden = torch.cat((hidden[0][-2,:,:], hidden[0][-1,:,:]), dim=1)
    x = self.fc(hidden[0])

    return x
</code></pre>
<p>What would a GRU version of this code look like? How do I learn how to code NLP? There’s tons of articles about what GRU is, but I have a hard time finding code tutorials explaining a GRU through code.</p>
Constrained Loss Function

gdx: <p>Suppose I have a neural network model which outputs a single positive scalar value.</p>
<p>Every training instance in my data has two ordered input tensors (x1, x2). The outputs from the model are as follows:</p>
<pre><code class="lang-auto">y1 = model(x1)
y2 = model(x2)
</code></pre>
<p>How to design a loss function to make sure that y1 - y2 &gt; 0? Additionally, I would like to achieve this without pushing the value of y2 towards 0. For my particular use-case, we don’t care about how large (y1-y2) is, just have to make sure that it is positive. What should be a suitable loss function to achieve this constraint on the outputs?</p>

KFrank: <p>Hi Gdx!</p>
<aside class="quote no-group" data-username="gdx" data-post="1" data-topic="123812" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gdx/48/37102_2.png" class="avatar"> gdx:</div>
<blockquote>
<p>How to design a loss function to make sure that y1 - y2 &gt; 0?</p>
</blockquote>
</aside>
<p>First a comment:</p>
<p>Because neural networks are trained by “backpropagating” a<br>
differentiable loss function using gradient-descent methods, it<br>
is much more natural to use a differentiable penalty to penalize<br>
violations of your desired constraint, rather than impose a hard<br>
constraint.  Note that such an approach does permit your constraint<br>
to be violated,  but if your penalty is large enough, your constraint<br>
won’t be violated by very much.</p>
<p>Along these lines, consider using:</p>
<pre data-code-wrap="python"><code class="lang-python">penalty_loss = torch.exp (alpha * (y2 - y1))
</code></pre>
<p>(This is a straightforward choice for the penalty function but many<br>
other choices would also make sense.)</p>
<p>The greater you make <code>alpha</code> the more you will penalize values<br>
for which <code>y1 - y2 &lt; 0</code> (and the less the network will care about<br>
the value of <code>y1 - y2</code>, as long as it is greater than zero).</p>
<p>I might imagine starting training with <code>alpha</code> set to a moderate value,<br>
and then increasing it as training progresses.</p>
<p>Best.</p>
<p>K. Frank</p>
Unfolding 5D input in memory-friendly manner?

Wouter_Durnez: <p>I’m currently trying to train a 3D U-Net, in which the convolutional layers are replaced with my own layers. In my own implementation, I unfold the (5D: batch size x channels x h x d x w) input to I can contract it with my kernel tensor, which means I use <code>torch.Tensor.unfold</code> three times per layer, like so:</p>
<p><code>   patches = input.unfold(2, kernel_dim, stride).unfold(3, kernel_dim, stride).unfold(4, kernel_dim, stride)</code></p>
<p>Unfortunately, this is also extremely taxing on the GPU’s RAM.</p>
<p>I’m currently training on an HPC, and have access to multiple GPUs, each with 32GB RAM. To save memory, I’m currently trying to checkpointing with cpu_offload (on each of my double convolution blocks), using the <code>fairscale</code> library.</p>
<p>I haven’t found a way to make it work just yet. I’m wondering: is there a memory-friendly alternative to unfolding 5D input this way? I found several posts of people having similar problems (which is how I turned to <code>fairscale</code>), but I haven’t found anything just yet.</p>
Update weight with same netowork's output

hyuntae: <p>I want to update the classifier’s weights twice with the two outputs of the classifier.<br>
To update, I wrote a code.<br>
But, the code gives me the error that <em>’ enable anomaly detection to find the operation that failed to compute its gradient,’</em></p>
<p>I saw the answer that this code works with the previous version of pytorch. But it seems weird.<br>
Can you tell me where I should fix it?<br>
I don’t want to backward with <code>(foreProb + backProb).backward()</code></p>
<pre><code class="lang-auto">foreData = data * mask
backData = data * (1-mask)

foreOutput = myClassifier(foreData)
backOutput = myClassifier(backData)                

foreProb = nn.CrossEntropyLoss(foreOutput, target)
backProb = nn.CrossEntropyLoss(backOutput, target)

self.optimizer['classifier'].zero_grad()
foreProb.backward()
self.optimizer['classifier'].step()
                
self.optimizer['classifier'].zero_grad()
backProb.backward()
self.optimizer['classifier'].step()
</code></pre>

ptrblck: <p>It seems your code tries to calculate the gradients in the second <code>backward</code> pass using “stale” intermediate forward activations, since the parameters were already updated, which is wrong. <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">This post</a> explains it in more detail.</p>

hyuntae: <p>Yes, I saw that post.<br>
But, still gives me the error although I fix the code to</p>
<pre><code class="lang-auto">self.optimizer['classifier'].zero_grad()
foreProb.backward(retain_graph=True)
self.optimizer['classifier'].step()
                
self.optimizer['classifier'].zero_grad()
backProb.backward()
self.optimizer['classifier'].step()
</code></pre>
<p>The error is<br>
<code>one of the variables needed for gradient computation has been modified by an inplace operation</code><br>
This error doesn’t appear with the old pytorch version…</p>

ptrblck: <p>That wouldn’t be a fix, as it’s still using the wrong behavior. Previous PyTorch versions allowed this wrong gradient calculations, which is why no errors were raised.</p>
Use DDP to split the model on two nodes (each node has one GPU)

Knight_Zhang: <p>Hello Guys,<br>
Thanks to the DDP, I could split the batch data across different GPUs on Different nodes, I could also split the model on different GPUs in one node. But for now, I need to split the model on different GPUs <strong>on Diffrent Nodes</strong>, let’s say two nodes, and one gpu per node. Could someone help me with this? I think the main difficult is that: for each node, the “local_rank” is 0, how to send different part of the model to different GPU on different node?<br>
Thanks a lot!</p>

cbalioglu: <p>Unfortunately cross-host model sharding is not supported yet, but we have plans to introduce it in a future version of PyTorch.</p>

Knight_Zhang: <p>Thanks for your quick response, Can!</p>
Get standard deviation of multiple tensors

David_Holomcik: <p>Hi, i want to get the standard deviations of multiple image tensors [10,1024,1024]. for each element (like from position [0,0,0] to [9,0,0] for each position in the tensors</p>
<p>Currently i´m using <code>uncertainty_prediction = torch.std(total_prediction,dim=0)</code> however this returns a tensor [1024,1024] with 0.5 everywhere. This is not correct, as there should be different std deviations at each element of the resulting tensors.</p>
<p>I´ve looked on google, and since I´m rather new to pytorch i probably missed the answere, but is there a build in solution to this? or am i just using the torch.std() wrong? I really want to avoid iterating over all the tensors.</p>
<p>Thanks a lot<br>
Greetings</p>

eqy: <p>Right, in this case it seems you actually want to reduce across the second and third dimensions (I think a good way to think about this is which dimensions would you be <em>averaging</em> <em>across</em> to calculate the corresponding mean?). Does <code>uncertainty_prediction = torch.std(total_prediction,dim=(1,2))</code> work?</p>
How to save variables in hard-disk to use them in pytorch dataloader?

krishna511: <p>Hi all,<br>
Sir I am using an online available code for my data. But its showing out of memory message everywhere, on my machine, kaggle GPU and google colab. The author suggested to</p>
<blockquote>
<p>" Process the data and save it on the hard disk and create [pytorch dataloader]"</p>
</blockquote>
<p>I have got the processed data as</p>
<pre><code class="lang-auto">Shape of X_train: (3441, 7, 1, 128, 128)
Shape of X_val: (143, 7, 1, 128, 128)
Shape of X_test: (150, 7, 1, 128, 128)
</code></pre>
<p>Now plz can someone guide how to use pytorch dataloader here, or If he meant to save these variables from python to my hard disk then only I can use dalaloader.<br>
plz guide.<br>
Regards</p>

ejguan: <p>For reference, you can take a look at <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Datasets &amp; Dataloaders — PyTorch Tutorials 1.8.1+cu102 documentation</a> or other repos like <a href="https://github.com/pytorch/vision/tree/master/torchvision/datasets" class="inline-onebox" rel="noopener nofollow ugc">vision/torchvision/datasets at master · pytorch/vision · GitHub</a><br>
Also, can you post a code snippet to better understand the problem?</p>

krishna511: <p>I have modified few things like Dataset for pytorchdataloader, so I am reframing the question again, in another question.</p>

ejguan: <p>Can you please link the another question you are referring to?</p>

krishna511: <p><a class="mention" href="/u/ejguan">@ejguan</a>  Actually I dont know the correct way so just copied the <a href="https://discuss.pytorch.org/t/stack-expects-each-tensor-to-be-equal-size-but-got-0-at-entry-0-and-3-at-entry-1/123375">link</a></p>
<p>Any guidance will be appreciated.</p>

Godoy56: <p>I will ensure that I bookmark your blog and will come back sometime soon.</p>
Error with save weakref object

PJBWFG: <h2>Hi there, I got an error when I call torch.save to save my pickle file but I got an error below, please let me know what’s does this mean and how to fix it. Thanks!</h2>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 ssd.export(‘123.pkl’)</p>
<p> in export(self, name_or_path)<br>
191<br>
192     def export(self, name_or_path):<br>
–&gt; 193         torch.save(self.learn,name_or_path)<br>
194<br>
195     def load(self, name_or_path):</p>
<p>D:\Software\envs\data245\lib\site-packages\torch\serialization.py in save(obj, f, pickle_module, pickle_protocol)<br>
222         &gt;&gt;&gt; torch.save(x, buffer)<br>
223     “”"<br>
–&gt; 224     return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))<br>
225<br>
226</p>
<p>D:\Software\envs\data245\lib\site-packages\torch\serialization.py in _with_file_like(f, mode, body)<br>
147         f = open(f, mode)<br>
148     try:<br>
–&gt; 149         return body(f)<br>
150     finally:<br>
151         if new_fd:</p>
<p>D:\Software\envs\data245\lib\site-packages\torch\serialization.py in (f)<br>
222         &gt;&gt;&gt; torch.save(x, buffer)<br>
223     “”"<br>
–&gt; 224     return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))<br>
225<br>
226</p>
<p>D:\Software\envs\data245\lib\site-packages\torch\serialization.py in _save(obj, f, pickle_module, pickle_protocol)<br>
295     pickler = pickle_module.Pickler(f, protocol=pickle_protocol)<br>
296     pickler.persistent_id = persistent_id<br>
–&gt; 297     pickler.dump(obj)<br>
298<br>
299     serialized_storage_keys = sorted(serialized_storages.keys())</p>
<p>TypeError: can’t pickle weakref objects</p>

albanD: <p>Hi,</p>
<p>The problem is that the model you try to save contains <code>weakref</code> python objects. Such objects are not supported by python’s pickle module.<br>
You might want to check your model and see why you have weakrefs in it.</p>

PJBWFG: <p>Thank you, what are wearef python objects? <a class="mention" href="/u/alband">@albanD</a></p>

albanD: <p>You can find it in the python documentation <a href="https://docs.python.org/3/library/weakref.html">https://docs.python.org/3/library/weakref.html</a> .</p>

ariyasaccautama: <p>then, how to change the weakref model, to strenght?</p>
<p>is it possible to get changed?</p>
Cuda out of memory error during forward pass

abhibha1807: <p>Hello,<br>
I am trying to implement a ‘one step gradient descent’ aproach wherein I accumulate the loss for the whole dataset, sum it, and then do a backpropagation. I have set my batch size to 8. The issue that I am facing is that after a few forward passes I obtain an OOM error. I think it is because pytorch is saving the forward computation graph for each instance. Is there any work around, where I can save the forward computation graph somewhere and access it when performing a backward pass? Or is there any other workaround?<br>
I have also tried deleting en_input, en_masks, de_output, de_masks after accumulating the loss but no avail.</p>
<pre><code class="lang-auto">#reproduce error
from transformers import BertModel, BertForMaskedLM, BertConfig, EncoderDecoderModel
import torch
import torch.nn.functional as F
model1 = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints
model1.cuda()

optimizer1 = torch.optim.Adam(model1.parameters(), lr=0.001)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print("Using device:", device)

def train1(batch_size):
  acc=torch.zeros(1)
  acc=acc.to('cuda')
  for i in range(20):
    optimizer1.zero_grad()

    #dummy inputs similar to my dataset
    en_input=torch.tensor([[i for i in range(50)] for i in range (batch_size)])
    en_masks=torch.tensor([[0 for i in range(50)] for i in range (batch_size)])
    de_output=torch.tensor([[i for i in range(50)] for i in range (batch_size)])
    de_masks=torch.tensor([[0 for i in range(50)] for i in range (batch_size)])
    lm_labels=torch.tensor([[i for i in range(50)] for i in range (batch_size)])
        
    en_input = en_input.to('cuda')
    de_output = de_output.to('cuda')
    en_masks = en_masks.to('cuda')
    de_masks = de_masks.to('cuda')
    lm_labels = de_output.clone().to('cuda') 
    
    out = model1(input_ids=en_input, attention_mask=en_masks, decoder_input_ids=de_output, 
                        decoder_attention_mask=de_masks, labels=lm_labels)

    prediction_scores = out[1]
    predictions = F.log_softmax(prediction_scores, dim=2)
    p=((predictions.sum() - de_output.sum())).sum() #some loss
    p=torch.unsqueeze(p, dim=0)
    acc = torch.cat((p,acc)) # accumulating the loss 

  loss=acc.sum()
  loss.backward(retain_graph=True) 
  optimizer1.step() 

train1(batch_size=8)
</code></pre>

ptrblck: <p>Yes, Autograd will save the computation graphs, if you sum the losses (or store the references to those graphs in any other way) until a <code>backward</code> operation is performed.<br>
To accumulate gradients you could take a look at <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">this post</a>, which explains different approaches and their computation as well as memory usage.</p>

abhibha1807: <p>Thank you so much! It helped me a lot!</p>
Floating point precision in libtorch

Dan_Sagher: <p>Hi,</p>
<p>I would like to get more accurate value when printing a tensor.<br>
I know that in python you can do something like that:</p>
<pre><code class="lang-auto">torch.set_printoptions(precision=10)
</code></pre>
<p>Do you know about something similar in libtorch?<br>
Thank you.</p>

Luca_Giacomoni: <p>Stuck at this, too.</p>
<p>Have you found a way to print with higher precision?</p>

Dan_Sagher: <p>not really, I just convert the tensor into array and than printed the array values.</p>

shyney7: <p>This is still an issue.<br>
The print behavior is really bad with double type numbers:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/748b7f3277a31c9f694621a34d55c2e060e4dd31.png" alt="Screenshot from 2021-06-10 15-12-22" data-base62-sha1="gD0fWLMSB60rWDu3Nzyx3G5uQgx" width="191" height="162"></p>
Transformer Encoder Self-Attention pad masking is applied to only one axis

DomainFlag: <p>I have a question regarding the <code>MultiheadAttention</code> and <code>multi_head_attention_forward</code>, it seems like the padding masking is applied only to one axis instead of being applied to two axes.</p>
<p>When attaching hook to the attention module, I receive the attention weights as so, when I would expect instead only a non-zero <strong>square</strong> attention map while the rest being zero ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d60c0c2c5a3da6d5f0031a49bcff56d3d8bf24c2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d60c0c2c5a3da6d5f0031a49bcff56d3d8bf24c2" title="map_29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d60c0c2c5a3da6d5f0031a49bcff56d3d8bf24c2_2_375x375.jpeg" alt="map_29" data-base62-sha1="uxxZpeNEavnrCahDyvgabMUmjAK" width="375" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d60c0c2c5a3da6d5f0031a49bcff56d3d8bf24c2_2_375x375.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d60c0c2c5a3da6d5f0031a49bcff56d3d8bf24c2_2_562x562.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d60c0c2c5a3da6d5f0031a49bcff56d3d8bf24c2_2_750x750.jpeg 2x" data-dominant-color="4E6482"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">map_29</span><span class="informations">800×800 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Looking at the source code:</p>
<pre><code class="lang-auto">    if key_padding_mask is not None:
        attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
        attn_output_weights = attn_output_weights.masked_fill(
            key_padding_mask.unsqueeze(1).unsqueeze(2),
            float("-inf"),
        )
        # **Expect Extra** =============&gt;
        attn_output_weights = attn_output_weights.masked_fill(
            key_padding_mask.unsqueeze(1).unsqueeze(2).transpose(-1, -2),
            float("-inf"),
        )
        # &lt;==================== End + Handle the lower zero-row softmax.
        attn_output_weights = attn_output_weights.view(bsz * num_heads, tgt_len, src_len)
</code></pre>
<p>Is it something I am missing or that’s intended like so which seems weird, as attention is shared between padding and audio features?</p>
Prediction has .grad = None

thaophan1: <pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(10, 16),
            nn.Sigmoid(),  
            nn.Linear(16, 16),
            nn.Sigmoid(),
            nn.Linear(16, 16),
            nn.Sigmoid(),
            nn.Linear(16, 3),
            nn.Sigmoid()
        )
    
    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

def nn_train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to('cpu'), y.to('cpu') 

        # # Compute prediction error
        pred = model.forward(X.float())
        loss = loss_fn(pred, y)

        # # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d} ]")
</code></pre>
<p>pred.grad is None during training, I am using pandas to read in word data, Word2Vec to convert word to vector, numpy to store the vectors, and created a dataloader using the array of numpy vectors to pass into the training function.</p>
<p>Also, I am using CrossEntropyLoss and torch.optim.SGD for optimizer</p>

soulitzer: <p>.grad is only populated for the leaf tensors (which usually are just the parameters of your model - and leaf just means that it is an input, and not the output of another operation requiring grad). .grad is only populated for leaf tensors (that require grad) because those are the weights that would need to be updated on each iteration, and if we had to save .grad for every intermediate result, that would be a lot more memory intensive.<br>
If you check <code>model._parameters[0]</code> you should see that the .grad field is populated there.</p>

Alex_Black: <p>What can I use instead of <a href="https://www.bestessaytips.com/review_essay.php" rel="noopener nofollow ugc">.</a>grad?</p>
Why LSTM has two bias parameters

genous110: <pre><code class="lang-auto">lstm = torch.nn.LSTM(10, 20,1)
lstm.state_dict().keys()
</code></pre>
<p>Output result：</p>
<pre><code class="lang-auto">Out[47]: odict_keys(['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0'])
</code></pre>
<p>According to the calculation process of LSTM, there should be only one bias. Why do we output two bias variables, that is,‘bias_ih_l0’and’bias_hh_l0’?</p>

Tony-Y: <p><a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html</a></p>
<p>It says that “Second bias vector is included for CuDNN compatibility. Only one bias vector is needed in standard definition.”</p>
<p><a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnRNNMode_t" class="onebox" target="_blank" rel="nofollow noopener">https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnRNNMode_t</a></p>

sh0416: <p>I think two bias term acts differently.</p>
<p>The main point is that <code>bias_ih</code> is applied once during the computation along time axis, while <code>bias_hh</code> is applied accumulated along the time axis.</p>
<p>I want to clarify this one using illustrative example, but the process is so complicate.</p>
Masking layer for RNN

Mostafa_Alishahi: <p>Is there any function similar to keras.layers.Masking which masks the rows of timeseries data which is filled with an specific value (e.g. padded with zero)?</p>

G.M: <p>By slicing perhaps.</p>
<pre><code class="lang-auto">t=tc.tensor(B,L,D)
t[ : , a:b , : ]=0
</code></pre>

Mostafa_Alishahi: <p>Thanks for your answer. in this way, I need to have the start and end index of the non padded values, isn’t there any other more straightforward way like Masking layer in keras?</p>
<p>one thing more, by slicing we feed the matrix with different shapes into RNN, is this acceptable in pytorch? (e.g. data[0].size() could be (timesteps=100, features=20) and  data[10].size() could be (timesteps=34, features=20))</p>

G.M: <ol>
<li>
<p>Idk how r you going to set values to zero if u don’t know where the values are. You can check torch.masked_select, but i’m quite sure u need to know what to mask. Just curious, are you doing ASR?</p>
</li>
<li>
<p>u have to pad them into a bigger tensor, u can’t even feed it into rnn if u don’t pad</p>
</li>
</ol>

Mostafa_Alishahi: <p>Ok, Thanks for the clarifications <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  . No I am not doing ASR.</p>

hadaev8: <p>Use this<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/Tushar-N/dfca335e370a2bc3bc79876e6270099e" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/Tushar-N/dfca335e370a2bc3bc79876e6270099e" target="_blank" rel="nofollow noopener">https://gist.github.com/Tushar-N/dfca335e370a2bc3bc79876e6270099e</a></h4>
<h5>pad_packed_demo.py</h5>
<pre><code class="Python">import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

seqs = ['gigantic_string','tiny_str','medium_str']

# make &lt;pad&gt; idx 0
vocab = ['&lt;pad&gt;'] + sorted(set(''.join(seqs)))

# make model</code></pre>
This file has been truncated. <a href="https://gist.github.com/Tushar-N/dfca335e370a2bc3bc79876e6270099e" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

Sanbot-kk: <p>Hello,</p>
<p>Were you able to find a solution for this?</p>
Optimizer step function with torch.no_grad() decorator

Rahul_Vashisht: <p>Hi,<br>
I am working on writing my own optimizer while going through the code for default optimizers such as RMSprop I found a torch.no_grad() decorator just before the step function. Is it necessary to use this decorator before the step function when I am writing my own optimizer’s step function? Can someone please explain why it is used over the step function.</p>

Luesch: <p>if you wouldnt call no_grad() the update operation would require gradients and you probably dont want to add your weight update in the computational graph.</p>

Rahul_Vashisht: <p>Understood, Thanks for the reply.</p>
Understanding model.to(device)

Ahmed11: <p>Hello,</p>
<p>I am new to PyTorch, I just want to ensure that I correctly understand how <code>model.to(device=device)</code> works before I start creating more complex models.</p>
<p>So far, almost all the tutorials that Ive followed create, train and evaluate their models in the same notebook/script.</p>
<p>However, what if I am creating my model, training it, and evaluating it using functions/operations in different scripts?</p>
<p>Suppose I have the following code under these different scripts.</p>
<p><strong>main.py</strong></p>
<pre><code class="lang-auto">model = EfficientNet.from_pretrained("efficientnet-b3")
model = model.to(device=device)

for epoch in range(EPOCHS):
	train(model, train_loader, ...)
	check_accuracy(model, val_loader, ...)
</code></pre>
<p><strong>train_model.py</strong></p>
<pre><code class="lang-auto">def train_model(model, loader, ...):
	losses = []	
	for batch_idx, (data, targets) in enumerate(tqdm(loader)):
		...		
		scores=model(data)		
		...
	print(f"Loss: {sum(losses)/len(losses)}")
</code></pre>
<p><strong>evaluation.py</strong></p>
<pre><code class="lang-auto">def check_accuracy(model, loader, ...):
	model.eval()
	with torch.no_grad():
		...
		...
	model.train()
	return accuracy
</code></pre>
<p>Since my model is moved to <code>device</code> by running <code>.to(device=device)</code>, I dont need to return the model from each function in different modules to <code>main.py</code>, is that right?</p>
<p>Simply passing the model to <code>train_model()</code> and <code>check_accuracy()</code> is enough run operations on the model, and once they finish running, the remaining operations in main.py will resume working on the <strong>same</strong> model, is that right?</p>
<p>Thank you all for your help.</p>

ptrblck: <p>Yes, your assumption should be correct as also seen in <a href="https://discuss.pytorch.org/t/model-eval-accuracy-is-0-and-running-corrects-is-0/123271/13">this post</a>, since the model reference would be passed and its parameters (and buffers) updated inplace.<br>
You could use the code snippet in the linked post to verify it using your setup.</p>

Ahmed11: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, will definitely check out the post.</p>