akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How could I swtith off gradient with c++/libtorch

username: coincheung
cooked: <p>Hi,</p>
<p>When I use python api, I can do that by <code>torch.set_grad_enabled(False)</code> which switches off the graident computation globally. How could I do that with libtorch/c++ api?</p>

username: ptrblck
cooked: <p>The RAII <code>torch::AutoGradMode enable_grad(true/false);</code> should work as described in the <a href="https://pytorch.org/cppdocs/api/typedef_namespacetorch_1a23e4c6c3b3c89d2a8051db46fc003159.html#_CPPv4N5torch12AutoGradModeE">docs</a>.</p>
<pre><code class="lang-auto">auto x = torch::tensor({1.}, torch::requires_grad());
{
  torch::AutoGradMode enable_grad(true);
  auto y = x * 2;
  std::cout &lt;&lt; y.requires_grad() &lt;&lt; std::endl; // prints `true`
}
{
  torch::AutoGradMode enable_grad(false);
  auto y = x * 2;
  std::cout &lt;&lt; y.requires_grad() &lt;&lt; std::endl; // prints `false`
}
</code></pre>

username: coincheung
cooked: <p>Thanks for telling this !!<br>
By the way, if I would like to use libtorch for inference, could I use other backends apart from default pytorch engine such as tvm ? If so, how could I do this please ?</p>

username: ptrblck
cooked: <p>I don’t know enough about TVM, but <a class="mention" href="/u/tom">@tom</a>, as a TMV developer, might know more about it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tom
cooked: <p>You can use dlpack to get from and to TVM with your tensors:</p>
<pre><code class="lang-auto">def tensor_to_tvm(t):
    return tvm.nd.from_dlpack(torch.utils.dlpack.to_dlpack(t))
def tensor_from_tvm(a):
    return(torch.utils.dlpack.from_dlpack(a.to_dlpack()))
</code></pre>
<p>This allows to feed PyTorch tensors to TVM models and recover the results.<br>
If you want to convert part of your PyTorch to TVM, you can use the JIT and the TVM PyTorch frontend.</p>
<p>I have an in-depth example that does funny tricks to also enable training in the <a href="https://lernapparat.de/transformers-pytorch-tvm/" rel="noopener nofollow ugc">Transformers, PyTorch and TVM blog post</a>.</p>
<p>In my courses, we also do converting to TVM and then running from C++ and this kind of stuff.</p>
<p>Best regards</p>
<p>Thomas</p>
Faster custom RNN loop sequence

username: Monne
cooked: <p>Hi,<br>
Currently i’m learning on HTR, and using CRNN with custom RNN cell model for it and using JIT, is there any <strong>‘pytorch way’</strong> to use PackedSequence for masking input in RNN loop sequence?</p>
<p>AFAIK only default RNN pytorch model (nn.RNN, nn.LSTM, nn.GRU) can accept it, and custom RNN can only use padded sequence which can be add more time for training</p>
An actual leaf tensor is reported " 'inputs' to backward is not a leaf Tensor" by pytorch

username: Hongkai_Zheng
cooked: <p>Hi,</p>
<p>I’m trying to compute the gradient of input data. I set requires_grad= True and then use autograd.backward(). Then pytorch says “One of the differentiated Tensors given as ‘inputs’ to backward is not a leaf Tensor”. But inputs.is_leaf is True. So what’s going on here? Any idea?</p>
<p>Here is my code to reproduce. (<a href="https://colab.research.google.com/drive/1-grLWtOVZjgo64CezAnVFgCHb-pf9jFg?usp=sharing" rel="noopener nofollow ugc">Run in colab</a>)</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.autograd as autograd

def linear_block(in_channel, out_channel):
    block = nn.Sequential(
        nn.Linear(in_channel, out_channel), 
        nn.Tanh()
    )
    return block

class FCNet(nn.Module):
  def __init__(self, layers=[2, 10, 1]):
    super(FCNet, self).__init__()
    
    fc_list = [linear_block(in_size, out_size) for in_size, out_size in zip(layers, layers[1:-1])]
    fc_list.append(nn.Linear(layers[-2], layers[-1]))
    self.fc = nn.Sequential(*fc_list)

  def forward(self, x):
    return self.fc(x)

layers = [2, 10, 1]

xt = torch.randn((100, 2))
model = FCNet(layers)

xt.requires_grad = True
ys = model(xt)

print(xt.is_leaf)
autograd.backward(tensors=ys.sum(), inputs=xt, create_graph=True)
grad_xt = xt.grad
</code></pre>

username: JuanFMontesinos
cooked: <p>It asks for a sequence of tensors, not for a tensor.</p>
<pre><code class="lang-auto">autograd.backward(tensors=[ys.sum()], inputs=[xt], create_graph=True)
</code></pre>
<p>The system is probably iterating over the tensor and the exception is not handled.</p>

username: Hongkai_Zheng
cooked: <p>You’re right. Thanks!</p>
Parallel two encoder in Pytorch

username: rashid67
cooked: <p>I have an encoder class that is defined in a class in PyTorch. I want to use this model for two parallel models as follows.  could you help me to solve that?</p>
<pre><code class="lang-auto">class  Encoder(nn.Module):
....```
</code></pre>
<p>model 1</p>
<pre><code class="lang-auto">optimizer        = torch.optim.Adam(model1.parameters(), lr=6e-4)
losses = train(model1, device, 100, data1, optimizer)
</code></pre>
<p>model2</p>
<pre><code class="lang-auto">model2           = Encoder(100,  1,   2,  10, device).to(device)
optimizer        = torch.optim.Adam(model2.parameters(), lr=6e-4)
losses = train(model2, device, 100, data2, optimizer)
</code></pre>
<p>Here is the figure which I want:</p>            <div class="onebox imgur-album">
              <a href="https://imgur.com/a/VB3RhKT" target="_blank" rel="noopener nofollow ugc">
                <span class="outer-box" style="width:600px">
                  <span class="inner-box">
                    <span class="album-title">[Album] en</span>
                  </span>
                </span>
                <img src="https://i.imgur.com/WlZXWRv.png?fb" title="en" height="315" width="600">
              </a>
            </div>


username: ptrblck
cooked: <p>You could use the two models in a new custom <code>nn.Module</code>, execute both forward passes, and concatenate the output.<br>
<a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">This post</a> was written for a slightly different use case, but might be a good starter.</p>
Slow data loading-perhaps by CUDA cache?

username: Simin.Li
cooked: <p>Hello,<br>
My pytorch program is working in a slow way. I determined the bottleneck through this code:</p>
<pre><code class="lang-auto">from timeit import default_timer as timer
t1=timer()
xxx(some codes here)
t2=timer()
print(t2-t1)
</code></pre>
<p>After running this code, I found the bottleneck:</p>
<pre><code class="lang-auto">    for batch_idx, (x1, x2, y) in enumerate(train_loader.get_augmented_iterator(model.training)):
        x1 = torch.Tensor(x1).to(device)  # slow in this line of code
        x1 = x1.transpose(1, 3)
</code></pre>
<p>Where get_augmented_iterator is some function to load the data I defined. In the first line, “x1 = torch.Tensor(x1).to(device)”, my program takes ~0.4s to execute this. The get_augmented_iterator() takes about ~0.1s to finish and includes some steps I have to preprocess it in this stage.</p>
<p>While the problem is certainly not in this line itself, I researched a little bit and found that if I add torch.cuda.empty_cache(), this line execute in normal time. However, torch.cuda.empty_cache() itself takes ~0.4s to execute, so I didn’t really solve this problem, but cache itself must be the problem.</p>
<p>I tried on another several projects where very similar dataloader were used, However, I didn’t reproduce my problem on their codes. So my question is: how can it possibly relate to any errors in my code and how can I solve it?</p>

username: Simin.Li
cooked: <p>Here is an update: I ran several more times and found that if I comment the “loss.backward()” line, the speed will be normal. Given that I have to keep this line for sure, how to make the training speed normal?</p>
<p>I believe it must be the data somehow overflowed, so pytorch have to clear the cache at every iteration. But why and how it caused this problem?</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so you need to synchronize the code manually before starting and stopping the timers via <code>torch.cuda.synchronize()</code>.<br>
Based on your current description, you are “moving” the accumulated time from one operation to the next blocking one by commenting them out.<br>
If you don’t synchronize the code, the next blocking operation will accumulate all previously executed (async) operations.</p>
Why can Numpy directly manipulate torch.tensor

username: DumnBird
cooked: <p>I am prepared to get rid of Tensorflow and join the club of Pytorch,<br>
and I found that Numpy can directly mannipulate torch.tensor,<br>
which is  not supported in tensorflow1.x.<br>
And I am wondering why  Numpy can directly manipulate  torch.tensor which are not exactly np.array type?</p>

username: ptrblck
cooked: <p>PyTorch tensors implement the <code>__array__</code> method, which is used for the interop between numpy and PyTorch, if I’m not mistaken.<br>
That being said, I would be careful with applying numpy operations directly on tensors especially if they require gradients and you want to backpropagate through them.</p>
Installation issue_22

username: ALI_AMJAD
cooked: <p>Hi there,</p>
<p>I am trying to install PyTorch on my laptop but it shows me the following error:</p>
<p>Your help is truly appreciated</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a67e92afeb75d0fddf45d60434ff73a0d4a8eab.png" alt="image" data-base62-sha1="1u3oXzcSyDMStMc5Lyrmm3KW7mb" width="661" height="418"></p>
<p>Thanks</p>

username: ptrblck
cooked: <p>It seems you are using a 32bit Python installation, which is not supported.<br>
Install a 64bit Python and retry to install PyTorch in this environment.</p>
Export libtorch model which support variable input tensor

username: WeNeedMore
cooked: <p>Do libtorch model support variable size of the tensor? Trying to train a classification on medical images(CT scans), but the number of slices are different for each file([512, 512, variable]). Is it possible to export the model witch support input of [1, variable, 512, 512]?</p>
<p>Every examples of libtorch use fix size input before export the model</p>
<pre><code class="lang-auto">img = torch.zeros((1,3,16,112,112)) #64,3,16,112,112; 16, 3, 1, 112, 112
traced_script_module.save("video_r3d_18.pt")
</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>Yes, it should be possible via <code>torch.jit.script</code>. Tracing via <code>torch.jit.trace</code> uses the passed input to trace all operations and might fail, if you are using different inputs in the deployment use case.</p>
Debugging the segmentation fault in multiprocessing

username: wentian_zhao
cooked: <p>I keep getting the same segmentation fault of dataloader worker process on a machine (the traceback is attached at the end). It only happens when <code>num_workers</code> &gt; 0. I tried to run the same code on 3 different machines and this only happens on one of them. Sometimes when <code>num_workers</code> == 0 but multiple GPUs are used (with <code>torch.nn.DataParallel</code>), the program crashes, simply showing “Segmentation fault (core dumped)” without showing a python traceback.</p>
<p>Unluckily I’m too busy to provide some sample that could reproduce this exception at this moment. I’m just wondering if there is a way to catch the exception in the dataloader worker process or show which part of the dataloader is faulty? It’s also okay if there is a way to get the core dump file so that I can inspect it with gdb.</p>
<pre><code class="lang-auto">  File "/home/library/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 872, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/library/miniconda3/lib/python3.7/multiprocessing/queues.py", line 108, in get
    res = self._recv_bytes()
  File "/home/library/miniconda3/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/library/miniconda3/lib/python3.7/multiprocessing/connection.py", line 411, in _recv_bytes
    return self._recv(size)
  File "/home/library/miniconda3/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/home/library/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 745428) is killed by signal: Segmentation fault.
</code></pre>

username: ptrblck
cooked: <p>You could try to get the backtrace using <code>gdb</code> via:</p>
<pre><code class="lang-auto">gdb --args python script.pt args
...
run
...
bt
</code></pre>
<p>This should give you the backtrace, which would point towards the failing operation and might be helpful to debug this issue further.</p>
Sparse tensors consume more memory than dense tensors

username: Anand_Krish
cooked: <p>I came across this quirky thing when playing around with sparse tensors where, a sparse tensor takes  lot more memory than a corresponding dense tensor.</p>
<h4>Example Code</h4>
<pre><code class="lang-python">import torch
from pytorch_memlab import MemReporter

DEVICE = 'cuda:0'

n = 1000
c = .6

t1 = torch.randn(100, n, n).to(DEVICE)
t1[torch.rand_like(t1) &gt; c] = 0
t2 = t1.to_sparse()
torch.cuda.empty_cache()

reporter = MemReporter()
reporter.report()
</code></pre>
<h4>Output</h4>
<pre><code class="lang-auto">Element type                                            Size  Used MEM
-------------------------------------------------------------------------------
Storage on cuda:0
Tensor0                                    (100, 1000, 1000)   381.47M
Tensor2                                        (3, 60000715)     1.34G
Tensor2                                          (60000715,)   228.88M
-------------------------------------------------------------------------------
Total Tensors: 340002860 	Used Memory: 1.94G
The allocated memory on cuda:0: 1.94G
Memory differs due to the matrix alignment or invisible gradient buffer tensors
-------------------------------------------------------------------------------
</code></pre>
<p><code>t2</code> occupies around 1.5GB compared to <code>t1</code>'s 380MB.</p>
<h4>Device Info</h4>
<pre><code class="lang-auto">Pytorch Version: 1.7.1
CUDA Version: 11.0
Device: GeForce RTX 2070
</code></pre>
<p>Is this a bug in the sparse memory format?<br>
I have a large sparse tensor (~ 30% sparsity) and I would like to reduce the GPU memory usage. Any help in reducing memory usage is appreciated.</p>

username: ptrblck
cooked: <p>I think the memory footprint is expected as described <a href="https://pytorch.org/docs/stable/sparse.html">here</a>.<br>
For your sparsity, this would be the memory usage:</p>
<pre><code class="lang-python">DEVICE = 'cuda:0'

n = 100
c = .6

t1 = torch.randn(100, n, n).to('cuda')
t1[torch.rand_like(t1) &gt; c] = 0
t2 = t1.to_sparse()

print((t2.indices().nelement() * 8 + t2.values().nelement() * 4) / 1024**2)
</code></pre>
<p>Note that I reduced the number of elements, but it should be the same relative usage compared to the dense tensor.</p>
Why pytorch module needs input shape

username: Jaideep_Valani
cooked: <p>Is it always necessary to specify the input shape of any module that is part of a Sequential Layers.<br>
Like in case of keras if you are building a seq layers u dont need to give the input shape of  hidden layers  just the output shape.</p>

username: Angry_potato
cooked: <p>I have just worked with CNN so i will just say for that. In CNN u just specify the input and output channels along with other parameters. Input size of image is not specified anywhere.</p>

username: akernel
cooked: <p>To the best of my knowledge, PyTorch does not perform implicit shape inference (which is what Keras does) unless each module in the Sequential model has implicit shape inference encoded in its <code>forward</code> method.</p>
<p>I’m not sure of the reasoning behind this, but it may be because TensorFlow (what Keras is built on) uses a static computational graph, while PyTorch uses a dynamic one. This would make it impossible for PyTorch to know the output shape of the 1st layer before it runs, while Tensorflow can analyze the static graph it builds to infer input sizes.</p>
<p>In short: you’ll have to specify input and output sizes.</p>

username: Scott7
cooked: <p>So, if I load someone else’s model and if I don’t know anything about that model, there is no way I can find out the input shape for that model?</p>

username: ptrblck
cooked: <p>I wouldn’t say there is no way to make the model work and you could of course try out different shapes and check for shape mismatches. Of course, it would be easier, if the author provides this shape information as well as all used preprocessing steps etc.</p>
Greyscale output to threshold binary tensor - how to write forward pass

username: oops
cooked: <p>Say the output of my forward pass is a greyscale image</p>
<pre><code class="lang-auto">x = torch.rand(2, 2)
</code></pre>
<p>The gold label is the binary image</p>
<pre><code class="lang-auto">gold = torch.tensor([[0, 1], [1, 0]])
</code></pre>
<p>I want to define my loss by comparing the binary threshold of this image against the gold label. How do I write this so that the computation graph is defined correctly. Here is what I came up with, but my loss doesn’t decrease</p>
<pre><code class="lang-auto">x = (x - torch.min(x)) / (torch.max(x) - torch.min(x))
x[x &lt; 0.5] = 0
x[x &gt;= 0.5] = 1
</code></pre>
<p>Loss is defined as</p>
<pre><code class="lang-auto">loss = criterion(x, gold)
</code></pre>

username: ptrblck
cooked: <p>The threshold operation would create a zero gradient on almost everywhere, which is most likely why the model isn’t training.<br>
The usual loss functions expect the model output to contain the logits, log-probabilities, etc. and apply a “smooth” function internally, such as <code>F.log_softmax</code>.</p>
Upsampling within a set of sizes dependent on the image in a nn.module

username: vmax
cooked: <p>In a nn.module, for a model which fuses feature maps of different strides, I am wondering if it is possible to do <code>nn.Upsample</code>, not with the <code>scale</code> parameter, but with the <code>size</code> parameter.</p>
<p>This <code>size</code> parameter would be dependent on which level of feature map is being upsampled, as well as the input image size which you only see during the forward pass and not when the module is instantiated.</p>
<p>The reason not to use <code>scale</code> is that I would like to be able to work with images of any sizes, without having to resize to some powers of 2.</p>

username: ptrblck
cooked: <p>If you want to use a constant <code>size</code>, you could create the <code>nn.Upsample</code> module in the <code>__init__</code> method of your model and directly use it for all images:</p>
<pre><code class="lang-python">up = nn.Upsample(size=(24, 24))

x = torch.randn(1, 3, 10, 10)
print(up(x).shape)
&gt; torch.Size([1, 3, 24, 24])

x = torch.randn(1, 3, 40, 40)
print(up(x).shape)
&gt; torch.Size([1, 3, 24, 24])
</code></pre>
<p>On the other hand, if you want to use different upsampling layers depending on the current input shape, you could use a condition and either create the <code>nn.Upsample</code> layer in the <code>forward</code> (as it doesn’t have any parameters, this would be fine), or use the functional API via e.g. <code>F.interpolate</code>.<br>
You could also create a <code>list</code> or <code>dict</code> of various <code>nn.Upsample</code> modules in the <code>__init__</code> and select the appropriate one in the <code>forward</code> method.</p>
Error while executing "Ninja all"

username: bucky
cooked: <p>Hey can anyone help me in resolving this error.</p>
<pre><code class="lang-auto">/home/user_name/Desktop/dgx/glow/thirdparty/folly/folly/ssl/OpenSSLPtrTypes.h:33:73: error: template argument 2 is invalid
/home/user_name/Desktop/dgx/glow/thirdparty/folly/folly/ssl/OpenSSLPtrTypes.h:105:34: error: ‘EvpMdCtxDeleter’ was not declared in this scope`Preformatted text`
</code></pre>

username: jfix
cooked: <p>Hi, this appears to be a an issue building folly. What is your build environment? I know some people have had issues on e.g. fedora.</p>

username: bucky
cooked: <p>Hey, I am using Ubuntu 18.04.5 LTS.</p>

username: bucky
cooked: <p><a class="mention" href="/u/jfix">@jfix</a>  I am using Openssl 1.1.1c could that the reason for this error? if yes what versions of Openssl does glow support?</p>

username: jfix
cooked: <p><a class="mention" href="/u/bucky">@bucky</a> This isn’t a Glow-specific issue, this is related to building folly which is a dependency of Glow. So I am not totally sure what the issue might be, or if it’s the version of OpenSSL you’re using. How did you install that version? It looks like on the folly page for 16.04 they just installed <code>libssl-dev</code> through <code>apt-get</code>, perhaps you could find out what version that is and use it too…</p>

username: bucky
cooked: <p>Hey <a class="mention" href="/u/jfix">@jfix</a>, you are correct the problem was with the version of openssl I was using, the issue was resolved after I switched to openssl 1.1.1g.</p>
Cache entire train/test data on GPU

username: mkarikom
cooked: <p>I have noticed that in some situations, a smaller batch size decreases error, but I want to minimize IO overhead for an epoch with many iterations.</p>
<p>It seems that the <code>Dataloader(...,pin_memory=True)</code> option is designed to speed up batch loading.  What if I know that most of my train/test data sets will fit on the device?  Is there a way to cache the entire data set on the GPU, so that I can tune batch size as needed without suffering from increased IO overhead for small batches?</p>

username: surya00060
cooked: <p>Hey!</p>
<p>If your data is small enough, it can be completely stored on GPU itself for computation.</p>
<pre><code class="lang-auto">data = data.to('cuda')
</code></pre>
<p>Else, you can use pin_memory=True to cache the data.</p>
<p>Please refer to this <a href="https://discuss.pytorch.org/t/how-to-load-all-data-into-gpu-for-training/27609">Q&amp;A</a></p>
<p>Hope this helps.</p>

username: mkarikom
cooked: <aside class="quote no-group" data-username="surya00060" data-post="2" data-topic="114396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c2a13f/48.png" class="avatar"> surya00060:</div>
<blockquote>
<p><code>data = data.to('cuda')</code></p>
</blockquote>
</aside>
<p>Thanks <a class="mention" href="/u/surya00060">@surya00060</a>!</p>
<p>I added an option to invoke this inside my <code>DataSet</code> class and it runs way faster now.</p>
Manual vs PyTorch calculation for BatchNormalization difference? Why?

username: GuokLiu
cooked: <p>Hi,<br>
In order to better understand BN of PyTorch, I did some manual calculations as follows, but the output differs. Could anyone help check and find the reason? Thank you very much : )</p>
<p><strong>Code</strong></p>
<pre><code class="lang-auto">import torch 
from torch import nn 

# PyTorch calculation
layer = nn.BatchNorm2d(3)
layer.train()
input = torch.ones(2,3,3,4)
output = layer(input)
print(layer.running_mean)


# Manual calculation
x_mean = input[:,0,:,:].mean()
momentum = nn.BatchNorm2d(3).momentum
running_mean = nn.BatchNorm2d(3).running_mean # initial values: 0
running_mean = momentum * running_mean + (1 - momentum) * x_mean # 0.1*0 + (1-0.1)*1 = 0.9 
print(running_mean)
</code></pre>
<p><strong>output</strong></p>
<pre><code class="lang-auto">tensor([0.1000, 0.1000, 0.1000])
tensor([0.9000, 0.9000, 0.9000])
</code></pre>

username: Eta_C
cooked: <pre><code class="lang-python">running_mean = (1 - momentum) * running_mean + momentum * x_mean
</code></pre>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm#torch.nn.BatchNorm2d" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm#torch.nn.BatchNorm2d</a></p>
<pre><code class="lang-python">.. note::
        This :attr:`momentum` argument is different from one used in optimizer
        classes and the conventional notion of momentum. Mathematically, the
        update rule for running statistics here is
        :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`,
        where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the
        new observed value.
</code></pre>

username: GuokLiu
cooked: <aside class="quote no-group quote-modified" data-username="Eta_C" data-post="2" data-topic="114394">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar"> Eta_C:</div>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm#torch.nn.BatchNorm2d" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm2d — PyTorch 2.1 documentation</a></p>
</blockquote>
</aside>
<p>Hi Eta_C<br>
Sincerely thanks for your timely reply.<br>
I followed your tips and everything goes as I wish.<br>
Sincrerely thanks for your help : )</p>
[DPP] define optimizer before DPP or after DPP (plus amp)

username: 111429
cooked: <p>Hi, a question about the appropriate ordering of defining optimizer in DPP + amp scenario.<br>
Define optimizer before DPP ( <code>torch.nn.parallel.DistributedDataParallel</code>) :</p>
<pre><code class="lang-auto">optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)
model, optimizer = amp.initialize(model, optimizer,  opt_level='O2')
model = DDP(model)
</code></pre>
<p>define optimizer after DPP :</p>
<pre><code class="lang-auto">model = DDP(model)
optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)
model, optimizer = amp.initialize(model, optimizer,  opt_level='O2')
</code></pre>
<p>The official video classification <a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L195" rel="noopener nofollow ugc">script</a> adopts the first logic.<br>
Yet I see some blog says that you should initialize an optimizer over a DPP model when you use DPP (though no amp context involved in that article).<br>
I wonder which one is correct. or dosen’t matter, both are fine. Thanks.</p>

username: rvarm1
cooked: <p>DDP does not change <code>model.parameters()</code>, and the optimizer works entirely locally, so defining the optimizer before or after wrapping model with DDP should not make a difference.</p>
Unexpected results with pre-training model - discussion

username: Aiman_Mutasem-bellh
cooked: <p>Hi all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>My project is a Grammar correction based on multi-head attention model. I have two datasets the first one is synthetic data in 3 GB and the second on the original data in 200 MB. The training strategy as:</p>
<ol>
<li>
<p>Load the pre-training data and start training to save the best version.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74690eb0e24a5603aec3af402d17d5e53dea483e.png" data-download-href="https://discuss.pytorch.org/uploads/default/74690eb0e24a5603aec3af402d17d5e53dea483e" title="save_model"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74690eb0e24a5603aec3af402d17d5e53dea483e.png" alt="save_model" data-base62-sha1="gBOtcsPCZKMTjQpDn9onJLMimn4" width="690" height="92" data-dominant-color="F1EFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">save_model</span><span class="informations">1042×140 6.56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>Change the learning rate and the number of epochs, load the pre-training model and continue training the model with the original dataset.</p>
</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/baba3822c1d87abeb042f6166da8a10d41ac9835.png" data-download-href="https://discuss.pytorch.org/uploads/default/baba3822c1d87abeb042f6166da8a10d41ac9835" title="load_model"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/baba3822c1d87abeb042f6166da8a10d41ac9835.png" alt="load_model" data-base62-sha1="qDRMvOTOdayLLFydBX0bCLo1rWB" width="690" height="187" data-dominant-color="ECEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">load_model</span><span class="informations">740×201 6.93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have a confusing problem related to the results when training the model based on pre-training model, which I get unexpected results (below score) compare with the same model that trained from scratch without pre-training. I have no explanation for this problem taking into account the different sizes between both datasets.</p>
<p>Any suggestion to overcome this issue?</p>
<p>Kind regards,<br>
Aiman Solyman</p>
4d matrix multiplcation using einsum

username: kevs
cooked: <p>I have two tensor, a and b, with 4 dimensions each of the same size and I want to write the following operation using einsum:</p>
<pre><code class="lang-python">output = torch.matmul(a, b.transpose(-2,-1))
</code></pre>
<p>This is what I have so far:</p>
<pre><code class="lang-auto">output = torch.einsum("ijkl,jmnl-&gt;imkn", [q,k])
</code></pre>
<p>But this is not correct, how do I write 4d tensor multiplication using einsum?</p>

username: kevs
cooked: <p>Solved, this seems to be give the correct results:</p>
<pre><code class="lang-python">torch.einsum("ijkl,ijml-&gt;ijkm", [q,k])
</code></pre>
Customizing generated code?

username: Linux-cpp-lisp
cooked: <p>Hi all,</p>
<p>In the context of this issue (<a href="https://github.com/pytorch/pytorch/issues/53517" class="inline-onebox" rel="noopener nofollow ugc">[FX] torch.fx generate code that is not torch.jit able · Issue #53517 · pytorch/pytorch · GitHub</a> which is since resolved, though not yet in the stable release) I was curious" is it possible to customize the generated code for a given node?</p>
<p>Thanks!</p>

username: James_Reed
cooked: <p>Hello <a class="mention" href="/u/linux-cpp-lisp">@Linux-cpp-lisp</a>,</p>
<p>This is a good question. We didn’t foresee the desire to do this, so the current APIs aren’t factored in a clean enough way to easily support something like this. (You can override fx.Graph.python_code, but it would involve a lot of copy-paste, which might be OK for some situations). This seems like a good idea, though, and I’ve filed a feature request: <a href="https://github.com/pytorch/pytorch/issues/53732" class="inline-onebox" rel="noopener nofollow ugc">[FX] Allow customizing code generation for certain Nodes · Issue #53732 · pytorch/pytorch · GitHub</a></p>
<p>For issue <span class="hashtag">#53517</span> in particular, the TorchScript fix is being rolled up into the 1.8.1 patch release, with release date to be announced</p>

username: Linux-cpp-lisp
cooked: <p>Hi <a class="mention" href="/u/james_reed">@James_Reed</a>:</p>
<p>That makes sense. I can think of a few cases where this would be useful:</p>
<ul>
<li>Circumventing compatibility issues (like above)</li>
<li>Code style</li>
<li>Inserting comments into generated code</li>
<li>Inserting things that cannot be represented as typical <code>Nodes</code>, such as profiling blocks with <code>with torch.autograd.profiler.record_function</code>, because they aren’t in the flow of data</li>
<li>Inserting <em>simple</em> control flow through custom nodes (i.e. a node that outputs</li>
</ul>
<pre><code class="lang-python">if my_input is None:
    my_output = torch.zeros(some_shape_node)
else:
    my_output = my_input
</code></pre>
<p>Conceptually this is a node — it has two arguments, one a Node and one a shape or a node representing a shape, and always one output. To do this in FX currently, as I understand it, this operation would have to be wrapped in a seperate function that you mark as a leaf node by making a custom <code>Tracer</code>. Doable, but not exactly convenient or modular.</p>
Optimize input instead of network weights

username: Mughees
cooked: <p>Hi,</p>
<p>I am trying to optimize the inputs in some required task and I don’t want to update my network as its freezed. I have written a minimal example but its not working as z has the same value in all iterations.  I am sure that I am doing some silly mistake in this process. Any guidance is highly appreciated.<br>
thanks.</p>
<pre><code class="lang-auto">import torch

z = torch.rand((1,6))
z.requires_grad_(True)
optimizer = torch.optim.SGD([z], lr= 0.1)

criteria = torch.nn.MSELoss()


for i in range(10):
    optimizer.zero_grad()
    print(z)
    loss = criteria(z, z+torch.rand(1))
    #print(loss)
    loss.backward()
    optimizer.step()

##output
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
</code></pre>

username: naveenvemy
cooked: <p>There is no learning happening here. You are adding a constant value to your z and using that as the target value while calculating MSE loss. In that case, the MSE will also be a constant and hence, the gradient of the loss function with respect to z will be 0. You can check the gradient values by printing them after each loop.</p>
<pre><code class="lang-auto">for i in range(10):
    optimizer.zero_grad()
    loss = criteria(z, z+torch.rand(1))
    #print(loss)
    loss.backward()
    optimizer.step()
    print(z.grad)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
</code></pre>
Set init weights to complicated model using weights from .pkl file

username: andreys42
cooked: <p>I’m wonder is that a correct way to init weights from pickled file for my model?<br>
.PKL file was saved after last epoch of previous training of the same model (unfortunatelly, .ckpt file lost)</p>
<pre><code class="lang-auto">model = FaultNetPL(batch_size = 5).cuda()
model_w = torch.load('model_weights.pkl')
model.load_state_dict(model_w )

trainer.fit(model)
</code></pre>
<p>I tried to look for analogous questions, but answer always is to use weight_init method (which is (as I understood) iterate over all of layers of the model and set weights to each if them semi-manually)</p>

username: ptrblck
cooked: <p>It depends, what <code>model_weights.pkl</code> contains. If it was stored using <code>model.state_dict()</code>, your approach would work. On the other hand, if it’s containing raw tensors or a custom <code>dict</code> etc. you would need to load it manually using the same logic which was applied while storing these values.</p>

username: andreys42
cooked: <p>Ty for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>. Is it ok that my model loss on the 1st epoch increased a little bit comparing to loss I got at the end of the last training (model_weights.pkl I got from)<br>
My intuition behind such behavior is that I didn’t restored optimizer and dataloader parameters …</p>

username: ptrblck
cooked: <p>Yes, both of your suggestions might be valid in case you are continuing the training.<br>
Assuming you are checking the training performance of the model, the order of the samples would matter, since each update depends on them.<br>
Also, if you are not restoring the optimizer, the first update step might increase the loss, especially if internal states were used in the optimizer.</p>
<p>However, the validation set should yield the same result up the the limited floating point precision.<br>
Make sure you are calling <code>model.eval()</code> before calculating the validation performance.</p>
Using both PyTorch and PyTorch Geometric

username: CorsBlady
cooked: <p>Hi there,</p>
<p>I’m currently working on a personal mini project and I wanted to utilise/combine both standard PyTorch as well as PyTorch Geometric, however, there are couple of things that seem to confuse me, primarily with regards to creating DataLoaders.</p>
<p>The project aims to predict a continuous quantity with respect to an agent that is surrounded by multiple other agents. Let’s call the agent of interest <strong>parent</strong> and other agents within the scene <strong>children</strong>.</p>
<p>The <strong>parent</strong> object has various attributes, some of which I want to process/encode with MLP and other with CNN. I also wanted to encode the relationship between <strong>parent</strong> and <strong>children</strong> through the use of Graph Neural Net. This would allow me to combine the output of MLP + CNN with the output of GNN (encoding of the nodes and graph) that could be processed further to get final prediction.</p>
<p>I have an issue with understanding how would one create a <code>torch.utils.data.DataLoader</code> that yields a batch of samples with regards to features of a <strong>parent</strong> agent as well as <code>torch_geometric.data.DataLoader</code> that yields the same batch with node features, edge indices etc, of both <strong>parent</strong> and <strong>children</strong>.</p>
<p>I went over couple of examples from the main PyTorch Geometric side but couldn’t find anything that would address my issues. Most of these example deal with data that is in pure graph form.</p>
<p>Thank you for any suggestions.</p>
Problem with batch size

username: nmourdou
cooked: <p>Hey guys,</p>
<p>I am facing a quite strange error, which I have not encountered before.</p>
<p>I have a data loader with batches of size 256. My training begins and everything is fine, meaning that I have the correct sizes for batch_x and and batch_y, being [256, 1, 28, 28] and [256] respectively.</p>
<p>Then something happens and these sizes become [96, 1, 28, 28] and [96].</p>
<p>Any ideas how this is even possible?</p>

username: matthewleigh
cooked: <p>Hard to say without seeing your dataset class.</p>
<p>But are you sure its not just the dataloader working as expected? If this happens at the end of the batch calls, then I would assume so.</p>
<p>Basically when the length of your whole dataset is not a hard multiple of the batch_size it means that when it comes to the last batch to load, you either have to load a smaller batch or drop it, using the <code>drop_last</code> argument in the dataloader.</p>
<p><a href="https://pytorch.org/docs/stable/data.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html</a></p>
<p>So if for example the training set was size 10 and the batch size was 3 you would have:</p>
<pre><code class="lang-auto">first batch  = 3
second batch = 3
third batch  = 3
fourth batch = 1
</code></pre>
List initialization of IntArrayRef. Why does it fail

username: Anders_G
cooked: <p>I have a function computing fft2 containing this code:</p>
<pre><code>  void DoSomeThing(Tensor *x)
  {
    c10::optional&lt;IntArrayRef&gt; n_opt = c10::nullopt;
    c10::optional&lt;IntArrayRef&gt; norm= c10::nullopt;
    IntArrayRef dim_arr = {-2, 1};
    Tensor z = torch::fft::fft2(*x, n_opt, dim_arr, norm);
  }
</code></pre>
<p>it will fail in release mode but not in debug mode. The issue seem to be that fft2 find huge values for dim_arr. E.g. values like 10032412311. (Note that if I print the content of dim_arr I got -2 and -1.) If I change it to</p>
<pre><code>  void DoSomeThing(Tensor *x)
  {
    c10::optional&lt;IntArrayRef&gt; n_opt = c10::nullopt;
    c10::optional&lt;IntArrayRef&gt; norm= c10::nullopt;
    int64_t arr[2] = {-2,-1};
    IntArrayRef dim_arr = IntArrayRef(arr, 2) ;
    Tensor z = torch::fft::fft2(*x, n_opt, dim_arr, norm);
  }
</code></pre>
<p>it works fine. Note that this also works fine:</p>
<pre><code>  void DoSomeThing(Tensor *x)
  {
    c10::optional&lt;IntArrayRef&gt; n_opt = c10::nullopt;
    c10::optional&lt;IntArrayRef&gt; norm= c10::nullopt;
    Tensor z = torch::fft::fft2(*x, n_opt,  {-2,-1}, norm);
  }
</code></pre>
<p>I probably missed some essential c++ here. That one cannot initialize a IntArrayRef using a list other than in a function argument list.</p>
<p>Someone who can explain what is happening?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/anders_g">@Anders_G</a><br>
IntArrayRef dim_arr = {-2, 1}; will most likely failed. Since {-2, 1} is temporary, ArrayRef won’t copy the value, it just reference the value. So the behavior of dim_arr is undefined if you do this. You might be able to print out the correct value, but the address is already released, so anything can happen.</p>

username: Anders_G
cooked: <p>Thanks, I suspected something along those line, but I was not sure.</p>
Changing Labels of Dataset in Pytorch for Continual Learning Setting

username: Akash_Tadwai
cooked: <p>We have CICIDS17 dataset that consists of 15 classes (1 Normal and 14 Attack labels). We are working in Continual Learning Setup in which we need to divide the data into a sequence of tasks (train and validation) i.e for example as we have 15 classes we used <code>pd.factorize</code> from pandas and converted the object labels into integer labels. For taskwise splitting the dataset, we looked at these integer labels and split the data with the task name added. The flow of our preprocessing steps is as follows,</p>
<p>Original Dataset → converting to torch.dataset.Dataset with 15 labels → Caching Dataset → Splitting Dataset based on Tasks.</p>
<p>Splitting Dataset based on Tasks consists of 2 steps:</p>
<ul>
<li>
<p>Subclass (A dataset wrapper that returns the task name and removes the offset of labels (Let the labels start from 0) )</p>
</li>
<li>
<p>Appendname (Appends the name of task with the dataset)</p>
</li>
</ul>
<p>Now, for our experiment we want the dataset to have the labels as 0 and 1 only (signifying whether the label is a normal class or attack class but the order should be maintained while creating tasks, i.e, in each task we are fitting exactly 3 different classes with each of its labels).</p>
<p>How to achieve this?</p>
<p>The code we have tried on 15 labels for 5 tasks is in <a href="https://github.com/ReethuVinta/AnomalyDetection/blob/696c0189c5dbc6f0f9ec18298440365480270bf6/iBatchLearn.py#L19" rel="noopener nofollow ugc">this</a>. Following the <a href="https://github.com/ReethuVinta/AnomalyDetection/blob/696c0189c5dbc6f0f9ec18298440365480270bf6/dataloaders/datasetGen.py#L6" rel="noopener nofollow ugc">SplitGen</a> function in the code would give a better idea of what we are trying to do.</p>
<p>We want to change the target labels for each task to either only 0 or 1. How to achieve this.</p>

username: Akash_Tadwai
cooked: <p>As I am a new user. It restricted me to put the links for the dataset and the Continual Learning related info in the original post. So here they go.<br>
CICIDS17 Dataset: <a href="https://www.kaggle.com/cicdataset/cicids2017" class="inline-onebox" rel="noopener nofollow ugc">CICIDS2017 | Kaggle</a><br>
Continual Learning Review: <a href="https://arxiv.org/pdf/1802.07569.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1802.07569.pdf</a></p>
PyTorch on Embedded Hardware

username: Werner2005
cooked: <p>Hi,</p>
<p>Background: My students and I currently build a small model sized autonomous car.<br>
At this moment we have Hardware form Texas Instrument (TI AM572x) with 2x ARM Cortex-A15 (Linux based), 2x DSP (C6000) and 4 x EVA(Vector Processors) like used in new BeagleBone AI (<a href="https://beagleboard.org/ai" class="inline-onebox" rel="noopener nofollow ugc">BeagleBoard.org - AI</a>)</p>
<p>Currently we can run PyTorch and the libtorch(we use the C++ Backend) on the Cortex-A15 Cores in Linux (Extremely slow the ARM Prozessors are not designend to execute code like this)<br>
The Vector Processors or the DSP on this Platform was original designed to run code linke this.</p>
<p>Problem:<br>
Texas Instrument support only OpenCL as Middelware for Computing on the Veotor Cores and on the DSPs. CUDA is not supported by now.<br>
I know there no plan to Support OpenCL in the future but:<br>
Is out there some working solution to run the PyTorch CUDA Code on a OpenCL Hardware?<br>
We found some Projekt but we are not sure this would work(<a href="https://github.com/hughperkins/coriander" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hughperkins/coriander: Build NVIDIA® CUDA™ code for OpenCL™ 1.2 devices</a>).</p>
<p>Best,<br>
Andy</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/werner2005">@Werner2005</a><br>
Can you create an issue in our github repo, we can have some CUDA experts answer this question there.</p>
Linear Regression loss increase and data scaling

username: Mohamed_Sharaf
cooked: <p>Hi everyone,<br>
I’m new to Pytorch and wanted to use it for simple Linear Regression for sensor data. The features are the longitude and latitude and the labels are pressure (values from 0-200) I found that when changing the scale of the data to values between 0 &amp; 1, the loss is decreasing. Is there a reason for that? The model is just linear regression, nothing else.</p>
<p><span class="hashtag">#Model</span></p>
<p>features=train_X.shape[1]<br>
input_size=features<br>
output_size=1<br>
model=nn.Linear(input_size,output_size)<br>
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
<span class="hashtag">#send</span> to GPU if available<br>
model.to(device)<br>
train_X=train_X.to(device)<br>
train_Y=train_Y.to(device)<br>
validate_X=validate_X.to(device)<br>
validate_Y=validate_Y.to(device)</p>
<p><span class="hashtag">#loss</span> &amp; optimizer<br>
learning_rate=0.01<br>
epochs=100<br>
criterion=nn.MSELoss()<br>
optimizer=torch.optim.SGD(model.parameters(),lr=learning_rate)</p>
<p><span class="hashtag">#training</span><br>
for epoch in range(epochs):<br>
<span class="hashtag">#forward</span><br>
y_predicted=model(train_X)<br>
loss=criterion(y_predicted,train_Y)<br>
<span class="hashtag">#backward</span><br>
loss.backward()<br>
<span class="hashtag">#updates</span><br>
optimizer.step()<br>
<span class="hashtag">#zero</span> gradients<br>
optimizer.zero_grad()<br>
<span class="hashtag">#print</span> info<br>
if(epoch+1) % 10 ==0:<br>
print(f’epoch: {epoch+1}, loss={loss.item():.4f}’)</p>

username: googlebot
cooked: <p>MSE formula is same as gaussian log-likelihood.<br>
So you’ve changed gaussian_lpdf(200, m=y_pred, std=1) to gaussian_lpdf(200/200, m=y_pred, std=1), and gradient magnitudes.</p>

username: Mohamed_Sharaf
cooked: <p>Thank you Alex, yes I got better results without scaling when used nn.L1Loss() instead</p>
LibTorch with CUDA 10.1?

username: baris
cooked: <p>Is it possible to use LibTorch with CUDA version 10.1 ? It seems like only 10.2 and 11.1 are supported.</p>

username: glaringlee
cooked: <p>You can find more versions here:<br>
‘<a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a>’</p>