akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Difference of : Direct assign vs. .copy_()

LeeDoYup: <p>Hello, in order to get EMA model, i tried the first line in the below block.</p>
<pre><code class="lang-auto">        for param_net, param_eval_net in zip(self.train_model.parameters(), self.eval_model.parameters()):
            # param_eval_net.data = param_eval_net.data * self.ema_m + param_net.data.detach() * (1. - self.ema_m)
            param_eval_net.copy_(param_eval_net * self.ema_m + param_net.detach() * (1. - self.ema_m))
</code></pre>
<p>However, i found that it didn’t actually update the parameter value.</p>
<p>When i tried to use <code>.copy_</code> method,  i found that the model is normally updated.<br>
What is the difference between two methods??</p>

albanD: <p>Hi,</p>
<p>In general, <code>.data</code> should never be used anymore <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  So you should not use the first line.<br>
Doing a copy_ is the right thing to do to update the value.</p>
Change CUDA device index via Libtorch API

Abrar_Zahin_Shahriar: <p>Libtorch C++ API seems to be having trouble detecting my Tegra CUDA device(Jetson Xavier NX), on Python3 it has no problems but on C++ it cannot. I want to set the CUDA device index using when using Libtorch on C++. How to do it?</p>

tom: <p>Are you using the same libtorch? (You can take the one from PyTorch itself.)</p>

Abrar_Zahin_Shahriar: <p>Using the one taken from Pytorch itself. I fixed the issue after building Pytorch from source and then adding – no-as-needed in the makefile.</p>
Generate one hot encoding, from probability matrix fast

tueboesen: <p>I asked this question on stack overflow with numpy in mind, but I’m curious to see whether pytorch have a good solution for this, as I’m intending to use it in pytorch. (Stackoverflow question here: <a href="https://stackoverflow.com/questions/63948265/generate-random-vectors-with-a-given-numerical-distribution-matrix" rel="nofollow noopener">https://stackoverflow.com/questions/63948265/generate-random-vectors-with-a-given-numerical-distribution-matrix</a> )</p>
<p>I’m trying to come up with a fast and smart way of generating random one-hot vectors from a distribution matrix.</p>
<p>To give a quick example of what I want to do: Given a one-probability matrix of shape (n,3):</p>
<pre><code class="lang-auto">p = torch.tensor([[0.2, 0.4, 0.4],[0.1, 0.7, 0.2],[0.44, 0.5, 0.06],...]) 
</code></pre>
<p>I wish to draw n elements, where each element can be either 0,1,2 and gets selected with the probability given in the above probability matrix.</p>
<p>Is there a fast way of doing this that doesn’t involve for loops, as I want to do this as data augmentation and therefore needs to be fast.</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>You can try</p>
<pre><code class="lang-auto">_, lbls = torch.max(p, 1) 
</code></pre>
<p>Thanks</p>

googlebot: <pre><code class="lang-auto">torch.zeros_like(p).scatter_(1, torch.multinomial(p,1), 1.)
</code></pre>
Is there any relation between index in __getitem__ method and what __len__ returns?

nivesh_gadipudi: <p>How pytorch manages to give index the value?, is it something depending on the length returned from <strong>len</strong> method?</p>

tom: <p>The dataloader will permust the sequence 0…len-1 and then use these as indices on getitem.</p>

nivesh_gadipudi: <p>Thanks for the clarification <a class="mention" href="/u/tom">@tom</a>. I initially thought of that but anyway thanks for making it clear.</p>
PyTorch docs and deep learning book

Guanqun_Yang: <p>I have completed most of the beginner and intermediate level tutorials and now I am trying to read the entire docs together with Bengio’s deep learning book. I have several questions:</p>
<ul>
<li>Issue of docs of PyTorch. I found that the documentation is not quite organized and I did not know where to start. So what is the relationship between different modules. For example, the “root” module determined the activity of all other modules and there are also similar master-slave relationship between other modules. Hopefully there would be a tree-view of the entire PyTorch package.</li>
<li>Issue of using PyTorch together with deep learning book. There are bunch of algorithms and fancy math in the deep learning book, and I could generally understand them. However, I do not know how there things are implemented in PyTorch and I want most of the details in the book make sense. Should I delve into the source code of PyTorch and how?</li>
</ul>

Lisa_McCoy: <p>Wow this is quite wonderful. I am still at the beginner level and reading the docs along with some <a href="https://www.bartleby.com/subject/math/advanced-math" rel="nofollow noopener">advanced math</a> work. Really enjoying going through the notes and gathering my own reflections. Thanks!</p>
How could I debug pytorch ops with Clion?

Bob_Liao: <p>Hi, here am I again. I have compiled pytorch from source code with the help of the CONTRIBUTTING.md.  And I want to know how can I use Pytorch within C++ source files without Python? And how can I debug this with some kind of IDE like Clion?</p>
<p>For example, I just want to know what happened during two tensors add together. So I can just write <code>a+b</code> （a &amp; b are Tensors), and debug into this operation.</p>
<p>I know this question may seem simple, but happy to have you guys help. Thanks!</p>
Placeholder in pytorch

pascal_notsawo: <p>What is the equivalent of tf.placeholder in pytorch?</p>

ptrblck: <p>You can pass tensors directly to modules in PyTorch without placeholder variables.<br>
Have a look at the <a href="https://pytorch.org/tutorials/">tutorials</a> for some examples and an introduction. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Different length sequences as batch input to LSTM

farhad-bat: <p>Hello,</p>
<p>I have an LSTM that outputs a float number in each time-step. (It’s regression, not classification). I want my batches have sequences with different sizes. But I don’t know how to implement such thing in PyTorch.</p>

Mazhar_Shaikh: <p>Hi <a class="mention" href="/u/farhad-bat">@farhad-bat</a>,<br>
Please have a look at this <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence" rel="nofollow noopener">link</a>. The pad sequence function will pad your sequences to make them all of the same length.</p>

vdw: <p>You have two basic alternatives:</p>
<ul>
<li>
<a href="https://discuss.pytorch.org/t/understanding-pack-padded-sequence-and-pad-packed-sequence/4099/17">Use padding (&amp; packing).</a> That’s a standard practice to handle batch sequences of different lengths.</li>
<li><a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">Organize your batches in such way that all sequences withing one batch have the same length.</a></li>
</ul>

seyeeet: <p>which oneis a beeter way though?</p>

vdw: <p>There’s probably not one best solution. For my work, I usually go for organizing my batches to avoid the hassle of padding and packing. But that’s just for basic research prototypes, and I’m the one looking at the code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>There are <a href="https://discuss.pytorch.org/t/is-padding-and-packing-of-sequences-really-needed/66478">valid arguments against this approach</a>. At least for practical purposes.</p>
Program freezes when doing distributed training

xdwang0726: <p>I am trying to add distributed training on my program as my model is relatively large. The program runs well without distributed training, however, when I add distributed training, the program freezes right at <code>model = torch.nn.parallel.DistributedDataParallel(model)</code> without returning any error messages. I am wondering have anyone else faced this situation before? Are there any possible solutions? Thanks!</p>

mrshenli: <aside class="quote no-group" data-username="xdwang0726" data-post="1" data-topic="96463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xdwang0726/48/17441_2.png" class="avatar"> xdwang0726:</div>
<blockquote>
<p>model = torch.nn.parallel.DistributedDataParallel(model)</p>
</blockquote>
</aside>
<p>When doing the above without specifying a <code>device_id</code>, it will try to replicate the model to all visible devices in each process (unless the model is on CPU). Is this intentional? The recommended use of DDP is to let each process exclusively operate on one GPU.</p>
<p>Beside, before v1.7 DDP will create communication buckets. The total size of those buckets will be the same as model size. So the GPU memory size needs to be at least 3X of the model size.</p>

xdwang0726: <p>Thanks for your reply! I have specified the cuda ids when training. When I trained in a single GPU, it returns the error message that ‘CUDA requires ~6G more memory’. I add one more 16G GPU to do the training. According to your suggestion, it seems like I need at least 4 GPUs in total? Thanks!</p>

xdwang0726: <aside class="quote no-group" data-username="mrshenli" data-post="2" data-topic="96463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>the GPU memory size needs to be at least 3X of the model size.</p>
</blockquote>
</aside>
<p>Thanks for your reply! I have specified the cuda ids when training. When I trained in a single GPU, it returns the error message that ‘CUDA requires ~6G more memory’. I add one more 16G GPU to do the training. According to your suggestion, it seems like I need at least 4 GPUs in total? Thanks!</p>

xdwang0726: <p>I got it work, never mind, thanks</p>

mrshenli: <p>Hey <a class="mention" href="/u/xdwang0726">@xdwang0726</a>, do you mind sharing what was the cause of the problem, and how it was resolved? In case future users hit the same issue.</p>

xdwang0726: <p>Before I was using <code> local_rank = torch.distributed.get_rank()</code> and the program freezes. I manually set <code>local_rank=-1</code> that solves the problem.</p>
Fasttext embedding with small window size

Aiman_Mutasem-bellh: <p>Dear all,</p>
<p>I’m using fasttext with a machine translation task.</p>
<p>The challenge I have is how to apply fasttext embedding with a small window less than the default value of [5].</p>
<p>url = ‘<a href="https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.ar.vec" rel="nofollow noopener">https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.ar.vec</a> ’</p>
<p>SRC.build_vocab(train_data, vectors=Vectors(‘wiki.ar.vec’, url=url), unk_init = torch.Tensor.normal_, min_freq = 2)</p>
<p>I have found the below code but I’m not sure about how to build vocab using <code>torchtext.data Field</code></p>
<pre><code class="lang-auto">from gensim.models import FastText
model_ted = FastText(sentences_ted, size=300, window=5, min_count=5, workers=4,sg=1)
</code></pre>
<p>Any suggestions?</p>
<p>Regards,</p>

ecdrid: <p>Can you share the pseudo code in complete with proper formatting?<br>
Also NB, if a model is pre-trained and you are going to use it, then we have to use those defaults with which the model was trained very likely.<br>
Plus if you are looking to train your own, then <a href="https://radimrehurek.com/gensim/models/fasttext.html" rel="nofollow noopener">here’s</a> a good starter!</p>
<p>Or is it that you want to only build the vocab with torchText?</p>
<p>Ty!</p>

Aiman_Mutasem-bellh: <p>Thank you <a class="mention" href="/u/ecdrid">@ecdrid</a> for your valuable comment.</p>
<p>My model is based on CNN seq2seq and I need to use fasttext embedding with a small window. Honestly, I’m not sure about the optimal way to do that.</p>
<p>I have read the article you cited, I think is better to use fasttext embedding for words representation as a separate feature.</p>
<p>Kind regard,</p>
Multi-Label Semantic Segmentation on imbalanced data

krioux5: <p>Hello,</p>
<p>I am working on a 9 label semantic segmentation on head CT scans. The data has a very imbalanced distribution. ie. Most frequent label: 98.8% of data corresponding to unlabeled structures in the scan, least frequent: 0.001% of data corresponding to the smallest structure in the scan. I have tried giving crossentropyloss weights corresponding to the inverse of their frequency with poor results. The largest structure (about 1% of the data) was FAR overrepresented in the segmentation, and some smaller structures were not identified. I then tried the same training with no crossentropyloss weights and the 1% structure had a more reasonable representation but was still not well identified, and all smaller structures remained unsegmented. Do you have any recommendations, perhaps specific training techniques I could use to handle a problem such as this?</p>
<p>Mandatory tag of the legendary <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you,</p>
<p>Kyle</p>
Warning: ignoring #pragma unroll during compilation of custom extension

gouthamvgk: <p>I am writing a custom extension that implements a modified grid_sample layer. I use the source code from Aten/native/cpu/GridSamplerKernel.cpp by making a few changes to it. Followed the steps in pytorch website and used the CppExtension to compile the layer. Since the source code in GridSamplerKernel.cpp involved <span class="hashtag">#pragma</span> unroll i also passed the “-O3” flag to the extra compile_args of CppExtension as follows</p>
<pre><code class="lang-auto">CppExtension(
            "alignSingle",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=e['-O3'],
        )
</code></pre>
<p>. But still, it shows that the pragma unroll is ignored during the compilation. Is there anything else to add for compiling the custom layers with pragma unroll?. To reproduce the warning run ./make.sh from the source code here <a href="https://drive.google.com/file/d/1hBmWyK3UvQMd3FRFD5_5tL50WvCtRegp/view?usp=sharing" rel="nofollow noopener">https://drive.google.com/file/d/1hBmWyK3UvQMd3FRFD5_5tL50WvCtRegp/view?usp=sharing</a> . NOTE: I am using ubuntu and have to set the flag AT_MKL_ENABLED() at include/ATen/Config.h to 0 before compiling. Otherwise it throws the error no &lt;mkl.h&gt; file during compilation.</p>

googlebot: <p>-funroll-loops perhaps?</p>

gouthamvgk: <p>It still throwing the same warning after adding -funroll-looops to extra_compile_args</p>

googlebot: <p>Actually, <span class="hashtag">#pragma</span> unroll form is for nvcc<br>
GCC form is: <span class="hashtag">#pragma</span> GCC unroll N</p>

gouthamvgk: <p>Unfortunately it also didn’t work</p>

googlebot: <p>clang has different syntax, but frankly that pragma should make almost no difference, and some unrolling is done anyway, based on optimization flags.</p>
How to Use Sentence Embeddings with Pytorch?

eprox: <p>I have been working with torchtext word embeddings (ex. GloVe) and now I would like to experiment with sentence embeddings.<br>
I am using FAIR LASER: <a href="https://pypi.org/project/laserembeddings/" rel="nofollow noopener">https://pypi.org/project/laserembeddings/</a><br>
With Word embeddings, I would carry out the following:</p>
<ol>
<li>Load Embedding Layer with Embedding.from_pretrained()</li>
<li>Feed to Linear Layer, LSTM and some other linear layers.</li>
</ol>
<p>I have simply replaced the word embeddings with the FAIR sentence embeddings, but the network is not learning. Is this an incorrect way to approach this?</p>

BramVanroy: <p>By default, pretrained embeddings are frozen. You have to add <code>freeze=False</code> to the options that you pass.</p>

eprox: <p>could you please elaborate on how this would help?</p>

BramVanroy: <p>When you “freeze” a model (or part of the model, or some layers in the model), you effectively disable learning. You set <code>requires_grad=False</code> meaning that no gradients will be calculate for that part in the model - so the model will not learn, i.e. the gradients will not be calculated and the optimizer will not update the weights. If you want to make those pretrained models “trainable” (or "fine-tune-able), you have to disable freezing.</p>
Access to saved context in backward function of a module

ttoosi: <p>Hi,</p>
<p>I want to change the saved input to a module that is used to calculate the gradients in the backward call.<br>
To be clear, let’s say I want <em>AdaptiveAveragePool2d</em> computes the gradients given an <em>input2</em> rather than the <em>input</em> that passed through it during <em>forward</em> call.<br>
One would write a custom autograd function where <code>input2</code> is saved in <em>ctx</em> instead of input. But I was wondering if there is a workaround? Implementing <em>backward</em> function of <em>AdaptiveAvgPool</em> from scratch has proven tricky.</p>
<p>Thanks,<br>
Tahereh</p>

albanD: <p>Hi,</p>
<p>I am afraid this is not really possible. This would compute the “wrong” gradients and so autograd will raise an error if you try to change the input inplace before computing the backward.</p>
<p>For the particular case of <code>AdaptiveAveragePool</code>, the input value is not actually needed when you compute the backward (just its size). So I am not sure what you expect to happen if you modify it?</p>

ttoosi: <p>Ahh! You’re right! I miss the point that I don’t need values for AdaptiveAveragePool!<br>
Good to know that there is no way to do this in general.</p>
Get exact formula used by autograd to compute gradients

Poornima_Jain: <p>I am implementing a custom CNN with some custom modules in it. I have implemented only the forward pass for the custom modules and left their backward pass to autograd.<br>
I have manually computed the correct formulae for backpropagation through the parameters of the custom modules, and I wished to see whether they match with the formulae used internally by autograd to compute the gradients.<br>
Is there any way to see this?<br>
Thanks</p>

albanD: <p>Hi,</p>
<p>I am afraid this will be tricky as autograd does not build a single formula but compute the gradient as a chain of operations.<br>
You can use things like <a href="https://github.com/szagoruyko/pytorchviz/">https://github.com/szagoruyko/pytorchviz/</a> to see what is the set of functions that will be executed in the backward pass. But you will have to manually “unpack” these and write down the formula for each and then combine all of them together.</p>
<p>Note that if you want to make sure your custom Function (with the manual gradient) computes the right thing, you can use <code>autograd.gradcheck()</code>.</p>

Poornima_Jain: <p>Thanks <a class="mention" href="/u/alband">@albanD</a>.<br>
I created a Pytorchviz visualization of my net. But it has some functions like ‘Slice backwards’ etc which I am not sure what they do.<br>
Is the list of functions used in Pytorchviz same as the list of 'grad_fn’s in Pytorch?<br>
Also, is the list of all 'grad_fn’s in Pytorch available somewhere?<br>
Thanks</p>

albanD: <blockquote>
<p>Is the list of functions used in Pytorchviz same as the list of 'grad_fn’s in Pytorch?</p>
</blockquote>
<p>Yes it is just a visualization of the grad_fn in PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  You can check the code, it just traverses the grad_fn and next_functions.</p>

Poornima_Jain: <p>Okay, thanks for the clarification.<br>
I have some grad_fns like ‘Slice backward’ in my pytorchviz graph, which I am not sure how it works. Where can I find what the function does or I need to get what it does on the basis of the operation in the forward pass?<br>
Thanks</p>

albanD: <p>The specification for what the backward does is <a href="https://github.com/pytorch/pytorch/blob/bee97d5be0fc89e415c611054d1f6645a35e5577/tools/autograd/derivatives.yaml#L937">here</a> And the functions it used can be found either <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp">here</a> or with the other aten functions.<br>
For slice in particular, it is <a href="https://github.com/pytorch/pytorch/blob/69f6d94caa3559d4f50745c26af5df041b83fee8/aten/src/ATen/native/TensorShape.cpp#L1059">here</a>.</p>
PyTorch equivalent of tf.where in Tensorflow 1

Niels_PyTorch: <p>I am converting code from Tensorflow 1 to PyTorch. I want to convert a <code>tf.where</code> operation. PyTorch’s <code>torch.where</code> function is equivalent to <code>tf.where</code> in Tensorflow 2, but not in Tensorflow 1, which does not yet support broadcasting similar to <code>np.where</code>, as one can see from the example below:</p>
<pre><code class="lang-auto">import torch

condition = torch.tensor([True, True, True, True])

x = torch.tensor([[0.],
        [0.],
        [0.],
        [0.]])

y = torch.tensor([[1.],
        [0.],
        [0.],
        [0.]])

torch.where(condition, x, y)
</code></pre>
<p>gives as result:</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]])
</code></pre>
<p>whereas in Tensorflow 1:</p>
<pre><code class="lang-auto">import tensorflow.compat.v1 as tf

condition = tf.convert_to_tensor([True, True, True, True])

x = tf.convert_to_tensor([[0.],
        [0.],
        [0.],
        [0.]])

y = tf.convert_to_tensor([[1.],
        [0.],
        [0.],
        [0.]])

tf.where(condition, x, y)
</code></pre>
<p>the result is</p>
<pre><code class="lang-auto">&lt;tf.Tensor: shape=(4, 1), dtype=float32, numpy=
array([[0.],
       [0.],
       [0.],
       [0.]], dtype=float32)&gt;
</code></pre>
<p>So my question is: how to get the same result as in Tensorflow 1 with PyTorch?</p>

Niels_PyTorch: <p>Turns out to use the following:</p>
<p><code>torch.where(condition.view(x.size()), x, y)</code></p>
Pronbelm while implementing the tiny YOLO v3 model

Rahul_rampa: <p>Hello guys,<br>
I have been trying to implement the tiny YOLO v3 model and when I try to load the pre-trained weights into the model,I get this error.</p>
<p>File “/home/rahulrampa/.spyder2/Yoloimplementation.py”, line 274, in loadweights<br>
conv_weights=conv_weights.view_as(conv.weight.data)</p>
<p>RuntimeError: shape ‘[512, 256, 3, 3]’ is invalid for input of size 723278</p>
<p>Any ideas on how to fix this problem?<br>
Thanks in advance</p>

Kuppharish: <p>I am getting the same error. Did you find any solution?</p>
Will 0 padded entries be updated during training for torch.nn.Embedding?

pmixer: <p>Hi Team,<br>
Just out of curiosity, for <code>torch.nn.Embedding</code>, if we set <em>padding_idx=0</em>, will the first entry of embedding table be updated during model training(cause its special entry just for alignment and dry-run in models and would be unnecessary if we really want to make all weights of embedding table trainable).</p>
<p>Regards,<br>
Zan</p>

ptrblck: <p>No, the weight row will be initialized as all zeros and won’t get any gradients as seen here:</p>
<pre><code class="lang-python">emb = nn.Embedding(10, 5, padding_idx=0)
x = torch.arange(10).view(1, -1)
out = emb(x)
out.mean().backward()
print(emb.weight)
print(emb.weight.grad)
</code></pre>

pmixer: <p>Thank you Piotr for the answer! PyTorch is really AWESOME <img src="https://discuss.pytorch.org/images/emoji/apple/metal/2.png?v=9" title=":metal:t2:" class="emoji" alt=":metal:t2:"></p>
Training model across multiple remote servers- each having multiple GPUs

motor_junkie: <p>Is it possible to train a model across multiple remote servers in my department? These servers are not connected to each other. I want to use GPUs of both the servers (with different IP addresses) so that I can train with larger batch size.</p>
<p>I have seen <a href="https://pytorch.org/docs/stable/nn.html#distributeddataparallel" rel="nofollow noopener"> <code>nn.DistributedDataParallel</code> </a> but how do I mention the IP address of multiple servers?</p>

mrshenli: <aside class="quote no-group" data-username="motor_junkie" data-post="1" data-topic="96312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/motor_junkie/48/38446_2.png" class="avatar"> motor_junkie:</div>
<blockquote>
<p>These servers are not connected to each other.</p>
</blockquote>
</aside>
<p>What does this mean? Their IPs are not reachable from each other?</p>
<blockquote>
<p>I have seen <a href="https://pytorch.org/docs/stable/nn.html#distributeddataparallel" rel="noopener nofollow ugc"> <code>nn.DistributedDataParallel</code>  </a>but how do I mention the IP address of multiple servers?</p>
</blockquote>
<p>If they can reach each other through network, yes, <code>DistributedDataParallel</code> can work across multiple machines. You need to provide the master address and master IP for all peers to do rendezvous. See <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md" rel="noopener nofollow ugc">this example</a>.</p>
<p>If you want to choose a specific network interface, you can configure the following two env vars (<a href="https://pytorch.org/docs/stable/distributed.html#common-environment-variables" rel="noopener nofollow ugc">more details</a>). You only need one of them depending on which backend you are using.</p>
<pre><code class="lang-auto">NCCL_SOCKET_IFNAME, for example export NCCL_SOCKET_IFNAME=eth0
GLOO_SOCKET_IFNAME, for example export GLOO_SOCKET_IFNAME=eth0
</code></pre>

motor_junkie: <p>I think I didn’t describe my situtation correctly. What I meant is that they are different systems. One has IP: a.b.c.<strong>d</strong>, and the other has IP: a.b.c.<strong>e</strong>.</p>
<p>Okay, thanks! I’ll try it out</p>

mrshenli: <blockquote>
<p>I think I didn’t describe my situtation correctly. What I meant is that they are different systems. One has IP: a.b.c. <strong>d</strong> , and the other has IP: a.b.c. <strong>e</strong> .</p>
</blockquote>
<p>I see. This should be fine. You only need to specify one of them as the master, and set <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> for all peers to point to that master. This will allow all peers to do rendezvous, and the rendezvous process will create connections between pairs.</p>
Faster way to partially add tensors (+ for only some channels)

Sharpy: <p>It seems that working with parts of Tensor like this</p>
<pre><code class="lang-auto">x[:,:channels] += y
</code></pre>
<p>is very time consuming for some reason.<br>
For example,</p>
<pre><code class="lang-auto">x += x
</code></pre>
<p>works much faster even when that requires much more GPU computation.<br>
Is there any faster more optimal way to change and address Tensors like this, or maybe get some “compilation”-type speedups? I viewed PyTorch implementations of PyramidNet, which is basically heavy on this kind of requirement, but everyone seems to just</p>
<pre><code class="lang-auto">zero-pad y
x += y
</code></pre>
<p>as in the paper, which is obviously a wasteful hack around crappy frameworks.</p>

Sharpy: <p>Oops, seems like that delay was mostly caused by me not removing</p>
<pre><code class="lang-auto">torch.autograd.set_detect_anomaly(True) 
</code></pre>
<p>after a debug. Might be “mostly”, as this</p>
<pre><code class="lang-auto">class HalfReLU(nn.Module):
	def __init__(self):
		super(HalfReLU, self).__init__()

	def forward(self, x):
		half = x.shape[1]//2
		y = nn.functional.relu(x[:, :half], inplace=True)
		return torch.cat(( y, x[:, half:] ), dim=1)
</code></pre>
<p>still works on the order of 20x slower than plain ReLU, but I’ll post back later only if I’ll find it to have severe impact on what I’m doing.</p>
Does Pytorch provide a ready to use ConvGRU?

Dipayan_Das: <p>Hello,<br>
I have been working on video compression lately. Even though 3D CNN is an option, I am eager to use something like ConvGRU (GRU with convolutional gates, instead of dense layer gates). The concept has been around for almost half a decade now, however I am unable to find any ready to use module of ConvGRU in torch.<br>
Even though i hav found Git repos containing ConvGRU models, the architecture assumes by default that there isnt any spatial maxpooling in between layers. So compression of data becomes tedious.<br>
Any help would be appreciated.</p>
<p>And on a second note, if I am to implement an autoencoder using ConvGRU layers, would replacing Conv2d by Conv2dTranspose inside the ConvGRU modules do the trick ? Or should i just carry on with ConvGRU and only use MaxUnpool2d() to build the decoder part ?</p>
Does amp training can use torch.nn.DataParallel at the sametime？

Feywell: <p>I try to use amp with pytorch1.6 to speed up my training code.<br>
But I have a problem ,when I use nn.DataParallel.</p>
<p>I print some Intermediate variable.<br>
I find the tensor is float16 in one gpu, but float32  in two gpus.<br>
Is it support DataParallel model to use mixed-precision training?</p>
<p>in one gpu:</p>
<blockquote>
<p>fake_image_orig:  torch.float16<br>
gen loss:  torch.float32<br>
discriminator_out dtype:  torch.float16<br>
pred_fake:  torch.float16<br>
amp discriminor<br>
discriminator_out dtype:  torch.float16<br>
self.get_zero_tensor(input) dtype:  torch.float16<br>
input dtype:  torch.float16<br>
self.get_zero_tensor(input) dtype:  torch.float16</p>
</blockquote>
<p>two gpus:</p>
<blockquote>
<p>discriminator_out dtype:  torch.float32<br>
self.get_zero_tensor(input) dtype:  torch.float32<br>
input dtype:  torch.float32<br>
self.get_zero_tensor(input) dtype:  torch.float32<br>
input dtype:  torch.float32<br>
discriminator_out dtype:  torch.float32<br>
self.get_zero_tensor(input) dtype:  torch.float32<br>
input dtype:  torch.float32<br>
self.get_zero_tensor(input) dtype:  torch.float32<br>
input dtype:  torch.float32<br>
fake_image_orig:  torch.float32<br>
gen loss:  torch.float32<br>
discriminator_out dtype:  torch.float32<br>
pred_fake:  torch.float32<br>
fake_image_orig:  torch.float32<br>
gen loss:  torch.float32<br>
discriminator_out dtype:  torch.float32<br>
pred_fake:  torch.float32<br>
fake_image_orig:  torch.float32<br>
gen loss:  torch.float32<br>
fake_image_orig:  torch.float32<br>
gen loss:  torch.float32<br>
discriminator_out dtype:  torch.float32</p>
</blockquote>

ptrblck: <p>Yes, amp is usable in <code>nn.DataParallel</code> as described <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#dataparallel-in-a-single-process">here</a>.<br>
I guess you might have missed this note about the <code>@autocast()</code> decorator for the <code>forward</code> method, if you are using <code>nn.DataParallel</code>.</p>

BramVanroy: <p>In cases where multi-GPU can be set by command line arguments, and the script thus has to work in single as well as multi-GPU scenarios, can we just put the autocast in forward for single GPUs as well?</p>

Feywell: <p>In face, I try to experiment  in another machine with pytorch-nightly.<br>
I find it is work!<br>
It is weird.</p>

ptrblck: <p>Yes, that wouldn’t be necessary, but shouldn’t hurt either.</p>

BramVanroy: <p>Just to be clear: it is necessary in the DP cases, but not so in single GPU scenarios?</p>

ptrblck: <p>Yes, that is correct.<br>
The <code>forward</code> method has to be annotated for <code>nn.DataParallel</code> and DDP - multiple GPUs per process, not for single GPU or DDP - single GPU per process (recommended and fastest approach) use cases.<br>
You should be able to do it anyway without any disadvantages, but let us know, if it’s breaking.</p>

Feywell: <p>It means I need use amo.autocast in eyery single forward for nn.DataParallel and DDP in multiGPUs</p>
Different result using torch.mean cpu and cuda

jo9sie: <p>Hi,<br>
I am using pytorch 1.3 but found torch.mean get different result when using cpu and cuda<br>
a = torch.ones(2,3,4)<br>
a.mean([1,1])<br>
If I give mean the same dim twice ,cpu will compute twice ,and cuda will only compute ones.<br>
but I also notice cpu sum only compute once, if I give the same dim twice.<br>
can anybody give me some help with this?</p>

tom: <p>Don’t pass the same dim multiple times, that is an error. If you upgrade PyTorch you get an error</p>
<pre><code class="lang-auto">RuntimeError: dim 1 appears multiple times in the list of dims
</code></pre>
<p>So it wasn’t properly defined before, and now we explicitly check.</p>
<p>Best regards</p>
<p>Thomas</p>
Faster-RCNN state_dict size compared to EfficientDet

yohann84L: <p>I made a detection model using Faster-RCNN from <code>torchvision</code>. I use as backbone <a href="https://github.com/rwightman/pytorch-image-models" rel="nofollow noopener">timm</a> library.<br>
Using for example <code>mobilenetv2_110d</code> I end up with a model size of 1.2Go by saving only the state_dict (<code>torch.save(model.state_dict(), "model.pth")</code>.</p>
<p>However, using EfficientDet with the same backbone (again from Ross Rightman <a href="https://github.com/rwightman/efficientdet-pytorch" rel="nofollow noopener">efficientdet-pytorch</a> ) I end up with a model of 47mb.</p>
<p>I understand there is a big difference between those 2 models in terms of params, but here saved model is 25x times bigger.<br>
Is there a better way to save weights of FasterRCNN ?</p>

tom: <p>If the file size is of concern to you, you could try to pack fp32 things into bfloat16 or so to spend less bytes per parameter.</p>

yohann84L: <p>This is not really a concern as I don’t really have a limit. However, it is kind of weird that FasterRCNN is 25x bigger, and I’m wondering if I did something wrong or maybe if there is a better way to do it.</p>
<p>But thank you for this tips, I hadn’t thought about it!</p>

tom: <p>OK, so something might be funny:<br>
When I look at my model cache, I have a ResNet50 at 98M and a MaskRCNN with ResNet50 backbone at 170M, so I’d expect a markup of, say 100M or maybe 200M at most.</p>
Accessing output scores instead of outputs classes

mbehzad: <p>Hi,</p>
<p>I am using a pre-trained ResNet18 model. How can I access/use the last FC-layer scores before softmax in <code>outputs = model(inputs)</code>.</p>

ptrblck: <p>The <code>torchvision</code> implementation of ResNet18 doesn’t use a softmax in the module and the output will contain logits.<br>
If you want to compute the predicted class labels, you can use <code>preds = torch.argmax(outputs, dim=1)</code>.</p>
Problem with narrowed tensor and torch.save()

navid_mahmoudian: <p>I noticed that when torch.save() is used on a narrowed version of a tensor, it saves the whole tensor, but it returns the narrowed tensor. To me, this looks like a bug because it saves something more than what it is intended, and we are not able to recover this redundant info. I wanted to discuss this issue before reporting it as a bug. For that, I wrote a simple example, and you can check the output to see what I mean:</p>
<pre><code class="lang-auto">import torch
import os


x= torch.randint(0, 100, size=(10000,8), dtype=torch.long)

y1 = x.narrow(dim=0, start=0, length=10)
torch.save({"y": y1},"y1.pth")

y2 = x.narrow(dim=0, start=0, length=10).detach().clone()
torch.save({"y": y2},"y2.pth")


loaded_y1 = torch.load("y1.pth")["y"]
loaded_y2 = torch.load("y2.pth")["y"]

if torch.equal(loaded_y1, loaded_y2):
    print("loaded_y1==loaded_y2")

if torch.equal(loaded_y1, y1):
    print("loaded_y1==y1")

print("size y1.pth (in Bytes)=", os.stat("y1.pth").st_size)
print("size y2.pth (in Bytes)=", os.stat("y2.pth").st_size)
</code></pre>
<p>This is the output of the above script:</p>
<blockquote>
<p>loaded_y1==loaded_y2<br>
loaded_y1==y1<br>
size y1.pth (in Bytes)= 640754<br>
size y2.pth (in Bytes)= 1394</p>
</blockquote>
<p>As you can see, both “y1.pth” and “y2.pth” are equal, but “y1.pth” is much larger because it saves the mother tensor, while we are not able to recover it.</p>

ptrblck: <p>That seems indeed like a bug. Could you create an issue on GitHub so that we could track and fix it, please?</p>

navid_mahmoudian: <p>Yes sure. Thank you very much for your support.</p>
Can I invert a sequence of Conv1d layers?

shamoons: <p>I have a pretrained model:</p>
<pre><code class="lang-auto">ModuleList(
      (0): Sequential(
        (0): Conv1d(1, 512, kernel_size=(10,), stride=(5,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
      (1): Sequential(
        (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
      (2): Sequential(
        (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
      (3): Sequential(
        (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
      (4): Sequential(
        (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
      (5): Sequential(
        (0): Conv1d(512, 512, kernel_size=(2,), stride=(2,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
      (6): Sequential(
        (0): Conv1d(512, 512, kernel_size=(2,), stride=(2,), bias=False)
        (1): Dropout(p=0.0, inplace=False)
        (2): Fp32GroupNorm(1, 512, eps=1e-05, affine=True)
        (3): ReLU()
      )
    )
</code></pre>
<p>Can I use the same weights for inverting with <code>ConvTranspose1d</code>? Or do I need to create an inverse network and retrain it?</p>

ptrblck: <p>If the input and output channels of the <code>nn.Conv1d</code> layers are reversed to the input and output channels of the transposed equivalent, you could try to reuse the weights.<br>
Usually you would create new layers and I don’t know how well this weight sharing approach would work, so let us know what you find out. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Then to use .clone() in optimizer computations?

hadaev8: <p>Should anyone explain why here clone() used?<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/HuangxingLin123/Learning-Rate-Dropout/blob/master/cifar10/adam_lrd.py#L79" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/HuangxingLin123/Learning-Rate-Dropout/blob/master/cifar10/adam_lrd.py#L79" target="_blank" rel="nofollow noopener">HuangxingLin123/Learning-Rate-Dropout/blob/master/cifar10/adam_lrd.py#L79</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="69" style="counter-reset: li-counter 68 ;">
<li>        #     # Use the max. for normalizing running avg. of gradient</li>
<li>        #     denom = max_exp_avg_sq.sqrt().add_(group['eps'])</li>
<li>        # else:</li>
<li>        denom = exp_avg_sq.sqrt().add_(group['eps'])</li>
<li>
</li><li>        bias_correction1 = 1 - beta1 ** state['step']</li>
<li>        bias_correction2 = 1 - beta2 ** state['step']</li>
<li>        step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1</li>
<li>
</li><li>        ##dropout learning rate</li>
<li class="selected">        lr_dropout = step_size / denom.clone()</li>
<li>        lr_dropout=lr_dropout * mask</li>
<li>
</li><li>        p.data.addcmul_(-1, exp_avg, lr_dropout)</li>
<li>
</li><li>        # p.data.addcdiv_(-step_size, exp_avg2, denom)</li>
<li>
</li><li>return loss</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Whats general rule about it?
<p>As far as I understand, this command copies the variable, in case I later modify it.<br>
But in this case, this variable is not used further.<br>
Is this an error in the code?</p>

ptrblck: <p>The <code>clone()</code> might not be necessary.<br>
Did you try to remove it and compare the output/training to the original implementation?</p>
Transfer Learning with Resnet 152

Luke_Kollmorgen: <p>I am following the tutorial here: <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>What I’m wondering is if I want to output the predictions a model is generating for the validation set to a dataframe how would I go about doing that?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#visualizing-the-model-predictions">Visualizing the model prediction</a> gives you an example how to get predictions during the validation or test phase.<br>
Could you describe where you’re currently stuck?</p>
Pytorch Lightning Profiler

aldeka12: <p>Hello,<br>
In a Pytorch Lightning Profiler there is a action called model_forward, can we use the duration for this action as an inference time? Of course, there is more to this action, than just an inference, but for comparison inference times of different models, would it be accurate to use the duration for this action?<br>
Thanks in advance!</p>

aldeka12: <p>Here is the example:</p>
<pre><code class="lang-auto">Action                  |  Mean duration (s)    |  Total time (s)
-----------------------------------------------------------------
on_epoch_start          |  5.993e-06            |  5.993e-06
get_train_batch         |  0.0087412            |  16.398
on_batch_start          |  5.0865e-06           |  0.0095372
model_forward           |  0.0017818            |  3.3408
model_backward          |  0.0018283            |  3.4282
on_after_backward       |  4.2862e-06           |  0.0080366
optimizer_step          |  0.0011072            |  2.0759
on_batch_end            |  4.5202e-06           |  0.0084753
on_epoch_end            |  3.919e-06            |  3.919e-06
on_train_end            |  5.449e-06            |  5.449e-06
</code></pre>

ptrblck: <p>CC <a class="mention" href="/u/williamfalcon">@williamFalcon</a> for Lightning questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Can this code be nicer?

pytorch_lzwhard: <pre><code class="lang-auto">if module.bias is not None:
    if len(out.shape) == 4:
        out = out + module.bias.view(1, -1, 1, 1)
    if len(out.shape) == 3:
        out = out + module.bias.view(1, -1, 1)
    if len(out.shape) == 2:
        out = out + module.bias.view(1, -1)
</code></pre>

ptrblck: <p>I don’t know what the original shape of <code>model.bias</code> is, but if you need to unsqueeze the dimensions, this code should work:</p>
<pre><code class="lang-python">bias = torch.randn(3)

out = torch.randn(2, 3)
view_shape = [1, -1] + [1] * (out.dim() - 2)
print(bias.view(view_shape).shape)
&gt; torch.Size([1, 3])

out = torch.randn(2, 3, 4)
view_shape = [1, -1] + [1] * (out.dim() - 2)
print(bias.view(view_shape).shape)
&gt; torch.Size([1, 3, 1])

out = torch.randn(2, 3, 4, 5)
view_shape = [1, -1] + [1] * (out.dim() - 2)
print(bias.view(view_shape).shape)
&gt; torch.Size([1, 3, 1, 1])
</code></pre>

pytorch_lzwhard: <p>Cool! it is nicer, tkx.</p>
Network inference on GPU under Android

Chame_call: <p>Does PyTorch support such type of inference?</p>

supriyar: <p>This is currently not supported, but we are actively looking into supporting inference on GPU on mobile devices.<br>
Stay tuned for future releases!</p>

Samuel_Menaker: <p>Any updates on this?</p>

seungjun: <p>I recently read about ONNX runtime and it might be an alternative.</p>
<p>PyTorch could export models to ONNX format and run the model with ONNX runtime.<br>
<a href="https://pytorch.org/docs/stable/onnx.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/onnx.html</a></p>
<p>According to the <a href="https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/NNAPI-ExecutionProvider.md" rel="nofollow noopener">onnxruntime</a>, NNAPI is supported including CPU and GPU inference.</p>
<p>I didn’t try it but looks like it is the only Android gpu-enabled accelerator for now.</p>
Implement about code

Walter_sh: <p>Hello</p>
<p>I’m a student who studies deep learning with Pytorch.</p>
<p>I have a question, but I want to learn only a specific character dataset (for example, alphabet V) when learning EMNIST dataset with Pytorch, but I want to know how to implement this with code.</p>
<p>Thanks!</p>

ptrblck: <p>You could manipulate the targets such that your target class uses the class index <code>1</code> while all other classes are assigned the class index <code>0</code>.<br>
Afterwards you can treat it as a binary classification and could use e.g. <code>nn.BCEWithLogitsLoss</code> as your criterion.</p>
Per Image Normalization

Sam_Xu: <p>How to perform similar normalization as tf.image.per_image_standardization in pytorch?</p>

ayushm-agrawal: <p>Any way to achieve this?</p>

ptrblck: <p>Could you explain, what this method is doing, as the <a href="https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization">docs</a> don’t seem to give much information:</p>
<pre><code class="lang-auto">Linearly scales each image in  `image`  to have mean 0 and variance 1.

For each 3-D image  `x`  in  `image` , computes  `(x - mean) / adjusted_stddev` , where
* `mean`  is the average of all values in  `x`
* `adjusted_stddev = max(stddev, 1.0/sqrt(N))`  is capped away from 0 to protect against division by 0 when handling uniform images
  * `N`  is the number of elements in  `x`
  * `stddev`  is the standard deviation of all values in  `x`
</code></pre>
<p>I don’t understand what the “per image” part of this normalization is, if the mean is the "average of all values in <code>x</code> and the <code>stddev</code> also seems to use all elements.</p>

ayushm-agrawal: <p>This is a replica that we created in PyTorch to use as a lambda function in our transforms. We compared these results with the tensorflow implementation and it seems to work the same.</p>
<p>I believe that this calculates the mean of the pixels in a single image and ‘x’ here refers to each pixel of the same image.</p>
<pre><code class="lang-auto">def per_image_standardization(image):
    """
    This function creates a custom per image standardization
    transform which is used for data augmentation.
    params:
        - image (torch Tensor): Image Tensor that needs to be standardized.
    
    returns:
        - image (torch Tensor): Image Tensor post standardization.
    """
    # get original data type
    orig_dtype = image.dtype

    # compute image mean
    image_mean = torch.mean(image, dim=(-1, -2, -3))

    # compute image standard deviation
    stddev = torch.std(image, axis=(-1, -2, -3))

    # compute number of pixels
    num_pixels = torch.tensor(torch.numel(image), dtype=torch.float32)

    # compute minimum standard deviation
    min_stddev = torch.rsqrt(num_pixels)

    # compute adjusted standard deviation
    adjusted_stddev = torch.max(stddev, min_stddev)

    # normalize image
    image -= image_mean
    image = torch.div(image, adjusted_stddev)

    # make sure that image output dtype  == input dtype
    assert image.dtype == orig_dtype

    return image
</code></pre>