akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does nn.Flatten not exist?

vainaijr: <p>which would do something like</p>
<pre><code class="lang-auto">out = inp.view(inp.size(0), -1)
return out
</code></pre>

ptrblck: <p>This module was recently <a href="https://github.com/pytorch/pytorch/pull/22245" rel="nofollow noopener">merged</a>.<br>
However, it looks like the <a href="https://pytorch.org/docs/stable/nn.html#flatten" rel="nofollow noopener">docs</a> are missing.</p>

vainaijr: <p>docs show in colab<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/db198d8bc0d0170d80d6e8621f1d95bde1e34efd.png" data-download-href="https://discuss.pytorch.org/uploads/default/db198d8bc0d0170d80d6e8621f1d95bde1e34efd" title="Screenshot%20(418)"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/db198d8bc0d0170d80d6e8621f1d95bde1e34efd.png" alt="Screenshot%20(418)" data-base62-sha1="vgfjoLhgWZFzeM3iVF7cN5e2Y2x" width="690" height="397" data-dominant-color="303031"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(418)</span><span class="informations">769×443 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
one more thing, how to implement show_batch without using matplotlib?<br>
I raised pull request, but I used matplotlib, and torchvision does not use matplotlib.<br>
Is there a way to show a batch of dataset, so it could be used like this</p>
<pre><code class="lang-auto">for batch, labels in dl:
  show_batch(batch)
</code></pre>

ptrblck: <p>The docs also show up in IPython, so maybe there is some issue on the website.</p>
<p>If you don’t want to use <code>matplotlib</code>, you could use e.g. <code>PIL</code> instead.<br>
However, in your code you are free to use whatever works for you.</p>

albanD: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> the function is missing there because you are on the stable doc, not master’s doc!</p>

ptrblck: <p>I’m pretty sure I was looking at both and <a href="https://github.com/pytorch/pytorch/pull/22245#issuecomment-526914935" rel="nofollow noopener">reported it</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_raised_eyebrow.png?v=9" title=":face_with_raised_eyebrow:" class="emoji" alt=":face_with_raised_eyebrow:"></p>
<p>However, the master docs are now there.<br>
Since this module is available in <code>1.2.0</code>, I think we might need to fix the stable docs as well.</p>
CUDA asynchronous execution

SPark9625: <p>According to <a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" rel="nofollow noopener">this</a>, operations that involve GPU are processed asynchronously and in parallel.</p>
<p>Does this mean that if I have many modules that can <strong>independently</strong> work on the same data, they will be processed in parallel, even if I call them sequentially?</p>
<p>For example, let’s say I have some code that looks like this:</p>
<pre><code class="lang-python">x = torch.randn(1,3,32,32).cuda()
conv_list = []
for i in range(30):
    conv_list.append(torch.nn.Conv2D(3,3,3,1).cuda())

# Is this executed in parallel?
output = sum(conv(x) for conv in conv_list)
</code></pre>
<p>Above, I’ve created 30 different conv layers that can work independent of each other. I’m calling them in a generator expression, but my understanding is that they are actually executed in parallel under the hood. Is my understanding correct?</p>
<p>I have a model where many sub-networks can operate independent of each other and I’d like to make it as parallel as possible. I was thinking of creating multiple <code>torch.cuda.Stream()</code> objects and use it for each independent module. But if my example code above does run in parallel, using multiple <code>torch.cuda.Stream()</code> objects would be silly.</p>
Understanding how loss and backward correctly for multiple leafs

orena1: <p>Hi,<br>
I have a simple question that I fail to understand, how does the loss which is usually a single value, propagate correctly in a network with multiple output nodes.</p>
<p>I’ll give an example, assuming I have a network with X inputs and 2 outputs. The network get pictures of cats or dogs or something else. And should output high value in output node 1 for dog (low value if no dog), and high value for output node 2 for cats. (And low values for no cat).<br>
Assuming I use MSE for my loss function, or some other loss function (CrossEntropyLoss) which outputs a single value, how does the loss propagate correctly for each node?</p>
<p>For example, let say I gave the network a batch with 1 pictures of a cat, and I got a vector of 2 (2 output nodes) which was (1,1) - therefor for node-2 I do not need to propagate any error but for node 2 I do need to propagate error. But when I do MSE I get a single loss value, how does this single loss value propagate correctly to each node.<br>
My intuition would be that I’ll have a loss which is a vector a loss for node-1 and a loss for node-2 and that each loss will propagate to his node. But as I said the MSE gives back a single loss value.</p>
<p>Thanks,</p>

spanev: <p>Hi <a class="mention" href="/u/orena1">@orena1</a>,</p>
<p>First, in autograd terms, leaf nodes are the input of the forward (the last  on which backpropagation will be applied) and the root the output of the forward network.</p>
<p>Remember that when you use <code>MSELoss</code>, the last operation to be applied is a reduction (by default <code>Mean</code> but you can also choose <code>sum</code> with the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MSELoss" rel="nofollow noopener">reduction</a> arg ).</p>
<p>So the first backward operation to be computed will the mean, as you can see here:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss(y, t)
tensor(0.1328, grad_fn=&lt;MeanBackward0&gt;) # &lt;==
</code></pre>
<p>an operation which will propagate the gradient to every output node contributing to this mean.</p>
<p>I hope everything is clear. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
C++ API with MEX compiler - Interfacing PyTorch and Matlab

Haydnspass: <p>Hey,</p>
<p>for some scientific application I need to interface PyTorch and Matlab (only for inference).<br>
I thought the best way would be not to mix the two interpreted languages (python and matlab) but rather load the trained models via the C++ API and interface this with the MATLAB C++ interface.<br>
Does someone have experience with this already?</p>
<p>Since MATLAB uses MEX “compiler” which is some kind of wrapper around the OS default compiler (clang++ in my case) it’s kind of hard to use cmake for this and I get tons of linking errors and I wondered if someone could help or has already tried something like this.</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

liangbright: <p>The best way is to call Python within Matlab, and use data files for communication.</p>

Pradeep_Rajendran: <p>I have managed to interface MEX and PyTorch using CMake. But, I am facing problems when I try to run module.forward(…) function. Matlab basically hangs when I run “forward” function. I may be able to help with CMake if you are still interested in that.</p>

Haydnspass: <p>Yeah would be interested! For the moment I</p>
<ul>
<li>Compile a shared library with my pytorch stuff (nicely in CMAKE)</li>
<li>have a quick and dirty solution for compiling the mex files via direct compiler command but that’s super ugly.</li>
</ul>
<p>Do you have an example?<br>
Forwarding works for me. if I can help, let me know.</p>

Haydnspass: <p>Any updates? Are you still active?</p>
Maximise (Shannon) Entropy of Network Output

Haydnspass: <p>Hey,</p>
<p>for some reason, I know that the output (i.e. the pixel values) of my network should follow a random uniform distribution. So I would like to aid the convergence by enforcing the output not only to have the correct values but also to be uniformly distributed. My idea was to have the loss as a combination of let’s say L2 between output and target as well as a regulariser / prior so as to enforce the output to be uniform.<br>
I am wondering whether this is possible, really simple or hard since the prior does not care about the actual values but only about its distribution. Any ideas or further reading?</p>
<p>Cheers<br>
Lucas</p>
Multiplying vector of embedding to image with shape (h * w * 1)

nobodykid: <p>Hello, I’m trying to migrate my GAN model code from Keras (backend TF) into Pytorch.</p>
<p>Currently I’m stuck at this operation, where I combined result from an embedding layer and image input thru element-wise multiplication.</p>
<p>In Keras, the code line would be such like this</p>
<pre><code class="lang-auto">label_embedding = Flatten()(Embedding(n_class, embed_dim)(label))
model_input = Multiply()([input_img, label_embedding])
</code></pre>
<p>Where <code>label_embedding</code> give a vector of size embed_dim and <code>input_img</code> is in shape of (h, w, 1), resulting in <code>model_input</code> in shape of (h, w, embed_dim)</p>
<p>In Pytorch, however, I can’t simply multiply the embedding result because of difference in dimension</p>
<pre><code class="lang-auto">self.embedding = nn.Embedding(n_class, embed_dim)

model_input = torch.mul(input_img, self.embedding(y))
</code></pre>
<p>How do I reshape the embedding vector to match with image input and able to be operated using <code>torch.mul</code>?</p>
How to put 2 images into my neural network

savior: <p>My neural network has two inputs , and I don’t know if I use the api rightly to put 2 images as inputs.This is the line of my code:</p>
<p>torch::jit::script::Module module = torch::jit::load("/media/usr515/psmnet_256.pt" , torch::kCUDA);<br>
torch::Tensor result = module.forward({tensor_image , tensor_image_1}).toTensor();</p>
<p>tensor_image and tensor_image_1 are the images having converted to tensors.</p>
Normalization of input data to Qnetwork

qLience: <p>Hello there,</p>
<p>I am well known with that a “normal” neural network should use normalized input data so one variable does not have a bigger influence on the weights in the NN than others.</p>
<p>But what if you have a Qnetwork where your training data and test data can differ a lot and can change over time in a continous problem?</p>
<p>My idea was to just run a normal run without normalization of input data and then see the variance and mean from the input datas of the run and then use the variance and mean to  normalize my input data of my next run.<br>
But what is the standard to do in this case?</p>
<p>Best regards Søren Koch</p>

alexis-jacq: <p>There is no standard as far as I know. What I usualy do is this :<br>
( this is from <a href="https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm">https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm</a> )</p>
<pre><code class="lang-python">class Normalizer():
    def __init__(self, num_inputs):
        self.n = torch.zeros(num_inputs)
        self.mean = torch.zeros(num_inputs)
        self.mean_diff = torch.zeros(num_inputs)
        self.var = torch.zeros(num_inputs)

    def observe(self, x):
        self.n += 1.
        last_mean = self.mean.clone()
        self.mean += (x-self.mean)/self.n
        self.mean_diff += (x-last_mean)*(x-self.mean)
        self.var = torch.clamp(self.mean_diff/self.n, min=1e-2)

    def normalize(self, inputs):
        obs_std = torch.sqrt(self.var)
        return (inputs - self.mean)/obs_std
</code></pre>
<p>Then each time I get a new state, I just do:</p>
<pre><code class="lang-python">normalizer.observe(new_state)
new_state = normalizer.normalize(new_state)
''' 
new_state must be a simple tensor, 
if it's a variable, use new_state.data
'''
</code></pre>

qLience: <p>thanks! btw. what is you data obs data type ? it is just because i get a error due to i am using a list</p>

alexis-jacq: <p>Ah I see, here there is a weird thing since the input of <code>observe</code> is a variable while the input of <code>normalize</code> is a tensor. Let me correct it so everything must be simple tensor (input and output).</p>

SunHaozhe: <p>Do we agree that this kind of “online” normalization is not injective ? In the sense that two distinct inputs that are observed and normalized at different time may be mapped to the same output value.</p>
<p>Furthermore, this mapping / filtering / normalization is not guaranteed to be monotonic (especially in the beginning when very few data have been observed).</p>

NotNANtoN: <p>A bit of a late reply, but:<br>
this kind of normalization made me struggle for quite some time. Using it, my DQN applied on cartpole could not learn properly. So you definitely need to be very careful.<br>
I will from now on either not normalize at all or freeze the update of the normalization parameters after some initial steps.</p>
How can i find the c++ implementation of pytorch function

saisai96: <p>I know C++ implementation code is in the folder ATen. But how can i find C++ implementation of function such as torch.tensor.half().</p>

tom: <p>In the end, you use <code>.to</code>, just like you would in the “modern” PyTorch idiom.</p>
<p>The variable function not founds in ATen can nonetheless be found using “rgrep” or somesuch on the source, in this case it is found in <a href="https://github.com/pytorch/pytorch/blob/ac7996ccd3a13528519beb398e418d7131a5ed52/tools/autograd/templates/python_variable_methods.cpp#L427" rel="nofollow noopener">tools/autograd/templates/python_variable_methods.cpp</a> or also, after you have built PyTorch in <code>torch/csrc/autograd/generated/python_variable_methods.cpp</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

saisai96: <p>Thank you very very much. Follow you suggest, I found what I need. Maybe you can give me some advice about how did you find where the torch.tensor.half() it is. How can I learn more about pytorch file structure</p>

tom: <p>I die use <code>rgrep</code>. In addition to the directory structure in <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md" rel="nofollow noopener">CONTRIBUTING.md</a>, you might look at <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="nofollow noopener">an old blog-post of mine</a> or <a class="mention" href="/u/ezyang">@ezyang</a>’s  <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="nofollow noopener">awesome talk</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Libtorch C++ model predict/forward propagation crashed on windows7 cuda9.0 GTX1050 4g

HZY302: <p>I used a model of  inceptionresnetv2 which trained in pytorch==1.0 and  scripting it in pytorch == 1.0. My model can load in libtorch But it can’t do forward in libtorch. Does anyone know why this happens?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/caab94d78ada06a57f9c8281858d63cf02e72499.png" data-download-href="https://discuss.pytorch.org/uploads/default/caab94d78ada06a57f9c8281858d63cf02e72499" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/caab94d78ada06a57f9c8281858d63cf02e72499.png" alt="image" data-base62-sha1="sUU3KCkjPSZ6Q9Be7azkPO0N7Hr" width="690" height="378" data-dominant-color="EBEDEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1053×578 38.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to control what goes in single batch

mayur_newase: <p>Hi,I wanted to control each batch data according to distribution of labels in it and current epoch.<br>
More specifically I wanted first few batches to have more number of images which are less in training set,and then gradually decrease their number in later batches.<br>
I have tried custom dataset and custom sampler,and what I understand is Dataset just provide a way to read individual item,and Sampler control indices for whole epoch.<br>
So to control each batch do i need custom DataLoader?<br>
If yes could you provide toy example for it,I could not find in documentation.</p>

alex.veuthey: <p>You’re close to the solution! Since the sampler gives indices for the whole epoch, you can disable shuffling in the DataLoader and decide on the indices you want in advance. By knowing the batch size, you can then know every index in every batch.</p>
shared_ptr<torch::jit::script::Module> module = torch::jit::load

savior: <p>When using pytorch1.2 ,one line of my code reports an error in make process which was not reported when using pytorch1.1.Is there anyone knowing how to solve it?</p>
<p>This is the error code:<br>
“shared_ptr<a>torch::jit::script::Module</a> module = torch::jit::load(”/media/usr515/ed65d8df-d015-4269-90bc-ac3cccb109a8/dsq/zz/stereo/psmnet_2.pt");"</p>
<p>This is the error report:<br>
error: conversion from ‘torch::jit::script::Module’ to non-scalar type ‘std::shared_ptr<a>torch::jit::script::Module</a>’ requested<br>
shared_ptr<a>torch::jit::script::Module</a> module = torch::jit::load("/media/us"</p>

dambo: <p>See this:</p><aside class="quote" data-post="3" data-topic="53462">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yf225/48/3418_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/broken-api-in-libtorch-1-2/53462/3">Broken API in libtorch 1.2</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    Yes the PyTorch C++ API is still considered “beta” stability. Specifically, we changed torch::jit::load(s_model_name) to return torch::jit::script::Module instead of std::shared_ptr&lt;torch::jit::script::Module&gt; because torch::jit::script::Module is now a reference type, and this change is documented in the release note ("[jit] script::Module is now a reference type" in <a href="https://github.com/pytorch/pytorch/releases/tag/v1.2.0" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/releases/tag/v1.2.0</a>).
  </blockquote>
</aside>


savior: <p>Thank you. It really helps.</p>
I want to serve my chatbot

zicxor: <p>Hi, I want to serve my chatbot. How can I do it? I have React-Node app, I think I will make an API (with Flask) and then connect with my app. Which ways should I use to build the Flask API? I trained my model on Colab. <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="nofollow noopener">PyTorch Chatbot Tutorial</a></p>

JamesTrick: <p>It’s good to see you’re wanting to productionize a model!</p>
<p>There’s now a <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html" rel="nofollow noopener">tutorial about writing an API around your model using Flask</a>. Then your react-node app would simply call it. Bearing in mind, that there’s probably a bit more you want to do before deploying to production, eg serve the Flask app with gunicorn or something similar.</p>
<p>Another approach would probably be to look into ONNX - I don’t know too much about that. I typically deploy models using Django or Flask.</p>
ViewBackward recursion

whoab: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://i.snipboard.io/H5ZJwN.jpg" title="H5ZJwN.jpg" rel="nofollow noopener"><img src="https://i.snipboard.io/H5ZJwN.jpg" alt width="372" height="500"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">H5ZJwN.jpg</span><span class="informations">372×986</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
<p>What could cause this? It grows every backprop step and is severely slowing down my code</p>

whoab: <p>I found out the reason. In my model, I have this tensor:<br>
<code>self.log_alpha = torch.zeros(1, requires_grad=True)</code></p>
<p>Using it in my loss is fine:</p>
<pre><code class="lang-auto">            alpha_loss = -(self.log_alpha * (log_pi + self.target_entropy).detach()).mean()
            self.alpha_optimizer.zero_grad()
            alpha_loss.backward()
</code></pre>
<p>But calling a view operation in my training loop, even with detach(), causes the ViewBackward recursion. Why?</p>
<pre><code class="lang-auto">    def _do_training(self):
        # self.log_alpha = self.log_alpha.view(1).detach()
</code></pre>

albanD: <p>Hi,</p>
<p>The graph above seems to be referencing the <code>layer_norms</code> layer, the first module in it, and a parameter called <code>bias</code> in that module. Are you sure it is this <code>log_alpha</code>?</p>

whoab: <p>The recursion was occurring in multiple scenarios. In the layer_norms example, I was passing the final probability distribution in an attention module outside of the training loop. But the log_alpha was another concrete example of how to cause the behavior.</p>
Data Dimension Processing (LSTM Time Series Prediction)

zyc0868: <p>When I use lstm, there is always a problem about data dimension(input_size,out_size). Now train_x(batch_zize,time_step,input_szie),  train_y(batch_size,time_step,input_size).  Lstm forward(batch_first is True) code:<br>
def forward(self,x):<br>
x, (h_n, h_c) = self.rnn(x, None)<br>
x = self.reg(x)<br>
return x<br>
Loss calculate code:<br>
var_x = Variable(train_x)<br>
var_y = Variable(train_y)<br>
out = net1(var_x[start:end]).view(-1)<br>
var_y=var_y[start:end].view(-1)<br>
loss = criterion(out ,var_y)<br>
Errors are about these two aspects.</p>
[CLOSED]DataLoader in parallel?

Petro_Key: <p>Hi, thank you always for your help.</p>
<p>I am writing a dataloader using torch.utils.data.DataLoader class.<br>
I want to load at once the same indexed files from different folders.</p>
<p>For example, provided I have folder 0, 1, …, 9, and each folder have the same indexed files from 00 to 99 like this:<br>
folder 0 — 00.txt, 01.txt, …, 99.txt<br>
folder 1 — 00.txt, 01.txt, …, 99.txt<br>
…<br>
folder 9 — 00.txt, 01.txt, …, 99.txt</p>
<p>I prepared a loading list file where the indices are shuffled, and then tried to load them as usual:<br>
for batch_index, inputs in enumerate(self.data_loader):<br>
process(inputs)<br>
What I expect is, if the loading list file indicates 07.txt, then the loader takes all the 07.txt files from the total 10 folders above. But it seems that all these files are shuffled. I guess this is due to the parallel computing.<br>
Is there any good way to load correctly?</p>
<p>Thank you in advance.</p>

ptrblck: <p>If I understand the use case correctly, you would like to return the samples from the .txt files having the same index.<br>
This would also mean that you would increase the batch size by 10 (number of folders).<br>
If that’s correct, you could use the <code>index</code> in your <code>__getitem__</code> and load the correcponding .txt file using this <code>index</code> for all 10 folders.<br>
Would that work or am I misunderstanding your use case?</p>

Petro_Key: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you always for your help!<br>
Yes, that is exactly what I want to do, and I see your suggestion about the batch size now, which I have not understood.<br>
I confirmed your idea in my code and obtained probably the correct items.<br>
Thank you very much!</p>
What does @ do?

isalirezag: <p>Have a silly question:</p>
<pre><code class="lang-auto">import torch
a = torch.rand(2,2)
b= torch.rand(2,2)
</code></pre>
<p>what <code>a @ b</code> do?</p>

ptrblck: <p>It’s a matrix multiplication. If I’m not mistaken it was introduced in Python3.</p>
Lack of gradient flow with torch.ge()

Abhishaike_Mahajan: <p>I may have found a bug in torch.ge(), but I just wanted to re-post it here in case it’s actually not a bug, but an issue with how I’m defining my model: <a href="https://github.com/pytorch/pytorch/issues/7322" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/7322</a>.</p>
<p>There’s a runnable and short code-sample in the link, along with my version numbers of relevant libraries.</p>

SimonW: <p>“greater than or equal” has zero gradient almost everywhere, and nondifferentiable at other points. It’s not a bug.</p>

Abhishaike_Mahajan: <p>Thank you for your answer! So is it not possible to have stable training with .ge when used as an intermediate layer?</p>

SimonW: <p>If there are no other path from input to output other than comparison ops, then there will be no gradients.</p>
<p>It’s not really an issue of stability because, if you think about the function, greater than or equal is just flat almost everywhere.</p>

Alex6D: <p>How do you solve this problem? I run into the same needs. Thanks</p>

Abhishaike_Mahajan: <p>Like SimonW said, this isn’t really a bug, just how the gradient surface of a piecewise operation is. I got around it by remaking my function into something that isn’t piecewise. I explain it in a bit more detail here: <a href="https://scientificattempts.wordpress.com/2018/06/03/conditionally-thresholded-cnns-for-weakly-supervised-image-segmentation/" rel="nofollow noopener">https://scientificattempts.wordpress.com/2018/06/03/conditionally-thresholded-cnns-for-weakly-supervised-image-segmentation/</a></p>
Pretrained model keypoint detection is outputting too much tensors

Hassan_Saeed: <p>I am trying to use the pretrained model keypoint rcnn model to detect poses, and make use of the poses results, but the keypoints output a tensor of (2,17,3), the 17 is number of joints, 3 is the coordinates and visibility, but what is that 2 for? I supposed it is the number of persons in the image but my image has only 1 person, so why is that?? I want to utilize the poses data but confused which one is the correct</p>

radeeplearning: <p>For the input to the network in target[“labels”] what have you passed as the number of class labels?</p>
DistributedDataparallel(DDP) questions

bobchennan: <p>In DDP broadcast_buffers is set to true as default. I am thinking if this is necessary? I used SyncBatchNorm and according to implementation I think during backward every process will get gradients from all nodes so statistics should always be consistent.</p>
<p>So far my validation accuracy is not normal similar to <a href="https://github.com/facebookresearch/maskrcnn-benchmark/issues/267" rel="nofollow noopener">https://github.com/facebookresearch/maskrcnn-benchmark/issues/267</a> . I am thinking to replace SyncBatchNorm with BatchNorm. So if broadcast_buffers is set to True I guess all nodes will use statistics from first node right?</p>
My policy gradient

slavavs: <p>A simple example will show how my NN works. Usually NN takes a step, then we give a label and do backprob. In my case, NN takes 10 steps (for example), after that I count how many points NN earned. If NN earned minus 5 points, then I need to reduce the gradient at all 10 steps, and if I gained NN plus 5 points, then I need to increase the gradient. How do I remember gradients and do backprob correctly?</p>

IAmIronMan: <p>Correct me, if I am wrong. Are you referring to batch size. Like to calculate loss for 10 inputs and then do backprop?<br>
If it is, then you can use batch size. It calculate loss for batch size (like 10) and calculate gradient and then backprop.</p>

slavavs: <p>You correctly understood the size of the party, this is the number of steps. I have 3 neurons in the NN output. I used to use CrossEntropyLoss. In this situation, NN scored minus 5 points, then for those neurons that showed the highest weights, I need to reduce the weight, and for the other two, increase. For example, for one neuron I decrease by 1, and for two I increase by 0.5. I can’t make such an implementation, I don’t understand how to adjust the weights manually.</p>

slavavs: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8ab149d403897f9e4b7f353762d803dcd6b24307.png" alt="pg" data-base62-sha1="jMVJenR0zvaHy5v4MQrPBMCGPCT" width="330" height="76"></p>
<p>I need to put such a formula after softmax. How to do it?</p>
_th_log_out not supported on CUDAType

_Ashraf: <p>Hi everyone,</p>
<p>I am trying to use KLDivLoss loss function with but I get the following Error:</p>
<blockquote>
<p>_th_log_out not supported on CUDAType for Long</p>
</blockquote>
<p>I don’t know if there is some bug with KLDivLoss, but as I see there is a problem when using it with GPU.</p>
<p>any hint or solution will be really useful</p>
Access 3d tensor by 2d index

Nick_8229: <p>I am trying to access a 3d tensor matrix by 2d matrix. The return should be an 2d matrix too. And the following is what i try to achieve.</p>
<pre><code class="lang-auto">
dim1 = 2
dim2 = 3
dim3 = 4
source = torch.FloatTensor(dim1, dim2, dim3)
source.normal_()

check = zn &gt; 0.0
index = torch.argmax(check, dim=0)

for i in range(dim2):
    for j in range(dim3):
        ret[i, j] = source[index[i, j], i, j]
</code></pre>

LeviViana: <p>I think you want to use <code>torch.gather</code>:</p>
<pre><code class="lang-python">import torch
dim1 = 2
dim2 = 3
dim3 = 4
source = torch.FloatTensor(dim1, dim2, dim3)
index = source.argmax(dim=0).unsqueeze(0)
source.gather(0, index).squeeze(0)
</code></pre>

Nick_8229: <p>It is working, thank you so much!</p>
Clueless halt while GPU is still running! 

youansheng: <p>While I use the distributed training, the training process encounters clueless halt. The GPU-Memory and GPU-Util (90%-100%) of all GPUs are normal with no pid being killed!!!</p>

jessemelpolio: <p>In my case, it shows like a 100% GPU utility occupation there on the GPU. And some cores of CPU are taken as well but it’s weird that actually nothing is ongoing and the training program seems just like “dead”. We use the same repo here: <a href="https://github.com/donnyyou/torchcv" rel="nofollow noopener">TorchCV</a>.</p>
<p>I tried to stop the program with Ctrl+C or simply kill the program. I was expecting some error info but nothing was printed.</p>

jessemelpolio: <p>Does anyone encounter this kind of problem before? It would be also great if anybody has some techniques to localize the problem or at least print something.</p>
Loss function problem

dreamer: <p>Hi everyone, i have a problem about loss function. I  have 2 tensors and x=[23177,1,3] and y=[23177,1]. They are my special datasets. So when I turn them into a dataloader:</p>
<pre><code class="lang-auto">train_dataset = TensorDataset(train_x, train_y)
train_generator = DataLoader(train_dataset, batch_size=100,shuffle=False)
</code></pre>
<p>and my criterion and optimizer:</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
</code></pre>
<p>and my nn class is:</p>
<pre><code class="lang-auto">class Modele(nn.Module):
        def __init__(self, D_in, H1, H2, D_out):
                super().__init__()
                self.linear1 = nn.Linear(D_in, H1)
                self.linear2 = nn.Linear(H1, H2)
                self.linear3 = nn.Linear(H2, D_out)
        def forward(self, x):
                x = F.relu(self.linear1(x))  
                x = F.relu(self.linear2(x))
                x = self.linear3(x)
                return x

</code></pre>
<p>and my computation loop:</p>
<p>epochs=15<br>
for e in range(epochs):</p>
<pre><code>    running_loss = 0.0
    running_corrects = 0.0
    val_running_loss = 0.0
    val_running_corrects = 0.0

    for inputs,out in train_generator:
            

          
            
            loss = criterion(model(inputs), out)
            #_,preds=torch.max(outputs,1)
            #outputss.append(preds.max().detach().numpy())
            losses.append(loss)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
                                                                    
            #outputss.append(outputs.detach().numpy())
            #print(loss.item())
</code></pre>
<p>And my loss function graph:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b4c97f0c4adb9d96682ba59f29ce91ef20d1bb07.png" alt="Figure_1" data-base62-sha1="pNjEcJamETWavx95yNs9GkstOsv" width="640" height="478"><br>
and I got this error</p>
<blockquote>
<p>C:\Users\gun\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\loss.py:431: UserWarning: Using a target size (torch.Size([100, 1])) that is different to the input size (torch.Size([100, 1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
C:\Users\gun\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\loss.py:431: UserWarning: Using a target size (torch.Size([12, 1])) that is different to the input size (torch.Size([12, 1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have<br>
the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
How can I solve this problem?</p>
</blockquote>

IAmIronMan: <p>Pad the target into 3d [ x, y , z ]. z-dim can be zero in this case. Loss can be computed only when they are of same size. But, check if loss is coming according to your desired expectations.</p>
How can i get the full layer name

arley: <p>I have followed this <a href="https://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks" rel="noopener nofollow ugc">forward-and-backward-function-hooks</a> to print the layer’s feature map’s size,just like:</p>
<pre><code class="lang-auto">Conv2d forward	 input: torch.Size([32, 3, 300, 300])	 output: torch.Size([32, 64, 300, 300])
Conv2d forward	 input: torch.Size([32, 64, 300, 300]) output: torch.Size([32, 64, 300, 300])
</code></pre>
<p>but this “Conv2d” name is unrecognized and I want to print the  name of the layer, how can i approach this?</p>
<p>for example:<br>
when i define a module, i use:</p>
<pre><code class="lang-auto">def conv_layers():
    conv1 = nn.Conv2d(1, 10, 5)
    pool1 = nn.MaxPool2d(2, 2)
    conv2 = nn.Conv2d(10, 20, 5)
    pool2 = nn.MaxPool2d(2, 2)
    layers = [conv1, pool1, conv2, pool2]
    return layers

class LeNet(nn.Module):
          def __init__(self,base):
                self.base= nn.Sequential(*base)
                self.fc1 = nn.Linear(320, 50)
                self.fc2 = nn.Linear(50, 10)
def forward():
        ……

net=LeNet(conv_layers)

def printnorm(self, input, output):
      print('{} forward\t input: {}\t output: {}\t'.format(
        self.__class__.__name__, input[0].size(), output.data.size()))

layers = []
for layer in net.children():
    layer.register_forward_hook(printnorm)

out = net(input)
</code></pre>
<p>I get something like this:</p>
<blockquote>
<p>Conv2d forward	 input: torch.Size([32, 3, 300, 300])	 output: torch.Size([32, 64, 300, 300])	<br>
ReLU forward	 input: torch.Size([32, 64, 300, 300]) output: torch.Size([32, 64, 300, 300])</p>
</blockquote>
<p>but how can i know the Conv2d’s name？<br>
because the layers in “self.base” are:<br>
base[0]-&gt;conv2d ,base[1]-&gt;maxpool2d, base[2]-&gt;conv2d ,base[3]-&gt;maxpool2d,<br>
I want get the full name in the hook function “printnorm” so i can find the specific Conv2d and do something else.</p>

JinyangGuo: <p>Hi,</p>
<p>I am also encounter this problem. Have you figure it out?</p>

Lizhen_Ji: <p>Hi, did you get the way to get the full name inside a module ? Thanks</p>
Tor.jit.trace to work on mask RCNN

TidelyPom: <p>Is there any chance to have the torch.jit.trace instrunction work on a maskrcnn_resnet50_fpn model since it doesn’t seem to work yet, see <a href="https://github.com/pytorch/vision/issues/1002" rel="nofollow noopener">https://github.com/pytorch/vision/issues/1002</a> ?</p>
<p>Thanks</p>
Buffering error

Jaideep_Valani: <p>hi all<br>
M unable to find out why i should buffer freed error with below code<br>
usig retain_Graph = true is expensive i cant use that option.<br>
Please suggest where to retain the previous value and use that in running calculation for next iterations.</p>
<pre><code class="lang-auto">for idx, (imgs, labels) in enumerate(tk0):
        imgs_train, labels_train = imgs.cuda(), labels.float().cuda()
        output_train = model(imgs_train)
        loss = criterion(output_train,labels_train)
        if idx&gt;=1:
            
            loss=loss*0.3+0.7*prev_loss
        with torch.no_grad():
            
            prev_loss=loss
        with amp.scale_loss(loss, optimizer) as scaled_loss:
            scaled_loss.backward( )
       
        optimizer.step() 
        
        optimizer.zero_grad()
</code></pre>

ptrblck: <p>Since you are setting <code>prev_loss=loss</code>, you will backpropagate through the computation graph multiple times, so you would need to set <code>retain_graph=True</code> in the <code>backward</code> call.<br>
Assigning <code>prev_loss = loss</code> in the <code>no_grad</code> block will not detach <code>prev_loss</code> from the computation graph, as just new operations won’t require gradients.</p>
Suddenly - RuntimeError: CUDA error: device-side assert triggered

lrpy_trainer: <p>I checked other suggestions, but other people used RNN Networks and data labels.<br>
In my case, everything was working fine yesterday, but suddenly my code is not working anymore.</p>
<p>I am trying to run this code on my local machine: <a href="https://colab.research.google.com/github/Curt-Park/rainbow-is-all-you-need/blob/master/08.rainbow.ipynb" rel="nofollow noopener">https://colab.research.google.com/github/Curt-Park/rainbow-is-all-you-need/blob/master/08.rainbow.ipynb</a></p>
<p>C:/w/1/s/tmp_conda_3.6_045031/conda/conda-bld/pytorch_1565412750030/work/aten/src/THC/THCTensorIndex.cu:189: block: [25,0,0], thread: [63,0,0] Assertion <code>dstIndex &lt; dstAddDimSize</code> failed.<br>
THCudaCheck FAIL file=C:/w/1/s/tmp_conda_3.6_045031/conda/conda-bld/pytorch_1565412750030/work/aten/src\THC/THCTensorMathCompareT.cuh line=69 error=59 : device-side assert triggered<br>
Traceback (most recent call last):<br>
File “rainbow.py”, line 763, in <br>
agent.train(epochs, horizon)<br>
File “rainbow.py”, line 629, in train<br>
loss = self.update_model()<br>
File “rainbow.py”, line 578, in update_model<br>
elementwise_loss_n_loss = self._compute_dqn_loss(samples, gamma)<br>
File “rainbow.py”, line 710, in _compute_dqn_loss<br>
dist = self.dqn.dist(state)<br>
File “rainbow.py”, line 386, in dist<br>
print(x)<br>
File “C:\Users\un_po\Anaconda3\envs\rainbowPy\lib\site-packages\torch\tensor.py”, line 82, in <strong>repr</strong><br>
return torch._tensor_str._str(self)<br>
File “C:\Users\un_po\Anaconda3\envs\rainbowPy\lib\site-packages\torch_tensor_str.py”, line 300, in _str<br>
tensor_str = _tensor_str(self, indent)<br>
File “C:\Users\un_po\Anaconda3\envs\rainbowPy\lib\site-packages\torch_tensor_str.py”, line 201, in _tensor_str<br>
formatter = _Formatter(get_summarized_data(self) if summarize else self)<br>
File “C:\Users\un_po\Anaconda3\envs\rainbowPy\lib\site-packages\torch_tensor_str.py”, line 87, in <strong>init</strong><br>
nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0))<br>
File “C:\Users\un_po\Anaconda3\envs\rainbowPy\lib\site-packages\torch\functional.py”, line 228, in isfinite<br>
return (tensor == tensor) &amp; (tensor.abs() != inf)<br>
RuntimeError: cuda runtime error (59) : device-side assert triggered at C:/w/1/s/tmp_conda_3.6_045031/conda/conda-bld/pytorch_1565412750030/work/aten/src\THC/THCTensorMathCompareT.cuh:69</p>
<p>Yesterday the same code just worked fine.</p>
<p>And the code still works on CPU.</p>

lrpy_trainer: <p>Downgrading pytorch and cuda helped. Strange</p>

ptrblck: <p>Did you change something else besides the PyTorch version?<br>
What CUDA, cudnn, and PyTorch versions are you using now and which one raised this error?<br>
Do you have a reproducible code snippet so that we could have a look?</p>
Confusion about torch.Tensor

chuang_ma: <p>First I define a torch Tensor, then I inspect it in Pycharm Debug Window, a odd problem occurs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/50699884/64088346-936f7880-cd73-11e9-89d5-ccea6710c09b.png" title="64088346-936f7880-cd73-11e9-89d5-ccea6710c09b.png" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/50699884/64088346-936f7880-cd73-11e9-89d5-ccea6710c09b.png" alt="image" width="689" height="480"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">64088346-936f7880-cd73-11e9-89d5-ccea6710c09b.png</span><span class="informations">1344×935</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div><br>
it seems that I can have endless t.T.T.T… , and they have the same value and structure, the same as t.data.data.data, could anyone give me an explansion.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/50699884/64088429-fcef8700-cd73-11e9-9f64-094d89e7079e.png" title="64088429-fcef8700-cd73-11e9-9f64-094d89e7079e.png" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/50699884/64088429-fcef8700-cd73-11e9-9f64-094d89e7079e.png" alt="image" width="690" height="429"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">64088429-fcef8700-cd73-11e9-9f64-094d89e7079e.png</span><span class="informations">1332×830</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
AlexNet vs ResNet efficiency as feature extractor

andrster: <p>Hello,<br>
I’ve been using a CNN-LSTM network for video description but the thing is that if I change the CNN architecture from AlexNet to any other better architecture, the results become worse! I did experiments were I kept a few CNN layers frozen, left it end-to-end trainable or froze the whole thing except the last CNN layer but still AlexNet yielded better results.<br>
Since I’m dealing with videos, I pass one video at a time but in order to go through the CNN I do the following:</p>
<pre><code class="lang-auto">bs, frames, channels, height, width = x.size()
x = x.view(bs*frames, channels, height, width)
</code></pre>
<p>Can the fault be that for each video the frame number isn’t the same and the BatchNorm layers that the other architectures have mess things us?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

LeviViana: <p>In the context of object tracking, until recently, the best feature extractor for Siamese Networks was AlexNet. <a href="https://arxiv.org/pdf/1812.11703.pdf" rel="nofollow noopener">This paper</a> sheds light on the reasons why standard ResNets worsen feature extractor performance, and they propose a solution to this problem.</p>
<p>I’m not sure if it will solve your case, but it is worth taking a closer look.</p>
Batch normalization and weight initialization in Seq2Seq

davidlee: <p>Hello PyTorch!<br>
I want to apply Batch Normalization and Weight initialization in Seq2Seq problem but,<br>
i don’t know how to apply it in my model.</p>
<p>Please help me guys!<br>
Those are my model below.</p>
<hr>
<p>class Encoder(nn.Module):</p>
<pre><code>def __init__(self,
             input_dim,
             hidden_dim,
             num_layers=1,
             dropout=0,
             bidirectional=False):
    super(Encoder, self).__init__()
    self.encoder = nn.LSTM(input_dim,
                           hidden_dim,
                           num_layers=num_layers,
                           dropout=dropout,
                           bidirectional=bidirectional)

def forward(self, x, hidden):
    encoder_output, encoder_state = self.encoder(x, hidden)
    return encoder_output, encoder_state
</code></pre>
<hr>
<p>class Decoder(nn.Module):</p>
<pre><code>def __init__(self, input_dim, hidden_dim, output_dim, num_layers=1, dropout=0, bidirectional=False):
    super(Decoder, self).__init__()
    self.decoder = nn.LSTM(input_dim, hidden_dim, num_layers=num_layers, dropout=dropout, bidirectional=bidirectional)
    self.linear = nn.Linear(hidden_dim, output_dim)

def forward(self, x, hidden):
    decoder_output, next_hidden = self.decoder(x, hidden)
    
    outputs = []
    for i in range(decoder_output.size()[1]):
        outputs += [self.linear(decoder_output[:, i, :])]
    return torch.stack(outputs, dim=1).squeeze(), decoder_output, next_hidden
</code></pre>
<hr>
<p>class Model(nn.Module):</p>
<pre><code>def __init__(self, input_dim, hidden_dim, output_dim, num_layers=1, output_length=3):
    super(Model, self).__init__()
    self.encoder = Encoder(input_dim, hidden_dim, num_layers=num_layers)
    self.decoder = Decoder(hidden_dim, hidden_dim, output_dim, num_layers=num_layers)
    self.output_length = output_length
    self.num_layers = num_layers
    self.hidden_dim = hidden_dim
    
def forward(self, x):
    encoder_output, encdoer_state = self.encoder(x, None) 
    decoder_input = torch.unsqueeze(encoder_output[-1], 0)
    
    seq = []
    next_hidden=None 
    next_input = decoder_input
    
    for _ in range(self.output_length):
        output, next_input, next_hidden = self.decoder(next_input, next_hidden)
        seq += [output]
    return torch.stack(seq, dim=0).squeeze(), torch.unsqueeze(encoder_output[-1],0)</code></pre>
Pytorch+CUDA Version

Suprem_Motu: <p>Does Pytorch work with CUDA 10.1? When I go to the site, it says get the version of pytorch with 10.0 and not 10.1.</p>

LeviViana: <p>I use it with CUDA 10.1 with no problems. In the worst case scenario, you can always build PyTorch from source.</p>
Lr Scheduler is not starting from the base_lr

IAmIronMan: <p>I used CLR scheduler and it is working weirdly. It is not starting from the base_lr.<br>
My base_lr = 1e-06 and max_lr =1. But it starts from 0.01. If I increase my step size it then, it starts from 0.001, which I think step size should not effect the start of lr scheduler.</p>
<p>I found a way around. For a range of 10^-6. I did 2 cycle with different base_lr and max_lr.<br>
1st cycle : base_lr = 1e-06 ; max_lr = 1e-.3<br>
2nd cycle : base_lr = 1e-03 ; max_lr = 1</p>
<p>It seems it is working when the range is less.</p>
<p>Did any one else found issue using CLR scheduler. ? Is it a bug or am I doing it incorrectly</p>