akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training Model_1 using output of Model_2 which is pretrained whether using model_2.eval() or not in train method

Nikronic: <p>Hi everyone,</p>
<p>I have two models, assuming <code>Model_1</code> and <code>Model_2</code>.<br>
<code>Model_2</code> is pretrained and I want to use its output to compute loss and train <code>Model_1</code>.<br>
Looks something like this:</p>
<pre><code class="lang-python">
def train(model1,model2, *args):
  Model_1.train()
  Model_2.eval()
  
  out1 = Model_1(input)
  out2 = Model_2(out1)
  loss = criterion(out2)
  ...

optimizer = torch.optim.SGD(Model_1.parameters(), *args)
</code></pre>
<p>Here is a <a href="https://discuss.pytorch.org/t/optimizing-based-on-another-models-output/6935/2">similar question</a> I’ve got some ideas.</p>
<p>My question is should I use <code>Model_2.eval()</code> even when I am passing only <code>Model_1</code> parameters to the optimizer?</p>
<p>Thanks for any advice</p>

MariosOreo: <p>Hi,</p>
<p>From albanD’s comment, you could pass only <code>Model_1</code> parameters to the optimizer (this is what you already did).</p>
<p>This thread is discuss if we should turn the pre-trained model to <code>eval</code> mode. In my shallow opinion, calling <code>Model_2.eval()</code> will have some difference as follows:</p>
<ul>
<li>the parameters of Model_2 will not update in optimizer.step(). This result is what your optimizer configuration already achieved.</li>
<li>change dropout layer and batchnorm layer into eval mode.</li>
</ul>
<p>So we can see that it will only have the second difference, but I have not met this problem, you could have a try on it whether turn off dropout and batchnorm layers in a fixed model will have influence on the post-networks.</p>
<p>If you get something different, please let me know, thanks!</p>

Nikronic: <p>Thank you,<br>
Basically, I want to my <code>Model_2</code> act exactly as a finalized model (I mean only testing and no train at all) so I just use its output for further usage.<br>
And based on your comment, I should use <code>model.eval()</code>.<br>
I will report differences when I finish the job.</p>
The same classifier has a different result in the same dataset

e8fa7736c3c1cf274dc1: <p>Hi,<br>
I test the resnet18 several times, but the accuracy and the loss value are different in every time I test.<br>
Does anyone can answer this situation, I am so confused…<br>
Thanks!</p>
<p>Here is my test function</p>
<pre><code class="lang-auto">def eval_model(device, model, loader):
    
    model.eval()
    
    avg_acc  = 0.0
    avg_loss = 0.0
    
    with torch.no_grad():
        for batch_idx, batch_data in enumerate(loader):
            images, labels = batch_data
            images = images.to(device)
            labels = labels.to(device)
            batch_size = images.shape[0]
            
            outputs = model(images)
            loss = F.cross_entropy(outputs, labels)

            # accuracy
            preds = torch.max(outputs, 1)[1]
            acc = (torch.eq(preds, labels).sum().item() / batch_size)
            
            avg_loss += loss.item()
            avg_acc += acc
        
    avg_loss /= len(loader)
    avg_acc  /= len(loader)
    print('Evaluation =&gt; Avg ACC: {:.4f}, Avg Loss: {:.4f}'.format(avg_acc, avg_loss))
</code></pre>
<p>Here is the result</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4b6c92ec0f909e84705fc24c23fcab8b0f9b03cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b6c92ec0f909e84705fc24c23fcab8b0f9b03cc" title="aaaaa"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4b6c92ec0f909e84705fc24c23fcab8b0f9b03cc.png" alt="aaaaa" data-base62-sha1="aLeqAorUXMsrOijEA9Zw92Mg40Y" width="690" height="264" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">aaaaa</span><span class="informations">920×353 17.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nutszebra: <p>The code looks okay.<br>
I’m not sure, but maybe your transform function in <code>loader</code> is stochastic.</p>

ptrblck: <p>Are you shuffling the <code>DataLoader</code>?<br>
If the length of your dataset is not divisible by the batch size without a remainder, you might see small differences in your validation accuracy. The last batch might be smaller, which creates a bias using your normalization (dividing by <code>len(loader)</code>).</p>
<p>Try this code instead:</p>
<pre><code class="lang-python">    acc = torch.eq(preds, labels).sum().item()
    avg_acc += acc
    avg_loss += loss.item() * images.size(0)

avg_acc /= len(loader.dataset)
avg_loss /= len(loader.dataset)
</code></pre>

e8fa7736c3c1cf274dc1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/nutszebra">@nutszebra</a><br>
Hi,<br>
Thank you very much, I solve the problem!!<br>
Yes, I’m shuffling the <code>DataLoader</code>, and my dataset is not divisible by the batch size.<br>
So, there are small differences in the validation accuracy.</p>
Conditional GAN with RNNs

Taxaria: <p>Hey people <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to build a GAN-model with a context vector as additional input, which should use RNN-layers for generating MNIST data.<br>
For the Generator I want to slice the noise vector into four pieces and it should generate MNIST data in the same way.<br>
For the Discriminator I want to do the same. The input should be sliced into four pieces.<br>
I have a conditional GAN model that works not that well, but it works <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> There is some work with the parameters to do.<br>
Do you have any ideas or example models for a conditional GAN with RNNs or for a GAN with RNNs?<br>
I drowned a lots of hours the last days to get by CGAN to become a CGAN with RNNs, but it’s not working. I’m missing some ideas, how I can realize the sliced input vector in addition to my context vector and how I can integrate the sliced input into the forward function.</p>
<p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Sort function behaviour for same values

mrmoody: <p>Hi everyone,</p>
<p>I have the following situation: I want to clamp a tensor with a given minimum value and then I want to sort the tensor, and for the positions of the same value, I want to get the locations sorted (first to last):</p>
<pre><code class="lang-auto">val is a tensor of dimension B x T x 1000

val = torch.clamp(val, min=0.01)
_, indices = val.sort(-1)
</code></pre>
<p>The indices for locations with the same values (e.g. 0.01) are NOT sorted (I think they are just random). Is there a way to enforce that I get the smallest indices first?</p>
<p>Thanks a lot!</p>
How to use my own datasets

Huhm302622377: <p>My previous work is to implement some neural network with PyTorch. The datasets for this network is mnist so it’s very convenient . But when I want to use my own datasets, I met some problems. The datasets is organized by folders. For instance, All the pictures in the folder named A1  belong to the same class called A and all the pictures in the folder named A2. How can I do it? Can anyone help me? Thanks a lot!<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cf6c72986eb9ed3d167911e02e4a9c3c005ca5ee.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cf6c72986eb9ed3d167911e02e4a9c3c005ca5ee" title="4"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cf6c72986eb9ed3d167911e02e4a9c3c005ca5ee_2_662x500.jpeg" alt="4" data-base62-sha1="tAXbqK8rhMRA3Ffr81RhLY4RFF4" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cf6c72986eb9ed3d167911e02e4a9c3c005ca5ee_2_662x500.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/c/cf6c72986eb9ed3d167911e02e4a9c3c005ca5ee.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/cf6c72986eb9ed3d167911e02e4a9c3c005ca5ee.jpeg 2x" data-dominant-color="E3DAD9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">737×556 52.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Nikronic: <p>Please read the documentation and look for “custom dataset pytorch” in forum. This question has a lot of answers already.</p>
<p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data Loading Tutorial</a></p>

Huhm302622377: <p>Thanks for your reply! I will have a look at the document:laughing:</p>

Nikronic: <p>If you still have any problem, feel free to reply and I’ll send you some examples which I have implemented myself.</p>
<p>Here is the <a href="https://discuss.pytorch.org/t/custom-dataset-with-some-preprocessing/30385">question</a> I asked for the first time I was working with custom <code>Dataset</code> class.<br>
I hope it helps you</p>

Huhm302622377: <p>Thanks again,you a really nice person！</p>

Nikronic: <p>You’re welcome mate. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"><br>
Did you solve your problem?</p>
Update part of word embeddings during training

lixin4ever: <p>Hi,</p>
<p>In my current project, I introduce auxiliary embeddings to discriminate functional word (e.g., Conjunctions, Determiners, Pronouns and so on).and non-functional word. Specifically, I set auxiliary embeddings of functional words as zero vectors and randomly initialize those of non-functional words. My goal is to fine-tune the latter during training and keep the former unchanged (i.e., always zeros).</p>
<p>Is it feasible for nn.Embedding  layer?</p>

Ooki: <p>Two ways:</p>
<ol>
<li>
<p>This could probably be done by introducing an additional masking embedding with n_dims=1 and have<br>
the embedding of a particular word ‘1.0’ if it should use aux embeddings and ‘0.0’ if not.<br>
Set the require_grads to False and do en element wise multiplication between the<br>
aux embedding and the mask embedding.</p>
</li>
<li>
<p>Calculate a mask (Boolen tensor) that determine if one should use the aux embedding for each particular word<br>
and use it to mask the input before embedding.</p>
</li>
</ol>
<p>Hope this helps <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

smth: <p>you can keep one separate Embedding layer for functional words, and one separate Embedding layer for non-functional words. Seems like a cleaner solution no?</p>

lixin4ever: <p>Thanks for your response!</p>
<p>In my previous implementation,  I adopted your second strategy but I am not sure if it works.</p>

lixin4ever: <p>Good idea!</p>
<p>I think it should work</p>

MrTuo: <p>Is it work? Can you share the code? Thanks a lot!</p>
The results is different when running on different gpus!

yunpei: <p>I use the same code and I run the code in gpu3 and gpu2 alternatively, but I got different results, I don’t know why?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0d7d41758a8b1a2c491a1b6e50a551261c3ebc2f.png" alt="image" data-base62-sha1="1Vkz6wSEyPyVe1cbPakjJPKniKX" width="451" height="174"></p>
<p>Can someone explain why it happens?</p>
<p>Thanks very much.</p>

rasbt: <p>I think this is normal if these are physically different GPUs since cuDNN implements several different algorithms for convolution etc. You can try to set the derministic flag for cuDNN to see if that helps. I.e., set</p>
<pre><code>torch.backends.cudnn.deterministic = True
</code></pre>
<p>in the beginning of your code</p>
How does the value get assigned in tuple in pack_padded_sequence funtion?

Pratik_Sharma: <p>Hello Everyone,</p>
<p>This is my first question in this forum. So,please bear with me.</p>
<p>I am a newbie to pytorch and I am trying to understand Hierarchical attention mechanism for text classification. Below is portion of the code which I downloaded from github to understand.</p>
<p>sentences, bs = pack_padded_sequence(documents,<br>
lengths=sentences_per_document.tolist(),<br>
batch_first=True)</p>
<p>Here, the parameter documents is of dimension 1* 7* 63. 1 being the no of document, 7 being the maximum no of sentence allowed and 63 being the maximum no of word allowed. The sentences_per_document is 4 which means that total no of sentences in the document is 4. Now after the execution of this code,<br>
the dimension of sentences is observed to be 4* 63 which means that there are 4 sentences and 63 is the maximum no of words allowed and bs value is [1,1,1,1].</p>
<p>Now I want to know how does the value get assigned in bs and why is it [1,1,1,1]?</p>
<p>Thank you</p>
How to get parameter tensor of corresponding param node in a traced jit graph?

traveller59: <p>I am working on a torch to tensorrt project. currently the major problem is impossible to get correct weight of an op. A traced resnet18 model produces following inputs:<br>
<code>node: %input.1 : Float(1, 3, 224, 224), %702 : Tensor, %703 : Tensor, %704 : Tensor, %705 : Tensor, %706 : Tensor, %707 : Tensor, %708 : Tensor, %709 : Tensor, %710 : Tensor, %711 : Tensor, %712 : Tensor, %713 : Tensor, %714 : Tensor, %715 : Tensor, %716 : Tensor, %717 : Tensor, %718 : Tensor, %719 : Tensor, %720 : Tensor, %721 : Tensor, %722 : Tensor, %723 : Tensor, %724 : Tensor, %725 : Tensor, %726 : Tensor, %727 : Tensor, %728 : Tensor, %729 : Tensor, %730 : Tensor, %731 : Tensor, %732 : Tensor, %733 : Tensor, %734 : Tensor, %735 : Tensor, %736 : Tensor, %737 : Tensor, %738 : Tensor, %739 : Tensor, %740 : Tensor, %741 : Tensor, %742 : Tensor, %743 : Tensor, %744 : Tensor, %745 : Tensor, %746 : Tensor, %747 : Tensor, %748 : Tensor, %749 : Tensor, %750 : Tensor, %751 : Tensor, %752 : Tensor, %753 : Tensor, %754 : Tensor, %755 : Tensor, %756 : Tensor, %757 : Tensor, %758 : Tensor, %759 : Tensor, %760 : Tensor, %761 : Tensor, %762 : Tensor, %763 : Tensor, %764 : Tensor, %765 : Tensor, %766 : Tensor, %767 : Tensor, %768 : Tensor, %769 : Tensor, %770 : Tensor, %771 : Tensor, %772 : Tensor, %773 : Tensor, %774 : Tensor, %775 : Tensor, %776 : Tensor, %777 : Tensor, %778 : Tensor, %779 : Tensor, %780 : Tensor, %781 : Tensor, %782 : Tensor, %783 : Tensor, %784 : Tensor, %785 : Tensor, %786 : Tensor, %787 : Tensor, %788 : Tensor, %789 : Tensor, %790 : Tensor, %791 : Tensor, %792 : Tensor, %793 : Tensor, %794 : Tensor, %795 : Tensor, %796 : Tensor, %797 : Tensor, %798 : Tensor, %799 : Tensor, %800 : Tensor, %801 : Tensor, %802 : Tensor, %803 : Tensor = prim::Param()</code><br>
It’s possible to get correct input nodes, but for parameter nodes, the only information I can get is “index” of slot, I don’t know how to get corresponding weight of a parameter node.<br>
A workaround is use torch.jit._unique_state_dict and remove all untracked variables to get a list of params, then assign them to param node in reversed order. but this isn’t work for models with unused modules such as torchvision.models.inception_v3 (it has a aux output).<br>
Thanks in advance!</p>

wanchaol: <p>Sorry I don’t quite get what you are asking for. what do you exactly mean <code>the weight of an op</code>? and <code>the correct input node</code>? If you can provide more context that will be good for us to answer your exact question</p>

traveller59: <p>torch.jit.trace create a graph, graph.inputs() return input nodes in net.forward and parameter nodes, the problem is there is no way to get corresponding weight tensor for a parameter node.<br>
I currently use this <a href="https://github.com/traveller59/torch2trt/blob/master/torch2trt/core.py#L395" rel="nofollow noopener">code</a> to get weight tensor to parameter node mapping, but this isn’t guaranteed by pytorch doc.</p>
How do I debug CUDNN_STATUS_EXECUTION_FAILED

ZeweiChu: <p>I got the following error when running my PyTorch code. My understanding is that this error comes from my “gather” function. However, the error message is not helpful for me to debug my code and I do not know exactly which line this error occurs. Is there any trick I can use to debug this kind of error?</p>
<p>The weird thing is that this error sometimes happens and sometimes does not happen, which makes it harder to debug and even to reproduce the bug.</p>
<blockquote>
<p>File “./model.py”, line 435, in update<br>
loss.backward()<br>
File “/home/zewei/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 93, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/zewei/anaconda3/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 89, in backward<br>
allow_unreachable=True) # allow_unreachable flag<br>
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED<br>
/pytorch/aten/src/THC/THCTensorScatterGather.cu:124: void &gt; THCudaTensor_scatterKernel(TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;Real, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [96,0,0] Assertion <code>indexValue &amp;gt;= 0 &amp;amp;&amp;amp; indexValue &amp;lt; tensor.sizes[dim]</code> failed.</p>
</blockquote>

Junwen_Chen: <p>I am encountering the same problem<br>
Did you solve it?</p>

Huhao123: <p>I’ve had this problem because of using topk(). topk  with “nan” is kind of undefined behavior, please refer<br>
<a href="https://github.com/open-mmlab/mmdetection/issues/313" rel="nofollow noopener">github/issues</a> and <a href="https://github.com/pytorch/pytorch/issues/1810" rel="nofollow noopener">pytorch/pytorch#1810</a>. Hoping this helps.</p>
Inception_v3 outdated tutorial

rxvine: <p>I’m following the finetuning tutorial for inception_v3 (all the other models work), but I have the following error: ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 768, 1, 1]).<br>
The code is currently the exact same as the tutorial, is this outdated or should I change something?</p>

ptrblck: <p>Most likely some batchnorm layer is throwing this error, since you are only using a single sample in the batch containing only one pixel.<br>
Try to increase the batch size and run it again.</p>
Computations happening on GPU

caissalover: <p>I’m a newbie and have come from tensorflow. I have a few noobie questions about PyTorch</p>
<p>Why are all models in pytorch made inside a class with nn.Module inherited? What is super(class_name, …).<strong>initi</strong>() ? Can I make a functional model which can fully run on GPU?</p>
<p>What determines whether a certain computation will be run on GPU? If a few tensors are placed on GPU, is it compulsory that all operations whether written in simple python(if, for) or torch.nn.something( gpu_tensor_1, gpu_tensor_2) are run on GPU “completely”? This is where the above model definition comes in, do only those defined in above model run completely on GPU?</p>
<p>I want to place a complete training aspect on GPU and wish to be clear on every aspect of it.</p>
Passing python object reference to c++ extension?

Lukas_Petravicius: <p>Good day everyone ,</p>
<p>Maybe some one knows how to properly handle torch.data_ptr() ? . I want to pass address of tensor to c++ extension of pytorch .</p>
<p>Main idea is to minimize host to device and device to host usage . Is it possible to pass address of pytorch tensor(python) to pytorch( c++) extension and then use it as normal tensor? .  Or it’s better just to rewrite everything in pytorch  c++ frontend</p>
Feeding model with list of inputs

Niki: <p>Would you please guide me regarding this error:</p>
<pre><code class="lang-auto"> output =  model(b)

  File "..", line 489, in __call__
    result = self.forward(*input, **kwargs)

  File "...", line 27, in forward
    output = F.relu(self.fc1(input))

  File "...", line 489, in __call__
    result = self.forward(*input, **kwargs)

  File "...", line 67, in forward
    return F.linear(input, self.weight, self.bias)

  File "...", line 1350, in linear
    if input.dim() == 2 and bias is not None:

AttributeError: 'list' object has no attribute 'dim'
</code></pre>
<p>For this code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
class Test(nn.Module):
        def __init__(self, input_size=2, hidden_size=100):
            super().__init__()
            self.fc1 = nn.Linear(input_size, hidden_size)
            self.fc2 = nn.Linear(hidden_size, hidden_size)
            self.fc3 = nn.Linear(hidden_size, 1)
           
            nn.init.normal_(self.fc1.weight,std=0.02)
            nn.init.constant_(self.fc1.bias, 0)
            nn.init.normal_(self.fc2.weight,std=0.02)
            nn.init.constant_(self.fc2.bias, 0)
            nn.init.normal_(self.fc3.weight,std=0.02)
            nn.init.constant_(self.fc3.bias, 0)
        
        def forward(self, input):
            output = F.relu(self.fc1(input))
            output = F.relu(self.fc2(output))
            output = self.fc3(output)
            return output

model = Test()
input1 = torch.randn(1, 3,16, 16)
input2 = torch.randn(1, 3,16, 16)

b = [input1, input2]

output =  model(b)

print(output)
</code></pre>

JuanFMontesinos: <p>You can pass whatever as input but you can feed built-in layers with tensors.<br>
to do what you are doing you need something like<br>
Analogous ways:<br>
<code>output = list(map(model,b))</code><br>
<code>output = [model(f) for f in b]</code></p>
<pre><code class="lang-auto">output = []
for f in b:
    output.append(model(f))
</code></pre>

Niki: <aside class="quote no-group quote-modified" data-username="JuanFMontesinos" data-post="2" data-topic="44931">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>output = <span class="chcklst-box fa fa-square-o fa-fw"></span> for f in b: output.append(model(f))</p>
</blockquote>
</aside>
<p>Thank you very much for your reply, <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>.<br>
It worked.</p>
PyTorch - CUDA Interoperability

akv: <p>Does anyone have any idea what the recommended way of going from PyTorch to CUDA is, either for the Python or the C++ side?</p>
<p>I want to be able to use my PyTorch tensors in CUDA, and I was trying to figure out whether to use CuPy or the C++ API, and how to correctly deal with this.</p>
<p>I was wondering if anyone has any insights into the management of context creations and streams with respect to PyTorch and CUDA, and the correct way to handle them if I want to be able to use the low-level CUDA functions. Any examples or pointers would be really helpful!</p>
How to convert Pytorch Image Tensor to OpenCV

Mark0502: <p>Hello everybody,<br>
i want to use a Pytorch Image Tensor for an ObjectDetector.detect_object_in_image() (from Nvidia/Dope) and need it as Opencv Image. But everything I try crashes… like the following":</p>
<pre><code class="lang-auto">  File "test2.py", line 1844, in &lt;module&gt;
    _runnetwork(epoch, testingdata, train=False)
  File "test2.py", line 1809, in _runnetwork
    config_detect
  File "/home/nils/catkin_ws/src/dope/src/inference/detector.py", line 263, in detect_object_in_image
    image_tensor = transform(in_img)
  File "/home/nils/catkin_ws/local/lib/python2.7/site-packages/torchvision/transforms/transforms.py", line 49, in __call__
    img = t(img)
  File "/home/nils/catkin_ws/local/lib/python2.7/site-packages/torchvision/transforms/transforms.py", line 76, in __call__
    return F.to_tensor(pic)
  File "/home/nils/catkin_ws/local/lib/python2.7/site-packages/torchvision/transforms/functional.py", line 44, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test2.py", line 1844, in &lt;module&gt;
    _runnetwork(epoch, testingdata, train=False)
  File "test2.py", line 1802, in _runnetwork
    rgb_image = pil2tensor(img)           # rgb_image = pil2tensor(images['img'])
  File "/home/user/catkin_ws/local/lib/python2.7/site-packages/torchvision/transforms/transforms.py", line 76, in __call__
    return F.to_tensor(pic)
  File "/home/user/catkin_ws/local/lib/python2.7/site-packages/torchvision/transforms/functional.py", line 44, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got &lt;type 'numpy.ndarray'&gt;
</code></pre>
Gradient penalty for lipchitzness not working with convolutional layers

rishubh: <p>I am trying to impose Lipchitzness using Grad Penalty like in WGAN on an encoder network. The backward on the gradient penatly term works and the model gets trained if the encoder network only consists of linear (fully connected) layers. On changing too / adding convolutional layers to the model, I get the error :<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>Loss and gradient penatly code:</p>
<pre><code class="lang-auto">recon_loss = F.mse_loss(X_sample, X, size_average=False) / mb_size

X_recon = P2(z_dis)
recon_loss_P2 = F.mse_loss(X_recon, X, size_average=False) / mb_size
loss = recon_loss + recon_loss_P2

# gradient penalization (effectively, second order derivative)
gradQ = grad(z_con.mean(), X, create_graph=True)
gradQ0 = gradQ[0]
gradQ_norm = gradQ0.norm()
gradient_penalty = (gradQ_norm - 0.0).pow(2)
loss.backward(retain_graph=True)
gradient_penalty.backward() #this line gives the error

</code></pre>
<p>Encoder code:</p>
<pre><code class="lang-auto">class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

Q = nn.Sequential(
    nn.Conv2d(1, 1, 4, stride=2, padding=1), # it works on removing this line of code
    Flatten(),
    nn.Linear(784, 10)
)

</code></pre>
<p>Why might this be happening? I am also open to imposing the gradient penalty in another way if that works with convolutional layers.</p>
How does distributions.entropy() work?

whoab: <p>distributions.Categorical(probs=ret_attn_softmax).entropy()</p>
<p>ret_attn_softmax is a [256, 2] vector representing 256 Bernoullii distributions.</p>
<p>If I stick it in the above code, it outputs something, but it doesn’t look correct at all for the entropy. Shouldn’t the entropy have a dimensional input so the code knows which rows correspond to distributions?</p>
Can I keep my CUDA version along with Anaconda-based Pytorch CUDA?

DGao: <p>I’m new to Pytorch (and Anaconda). I’ve read related topics in the forum but am still confused about the compatibility. My ubuntu has already got CUDA 10.1 + CuDNN installed with Nvidia 418.56 driver. I’ve also had CUDA environment and library path setup. Then I installed Pytorch through Anaconda which comes with CUDA toolkit 10.0. I’m wondering:</p>
<ol>
<li>Am I correct that Pytorch only calls CUDA tools purely within the Anaconda library path,  without looking for CUDA tools in the OS environment?</li>
<li>For long term, is it safe to keep two versions of CUDA like this? I’d like to keep the OS-based CUDA for some graphic works, as its feature is a bit more up-to-date. But I don’t want to mess things up.<br>
Any suggestion? Thanks.</li>
</ol>
Gradient behavior with multi-layer loss

williamFalcon: <p>I have a loss where each layer plays into the loss. Which is the correct approach in terms of making sure the weights are updated properly?</p>
<pre><code># option 1
x2 = self.layer1(x1)
x3 = self.layer2(x2)
x4 = self.layer3(x3)
</code></pre>
<p>In this option, I detach when feeding into each subsequent block</p>
<pre><code>    # option 2
    # x2 = self.layer1(x1.detach())
    # x3 = self.layer2(x2.detach())
    # x4 = self.layer3(x3.detach())
</code></pre>
<p>shared ops which calculate 4 losses and sum them.</p>
<pre><code>    x4 = F.relu(self.bn1(x4))
    loss = some_loss([x1, x2, x3, x4])</code></pre>

MariosOreo: <p>Hello,</p>
<p>I think option 2 with <code>.detach()</code> is not the correct way. Option 1 seems like a separate version of the third code snippet, you could have a try on these two ways (<span class="hashtag">#1</span>, <span class="hashtag">#3</span>).</p>
Why does tensor type cast not work sometimes

xiaotou: <p>After tensor.half() called, the tensor’s dtype is not casted to half.<br>
I found some docs as below:<br>
" If the <code>self</code> Tensor already has the correct [ <code>torch.dtype</code> ] and [ <code>torch.device</code> ], then <code>self</code> is returned. Otherwise, the returned tensor is a copy of <code>self</code> with the desired [ <code>torch.dtype</code> ] and [ <code>torch.device</code> ]."<br>
So, what does the “correct [ <code>torch.dtype</code> ] and [ <code>torch.device</code> ]” real mean?</p>

tom: <p>For me the tensor is certainly cast.</p>
<pre><code class="lang-auto">torch.randn(5).half()
tensor([ 0.1776,  0.1658,  0.0118, -0.0336, -0.1261], dtype=torch.float16)

torch.randn(5, device='cuda').half()
tensor([ 0.5459,  0.3481, -1.5215, -0.3481,  0.2035], device='cuda:0',
       dtype=torch.float16)
</code></pre>
<p>What the sentence you quote means is that when the input (self) already has the desired type / device, then you are getting the same tensor, not a copy (while when a cast is needed, that necessarily implies copying).</p>
<p>Best regards</p>
<p>Thomas</p>

xiaotou: <p>Thanks for your relay.</p>
<p>Here is my code:<br>
print("tensors.dtype: ", tensors.dtype)   -&gt;  tensors.dtype:  torch.float32<br>
self.tensors = tensors<br>
tensors.half()<br>
print("tensors.dtype: ", tensors.dtype)   -&gt;  tensors.dtype:  torch.float32  , not torch.float16<br>
self.image_sizes = image_sizes<br>
very confused!</p>

tom: <p>Tensor.half is not an inplace operation (on Tensors (as opposed to modules) only functions ending with <code>_</code> are).<br>
So <code>cast_tensors = tensors.half()</code> or so.</p>
<p>Best regards</p>
<p>Thomas</p>
Add_scalars tensorboard

JuanFMontesinos: <p>I was wondering the reason why add_scalars warns about exploiding memory. What’s the point of saving scalars in RAM? planning to let this to be optional?</p>
<pre><code class="lang-auto">    def add_scalars(self, main_tag, tag_scalar_dict, global_step=None, walltime=None):
        """Adds many scalar data to summary.

        Note that this function also keeps logged scalars in memory. In extreme case it explodes your RAM.
</code></pre>
<p>EDIT:<br>
is it possible to use add_scalars passing values at different times?<br>
Rewriting the question<br>
provided example sends a single dictionary with 3 values. Is it possible to send 3 dictionaries to the same timestap with different subtags getting them properly plotted?</p>
Trouble with a custom autograd function

jfsantos.ds: <p>Hello,</p>
<p>I am trying to make a custom loss function based on the probability mass function of a negative binomial distribution.<br>
The parameters total_count and probs are a size (2,1) tensor. It comes as a product of the previous layers of my model in the forward pass.</p>
<p>Using these parameters to define the the negative binomial distribution, I want to take the negative log of it’s probability mass function as loss. I computed the gradient of this function wrt. total_count and probs but I can’t make the backward method work.<br>
So far I have this:</p>
<pre><code>class NBLL_loss(Function):
@staticmethod
def forward(ctx, input, target):
    tc, prb = input
    n_b=NegativeBinomial(total_count=tc, probs=prb)
    loss=-n_b.log_prob(target)        
    ctx.save_for_backward(loss,tc,prb,target)
    return loss

@staticmethod
def backward(ctx, grad_loss):
    loss,tc,prb,target, = ctx.saved_tensors
    dg=digamma()
    grad_loss[0] = -dg(target+tc)+dg(tc)-np.log(1-prb)
    grad_loss[1] = -target*(1/prb)+tc*(1/(1-prb))
    return grad_loss, None
</code></pre>
<p>I’m trying it out with the following lines:</p>
<pre><code>loss=NBLL_loss.apply(torch.tensor([[5.],[0.5]], requires_grad=True),torch.tensor([5.]))
loss.backward()
</code></pre>
<p>Forward method is working fine but I’m getting “RuntimeError: No grad accumulator for a saved leaf!” on the backward call.</p>
<p>Can someone help me understand what am I doing wrong?</p>
<p>Thank you,</p>
How to save a model attribute in with state_dict()

Hahnbosch1: <p>Hi,</p>
<p>I have a class <code>DiscreteMLP(torch.nn.Module)</code>, in which I overwrite <code>state_dict()</code> and <code>load_state_dict()</code> to save also the class attribute <code>self.theta</code> and load it again.</p>
<pre><code class="lang-auto">
    def state_dict(self, destination=None, prefix='', keep_vars=False):
        """ Overrides state_dict() to save also theta value"""
        original_dict = super().state_dict(destination, prefix, keep_vars)
        original_dict[prefix+'theta'] = self.theta
        return original_dict

    def load_state_dict(self, state_dict, strict=True):
        """ Overrides state_dict() to load also theta value"""
        theta = state_dict.pop('theta')
        self.set_theta(theta)
        super().load_state_dict(state_dict, strict)
</code></pre>
<p>It works fine if I save and load this mlp only. However, when I integrate this mlp into a bigger model, I have a problem:</p>
<pre><code class="lang-auto">if len(error_msgs) &gt; 0:
            raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
&gt;                              self.__class__.__name__, "\n\t".join(error_msgs)))
E           RuntimeError: Error(s) in loading state_dict for OptionsGraph:
E           	Unexpected key(s) in state_dict: "0.theta".
</code></pre>
<p>The bigger model is a model with multiple layers, where each layer is a mlp.</p>
<p>I could not find a proper pytorch tutorial, how to overwrite <code>state_dict()</code> and <code>load_state_dict()</code> that can handle this case. Could you please help?</p>
<p>Thanks a lot!</p>

nutszebra: <p>If I were you, I set <code>theta</code> by <code>register_buffer</code> in the custom mlp like <code>running_mean</code> in batch norm (here is the <a href="https://github.com/pytorch/pytorch/blob/99874f87cbd3e0004c23f8514311fe2a336fd6de/torch/nn/modules/batchnorm.py#L35" rel="nofollow noopener">link</a> of that part).<br>
In this case, it’s not necessary to overwrite <code>load_state_dict</code> and <code>state_dict</code>.</p>

Hahnbosch1: <p>Thank you, that’s exactly what I am looking for. A clean way to register theta.</p>
<pre><code class="lang-auto">self.register_buffer('theta', torch.tensor([theta]))
</code></pre>
Can torch.nn.AdaptiveLogSoftmaxWithLoss specifies a target value that is ignored and does not contribute to the input gradient like torch.nn.CrossEntropyLoss?

cswangjiawei: <p>Can <code>torch.nn.AdaptiveLogSoftmaxWithLoss</code>   specifies a target value that is ignored and does not contribute to the input gradient like <code>torch.nn.CrossEntropyLoss</code>?  In many cases, we need to pad the text so that all the sequences are the same length, so we can process them in batch. So I think specifying a target value that is ignored and does not contribute to the input gradient is necessary.</p>
Modifying data in batch

GeoffNN: <p>Hi all,</p>
<p>I have a use case where I need to optimize over dual variables of the size of my dataset. Namely, I would like to load <code>(input, duals, targets)</code> at each batch, and then optimize both over <code>duals</code> and my model’s parameters. My loss function takes <code>(model(input), duals, targets)</code> as input. At the end of the optimization, I would like to update <code>duals</code> in my dataset.</p>
<p>How can I do this?</p>
<p>Thanks!</p>

MariosOreo: <p>I don’t understand your use case clearly, especially the dual variables of the size of the dataset.<br>
If the dual variable you mentioned is learnable, you could set <code>requires_grad=True</code> and add them to the optimizer.<br>
Does this method fit your use case?</p>

GeoffNN: <p>I’m optimizing the dual of a convex optimization problem, that yields an unconstrained optimization problem of the above form. My <code>dual</code> variable is of size <code>n_data x m</code> since originally I had <code>n_data x m</code> constraints in my primal constrained convex problem and each row of my <code>dual</code> variable corresponds to a specific datapoint. I would prefer batching these variables with the datapoints to allow for easy shuffling.</p>
Why should I set this decoder_learning_ratio parameter in Pytorch Chatbot Tutorial

doragd: <p>Why should I set this decoder_learning_ratio parameter?<br>
why does we set decoder_learning_ratio to 5?</p>
<pre><code class="lang-auto">learning_rate = 1e-3
decoder_learning_ratio = 5.0

decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate * decoder_learning_ratio)
</code></pre>
Where is torch._C._functions.BatchNorm located?

Nabarun_Goswami: <p>Hi,</p>
<p>In nn/functional.py,<br>
<code>f = torch._C._functions.BatchNorm(running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled)</code></p>
<p>Can I get the current value of running_mean and running_var from this call? If yes, can I also get the current mini-batch mean and var.</p>
<p>I am looking where this function comes from so that I can figure it out. Any pointers would be most helpful.</p>
<p>Regards<br>
Nabarun</p>

albanD: <p>Hi,</p>
<p>This is one of the functions that has been moved to cpp.<br>
It is implemented in <a href="https://github.com/pytorch/pytorch/tree/e5b5154768ca2ba26742d75d13df4ea89a0d814b/torch/csrc/autograd/functions">this</a> folder.<br>
Especially, the argument parsing is <a href="https://github.com/pytorch/pytorch/blob/e5b5154768ca2ba26742d75d13df4ea89a0d814b/torch/csrc/autograd/functions/init.cpp#L19-L25">here</a>.<br>
And the definition on how to access elements such as <code>running_mean</code> or <code>running_var</code> is <a href="https://github.com/pytorch/pytorch/blob/e5b5154768ca2ba26742d75d13df4ea89a0d814b/torch/csrc/autograd/functions/init.cpp#L121-L136">here</a>.</p>
Using torch.nonzero to index into a larger tensor

david.bernstein: <p>Relatively new to pytorch so this might already be answered. Suppose I have a tensor y of shape (H, W) so that</p>
<p>idx = torch.nonzero(y)</p>
<p>has shape (N, 2) where N is the number of nonzero entries of y. I also have a tensor x of shape (4, H, W) and I want to create an output tensor z of shape (N, 4) where I’ve used idx to index into the last two dimensions of x. In my code I want to return idx and z as parallel tensors. There has to be a neat way to do this but I’m not there yet with pytorch.</p>
<p>Thanks,<br>
Dave</p>

MariosOreo: <p>Hi Dave,</p>
<p>You could have a try on the code snippet as follows, I’m not sure whether it is efficient, but it works.</p>
<pre><code class="lang-auto">z = x[:, idx[:, 0], idx[:, 1]]
z = torch.transpose(z, 0, 1)
</code></pre>
PyTorch TPU support on Google Colab

tarunpaparaju: <p>When can I train PyTorch models on Google Colab Cloud TPUs ? I’m trying to avoid shifting to tensorflow for my project just for the TPUs.</p>

marioviti: <p>I guess we’ll have to wait a really long time…</p>

sakaia: <p>You can try Google Colab by using following jupyter notebook.<br>
<a href="https://github.com/pytorch/xla/tree/master/contrib/colab" rel="nofollow noopener">https://github.com/pytorch/xla/tree/master/contrib/colab</a></p>

sakaia: <p>From seeing following url, nightly build package for pytorch_xla exist.<br>
<a href="http://storage.googleapis.com/pytorch-tpu-releases/" rel="nofollow noopener">http://storage.googleapis.com/pytorch-tpu-releases/</a></p>
<p>Which combination with torch_xla &amp; torch is latest?<br>
for example, which torch version should be work with wheels/torch_xla-20190508-0.1+d581df3-cp35-cp35m-linux_x86_64.whl?</p>
Can I directly load pre-trained weights from common convolution to dilated convolution?

lity: <p>Hi, I want to do experiment on dilated convolution. However, I have no time and GPUs to pre-trained on ImageNet. So I want to load pre-trained weights on ImageNet though they are common convolution which have 1 rate. From 3x3 common conv whilch has 1 rate to 3x3 dilated conv which has 2 rate, the numbers of parametres are same.</p>
<p>I have a try and it seems that dilated convolution can load weights successfully. However, the performance is very bad and the net cannot work normally. So I want to know, can I directly load pre-trained weights from common convolution to dilated convolution? It will cause some problem？Or just because there is no real pre-train on Imagenet, the performance is very poor.</p>

ptrblck: <p>I guess the performance is bad, as the activations might be completely different using dilated convolutions.<br>
Are you just checking the performance or are you also fine tuning the model?<br>
The latter might help adapt the dilated convolutions to your use case.</p>

lity: <p>I fine tune the model and the result is a little lit bad. Perhaps I need to train from scratch in ImageNet?</p>

MingHongL: <p>Hi,<br>
I also did some experiment on dilated convolution. but I found that using the dilated convolution will cause several times the training time than convolution without dilation. I want to know if this situation is normal.<br>
if you know  the answer, could you tell me？Thanks!</p>

ptrblck: <p>You might solve the performance issue setting <code>torch.backends.cudnn.benchmark=True</code>, if your input sizes are static (i.e. they don’t change in each iteration).<br>
Have a look at <a href="https://github.com/pytorch/pytorch/issues/15054">this issue</a> for more information.</p>
<blockquote>
<p>This issue might be fixed by cudnn 7.5.1, which was shipped in pytorch 1.1.<br>
Please give it a try</p>
</blockquote>

MingHongL: <p>Thanks for your reply！It is really helpful.</p>
Extending module and function pytorch

foudam: <p>Hello All,</p>
<p>I’m trying to extend pytorch by adding a new module and function.</p>
<pre><code class="lang-auto">class Linear(torch.autograd.Function):
    @staticmethod
    def forward(self, input, weight_Pos, weight_Neg, bias=None):
        self.save_for_backward(input,weight_Pos, weight_Neg, bias)
        weight =weight_Pos - weight_Neg
        if input.dim() == 2 and bias is not None:
            # fused op is marginally faster
            ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
        else:
            output = input.matmul(weight.t())
            if bias is not None:
                output += torch.jit._unwrap_optional(bias)
            ret = output
        return ret    
    
    @staticmethod
    def backward(self, grad_output):
        input, weight_Pos, weight_Neg, bias = self.saved_tensors
        grad_input = grad_weight_Pos= grad_weight_Neg = grad_bias = None
        if self.needs_input_grad[0]:
            grad_input = grad_output.mm(weight_Pos)
        if self.needs_input_grad[1]:
            grad_weight_Pos= grad_output.t().mm(input)
        if self.needs_input_grad[2]:
            grad_weight_Neg = -1*grad_output.t().mm(input)
        if bias is not None and self.needs_input_grad[3]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight_Pos, grad_weight_Neg, grad_bias
</code></pre>
<p>But, I’m getting an error and I don’t know the problem.</p>
<pre><code class="lang-auto">&lt;ipython-input-205-f9f4df5e472f&gt; in backward(self, grad_output)
     29             grad_input = grad_output.mm(weight_Pos)
     30         if self.needs_input_grad[1]:
---&gt; 31             grad_weight_Pos= grad_output.t().mm(input)
     32         if self.needs_input_grad[2]:
     33             grad_weight_Neg = -1*grad_output.t().mm(input)

RuntimeError: t() expects a 2D tensor, but self is 1D
</code></pre>