akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the most common way to deply pytorch model?

username: juhyung
cooked: <p>I found that there is no third party for serving. Like tf serving server.</p>
<p>So I searched how pytorch users serve model and there are some ways like this</p>
<ol>
<li>pytoch model -&gt; onnx -&gt; tensorrt inference server</li>
<li>pytorch model -&gt; jit -&gt; tensorrt inference server</li>
<li>pytoch model -&gt; load in python -&gt; serve with flask etc.<br>
and so on</li>
</ol>
<p>I wonder what is the common or preferred way to serving pytorch model.</p>

username: RicCu
cooked: <p>Thre’s also <a href="https://github.com/microsoft/onnxruntime" rel="nofollow noopener">onnx runtime</a> which has a model server in beta.<br>
And there’s an <a href="https://github.com/pytorch/pytorch/issues/27610" rel="nofollow noopener">RFC</a> open for a production serving framework for PyTorch.</p>
Jit.scripting nn.Module with function attributes

username: AustenLamacraft
cooked: <p>Suppose I want to instantiate a Module with a function:</p>
<pre><code class="lang-auto">class Foo(nn.Module):
    def __init__(self, fn):
        super(Foo, self).__init__()
        self.fn = fn
        
    def forward(self, x):
        return self.fn(x)
</code></pre>
<p>This works fine</p>
<pre><code class="lang-auto">def square(x):
    return x**2

foonet = Foo(square)
</code></pre>
<p>But attempting to script the module fails</p>
<pre><code class="lang-auto">foonet_script = torch.jit.script(foonet)
</code></pre>
<p>giving</p>
<pre><code class="lang-auto">RuntimeError: 
module has no attribute 'fn':
at &lt;ipython-input-25-24af853ed724&gt;:7:15
    def forward(self, x):
        return self.fn(x)
               ~~~~~~~ &lt;--- HERE
</code></pre>
<p>It doesn’t help if I first script the function</p>
<pre><code class="lang-auto">square_script = torch.jit.script(square)
</code></pre>
<p>and then instantiate with the resulting <code>torch._C.Function</code>. On the other hand, if I pass in another <code>nn.Module</code>, everything is fine. Setting up a whole new Module when all I want is the function call seems overkill though.</p>

username: AustenLamacraft
cooked: <p><strong>Update:</strong> It’s a known bug see <a href="https://github.com/pytorch/pytorch/issues/27495" rel="nofollow noopener">this issue</a> for function attributes and <a href="https://discuss.pytorch.org/t/jit-scripted-attributes-inside-module/60645">this post</a> for class attributes.</p>

username: driazati
cooked: <p>This bug should be fixed in <a href="https://github.com/pytorch/pytorch/pull/28569" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/28569</a>, you should be able to try out the fix if you build master from source or use the nightly PyTorch pip package</p>
64-bit Double math based inference in PyTorch

username: hashbrown
cooked: <p>Hi,</p>
<p>I have used the default 32-bit float to train a network in PyTorch and saved the model state dict. I would like to load it for testing purpose and I would like all the math be done in double precision.</p>
<p>What is the way to do this?</p>
<p>I put the following in the beginning of my code:</p>
<pre><code class="lang-auto">torch.set_default_tensor_type(torch.DoubleTensor)
torch.set_default_dtype(torch.float64)
</code></pre>
<p>But I get this error message:</p>
<pre><code class="lang-auto">  File "lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "lib/python3.6/site-packages/torch/nn/functional.py", line 1656, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: Expected tensor for argument #1 'input' to have the same type as tensor for argument #2 'weight'; but type torch.cuda.FloatTensor does not equal torch.cuda.DoubleTensor (while checking arguments for cudnn_batch_norm)
</code></pre>
<p>Appreciate your inputs, thanks!</p>

username: Yaroslav_Bulatov
cooked: <p>You also need to convert your model parameters to be double. IE, call <code>module.double();</code></p>

username: hashbrown
cooked: <p>Thanks! Let me give it a shot.</p>
How to deploy different scripts on different GPUs?

username: Huimin_ZENG
cooked: <p>Hello everybody,</p>
<p>I am trying to deploy different pytorch-based training scripts on different GPUs. However, the information I could find is about training a model on multiple GPUs.</p>
<p>Could some tell me how to do this?<br>
I tried the ‘spawn’ trick, ‘.cuda(0)/.cuda(1)’ trick… But they were not working.</p>
<p>Sorry if this is a bad question</p>
<p>Thank you!</p>

username: JuanFMontesinos
cooked: <p>just need to pass CUDA_VISIBLE_DEVICES=0 python script.py<br>
replace 0 by the gpu you want</p>

username: TinfoilHat0
cooked: <p>You could pass the device you want to train on as an argument to the script.</p>
<p>For example, ‘cuda:0’ corresponds to the 1st GPU in your system, ‘cuda:1’ corresponds to the 2nd GPU and so on.</p>
<p>Then assuming you store the passed argument in a variable named <code>device</code>, all you have to do is to call <code>.to(device)</code> on your tensors etc.</p>
Pytorch with complex tensors

username: cosmmb
cooked: <p>Is there a solution for dealing with complex tensors yet? Is there a similar function to numpy.angle in PyTorch?</p>
<p>thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check this issue to see the internal work on complex tensors: <a href="https://github.com/pytorch/pytorch/issues/755" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/755</a></p>
<p>Also third party libraries might give some partial support for complex tensors. A quick search points to <a href="https://github.com/williamFalcon/pytorch-complex-tensor" rel="nofollow noopener">https://github.com/williamFalcon/pytorch-complex-tensor</a> for example.</p>

username: cosmmb
cooked: <p>thanks! I just found a way around complex number with some certain precision loss.</p>
How to make ImageFolder ignore .ipynb_checkpoints

username: eric01300
cooked: <p>Hello, I am having a problem of using ImageFolder in Jupyter Notebook. Jupyter Notebook automatically creates .ipynb_checkpoints file, and ImageFolder recognizes it as one of the classes. I can remove it, but everytime I run the code it’s there again. Anyway for ImageFolder to completely ignore it?</p>

username: ptrblck
cooked: <p>I couldn’t reproduce this issue. Also, <code>ImageFolder</code> should only detect the allowed file extensions.<br>
Could you post your folder structure so that I could have a look?</p>

username: whydoineedausername
cooked: <p>I’m having this same issue. My folder structure is simply images_dir -&gt; many images with a .jpeg extension. Whenever I try to load this dir in a notebook with <code>datasets.ImageFolder</code>, the notebook creates a .ipynbcheckpoints folder and starts placing loaded images in there.</p>
<p>Edit: I think I see the issue. The ImageFolder loader is expecting images to be separated into subdirectories. I have no need of a subdirectory, but I guess I need to make one anyway to avoid this.</p>

username: ptrblck
cooked: <p>As you explained, <code>ImageFolder</code> uses a subdirectory for each class.<br>
If you don’t need it, you could write your custom <code>Dataset</code> and load the data using your own logic.</p>
I have a error can you help me?

username: 1415140539
cooked: <blockquote>
<blockquote>
<blockquote>
<p>net.cuda()<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “E:\python\lib\site-packages\torch\nn\modules\module.py”, line 305, in cuda<br>
return self._apply(lambda t: t.cuda(device))<br>
File “E:\python\lib\site-packages\torch\nn\modules\module.py”, line 224, in _apply<br>
param_applied = fn(param)<br>
File “E:\python\lib\site-packages\torch\nn\modules\module.py”, line 305, in <br>
return self.<em>apply(lambda t: t.cuda(device))<br>
File "E:\python\lib\site-packages\torch\cuda_<em>init</em></em>.py", line 193, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: generic_type: cannot initialize type “_CudaDeviceProperties”: an object with that name is already defined</p>
</blockquote>
</blockquote>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>Could you give more informations about your system, your hardware, OS and how you installed pytorch please?</p>
Optimizing the Wrong Model in Pytorch

username: Nikhil_Pinnaparaju
cooked: <p>I am somewhat new to this so please bear with me. Recently, I was training a bunch of models in Pytorch, when I ran into this problem. Suppose, I have some models m1 and m2.</p>
<pre><code class="lang-auto">m1 = modelClass1()
optimizer = torch.optim.Adam(model.parameters())
m1.train()
optimizer.step()

m2 = modelClass2()
m2.train()
optimizer.step()
</code></pre>
<p>I forgot to reinitialize my optimizer and the loss and accuracies were changing for my first model but not for my second model (which makes sense because I didn’t set the optimizer to optimize the 2nd model’s parameters).</p>
<p>Considering the modelClass1() and modelClass2() are different, why is it that there was no error or anything thrown when I forgot to optimize different models with different parameters. What exactly is optimizer doing with the parameters passed as an input to it. Was looking for some insight into understanding this.</p>

username: albanD
cooked: <p>When step is called, it will iterate over all the parameters you gave it and make a gradient step for these parameters.<br>
In the case of your second step, since you did not zero out gradients, it will do another step using the previously computed gradients.</p>
FasterRCNN & Open Images

username: dusted996
cooked: <p>Hi, I’m retraining FasterRCNN on Openimages and have a couple of queries:</p>
<p>What image size should I be using for the resnet50_fpn backbone in the data loader?  (I’ve seen both 128x128 and 256x256 on Kaggle, but the correct value for resnet50 I thought was 224x224?</p>
<p>When calculating IoU how do I know which object in validation matches the inferred target object for any given image?  Do I just check all boxes of the given classification and select the best IoU as the correct object?</p>
<p>Any help really appreciated!</p>
How to get pytorch concat function to work by index

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>I have two models that take in images from a house, one being the house exterior and one being the roof.  Each observation has multiple photos and not every observation has the same amount of photos.  Is it possible to concat the two models by house ID?</p>
<p>If so, how would I do that.  My current custom dataclass object returns the image, target label and ID.</p>
Softmax gives output vector whose sum is greater than 1

username: Oormila_Ghantasala
cooked: <p>Hi,<br>
I am a newbie to PyTorch. I was trying out the following network architecture to train a multi-class classifier. I used Softmax at the output layer and cross entropy as the loss function. However, the output doesn’t look like probabilities. For example, one of the outputs looks like this [2.0032e-10, 1.798e-8, …1.0000e+0,…2.112e-4]. My question is, how can one of them be 1 when their sum has to be equal to 1.</p>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 13 features in input layer
        self.fc1 = nn.Linear(13, 512)
        self.fc2 = nn.Linear(512, 512)
        self.fc3 = nn.Linear(512, 512)
        self.fc4 = nn.Linear(512, 512)
        self.fc5 = nn.Linear(512, 40)
        self.bn1 = nn.BatchNorm1d(512)
        self.bn2 = nn.BatchNorm1d(512)
        self.bn3 = nn.BatchNorm1d(512)
        self.bn4 = nn.BatchNorm1d(512)
        

    def forward(self, x):
        x = self.bn1(F.relu(self.fc1(x)))
        x = self.bn2(F.relu(self.fc2(x)))
        x = self.bn3(F.relu(self.fc3(x)))
        x = self.bn4(F.relu(self.fc4(x)))
        x = nn.Softmax(dim=1)(self.fc5(x))
        return x
    
</code></pre>
<p>Please correct me if I am wrong and help me out.</p>

username: ptrblck
cooked: <p>You are most likely seeing some floating point precision issues.<br>
That being said, note that <code>nn.CrossEntropyLoss</code> expects logits, as internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied, so you should remove the softmax for this criterion.</p>

username: Oormila_Ghantasala
cooked: <p>If softmax is removed, output range is not between 0 and 1. It contains negative values as well like [-12.098, 2.0988, -12.121…, 0.87, 0.21]. But I need probabilities for each of the classes.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects these logits.<br>
For debugging purposes you could still apply <code>softmax</code> on the output. Just don’t pass it to the criterion.</p>

username: Oormila_Ghantasala
cooked: <p>So how do I get the class probabilities? I need the output as probabilities. How can I achieve that without/with using Softmax?</p>

username: Oormila_Ghantasala
cooked: <p>Solved. I got output as probabilities by sending the predicted values to softmax but didn’t include softmax in the net architecture as cross-entropy already applies softmax internally. Thank you.</p>
Use packed padded sequence to RNN cell

username: zhdkffk533
cooked: <p>HI ! I’m biginner to pytorch!</p>
<p>And I’m trying to use packed padded sequence to torch.nn.LSTMCell,</p>
<p>but there is no tutorials about using packed padded sequences to torch.nn.LSTMCELL(not torch.nn.LSTM)</p>
<p>is there any way to use packed padded sequences to torch.nn.LSTMCELL?</p>
[Android] Build nightly from source and import

username: zetyquickly
cooked: <p>Hello there,</p>
<p>With help (<a href="https://discuss.pytorch.org/t/android-build-failed/60873" class="inline-onebox">[Android] Build failed</a>) I’ve managed to built AAR files but using the explanation on PyTorch website it impossible to build android project.<br>
Here is relevant issues (<a href="https://github.com/pytorch/android-demo-app/issues/31" rel="nofollow noopener">https://github.com/pytorch/android-demo-app/issues/31</a>) and (<a href="https://github.com/pytorch/android-demo-app/issues/29" rel="nofollow noopener">https://github.com/pytorch/android-demo-app/issues/29</a>)<br>
The first error appears when you try to build from terminal and the second when you try to build from Android Studio.</p>
<p>How to properly add AAR files to android project?</p>

username: zetyquickly
cooked: <p>This was a small heap issue so add:</p>
<pre><code class="lang-auto">org.gradle.jvmargs=-Xmx4096m
</code></pre>
<p>to your  <code>gradle.properties</code></p>
Compile libtorch.a without pthreadpool?

username: divinho
cooked: <p>I’m using pytorch and lib A in an app. LibA depends on pthreadpool, and pytorch seems to as well except it bundles what it needs into libtorch.a.</p>
<p>What this leads to is if I include pthreadpool as a dependency I get a “multiple definitions” error, because it’s defined in both libpthreadpool.a and libtorch.a. But when I don’t then libA is missing the definitions of functions it uses.</p>
<p>The only way I see out of this is to somehow stop the building process from including pthreadpool stuff into libtorch.a, but I can’t find out how it’s doing that, there are no references to pthreadpool in the main <code>CMakeLists.txt</code> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: divinho
cooked: <p>I got it to work by commenting out the pthreadpool files under caffe2/utils/CMakeLists.txt</p>
How to properly use Pytorch multiprocessing to accelerate model training?

username: DKSG
cooked: <p>I try to accelerate my model training by using multiple CPUs. My code is like</p>
<pre><code class="lang-auto">def train(model, rank):
    for it,batch in enumerate(data_loader):
        optimizer.zero_grad()
        loss(model(batch)).backward()
        print("iter: {} at core: {}, current loss: {:10.4f}".format(it, rank, loss.item()/len(batch)), flush=True)
        optimizer.step()

model = nnet() # inherit from nn.Module
num_processes = 3
model.share_memory()
processes = []
for rank in range(num_processes):
    p = mp.Process(target=train, args=(func, rank))
    p.start()
    processes.append(p)
for p in processes:
    p.join()
</code></pre>
<p>and the output looks like (whether I select num_workers = 0 or 3 in dataloader)</p>
<blockquote>
<p>iter: 0 at core: 0, current loss:   144.0524<br>
iter: 0 at core: 1, current loss:   144.0524<br>
iter: 0 at core: 2, current loss:   144.0524<br>
iter: 1 at core: 0, current loss:   137.5519<br>
iter: 1 at core: 1, current loss:   137.6043<br>
iter: 1 at core: 2, current loss:   137.6081<br>
iter: 2 at core: 0, current loss:   131.0690<br>
iter: 2 at core: 1, current loss:   131.0943<br>
iter: 2 at core: 2, current loss:   131.0782<br>
…</p>
</blockquote>
<p>It seems they independently train the model (each core did their own backpropagation) instead of trying to combine the gradient from other sub-processes to do backpropagation in each iteration (given the starting parameter values during loss calculation are the same). This does not seem to decrease the training time.<br>
I would like to know how I should use multiple CPUs to accelerate training in Pytorch properly.</p>
<p>Thank you.</p>
<p>Edit: Even worse, multiprocessing updates parameter values incorrectly. For example, core 1 and core 2 use parameter p1 to calculate loss and gradient. When core 1 finished and updated the parameter value to p2, core 2 will update parameter based on p2 instead of p1 even though core 2 uses parameter p1 to calculate their loss and gradient! Also, they use exactly the same data from dataloader in each iteration to calculate loss and gradient (cannot be fixed easily by randomly sampling data because each core also uses the same random seed).</p>

username: JuanFMontesinos
cooked: <p>I would say that there is no such a way already implemented.<br>
The tool you are looking for is DataParallel but that’s implemented only for GPU parallelization.</p>
<p>You can try to dig into that code to see if you can manually emulate it for CPU.<br>
I think the standard pipeline is computing a forward-backward, average gradients and use optim.step()</p>
I want to print the number of images for each class for training

username: sampa
cooked: <p>I want to print the no. of images for each class</p>

username: ptrblck
cooked: <p>We would need some more information about what’s not working.<br>
E.g. how are you loading the data, how did you store it, and what kind of data are you trying to use?</p>

username: sampa
cooked: <p>Image type of data using data loader</p>
Re-arrange MNIST into two classes

username: rahulvigneswaran
cooked: <p>I want to re-arrange the MNIST into to 2 classes - odd numbers in one class and even numbers in another class, then use this rearranged dataset for train and test. How to do it?</p>

username: ptrblck
cooked: <p>You could try to adapt <a href="https://discuss.pytorch.org/t/how-to-use-one-class-of-number-in-mnist/26276/2">this code</a> for your use case.</p>

username: rahulvigneswaran
cooked: <p>I will try it and get back.</p>

username: rahulvigneswaran
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> The following works for me. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">for i in range(10):
  idx = (dataset.targets==i)
  if (i == 0) or ((i % 2) == 0): dataset.targets[idx] = 0

  else: dataset.targets[idx] = 1
</code></pre>
Auc stay at 0.5, not change with iterations increase

username: cil123
cooked: <p>I’m new to Pytorch. I have a sparse dataset in libsvm format. I used a simple NN model for binary classification. In training procedure, AUC of my model kept at 0.5, which means random selecting. I don’t know what’s the problem(s). I’m not sure if it is because the network is too simple.</p>
<p>My features are already scaled to 0 - 1.</p>
<p>My code:</p>
<p>import torch<br>
import sklearn.datasets<br>
import numpy as np</p>
<p>import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>data = sklearn.datasets.load_svmlight_file(‘train.libsvm’)<br>
coo = data[0].tocoo()<br>
y = data[1]</p>
<p>indices = np.vstack((coo.row, coo.col))<br>
values = coo.data</p>
<p>i = torch.LongTensor(indices)<br>
v = torch.FloatTensor(values)<br>
shape = coo.shape<br>
x_data = torch.sparse.FloatTensor(i, v, torch.Size(shape))<br>
y_data = torch.from_numpy(y).type(torch.LongTensor)</p>
<p><span class="hashtag">#our</span> class must extend nn.Module<br>
class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net,self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(393216,1024)<br>
self.fc2 = nn.Linear(1024,2)</p>
<pre><code>def forward(self,x):
    x = self.fc1(x)
    x = F.tanh(x)
    x = self.fc2(x)
</code></pre>
<h1>x = F.relu(x)</h1>
<h1>x = self.fc3(x)</h1>
<pre><code>    return x

#This function takes an input and predicts the class, (0 or 1)
def predict(self,x):
    #Apply softmax to output.
    pred = F.softmax(self.forward(x))
    ans = []
    #Pick the class with maximum weight
    for t in pred:
        if t[0]&gt;t[1]:
            ans.append(0)
        else:
            ans.append(1)
    return torch.tensor(ans)
</code></pre>
<p>model = Net()<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, weight_decay= 1e-6, momentum = 0.9, nesterov = True)</p>
<p><span class="hashtag">#Number</span> of epochs<br>
epochs = 100<br>
<span class="hashtag">#List</span> to store losses<br>
losses = []<br>
for i in range(epochs):<br>
y_pred = model.forward(x_data)<br>
print(y_pred)<br>
loss = criterion(y_pred,y_data)<br>
losses.append(loss.item())</p>
<pre><code>#Clear the previous gradients
optimizer.zero_grad()
#Compute gradients
loss.backward()
#Adjust weights
optimizer.step()

from sklearn.metrics import roc_auc_score
print(roc_auc_score(y_data, model.predict(x_data)))</code></pre>
Obtain mask from F.dropout

username: JakeStevens
cooked: <p>Is it possible to obtain the specific mask used in F.dropout explicitly? Originally, I tried to simply check what became zero after dropout, but this will clearly fail when there are zeros in the original input that are not dropped out.</p>

username: ptrblck
cooked: <p>I don’t think it’s currently possible to get the mask from <code>F.dropout</code>.<br>
If you need it for debugging reasons, you could write a custom <code>nn.Module</code> and apply your own mask on the input.</p>
Implementing half-way branch in the network

username: MrRobot
cooked: <h1><a name="question-1" class="anchor" href="#question-1"></a>Question</h1>
<p>I would like to implement a network architecture similar to Inception shown below. They are similar in the sense that there are multiple half-way branches (the part within in blue block).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/077e4cb3ddb72d8a33c67b94565ec37530bf06ca.png" data-download-href="https://discuss.pytorch.org/uploads/default/077e4cb3ddb72d8a33c67b94565ec37530bf06ca" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/077e4cb3ddb72d8a33c67b94565ec37530bf06ca_2_690x250.png" alt="image" data-base62-sha1="14hW9AU5UyCfrpmNjnoF0IYALa2" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/077e4cb3ddb72d8a33c67b94565ec37530bf06ca_2_690x250.png, https://discuss.pytorch.org/uploads/default/original/3X/0/7/077e4cb3ddb72d8a33c67b94565ec37530bf06ca.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/7/077e4cb3ddb72d8a33c67b94565ec37530bf06ca.png 2x" data-dominant-color="EBEEF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">828×301 61 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In order to implement this architecture, I could write those <code>HalfWayBranch</code> within <code>MainBranch</code>. However, I would like different modules to be least entangled together.</p>
<p>Then I have</p>
<pre data-code-wrap="python"><code class="lang-python">class MainBranch(nn.Module):
# ...

class HalfWayBranch(nn.Module):
# ...
</code></pre>
<p>Now I have two questions</p>
<ul>
<li>How do I write my training loop?</li>
<li>Could the loss be backproped correctly?</li>
</ul>

username: ptrblck
cooked: <p>If I understand your use case correctly, you would like to add these auxiliary modules into your model.<br>
I would recommend to write a main module and initialize the base model as well as these aux modules inside the <code>__init__</code>.<br>
Then in the <code>forward</code> you could simply use an intermediate output, feed it to the next module in your base model as well as one of the aux modules, and return all necessary outputs at the end. This would be similar to the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py" rel="nofollow noopener">Inception implementation</a>.</p>
What is leaf node

username: dali_wang
cooked: <pre><code class="lang-auto">def main():
    #order2 - MmBackward
    A = torch.tensor([1.,2,3,4,5,6],requires_grad=True).reshape(2,3)
    B = torch.tensor([1.,2,
                      3,4,
                      5,6],requires_grad=True).reshape(3,2)

    b = torch.tensor([1.,2,3,4],requires_grad=True).reshape(2,2)

    #A.retain_grad()
    #B.retain_grad()
    #b.retain_grad()

    z_ = A@B
    z = z_ +b
    z[0][0].backward()

    print('A.grad ',A.grad)#None
    print('B.grad ',B.grad)#None 
    print('z_.grad ',z_.grad)#none
    print('z.grad ',z.grad)#none

</code></pre>
<p>why A and B are both None?? Aren’t they  the leaf node??  If not,  What is a leaf node?<br>
pytorch version = 1.2</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751/2">this post</a> for more information.</p>
<p>PS: I enjoy the friendly discussion style in this forum and would like to keep it this way, thus editing your topic name.</p>
Data type in Pytorch C++ API

username: Taehun_Kim
cooked: <p>Hi,</p>
<p>I think the tensor data type is a little bit weird in C++.<br>
I made a tensor variable which is kFloat32 and I thought that it was 4 bytes.<br>
However, when I check the size of it with sizeof(), it says that it is 8 bytes.</p>
<p>And I’ve tried other types such as kFloat64, kInt8, kInt16 and so forth.<br>
But the sizeof() always says that the variable is 8 bytes.</p>
<p>Is tensor variable always 8 bytes in c++? or did I do something wrong?</p>

username: albanD
cooked: <p>A Tensor is a Tensor in some sense. So I am not surprised they all have the same size.<br>
If you’re looking for the size of each element in the Tensor, you have a special function for that. It is <code>.element_size()</code> if I recall correctly.</p>

username: Taehun_Kim
cooked: <p>With <code>.element_size()</code>,  I can see the expected size of variables.<br>
Thanks a lot.</p>
Trainig loss is decreasing but testing loss is not

username: WeiQin_Chuah
cooked: <p>I am training a network that performs stereo depth estimation from a stereo pair. The training loss of my network is decreasing as expected but my testing loss and errors are increasing.</p>
<p>Is this considered as overfitting? I have included random cropping and normalizing my input images. Also, I included weight decay of 0.01 and my learning rate is 0.0005. I am running out of idea what could possibly be wrong. My network has a total of 4120322 parameters and all are convolutional layers with batch norms and relu except the last layer.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99f085e15406dd60182de3cdc7b9d12bfd1a906c.png" alt="image" data-base62-sha1="lXOn0mZoYKvaNcig1yUuEesy4ig" width="647" height="486"></p>

username: JuliousHurtado
cooked: <p>A really good way to observe if your network is overfitting, change your validation dataset for you train dataset and it should obtain almost perfect accuracy.</p>
<p>Another experiment that you could try, is changing the Validation dataset to a random dataset, and verify if it changes something, maybe is something wrong with the generation of the validation images (one must make sure the pre-processing or creation of train and val images are the same)</p>
<p>I hope this could help.</p>

username: WeiQin_Chuah
cooked: <p>Regarding the images, does the image dimension matters? I am using <code>[3,256,512]</code> for training and <code>[3,540,960]</code> for testing/validation.</p>

username: cyanM
cooked: <p>You should preprocess images in the same way  (except for data enhancement operations) , such as resize and normalize. Not only the dimensions of the input data will affect the results, but also color channels’ order (such as RGB and BGR,  but this may not affect for stereo depth estimation).</p>
M620 GPU cuda not available

username: Cameron_McWilliams
cooked: <p>NVIDIA’s website says cuda is enabled on M620 GPU, but i have not been able to get torch.cuda.is_available() to return True. I notice that when I try download pytorch using: ‘conda install pytorch torchvision cudatoolkit=10.1 -c pytorch’ the build of pytorch that is downloaded is py3.7_cpu_0 which I think may be the problem?</p>
<p>I am using windows 10 and python 3.7</p>

username: smth
cooked: <p>Maybe your NVIDIA driver needs to be updated.</p>

username: Cameron_McWilliams
cooked: <p>I updated them according to this page (<a href="https://www.nvidia.com/content/DriverDownload-March2009/confirmation.php?url=%2FWindows%2FQuadro_Certified%2F440.97%2F440.97-quadro-desktop-notebook-win10-64bit-international-whql.exe&amp;lang=us&amp;type=Quadro&amp;aliId=eyJpIjoiVzVPKzQ1VkJ6VGpHM3pxVSIsInQiOiJWdUpuUFZcL2Q2SkZUZ21FbFNTM0l6dz09In0%253D" rel="nofollow noopener">https://www.nvidia.com/content/DriverDownload-March2009/confirmation.php?url=%2FWindows%2FQuadro_Certified%2F440.97%2F440.97-quadro-desktop-notebook-win10-64bit-international-whql.exe&amp;lang=us&amp;type=Quadro&amp;aliId=eyJpIjoiVzVPKzQ1VkJ6VGpHM3pxVSIsInQiOiJWdUpuUFZcL2Q2SkZUZ21FbFNTM0l6dz09In0%3D</a>)</p>
<p>but still no luck getting cuda enabled unfortunately. Wondering why my install of pytorch defaults to a cpu version?</p>

username: ptrblck
cooked: <p>Is <code>nvidia-smi</code> detecting the card correctly and shows the right driver version?</p>

username: Cameron_McWilliams
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74471d5d21267efa2d27e26f28f66964f41e96be.png" alt="image" data-base62-sha1="gADKrC30wry1w0Am0utg3n4XGV0" width="274" height="204"></p>

username: Cameron_McWilliams
cooked: <p>I think it may be seeing my first GPU not my second one?</p>

username: Cameron_McWilliams
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5305edf11bab20fcd3ea55bfc61f269b3c5d443.png" alt="image" data-base62-sha1="nzkpNDjSjTml90w3n9KVy1Hr99V" width="676" height="144"></p>
<p>I updated the drivers recently</p>
How to jointly embed multi-dimensional discrete features?

username: Ahmed_m
cooked: <p>Hi,</p>
<p>I am trying to use embedding layers to learn continuous representations for discrete input features. Each of these discrete features is multidimensional. That is, one sample in the input batch contains two integer items for example. Is there a possibility to use one embedding layer whose continuous representation dictionary can be accessed using the two integer values?</p>
<p>I have tried to use two separated embedding layers. But, unfortunately, I get an issue of “RuntimeError: CUDA error: device-side assert triggered”</p>
About RandomResizedCrop(224)

username: Omar2017
cooked: <p>Why we use random cropping for the image in deep learning, is it one of data augmentation? What if an important portion of the image is cropped?</p>

username: ptrblck
cooked: <p>Yes, it’s a data augmentation method.<br>
Often your crop size is not much smaller than the original image size, e.g. you are cropping <code>224x224</code> patches out of a <code>256x256</code> image.<br>
However, some techniques (IIRC in Inception) use larger differences between the crop size and original size.</p>
How to propely unsort an unpacked sequence to get hidden states also

username: Ameen_Ali
cooked: <p>In my network, I’m using packing for variable-length sequence inputs for the gru.<br>
please see code below<br>
. I still have my indices from sorting (variable ‘order’)<br>
I’m doing the unpacking and unsorting as in the following code :</p>
<pre><code class="lang-auto">  new_s , new_s_lengths = nn.utils.rnn.pad_packed_sequence(s) # s is the PackedSequence
    output = unscramble(new_s , new_s_lengths, order, batch_size)
</code></pre>
<p>The function <code>unscramble</code> does the unsorting stuff and is defined as follows</p>
<pre><code class="lang-auto">def unscramble(output, lengths, original_indices, batch_size):
"""
Takes the output from the model, the lengths, and original_indices, and batch size.
Unscrambles the data, which had been sorted to make pack_padded_sequence work. 
Returns the unsscrambled and unpadded outputs. 
"""
final_ids = (Variable(torch.from_numpy(np.array(lengths) - 1))).view(-1,1).expand(output.size(1),output.size(2)).unsqueeze(0)
if cuda:
    final_ids = final_ids.to('cuda')
final_outputs = output.cpu().gather(0, final_ids.cpu()).squeeze()
unscrambled_outputs = final_outputs[original_indices]
return unscrambled_outputs
</code></pre>
<p>However this is not my problem, the problem this function returns only the output of the last hidden state of the GRU, let’s say I have defined a GRU in the following way:</p>
<pre><code class="lang-auto">self.lstm = nn.GRU(300 , 400 ,  1)
</code></pre>
<p>and having an input of shape : <code>[48 , 128 , 300]</code> (48 is the longest sequence and 128 is the batch size) so after packing the sequence and unpacking and calling unscramble function which I have defined I will get a new tensor of shape : <code>[128 , 400]</code> (the last hidden state with the original indices as before sorting by lengths).<br>
my question here is how can I get the output of all of the hidden states? (in my example a tensor of shape <code>[128 , 48 , 400]</code> .<br>
Thanks.</p>
Can model with batch forward update predict single example?

username: dokato
cooked: <p>Hi PyTorch users,</p>
<p>I’m still quite new to pytorch, but I’ve spent on this problem sometime already.</p>
<p>So I’ve got this demo model of LSTM which works on batches.</p>
<pre><code class="lang-python">
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, batch_size, output_dim=1, num_layers=2):
        super(LSTM, self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.batch_size = batch_size
        self.num_layers = num_layers

        self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)

        self.linear = nn.Linear(self.hidden_dim, output_dim)

    def init_hidden(self):
        return (torch.zeros(self.num_layers, self.batch_size, self.hidden_dim),
                torch.zeros(self.num_layers, self.batch_size, self.hidden_dim))

    def forward(self, input):
        lstm_out, self.hidden = self.lstm(input.view(-1, self.batch_size, self.input_dim))
        y_pred = self.linear(lstm_out[-1].view(b_size, -1))
        return y_pred.view(-1)
</code></pre>
<p>I tried it on some sine signal and looks like it learns okay.</p>
<p>But I give as input tensor batches of <em>batch_size</em> length. Now I was wondering how to achieve similar to keras method <em>model.predict(X_test)</em> that I can feed to the LSTM model only single example. Any simple solution?</p>

username: ptrblck
cooked: <p>You could just set the batch size to 1 and get a single sample output.</p>

username: BramVanroy
cooked: <p><code>.view(-1)</code> when returning seems superfluous no?</p>

username: dokato
cooked: <p>But then I’d have to train using stochastic gradient descent, right? By showing one training example at a time? What I want to achieve is: to train on batches and predict a single example.</p>
<p>I think that key to my concern is a lack of proper understanding of what <code>init_hidden</code> does, as it includes in return dimensions <code>self.batch_size</code>, which are not needed for me at a prediction stage.</p>

username: matthias.l
cooked: <p>This sounds like a standard problem: Writing a model for single samples. But on training mini-batches should be the default input for the model.<br>
Is there something like <code>myTrainModel = torch.convertForBatchInputs(myInferenceModel)</code></p>
CUDA out of Memory if GPU is not "warmed up"?

username: matohak
cooked: <p>I have a simple script:</p>
<pre><code class="lang-auto">import torch
LOSS_WEIGHTS = [1,2,3]
LOSS_WEIGHTS = torch.Tensor(LOSS_WEIGHTS)
LOSS_WEIGHTS = LOSS_WEIGHTS.to(0)
</code></pre>
<p>If I start the script while the computer is idle, I often get “CUDA error: out of memory”<br>
The error somehow always goes away after I repeatedly relaunch the script several times. Does anyone know what can I do to prevent this error? Am I suppose to initialise my cuda device before starting the script?</p>
<p>pytorch 1.2.0 (Tried several versions)<br>
cuda 10.1 (Also tried cuda 9)<br>
Nvidia Driver 430<br>
Hardware: 1 x GTX 1070<br>
Ubuntu 18.04</p>
from torch._C import * ImportError: DLL load failed: 找不到指定的模块。

username: LinpengPan
cooked: <p>from torch._C import *<br>
ImportError: DLL load failed: 找不到指定的模块。</p>

username: albanD
cooked: <p>Hi,</p>
<p>This usually happens if you try to import pytorch from the root folder of the repo. It confuses python import system as there is a folder called torch there.<br>
You should try to run python from another folder.</p>
Model weight cannot update when gradients are computed from distributed machines

username: jia
cooked: <p>I am trying to train the global model by receiving gradients from distributed devices. But, the model’s weight is not updated. I have no idea…Thanks in advance.</p>
<p>mod = Model()<br>
cp = torch.load(rep)<br>
mod.load_state_dict(cp[’ model_state_dic’ ])<br>
optimizer = optim.Adam(mod.parameters(), lr=0.001)<br>
optimizer.load_state_dict(cp[‘optimizer_state_dict’])<br>
G = ComputeGradient()<br>
mod.train()<br>
for u in range(len(G)):<br>
optimizer.zero_grad()<br>
loss = Variable(G[u], requires_grad=True)</p>
<pre><code>loss.backward()
optimizer.step()</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The gradient history is based on Variables. So if you make the loss a Variable at the last moment, the only history you have is empty.<br>
At the moment the autograd engine only works with Tensors that are within the current process. Adding support for distributed is currently being done. You can find the PR with all the doc <a href="https://github.com/pytorch/pytorch/pull/29276" rel="nofollow noopener">here</a>.</p>

username: jia
cooked: <p>Hi,<br>
Thanks for the reply. I am not sure if i totally got it. Instead of PR, is there any cheap trick to solve this problem? Thanks in advance.</p>

username: albanD
cooked: <p>If there was, we wouldn’t have made a whole package to support this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
The PR I linked corresponds to the doc for functions that are already in pytorch master.</p>
<p>You can try and emulate a similar thing for your particular use case. But that would require calling backward yourself on each node one after the other, after having transfered the gradients for the shared Tensors.<br>
This can be tricky to do.</p>
Grid_sample() returns ZERO gradient with misordered grid

username: halfjoe
cooked: <p>Hi, thanks first for the amazing tools. I am currently trying to use grid_sample() to sample colors from a texture image to be the color of vertices from a 3D .obj model (part of a rendering process), and then the color is used to compute loss and gradients are expected to back-propagate to the texture image. However, in the experiment gradients on the texture image are all zero.</p>
<p>To find out the reason, I replace the grid with an “ordered” one, like those generated by affine_grid(), then the gradients are non-zero. Thus, I am wondering if this is because the original grid is defined by the UV coordinates corresponding to each vertice in the 3D model, and thus there is no regularity in it. I would appreciate if anyone could tell me whether there are certain constraints on defining the grid, or if there may be other reasons that make the grad zero.</p>
<p>Many thanks!!</p>

username: albanD
cooked: <p>Hi,<br>
Would you have a small code sample that reproduces this behavior please?</p>

username: halfjoe
cooked: <p>It turns out the gradients are not totally zeros. It’s just updating in a very very low speed. This is a stupid question and I will delete it soon. Thanks and sorry for your time!</p>
Loss Function for Multi-class with probabilities as output

username: marcio
cooked: <p>Hello!<br>
I’m working on a Multi-class model where my target is a one-hot encoded vector of size C for each input sample. Since the output should be a vector of probabilities with dimension C, I’m having trouble finding what combination of <strong>output layer activation</strong> and <strong>Loss Function</strong> to use.</p>
<p>Based on what I’ve read so far, vanilla <code>nn.NLLLoss</code> and <code>nn.CrossEntropyLoss</code> can’t be used since the output is a label. My guess is that I would either need to tweak these loss functions to use one-hot encoded target or write my own loss. I’m somewhat confused on how to proceed form here since I don’t know how each of these options are going to impact the final model performance.</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>In the docs, we can see that <code>nn.CrossEntropyLoss</code> is the combination of <code>nn.LogSoftmax</code> (responsible for converting the output of your network to a probability distribution) and <code>nn.NLLLoss</code>. Therefore, if you want to use <code>nn.CrossEntropyLoss</code>, you do not need any output layer activation as it is included in the loss function. If you prefer to use <code>nn.NLLLoss</code>, then a <code>nn.LogSoftmax</code> would be a good choice.</p>
<p>You can convert your one-hot encoded target vector to a label in order to use either loss function as above. Here is an example for cross-entropy:</p>
<pre><code class="lang-auto">def one_hot_ce_loss(outputs, targets):
    criterion = nn.CrossEntropyLoss()
    _, labels = torch.max(targets, dim=1)
    return criterion(outputs, labels)
</code></pre>
<pre><code class="lang-auto">targets = torch.tensor([[0, 0, 1], [1, 0, 0], [0, 0, 1], [0, 1, 0]], dtype=torch.int32)
outputs = torch.rand(size=(4, 3), dtype=torch.float32)
loss = one_hot_ce_loss(outputs, targets)
print(loss)
</code></pre>
<p>The example supposes a batch size of 4 and the number of possible classes © as 3.</p>
<p>Hope this helps!</p>

username: marcio
cooked: <p><a class="mention" href="/u/beaupreda">@beaupreda</a>,</p>
<p>Big thanks for commenting on my thread, your reply solved my problem!<br>
I forgot that <code>nn.CrossEntropyLoss</code> already performs both <code>nn.LogSoftmax</code> and <code>nn.NLLLoss</code>, so I was adding an extra logit transformation to the output layer. Plus the snippet your provided to transform one-hot labels into scalars for the Loss function was key for me.</p>
<p><span class="mention">@edit</span>: Added more details to reply.</p>
Problems when implementing pooling layer

username: G.M
cooked: <p>I’m trying to implement <a href="https://arxiv.org/abs/1301.3557" rel="nofollow noopener">Stochastic pooling</a> with pytorch. I’m implementing Stochastic max pooling and somehow my code doesn’t compute the gradients correctly and won’t work when propagating backward. The code can give the correct output but the gradients seem to be removed somewhere.</p>
<pre><code class="lang-auto">from torch.distributions.one_hot_categorical import OneHotCategorical as ohc

class RandomPool2d(nn.Module):
    def __init__(self, kernal_size=2):
        super().__init__()
        self.ksize = tuple(kernal_size) if type(kernal_size) == int else kernal_size
        self.p = self.ksize[0] * self.ksize[1]#self.p is the area of kernel size

    def forward(self, inp: tc.Tensor):
        if self.training:
            
            bsz, chn, h, w = inp.size()
            inp: tc.Tensor = tcf.unfold(inp, self.ksize, stride=self.ksize)
            inp = inp.view(bsz, chn, self.p, -1).transpose_(2, 3).flatten(0, 2)
            # mask=ohc(inp.softmax(1)).sample().to(dtype=tc.uint8) #this is just another way to create a one-hot represation of the indexes drawn
            mask: tc.Tensor = tc.zeros_like(inp, dtype=tc.uint8)
            mask.scatter_(1, tc.multinomial(inp.softmax(1), 1), 1) #generate a one-hot  representation of the index drawn
            inp = inp.masked_select(mask)# use the indexs to choose from orginial tensor
            inp = inp.view(bsz, chn, -1, 1).transpose_(2, 3)
            return inp.view(bsz, chn, h // self.ksize[0], w // self.ksize[1])
        else:
            return tcf.max_pool2d(inp, self.ksize)

</code></pre>
<p>The usage of this code is exactly like other 2d pooling layers except the stride are always equal to kernel size.<br>
I am quite new with pytorch and is unable to find out why is the gradients not computed correctly <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
Can anyone help me solve this problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What do you mean by “gradients are not computed correctly”? What do you check to see that? What do you expect?</p>

username: G.M
cooked: <p>Thx 4 ur reply. the behavior was that my model wasn’t learning anything, the loss wasn’t changing a lot and the accuracy(measured per epoch) wasn’t even changing. So that’s y I thought the gradients wasn’t computed correctly.<br>
However, I solved it already, it’s due to a problem in other part of my model; in fact, it was because I used too many Stochastic pooling and that the model was learning very slowly.</p>
<p>Thank you so much:smiley:</p>