akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Freeze model weights only on the second call of layer

batrobin: <p>I’m currently implementing a network architecture where I have to pass my data through a certain layer for multiple times.<br>
The network can be simplified as:</p>
<pre><code class="lang-auto">Input -&gt;  Layer A -&gt; Layer B -&gt; Layer A -&gt; Output
</code></pre>
<p>However I dont want Layer A to be updated by the gradient, i.e. freezed, on the latter call, while still be updated by the first call. Is this possible with pytorch? I’ve tried to come up with something using no_grad, requires_grad or detach() but I couldn’t.</p>
<p>A hacky workaround Im thinking of is to do gradient scaling during the backward pass:</p>
<pre><code class="lang-auto">Input Gradients &lt;- Layer A &lt;- Layer B &lt;- Scale up gradient by λ &lt;- Layer A &lt;- Scale down gradient by λ &lt;- Output Gradients
</code></pre>
<p>Is there a cleaner implementation? Thanks.</p>

ptrblck: <p>Wouldn’t <code>torch.no_grad()</code> work?</p>
<pre><code class="lang-python">x = layerA(x)
x = layerB(x)
with torch.no_grad():
    x = layerA(x)

loss = criterion(x, target)
</code></pre>
<p>This would make sure to not track the second call on <code>layerA</code>.</p>

batrobin: <p>If this is the case, I may have misunderstood how no_grad works. I was under the impression that with no_grad, the gradient would not be computed for the second call of layerA and thus the gradient cannot be propagated to layerB and also the first call of LayerA. Can you confirm this is not the case? Thanks for the quick reply.</p>

ptrblck: <p>No, sorry, I’m wrong and the output won’t require gradients.</p>
<p>In that case, you could detach the parameters of <code>layerA</code> and use the functional API, which should work:</p>
<pre><code class="lang-python">torch.manual_seed(2809)

layerA = nn.Linear(10, 10)
layerB = nn.Linear(10, 10)

x = torch.randn(1, 10)

out = layerA(x)
out = layerB(out)

out = F.linear(out, layerA.weight.detach(), layerA.bias.detach())

out.mean().backward()

print(layerA.weight.grad)
</code></pre>

batrobin: <p>In my case, my layerA is much more complicated than a nn.Linear layer, so simply detaching weightings would not be practical. I am thinking of deepcopying layerA instead, however im not familiar with the interactions of it with pytorch. Would you know if this will work?</p>
<pre><code class="lang-auto">from copy import deepcopy
import torch

x = torch.randn(1, 10)
layerA = ...
layerB = ...
layerA_2 = deepcopy(layerA)

out = layerA(x)
out = layerB(out)
out = layerA_2(out)

out.mean().backward()


</code></pre>

ptrblck: <p>Yes, your approach should also work.<br>
I compared it to the functional API run and get the same gradients for the copied module.</p>
Convert Grayscale numpy image shape (H x W x C) into a torch image shape (C x H x W)

minmax: <p>Hey I have a gray scale numpy ndarray of shape 224,224 [ Which I assume is in (H x W x C ) format. However I need to convert this into (C x H x W) format. When I printed the shape it showed only the H x W. Where is the channel, or is it not shown because the image is in gray scale?</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji only-emoji" alt=":smiley:"></p>

simaiden: <p>Use</p>
<pre><code class="lang-auto">numpy_array = ... # your numpy array image with shape (224,224)
tensor = torch.from_numpy(numpy_array) #torch.Size([224, 224])
tensor = tensor.unsqueeze(dim=0) # torch.Size([1, 224, 224])
</code></pre>

minmax: <p>Thank you so much <a class="mention" href="/u/simaiden">@simaiden</a> It worked.</p>

simaiden: <p>Also you can use transforms.ToTensor() <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor" rel="nofollow noopener">docs</a>, this  transforms a PIL or numpy image in range [0,255] to tensor in range [0,1]</p>
<pre><code class="lang-auto">from torchvision.transforms import ToTensor
import torch
numpy_array = ... # your numpy array image with shape (224,224)
tensor = ToTensor()(numpy_array) #torch.Size([1, 224, 224])

</code></pre>

minmax: <p>Hey is transforms.ToTensor() equivalent to numpy_array/255 . Does this normalize the values in the range [0,1].<br>
P.S thanks for the docs link <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"></p>

simaiden: <p>According to the docs:</p>
<blockquote>
<p>Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0] if the PIL Image belongs to one of the modes (L, LA, P, I, F, RGB, YCbCr, RGBA, CMYK, 1) or <strong>if the numpy.ndarray has dtype = np.uint8</strong></p>
</blockquote>
<p>You could see the difference doing:</p>
<pre><code class="lang-auto">numpy_array = np.ones((224,224))
numpy_array_uint8 = np.ones((224,224,1),dtype=np.uint8)
tensor = ToTensor()(numpy_array)
tensor_uint8 = ToTensor()(numpy_array_uint8)
print(tensor)
print(tensor_uint8)
</code></pre>
<p>So only is equivalent if the numpy array has <code>dtype = np.uint8</code></p>
Different behavior between Python and C++

Daewoong.Ahn: <p>I was uploaded on <a href="https://github.com/pytorch/pytorch/issues/35606" class="inline-onebox" rel="noopener nofollow ugc">Could not find any similar ops to "foo..." in the `Libtorch` · Issue #35606 · pytorch/pytorch · GitHub</a></p>
<p>Error Trace</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'torch::jit::script::ErrorReport'
  what():  
Unknown builtin op: tomo::morph_pool.
Could not find any similar ops to tomo::morph_pool. This op may not exist or may not be currently supported in TorchScript.

....

Serialized   File "code/__torch__/torch/nn/modules/module/___torch_mangle_162.py", line 9
    argument_1: Tensor) -&gt; Tensor:
    _0 = self.morph
    x_min_morph = ops.tomo.morph_pool(argument_1, _0)
                  ~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    x, _1 = torch.min(x_min_morph, 2, False)
    x_max_morph = ops.tomo.morph_pool(x, _0)

[1]    57316 abort (core dumped)  ./forward 
</code></pre>
<p>But In python, working fine as I expected.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/4649a0b71c6ec99eee729d0880d21c05b8b2ff23.png" alt="image" data-base62-sha1="a1NawwOw7KpgD9JKvffw94sdvr5" width="557" height="279"></p>
<p>I would like a hint as to what part to look at.</p>
Loading saved model

PURAV_SHAH: <p>Hi,</p>
<p>I have used an alexnet model that is available in pytorch models  and modified the last layer of the alexnet to suit my requirements and have trained this model and saved the trained weights.<br>
Now whenever i try to load the saved weights i keep getting the error given below.</p>
<p>Some help and insight on this would be appreciated.</p>
<p>Thanks.</p>
<pre><code class="lang-auto">torch.save(alexNet.state_dict(),'./cifar_net.pth')
new_models=models.alexnet()
new_models.classifier[6]=nn.Sequential(nn.Linear(4096,10))
new_models=torch.load('./cifar_net.pth')
new_models.eval()
</code></pre>
<p>and the output i get is-</p>
<pre><code class="lang-auto">AttributeError: 'collections.OrderedDict' object has no attribute 'eval'
</code></pre>
<p>I also get the same error when i try to load the model using-</p>
<pre><code class="lang-auto">new_models.load_state_dict(torch.load('./cifar_net.pth'))
</code></pre>

simaiden: <p>You should’t get that error using <code>new_models.load_state_dict(torch.load('./cifar_net.pth'))</code>, because this is the right way to load a model, not <code>new_models=torch.load('./cifar_net.pth')</code>, beacuse this doesn’t load the weights to the model, but only loading the dict on the <code>new_model</code> variable</p>
Loss is nan for CrossEntropyLoss

thingsofleon: <p>Very new to this. Trying to get this to work. Pretty sure I am making an easy mistake, I just can’t find it. Have spent hours today trying to figure it out. So now I am humbly asking for help.</p>
<p>Data: 250 Signals consisting of 30,000 samples (250,30000)<br>
Locations: 10 different locations<br>
NN: 2 Linear Layers with the first going through RELU.</p>
<p>As far as I know all of my dimensions are correct.</p>
<p>Link to Notebook: <a href="https://drive.google.com/open?id=12dOdHRdJR87pVUOQ8h9_XtM4AucnHy_W" rel="nofollow noopener">https://drive.google.com/open?id=12dOdHRdJR87pVUOQ8h9_XtM4AucnHy_W</a></p>
<p>Any help would be amazing.</p>

charan_Vjy: <p>It seems that you have randomized the inputs and the labels. The model is getting confused which is expected behaviour.</p>

thingsofleon: <p>This is just dumb made up data. Was hoping to not get a good model, but one that would run. Is this a bad assumption?</p>

dhecloud: <p>Your loss is exploding. reduce your learning rate</p>

charan_Vjy: <p>Contrary to my initial assumption, you should try reducing the learning rate. Loss should not be as high as Nan. Having said that, you are mapping non-onto functions as both the inputs and outputs are randomized. There is a high chance that you should not be able to learn anything even if you reduce the learning rate.</p>

thingsofleon: <p>Thanks so much for the input. I will give it a try and let you know!</p>

thingsofleon: <p>OK. I have changed the dummy data.</p>
<ul>
<li>more examples, know 25000 instead of 250</li>
<li>calculate the locations first (this is random ints between 0-9)</li>
<li>calculate the signals with a different STD_DEV for each location<br>
There now should be a pattern between locations and signal since they have different distributions.</li>
</ul>
<p>I am also running the entire training sequence through multiple times using different learning_rates. 1e-4 is the first one that doesn’t make the number explode or decrease (also seems to be the best).</p>
<p>Am I on the right track here?<br>
Am I calculating the loss correctly?<br>
When do I know that the model is good? What does the loss value mean? Do I need to calculate accuracy to know if the model is good?</p>

charan_Vjy: <p>Now that you localized your distribution for a particular class, the training should be better. It is advisable for the loss to decrease every subsequent epoch (at least for the first few runs). Set a learning rate for which this would happen.</p>
<ul>
<li>It seems to me that you are computing the loss correctly. The network is a 2 layer MLP. Try playing around with the model architecture, a little more. Add in more layers, preferably <code>Conv1d</code> layers</li>
<li>As for knowing when to stop training, for a particular set of hyperparameters, the training loss would have converged (remains stagnant).</li>
<li>Ideally you should test your trained model on a validation set and choose the model that shows best accuracy on the validation set.</li>
</ul>
<p>All the best</p>

thingsofleon: <p>Really appreciate the help. Thank you!</p>
The scope of model.to('cuda')

Sriharsha-hatwar: <p>While reading a pytorch code there was comment where it said we have to once again move the model trained in the GPU to device using <code>.to('CUDA')</code>. So I was wondering the scope of <code>.to('CUDA)</code> ? Will it persist in the cuda memory until the program is terminated or the scope of function is terminated? what happens if we pass the <strong>model</strong> variable to another function? will it be affected?<br>
Is transferring the model to run in GPU just a one time activity?</p>

albanD: <p>Hi,</p>
<p>Yes it is a one time thing. And the weights will stay on the GPU until you delete the model, or move it back to the cpu().</p>
<p>This comment might be related to loading of a saved model maybe? With the saved model being on CPU.</p>
Skip a gradient computation?

pytorchnewbie: <p>Hi, so if I want to learn a matrix M, that will transform an image I into I’ that is ideally very close to the ground_truth image I_g, the forward pass is the following:</p>
<p>M = [1<em>scale 0 dx<br>
0 1</em>scale dy]</p>
<p>The only variables in M are scale, dx, dy</p>
<p>I*M    = I’<br>
loss = I_g - I’</p>
<p>note that this * is actually using grid_sample, and grid_affine but it doesn’t matter here</p>
<p>when I back prop the loss though the gradient will first go to M, and then to dx,dy, scale.<br>
I don’t want it to consider the entire matrix of M as a parameter to update, because for example the first row second column’s value is a fixed 0. For the moment though, its considering 0 to be updatable and I don’t want that. However, if I do M.detach() then dx,dy, and scale won’t be updated/learned…</p>
<p>Anyone know how to do this? Thanks!</p>

albanD: <p>Hi,</p>
<p>How do you create M exactly?<br>
If you make sure to recreate it at each forward based on your parameters, you should not have any issue.</p>

pytorchnewbie: <p>You’re right I just realized that the gradients in the a and e position sum together to the scale gradient. Thanks! But since I’m here I wanted to ask a related question - why is it that when the gradients are largest, the values are affected the least? I’m noticing that when I plot my gradients of x, the larger they are (they oscillate between super big and super small) yet the value of x doesn’t change at all… I’m using an Adam optimizer and was wondering if thats why? Not sure how Adam affects it but just a guess…</p>

albanD: <p>When you say the value doesn’t change at all, you mean after doing the optimizer step?<br>
If so yes, that might be due to Adam that regularizes a lot the trajectory if it is very noisy. You can try with SGD to see if you get the same behavior.</p>
Question about how the -m.log_prob() function in torch.distributions.bernoulli works?

paulsteven: <p>Hi,</p>
<p>I am trying to figure out what the -m.log_prob() function is actually doing when implementing Policy gradients.</p>
<p>I have stepped through the m.log_prob() function in the debugger <strong>many</strong> times - and fail to see how it is working. My understanding is that m.log_prob() calls</p>
<p>m.log_prob() &gt;&gt; which calls logits() &gt;&gt; when then calls cross_entropy_with_logits()</p>
<p>this is all fine, except that I cannot recreate the value being created in the logits function.</p>
<p>I get that logits() is just the log odds, so log( p /(1 −  p ) but when inputting the same value for p (generated by the net on line 1 below , probs = p) in into the equation for logits by hand, fail to recreate the same value produced by PyTorch’s logit function!</p>
<p>The reason for all of this being that I think this function is at the core of what is going on - and because Pytorch is so efficient-obscures “the magic” of what is actually going on under the hood in Vanilla Policy Gradients (aka REINFORCE).</p>
<p>specifically, I am referring to what is going on inside the <strong>-m.log_prob()</strong> function here:</p>
<ol>
<li>probs = policy_network(state)</li>
<li>Note that this is equivalent to what used to be called multinomial</li>
<li>m = Categorical(probs)</li>
<li>action = m.sample()</li>
<li>next_state, reward = env.step(action)</li>
<li>loss = <strong>-m.log_prob(action)</strong> * reward</li>
<li>loss.backward()</li>
</ol>
<p>this example is straight out of the pytorch documentation here: <a href="https://pytorch.org/docs/stable/distributions.html?highlight=reinforce" rel="nofollow noopener">https://pytorch.org/docs/stable/distributions.html?highlight=reinforce</a></p>

penguinshin: <p>Just checking in here - does it not just call .gather on the logits?</p>
Queue get function get stuck when using multiprocessing

cocoa_kang: <p>Hi everyone,<br>
I found that when getting Tensors from a multiprocessing queue, the program will be stuck randomly. I wrote a snippet to reproduce this problem:</p>
<pre><code class="lang-auto">import torch
import time
from torch.multiprocessing import set_start_method,Queue

try:
    set_start_method('spawn')
except RuntimeError:
    pass

def f(nproc,q,generate_device_list):
    while True:
        training_data_map = {
            "multiview_lumitexel_list":[torch.randn(50,24576,1,device=generate_device_list[i%len(generate_device_list)]) for i in range(8)],
            "rendered_slice_diff_gt_list":[torch.randn(50,24576,1,device=generate_device_list[i%len(generate_device_list)]) for i in range(8)],
            "rendered_slice_spec_gt_list":[torch.randn(50,384,1,device=generate_device_list[i%len(generate_device_list)]) for i in range(8)],
            "normal_label":torch.randn(50,3,device=generate_device_list[0]),
            "input_positions":torch.randn(50,3,device=generate_device_list[0]),
            "geometry_normal":torch.randn(50,3,device=generate_device_list[0])
        }
        # print("[PROCESS] before putting...")
        q.put(training_data_map)
        # print("[PROCESS] put one item")
        # time.sleep(0.5)

if __name__ == '__main__':
    torch.random.manual_seed(2333)
    q = Queue(100)
    generate_device_list = [torch.device("cuda:0"),torch.device("cuda:1"),torch.device("cuda:2"),torch.device("cuda:3")]
    training_device = torch.device("cuda:0")

    ctx = torch.multiprocessing.spawn(f, args=(q,generate_device_list), nprocs=1, join=False, daemon=True)
    time.sleep(30.0)
    print(q.qsize())
    counter = 0
    while True:
        print("[MAIN] before getting...")
        batch_data = q.get()

        input_positions = batch_data["input_positions"].to(training_device,copy=True)
        multiview_lumitexel_list = [a.to(training_device,copy=True) for a in batch_data["multiview_lumitexel_list"]]
        rendered_slice_diff_gt_list = [a.to(training_device,copy=True) for a in batch_data["rendered_slice_diff_gt_list"]]
        rendered_slice_spec_gt_list = [a.to(training_device,copy=True) for a in batch_data["rendered_slice_spec_gt_list"]]
        normal_label = batch_data["normal_label"].to(training_device,copy=True)
        geometry_normal = batch_data["geometry_normal"].to(training_device,copy=True)

        del batch_data
        print("[MAIN] got one item:{}".format(counter))
        counter+=1

    ctx.join()
</code></pre>
<p>However, this problem is not dedicated. I got stuck when counter equals 607 or 18901.<br>
My environment:<br>
Linux version 4.15.0-74-generic<br>
Ubuntu 16.04<br>
python3.6.7 or python 3.7.0<br>
pytorch 1.4.0</p>
<p>Can anyone help me? Many thanks before! <img src="https://discuss.pytorch.org/images/emoji/apple/dizzy_face.png?v=9" title=":dizzy_face:" class="emoji" alt=":dizzy_face:"></p>
How to pass context vector of Encorder to decorder without using attention

sgaur: <p>Hi<br>
I am going through several tutorials about seq2seq in pytorch but most of them are with using attention. I want to create simple seq2seq model.</p>
<p>I just need to know how to pass context vector into decorder.<br>
Working on english to French dataset.</p>
<p>Below is my code:</p>
<blockquote>
<p>class Encorder(nn.Module):<br>
def <strong>init</strong>(self,n_vocab,n_embed,hidden_size):</p>
<pre><code>    super(Encorder,self).__init__()
    
    self.hidden_size = hidden_size
    self.embed = nn.Embedding(n_vocab+1,n_embed)
    self.RNN = nn.RNN(n_embed,hidden_size,batch_first = True)

def forward(self,x):
    x = self.embed(x)
    x,hidden = self.RNN(x)
    assert x[:,-1,:].shape == hidden.squeeze(0).shape
    return x[:,-1,:]

def initHidden(self):
    return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>
</blockquote>
<p>For Decorder</p>
<blockquote>
<p>class decorder(nn.Module):<br>
def <strong>init</strong>(self,n_vocab,n_embed,hidden_size,output_size):<br>
super(decorder,self).<strong>init</strong>()</p>
<pre><code>    self.hidden_size = hidden_size
    self.embed = nn.Embedding(n_vocab+1,n_embed)
    self.RNN = nn.RNN(n_embed,hidden_size,batch_first = True)
    self.fc1 = nn.Linear(hidden_size,output_size)
    
    
def forward(self,x,hidden):
    #x = x.unsqueeze(0)
    x = self.embed(x)
    x,hidden = self.RNN(x,hidden)
    x = self.fc1(x)
    
    return x
def initHidden(self):
    return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>
</blockquote>
<p>Please let me know if some one need more info. I just want to make basic seq2seq (without attention)</p>
<p>Thanks in advance.<br>
Stay safe. !!</p>

vdw: <p>Without attention, the context vector is usually just the last hidden state – the most basic encoder-decoder setup. And you’re already return the last hidden state in your <code>forward()</code> method of your encoder. Your current code is working, right?</p>

sgaur: <p>I just need to find out how to put that last context vector into the initial state of Decorder.</p>
<p>Example::</p>
<blockquote>
<p>n_vocab = 3160<br>
n_embed = 16<br>
hidden_size = 16<br>
model = Encorder(n_vocab,n_embed,hidden_size)<br>
model(input_[0:3])</p>
</blockquote>
<p>Results are :<br>
tensor([[ 0.2726, -0.5882, -0.9438, -0.8380,  0.2866,  0.7537,  0.8387, -0.9315,<br>
-0.3624, -0.6164,  0.6472,  0.3157, -0.9453,  0.6535,  0.5850, -0.7729],<br>
[-0.1652, -0.9067, -0.7611,  0.2591,  0.0118, -0.1394,  0.8526, -0.1523,<br>
-0.7598, -0.3685, -0.5124, -0.3204,  0.2357,  0.2204,  0.5002, -0.6450],<br>
[-0.1652, -0.9067, -0.7611,  0.2591,  0.0118, -0.1394,  0.8526, -0.1523,<br>
-0.7598, -0.3685, -0.5124, -0.3204,  0.2357,  0.2204,  0.5002, -0.6450]],<br>
grad_fn=)</p>
<p>Now I got 3 hidden state for 3 sentences Now how to send those state into initial_state of decorder part/</p>

vdw: <p>Isn’t the basic Seq2Seq tutorial what you’re looking for; before they introduce the Attention Decoder?</p>

sgaur: <p>Yeah thats the same thing I am looking for but all the tutorials I got are just related to attention mechanis</p>
Cuda extension install without rebuilding

2CuteChiquita: <p>Hello,</p>
<p>How do I move a CUDA extension after it has been build?<br>
Right now I rebuild the extension every time i rerun the code but as the code inside the extension is unchanged this shouldn’t be necessary.</p>
<p>Is there an easy way to do this?</p>

albanD: <p>You mean when using the cpp extensions?<br>
If so, the build process is cached and only the first one should compile. All the other ones will detect that nothing changes and just use it. You can find these cached modules in <code>/tmp</code> on unix machines.</p>

2CuteChiquita: <p>Thanks for the response.</p>
<p>It is because I’m compiling it in my docker container, and so I don’t have any caching functionality.<br>
With Tensorflow I could built it in a build image but I seem to be unable to do the same with Pytorch.<br>
So I build it in my container each time. What I was wondering were whether I could save the output of the compilation to a shared folder and then copy it in during image build in the following builds.</p>
<p>Do you know whether this is possible?</p>

albanD: <p>You can specify the build directory if you want and share that directory between your docker images (see <a href="https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load_inline">doc</a>).</p>

2CuteChiquita: <p>Thanks, I wasn’t aware of that possibility.</p>
Linear_dynamic has some problems with qnnpack

huoge: <p>Hello everyone. I recently use the torch.quantization.quantize_dynamic(model, dtype=torch.qint8) to dynamic quantize my model. When use fbgemm（the default engine） to quantize the model, it works. However, when I change the engine from fbgemm to qnnpack, it has some problems. The way I use qnnpack is like this: model.qconfig = torch.quantization.get_default_qconfig(‘qnnpack’)<br>
torch.backends.quantized.engine = ‘qnnpack’.<br>
qnnpack causes the problem like that:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f19d7843b03616ae933ed2835f9d3e0a6ce1f2e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/f19d7843b03616ae933ed2835f9d3e0a6ce1f2e5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f19d7843b03616ae933ed2835f9d3e0a6ce1f2e5_2_690x106.png" alt="image" data-base62-sha1="ytqrLsTZpepZCJZmA49W4c99Gmx" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f19d7843b03616ae933ed2835f9d3e0a6ce1f2e5_2_690x106.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f19d7843b03616ae933ed2835f9d3e0a6ce1f2e5_2_1035x159.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f19d7843b03616ae933ed2835f9d3e0a6ce1f2e5.png 2x" data-dominant-color="3C1831"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1200×186 32.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Has anyone ever done any related work or met the same problem?<br>
by the way, the torch version is 1.5.0a0+b336deb</p>
<p>I’d appreciate if anybody can help me! Thanks in advance!</p>

jerryzh168: <p>cc <a class="mention" href="/u/supriyar">@supriyar</a> do you know?</p>

supriyar: <p>You don’t need to necessarily set the <code>qconfig</code> in this case to qnnpack. Please try using <code>default_dynamic_qconfig</code> instead and see if that helps to solve the issue. Also make sure if your build has qnnpack enabled.</p>
<p>I was able to get this example model to work with qnnpack backend.</p>
<pre><code class="lang-auto">class SingleLayerLinearDynamicModel(torch.nn.Module):
    def __init__(self):
        super(SingleLayerLinearDynamicModel, self).__init__()
        self.qconfig = default_qconfig
        self.fc1 = torch.nn.Linear(5, 5).to(dtype=torch.float)

    def forward(self, x):
        x = self.fc1(x)
        return x

torch.backends.quantized.engine = 'qnnpack'
base = SingleLayerLinearDynamicModel()
model = quantize_dynamic(base, dtype=torch.qint8)
</code></pre>

huoge: <p>thx for your reply! I found this was a bug in my code，in some iteration, I pass the tensor with the shape (0, 20, 80). In fbgemm, it does not affect the result, while in qnnpack, it will report that error in the picture. so I fix the bug in my code, then everything goes well.</p>
Multiclass Semantic segmentation in pytorch

Mohit: <p>Greetings everyone</p>
<p>I am new to using PyTorch. I am trying to implement semantic segmentation for identifying multi-class objects for eg if I pass an image with 2 laptops and one knife, I want my model to categorise both the laptops in one class, the knife in another and background being the last class. I have understood the concepts intuitively and was able to predict binary classification using DeepLab V3. But after several tries I am failing to do the same when it comes to multi-channels. I have customised my images and masks using Gimp. <strong>I need help with the code</strong> if someone could send me a running code or a repository it’ll be a huge favour.</p>
<p>Regards</p>
Which number corresponds to which label?

Ajna_Ram: <p>Using one hot encoded labelling, which label is assigned to which number? Does it depend on how you define the variable ‘classes’?</p>

ptrblck: <p>The meaning of the class labels depend on your definition.<br>
I.e. you can create the mapping manually or use any predefined mapping for standard datasets.</p>
<p>However, note that for a vanilla multi-class classification use case, you won’t use one-hot encoded labels, but instead the class indices directly as explained in the <code>nn.CrossEntropyLoss</code> docs.</p>
Indexing torch tensors with a list of indices

SpandanMadan: <p>I populate a tensor with dimensions that are not fixed. Then, I try to index this tensor. I would like to pass the indices as a list, but that doesn’t seem to work.</p>
<p>For ex:</p>
<pre><code class="lang-auto">A = torch.rand((5,5,4,18))
B = [3,3,2]
</code></pre>
<p>Instead of indexing it as</p>
<pre><code class="lang-auto">A[3,3,2]
</code></pre>
<p>why can’t I do</p>
<pre><code class="lang-auto">A[B]
</code></pre>
<p>I need this because in my case the dimensions of B keep changing.</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">A = torch.rand((5,5,4,18))
B = [3, 3, 2]
print((A[3, 3, 2]==A[torch.tensor(B).split(1)]).all())
&gt; tensor(True)
</code></pre>
Torchvision expected release dates of future versions

syedb93: <p>Hi, I didn’t know what was the best place to ask this but when is torchvision==0.5.1 coming out?</p>
<p>Also is it going to address GitHub issue <a href="https://github.com/pytorch/vision/issues/1607" rel="nofollow noopener">#1607</a> by PR <a href="https://github.com/pytorch/vision/pull/1976" rel="nofollow noopener">#1976</a></p>
<p>Thanks</p>

ptrblck: <p>Since the PR was merged ~2 weeks ago, I would assume it’ll land in the next release.<br>
CC <a class="mention" href="/u/fmassa">@fmassa</a>, who might give you some information about the release date.</p>
<p>You can see the general PyTorch <code>1.5</code> release tracker <a href="https://github.com/pytorch/pytorch/issues/35009">here</a>.</p>
Fix Model Output for Same Input

musadaqm: <p>Following is the picture of my model<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b81501cf705a4bcde1a8bdd06378acadf9a86af.png" alt="image" data-base62-sha1="jU7q2PSX2NI6VCHd0Xb3DoOSJST" width="661" height="488"></p>
<p>When I give one input two times, the output is different. I want the output to be same for same input.</p>
<p>If anyone can help please? I have tried the following but didn’t find helpful</p>
<p>torch.manual_seed(123)<br>
torch.cuda.manual_seed(123)<br>
torch.cuda.manual_seed_all(123)<br>
np.random.seed(123)<br>
torch.backends.cudnn.deterministic=True</p>
<p>Thank you</p>

ptrblck: <p>Did you make sure to call <code>model.eval()</code> before running the comparison?<br>
This will make sure to disable dropout layers and use the running stats from batchnorm layers instead of the current batch statistic.</p>
<p>Also, note that some operations include <code>atomicAdd</code> as described <a href="https://pytorch.org/docs/stable/notes/randomness.html">here</a>, which will not yield deterministic results.</p>
Question about convolutional layer output size calculation

ljj7975: <p>When I tried to estimate the output size of conv layer using the formula in the doc,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d57cc555366c4e399e0b2f87ab669588b7d02c9.png" alt="Screen Shot 2020-03-29 at 10.54.55 AM" data-base62-sha1="hSPN0O5uk3yzIZ9EEbmTU84zRSV" width="673" height="75"><br>
the number I got from the formula is actually different from what I get from the layer.</p>
<p>for the 2d input of size (101, 40)<br>
with kernel(16,8), stride (2,1) padding (0,0), dilation (1,1),<br>
torch.2dconv generates (43,33)<br>
while formula seems to give (44,33)</p>
<p>where is this mismatch coming from?</p>
<p>Also I tried to understand the formula but couldn’t figure out where the last -1 operation comes from.<br>
we should be following basic math ordering (Brackets, Orders, Division and Multiplication, and Addition and Subtraction) here right?</p>
<p>Can anyone clarify?</p>

simaiden: <p><code>Hout = (101 +2*0 - 1*(16-1) -1)/2 +1</code> gives me 43</p>

ljj7975: <p>opps thanks, I actually for got to add -1 at the end.<br>
But still… can you explain why we need the last -1?</p>

ljj7975: <p>I actually found out why I got so confused.</p>
<p>without dilation,<br>
out = (in + 2*p - k)/s + 1</p>
<p>however, with dilation new kernel size k_d = d * (k-1) + 1,<br>
Plugging this back into the above formula we have</p>
<p>out = (in + 2<em>p - k_d)/s + 1<br>
= (in + 2</em>p - (d * (k-1) + 1))/s + 1<br>
= (in + 2*p - d * (k-1) - 1)/s + 1</p>
<p>same as the one from doc.</p>
Architecture for LSTM autoencoder

MehdiZouitine: <p>Hi everybody ! I’m currently working on anomaly detection in time series,(1D signal anomaly).</p>
<p>To deal with it i want to build an LSTM autoencoder.</p>
<p>I’ve no trouble with the code but I CANT find an architecture (I found no paper).</p>
<p>Someone can advice me ?</p>
<p>Thanks you</p>

ebarsoum: <p>There are multiple ways to do autoencoder with LSTM. At the high-level:</p>
<ol>
<li>Encoder: will output a state.</li>
<li>Decoder: will use this state as the initial state, and will try to construct the input to the encoder.</li>
</ol>
<p>The state from the encoder, can be:</p>
<ol>
<li>Final state.</li>
<li>Average state for the last K timesteps.</li>
<li>Weighted average --soft-attention.</li>
<li>The output of a NN, which take all the encoder states and learn the best output (a more advance version of the soft-attention).</li>
</ol>
<p>Also, because you are dealing with sequences, you can have multiple decoder to improve the presentation power of the encoder. Such as, decoder that construct current sequence, another decoder that predict the future…etc.</p>
RuntimeError: No such operator torchvision::nms

luodahei: <p>i use pytorch1.4+torchvision0.5(CPU in Windows)。when i use run detection model,it show that.Can someone tell me how to solve it? Thank you very much</p>

peterjc123: <p>Please refer to this post. <a href="https://github.com/pytorch/vision/issues/1916" rel="nofollow noopener">https://github.com/pytorch/vision/issues/1916</a></p>

nicolas-gervais: <p>I had this issue with Python 3.6.9 so it doesn’t seem to be a Python 3.8 problem. I solved my problem adding this before importing <code>torch</code> and <code>torchvision</code>:</p>
<pre><code class="lang-auto">import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.environ['PATH'] += py_dll_path
</code></pre>
Libtorch 1.4 kernel_lambda unhandled exception when calling backward

Cosmin1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f4789b48a09df2d6d12d567c158b87508bd076.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1f4789b48a09df2d6d12d567c158b87508bd076" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f4789b48a09df2d6d12d567c158b87508bd076.png" alt="image" data-base62-sha1="weTcSOEKgizBSGzO0B2aBkYj3TM" width="621" height="500" data-dominant-color="232726"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">805×648 45.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Cosmin1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e0e0190389a28ab2d1fb22c78e76af07bcf22a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/4e0e0190389a28ab2d1fb22c78e76af07bcf22a7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e0e0190389a28ab2d1fb22c78e76af07bcf22a7.png" alt="image" data-base62-sha1="b8vfxsGkFB8kP2LhvnyN2b6GsqX" width="690" height="454" data-dominant-color="242524"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">714×470 23.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Download Imagenet-1k for training ResNet model

zzsunshine: <p>Hi all,</p>
<p>I am a beginner for machine learning area. I want to train model with Imagenet-1k dataset, but I don’t know where can i download that (I assume it is different from Imagenet because it is smaller). I went to the Imagenet website but I cannot download dataset from here.</p>
<p>Appreciate for any response,<br>
Thanks</p>
Optimizer.step() not updating correctly

pytorchnewbie: <p>As you can see below, I print the value of scale.grad as well as the value of scale before and after the optimizer.step(),and my learning rate = 0.01 with adam optimizer, but 1 - .01*13 = 0.87, not 0.99. What could be going on here?  I know adam optimizer has some other tweaks to this equation but I didnt expect it be different by a magnitude of .11, which relative to 1 is 10% difference! Thanks.</p>
<p>Below is the photo and the code.</p>
<p><a href="/uploads/short-url/2JONAB7uZUCsSDQ60OUD9Kf72Ib.png">Screen Shot 2020-02-27 at 11.03.06 AM|690x308</a></p>
<pre><code class="lang-auto">cat= TF.to_tensor(np.array(Image.open("just_cat_above_dog.png").convert('RGB')))
cat_dog = TF.to_tensor(np.array(Image.open("cat_above_dog.png").convert('RGB')))

translated_params = torch.unsqueeze(torch.tensor([0.0,0.0]),1)
translated_params.requires_grad_(True)

scale = torch.unsqueeze(torch.tensor([1.0]),1)
scale.requires_grad_(True)

def forward2(x,dxdy,the_scale):
  dxdy = torch.tanh(dxdy)
  the_scale = torch.tanh(the_scale)
  M= torch.cat((torch.eye(2)*the_scale, dxdy),dim=1)[None,:,:]
  #print(M)
  grid = F.affine_grid(M,[1] + list(x.shape))
  resampled = F.grid_sample(x[None,:,:,:], grid, mode='bilinear')[0]
  return resampled

optimizer = torch.optim.Adam([scale,translated_params], lr=0.01)

for i in range(3000): 
    optimizer.zero_grad()
    predicted= forward2(cat_dog,translated_params,scale)
    #criterion = LapLoss()
    #loss = criterion.forward(torch.unsqueeze(predicted,0),torch.unsqueeze(cat,0))
    fft_predicted = torch.rfft(predicted,3)
    #pdb.set_trace()
    fft_gt = torch.rfft(cat,3)
    loss = torch.nn.functional.l1_loss(fft_gt[:,:,:,0],fft_predicted[:,:,:,0]) - 0.0001*torch.sum(torch.abs(fft_predicted))
    #absdiff = (predicted-cat).abs()
    #loss = absdiff[:,100:300,:].mean()

    loss.backward()
    print(loss, scale.grad.data, scale)
    #pdb.set_trace()
    
    optimizer.step()
    print(loss, scale.grad.data, scale)

    if i % 300 == 0:
      
      img = predicted.data.cpu().numpy().transpose((1,2,0))
      actualimg= cat.cpu().numpy().transpose((1,2,0))
      plt.figure()
      plt.imshow(img)
      plt.figure()
      plt.imshow(actualimg)
</code></pre>

ptrblck: <p>I’m not sure to understand the posted numbers correctly.<br>
Is <code>SGD</code> yielding the expected result?<br>
If so, your guess of the internal states in <code>Adam</code> might be right.</p>

pytorchnewbie: <p>sorry i rephrased my question in a new post, hopefully its clearer now… thank you</p>
Resize image data as a part of preprocessing

ang: <p>I have image of size (320,576,3)( 3 indicating RGB image) and their respective masks of size (640,1176)(Grayscale). I need to bring them to a common size for further processing but I am unable to figure out how to achieve it.<br>
Also this has to be performed on the complete dataset of 417 images. Can someone suggest any way?</p>

charan_Vjy: <p>This might do the trick</p>
<pre><code class="lang-auto">import torchvision.transforms as transforms
compose_img = transforms.Compose(transforms.Resize((x,y))]) ## x,y is the common size of your choice
img = compose_img(img)
</code></pre>

Angry_potato: <p>You can use cv2 to resize the images.<br>
‘’’ im=cv2.resize(image,(height ,width))’’’</p>
Attention implemented using encoder outputs rather than hidden states

karsal: <p>I have observed in the implementations by Pytorch and others, encoder outputs are used instead encoder   hidden states to calculate attention weights. Does the make no difference? One reason that I’ve observed  is that using LSTM without manually looping through the sequence results in the entire sequence at the output but only final hidden state. Are there other reasons for using encoder outputs instead of encoder hidden states ?</p>

vdw: <p>The encoder outputs ARE the hidden states at each time step. Look at the following sketch of an LSTM cell:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fe697b28ed127cc9d294c496fd06fce4a65289d.png" data-download-href="https://discuss.pytorch.org/uploads/default/9fe697b28ed127cc9d294c496fd06fce4a65289d" title="1024px-The_LSTM_cell"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fe697b28ed127cc9d294c496fd06fce4a65289d_2_690x452.png" alt="1024px-The_LSTM_cell" data-base62-sha1="mOxXwCQbZteaYhcCBWs3B0YY4c5" width="690" height="452" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fe697b28ed127cc9d294c496fd06fce4a65289d_2_690x452.png, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fe697b28ed127cc9d294c496fd06fce4a65289d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fe697b28ed127cc9d294c496fd06fce4a65289d.png 2x" data-dominant-color="C2E0D2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1024px-The_LSTM_cell</span><span class="informations">1024×672 75.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There is not “extra” output, it’s just the current hidden state <code>h_t</code>. For example, if you define an <code>lstm</code> with <code>num_layers=0</code> and <code>batch_first=False</code>, and given, say,</p>
<pre><code>output, (h, c) = lstm(inputs, hidden)
</code></pre>
<p>then <code>output[-1] = h_n</code>, simply meaning that you can get the final hidden state directly from <code>h</code> or as the last hidden state from <code>output</code>. If you have more layers (<code>num_layers&gt;1</code>), things are a bit different: <code>output</code> will contain only the hidden states for each time stamp after the last layer, while <code>h</code> will contain the last hidden states for each layer.</p>

karsal: <p>Thanks, I totally missed out on this. Surprisingly, I have seen this question asked by others and people are suggesting to manually loop over sequence length to capture the hidden states.</p>
How to implement a beam search for translation?

IamAdiSri: <p>Basically what the title says. I know what a beam search does but cannot understand how to implement it efficiently in PyTorch. I did find a couple of implementations online, but couldn’t understand how they worked. Any help would be appreciated.</p>
<p>If you happen to know of some way to add beam search decoding to a regular neural network, let me know of the same as well.</p>
How to reshape tensors for LSTM input?

odats: <p>I have LSTM with 3 inputs and 1 output. Dataset returns ((tensor1, tensor2, tensor3), tensor_target). How to reshape my Dataset outputs to make in work with LSTM input?</p>
<pre><code class="lang-auto">batch_size = 5
n_hidden = 1
encoder_size = 64*2*2
seq_len = 3

rnn = nn.LSTM(encoder_size, encoder_size, n_hidden)

input = torch.randn(seq_len, batch_size, encoder_size) # how to reshape to this shape?

output, (hn, cn) = rnn(input)
</code></pre>

Rohan_Kumar: <p>this looks like a simple lstm, cant you use tensor.view(seq_len, batch_size, encoder_size)?</p>

odats: <p>I can not use tensor.view because</p>
<aside class="quote no-group" data-username="odats" data-post="1" data-topic="74626">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/odats/48/21970_2.png" class="avatar"> odats:</div>
<blockquote>
<p>(tensor1, tensor2, tensor3)</p>
</blockquote>
</aside>
<p>is not a tensor</p>

vdw: <p>It’s less about reshaping but converting your input tuple <code>(tensor1, tensor2, tensor3)</code> to a tensor. What you can do is, for example:</p>
<pre><code>input = torch.zeros(3, batch_size, encoder_size)
tensor[0], tensor[1], tensor[2] = tensor1, tensor2, tensor3
</code></pre>
<p>This assumes that <code>tensor1.shape = (batch_size, encoder_size)</code>; same of <code>tensor2</code> and <code>tensor3</code>, of course. If the shapes are <code>(seq_len, encoder_size)</code> then you have to adopt the code a bit. But I cannot tell which shapes your tensors have.</p>
How does masked_lm_labels work

hao_wang2: <p>The BertForMaskedLM model in the Bert model has a parameter masked_lm_labels and I want to know what he expresses and how can we get it<br>
This is it’s introduction<br>
masked_lm_labels (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) – Labels for computing the masked language modeling loss. Indices should be in [-100, 0, …, config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, …, config.vocab_size]</p>
How to use bert to train my own dataset

hao_wang2: <p>I just learned bert these days, and want to use bert to train my corpus, but the code on the Internet uses Google-trained models. I want to ask how to train my corpus through bert</p>

damaoooo: <p>you can download un pre-trained models from Internet</p>

hao_wang2: <p>Can you say more in detail, how do I train the model, does bert have a corresponding method, thank you very much</p>

damaoooo: <p>you can download it by github searching <code>pytorch-bert</code> there are so many codes that can be used which include the model download</p>

hao_wang2: <p>thank you ,i will try it</p>
Bayesian optimization libraries for tuning and

davidlee: <p>Hi,</p>
<p>I am looking ‘Bayesian optimization’ for hyper-parameter tuning<br>
for my multi-layered neural networks.<br>
What I would like to optimize are like, ‘learning rate, batch size, number of layers and neurons’</p>
<p>Can you guide me any example or tutorial for bayesian optimization?<br>
Also, Can you recommend any library for this?<br>
I’ve been looking through ‘hyper-opt’ or ‘bayesian-optimization’ libraries and wondering if they works for pytorch.</p>
<p>Thanks in advance.</p>

charan_Vjy: <p>Check out Pyro(Provides api for probabilistic programming) -&gt; <a href="https://pyro.ai/" rel="nofollow noopener">https://pyro.ai/</a></p>

davidlee: <p>Hi, Thank you for your suggestion,<br>
Do you see any pros or cons with hyperopt, skopt, bayesian-optimization or botorch?</p>

charan_Vjy: <p>From a quick browse of all above mentioned frameworks,</p>
<ol>
<li>BoTorch (Recommended) - Built on top of Pytorch with autograd features. All basic Bayesian optimization tools are included. This should be preferred if you are using Pytorch<br>
Pros - Modular, Simple and Scalable.<br>
Cons - Not extensive</li>
<li>scikit-optimize - Integrated with scikit learn. Has extensive API and good example.<br>
Pros - More extensive than BoTorch.<br>
Cons - Not sure how easy it is to run in conjunction with Pytorch</li>
<li>Hyperopt - Provides parallelization capabilities with MongoDB and Apache Spark<br>
Pros - Might be useful for if the database is huge and parallelization is required.<br>
Cons- Might not be the right tool for prototyping.</li>
<li>bayesian-optimization - Runs on top of scipy and scikitlearn.<br>
Pros - good documentation, Clear examples, should be good for basic prototyping<br>
Cons - Not sure how it would go along with pytorch.</li>
</ol>

davidlee: <p>I really appreciate your help!</p>

charan_Vjy: <p>Thanks. The pleasure is mine.</p>