akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Spurious HTML code in neural_networks_tutorial.ipynb

Rick_Szeliski: <p>I’m not sure if this is the right forum to report a formatting bug in one of the JuPyter Noteboks in the tutorial, but one of the Markdown text cells has some HTML in it instead of pure Markdown.</p>
<p>In particular, it’s the cell that’s titled Note and starts with the following Markdown:</p>
<p>&lt;div class=“alert alert-info”&gt;&lt;h4&gt;Note&lt;/h4&gt;&lt;p&gt;&gt;``torch.nn`` only supports mini-…</p>
<p>which should be replaced with a simple</p>
<p>**Note** ``torch.nn`` only supports mini-…</p>
<p>Markdown command.</p>

ptrblck: <p>Thanks for reporting this, Rick!</p>
<p>Would you mind creating an issue <a href="https://github.com/pytorch/tutorials/issues">here</a> so that we can fix it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torchvision pretrained vgg16 doesn't accept 224*224*3 image?

billconan: <p>I’m trying to use the pretrained vgg16 model provided by torchvision.</p>
<p>But it seems that the expected input image is not a 3<em>224</em>224, as I got the following?</p>
<pre><code class="lang-auto">RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 3 3 3, but got 3-dimensional input of size [3, 224, 224] instead
</code></pre>
<p>My code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchvision import models

original_model = models.vgg16(pretrained=True)

ran = torch.rand((3, 224,224))

original_model.features(ran)
</code></pre>

ptrblck: <p>The input is expected to have the shape <code>[batch_size, channels, height, width]</code><br>
You would have to add a batch dimension in dim0 via:</p>
<pre><code class="lang-python">ran = ran.unsqueeze(0)
</code></pre>
<p>or initialize the batch dimension directly via:</p>
<pre><code class="lang-python">ran = torch.rand((1, 3, 224, 224))
</code></pre>

billconan: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="78911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">ran = torch.rand((1, 3, 224, 224))
</code></pre>
</blockquote>
</aside>
<p>Thank you, this works. However, why does the error message say  64 3 3 3 ? instead of 1,3,224,224?</p>

billconan: <p>oh, I got it. it’s 4 dimensional weights …</p>

ptrblck: <p>This is the kernel shape of the conv layer (64 filters, 3 input channels, 3x3 spatial size).<br>
The error message mentions the 3-dimensional input as:</p>
<blockquote>
<p>but got 3-dimensional input of size [3, 224, 224] instead</p>
</blockquote>
<p>EDIT: I was a second too late <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Implement “Soft Pooling”

shirui-japina: <p>In the traditional pooling layer, the side length during pooling is fixed.<br>
At present, I want to try to implement pooling with variable side length, tentatively called “soft pooling”.</p>
<p>The details are as follows.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dfca32f02c4d685861afe01c8118e0ae790dece7.png" data-download-href="https://discuss.pytorch.org/uploads/default/dfca32f02c4d685861afe01c8118e0ae790dece7" title="traditional pooling"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/dfca32f02c4d685861afe01c8118e0ae790dece7_2_345x192.png" alt="traditional pooling" data-base62-sha1="vVJGtZl5TGJy3Q7pH1T2kkqUVaT" width="345" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/dfca32f02c4d685861afe01c8118e0ae790dece7_2_345x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/dfca32f02c4d685861afe01c8118e0ae790dece7_2_517x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/dfca32f02c4d685861afe01c8118e0ae790dece7_2_690x384.png 2x" data-dominant-color="90B4DB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">traditional pooling</span><span class="informations">930×520 9.88 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
traditional pooling</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47eb6d6a2e9358feeee86c8739b25eca59ba8387.png" data-download-href="https://discuss.pytorch.org/uploads/default/47eb6d6a2e9358feeee86c8739b25eca59ba8387" title="soft pooling"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47eb6d6a2e9358feeee86c8739b25eca59ba8387_2_345x192.png" alt="soft pooling" data-base62-sha1="ageiG8eiuMCgORJkvFqAAs4n4Ld" width="345" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47eb6d6a2e9358feeee86c8739b25eca59ba8387_2_345x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47eb6d6a2e9358feeee86c8739b25eca59ba8387_2_517x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47eb6d6a2e9358feeee86c8739b25eca59ba8387_2_690x384.png 2x" data-dominant-color="8FB3DA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">soft pooling</span><span class="informations">930×520 9.94 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
“soft pooling”</p>
<p>How can I implement such a model layer?</p>

ptrblck: <p>How would this approach work, if more than two windows fit into each spatial dimension?<br>
If the <code>3x3</code> kernels stays constant and just the last part if smaller, you could probably add padding on the right and down border of the input.</p>
Expected object of device type cuda but got device type cpu for argument #1 ‘self’ in call to _thnn_conv2d_forward

statstom: <p>Given a tensor loaded with data, I am trying to convert it to CUDA and then forward it through a module that was loaded in CUDA.</p>
<p>For example:</p>
<pre><code class="lang-auto">input.to(torch::kCUDA);
vector&lt;torch::jit::IValue&gt; inputs;
inputs.emplace_back(input);  //Input defined at at::Tensor that was .to(torch::kCUDA)
module.to(torch::kCUDA);
result = module.forward(inputs);
</code></pre>
<p>This results in</p>
<blockquote>
<p>Expected object of device type cuda but got device type cpu for argument <span class="hashtag-raw">#1</span> ‘self’ in call to _thnn_conv2d_forward<br>
The above operation failed in interpreter.</p>
</blockquote>
<p>Do I need to convert the IValue to cuda somehow?  Or do the values in IValue hold their original device?  When I check the device of the tensor I load into the IValue vector, it is of device CUDA.</p>

ptrblck: <p>Could you try to reassign <code>input</code> via:</p>
<pre><code class="lang-auto">input = input.to(torch::kCUDA);
</code></pre>
<p>and rerun the code, please?</p>

statstom: <p>Thanks this worked.</p>
<p>I’m curious, why is the module to function in place but the input is a return value?</p>

ptrblck: <p>It’s reflecting the Python frontend, which works in the same way.<br>
<code>.to()</code> will be called recursively on all sumbodules and their parameters as well as buffers, which is why you don’t have to reassign modules.</p>
nn.TransformerEncoderLayer 3D Mask Doesn't Match the Broadcast Shape

HHousen: <p>Hello. I am trying to use a 3D mask with a <code>nn.TransformerEncoderLayer</code>. However, I can not seem to get the dimensions right. The <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Transformer.forward" rel="nofollow noopener"><code>nn.Transformer</code> documentation</a> dictates the source size as (source sequence length, batch size, feature number) which I use below. The <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention" rel="nofollow noopener"><code>MultiHeadAttention</code> code</a> shows the shape of a 3D mask should be "<code>(N*num_heads, L, S)</code> where N is the batch size, L is the target sequence length, S is the source sequence length," which I also use below.</p>
<p>Reproducible Example Code:</p>
<pre><code class="lang-python">import torch
from torch import nn

num_heads = 8
bs = 16
num_features = 768
src_and_target_size = 21

encoder_layer = nn.TransformerEncoderLayer(num_features, num_heads, dim_feedforward=2048, dropout=0.1)

src = torch.rand(src_and_target_size, bs, num_features)

mask = torch.ones(bs*num_heads, src_and_target_size, src_and_target_size)

print(src.shape)
print(mask.shape)

encoder_layer(src, mask)
</code></pre>
<p>However, I recieve the below error message. Can you please help me find the error in my mask dimension sizes?</p>
<p>Error Message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 19, in &lt;module&gt;
    encoder_layer(src, mask)
  File "/home/user/anaconda3/envs/test_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/envs/test_env/lib/python3.7/site-packages/torch/nn/modules/transformer.py", line 283, in forward
    key_padding_mask=src_key_padding_mask)[0]
  File "/home/user/anaconda3/envs/test_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/envs/test_env/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 819, in forward
    attn_mask=attn_mask)
  File "/home/user/anaconda3/envs/test_env/lib/python3.7/site-packages/torch/nn/functional.py", line 3362, in multi_head_attention_forward
    attn_output_weights += attn_mask
RuntimeError: output with shape [128, 21, 21] doesn't match the broadcast shape [1, 128, 21, 21]
</code></pre>
<p>Thanks.</p>

HHousen: <p>The PyTorch code that produces the error:</p>
<p><code>torch/nn/functional.py", line 3362, in multi_head_attention_forward</code>:</p>
<pre><code class="lang-python">if attn_mask is not None:
    attn_mask = attn_mask.unsqueeze(0)
    attn_output_weights += attn_mask
</code></pre>
<p>Commenting out the <code>attn_mask.unsqueeze(0)</code> line makes the code run without errors. Is this a bug in PyTorch? Why add a dimension to the beginning of the attention mask?</p>

HHousen: <p>Solution: Upgrade to PyTorch 1.5</p>
What do I need to inherit to make a custom nn module?

Brando_Miranda: <p>I want to make a class that could be any type of nn module (though right now I am using it to play around with only fully connected NNs while I learn pytorch). I was doing my class but now I’m getting python attribute errors e.g.:</p>
<pre><code>*** AttributeError: 'NN' object has no attribute '_modules'
</code></pre>
<p>Which makes me think I am not calling the init method of something I clearly need to be calling. The example in the tutorials have as an example:</p>
<pre><code> super(TwoLayerNet, self).__init__()
</code></pre>
<p>but my FC NN might have more than 2 layers. Thus, I was wondering, is there some general super I can call that doesn’t restrict the types of NNs I might make? If not I’d like restrict to any type of feedforward NN (with convolutions or not). An example of the super I should call for an RNN would also be nice.</p>

fmassa: <p>Just use the construct you added. You only need to inherit from <code>nn.Module</code>, and call the <code>__init__</code> from it</p>
<pre><code class="lang-python">class MyComplexModel(nn.Module):
    def __init__(self, net1, net2, net3):
        super(MyComplexModel, self).__init__()
        self.net1 = net1
        self.net2 = net2
        self.net3 = net3

    def forward(self, input):
        return self.net3(self.net2(self.net1(x) + 1)**2) / x
</code></pre>

Brando_Miranda: <aside class="quote no-group quote-modified" data-username="fmassa" data-post="2" data-topic="5896">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar"> fmassa:</div>
<blockquote>
<p>super(MyComplexModel, self).<strong>init</strong>()</p>
</blockquote>
</aside>
<p>can you explain what that does?</p>

fmassa: <p>This is calling the <code>__init__()</code> method of the parent class of <code>MyComplexModel</code>, in this case, <code>nn.Module</code>. For more details, see for example <a href="http://www.python-course.eu/python3_inheritance.php">http://www.python-course.eu/python3_inheritance.php</a></p>

Brando_Miranda: <p>actually <code>super(type(self),self).__init__()</code> is better</p>

financial_physician: <p>Could you elaborate on why that is?</p>
Bayesian optimization with TPE or Gaussian process?

davidlee: <p>Hi all,</p>
<p>I want to optimize [the number of hidden layers, hidden neurons and learning rate]</p>
<p>Since the searching space is both continuous and discrete,<br>
What surrogate model do I have to use?<br>
I know Gaussian process is widely used for Bayesian optimization, but would it be possible to apply Gaussian process in this case?</p>
<p>Thanks in advance!</p>
Loaded Model Returns Different Predictions

murphinator: <p>I performed transfer learning on a VGG-16 and re-trained the classifier portion. Training worked, the predictions work. However, I am puzzled as to why when I continue to run a prediction on the same image file, the prediction oscillates between two classifications that are very similar to each other. I must have forgotten to change a setting, but I’m not sure which one.</p>
<pre><code class="lang-auto">def predict_breed_transfer(img_path, prob=None):
    # load the image and return the predicted breed 
    
    img = Image.open(img_path)
    
    myResize         = transforms.Resize((255,255))
    myCenterCrop     = transforms.CenterCrop(224)
    myRandomRotation = transforms.RandomRotation(35)
    myToTensor       = transforms.ToTensor()
    myNormalize      = transforms.Normalize(mean=(0.485,0.456,0.406), 
                                            std=(0.229,0.224,0.225))
    
    transform  = transforms.Compose([myResize, 
                                     myCenterCrop,
                                     # myRandomRotation,
                                     myToTensor, 
                                     myNormalize])
    
    img = transform_test(img)
    img = img.unsqueeze(0)
    img = img.cuda()
    
    prediction = model_transfer(img)
    prediction = prediction.cpu()
    
    predicted_class_idx = prediction.data.numpy().argmax()
    predicted_class_str = class_names[predicted_class_idx]
    
    if prob == None:
        return predicted_class_str
    elif prob != None:
        return prediction.data.numpy().max()

</code></pre>

ptrblck: <p>You might have forgotten to call <code>model_transfer.eval()</code>, which will disable dropout and will use the running stats in batchnorm layers instead of the current batch stats.<br>
Let me know, if that’s the case or if you are still seeing shaky outputs.</p>

murphinator: <p>You are correct. Calling .eval() stopped the oscillation of predicted outputs. Many thanks!</p>

murphinator: <p>Hi again, I was thinking about this more. If I am simply calling my prediction function to run an image through the net, meaning I am not backpropping or updating gradients, then I guess I still don’t understand why the output would change?</p>

ptrblck: <p>The gradient calculation is independent from <code>model.train()/eval()</code>.<br>
<code>train()</code> and <code>eval()</code> will change the behavior of some modules, such as dropout and batch norm layers.<br>
The gradient calculation (frozen parameters or not) won’t be changed.<br>
So even if you wrap the code in a <code>with torch.no_grad()</code> block, which will disable the gradient calculation, the batch norm stats will be updated, if the modules in still in <code>training</code> mode.</p>
Deployment of PyTorch model in production

vishalagarwal: <p>I have a couple of questions related to using PyTorch models in production</p>
<ol>
<li>
<p>How can we deploy PyTorch models in a stateless server on-cloud?</p>
</li>
<li>
<p>What’s the best practice to deploy a model offline to do inference on phone? I mean I know about the tracing and direct embedding method but from a security perspective is it safe to roll out an APK by tracing? By safety I mean can someone reverse engineer the weights of my model from the APK?</p>
</li>
</ol>

thvasilo: <p>I’d suggest taking a look at <a href="https://github.com/pytorch/serve/blob/master/README.md#serve-a-model" rel="nofollow noopener">torchserve</a> for model serving. Not sure if it’s possible to do stateless though, I assume you mean something like a Lambda function?</p>
Visualize torchvision data

oasjd7: <p>I can load svhn dataset, but how can I plot some images?<br>
I tried to plot with cv2, but the data is torch.tensor type so I failed.<br>
Anyone can help ?</p>
<pre><code class="lang-auto">from torchvision import datasets, transforms
from torch.utils.data import DataLoader


def svhn_train_loader():
    pre_process = transforms.Compose([
                                      transforms.ToTensor(),
                                      transforms.Normalize(
                                          mean=(0.5, 0.5, 0.5),
                                          std=(0.5, 0.5, 0.5))])

 
    svhn_dataset = datasets.SVHN(root='svhn',
                                 transform=pre_process,
                                 download=True)

    svhn_data_loader = DataLoader(
        dataset=svhn_dataset,
        batch_size=params.batch_size,
        shuffle=True)

    return svhn_data_loader
</code></pre>

ptrblck: <p>You can get the numpy array using <code>data.numpy()</code>.<br>
Then you can plot this array with OpenCV, matplotlib etc.</p>

oasjd7: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for quick reply. I tried data.numpy()</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
for batch_idx, (inputs, labels) in enumerate(svhn_train_loader):
     plt.figure()
     plt.imshow(inputs.numpy())
     plt.show()
</code></pre>
<p><code>TypeError: Invalid dimensions for image data</code><br>
How can I fix  this error?</p>

ptrblck: <p><code>matplotlib</code> needs the channel in dim2.<br>
Permute your image before passing it to <code>imshow</code>.<br>
<code>inputs = inputs.permute(1, 2, 0)</code></p>
Grad is none when the grad_requires=True after adding the retain_grad()

Maxxingxing: <h2>Hello,guys~~I can not get the grad of the input of my net. I have tried a lot of solutions provided in the forum, but it can not work. Following is part of my code, and anybody can help? Thanks a lot!</h2>
<p>X_emb = Variable(X_emb.data,requires_grad=True)<br>
X_emb = X_emb.to(device)<br>
print(X_emb.dtype)<br>
<span class="hashtag">#X_emb</span>.requires_grad_()<br>
print(X_emb.requires_grad)<br>
X_emb.retain_grad()<br>
T = opt.T<br>
epsilon = opt.epsilon</p>
<p>net.zero_grad()<br>
<span class="hashtag">#y_hat</span> = net(X,seq_lengths)<br>
y_hat = net_rest1(net,X_emb,seq_lengths)<br>
y_hat.requires_grad_()<br>
print(y_hat.requires_grad)<br>
<em>, pred_idx = torch.max(y_hat.data,1)<br>
labels = Variable(pred_idx)<br>
y_hat = y_hat / T<br>
y_hat.requires_grad</em>()<br>
print(y_hat.requires_grad)<br>
loss = xent(y_hat,labels)<br>
loss.requires_grad_()<br>
X_emb.retain_grad()<br>
loss.backward()</p>
<p>print(type(X_emb))<br>
print(X_emb.grad)<br>
X_emb = X_emb - epsilon * torch.sign(X_emb.grad) # here occures the error because X_emb.grad is None</p>
<hr>
<p>loss.backwarc() can pass but I can’t get the grad. I am very confused with the result…</p>

chetan06: <p>Your Variable X_emb has no value so how its grad can be calculated.</p>

Maxxingxing: <p>this is just a part of my code, it must have value…</p>

albanD: <p>Hi,</p>
<p>The issue is that you have some non-differentiable operations.<br>
More generally. You should never need to call <code>.requires_grad_()</code> unless for the Tensor for which you want .grad to be populated.</p>
<p>Here is updated code with comments</p>
<pre><code class="lang-auto"># If you want to break the graph, use `.detach()`
X_emb = X_emb.detach()
# Move to the right device
X_emb = X_emb.to(device)
# Tensor still a leaf as it does not require gradients. Make it now
X_emb.requires_grad_()
# No need for retain_grad() as it is already a leaf

T = opt.T
epsilon = opt.epsilon

net.zero_grad()
#y_hat = net(X,seq_lengths)
y_hat = net_rest1(net,X_emb,seq_lengths)
# y_hat.requires_grad_() don't add extra requires_grad they are never needed
, pred_idx = torch.max(y_hat.data,1)
labels = pred_idx
y_hat = y_hat / T
print(y_hat.requires_grad)
loss = xent(y_hat,labels)
# loss.requires_grad_()
loss.backward()

print(type(X_emb))
print(X_emb.grad)
</code></pre>
<p>If a Tensor in the middle does not require gradients even though the input to the op does, that means the op is not differentiable and so no gradient can flow back.</p>
Quantize activations with qinit8

Reuven_Peretz: <p>Hello,<br>
I was wondering if I can use observers for activations with dtype of qint8.<br>
It seems by the <a href="https://pytorch.org/docs/stable/quantization.html#operation-coverage" rel="nofollow noopener">documentation</a> that it cannot be done (I want to make sure I got it correctly).</p>
<p>When I tried it with different observers, it failed for this kind of error when evaluating:<br>
RuntimeError: expected scalar type QUInt8 but found QInt8 (data_ptr<a>c10::quint8</a> at /pytorch/build/aten/src/ATen/core/TensorMethods.h:6322)</p>
<p>Does qint8 supported for activation quantization?</p>
<p>Thanks!</p>

hx89: <p>The default observer for activation is quint8, but you may overwrite it by creating a new activation observer such as:<br>
new_activation_observer = MinMaxObserver.with_args(dtype=torch.qint8)</p>
<p>and then assign this observer to the model qconfig before quantizing the model:<br>
myModel.qconfig = QConfig(activation=new_activation_observer,<br>
weight=default_weight_observer)</p>

Reuven_Peretz: <p>Thanks.<br>
Did you try it?<br>
Because I just tried what you offered and got again the above-mentioned error:<br>
RuntimeError: expected scalar type QUInt8 but found QInt8 (data_ptr<a>c10::quint8</a> at /pytorch/build/aten/src/ATen/core/TensorMethods.h:6322)</p>
<p>I think it’s not supported, but I don’t understand why…</p>

jerryzh168: <p>which quantized op are you calling? some ops only works for quint8</p>

onesnow123q: <aside class="quote no-group" data-username="hx89" data-post="2" data-topic="74839">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hx89/48/28407_2.png" class="avatar"> hx89:</div>
<blockquote>
<p>activation=new_activation_observer,<br>
weight=default_weight_observer</p>
</blockquote>
</aside>
<p>I tried this, but got the same error<br>
RuntimeError: expected scalar type QUInt8 but found QInt8 (data_ptr<a>c10::quint8</a> at /pytorch/build/aten/src/ATen/core/TensorMethods.h:6322)<br>
the op is<br>
self.conv0 = nn.Conv2d(3, 3, kernel_size=1, stride=1, padding=0, bias=False)</p>
<p>which quantized op are supported qint8??</p>

jerryzh168: <p>conv only support quint8 as input, I think right now most of the ops that doing some computation on the data itself will require quint8.</p>
<p>for example, conv expects quint8 activations and qint8 weight, so you need to set the dtype to quint8 for activation observer and qint8 for weight observer</p>
DCGAN tutorial -> make D see only grayscale

joshgiesbrecht: <p>I’ve been working through the DCGAN tutorial and want to work with new datasets. I’ve brought in a grayscale set of Lego images, and while I <em>could</em> do the sensible thing and figure out how to convert the network to operate with a single color channel, I realized it would be <em>much funnier</em> to make the Detector colorblind so that the Generator could get away with making psychedelic colorful Lego instead of this boring gray.</p>
<p>I’m having a heck of a time doing it though. I tried doing a manual averaging of the color channels, like such:</p>
<pre><code class="lang-auto">        with torch.no_grad():
            for f in fake:
                f[0] = (f[0] + f[1] + f[2]) / 3
                f[1] = f[0]
                f[2] = f[0]
</code></pre>
<p>between when the fakes are generated and when they’re passed along to D. But I get an error complaining that it’s noticed an anomaly despite the no_grad.</p>
<p>Is there an easier way to do this? Or a fix I can use to make it not break?</p>
<p>thanks!</p>

ptrblck: <p>Your idea sounds like a fun project! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Try to avoid the inplace operations and create a new tensor via:</p>
<pre><code class="lang-python">f2 = f1.mean(1, keepdim=True)
</code></pre>
<p>and pass <code>f2</code> to the discriminator.</p>

joshgiesbrecht: <p>Thanks! I used that and changed the D to just 1 color channel elsewhere and it worked!</p>
<p>The whole thing came out more uniformly pink-plaid than I expected <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Data:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4e09c98c2e4c8874cbfe5568f090d33d3aef27b.png" alt="lego-start" data-base62-sha1="nwzx4PnLAuqJ6gSBJNc7AFpBmqf" width="449" height="465"></p>

joshgiesbrecht: <p>Result:</p>
<p>Result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39e3e4346499521353427cc1363e3484f8307e02.png" alt="pinkplaidlego" data-base62-sha1="8g7tcilHvoWlI6zWbIvCzmt3Sa6" width="449" height="449"></p>

joshgiesbrecht: <p>As maybe all of you would’ve guessed, it picks a hue at random each time I train it. I was hoping for blobs of color left over from early inaccurate generation, but thus far it’s consistently one major color with that same stripey-grid plaid texture within.</p>
Final layer of Unet for multiple classes?

sgaur: <p>Hi Guys,</p>
<p>I am trying to build Unet for one of my project from scratch but my label consists Multiple Classes(10 Classes).I gone through multiple source but unable to understand the final layer of Unet Network(Last layer of Decorder/Upscaling part in Unet ).</p>
<p>Currently I am using Linear Transformation in the end I don’t know it is a correct solution or not ??</p>
<p>Initial Layer of Down scaling</p>
<blockquote>
<p>self.conv2d_1 = nn.Conv2d(1,32,kernel_size = 3,padding =1,stride=1)</p>
</blockquote>
<p>Final Layer of Up scaling.</p>
<blockquote>
<p>self.conv2d_10 = nn.Conv2d(32,32,kernel_size = 3,padding =1,stride=1)<br>
self.Fc1 = nn.Linear(32,10)</p>
</blockquote>

vishalagarwal: <p>In UNet, the output layer is a conv2d layer with kernel size 1 followed by a sigmoid activation</p>
<p><code>out_layer = nn.Conv2d(in_ch, num_classes, 1)</code></p>
Understanding workers

carlogarro: <p>Hello, I have a doubt with dataoader. If batch size = 1 and workers = 0, when iterating the dataset, when printing I receive one print output.<br>
But when batch size = 1 and workers = 1 I receive two print outputs, is it normal? I thought I should receive also one. And when, for example, batch size = 2 and workers = 2 I should receive 4 but I receive 8… Could you please explain it to me?</p>
<p>Maybe the image will help you to understand my question.<br>
Thank you.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f985823bd1c038882ab224ddbd334241d9c0475.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f985823bd1c038882ab224ddbd334241d9c0475" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f985823bd1c038882ab224ddbd334241d9c0475.png" alt="image" data-base62-sha1="icL2a8k7p04wKi546yjJ4V0f5XL" width="551" height="500" data-dominant-color="1E1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">673×610 13 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>When workers = 0, the loading happens synchronously on the same process and so each batch is loaded when asked for.</p>
<p>When workers &gt; 0, since loading is done in other processes, we have more freedom. In particular, we preload future batches in advance so that when you ask for them, they are (hopefully) already ready. And we load the next batch while you’re processing the one you just got.</p>

carlogarro: <p>Thanks albanD,</p>
<p>Is it defined how many future batches in advance are loaded? Just curiosity.</p>

albanD: <p>IIRC <code>2 * workers</code>, see code <a href="https://github.com/pytorch/pytorch/blob/a51f047c7e54d8e96837706428d322e3099ac109/torch/utils/data/dataloader.py#L745">here</a>.</p>
Keeping gradients of past iterations

deepfailure: <p>Hello,</p>
<p>I am exploring a RL agent and want to keep gradients of previous iterations in the following way:</p>
<p>grad_current_iter = grad_from_loss_current_iter + grad_from_loss_iter_-1 + grad_from_loss_iter_-2 …</p>
<p>reset every N iter</p>
<p>Would it be just enough to deactivate optimizer.zero_grad():</p>
<pre><code class="lang-auto">loss = obtain_loss()
if self.iteration % 10:
    optimizer.zero_grad()
loss.backward()
self.optimizer.step()
</code></pre>

albanD: <p>Hi,</p>
<p>If you do this, then all the gradients will accumulate. It may or may not be what you want.<br>
From your description, it seems like you want a sliding window? Or is just reseting every N iterations enough?</p>

deepfailure: <p>I want it to reset every episode (and the episodes have always the same number of steps).</p>
<p>So if my episode has 100 steps.</p>
<p>The step 50 would have 50 gradients accumulated and step 101 would have just one gradient.</p>
<p>I guess it will work, thanks.</p>

albanD: <p>In that case yes, your example will work just fine !</p>
How to freeze a specific layer in pytorch?

Peter_Ham: <p>I found one post here: <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088">How the pytorch freeze network in some layers, only the rest of the training?</a> but it does not answer my question. If I create a layer called <code>conv1 = nn.Conv2d(3,3,2)</code>, then how do I freeze this specific layer?</p>

BartolomeD: <p>This should do the trick:</p>
<pre><code class="lang-auto">for param in conv1.parameters():
    param.requires_grad = False
</code></pre>

Benat_Mencia: <p>As defined above, conv1 contains 3 filters of dimensions (3,2,2). What should I do if I wanted to freeze the second filter only?</p>
How is gradient of multi-gpu merged?

coincheung: <p>Hi,</p>
<p>When I use two gpus to train my model, in the backward computation, are the gradients of the two gpus get added or averaged to for us to obtain the gradient for updating the model parameters?</p>
<p>As for the moving average parameters of batchnorm layers, if I do not use sync-bn, how is the moving average status of the final model computed?</p>

albanD: <p>Hi,</p>
<p>It depends how you do multigpu. If you use DataParallel, then the gradients are accumulated on the first device in the list and then the new weights are shared to all the gpus.<br>
For batchnorm when not synced I am not sure but I would guess it uses the statistics computed on the first device in the list.</p>

coincheung: <p>I used distributeddataparallel, does this method sum or average the gradients of multi-gpus?</p>
<p>I think this settings will have influence on adjusting the learning rate. According to the learning rate linear rule, when batch size is double, the learning rate should also be doubled. If the gradients are summed, the amplitude of the gradients are thus double, which means that I no longer need to double the learning rate, and I only need to adjust the learning rate according to the change of batch size on one single gpu, rather than the change of batch size brought by increasing gpu numbers. Would you please give some suggestions on how to adjust learning rate in the multi-gpu mode?</p>

albanD: <p>The gradients are always summed, never averaged.<br>
To adapt the learning rate, I would say it’s not as easy as adapting linearly to the learning rate. Larger batch sizes will reduce the noise in your gradients as well, so you might need to change your learning rate schedule to make larger steps?<br>
I don’t think there is a one true rule on how to change the learning rate when you change the batch size unfortunately.</p>

dio_din: <p>The <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">doc</a> says:</p>
<blockquote>
<p>When this is done, <strong>averaged</strong> gradients are written to the  <code>param.grad</code>  field of all parameters.</p>
</blockquote>

albanD: <p>Hi,</p>
<p>Indeed, the DDP is doing something special here. It actually averages the gradients computed from each worker with equal weights.</p>

dio_din: <p>So, is the weights 1/world_size or 1? Could you please provide a link to the code line?</p>

albanD: <p>1/world_size the code is <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L357">here</a>.</p>
How to ensure GPU memory is freed at the end of a program?

TinfoilHat0: <p>Hi,</p>
<p>I’m using a bash script to do grid-search for my pytorch code.<br>
Something like as follows:</p>
<pre><code class="lang-auto">for lr in 0.1 0.01 0.001
do
    for wd in 0.1 0.01 0.001
    do
       # call pytorch script for the current setting of hyperparams
       python my_pytorch_script.py --lr=$lr --wd=$wd
    done
done
</code></pre>
<p>I noticed that, sometimes, a process finishes without freeing the GPU memory. It eventually causes the GPU to go OOM even though I execute each program sequentially.</p>
<p>How can I fix this?</p>

donJuan: <p>maybe some gpu cache memory is not emptied? check torch.cuda.empty_cache()</p>

ptrblck: <p>Was each script execution correctly exited or did you see some errors?<br>
If the GPU memory is not released after finishing the execution, some zombie processes might be alive.<br>
Could you check it via <code>htop</code> or <code>ps</code>?</p>

TinfoilHat0: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Indeed there seems to be zombie processes. Btw, previously called scripts run fine however, latter ones yield an error due to GPU running OOM.</p>
What is the relationship between Torch Script and TensorRT

Alex_Luya: <p>Is possible to use <strong><a href="https://github.com/NVIDIA-AI-IOT/torch2trt" rel="nofollow noopener">torch2trt</a></strong> to convert trained model to TensorRT,then convert the result to TorchScript?</p>

ptrblck: <p>Usually TensorRT would be the last deployment stage.<br>
What is your use case that you would like to transform a TensorRT model back to TorchScript?</p>

Alex_Luya: <p>We have a  c++ based system,and I am not sure TensorRT-ed model can work with C++</p>

ptrblck: <p>TensorRT is a C++ library (with a Python interface) as described <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#python_example_unsupported">here</a>, so your model should work in C++.</p>

Alex_Luya: <p>That will be fine,thank you</p>

solarflarefx: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> So when would TorchScript be used if not as a deployment solution?  From my understanding most do training in Python but may want to run an optimized inference in another language like C++.  I thought this was the function of TorchScript (to run an inference in C++), though I definitely see overlap with TensorRT.  I believe that TensorRT has many more optimizations for runtime inference.  But there may also be some things that TorchScript can do over TensorRT that I am not aware of.  I’m still not 100% sure when to use one over the other.</p>

ptrblck: <p>A scripted model could still be used for training, no?<br>
The major use case might be inference, but I also see the advantage of “porting” the model from Python to C++, where fine tuning the model would also be possible.</p>
<p>Yes, TensorRT is an inference engine and you could use it to speed up your inference further.<br>
Have a look at <a href="https://devblogs.nvidia.com/how-to-deploy-real-time-text-to-speech-applications-on-gpus-using-tensorrt/">this blog post</a> for more information.</p>
How to give different L2 regularization factors on different parameters?

joshua2: <p>Hi, I’m a beginner at Pytorch. I figured out how to give L2 regularization, using weight_decay arguments of optimizers.</p>
<pre><code class="lang-auto">optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9,weight_decay=1e-4)
</code></pre>
<p>However, I’m not sure how to give different L2 regularization factors to different parameters. Suppose the model has two convolution layers, and give 1e-4 &amp; 2e-4 in each layer’s weight.<br>
Then should I have to use optimizer1 (first layer params) &amp; optimizer2 (second layer params)? or is there any way to exclude L2 regularization in specific parameters, and give different L2 factors?</p>

googlebot: <p>See <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#per-parameter-options</a>, there are no facilities to create parameter groups though <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Backpropogation Through Time with looping statements in `forward()`

Vigneswaran: <p>Hi community, lately I am having trouble in understanding the relationship between looping statements inside <code>forward()</code> function and Backpropagation Through Time (BPTT).</p>
<p>In brief, I would like to create a custom recurrent model, similar to RNNs/LSTMs, in which the <code>forward()</code> would get sequential input data (of shape: (sequence length, batch size, input dimension)) and process the data by looping over each sequence, something similar to,</p>
<pre><code class="lang-python">def forward(seq_data, hidden_states):
    for sequence in seq_data:
        (some simple operations ...)
        out = torch.sigmoid(seq_data) + hidden_states
        hidden_states = torch.tanh(out)
        .....
    return out, hidden_states
</code></pre>
<p>Now, does PyTorch automatically detect sequential processing in forward() and perform BPTT on its own when doing <code>loss.backward()</code> (i.e. without writing backward())?</p>
<p>Any insights and corrections are much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

googlebot: <aside class="quote no-group" data-username="Vigneswaran" data-post="1" data-topic="78801">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vigneswaran/48/12847_2.png" class="avatar"> Vigneswaran:</div>
<blockquote>
<p>Now, does PyTorch automatically detect sequential processing in forward() and perform BPTT on its own when doing <code>loss.backward()</code> (i.e. without writing backward())?</p>
</blockquote>
</aside>
<p>No, nothing is “detected”, but backprop works for implementations without inplace operations (that usually accumulate outputs in a python list first). There was a blog post on this site about how JIT speeds up rnn unrolling, but c++ coded rnns are faster. You may check <a href="https://github.com/asappresearch/sru" rel="noopener nofollow ugc">SRU</a> as an example that has python and c++ implementations.</p>
Return vector but take loss function on element

janistrov: <p>Hi all,</p>
<p>I have a weird problem (so I guess). I want to train a model many times on different elements of the output. This works only if I output the respective element directly and it doesn’t work if I output a vector with all elements and calculate the loss only on one.<br>
Lets say I have an output tensor r with size 1x4 and I’m interested in the loss on the last element r[3].</p>
<p>If I return from my forward path:</p>
<p>return r[3]</p>
<p>And then train:</p>
<p>y_hat = model(X)<br>
loss = criterion(y_hat, y)</p>
<p>The predictions on the test set are perfectly fine. But if i return all elements:</p>
<p>return r</p>
<p>And train:</p>
<p>y_hat = model(X)<br>
loss = criterion(y_hat[3], y)</p>
<p>In the ladder case the prediction is way off. In debugging mode, I see that the value which is passed to the loss function is in both cases exactly the same. Do I miss something here?</p>
<p>Thank you very much for your help!<br>
Greetings</p>

ptrblck: <p>This shouldn’t make any difference as seen here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(10, 10, bias=False)
        self.fc2 = nn.Linear(10, 3, bias=False)
        
    def forward(self, x, return_slice=False):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        if return_slice:
            return x[:, -1]
        else:
            return x

model = MyModel()
x = torch.randn(10, 10)

# Complete batch approach
out = model(x)
out[:, -1].mean().backward()

for name, param in model.named_parameters():
    print(name, param.grad.abs().sum())

# Sliced approach
model.zero_grad()
out = model(x, return_slice=True)
out.mean().backward()

for name, param in model.named_parameters():
    print(name, param.grad.abs().sum())
</code></pre>
<p>You will get the same gradients for both methods.</p>
<p>How reproducible is this issue using your code?<br>
I.e. are you always getting the good results in the former approach and bad ones in the latter?</p>

janistrov: <p>Thank you for your help, it made me search at other places and in the end I found a mistake in the testing code. It all works fine now.</p>
Python setup.py install is not created lib.linux-armv7l-3.7 for arm architecture while cross compiling from linux x86

aaripakaprashad: <p>cross compiling for arm architecture, "python setup.py install " observes  lib.linux-armv7l-3.7 is not created .</p>
<p>but found arm compiled libraries under “pytroch/lib/”</p>

(base) prashad@ubuntu:~/b/mypytorch/lib$ file libshm.so 
libshm.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=fbbdfe380aaf1de9035d6f7ee05a7f4907e66eb8, with debug_info, not stripped
(base) prashad@ubuntu:~/b/mypytorch/lib$ 

<p>is this known issue for cross compiling</p>
PyTorch Cross-platform Development (Android & iOS)

Untitled: <p>Hi everyone !<br>
I’m planning to do an app that will be using a PyTorch model. I would like this app to be compatible with both Android and iOS (like a React Native app).<br>
What are the best solutions to do that (considering the use of a PyTorch model) ?</p>

Untitled: <p>Up! Is there any way to do a React Native App with a working PyTorch model inside ?</p>

Joachim: <p>I have not used PyTorch with React Native, but what you can do is to write your model using PyTorch, adapt it to ONNX (for Android) and CoreML (for iOS). You can export to ONNX directly from PyTorch, and you can convert from ONNX to CoreML using <a href="https://github.com/onnx/onnx-coreml" rel="nofollow noopener">https://github.com/onnx/onnx-coreml</a> .</p>

Untitled: <p>And so you think that if I’m detecting the OS inside the React Native app and then I use the proper model (ONNX for Android, CoreML for iOS) it could potentially work ?</p>

Joachim: <p>I would presume so, yes. I have managed to get a TensorFlow model working on Android / iOS using a very similar approach. I have to imagine it would have been possible with PyTorch as well. I do need to stress that I am not not up-to-date with PyTorch and mobile devices, so it is entirely possible that there exists an easier approach. I encourage you to do further research on your own just to be sure. However, checking the ONNX repository, this example should get you started: <a href="https://github.com/onnx/tutorials/blob/master/tutorials/PytorchCaffe2MobileSqueezeNet.ipynb" rel="nofollow noopener">https://github.com/onnx/tutorials/blob/master/tutorials/PytorchCaffe2MobileSqueezeNet.ipynb</a></p>

Untitled: <p>Okay, thank you very much!</p>
Could I use 'torch.utils.tensorboard' to see the compute time and memory of each step in model? If yes, how to do it? Are there any other ways to do?

AugF: <p>I want to check the step’s compute time and memory of model,  and I find tensorboard in tensorflow can do it. But I run the document in official website (<a href="https://pytorch.org/docs/stable/tensorboard.html?highlight=tensorboard" rel="nofollow noopener">https://pytorch.org/docs/stable/tensorboard.html?highlight=tensorboard</a>), it’s not clearly. Could I use it to see network graph’s compute time and memory? If yes, how to do it?</p>
isInt() INTERNAL ASSERT FAILED

Joachim: <p>Hi</p>
<p>I am receiving the following <code>RuntimeError</code></p>
<pre><code class="lang-auto">RuntimeError: isInt() INTERNAL ASSERT FAILED at /tmp/pip-req-build-xd7oc4a9/aten/src/ATen/core/ivalue.h:221, please report a bug to PyTorch. 
The above operation failed in interpreter, with the following stack trace:
at /network_extended.py:49:30
        outputs = []
        for i in range(len(inputs)):
            state = self.cell(inputs[i], state)
                              ~~~~~~~~ &lt;--- HERE
            outputs += [state]
        return torch.stack(outputs)
</code></pre>
<p>when attempting to train on a machine with <a href="https://gist.githubusercontent.com/x775/99521adbb10f80b05b866a729b4b9957/raw/4c0c01f8d5ff7c3036e8ca092b125c5c2d1ad571/environmentdump" rel="nofollow noopener">this environment</a>. Note that I am executing in a Singularity container built using Nvidia’s <code>pytorch_20.01-py3</code> image. As such, whilst the Tesla V100 allocated to the system appears unavailable in the above dump, it is very much available and works with other PyTorch models trained the same way.</p>
<p>However, curiously, when executing the exact same code on an older Windows 8.1 machine with <a href="https://gist.githubusercontent.com/x775/b1485ecb4b0cd10b7f87c9da349569a7/raw/e5932c3473829036c6faa50dcb99ac748b623377/environmentdump2" rel="nofollow noopener">this environment</a>, everything works as expected. I can thus only assume that it is due to the specific environment on my Linux machine as opposed to my code.</p>
<p>I have for this reason opted for writing here as opposed to submitting an issue. I am naturally happy to close this thread and submit an issue with additional information if that is more appropriate.</p>
<p>Any thoughts on what this could be are greatly appreciated!</p>
<p>Thanks!</p>

ptrblck: <p>Could you wipe the PyTorch installation inside the container via <code>pip uninstall torch -y</code> and install the latest binary?<br>
Besides the OS change, PyTorch is also in another version (<code>1.4.0</code> in Windows, while <code>20.01</code> uses a pre-1.4.0 build), so this might have been a known issue, was could have been fixed in <code>1.4.0</code>.</p>

Joachim: <p>Thanks for your response. It was indeed an issue with the pre-1.4.0 build. Upgrading the entire Singularity container using the <code>pytorch_20.03-py3</code> image resolved the issue. Thanks!</p>
Thnn_fused_gru_cell does not exist

bridgetteSong: <p>When I export onnx model using GRUCell, I got a warning:</p>
<pre><code class="lang-auto">UserWarning: ONNX export failed on ATen operator _thnn_fused_gru_cell because torch.onnx.symbolic_opset10._thnn_fused_gru_cell does not exist
KeyError: '_thnn_fused_gru_cell'
</code></pre>
<p>and I failed to export, Anyone can give some advice?</p>