akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unused Parameters when using nn.TransformerEncoder

Semir_Elezovikj: <p>I am using nn.TransformerEncoder with a DDP wrapper - and getting an “unused parameters” error.</p>
<pre><code class="lang-auto">self.ego_encoder_layer = nn.TransformerEncoderLayer(d_model=2048, nhead=16, activation='gelu')
self.ego_transformer_encoder = nn.TransformerEncoder(self.ego_encoder_layer, num_layers=6)
</code></pre>
<p>And do a forward pass using:</p>
<pre><code class="lang-auto">ego_transformer_features = self.ego_transformer_encoder(ego_sequences, src_key_padding_mask=src_padding_mask)
</code></pre>
<p>I used the following code to print the unused parameters :</p>
<pre><code class="lang-auto">loss.backward()
print('-- FIND UNUSED PARAMETERS --')
for name, param in model.named_parameters():
    if param.grad is None:
        print(name)
print('--' * 30)
optimizer.step()
</code></pre>
<p>The unused parameters are listed below:</p>
<pre><code class="lang-auto">ego_encoder_layer.self_attn.in_proj_weight
ego_encoder_layer.self_attn.in_proj_bias
ego_encoder_layer.self_attn.out_proj.weight
ego_encoder_layer.self_attn.out_proj.bias
ego_encoder_layer.linear1.weight
ego_encoder_layer.linear1.bias
ego_encoder_layer.linear2.weight
ego_encoder_layer.linear2.bias
ego_encoder_layer.norm1.weight
ego_encoder_layer.norm1.bias
ego_encoder_layer.norm2.weight
ego_encoder_layer.norm2.bias
</code></pre>

Semir_Elezovikj: <p><a class="mention" href="/u/alband">@alband</a>:</p>
<pre><code class="lang-auto">self.ego_encoder_layer = nn.TransformerEncoderLayer(d_model=2048, nhead=16, activation='gelu')
self.ego_transformer_encoder = nn.TransformerEncoder(self.ego_encoder_layer, num_layers=6)
</code></pre>
<p>Removing the <code>self</code> designator in the <code>ego_transformer_encoder</code> resolved the issue? Not sure why:</p>
<pre><code class="lang-auto">ego_encoder_layer = nn.TransformerEncoderLayer(d_model=2048, nhead=16, activation='gelu')
self.ego_transformer_encoder = nn.TransformerEncoder(ego_encoder_layer, num_layers=6)
</code></pre>
How do we write CUDA kernels without writing CUDA today?

Enamex: <p>Tensor Comprehensions are discontinued (rather quietly). Triton seems too complicated? Or not, if you have experience to share on it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>So what do we use today like TC? Anything on the horizon, even?</p>
<p>Perhaps some documentation on guiding the JIT towards similar performance to writing fused kernels?</p>

ptrblck: <aside class="quote no-group" data-username="Enamex" data-post="1" data-topic="132519">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/enamex/48/3329_2.png" class="avatar"> Enamex:</div>
<blockquote>
<p>Perhaps some documentation on guiding the JIT towards similar performance to writing fused kernels?</p>
</blockquote>
</aside>
<p>There is indeed ongoing work in the different JIT backends, which use code generation to create (fused) kernels. I’m a bit familiar with the internals of the <code>nvfuser</code> work, but unfortunately cannot link to a proper documentation, as it’s still in an early stage.<br>
In any case, I believe to see more code generation approaches in the future, which would make writing custom operations easier in the framework <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

Enamex: <p>So the current plan is JIT all the way? No lower-level script language for kernels specifically (like TCs)?</p>
<p>And if it is going to be JIT-based, are there any plans/work for a static diagnostic that tells you how well certain parts of the source TorchScript have managed to generate fused low-level code?</p>
<p>Something like a function <code>summarize_compilation(script(model, method))</code> that returns a report on how certain sections fused or didn’t. Or something. You can definitely tell I’ve no idea about this area except that “fused–&gt;probably good” <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
What is the difference between torch.vmap and functorch?

kfxw: <p>Just out of curiosity.<br>
It seems that the torch.vmap in the unstable pytorch version has very similar functionality with the functorch. Do you have any suggestion on which one to use in practice?</p>

albanD: <p>Hi,</p>
<p>The torch.vmap works fine even though it lacks some kernels for perf.<br>
It is limited though on how you can compose it with the autograd engine.</p>
<p>functorch is a trying alleviate this limitation.</p>

kfxw: <p>Thanks! It’s a very helpful message!</p>
DQN saved model doesn't play correct

Victor: <p>I have the following dqn model:</p>
<pre><code class="lang-auto">class DQN(nn.Module):

    def __init__(self, input_shape, n_actions):
        super(DQN, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1),
            nn.ReLU()
        )

        conv_out_size = self._get_conv_out(input_shape)
        self.fc = nn.Sequential(
            nn.Linear(conv_out_size, 512),
            nn.ReLU(),
            nn.Linear(512, n_actions)
        )

    def _get_conv_out(self, shape):
        o = self.conv(Variable(torch.zeros(1, *shape)))
        return int(np.prod(o.size()))

    def forward(self, x):
        fx = x.float() / 256.0
        conv_out = self.conv(fx).view(fx.size()[0], -1)
        return self.fc(conv_out)
</code></pre>
<p>I trained it for Pong and got more than 18.0 mean reward for the last 100 games.<br>
It was saved:</p>
<pre><code class="lang-auto">torch.save(net.state_dict(), 'pong_model.pt')
</code></pre>
<p>But when I try to load it and play some games I get wrong results.</p>
<pre><code class="lang-auto">net = models.DQN(env.observation_space.shape, env.action_space.n)
net.load_state_dict(torch.load('pong_model.pt', map_location=lambda storage, loc: storage))
for i in range(20):
    state = env.reset()
    while True:
        env.render()
        action = agent(state, net)
        next_state, _, done, _ = env.step(action)
        if done:
            break
        state = next_state
</code></pre>
<p>It can’t win even once. What could be wrong?</p>

keerthanpg: <p>Can you post the code for ‘agent()’?</p>

Victor: <pre><code class="lang-auto">class EpsilonGreedyAgent(object):

    def __init__(self, actions_num):
        self.actions_num = actions_num
    
    def _get_state_value(self, state, model):
        state = np.expand_dims(state, 0)
        state = Variable(torch.from_numpy(state), volatile=False)
        if use_cuda:
            state = state.cuda()
        return model(state).data.cpu().numpy()

    def __call__(self, state, model, eps):
        state_value = self._get_state_value(state, model)
        sample = random.random()

        if sample &gt; eps:
            action = np.argmax(state_value)
        else:
            action = random.randrange(self.actions_num)
        return action
</code></pre>
<p>I found my mistake. I didn’t use all wrappers for environment when I was testing model.</p>

Ashwini_Magar: <p>Hi ,</p>
<p>I have trained model for robotic arm  in Gazebo simulation. But when I am loading that model, I can see all<br>
trained weights in place and model is not winning single time. How should I wrap Gazebo environment. Any suggestions.</p>
Onnx conversion - Dense layer instead of gemm

Yorgus: <p>Hi,<br>
lately I converted a pytorch model into onnx (please see model and conversion code below). It is a model with several Dense layers in a row. The model structure itself is garbage, please focus on the translation. The model is translated into a sequence of gemm, non-linearity and eltwise operations. I expected the onnx-model to contain Dense Layer. In fact, I need Dense layers for a tool that takes onnx as input. However, I was not able to generate it that way.</p>
<p>My question now is: How to convert the model into a sequence of Dense layers instead?</p>
<p>The shape of input_for_onnx is “torch.Size([75])”. I have tried dummy inputs as well with shape (1,75).</p>
<pre><code class="lang-auto">class vanilla(nn.Module):
    def __init__(self):
        super(vanilla,self).__init__()
        #self.bn = nn.BatchNorm1d(num_features=75)
        self.linear1 = Linear(75,75)
        self.linear2 = Linear(75,75)
        ...
        self.linear18 = Linear(75,75)
        self.linear19 = Linear(75, starting_neurons)
        self.linear20 = Linear(starting_neurons, 43)

    def forward(self, X):
        origin = X
        X = F.relu(self.linear1(X))
        X = F.relu(self.linear2(X))
        X = F.relu(self.linear3(X))
        X += origin
        X = F.relu(self.linear4(X))))
        ...
        X = F.relu(self.linear19(X))
        X = self.linear20(X)
        return X
        
</code></pre>
<pre><code class="lang-auto">batch_size= 1

torch.onnx.export(model.cpu(),                     # model being run
                      input_for_onnx.cpu(),         # model input (or a tuple for multiple inputs)
                      "gtsrb.onnx",   # where to save the model (can be a file or file-like object)
                      export_params=True,        # store the trained parameter weights inside the model file
                      opset_version=10,          # the ONNX version to export the model to
                      do_constant_folding=False,  # whether to execute constant folding for optimization
                      input_names = ['input'],   # the model's input names
                      output_names = ['output'], # the model's output names
                      dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                    'output' : {0 : 'batch_size'}})

</code></pre>
<p>PS: I posted in onnx github but was redirected to pytorch forum, as this is not a matter of onnx itself, but how the model is translated by pytorch.</p>
The results are not correct when load_state_dict on same model

Amadeus-AI: <p>I try to validate several state dict, let’s say A and B.<br>
At first, I do it in the following order:</p>
<blockquote>
<p>net = Net()<br>
net = net.to(device)<br>
net = nn.DataParallel(net, device_ids = [0])<br>
net.load_state_dict(torch.load(A))<br>
torch.no_grad()<br>
net.eval()<br>
retA1 = net(input)<br>
net = Net()<br>
net = net.to(device)<br>
net = nn.DataParallel(net, device_ids = [0])<br>
net.load_state_dict(torch.load(B))<br>
torch.no_grad()<br>
net.eval()<br>
retB1 = net(input)</p>
</blockquote>
<p>The answer retA1 and retB1 is correct.</p>
<p>And I try the following order:</p>
<blockquote>
<p>net = Net()<br>
net = net.to(device)<br>
net = nn.DataParallel(net, device_ids = [0])<br>
net.load_state_dict(torch.load(A))<br>
torch.no_grad()<br>
net.eval()<br>
retA2 = net(input)<br>
net.load_state_dict(torch.load(B))<br>
torch.no_grad()<br>
net.eval()<br>
retB2 = net(input)</p>
</blockquote>
<p>retA2 is same as retA1, but retB2 is not same as retB1.</p>
<p>Is this behavior expected or I did something wrong?</p>
Why those tensors are different?

mathwseg: <p>what is the difference between those tensor’s shapes?</p>
<pre><code class="lang-auto">Tesnor tensor([[-6.3570e-01, -9.6387e-01,  9.9367e-01, -1.2536e+00,  1.1033e+00,
         -1.4880e+00, -5.9717e-01,  6.9657e-01, -2.1082e-01, -1.3269e-02,
          7.7475e-01,  8.2461e-01, -4.4550e-01, -3.7213e-01, -2.5947e-02,
          1.1908e+00,  9.8890e-01,  4.9845e-01, -6.2397e-01, -7.4751e-01,
         -3.7960e-01, -1.0479e+00,  9.1659e-01,  2.0286e-01, -3.0425e+00,
          8.9924e-01,  4.9533e-01,  1.2649e-01, -7.8557e-01, -1.9877e+00,
          1.5009e+00,  6.3702e-01,  6.0425e-01, -2.5740e-01, -8.3749e-01, ....  -9.3429e-01,  3.5320e-01,
         -1.0368e-01, -6.1338e-01]], grad_fn=&lt;SelectBackward&gt;
</code></pre>
<p>and</p>
<pre><code class="lang-auto">
tensor([[0.1703, 0.1442, 0.0332,  ..., 0.1482, 0.0598, 0.1676]],
       grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>this is for two different models with the same image but does this different form structure of the model or can I control the shape ?</p>
Reverse of torch.clamp

Mriganka_Nath: <p>Torch.clamp clamps a tensor in the range of [min,max].</p>
<p>I want to do the reverse action.<br>
For example a tensor of shape (N x M), and given some min and max, I want <strong>all the values in the tensor in the range [min,max] to be zero</strong>.</p>
<p>How to do that?</p>

Mriganka_Nath: <p>Just had to do this<br>
tensor[tensor==torch.clamp(tensor, min , max)]=0<br>
sorry for my foolish question</p>
Can't init a tensor with `randint` while `requires_grad=True`

manhdung20112000: <p><strong>Describe</strong><br>
Recently, I found that I can not initialize a random int tensor with <code>randint</code> while <code>requires_grad</code> is set to True.</p>
<p><strong>To Reproduce</strong></p>
<pre><code class="lang-python">torch.randint(1, (10, 10), requires_grad=True)
</code></pre>
<p><code>RuntimeError: Only Tensors of floating point and complex dtype can require gradients</code></p>
<p><strong>Additional information</strong></p>
<ul>
<li>Windows 10</li>
<li>
<code>torch</code> version 1.9.0</li>
<li>
<code>python</code> version 3.9.6</li>
</ul>

manhdung20112000: <p>I don’t think this is my mistake in the usage way.<br>
If this is a real bug, can I raise an issue in github torch repository? Thanks.</p>

ptrblck: <p>This is not a bug and expected behavior. As the error message explains, only floating point types (and complex types, which are also using floating point types internally) can require gradients.</p>
RuntimeError: Given groups=1, weight of size [244, 244, 3, 3], expected input[8, 1, 244, 244] to have 244 channels, but got 1 channels instead

kasidkhansbp: <p>I used Pytorch transform to transform the data and Pytorch model to train them. But, I am geting the error message as “RuntimeError: Given groups=1, weight of size [244, 244, 3, 3], expected input[8, 1, 244, 244] to have 244 channels, but got 1 channels instead”.</p>
<p>Any suggestion on how to fix weight shape to accept the input?</p>
<p>Note:<br>
I had converted the image to grayScale while tranforms.</p>
<pre><code class="lang-auto">train_transforms = transforms.Compose([
             transforms.ToPILImage(),
             transforms.Grayscale(num_output_channels=1),
             transforms.Resize((244,244)),
             transforms.RandomHorizontalFlip(),
             transforms.ToTensor(),
             transforms.Normalize((0.5,), (0.5,))
             ])

train_df = CustomImageDataset(train_data,train_transforms)
train_dataloader = torch.utils.data.DataLoader(
            train_df,
            batch_size=8,
            num_workers=1,
            shuffle=True,
        )

train_features, train_labels = next(iter(train_dataloader))
print(f"Feature batch shape: {train_features.size()}")
# this code output: Feature batch shape: torch.Size([8, 1, 244, 244])

</code></pre>
<pre><code class="lang-auto">import torch
from torch import nn
from d2l import torch as d2l

net = nn.Sequential(
    nn.Conv2d(244, 244, kernel_size=3, stride=1, padding=1), nn.ReLU(),
    nn.Dropout(p=0.5),
    nn.MaxPool2d(kernel_size=3, stride=1),
    nn.Dropout(p=0.5),
    nn.Conv2d(244, 256, kernel_size=3, padding=1), nn.ReLU(),
    nn.Dropout(p=0.5),
    nn.MaxPool2d(kernel_size=3, stride=1),
    nn.Dropout(p=0.5),
    nn.Conv2d(256, 256, kernel_size=3, padding=1), nn.ReLU(),
    nn.Linear(256, 3))

lr, num_epochs = 0.01, 10
d2l.train_ch6(net, train_dataloader, test_dataloader, num_epochs, lr, d2l.try_gpu())
</code></pre>

ptrblck: <p>The first conv layer:</p>
<pre><code class="lang-python">nn.Conv2d(244, 244, kernel_size=3, stride=1, padding=1)
</code></pre>
<p>expects the input to have <code>244</code> channels, since <code>in_channels=224</code>.<br>
If you are passing an input with a single channel, change <code>in_channels</code> to <code>1</code>:</p>
<pre><code class="lang-python">nn.Conv2d(1, 244, kernel_size=3, stride=1, padding=1)
</code></pre>

kasidkhansbp: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  I just changed it and it is thowing an error “RuntimeError: mat1 and mat2 shapes cannot be multiplied (491520x240 and 256x3)”. Any idea why it is doing it?</p>

ptrblck: <p>This error is most likely raised in the linear layer.<br>
Flatten the activation via <code>nn.Flatten()</code> after the last conv layer:</p>
<pre><code class="lang-python">    nn.Conv2d(256, 256, kernel_size=3, padding=1), nn.ReLU(),
    nn.Linear(256, 3))
</code></pre>
<p>and make sure the <code>in_features</code> of <code>nn.Linear</code> match the activation features.</p>
Is it possible to use rtx 3080 gpu with pytorch 1.6

JamesDickens: <p>I have an rtx 3080 gpu, using Windows 10. Is it possible to use this gpu with<br>
pytorch version 1.6 and any version (as necessary) of CUDA?</p>
<p>The reason I am asking is using the command<br>
conda install pytorch torchvision cudatoolkit=11 -c pytorch-nightly<br>
(recommended from the github issue <a href="https://github.com/pytorch/pytorch/issues/45028" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/45028</a>)<br>
resulted in an install of a 1.8 dev nightly build that was having errors I was not encountering previously with the torchvision.models.detection repository where they<br>
seemed to have added a new file anchor_utils.py . The authors seem to have refactored the anchor generator to its own file I guess, where previously the anchor_generator class was in rpn.py.</p>
<p>Thanks</p>

ptrblck: <p>You could use the <code>1.6</code> binaries with e.g. CUDA10.2, which will then however JIT compile the CUDA kernels in the first CUDA operation and will thus add a massive overhead to your runs, so I would recommend to adapt the source code and use the nightly binaries with CUDA11.</p>

JamesDickens: <p>What is the latest stable build of pytorch that supports RTX 3080/ CUDA 11?</p>

ptrblck: <p><code>1.7.0</code> is the latest stable release, which ships with CUDA11.0.</p>

Glory_Chen: <p>Yeah. So it can be “interpreted” as RTX 30 Series are still not well compatible with relative “old” PyTorch versions (&lt; 1.7.0). Am I right?</p>

ptrblck: <p>Yes, and since the framework is not forwards compatible the CUDA11 support will not be backported to already released versions.</p>
How can I crop away a tensor's constant value padding (padding height and width are the same) with an unknown padding value and size?

ProGamerGov: <p>How can I crop away a tensor’s constant value padding (padding height and width are the same) with an unknown value and size?</p>
<p>I would think that because the padding surrounding my tensor has a constant value, and the same height / width, that it should be possible to know where to crop the tensor to remove the padding.</p>
<pre><code class="lang-auto">import torch

# Test tensor with NCHW dimensions
a = torch.randn(1,4,5,5) # Can have any H &amp; W size

# Create padding for testing
b = torch.nn.functional.pad(a, (2,2,2,2), 'constant', value=1.2) # Value can be any number

c = # equal to a, without being able to use the variables a or b (or their argument values)
</code></pre>

ptrblck: <p>It depends a bit on the padding itself. E.g. if you know that the padding will be applied to all spatial sides with a constant value, you could just grab the very first value, and remove it via indexing. However, since you cannot access <code>a</code> and <code>b</code>, I assume you know the padding size?<br>
If so, you could index <code>a</code> using this size, but again you are apparently not allowed to use <code>a</code>.<br>
Could you clarify this restriction a bit, since you won’t be able to create <code>c</code> without using <code>a</code> or <code>b</code> at all?</p>

John_Grabner: <p>I have a similar, maybe the same problem.</p>
<p>An image_hw with values between 0.0 and 1.0, where the non-zero pixels are somewhere in the center (i.e. unknown padding on left, right, top, and bottom, maybe no padding).</p>
<p>An example image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/11385d19302f7aed195bfb419cc367cbebd5b309.png" alt="not_trimmed_mask_1_0_11" data-base62-sha1="2skShMCz8MmIbmNFYjBE9hDi7c5" width="113" height="98"></p>
<p>I need a bounding box coordinates so that is can rectangle crop center region with non-zero pixels. image_hw[top:bottom, left:right]</p>
<p>I can do this with 4 for loops, similar to:</p>
<pre><code class="lang-auto">for top_pad in range(image_hw.shape[0];
   if image_hw[top_pad,:].sum()&gt;0:
      break;
</code></pre>
<p>I suspect pytorch and a function that could accomplish this more concisely.</p>

ptrblck: <p>I think you could use <code>nonzero()</code> on the image and grab the min. and max. values as see here:</p>
<pre><code class="lang-python">x = torch.zeros(24, 24)
x[3:7, 5:9] = 1.

idx = x.nonzero()
x_min = idx[:, 0].min()
x_max = idx[:, 0].max()
y_min = idx[:, 1].min()
y_max = idx[:, 1].max()
</code></pre>
Access to a single step of a conv2d

Ilaria_Bragaglia: <p>Hi, i want to acces to a single result of the conv2d Pytorch. In particular for a certain set of neurons, i would like to repeat the computation kernel*input feature map.<br>
i try to recupute manually each multiplication and then the summation but the program takes too much time w.r.t the one without the repeatition.<br>
Which function can i use to doo this quickly? From the Pytorch documentation i show the ‘group’ parameter for Cov2d , but i don’t understand whether this can help me.</p>

ptrblck: <p>I’m not sure what your actual use case is, but if you are trying to implement a convolution “manually” via a matrix multiplication, take a look at <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html?highlight=unfold#torch.nn.Unfold"><code>nn.Unfold</code></a>, which gives an example of a conv in the docs.</p>

Ilaria_Bragaglia: <p>Can you explain to me what is the role of ‘groups’ for Conv2d? Thaks so much</p>

ptrblck: <p>By default a conv kernel will use <em>all</em> input channels of the incoming activation to compute the activation map. If you use the <code>groups</code> argument, the kernel’s channels as well as the input channels will be “split” so that the convolution operation would be equivalent to having multiple conv layers side by side which process only the corresponding subset of the input activation. The result will be concatenated afterwards.</p>
<p>E.g. if <code>groups=in_channels</code>, each channel of the input activation would get its own set of conv filters.</p>
Tesnor operation with multiple condition indexing involving two tensors

7029279: <p>I have tensors <code>a_angle</code> and <code> b_angle</code> each containing 0.0-1.0 values representing 0-360 degrees.</p>
<p>I want to prevent a case where distance between two points become larger than the actual distance due to the discontinuity around 0 degree.<br>
To this end, I am addting or subtracting 1 (360 degree) when the distance between two points cross over 0.<br>
I have the following code, but it is very slow.<br>
Is there anyway to do this more efficiently? Perhaps I should use masks of indices or torch.where but not sure how to perform that computation  in my use case.</p>
<pre><code class="lang-auto">for i in range(len(a_angle)):
        if b_angle[i] &gt; 0.5 and  0.5 &gt; a_angle[i] and b_angle[i]-a_angle[i] &gt; 0.5:
            b_angle[i] = torch.add(b_angle[i], -1)
        elif a_angle[i] &gt;0.5 and 0.5 &gt; b_angle[i] and a_angle[i]-b_angle[i] &gt; 0.5:
            a_angle[i] = torch.add(a_angle[i], -1)
</code></pre>

chetan_patil: <p>You can use this code</p>
<pre><code class="lang-auto">condition1 = ((b&gt;0.5) &amp; (0.5&gt;a) &amp; ((b-a)&gt;0.5))
b[condition1]-=1

condition2 = ((a&gt;0.5) &amp; (0.5&gt;b) &amp; ((a-b)&gt;0.5))
a[condition2]-=1
</code></pre>

7029279: <aside class="quote no-group" data-username="chetan_patil" data-post="2" data-topic="85297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chetan_patil/48/25122_2.png" class="avatar"> chetan_patil:</div>
<blockquote>
<pre><code class="lang-auto">condition1 = ((b&gt;0.5) &amp; (0.5&gt;a) &amp; ((b-a)&gt;0.5))
b[condition1]-=1

condition2 = ((a&gt;0.5) &amp; (0.5&gt;b) &amp; ((a-b)&gt;0.5))
a[condition2]-=1
</code></pre>
</blockquote>
</aside>
<p>I did not know that you can pass conditions like that to a tensor. Let me give it a try! Thank you so much.</p>

7029279: <p>wroking like a charm. Thank you so much!!!</p>

Amik-tj: <p>Can anyone help me to convert this code using pytorch tensor operation</p>
<pre><code class="lang-auto">import torch

outputs = torch.rand(3,5)
labels = torch.randint(0, 4, (3,))

def loss_swap(outputs, labels):
    _, predicted = outputs.max(0)
    correct = predicted.eq(labels)
    for i in range(correct.shape[0]):
        if correct[i].item() == False:
            c_index = labels[i].item()
            p_index = predicted[i].item()
            tmp1, tmp2 = outputs[i, p_index].item(), outputs[i, c_index].item()
            outputs[i, c_index] = tmp1
            outputs[i, p_index] = tmp2
    return outputs
</code></pre>
How to access different layers of a sequential model?

Kai123: <p>Hi all…I have a deep sequential CNN model consisting of Conv, relu, batchNorm, maxPool stacked one after another. I was thinking if I could access the layers(like indexing) through a loop or by some other mechanism, so that I can perform some different operation to different layers.</p>
<p>Thank you</p>

zetyquickly: <p>Hi <a class="mention" href="/u/kai123">@Kai123</a></p>
<p>To get an item of the <code>Sequential</code> use square brackets. You can even slice <code>Sequential</code></p>
<pre><code class="lang-auto">import torch.nn as nn

my_model = nn.Sequential(nn.Identity(), nn.Identity(), nn.Identity())
print(my_model[0:2])

</code></pre>
Questions about torch.cuda.amp

Ford: <p>To train my model using less GPU memory, I tried to use the “amp” package. However,an error occurred when importing this package, which is:</p>
<p><code> "ModuleNotFoundError: No module named 'torch.cuda.amp.autocast'".</code></p>
<p>My pytorch verision is 1.8.1, and the codes for importting the package are written as:</p>
<pre><code class="lang-auto">import torch.cuda.amp.autocast
import torch.cuda.amp.GradScaler
</code></pre>
<p>I wonder why this error occurred, and how to solve it. Thanks</p>

ptrblck: <p>You could directly use these methods and objects via:</p>
<pre><code class="lang-python">import torch
torch.cuda.amp.autocast
torch.cuda.amp.GradScaler
</code></pre>
<p>or could import them via:</p>
<pre><code class="lang-python">from torch.cuda.amp import autocast
from torch.cuda.amp import GradScaler
</code></pre>
Build Pytorch from source with cuda 9.2 or lower version on windows 10

somex: <p>I have Cuda 9.2</p>
<p>while building i am getting</p>
<pre><code class="lang-auto">-- Caffe2: CUDA toolkit directory: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2
CMake Error at cmake/public/cuda.cmake:42 (message):
  PyTorch requires CUDA 10.2 or above.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1191 (include)
  CMakeLists.txt:664 (include)
</code></pre>
<p>I want to build it with cuda 9.2 only how to proceed?</p>
<p>on git page I was not able to find any relevant document.</p>

ptrblck: <p>You would need to checkout an older branch, as the current master needs CUDA&gt;=10.2 as seen <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>
Applying random transformations with probability p=1

hussain: <p>Hi all,</p>
<p>Do applying Random Transformation (flip, crop rotate or any etc.) with p=1 means it is applied to all tensors/PIL images that are passed?</p>
<p>OR</p>
<p>it is like, if p=1 then any randomly chosen PIL image/Tensor will be be transformed because it is now selected and those skipped tensors/PIL images will be preserved as they were before?</p>
<p>I am confused in this command, please help me out.</p>
<p>Thanks in advance.</p>

hussain: <p>dear <a class="mention" href="/u/ptrblck">@ptrblck</a>, please guide me through this.</p>

ptrblck: <p>The probability <code>p</code> will be used to determine, if the current image passed through the transformation will be transformed or not as seen e.g. <a href="https://github.com/pytorch/vision/blob/74de51d6d478e289135d9274e6af550a9bfba137/torchvision/transforms/transforms.py#L615-L617">here</a>.</p>

sssingh: <p>why not all Random transforms have probability (p) argument?<br>
For example if I use RandomRotation(10), does this mean p=1 is explicitly assumed and rotation will be applied between -10 to 10 degrees to ALL images?</p>

ptrblck: <p>Yes, <code>RandomRotation</code> will transform all inputs and uses a uniform probability to sample the angle in the specified range. <code>Random*Flip</code> uses the passed probability to apply the flipping or to keep the original image, so both are using a probability in a similar sense where the former is uniform and the latter is binary. If you want to randomly apply a random transformation (such as <code>RandomRotation</code>), you could check <a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.RandomApply"><code>transforms.RandomApply</code></a>.</p>
Should I ever use non_blocking=False?

Vadim: <p>What is the reason for having the <code>non_blocking</code> flag in various functions that <a href="https://pytorch.org/docs/1.8.1/tensors.html#torch.Tensor.to" rel="noopener nofollow ugc">transfer</a> tensors between devices?    And why is it set to <code>False</code> by default?</p>
<p>It is my understanding that all GPU operations are completed asynchronously by default (and, in fact, this is vital for achieving a good performance).   So why does pytorch make an exception for transfers?</p>
<p>The only reasons for not using non_blocking that are mentioned in the docs are benchmarking and CUDA streams.  However these are pretty niche use-cases, and one should probably use an explicit synchronization for those anyway.</p>
<p>I’ve been spraying my code with <code>non_blocking=True</code> on assumption that this will never hurt and might occasionally speed things up.  Am I wrong about that?</p>

ptrblck: <p>For the async transfer you would need to use pinned memory, which is a limited system resource and which is thus the reason it’s not used by default.</p>

Vadim: <p>Does <code>non_blocking</code> force pinning?</p>
<p>Other answers on this forum (<a href="https://discuss.pytorch.org/t/slow-cpu-gpu-transfer/71935">1</a>, <a href="https://discuss.pytorch.org/t/why-set-cuda-non-blocking-false-for-target-variables/16943/2">2</a>, <a href="https://discuss.pytorch.org/t/should-we-set-non-blocking-to-true/38234/8">3</a>) seem to imply <code>non_blocking=True</code> <em>might</em> make the transfer asynchronous, if the CPU tensor is already pinned.  Otherwise, it’ll just silently turn into synchronous transfer.    But, in that case, there wouldn’t be any extra pinned memory used…<br>
What am I missing here?</p>

ptrblck: <p>No, it doesn’t use pinned memory automatically and is thus set to <code>False</code>. I would see this default as sticking to the “Zen of Python” - “<em>Explicit</em> is better than <em>implicit</em>”. Setting <code>non_blocking=True</code> and speculate for pinned memory sounds wrong.</p>

Vadim: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="133293">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would see this default as sticking to the “Zen of Python” - “<em>Explicit</em> is better than <em>implicit</em>”.</p>
</blockquote>
</aside>
<p>It seems to be falling short of this goal anyway.  If the idea was to avoid obscure performance problems caused by a distant change, torch should have thrown an exception in case async transfer is not possible.  Or at least printed a warning.</p>
Different input and output shapes of custom autograd.function forward

nihilistsumo: <p>It seems that when we write our custom autograd function, we have to make sure the tensor shapes of input and output in the forward function stay the same. Otherwise, it will cause RuntimeError while calling backward. Here is a minimal example:</p>
<pre><code class="lang-auto">class MyFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        y = torch.sum(x, dim=1)
        return y
    @staticmethod
    def backward(ctx, grad_output):
        g = grad_output * 2
        return g

f = MyFunc()
a = torch.randn((3, 4), requires_grad=True)
b = f.apply(a)
c = torch.sum(b)
c.backward()
</code></pre>
<p>This example will cause the following RuntimeError:</p>
<pre><code class="lang-auto">RuntimeError: Function MyFuncBackward returned an invalid gradient at index 0 - got [3] but expected shape compatible with [3, 4]
</code></pre>
<p>So does it mean in the forward function, input and output (x and y in the example) have to be of the same shape? What if I want to write a function that changes the input shape in the forward?</p>
Efficient way to get index of values from one tensor in another

YutongZheng: <p>Hi,<br>
I have two very large tensors <strong>a</strong> and <strong>b</strong>, each has about one million values.<br>
Now I want to see if the elements in <strong>b</strong> exist in <strong>a</strong>, and if yes, return their indices in <strong>a</strong>. I know the easiest way to implement is</p>
<pre><code class="lang-auto">exist_elements_id, index_in_a = b.view(-1, 1).eq(a).nonzero(as_tuple=True)
</code></pre>
<p>However, this will result in cuda out of memory (1TB memory) issue because the eq() result is too large.</p>
<p>Is there any other memory-efficient and fast way to implement this? Thanks!</p>

googlebot: <p>1)concatenate tensors into tensor C<br>
2)stable sort C on CPU, keep indices<br>
3)do torch.unique_consecutive, return counts &amp; indices<br>
4)select indices where count is 2<br>
5)map indices back to indices in C, due to stable sort they should only point to the first tensor (a)</p>
<p>disclaimer: I haven’t done this in practice</p>
Transfer part layers of weights/biases/optimizer to another neural network

Z_Huang: <p>Let’s say I have two cov neural networks (network A and network B) where the architectures (network A has 19 layers and network B has 20 layers) are almost identical.</p>
<p>How could  I transfer the first five layers of weights/biases/optimizer from network A to network B?</p>
<p>I need some help with coding. Thanks.</p>

ptrblck: <p>I think the cleanest approach would be to create a mapping between the layer names of <code>modelA</code> and <code>modelB</code> and load each <code>state_dict</code> of these layers separately as described <a href="https://discuss.pytorch.org/t/loading-pretrained-weights-into-new-model/104159/9">here</a>.<br>
This approach would need an explicit definition of which layers should be loaded and would not rely on e.g. using the <code>strict=False</code> argument, which might yield unexpected results if the output is ignored.</p>
How to extract moment estimates from torch.optim.Adam

blakebullwinkel: <p>The Adam weight update is defined as</p>
<p>w_{t+1} = w_t - \eta \left( \frac{1}{\sqrt{v_t} + \epsilon} \right) m_t</p>
<p>(see <a href="https://quicklatex.com/cache3/cf/ql_7cd36f4e1fa3681600c821ddabf8e2cf_l3.png" rel="noopener nofollow ugc">here</a> for rendered latex)</p>
<p>I would like to plot the “effective learning rate” of my optimizer during training, which requires knowing the moment estimates v_t and m_t. Does anyone know how I can extract these from an Adam optimizer?</p>

AlphaBetaGamma96: <p>You can grab these values by doing this,</p>
<pre><code class="lang-auto">def get_betas(optim):
  for group in optim.param_groups:
    return group['betas']
</code></pre>

blakebullwinkel: <p>These are just the beta1 and beta2 hyperparameters used to calculate running averages of the gradients, right? I want to know the actual moment estimates, which incorporate these betas along with the gradients.</p>

AlphaBetaGamma96: <p>By reading through the source (<a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html#Adam" class="inline-onebox" rel="noopener nofollow ugc">torch.optim.adam — PyTorch 1.9.1 documentation</a>), these values are defined as <code>exp_avg</code> and <code>exp_avg_sq</code>, you should be able to access via something like this,</p>
<pre><code class="lang-auto">for group in optim.param_groups:
  for p in group['params']:
    state = optim.state[p]
    print(state['exp_avg'], state['exp_avg_sq'])
</code></pre>

blakebullwinkel: <p>Thanks, I’ll try this</p>
Error: checkpointing is not compatible with .grad(), please use .backward() if possible

111282: <p>hi,<br>
I implement a rpc distributed model like <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-rnn-using-distributed-autograd-and-distributed-optimizer" rel="noopener nofollow ugc">the tutorial example</a>;</p>
<p>I run my model on one gpu and use checkpointing to reduce the usage of my gpu. After I get the loss, I do the backward using torch.distributed.autograd.backward(context_id, [loss]), then the error above occurs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e84e3d01a899d18265a64568dd170352c875a853.png" data-download-href="https://discuss.pytorch.org/uploads/default/e84e3d01a899d18265a64568dd170352c875a853" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e84e3d01a899d18265a64568dd170352c875a853_2_636x500.png" alt="image" data-base62-sha1="x94oFGW3weZKyyKf6gJ8Ah95Eif" width="636" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e84e3d01a899d18265a64568dd170352c875a853_2_636x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e84e3d01a899d18265a64568dd170352c875a853.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e84e3d01a899d18265a64568dd170352c875a853.png 2x" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">658×517 44.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t use torch.distributed.autograd.grad() anywhere, so I wonder if torch.distributed.autograd.backward() calls grad() internally? If not, I should have checked it again.</p>
<p>Thanks.</p>

Varal7: <p>Hi, yes distributed uses grad() internally.<br>
There are plans to make checkpointing compatible with distribute here: <a href="https://github.com/pytorch/pytorch/issues/65537" class="inline-onebox" rel="noopener nofollow ugc">New utils.checkpoint rollout. · Issue #65537 · pytorch/pytorch · GitHub</a></p>
Training issues

lifeblack: <p>Hi, I tend to learn a model that can map X=[400,100] to y=[400,180], in which each row of y has just one non-zero element = 1. A model with binary cross-entropy loss and adam optimizer could map that, but when I replace y by y=0.5*y, the model does not train at all and just returns a zero output (output=&gt; zeros([400,180]) ).<br>
Could you mind helping me fix this problem?</p>

ptrblck: <p>If I understand your use case correctly, you are struggling with getting good performance out of your model using soft labels and in particular if the target is set to <code>0.5</code>? If so, have a look at <a href="https://discuss.pytorch.org/t/cross-entropy-for-soft-label/16093/2">this post</a> which shows that the cost of binary crossentropy is non-zero for this use case.</p>

lifeblack: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your reply… As I understand from the post you mentioned, Binary cross-entropy is not good for my problem. As one of the replies suggested I changed the loss function to <strong>categorical_crossentropy</strong> and the activation function to sigmoid, it does not work either.(it distinguishes the place of the nonzero element, But with the wrong magnitude(0.98 instead of 0.5)) <strong>Could you mind helping me with how to fix it?</strong><br>
The aim of the model is to find out that the magnitude of the non-zero element in each output (each row of y ) which in this case is 0.5 and should find the place of the non-zero element.  I used the <strong>MSE</strong> but in the best case, it find the magnitude as 0.1 instead of 0.5.</p>

ptrblck: <p>I don’t know what the best approach would be and would have assumed that <code>nn.MSELoss</code> should work, so let’s wait for the experts in this topic.</p>
'int' object is not subscriptable error on using BucketIterator

JACOBIN-SCTCS: <p>I am getting the following error on using BucketIterator</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a1f03c22e48f55556596363b1a1636fa246150c.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a1f03c22e48f55556596363b1a1636fa246150c" title="Screenshot_20200126_121842"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a1f03c22e48f55556596363b1a1636fa246150c_2_690x217.png" alt="Screenshot_20200126_121842" data-base62-sha1="lZpYHEwZibBDl1YkNWXSmAu9CR6" width="690" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a1f03c22e48f55556596363b1a1636fa246150c_2_690x217.png, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a1f03c22e48f55556596363b1a1636fa246150c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a1f03c22e48f55556596363b1a1636fa246150c.png 2x" data-dominant-color="3C3D3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20200126_121842</span><span class="informations">941×296 34.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

zhangguanheng66: <p>Which dataset are you using in this case?</p>

JACOBIN-SCTCS: <p>Sorry sir for replying late .I was applying it on the aggregator label in WIkiSQL Dataset</p>

zhangguanheng66: <p>Could you copy/paste your code so I can reproduce? You could also open an issue in <a href="https://github.com/pytorch/text" rel="nofollow noopener">pytorch/text</a>.</p>
<p>In general, we would like to retire <code>iterator</code> in torchtext. Instead, the new abstraction uses <code>DataLoader</code> in <code>torch.utils.data</code>, like <a href="https://github.com/pytorch/text/blob/master/torchtext/datasets/text_classification.py" rel="nofollow noopener">text_classification_dataset</a></p>

JACOBIN-SCTCS: <p>Thank you for your concern and sorry again for replying late. It was a problem in the parameters to the BucketIterator . The error has been solved.  Thank you sir</p>

shahriarshayesteh: <p>Hi <a class="mention" href="/u/jacobin-sctcs">@JACOBIN-SCTCS</a>, Could share here how you addressed the problem? Thanks!</p>

furkat: <p>Here the datasets contain only one dataset which is train_set. However in my case I had three datasets-train, valid and test. So they are given in a tuple. Thus the batch size should also be given in a tuple(even in the case of the same batch size for all datasets). This way, I solved the problem.</p>

arminshef: <p>Actually, I know what was the problem. The ‘batch_size’ parameter in BucketIterator automatically changes to ‘batch_sizes’ which is wrong and there is no need to use ‘s’ at the end of the parameter.</p>
Assertion error in generalized_rcnn.py

gustavo_scholze: <p>I am using pretrained models in order to perform a detection pipeline in Pytorch. The problem is that, when I send the image to the model, the <code>generalized_rcnn.py</code> raises an <code>assert len(val) == 2</code>. The image is a tensor of shape <code>torch.Size([1, 3, 300, 550])</code>, and its length is, in fact:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcd1653fc22184d8013a88a096cf1f2906df983d.png" alt="image" data-base62-sha1="A4wWV9q1Tn4CU8Kqo9dXWuIodIx" width="205" height="72"></p>
<p>Given that, I would to know if there is any error from my part.</p>
Map-style dataset with unknown length

FreddyJ: <p>Hi,</p>
<p>I have a relatively large dataset, which will be assembled from raw data at train time (raw data is downloaded and preprocessed live, the preprocessed tensors are going to be “cached” on the hard drive for the training, as they fit on the hard drive but not into the memory).<br>
Due to data corruption or network issues, I expect a nonzero amount of data that will not reach me.</p>
<p>Is it possible to make a map-style dataset robust against this problem? After the first epoch, the length is well defined and I can set it, but during the first epoch, I can not.<br>
Does that mean I am forced to preprocess all data ahead of time in this case?</p>
<p>If at all possible, I would like to avoid using an Iterable-style dataset.</p>
Add_embedding tensorboard colour image displaying issues

ArshadIram: <p>Hello,</p>
<p>I am following PyTorch provided code from the official site to see embedding images on tensorbaord.</p>
<pre data-code-wrap="import"><code class="lang-import">import torch
from torch.utils.tensorboard import SummaryWriter
import tensorflow as tf
import tensorboard as tb
tf.io.gfile = tb.compat.tensorflow_stub.io.gfile
writer = SummaryWriter('logs/experiments')
meta = []

while len(meta)&lt;100:
    meta = meta+keyword.kwlist # get some strings
meta = meta[:100]

for i, v in enumerate(meta):
    #print(i)
    meta[i] = v+str(i)


label_img = torch.rand(100, 3, 32, 32)


for i in range(100):
    label_img[i]*=i/100.0

writer.add_embedding(torch.randn(100, 5), metadata=meta, label_img=label_img)
</code></pre>
<p>However, when I am doing it for CIFAR10 dataset, labels are displaying, but images are not displaying correctly.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6f5e6faf73a14c3af9d1c7a329d60ce1a6d1a7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6f5e6faf73a14c3af9d1c7a329d60ce1a6d1a7f" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6f5e6faf73a14c3af9d1c7a329d60ce1a6d1a7f_2_537x500.png" alt="pytorch" data-base62-sha1="nP06oYwVFhtfGWtPzmrSleGeMFx" width="537" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6f5e6faf73a14c3af9d1c7a329d60ce1a6d1a7f_2_537x500.png, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6f5e6faf73a14c3af9d1c7a329d60ce1a6d1a7f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6f5e6faf73a14c3af9d1c7a329d60ce1a6d1a7f.png 2x" data-dominant-color="FAF6F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">691×643 66.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the images are displaying like this:</p>
<p>I want to visualize images correctly. I really appreciate any help you can provide.</p>

ptrblck: <p>The image is unfortunately not visible, so could you post it again outside of the code block, please?</p>

ArshadIram: <p>Yes, I uploaded the images out side of block.</p>