akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can someone help me get my code to work?

Patchie: <p>I created a colab to share the code and a sample dataset to more easily run my code: <a href="https://colab.research.google.com/drive/1eh0vN1o4gEGjDquJQXo0sH4YkBIW_0za?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>I also made a comment in the code to explain the dataset and what i am trying to do.</p>
<p>Thanks in advance.</p>
Model giving same output for all video while creating a pytorch Video Captioning system using GRU encoder and GRU language decoder

Chhatra: <p>Hello people ,<br>
sorry to bother you guys but I am just getting started implementing some complex begginner friendly projects with pytorch . So to code a sequence to sequence artitecture with pytorch. I thought of coding a video captioning system . I have used kinematics dataset only about (1200) samples.<br>
My video captioning model predicts same caption for any video so I want help</p>
<p>this is my colab file link<br>
:<a href="https://colab.research.google.com/drive/1eeoDktRG0If4__X9DCuLpPwCCqCqMP4E" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a><br>
please help</p>
<p>I think there is problem in my training loop</p>

ptrblck: <p>It seems you are using <code>nn.CrossEntropyLoss</code>, which expects raw logits as the model output, while you are applying a <code>softmax</code> on the decoder’s <code>output</code> tensor thus creating probabilities.<br>
Remove the <code>softmax</code> and see if this would help training the model.</p>

Chhatra: <p>I removed the softmax function but it doesnot help in training My model still classifying same text for all videos.  Actually I have used this <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.13.1+cu117 documentation</a><br>
tutorial and modified encoder layer to take video as input instead of sentence and modified code little to use accelerator , rest other things are same. I tried both attention decoder and simple decoder given in this tutorial still I got same result. My model is not learning , the loss plot is random up and down.</p>
In loss.item() getting error: ValueError: only one element tensors can be converted to Python scalars

Garry_Santana: <p>class FocalLoss(nn.Module):</p>
<pre><code>def __init__(self, weight=None, 
             gamma=2., reduction='none'):
    nn.Module.__init__(self)
    self.weight = weight
    self.gamma = gamma
    self.reduction = reduction
    
def forward(self, input_tensor, target_tensor):
    target_tensor = torch.argmax(target_tensor ,axis=1)
    log_prob = F.log_softmax(input_tensor, dim=-1)
    prob = torch.exp(log_prob)
    return F.nll_loss(
        ((1 - prob) ** self.gamma) * log_prob, 
        target_tensor, 
        weight=self.weight,
        reduction = self.reduction
    )
</code></pre>
<p>—&gt; 33             batch_loss += loss.item()<br>
34             total_loss += loss.item()<br>
35</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>

KFrank: <p>Hi Garry!</p>
<aside class="quote no-group quote-modified" data-username="Garry_Santana" data-post="1" data-topic="170880" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/garry_santana/48/56660_2.png" class="avatar"> Garry_Santana:</div>
<blockquote>
<pre><code>def __init__(self, weight=None, 
             gamma=2., reduction='none'):
</code></pre>
<p>…<br>
return F.nll_loss(<br>
((1 - prob) ** self.gamma) * log_prob,<br>
target_tensor,<br>
weight=self.weight,<br>
reduction = self.reduction<br>
)</p>
<p>—&gt; 33             batch_loss += loss.item()</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>
</aside>
<p>Because you use <code>reduction = 'none'</code> in <code>nll_loss()</code>, it will (most likely)<br>
return a batch of loss values and therefore <code>loss</code> will be a tensor that<br>
contains more than one element.</p>
<p>The purpose of <code>.item()</code> is to convert a <em>single-element</em> tensor into a regular<br>
python scalar, hence the error.  Try using <code>reduction = 'mean'</code> (the default)<br>
and <code>loss</code> should now be a single-element tensor for which <code>.item()</code> will work.</p>
<p>Best.</p>
<p>K. Frank</p>

Garry_Santana: <p>Thank you very much, Frank that fixed my problem!</p>
Clarification on at:parallel_for needed

Matthias_Moller: <p>In <a href="https://web.mit.edu/~ezyang/Public/pytorch-internals.pdf" rel="noopener nofollow ugc">this</a> document slide 43 I read that it is recommended to use <code>at::parallel_for</code> over OpenMP pragmas.</p>
<p>In another post <a href="https://discuss.pytorch.org/t/using-at-parallel-for-in-a-custom-operator/82747">here</a> the individual elements of the tensor are accessed by the <code>operator[]</code>, e.g.</p>
<pre><code class="lang-auto">torch::Tensor z_out = at::empty({z.size(0), z.size(1)}, z.options());
  int64_t batch_size = z.size(0); 

  at::parallel_for(0, batch_size, 0, [&amp;](int64_t start, int64_t end) {
    for (int64_t b = start; b &lt; end; b++) {
      z_out[b] = z[b] * z[b];
    }
  });
</code></pre>
<p>Is this the right way to do or should one still use a tensor accessor (even when using <code>at::parallel_for</code>)?</p>

Matthias_Moller: <p>Thanks, let me ask two follow-up questions.</p>
<ol>
<li>Is there a recommended way to implement nested loops with <code>at::parallel_for</code>? With OpenMP you can (if the logic of the loops allows for it) use the <code>nested(2)</code> keyword.</li>
<li>Is there a similar approach for tensors residing in GPU memory or do I need to implement a CUDA kernel?</li>
</ol>
<p>Many thanks in advance.</p>
Fine-Tuning on same task leads to worse results

blueeagle: <p>Hi together,</p>
<p>I just finished experimenting with an encoder-classifier architecture and found a behavior that I cannot fully explain.</p>
<p>The architecture consists of a transformer encoder followed by a dense layer classifier.</p>
<p>First I trained the model in an end-to-end supervised pipeline, meaning by backpropagation of an MSE loss through the whole model.</p>
<p>After training, I freezed the parameters of the encoder network and reinitialized the classifier. Then, I trained the model again (only updating the classifiers parameters) on the same task, in order to simulate a transfer learning scenario.</p>
<p>During both training phases, the loss does converge and everything seems sound.</p>
<p>What seems strange to me now, is that the results the model achieves are much better after the first trining phase. The results after transfer learning are still reasonable but worse.</p>
<p>I would argue, that it is probably harder for the model to learn the task in the transfer learning scenario, as the encoder is fixed, meaning only the classifiers parameters can be used for optimization and thus the complexity that the architecture can model is lower.</p>
<p>However, the first training is actually a proof that the classifiers complexity is high enough to achieve better results.</p>
<p>Can anybody give me an intuition why the results are not the same?</p>
<p>Thank you already!</p>
Efficient FFT padding

Florian_Bruckner: <p>Hi,</p>
<p>performing an fft-based convolution in 3D requires zero-padding of the input data in 3D and then performing an fftn in all three dimensions. since there is only data in one octant of the input data, the first 1D fft needs to be performed only for half of the data.</p>
<p>I am wondering whether pytorch uses this optimization when i use the s-parameter for extending the input dimensions. Unfortunately, separating the 3D fft into three 1D ffts in python leads to additional overhead.</p>
<p>For the ifft the situation is similar. In this case the output data is only needed in the first octant of the data, so e.g. after the first 1D fft half of the data could be cropped. Since the s-parameter of ifftn seems to crop data before the FFT, I don’t see an efficient way for an implementation.</p>
<p>I’m not sure if this is the right location for this sort of questions or if I should better create an issue for a feature request!?</p>
<p>thanks and best wishes<br>
Florian</p>
How can I set the x-axis range of each histogram in TensorBoard?

BobKim: <p>Hello, everyone.</p>
<p>I’d like to visualize the weights of a network using TensorBoard.<br>
However, as you can see, it is difficult to analyze each histogram because each histogram has a different x-axis range.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f027a264ea2daf5377a8d43301c9d306b61f7063.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f027a264ea2daf5377a8d43301c9d306b61f7063" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f027a264ea2daf5377a8d43301c9d306b61f7063_2_517x255.jpeg" alt="image" data-base62-sha1="ygvvz2BsbE4EYKfxjv2POc0bb4n" width="517" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f027a264ea2daf5377a8d43301c9d306b61f7063_2_517x255.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f027a264ea2daf5377a8d43301c9d306b61f7063_2_775x382.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f027a264ea2daf5377a8d43301c9d306b61f7063_2_1034x510.jpeg 2x" data-dominant-color="F8EAE1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1909×944 267 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, how can I set the x-axis range of each histogram in TensorBoard?</p>
Slice multiple regions simultaneously

xeTaiz: <p>Hey,<br>
I’m trying to slice multiple contiguous regions from a larger tensor, preserving their shape.<br>
Currently I manage to do this using the following function</p>
<pre><code class="lang-auto"># volume is (C, D, H, W)
# centers is (N, 3)
def gather_receiptive_fields(volume, centers, ks=3):
    L = ks // 2
    R = L+1  # slice end indices are exclusive, thus +1
    pad_vol = F.pad(volume, tuple([L]*6)) # Pad in case a center is at volume border
    return torch.stack([pad_vol[...,       # keep channel dims as is
            coord[0]-L : coord[0]+R,
            coord[1]-L : coord[1]+R,
            coord[2]-L : coord[2]+R] for coord in centers + L  # Shift by ks//2 to account for padding
    ])
</code></pre>
<p>The result being a tensor of shape <code>(N, C, KS, KS, KS)</code> that contains all N <code>ks^3</code> bricks around the given centers.<br>
My problem is now that for large N this becomes quite slow, so I wonder whether there is a PyTorch function I could abuse to do such indexing for all N in one operation, hopefully getting around that Python loop there.</p>
<p>I’d appreciate any suggestions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Cheers</p>

eqy: <p>From the description it sounds like this is similar to an <code>Unfold</code> operation: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html#torch.nn.Unfold" class="inline-onebox" rel="noopener nofollow ugc">Unfold — PyTorch 1.13 documentation</a></p>

xeTaiz: <p>Thanks for your answer! This does indeed look very related, unfortunately it only supports 2D image-like tensors at the moment, but I will investigate if I can abuse it for my task as well</p>

xeTaiz: <p>One way to speed things up for my case (<code>N ~ 65k</code> with <code>ks = (9,9,9)</code>) is definitely to loop over the <code>9x9x9 = 729</code> brick entries (getting <code>N</code> values every iteration)  instead of looping over <code>N</code> (getting only 729 entries per iteration).<br>
This change gives me a speed up of around <code>~40x</code> and gets me quite close to usable speed. Here’s the impl:</p>
<pre><code class="lang-auto">def gather_receiptive_fields2(volume, centers, ks=3):
    L = ks // 2
    offsets = [[L+i,L+j,L+k] for i in range(-L, L+1) for j in range(-L, L+1) for k in range(-L, L+1)]
    pad_vol = F.pad(volume, tuple([L]*6))
    return torch.stack([pad_vol[...,
            centers[:, 0] + off[0],
            centers[:, 1] + off[1],
            centers[:, 2] + off[2]] for off in offsets
    ]).permute(2,1,0).reshape(centers.size(0), volume.size(0), ks, ks, ks).contiguous()
</code></pre>
<p>I still feel like there should be a faster approach tho</p>

eqy: <p>I would also take a look at external repos that have implemented an “Nd” Unfold e.g., <a href="https://github.com/f-dangel/unfoldNd" class="inline-onebox" rel="noopener nofollow ugc">GitHub - f-dangel/unfoldNd: (N=1,2,3)-dimensional unfold (im2col) and fold (col2im) in PyTorch</a></p>
Pytorch setting elements to zero with a "tensor index"

biubiu: <p>Hello Pytorch coders,</p>
<p>I’ve used Pytorch for a few months. But I recently want to create a customized pooling layer which is similar to the “Max-Pooling Dropout” layer and I think Pytorch provided us a bunch of tools to build such a layer I need. Here is my approach:</p>
<ol>
<li>use MaxPool2d with indices returned</li>
<li>set tensor[indices] to zero</li>
<li>I want it behaves like torch.take (without flatten) if possible.</li>
</ol>
<p>here is how to get the “index tensor”. (I think it is called “index tensor”. correct me if I was wrong)</p>
<pre><code class="lang-auto">input1 = torch.randn(1, 1, 6, 6)
m = nn.MaxPool2d(2,2, return_indices=True)
val, indx = m(input1)
</code></pre>
<p>indx is the “index tensor” which can be used easily as</p>
<pre><code class="lang-auto">torch.take(input1, indx)
</code></pre>
<p>No flatten needed, no argument needed to set dimension. I think it make sense since indx is generated from input1.</p>
<p>Question: how do I set the values input1 pointed by indx to 0 in the “torch.take” style? I saw some answers like <code> input1.flatten().scatter_(dim=-1, index=indx.flatten(), value=0.).reshape_as(input1)</code>. But I don’t think PyTorch returning such “index tensor” thing which cannot be applied directly. (Maybe I was wrong.)</p>
<p>Is there something like</p>
<pre><code class="lang-auto">torch.set_value(input1, indx, 0) ?
</code></pre>
<p>Update: if I use <code> input1.flatten().scatter_(dim=-1, index=indx.flatten(), value=-5.).reshape_as(input1)</code> with GPU (torch.<strong>version</strong>=1.12.1+cu113), then I got the inplace error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "v1.1c.py", line 120, in &lt;module&gt;
    loss.backward()
  File "/opt/software/anaconda3/lib/python3.7/site-packages/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/opt/software/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 512, 6, 6]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>

eqy: <p>Does it work if you simply replace the inplace <code>scatter_</code> with the out-of-place version?</p>
<pre><code class="lang-auto">from torch import nn

input1 = torch.randn(1, 1, 6, 6, requires_grad=True)
m = nn.MaxPool2d(2,2, return_indices=True)
val, indx = m(input1)
out = input1.flatten().scatter(dim=-1, index=indx.flatten(), value=-5.).reshape_as(input1)
print(out)
loss = out.sum()
loss.backward()
</code></pre>

biubiu: <p>Thanks eqy for your solution! It does solve the GPU error. But I am still wondering if there is a nice way to use the “tensor index”.</p>
Initalizing ResNet18 with Imageet pretrained weights

papoo13: <p>I am trying to initialize my original PyTorch ResNet18 model with pre-trained weights. I’m targeting all the layers but the last weight and bias. My code in the following does not work. I am not getting any errors but the model is not getting pre-trained. Is this incorrect? I am not bale to find the bug in there.<br>
I have different types for model state dictionary tensors than the pretrained weights:</p>
<p>pretrained tensors are of type &lt;class ‘torch.nn.parameter.Parameter’&gt;<br>
model tensors are of type &lt;class ‘torch.Tensor’&gt;</p>
<pre><code class="lang-auto">        if self.dataset=='cub200':
            pdb.set_trace()
            # load pretrained resnet18 weights
            state_dict = load_state_dict_from_url(model_urls[self.model_name] , progress=True) #&lt;class 'collections.OrderedDict'&gt;
            #update the model state_dict
            for key, value in state_dict.items():
                if key !='fc.weight' and key!='fc.bias':
                    assert self.model.state_dict()[key].shape == value.shape
                    self.model.state_dict()[key]=value
                else:
                    continue
</code></pre>

ptrblck: <p>This assignment:</p>
<pre><code class="lang-python">self.model.state_dict()[key]=value
</code></pre>
<p>will not work and you should <code>copy_</code> the <code>value</code> into the <code>parameter</code> directly  via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layer.parameter.copy_(value)
</code></pre>
<p>or manipulate the <code>state_dict</code> separately and load it into the model:</p>
<pre><code class="lang-python">sd = model.state_dict()
...
sd[key] = value
...
model.load_state_dict(sd)
</code></pre>
Unknown custom class type quantized.Conv2dPackedParamsBase. Please ensure it is registered

Saravana_Rathinam: <p>After converting a pytorch model by using <code>torch.jit.script</code> and calling <code>optimize_for_mobile</code>, the model fails to load on <code>iOS</code> on a real device with the following error.</p>
<pre><code class="lang-auto">libc++abi: terminating with uncaught exception of type torch::jit::ErrorReport: 
Unknown custom class type quantized.Conv2dPackedParamsBase. Please ensure it is registered.:
quantized::conv1d(Tensor qx, __torch__.torch.classes.quantized.Conv2dPackedParamsBase packed_weight, float output_scale, int output_zero_point) -&gt; Tensor
                                                               ~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE

terminating with uncaught exception of type torch::jit::ErrorReport: 
Unknown custom class type quantized.Conv2dPackedParamsBase. Please ensure it is registered.:
quantized::conv1d(Tensor qx, __torch__.torch.classes.quantized.Conv2dPackedParamsBase packed_weight, float output_scale, int output_zero_point) -&gt; Tensor
</code></pre>
<p>How should I start debugging this? We do not use any quantization of any sort in our model.</p>
<p>I have tried disabling the optimzation’s using:</p>
<pre data-code-wrap="optimized_model"><code class="lang-plaintext">                                              optimization_blocklist={
                                                  HOIST_CONV_PACKED_PARAMS,
                                                  CONV_BN_FUSION,
                                                  INSERT_FOLD_PREPACK_OPS,
                                                  FUSE_ADD_RELU
                                              })
</code></pre>
Switching between torch.flatten(x) and x.view(-1, 256)

JkimIshere: <pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, kernel_size=5) # 1 * 28 * 28 -&gt; 6 * 24 * 24
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5) # 6 * 12 * 12 -&gt; 16 * 8 * 8
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(256, 100)
        self.fc2 = nn.Linear(100, 10)
    def forward(self,x):
        x = self.pool(torch.relu(self.conv1(x)))
        x = self.pool(torch.relu(self.conv2(x)))
        x = torch.flatten(x)
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, -1)
</code></pre>
<pre><code class="lang-auto">def training_loop():
    for epoch in range(epochs):
        model.train() 
        for img, label in train_loader:
                img, label = img.to(device), label.to(device)
                out = model(img)
                loss = loss_fn(out, label)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
        print(loss)
        print(f'epoch {epoch} ended')
        print('training done!')

    torch.save(model.state_dict(), './MNIST.pth')
</code></pre>
<p>I have this code here.<br>
I used torch.flatten(x) at first. the fc1 layer expects 2560 input and crossentropyloss expects float data values. After training, the loss is around 100.<br>
I changed torch.flatten(x) to x.view(-1, 256), fc1 layer expects 256 input and crossentropyloss wants Long values. the loss value looks good. (0.01 or something)</p>
<p>Why does this happen?</p>

ptrblck: <p>The usage of <code>torch.flatten(x)</code> sounds wrong as it would flatten the entire tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 256)
print(torch.flatten(x).shape)
# torch.Size([2560])
</code></pre>
<p>Using <code>x = x.view(-1, 256)</code> sounds better in case you can guarantee the tensor has a feature size of <code>256</code>. The better approach would be to use <code>x = x.view(x.size(0), -1)</code> to make sure the batch size does not change.</p>

JkimIshere: <p>Thanks for the help!<br>
This solved my problem, but why does switching between flatten() and view(-1, 256) affect the loss values?</p>

ptrblck: <p>I guess because your loss calculation was wrong in the <code>flatten()</code> use case and e.g. unexpected broadcasting might have been used.</p>
Empty batch recieved

nmn2314242: <p>finetuning gptj on 20newsgroup with dataparallel.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbcedd49ef4ac45056258ca7eef9fc11ad11b57f.png" data-download-href="https://discuss.pytorch.org/uploads/default/bbcedd49ef4ac45056258ca7eef9fc11ad11b57f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbcedd49ef4ac45056258ca7eef9fc11ad11b57f_2_690x223.png" alt="image" data-base62-sha1="qNquuXZRsXgwf7oAFjH8EBJqGyX" width="690" height="223" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbcedd49ef4ac45056258ca7eef9fc11ad11b57f_2_690x223.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbcedd49ef4ac45056258ca7eef9fc11ad11b57f_2_1035x334.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbcedd49ef4ac45056258ca7eef9fc11ad11b57f_2_1380x446.png 2x" data-dominant-color="141B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1915×619 63.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to edit and compile pytorch source code?

Vinayaka_Hegde: <p>Hi, sorry if this question was asked before…</p>
<p>I want to make few changes to the file <code>utils.data.dataloader.py</code> and <code>utils.data._utils.worker.py</code>. Precisely, I want to add some log statements to both the files.</p>
<p>I have downloaded and installed pytorch in Windows 10 using <code>pip3 install torch...</code>. Now, I wanted to know :</p>
<ol>
<li>how can I edit files and re-compile pytorch from source ?</li>
<li>Do I need to install any C++ compiler on windows since pytorch uses C++ under the hood ?</li>
</ol>
<p>Please help me, Thanks !!</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="Vinayaka_Hegde" data-post="1" data-topic="170962">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinayaka_hegde/48/56332_2.png" class="avatar"> Vinayaka_Hegde:</div>
<blockquote>
<ul>
<li>how can I edit files and re-compile pytorch from source ?</li>
</ul>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/vinayaka_hegde">@Vinayaka_Hegde</a>,</p>
<p>If you want to edit source, you’ll need to install pytorch from source rather than pip or conda. The instructions can be found on their GitHub repo <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">here</a>.</p>

Vinayaka_Hegde: <p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> thanks for replying. I have one doubt : I want to install pytorch from source but I dont want the cuda option enabled for now. I want to try out some minor things by chaning the pytorch source code, and I dont want to install nvidia CUDA sdk or cuDNN packages for now.</p>
<p>Is it possible to still install and compile/build source by just installing conda and other packages as mentioned in github (without installing cuDNN or Nvidia CUDA SDK )  ?</p>

AlphaBetaGamma96: <p>There are cpu-only install instructions in the github link I shared, but note you won’t be able to use CUDA with that environment.</p>

albanD: <p>hi,</p>
<p>If you only want to change some python code, then there is a hacky way to do this by modifying directly the intall you got from pip.<br>
You can locate the file from the install by running <code>python -c "import torch; print(torch.utils.data.dataloader.__file__)"</code><br>
Then if you modify this python file, it will be automatically picked up after you restart your python proces.<br>
There is no need to recompille if you change only python code.</p>

Vinayaka_Hegde: <p><a class="mention" href="/u/alband">@albanD</a> thanks as this worked, However, I’m facing some more issues, would you please take a look at this: <a href="https://discuss.pytorch.org/t/dataloader-multiprocessing-runtime-error/170975/1" class="inline-onebox">DataLoader Multiprocessing RunTIme Error</a><br>
Thanks</p>
PyTorch and 'OpenCL'

dbl001: <p>Is there a version of PyTorch which works with ‘OpenCL’ on MacoOS Ventura?<br>
If yes, does it support torch.float64?, torch.cfloat? torch.cdouble?</p>
<p>I have an AMD Radeon Pro 5700 XT GPU.</p>

albanD: <p>Hi,</p>
<p>I think you have a few options here:</p>
<ul>
<li>You can try to use the MPS backend that is available with the regular macOS install. I am not 100% sure if your OS is recent enough though.</li>
<li>You can take a look at this OpenCL backend for PyTorch: <a href="https://github.com/artyom-beilis/pytorch_dlprim" class="inline-onebox">GitHub - artyom-beilis/pytorch_dlprim: DLPrimitives/OpenCL out of tree backend for pytorch</a> I’m not sure what’s the status of macOS support there though.</li>
</ul>
What is TORCH_API?

wnnaone: <p>Found here for example: <a href="https://github.com/pytorch/pytorch/blob/6084908287ab88b73f8af8d6469aac0c57cb1b43/torch/csrc/jit/fuser/interface.h#L19" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/6084908287ab88b73f8af8d6469aac0c57cb1b43/torch/csrc/jit/fuser/interface.h#L19</a></p>

albanD: <p>These macros are used to properly set the visibility of the cpp functions. In particular, by default, functions won’t be exposed in the shared library. Adding this macro will make them available.</p>

yaohwang: <p>where is this macro’s definition?</p>

albanD: <p>Here it is: <a href="https://github.com/pytorch/pytorch/blob/0fe5367058a1d67134aee510ed81691cf9e61e33/c10/macros/Export.h#L94-L99" class="inline-onebox">pytorch/Export.h at 0fe5367058a1d67134aee510ed81691cf9e61e33 · pytorch/pytorch · GitHub</a></p>
The same model with the same data sometimes works, sometimes not

xlaxc: <p>Hello everyone,</p>
<p>I’m facing a certain problem and I can’t figure out why.</p>
<p>I am trying to train a binary classification model, which seems to work with high accuracy most of the time through multiple re-trainings (about 90% on the validation set). However, in some runs, the model does not give better accuracy than a random estimate (50%). I made sure that all runs use the same data and even fixed the seed for the data and batch sampling. The only difference between the runs is the random network initialization (same hyperparameters, same everything else).</p>
<p>I can’t figure out what the problem is, especially since I haven’t observed any difference in performance between the good and random ones (either ~90% or 50%).</p>
<p>Can network initialization have such an impact? Also, I don’t think it’s the size of the dataset, as I think I have more than enough data for this problem (it could still be the problem, but I think it’s unlikely).</p>
<p>Thanks for your help.</p>

ptrblck: <p>Yes, the model initialization could have a significant effect on the success rate of the model training.<br>
I don’t know how long it would take to determine if the training gets stuck or if you are seeing the expected good accuracy, but in case it’s fast you could use different seeds and check the success rates or different initializations in order to get a better picture of the training stability.</p>

xlaxc: <p>Hello, thanks for your reply.</p>
<p>Yes, that’s what I did, I tried different seeds, and depending on the seed the model either train and give a good accuracy on validation or not train at all.</p>
<p>Sorry, I want to add some more information I should have put in the first post. When it doesn’t work, it doesn’t train at all, always predicting the same class and even the train loss isn’t decreasing at all (Hence, 0.5% accuracy).</p>
<p>Since my first post, I tried different hyperparameters (number of neurons and layers)  and for each of these there is a different set of seeds where it doesn’t train. (for example, with the first hyperparameters’ configuration, the model train well and give the expected good performances with seed=2 and doesn’t train at all with seed=3. With the second hyperparameters’ configuration, the model doesn’t train with seed=2, but it does with seed=3)</p>
Categorical distribution in LibTorch?

m2q: <p>In python we can do</p>
<pre><code class="lang-auto">m = Categorical(probs)  &lt;----
action = m.sample()
</code></pre>
<p>How can I sample from a given categorical distribution in libtorch? I’ve searched the docs but I can’t find any matching function in the C++ frontend. If it’s not implemented yet, <em>what would be the most efficient way to sample that way for now</em>? I could use std’s discrete_distribution, but seems a bit too much like a workaround</p>

m2q: <p>bumping this…</p>

Bojacob: <p>Did you ever figure this out?  I thought of porting the categorical distribution from Python but reading the Python source code makes me think this isn’t a trivial task.</p>

afshin67: <p>Still no update on this?</p>

Ruslan_Mukhamadiarov: <p>bumping this again…<br>
(also why 20char limit…)</p>
Error after installing pytorch

Vinayaka_Hegde: <p>Ive installed pytorch version below and tried running a piece of code which is producing the below error. Please help !!</p>
<p>Error log when I ran the code below:<br>
<code>File "D:\DREAM\pytorch\mycode\dataloader_test_1.py", line 15, in &lt;module&gt;     transforms.toTensor(), AttributeError: module 'torchvision.transforms' has no attribute toTensor</code></p>
<p>Code used :<br>
<code>import torch</code></p>
<p><code>import torchvision</code></p>
<p><code>import torchvision.transforms as transforms</code></p>
<p><code>import torch.nn as nn</code></p>
<p><code>from torch.utils.data import DataLoader</code></p>
<p>FYI :<br>
<code>import torch torch.__version__</code><br>
<code>1.13.1+cu116</code></p>

ptrblck: <p>You have a typo and <code>transforms.ToTensor()</code> should work (note the capital <code>T</code>).</p>
CrossEntropyLoss crashes on soft targets in various PyTorch versions

Huxwell: <p>I am trying to pass soft targets to mobilenet v2 (with the only change that I am using 2 classes instead of 1000).<br>
According to <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 2.1 documentation</a> and various similar posts, since ~2022 float targets are accepted by CrossEntropyLoss. However, for me it still fails.</p>
<p>The Last layer is:</p>
<blockquote>
<p>nn.Linear(in_features=1280, out_features=2, bias=True)</p>
</blockquote>
<p>It crashes though, if my labels are floats. I have prepared the minimal failing example using values returned by my network (batch_size = 4):</p>
<pre><code class="lang-auto">activations = torch.FloatTensor([[-0.3139, -0.0486],
[-0.0510,  0.0470],
[ 0.0963,  0.0143],
[-0.2151, -0.0576]])

targets_float = torch.FloatTensor([1.0, 1.0, 0.0, 0.5])
targets_long = torch.LongTensor([1, 1, 0, 0])
crit = nn.CrossEntropyLoss()
crit(activations,targets_long) # returns tensor(0.6606) as expected
crit(activations,targets_float) 
# return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) #crashes with
# RuntimeError: expected scalar type Long but found Float
</code></pre>

ptrblck: <p>As described in the docs the shape for “soft” targets is expected to be the same as the model output:</p>
<blockquote>
<p>If containing class probabilities, same shape as the input and each value should be between <code>[0,1]</code>.</p>
</blockquote>
<p>This would thus work:</p>
<pre data-code-wrap="python"><code class="lang-python">activations = torch.FloatTensor([[-0.3139, -0.0486],
[-0.0510,  0.0470],
[ 0.0963,  0.0143],
[-0.2151, -0.0576]])

targets_float = torch.FloatTensor([1.0, 1.0, 0.0, 0.5])
targets_long = torch.LongTensor([1, 1, 0, 0])
crit = nn.CrossEntropyLoss()
crit(activations,targets_long) # returns tensor(0.6606) as expected
crit(activations,F.one_hot(targets_long, num_classes=2).float())
</code></pre>

Huxwell: <p>Thanks a lot, it works. I thought at first that you mean that the batch size must be 1 (it’s 4 in the example), but if my targets are [4,2] the softmax is computed fine, so I am doing smth like this for now:</p>
<p><code>targets_float = torch.FloatTensor([[t, 1-t] for t in targets_float])</code><br>
since F.one_hot would crash on vector that has floats.</p>
What are the best ways to decrease the inference time of a CNN network?

Alireza: <p>Hi,</p>
<p>I have trained a fully convolutional neural network for monocular depth estimation, and its performance is quite satisfactory. I have used depthwise separable convolution instead of all normal convolutions in order to decrease the trainable parameters. However, the speed of testing a new image is relatively slow. The network can handle almost 7.4 frames per second on GPU. I have changed all weights type to float16. This solution could speed up the network to around 17 FPS. I want to know whether there are any techniques that can further speed up my network.</p>
<p>Thanks.</p>

ptrblck: <p>Take a look at the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">Performance Guide</a>, which explains that e.g. <code>cudnn.benchmark=True</code> could be used for static inputs etc.</p>

Alireza: <p>Thanks <span class="mention">@ptrblck_de</span> some guides in the provided link were helpful.</p>

sardanian: <p>I have found that converting to a tensorrt via torch_tensorrt has been the fastest for inference.</p>
Heterogeneous Graphs Graph Embedding (global_mean_pool)

MichZipp: <p>I’m working with PyTorch-geometrics and heterogeneous graphs.<br>
How can I aggregate the node embeddings by node types to receive the overall graph embedding?</p>
<pre><code class="lang-auto">class GNN(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)

        return global_mean_pool(x, batch=None)

model = GNN(hidden_channels=64, out_channels=dataset.num_classes)
model = to_hetero(model, data.metadata(), aggr='sum')

with torch.no_grad():  # Initialize lazy modules.
    out = model(data.x_dict, data.edge_index_dict)

model(data.x_dict, data.edge_index_dict)
</code></pre>
<p>Result:</p>
<pre><code class="lang-auto">{'node_type_1': tensor([[-0.0687, -0.2087]], grad_fn=&lt;MeanBackward1&gt;),
 'node_type_2': tensor([[ 0.0210, -0.0251]], grad_fn=&lt;MeanBackward1&gt;),
 'node_type_3': tensor([[ 0.1923, -0.0379]], grad_fn=&lt;MeanBackward1&gt;),
 'node_type_4': tensor([[-0.0792,  0.1417]], grad_fn=&lt;MeanBackward1&gt;)}
</code></pre>
Error converting PyTorch scripted model to ONNX

snucklebarr: <p>I’m trying to convert a Pytorch model to ONNX using this code:</p>
<pre><code class="lang-auto">import onnx
import torch
import onnxruntime
import numpy as np

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


batch_size = 1
model = torch.jit.load('../saved_weights/autotissue_mobilenetv3_lrassp.pt')

x = torch.randn(1, 3, 224, 224, requires_grad=True)
x = x.to(device)
torch_out = model(x)
torch_out = torch_out['out']


torch.onnx.export(model, x, "../saved_weights/autotissue_mobilenetV3.onnx",
              export_params=True,
              opset_version=12,
              do_constant_folding=True,
              input_names=['input'],
              output_names=['out'],
              dynamic_axes={ 'input': {0 : 'batch_size'},
                            'output': {0 : 'batch_size'}})
</code></pre>
<p>However I’m getting the following error:</p>
<pre><code class="lang-auto">raise errors.UnsupportedOperatorError(
 torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 
 'aten::dict' to ONNX opset version 12 is not supported. 
</code></pre>
<p>Is this due to:</p>
<ol>
<li>The fact that model was saved as  a scripted model?</li>
<li>The torchvision model outputs an OrderedDict ?</li>
</ol>
What is extreme multi-label classification(XMLC)

Rexedoziem: <p>I tried solving a problem in recommendation systems using PyTorch, and i came this in an article, so ill like to how to use it and apply it to solve problems in recommendation systems?<br>
Thanks</p>

ptrblck: <p>Based on <a href="https://paperswithcode.com/task/extreme-multi-label-classification">this post</a> by <code>PapersWithCode</code> it seems that Amazon provided a library based on PyTorch called <a href="https://github.com/amzn/pecos"><code>PECOS</code></a> for this use case, so you might want to take a look at their implementation and how the large space of labels is taken care of.</p>
Ignore loss on some outputs depending on others?

supagu: <p>I’m trying to guess a bounding box + a few sets of class labels based on an image.<br>
It seems my bounding box code is struggling to learn, and I suspect it might be because I have the situation where, if a certain class label is set, then there is no need for a bounding box, so I have just returned [0,0,0,0] for the bounding box in that case.</p>
<p>here is how I make a prediction:</p>
<pre><code class="lang-auto">pred_bounds, pred_node, pred_display = model(X)

loss_1 = criterion_1(pred_bounds, Y_bounds)
        loss_2 = criterion_2(pred_node, Y_node_class)
        loss_3 = criterion_3(pred_display, Y_display_class)
        loss_total = loss_1/1000.0 + loss_2 + loss_3
</code></pre>
<p>I’m trying to predict the bounding box + a class label called “node” + another class label called “display” (these labels are one-hot encoded).<br>
If node == [0,1] then I actually don’t care about any loss value for the bounding box is, as this class label means I can ignore the bounding box.<br>
Should I some how conditionally ignore bounding box loss (loss_1) for those data samples that I don’t care what the bounding boxes should be?<br>
How do I do that?</p>
<p>Also, I’m scaling my bounding box error down (see the /1000) with the thinking that perhaps it will make smaller adjustments to the bounding box nodes…I’m not sure if I need to/should be doing that also?</p>

eqy: <p>I would check if simply setting the loss to zero for those examples would make a difference, provided that you can pass or compute this information in your criterion function.</p>
<p>Yes, scaling the relative weight of the different parts of your loss calculation is a common practice, even if it does introduce additional hyperparameters. A somewhat famous example: <a href="https://www.youtube.com/watch?v=IHH47nZ7FZU&amp;t=12m20s" class="inline-onebox" rel="noopener nofollow ugc">Andrej Karpathy: Tesla Autopilot and Multi-Task Learning for Perception and Prediction - YouTube</a></p>

supagu: <p>Thanks for that egy!</p>
Getting Gradients from Gym Environment Reward

kayuksel: <p>I am trying to train a neural network based on gradients of a loss function calculated in a Gym environment. However, the gym environment requires me to detach the tensor (as the environment only supports numpy) to get the reward. How, can I use Autograd to do gradient-based training in this case? I have checked other policy gradient examples but didn’t understand how they get a gradient to train their models. I would appreciate it if you could help me understand the trick here.</p>
<pre><code class="lang-auto">def reward_func():
    observation, info = env.reset(seed=42)

    total_reward = torch.zeros(1).cuda().requires_grad_()

    terminated, truncated = False, False
    while not (terminated or truncated):
        action = agent(torch.from_numpy(observation).float().cuda())
        observation, reward, terminated, truncated, info = env.step(action.detach().cpu().numpy())
        total_reward = total_reward + reward
        
    return total_reward
</code></pre>

vmoens: <p>Unfortunately, as of today you can’t do that in the general case. Some libs like mujoco offer finite difference gradient, and others like brax offer built-in differentiable operations, but the general case in gym is that you can’t backprop through the ops unless clearly states.<br>
Sorry if this is disappointing.<br>
One work-around could be to use finite difference but that would require an env where you can easily set the state, which is not always that easy.<br>
Hope that helps</p>

kayuksel: <p>Thanks. I learn that the log_prob trick is also usable for continuous action spaces.<br>
I previously used Brax with PyTorch, but this was the first time that I used Mujoco.</p>

vmoens: <p>The log-prob trick (REINFORCE, log-derivative trick etc) gives you an estimator of the gradient of an integral taken over a probability space: it won’t give you the Jacobian over the transform or work to backprop through an env dynamic, in general.</p>
TypeError: max_pool2d(): argument 'input' (position 1) must be Tensor, not tuple

ays: <p>I keep getting this error, but I’m not sure why. I’ve checked the shape of my input and its a tensor of shape [1, 256, 20, 24]. SO I dont know why it gives this error. Below is the code. Thanks</p>
<pre><code class="lang-auto">class MLP_model_lvl1(torch.nn.Module):
        def __init__(self, pretrained_model_lvl1):
            super(MLP_model_lvl1, self).__init__()
            self.pretrained_model_lvl1 = pretrained_model_lvl1
            self.hidden = 64
            output_dims = 1
            self.relu = torch.nn.ReLU()
            self.fc2 = torch.nn.Linear(self.hidden, self.hidden)
            self.fc3 = torch.nn.Linear(self.hidden, output_dims)
            self.max_pool = nn.MaxPool2d(kernel_size = 3, stride= 2, padding = 1)
            
        
        def forward(self, x, y, reg_code, pretrained):
            e0 = self.pretrained_model_lvl1(x, y, reg_code, pretrained) #shape [1, 256, 20, 24]
            # print(e0.shape, e0)
            e0 = self.max_pool(e0) #the error is here
            encoded = e0.flatten(1, -1)
            return encoded


</code></pre>

ptrblck: <p>The layer itself works fine with a single tensor input:</p>
<pre><code class="lang-python">max_pool = nn.MaxPool2d(kernel_size = 3, stride= 2, padding = 1)
x = torch.randn(1, 256, 20, 24)            
out = max_pool(x)
print(out.shape)
# torch.Size([1, 256, 10, 12])
</code></pre>
<p>and the error claims that <code>e0</code> is a <code>tuple</code> so make sure it’s a tensor instead.</p>

ays: <p>This is the shape I get(which is a tensor), I’m not sure why it gives that error.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9326b54674eafd5729db683cd7109a597a3dfd32.png" alt="image" data-base62-sha1="kZL7qRsW7yYPtjKBKMnLJ9VkkOS" width="303" height="18"></p>

ays: <p>And the type is a tensor (image is attached below)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a55e496b957ff6926b8e0968db047454bc8d142.png" alt="image" data-base62-sha1="aBBmarCd47gZKiQf6pb2EL1884W" width="173" height="16"></p>

ptrblck: <p>That’s weird as it should work in this case as you can see in my code.<br>
Could you post a minimal and executable code snippet reproducing your error, please?</p>

ays: <p>It stopped giving me an error after I ran it multiple times. (Not sure what the problem was or how it got fixed.</p>
How to consider weights in cross entropy loss

hossein: <p>Hi<br>
There is a <code>weight</code> option in <code>nn.CrossEntropyLoss</code> class. What I don’t understand is whether I should calculate weights for the whole dataset and then feed it to the loss function, or I must calculate it separately for every batch?!</p>
<p>Thanks</p>

eqy: <p>In theory you would only set it once according to how imbalanced the classes in your training dataset are. This is also implied as it is a parameter you can pass at initialization time, rather than during invocation of the <code>forward</code> method.</p>
Einsum Matrix Multiplication Question

armal: <p>I currently have a tensor of size ([10, 5]) which represents 10 5-D vectors and a tensor of size ([10, 5, 5]) which represents 10 5x5 matrices. I would like to perform<br>
V_i @ M_i @ V_i.T for each matrix and vector so it returns back a ([10, 1]) tensor and I think this could be done via einsum. Currently I can do this easily via a for loop but it is very slow and I am wondering if this calculation is possible to do using einsum or some other torch function?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/armal">@armal</a>,</p>
<p>Here are some example codes that solve your problem.</p>
<pre><code class="lang-auto">import torch

_ = torch.manual_seed(0)

V = torch.randn(10,5)
M = torch.randn(10,5,5)

def einsum_op(V, M):
  out = torch.einsum("bi,bij,bj-&gt;b", V, M, V)
  return out

def unsqueeze_op(V, M):
  out = V.unsqueeze(-2) @ M @ V.unsqueeze(-1)
  return out.squeeze() #reshape to size [B,]

out = einsum_op(V,M)
print(out)
#returns 
#tensor([  4.7036,   6.2151,   1.1999,  -2.0412,  -6.3517,  -5.3721,  -2.5428,
#         -0.4131, -24.2325,  -2.3242])
out = unsqueeze_op(V, M)
print(out)
#returns 
#tensor([  4.7036,   6.2151,   1.1999,  -2.0412,  -6.3517,  -5.3721,  -2.5428,
#         -0.4131, -24.2325,  -2.3242])
</code></pre>
<p>If you want to run a function over batches of inputs, you might want to have a look at the functorch library (which is specifically built for this). Documentation <a href="https://pytorch.org/functorch/stable/" rel="noopener nofollow ugc">here</a>.</p>

armal: <p>Thank you so much for this! I will definitely take a look at functorch as well, looks like exactly what I need.</p>
SyntaxError: invalid syntax 'nn.BatchNorm2d(32)'

Shahriar: <p>model = nn.Sequential(<br>
nn.Conv2d(3, 32, (3, 3), padding=‘same’),<br>
nn.ReLU(),<br>
nn.init.kaiming_uniform_()<br>
nn.BatchNorm2d(32)<br>
)</p>
<p>nn.BatchNorm2d(32)<br>
^<br>
SyntaxError: invalid syntax</p>

ptrblck: <p>The syntax error is raised because of the usage of <code>nn.init.kaiming_uniform_()</code> without a comma afterwards.<br>
Remove the <code>nn.init.kaiming_uniform_()</code> call since this is an inplace function on a tensor and it should work.<br>
If you want to initialize parameters of the model, use the <code>model.apply</code> method or call the <code>nn.init</code> methods on parameters directly after creating the object.</p>