akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to load model in Pytorch in another notebook

username: bkh6722
cooked: <p>I have trained M5 model for the purposes of sentence classification. As I have saved it using <code>torch.save(PATH)</code> , I am no more able to load it in another notebook using <code>torch.load(PATH)</code> . The error, that I see is following:</p>
<pre><code class="lang-auto">AttributeError: Can't get attribute 'M5' on &lt;module '__main__'&gt;
</code></pre>

username: Samuel_Bachorik
cooked: <p>You should use</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), PATH)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load(PATH))
</code></pre>

username: bkh6722
cooked: <p>Thanks! Thats working.</p>
Expand tensor dimension with ones

username: sondisonda
cooked: <p>Hi, is there any simple way to expand tensor dimension to get something similar</p>
<p>input tensor</p>
<pre><code class="lang-auto">tensor([[25, 11],
        [25, 12],
        [25, 13],
        [25, 14]])
</code></pre>
<p>output tensor</p>
<pre><code class="lang-auto">tensor([[25, 11,  1],
        [25, 12,  1],
        [25, 13,  1],
        [25, 14,  1]])
</code></pre>

username: Samuel_Bachorik
cooked: <p>Hi, you can try like this-</p>
<pre><code class="lang-auto">x = torch.randn((10,2))
y = torch.ones((x.shape[0],1))

final = torch.cat((x,y),1)

</code></pre>

username: sondisonda
cooked: <p>Thanks, It’s what i needed</p>
Indexing during runtime into a JITed ModuleList

username: avitase
cooked: <p>Hi,</p>
<p>I created a module that holds a list of different modules and randomly applies one of them on each forward path. This is my current (ugly!) implementation:</p>
<pre><code class="lang-auto">class OneOf(nn.Module):
    def __init__(self, *args) -&gt; None:
        super().__init__()

        self.layers = nn.ModuleList(list(args) if len(args) else [nn.Identity()])

        n_layers = len(self.layers)
        self._weights = nn.parameter.Parameter(
            torch.rand(n_layers), requires_grad=False
        )

    def forward(
        self, x: torch.Tensor, sample: bool = True
    ) -&gt; Tuple[torch.Tensor, torch.Tensor]:
        log_prob = nn.functional.log_softmax(self._weights, dim=0)

        idx = (
            torch.multinomial(torch.exp(log_prob), 1)
            if sample
            else torch.argmax(log_prob)
        )

        y = torch.zeros_like(x)
        for i, f in enumerate(self.layers):  # change me!
            if i == idx:                     # change me!
                y = f(x)

        return y, log_prob[idx]


if __name__ == '__main__':
    f = OneOf()
    f = torch.jit.script(f)

    x = torch.rand(2, 3, 5)
    y, log_prob = f(x)

    assert torch.allclose(x, y)
    assert np.isclose(log_prob.item(), 0.0)

</code></pre>
<p>This hideous iteration &amp; <code>idx</code> comparison is my workaround to avoid the indexing with a non-literal number. Is there a better way to do this w/o breaking the compilation to TorchScript?</p>
<p>Thanks for your help!</p>
Torchscript model occupies different memory in different version of PyTorch

username: Vidit_Goel
cooked: <p>I am using a torchscript model and noticed that it occupies different memory on GPU for different versions of PyTorch. The torchscript model was created using torch version 1.8.0</p>
<pre><code class="lang-auto">| Torch version | Memory  |
|---------------|---------|
| 1.8.0         | 2879 MB |
| 1.9.1         | 3383 MB |
</code></pre>
<p>Is this an expected behavior? Shouldn’t the memory consumption be the same across different versions of PyTorch.</p>

username: ptrblck
cooked: <p>No, the memory consumption on the GPU depend on various things:</p>
<ul>
<li>the largest portion would probably come from CUDA libraries (the driver, cuDNN, etc.)</li>
<li>each PyTorch kernel will be loaded into the CUDA context, which could also increase it assuming the number of kernels grew between <code>1.8.0</code> and <code>1.9.0</code>
</li>
<li>changes in PyTorch might use other operations e.g. broadcasting could be used where it wasn’t previously (and might have raised a warning)</li>
</ul>

username: Vidit_Goel
cooked: <p>Thanks for the quick reply. It clears my doubt.</p>
Beginner - Running a PyTorch program

username: hugo011
cooked: <p>Hello, since this is my first time using PyTorch, I don’t know much about the architecture, and I want to run a project from GitHub (StyleGAN - this is for my bachelor thesis), but I’m not sure how to do it. I’ve seen a bit of documentation and it doesn’t take much more than the importion of the library and such stuff, but when I read the requirements file of this project (written down below), I  stumbled upon Docker and NVIDIA Containers. Now, I’ve read a bit about them, but I still don’t know why do I need them and how to run them so I can run that program. If anyone can explain a bit of it or send me some useful links I would appreciate it.<br>
Thanks.</p>
<h2>
<a name="requirements-1" class="anchor" href="#requirements-1"></a>Requirements</h2>
<ul>
<li>Docker</li>
<li><a href="https://github.com/NVIDIA/nvidia-container-runtime" rel="noopener nofollow ugc">NVIDIA Container Runtime</a></li>
<li>Conda (see <a>./conda_env.yml</a>)</li>
<li>NVIDIA GPU, NVIDIA drivers</li>
<li>Create copy of the <code>.env</code> file, name it <code>dev.env</code> and fill the values</li>
</ul>
Make specific parameters disappear in state_dict

username: Julian_Buchel
cooked: <p>I have written a custom convolutional layer that has a <code>Linear</code> layer as a member. However, the <code>Linear</code> layer’s weights are just a <code>.view</code> of the weights of the convolutional layer.</p>
<pre><code class="lang-auto">class Conv2d(AISBase):
   
    def __init__(
        self,
        in_channels : int,
        out_channels : int,
        kernel_size : _size_2int,
        stride : _size_2int = 1,
        padding : Union[str, _size_2int] = 0,
        dilation : _size_2int = 1,
        groups : int = 1,
        bias : bool = False,
        device = None,
        dtype = None,
    ):
       
        super(Conv2d, self).__init__()
        
        # ... standard stuff here
        self.weight = Parameter(
            torch.empty((out_channels, in_channels, *self.kernel_size), **factory_kwargs)
        )
        if bias:
            self.bias = Parameter(torch.empty(out_channels, **factory_kwargs))
        else:
            self.register_parameter('bias', None)
        # ... more stuff
        self.reset_parameters()

        n_rows = self.kernel_size[0] * self.kernel_size[1] * in_channels
        n_cols = out_channels
        self.linear = Linear(
            in_features=n_rows,
            out_features=n_cols,
            bias=bias,
            device=device,
            dtype=dtype
        )
        # Link the parameters of the conv layer with the one of the linear through a view
        self.linear.weight = Parameter(self.weight.view(self.weight.size(0),-1))
        self.linear.bias = None if not bias else Parameter(self.bias.view(-1))
</code></pre>
<p>Viewing the network from outside, this linear layer should not be visible. Therefore, it should not appear when I look at the <code>state_dict</code> of the model. However, I have to use the Linear class and for that the weight has to be a <code>Parameter</code>.<br>
I also want to be able to use a <code>state_dict</code> in <code>load_state_dict</code> that does not have these linear parameters. If I now try to load one, PyTorch of course complains that the <code>Parameter</code>s of the <code>self.linear</code> are not matched.</p>
<p>Is there maybe a fancy way through hooks or smth. similar that solves my problem?</p>

username: Julian_Buchel
cooked: <p>Ok, this is resolved now. I added this to my <code>Conv</code> module:</p>
<pre><code class="lang-auto">def state_dict_hook(self, destination, prefix, local_metadata):
        prefixes_removed = [k for k in destination if "linear" in k]
        for p in prefixes_removed:
            destination.pop(p)
        return destination
self._register_state_dict_hook(state_dict_hook)
</code></pre>
<p>Maybe not the most elegant, but it works fine.</p>
Timeseries and DataPipes

username: tcapelle
cooked: <p>Hello,</p>
<p>I started playing with torchdata recently and have been implementing some TimeSeries pipes. Would more people be interested on persuing the creation of different TimeSeries related pipes?</p>
<p>Who one would do this without loading the full pandas dataframe in memory?</p>
<p>I created a rolling window one <a>here</a>, give it a try.</p>
<p>Would love to work with more poeple on making timeseries friendly in pytorch.</p>
Exporting batchnorm layer to onnx with autocast

username: xyz
cooked: <p>Hi, I want to try mixed precision at inference time with onnx models. It seems, that there is a problem with the batchnorm layers:</p>
<pre><code class="lang-auto">import torch
from torch import nn

import onnx
import onnxruntime as ort

class TestModel(nn.Module):
    def __init__(self, do_bn=False):
        super().__init__()
        self.conv = nn.Conv2d(3, 1, 3)
        self.bn = nn.BatchNorm2d(1)
        self.do_bn = do_bn
        return
    
    def forward(self, x):
        with torch.cuda.amp.autocast(True):
            x = self.conv(x)
            if self.do_bn:
                x = self.bn(x)
            return x



def test(do_bn):
    model = TestModel(do_bn=do_bn).cuda()
    i = torch.randn(2, 3, 50, 50).cuda()
    torch.onnx.export(model.cuda(), (i,), 'test.onnx', input_names=['input'])
    onnx.checker.check_model("test.onnx")
    s = ort.InferenceSession("test.onnx", providers=['CUDAExecutionProvider'])
    s.run(None, {'input': i.cpu().numpy().astype('float32')})[0]
    return

test(False) # works 

test(True) # Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from test.onnx failed:Type Error: Type parameter (T) of Optype (BatchNormalization) bound to different types (tensor(float16) and tensor(float) in node (BatchNormalization_2).
</code></pre>
<p>I’m using<br>
torch==1.11.0+cu113<br>
onnxruntime-gpu==1.10.0</p>
<p>Is this a bug, or is there a better way to do this?<br>
Thanks in advance</p>

username: xyz
cooked: <p>Has really no one an idea, or should I open an issue?</p>

username: xyz
cooked: <p>This is discussed in <a href="https://github.com/pytorch/pytorch/issues/72494" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/72494</a> and using opset version 15 fixes this:</p>
<pre><code class="lang-auto">torch.onnx.export(model.cuda(), (i,), 'test.onnx', input_names=['input'], opset_version=15)
</code></pre>
Loss function involving max of n inputs

username: hnakao
cooked: <p>I am trying to implement a loss function <code>max(f_1(x),f_2(x),...)</code>.<br>
I know that</p>
<pre><code class="lang-auto">max(a,b)=max(a-b,0)+b
</code></pre>
<p>so this can be implemented with O(log(n)) layers of ReLU. Is there a function in pytorch that already implements this?</p>

username: hnakao
cooked: <p>I haven’t verified but it should look like this</p>
<pre><code class="lang-auto">def hardmax(ls):
    half = []
    for i in range(0,len(ls),2):
        if i+1 == len(ls):
            half.append(ls[i])
        else:
            relu = F.relu(ls[i]-ls[i+1])+ls[i+1]
            half.append(relu)
    if len(half) == 1:
        return half[0]
    else:
        return hardmax(half)
</code></pre>

username: Matias_Vasquez
cooked: <p>Hi,</p>
<p>if I understand correctly, you want to get the maximum of two (or more) values.</p>
<p>Let´s say something like this</p>
<pre><code class="lang-python">ab = torch.randint(-5, 5, (5, 2))
# Output:
# tensor([[-3, -5],
#         [ 1, -4],
#         [-3, -2],
#         [ 1, -3],
#         [ 4, -4]])
</code></pre>
<p>Where the left column would be <code>a</code> and the right one would be <code>b</code> in your example.</p>
<p>You can then use the <code>max</code> function to get the maximum value.</p>
<pre><code class="lang-python">val, ind = ab.max(dim=1)

print(val)
# Output:
# tensor([-3,  1, -2,  1,  4])
</code></pre>
<p>By defining the dimension to 1, it will look for the maximum value in each row. (You also get the indices for the column where the maximum value is located if you want/need them).</p>
<p>Please let me know if this is not the answer you were looking for.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Adapting wav2vec2 to stream of int16

username: barakb
cooked: <p>Hi, Im trying to use wav2vec2 on a stream of audio built in the following way (this is not the all pipeline, just the important lines):</p>
<pre><code class="lang-auto">stream = pyaudio.PyAudio().open(
            format=pyaudio.paInt16,
            channels=self.CHANNELS,
            rate=16000,
            input=True,
            frames_per_buffer=self.INPUT_FRAMES_PER_BLOCK,
        )
data = stream.read(self.INPUT_FRAMES_PER_BLOCK)
data = np.frombuffer(data , np.int16)

</code></pre>
<p>It is important to mention that using my framework of streaming I enabled other deep algorithms, that works great.<br>
I’m followed this <a href="https://pytorch-tutorials-preview.netlify.app/intermediate/speech_recognition_pipeline_tutorial.html" rel="noopener nofollow ugc">tutorial</a>.<br>
Using the same pipeline (both in streamer and in offline) the only difference is that the offline is getting audio from torchaudio.load and other from stream which is pyint16.<br>
Before I’ll start to talk about the normalization , I’ll first mention that I’m sure that loading the weights is completed successfully and even tested using dummy inputs.</p>
<p>My is that it will work on online stream, how would I know that it is working, I’ll see some “make sense” transcripts.<br>
I saw that torchaudio used a normalization using a factor of (1&lt;&lt;31).<br>
Since I’m using a stream with format pyint16, I tried using (1&lt;&lt;15).<br>
Didn’t work so I tried to stream in a int32 format, and use the original factor, also didn’t work.<br>
I further tried all kinds of normalizations, but nothing work.<br>
Is there anything else I’m missing in order to make input stream aligned with what wav2vec2 expects to see?</p>
<p>Thanks for your help!</p>
Is there a way to compute the gradient w.r.t a part of samples?

username: chenyaofo
cooked: <p>Hi, I am looking for computing the gradient w.r.t a part of samples in PyTorch.</p>
<p>For example, I first compute the loss on 4 samples (batch_size=4). Then I want to compute the gradients of 2 samples among them depend on a <code>mask</code> (see the code). Note that the backward time of 2 samples should be around half of that of the whole 4 samples.</p>
<p>Is there a way to perform backward on a part of samples with less time cost in PyTorch? I have found that PyTorch realeased ‘functorch’ library, but it seems can not meet my needs.</p>
<pre><code class="lang-auto">
import torch

from torchvision import models

samples = torch.rand(4, 3, 224, 224) # batch_size = 4

model = models.resnet18()

outputs = model(samples)

loss = outputs.mean()

mask = torch.tensor([1,1,0,0]) 
# the mask means the gradients of the first two samples are required

# how to perform backward according to the mask with less time cost

</code></pre>
Median based loss instead of mean

username: redevil6990
cooked: <p>Is there a way to use the <em>median</em> as the reduction operation when computing the loss? I have this piece of code:</p>
<pre><code class="lang-auto">    output = model(data)
    loss = F.cross_entropy(output, target)
    loss.backward()
</code></pre>
<p>I would like to take the median of losses in a batch, instead of the mean, and get gradients based on that.</p>

username: mxahan
cooked: <p>In that case, you can write a custom function that does the following pseudocode</p>
<pre><code class="lang-auto">loss = elementwise_cross_entropy(output, target)
median_val_index = index of loss median value  
return loss[median_val_index]
</code></pre>

username: redevil6990
cooked: <p>Is there more detail on how to create a new loss function using this approach? I cannot find the elementwise_cross_entropy in functional.py.</p>

username: Matias_Vasquez
cooked: <p>You can specify <code>none</code> as the reduction method and get the median from the result.</p>
<pre><code class="lang-python">y_hat = torch.randn(3, 5, requires_grad=True)
y = torch.empty(3, dtype=torch.long).random_(5)

print(f"Outputs: \n{y_hat}\n")
print(f"Targets: \n{y}\n")

loss = torch.nn.CrossEntropyLoss()
loss_none = torch.nn.CrossEntropyLoss(reduction="none")

l_mean = loss(y_hat, y)
l_none = loss_none(y_hat, y)

print(f"Loss with 'mean' reduction: \n{l_mean}\n")
print(f"Loss with 'none' reduction: \n{l_none}\n")
print(f"Median loss with 'none' reduction: \n{torch.median(l_none)}")
</code></pre>
<pre><code class="lang-python"># Output:
Outputs: 
tensor([[ 2.0904,  0.2319,  0.0346,  0.5896,  1.3256],
        [ 0.4279,  0.6223, -0.1241,  1.8375,  0.7244],
        [-0.4159,  0.0537, -0.2241,  0.4073,  0.6651]], requires_grad=True)

Targets: 
tensor([1, 3, 1])

Loss with 'mean' reduction: 
1.6558622121810913

Loss with 'none' reduction: 
tensor([2.5377, 0.6982, 1.7317], grad_fn=&lt;NllLossBackward0&gt;)

Median loss with 'none' reduction: 
1.7316656112670898
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
How to push the torch::KInt() to GPU?

username: MartinZhang
cooked: <p>Today I define a custom c++ ops for pytorch</p>
<p>I use a constant with type is torch::KInt()</p>
<p>When I  load my module to the GPU</p>
<p>but there will a error about :  two device of cuda:0 and cpu</p>
<p>so , how to push the torch::KInt to CUDA?</p>
<p>thx.</p>

username: ptrblck
cooked: <p><code>tensor = tensor.to(torch::kCUDA);</code> should work.</p>

username: MartinZhang
cooked: <p>awesome! thx <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: MartinZhang
cooked: <p>Dear friend</p>
<p>Is there a way to specify the gpu id in C++, like python : <code>.to('cuda:0')</code>?</p>
<p>Or how to get a tensor’s device in C++?</p>
<p>thx, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>Yes, <code>.to({torch::kCUDA, 0});</code> should work where the latter <code>int</code> is the device id.</p>

username: MartinZhang
cooked: <p>awesome! , it can work.</p>
<p>thx <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Pytorch custom CUDA extension - no for loop needed?

username: sebastian-sz
cooked: <p>I’m following the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="noopener nofollow ugc">official tutorial</a> on writing custom CUDA extension I’m noticing and example CUDA kernel:</p>
<pre><code class="lang-auto">template &lt;typename scalar_t&gt;
__global__ void lltm_cuda_forward_kernel(
    const scalar_t* __restrict__ gates,
    const scalar_t* __restrict__ old_cell,
    scalar_t* __restrict__ new_h,
    scalar_t* __restrict__ new_cell,
    scalar_t* __restrict__ input_gate,
    scalar_t* __restrict__ output_gate,
    scalar_t* __restrict__ candidate_cell,
    size_t state_size) {
  const int column = blockIdx.x * blockDim.x + threadIdx.x;
  const int index = blockIdx.y * state_size + column;
  const int gates_row = blockIdx.y * (state_size * 3);
  if (column &lt; state_size) {
    input_gate[index] = sigmoid(gates[gates_row + column]);
    output_gate[index] = sigmoid(gates[gates_row + state_size + column]);
    candidate_cell[index] = elu(gates[gates_row + 2 * state_size + column]);
    new_cell[index] =
        old_cell[index] + candidate_cell[index] * input_gate[index];
    new_h[index] = tanh(new_cell[index]) * output_gate[index];
  }
}
</code></pre>
<p>Usually, when reading other CUDA examples, one can notice that the for loop is used. For example:</p>
<pre><code class="lang-auto">  for (int i = blockIdx.x * blockDim.x + threadIdx.x; i &lt; size;
       i += blockDim.x * gridDim.x) {
       //  (...)
</code></pre>
<p>I have two questions:</p>
<ol>
<li>Why is it not used (not necessary?) in the above case?</li>
<li>Is such for loop necessary if I wanted to create something simpler - a custom activation function, applied for each tensor element?</li>
</ol>
<p>Best regards,<br>
Sebastian</p>

username: ptrblck
cooked: <p>It depends on your use case and if you are writing a “monolithic kernel” or a “grid-stride loop”.<br>
<a href="https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops/">This tutorial</a> explains it using the <code>saxpy</code> example.</p>

username: sebastian-sz
cooked: <p>Thank you for the reply. This does answer my question.</p>
Autocast doesn't cast all FloatTensor to HalfTensor

username: Toby
cooked: <p>Hi, I am trying to use Automatic Mixed Precision of Pytorch and I’m facing a problem just like the title. I wrote a small example below, can someone tell me what’s wrong with my code?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; from torch.nn import functional as F
&gt;&gt;&gt; from torch.cuda.amp import autocast
&gt;&gt;&gt; torch.manual_seed(30)
&gt;&gt;&gt; torch.cuda.manual_seed(30)
&gt;&gt;&gt; x = torch.randn(3,4).cuda()
&gt;&gt;&gt; y_true = torch.tensor([3.,3.,3.]).cuda()
&gt;&gt;&gt; criterion = torch.nn.MSELoss()
&gt;&gt;&gt; model = torch.nn.Linear(4,1, bias = False)
&gt;&gt;&gt; model.cuda()
&gt;&gt;&gt; with autocast(enabled=True):
&gt;&gt;&gt;     out = model(x)
&gt;&gt;&gt;     loss = criterion(out, y_true)
&gt;&gt;&gt; print(out.type())
&gt;&gt;&gt; print(loss.type())
&gt;&gt;&gt; print(isinstance(out, torch.cuda.HalfTensor))
&gt;&gt;&gt; print(isinstance(loss, torch.cuda.HalfTensor))
torch.cuda.HalfTensor
torch.cuda.FloatTensor
True
False
</code></pre>
<p>I expect the <code>loss</code> should be in <code>float16</code> or am I wrong in the definition of AMP?</p>

username: ptrblck
cooked: <p><code>torch.cuda.amp.autocast</code> will cast to <code>float16</code> where possible and will cast or keep the precision in <code>float32</code> where it’s necessary as described <a href="https://pytorch.org/docs/stable/amp.html#op-specific-behavior">here</a>.<br>
The <code>float32</code> list contains <code>mse_loss</code> so the output is expected.</p>

username: Toby
cooked: <p>Thank you for your answer,<br>
My problem is the <code>RuntimeError: expected scalar type Half but found Float</code> at the line <code>scaler.scale(loss).backward()</code>. I want to print the internal params to check which are <code>HalfTensor</code> by:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; def check_half(model):
&gt;&gt;&gt;     for name, param in model.named_parameters():
&gt;&gt;&gt;         if isinstance(param, torch.cuda.HalfTensor):
&gt;&gt;&gt;             print(name)
&gt;&gt;&gt; with autocast(enabled=True):
&gt;&gt;&gt;     out = model(x)
&gt;&gt;&gt;     check_half(model)
</code></pre>
<p>However, it doesn’t work, do you have any suggestions?</p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet reproducing the issue so that I could take a look, please?</p>
When should I normalize my image data?

username: MUnique
cooked: <p>Hi,<br>
I’m working on a medical image database and I’ve trained a network that works fine on it without applying any normalization. But when I normalize the data the results deteriorate a little bit. Why should I normalize the images? Does it have anything to do with testing on unseen images from a new dataset?</p>

username: mxahan
cooked: <p>Normalization improves learning efficiency as the input and activation values fall in close ranges.</p>
Pytorchvideo create_r2plus1D

username: mxahan
cooked: <p>what is the input shape of pytorchvideo r(2+1)d model?</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/facebookresearch/pytorchvideo/blob/main/docs/source/models.md#models">these docs</a> I would assume the input shape to be <code>[batch_size, nb_channels=3, seq_len, height=224, width=224]</code>.</p>

username: mxahan
cooked: <p>Thank you very much. I almost gave up on this version.</p>
<p>It works for the following shape<br>
[B, 3, seq_len = 16, 224, 224]</p>
<p>Initially, I got confused about two things. The document in pytorchvideo shows their mode with ResNet shape of (B, 3, 8, 224, 224). And “<a href="https://pytorch.org/vision/0.8/_modules/torchvision/models/video/resnet.html#r2plus1d_18" rel="noopener nofollow ugc">Pytorchvision.models</a>” r2plus1D works with an input shape of [B, 3, 16, 112, 112].</p>
AudioMNIST Dataset with Pytorch

username: Sairam954
cooked: <p>Hi All,</p>
<p>Does anyone have experience working with AudiMNIST with pytorch, It is not available in torchaudio datasets, any information regarding this or a reliable git repo will be great.</p>
<p>Thanks,<br>
Sairam</p>

username: payam_jome_yazdian
cooked: <p>I was looking for the same thing and I found <a href="https://www.kaggle.com/code/jokekling/pytorch-study-audio/notebook" rel="noopener nofollow ugc">link1</a> and <a href="https://github.com/jayrodge/AudioMNIST-using-PyTorch" rel="noopener nofollow ugc">link2</a>.</p>
Weight/neuron based learning rate

username: razor
cooked: <p>I am trying to have individual learning rates for each neuron, that is, the learning rate becomes a matrix. How would anyone implement that for linear layers only?<br>
Thanks!</p>

username: mxahan
cooked: <aside class="quote no-group" data-username="razor" data-post="1" data-topic="148384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/razor/48/10543_2.png" class="avatar"> razor:</div>
<blockquote>
<p>am trying to have individual learning rates for each neuron, that is, the learning rate becomes a matrix. How would anyone implement that for linear layers only?<br>
Thanks!</p>
</blockquote>
</aside>
<p>You can do that by defining different optimizers for each of the neurons. This <a href="https://discuss.pytorch.org/t/how-to-set-different-learning-rate-for-weight-and-bias-in-one-layer/13450">link</a> may help you.</p>

username: JuanFMontesinos
cooked: <p>I think it’s not currently possible (in an automatic way). LR affects parameter groups (this is, objects belonging to a nn.Module). In this case you want different learning rates for each element in a tensor which is not supported (I think).<br>
It could be manually done defining each of the values of the tensor as a nn.parameter.</p>

username: AlphaBetaGamma96
cooked: <p>When you use optimizers like Adam or RMSprop? Don’t they effectively already do this? As they modify the raw gradient by some preconditioning which is dependent on each given weight?</p>

username: razor
cooked: <p><a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> No, they are not as far as I have checked their source.<br>
What happens when I try to add a matrix in the ‘lr’ parameter group (no matter if SGD, RMSProp etc), the optimizer will validate the value and fail before going further…</p>

username: AlphaBetaGamma96
cooked: <p>The learning rate of Adam or RMSprop is the ‘global’ learning rate the <em>actual</em> learning rate is determined by the exponential moving averages of each weight. This is what I mean, if you print out the exp. moving averages they should be the same shape as <code>param</code> and that gives you a learning rate based on each weight/bias of your network.</p>
Load, augment, split, batch pipeline with TorchData

username: the-dharma-bum
cooked: <p>Hi,</p>
<p>I’ve been toying with the new TorchData lib, and I came up with an implementation that seems overly complicated for something pretty simple, so I wonder if i missed a simpler way of doing things.<br>
Let’s say I’m doing semantic segmentation and I have a dataset folder containing two subfolders ‘images’ and ‘masks’ with identically named png files. I want to load a couple (image, mask) as PIL Image and augment the couple.  That would be the old <strong>getitem</strong>() method. Then I want to split into train and validation sets, shuffle and batch.  That would be the old <strong>DataLoader.</strong></p>
<p>Currently I’m doing this:</p>
<pre><code class="lang-python">image_files = FileLister([rootdir / 'images'], masks='*.png')
mask_files  = FileLister([rootdir / 'masks'],  masks='*_multiclass.png')
images = image_files.map(Image.open)
masks  = mask_files.map(Image.open)
dp = Zipper(images, masks).map(transform)
def train_val_split(x: dict) -&gt; bool: return x['index'] &gt;= split * length
train_dp, val_dp = dp.add_index().demux(num_instances=2, classifier_fn=train_val_split)
train_dp = train_dp.shuffle().batch(batch_size).collate(default_collate)
val_dp = val_dp.shuffle().batch(batch_size).collate(default_collate)
</code></pre>
<p>where <code>split</code> is a float in [0, 1], <code>length</code> is the number of couples (image, mask), <code>transform</code> is a function returning a dict[‘image’: tensor[3, H, W], ‘mask’: tensor[H, W]], and <code>default_collate</code> is <code>torch.utils.data.default_collate</code>.</p>
<p>So is this indeed too complicated and i there a cleaner way of doing things ?</p>
<p>Thanks !</p>

username: nivek
cooked: <p>Hi, thanks for trying out <code>torchdata</code>. I think your code snippet seems reasonable, readable, and isn’t too complicated. The only nit improvement I can think of to make it more succinct (but maybe less readable) is this:</p>
<pre><code class="lang-python">dp = Zipper(image_files, mask_files).map(open_and_transform)
</code></pre>
<p>You can still use <code>DataLoader</code>'s <code>shuffle</code>/<code>batch</code>/<code>collate</code> functionalities if you would like. Then you will not need <code>.shuffle().batch(batch_size).collate(default_collate)</code>.</p>
<p>Let us know if you have any suggestion to improve the API and make things more convenient for your use case!</p>
Why is autograd on by default?

username: robotrapta
cooked: <p>Sorry if this is an FAQ or completely irrelevant in 2022, but I’m wondering why autograd is enabled by default.  Wouldn’t it be better to have it off by default, and only enabled when needed?  In that case, any time you called <code>backward()</code> you would get a very clear error message telling you what you need to do.  Yes a lot of people would hit it, but there would be essentially zero confusion on the issue, and people would learn to do the right thing very fast.</p>
<p>Whereas today I feel like I’m always chasing down memory leaks caused by autograd being enabled when I don’t want it.  When anybody accidentally leaves autograd on during an evaluation, their workstation runs out of memory and becomes unresponsive needing a hard reboot.  In the big code bases we work in, it seems like the only way I can be sure there aren’t memory leaks is to turn off autograd when the program first loads, and explicitly turn it on when I need it.</p>
<p>I realize it would be a big breaking change at this point.  But I’m wondering if there’s good reasons I don’t see for having it be on by default?  Because I see genuine pain this causes today, and it really seems the other option would be much easier to manage.</p>
<p>Cheers</p>
Different results without optimizer.step()

username: cbd
cooked: <p>I used batch size of 8 and adam optimizer. In testing phase, i pass the images and got PSNR and SSIM. Now before testing i have pass some batches but make the loss manually 0 and do backpropagation without optimizer.step(). The results are different even loss is <strong>0</strong> and <strong>without</strong> <strong>optimizer.step().</strong></p>
<p>How the weights are changing?</p>

username: Matias_Vasquez
cooked: <p>Hi, would you mind sharing how your model architecture is built?</p>
<p>There are some layers that behave different in each pass in order to reduce overfitting. One example is the Dropout layer. With this little example you can see how the same layer behaves differently each time during training mode.</p>
<pre><code class="lang-python">drop = torch.nn.Dropout()
ones = torch.ones(3, 3)

drop.train()
print(f"Train: 1\n{drop(ones)}")
print(f"Train: 2\n{drop(ones)}")
print(f"Train: 3\n{drop(ones)}")
drop.eval()
print(f"Eval: \n{drop(ones)}")
</code></pre>
<pre><code class="lang-python">#Output:
Train: 1
tensor([[2., 0., 2.],
        [2., 0., 2.],
        [0., 0., 2.]])
Train: 2
tensor([[2., 2., 2.],
        [0., 2., 0.],
        [0., 2., 2.]])
Train: 3
tensor([[0., 2., 0.],
        [0., 0., 2.],
        [0., 2., 2.]])
Eval: 
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

username: cbd
cooked: <p>No drop out in architecture. The difference between 2 run is thant in 2nd run i just do backward() without optimizer.step() before testing so weight shoud not change. Even i manually changed the loss to 0 so there should not be any gradient flow during backpropagation. Even though PSNR SSIM is different. Used batchnorm as “nn.BatchNorm2d, affine=True, track_running_stats=True”</p>

username: Matias_Vasquez
cooked: <p>Could you post a minimal executable code that preproduces this behavior?</p>
<p>That way it will be easier to look into it.</p>

username: cbd
cooked: <p>After debugging, i come to know the things change in forward pass even if i comment backward. So in 2nd run i just do forward pass with few batches before testing without backpropagation and optimizer.step().  Any parameter (when batch norm is used or in some other case) change during forward pass?</p>

username: JuanFMontesinos
cooked: <p>If you “track running stats” with batchnorm it basically will update the statistics of the batchnorm module depending on the batch and the momentum.</p>
<p>use model.eval() for these comparisons.</p>

username: cbd
cooked: <p>Thanks. It really helps.</p>
Avoid tensor split with nn.DataParallel

username: Vardaan_Pahuja
cooked: <p>I am using a model whose forward pass contains some auxiliary tensors (which are agnostic of batch). I return them by updating in the forward pass. When I use <code>nn.DataParallel</code>, PyTorch splits them across 8 GPUs in the dim 0. Is there any way I can tell PyTorch not to split those tensors across the batch dim.? These auxiliary tensors are not a part of the computational graph.</p>

username: alibabadoufu
cooked: <p>Hi <a class="mention" href="/u/vardaan_pahuja">@Vardaan_Pahuja</a>, have you found the solution? I am having the same problem here.</p>

username: tshrjn
cooked: <p>I’m also looking for solution, facing the same problem.<br>
Have either of you been able to find a solution?</p>

username: ptrblck
cooked: <p>Are these auxiliary tensors large or could you repeat them?</p>

username: tshrjn
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, they’re quite large.</p>

username: cetinsamet
cooked: <p>Hi, <a class="mention" href="/u/tshrjn">@tshrjn</a> <a class="mention" href="/u/ptrblck">@ptrblck</a>, it has been a long time but did you find any proper way to handle this problem?</p>
<p>Let’s say i have an aux. tensor with shape (N, D) and i have 4 GPU device. I’m thinking of repeating my aux. tensor in a way that i’ll have a new (repeated) tensor with shape (4N, D) and when this new tensor is shared, each GPU process will end up with my original tensor (with shape (N,D) obviously).</p>
<p>I’m not sure if this is feasible (I’ll try it) and I’m wondering if there is an easy solution. Thanks.</p>

username: ptrblck
cooked: <p>Your aux tensors seem to include the batch dimension, so they should be chunked appropriately.<br>
The original post dealt with aux tensors, which are independent of the current batch size, thus would have to be repeated.</p>

username: mirbagheri
cooked: <p>For those of you still looking for a solution, I found out if the input data to the DataParallel module forward is a numpy array it won’t be split, and also no additional copies of it made on CPU memory. The transfer to GPU still needs to happen inside the forward. Also if you want to avoid split for only part of your data, if the input is a dict only those items with Tensor values will be split and the rest will be passed intactly.</p>
Correct way to subclass DataLoader __iter__ method to apply transforms to minibatch?

username: jlehrer1
cooked: <p>Hi,</p>
<p>I’m trying to add a transform to minibatches as they’re yielded from the <code>DataLoader</code> class. <em>I understand that I can do this in the <code>Dataset</code></em>, but in my case, I’m applying a numpy function whose performance is nearly identical sample-wise and mini-batch wise. Therefore, with <code>batch_size=4</code>, it’s 4x faster for me to apply this transform after the <code>DataLoader</code> is created.</p>
<p>The most obvious option is</p>
<pre><code class="lang-python">for X, y in dataloader:
     X = transform(X)
</code></pre>
<p>But I’d also like to use this DataLoader with <code>PyTorch-Lightning</code>, so I have to subclass it. I’ve done the following:</p>
<pre><code class="lang-auto">class SampleLoader(torch.utils.data.DataLoader):
    def __iter__(self):
        for batch in super().__iter__():
            yield transform(batch[0]), batch[1]
</code></pre>
<p>My question is, is this safe with <code>num_workers&gt;0</code> and is this correct? It appears to be with limited testing. Thanks in advance!!</p>

username: nivek
cooked: <p>You can also write a custom collation function that performs the transformation as part of collation and pass that to <code>DataLoader</code> (the argument name is <code>collate_fn</code>).</p>
<p>This <a href="https://pytorch.org/docs/stable/data.html#automatic-batching-default" rel="noopener nofollow ugc">part of the documentation</a> talks about automatic batching and the application of <code>collate_fn</code> within <code>DataLoader</code>.</p>
How to judge an empty tensor?

username: lxtGH
cooked: <p>I want to judge whether a tensor is an empty tensor</p>

username: jpeg729
cooked: <p>How about this?</p>
<pre><code class="lang-auto">len(tensor.size()) == 0</code></pre>

username: lxtGH
cooked: <p>Yes, it works<br>
empty_tensor = Variable(torch.FloatTensor())<br>
len(empty_tensor) ==0<br>
True<br>
,I mean if there are other ways like if pytorch keyword about it</p>

username: richard
cooked: <p><code>x.nelement() == 0</code> works too.</p>

username: jpeg729
cooked: <p>Hmmm, if length is an issue then <code>len(x)==0</code> wins. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=5" title=":grin:" class="emoji" alt=":grin:"><br>
Plus, it is pythonic, though not specifically pytorchic.</p>

username: drevicko
cooked: <p>just beware that you can have a tensor with no elements and non-trivial size.</p>
<pre><code class="lang-auto">t = torch.zeros(size=(1,0))
t.nelement()   # returns zero, empty in this sense
len(t.size())  # returns two, not empty in this sense!!
len(t)         # returns one, not empty in this sense!!

t = torch.zeros(size=(0,1))
len(t)         # returns zero for this one!
</code></pre>

username: yingda.yin
cooked: <p>as <a class="mention" href="/u/drevicko">@drevicko</a> mentioned<br>
<code>len(x) == 0</code> may not be a good choice, since a tensor may have more than 0 dimensions, but still be empty.<br>
<code>x.nelement()</code> or <code>x.numel()</code> should be good.</p>
I don't know the object detection error

username: NagaYu
cooked: <p>I haven’t been able to learn. I don’t know what’s wrong because no error occurs.</p>
<pre><code class="lang-auto">model.train()
for epoch in range(num_epoch):
    loss_hist.reset()
    
    for i, (images, targets, ImageIDs) in enumerate(train_loader):

        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)
        
        losses = sum(loss for loss in loss_dict.values())
        loss_value = losses.item()
        
        loss_hist.send(loss_value)
        
        optimizer.zero_grad()
        losses.backward()
        optimizer.step()

        if i%100 == 0:  
            print("epoch:", epoch,  "iteration:", i,  "loss:", losses.item())
</code></pre>
<p>/Users/nagaoyuuta/opt/anaconda3/lib/python3.9/site-packages/torch/nn/functional.py:780: UserWarning: Note that order of the arguments: ceil_mode and return_indices will changeto match the args list in nn.MaxPool2d in a future release.<br>
warnings.warn(“Note that order of the arguments: ceil_mode and return_indices will change”</p>
<p>epoch: 0 iteration: 0 loss: 66.70726178733568</p>

username: Andrei_Cristea
cooked: <p>NagaYu - is that your entire stdout? It looks like you’re running your model for a small number of steps (&lt; 100) since you’re only printing out the loss once.</p>

username: NagaYu
cooked: <p>Yes. The output layer. Do the loss functions need to be the same?</p>

username: Andrei_Cristea
cooked: <p>It seems like you are training your model for too few steps. You need to train it for more steps before the training loss converges. Your current example only includes output from the very first model run, when it’s totally untrained.</p>

username: NagaYu
cooked: <p>Need to step over 100 k?</p>

username: Andrei_Cristea
cooked: <p>If this is the only output you get from your script:</p>
<p><code>epoch: 0 iteration: 0 loss: 66.70726178733568</code></p>
<p>Then it means you only ran your training for at most 100 steps, since otherwise this condition would have triggered again and you would have gotten another line printed out:</p>
<pre><code class="lang-auto">if i%100 == 0:  
    print("epoch:", epoch,  "iteration:", i,  "loss:", losses.item())
</code></pre>
<p>Maybe this is not the problem and you truncated your output before posting it here, but if you didn’t truncate then you should double-check that your <code>train_loader</code> has as much data in it as you think, and that your <code>num_epochs</code> are high enough. If you did truncate, please post more of the output (meaning, losses for more iterations and more epochs than just epoch=0, iteration=0).</p>
<p>Put differently, what you posted does not show whether or not your model is training.</p>

username: NagaYu
cooked: <p>I understand. Check where to check. Thank you for your polite explanation.</p>
Torch::save and torch::load compilation errors - no overloaded operator definition

username: cdcdcd
cooked: <p>I’ve successfully created a model using libTorch and visual studio 2019 (compiler version: 19.29.30141).</p>
<p>However, when I try to save it via torch::save(model, “model.pt”):</p>
<p><em>torch::save(model, “model.pt”);</em></p>
<p>model is of type Net:</p>
<p><em>struct Net : torch::nn::Module {}</em></p>
<p>I get the following visual studio compilation error:</p>
<p><em>|Error|C2679|binary ‘&lt;&lt;’: no operator found which takes a right-hand operand of type ‘const Value’ (or there is no acceptable conversion)</em></p>
<p>When I dig into this I find the error is actually in serialise.h:</p>
<p><em>template &lt;typename Value, typename… SaveToArgs&gt;</em><br>
<em>void save(const Value&amp; value, SaveToArgs&amp;&amp;… args) {<br>
serialize::OutputArchive archive(<br>
std::make_shared<a>jit::CompilationUnit</a>());<br>
archive &lt;&lt; value;<br>
archive.save_to(std::forward(args)…);</em>}*</p>
<p>The issue is the overloading of the “&lt;&lt;” operator (<em>archive &lt;&lt; value</em>).</p>
<p>When I look further I find OutputArchive (which is a closed class) has no overload definition for “&lt;&lt;” so this will never build. The documentation suggests that the overload can be defined elsewhere but this is a <strong>final</strong> class (closed).</p>
<p>Is this a bug in libTorch or am I missing something - which is the usual case (sorry).</p>
<p>Any help much appreciated.</p>

username: cdcdcd
cooked: <p>The solution is here from mhubi (builds but I haven’t tested it):</p>
<aside class="quote quote-modified" data-post="5" data-topic="34234">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mhubii/48/71033_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/libtorch-how-to-save-model-in-mnist-cpp-example/34234/5">(libtorch) How to save model in MNIST cpp example?</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" style="--category-badge-color: #9EB83B; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    I think, the example was written prior to the stable release of libtorch. The way you would implement the torch::nn::Module now is as follows 
struct NetImpl : torch::nn::Module {       // replaced Net by NetImpl
  NetImpl()                                // replaced Net by NetImpl
      : conv1(torch::nn::Conv2dOptions(1, 10, /*kernel_size=*/5)),
        conv2(torch::nn::Conv2dOptions(10, 20, /*kernel_size=*/5)),
        fc1(320, 50),
        fc2(50, 10) {
    register_module("conv1", conv1);
 …
  </blockquote>
</aside>

<p>The key here is inclusion of TORCH_MODULE(Net). See example.</p>
Error: to many values to unpack

username: Pytorch_developer
cooked: <p><strong>I have some code:</strong></p>
<p>from torchvision.datasets import ImageFolder<br>
import torchvision.transforms as T<br>
from torch.utils.data import DataLoader<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as f<br>
import torch.optim as optim</p>
<p>path ="/Users/edenbrown/Downloads/kagglecatsanddogs_3367a/PetImages"<br>
transform = T.Compose([T.Resize((50, 50)),T.ToTensor()])</p>
<p>dataset = ImageFolder(root=path, transform=transform)</p>
<p>dataloader = DataLoader(dataset, batch_size=10, shuffle=True)</p>
<p>for batch_number, (images, labels) in enumerate(dataloader):<br>
break</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(50 * 50, 64)
    self.fc2 = nn.Linear(64, 64)
    self.fc3 = nn.Linear(64, 64)
    self.fc4 = nn.Linear(64, 2)

def forward(self, x):
    x = f.relu(self.fc1(x))
    x = f.relu(self.fc2(x))
    x = f.relu(self.fc3(x))
    x = self.fc4(x)
    return f.log_softmax(x, dim = 1)
</code></pre>
<p>net = Net()</p>
<p>optimizer = optim.Adam(net.parameters(), lr = 0.001)</p>
<p>Epochs = 3</p>
<p>for epoch in range(Epochs):<br>
for data in labels, images:<br>
a, b = data<br>
net.zero_grad()<br>
output = net(a.view(-1, 50 * 50))<br>
loss = f.nll_loss(output, b)<br>
loss.backward()<br>
optimizer.step()<br>
print(loss)</p>
<p><strong>I am getting this error:</strong></p>
<p>Traceback (most recent call last):<br>
File “/Users/edenbrown/sample.ws45/new.py”, line 45, in <br>
a, b = data<br>
ValueError: too many values to unpack (expected 2)</p>
<p><strong>Is there a way to fix this error.</strong></p>

username: ptrblck
cooked: <p>Based on the code and error it seems that you are trying to iterate:</p>
<pre><code class="lang-python">for data in labels, images:
</code></pre>
<p>instead of the <code>dataloader</code> and later unpack <code>data</code> via:</p>
<pre><code class="lang-python">a, b = data
</code></pre>
<p>Check that <code>data</code> can be unpacked and make sure you really want to iterate <code>labels, images</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

username: Pytorch_developer
cooked: <p><strong>Just so you know I changed:</strong></p>
<p>for data in labels, images:</p>
<p><strong>into:</strong></p>
<p>for data in images, labels</p>
<p><strong>What I am trying to do is make a = labels, b = images but for the one image which is being processed through the for loop. So basically, if the image is a dog the label should be one and the image variable shall be the tensor of that image. Therefore for that one image, a should = 1 and b should = the correct tensor.</strong></p>

username: Pytorch_developer
cooked: <p>I do not need a answer anymore I figured this out:</p>