akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The difference between torch.tensor.data and torch.tensor

auroua: <p>I write the following code:</p>
<pre><code class="lang-auto">a = torch.rand(3, 5)
type(a)                      #  torch.Tensor
type(a.data)              #  torch.Tensor
id(a)                          #  4493764504
id(a.data)                 #  4493475488
</code></pre>
<p>I don’t understand the difference between Tensor <code>a</code> and Tensor <code>a.data</code>, and when to use <code>a</code> and when to use <code>a.data</code>.<br>
Thanks.</p>

albanD: <p>Hi,</p>
<p>The <code>.data</code> field is an old field that is kept for backward compatibility but should not be used anymore as it’s usage is dangerous and can make computations wrong. You should use <code>.detach()</code> and/or <code>with torch.no_grad()</code> instead now.</p>

mital: <p>I noticed it’s still in <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="noopener nofollow ugc">this tutorial</a></p>
<blockquote>
<p>_, predicted = torch.max(outputs.data, 1)</p>
</blockquote>

Ido_Amos: <p>Hi <a class="mention" href="/u/alband">@albanD</a> , if data should not be used can you please explain how to execute the following while still getting the model with the adjusted weights?<br>
when I run:</p>
<pre><code class="lang-auto">i = 1
for w in LinModel.parameters():
    torch.nn.init.eye_(w)
    w.data = w.data*i
    i += 1
</code></pre>
<p>the model weights are adjusted properly by a factor of i after the loop ends. if I don’t use data then just the identity init is saved.</p>

albanD: <p>Hi,</p>
<p>You can do either</p>
<pre><code class="lang-auto">with torch.no_grad():
  w *= i
</code></pre>
<p>Or if you need cannot do it inplace directly:</p>
<pre><code class="lang-auto">with torch.no_grad():
  w.copy_(w * i)
</code></pre>

bpfrd: <p>if we do an inplace operation on the data property of a leaf variable with requires_grad=true  it seems fine. But what is dangerous exactly? is it safe to use it in pixelcnn code (<a href="https://github.com/axeloh/pixelcnn-pytorch/blob/master/masked_cnn_layer.py#L17" class="inline-onebox" rel="noopener nofollow ugc">pixelcnn-pytorch/masked_cnn_layer.py at master · axeloh/pixelcnn-pytorch · GitHub</a>)? does it still track the gradients?</p>

soulitzer: <p>Normally if you make in-place updates on a tensor it would update a version tracker that would error out if autograd realizes that the original tensor is needed for gradient computation during backward.</p>
<p>But if you save a tensor for backward, and then mutate its data property in-place, and then backward, you lose the protection of version counter checking and the results would be silently incorrect.</p>
Hooks in PyTorch

Suraj520: <p>I was trying to debug a CRNN model and in one of the modular class implementation of model inherited from nn.module, I saw forward and backward hooks.</p>
<p>What are they used for, Practically ? Saw some documentation w.r.t gradients flow but couldn’t foresee its practical usecase.</p>
<p>Thanks!</p>

soulitzer: <p>You can use them to inspect the values are they are passed during forward, or inspect the gradients as they flow during backward to see if they are what you expect.</p>
pytorch model eval shows progress bar and epochs

mossishahi: <p>after loading a pretrained torch lightning model, I assume that I should run model.eval() and it turns off the gradiants and dropouts, still, Unexpectedly, the progress bar is shown and some epochs start to proceed.  Does anyone have any idea what’s the probable re</p>

ptrblck: <p>Calling <code>model.eval()</code> will disable dropout layers but will not freeze the model. To freeze parameters you would need to set their <code>requires_grad</code> attribute to <code>False</code>.<br>
I’m not familiar enough with Lightning and don’t know what this progress bar represents.</p>
Is there any reason ios metal libtorch using MPSImage to make tensor?

seastar105: <p>It seems MetalTensor is created using MPSImage, from <a href="https://github.com/pytorch/pytorch/blob/d253d6ec0c1c086b9d3be98b421d224ff20b734e/aten/src/ATen/native/metal/mpscnn/MPSImageWrapper.mm" rel="noopener nofollow ugc">here</a></p>
<p>due to its limitation it seems not possible to create tensor whose length is longer than 16384, or 8192, due to MTLTextureDescriptor (width, height limit)</p>
<p>this is problematic when processing speech data, usually longer than such limit. with error,</p>
<pre><code class="lang-auto">-[MTLTextureDescriptorInternal validateWithDevice:], line 1344: error 'Texture Descriptor Validation
MTLTextureDescriptor has width (160000) greater than the maximum allowed size of 16384.
</code></pre>
<p>here’s question, is there any reason to implement MetalTensor using MPSImage, not using MTLBuffer? let me know if i understand wrong</p>

ssjia: <p>Hi, so the primary reason MetalTensor uses MPSImage is that we had image processing models in mind when creating the Metal backend, and in that context image tensors were more performant especially since we were using MPSCNN for key operators such as Convolution which <a href="https://developer.apple.com/documentation/metalperformanceshaders/mpscnnkernel" rel="noopener nofollow ugc">consumed and produced MPSImages</a>.</p>
<p>Of course, as you’ve mentioned MPSImage does not work well for speech models due to the tensor shapes having different properties, and the plan was to eventually offer the option to use MTLBuffer for these use cases but unfortunately we never got around to that due to the focus shifting to enabling the <a href="https://pytorch.org/tutorials/prototype/ios_coreml_workflow.html" rel="noopener nofollow ugc">CoreML delegate</a> which we found to be more performant than the Metal backend in most cases. I would recommend you give the CoreML delegate a try since you are having issues with Metal.</p>

seastar105: <p>Thanks for answer, it seems convert model using CompileSpec seems really similar to converting pytorch model into .mlmodel using coremltools. what’s the difference between <code>_jit_to_backend('coreml')</code> and <code>coremltools.convert</code>?</p>

ssjia: <p>I believe <code>jit_to_backend('coreml')</code> calls <code>coremltools.convert</code> under the hood. The primary difference is that <code>coremltools.convert</code> produces a CoreML model, but <code>jit_to_backend('coreml')</code> will wrap the CoreML model with a torchscript wrapper so that you can still use torchscript to execute the model.</p>
How can I normalize after masking?

Ahmad_Pouramini: <p>Suppose I have a tensor and a mask.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a
tensor([[0.6000, 0.7000, 0.4000],
        [0.6000, 0.7000, 0.4000]])
&gt;&gt;&gt; mask = torch.tensor([[1,0,0],[1,1,1]])
&gt;&gt;&gt; a * mask
tensor([[0.6000, 0.0000, 0.0000],
        [0.6000, 0.7000, 0.4000]])
&gt;&gt;&gt; F.softmax(a, 1)
tensor([[0.3420, 0.3780, 0.2800],
        [0.3420, 0.3780, 0.2800]])
</code></pre>
<p>However, I desire:</p>
<pre><code class="lang-auto">
tensor([[1., 0., 0.],
        [0.3420, 0.3780, 0.2800]])
</code></pre>

Ahmad_Pouramini: <p>I got my answer in StackOverFlow:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/75388512/how-can-i-normalize-after-masking/75388569#75388569">
  <header class="source">

      <a href="https://stackoverflow.com/questions/75388512/how-can-i-normalize-after-masking/75388569#75388569" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/21173703/outgaze" target="_blank" rel="noopener nofollow ugc">
    <img alt="outgaze" src="https://www.gravatar.com/avatar/0123963c8fef97ec13b6a839c3c72efd?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/75388512/how-can-i-normalize-after-masking/75388569#75388569" target="_blank" rel="noopener nofollow ugc">How can I normalize after masking?</a>
</h4>

<div class="tags">
  <strong>python, torch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/21173703/outgaze" target="_blank" rel="noopener nofollow ugc">
    outgaze
  </a>
  on <a href="https://stackoverflow.com/questions/75388512/how-can-i-normalize-after-masking/75388569#75388569" target="_blank" rel="noopener nofollow ugc">04:07PM - 08 Feb 23 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Memory/Performance of Training Wav2vec2 Model

Douglas_Python: <p>Hey All,</p>
<p>I am rather new with Wav2Vec 2.0 working with it since december for a speech emotion recognition project. I started with torchaudio.pipelines bundle of Wav2vec 2 base model and put a Classification Head on top of it and freeze the transformers CNN-Layers, but fine-tune the rest on my emotion recognition task. I could work with the last layer (12) output, but I’d like to rather acces output of layer 10.<br>
I did this with model.extract_features(). How to use the model without storing each layers Output? I have a batch of 32 of length 400000… Therefore ( the reason I suspect) I can’t make it run on a GPU (Feature Layers leading to 12 x 32 x 1400 x 768 tensor storage). There is the argument num-layers, which outputs only the layers output up to a number (why only int instead of list??). I manipulated components.py to only return the layer I want ( nice! I am able to run batches of 32 instead of previous maximum of 25 on my local CPU). I also tried the transformer method and used model(x).last_hidden_state inside my Classification head model… With everything I tried I still get Memory overload of more than 80GB on gpu. I guess I am not using the right method for what I want to achieve… please help me! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I don’t know how the <code>extract_features</code> method is implemented, but your approach of returning the desired activation/feature sounds valid.<br>
Alternatively, you could also try to use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> assuming you are interested in an output of a specific <code>nn.Module</code>.<br>
If you don’t want to train the model, you could also wrap the forward pass into a <code>with torch.no_grad()</code> statement to delete the intermediate activations which would otherwise be needed for the gradient computation.</p>

Douglas_Python: <p>Thank you very much for the quick reply!<br>
I think I can resolve this<br>
Yes, the storing of all transformer layers output is bad (If you only need certain layers).<br>
I implemented a method where I slice the original Wav2vec2 base model up to the transformer layer, which I desire the output from.<br>
By doing this and freezing the first 20 layers (I’d like to only freeze the CNN layers) for training and setting batch to 20 instead of 32 i am able to run it on a GPU with ~80GB Memory. Wish I had more memory <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
Module 'torch' has no attribute 'amp'

Teddy_547: <p>Hey, I’ve been having trouble to see what’s wrong here. I have torch 1.8.1+cu111<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8eba7e65cf7c4c94c662e1897dba3dd31e4f452.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a8eba7e65cf7c4c94c662e1897dba3dd31e4f452" title="Screen Shot 2023-02-07 at 10.37.39 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8eba7e65cf7c4c94c662e1897dba3dd31e4f452_2_689x143.jpeg" alt="Screen Shot 2023-02-07 at 10.37.39 AM" data-base62-sha1="o6l6x63zjvka6XIgj6gweSrQ8dY" width="689" height="143" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8eba7e65cf7c4c94c662e1897dba3dd31e4f452_2_689x143.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8eba7e65cf7c4c94c662e1897dba3dd31e4f452_2_1033x214.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8eba7e65cf7c4c94c662e1897dba3dd31e4f452_2_1378x286.jpeg 2x" data-dominant-color="F6E6E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-02-07 at 10.37.39 AM</span><span class="informations">2327×485 94.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I believe the <code>torch.amp</code> namespace was added in PyTorch <code>1.12.0+</code> after mixed-precision training was implemented for the CPU.<br>
In older versions, you would need to use <code>torch.cuda.amp</code> instead.</p>

Teddy_547: <p>Thank you! The issue was resolved</p>
Understanding __torch_function__

Johannes_Messner: <p>Hi folks,</p>
<p>I am subclassing <code>torch.Tensor</code>, and am struggling to understand <code>__torch_function__</code>, and specifically its <strong><code>types</code></strong> argument.</p>
<ul>
<li>What are expected inputs?</li>
<li>And what is the intended use?</li>
</ul>
<p>I did read through <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-with-a-tensor-like-type" rel="noopener nofollow ugc">this</a> page, where it says:</p>
<blockquote>
<p>The <code>__torch_function__</code> method takes four arguments: <code>func</code>, a reference to the torch API function that is being overridden, <strong><code>types</code>, the list of types of Tensor-likes that implement <code>__torch_function__</code></strong>, <code>args</code>, the tuple of arguments passed to the function, and <code>kwargs</code>, the dict of keyword arguments passed to the function.</p>
</blockquote>
<p>Unfortunately, I can’t gain much from that explanation. Where do these types come from, and what are they the types of?</p>
<p>From playing around a little, I observed that <code>types</code> always seems to be a 1-tuple containing the type of my subclass, never anything else.</p>
<p><strong>Motivation for this question:</strong><br>
I have encountered a bug in my project, and the following seems to fix it (in my subclass):</p>
<pre><code class="lang-auto">@classmethod
def __torch_function__(cls, func, types, args=(), kwargs=None):
    return super().__torch_function__(func, (torch.Tensor,), args, kwargs)
</code></pre>
<p>But not understanding <code>__torch_function__</code> and its <code>types</code> argument properly, I am not confident about how good of an idea this is.</p>
<p>Any help appreciated, thanks in advance!</p>

samsja: <p>I am also interested in understanding what the types argument represent. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

Johannes_Messner: <p>In case someone else comes across this, I took some time to write down what I learned since asking this question <a href="https://github.com/docarray/notes/blob/main/blog/02-this-weeks-in-docarray-01.md#__torch_function__--or-how-to-give-pytorch-a-little-bit-more-confidence" rel="noopener nofollow ugc">here</a></p>
Expected all tensors to be on the same device, but found at least two devices , cpu and cuda:0!

ays: <p>I keep on getting this error(shown below), but when I print the location of the tensors, its all on cuda, kindly assist.</p>
<pre><code class="lang-auto">map_reg_code = self.mapping(reg_code)
return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)
</code></pre>
<pre><code class="lang-auto">class MLP_pretrained_model(torch.nn.Module):
    def __init__(self, pretrained_model_lvl3):
        super(MLP_pretrained_model, self).__init__()
        self.pretrained_model_lvl3 = pretrained_model_lvl3
        self.hidden = 64
        
        self.input_dims =  491776
        self.output_dims = 2 #predicting the dice and jacobian


        mapping_input = 64
        self.mapping_output = 256

        self.max_pool = torch.nn.MaxPool2d(kernel_size = 3, stride= 2, padding = 1)
        self.fc1 = torch.nn.Linear(self.input_dims, self.hidden) #added
        self.relu = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(self.hidden, self.hidden)
        self.fc3 = torch.nn.Linear(self.hidden, self.output_dims)

        self.mapping = nn.Sequential(
        # nn.Linear(1, mapping_input),
        nn.Linear(3, mapping_input),
        nn.ReLU(),
        nn.Linear(mapping_input, mapping_input),
        nn.ReLU(),
        nn.Linear(mapping_input, mapping_input),
        nn.ReLU(),
        nn.Linear(mapping_input, self.mapping_output),
    )
    
    def forward(self, x, y, reg_code, pretrained):

        if pretrained == True:
            e0 = self.pretrained_model_lvl3(x, y, reg_code, pretrained) 
            e0 = self.max_pool(e0)
            encoded = e0.flatten(1, -1)
            map_reg_code = self.mapping(reg_code)
            encoded_reg_code = torch.cat([encoded, map_reg_code], dim= 1)
            # fc = nn.Linear(encoded_reg_code.shape[1], self.hidden)
            encoded_reg_code = self.fc1(encoded_reg_code)
            encoded_reg_code = self.relu(encoded_reg_code)
            encoded_reg_code = self.fc2(encoded_reg_code)
            encoded_reg_code = self.relu(encoded_reg_code)
            encoded_reg_code = self.fc3(encoded_reg_code)
        
            

            return encoded_reg_code
        else:
            return self.pretrained_model_lvl3(x, y, reg_code, pretrained)
            

</code></pre>

thecho7: <p>Check where both model and <code>reg_code</code> are allocated by<br>
<code>print(your_model.device)</code> and <code>print(reg_code.device)</code> outside the forward function.</p>
<p>If they are the different, copy the one which is allocated on CPU by</p>
<pre><code class="lang-auto">device = torch.device('cuda:0')
if reg_code.device ==  torch.device("cpu"):
    reg_code = reg_code.to(device)
elif your_model.device == torch.device("cpu"):
    your_model = your_model.to(device)</code></pre>

ays: <p>Thanks for your help!</p>
Is there any way to do multi-embedding fastly?

chenfei: <p>I have 100-dim category features and each value range is different. I want to map them to 16-dimensional vectors respectively:</p>
<pre><code class="lang-auto">embedding_layer_list = nn.ModuleList()
for i in range(100):
  embedding_layer_list.append(nn.Embedding(feature_range, 16))

# forward
for i in range(100):
  # forward each feature
</code></pre>
<p>Is there any way to do multi-embedding instead of for-loop?</p>

J_Johnson: <p>Given the nature of embedding layers, why not just use:</p>
<pre><code class="lang-auto">embedding_layer=nn.Embedding(100*feature_range, 16)
</code></pre>
<p>At the end of the day, you are either calling 100 smaller tables or one bigger table. The result is the same.</p>
<p>Then just ensure your tokens are in the range of 0 to 100*feature_range - 1.</p>
How can I use a multi-indexed Dataset?

dryan: <p>I have several timeseries that I want to classify with RNNs. I would like to have the get_item method take in either a dict or tuple, specifying both time idx and sequence idx. I have implemented that here.</p>
<pre><code class="lang-auto">class MLDataWrangler(zrfr.ZRFReader, torch.utils.data.Dataset):
   ...
   ...
  
  def __len__(self) -&gt; int:
    # return len(self._all_flexions) # number of timeseries's
    return len(self._all_flexions), self._sequence_length # can I return multiple lengths?

  def __getitem__(self, idx) -&gt; Tuple[np.ndarray, Dict[str, bool]]:
    if isinstance(idx, dict):
      frames = idx["frames"]
      sequence = idx["flexion"]
    elif hasattr(idx, "__iter__"):
      frames = idx[1]
      sequence = idx[0]
    

    zrf_path = self._all_flexions[sequence]
    video_pixels = self.read(zrf_path).data


    # signal level (input)
    if self._signame == SIGNAL_NAMES[1]:
      img_pixels = video_pixels.zsig[frames].astype(np.float32) # new since stacked
    elif self._signame == SIGNAL_NAMES[0]:
      img_pixels = video_pixels.rfsig[frames].astype(np.float32) # new since stacked


    # get finger ground truth
    labels = self._data_labels[sequence][1] # since stacked
    labels = np.array([labels[fing] for fing in FINGER_NAMES]).astype(np.float32) # convert dict to np array

    # transform as necessary
    if self.transform:
      img_pixels = self.transform(img_pixels)
    if self.target_transform:
      labels = self.target_transform(labels)
    return img_pixels.astype(np.float32), labels
</code></pre>
<p>My question is how do I use DataLoader on this? I assume I need a custom Sampler but I’ve tried searching and can’t figure out what I need to implement.</p>

ptrblck: <p>I don’t know if creating a custom <code>sampler</code> yielding multiple indices would work, as I would expect to see issues in the <code>fetch</code> method, e.g. <a href="https://github.com/pytorch/pytorch/blob/230c4fe93d370e2548ab9cfc74a6810905bf7680/torch/utils/data/_utils/fetch.py#L53-L61">here</a>.<br>
However, you might be able to use a <code>BatchSampler</code> as described in <a href="https://discuss.pytorch.org/t/force-dataloader-to-fetch-batched-index-from-custom-batch-sampler/86656/2">this post</a> which would allow you to pass e.g. a <code>list</code> of indices to the <code>Dataset.__getitem__</code> method. You might need to “encode” this list (e.g. first part is <code>index0</code>, second part is <code>index1</code> etc.), but it could work.</p>
Onnx export from torch quantized model

chowk1109: <p>Hello</p>
<p>i made an own my quantized resnet50 classification model.<br>
i already checked that i can export fp32 trained model to onnx model.<br>
and i tried to export int8 quantized model to onnx.<br>
it works, but i wanna know that I wonder if Pytorch actually supports it now(export Quantized model to onnx).</p>
<p>i am using torch version 1.13</p>
Remove tensors elements by mask

Ameen_Ali: <p>Let’s say I have a tensor of shape a = [Batchsize , 80 , 2048].<br>
and I have another mask tensor of shape b = [Batchsize] containing 0 , 1. I want to remove the element i from tensor a which satisfy b[i] = 1 , and keep the other tensors in the same order.<br>
so far I did it in for loops, but I’m sure there is another fast way to do it. Thanks</p>

ptrblck: <p>You could index <code>a</code> by keeping all samples, which meet the requirement <code>b!=1</code>:</p>
<pre><code class="lang-python">a = torch.randn([10, 80, 2])
b = torch.randint(0, 2, (10,))

res = a[b!=1]
</code></pre>

Noob_Coding: <p>Sorry for bothering you, but I have a similar problem. For example:<br>
a = [Batchsize , 80 , 2048]<br>
b = [Batchsize] containing labels 0, 1,2,3, or 4. For each batch, I want to remove the element ith from the tensor, which satisfies i = b[i] (label of the corresponding batch) and keeps the other tensors in the same order.<br>
I would like to know the effective way to solve this problem. Thank you for your support.</p>
`torch.tanh` does not appear to be working properly

ha2ka: <p>Hi,</p>
<p>I am currently experimenting with automatic differentiation.<br>
I wrote the following script, but the error signal returned from <code>torch.tanh</code> is not what I expected.</p>
<p>I expect the error signal returned from tanh to be 0.9151. Because the input of <code>torch.tanh</code> is 0.3 of tensor in the following script, so its derivative must be <code>1.0 - tanh(0.3)^2.0 = 0.9151</code> of tensor.</p>
<p>Please let me know if any part of my understanding is incorrect.</p>
<pre><code class="lang-python">import torch


class LinearImpl(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, w):
        y = x @ w
        ctx.save_for_backward(x, w)
        return y

    @staticmethod
    def backward(ctx, e):
        x, w = ctx.saved_tensors
        grad_x = grad_w = None
        print(e)

        if ctx.needs_input_grad[1]:
            grad_w = x.t() @ e

        if ctx.needs_input_grad[0]:
            grad_x = e @ w.t()

        return grad_x, grad_w


class Linear(torch.nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.weight = torch.nn.Parameter(torch.zeros(in_dim, out_dim) + 0.3)

    def forward(self, x):
        return LinearImpl.apply(x, self.weight)


x = torch.zeros(2, 2, requires_grad=True) + 0.5
fc1 = Linear(2, 3)
fc2 = Linear(3, 2)

y = fc2(torch.tanh(fc1(x)))
y.sum().backward()
</code></pre>

ptrblck: <p>The <code>tanh</code> derivative shows the expected value using:</p>
<pre><code class="lang-python">x = torch.tensor(0.3, requires_grad=True)
y = torch.tanh(x)
y.backward()
print(x.grad)
# tensor(0.9151)
</code></pre>
<p>and also shows the same value in your code if you remove <code>fc2</code>.</p>

ha2ka: <p>Thank you for you reply.</p>
<p>I forgot that the derivative of tanh is multiplied by the error signal from the previous node.<br>
Sorry for the obvious question.</p>
Given groups=1, weight of size [256, 2048, 1, 1], expected input[2, 960, 34, 46] to have 2048 channels, but got 960 channels instead

Skylixia: <p>Hi all,</p>
<p>I’m trying to train the deeplabv3_mobilenet_v3_large model (pretrained). When, I do model(imgs) with imgs of size: torch.Size([2, 3, 544, 728]) thus batch of size 2, RGB and image dimensions. I get this error: Given groups=1, weight of size [256, 2048, 1, 1], expected input[2, 960, 34, 46] to have 2048 channels, but got 960 channels instead.<br>
Why is this error raised ? I don’t have this issue with the other deeplablab models.</p>
<p>Thanks in advance for the help!</p>

ptrblck: <p>I cannot reproduce the issue with the given input shape:</p>
<pre><code class="lang-python">model = models.segmentation.deeplabv3_mobilenet_v3_large()
x = torch.randn(2, 3, 544, 728)
out = model(x)
print(out['out'].shape)
# &gt; torch.Size([2, 21, 544, 728])
</code></pre>

MADHAVENDRA_SHARMA: <p>Hey, did you solved this problem? I also got the same issue.</p>
<p>I started with:</p>
<blockquote>
<p>def deeplabv3_mobilenet(output_classes):<br>
model = models.segmentation.deeplabv3_mobilenet_v3_large(weights=“DeepLabV3_MobileNet_V3_Large_Weights.COCO_WITH_VOC_LABELS_V1”)<br>
model.classifier = DeepLabHead(960, output_classes)<br>
return model</p>
</blockquote>
<blockquote>
<p>model = deeplabv3_mobilenet(9)<br>
img = d1[0][“image”].unsqueeze(dim =0)<br>
print(img.shape)<br>
<a href="//torch.Size" rel="noopener nofollow ugc">//torch.Size</a>([1, 3, 1022, 1820])</p>
</blockquote>
<p>Then, when I used</p>
<blockquote>
<p>model(d1[0][“image”].unsqueeze(dim = 0))</p>
</blockquote>
<p>I am also receiving the same error:<br>
RuntimeError: Given groups=1, weight of size [256, 2048, 1, 1], expected input[1, 960, 64, 114] to have 2048 channels, but got 960 channels instead</p>
<p>Thanks in advance. Sorry for any typo, this is my first post here.</p>

ptrblck: <p>I cannot reproduce the reported issue and see another error instead for an input of <code>[1, 3, 1022, 1820]</code>:</p>
<pre><code class="lang-python">output_classes = 10
model = models.segmentation.deeplabv3_mobilenet_v3_large()
model.classifier = models.segmentation.deeplabv3.DeepLabHead(960, output_classes)

x = torch.randn(1, 3, 1022, 1820)
out = model(x)
# ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])
</code></pre>
<p>which indicates a failure in a batchnorm layer as it cannot compute the stats from a single value.<br>
Increasing the batch size works for me:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 1022, 1820)
out = model(x)
print(out["out"].shape)
# torch.Size([2, 10, 1022, 1820])
</code></pre>
Can torch.load run lazy to save memory?

tarjintor: <p>Hello,I use a user defined trainer class to train a torch model with other regular method such as eval, predict etc.<br>
The trainer <strong>init</strong> method will check certain checkpoint path from args,if exists it will load the checkpoint,so object such as model,optimizer,lr_scheduler will all loaded.This is for failover so other framework like ray actor can reload the checkpoint to continue other operating such as fit.<br>
Then I will call the trainer method such as fit, eval, predict etc.<br>
But I only need optimizer, lr_scheduler when I call fit.If I call predict,I will only need model.<br>
So I want to use some lazy operation to wrap the checkpoint so that I will only load the object when I really need it,this will consume more io, but can save memory.<br>
But it seem the default torch load can’t do this,how can I do so?</p>
Static Quantizatied model

chowk1109: <p>Hello</p>
<p>i wanna make a quantization model, so i followed FX GRAPH MODE POST TRAINING STATIC QUANTIZATION tutorial on my resnet50 model.<br>
so  I checked my model weight capacity reduced.</p>
<p>here is my question.</p>
<ol>
<li>After quantizing the trained model, do I need to convert it into int8 and put it in when inference?</li>
<li>is my quantized model weight type int8?</li>
</ol>

111357: <p>Short answer:</p>
<ol>
<li>No</li>
<li>Yes</li>
</ol>
<p>After quantization, the weight of the quantized model has been converted int8 already.</p>
<p>You can check it by <code>quantized_model.layer_name.weight().dtype</code>.<br>
More details can be can found here <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html#debugging-quantized-model" rel="noopener nofollow ugc">Debugging Quantized Model</a>.</p>

chowk1109: <p>Thanks a lot.<br>
Really helpful!!</p>
<p>So, in Q1, don’t i have to change my input data type fp32 to int8?</p>

111357: <p>Yes, PyTorch will do it automatically in Fx mode.</p>

chowk1109: <p>Really helpful for my works.<br>
Thanks!!!</p>
Is num_workers>0 compatible on M1 MacBooks with latest macOS?

Sam_Lerman: <p>I get the following error on my M1 MacBook running macOS Ventura when I set <code>num_workers=1</code> (or any value &gt;0):</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/sam/IdeaProjects/Template/Run.py", line 48, in &lt;module&gt;
    for i, (x, y) in enumerate(train_loader):
  File "/Users/sam/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/Users/sam/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1316, in _next_data
    idx, data = self._get_data()
  File "/Users/sam/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1282, in _get_data
    success, data = self._try_get_data()
  File "/Users/sam/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1133, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 11208) exited unexpectedly
</code></pre>
Libtorch_cpu.so size

tepperson: <p>I just managed to compile libtorch under yocto. Is libtorch_cpu.so supposed to be huge? Mine is 2.2 gigabytes.</p>

ptrblck: <p>This seems to be unexpected as my local source build shows ~241MB and the nightly pip wheels ~490MB for <code>libtorch_cpu.so</code>.</p>

tepperson: <p>is that with debug symbols?</p>

ptrblck: <p>This could be the case, but you would need to specify some build arguments such as <code>DEBUG=1 python setup.py install</code>. By default debug symbols won’t be added.</p>
Advice for using DDP for dictionary input to models

jasperhyp: <p>Hi!  I’m trying to set up multi-GPU training for my sort of complicated model, but I’m not sure if I’m pursuing the correct direction.  I’ve been reading tutorials including <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">this</a> and <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">this</a>, but I just realized according to <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel" rel="noopener nofollow ugc">comments in the class</a>, DDP is splitting the first dimension of input, while I have a dictionary as input.  My model is like a multi-encoder-decoder framework over entities and relations, where we have a number of encoders for entities and a number of pair-wise decoders for relations, and the objective is link prediction.  Each batch of input would be a dictionary like</p>
<pre><code class="lang-python">{
  'tokens':{
    'entity_A': some_tensor,  # size is [num_A, seq_len_A]
    'entity_B': some_tensor,  # size is [num_B, seq_len_B]
    ...
  },
  'pairs':{
    'A_B': some_tensor,  # size is [num_AB, 3], 3 = head, relation, tail
    'A_C': some_tensor,  # size is [num_AC, 3], 3 = head, relation, tail
    ...
  }
}
</code></pre>
<p>I’m thinking of manually requiring DDP to only split the relations to each device, and all entities in the batch will be input to the <code>forward</code>. Then, in the model, we only keep the entities that those relations encompass (including remapping the indices).  I’m not sure if this is the best way though, and am not familiar with how to “require DDP to only split certain inputs to <code>forward</code>” either.</p>
<p>I am not using <code>pytorch-geometric</code>, but I am looking at their usage of DDP <a href="https://github.com/pyg-team/pytorch_geometric/blob/master/examples/multi_gpu/distributed_sampling.py" rel="noopener nofollow ugc">here</a>.  It looks like one can customize how data is fed into each machine in <code>DistributedSampler</code> (?) and DDP isn’t directly truncating the multiple inputs in the same way.  But I’m still a bit unsure how DDP would work in those cases.  Any pointers &amp; references will be greatly appreciated!</p>

wanchaol: <p>DDP would not split the input, it would receive a mini-batch input from the dataloader directly. But I think it would require the input be mostly tensor or tuple of tensors. One thing you can try is to extract the input tensor from the dict and feed the input tensor to a DDP-wrapped model.</p>

jasperhyp: <p>Oh yes that makes sense, so the customization needs to be made in the <code>DistributedSampler</code>?  Could you point me to some codebases where they made such edits?  Thank you!</p>
Unsure how to migrate my containerized torch.distributed.launch script to torchrun

fclingen-pet: <p>I run Pytorch from a container on an HPC system.  A simplified version of my usual launch script looks something like this:</p>
<pre><code class="lang-auto">RANK=1
for i in ${HOSTS// /}
do
        if [[ "$i" != "$HOST0" ]] ; then
                ssh  $i $CONATINER_EXEC $CONTAINER_EXEC_ARGS python -m torch.distributed.launch --nproc_per_node=$NGPU --nnodes=$NHOST --node_rank=$RANK --master_addr="$HOST0" --master_port=12581 train.py $CONFIG &amp;
                RANK=$((RANK+1))
        fi
done

$CONTAINER_EXEC $CONTAINER_EXEC_ARGS python -m torch.distributed.launch --nproc_per_node=$NGPU --nnodes=$NHOST --node_rank=0 --master_addr="$HOST0" --master_port=12581 train.py $CONFIG
</code></pre>
<p>This works great for me, because it means I don’t have to play games trying to keep a version of mpirun or gloo or whatever on the host system synchronized with those same tools in the container.</p>
<p>According to the distributed docs, <code>torch.distributed.launch</code> is about to be deprecated, in favor of <code>torchrun</code> which uses Elastic Launch instead of the old way of launching distributed torch.  This is not really compatible with my workflow, but there doesn’t seem to be a non-deprecated way to replicate my manner of launching with <code>torch.distributed.launch</code>?  Is my old workflow about to be deprecated?</p>

kumpera: <p>Maybe Tristan can help here? <a class="mention" href="/u/d4l3k">@d4l3k</a></p>
Hi folks, am trying to build pytorch for RISC-V , after 89% completion of build am facing this below error, pls help me

Achyuth: <p><strong>[ 89%] Building CXX object test_jit/CMakeFiles/test_jit.dir/__/common/main.cpp.o</strong><br>
<strong>[ 89%] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_add_if_then_else.cpp.o</strong><br>
<strong>In file included from /home/root/version-down/pytorch/torch/csrc/Exceptions.h:18,</strong><br>
**                 from /home/root/version-down/pytorch/torch/csrc/autograd/python_cpp_function.h:7,**<br>
**                 from /home/root/version-down/pytorch/torch/csrc/autograd/generated/python_functions_0.cpp:10:**<br>
<strong>/home/root/version-down/pytorch/torch/csrc/utils/pybind.h: In member function ‘bool pybind11::detail::type_caster<a>c10::DispatchKey</a>::load(pybind11::handle, bool)’:</strong><br>
<strong>/home/root/version-down/pytorch/torch/csrc/utils/pybind.h:174:29: error: ‘py::module_’ has not been declared</strong><br>
**  174 |                    src, py::module_::import(“builtins”).attr(“str”))) {**<br>
**      |                             ^~~~~~~**<br>
<strong>[ 89%] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_alias_analysis.cpp.o</strong><br>
<strong>make[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/build.make:82: caffe2/torch/CMakeFiles/torch_python.dir/csrc/autograd/generated/python_functions_0.cpp.o] Error 1</strong><br>
<strong>make[1]: *** [CMakeFiles/Makefile2:7624: caffe2/torch/CMakeFiles/torch_python.dir/all] Error 2</strong><br>
<strong>make[1]: *** Waiting for unfinished jobs…</strong><br>
<strong>[ 89%] Building CXX object test_tensorexpr/CMakeFiles/test_tensorexpr.dir/test_conv.cpp.o</strong></p>
FX.graph generate CUDA code for customized layers

DanielWang: <p>Is there any way to add a customized CUDA kernel generator based on the output from <code>Torch.FX.graph</code>.<br>
For instance, I define a simple CNN model with two layers, and my goal is to generate CUDA  code for a specialized convolutional kernel (let’s say <code>conv_new</code>) to replace the original convolution based on FX.graph IR.</p>
<p>Should I first build a customized operator in Pytorch as an extension and then replace the original model kernel with the new one? or is there any code gen for FX.graph to generate the kernel directory from templates?</p>

mszsorondo: <p>I don’t know if PyTorch specifies a standard interface to set certain torch.fx nodes to run as CUDA kernels, but you could try by binding a python function to the kernel, inserting the node and setting the function as its target.</p>
Node fusion via leaf nodes

mszsorondo: <p>Hello everyone!<br>
I’m perfoming node fusions and am thinking about using the following procedure:</p>
<ol>
<li>Take two subsequent nodes that perform an operation (say linear → relu or add → mul)</li>
<li>Implement a fused module (LinearWithRelu with appropriate forward or addAndMul respectively)</li>
<li>Set new fused module as leaf (or wrap function, in the case of addAndMul)</li>
<li>Replace old nodes by fused ones.</li>
<li>Recompile the GraphModule</li>
</ol>
<p>Could this new IR have an impact (positive or negative) on the way the module is evaluated? Will the fused nodes be compiled and/or scheduled differently? Or this only affect the symbolic tracing?</p>

mszsorondo: <p>Ok. Got no answer here but I already found one in the PyTorch Dev Forum, so I’m leaving it in case someones has a similar question:</p>
<h1>
<a name="the-nuances-of-pytorch-graph-capture-httpsdev-discusspytorchorgtthe-nuances-of-pytorch-graph-capture501-1" class="anchor" href="#the-nuances-of-pytorch-graph-capture-httpsdev-discusspytorchorgtthe-nuances-of-pytorch-graph-capture501-1"></a><a href="https://dev-discuss.pytorch.org/t/the-nuances-of-pytorch-graph-capture/501" rel="noopener nofollow ugc"> The nuances of PyTorch Graph Capture </a>
</h1>
<p><strong>TL,DR</strong><br>
Could this new IR have an impact (positive or negative) on the way the module is evaluated? Yes, and that is the main purpose of changing the IR.<br>
Will the fused nodes be compiled and/or scheduled differently? Or this only affect the symbolic tracing? Changing the graph will probably alter the scheduling of the node since “fused” nodes will be scheduled as a whole kernel, while the folding of such fusions will have separate schedules. Symbolic tracing is one procedure to capture the graph and thus changing the graph does not affect the symbolic tracing procedure per-se, but it does affect the result of the symbolic tracing.</p>
Function train transfer learning

brahimou: <p>hello everyone, i’m in  a big troble. does someone can explain to me this function line by line pleaaaase, it really blocks me i still a beginner</p>
<p>def train_model(model, criterion, optimizer, scheduler, num_epochs=25):<br>
since = time.time()</p>
<pre><code>best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0

for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode

        running_loss = 0.0
        running_corrects = 0

        # Iterate over data.
        for inputs, labels in dataloaders[phase]:
            inputs = inputs.to(device)
            labels = labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)
        if phase == 'train':
            scheduler.step()

        epoch_loss = running_loss / dataset_sizes[phase]
        epoch_acc = running_corrects.double() / dataset_sizes[phase]

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))

        # deep copy the model
        if phase == 'val' and epoch_acc &gt; best_acc:
            best_acc = epoch_acc
            best_model_wts = copy.deepcopy(model.state_dict())

    print()

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

# load best model weights
model.load_state_dict(best_model_wts)
return model
</code></pre>

ptrblck: <p>It’s unclear how much details you would need to understand each line of code. Instead of directly describing the code, could you let us know which lines are unclear?<br>
I would expect that e.g.:</p>
<pre><code class="lang-python">running_loss = 0.0 # initializes the running_loss variable with 0.0
</code></pre>
<p>would be clear, but:</p>
<pre><code class="lang-python">with torch.set_grad_enabled(phase == 'train'): # enabled gradient computation if phase=='train' is used
</code></pre>
<p>might be more complicated to understand.</p>

brahimou: <p>i wan to know waht this function do and what’s the output</p>
Unpack tuple of tensors

Aditya_Raj_P15VSS201: <p>I am working on pytorch code of time series forecasting using TFT method. Here</p>
<ol>
<li>First I have taken a csv file to read my data that has 4 columns with numeric values. (stock prices)</li>
<li>I converted the data into TimeSeriesDataSet() format.</li>
<li>Then converted the datasets to dataloaders for training val, and testing using to_dataloader() method.</li>
<li>I am trying to extract the feature (x) and label (y) using below sample code
<h1>
<a name="for-x-y-in-iterval_dataloader-1" class="anchor" href="#for-x-y-in-iterval_dataloader-1"></a>for x, y in iter(val_dataloader)</h1>
</li>
</ol>
<p>Here y represent the target column for which I want my model to make predictions. I have options to make declare single and multiple target columns like  this<br>
a:  target_colum = “colum1”<br>
b:  target_colum = [‘colum1’, colum2’]     etc.</p>
<p>in case of “a” where I have only one column to predict I can extract the actual labels<br>
like this<br>
<span class="hashtag">#actuals</span> = torch.cat([y[0] for x, y in iter(test_dataloader)])</p>
<p>However in case of “b” where I have multiple columns I am not able to extract the actual labels</p>
<p>when I explored I found that “y” is a tuple and in my case there are two target columns then in the “y” data is stored like this</p>
<p>horizon = n<br>
print(y) = ( [  tensor( [ [1], [2],…[n no. of predictions for column 1] ]  ) , tensor( [ [1], [2],…[n no. of predictions for column 2] ]  )  ] , none)</p>
<p>and if there are multiple test data like if n = 30 and I have 120 rows in the test data then this list continues with 4 such data.</p>
<p>my problem is I want to extract this data from “y”. Kindly help I am trapped in these square and small brackets. My intention is to store all the values in a numpy array with size (n, 2)  .</p>

ptrblck: <p>Something like this should work:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(100, 1)
        self.target1 = torch.arange(100)
        self.target2 = torch.arange(100, 200)
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, index):
        x = self.data[index]
        y1 = self.target1[index]
        y2 = self.target2[index]
        return x, (y1, y2)
    

dataset = MyDataset()
loader = DataLoader(dataset, batch_size=10)

targets = [torch.stack((y1, y2), dim=1) for _, (y1, y2) in loader]
targets = torch.cat(targets, dim=0)
print(targets.shape)
# torch.Size([100, 2])
</code></pre>
Best way to split long text into logical pieces

Robert_Lukoshko: <p>Does anyone know what is the best splitting text solution if I want to perform a search on the long RAW text?<br>
I want to split the text into logical pieces \ paragraphs and search among them. I wonder if someone knows a lib \ hugging face model to do it.<br>
I struggle to find a working solution.<br>
Thanks!</p>

vdw: <p>I guess it depends what you mean by logical pieces and how your dataset looks like.</p>
<p>For example, a paragraph is “a short part of a text, consisting of at least one sentence and beginning on a new line. It usually deals with a single event, description, idea, etc.” So if that’s what you’re after, you could just split our string by <code>\n</code>.</p>
<p>But I vaguely assume you mean something else :).</p>
Do we support inference on video stream?

Hui_Liu: <p>Hi,</p>
<p>I only see examples at <a href="https://pytorch.org/mobile/home/#deployment-workflow" class="inline-onebox" rel="noopener nofollow ugc">Home | PyTorch</a> on static images.</p>
<p>Do we have plan to support inference on video stream ( and automatically drop frame before inference finishes etc)?</p>
<p>I do see this: <a href="https://github.com/pytorch/vision/issues/2920" class="inline-onebox" rel="noopener nofollow ugc">[Feat] Camera Stream API proposal · Issue #2920 · pytorch/vision · GitHub</a> but just to reconfirm <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thanks!</p>

marksaroufim: <p>Sure <a href="https://github.com/pytorch/ios-demo-app/tree/master/TorchVideo" class="inline-onebox" rel="noopener nofollow ugc">ios-demo-app/TorchVideo at master · pytorch/ios-demo-app · GitHub</a></p>

Hui_Liu: <p>Thanks, Mark! Will see if I could accomodate that to do object detection directly on camera stream</p>