akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LayerNorm's mean and std div are not fixed while inferencing

hibama: <p>I’m currently working on recreating the input after LayerNorm. As far as I know, the mean and standard deviation for LayerNorm are fixed during the inference phase. Therefore, I thought I could extract these factors and recreate the original input from the LayerNorm output.</p>
<p>I have successfully extracted the weight and bias, which are not necessarily identical to the mean and standard deviation because LayerNorm has its own weight and bias parameters. My weight and bias parameters are fused from various factors, but they successfully recreate the original input from the LayerNorm output.</p>
<p>However, when I applied these extracted weight and bias parameters to another input tensor and expected LayerNorm to work in the same way as with the previous input, I obtained a completely different output. I assumed that LayerNorm calculated new mean and standard deviation values for the second input, causing the difference. But I’m puzzled as to why LayerNorm computed the mean and standard deviation for the second input; they should have remained fixed during inference.</p>
<pre><code class="lang-auto">import torch
from torch import nn

input_data = torch.randn(1, 577, 768)
input_remade = torch.randn(1, 577, 768)

class layer(nn.Module):
    def __init__(self):
        super().__init__()
        self.norm1 = nn.LayerNorm(768, eps=1e-6)
        #self.norm1 = nn.LayerNorm(768, eps=0)
    def forward(self, x):
        x = self.norm1(x)
        return x

layer = layer().eval()
with torch.inference_mode():
    out = layer(input_data)

w = torch.zeros(len(out[0, :, 0]))
b = torch.zeros(len(out[0, :, 0]))

for i in range(len(out[0, :, 0])):
    w[i] = (input_data[0, i, 0] - input_data[0, i, 10]) / (out[0, i, 0] - out[0, i, 10])
    b[i] = (input_data[0, i, 0] * out[0, i, 10] - input_data[0, i, 10] * out[0, i, 0]) / (out[0, i, 10] - out[0, i, 0])

for i1 in range(len(input_remade[0, :, 0])):
    input_remade[0, i1, :] = out[0, i1, :] * w[i1] + b[i1]
print(torch.sum(input_remade - input_data))


input_data2 = torch.randn(1, 577, 768)
input_remade2 = torch.randn(1, 577, 768)
with torch.inference_mode():
    out2 = layer(input_data2)

for i1 in range(len(input_remade2[0, :, 0])):
    input_remade2[0, i1, :] = out2[0, i1, :] * w[i1] + b[i1]
print(torch.sum(input_remade2 - input_data2))

w1 = torch.zeros(len(out2[0, :, 0]))
b1 = torch.zeros(len(out2[0, :, 0]))

for i in range(len(out2[0, :, 0])):
    w1[i] = (input_data2[0, i, 0] - input_data2[0, i, 10]) / (out2[0, i, 0] - out2[0, i, 10])
    b1[i] = (input_data2[0, i, 0] * out2[0, i, 10] - input_data2[0, i, 10] * out2[0, i, 0]) / (out2[0, i, 10] - out2[0, i, 0])

for i1 in range(len(input_remade2[0, :, 0])):
    input_remade2[0, i1, :] = out2[0, i1, :] * w1[i1] + b1[i1]
print(torch.sum(input_remade2 - input_data2))
</code></pre>
<pre><code class="lang-auto">tensor(-0.0061)
tensor(1280.9966)
tensor(0.0014)
</code></pre>
Using MViT as an encoder

mohammad995: <p>Hello all,</p>
<p>I am trying to use video.mvit_v2_s as an encoder to get the encodings of frames. more specifically, I want to omit the “head” from the last block of this model. Is there any easy way to do it?<br>
I am trying to create a class and inherit from MViT class. but get errors. can anyone help me with this?</p>
<p>Thanks in advance</p>

J_Johnson: <p>Probably the least amount of code would be defining a forward module and replacing the head with that:</p>
<pre><code class="lang-auto">class Fwd(nn.Module): 
    def forward(self, x):
        x
</code></pre>
<p>Then redefine the head:</p>
<pre><code class="lang-auto">model.head = Fwd()
</code></pre>

mohammad995: <p>Thank you for your answer. Is this the same as Identity function? meaning that I need to replace the previous classification head with an Identity func?</p>

J_Johnson: <p>No. It’s not. The identity matrix takes up ram and has some calculation overhead.</p>
<p>The above Fwd function does basically nothing. No memory increase during run time and no compute.</p>

minhnh: <p>Yes, just use nn.Identity()</p>
How link prediction model output multiple different predictions?

qwsdeef: <p>I have a graph dataset with 500 small graphs(each graph have about 5 nodes). A graph neural network is trained by the dataset for link prediction task. However, for an input such as (3 nodes), it can only get a fixed prediction result. How can I get multiple different predictions? Some details you can see in Fig below. Thanks for your reply！<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/671d71f0e867a8711e4ecac9f7e7999e8bf60ba5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/671d71f0e867a8711e4ecac9f7e7999e8bf60ba5" title="MMMM1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/671d71f0e867a8711e4ecac9f7e7999e8bf60ba5_2_690x462.jpeg" alt="MMMM1" data-base62-sha1="eIcgrDrnExlF8Y2xmF4pcG0UiMZ" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/671d71f0e867a8711e4ecac9f7e7999e8bf60ba5_2_690x462.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/7/671d71f0e867a8711e4ecac9f7e7999e8bf60ba5.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/7/671d71f0e867a8711e4ecac9f7e7999e8bf60ba5.jpeg 2x" data-dominant-color="F3F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MMMM1</span><span class="informations">980×657 80.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is there a common way of finding feasible word compositions?

hiru: <p>I am working on vision but trying multi modality rn. So I need to figure out if a given word composition exists within everyday context (eg. ‘ripe cat’ would never occur but a ‘wet cat’ would). I was thinking of finding bigram and trigram possibilities within some big nlp context text but have no idea where to begin. Any topic related to this or paper suggestions highly appriciated.</p>

vdw: <p>It kind of sounds like you’re looking for n-gram language models – e.g.: for bigrams: What is the probability that the word “ripe” is followed by the word cat. And yes, it basically requires to go through a large corpus an count n-grams. I have to basic Jupyter notebooks <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/1-foundations/Section%203.1%20-%20n-Gram%20Language%20Models%20(basic).ipynb" rel="noopener nofollow ugc">here</a> and <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/1-foundations/Section%203.2%20-%20n-Gram%20Language%20Models%20(advanced).ipynb" rel="noopener nofollow ugc">here</a>.</p>
<p>Keep in mind that most language models perform smoothing to avoid zero-probabilities. So you need to find some notion which probability is low enough to be considered “does not exist in everyday context”. For example, maybe a non-native speaker has translated “fertile cat” with “ripe cat” :).</p>

hiru: <p>Thank you! This is what I was playing with last few days but couldn’t find any large corpus that would fit everything I wanted with good generalizability. That’s what I am struggling rn.</p>

vdw: <p>Welcome to the tedious 80% of machine learning / deep learning: collecting, cleaning, processing, preparing etc. good data :).</p>
Non-blocking device to host transfer

sinafz: <p>On cuda-semantics page, there is information on non-blocking host to device data transfer. Is it possible to do non-blocking host to device data transfer?</p>

ptrblck: <p>You can use non-blocking data transfers using the <code>non_blocking=True</code> argument in e.g. <code>tensor = tensor.to()</code>. <a href="https://devblogs.nvidia.com/how-overlap-data-transfers-cuda-cc/" rel="nofollow noopener">This NVIDIA blog post</a> gives you some information what’s going on under the hood.</p>
<p>In case I misunderstood your question, could you clarify it a bit?</p>

sinafz: <p>Sorry <a class="mention" href="/u/ptrblck">@ptrblck</a> for not following up on this. It dropped down in my backlog for a while. I think I’ve finally figured out how to do GPU-&gt;CPU asynchronously. Calling <code>tensor.to()</code> doesn’t seem to allow you to specify the output buffer or request output to be placed in pin-memory. So using <code>tensor.to()</code> for GPU-&gt;CPU transfer is always synchronous. However I think I was able to get async transfer by creating a pin-memory buffer in cpu and using <code>pinned_cpu_tensor.copy_(gpu_tensor, non_blocking=True)</code>. Can you confirm this is the correct way to achieve asynchronous GPU-&gt;CPU data transfer?</p>

Gun_Park: <p>I’m concerned about this as well. Can anyone confirm?</p>

GuillaumeTong: <p>Here is a related discussion FYR:</p><aside class="quote" data-post="6" data-topic="173855">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aifartist/48/57106_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-maximize-cpu-gpu-memory-transfer-speeds/173855/6">How to maximize CPU &lt;==&gt; GPU memory transfer speeds?</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/ptrblck">@ptrblck</a> 
If I “strace -f” my GPU app I find not a single mlockxxx call nor a mmap with MAP_LOCKED call. 
I presume there isn’t a major bug in pytorch where they are missing doing the memory pin such that crashes might occur under stress if the system starts swapping pages.  I only know of the two ways mentioned to pin memory pages.
  </blockquote>
</aside>

<p>Here is the TLDR:</p>
<ol>
<li><code>gpu_tensor.to(device="cpu", non_blocking=True)</code> will asynchronously copy a tensor from GPU to CPU pinned memory</li>
<li>As you point out, <code>pinned_cpu_tensor.copy_(gpu_tensor, non_blocking=True)</code> will asynchronously copy a tensor from GPU to an <em>existing</em> CPU pinned memory tensor</li>
<li>Once the asynchronous copy has been launched, you need to use <code>torch.cuda.Stream.synchronize()</code> or <code>torch.cuda.synchronize()</code> to ensure the copy has finished before using its content</li>
</ol>
Forward hook - any way to differentiate by model's input?

amperie: <p>Hi there,</p>
<p>I’m using forward hooks to look at the intermediate activations of a net as it goes through, works great.</p>
<p>I’d like to be able to correlate/separate these activations by the input, so I can see just the activations for a particulr input, even if multiple inputs are being processed at the same time or while the hooks are in place. In essence I’d like to be able to separate the data that the hooks give me into something like:</p>
<p>For input <span class="hashtag-raw">#1</span> → activations <span class="hashtag-raw">#1</span><br>
input <span class="hashtag-raw">#2</span> → activations <span class="hashtag-raw">#2</span></p>
<p>As I understand it now, the hooks give me the activations but won’t tell me for which input they are.</p>
<p>Make sense? Thanks in advance</p>

ptrblck: <p>The activations will have a batch dimension containing containing the activation for each corresponding input. I.e. the activation at batch index <code>i</code> corresponds to input at index <code>i</code>.</p>

amperie: <p>Thanks! I’ll check that out for batches.</p>
<p>How about for separate batches? ie: if I run one batch through, then another right after - is there a way to differentiate them in that case?</p>

ptrblck: <p>The forward hooks will be triggered during the forward pass so you should be able to process the intermediate activations between separate forward passes.</p>

amperie: <p>thanks, much appreciated!</p>
WSL Version Confusion

Brad_Bellomo: <p>I am unsuccessfully trying to install pytorch with GPU on WSL.  Plenty of good tutorials and instructions to install specific versions of CUDA and Tensorflow, but these are always out of date once someone release a new version.  I can’t find any good documentation on what versions of Tensorflow are supported or required for pytorch.</p>
<p>My install currently has ‘torch.<strong>version</strong>’ returning ‘2.1.2+cu121’.  I assume this means it is using pytorch 2.1.2.  But I don’t know if this means it thinks it is using Cuda 12.1, or it requires 12.1 or something else.</p>
<p>Do I need to downgrade to CUDA 12.1 to use pytorch 2.1.2?  If so, which version of Tensorflow should I be on?</p>

ptrblck: <p>PyTorch binaries ship with their own CUDA dependencies, in your case using CUDA 12.1U1. You would only need to install an NVIDIA driver allowing PyTorch to communicate with your GPU. I don’t know what TensorFlow requires.</p>

Brad_Bellomo: <p>Thanks!  But please explain further.  I thought pytorch was just a library that called TensorFlow, which had a dependency on CUDA, which calls NVidia’s driver.</p>
<p>Are you saying I should be able to use pytorch with GPU acceleration WITHOUT either installing TensorFlow or CUDA seperately?</p>

ptrblck: <p>PyTorch does not depend on TensorFlow and does not call it.<br>
Yes, you can install the PyTorch binaries, an NVIDIA driver, and can directly run PyTorch scripts on the GPU.</p>

Brad_Bellomo: <p>Thanks!  Wow, I never realized this and haven’t really used it much until now.  I thought it was just a different syntax flavor of Keras.</p>
<p>Much much easier to install, although so far, harder to know if it installed right.</p>
Accessing intermediate objects in `forward()`

sheim: <p>We’re playing around with some custom neural nets that include operations in the <code>forward</code> such as building a matrix and multiplying the input, e.g.</p>
<pre><code class="lang-auto">def forward(self, x):
        output = self.connection_1(x)
        output = self.activation_1(output)
        output = self.connection_2(output)
        output = self.activation_2(output)
        output = self.connection_3(output)
        output = self.activation_3(output)
        M = self.create_cholesky(output)
        # * create symmetric matrix A out of predicted
        # * Cholesky decomposition
        A = M.bmm(M.transpose(1, 2))
        return A
</code></pre>
<p>So far, the module outputs the matrix, and then we use this outside to compute the actual value of interest, <code>y = x.T A x</code>.<br>
It seems sensible to me that this final step should also be in the <code>forward()</code>, so that the final output is already <code>y</code>, which would also make comparing it to vanilla MLP much more straightforward.<br>
The catch is that we also want access to the matrix <code>A</code> (among other things to do custom losses), and it’s not clear to me how best to approach this.</p>
<ol>
<li>We could simply assign <code>self.A = A</code> so that it’s accessible, but I wonder if that might invite problems with autograd.</li>
<li>We could have a separate method to output only <code>A</code> and wrap that in <code>torch.no_grad()</code>, but that seems wasteful.</li>
<li>We could have <code>forward()</code> return both <code>y</code> and <code>A</code>, but that changes the how we interface with a model, which is a bit gross.</li>
<li>We could use a forward hook. We haven’t used this before, and it seems like this might be meant only for debugging (?), but it seems like the correct choice?</li>
</ol>
PyTorch Gradient Computation without calling loss.backward() function

rajoy99: <p>The problem I am solving does not have an explicit loss term. I need to calculate the gradient and update the weights manually. That means I cannot use loss.backward() function; since the loss term does not even exist.</p>
<p>How can I calculate gradient w.r.t the Neural Network parameters ? The Neural Network is of type torch.nn.Module.</p>
<p>I am using this approach to calculate the gradient manually.</p>
<p>gradient = torch.autograd.grad(term,NN.parameters()) ; here term can be a scaler or vector</p>
<p>However, the shape of the gradient that I receive is unexpected.</p>
Slicing with an arbitary sized tuple

bloodymeli: <p>Can I use a tuple with an unspecified length for slicing (or equivalently, a matrix with an unspecified number of columns)?<br>
For example, I want to slice x using <code>ind</code> and <code>ind2</code>.  <code>ind2</code> is a 1D tensor. If <code>ind</code> is a tuple containing a single 1D vector, than I want to get <code>x[ind[i],ind2[i]]</code> . If<code> ind</code> contains two 1D tensors, I want to obtain <code>x[ind[0,i], x[ind[1,i], ind2[i]]</code> , etc.</p>
Normalzing images

satinder147: <p>I am trying to train a UNet for road segmentation. When I normalize the images using transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))  the loss decreases but if I normalize the images by dividing them by 255.0 then the loss stops decreasing after a certain point. What is the difference between the two?</p>

ptrblck: <p><code>transforms.Normalize</code> subtracts the provided mean and divides by the stddev to create standardized tensors (<a href="https://en.wikipedia.org/wiki/Standard_score">Wikipedia - Standard Score</a>), while the latter approach normalizes the values to the range <code>[0, 1]</code>.</p>

satinder147: <p>but why the first one works efficiently, but the loss for the latter stops decreasing after some time ?</p>

ptrblck: <p>It’s not generally true, I think, and might depend on your application.<br>
You can find literature about how standardizing whitens the input data and this creates a “round” loss surface, but I’m not sure if these simple abstractions are applicable for deep neural networks.<br>
Have a look at <a href="https://sebastianraschka.com/Articles/2014_about_feature_scaling.html#about-standardization">this post</a> written by <a class="mention" href="/u/rasbt">@rasbt</a> for some more information.</p>

rasbt: <p>In practice, I often find that dividing by 255 works similarly well as z-score standardization (the Normalize approach with means and standard deviation) for images. However, many gradient-based optimization algorithms benefit from the symmetry of having the data centered at 0 with positive and negative values</p>

Foram: <p>Hello, I am working with BreakHIs dataset ( Breast Histopathology images), and on computing the mean and standard deviation, I get the following values mean=(0.7941, 0.6329, 0.7694), std=(0.1024, 0.1363, 0.0908). However, on using values for normalization, I noticed that the min and max values for my normalized image are around -8 and +3 respectively. Are these values correct? The following is the histogram for one of the original images, it can be seen that the histogram is skewed towards the brighter end.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36684eb97e41bcb19c343aaa917908faa37c2dbe.png" alt="image" data-base62-sha1="7LjgcaAmp9R9pWVkkxFVaNwYUoe" width="483" height="376"></p>
Why are 1x1 conv used in SE layers instead of linear layers

Foram: <p>I am implementing SE-ResNet for a binary classification problem.  I noticed that, in the description of the SE layers, linear layers were used to compute the attention map. However, in the code, 1x1 conv layers are used instead.</p>
<p>Is there a specific reason for this? Are the 1x1 conv more stable than linear layers? Or it that both can be used interchangeably, and it does not matter, which one of them is used?</p>
<p>Thank you!</p>

InnovArul: <p>It will be great if you could link the code as well in the question.<br>
To your question, the short answer would be, in the context of SE-layer, both 1x1 conv and linear layer will do the same job.</p>
<p>Consider that the global average pooling of 2D features gives the output of dimension <code>B x P x 1 x 1</code>. B = batch, P = num channels. In the SE-layer, the next step is to convert the <code>P</code> dimensional vector to <code>Q</code> dimensional vector. (P, Q can be any number)</p>
<p>This can be achieved either by,</p>
<ul>
<li>using a linear layer<br>
<code>nn.Linear(in_features=P, out_features=Q)</code></li>
</ul>
<p>(or)</p>
<ul>
<li>by using a 1x1 conv layer<br>
<code>nn.Conv2d(in_channels=P, out_channels=Q, kernel_size=1)</code></li>
</ul>
<p>In the case of the linear layer, the global average pooling output (<code>B x P x 1 x 1</code>) needs to be squeezed to have the correct dimension (2D = <code>B x P</code>) to pass through the linear layer, then again convert back to 4D tensor. Instead, the 1x1 conv layer works with a 4D tensor without intermediate squeeze operation.</p>

J_Johnson: <p>A Conv2d 1x1 kernel is identical mathematically to a Linear operation. Please see here for a code demonstration:</p>
<aside class="quote quote-modified" data-post="3" data-topic="191179">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/help-me-rewrite-this-from-tensorflow-dense-layers/191179/3">Help me rewrite this from tensorflow (Dense layers)</a> 
  </div>
  <blockquote>
    Welcome to the forums! 
There is a lot to unpack here. 

When you are calling nn.AdaptiveAvgPool2d(1), you are telling PyTorch that whatever size image comes into that, make it 1x1. So the output of that layer would then be (batch_size, channels, 1, 1). Let’s assume you want that. Then you should call a flatten() afterward.

_x = self.global_avg_pool(inputs).flatten(1) 


When you get down to your convolutional layer, you will need to add those two dims back.

_x = self.conv(_x.unsqueeze(2).unsq…
  </blockquote>
</aside>


Foram: <p>Hi, I apologize for my delayed response. Thank you for your replies <a class="mention" href="/u/innovarul">@InnovArul</a> and <a class="mention" href="/u/j_johnson">@J_Johnson</a>, this gave me a better insight into the questions I asked.</p>
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1695, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

Julie_Martin: <p>When i try to implement ML_decoder from <a href="https://github.com/Alibaba-MIIL/ML_Decoder" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Alibaba-MIIL/ML_Decoder: Official PyTorch implementation of "ML-Decoder: Scalable and Versatile Classification Head" (2021)</a> i have this error than seems to come from pytorch : File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1695, in <strong>getattr</strong><br>
raise AttributeError(f"‘{type(self).<strong>name</strong>}’ object has no attribute ‘{name}’"). Whant can i do ?</p>

ptrblck: <p>Could you check which line of code triggers this <code>AttributeError</code> as it seems you are trying to access an attribute which is undefined.</p>
Problem fixing seeds in pytorch 2.1

0sum: <p>Hi, I’m doing some experiments with pytorch and have a problem of reproducing the same value for each different runs.</p>
<p>I created conda env by “conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia”, and tried to fix the seeds by following 5 lines of code.</p>
<p>torch.manual_seed(seed)<br>
np.random.seed(seed)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False<br>
random.seed(seed)</p>
<p>When I print the feature values or loss, the values are the same at first several iterations, then there occurs small differences (1e-4) which affects the final prediction values.</p>
<p>One strange thing is that when the same experiments (exactly the same codes) are tested in pytorch 1.12.0, I could successfully fix the seeds.</p>
<p>Has anyone got a problem like me? Or would there be additional nondeterministic operations (that I should fix) in pytorch 2.1.0 compared to pytorch 1.12?</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue, please?</p>
Different interpolation types in transforms v2

agaldran: <p>Hello!</p>
<p>I was wondering if when using transforms v2 in torchvision we are allowed to specify different interpolation modes for the list of intputs. Like, if I have an RGB and a binary image, I may want to interpolate the former bilinearly and the latter with nearest neighbors, but I am getting an error when trying to dot his:</p>
<pre><code class="lang-auto">import torchvision.transforms.v2 as tr
from torchvision.transforms.functional import InterpolationMode as mode

# load an image and a mask into im, mask and try this:
x, y = tr.RandomRotation(45, interpolation = (mode.BILINEAR,  mode.NEAREST))(im, mask) 
# throws error
</code></pre>
<p>Any clue on how to do this? Thanks!</p>
Parallel grad on different cuda stream

jeffry1829: <p>When I do this:</p>
<pre><code class="lang-auto">    streams = [torch.cuda.Stream() for i in range(elemsize)]
    for ii in range(elemsize):
        streams[ii].wait_stream(torch.cuda.current_stream())
    for ii in range(elemsize):
        loc = [0 for jj in range(len(shp))]
        n = ii
        for jj in range(len(shp)):
            loc[jj] = n//accu[jj]
            n = n%accu[jj]
        with torch.cuda.stream(streams[ii]):
            print(ii)
            NormMat[(...,)+tuple(loc)] = torch.autograd.grad(Norm[(0,0)][tuple(loc)], B_grad, create_graph=False, retain_graph=True)[0]
            print("end", ii)
</code></pre>
<p>I want to parallelize the slice and copy operation on gpu.<br>
However, this code will run sequentially.<br>
I’m not sure how to solve this, seems like there’re some sync inside autograd…</p>
<p>Thanks!</p>

jeffry1829: <p>Actually, seems like it works…</p>
Question regarding Quantization Aware Training(QAT)

BRH: <p>Apologies,  I am new to quantization.<br>
The current QAT tutorial on the official website runs the training entirely on CPU.<br>
I realized the QAT model can be trained on GPU before calling torch.ao.quantization.convert on the model.</p>
<p>Will training the QAT model in GPU mode affect anything?</p>
How can I get fixed mean and standard-deviation from layernorm?

hibama: <p>Hi, I’m currently working on extracting the mean and standard deviation values from LayerNorm.<br>
I understand that LayerNorm operates as a fixed-weight layer during model inference, so I assume I can extract the fixed mean and standard deviation values.<br>
However, I have only found additional weight and bias objects. Is there any way to obtain these mean and standard deviation values?<br>
Thanks as always.</p>
Why what use Transformer in cv project?

yinjun1131: <p>I am currently studying NLP and have learned about the Transformer section. After a brief study, I believe that this is because the NLP field has a large amount of input text content and cannot fix the range of numbers/letters. Therefore, I use Transformer to extract the input text content first, while the CV field is extremely constant. Regardless of the final result, each pixel is fixed at 1-255 * 3, or 1-255. In this case, Why is it said that Transformer can improve the effectiveness of CV projects</p>
Tutorials for neural models related to the brain?

Mah_Neh: <p>Here just asking for some advice. I’ll describe the situation below:</p>
<p><strong>TL;DR</strong></p>
<p>I work with neural networks mostly variations of CNNs (object detection and classification, and this type of tasks) and know linear algebra quite well. This is just to say: I would like to code some stuff (for example, neural nets that would explain conscious processes) and stop reading purely verbal theories of consciousness.</p>
<p><strong>A bit of context</strong></p>
<p>Probably some here are quite into the details, and maybe some have advice for someone trying to get started with the implementation of some models. What I have in mind is that there may be many theories for many processes: how the visual detection works, how each modality processes data and produces and output, how different areas communicate, and also how the whole thing happens (like IIT.) I assume there will be different levels of complexity, and I wonder what could be a good start given my background (maybe some specific architectures that are related to some specific conscious experience like visual experience?)</p>
<p>I wonder if some of you could give me some hints as to which types of architectures or models, or courses could I start, to get some hands on rather than keep reading so much stuff. The people I’m familiar with are Chalmer, Dennett, Blackmore and this type of philosophical/conceptual books but no hands on experience since this books do not implement their ideas.</p>
<p>I know about some theories like IIT, but I doubt it’s a good starting point.</p>
<p>Thanks !</p>
ParameterListImpl

amayr: <p>I wonder, whether cloning a ParameterList can work.</p>
<p>In cloneable.h there is the sequence:</p>
<blockquote>
<pre><code>copy-&gt;parameters_.clear();
copy-&gt;buffers_.clear();
copy-&gt;children_.clear();
copy-&gt;reset();
TORCH_CHECK(
    copy-&gt;parameters_.size() == parameters_.size(),
    "The cloned module does not have the same number of "
    "parameters as the original module after calling reset(). "
    "Are you sure you called register_parameter() inside reset() "
    "and not the constructor?");
</code></pre>
</blockquote>
<p>reset in parameterlist.h shows</p>
<blockquote>
<p>/// <code>reset()</code> is empty for <code>ParameterList</code>, since it does not have parameters<br>
/// of its own.<br>
void reset() override {}</p>
</blockquote>
<p>To my understanding all the parameters, children, etc. are cleared first. Reset does nothing. Then the assertion fails. Or do I get this wrong?</p>
Setting seeds for multi-GPU training

seer_mer: <p>When setting a seed for multi-GPU training (DDP), should I set the same seed (e.g. seed=42) for all ranks? or should I set different seeds for different ranks? Or it doesn’t matter?</p>
<p>I saw some people setting <code>seed = args.seed + rank</code> and also some people setting <code>seed = args.seed</code>. Would either of these cause any problem with the dataset? e.g., will the dataset get shuffled and distributed to different processes correctly with both methods? Will different processes accidentally train on the same portion of the dataset due to using the same/different seed?</p>
<p>Thanks in advance.</p>

ptrblck: <p>In a <code>DDP</code> setup the <code>DistributedSampler</code> is used to split the dataset and the local seed won’t break it.</p>
AttributeError: 'Linear' object has no attribute '_lazy_load_hook'

Sarthak_Jain: <p>Hello all,<br>
I am facing a very weird issue when loading a saved model using pickle.load. I had trained a pyG GCN model and saved it like this:</p>
<pre><code class="lang-auto">import pickle
path = '/content/drive/MyDrive/'
file = open(path + 'bamodel', 'wb')
pickle.dump(model, file)
file.close()
</code></pre>
<p>I am loading it like this:</p>
<pre><code class="lang-auto">import pickle

# open a file, where you stored the pickled data
path = '/content/drive/MyDrive/'
file = open(path+ 'bamodel', 'rb')

model = pickle.load(file)
</code></pre>
<p>But, when I try to load the model, I receive the error:</p>
<p>AttributeError: ‘Linear’ object has no attribute ‘_lazy_load_hook’<br>
I have started facing the issue today, while I was not facing the issue earlier</p>
<p>Neither did I change the ‘bamodel’ file, nor anything else, so I am pretty surprised at why is this error coming up. I searched the internet but no on else seems to have faced this issue before. Any help is greatly appreciated!</p>

JuyiLin: <p>I also meet this problem, I regenerate the pickled file and use the same path. It works</p>

Sarthak_Jain: <p>Thanks for your reply! The issue is that, in my case I only want to use the saved model(model training is stochastic, so it is difficult to regenerate the same model state) in the pickle file. So, if I regenerate the pickle file, then I will lose the model that I had saved. Do you know any way by which I can use the same pickle file?</p>

JuyiLin: <p>It might be difficult if you have different path with the user who train the model. If you train it, you could maintain it. Save the whole model is tricky. You could save the static dict.</p>
<p>Refer : <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p>This save/load process uses the most intuitive syntax and involves the least amount of code. Saving a model in this way will save the entire module using Python’s <a href="https://docs.python.org/3/library/pickle.html" rel="noopener nofollow ugc">pickle</a> module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.</p>
GPU RAM out of memory

Devis: <p>Hey guys, i’m trying to finetune a sharded llama 2 model for a college project, but i keep running out of GPU RAM instantly</p>
<p>This code is based on a template i found online…</p>
<p>I tried setting max_split_size_mb based on the error message but that doesnt seem to be helping either</p>
<pre><code class="lang-auto">from transformers import TrainingArguments

output_dir = "./results"
per_device_train_batch_size = 4
gradient_accumulation_steps = 4
optim = "paged_adamw_32bit"
save_steps = 100
logging_steps = 10
learning_rate = 2e-4
max_grad_norm = 0.3
max_steps = 100
warmup_ratio = 0.03
lr_scheduler_type = "constant"

training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    fp16=True,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    group_by_length=True,
    lr_scheduler_type=lr_scheduler_type,
)

from trl import SFTTrainer

max_seq_length = 512

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    tokenizer=tokenizer,
    args=training_arguments,
)
</code></pre>
<p>Then I’m just simply calling <code>trainer.train()</code></p>
<p>I’m using Google Colab free with T4 GPU, i don’t want to upgrade to the pro version considering I’m a cash-strapped college student lmao and i lack the experience to know how much that would even help in the first place</p>
<p>Any suggestions on how to properly manage resources are appreciated</p>
<p>Here’s the error message for reference-</p>
<p>OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB. GPU 0 has a total capacty of 14.75 GiB of which 51.06 MiB is free. Process 11288 has 14.70 GiB memory in use. Of the allocated memory 14.27 GiB is allocated by PyTorch, and 304.69 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>

ptrblck: <p>Do you know how much memory the model would need during training? 15GB might not be enough.</p>

Devis: <p>I don’t really know… it was spiking up to 14.7gb almost instantly…</p>