akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Regarding the default weight decay of AdamW

Lin_Jia: <p>Hi Folks:</p>
<p>I was playing with AdamW. The default value for the weight decay (L2 regulation) is 0.01. I believe it is a wrong value. I tried to use the default value, and fail to train the model.</p>
<p>Some interesting points after carefully reading the paper of AdamW: <a href="https://arxiv.org/pdf/1711.05101v2.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1711.05101v2.pdf</a></p>
<p>For a computation vision example, the recommended value is 0.00025: Figure 2</p>
<p>It suggests that for different computation tasks, the best weight decays are different. Therefore they proposed the concept of normalized weight decay, which you can use to compute the actual weight decay in formula 6. The normalized weight decay is much bigger than the weight decay.</p>
<p>I believe the 0.01 used in pytorch implementation of AdamW comes from the normalized weight decay. I suspect pytorch just uses that value directly as weight decay.</p>
<p>I hope this is helpful for other folks.</p>
Two models with .eval() and .train() separately but always incompatible

Raymond10153: <p>Hi,</p>
<p>I have two models with .eval() (model_A) and .train() (model_B) separately. I am inputting the output of model_A, a trained model, to model_B which I am training, as followed.</p>
<p>model_A.eval()<br>
for epoch in range(epoch_num):<br>
…<br>
model_B.train()<br>
output_model_A = model_A(input)<br>
output_mobel_B = model_B(output_model_A)<br>
…<br>
batch_loss.backward()<br>
optimizer.step()</p>
<p>It popped up an error “RuntimeError: cudnn RNN backward can only be called in training mode”. I have set the model_B to train mode. I don’t know why I still get this kind of error.</p>
<p>Does anyone have a solution? Many thanks!!!</p>

J_Johnson: <p>Wrap model_A with <code>torch.no_grad()</code> or use <code>.detach()</code> on the outputs(preferably the former for avoiding memory issues).</p>
<pre><code class="lang-auto">with torch.no_grad():
    data=model_A(input)

#rest of training not wrapped
</code></pre>
Torch.view is painful to use. Is there a function for spliting or merging dimensions?

ZdsAlpha: <p>Most of the time when I am trying to define a custom operation, mostly I am not concerned with rest of dimensions i.e defining multi-head attention for convs. I am not concerned with batch, width, height or even what conv operation is 1d, 2d or 3d. I am just concerned with feature dimension (dim=1) and I am not concerned with the rest of dimensions. This is where view and permute become really painful to use. Lets just limit this thread to torch.view.</p>
<p>Now I just want to split feature dimension into two without converting it to list and having to concat it back again? Why is there no function to do so? or at least I couldn’t find one. Lets suppose I have tensor like (B, C, H, W) and I want to split channel into into N chunks.</p>
<p>There should be a function like torch.split(x, dim=1, (C // N, N)) which should result in (B, C//N, N, H, W) shape. Using torch.view makes this so painful especially when you are writing a generalized operation for all convolutions. I have to split shape and do multiple operations just to achieve this result. And on top of that it ruins pytorch tracing when you involve shapes. There should also be torch.merge function that should merge dimensions (reverse of split). torch.merge(x, (1, 2)) should revert (B, C//N, N, H, W) to (B, C, H, W) it can save so much time.</p>
<p>Same issue with permute. If I want to swap two dimensions I have to specify all dimensions. I am not concerned with W,H or L at the end. It become really hard to write generalized operations due to hard-coded approach we currently have for view and permute.</p>

J_Johnson: <p>In your specific example given, there is a chunk method:</p>
<pre><code class="lang-auto">x=torch.rand(16, 3, 32, 32)

x1, x2, x3 = x.chunk(3, dim=1)
</code></pre>
<p><code>torch.cat</code> can reverse the op.</p>
<pre><code class="lang-auto">y = torch.cat([x1, x2, x3], dim=1)
</code></pre>
<p>You can also simply specify the ranges, like in numpy:</p>
<pre><code class="lang-auto">x1, x2, x3 = x[:,0:1,...], x[:, 1:2, ...], x[:, 2:3,...]
</code></pre>
<p>Additionally, <code>torch.einsum</code> is a good kit to use for miscellaneous tensor operations.</p>

ZdsAlpha: <p>That sounds very very inefficient. The computation graph is going to be mess and this operation alone would be huge bottleneck in training if we have to split tensor into many chunks i.e 64, 128.</p>
<p>Its also going to rely on python list and GIL so might not work with pytorch tracing. This might be way more inefficient than view method.</p>

J_Johnson: <p>Pytorch is an open source community and developed by community members. If you have an idea that you believe would help improve the ecosystem, you can submit the code for inclusion:</p>
<p><a href="https://pytorch.org/docs/stable/community/contribution_guide.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/community/contribution_guide.html</a></p>
Small NNs to reuse the pytorch impl to do my own Object detector?

Mah_Neh: <p>I have read that YOLO, Retina with Resnet, SSD, etc can do Object Detection.</p>
<p>Now I struggle Implementing these from scratch, so I wonder if it is possible to re train some implementation of these from Pytorch, but with a new dataset not included in the pre-trained classes.</p>
<p>I am looking for some NNs that would run in not so powerful devices, so light ones.</p>
<p>Im kinda directionless atm.</p>

ptrblck: <p>Yes, fine tuning a model with a new dataset is possible.<br>
<code>torchvision</code> provides implementations for <a href="https://pytorch.org/vision/main/models/retinanet.html#retinanet"><code>RetinaNet</code></a> and <a href="https://pytorch.org/vision/main/models/ssd.html#ssd"><code>SSD</code></a>.<br>
The <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">Object detection finetuning tutorial</a> could be a good starter.</p>

Mah_Neh: <p>But I am trying to find a small model like <strong>tiny yolo</strong> or any <strong>ssd mobile net</strong> that does object detection, is this feasible as well? I dont really see anything similar in the docs, after days reading the all other websites as well. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Maybe <a href="https://pytorch.org/vision/stable/models/ssdlite.html#ssdlite"><code>SSDLite</code></a> would work for you. I don’t think Yolo is available in <code>torchvision</code>.</p>

Mah_Neh: <p>Do you now the reason or if it will be ? Seems like an interestin architecture and quite successful</p>

ptrblck: <p>I believe it was planned and discussed <a href="https://github.com/pytorch/vision/issues/6341">here</a> but it seems the author is busy with other work:</p>
<blockquote>
<p>The only bad news for us is that I transferd to a new work group and my new leader only allow me to share only a half day per week to complete this job. So it might takes me more time than expected.</p>
</blockquote>
What's the point of datapipe.fork if it doesn't copy the underlying data?

Gavin_Sellers: <p>I’m trying to fork the WikiText2 datapipe, so that I can do a different set of transforms to two instances of the data, but if I use datapipe.fork described here: <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Forker.html#torchdata.datapipes.iter.Forker" class="inline-onebox" rel="noopener nofollow ugc">Forker — TorchData main documentation</a>, the underlying data is identical (confirmed using data_batch is data2_batch). Am I just missing the use case for .fork? And is there a better way to split datapipes?</p>

nivek: <p>I don’t 100% understand your question. Can you elaborate or provide a code snippet of what isn’t working?</p>
<p>In your case, you can iterate through <code>data_batch</code> and <code>data2_batch</code> separately and perform different sets of transforms. Then, you can later combine the outputs together again if you’d like.</p>

Gavin_Sellers: <p>Yeah sure. So if I do something like:</p>
<pre><code class="lang-auto">dp1, dp2 = source_dp.fork(num_instances=2)
dp1 = dp1.map(dp1_transform)
dp2 = dp1.zip(dp2)
</code></pre>
<p>Then I get a series of tuples with the same underlying data even though dp1 had a different transform applied than dp2, because the underlying data returned by fork is the same. In my case, I’m trying to split the WikiText2 data into two identical copies and then apply random masking to one set and rejoin the two before passing into a dataloader. With the fork operation, I can’t seem to apply the masking operation to one fork without affecting the other.</p>

nivek: <p>What are your objects inside? This would work as expected.</p>
<pre><code class="lang-python">from torchdata.datapipes.iter import IterableWrapper

dp = IterableWrapper(list(range(10)))
dp1, dp2 = dp.fork(num_instances=2)
dp1 = dp1.map(lambda x: x + 100)
dp3 = dp1.zip(dp2)

print(list(dp3))  # [(100, 0), (101, 1), (102, 2), (103, 3), (104, 4), (105, 5), (106, 6), (107, 7), (108, 8), (109, 9)]
</code></pre>
<p>I assume you have a mutable object? If so, you can make a copy of it within <code>dp1_transform</code> to avoid affecting the copy in <code>dp2</code>.</p>

nivek: <p>I can see how this is confusing, feel free to open an issue on GitHub and suggest adding an argument to deep copy outputs coming out of <code>fork</code>. We can discuss further there and see if there is enough interest for a PR/change. I think there is certainly some uses case for that. Thanks for bringing this up.</p>

Gavin_Sellers: <p>Thanks. I’ll open an issue when I get the chance.</p>
Dataloader stuck issue

Jaideep_Valani: <p>/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py in <strong>next</strong>(self)<br>
519             if self._sampler_iter is None:<br>
520                 self._reset()<br>
 → 521             data = self._next_data()<br>
522             self._num_yielded += 1<br>
523             if self._dataset_kind == _DatasetKind.Iterable and \</p>
<p>/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_data(self)<br>
1184<br>
1185             assert not self._shutdown and self._tasks_outstanding &gt; 0<br>
 → 1186             idx, data = self._get_data()<br>
1187             self._tasks_outstanding -= 1<br>
1188             if self._dataset_kind == _DatasetKind.Iterable:</p>
<p>/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _get_data(self)<br>
1150         else:<br>
1151             while True:<br>
 → 1152                 success, data = self._try_get_data()<br>
1153                 if success:<br>
1154                     return data</p>
<p>/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)<br>
988         #   (bool: whether successfully get data, any: data if successful else None)<br>
989         try:<br>
 → 990             data = self._data_queue.get(timeout=timeout)<br>
991             return (True, data)<br>
992         except Exception as e:</p>
<p>/opt/conda/lib/python3.8/multiprocessing/queues.py in get(self, block, timeout)<br>
105                 if block:<br>
106                     timeout = deadline - time.monotonic()<br>
 → 107                     if not self._poll(timeout):<br>
108                         raise Empty<br>
109                 elif not self._poll():</p>
<p>/opt/conda/lib/python3.8/multiprocessing/connection.py in poll(self, timeout)<br>
255         self._check_closed()<br>
256         self._check_readable()<br>
 → 257         return self._poll(timeout)<br>
258<br>
259     def <strong>enter</strong>(self):</p>
<p>/opt/conda/lib/python3.8/multiprocessing/connection.py in _poll(self, timeout)<br>
422<br>
423     def _poll(self, timeout):<br>
 → 424         r = wait([self], timeout)<br>
425         return bool(r)</p>
During training evaluation in first epoch I am getting image loading fault

abdulkadirkaraca: <p>During training evaluation in first epoch I am getting image loading fault and It is collapsing the training stage. Please where should I look?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/adf5b73666e5a07d3a646f3f5f96488eff5ca5f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/adf5b73666e5a07d3a646f3f5f96488eff5ca5f5" title="Ekran 2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/adf5b73666e5a07d3a646f3f5f96488eff5ca5f5.png" alt="Ekran 2" data-base62-sha1="oOV2Q1h8l5OpTQBhEPfeja91wih" width="690" height="465" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Ekran 2</span><span class="informations">1262×852 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help me. Thanks.</p>

ptrblck: <p>Remove the paths to invalid images from your <code>Dataset</code> and make sure each sample can be properly loaded to avoid these issues.</p>
Error in repeated use of autograd.grad on inputs

LukeTri: <p>I am training a model that returns a gradient of the final fully connected layer with respect to the inputs for each minibatch (i.e. first element of returned value is gradient of first value of self.fc3(out).squeeze() with respect to the x[0]):</p>
<pre><code>def forward(self, x):
    out = self.conv1(x)
    out = self.relu1(out)
    out = self.conv2(out)
    out = self.relu2(out).view(x.shape[0], -1)
    out = self.fc1(out)
    out = self.relu3(out)
    out = self.fc3(out).squeeze()
    out = torch.autograd.grad(out, x, grad_outputs=torch.ones_like(out), create_graph=True)[0]
    return out
</code></pre>
<p>Here, the input is of size (10, 1, 10000) and the output is of size (10, 1, 10000). This part seems to work correctly, but in the main loop I need to compute the gradient again in a similar way:</p>
<pre><code>    samples = samples.reshape((samples.shape[0], 1, samples.shape[1] * samples.shape[2])).float().requires_grad_(True)
    outputs = model(samples)
    out1 = (outputs @ v).squeeze()
    out1 = torch.autograd.grad(out1, samples, grad_outputs=torch.ones_like(out1), create_graph=True)[0].float()
</code></pre>
<p>(outputs @ v).squeeze() results in a 1d tensor with size 10. Since, I set create_graph to true in the forward method, I would expect that I can call autograd.grad on out1 with respect to samples and get a non-zero value. However, the result is that out1 is a (10, 1, 10000) of all 0s, rather than one that is filled out with the correct gradients.</p>
Different results on different training using single 4090 gpu

sealynndev: <p>I am training Informer on a private dataset based on the pipeline <a href="https://github.com/MAZiqing/FEDformer" rel="noopener nofollow ugc">MAZiqing/FEDformer (github.com)</a>. I have fixed the random seed and the num_workers is set to 1. The problem is, I get same results on different training using single 3090 or 3090ti gpu, even on different devices(Ubuntu on clound server or Ubuntu(WSL) on local PC). But I get different results on different training using single 4090 gpu.<br>
We conduct the training both on Ubuntu clound server and local PC(Ubuntu, WSL). All the code in training is same.<br>
Is there any solutions to fixed this problem?</p>

ptrblck: <p>Did you set all needed variables as explained in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>? If so, could you post a minimal and executable code snippet which would show this behavior, please?</p>
Cannot load torchvision model in local machine

Peterwisu: <p>Hi, I am trying to use transfer learning, but the problem is I can’t load the model on my local machine. It takes forever to load a model.</p>
<p>It keeps showing <code>Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /home/peter/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth</code></p>
<p>This is my code</p>
<pre><code class="lang-auto">import torchvision 
from torchvision.models import resnet50, ResNet50_Weights

model = torchvision.models.resnet50(weights=ResNet50_Weights.IMAGENET1K_V1
</code></pre>
<p>However, I tried it on google Colab my code seems to work fine. I also tried several models in Torchvision, but I can’t load them. Thank you.</p>

ConvolutionalAtom: <p>I can download torchvision models without a problem. The file is about 90 mb, so it should take a few minutes (worst case) with decent internet connection.<br>
Is there any network setting that would prevent this, e.g. VPN?</p>
<p>Maybe, you can also run the following to see the problem is about your network setting</p>
<pre><code class="lang-auto">wget https://download.pytorch.org/models/resnet50-0676ba61.pth
</code></pre>
<p>This will download the model to the current working directory</p>
How to construct a tensor with parameters without breaking graph

Ben_K: <p>I have discovered that I am severing the computational graph when constructing a parameterized tensor:</p>
<pre><code class="lang-auto">class Model(nn.Module):
 def __init__(self) -&gt; None:
        super().__init__()
        self.A_params = nn.Parameter(torch.tensor([0.3, 0.1]), requires_grad=True)
        self.update_ss()
def update_ss(self):
        k_a, k_b = torch.sigmoid(self.A_params)  # constrain to [0, 1]
        self.A = torch.tensor([[-k_a, 0], [k_a, -k_b]]) # does not preserve gradient??
</code></pre>
<p>How do I construct my tensor (A) while preserving the gradient from A_params to A</p>

Ben_K: <pre><code class="lang-auto">A= torch.zeros((2,2))
A[0,0] = -k_a
A[1,0] = k_a
A[1,1] = -k_b
</code></pre>
<p>This works, but I feel like there is a better way…</p>

srishti-git1110: <p>Please see if <a href="https://pytorch.org/docs/stable/generated/torch.stack.html" rel="noopener nofollow ugc">torch.stack()</a> will help you?<br>
See-</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([3.0, 4], requires_grad=True)
x = (a+2)*3*torch.exp(a)
y = torch.tensor([5.0, 6], requires_grad=True)

z = torch.stack((x, y)) # graph not disturbed
z.sum().backward()
print(a.grad) # tensor([ 361.5397, 1146.5612])
</code></pre>
<p>Feel free to post if you face errors with this.</p>
Use Network developed for Ubuntu on Windows

poseestimationuser: <p>Hello there,</p>
<p>I want to train the PoseCNN network with my own synthetic images and depth data and I am using Windows 10 on my computer. I saw on the <a href="https://github.com/NVlabs/PoseCNN-PyTorch" rel="noopener nofollow ugc">GitHub</a> Readme of PoseCNN that the required environment is Ubuntu. So do I need to use a virtual machine with Ubuntu or should I use something like WSL? I am afraid that if I use a virtual machine the performance will be way worse.</p>
<p>I am very new to these topics so please don’t be mad if this question is incredibly stupid <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thank you so much</p>

ptrblck: <p>Your question isn’t stupid and nobody will get mad at you for asking as we are here to help <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>I would probably use WSL2 and install the PyTorch binaries there to get the model running.</p>

tiramisuNcustard: <p>You can get a Ubuntu image with cuda installed on Docker Hub and run the container on your Windows host machine. Remember to install the Nvidia Docker tool.</p>
Is the Torch 2.0 compile feature adapted to quantization?

EdouardVilain-Git: <p>Hi everyone,<br>
I’ve been running tests with the compile feature of Torch 2.0 and I’m observing impressive gains on inference speed for Transformer. Nonetheless, I’ve tried quantizing compiled models without success (there’s no gain observed between the normal and quantized model).<br>
Is that expected? If not, is there future work aiming to tackle this issue?<br>
Thank you very much!</p>

supriyar: <p>hi,<br>
Are you trying to compare gains between FP32 model and INT8 model, both with torch.compile enabled?<br>
What are the results of quantizing the model without torch.compile?<br>
Which backend are you running the model on (server CPU or something else)?</p>
<p>We haven’t added support to accelerate quantized models with torch.compile yet. Typically the compute intensive quantized ops have device specific implementation (fbgemm for server cpu, qnnpack/xnnpack for edge) which might not benefit from this. For others, we might get some speedup with torch.compile but this hasn’t been explored yet.</p>

EdouardVilain-Git: <p>Hi supriyar,<br>
Thanks for getting back.<br>
Yes that’s it, I’m trying to compare between FP32 and INT8 models both with torch.compile. I was hoping to observe a similar gain than what we get for FP32 and INT8 models without using torch.compile.<br>
I’m running tests on a XLM-RoBERTa transformer on server CPU and observe a 45% gain in inference between unquatized and quantized models.<br>
When using torch.compile, inference speed stays the same but I guess this is expected given your reply. Nonetheless, it would be interesting for torch.compile to support quantized models to obtain even better results!<br>
Thanks again</p>

Jiong_Gong: <p>We (Intel) are working on the design (collaborating with Meta folks) and PoC (<a href="https://github.com/pytorch/pytorch/pull/91226" class="inline-onebox" rel="noopener nofollow ugc">[Experiment][Inductor][Quant] inductor as a quantization backend for PT 2.0 quantization by Xia-Weiwen · Pull Request #91226 · pytorch/pytorch · GitHub</a>) to enable x86 CPU quantization optimization in PT2.0 TorchInductor. It is still in an early stage though. Welcome to any feedbacks.</p>

EdouardVilain-Git: <p>Hey Jiong, thanks a lot for letting me know, I’m looking forward to seeing the results.</p>
Train on numpy arrays

HelixPiano: <p>Hello everyone,<br>
I wanna say that I am a total beginner to the entire topic of NN.<br>
I am trying to train a CNN on classifying numpy arrays into one of two categories.</p>
<p>So the first question is if a CNN is even the right approach or if I should use something else.</p>
<p>I have one 3D numpy array (third component is time) and one that contains the labels (either True or False).</p>
<p>What I do with them is currently:</p>
<pre><code class="lang-auto">  tensor_training_data = torch.Tensor(training_data)
  tensor_training_label = torch.Tensor(training_label)
  training_dataset = TensorDataset(tensor_training_data, tensor_training_label)
</code></pre>
<p>Second question: Is this the right way to handle the data?</p>
<p>Third question: Is there some tutorial for handling data that are not images? All I saw always utilizes images and therefore 3 channels. My understanding of this topic is at the moment not good enough to figure out how to proceed.</p>
<p>To clarify: I don’t want anyone to do the work for me, I just need someone pointing me in the right direction.</p>
Multiprocessing loading a lot of files is stucked in the traning loop

xwjBupt: <p>Hi, the thing is that i use multiprocessing to load my training samples into the RAM in the init function of my dataset, when i test my dataset, everything works just fine, but when i use the dataset in my training loop, it stucks in the preloading files into RAM, it do not report errors, just stuck. The code is shown as following:</p>
<pre><code class="lang-auto">from torch.utils import data
from multiprocessing import Pool
import tqdm

class MYDATA(data.Dataset):
    def __init__(
        self,
        datadir
        **kwargs
    ):
    self.samples = len(datadir)
    self.content = preload(self.samples)
   def __len__(self):
         return len(self.samples)
   def __getitem__(self,index):
        return self.content[index]

def preload(sampledirs):
   with Pool(4) as p:
                  tmp_content = list(
                      tqdm.tqdm(
                          p.imap(loading_function, sampledirs),
                          total=len(sampledirs),
                          desc="Pre loading views to ram with %d thread &gt;&gt;&gt; " % 4,
                          ncols=100,
                      )
                  )
    return tmp_content 
</code></pre>
<p>what is wrong? thank you so much!!</p>

nkdatascientist: <p>Hi <a class="mention" href="/u/xwjbupt">@xwjBupt</a>,<br>
I think you need to use list manager from multiprocessing as show below, Can you test this code to see if it works for you?</p>
<pre data-code-wrap="py"><code class="lang-plaintext">from torch.utils import data
from multiprocessing import Pool, Manager
import tqdm

class MYDATA(data.Dataset):
    def __init__(
        self,
        datadir,
        **kwargs
    ):
        self.samples = len(datadir)
        with Manager() as manager:
            self.content = manager.list()
            self.content += preload(datadir)
    
    def __len__(self):
        return self.samples
    
    def __getitem__(self, index):
        return self.content[index]

def preload(sampledirs):
    with Pool(4) as p:
        tmp_content = list(
            tqdm.tqdm(
                p.imap(loading_function, sampledirs),
                total=len(sampledirs),
                desc="Preloading views to RAM with %d threads &gt;&gt;&gt; " % 4,
                ncols=100,
            )
        )
    return tmp_content
</code></pre>

xwjBupt: <p>Thank you so much for replying, but it does not help and is still the same</p>
Memory leak during inference?

tshmak: <p>My model trains without memory issues, but when being deployed, it has a memory leak which I cannot track down. I’m not sure whether it’s related to pytorch. But to help me narrow down the options, I would just like to know, for example, in a loop such as the following, is there potential for memory leak in pytorch? If so, can you suggest some possibilities?</p>
<pre><code class="lang-auto">while(True): 
    model = SomePytorchModel()
    model.eval()
    for i in range(100): 
        with torch.no_grad(): 
            result = model()
</code></pre>

ptrblck: <p>Your code is a bit odd as it’s recreating the model in a loop, but this should not leak memory as the garbage collector should be cleaning the old variables after they were replaced by the new objects.</p>
Inner loops on GPUs vs CPUs

Ascrypto: <p>In the training step and part of the model’s architecture, the code consists of an inner for loop that iterates over steps in a single sequence, one step at a time.</p>
<p>By profiling this part on both CPUs and GPUS, I noticed a big difference in the time it takes to iterate over the sequence, on CPU: 0.2-0.4s; on GPU: every single iteration takes around 0.65s.  This is slowing down the overall execution of the code on the Slurm (I am using DistributedParallelData).</p>
<p>As far as I know, data transfer from GPU to CPU mid-training is not worth the struggle.  I do not know exactly what should be done in this case.  I tried both nccl and gloo backends, but both are slow.</p>

Ascrypto: <p>I made a naive mistake where inside the for loop, I used to(device) to transfer each data point to the GPU, separately, instead of at once, which I believe contributed to the overhead and time.</p>
Some questions about custom autograd function with parameter list

ha2ka: <p>Hi,</p>
<p>I am currently having some troubles with <code>torch.nn.ParameterList</code>.<br>
The following code is not working as I expcted.</p>
<p>There are 2 questions below:</p>
<ol>
<li>I assume that if some argument of custom autograd functions is a list of tensors, the system does not check <code>requires_grad</code> of the tensors in it, is that correct?</li>
<li>When some argument of custom function is a list of tensors, how do I return the gradients for it?</li>
</ol>
<pre><code class="lang-python">import torch

# Normal matrix multiplication works fine for inputs
# with `requires_grad = False`
weight = torch.nn.Parameter(torch.zeros(4, 4))
x = torch.randn(10, 4)
y = x @ weight
y.sum().backward()
print(weight.grad.max())


class MultiLinearFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, weights):
        ctx.save_for_backward(input, *weights)

        y = []
        for weight in weights:
            y.append(input @ weight)
        return torch.stack(y)

    @staticmethod
    def backward(ctx, grad):
        input, *weights = ctx.saved_tensors

        grads_weight = [input.t() @ grad for _ in range(len(weights))]
        grads_input = []
        for weight in weights:
            grads_input.append(grad @ weight.t())

        return torch.stack(grads_input).mean(dim=0), None, None

        # This unpack statement does not work
        # return torch.stack(grads_input).mean(dim=0), *grads_weight


weights = torch.nn.ParameterList([torch.zeros(4, 4) for _ in range(2)])
x = torch.randn(10, 4)
x.requires_grad = True  # This is needed for triggering to call backward

y = MultiLinearFunction.apply(x, weights)
y.sum().backward()
</code></pre>
<p>environments:</p>
<ul>
<li>Python 3.9.7</li>
<li>PyTorch 1.13.1</li>
</ul>