akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Stateful LSTM problem with last batch

sava_delchev: <p>Hello everyone,</p>
<p>I am trying to create a LSTM model to forecast financial time series and the data is split into train,validation and test. The model is trained on the train, fine-tuned on the validation and then 14 observations are left in the test set, to check it on unobserved data.</p>
<p>I have configured my lstm network to be stateful, i.e. i have an initial hidden state, which is updated in the batch and passed to the next batch. Of course, this makes sense for time-series data since it is sequential. I would like to keep those “warm” hidden states to be passed to the validation data as well and eventually to the test as well.</p>
<p>However, i run into the following problem:<br>
The training/validation data is split into equal batches, but of course the last batch in each set has a different size. So, this means that for the last batch in the training i need a different shaped hidden and cell states. Then those need to be passed to the validation as well, with the original shape, since the first batches of the validation are the same as the train batches.</p>
<p>At the moment, i just drop the last batches from the train and validation data, but is there a more sustainable solution?</p>
<p>Thank you in advance!</p>
Image classifier for dataset of images with some attributes

2U3_1967: <p>The CSV file for the dataset is like as below.<br>
“index, path to image, value, class”</p>
<p>For example, image is that of insect and value is elevation where a picture of the insect has been taken.</p>
<p>Could anyone tell me a network (or model) that classifies the images in the dataset?</p>
Expected input batch_size (220) to match target batch_size (63)

Hana_AlRasheed: <p>Hello, I am encountering the following error while fine-tuning a large language model. The code initially ran without issues, but when I executed it again two hours later, I received this error: Expected input batch_size (220) to match target batch_size (63).</p>

ptrblck: <p>This error is often caused by <code>view</code> or <code>reshape</code> operations manipulating the batch size while in fact other dimensions should be changed.<br>
E.g. are you using <code>x = x.view(-1, ...)</code> somewhere in your model (where <code>x</code> could be an activation or target)? If so, replace it with <code>x = x.view(x.size(0), -1, ...)</code> to keep the batch size.</p>

Hana_AlRasheed: <p>I can’t find where this applies in my code. Could you help me locate it?</p>
Additional layer in the conv weight after quantization

HIMANSHU_LAL: <p>key_encoder = KeyEncoder()<br>
key_checkpoint = torch.load(‘saves/key.pth’, map_location=‘cpu’)<br>
key_encoder.qconfig = key_checkpoint[‘qconfig’]<br>
key_encoder = prepare(key_encoder)<br>
key_encoder = convert(key_encoder, inplace=True)<br>
key_encoder.load_state_dict(key_checkpoint[‘model_state_dict’])</p>
<p>I first quantized the model and then saved it and after again loading it is showing the below error.<br>
RuntimeError: [QConv2D Given groups=1, weight of the size 64, 7, 7, 4, expected input (NCHW) 1, 3, 480, 864 to have 4 channels, but got 3 channels</p>

Jingyun_Liu: <p>Have you solved the problem? I encountered the same issue.</p>
Seeking insights: Strange torch.compile() behavior with RegionViT in inference_mode()

hassonofer: <p>I’ve encountered an interesting issue while using <code>torch.compile()</code> for inference with a RegionViT model and would appreciate help debugging it.</p>
<p>I’ve filed a detailed bug report here: <a href="https://github.com/pytorch/pytorch/issues/146780" class="inline-onebox" rel="noopener nofollow ugc">TypeError when using torch.compile with RegionViT under torch.inference_mode() · Issue #146780 · pytorch/pytorch · GitHub</a></p>
<p>My main goal is to understand which part of the code is causing the issue and how to fix it in my implementation.</p>
<h4><a name="p-465605-reproduction-steps-1" class="anchor" href="#p-465605-reproduction-steps-1"></a>Reproduction Steps</h4>
<p>Install required packages:</p>
<p>pip install birder torch torchvision torchaudio</p>
<p>Run the following minimal example:</p>
<pre><code class="lang-auto">from birder.model_registry import registry
import torch

net = registry.net_factory("regionvit_t", input_channels=3, num_classes=1000)
net.to(torch.device("cuda"))
net.eval()
net = torch.compile(net)
with torch.inference_mode():
    net(torch.rand(1, 3, 256, 256, device=torch.device("cuda")))
</code></pre>
<h4><a name="p-465605-debugging-attempts-2" class="anchor" href="#p-465605-debugging-attempts-2"></a>Debugging Attempts</h4>
<p>I’ve added print statements to locate the failure point and observed some unexpected behavior. In the main forward pass (reference: [<a href="https://gitlab.com/birder/birder/-/blob/c1f5f9adb6b552af7f1cd2b9603e8a5abf8d3340/birder/net/regionvit.py#L388" class="inline-onebox" rel="noopener nofollow ugc">birder/net/regionvit.py · c1f5f9adb6b552af7f1cd2b9603e8a5abf8d3340 · birder / birder · GitLab</a>)):</p>
<pre><code class="lang-auto">def forward(self, cls_tokens: torch.Tensor, patch_tokens: torch.Tensor) -&gt; tuple[torch.Tensor, torch.Tensor]:
    (cls_tokens, patch_tokens) = self.proj(cls_tokens, patch_tokens)
    (out, mask, p_r, p_b, B, C, H, W) = convert_to_flatten_layout(cls_tokens, patch_tokens, self.ws[0])
    for blk in self.blocks:
        # print("A")
        out = blk((out, B), mask)
        # print("B")

    # print("C")
    (cls_tokens, patch_tokens) = convert_to_spatial_layout(out, B, C, H, W, self.ws, mask, p_r, p_b)

    return (cls_tokens, patch_tokens)
</code></pre>
<p>Interestingly:</p>
<ul>
<li>Adding print statements at points A or B makes the code succeed</li>
<li>Adding print statements at point C still results in failure</li>
<li>Any print statement inside the <code>blk</code> forward pass also leads to success</li>
</ul>
<p>Does anyone have any ideas on how to zero in on the issue? Any insights or suggestions would be greatly appreciated.</p>
Loading and retraining Peft model error ( grad and does not have a grad_fn)

Sourabh_Yadav: <p>pretty much the title. I am low on resources so I am training Peft adapters then saving them and loading them to retrain them on next data. but when I load adapters after training I get error also when I check the trainable parameters it prints the number of adapter correctly.</p>
<p><strong>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</strong></p>
<p>device = “cuda” if torch.cuda.is_available() else “cpu”<br>
model_id = “Qwen/Qwen2.5-VL-3B-Instruct”</p>
<p>base_model = AutoModelForImageTextToText.from_pretrained(<br>
model_id,<br>
torch_dtype=torch.float16,<br>
load_in_8bit=True,  # Load in 8-bit precision<br>
device_map=“auto”<br>
)</p>
<p>min_pixels = 224<em>28</em>28<br>
max_pixels = 224<em>28</em>28<br>
processor = AutoProcessor.from_pretrained(model_id, min_pixels=min_pixels, max_pixels=max_pixels)<br>
processor.tokenizer.padding_side = “right”</p>
<p>model = PeftModel.from_pretrained(base_model, “./docmat/iter-1”, is_trainable=True)</p>
<p>optimizer = AdamW(model.parameters(), lr=2e-6)</p>
<p>training_args = TrainingArguments(<br>
output_dir=“./results”,<br>
learning_rate=2e-6,<br>
per_device_train_batch_size=6,<br>
per_device_eval_batch_size=4,<br>
num_train_epochs=1,<br>
weight_decay=0.01,<br>
logging_dir=“./logs”,<br>
logging_steps=50,<br>
save_steps=1500,<br>
gradient_checkpointing=True,<br>
gradient_accumulation_steps=6,<br>
max_grad_norm= 1,<br>
remove_unused_columns= False,<br>
no_cuda=False<br>
)</p>
<p>def batch_collate_fn(examples):<br>
texts = [processor.apply_chat_template(example, tokenize=False) for example in examples]<br>
image_inputs = [example[1][“content”][0][“image”] for example in examples]<br>
inputs = processor(<br>
text=texts,<br>
images=image_inputs,<br>
return_tensors=“pt”,<br>
padding=True<br>
)<br>
inputs[“labels”] = inputs[“input_ids”].clone()<br>
return inputs</p>
<p>trainer = Trainer(<br>
model=model,<br>
args=training_args,<br>
train_dataset=train_dataset,<br>
eval_dataset=eval_dataset,<br>
optimizers=(optimizer, None),<br>
data_collator=batch_collate_fn<br>
)</p>
<p>print(“Starting training…”)<br>
trainer.train()<br>
print(“-”*30)</p>
<p>print(“Final evaluation…”)<br>
metric = trainer.evaluate()<br>
print(f"Final Evaluation metrics: {metric}")</p>
Pytorch CUDA for Jetson orin nano

PhantatP: <p>I want to use my Jetson orin nano to train YOLO but it can’t use GPU for training.<br>
My Jetson is installed with Jetpack 6.2 and CUDA 12.6. Is there pytorch version that compatible with this setup?</p>

ptrblck: <p>You can find the wheels for JetPack6 <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>
Pytorch install is that large? 5 Gbs?

JonathanAlis: <p>I just installed torch via pip in a virtualenv.<br>
(a simple pip install torch, using python 3.10, in a linux with cuda 12.6)<br>
I checked the total venv disk size before and after the pip install torch.<br>
Before it was ocupying around 600 Mb<br>
After, it ocupies new 5.7Gb</p>
<p>So, all installation files get a total of 5Gb!<br>
Probably there is plenty of cuda stuff too, but I already have all drivers before. It is really needed to download all that stuff?<br>
There are other ways to get a smaller install while keep cuda support?</p>

ptrblck: <p>No, the PyTorch binaries with CUDA support should use around 2.3GB and will install the CUDA runtime dependencies from PyPI, which other packages can reuse.</p>

N_N_miko: <p>Maybe you’ve installed other lib as well, torch along without torchtext is 2.3GB I think, like what <a class="mention" href="/u/ptrblck">@ptrblck</a> said, I think for torchtext along it takes 1.1GB? Just leave enough space for your env, if you are new in ML or DL field, try to split one env as ‘one for all’ environment and depend on case to use which env. If you are working on linux, try docker, much easier for building a env and deploy.</p>

dariox: <p><a class="mention" href="/u/jonathanalis">@JonathanAlis</a> your appreciation is correct; I’ve tried to install pytorch while having at least 5 Gb free, and it stopped throwing “not enough space”. After the error, some of the space is restored; however a “pip cache purge” is needed to recover all the used space after a failed installation.<br>
While the sum of “declared” download size (as is informed by pip, including cuda support) is near 2 Gb, as others users mentioned, the full amount of space used is certainly above 5 Gb. Unless someone knows an alternative solution…</p>
Too many compile workers and huge RAM consumption

PierreId: <p>With 4 GPU and 3 dataloader workers per GPU, I end up with 4 x (1 + 3) x 32 = 512 compile worker processes that use ~400MB each, leading to a 200GB RAM consumption.</p>
<ul>
<li>4 GPU</li>
<li>1 main process + 3 dataloader workers</li>
<li>32 compile worker process each</li>
</ul>
<p>Why ? Can I specify the number of workers created ? Can I disable the compile workers in the dataloader workers ?</p>
<p>Note : I have torch2.5.1 and I use torch.compile() with no extra params (=the default ones)</p>

ptrblck: <p>Are you calling <code>torch.compile</code> inside your <code>Dataset</code> or <code>DataLoader</code>? If not, I would not expect to see any additional workers related to <code>torch.compile</code>.</p>

PierreId: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>It looks like it is related to the <code>@torch.compile()</code> decorator that I put on a function that is part of the imported files of my <code>DataLoader</code>.</p>
<p>Is my only option to remove this decorator ? Shouldn’t it start the compile worker processes when it actually starts compiling ? In my case, this function is never called in the <code>DataLoader</code>.</p>

PierreId: <p>So, the problem occurs when my <code>DataLoader</code> imports a file with a decorated function:</p>
<pre><code class="lang-auto">@torch.compile
def do_something(A, B):
    return A + B
</code></pre>
<p>A simple workaround to avoid creating 32 compile processes in each <code>DataLoader</code> worker is:</p>
<pre><code class="lang-auto">def do_something(A, B):
    @torch.compile
    def _do_something(A, B):
        return A + B
    return _do_something(A, B)
</code></pre>
<p>When <code>do_something()</code> is called, it will compile as expected. If it’s not called, no compile processes are created.<br>
That would be nice to not have to do that.</p>
Potential issues with very large map style dataset

ChrisLiu2: <p>I have a very large amount of files on disk (on the order of billions of samples) which I have an index and I can construct a map style dataset out of. My question is, is it a good idea to do this? E.g., during training, will the shuffle index mapper be a problem? I’m trying to gauging whether there will be problems with this approach, and I would appreciate any inputs/experience here. Thanks!</p>
2 gpu model loading cuda error

Sourabh_Yadav: <p>I am trying to load qwen model in 8 bit and perform lora. I jave 2x V100 32GB. Can some one provide any idea plz. (sorry to disturb you but <a class="mention" href="/u/ptrblck">@ptrblck</a> can you help)</p>
<p>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!</p>
<p>the code I am using (I didn’t save the original code I was using but can someone help with modifications in this code so that I can split my model properly and utilize both my gpus)</p>
<p>device = “cuda” if torch.cuda.is_available() else “cpu”<br>
model_id = “Qwen/Qwen2.5-VL-3B-Instruct”</p>
<p>model = AutoModelForImageTextToText.from_pretrained(<br>
model_id,<br>
torch_dtype=torch.float16,<br>
load_in_8bit=True,  # Load in 8-bit precision<br>
device_map=“auto”<br>
)</p>
<p>min_pixels = 224<em>28</em>28<br>
max_pixels = 224<em>28</em>28<br>
processor = AutoProcessor.from_pretrained(model_id, min_pixels=min_pixels, max_pixels=max_pixels)<br>
processor.tokenizer.padding_side = “right”</p>
<p>Also my gpu-0 gets used to full and throws cuda out of memory error while gpu-1 remains at 9 GB use only.</p>
Should I install the extra cudatoolkit and cudnn?

qi_jing: <p>I install the latest pytorch from the official site with the command “conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia”. When I run the code “torch.cuda.is_available()”, the output is True. Does it mean that I don’t have to install the cudatoolkit and cudnn if I wanna run my model on GPU ? My computer is brand new and I don’t install the cudatoolkit and cudnn additionally.</p>

ptrblck: <p>The PyTorch binaries ship with all CUDA runtime dependencies and you don’t need to locally install a CUDA toolkit or cuDNN. Only a properly installed NVIDIA driver is needed to execute PyTorch workloads on the GPU.</p>

Laser_Cho: <p>Seriously? I know I should trust what you say because you are an expert, but your opinion contradicts other Google search results and other posts here on the forum.</p>

ptrblck: <p>You don’t need to trust me of course and can stick to other people’s opinions. If you check my posts here I’ve verified multiple times that no CUDA toolkit is needed and the binaries will work fine with their own shipped dependencies e.g. by installing them in a clean Ubuntu docker container. I’m also updating the CUDA dependencies in the binaries in the <code>pytorch/builder</code> status and should thus know how dependencies are used.</p>

NiDHanWang: <p>Shoud NVIDIA driver be installed on host machine or in container?</p>

ptrblck: <p>The driver needs to be installed on the host. The container can ship with a newer driver in addition, if you want to use CUDA’s forward compatibility for DataCenter GPUs.</p>

ShiZhou_Huang: <p>I think that the cudatoolkit do not need to be installed by pip, because the command is similar as <code>pip install torch==1.10.1+cu111</code>, where the cudatoolkit is installed with pytorch.<br>
While we use conda to install torch, such as <code>conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=10.2 -c pytorch</code> will install cudatoolkit.<br>
But I am confused about <code>conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 -c pytorch</code>, which seems like do not need cudatoolkit anymore.</p>

kimminw00: <p>Thank you for the clarification regarding the CUDA toolkit! I have a follow-up question: when using vLLM, Hugging Face, or Lightning with pytorch+cu121, is installing them in a clean Ubuntu Docker image similarly sufficient?</p>
Too much GPU memory usage for input/model size?

reykoki: <p>I’m using DistributedDataParallel and have 8 P100 GPUs with 16GB each, all located on a single node. I’m training DeepLabV3Plus semantic segmentation model with the effiicentnet-b0 encoder (trained model is 54MB), the input data samples are 1024x1024x3 (25MB uncompressed tiff files) output is the same size as input data.</p>
<p>The batchsize=4, num_workers=2, using the memory snapshot tool, it seems like I’m using 13.4GB on each GPU, why is the memory usage so high, is this expected? Can I approximate GPU memory needed given model/data/output sizes?</p>
DataLoader prefetches batches for the next epoch after being consumed once?

vadimkantorov: <p>When I have many small epochs, can it prefetch the first batches for the next epoch? I would assume this could be done if <code>persistent_workers = True</code></p>

ptrblck: <p>Yes, <code>persistent_workers=True</code> will not shut down all workers after the epoch end and will not restart them for the new epoch and the batches will be continuously loaded.</p>

vadimkantorov: <p>I guess there is a problem with that given that we often have <code>sampler.set_epoch(epoch)</code> especially in distributed context before the new epoch, so prefetched batches need to be discarded somehow and the sampler needs to be re-evaluated.</p>

ptrblck: <p>That’s a good point and I guess I’m wrong.<br>
Based on <a href="https://github.com/pytorch/pytorch/blob/14baca38c58084ce88f67eadc622951309490a68/torch/utils/data/dataloader.py#L353-L368">this code</a> the <a href="https://github.com/pytorch/pytorch/blob/14baca38c58084ce88f67eadc622951309490a68/torch/utils/data/dataloader.py#L515"><code>reset</code></a> method is called, which seems to grab the new sampler.<br>
You could test the behavior of the sampler with a pre-defined code snippet and check if persistent workers behave the same or not.</p>

f_g: <p>In this case, would there be a way to have the custom sampler that gets the next epoch, without ending the current one?<br>
A naive way would be to just have the sampler to generate indices for all epochs in one go. But it’s very hacky as we don’t have proper knowledge of the current epoch anymore</p>
How does autograd merges 'parallel paths'?

loryruta: <p>I’m trying to manually compute the gradients of the loss function at the top:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/639e003dd19c00e78b46ff00dc8344cb98f2d667.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/639e003dd19c00e78b46ff00dc8344cb98f2d667" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/639e003dd19c00e78b46ff00dc8344cb98f2d667_2_375x500.jpeg" alt="image" data-base62-sha1="edfMFFDmaSR25HjaZokB4uaHpgr" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/639e003dd19c00e78b46ff00dc8344cb98f2d667_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/639e003dd19c00e78b46ff00dc8344cb98f2d667_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/639e003dd19c00e78b46ff00dc8344cb98f2d667_2_750x1000.jpeg 2x" data-dominant-color="95948F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×2560 383 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The loss is the combination of a L1 loss and L_{dssim} loss and both are function of my model’s prediction y(θ).</p>
<p>Therefore I have two ways of using the Chain Rule to calculate dL/dθ. As I have highlighted in blue. Which I imagine to be two parallel paths in the computational graph.</p>
<p>How do I combine them? – Or better, how does PyTorch do so I can replicate it?</p>

soulitzer: <p>The total gradient is the sum of the gradients from each of the paths</p>

Geremia: <aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="216527">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>total gradient is the sum of the gradients</p>
</blockquote>
</aside>
<p>Why sum and not product? The chain rule involves products of derivatives.</p>

soulitzer: <p>yes, connecting two paths <strong>sequentially</strong> would be product, per the chain rule, but connecting two paths in <strong>parallel</strong> would be sum</p>
Weight normalization of lazily initialized modules

davidkwho: <p>Hi experts,<br>
I am trying to use weight normalization (torch.nn.utils.parametrizations.weight_norm) on the SAGEConv.<br>
So inside the SAGEConv class, I added the weight_norm call to each of the Linear() instances.<br>
However, SAGEConv is lazily initialized with SAGEConv(-1,…).   So I cannot do this because the parameters are uninitialized.<br>
Would you know how I can make this work?<br>
Regards,<br>
David</p>
PyTorch Not Automatically Utilizing Multiple GPUs

Sai_Yaaminie_Ganda: <p>I am working on the Lambda Vector Machine, which has 2 NVIDIA RTX 600 Ada GPUs. However, PyTorch is using only 1 GPU automatically, and I have to <strong>manually</strong> split the workload to run on both GPUs.</p>
<p>I expected PyTorch to automatically parallelize computations across both GPUs using <code>DataParallel()</code> or <code>DistributedDataParallel()</code>, but the workload remains restricted to a single GPU.</p>
<p>Are there any specific system configurations or PyTorch settings I need to adjust?</p>
<p>Thank you.</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">DDP Tutorial</a> shows how multiple GPUs can be used in DDP. Did you stick to this example or how do you execute your script?</p>

Sai_Yaaminie_Ganda: <p>Thank you for sharing the resources. I haven’t used Distributed Data Parallel (DDP) for machine learning before. I was trying to parallelize matrix multiplication for very large matrices (N &gt; 10,000 ). For this, I defined the operation as a <strong>model(N, N)</strong> and then passed it through <strong>torch.nn.DataParallel()</strong></p>

ptrblck: <p><code>DistributedDataParallel</code> will not shard matmuls or other operations. It will parallelize the workload by cloning the model to each (rank) and will execute the forward/backward passes in parallel on each rank using different data samples.<br>
If you want to apply model sharding approaches you might want to check e.g. <a href="https://pytorch.org/docs/stable/distributed.tensor.parallel.html">Tensor Parallelism</a>.</p>

Sai_Yaaminie_Ganda: <p>Thank you for the clarification. I went through the documentation and had one question - does Tensor Parallelism automatically synchronize all computations in the correct order, or do I need to manually handle synchronization to prevent race conditions or stale data?</p>
How to deal with split objects due to tiling

matiasV-TYN: <p>What is the correct way of dealing with target bounding boxes being split due to the use of tiling? Would you still keep a bounding box annotation on a tile even if a very small portion of the original object is showing? Or is there some threshold you establish that would work as another hyperparameter were you only keep the annotation if X% or more of the original bounding box is showing?<br>
I suppose there are different approaches I’m just curious about what some pitfalls might be. With the threshold approach I’m just afraid that it can feel very arbitrary and can lead to conflicting annotations and bad learning.</p>
<p>Thanks.</p>
[Solved][Pytorch1.12] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

briwa: <p>I tried many ways but nothing seems to work. As a newbie, I really don’t understand how to modify the code to make it work.<a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256">https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256?u=peter_pack</a><a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256">https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256?u=peter_pack</a></p>
<p>I found that it is because LightCNN and MobileFacenet change the input x_bar<br>
It works as follows:</p>
<blockquote>
<p>x_bar_ID_fea = LightCNN(x_bar.detach())<br>
mbf_x_bar = resize_img_to_MBF_input(args, x_bar.detach())</p>
</blockquote>
Convolution with all kernel columns same

syomantak: <p>I want to do a conv1D operation but on an image. Let’s say the data is of size <code>(1,1,H,W)</code>. Treat this as a data of size <code>(1,1,H)</code> W times, and do <code>Conv1D(1,1,k)</code> on each and stacking the outputs back. Basically, it is equivalent to doing a Conv2D with kernel size = <code>(k,W)</code>, however, with each column of the kernel equal. It is like doing <code>np.tile(kernel,(1,W))</code> on the regular Conv1D kernel and then using it on the image data.<br>
How can I do this efficiently?</p>

ptrblck: <p>Wouldn’t your approach be equivalent to use a <code>nn.Conv2d</code> with a kernel size of <code>(k, 1)</code>?<br>
This would also only use the height of the kernel and apply it basically to each “column”.<br>
Let me know, if I misunderstood your use case.</p>

syomantak: <p>Oh yeah, I guess that’s an easy way to do it. The difference is, I would need to sum all the columns after the Conv2D operation since I was basically talking about a 2D kernel of size (k,W). But yea, thanks. This pointed me in the right direction!</p>

Andrew_Ulmer: <p>Can you use conv2d in this way to apply a different kernel to each column? Like have a filter for column 1, another filter for column 2 and so on</p>

ptrblck: <p>No, the filters would convolve over the spatial input dimensions.<br>
If you want to use separate filters you could try to reshape the input and apply a depthwise convolution.</p>
Issues with exporting Pytorch model as ONNX

the-ml-bulldog: <p>I am following the instructions from this tutorial <a href="https://pytorch.org/tutorials/beginner/onnx/export_simple_model_to_onnx_tutorial.html" rel="noopener nofollow ugc">on Pytorch</a> for exporting to ONNX, so I can use Netron visualization.</p>
<p>Unfortunately, there seems to be a problem with <code>onnxscript</code>, even though I have already installed it through conda.</p>
<pre><code class="lang-auto">torch_model = torch.load(file_path)
Traceback (most recent call last):
  File "/home/user/export.py", line 22, in &lt;module&gt;
    main(FILE_PATH, shape)
  File "/home/user/export.py", line 10, in main
    onnx_program = torch.onnx.dynamo_export(torch_model, torch_input)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/test/lib/python3.12/site-packages/torch/onnx/_internal/exporter.py", line 1499, in dynamo_export
    resolved_export_options = ResolvedExportOptions(ExportOptions(), model=model)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/test/lib/python3.12/site-packages/torch/onnx/_internal/exporter.py", line 365, in __init__
    from torch.onnx._internal.fx import (  # TODO: Prevent circular dep
  File "/home/user/miniconda3/envs/test/lib/python3.12/site-packages/torch/onnx/_internal/fx/diagnostics.py", line 11, in &lt;module&gt;
    import onnxscript  # type: ignore[import]
    ^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'onnxscript'
</code></pre>
<p>Has anybody run into this issue?</p>

onedeadmatch: <p>The stacktrace is telling you that onnxscript is not found in your virtual environment, as stated in the tutorial, you should install:</p>
<pre><code class="lang-auto">pip install onnx
pip install onnxscript
</code></pre>
<p>If you have already installed both in the correct virtual environment it should work. Otherwise you may try this solution, it worked for me (adapt the code for your needs):</p>
<pre><code class="lang-auto">import torch
example_input = torch.randn(1, 3, args.img_height, args.img_width).to(device)

        torch.onnx.export(
            model,
            example_input,
            export_path,
            export_params=True,
            opset_version=args.onnx_opset_version,
            input_names=['input'],
            output_names=['output'],
            dynamic_axes={
                'input': {0: 'batch_size'},
                'output': {0: 'batch_size'}
            }
        )
</code></pre>
Advice to _overfit_ a model

emerth: <p>If you were to overfit training of Googlenet using Imagenet data, what would be your strategy?</p>
<p>I am trying to do this and all can I get are fairly well generalized results.</p>
<p>My data is the original ImageNet2012 1000-class set and I preprocessed it by taking 3 maxspect subcrops of each image and scaling them to 224x224. This is data I have used in the past with Caffe.</p>
<p>I took the script</p>
<blockquote>
<p><a href="http://github.com" rel="noopener nofollow ugc">github.com</a> / pytorch/examples/blob/main/imagenet/main.py</p>
</blockquote>
<p>and turned off augmentation via Transform because my files are preprocessed. I am manipulating the learning rate using StepLR or ReduceLROnPlateau.</p>
<p>Strategies I have tried:</p>
<p>StepLR - train for up to 100 epochs with combinations of step_size and gamma s.t. LR is ~10^-6 at 100th epoch. Gamma values like 0.1, 0.5, 0.67, 0.9, 0.925 and step_size derived from that.</p>
<p>ReduceLROnPlateau - this I am using with default parameters, and mode ‘min’.  I’m only on my first attempt with this TBH.</p>
<p>With StepLR, I have used 500 images/class, 1000 images/class, 5000 images/class. With ReduceLROnPlateau I’m using 1000 images/class.</p>
<p>Either way I get extremely well behaved training. StepLR I can make overfit just a bit: val accuracy falls a bit off best result. ReduceLROnPlateau is like super extremely well behaved: val accuracy plateaus and stays there.</p>
<p>It is as if PyTorch is so well made that it really does not want to overfit a model! How do I defeat it?</p>

KFrank: <p>Hi emerth!</p>
<aside class="quote no-group" data-username="emerth" data-post="1" data-topic="216542">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emerth/48/75379_2.png" class="avatar"> emerth:</div>
<blockquote>
<p>If you were to overfit training of Googlenet using Imagenet data, what would be your strategy?</p>
</blockquote>
</aside>
<p>I would train for a lot of epochs on a <em>small</em> dataset.  Imagenet is pretty big, so I<br>
would recommend that you start with a quite small subset of the Imagenet data.<br>
Then, depending on how much you want to experiment, see how much you can<br>
increase the size of your training subset while still getting obvious overfitting in a<br>
reasonable amount of time.</p>
<aside class="quote no-group">
<blockquote>
<p>StepLR - train for up to 100 epochs with combinations of step_size and gamma s.t. LR is ~10^-6 at 100th epoch.</p>
</blockquote>
</aside>
<p>If I read this correctly, you are only training for 100 epochs.  For any reasonably large<br>
model, this is not much training and I wouldn’t expect to see overfitting.  Try training<br>
(on a small subset) for <em>a lot</em> longer.</p>
<p>As an aside, although I haven’t looked at GoogLeNet, I would imagine it uses techniques<br>
to make it somewhat resistant to overfitting (such as dropout).</p>
<p>Best.</p>
<p>K. Frank</p>

emerth: <p>Thanks K! I will give these approaches a try.</p>
Quantized::linear (xnnpack): xnn create operator failed(2)

GangnamStyle: <p>Hello,<br>
Passing certain quantized inputs into a quantized linear layer gives the error: <code>quantized::linear (xnnpack): xnn create operator failed(2)</code>. I am on torch version 2.3.1, and using the qnnpack backend. It seems to occur when the input scale is less than or equal to 1e-38. The following is a minimal example.</p>
<pre><code class="lang-auto">import torch as th
th.backends.quantized.engine = "qnnpack"

# Create input
x = th.tensor((0.1)).reshape(1, 1)
xq = th.quantize_per_tensor(x, scale=1e-38, zero_point=0, dtype=th.qint8)

# Create FC layer
fc = th.nn.quantized.Linear(1, 1)

# Fails here
fc(xq)
</code></pre>
<p>I think it’s a case of the scale being so small that it can’t even be represented in float. However, I ran into this issue during QAT training (in fact the scale was 1e-40), and am not sure how I can force the model to only use representable values. Is it correct that the error is being caused by the input scale being too small? Is there a workaround to prevent the scale from becoming too small during training? Thank you very much.</p>

GangnamStyle: <p>I was able to fix the issue. The problem was being caused by a softmax layer producing very small non-zero outputs. For example, if given two values it could potentially output 1e-40 and 0.9999, and it seems like the 1e-40 was throwing off quantization. To fix this I’ve clipped the minimum value to 0.00001.</p>
Limiting PyTorch Worker Thread Memory Usage

saff: <p>I’ve noticed that each of my worker threads in PyTorch consumes up to 4% of memory, and I have 10 worker threads running. This adds up quickly, and I’m concerned about excessive memory usage.</p>
<p>To be on the safe side, is there a way to <strong>limit the memory usage of worker threads in PyTorch</strong>?</p>
MATLAB's Neural Network Toolbox The "Trainbr" algorithm

saleh1: <p>i want The “Trainbr” algorithm that seems to be specific to MATLAB’s Neural Network Toolbox.<br>
so what does it equal to in pytorch</p>
<p>for context i’m a researcher that doesn’t know anything about Neural Network’s just trying to make one To predict the thermal conductivity of a nanofluid based on given input’s<br>
and there seems to be a trend of using Back Propagation with Trainbr</p>

ai-mannamalai: <p>I thin you could read their documentation; its unlikely you can get help here.</p>
<p>Maybe try one of the LLM’s ?</p>
Generate Triton kernels for CPU

fhossein-quic: <p>Hi,</p>
<p>Is it possible to generate Triton kernels without having a GPU device using <code>torch.compile</code>? Can we achieve this by registering a new Dynamo backend and enabling torch-to-triton translation for the CPU?</p>
<p>Context: I want to generate triton kernels out of pytorch model for a non-GPU device, and would like to know where to start? Any docs on how this conversion works for GPU devices?</p>

fhossein-quic: <p>Any idea on this? If generating triton for cpu is not available in inductor, what would be the best way to go from torch to triton?</p>

jleeleee: <p>Do you mean running the generated Triton kernel in the CPU? You can set <code>TRITON_INTERPRET=1</code> in your environment variables, which will run the kernels in the CPU for debug mode <a href="https://github.com/openai/triton?tab=readme-ov-file#tips-for-hacking" class="inline-onebox" rel="noopener nofollow ugc">GitHub - openai/triton: Development repository for the Triton language and compiler</a></p>

fhossein-quic: <p>I mean generating triton kernels from pytorch models. Currently, torchinductor’s codegen can do this but it only works for GPU devices. For CPU, torchInductor generates C++/OpenMP code. My question is basically, how to generate triton out of torch models if I don’t have cuda device?</p>

111357: <p>Hi, as far as I know, triton was designed to generate code that ran on GPUs(Nvidia/Intel/AMD GPUs). So It can <strong>not generate code</strong> if you do not have GPU device.</p>

trusira: <p>For what it’s worth, there’s a Triton CPU backend available now. <a href="https://github.com/triton-lang/triton-cpu" class="inline-onebox" rel="noopener nofollow ugc">GitHub - triton-lang/triton-cpu: An experimental CPU backend for Triton</a><br>
I wonder if you could seamlessly replace Pytorch’s default Triton with this.</p>
Conda Pytorch keeps downgrading itself from cuda -> cpu

wilhelm: <p>I know it is a silly question and probably I’m not the only one experiencing this problem.<br>
But I really get frustrated by the fact that once I have a working PyTorch installation, that uses CUDA, then any change in my conda environment leads to an unwanted downgrade of PyTorch from cuda to cpu.</p>
<p>Now it happened again: I was using Pytorch since month. Then I needed scipy and installed it using the following command:</p>
<p><code>$conda install scipy</code></p>
<p>and then during the installation process it did the following:</p>
<p><code>  pytorch            pytorch::pytorch-2.5.1-py3.12_cuda12.~ --&gt; pkgs/main::pytorch-2.3.1-cpu_py312h4f03144_0</code></p>
<p>So, cuda is not recognized anymore. I have to reinstall my system from scratch again.</p>
<p>Why?<br>
How prevent such problems in the future?<br>
It happens 7 times of 10 when I need to install another package in my environment.</p>
<p>Any suggestion to fix it once for all?</p>

ptrblck: <p>Before confirming the updates (in this case <code>scipy</code>) your package manager should show which dependencies would be changed and you should check which packages are downgraded.<br>
These chains of dependency can be annoying if too many packages use strict requirements. PyTorch itself does not depend on <code>scipy</code> and would not care if you change its version.</p>

wilhelm: <p>Ok, but how to stop the downgrade of a package?<br>
Why should conda downgrade a package (in this case pytorch) from cuda to cpu version?</p>

ptrblck: <p>I don’t know as you didn’t show any logs, which would indicate what kind of dependencies you are changing.<br>
I can only speculate that e.g. <code>scipy</code> downgraded <code>numpy</code> that downgraded <code>transformers</code> that downgraded <code>PyTorch</code> (I don’t know if this dependency chain exists, so use this as a pure theoretical example).</p>
<aside class="quote no-group" data-username="wilhelm" data-post="3" data-topic="216525">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wilhelm/48/70280_2.png" class="avatar"> wilhelm:</div>
<blockquote>
<p>Why should conda downgrade a package (in this case pytorch) from cuda to cpu version?</p>
</blockquote>
</aside>
<p>The <code>conda install</code> command for PyTorch binaries needed the <code>pytorch-cuda==version</code> metapackage defining all CUDA runtime libraries.<br>
If you don’t specify this, the default will be used which might be CPU-only.<br>
<code>conda</code> binaries are also deprecated, so you might want to consider switching to pip wheels (which use CUDA as the default when running <code>pip install torch</code>).</p>
L-BFGS memory increasing with number of classes, not parameters?

dherrera1911: <p>I have a model that takes as input a tensor of shape <code>(c,n,n)</code> and transforms it into a tensor of shape <code>(c,m,m)</code> by multiplying on both sides with the parameter <code>filters</code> that has shape <code>(m,n)</code>, where <code>m&lt;&lt;n</code>.</p>
<p>For the loss, the output tensor of shape <code>(c,m,m)</code> is used to compute a pairwise distance matrix of size <code>c</code>-by-<code>c</code> (in this problem, <code>c</code> is the number of classes, each having an associated <code>m</code>-by-<code>m</code> matrix).</p>
<p>I am using the L-BFGS algorithm to optimize this, and it works great for most problems. However, in one problem where <code>c</code> has a large value (1700), I’m running into memory problems. What I am confused about is that, according to the L-BFGS documentation, memory usage should be determined by the size of the parameter. However, here reducing the size of the parameter <code>filters</code> by reducing <code>n</code> or <code>m</code> doesn’t prevent the memory crash, only reducing the number of classes <code>c</code> does. And the number of parameters (that is, <code>m</code> and <code>n</code>) is comparable to other problems where the model was working, that had fewer classes <code>c</code>.</p>
<p>Is there some reason why this might happen? Is there a way to try to find a work-around, since it doesn’t seem to be the number of parameters what’s causing the issue?</p>
<p>The code is in this repo <a href="https://github.com/dherrera1911/blur-spectral-sqfa" class="inline-onebox" rel="noopener nofollow ugc">GitHub - dherrera1911/blur-spectral-sqfa</a>, the model + data seem a bit complicated to provide a minimalistic example, but I can try if needed.</p>
Calling autograd.Function in autograd.Function

HanGuo97: <p>I find myself needing to manually implement gradient checkpointing for several custom autograd functions. This involves manually “merging” a few <code>autograd.Function</code> instances into a new <code>autograd.Function</code>. How can I achieve this? I would like to reuse each of the forward and backward functions for the nested <code>autograd.Function</code>, and each of these functions might save different data.</p>

soulitzer: <p>One way of doing it is just factor out the reusable logic into helper functions</p>
<p>Have you looked at <a href="https://pytorch.org/docs/stable/checkpoint.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.checkpoint — PyTorch 2.6 documentation</a> by the way, curious why that doesn’t work for your case.</p>

HanGuo97: <p>Thanks, my use case involves certain things that makes it easier to manually do it. I’m not entirely sure if <code>checkpoint</code> is the right way, but similar codebase does it manually, and I’m following them.</p>
<p>Would something like this work?</p>
<pre data-code-wrap="python"><code class="lang-python">class CustomFunction(torch.autograd.Function):

    @staticmethod
    def forward(ctx, *args, **kwargs):
        # Creating separate context objects
        ctx_hack1 = torch.autograd.function.FunctionCtx()
        ctx_hack2 = torch.autograd.function.FunctionCtx()
        ctx_hack3 = torch.autograd.function.FunctionCtx()

        # Some operations...
        out = SomeOtherFunction.forward(ctx_hack1, *args)
        out = AnotherFunction.forward(ctx_hack2, out, *args)

        # Context hacks
        ctx.save_for_backward(*ctx_hack1.to_save, *ctx_hack2.to_save)
        ctx.sub_contexts = (ctx_hack1, ctx_hack2, ctx_hack3)
        ctx_hack1.to_save = None
        ctx_hack2.to_save = None

        return out

    @staticmethod
    def backward(ctx, *args, **kwargs):
        # ...
</code></pre>

soulitzer: <p>That should work, yes</p>

HanGuo97: <p>Thank you, closing this!</p>
How to get weights and biases

J0hn: <p>Hello,</p>
<p>I have pretrained model in c++ that I trained in python. I now need to extract the weights to do my own arithmetic with it. I see that several methods exist for python. I tried using one called named_parameters() but it did not work. Does anyone have any other methods that i can try to use. Idealy I would want it to be return as a tensor.</p>

ptrblck: <aside class="quote no-group" data-username="J0hn" data-post="1" data-topic="216327">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a9a28c/48.png" class="avatar"> J0hn:</div>
<blockquote>
<p>I tried using one called named_parameters() but it did not work.</p>
</blockquote>
</aside>
<p>What kind of issue did you see?</p>
<p><code>Module::parameters()</code> is defined <a href="https://github.com/pytorch/pytorch/blob/3008368b126f81e183a0dd157e880030b28bf77d/torch/csrc/api/src/nn/module.cpp#L133">here</a> and will return a <code>std::vector&lt;Tensor&gt;</code> while <code>Module::named_parameters()</code> is defined <a href="https://github.com/pytorch/pytorch/blob/3008368b126f81e183a0dd157e880030b28bf77d/torch/csrc/jit/mobile/module.cpp#L179">here</a> and will return a <code>std::map&lt;std::string, at::Tensor&gt;</code>.</p>

J0hn: <p>Sorry I just saw this and been deleting code all day so I do not remember. My goal right now is to just get the weights and store them so I could use my own multiplication method. majority of the issue say ‘no member’ or that the function is not a member.</p>

J0hn: <pre><code>torch::jit::script::Module module;
std::vector&lt;torch::Tensor&gt; params = module.parameters();
for (auto&amp; p : params) {
    std::cout &lt;&lt; p.sizes() &lt;&lt; std::endl;
}
</code></pre>
<p>This gave me this error</p>
<p>" error: conversion from ‘torch::jit::parameter_list’ {aka ‘torch::jit::slot_list_impl<a>torch::jit::detail::ParameterPolicy</a>’} to non-scalar type ‘std::vector<a>at::Tensor</a>’ requested"</p>

J0hn: <p>Got it! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> I should be able to continue now Thank you !!!</p>
[Pytorch1.12] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

briwa: <p>I have tried many methods but nothing seems to work. As a newbie, I really don’t understand how to modify the code to make it work. Did I modify it wrong?</p>
<p><a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256">same problem</a></p>
<pre><code class="lang-auto">torch.autograd.set_detect_anomaly(True)
        
        # 1st path update D_xvs
        d_x_loss = x_bar_bar_loss_s - x_loss_s + 10. * gp_loss + v_loss_x
        
        D_xvs_solver.zero_grad()
        
        d_x_loss.backward()
        
        D_xvs_solver.step()
        
        # 2nd path, update G_xvz
        x_bar_bar_loss_v, x_bar_bar_loss_s = D_xvs(x_bar_bar)  # 
        x_bar_bar_loss_s = x_bar_bar_loss_s.mean()
        
        if reconstruct_fake:
            x_l1_loss = L1_loss(x_bar_bar, x_bar)
            v_loss_x_bar_bar = crossEntropyLoss(x_bar_bar_loss_v, vv1)  # ACGAN loss of x_bar_bar(v1)
        else:
            x_l1_loss = L1_loss(x_bar_bar, x2)
            v_loss_x_bar_bar = crossEntropyLoss(x_bar_bar_loss_v, vv2)  # ACGAN loss of x_bar_bar(v2)
        
        # id loss
        v_loss_x = crossEntropyLoss(v_bar, vv1)
        
        if not reconstruct_fake:
            l_g_rec = ID_weight * cri_rec(
                fake_fea, real_fea,
                torch.ones((real_fea.shape[0]), device=device)  # 
            )
        
            symmetry_128_loss = args.symmetry_loss_weight * Sym_loss(x_bar_bar)
        
            g_loss = -x_bar_bar_loss_s + 4 * x_l1_loss + v_loss_x_bar_bar + 0.01 * v_loss_x + l_g_rec + symmetry_128_loss
        else:
            g_loss = -x_bar_bar_loss_s + 4 * x_l1_loss + v_loss_x_bar_bar + 0.01 * v_loss_x

        G_vzx_solver.zero_grad()
        G_xvz_solver.zero_grad()

        g_loss.backward()

        if not reconstruct_fake:
            
            G_vzx_solver.step()
        
        G_xvz_solver.step()
</code></pre>