akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training stuck after first epoch

StellaJourney_Huang: <p>Hi,</p>
<p>I am using two GPUs and six number_workers to train my model. But it hangs after 1st epoch as the screenshot below. How can I solve this issue?  many thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc7f26f806235ff1f6157ba0e20e0cfe00def01d.png" data-download-href="https://discuss.pytorch.org/uploads/default/fc7f26f806235ff1f6157ba0e20e0cfe00def01d" title="Screenshot from 2023-08-22 14-49-38"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7f26f806235ff1f6157ba0e20e0cfe00def01d_2_690x192.png" alt="Screenshot from 2023-08-22 14-49-38" data-base62-sha1="A1GKbwgQE3lzch3iMTrjFS2gdWZ" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7f26f806235ff1f6157ba0e20e0cfe00def01d_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7f26f806235ff1f6157ba0e20e0cfe00def01d_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7f26f806235ff1f6157ba0e20e0cfe00def01d_2_1380x384.png 2x" data-dominant-color="35122B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-08-22 14-49-38</span><span class="informations">1388×388 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

HELLORPG: <p>Maybe you can use a <code>break</code> in your training loop to early skip the first epoch and verify whether the second epoch can be executed correctly.</p>
<p>According to my experience, this seems like something is wrong in your data processing (dataset, dataloader or datasampler). You can check some of these parts or provide some demo code to reproduce this situation.</p>

StellaJourney_Huang: <p>Hi,</p>
<p>It keeps showing errors caused from site-packages files when I debug my code deeply line by line.<br>
But when I run the original code, it’s training successfully~~~ It’s a bit weird, because I only add a new robot environment.</p>
<p>Anyway, it’s training finally. Thanks for your help!</p>
<p>Best regards</p>
Loss and Accuracy changed after saving and loading model

HELLORPG: <p>Hey guys, I have encountered a complicated problem.</p>
<p>During model training, exactly at the end of each epoch, I first save the model (and optimizer, of course) and then evaluate the model. And I try to resume this model, but get some troubles:</p>
<ol>
<li>
<p>When I try to continue training, the loss changes compared to the previous results (increase a lot). But if I only use 1 GPU for training, the loss looks unchanged. However, as I use more GPUs, the loss increases more than the previous results (I only tried on 1/2/8 GPUs).</p>
</li>
<li>
<p>When I try to evaluate it again, the accuracy significantly drops (27~ vs. 3~, drops a lot!), no matter I use how many GPUs.</p>
</li>
</ol>
<p>My project is so large that I can’t show it here. But I use the same framework with ResNet18 for Image Classification, and it works well.</p>
<p>Does anyone have some clues about what situation may cause these problems? Though I can’t provide my code here, I’ll check or try what you said.</p>
<p>Thanks a lot. It keeps me up all night.</p>

HELLORPG: <p>I tried a lot. And I found that the model parameters on GPU 0 and GPU 1 are inconsistent. That makes me really confused.</p>

InnovArul: <p>It will be easier to get an answer if you provide some code snippet on how you save and load the models,  how do you convert from single GPU training to multiGPU training etc. Without a reference,  it’s difficult to hypothesize on what might be wrong.</p>

HELLORPG: <p>Thanks for your reply!</p>
<p>I’m sorry for not providing some code snippets for this discussion (at that time, I make sure that my framework, like save and load methods are correct since I already use this framework for a long time).</p>
<p>Now I have identified the cause of this problem, and I will explain all this in my next reply.</p>

HELLORPG: <p>My mistake was that I didn’t call <code>forward()</code> after using DDP. That’s because my model design is complicated. Thus, combining the forward processes of all modules into the same <code>forward()</code> function is difficult.</p>
<p>If using DDP without calling the DDP model’s <code>forward()</code> function, all parameters in the model will not be synced between multiple GPUs. This will cause serious problems.</p>
<p>More explanation in <a href="https://discuss.pytorch.org/t/is-it-ok-to-use-methods-other-than-forward-in-ddp/176509/3">another discussion</a>.</p>
Compile error after revising ParallelOpenMP.h

sunjnnHuawei: <p>For my work purpose, I modified file <code>ParallelOpenMP.h</code> and add some functions and classes in file <code>ParallelOpenmp.cpp</code>. <code>ParallelOpenMP.h::invoke_parallel</code> call these functions. But I get some compile error which means that functions in <code>ParallelOpenMP.cpp</code> are used but never defined.<br>
Please help correct this error.</p>
Pickle error when try to save a trained mode

maryma: <p>I use pycox repository on Github for my project that is about survival analysis. This repository is based on Pytorch and torch tuples. After training a model we can save the model with</p>
<p><code>model.save_net('mynet.pt')</code></p>
<p>But when I use this code I face Pickle error that is not usual for other people. This error:</p>
<pre><code class="lang-auto">File ~/anaconda3/envs/myenv/lib/python3.9/site-packages/torchtuples/base.py:681 in save_net
    return torch.save(self.net, path, **kwargs)

  File ~/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/serialization.py:380 in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)

  File ~/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/serialization.py:589 in _save
    pickler.dump(obj)

AttributeError: Can't pickle local object
</code></pre>
<p>Can anyone help me what could be the reason for this error?</p>

ptrblck: <p>I don’t know how <code>save_net</code> is implemented, but I would recommend to store the <code>state_dict</code> of the model instead of trying to pickle the model directly as it can fail in multiple ways during the loading (e.g. if you’ve changed the file structure or any function/class definition).</p>

maryma: <p>Because I use a repository and I do not save the model myself it is hard for me to understand how can I save it with state_dict. This is the save_net part of code in repository:</p>
<pre><code class="lang-auto"> def save_net(self, path, **kwargs):
        """Save self.net and baseline hazards to file.
        Arguments:
            path {str} -- Path to file.
            **kwargs are passed to torch.save
        Returns:
            None
        """
        path, extension = os.path.splitext(path)
        if extension == "":
            extension = '.pt'
        super().save_net(path+extension, **kwargs)
        if hasattr(self, 'baseline_hazards_'):
            self.baseline_hazards_.to_pickle(path+'_blh.pickle')
</code></pre>

Vaderrulez: <p>Hi <a class="mention" href="/u/maryma">@maryma</a>,</p>
<p>I am working with the DeepHit model in the <em>pycox</em> package.</p>
<p>Lets say that you trained a model “<em>M</em>”. You can save the trained weights as follows -</p>
<blockquote>
<p>M.save_model_weights(PATH)</p>
</blockquote>
<p>Here, “<em>PATH</em>” is the location where you want to save the weights.</p>
<p>To load these weights back into an empty model (say M*), you can just do -</p>
<blockquote>
<p>M*.load_model_weights(PATH)</p>
</blockquote>
<p>Both these methods are part of the <em>DeepHit</em> class. In Deep Learning, it is a common practice to not save the entire model architecture but just the weights, which makes deploying these models very efficient.</p>
<p>Hope this helps.<br>
Ani</p>
Can pytorch still work on old GPU if build from source?

Ewan_Dao: <p>My environment is</p>
<ul>
<li>NVIDIA GeForce 930MX + latest Nvidia driver</li>
<li>win10 + WSL2 + Docker Desktop + <a href="http://nvcr.io/nvidia/pytorch:23.07-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:23.07-py3</a></li>
</ul>
<p>What works:</p>
<ul>
<li>compare running <code>cuda-sample:nbody</code> container on GPU vs CPU shows GPU acceleration is available:<br>
** GPU 450.378 single-precision GFLOP/s<br>
** CPU 0.626 single-precision GFLOP/s</li>
<li>torch.cuda.is_available() returns True.</li>
</ul>
<p>What doesn’t work:</p>
<blockquote>
<p>NVIDIA GeForce 930MX which is of cuda capability 5.0.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability supported by this library is 5.2.</p>
</blockquote>
<p>I heard build from source can make old GPU work, so I tried:</p>
<blockquote>
<p>python setup.py develop</p>
</blockquote>
<p>It took a few hours to build, surprisingly there was no error, pytorch is successfully built, thanks to docker.<br>
However, pytorch still does not work on the old GPU.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4d96a8882a39d242014ace17d1b8d2b0363b422.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4d96a8882a39d242014ace17d1b8d2b0363b422" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4d96a8882a39d242014ace17d1b8d2b0363b422.png" alt="image" data-base62-sha1="nwk7g7vbgJm5WOucWh30p05DNBg" width="690" height="306" data-dominant-color="F5EBEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">992×440 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can pytorch still work on old GPU if build from source?<br>
Would recompile with <code>TORCH_USE_CUDA_DSA</code> make it work? What other flags should I enable/disable when recompile?</p>

ptrblck: <p>How did you install PyTorch as the latest nightly with CUDA 12.1 should support compute capabilities 5.0+ and the nightly with 11.8 should support 3.7+:</p>
<pre><code class="lang-python">torch.cuda.get_arch_list()
['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
</code></pre>

Ewan_Dao: <p>For me, it returns:<br>
<code>['sm_52', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_90']</code><br>
Guess I should use cuda toolkit 11.8 instead.<br>
Thank you very much.</p>
Poor performance with eval() and batch normalization

Tim_Henderson: <p>Hi,</p>
<p>I have an simple dataset where the input is 55 floating point features and each sample has a classification output as 0 or 1. The dataset is split roughly equally in terms of how many samples of each class there are.</p>
<p>I can get good performance during training, but when using eval() my model will only output zeros.</p>
<p>The model is as follows:</p>
<blockquote>
<p>model = nn.Sequential(<br>
nn.Linear(55,128),<br>
nn.BatchNorm1d(128),<br>
nn.ReLU(),<br>
nn.Linear(128,64),<br>
nn.BatchNorm1d(64),<br>
nn.ReLU(),<br>
nn.Linear(64,8),<br>
nn.BatchNorm1d(8),<br>
nn.ReLU(),<br>
nn.Linear(8,1),<br>
nn.Sigmoid()<br>
)</p>
</blockquote>
<p>I have tried setting track_running_stats to False for the bn layers, and when using eval() the model outputs actual values, but they are all roughly between 0.2 &amp; 0.3 so the classification as 0 or 1 is still poor.</p>
<p>Is there something I am fundamentally misunderstanding? I should add that my batch size is 128.</p>

eqy: <p>Is there some distribution shift between your eval and train data setups (e.g., perhaps due to normalization differences)? You can inspect whether the statistics of each batch are varying at training vs. eval time by just computing the mean and variance manually in the forward pass.</p>
FSDP with high batch size makes GPU memory usage to flunctuate

Mansu_Kim: <p>When I use FSDP, I saw some weird things.</p>
<p>When I use FSDP with 14 batch per device, GPU memory usage fixed. But, with 16 batch, GPU memory has fluctuated.</p>
<p>With 16 batches, training works fine. But sometimes train failed with OOM. (No dynamic shapes are used.)</p>
<p>Could someone give some insight into the reasons behind the fluctuation in memory usage?</p>
<h2><a name="h-14-batch-1" class="anchor" href="#h-14-batch-1"></a>14 batch</h2>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/8017a1cfbe87ac1dd1569e193bd811dd82240460.png" data-download-href="https://discuss.pytorch.org/uploads/default/8017a1cfbe87ac1dd1569e193bd811dd82240460" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8017a1cfbe87ac1dd1569e193bd811dd82240460_2_690x254.png" alt="image" data-base62-sha1="ih9KmbyscIHhDxOFgRwiQxadAYM" width="690" height="254" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8017a1cfbe87ac1dd1569e193bd811dd82240460_2_690x254.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8017a1cfbe87ac1dd1569e193bd811dd82240460_2_1035x381.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8017a1cfbe87ac1dd1569e193bd811dd82240460_2_1380x508.png 2x" data-dominant-color="3B2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1878×692 87.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h2><a name="h-16-batch-2" class="anchor" href="#h-16-batch-2"></a>16 batch</h2>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fed5d8eb8c7908e7f2c337ab0c9137351974ca13.png" data-download-href="https://discuss.pytorch.org/uploads/default/fed5d8eb8c7908e7f2c337ab0c9137351974ca13" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed5d8eb8c7908e7f2c337ab0c9137351974ca13_2_690x254.png" alt="image" data-base62-sha1="Amnru6BlS0UriRJRH40qCG0GnAf" width="690" height="254" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed5d8eb8c7908e7f2c337ab0c9137351974ca13_2_690x254.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed5d8eb8c7908e7f2c337ab0c9137351974ca13_2_1035x381.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fed5d8eb8c7908e7f2c337ab0c9137351974ca13_2_1380x508.png 2x" data-dominant-color="3B312E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1878×692 94.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Upgrading to PyTorch 2.01 gives torch.cuda.OutOfMemoryError

vineetk1: <p>I have a Nvidia GTX 1080. I was using PyTorch 1.8 and my application was working fine. After upgrading to PyTorch 2.01 and Cuda 11.7, my application gives the following error:<br>
<strong>torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 7.93 GiB total capacity; 6.98 GiB already allocated; 32.38 MiB free; 7.09 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</strong></p>
<p>Setting the environmental variable to <strong>garbage_collection_threshold:0.6,max_split_size_mb:256</strong> gives the following error:<br>
<strong>torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 7.93 GiB total capacity; 6.79 GiB already allocated; 37.06 MiB free; 6.83 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</strong></p>
<p>I have tried various combinations of <strong>max_split_size_mb</strong>  but I still get an error. Is it possible to fix this error, or I should go back to PyTorch 1.8?</p>
Training from a checkpoint

StellaJourney_Huang: <p>Hi,</p>
<p>When I tried to continue training from a checkpoint<br>
<code>python scalingup/train.py dataset_path=scalingup/wandb/run-20230808_082724-ioa7gmrt/files/ evaluation=bin_transport_test algo=diffusion_default algo.replay_buffer.batch_size=256 tags='[sim,transport,diffusion]' load_from_path=scalingup/wandb/run-20230808_205236-ib7bi2uq/files/checkpoints/last.ckpt</code>, the following error happened:</p>
<pre><code class="lang-auto">Error executing job with overrides: ['dataset_path=scalingup/wandb/run-20230808_082724-ioa7gmrt/files/', 'evaluation=bin_transport_test', 'algo=diffusion_default', 'algo.replay_buffer.batch_size=256', 'tags=[sim,transport,diffusion]', 'load_from_path=scalingup/wandb/run-20230808_205236-ib7bi2uq/files/checkpoints/last.ckpt']
Traceback (most recent call last):
File "/home/yan/Documents/scalingup/scalingup/train.py", line 99, in train
trainer, algo = setup_trainer(
File "/home/yan/Documents/scalingup/scalingup/train.py", line 46, in setup_trainer
algo = ScalingUpAlgo.load_from_checkpoint(
File "/home/yan/mambaforge/envs/scalingup/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1531, in load_from_checkpoint
loaded = _load_from_checkpoint(
File "/home/yan/mambaforge/envs/scalingup/lib/python3.10/site-packages/pytorch_lightning/core/saving.py", line 90, in _load_from_checkpoint
return _load_state(cls, checkpoint, strict=strict, kwargs)
File "/home/yan/mambaforge/envs/scalingup/lib/python3.10/site-packages/pytorch_lightning/core/saving.py", line 136, in _load_state
obj = cls(_cls_kwargs)
TypeError: Can't instantiate abstract class ScalingUpAlgo with abstract method get_policy
</code></pre>
<p>Is it a bug that can be fixed?</p>
<p>Best regards</p>

ptrblck: <p>This issue seems to be Lightning-specific so you might want to post the question in their discussion board.</p>
How to go from a lower output shape to higher output shape Conv2d?

lint_weaver: <p>I’m trying to implement an Image classification model, but I’m having trouble understanding how to go from a lower output shape to a higher output shape. Below is a picture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccaacdef9a3542ba0eaaa9bea2102d1e57e73a21.png" data-download-href="https://discuss.pytorch.org/uploads/default/ccaacdef9a3542ba0eaaa9bea2102d1e57e73a21" title="SCR-20230822-macz"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccaacdef9a3542ba0eaaa9bea2102d1e57e73a21_2_690x225.png" alt="SCR-20230822-macz" data-base62-sha1="tczlKrgnvyXJdGJN7oiwcP47X2h" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccaacdef9a3542ba0eaaa9bea2102d1e57e73a21_2_690x225.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/ccaacdef9a3542ba0eaaa9bea2102d1e57e73a21_2_1035x337.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccaacdef9a3542ba0eaaa9bea2102d1e57e73a21.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">SCR-20230822-macz</span><span class="informations">1274×416 21.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How does the author of this paper go from Maxpool2d_2 to conv2d_3? The only way that I can imagine this happening is if you use nn.ConvTranspose2d(), however this is not mentioned in the paper. Is there an alternative way of doing this step?</p>
<p>The paper: <a href="https://www.sciencedirect.com/science/article/pii/S0960077920308870#sec0002" rel="noopener nofollow ugc">https://www.sciencedirect.com/science/article/pii/S0960077920308870#sec0002</a></p>

ptrblck: <p>It’s unclear as the authors didn’t share enough information, but they might have used a small stride and large padding.<br>
Here is an example:</p>
<pre><code class="lang-python">conv = nn.Conv2d(128, 64, 3, 1, padding=7)
x = torch.randn(1, 128, 9, 9)
out = conv(x)
print(out.shape)
# torch.Size([1, 64, 21, 21])
print(torch.tensor([p.nelement() for p in conv.parameters()]).sum())
# tensor(73792)
</code></pre>
<p>As you can see it’s just a guess and I didn’t consider any other <code>dilation</code> argument, which might also be different. The number of parameters also doesn’t match, so take this code with a grain of salt.</p>
Performance changes when the model is loaded on another machine with a different GPU

augusmaa: <p>Hello everyone,</p>
<p>I have a question about loading a saved model.<br>
I trained a model on a server with 8 GPU using nn.Dataparallel I got a performance of 78.07% accuracy on the validation set. But I reload the saved model on another server with 2 GPU (still using Dataparallel) to run the validation script (of course with the same dataset), and the performance decreases to 76.86% accuracy. I am wondering why.</p>
<p>Note: When I go back to the server where the model has been trained everything works normally, I found exactly the same performance by running the validation script.</p>
<p>I would like to know if I did something wrong either in saving the model or in re-loading the saved-model.<br>
If you have any advice that could help will be welcome.</p>
<p>To save the model I used:</p>
<pre><code class="lang-auto">torch.save({
             'epoch':epoch,
             'model_state_dict': model.state_dict(),
             'optimizer_state_dict':optimizer.state_dict(),}, ‘./save_model_folder/model.tar')
</code></pre>
<p>To load the saved model I used:</p>
<pre><code class="lang-auto">checkpoint = torch.load(‘./save_model_folder/model.tar’)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
</code></pre>
<p>Thank you in advance.</p>

marksaroufim: <p>Are you adjusting the global batch size to account for the smaller number of GPUs? The numerics of your program will change if you change the number of GPUs and nothing else</p>

augusmaa: <p>Yes, I did. I think there is something related to the server that I used,  because when I evaluated the model on a third server (another one, different of 2 others) , I found the right results.</p>
Starting backward from any convolutional layer

hamedB: <p>Hi<br>
The backward function starts the gradient operation from the output respect to the input.<br>
How can I compute gradient from specific convolution layer in specific layer, instead from output, respect to input?Is there any solution in Pytorch or should I create my own backward function?</p>
<p>Thanks you in advance.</p>

Darleen2019: <p>I have encountered the same problem. Have you solved the problem? Can you teach me some experience</p>
1x1 convolution in PyTorch

Mah_Neh: <p>Suppose you have a CNN to deal with images, and your input images can be 1 (grey), 3(color) or 4 channels (color + transparency).</p>
<p>Would a 1x1 convolution and (maybe with 3 filters) in a first layer, be able to handle those cases, and make the output channels after the convolution always the same (3 in this case), so that we do not have to deal with images transformations elsewhere?</p>

ptrblck: <p>Could you explain the idea a bit more? Why should the conv layer create the same outputs and why would you want that?</p>

Mah_Neh: <p>Yes, I was reading this paper <a href="https://arxiv.org/pdf/2105.05787.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2105.05787.pdf</a>, and I thought that the convolutions of 1x1 could help to deal with images with different number of channels, but i must be wrong, because my model includes that the depth will always be the number of channels of the image.</p>
How to get rid of a message while running torch 2.0

Nirmai: <p>============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============<br>
verbose: False, log level: Level.ERROR<br>
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================</p>
<p>is there a way to get rid of this message</p>

ATYslh: <p>I don’t know the reason for the message, but if you simply want to surpress it you can do the following:</p>
<pre><code class="lang-auto">import warnings
warnings.filterwarnings("ignore", ".*Diagnostic Run.*")
</code></pre>
Number of initial input channels

ATYslh: <p>Hello everyone,<br>
I have a question about the number of initial input channels.<br>
Currently the number of channels in my cnn looks like this for the Conv2D layers:<br>
data input 2 → 32 → 64 → 128 → 256</p>
<p>I get quite good results with this. When I increase the number of initial input channels (from 2 to 4 for example since I have a lot of different variables from a climate model) I don’t see any real improvement in the classification. Can I draw the conclusion that the additional parameters don’t contribute in any meaningful way, or might it have something to do with the number of network channels? I just want to be sure that I don’t mess things up.</p>
Creating a multi-dimensional MultivariateNormal

ekosman: <p>Hi,</p>
<p>I want to create a multivariate normal distribution that spans features across multiple dimensions. For example, the mean tensor is of shape <code>(10, 2)</code> and I want all entries to be possibly correlated, which means I provide a tensor with 20^2 entries that represent the covariance. Is this achievable using the <code>MultivariateNormal</code> object of torch distribution? I would expect a distribution of event shape <code>(10, 2)</code> then.</p>
Exclusive encoder-decoder architecture

Peter_Featherstone: <p>How do you train an encoder-decoder pair such that they are exclusive? Specifically, I would like 2 things:</p>
<ul>
<li>when you train 2 pairs of encoder-decoder networks, you cannot mix the encoder and decoder from different pairs</li>
<li>if you have a pretrained encoder, you cannot retrain a decoder from scratch and likewise the other way round. I.e you can only train them together end-2-end</li>
</ul>
<p>Cheers</p>

vdw: <p>I’m not really sure what you’re looking for, so my reply might be a bit off.</p>
<p>Regarding your first point, I would argue this is the default situation. Let’s assume you use the exact same encoder-decoder architecture to train two models using the exact same dataset. If both models are initialized randomly and trained with randomized batches, the loss for both models is likely to converge to different minima. Thus, say, then using the encoder of Model 1 and the decoder of Model 2 should yield subpar results. Of course, you <em>can</em> still do it.</p>
<p>Regarding your second point, this seems to contradict the idea of pretraining and transfer learning. If you have a pretrained encoder-decoder model, and then use the pretrained encoder to train a new decoder, I cannot see why the new decoder – at least in principle – might not converge to pretrained decoder (i.e., have more or less the same weights in the end). Of course, the new decoder might also be very different but also yield equally good results.</p>

Peter_Featherstone: <p>So let’s imagine you’re trying to build NN encryption. This is not my use-case but let’s entertain it for now. (i don’t think a neural network could ever provide a cryptographically secure algorithm…) Let’s imagine your encoder is the encryption key and the decoder is the decryption key. Because you can retrain your decoder from scratch from a frozen encoder, and likewise the other way round, if you intercept the encryption key, you can rebuild your decryption key. Likewise the other way round. This is not great. So you have to treat them as the same key. So you end up with something more similar to symmetric encryption like AES. However, I want something like asymmetric key encryption with a public and private key. You cannot “retrain” a private key from a public key. So I want to train an encoder and decoder such that, once trained, if i intercept the encoder, there is no way to retrain the decoder. And ideally the same the other way round.<br>
The only thing I could think of was to introduce non-differentiable layers, which you could somehow train end-to-end the first time  round, but not easily the second time round.<br>
Does this make sense?</p>
How to do multitask learning for seq2seq models (NLP tasks)

Arij-Aladel: <p>Hello I am trying to figure out the following moments in multitask learning process</p>
<ul>
<li>Dalaloader for generating batches with one task every time</li>
<li>calculating loss and metrics and importance of each according to datasets(tasks) rates and datasets(task) importance</li>
</ul>
<p>How can we do that in the right way for seq2seq models?</p>
Why is Elastic designed to restart all processes?

adepp: <p>I have two questions regarding the design of Elastic：</p>
<ol>
<li>How can we differentiate between scaling down and a fault (i.e., whether the process automatically restarts after being killed) in Elastic?</li>
<li>Why is it necessary to restart all processes in Elastic instead of making modifications while keeping the existing processes intact? What are the main factors considered in making this decision?  What are the differences between automatic restart and manually loading the checkpoint to restart the job after an interruption?</li>
</ol>
Optimize_for_mobile applied for segmentation task gives me the exact same mask

Eddie_Carraro: <p>i’m using a DeepLabv3 with ResNet50 model to apply segmentation to a set of portraits. When i quantize and optimize it for mobile, i would expect a reduction of the precision of the segmentation, but contrarily i obtain the same score(i’m using IoU and Dice to verify this, and the scores for both the models are identical).</p>
Gradients type in torch.cuda.amp

fiona: <p>Hi, I have some questions about torch.cuda.amp. I want to use mixed precision training in my project and I notice that the grads of network parameters is still float32, which is really bothering me. I consider that the fp16 params should have fp16 gradients and fp32 params should have fp32 gradients in mixed precision training. So ,is there any wrong with me? Can anyone help me ?<br>
The following is my test code:</p>
<pre><code class="lang-auto">class CustomModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(CustomModel, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc3 = nn.Linear(hidden_size, num_classes)
        
    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out

# 初始化模型
input_size = X_train_tensor.shape[1]
hidden_size = 16000
num_classes = 2000
model = CustomModel(input_size, hidden_size, num_classes).to('cuda')
scaler = torch.cuda.amp.GradScaler()
# 定义损失函数和优化器
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# 训练模型
num_epochs = 1
for epoch in range(num_epochs):
    optimizer.zero_grad()
    with torch.cuda.amp.autocast(dtype=torch.float16, enabled=True):
        outputs = model(X_train_tensor)
        loss = criterion(outputs, y_train_tensor)
        
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    
    print(outputs.dtype)
    print(model.fc1.weight.grad.dtype)
    print(model.fc2.weight.grad.dtype)

print("Done!")
</code></pre>
<p>and the output is</p>
<pre><code class="lang-auto">torch.float16
torch.float32
torch.float32
Done!
</code></pre>

fiona: <p>@ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> Do you have any comments? Thank you very much!</p>

ptrblck: <aside class="quote no-group" data-username="fiona" data-post="1" data-topic="186748">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fiona/48/60181_2.png" class="avatar"> fiona:</div>
<blockquote>
<p>I consider that the fp16 params should have fp16 gradients and fp32 params should have fp32 gradients in mixed precision training.</p>
</blockquote>
</aside>
<p>All parameters are in <code>float32</code> using <code>amp</code>. If you want <code>float16</code> parameters you would need to manually cast them and might run into the risk of divergence etc.</p>

fiona: <p>I don’t understand why all parameters are in float32 using amp. And where’s the mixed precision? You mean that as all parameters are in float32, and the gradients are in float32? If so, is it necessary to use grad scaler? I consider that grad scaler makes sense in gradients which is float16.</p>
Device-aware Tensor hash

gonzrubio: <p>Is there a GPU equivalent hash implementation for the Tensor wrapper bellow? I want sto store unique multi-dimensional tensors in a set.</p>
<pre><code class="lang-auto">"""Wrapper to store tensors in a set."""

import torch


class HashTensor:
    def __init__(self, obj):
        self.obj = obj

    def __hash__(self):
        return hash(self.obj.cpu().numpy().tobytes())

    def __eq__(self, other):
        if isinstance(other, HashTensor):
            return torch.equal(self.obj, other.obj)
        elif isinstance(other, torch.Tensor):
            return torch.equal(self.obj, other)
        return False

    def __repr__(self):
        return repr(self.obj)
</code></pre>
PCA on torch.Datasets.SVHN

hormone03: <p>I am working on a machine-learning classification model. It works fine on greyscale data set. But the performance on RGB images is poor. So, I decided to apply <code>torch.pca_lowrank</code> on the RGB data set to see if this can improve the feature representations and the model’s performance.</p>
<p>Here is my code sample:</p>
<pre><code class="lang-auto">U, S, V = torch.pca_lowrank(torch.as_tensor(train_dataset) , q=None, center=True, niter=3)
train_dataset = torch.matmul(train_dataset, V[:, :n_components])
</code></pre>
<p>I am getting this error:</p>
<pre><code class="lang-auto">ValueError: only one element tensors can be converted to Python scalars
</code></pre>
<p>Any suggestions, please? Thanks.</p>

ptrblck: <p><code>torch.as_tensor(train_dataset)</code> looks wrong as it seems you are trying to transform the entire dataset into a single tensor. If you’ve preloaded all samples in your <code>Dataset</code>, access this data directly and pass it to <code>pca_lowrank</code> instead.</p>

hormone03: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
That saves my day.</p>
How to pass gradients through a manually set model?

rao: <p>Consider the following code:</p>
<pre><code class="lang-auto">n_inputs = 2
n_outputs = 2
model = nn.Sequential(
    nn.Linear(n_inputs,32),
    nn.ReLU(inplace=True),
    nn.Linear(32, 64),
    nn.ReLU(inplace=True),
    nn.Linear(64, 32),
    nn.ReLU(inplace=True),
    nn.Linear(32, n_outputs),
    nn.Tanh(),)

output = state(x)
print(output.grad_fn)
</code></pre>
<p>Rightfully prints: <code>&lt;TanhBackward0 at 0x...&gt;</code></p>
<p>But, now when I use some other function F (could be a neural network) to generate weights for the model and use load_state_dict or any other inplace weights assignment the gradient flow is broken. For example:</p>
<pre><code class="lang-auto">new_weights = F(some_input) 
new_state_dict = OrderedDict()
for name,layer in model.state_dict().items():
    end += np.prod(layer.shape)
    new_state_dict[name] = new_weights[start:end].view(layer.shape)
    start = end

model.load_state_dict(new_state_dict)
output = state(x)
output.grad_fn
</code></pre>
<p>None.</p>
<p>May be load_state_dict is not a part of autograd, which is understandable, but I can not think of an non-equivalent way to reset weights manually through another gradient-requiring process. I can do it for a simple neural network explicitly defining the layers but still it doesnt scale.</p>
<p>Any help? Thanks</p>
Extracting a submatrix from a matrix

vadimkantorov: <p>Given a 2-d tensor <code>x</code>, 1-d index tensors <code>I</code> and <code>J</code>, I knew of the way of extracting submatrix as <code>x[I][:, J]</code> which works.</p>
<p>But at <a href="https://stackoverflow.com/a/19170890/445810" class="inline-onebox" rel="noopener nofollow ugc">python - Numpy extract submatrix - Stack Overflow</a> I also found for NumPy a way of <code>x[I[:, None], J[None, :]]</code>. This seems to also work for PyTorch. Is it documented anywhere? Is it officially supported?</p>
<p>Which way is more performant in terms of extra allocations? (the first way would seem to reallocate first <code>x[I]</code>, but I don’t know if the second way does any hidden reallocations due to the index broadcasting)</p>
About ROCm 5.6.0 on Windows

Sorpfenvier: <p>It is said that, the newest ROCm version, 5.6.0 Alpha, supports some AMD consumer GPUs on Windows now.<br>
Would Pytorch be supporting AMD GPUs on Windows soon?</p>

Lesha: <p>I am also interested. Here is the article if anyone is interested in reading: <a href="https://www.tomshardware.com/news/amd-rocm-comes-to-windows-on-consumer-gpus" class="inline-onebox" rel="noopener nofollow ugc">AMD ROCm Comes To Windows On Consumer GPUs | Tom's Hardware</a></p>

Alex_Hawk: <p>Part of the AMD ROCm Platform, the HIP SDK is now available on Windows. 5.5 was released on July 27th, 2023.<br>
now it’s online: <a href="https://www.amd.com/en/developer/rocm-hub/hip-sdk.html" rel="noopener nofollow ugc">https://www.amd.com/en/developer/rocm-hub/hip-sdk.html</a><br>
<a href="https://docs.amd.com/en/latest/release/windows_support.html" class="inline-onebox" rel="noopener nofollow ugc">GPU and OS Support (Windows) — ROCm Documentation Home</a></p>

silvia95guy: <p>Yes, I am very interested too about this, and hope it is supported soon with pytorch, so we can get the most out of the newer AMD cards.</p>

Ay_Person: <p>Being curious and impatient, I installed the latest ROCm windows SDK and grabbed the source to build pytorch for Windows and ROCm, this is what I found.</p>
<p>There are a lot of assumptions in the build files and in the code made around which libraries are available on which platforms, eg: CUDA, only if NOT_MAC, ROCm must be on linux. that is one thing that will require some maintenance but that’s not so bad.</p>
<p>There was one place in code that would be used on windows that wasn’t where a thread exposes a bool that needs changed to get it to build.</p>
<p>That’s about it, although I got it to build, there was a lot of those assumptions with definitions and the package I produced resulted in a “working” pytorch that spewed errors about CUDA not being available if you tried to do any heavy lifting with it. This satisfied my curiosity about if the ROCm windows SDK even had all the math libraries one would need to build pytorch and it appears that it does. It shouldn’t be far off that someone familiar with this project could pick off this low hanging fruit.</p>
CUDA memory unavailable despite clearing cache and garbage

kanishkanarch: <p>I am using AirSim with an ESPNet segmentation inference model on a <strong>GeForce RTX 2060 SUPER</strong>. It was working until CUDA started showing the OOM error. Despite <code>nvitop</code> and <code>nvidia-smi</code> showing free GPU memory, <code>torch.cuda.mem_get_info()</code> is showing that atleast 70% of my memory is filled (even going above 90% in sometimes).<br>
I tried <code>torch.cuda.empty_cache</code> and <code>gc.collect()</code> outside the program but they aren’t helping me.</p>
<p>How can I free up my filled up memory?</p>
<p>What can be the reason <code>torch.cuda.mem_get_info()</code> is showing filled memory? Please suggest some solutions to free up GPU memory that’s occupied with previously ended programs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96f92cfda9738b26ce736e68844211c66e4919b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/96f92cfda9738b26ce736e68844211c66e4919b9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f92cfda9738b26ce736e68844211c66e4919b9_2_464x500.png" alt="image" data-base62-sha1="lxztA7QS83LXZ0ZEJcL1OEFywvT" width="464" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f92cfda9738b26ce736e68844211c66e4919b9_2_464x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f92cfda9738b26ce736e68844211c66e4919b9_2_696x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96f92cfda9738b26ce736e68844211c66e4919b9_2_928x1000.png 2x" data-dominant-color="F5F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1079×1162 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="kanishkanarch" data-post="1" data-topic="186742">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/7993a0/48.png" class="avatar"> kanishkanarch:</div>
<blockquote>
<p><code>torch.cuda.mem_get_info()</code> is showing that atleast 70% of my memory is filled (even going above 90% in sometimes).</p>
</blockquote>
</aside>
<p><code>torch.cuda.mem_get_info()</code> returns:</p>
<blockquote>
<p>Returns the global free and total GPU memory for a given device using <code>cudaMemGetInfo</code>.</p>
</blockquote>
<p>Dividing the first by the second value calculates the relative free memory, not the occupied one.</p>
Loading the checkpoint doesn't lead to identical results

papoo13: <p>I am using a ResNet18 in a continual learning setting. In such a setting, we train the model on multiple tasks. so after training task_1, we use the weights of the trained model and start training task_2, and so on. In such a setting I would like to be able to start training from a specific task, aka, use the checkpoints of a model trained on task_n and the information about the coreset, I start training from task_n+1 by loading the checkpoint. What I expect is to see the exact same results compared to when I continue training from task_n to task_n+1 because the seeds are set and determinism is on. However, I see slightly different results when plotting accuracy and loss over the training and test sets. When I run my experiments twice, both times from task_0 to task_n, I get identical results.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/6088cc67d90c3382632fcf9b9a26e95bc6cd59de.png" data-download-href="https://discuss.pytorch.org/uploads/default/6088cc67d90c3382632fcf9b9a26e95bc6cd59de" title="ckpreproduce"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6088cc67d90c3382632fcf9b9a26e95bc6cd59de_2_690x234.png" alt="ckpreproduce" data-base62-sha1="dLYVrvwUk5V7l0TEf9eOWswcjRs" width="690" height="234" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6088cc67d90c3382632fcf9b9a26e95bc6cd59de_2_690x234.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6088cc67d90c3382632fcf9b9a26e95bc6cd59de_2_1035x351.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/0/6088cc67d90c3382632fcf9b9a26e95bc6cd59de.png 2x" data-dominant-color="EBE9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ckpreproduce</span><span class="informations">1149×391 31.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the above figure, you can see that two runs, GPUoriginal and GPUoriginal2 show identical results on task_1 (we start from task_0). Still, GOUoriginalckp, when using the checkpoint of task_0 of GPUoriginal and its updated memory, shows a different performance.<br>
I checked the model weights at the start of task_1 for GPUoriginal and GPUoriginalckp, and they were the same. All the seeds are set as well and I’m running all the experiments on the same hardware.<br>
Is there anything else I need to consider? as I expect identical results.<br>
I appreciate any help.</p>