akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
OutOfMemoryError: CUDA out of memory. Despite plenty of free memory

FloconDeNeige: <p>I don’t know much about Pytorch, I’m just using it for stable diffusion, but I’m facing an annoying issue. I am plagued by OOM errors, despite having more than enough VRAM for what I’m doing.</p>
<p>Example:</p>
<blockquote>
<p>OutOfMemoryError: CUDA out of memory. Tried to allocate 2.29 GiB (GPU 0; 24.00 GiB total capacity; 10.43 GiB already allocated; 12.05 GiB free; 10.65 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>As you can see pytorch tries to allocate much less memory than what is free. Sometimes it can fail to allocate even smaller chunks of memory (~1GiB), when more than 18GiB are free.<br>
Pytorch seems incapable of reaching 50% vram utilization, it always crashes before. And it’s not normal, others don’t seem to be getting this issue.<br>
The supporter of the app couldn’t help, so I am turning to here instead.</p>
<p>It happens around the scaled_dot_product_attention there:</p>
<blockquote>
<p>def sdp_attnblock_forward(self, x):<br>
h_ = x<br>
h_ = self.norm(h_)<br>
q = self.q(h_)<br>
k = self.k(h_)<br>
v = self.v(h_)<br>
b, c, h, w = q.shape # pylint: disable=unused-variable<br>
q, k, v = (rearrange(t, ‘b c h w → b (h w) c’) for t in (q, k, v))<br>
dtype = q.dtype<br>
if shared.opts.upcast_attn:<br>
q, k, v = q.float(), k.float(), v.float()<br>
q = q.contiguous()<br>
k = k.contiguous()<br>
v = v.contiguous()<br>
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, dropout_p=0.0, is_causal=False)<br>
out = out.to(dtype)<br>
out = rearrange(out, ‘b (h w) c → b c h w’, h=h)<br>
out = self.proj_out(out)<br>
return x + out</p>
</blockquote>
<p>This is the version I am using on python 3.10.11:</p>
<p>10:18:37-117553 INFO Torch 2.0.1+cu118<br>
10:18:37-129554 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700</p>
<p>Nvidia-smi also seems to agree with the numbers given, and shows plenty of memory free.</p>
<p>I don’t know what info you would need to help elucidate the issue. Just ask and I will do my best to provide.</p>
<p>Thank you,</p>
Training freezes at “initializing ddp: GLOBAL_RANK.."

nonathecoda: <p>I’m trying to <a href="https://github.com/zju3dv/LoFTR/blob/master/docs/TRAINING.md" rel="noopener nofollow ugc">train LoFTR</a> on 4 RTX 3090 GPUs on Ubuntu 18.04. When I start training, the output gets stuck on “initializing ddp: GLOBAL_RANK” and the terminal freezes (Ctrl + C won’t work anymore).</p>
<p>I saw that others had that problem with certain Pytorch / Pytorch Lightning versions, however i’m using pytorch-lightning==1.3.5 and pytorch=1.8.1, which noone else seemed to have problems with. Also, the authors trained LoFTR with the same environment, so I think the problem has to be somewhere else.</p>
<p>Does anyone have an idea what else apart from the Pytorch / Pytorch Lightning versions could be the problem?</p>
<p>Thanks!</p>
<p>P.S. I ran the cuda p2pBandwithLatencyTest and it shows that GPUs can’t access each other. However I’m not sure if this is needed in this case or if they can just pass the information via CPU.</p>

ptrblck: <aside class="quote no-group" data-username="nonathecoda" data-post="1" data-topic="182480">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nonathecoda/48/61469_2.png" class="avatar"> nonathecoda:</div>
<blockquote>
<p>P.S. I ran the cuda p2pBandwithLatencyTest and it shows that GPUs can’t access each other.</p>
</blockquote>
</aside>
<p>Try to disable p2p access via <code>NCCL_P2P_DISABLE=1</code> and check if this would help.</p>

nonathecoda: <p>When i ran <code>torch.tensor(1).cuda()</code>, it returned</p>
<blockquote>
<p>NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.</p>
</blockquote>
<p>So seems to be that I have to use a higher CUDA version. Thank you!</p>

ptrblck: <p>Yes, the error shows you’ve installed a PyTorch binary with CUDA 10.2, which is not compatible with your 3090. Update PyTorch to any binary in the current stable or nightly release and it will work (as all new binaries use CUDA 11.7, 11.8, or 12.1).</p>
Accuracy ~ 0.009 on test set after not touching my code for a week

Nico_Grassetto: <p>Hello,<br>
I trained a model a while ago now, I got about 0.84 on the test set while testing the same day I trained.<br>
After pausing my work for a few weeks I decided to get back to it and retested everything. Loaded my model and whatnot. eval mode, inference, blablabla and my accuracy seems to be around 0.009 (compared to 0.65 when I tested it)<br>
Is it the dataset splits? No, this has been verified and verified again by both my supervisor and myself.</p>
<p>Some help would be immensely appreciated!</p>
<p>I think the code is quite self-explanatory and compartmented so here’s the class I use for anything linked to training, testing etc.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1vSb36pmH_-iEuwrvSIxgy9bsA6BgMvzS?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/8c72f9237abda059c2d76387994229e9/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1vSb36pmH_-iEuwrvSIxgy9bsA6BgMvzS?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1vSb36pmH_-iEuwrvSIxgy9bsA6BgMvzS?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><strong>Context:</strong> The code trains multiple backbones on different modalities for action recognition in videos. Do not mind the code for consensus methods, that one isn’t the issue here.</p>
<p>Thanks for your support!</p>
<p>Cheers,</p>

ptrblck: <p>Usually these issues arise if something changed in the data loading or processing stage or if the wrong pre-trained parameters are loaded. You could thus make sure to load the proper <code>state_dict</code> (assuming you stored multiple ones) and also check the training accuracy to make sure this metric is at least looking expected. If that’s the case, check for differences between the training and test dataset processing steps.</p>

Nico_Grassetto: <p>I doubt anything changed in the data loading stage. I also made sure to use the same seed.</p>
<blockquote>
<p>You could thus make sure to load the proper <code>state_dict</code> (assuming you stored multiple ones)</p>
</blockquote>
<p>I did make sure that I wasn’t loading the wrong model and sadly I am not.</p>
<blockquote>
<p>check the training accuracy to make sure this metric is at least looking expected.</p>
</blockquote>
<p>I also checked the training accuracy. The training accuracy must be correct since it worked a few weeks ago and I have not changed anything in the code since then.</p>
<p>Thank you for your help anyway!</p>
RedisAI inference attempt

Robert_Grzelak: <p>Hello, I am getting redis.exceptions.ResponseError when trying to make inference on RedisAI using exported yolov5 torchscript  model, but got;</p>
<p>isTuple()INTERNAL ASSERT FAILED at “…/aten/src/ATen/core/ivalue_inl.h”:1916, <strong>please report a bug to PyTorch</strong>. Expected Tuple but got String Exception raised from toTupleRef at …/aten/src/ATen/core/ivalue_inl.h:1916 (most recent call first): frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x6b (0x7f02fbb747ab in /home/robb/Desktop/nn/RedisAI/deps/linux-x64-cpu/libtorch/lib/libc10.so) frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits,…</p>
<p>What I may missing?</p>
ONNX export with constant upscaling dimensions

tal123: <p>I am trying to export an FPN model with a constant input size, meaning that the feature maps have constant sizes and so are the target dimensions of the upscaled maps. To determine the upscaling target dimensions, I do use the shape of the “parent” feature map, but I don’t want the ONNX model to include that tensor-shape query part in its graph.</p>
<p>Whenever I use <code>opset_version&gt;9</code> when calling <code>onnx.export</code>, it always keeps track of the query of the tensor’s shape, which I want to avoid. I want to have constants for upscaling.</p>
<p>I tried to make the tracer lose track of the shape query using many hacks, but none of them worked. Only thing that worked was using <code>opset_verison=9</code>. Things I tried as hacks included converting to string and then back to int, converting to float and adding some constant &lt; 1 and then flooring it back to int, using a for loop on a range near the dimension and saving the iterator value… but the tracer is really really sticky, non of my hacks worked, it still includes the shape query in the ONNX graph…</p>
<p>Is there a way to make it use constant target dimensions for upscaling without using opset version 9?</p>
How to make input channel "Cin" to be the same size as the shape of the second index of input data without fixing?

ashishjv11: <p>Let’s say i have a data “x” with shape  [32, 3, 32, 32] and i split it into 3 parts channel-wise/second index, with shapes  [32,1,32,32] each. My current model has 3 depth-wise linear layers facing the input each of which takes a copy of the original data x . Now, I want each of the 3 depth-wise layers to take a slice of data without needing to fix the number of input channels. Let’s say self.cin1 = x1.shape[1], self.cin2 = x2.shape[2] and self.cin3=x3.shape[3] where x1,x2,x3 are the 3 data splits. I split the data in the forward funtion as :</p>
<p>def forward(self, x):<br>
x1, x2, x3 = data_splitter(x)<br>
l1 = self.linear(x1)<br>
l2 = self.linear(x2)<br>
l3 = self.linear(x3)<br>
…</p>
Is there any tutorial on attention, especially how to use cross_attention

Gorgen: <p>Is there any tutorial on attention, in particular how to use cross_attention?</p>
<p>My task is multimodal emotion recognition, I want to use cross-attention to capture some dynamic relevant-information about audio and text modality.</p>
<p>could you please give me some tutorials or suggestions?<br>
Thanks<br>
best wishes</p>

muammar: <p>Did you find anything?</p>
Distributions and generators on GPU (torch.normal, torch.linspace, etc)

amiasato: <p>Quick question related to this StackOverflow answer: <a href="https://stackoverflow.com/questions/59179609/how-to-make-a-pytorch-distribution-on-gpu/59179927#59179927" class="inline-onebox" rel="noopener nofollow ugc">python - How to make a PyTorch Distribution on GPU - Stack Overflow</a></p>
<p>The first question is: is it really needed to call torch.[rand,normal,linspace] with GPU tensor arguments as stated in the question? More directly, are these</p>
<pre><code class="lang-python">torch.normal(0, 1, shape, device="cuda")
torch.linspace(-n, n, shape, device="cuda")
</code></pre>
<p>And these:</p>
<pre><code class="lang-python">torch.normal(torch.Tensor(0, device="cuda"), torch.Tensor(0, device="cuda"), shape, device="cuda")
torch.linspace(torch.Tensor(-n, device="cuda"), torch.Tensor(n, device="cuda"), shape, device="cuda")
</code></pre>
<p>Any different in terms of the origin of the generated tensors (cpu vs gpu)? Is it dependent on each function?</p>
<p>Thanks,</p>

vahvero: <p>Did you ever find a general solution for calling <code>to()</code>  for all distributions?</p>
<p>This is still on going issue <a href="https://github.com/pytorch/pytorch/issues/7795" class="inline-onebox" rel="noopener nofollow ugc">[feature request] Add cudaification API for distributions · Issue #7795 · pytorch/pytorch · GitHub</a>.</p>
Calculate Loss for Specific classes only

aditya_dhaulakhandi: <p>I’m working on training a model where I have to calculate the loss only for specific classes, i.e. all ground truth masks (targets) doesn’t contain all the classes, so I want to calculate the loss for only those class that is present in the target. For this, I made a vector that holds the indices of the channels that are present in the target, and when I’m using the CrossEntropyLoss function, I’m passing my predictions and targets as:</p>
<pre><code class="lang-auto"> loss += loss_fn( pred[instance,class_present,:,:],targets[instance,class_present,:,:] ) 
</code></pre>
<p>here instance is the index in the batch and class_present is a vector containing the indices of the classes present in targets. My problem is, this is giving significantly very high training error (ex: 5000.5…). Is my approach correct? Can anyone help me to find the problem? Thanks!</p>
<p>Edit: Can I use the <code>weight</code> parameter of the <code>CrossEntropyLoss</code> function to specify the above?</p>
How to stream a folder of CSV data for multi-GPU training using DistributedSampler?

shengy90: <p>Hi <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"> I have a folder of a lot of CSV files which cannot be loaded into memory. How do I load this data into multi-GPU training?</p>
<p><strong>What I have done</strong></p>
<ol>
<li>At the moment, I’m loading this data into my training application using TorchData datapipes (<a href="https://pytorch.org/data/main/dp_tutorial.html?highlight=parse_csv" rel="noopener nofollow ugc">Example tutorial</a>).</li>
<li>As my full production dataset is too big to fit all into memory and I’d like to accelerate my training, I’d like to train it using multiple GPUs, which each GPU only reading a subset of the data.</li>
<li>To do that, I’m trying to use <a href="https://pytorch.org/docs/stable/data.html?highlight=distributedsampler#torch.utils.data.distributed.DistributedSampler" rel="noopener nofollow ugc">DistributedSampler</a> alongside with DistributedDataParallel strategy</li>
</ol>
<p>However, I’m getting the error " MapperIterDataPipe has no <em>len</em>" which of course it hasn’t because the full data size is not known in advanced. What’s the best strategy then to stream CSV records into Distributed GPU training in Pytorch?</p>
<p>In tensorflow, there’s a DistributedDataset and DistributedMirrorStrategy which takes care of everything. What’s the best practice for Pytorch?</p>

shengy90: <p>I also attempted to use Pytorch Lightning to do MultiGPU training, and then discovered that Pytorch Lightning doesn’t automatically distribute data across multiple GPU devices for IterableDatasets. Pytorch Lightning only automatically does so for Map Datasets.</p>
<p>Wondering what’s the best practice for production datasets (very large data, sharded across many CSV files) + MultiGPU training in Pytorch!</p>
Data loader with large ram variable

aswamy: <p>Hello,</p>
<p>I have a large(3GB) dictionary to load in init of the Dataset class. I want to use this dictionary in <strong>getitem</strong><br>
but I am afraid that loading such a large file in init method would slowdown my training.<br>
is there a better way to handle it?</p>
<p>Below is the sudo code of what I am trying to do…</p>
<pre><code class="lang-auto">class DataSetClass(Dataset):
    def __init__(self, dict_file_pth):
        self.large_dict = load_pkl(dict_file_pth)

    def __getitem__(self, index):
        data = self.large_dict[index] 
        return data
    
    def __len__(self,):
        return len(self.large_dict)
</code></pre>
Different implementation of square function

WeiQin_Chuah: <p>I am working on writing a custom convolution function that involves taking the square of some inputs (x). I am wondering what are the differences between different square implementations:</p>
<ol>
<li>torch.square(x)</li>
<li>torch.pow(x)</li>
<li>x**2</li>
<li>x*x</li>
</ol>
<p>I would like to know how PyTorch handles each of these operations and if there is any difference in term of gradient computation and efficiency.</p>
<p>Thank you</p>

ptrblck: <p>You can use any of these as they should all dispatch to the same kernel.<br>
A similar question was also asked <a href="https://discuss.pytorch.org/t/purpose-of-torch-neg-torch-square-and-torch-sqrt/182215/2">here</a>.</p>
Channel class mapping in semantic segmentaiton

aditya_dhaulakhandi: <p>I’m using the deeplab Mobilnet pre-trained model for semantic segmentation where I have 4 classes. The prediction of my model is [batch_size,num_classes, H, W] and to calculate the loss I have made my ground truth of the same dimension having 4 binary masks and using the CrossEntropyLoss function.I want to know which channel,the prediction corresponds to which class?, and also I’m getting a massive loss of about 5000 for 1 epoch during training. Can anyone help me? Thanks!</p>

ptrblck: <p>The channel index corresponds directly to the class index and you are defining it by creating the targets.<br>
I.e. the logits of <code>output[:, 0]</code> will correspond to <code>class0</code>, <code>output[:, 1]</code> to <code>class1</code>, etc.<br>
You can of course change the mapping by creating the target indices in the desired order.</p>

aditya_dhaulakhandi: <p>To make the model return 4 classes,.I have changed the last classifier layer from 22 to 4.<br>
For example, if my classess are background, person, car and licence plate, I took the rgb mask and converted it into 4 binary mask, one for each class. Here I made sure that all targets have the same order of channels i.e. first channel is background, then person then car and then license plate. This is what is meant by defining the order in the targets sir? And will the model return the output in the same order? Thanks!</p>

ptrblck: <p>Yes, you are defining the order of the targets and the model doesn’t have any knowledge which class corresponds to which index as it just tries to learn the mapping between the input and the target.<br>
If you define that e.g. the background class corresponds to <code>class0</code> (and thus also the 0-th channel of the model output) the model will try to learn it.</p>

aditya_dhaulakhandi: <p>Thanks for the help!</p>
Torchserve : java not found, please make sure JAVA_HOME is set properly

Hari_Krishnan: <p>I am trying to run torchserve for a BERT based token classfier on Mac M1 Pro. I am getting the following error when i start tochserve with the command.</p>
<p><code>torchserve --start --ncs --model-store model_store --models all</code></p>
<blockquote>
<p>java not found, please make sure JAVA_HOME is set properly.</p>
</blockquote>
<p><code>JAVA_HOME</code> is set and java is installed. These are the outputs for the following commands</p>
<ol>
<li><code>java -version</code></li>
</ol>
<blockquote>
<p>openjdk version “19.0.2” 2023-01-17<br>
OpenJDK Runtime Environment Homebrew (build 19.0.2)<br>
OpenJDK 64-Bit Server VM Homebrew (build 19.0.2, mixed mode, sharing)</p>
</blockquote>
<ol start="2">
<li><code>echo $JAVA_HOME</code></li>
</ol>
<blockquote>
<p>/opt/homebrew/opt/openjdk/bin</p>
</blockquote>
<p>How can I solve this issue ?</p>

marksaroufim: <p>Not an an M1 right now but does this solve your problem?</p>
<pre><code class="lang-auto">export JAVA_HOME=$(/usr/libexec/java_home)
</code></pre>

lsmith77: <p>I have been unable to persistenly update JAVA_HOME on OSX so I am using the following to start TorchServe:</p>
<p><code>export JAVA_HOME=$(/usr/libexec/java_home -v 17); torchserve --start ..</code></p>
<p>However, I am unable to get any models to work, likely because M1 isn’t supported.</p>
Run out of GPU memory when generating sequences using multi-head attention model in an autoregressive way

Jingge: <p>I am working on time series forecasting using GPT-like model. The idea is similar to training a language model that, given the beginning part of a sentence, the model can generate the rest of that sentence.</p>
<p>“I’m hungry, I want a hamburger and a cup of cola.”</p>
<p>Input: I’m hungry</p>
<p>Predict: I want a hamburger and a cup of cola.</p>
<p>An autoregressive language model will generate words step by step.</p>
<p>I’m hungry, I<br>
I’m hungry, I want<br>
I’m hungry, I want a<br>
……<br>
I’m hungry, I want a hamburger and a cup of cola.</p>
<p>That is, the newly generated word will be appended to the end of the previous input sequence to construct the new input sequence. During training, I will calculate the loss on the generated content “I want a hamburger and a cup of cola” and use back-propagation to update model parameters. The generation process can be implemented through a for-loop and a “decoder-only” module.</p>
<p>However, the GPU memory usage always spikes in this for-loop, and causes out-of-GPU-memoery error. If I set “<span class="mention">@torch.no_grad</span>()”, there will be no such problem. So I guess maybe the problem is caused by stored intermediate data for back-propagation.</p>
<p>Do you think my implementation is the right way to generate word sequences? Do you have any suggestions for optimizing my implementation?</p>
<p>My time series forecasting sequence contains around 100 elements, that is, the for-loop repeat operation 100 times.</p>

ptrblck: <p>The increase in memory is expected as you are storing all computation graphs including all intermediate tensors needed for the gradient computation.<br>
I don’t know if your auto-regressive approach is suitable for your problem, but you won’t be able to reduce the memory usage as the memory is indeed needed. Reducing the number of iterations, the model size, etc. might help but also changes your use case.</p>

vdw: <p>There is a PyTorch tutorial <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" rel="noopener nofollow ugc">“Language Translation with nn.Transformer and torchtext”</a>. As the decoder does essentially the same as for you, i.e., generating text, maybe this implementation might be worth look it. If you look at the <code>greedy_decode()</code> method, there’s the for loop generating the next word. I ran it locally and it works just fine. To be fair, the translations – and with that the number of iterations in <code>greedy_decode()</code> are – much shorter than 100 steps.</p>
Input to 1D cnn when input is skeleton data

lola1: <p>Hi,</p>
<p>as input data I am using 20 npz files. Each npz file contains 64 rows with 51 data points.</p>
<p>I want to implement 1D CNN, but I am getting confused what should in_channels be in my case:</p>
<pre><code class="lang-auto">layer1 = torch.nn.Conv1d(in_channels, out_channels = 32, kernel_size = 3)
</code></pre>
<p>Can someone give me tip here? Thanks!</p>

ptrblck: <p>The expected input to <code>nn.Conv1d</code> is a 3D tensor in the shape <code>[batch_size, in_channels, seq_len]</code>.<br>
Your loaded data should thus already define the <code>in_channels</code> size. If it’s not in the right format yet, you could <code>unsqueeze</code> the channel dimension and set <code>in_channels</code> to <code>1</code>.</p>
Slow aws S3 data loading using TorchData open_files_by_fsspec

knoriy: <p>Hi All,</p>
<p>I am currently utilizing AWS for storing a large dataset, which has been divided into tar files. While trying to scale up my training process, I found that the main bottleneck lies in the <code>fsspec</code> component. It takes approximately 5 seconds to load the data stored in the tar files.</p>
<p>If you have encountered a similar challenge and have any experience or insights to share, I would greatly appreciate your assistance.</p>
<p>Thank you for your time!</p>
<p>Setup:</p>
<pre><code class="lang-auto">torch==2.0
pytorch_lightning==2.0.2
torchdata==0.6
</code></pre>
<p>TorchData Datapipe.</p>
<pre><code class="lang-python">datapipe = torchdata.datapipes.iter.IterableWrapper(data_dir)\
			.shuffle()\
			.open_files_by_fsspec(mode='rb')\
			.load_from_tar() \
			.groupby(group_by_filename, group_size=2, guaranteed_group_size=2)\
			.sharding_filter()\
			.shuffle(buffer_size=self.batch_size)\
			.map(self.to_sampels) \
</code></pre>

shengy90: <p>I had the same issue with bottleneck when streaming data from a Google Cloud Storage bucket. It looks like if you’re already using tar files, you could use WebDataset to stream the data - might be faster than using IterableWrapper.</p>
<p>I found this blog by GCP, but I’ve never tried it as my data is in CSV format.</p>
Initialising weights in nn.sequential

arjun_pukale: <p>how to implement weight initializing techniques like xavier, He while using nn.sequential. for eg for the image given below<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba9a9def100944fb96e122217c637692a673fc2d.jpeg" alt="doubt" data-base62-sha1="qCM4Dk7G8JHL1B17hGoTgbAJRmd" width="594" height="489"></p>

ptrblck: <p>You could define a method to initialize the parameters for each layer via e.g.:</p>
<pre><code class="lang-python">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform_(m.weight)
        torch.nn.init.zero_(m.bias)

net.apply(weights_init)
</code></pre>
<p>Inside this method, you could add conditions for each layer and use the appropriate weight init method.</p>

arjun_pukale: <p>I am getting error please help!!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9ffac7ff2d06ba471c37cace2eea85ca88391b67.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9ffac7ff2d06ba471c37cace2eea85ca88391b67" title="doubt2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9ffac7ff2d06ba471c37cace2eea85ca88391b67_2_612x500.jpeg" alt="doubt2" data-base62-sha1="mPfdh5T0ING8u8A28FlGMoIBfaD" width="612" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9ffac7ff2d06ba471c37cace2eea85ca88391b67_2_612x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9ffac7ff2d06ba471c37cace2eea85ca88391b67_2_918x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9ffac7ff2d06ba471c37cace2eea85ca88391b67.jpeg 2x" data-dominant-color="2A2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">doubt2</span><span class="informations">978×799 60.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Sorry for the misleading code, but you cannot use <code>xavier_uniform_</code> on 1-dimensional tensors, which would be the case for the <code>bias</code> parameter in the linear layers.</p>
<p>I’ve edited the previous example.</p>

apeard: <p>How can this be done for a single layer? For example in my nn.Sequential if I want to initialise the weights of the first Conv2d to Xavier uniform and leave the other Conv2ds at their default values.</p>

ptrblck: <p>You could call the <code>.apply</code> method on this layer only. Here is a simple example:</p>
<pre><code class="lang-python">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform_(m.weight)
        if m.bias:
            torch.nn.init.zeros_(m.bias)

model = models.resnet50()
model.layer4[1].conv2.apply(weights_init)
</code></pre>
Changing the architecture of a model & using old pretrained weights

Amur_Saqib_Pal: <p>I have a pytorch model and it’s weights in a .pth format. I would like to remove two layers of the model, both from the architecture and the parameters so I can continue using the pretrained weights. I’ve changed the architecture and deleted the weights and biases of those two from the state dictionary, however upon trying to load it into the new architecture I get an error about the parameter group not being the same.</p>
<pre><code class="lang-auto">    Traceback (most recent call last):
  File "main.py", line 318, in &lt;module&gt;
    main()
  File "main.py", line 165, in main
    net.load_state_dict(new_state_dict)#, strict=False)
  File ".../python3.7/site-packages/torch/optim/optimizer.py", line 171, in load_state_dict
    raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group
</code></pre>
<p>This is what I did to the state dictionary to get rid of the parameters.</p>
<pre><code class="lang-auto">
del state_dict['classifier.0.weight']
del state_dict['classifier.0.bias']
del state_dict['classifier.1.weight']
del state_dict['classifier.1.bias']
del state_dict['classifier.1.running_mean']
del state_dict['classifier.1.running_var']
del state_dict['classifier.1.num_batches_tracked']
del state_dict['classifier.2.act_quant.fused_activation_quant_proxy.tensor_quant.scaling_impl.value']
del state_dict['classifier.4.weight']
del state_dict['classifier.4.bias']
del state_dict['classifier.5.weight']
del state_dict['classifier.5.bias']
del state_dict['classifier.5.running_mean']
del state_dict['classifier.5.running_var']
del state_dict['classifier.5.num_batches_tracked']
del state_dict['classifier.6.act_quant.fused_activation_quant_proxy.tensor_quant.scaling_impl.value']
</code></pre>
<p>However, the optimizer state dict has different, ambiguous key names that I cannot tell which ones need to removed.</p>
<pre><code class="lang-auto">optimizer_dict['state'].keys()
dict_keys([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126])
</code></pre>
<p>How can I fix this or go about it in a better way?</p>

JuanFMontesinos: <p>It seems you are using a framework that is also saving the optimizer’s state dict.<br>
The issue is not about model parameters, but optimizer group’s parameters. So basically for each layer in the net you have some optimizer-related paramters that you should remove as well. Or just modify your saving/loading funcs not to track this.</p>
<p>If you use another one like SGD you won’t have this problem.</p>
Libtorch CUDA predictions not working on .NET 6.0 debug version

DejNea: <p>I have some strange problem. I have created C++ DLL with LibTorch. It has two functions. One for loading model and one for predicting. It’s used for segmentation with U-net. C++ code is working fine but the problem occurs when I want to use it in C# with CUDA.</p>
<p>I tried to use it in C# .NET framework 4.7.2 and it works fine in both release and debug version. When I tried to use it in .NET 6.0 for some reason release version works, but debug doesn’t. Program crashes when it is trying to predict.</p>
Automatically install the latest PyTorch version?

RubixCube: <p>Hi all,</p>
<p>I noticed that the pip installation of PyTorch depends on the supported CUDA version, e.g. <code>https://download.pytorch.org/whl/cu117</code> for 11.7 or <code>https://download.pytorch.org/whl/cu118</code> 11.8, respectively.<br>
I am looking for a <em>give me the most recent CUDA version supported</em> version that in particular updates the used CUDA version by automatically resolving to the latest version available. Is there such an option or, alternatively, how can this latest supported CUDA version be determined by <code>curl</code> or something similar?<br>
I would like to use this as part of an installation / update script to not only update the pip packages but also PyTorch’s CUDA version to the latest supported one.</p>
<p>Thanks in advance!</p>
Multiple sampling of data

STR720: <p>I have a small question. I am using</p>
<pre><code class="lang-auto">torch.utils.data.WeightedRandomSampler
</code></pre>
<p>to balance the classes between cats and dogs.<br>
But before doing that, I want to balance between different breeds of dogs first, then I want to apply  <code>torch.utils.data.WeightedRandomSampler</code> to balance cats and dogs.</p>
<p>Can anyone help me with How to do it?</p>
Auto grad for a matrix with only a parameter

YourLili: <p>I want to get the gradient of a parameter. For example:<br>
parameter: <code>t</code><br>
<code>A = [[cos(t), -sin(t)], [sin(t), cos(t)]]</code><br>
<code>x = [[1], [10]]</code></p>
<p><code>y = A x</code><br>
<code>z = ||y||^2</code></p>
<p>I want to know how to code to get the gradient of <code>theta</code> where loss function is <code>z</code>.</p>
<pre><code class="lang-auto">theta = torch.tensor(1.0, dtype=torch.float32, requires_grad=True)
A = torch.Tensor([
    [torch.sin(theta), torch.cos(theta)],
    [torch.tan(theta), torch.exp(theta)]
])

x = torch.tensor([[1], [10]], dtype=torch.float32)
y = torch.matmul(A, x)
z = torch.matmul(y.T, y)
</code></pre>
<p>But I found <code>A.requires_grad</code> is <code>False</code>, and the same to <code>z</code>.</p>
'NoneType' object has no attribute 'size'

pytorchcv: <p>idx = 17<br>
model.load_state_dict(torch.load(‘/content/best_model.pt’))<br>
image, mask = validset[idx]</p>
<p>print(image.shape)</p>
<p>logit_mask = model(image.to(DEVICE).unsqueeze(0))<br>
print(logit_mask.shape)</p>
<p>pred_mask = torch.sigmoid(logit_mask)<br>
print(pred_mask.shape)</p>
<h2>
<a name="pred_mask-pred_mask-0510-1" class="anchor" href="#pred_mask-pred_mask-0510-1"></a>pred_mask = (pred_mask &gt; 0.5)*1.0</h2>
<h2>
<a name="error-torchsize3-512-512-2" class="anchor" href="#error-torchsize3-512-512-2"></a>ERROR —&gt;&gt;&gt;<br>
torch.Size([3, 512, 512])</h2>
<p>AttributeError                            Traceback (most recent call last)<br>
 in &lt;cell line: 14&gt;()<br>
12 <span class="hashtag-raw">#logit_mask</span> = torch.unsqueeze(image, dim=0)<br>
13 <span class="hashtag-raw">#logit_mask</span> = image.to(DEVICE).unsqueeze(0)<br>
—&gt; 14 logit_mask = model(image.to(DEVICE).unsqueeze(0))<br>
15 print(logit_mask.shape)<br>
16</p>
<p>3 frames<br>
/usr/local/lib/python3.10/dist-packages/segmentation_models_pytorch/losses/dice.py in forward(self, y_pred, y_true)<br>
57     def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor) → torch.Tensor:<br>
58<br>
—&gt; 59         assert y_true.size(0) == y_pred.size(0)<br>
60<br>
61         if self.from_logits:</p>
<p>AttributeError: ‘NoneType’ object has no attribute ‘size’</p>
<p>Can Someone help me out… ???</p>

ptrblck: <p>Check the <code>forward</code> method of your model as it seems either <code>y_true</code> or <code>y_pred</code> is set to <code>None</code> (or both).</p>

pytorchcv: <p>def forward(self, images, masks = None ):<br>
logits = self.backbone(images)</p>
<pre><code>if mask != None:  # then return logits and also calculate Loss
  return logits, DiceLoss(mode = 'binary')(logits, masks) + nn.BCEWithLogitsLoss()(logits, masks)
</code></pre>
<p>This is runs fine with from where i am learning… but in my Google colab giving above error as i mentioned in my post…<br>
Thanks for your reply…</p>

ptrblck: <p>Add print statements to narrow down which line of code exactly raises the error. As can be seen in the stacktrace the <code>DiceLoss</code> expects two valid tensor as its input while the <code>masks</code> seem to use the default <code>None</code> value.</p>

pytorchcv: <p>if mask != None:  ----&gt;&gt;&gt;&gt; Typo error basically,… I have written mask but its actually ‘masks’<br>
Only for this typo got that error… its pathetic…</p>
<p>Thank for your response…</p>

ptrblck: <p>Good catch as I didn’t see it!</p>

pytorchcv: <p>Thank you so much for your concern…</p>
Can I nest nn.ModuleDict()?

EmmaRocheteau: <p>I want to organise my submodules into groups of modules. Within the groups the modules aren’t necessarily applied sequentially but the groups are then applied one after another. Can I nest a ModuleDict so that I can index a group, which itself is a ModuleDict?</p>
<p>Also, can I add non-module attributes to a ModuleDict? Like a normal string? Or would I need a separate dictionary to track non-module items?</p>

albanD: <p>Hi,</p>
<p>You can nest <code>ModuleDict</code>, they are nn.Module and can contain any nn.Module.<br>
I think you can store anything in a Module dict, I can’t see any restriction.</p>

NagabhushanSN95: <p>I am not able to add non-module elements to ModuleDict. Getting the following error:</p>
<p><code>TypeError: str is not a Module subclass</code></p>
A set of data, sum() as the dividend, how to find grad

windlinekk: <p>Hi,guys,<br>
A set of data, sum() as the dividend, how to find grad.code like this:</p>
<pre><code class="lang-auto">x = torch.tensor([1.,2.,3.],requires_grad=True)
sum = x.sum()
y = x**2/sum
z = y[1]
z.sum().backward()
print(x.grad)

tensor([-0.1111,  0.5556, -0.1111])

x1 = torch.tensor([1.,2.,3.],requires_grad=True)
sum1 = x1.sum()
y1 = x1**2/sum1
z1 = y1
z1.sum().backward()
print(x1.grad)

tensor([-0.0556,  0.2778,  0.6111])

</code></pre>
<p>Any answer or idea will be appreciated!</p>

windlinekk: <p>I know how to get the first code array[1] grad,is 0.5556<br>
it is sum = 6 sum’ = 1<br>
so<br>
uv/w = (u’w-uw’)/(w * * 2) =( 2 * 2 * 6-2 * * 2)/(6 * 6) = 0.5556<br>
but others no index i don’t know</p>

InnovArul: <p>I tried to answer in a gist, as it involves math expressions.<br>
Hopefully, it helps.</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/InnovArul/14af4577b6d8cd19efd6a5b275cfc189">
  <header class="source">

      <a href="https://gist.github.com/InnovArul/14af4577b6d8cd19efd6a5b275cfc189" target="_blank" rel="noopener">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/InnovArul/14af4577b6d8cd19efd6a5b275cfc189" target="_blank" rel="noopener">https://gist.github.com/InnovArul/14af4577b6d8cd19efd6a5b275cfc189</a></h4>

  <h5>math.md</h5>
  <pre><code class="Markdown">Answering https://discuss.pytorch.org/t/a-set-of-data-sum-as-the-dividend-how-to-find-grad/182303:

We can find $grad_x = \frac{dz}{dx}$ in the following way:

$$grad_x = \frac{dz}{dx} = \left[\frac{dz}{dx_0},\frac{dz}{dx_1}, \frac{dz}{dx_2}\right]$$

$$   = \frac{dz}{dx} = \left[\frac{dy}{dx_0}\frac{dz}{dy},\frac{dy}{dx_1}\frac{dz}{dy},\frac{dy}{dx_2}\frac{dz}{dy}\right]$$

Now let's concentrate on finding one element $\frac{dz}{dx_0} = \frac{dz}{dy}\frac{dy}{dx_0}$
</code></pre>
    This file has been truncated. <a href="https://gist.github.com/InnovArul/14af4577b6d8cd19efd6a5b275cfc189" target="_blank" rel="noopener">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


windlinekk: <p>I see, thank you very much</p>