akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch: pin_memory=True Not Transferring Data to GPU

ekrx: <p>I’m encountering an issue where even though I have pin_memory=True set in my DataLoader, the data remains on the CPU during training. My CUDA GPU is available (torch.cuda.is_available() returns True).</p>
<pre><code class="lang-auto">train = MyDataSet() # Instance of an IterableDataset
train_loader = DataLoader(dataset=train, batch_size=batch_size, prefetch_factor=10, num_workers=4, pin_memory=True)
for batch_index, (x_batch, y_batch) in enumerate(train_loader):
    print(x_batch.device)  # Shows "cpu"
    print(torch.cuda.is_available()) # Shows 'True
    output = model(x_batch)
    # ... training code ...
</code></pre>
<p>This results in a RuntimeError stating that input and parameter tensors are on different devices (CPU vs. GPU).</p>
<p>I want to avoid having to explicitly move the data to the GPU because its causing a bottleneck.</p>

ptrblck: <p>Using <code>pin_memory=True</code> will use page-locked host memory and will not move the data behind your back. You still need to move the data to the device.</p>

ekrx: <p>Is there a way to prevent this being the bottleneck?  What are the best practices for this?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6bb010aa34728e203476850c431972a6f744f660.png" alt="image" data-base62-sha1="fmEj2n7wEwMgoIPWGjK0Csssrp6" width="497" height="285"></p>

ekrx: <p>I’m thinking I could create a new process that would be responsible for putting the data into the GPU and then push those references to a mp.Queue. Then the training loop, instead of using a for loop on top of the Dataloader, consume the references from the Queue. This way I could overlap data transfers and fully utilize my GPU.</p>
<p>I have not tried this, but I’m wondering if this would be a good approach, or there is another recommended way to overlap data transfers to GPU. Also wonder if once moved data to the GPU, can be put in a Queue.</p>

ptrblck: <aside class="quote no-group" data-username="ekrx" data-post="4" data-topic="203254">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ekrx/48/68501_2.png" class="avatar"> ekrx:</div>
<blockquote>
<p>Then the training loop, instead of using a for loop on top of the Dataloader, consume the references from the Queue.</p>
</blockquote>
</aside>
<p>Using <code>num_workers&gt;0</code> will already use multiple processes in the background to create batches of data. Pinning the memory allows you to move the data asynchronously to the device w.r.t. the host. However, since you don’t have any overlapping work to do, the benefits might be insignificant.</p>
Weird Behavior of F.scaled_dot_product_attention

ahhhh: <p>import torch<br>
import torch.nn.functional as F</p>
<p>k  = torch.randn(2, 4, 4, 20).cuda()<br>
mask = torch.ones(2, 4, 1, 1).cuda()<br>
result = F.scaled_dot_product_attention(k, k, k, mask)</p>
<p>I run the above code got: *** RuntimeError: (*bias): last dimension must be contiguous</p>
<p>Is there any restrictions on the shape of the tensors?</p>

konan009: <p>Hi ! the last axis of your <em>mask</em> tensor should <strong>always</strong> match the <em>3rd axis</em> or <em>2nd from the last axis</em> of your <em>k</em> tensor.</p>
<blockquote>
<p>torch.manual_seed(0)<br>
k = torch.randn(2, 4, <strong>4</strong>, 20).cuda()<br>
mask = torch.ones(2, 4, 1, <strong>4</strong>).cuda()<br>
result = F.scaled_dot_product_attention(k, k, k, mask)</p>
</blockquote>
<p>There is no shape restriction, you just need at least to have the same dimensions of tensors for your key, query and values that you pass on your function. You can refer to this <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">link</a> in terms of the implementation of the <em>F.scaled_dot_product_attention</em>.</p>
How to downsampling in a GAN model using Pytorch

Mohammed_Alruqimi: <p>I am building a conditional GAN for time series forecasting. My dataset consists of multivariate time series data, including historical gold prices and historical USD prices as features. My goal is to forecast the next day’s gold prices.</p>
<p>The forward function in the generator takes the following inputs:</p>
<pre><code class="lang-auto">X[16,10,2] -&gt; [batch_size, time_steps, features_no(gold, USD)]` `nosie[16,32] -&gt; [batch_size, noise_size]` While the out put should be ` y[16,1].
</code></pre>
<p>I am unsure about the correct way to concatenate the noise and the input 𝑋 before passing the output to an LSTM unit.</p>
<p>Here is my attempts:</p>
<pre><code class="lang-auto">class Generator_LSTM_LEVY(nn.Module):
        def __init__(self, hidden_dim, feature_no, time_steps):
            super().__init__()

           # LSTM Unit
           self.lstm = nn.LSTM(self.input_dim + noise_size, self.hidden_dim,self.layer_dim, 
           batch_first=True,bidirectional=True)

       def forward(self, x, noise, batch_size): # x = [64, 10, 2]

           noise_exp = noise.unsqueeze(1).expand(-1, self.time_steps, self.output_dim)
           x_n = torch.cat((x, noise_exp), dim=1)  

           out, (hn, cn) = self.lstm(x_n, (h0, c0))

           # Reshaping the outputs in the shape of (batch_size, seq_length, hidden_size)
           # so that it can fit into the fully connected layer
           out = out[:, -1, :]
           #.....
           return out
</code></pre>
<p>I got error message that</p>
<pre><code class="lang-auto">RuntimeError: The expanded size of the tensor (1) must match the existing size (32) at 
non-singleton dimension 2.  Target sizes: [-1, 10, 1].  Tensor sizes: [16, 1, 32]
</code></pre>
<p>Additionaly, I’m not sure if this a meaningful way to do so or not.</p>
<p>Can you please help? any insights will be highly appreciated.</p>
The gradient from loss.backward() is different from the gradient computed manually

3yuan: <p>In order to find the reason why nan or inf produced in the training, I compute the gradident manually. But I found that the gradient from loss.backward() is different from the gradient computed manually.</p>
<p>I want to know what happen in backward() when training on multi gpu nodes.<br>
I did the following experiment.<br>
Steps:</p>
<ol>
<li>select layer: stem conv</li>
<li>hooks: Hook the input, output, weight of the layer using register_forward_hook, use retain_grad() to hook the output grad</li>
<li>compute the weight grad manually on each node</li>
</ol>
<pre><code class="lang-auto">
def compute_weight_grad(self, input_tensor, grad_output, layer_weight):
    kernel_size = 3
    stride = 2
    padding = 1
    fold_params = dict(kernel_size=kernel_size, stride=stride, padding=padding)
    unfold = torch.nn.Unfold(**fold_params)
    unfold_input = unfold(input_tensor)
    grad_output = grad_output.flatten(2)
    grad_weight = grad_output @ unfold_input.transpose(1, 2)
    grad_weight = grad_weight.sum(0)
    grad_weight = grad_weight.view_as(layer_weight)
</code></pre>
<ol start="4">
<li>scale and average the gradients. Becuase of the amp training, the loss is scaled by 256, so here scale back the gradient, and then average the gradients from multi nodes.</li>
<li>Questions<br>
The manually computed weight grad is different with the grad from model.parameters<br>
Define:</li>
</ol>
<pre><code class="lang-auto">ratio = manually_grad / grad_from_parameter
</code></pre>
<p>I find that for each iteration, the ratio is const, and different iterations have different ratios<br>
I want to know what happened? Thanks</p>

3yuan: <p>clip_grad_norm is used</p>
Loading model weights layer by layer

Aniruth_Sundararaja1: <p>I have a model definition , and i have its pre trained weights.<br>
I have now done modifications to the original model , by adding some new layers.<br>
now , i want to load the pre trained weights into the new model such that , the original layers , are loaded with their corresponding weights, while the new layers are randomly initialized.</p>
<p>making strict  = False , is also not giving the correct weights.<br>
can someone help me to manually initialize the weights accordingly.</p>

JuanFMontesinos: <p>Just get the state dict of your new model. It’s a dictionary in the end. Replace the layers that you need with the stored ones. Then just load the state dict normally.</p>

Aniruth_Sundararaja1: <p>thank you , its working now for me .</p>
How to fix all random of training?

Ardeal: <p>Hi,</p>
<p>I am training resnet18 with pytorch. I would like to do the following experiments:<br>
step 1: trian resnet18.<br>
step 2: train resnet18 again.<br>
how to remove all random and use fixed in my code?</p>

techkang: <p>You can set the same random seed for both training process.</p>
<p><a href="https://pytorch.org/docs/stable/notes/randomness.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/randomness.html</a></p>

Ardeal: <p><a class="mention" href="/u/techkang">@techkang</a> ,</p>
<p>in my code, I do:<br>
`import random<br>
random.seed(5)</p>
<pre><code>torch.random.initial_seed()  
torch.cuda.manual_seed_all(5)

torch.cuda.manual_seed_all(5)
torch.backends.cudnn.deterministic=True
random.seed(5)
torch.random.initial_seed()  
torch.cuda.manual_seed_all(5)`
</code></pre>
<p>there is still something randomly changes sometimes. did I missed anything else?</p>
<p>or should I do the upper code in all related files? or I only need to do the upper code in the beginning of my project code?</p>

DOUBLE: <p>Hi, Ardeal. Have you solved the problem？I also have the same problem.Could you give me some advice?</p>

Ardeal: <p><a class="mention" href="/u/double">@DOUBLE</a><br>
I referred to this page, and successfully fixed all parameters in NN:<br>
<a href="https://pytorch.org/docs/stable/notes/randomness.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/randomness.html</a></p>
Help Getting Backprop Memory to Deallocate?

Jonezia: <p>Hello.</p>
<p>I have a scenario where I am running multiple runs where I create a new model, train it for a number of epochs in a loop, and then test the trained model after all these epochs. I am running multiple runs to gather averaged statistics. I would like to deallocate all unnecessary memory between each run to ensure that the timing is “fair”. I have managed to deallocate the memory (except the memory marked pretraining memory) between runs but using the memory snapshot there are two memory allocations which are persisting across runs. Please see the image.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69500077d0f707cf5cd1970148a1f271eae47e3e.png" data-download-href="https://discuss.pytorch.org/uploads/default/69500077d0f707cf5cd1970148a1f271eae47e3e" title="mem_with_annotations"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69500077d0f707cf5cd1970148a1f271eae47e3e_2_690x235.png" alt="mem_with_annotations" data-base62-sha1="f1DxkeWrx1GM1vR7hCDkd9hDIDA" width="690" height="235" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69500077d0f707cf5cd1970148a1f271eae47e3e_2_690x235.png, https://discuss.pytorch.org/uploads/default/original/3X/6/9/69500077d0f707cf5cd1970148a1f271eae47e3e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/9/69500077d0f707cf5cd1970148a1f271eae47e3e.png 2x" data-dominant-color="AFBDC1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mem_with_annotations</span><span class="informations">1001×342 74.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>One of these seems to be memory allocated for backprop. When I comment out loss_train.backward() it isn’t allocated. Another seems to be memory allocated for the forward pass of the model as when I replace output = model.forward() with a random tensor it isn’t allocated.</p>
<p>To try and deallocate the memory I have tried</p>
<pre><code>optimizer.zero_grad(set_to_none=True)
model.zero_grad(set_to_none=True)
torch.cuda.synchronize()
for param in model.parameters():
    if param.grad is not None:
        del param.grad  # Delete gradients
del model
del optimizer
gc.collect()
torch.cuda.empty_cache()
</code></pre>
<p>Using del + clearing cache has managed to free most memory between runs except these two blocks. Can anyone help with a way to force this memory to be deallocated between runs? I can post the memory snapshot information for the blocks if this is helpful. Thanks very much.</p>

ptrblck: <p>Based on the screenshot it seems the allocations in questions have an approx. size of ~15MB. If so, you could try to clear the cuBLAS workspace via <code>torch._C._cuda_clearCublasWorkspaces()</code> and check if this memory is related to it.</p>

Jonezia: <p>That worked perfectly thank you so much!</p>
Correct way to serialize aten calls?

samkg: <p>Hello,</p>
<p>I’m trying to do something a bit weird: I’m trying to distribute functions in a graphmodule graph over a cluster (e.g. through some compile analysis passes + replacing the node with logic that serializes inputs, runs the function on a worker, and returns/deserializes the result). This means I need a way to serialize the torch OpOverload object to send to the worker node on the cluster.</p>
<p>Is there a way to cleanly do this? I’m hoping in particular that there’s some insights from torch’s RPC stack, but I don’t have a deep enough understanding of how it is built to know otoh.</p>
<p>Greatly appreciate your help!</p>
Performing adaptations on MONAI "2d stable diffusion v2 super resolution"

mamil: <p>Before delving into the proper questions I would like to apologize if the questions are simple. All my previous deep learning works were made based on tensorflow so I am quite a novice at using pytorch.</p>
<p>While working on my master’s thesis, my coordinators recomended me to try using some of the examples made available by <a class="mention" href="/u/monai">@MONAI</a> as a starting point. My objective is to create a network that receives as input noisy medical images, with the present artifacts being a result of undersampling and movement mainly, and it then outputs the clear images. My coordinators were also very interested in me using a diffusion network to achieve this. With all of this in mind, I tried to adapt “2d_stable_diffusion_v2_super_resolution” network made available by the MONAI team.<br>
However, I have come to face some issues that I am not being able to solve. If anyone would be able to help me I would be very thankfull.</p>
<ol>
<li>In the original example, the noisy images are created as a transformation of the ground truth images. For me this is not the case, since I have both noisy and ground truth images saved in different directories. I was importing the data as follows:</li>
</ol>
<p>directory_und = os.fsencode(directory_tr_und)<br>
directory_gt = os.fsencode(directory_tr_gt)<br>
train_datalist =<span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>for file in os.listdir(directory_gt):<br>
filename_gt =‘/home/mamil/monai/GenerativeModels/data/multicoil_train_proc_128/’ + os.fsdecode(file)<br>
filename_und =‘/home/mamil/monai/GenerativeModels/data/multicoil_train_proc_und_128/’ + os.fsdecode(file)<br>
train_datalist+= [{“image”:filename_gt } , {“low_res_image”:filename_und }]</p>
<p>However, when using CacheDataset I would get an error. How could I perform the same transformations as the example, mainly the normalization of the figures, while having already both noisy and ground truth images imported, instead of creating the noisy images as a transformation of the ground truth images?</p>
<ol start="2">
<li>The diffusion model that is trained is created with 4 input channels and 3 output channels. However, I do not understand why this is the case. I was thinking about not using the latent representation created by the autoencoder but instead just directly give the ground truth image as conditional information to the diffusion model. However, since I do not understand why there are 4 input channels, I am not sure what additional information I should give to the model.</li>
</ol>
<p>Thanks to anyone that took their time to read this post.</p>

ptrblck: <p>Could you post the error for 1 as I cannot find the mentioned model in the <a href="https://github.com/search?q=repo%3AProject-MONAI%2FMONAI+2d_stable_diffusion_v2_super_resolution&amp;type=code">repository</a> (or could you post a link to the MONAI implementation)?</p>
AttributeError: module 'torch' has no attribute 'unravel_index'

songsong0425: <p>Hi, I tried to use <code>torch.unravel_index</code> to find the original indices from the flattened tensor, but there was an error as below:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[61], line 1
----&gt; 1 torch.unravel_index

File /scratch/r902a02/.conda/envs/PyG_scratch1/lib/python3.9/site-packages/torch/__init__.py:1833, in __getattr__(name)
   1830     import importlib
   1831     return importlib.import_module(f".{name}", __name__)
-&gt; 1833 raise AttributeError(f"module '{__name__}' has no attribute '{name}'")

AttributeError: module 'torch' has no attribute 'unravel_index'
</code></pre>
<p>But I found that there was a documentation about it in <a href="https://pytorch.org/docs/stable/generated/torch.unravel_index.html" rel="noopener nofollow ugc">here</a>.<br>
How can I use <code>torch.unravel_index</code>?</p>

ptrblck: <p>It works for me:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.unravel_index
Out[40]: &lt;function torch.functional.unravel_index(indices: torch.Tensor, shape: Union[int, Sequence[int], torch.Size]) -&gt; Tuple[torch.Tensor, ...]&gt;
</code></pre>
<p>using <code>torch==2.4.0.dev20240506+cu124</code>.</p>
Self = reduction.pickle.load(from_parent) _pickle.UnpicklingError: pickle data was truncated

Md_Minhazur_Rahman: <p>This issue is happening during inference.</p>
<ul>
<li>I have set num_worker=3 in dataloader.</li>
<li>Using distributed data parallel.</li>
<li>My batch size is 256.</li>
<li>Each image is 256x256</li>
<li>Using A100, 8 gpus,</li>
<li>each gpu has 40gb memory.</li>
<li>total dataset size is=20gb</li>
<li>System: Ubuntu</li>
</ul>
<p>Still I am getting this error:<br>
Exception has occurred: UnpicklingError       (note: full exception trace is shown but execution is paused at: )<br>
pickle data was truncated<br>
File “”, line 1, in  (Current frame)<br>
_pickle.UnpicklingError: pickle data was truncated</p>
<p>Sometimes it looks like this:<br>
self = reduction.pickle.load(from_parent)<br>
_pickle.UnpicklingError: pickle data was truncated</p>
nn.Linear weight shape initialization confusion

dmplot: <p>I have an input tensor with the shape of (32, 5). To create a <code>nn.Linear</code> layer from the input to the hidden layer (lets say we want our hidden layer to have 64 neurons), we would use <code>linear1 = nn.Linear(5, 64)</code>.  However, when I print <code>linear1.weight.shape</code>, it prints <code>torch.Size([64, 5])</code>. This means that during forward propagation, does <code>Z = matmul(X, W.T)</code>.</p>
<p>My question is, why do they initialize the weight with the shape <code>(output_nodes, input_nodes)</code>, rather than doing <code>(input_nodes, output_nodes)</code>, which removes the need for transposing? If I decide to make a neural network using just NumPy, would initializing the weights with a shape <code>(input_nodes, output_nodes)</code> and doing <code>Z = matmul(X, W)</code> be incorrect?</p>

KFrank: <p>Hi Daniel!</p>
<aside class="quote no-group" data-username="dmplot" data-post="1" data-topic="202891" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/f07891/48.png" class="avatar"> dmplot:</div>
<blockquote>
<p>This means that during forward propagation, does <code>Z = matmul(X, W.T)</code>.</p>
<p>My question is, why do they initialize the weight with the shape <code>(output_nodes, input_nodes)</code>, rather than doing <code>(input_nodes, output_nodes)</code>, which removes the need for transposing?</p>
</blockquote>
</aside>
<p>I speculate as follows:</p>
<p>First, we wish to multiply the input (your <code>X</code>) from the right by the weight (your <code>W</code>)<br>
because <code>X</code> may have a leading batch dimension (or multiple leading “batch”<br>
dimensions).</p>
<p>Second, when performing matrix multiplication from the right (not using the<br>
transpose), you stride through <code>W</code> with a stride of <code>out_features</code> (<code>64</code>, in your<br>
example).  So you are moving through <code>W</code> non-locally.  As a general rule of thumb,<br>
this will be less efficient than striding locally through <code>W</code> with a stride of one.  To<br>
gain this efficiency, pytorch stores the <code>weight</code> matrix of a <code>Linear</code> in “transposed”<br>
form (shape <code>[64, 5]</code>, in your example).</p>
<p>How much (or whether) this helps depends on the details of your cpu or gpu<br>
pipeline and the specific matrix-multiplication kernel used for the operation.<br>
But, as a general rule, this optimization should help, sometimes significantly.</p>
<p>(Just to be clear, the transpose, <code>W.T</code>, is never explicitly computed nor stored.<br>
<code>W.T</code> would be a view into <code>W</code> that then lets the matrix multiplication use a stride<br>
of one.)</p>
<p>Best.</p>
<p>K. Frank</p>
How to remove the grad_fn=<SelectBackward> in output array

Mriganka_Nath: <p>I am trying to put all my model’s output in an array, though I  am using it in eval mode,<br>
computing the output <code>with torch.no_grad():</code>  and detaching it the array is showing<br>
<code>grad_fn=&lt;SelectBackward&gt;</code><br>
My code</p>
<pre><code class="lang-auto">m.eval() # m is my model
for vec,ind in loaderx:

    with torch.no_grad():
        opp,_,_ = m(vec)
        opp = opp.detach().cpu()
    for i in range(ind.shape[0]):
        ar[i] = torch.cat((torch.tensor([ind[i]]),opp[i]),dim =0)
    break
</code></pre>
<p>when I try to output the array where my outputs are<br>
<code>ar[0][0] #shown only one element since its a big array </code><br>
output → <br>
<code>tensor(3239., grad_fn=&lt;SelectBackward&gt;)</code></p>

albanD: <p>Hi,</p>
<p>The detach() in the no_grad block is not needed.<br>
You will need to move all the ops into the no_grad block though to make sure no gradient is tracked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Mriganka_Nath: <p>Hi!<br>
I did it like this, it worked. Thanks</p>
<pre><code class="lang-auto">m.eval()
for vec,ind in loaderx:
    with torch.no_grad():
        opp,_,_ = m(vec)    
    for i in range(ind.shape[0]):
        ans = torch.cat((torch.tensor([ind[i]]),opp[i]),dim =0)
        ar[ind[i]] = ans
</code></pre>
<p>But can you solve another query, detach() should have been cut it in the computational graph then why it didn’t work?</p>

albanD: <p>At the place where you detach() is, there is no computational graph because you are running in no_grad mode. So it is just a no-op where it is.</p>

zeh_gobbes: <p>Hi!</p>
<p>My tensor x had a grad_fn, so I copied data of tensor x in a new tensor x1:</p>
<p>x1 = x.data</p>
<p>So, the new tensor x1 staied without grad_fn.</p>
Create a face verification app

Ahmed_Rashad: <p>Hello, I need to create  a face verification app that takes as an input the person’s images at different angles, then compare these images with a saved biometric versions. When searching to solve this problem, i always get stucked with a query search on face reconginition instead. I am new to the face verification task. Can any one helps me to slove this issue by suggesting  a pretrained model to  try which achevies a high accuracy? thanks a lot in advance.</p>
[WinError 126] The specified module could not be found

yuciin: <p>Can someone help me?<br>
I am trying to use torch in my project but it is not working.<br>
To test the torch I have tried following simple code.<br>
import torch<br>
x = torch.rand(5, 3)<br>
print(x)</p>
<p>and I have got following error. “shm.dll” is in the right directory and the directory is included in the system path.</p>
<p>Error:<br>
OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users\BERAY\AppData\Local.…\site-packages\torch\lib\shm.dll” or one of its dependencies.</p>

ptrblck: <p>Seem to be related to <a href="https://discuss.pytorch.org/t/error-loading-lib-site-packages-torch-lib-shm-dll-or-one-of-its-dependencie/201695/9">this issue</a>. You could check the corresponding GitHub issue suggesting workarounds or you could update to the latest nightly release where the issue was already fixed.</p>

yuciin: <p>It is working now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Thank you</p>

chenming7777: <p>help what did you do teach me mine is not working</p>
Importing RRef, rpc_async, remote from RPC

CollinsJnr: <p>Hey guys</p>
<p>I just started using RPC yesterday for SEED_RL and I have a problem importing RRef, rpc_async, remote from torch.distributed.rpc</p>
<p>if I run<br>
from torch.distributed.rpc import RRef, rpc_async, remote</p>
<p>It keeps saying, "cannot import name ‘RRef’ or ‘rpc_async’ or ‘remote’ from ‘torch.distributed.rpc’</p>
<p>I tried it on Windows and also WSL (Ubuntu)</p>

CollinsJnr: <p>Hey<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Can you make some suggestions, please?</p>
<p>Thank you</p>

ptrblck: <p>I cannot reproduce the issue on Linux:</p>
<pre data-code-wrap="python"><code class="lang-python">from torch.distributed.rpc import RRef, rpc_async, remote
RRef
# torch.distributed.rpc.api.RRef
rpc_async
# &lt;function torch.distributed.rpc.api.rpc_async(to, func, args=None, kwargs=None, timeout=-1.0)&gt;
remote
# &lt;function torch.distributed.rpc.api.remote(to, func, args=None, kwargs=None, timeout=-1.0)&gt;
</code></pre>
<p>using <code>2.4.0.dev20240423+cu121</code>.</p>

Mathieu-van-Luijken: <p>Hello<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> I am also having a similar issue for Windows 11 with python 3.8 and pytorch 2.3.0 where I am unable to import any of the three imports mentioned. When testing I also find rpc.is_available() to be False but cannot seem to find out why.<br>
Can you make some suggestions as to what I can do to solve this</p>
<p>Thank you</p>

ptrblck: <p>I’m not familiar with the Windows support and don’t know if RPC is implemented using Gloo, but <a class="mention" href="/u/kwen2501">@kwen2501</a> might know.</p>
The correct way to have split input to a model

SystemLord: <p>I am using a DataLoader and then</p>
<p><code>for input_batch, input_labels in data_loader:</code></p>
<p>as is torch-onic.</p>
<p>However, my model needs to use “split input”, ie it’s forward function needs to do something like this:</p>
<pre><code class="lang-auto">def forward(x):
   resnet_input, fc_input = x

   resnet_out = self.resnet(resnet_input)
   out = self.fc_layer(torch.concat(resnet_out, fc_input))

   return out
</code></pre>
<p>But torch.utils.data.Dataset::<strong>getitem</strong> is expected to return tensor, tensor, so I don’t know how to do this.</p>
<p>Maybe I should be constructing an input tensor which has another channel which has the extra information in it?  And then in forward doing something like:</p>
<pre><code class="lang-auto">def forward(x):
   resnet_input = x[:,:,:,:3]
   fc_input = x[:,:,:,3][0:4,0,0]
</code></pre>
<p>That’s pretty ugly though (and I’m just assuming the backpropogation all still works when you’re slicing the input like that…)</p>
<p>This feels like something which has a standard way of doing things, can you please educate me?</p>

ptrblck: <aside class="quote no-group" data-username="SystemLord" data-post="1" data-topic="203146">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e19b73/48.png" class="avatar"> SystemLord:</div>
<blockquote>
<p>But torch.utils.data.Dataset::<strong>getitem</strong> is expected to return tensor, tensor, so I don’t know how to do this.</p>
</blockquote>
</aside>
<p>You can return a <code>tuple</code> or <code>list</code>, too:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data1 = torch.randn(10, 1)
        self.data2 = torch.randn(10, 1)
        self.target = torch.randn(10, 1)
        
    def __len__(self):
        return len(self.data1)
    
    def __getitem__(self, index):
        x1 = self.data1[index]
        x2 = self.data2[index]
        y = self.target[index]
        return (x1, x2), y

dataset = MyDataset()
loader = DataLoader(dataset, batch_size=5)

for (input_batch1, input_batch_2), input_labels in loader:
    print(type(input_batch1))
    print(type(input_labels))
# &lt;class 'torch.Tensor'&gt;
# &lt;class 'torch.Tensor'&gt;
# &lt;class 'torch.Tensor'&gt;
# &lt;class 'torch.Tensor'&gt;
    
for input_batch, input_labels in loader:
    print(type(input_batch))
    print(type(input_labels))
# &lt;class 'list'&gt;
# &lt;class 'torch.Tensor'&gt;
# &lt;class 'list'&gt;
# &lt;class 'torch.Tensor'&gt;
</code></pre>

SystemLord: <p>Thanks Ptr.  I thought I was having a problem with sending a tuple to module::forward (which I’d need to do), but I’ll try again.</p>
Is there any roadmap to NVDIA cuLibrary API?

Forsworns: <p>There is a new collection API named <a href="https://developer.nvidia.com/blog/cuda-context-independent-module-loading/" rel="noopener nofollow ugc">cuLibrary</a> in the CUDA12, but I cannot find any related roadmap in the pytorch repo. Does pytorch have a plan to use these API? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m working on the AI infrastructure and wonder if we need to support cuLibrary in time, in case that our user with pytorch will need them. Or I can postpone this if most frameworks do not need cuLibrary support.</p>

Forsworns: <p>I captured these API in the samples with cuda 12. Seems nvcc calls these API during kernel registration in its preprocessing steps. But I guess torch won’t call API such as <code>cuLibraryGetGlobal</code> during runtime?</p>

ptrblck: <p>We don’t plan to use these APIs soon, but of course can add them for interesting use cases. I don’t understand your concern and what exactly you want to maintain as these are CUDA driver APIs, which are already needed to be present if you want to execute any GPU workload.</p>

Forsworns: <p>Thanks for your reply. Yes you are right that I found it’s necessary to support <code>cuLibraryLoadData</code> and <code>cuLibraryUnload</code>, etc… But I guess APIs such as <code>cuLibraryGetUnifiedFunction</code> are barely called.</p>
In-place torch.nn.functional.conv

Manvel_Petrosyan: <p>Hi!<br>
I have huge 2D tensor A and I want to convolve each row by same filter, for this purpose I use torch.nn.functional.conv2D, but there is a problem, sometimes I A is such that I can’t place 2 As in my GPUs memory, so I am interested is there any way that I can make in-place convolution to use less memory?</p>
Why slice may affect the nn.linear output?

JesseL1: <p>Hi all, I am curious why the following code will return false. Slice seems to affect the linear layer output.</p>
<pre><code class="lang-auto">torch.manual_seed(1234)
a = torch.randn((50, 4096)).float()
idx = [0, 2]
b = a[idx,:]
w1 = torch.nn.Linear(4096, 4096, bias=False)
w2 = torch.nn.Linear(4096, 4096, bias=False)
w3 = torch.nn.Linear(4096, 4096, bias=False)

act = torch.nn.SiLU()
out_a = w3(act(w1(a)) * w2(a))
out_b = w3(act(w1(b)) * w2(b))
print(torch.equal(out_a[idx,:], out_b))
</code></pre>

ptrblck: <p>Changing the input shape, stride, and memory layout can create different results caused by the limited floating point precision and a different order of operations. Also, different mathematical algorithms can be selected based on the aforementioned metadata of the tensors.</p>
How to configure conda environment for using GPU in Nvidia GTX 1650?

rashadkp: <p>I have a NVIDIA GTX1650 (not 1650 Ti). Iam unable to use my GPU for training in my local system. I tried installing Cuda toolkit 11.2, but Py torch is not using GPU on the cuda environment iam using. in the same cuda environment tensorflow is able to use gpu. how to train pytorch model using gpu</p>

ptrblck: <p>Create a new and empty conda environment, install PyTorch with CUDA support by using the provided install commands from <a href="https://pytorch.org/get-started/locally/">here</a>, and make sure the NVIDIA driver is properly installed.</p>
No kernel image is available for execution on the device: what to do?

mattiasospetti: <p>I know there are several other questions about this, but obviously none of them helped.<br>
I have the following:</p>
<p>-NVIDIA 960M (cc=5.0)<br>
-pytorch: 2.1.1<br>
-Cuda: 11.8</p>
<p>However,</p>
<pre><code class="lang-auto">YOLOv5  2023-12-12 Python-3.9.5 torch-2.1.1+cu118 CUDA:0 (NVIDIA GeForce GTX 960M, 2048MiB)
024-05-22 11:42:51.0616616 [E:onnxruntime:, sequential_executor.cc:514 onnxruntime::ExecuteKernel] Non-zero status code returned while running QuickGelu node. Name:'QuickGelu' 
Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device
YoloV5 failed --&gt; Exception raised: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running QuickGelu node. Name:'QuickGelu' Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device
</code></pre>
<p>I swear I have no idea on why it’s happening.<br>
The torch version is stable and cuda is supported… what’s going on?</p>

ptrblck: <p>The error seems to be raised by <code>ONNXRuntime</code> so you might want to check their GPU architecture support.</p>
Segmented input area

SalCometa: <p>Hello all,<br>
I have my input image segmented into water region and land region.<br>
I have my masked image:<br>
image = Image.open(image_path).convert(“RGB”)<br>
mask = Image.open(mask_path) .convert(“L”)</p>
<h1><a name="convert-to-numpy-arrays-1" class="anchor" href="#convert-to-numpy-arrays-1"></a>Convert to numpy arrays</h1>
<p>image_np = np.array(image)<br>
mask_np = np.array(mask)<br>
binary_mask = (mask_np &gt; 0).astype(np.uint8)</p>
<h1><a name="apply-mask-to-the-image-2" class="anchor" href="#apply-mask-to-the-image-2"></a>Apply mask to the image</h1>
<p>segmented_part = image_np * np.expand_dims(mask_np, axis=2)<br>
segmented_part.astype(“float32”)<br>
segmented_part_image = Image.fromarray(segmented_part)</p>
<p>I have a U-Net network and my input is the masked image the  tensor size is  (batch size, channels (I have 3), width, height) like below<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07c9a713f3c5861132704b0b57d7f565c23bfd4d.png" alt="plot" data-base62-sha1="16TnCWEfiN79yhpjeLu31r6kJqJ" width="200" height="200"></p>
<p>How can I work  with the features that are only in the water region, just predict the water region and have the loss for that region?</p>
<p>Thank you for your help</p>

Sam_Tux: <p>Write your own Dataset class that works with three things: image, object_mask, water_mask (where water pixel =1, land_pixel=0)</p>
<p>Do augmentation (I use albumentations lib) like this:<br>
sample = self.augmentation(image=image, masks=[object_mask, water_mask ])</p>
<p>Write your own loss (likely IoU + BCE) and metrics that takes three objects (pred, object_mask, water_mask), then do<br>
pred = pred *  water_mask<br>
object_mask= object_mask * water_mask<br>
and then do standard IoU_loss(pred, object_mask)  + BCE(pred, object_mask) on these new entities.</p>
<p>ChatGPT writes decent code for this task.</p>
Return a pair/tuple in C++ nn.Module forward method

hharryyf: <p>I saw almost all of the examples in the libtorch tutorial have returned a single Tensor in the forward method.</p>
<pre><code class="lang-auto">struct Net : torch::nn::Module {
   torch::Tensor forward(torch::Tensor x) {
     x = torch::relu(batch_norm1(conv1(x)));
     x = torch::relu(batch_norm2(conv2(x)));
     x = torch::relu(batch_norm3(conv3(x)));
     x = torch::tanh(conv4(x));
     return x;
   }
}
</code></pre>
<p>However, I’m confused about how can we let the forward method return a tuple/pair? I’m currently doing a reinforcement learning task, and I need the network forward method to return both the policy and the position value. How can I translate the following code to C++? Is it possible to do it without the JIT stuff?</p>
<pre><code class="lang-auto">def forward(self, state_input):
        x = self.common_layer(state_input)
        x_action = self.policy_layer(x)
        x_value = torch.tanh(self.value_layer(x))
        return x_action, x_value
</code></pre>
<p>Thanks a lot!</p>

Xavier31: <p>Hi. I do not use libtorch for training so I may be completly off, but what happens if you replace the return line in the forward with:</p>
<pre><code class="lang-auto">return std::make_tuple(x_action, x_value);
</code></pre>
<p>at inference, retrieve the tuple with:</p>
<pre><code class="lang-auto">auto outputs = model.forward(inputs).toTuple();
at::Tensor action= outputs-&gt;elements()[0].toTensor();
at::Tensor value= outputs-&gt;elements()[1].toTensor();
</code></pre>

Gonen_Raveh: <p>Take a look at the full function here: <a href="https://discuss.pytorch.org/t/preallocating-cuda-buffers-as-preparation-for-libtorch-model-forward/203203/1" class="inline-onebox">Preallocating CUDA Buffers as preparation for libtorch model.forward</a></p>
Memory leak in Pytorch 1.10

Sam_Tux: <p>I run the following code Ind I have a memory leak</p>
<pre><code class="lang-auto">from time import sleep
import cv2
from torch import from_numpy, load as torch_load, cuda

def predict(image):  # numpy image
        print(f'memory before: {cuda.memory_stats()}')
        x_tensor = from_numpy(image).to(DEVICE).unsqueeze(0)
        del x_tensor
        sleep(20)
        cuda.empty_cache()
        sleep(20)
        cuda.synchronize()
        sleep(20)
        cuda.empty_cache()
        sleep(20)
        cuda.reset_max_memory_allocated()
        sleep(20)
        print(f'memory after: {cuda.memory_stats()}')


image = cv2.imread('my_image.png')
for i in range(10):
    predict(my_image)
</code></pre>
<p>The output is like for ‘active.all.allocated’ key</p>
<p>memory before: 0<br>
memory after: 1<br>
memory before: 1<br>
memory after: 2<br>
…<br>
memory before: 9<br>
memory after: 10</p>
<p>Is thre a workaround for the given version? It would be time consuming to retrain all the models and the models are incompatible with a new smp/PyTorch.<br>
What can cause this memory leak? Is it an old bug that is fixed in more recent versions of PyTorch?</p>

ptrblck: <p>Using <code>print(f'summary: {cuda.memory_summary()}')</code> shows the expected results of <code>0B</code> in the current usage and an increased total allocation count.</p>

Sam_Tux: <p>Thank you for your response.<br>
So, how do I check for a memory leak? Situation. When I analyze some 10 photos one by one it works fine. Then the application fails with memory allocation error. I do shrink all photos to 10 megapixel size and I do know that my GPU can hold and process 10- Mpixel photos.<br>
But there is a clear memory leak buildup. Possibly some memory fragmentation issues.<br>
I do</p>
<pre><code class="lang-auto">del tensor
cuda.empty_cache()
</code></pre>
<p>after each analysis.<br>
What can cause the problem and how to fix it?<br>
PS We are working on the analysis of skin health if it helps.</p>
<p>PPS. In the actual application I load my models (takes some 2 Gb out of 8 Gbs). Then I receive and analize photos. Each photo is shrunk to 10- Mpixels prior to analysis. Works fine on the first ten photos, then fails with</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. 
Tried to allocate 1.17 GiB (GPU 0; 7.93 GiB total capacity;
3.47 GiB already allocated; 
1.03 GiB free; 6.14 GiB reserved in total by PyTorch) 
If reserved memory is &gt;&gt; allocated memory try setting 
max_split_size_mb to avoid fragmentation.  
See documentation for Memory Management and 
PYTORCH_CUDA_ALLOC_CONF
</code></pre>

Sam_Tux: <p>Can you please expand your explaination?<br>
Do I have a memory leak or not?</p>
<p>When I run my app and analize the same photo hundred times it crashes with “out of memory. Tried to allocate 1.2 Gb of memory, only have 1.1 free memory”.</p>
Pre trained model performance is affected by other models

Reef: <p>I am trying to integrate Mask RCNN to another code, defining it on the constructor of the class. I am using pre trained weights and using it in the evaluation mode. but I noticed that its predictions are affected by other functions inside the constrctor of the class.</p>
<p>this is my line of initializing the model inside the base code’s constrctor:<br>
self.model_mask = torchvision.models.detection.maskrcnn_resnet50_fpn(weights = “MaskRCNN_ResNet50_FPN_Weights.DEFAULT”)</p>
<p>these are other functions that I think they have a direct effect on its performance that are defined in the constrctor of the class.<br>
fill_fc_weights(…)<br>
pass</p>
<p>when I use mask rcnn model separately it works fine but when I integrate it to the base code it does not give the same predictions even for the same images. I tried to freeze the weights but that did not work. your help is greatly appreciated</p>

ptrblck: <aside class="quote no-group" data-username="Reef" data-post="1" data-topic="202730">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/reef/48/69994_2.png" class="avatar"> Reef:</div>
<blockquote>
<p>these are other functions that I think they have a direct effect on its performance that are defined in the constrctor of the class.<br>
fill_fc_weights(…)</p>
</blockquote>
</aside>
<p>If you are initializing (some) weights in a custom way, differences in the output is expected so check what e.g. <code>fill_fc_weights</code> does and if it’s initializing some parameters.</p>

Reef: <p>Sorry for the inconvenience. I realized that the problem was that the colors of the image were distorted which was the cause of the model’s unprecise predictions.</p>
<p>Thank you for your attention</p>
Accuracy of Quantized model drop very much compare with normal model

kent252: <p>I am a newbie of quantization. I trained a Resnet model and quantized it by using this instruction<br>
The original model possesses 96% accuracy, however, model_quantized got 49% accuracy. Why this happened and how can I address it? Thank all of you</p>
<h2><a name="quantize-model-post-training-static-quantization-fx-graph-mode-1" class="anchor" href="#quantize-model-post-training-static-quantization-fx-graph-mode-1"></a>Quantize model (post-training Static quantization FX Graph mode)</h2>
<blockquote>
<p>torch.backends.quantized.engine = “x86”</p>
<p>model.to(“cpu”)<br>
model.eval()</p>
<p>prepared_model = prepare_fx(model, {“”: get_default_qconfig(“x86”)})</p>
<p>with torch.no_grad(): # calibrate using random data</p>
<blockquote>
<pre><code>  data = torch.rand((7,1,10,20))
  prepared_model(data)
</code></pre>
</blockquote>
<p>model_quantized = convert_fx(prepared_model)</p>
</blockquote>

jcaip: <p>Hi <a class="mention" href="/u/kent252">@kent252</a></p>
<p>This is expected if you calibrate using random data. During calibration we calculate the optimal <code>scales</code> and <code>zero_point</code> using the data provided. Can you try calibrating on the representative input data of whatever dataset you are using to evaluate accuracy?</p>

kent252: <p>Thank you for your information. I appreciate</p>
Is there any problem with using "sub models"?

SystemLord: <p>I wish to create a model which uses a resnet as part of the architecture</p>
<p>It will eventually have other layers, but I’ve put together a quick prototype:</p>
<pre><code class="lang-auto">class CombinedModel(torch.nn.Module):
    def __init__(self):
        super(CombinedModel, self).__init__()

        self.resnet = torch.hub.load('pytorch/vision:v0.10.0', 'resnet101', pretrained=True)

    def forward(self, x):
        x = self.resnet(x)

        return x
</code></pre>
<p>This seems to learn fine with my (very standard) training code.  Are there any things I need to be aware of though?  Do I need to forward, for example, .eval and .train calls to the sub-model?</p>

Eduardo_Lawson: <p>Will depend if the resnet is already trained with weights that are “good” for your data For the evaluation part you will definitely have to call to sucessfully run it.</p>

ptrblck: <aside class="quote no-group" data-username="SystemLord" data-post="1" data-topic="203142">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e19b73/48.png" class="avatar"> SystemLord:</div>
<blockquote>
<p>Do I need to forward, for example, .eval and .train calls to the sub-model?</p>
</blockquote>
</aside>
<p>No, all properly registered submodules will automatically receive the <code>.train()</code>, <code>.eval()</code>, and <code>.to()</code> calls and you can double check it:</p>
<pre data-code-wrap="python"><code class="lang-python">model = CombinedModel()
print(model.resnet.training)
# True

model.eval()
print(model.training)
# False
print(model.resnet.training)
# False
</code></pre>