akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch+cuda problem

emilio_hermosa: <p>Hello, I’ve been having problems this days while trying to import torch after installling it with the following line:</p>
<pre><code class="lang-auto">pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
</code></pre>
<p>When I try to import torch I have the follwing error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[1], line 2
      1 # Importar librerías clásicas
----&gt; 2 import torch
      3 import numpy as np
      4 import pandas as pd

File c:\Users\2002e\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\__init__.py:148
    146                 err = ctypes.WinError(ctypes.get_last_error())
    147                 err.strerror += f' Error loading "{dll}" or one of its dependencies.'
--&gt; 148                 raise err
    150     kernel32.SetErrorMode(prev_error_mode)
    153 def _preload_cuda_deps(lib_folder, lib_name):

OSError: [WinError 126] No se puede encontrar el módulo especificado. Error loading "c:\Users\2002e\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies.
</code></pre>
<p>I have seen that the fbgemm.dll file is on the enviroment and that I have everything good configured. I’ve tryed with several cuda and torch versions but is always giving me the error. If I install torch for cpu and import it it works correctly.<br>
I hope someone can help me.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969">This topic</a> might help.</p>
Which is better TorchServe Batching or ClientSide Batching for text classification

OmniaZayed: <p>Hi,<br>
I am working on a text classification project, and I am researching the best way to do batching if I want to send a csv file to the model to classify at once.  I couldn’t find examples that show batching for text, but I came across <a href="https://github.com/pytorch/serve/tree/master/examples/image_classifier/near_real_time_video" rel="noopener nofollow ugc">this example</a> for Image Classification  that describes batching using two methods:</p>
<ul>
<li>TorchServe Batching</li>
<li>Client-Side Batching</li>
</ul>
<p>At some stage, instead of using <code>curl</code>, I want to implement a UI application that sends either raw text or csv file to be classified. My idea is that the csv file would be batched on the client side and then sent as a payload in JSON to <code>requests.post(url, data=payload, headers=headers)</code></p>
<p>I wonder which method is better for my use case. Also, if the TorchServe Batching is better, can you please recommend any pointers for its implementation?</p>
Why do different data splits lead to significant differences in the fit of my LSTM model?

ives_chenn: <p>Hello everyone, today I was trying to use LSTM for time series prediction. Initially, I used an 80% training set and a 20% test set, and everything fit well. However, when I reduced the training set to 70%, the model started severely underfitting, with an MAE close to 14 under normalization. What could be causing this?</p>
How to calculate PDEs(via torch.fft.fftn) from the output of FNO model using model parallel?

Gaijinliu_Gangmei: <p>I split my Fourier Neural Operator(FNO) model using model parallel(<a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Single-Machine Model Parallel Best Practices — PyTorch Tutorials 2.4.0+cu124 documentation</a>). The output from the model was used for calculating PDEs using Fourier Transform(torch.fft.fftn).<br>
I was able to acquire the required PDE solution when I used only one GPU. However, the PDE solution increased significantly when the same code was run on two GPUs.</p>
<p>Is there a solution for this issue?</p>

LilyAvo: <p>Compute the FFT of the model’s output, apply the differential operator in the frequency domain, and then perform an inverse FFT. Here’s an example:</p>
<pre><code class="lang-auto">import torch.fft

# Assuming output is your FNO model's output
fft_output = torch.fft.fftn(output)

# Apply the differential operator in the frequency domain
# Example for a second-order derivative: multiply by -k^2
k = torch.fft.fftfreq(output.size(-1))
k_squared = -k ** 2
fft_derivative = fft_output * k_squared

# Inverse FFT to get the derivative in spatial domain
derivative = torch.fft.ifftn(fft_derivative).real

</code></pre>

Gaijinliu_Gangmei: <aside class="quote no-group" data-username="LilyAvo" data-post="2" data-topic="206818">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/97f17d/48.png" class="avatar"> LilyAvo:</div>
<blockquote>
<p>Compute the FFT of the model’s output, apply the differential operator in the frequency domain, and then perform an inverse FFT</p>
</blockquote>
</aside>
<p>The results I get from the above steps is different when I use 2 GPUs instead of 1 GPU.</p>
<p>Do you know why I am not getting the same result when I use 2 GPUs ?</p>

anantguptadbl: <p><a class="mention" href="/u/gaijinliu_gangmei">@Gaijinliu_Gangmei</a> It is quite possible that you are not setting the same manual_seed</p>
<pre><code class="lang-auto">torch.manual_seed(42)
</code></pre>

Gaijinliu_Gangmei: <p>The problem still exists even after using the same manual seed( torch.cuda.manual_seed_all(42) and torch.manual_seed(42)).</p>
Why do I get a single hiddenstate for the whole batch when I try to process each timestep separately?

Shisho_Sama: <p>Hello everyone, hope you are doing great.<br>
I have a simple question, that might seem dumb, but I here it goes.</p>
<p>Why do I get a single hiddenstate for the whole batch when I try to process each timestep separately?<br>
consider this simple case:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_size, num_layers=1, bidirectional=False, dropout=0.3):
        super().__init__()
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bidirectional = bidirectional
        self.dropout = dropout
        self.lstm = nn.LSTM(input_size=embedding_dim, 
                            hidden_size=self.hidden_size, 
                            num_layers=self.num_layers,
                            batch_first=True, 
                            dropout=self.dropout, 
                            bidirectional=self.bidirectional)
        self.embedding = nn.Embedding(self.vocab_size, embedding_dim)
        
        
    def forward(self, x, h):
        x = self.embedding(x)
        out = []
        for t in range(x.size(0)):
            xt = x[:,t]
            print(f'{t}')
            outputs, hidden = self.lstm(xt, h)
            out.append((outputs,hidden))
            print(f'{outputs.shape=}')   # prints (batchsize, timestep, hiddensize)
            print(f'{hidden[0].shape=}') # prints (1,hiddensize)
            print(f'{hidden[1].shape=}') # prints (1,hiddensize)
        return out


enc = Encoder(vocab_size=50,embedding_dim=75, hidden_size=100)
xt = torch.randint(0,50, size=(5,30))
h = None
enc(xt, None)
</code></pre>
<p>Now I’m expecting to get <code>(batchsize, hiddensize)</code> for my hiddensize, the same way my outputs come out ok as <code>(batchsize, timestep, hiddensize)</code>. but for some reason, the hiddenstate shape is <code>(1,hiddensize)</code>, not <code>(5,hiddensize)</code> which is the batchsize.<br>
basically I’m getting a single hiddenstate for the whole batch at each iteration, but I get correct outputs for some reason!?</p>
<p>Obviously this doesn’t happen if I feed the whole input sequence all at once, but I need to grab each timestep for my Bahdanau attention mechanism. I’m not sure why this is happening?<br>
any help is greatly appreciated.</p>

Shisho_Sama: <p>The issue is the way PyTorch requires the input to be!<br>
Basically the input needs to be in the form of <code>(batchsize, timestep, input_dim)</code> for this to work.<br>
All we had to do was :</p>
<pre><code class="lang-auto"># add a timestep dimension
xt = xt[:, None, :]
</code></pre>
Is there a Pytorch version With 3.0 Compability?

Muhammad_Hilal: <p>I’ve Nvidia Quadro K400. And I’m just new to CUDA &amp; Pytorch. So I’ve downloaded Cuda toolkit version 12.1, and Pytorch that supports Cuda 11.8 by following this command :<br>
<strong>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></strong><br>
Now to check whether I’ve successfully downloaded CUDA and pytorch I’ve ran the below Script:</p>
<p><strong>import torch</strong></p>
<p><strong>print(torch.cuda.is_available())</strong></p>
<p><strong>print(torch.cuda.current_device())</strong></p>
<p><strong>print(torch.cuda.device(0))</strong></p>
<p><strong>print(torch.cuda.device_count())</strong></p>
<p><strong>print(torch.cuda.get_device_name(0))</strong></p>
<p>And it gives me the following results:<br>
<strong>True</strong><br>
<strong>D:\TorchEnviornment\TorchEnv\torch_env\Lib\site-packages\torch\cuda_<em>init</em>_.py:184: UserWarning: **<br>
**    Found GPU0 Quadro K4000 which is of cuda capability 3.0.</strong><br>
**    PyTorch no longer supports this GPU because it is too old.**<br>
**    The minimum cuda capability supported by this library is 3.7.**<br>
**  warnings.warn(**<br>
<strong>0</strong><br>
<strong>&lt;torch.cuda.device object at 0x00000255B1D23B30&gt;</strong><br>
<strong>1</strong><br>
<strong>Quadro K4000</strong><br>
Now the problem is that it won’t allow me to run Python libraries that support the CUDA for the operations rather it falls back to CPU which takes alot of time. Rather than switching to hardware at the moment, I’m thinking of downgrading the pytorch version that supports the compute compatibility of 3.0, but I’m unable to find such relevant information on the internet, so it would be great if someone contribute.</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/releases/tag/v0.3.1">these release notes</a> it seems <code>v0.3.1</code> removed compute capability 3.0.</p>

Muhammad_Hilal: <p>So the only solution left is to get rid of the hardware? or there is other solution that I can switch towards maybe downloading some crack version, I guess?</p>

ptrblck: <p>No, I don’t think any newer PyTorch release can support your GPU from 2013, as sm_30 was dropped in CUDA 11 and we also dropped the support for CUDA 10.</p>
Train/Validation separation before data augmentation?

L_Z: <p>My question is as my topic. I should have separate my dataset into train and validation before applying any data augmentation and randomization right? Otherwise, it will be some sort of data leakage?<br>
Thanks in advance.</p>

ptrblck: <p>It depends on the actual transformation and if it’s stateful.<br>
E.g. if you are using <code>Normalize</code> on the tensors the stats should be computed based on the training dataset and applied to all splits. Computing the stats using all samples would be considered a data leak.</p>
Worker Process hanging on recv using gloo backend

ChuckGarcian: <p>Howdy all,</p>
<p>I’m working on a distributed application using the PyTorch distributed package, but I’m encountering issues with <code>isend</code> and <code>recv</code>. I’ve simplified my code to highlight the problem:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist

# Host (Rank 0) code
def host_distribute_data(batches, tensor_sizes_data):
    for dst_rank, batch in enumerate(batches):
        print(f"Sending to rank {dst_rank+1}", flush=True)
        dist.isend(torch.tensor(tensor_sizes_data.shape, dtype=torch.int64), dst=dst_rank+1, tag=0)
        dist.isend(tensor_sizes_data, dst=dst_rank+1, tag=0)
        dist.isend(torch.tensor(batch.shape, dtype=torch.int64), dst=dst_rank+1, tag=0)
        dist.isend(batch, dst=dst_rank+1, tag=0)

# Worker (Rank &gt; 0) code
def worker_receive_data():
    while True:
        with torch.no_grad():
            tensor_sizes_shape = torch.zeros([1], dtype=torch.int64)
            dist.recv(tensor=tensor_sizes_shape, src=0, tag=0)
            tensor_sizes = torch.zeros(tensor_sizes_shape, dtype=torch.int64)
            dist.recv(tensor=tensor_sizes, src=0, tag=0)
            
            # Receive batch shape and data
            batch_shape = torch.zeros_like(tensor_sizes_shape)
            dist.recv(tensor=batch_shape, src=0, tag=0)
            batch = torch.zeros(batch_shape)
            dist.recv(tensor=batch, src=0, tag=0)
            
            # Process batch here...

# Main process
def main():
    # Initialize process group, prepare data, etc.
    if dist.get_rank() == 0:
        host_distribute_data(batches, tensor_sizes_data)
    else:
        worker_receive_data()

if __name__ == "__main__":
    main()
</code></pre>
<p>In this simplified example, I’m trying to distribute batches of data from a host (rank 0) to multiple worker processes. The host uses isend to send data, while workers use blocking recv to receive it. However, I’m experiencing a specific issue where the worker nodes hang on the second recv and never continue execution. This problem only occurs when using the Gloo backend, but works fine with the NCCL backend.</p>
<p>For context I found this <a href="https://github.com/pytorch/pytorch/issues/30723" rel="noopener nofollow ugc">github issue</a> that describes my exact issue but no solution was ever posted.</p>
<p>Any insights or suggestions would be greatly appreciated. Thank you in advance for your help!</p>
Self-supervised learning weights initialization "after" projection head

grid_world: <p>For most Self-supervised learning algorithms: SimCLR, MoCo, BYOL, SimSiam, SwAV, etc., its common to have a projection head after the base encoder (which in most cases is a vanilla ResNet-50 CNN). An example of such a projection (taken from SwAV) is:</p>
<pre><code class="lang-auto">projection_head = nn.Sequential(
    nn.Linear(2048, 512),
    nn.BatchNorm1d(512),
    nn.ReLU(inplace=True),
    nn.Linear(512, 128),
)
</code></pre>
<p>The output of this projection head is L2-normalized:</p>
<pre><code class="lang-auto">x = projection_head(x)
x = nn.functional.normalize(x, dim = 1, p = 2)
</code></pre>
<p>I am trying to initialize a layer after the projection head as:</p>
<pre><code class="lang-auto">wts = nn.Parameter(data = torch.empty(40 * 40, 128), requires_grad = True)
# The projection head outputs weights in the range [-1, 1], so initialize SOM weights to be in that range-
wts.data.uniform_(-1.0, 1.0)
</code></pre>
<p>Since the output of the projection head is L2-normalized, I am assuming that the input range to “wts” ∈ [-1, 1] and therefore use the uniform initialization above.<br>
Is this a correct approach or am I missing something?</p>
Does torch.compile support inter-op parallelism?

andreyyashkin: <p>I tried <a href="https://pytorch.org/tutorials/advanced/torch-script-parallelism.html" rel="noopener nofollow ugc">dynamic inter-op parallelism in TorchScript</a> tutorilal with torch.compile and found that it makes forward and backward LSTM to run sequentially. Here is my <a href="https://colab.research.google.com/drive/1ecMYqhdhXFzOeIi3VeOZrx5beixT3mQB?usp=sharing" rel="noopener nofollow ugc">notebook</a></p>

MoFHeka: <p>Any progress? In Recommender model, there are usually thousands of input feature. And these features have their own preprocess methods, which are able to parallel.</p>
Even = total_real[:, :, :, :, 0]

TaeYeon_Lee: <p>In pytorch,</p>
<p>even = total_real[:, :, :, :, 0]    # extract even elements<br>
odd = total_real[:, :, :, :, 1]      # extract odd elements</p>
<p>total_real.shape<br>
torch.Size([1, 1, 1023, 64, 2])</p>
<p>even.shape<br>
torch.Size([1, 1, 1023, 64])</p>
<p>odd.shape<br>
torch.Size([1, 1, 1023, 64])</p>
<p>In C++ Torch lib,<br>
How to convert it as same ?</p>

LilyAvo: <p>To convert the output to be the same, you can ensure each step in the operation is valid. If you’re trying to get the real part of the cosine of freqs, you might need to separately apply the operations and ensure they’re compatible. Here’s an example:</p>
<pre><code class="lang-auto">cos_freqs = freqs.cos()  # Apply cosine
real_cos_freqs = cos_freqs.real()  # Get the real part
</code></pre>
<p>If <code>freqs</code> is a complex tensor, this should work. If not, please provide more details about <code>freqs</code> and the desired output.</p>

TaeYeon_Lee: <p>I want to extract even elements, in C++ torch lib.</p>

ptrblck: <pre data-code-wrap="python"><code class="lang-python">auto even = total_real.index({"...", 0});
auto odd = total_real.index({"...", 1});
</code></pre>
<p>should work based on the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html">docs</a>.</p>

TaeYeon_Lee: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="206712">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>total_real.index({"...", 0});</code></p>
</blockquote>
</aside>
<p>It’s working.<br>
Thank you expert for the info.</p>
DPP no_sync not equivalent

jamied157: <p>Hi there,</p>
<p>Just wanted to check some logic, I’m seeing cases where the gradients for my model using DDP with <code>no_sync</code> set aren’t equivalent to DDP without <code>no_sync</code> set when using gradient accumulation. I’m seeing this in the 2nd decimal place of the gradient norm which is obviously quite a small difference, so my guess is that we’re hitting up against some of the limitations of floating point and because we’re now summing the gradients in a slightly different order, this is causing differences.</p>
<p>Does this make sense as an explanation? Is there anything else that I could be missing?</p>
<p>Thanks!</p>

agu: <p>This seems somewhat reasonable to me. Is this using fp32 gradients?</p>
Tensor Parallelism Behavior in FSDP Example (fsdp_tp_example.py)

SeungyounShin: <p>I am running the <a href="https://github.com/pytorch/examples/blob/26de41904319c7094afc53a3ee809de47112d387/distributed/tensor_parallelism/fsdp_tp_example.py" rel="noopener nofollow ugc">fsdp_tp_example</a> from the PyTorch examples repository.</p>
<p>I understood that tensor parallelism distributes weights and activations across devices. For example, if we apply ColWiseLinear to a Linear layer, I expected it to distribute parts of the weight tensor, such as <code>weight[:tp_size, :]</code>, across multiple devices. However, after running the fsdp_tp_example, I observed that it actually keeps the entire model parameters on each GPU. For instance, in the case of the LLaMA 7B model, the full 24GB of parameters remain on each GPU even after initializing with <code>parallize_module</code>.</p>
<p>Is this the expected behavior? I thought the main purpose of tensor parallelism was to fit large models on multiple GPUs by splitting the weights, but it seems like the full weights reside on each device, and only sub-parts of the weights are used for calculations.</p>

agu: <p>Can you explain a bit how you concluded that the full 24 GB of parameters are on each GPU? Maybe you could share some repro script?</p>
DDP/ FSDP across geo-distributed machines

Sanjeev_Rampal: <p>Hello,</p>
<p>While it is clear that most scenarios for using DDP or FSDP have all training data and all compute resources in the same region/ same datacenter, in theory there could be cases where data is distributed over a WAN. So either the data cant be moved due to policy/ sovereignty/ security reasons or it could have been moved in theory but it is better/ more efficient in some particular scenario to not move the data and instead perform distributed training and then aggregate/ combine the partial models from each location without ever having moved the raw training data out of each location. And yet all these locations belong to the same enterprise so this is not a cross-enterprise federated training scenario. … more of geo-distributed training across multiple locations within a single enterprise.</p>
<p>There may be a moderately high speed network connection between the sites (10G or even 100G for example) but not as much of a high speed network as within a single datacenter which could have a 400G network for example).</p>
<p>I’m trying to determine are there real world use cases where such an approach has been used ? And if yes, was some variation of Pytorch DDP/ FSDP considered ? or was it a “Federated training framework product”  ? Or has someone done a theoretical analysis of the tradeffs to determine what combination of compute capacity/ training data volume/ network bandwidth and latency is needed to prefer the use of such geo-distributed training design even within a single enterprise.</p>
<p>Perhaps this question has been asked before (though I couldnt find anything from search) and I welcome your responses.</p>

agu: <p>This is not a direct answer, but maybe you will find <a href="https://www.primeintellect.ai/blog/opendiloco" class="inline-onebox" rel="noopener nofollow ugc">Prime Intellect</a> interesting.</p>
PyTorch C++ extension: efficiently accumulate results from threads in a tensor

Jonas_De_Schouwer: <p>I am implementing a PyTorch C++ extension that employs parallellism. The desired result is a N x D matrix, and each thread computes one row of this result. What is the recommended way to accumulate these rows?</p>
<hr>
<p>To put this into context, my code looks as follows (simplified):</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">at::Tensor nearestKKeys(at::Tensor queries, at::Tensor keys, int k, int maxLeafSize) {
    /*
    queries: (Nq, D)
    keys: (Nk, D)
    k: int
    return: (Nq, k)
    */
    int Nq = queries.size(0);

    at::Tensor result = at::empty({Nq, k}, at::ScalarType::Int);
    at::Tensor indices = at::arange({Nk}, keys.device())
    BallTreePtr pBallTree = buildBallTree(
        keys,
        indices,
        maxLeafSize,
        0
    );
    
    #pragma omp parallel for
    for (int n=0; n&lt;Nq; n++) {
        at::Tensor query = queries.index({n});
        
        BestMatchesPtr pBestMatches = std::make_shared&lt;BestMatches&gt;(k);
        // places the desired result for this query in *pBestMatches
        pBallTree-&gt;query(query, query_norm, pBestMatches, k);
        
        // place the desired row in results
        at::Tensor matches = pBestMatches-&gt;getMatches();
        result.index_put_({n}, matches);
   }
}
</code></pre>
<p>This is currently not very efficient, and my profiler tells me that the threads spend a lot of time waiting on each other. I believe this is because of the <code>result</code> variable being shared between threads. I don’t think that <code>pBallTree</code> being shared really matters (because it’s a pointer), but do tell me if I’m mistaken. So, hence my question: What is the recommended way to accumulate these rows?</p>
Offloading mmap state_dict to cpu

Sohaib_Ahmed: <p>I have<br>
<code>checpoint=torch.load(chkpt_path, mmap=True)</code><br>
Which after merging different checkpoints of that type and changing keys, I load it into model using<br>
<code>model.load_state_dict(checkpoint)</code><br>
Now the problem is that model weights are memory mapped. Using them for forward pass will be very slow. Doing<br>
<code>model.to("cpu")</code><br>
does not offload weights to cpu which is required.<br>
<code>to("cuda")</code> works though but I want testing on cpu.<br>
How to do that.</p>

mikaylagawarecki: <p>Are you calling <code>model.load_state_dict(checkpoint)</code> with <code>assign=True</code> or <code>False</code>?</p>
<p>If calling with <code>assign=False</code> (the default value for <code>assign</code>) I would expect that for <code>cpu_model.load_state_dict(mmaped_checkpoint, assign=False)</code> , the <code>model.param.copy_(checkpoint['param_key'])</code> within the implementation of <code>load_state_dict</code> (<a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/module.py#L2336" rel="noopener nofollow ugc">this line</a>) to cause all the mmap-ed weights to be loaded into CPU RAM.</p>

Sohaib_Ahmed: <p>Detailed code:</p>
<pre data-code-wrap="python"><code class="lang-python">#After processing the mmap checkpoint
With torch.device("meta"):
         model = Model()
model.load_state_dict(checkpoint, assign=True)
model.to("cpu") # model state_dict is still mmaped. No effect
</code></pre>

mikaylagawarecki: <p>Got it,</p>
<p>Yea I do expect the <code>model.to("cpu")</code> not to load weights into RAM for that case.</p>
<p>I would expect</p>
<pre><code class="lang-auto">checkpoint = torch.load(chkpt_path, mmap=True)
# do pre-processing with mmaped weights
# ...
model = Model() # 1x model weights in CPU RAM
model.load_state_dict(checkpoint)
</code></pre>
<p>to load all the mmap-ed weights into CPU RAM like you want.</p>
<p>Unlike the previous approach with the <code>meta</code> device and <code>load_state_dict(assign=True)</code>, I would expect the peak CPU RAM used be 1x model weights + size_of_largest_parameter, would that work for your use case?</p>

Sohaib_Ahmed: <p>Thank you for giving the idea to remove meta device when offloading to cpu. I can use condition to only use meta device when offloading the model to “cuda” in order to save ram.</p>
Is there a way to train a CNN with the loss of another net?

cabamarcos: <p>I have 2 CNNs. One is called net and the other is called mask. Both have the same architecture but are initialised differently. My training loop looks like this:</p>
<pre><code class="lang-auto">for epoch in range(epochs):

  print(f"Epoch {epoch + 1}")
  
  net.train()
  mask.train()
  
  # I use the mask to prune the net
  net_masked = apply_mask(net, mask)
  
  running_loss = 0.0
  
  for images, labels in train_loader:
    images, labels = images.to(device), labels.to(device)
  
    # I use the images to calculate the loss of the net
    outputs_net = net_masked(images)
    loss_net = criterion(outputs_net, labels)
  
    outputs_mask = mask(images) # I also get the loss of the mask (I don't use loss_mask.item())
    loss_mask = criterion(outputs_mask, labels)
  
    loss = loss_net + loss_mask*0 # Now loss looks like this : tensor(2.2969, grad_fn=&lt;AddBackward0&gt;)
  
    optimizer_mask.zero_grad()
  
    loss.backward() # Here I try to calculate the gradients
  
  
    optimizer_mask.step()  # Update weights
  
    running_loss += loss_net.item()
  
  train_loss.append(running_loss / len(train_loader))
</code></pre>
<p>What I am trying to do is update the mask weighs to prune net in different ways each time. My problem is that when I try to compute the gradients I get “None” in all of them.</p>
<p>I don’t exactly know why the gradients are not beeing computed maybe is the way I am computing the loss.</p>
<p>If someone can help me I would be so grateful</p>
transforms.ToTensor ignores set_default_device("cuda")

jfrancis79: <p>It seems that transforms.ToTensor ignores set_default_device.</p>
<p>Here is an example:</p>
<pre><code>import torch
import torchvision.transforms as transforms
import torchvision.datasets as datasets


torch.set_default_device("cuda")
transform = transforms.ToTensor()
dataset = datasets.MNIST("~/datasets", train=True, download=False, transform=transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
batch = next(iter(dataloader))
print("Batch[0] = ", batch[0].device)
print("Batch[1] = ", batch[1].device)
</code></pre>
<p>When run, I have batch[1] on cuda, but batch[0] is on the CPU. I expected batch[0] to be on cuda.</p>
<p>Interestingly, from examining the transforms code, it seems that it is using torch.from_numpy and this maybe a problem:</p>
<pre><code>import torch
import numpy as np


torch.set_default_device("cuda")
t = torch.from_numpy(np.array(5.0))
print(t)
</code></pre>
<p>I get:<br>
tensor(5., dtype=torch.float64)</p>
<p>I would have expected this to be on the CUDA device, and I believe it may be the source of the above problem.</p>

ptrblck: <p>Yes, this is expected as explained in the <a href="https://pytorch.org/docs/stable/generated/torch.set_default_device.html#torch.set_default_device">docs</a>:</p>
<blockquote>
<p>This doesn’t affect functions that create tensors that share the same memory as the input, like: torch.from_numpy() and torch.frombuffer()</p>
</blockquote>

jfrancis79: <p>Thanks for the doc reference, that explains it.</p>
<p>Presently I do:</p>
<pre><code>class DevicePlace:
    def __call__(self, x):
        return x.to(torch.get_default_device())
</code></pre>
<p>and then in the transform:</p>
<pre><code>transform = transforms.Compose([transforms.ToTensor(), DevicePlace()])
</code></pre>
<p>It works, but I find it a bit ugly. I guess I could make it a function call, but I was hoping to remove it entirely. It was not intuitively obvious to me that a ToTensor would not transfer tensor onto default device.</p>
<p>I have been moving my code to use set_default_device and it has vastly simplified all my device management and is working very well generally.</p>
Precision issues with torch.mm functions

pilipala5: <p>In my code, I’ve identified two lines that are mathematically equivalent, yet their outcomes significantly diverge. I am seeking to understand the underlying reasons for this discrepancy？<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6cc3fc6c925a627ff174c66044e62d46787c8cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6cc3fc6c925a627ff174c66044e62d46787c8cd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6cc3fc6c925a627ff174c66044e62d46787c8cd_2_690x374.png" alt="image" data-base62-sha1="nNyRqSqW5LMilvb9T2EXOZgCt0F" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6cc3fc6c925a627ff174c66044e62d46787c8cd_2_690x374.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6cc3fc6c925a627ff174c66044e62d46787c8cd_2_1035x561.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6cc3fc6c925a627ff174c66044e62d46787c8cd.png 2x" data-dominant-color="292A2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1182×642 53 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue, please?</p>
Model Convergence Issue in Custom Training Loop

LilyAvo: <p>There are issues with model convergence when using a custom training loop in PyTorch. The loss stagnates after a few epochs. The optimizer is Adam with a learning rate of 0.001. Issue persists even after adjusting the learning rate and changing the network architecture. Any advice on what might be causing this and potential solutions?</p>

Eduardo_Lawson: <p>If you can post here a snippet of your code it will be easier to give you a solution.</p>
Query on Asynchronous Execution and Graph Splitting in PyTorch for Embedding Lookup Performance Optimization

yu_feng1: <p>Hello everyone,</p>
<p>We are in the process of migrating our CTR training framework from TensorFlow to PyTorch, which involves tasks on GPUs. However, we are currently facing performance bottlenecks with PyTorch’s embedding lookup. In TensorFlow, we divided the computation graph into an embedding graph and a default graph. The embedding graph asynchronously pre-fetches the embeddings and updates the embedding parameters. I would like to ask if it is possible to split the graph and execute it asynchronously in PyTorch?</p>
Tensor sharing between CPU and GPU processes

zizhao.mo: <p>Hi,</p>
<p>I have a use case, where I have two processes, one for execution in GPU and another one is responsible for CPU execution. I want to send PyTorch tensors from GPU to CPU asynchronously, i.e., the data transfer to the CPU process will not block the workflow on GPU-hosted process.</p>
<p>Is this idea viable? If yes, is torch.multiprocessing library has such data structure that allows me to share data from GPU to another process in CPU?</p>
<p>Thanks.</p>
How realistic to implement redice for sparse.mm over COO tensors? Where to start?

alexj: <p>The massive gap in the sparse capability is that you need to use COO for Autograd to work, but CSR for <code>reduce</code> parameter to work. It sounds like the easiest way to solve it would be to migrate <code>reduce</code> implementation to COO, at least for CPU backend.<br>
Where would one start to at least investigate the complexity? Or the fix is already in the works?</p>
<p>Thank you</p>
Tracking statistics of operators without overwriting forward pass?

Whisper: <p>I’m looking to track output statistics of operators within networks. Take the following network:</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(3, 3)
    self.fc2 = nn.Linear(3, 3)
    self.fc3 = nn.Linear(3, 3)

  def forward(self, x):
    x = self.fc1(x)
    y = self.fc2(x)
    print("beginning addition")
    z = x + y
    return self.fc3(x)
</code></pre>
<p>I want to be able to track the range of values <code>z</code> takes on over time. I could always instantiate the model with a custom layer like <code>self.add = Add()</code>, but my goal is for other people to interact with what I’m working on, and forcing people to declare a new module for these operations isn’t something I’d like to have happen. I’d much rather have a way to automatically convert instances of <code>x+y</code> into <code>nn.Module</code> (like below) and then use forward hooks under the hood.</p>
<pre><code class="lang-auto">import torch.nn as nn

class Add(nn.Module):
  def __init__(self):
    super().__init__()

  def forward(self, x, y):
    return x + y
</code></pre>
<p>It also seems pretty straightforward to use the symbolic trace feature to generate and modify the network graph (also included below), but doing so overwrites my original forward pass. Therefore, any print statements I’ve added get removed, which I can’t really allow either.</p>
<p>I’ve put together a minimal working example that includes a symbolic trace, however is there any other better (general) approach that retains the original forward function, just with all instances of “+” replaced with Add()? This way, users don’t need to modify the model instantiations and print statements remain?</p>
<pre><code class="lang-auto">import operator
import torch.fx as fx

def replace_add(model):
  traced = fx.symbolic_trace(model)

  num_adds = 0
  for node in traced.graph.nodes:
    if node.op == "call_function" and node.target == operator.add:
      module_instance = Add()
      module_name = f"add_{num_adds}"

      traced.add_module(module_name, module_instance)
      with traced.graph.inserting_after(node):
        add_node = traced.graph.call_module(module_name, args=node.args)
        node.replace_all_uses_with(add_node)
        traced.graph.erase_node(node)
        num_adds += 1

  traced.recompile()
  return traced # no longer includes print statements

x = torch.randn(3)

original_net = Model()
original_out = original_net(x)

modified_net = replace_add(original_net)
print("we must see 'beginning addition' after this print")
modified_out = modified_net(x)
</code></pre>
Torch.ops.aten operators

trusira: <p>Hi,<br>
Where can I find documentation for the complete torch.ops.aten operator space?</p>
<p>Thanks!</p>

trusira: <p>I found this listed under IRs.</p>
<p><a href="https://pytorch.org/docs/stable/torch.compiler_ir.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torch.compiler_ir.html</a></p>

soulitzer: <p>That’s not really documented anywhere but you can check <code>native_functions.yaml</code> in the repo. The link that you gave is for the “core aten ops”, a subset of all aten ops, and is specific to export.</p>

Abhishek_Ghosh: <p>It took me some time to find out the file. Link to the file:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/500cbb5b9013f842ba0a15ef61bbf0b079ed99ff/aten/src/ATen/native/native_functions.yaml">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/500cbb5b9013f842ba0a15ef61bbf0b079ed99ff/aten/src/ATen/native/native_functions.yaml" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/500cbb5b9013f842ba0a15ef61bbf0b079ed99ff/aten/src/ATen/native/native_functions.yaml" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/500cbb5b9013f842ba0a15ef61bbf0b079ed99ff/aten/src/ATen/native/native_functions.yaml</a></h4>


      <pre><code class="lang-yaml"># See README.md in this directory for more guidance

# *********NB: _cast_* operators are DEPRECATED and will be removed
# eventually. These were previously used before TorchScript IR supported
# representing ScalarType's. They are now superseded by usage of
# `aten::to()`. The ops remain here for backward compatibility purposes.

# DEPRECATED. DO NOT USE
- func: _cast_Byte(Tensor self, bool non_blocking=False) -&gt; Tensor
  variants: function

# DEPRECATED. DO NOT USE
- func: _cast_Char(Tensor self, bool non_blocking=False) -&gt; Tensor
  variants: function

# DEPRECATED. DO NOT USE
- func: _cast_Double(Tensor self, bool non_blocking=False) -&gt; Tensor
  variants: function

# DEPRECATED. DO NOT USE
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/500cbb5b9013f842ba0a15ef61bbf0b079ed99ff/aten/src/ATen/native/native_functions.yaml" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Best way to support multiple loss/metrics without touching the source code

ado_sar: <p>I am trying to build a model which can be trained/evaluaed with various combinations of loss functions/metric functions. E.g. I want a model of the form:</p>
<pre data-code-wrap="python"><code class="lang-python">class Model(nn.Module):
    def __init__(self, loss_fn, metric_fn):
    ...
</code></pre>
<p>However, since the signature of loss functions (e.g. <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html" rel="noopener nofollow ugc">Cross Entropy</a>) is different than these of a metric (e.g. <a href="https://lightning.ai/docs/torchmetrics/stable/classification/accuracy.html#functional-interface" rel="noopener nofollow ugc">Accuracy</a>), one should reshape <code>preds</code> and <code>targets</code> according to the loss/metric signature. That is, the following might not always work:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = loss_fn(input=preds, target=target)
metric = metric_fn(preds=preds, target=target)
</code></pre>
<p>since the required shapes might be different in <code>loss_fn</code> compared to <code>metric_fn</code>.</p>
<p>This means that the source code should updated each time a different loss and/or metric is to be used. Obviously, this is something we would like to avoid. Is there any solution that I am missing or I need to handle each special case in the source code?</p>

ptrblck: <p>I don’t know if I understand your concern about “source code” changes correctly, but yes the changes have to land somewhere in your script. If you don’t want to add them to the model, you could add them into e.g. the training loop.</p>

gradientlord: <p>Maybe have a look at hooks?<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html</a></p>
Using LSTM to output one classification per timestep

r_m_rbt: <p>I’m trying to use an LSTM to detect anomalies in timeseries data</p>
<ol>
<li>5 inputs/features per timestep</li>
<li>Single True/False classification output at each timestep</li>
<li>Pattern in question can be 5 to 8 timesteps long</li>
<li>Pattern can occur multiple times in a large stream of timeseries data</li>
</ol>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

# Define LSTM classifier model
class LSTMClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMClassifier, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        out, _ = self.lstm(x, (h0, c0))
        out = self.fc(out[:, -1, :])
        return out

# Input - 100 examples containing 5 data points per timestep (where there are 10 timesteps)
X_train = np.random.rand(100, 10, 5)
# Output - 100 examples containing 1 True/False output per timestep to match the input
y_train = np.random.choice(a=[True, False], size=(100, 10))  # Binary labels (True or False)

# Convert data to PyTorch tensors
X_train_tensor = torch.tensor(X_train, dtype=torch.float32)
y_train_tensor = torch.tensor(y_train, dtype=torch.bool)

# Define model parameters
input_size = X_train.shape[2] # 5 inputs per timestep
hidden_size = 8 # Pattern we are trying to detect is usually 8 timesteps long
num_layers = 1
output_size = 1 # True/False

# Instantiate the model
model = LSTMClassifier(input_size, hidden_size, num_layers, output_size)

# Define loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Train the model
num_epochs = 100
for epoch in range(num_epochs):
    optimizer.zero_grad()
    outputs = model(X_train_tensor)
    loss = criterion(outputs, y_train_tensor)
    loss.backward()
    optimizer.step()
    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item()}')

# Test the model
X_test = np.random.rand(10, 10, 5) # Generate some test data - same dimensions as input
X_test_tensor = torch.tensor(X_test, dtype=torch.float32)
with torch.no_grad():
    predictions = model(X_test_tensor)
    predicted_outputs = torch.argmax(predictions, dim=1)
    print("Predicted Outputs:", predicted_outputs)
</code></pre>
<p>Unfortunately it seems like CrossEntropyLoss only supports 0D or 1D target tensors - how can I achieve the desired output dimensions?</p>
<p>Thanks!!</p>
How can i apply Lecun weight initialization for my Linear Layer?

hoangle_tttm: <p>Im using SELU as my activation function. As i have known pytorch is using He as weight initialization for Linear Layer. How can i apply Lecun weight initialization for my Linear Layer?</p>

ptrblck: <p>You can apply the <code>torch.nn.init</code> method (or any other custom weight initialization) on the modules directly or e.g. via <code>model.apply()</code> and a <code>weight_init</code> method as described in <a href="https://discuss.pytorch.org/t/initialising-weights-in-nn-sequential/76553/2">this post</a>.</p>

hoangle_tttm: <p>Thank you very much!</p>

guacamole: <p>Hoping someone might be able to elaborate a little further on this as I’m trying to initialize a Conv2d layer’s weights with Lecun Normal (for selu), but torch.nn.init doesn’t seem to have a lecun_normal_() function. Is there a recommended way of achieving this?</p>
Preferred way to update/set parameters in custom optimizer?

qq-me: <p>Before posting a query, check the <a href="https://discuss.pytorch.org/c/faq/30">FAQs</a> - it might already be answered!</p>
<p>I have an optimizer that I need to set model parameters to a new tensor.</p>
<p>I have found that on GPU</p>
<pre data-code-wrap="py"><code class="lang-py">param.set_(new_weights)
</code></pre>
<p>is about 40% faster than</p>
<pre data-code-wrap="py"><code class="lang-py">param.clone_(new_weights)
</code></pre>
<p>And then I found that</p>
<pre data-code-wrap="py"><code class="lang-py">param.data = new_weights
</code></pre>
<p>Is a bit faster even.<br>
So I have decided to use it, that being said, I am not sure if it is better, like more stable, etc, and if my tests were robust enough.</p>
<p>And then there is a question of how to best do stuff like subtracting gradients. Is this preferred?</p>
<pre data-code-wrap="py"><code class="lang-py">p.sub_(p.grad)
</code></pre>
<p>Official implementations of optimizers use <code>_foreach_add_</code> though. And I don’t know what that function uses, so I am unsure what method that may be equivalent to.</p>

AlphaBetaGamma96: <p>I’d avoid using <code>param.data = new_weights</code> as that doesn’t copy over any of the meta data of the particular tensor and is deprecated, I believe.</p>
Am I permitted to use functions such as `torch._foreach_add`

qq-me: <p>Before posting a query, check the <a href="https://discuss.pytorch.org/c/faq/30">FAQs</a> - it might already be answered!</p>
<p>This function <code>torch._foreach_add</code> is not mentioned anywhere, like on documentation, etc, except a few of foreach functions are mentioned in the Foreach section of this page <a href="https://pytorch.org/docs/stable/torch.html" class="inline-onebox" rel="noopener nofollow ugc">torch — PyTorch 2.3 documentation</a>, but most like <code>_foreach_add</code> are not mentioned there either, so I am unsure whether I am allowed to use them, or if they are subject to change, etc. I do notice that the official optimizer implementations use them, and as I understand, they are a bit faster than doing for loops. So is it okay if I use that?</p>

ptrblck: <p>These methods are tagged as internal/private indicated by the underscore at the front of the function names, since their interface can easily change and break.<br>
You can of course use these, but should not depend on backwards compatibility and might need to change some calls in future versions if their interfaces change.</p>
PyTorch with Metal Support Contribution Guidance

ashwani_rathee: <p>I want to specifically contribute to Metal support for PyTorch(as I only have a mac rn) especially low level primitives section in C++ like the ones here: <a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native/metal" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/metal at main · pytorch/pytorch · GitHub</a></p>
<p>I have started attempting to build PyTorch from source. Can anyone guide what’s a good first step and can anyone guide?</p>
<p>I have previous open source experience in Julia in GSOC,OSPP etc so comfortable with navigating problems with some mentorship. My work experience was C/C++ last year hence the interest.</p>
Have a question on `W_ii @ x_t` in lstm

Seungjun_Lee: <p>I have a question of the shape of weights of lstm.</p>
<p>If the input to the lstm layer is <code>(batch_size, seq_length, input_size)</code>, then <code>x_t</code> would be <code>(batch_size, input_size)</code>.</p>
<p>And as the documentation say so, the shape of <code>W_ii</code> is <code>(hidden_size*4, input_size)</code>.</p>
<p>But here how possibly matrix multiplication of <code>W_ii</code> and <code>x_t</code> can be happen?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a97e2c35484167dacc9481dad75a0998de8846d6.png" alt="Screenshot 2024-07-20 at 12.32.47 PM" data-base62-sha1="obp10rdQwiH5IgCgXrQFKF0Dgc6" width="371" height="146"></p>
<p>What did I miss here? Can anyone help me please</p>