akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
There is no option pytorch for CUDA 12.2 and latest 12.3

username: minh282906
cooked: <p>Hi,</p>
<p>I want to install pytorch with GPU in WSL Linux Ubuntu in my Windows computer. When I go to the start locally, it only has option for CUDA 11.8 and CUDA 12.1. I use CUDA 12.2, which option should I choose.</p>
<p>Thanks,<br>
Minh Nguyen</p>

username: ptrblck
cooked: <p>The PyTorch binaries ship with their own CUDA runtime dependencies and you can select any version you want. For Hopper GPUs we recommend sticking to CUDA 12. If you need CUDA 12.3 for some reasons you would need to build from source.</p>

username: minh282906
cooked: <p>Thank you very much.</p>

username: Hyperx_zero
cooked: <p>“If you need CUDA 12.3 for some reasons you would need to build from source.”</p>
<p>How can I build from source? I don’t know what build from source means.</p>
<p>I will explain my problem from the beginning. I recently downloaded pytorch for AI work. This morning when I looked at pytorch, I saw that it was using the CPU (also I didn’t have CUDA on my computer). Then, I deleted all pytorch versions and all pytorch related packages from my computer, downloaded the latest CUDA (with CUDA toolkit) for my video card (RTX 3050 8GB) and got version “12.3.103” (aka “12.3”). But when I looked at pytorch’s own website, I saw that the latest version available for download is 11.8 CUDA. Is there a torch for version 12.3 CUDA? Where can I download it?</p>

username: ptrblck
cooked: <p>You don’t need a locally installed CUDA toolkit as PyTorch ships with all needed CUDA runtime dependencies as already mentioned. Only a properly installed NVIDIA driver is needed.</p>

username: beishangeyu
cooked: <p>Hi, i use <code>pip install torch</code> and my torch version = 2.4, and i got cuda 12.3. What should i do to use GPU?</p>

username: ptrblck
cooked: <p>Running <code>pip install torch</code> will download the latest PyTorch binary with CUDA support and is thus sufficient to run PyTorch applications on your GPU assuming the NVIDIA driver is properly installed.</p>
Visualize Model

username: pascalm
cooked: <p>Hello,</p>
<p>I’d like to visualize a preprainted bert model that’s very tiny. It inherits <a href="https://github.com/huggingface/transformers/blob/eb5b968c5d80271ecb29917dffecc8f4c00247a8/src/transformers/modeling_utils.py#L1298" class="inline-onebox" rel="noopener nofollow ugc">transformers/src/transformers/modeling_utils.py at eb5b968c5d80271ecb29917dffecc8f4c00247a8 · huggingface/transformers · GitHub</a></p>
<p>I tried using tensorboard but wasn’t successfull so far. I can get a summary but I’d like a bit more details. Any package capable of that?</p>
<p>Thanks in advance</p>

username: Soumya_Kundu
cooked: <aside class="onebox githubrepo" data-onebox-src="https://github.com/jessevig/bertviz">
  <header class="source">

      <a href="https://github.com/jessevig/bertviz" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/1c752cc53cb442565f7feba4595e8ce3/jessevig/bertviz" class="thumbnail">

  <h3><a href="https://github.com/jessevig/bertviz" target="_blank" rel="noopener nofollow ugc">GitHub - jessevig/bertviz: BertViz: Visualize Attention in NLP Models (BERT,...</a></h3>

    <p><span class="github-repo-description">BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.) </span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Index out of range after using Subset()

username: trashpanda
cooked: <p>Hey,</p>
<p>I am using a custom Dataset class because I need the index in my data. But if I subset the data, I can’t use it anymore. I need to remove parts of the data by index, and I think the <strong>getitem</strong> might be the reason. But I am really stuck and would love some hints <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Subset
from torchvision import datasets
N_train = 300


def make_dataset_with_index(
    dataset_class,
):
  
    class DatasetWithIndex(dataset_class):
        def __init__(self, *args, **kwargs):
            super(DatasetWithIndex, self).__init__(*args, **kwargs)

            self.dataset = dataset_class(*args, **kwargs)

        def __getitem__(self, index):
            image, label = super(DatasetWithIndex, self).__getitem__(index)
            return (image, label, index)

        def __len__(self):
            return len(self.dataset)
    return DatasetWithIndex



train_dataset = make_dataset_with_index(
        datasets.MNIST,
    )(
        "/tmp/mnist",
        train=True,
        download=True,
    )


    # subsampling to reduce data
train_dataset, _ = torch.utils.data.random_split(
        train_dataset,
        [N_train, len(train_dataset) - N_train],
        generator=torch.Generator().manual_seed(1),
    )

indices_to_remove = [35845,27522,33254]  # These are indices within the original dataset

new_train_indices = [i for i in train_dataset.indices if i not in indices_to_remove]

# Create new Subsets
new_train_dataset = Subset(train_dataset, new_train_indices)

loader = torch.utils.data.DataLoader(new_train_dataset, batch_size=32, shuffle=False)
for batch in loader:
    print(batch)
</code></pre>
How to run multiple models in different C++ threads with different num_threads

username: James_Pinkerton
cooked: <p>Hi. I have a multi-threaded program with a different torch model in each thread. For some of the models I’d like to pin the thread to multiple cores and use set_num_threads(2) or more, and for some smaller models I’d like to pin them to one core and set_num_threads(1). However, my understanding is that this is a global property. Is there any way to accomplish this?</p>
<p>Thanks,<br>
James</p>
Replicate training of wav2vec2 available?

username: pytorcher
cooked: <p>For torchvision there are tutorials on how the pretrained models are created so that the results can be replicated <a href="https://github.com/pytorch/vision/tree/main/references/classification#mobilenetv3-large--small" rel="noopener nofollow ugc">here</a>.</p>
<p>I was wondering if the same thing exists for torchaudio.  Specifically looking to train <a href="https://pytorch.org/audio/main/generated/torchaudio.models.Wav2Vec2Model.html#torchaudio.models.Wav2Vec2Model" rel="noopener nofollow ugc">wav2vec2</a> from scratch.</p>
Parameter matched Transformer is slower than RNN

username: psoulos
cooked: <p>I shared a script below that measures the inference speed of parameter matched Transformers and LSTMs, both with the same number of layers. I expected that the Transformer would be faster as the sequence length increases since the Transformer can use teacher forcing for better parallelizability. However, I find that the LSTM is faster. Is this expected? I tested on both A100 and V100 GPUs.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import tqdm

ARCHITECTURE = 'transformer'
transformer_dim = 256
lstm_dim = 312
nhead = 4
num_layers = 5

if ARCHITECTURE == 'transformer':
    d_model = transformer_dim
    encoder_layer = nn.TransformerEncoderLayer(
        d_model=transformer_dim,
        nhead=nhead,
        dim_feedforward=transformer_dim*4,
        dropout=0,
        batch_first=True
    )
    model = nn.TransformerEncoder(
        encoder_layer=encoder_layer,
        num_layers=num_layers
    )
elif ARCHITECTURE == 'lstm':
    d_model = lstm_dim
    model = nn.LSTM(
        input_size=lstm_dim,
        hidden_size=lstm_dim,
        num_layers=num_layers,
        batch_first=True,
        dropout=0
    )

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = model.to(device)

parameters = sum(p.numel() for p in model.parameters())
print(f'{parameters} parameters')

batch_size = 128
sequence_length = 1024
warmup_steps = 50
steps = 1000
log_frequency = 100

print('Warming up')
warmup_steps = tqdm.tqdm(range(warmup_steps))
for i in warmup_steps:
    batch = torch.rand((batch_size, sequence_length, d_model), device=device)
    out = model(batch)

torch.cuda.synchronize()
print('Finished warming up')
start = torch.cuda.Event(enable_timing=True)
start.record()
steps = tqdm.tqdm(range(steps))
for i in steps:
    batch = torch.rand((batch_size, sequence_length, d_model), device=device)
    out = model(batch)
    if i % log_frequency == 0 and i &gt; 0:
        end = torch.cuda.Event(enable_timing=True)
        end.record()
        torch.cuda.synchronize()
        elapsed_time_ms = start.elapsed_time(end)
        elapsed_time_per_step_ms = elapsed_time_ms / log_frequency
        print(f'Elapsed time per step: {elapsed_time_per_step_ms}ms')
        start = torch.cuda.Event(enable_timing=True)
        start.record()
</code></pre>
Calculating loss consumes a lot of RAM(cpu)

username: asz18351
cooked: <p>I wrote a custom loss and used GPU to train my model, but when calculating the loss, the CPU RAM usage continued to rise. I would like to know what caused this problem. (Is it related to computational graphs?)</p>
<p>The following is the code to reproduce the problem. Similar operations are performed in the loss function I designed.</p>
<pre data-code-wrap="code"><code class="lang-code">import torch
import sys
import tqdm

b = torch.zeros(5).cuda()
d1 = 3*torch.zeros(1,requires_grad=True).cuda()
d2 = 3*torch.zeros(1,requires_grad=True).cuda()
d3 = 3*torch.zeros(1,requires_grad=True).cuda()
d4 = 3*torch.zeros(1,requires_grad=True).cuda()
d5 = 3*torch.zeros(1,requires_grad=True).cuda()
d6 = 3*torch.zeros(1,requires_grad=True).cuda()
d7 = 3*torch.zeros(1,requires_grad=True).cuda()
d8 = 3*torch.zeros(1,requires_grad=True).cuda()

for i in tqdm(range(1000000)):
    b +=(d1*d2*d3*d4*d5*d6*d7*d8)</code></pre>

username: ptrblck
cooked: <p>Yes, the computation graph will be created and attached to <code>b</code> in each iteration thus storing all needed metadata as well. How large is the increase in each iteration?</p>

username: asz18351
cooked: <p>Sorry for the late reply.<br>
After 1,000,000 iterations, RAM usage increased by 12.2GB.</p>

username: ptrblck
cooked: <p>Thanks for checking! I would assume this increase might be expected as it comes down to <code>12.2 * 1024**2 / 1000000 = 12.7926272 kB</code> per iteration, which might fit the metadata. CC <a class="mention" href="/u/alband">@albanD</a> to correct me in case ~12kB are unexpected.</p>

username: asz18351
cooked: <p>Thank you for your reply. So if my custom loss function requires a similar calculation process (finally requiring backpropagation), is there a better way to write the program that can reduce RAM usage?</p>
How can I return a list of tensors in a Pytorch autograd function?

username: Vedant_Roy
cooked: <p>Is it possible to write a custom autograd function that returns a <em>list</em> of tensors? For context, I’m trying to manually write the backward pass for <code>all_gather</code>.</p>
<p>This code:</p>
<pre><code class="lang-auto">class AllGatherFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, tensor: torch.Tensor, dim, reduce_dtype):
        ctx.reduce_dtype = reduce_dtype
        ctx.dim = dim

        output_list = list(torch.empty_like(tensor) for _ in range(_CONTEXT_PARALLEL_GROUP_SIZE))
        dist.all_gather(output_list, tensor, _CONTEXT_PARALLEL_GROUP)
        return output_list
        # return torch.cat(output_list, dim=dim)
</code></pre>
<p>fails because I can’t return a list of tensors.</p>

username: KFrank
cooked: <p>Hi Vedant!</p>
<aside class="quote no-group" data-username="Vedant_Roy" data-post="1" data-topic="208810" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vedant_roy/48/51087_2.png" class="avatar"> Vedant_Roy:</div>
<blockquote>
<p>Is it possible to write a custom autograd function that returns a <em>list</em> of tensors?</p>
</blockquote>
</aside>
<p>Yes.  For example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.4.0'
&gt;&gt;&gt; class Func (torch.autograd.Function):
...     @staticmethod
...     def forward (ctx, t):
...         return [t**2, t**3]
...
&gt;&gt;&gt; Func().apply (torch.arange (3.))
[tensor([0., 1., 4.]), tensor([0., 1., 8.])]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
PyTorch Conference 2024 - 30% Discount on Registration for our Discuss Community!

username: kyliewd
cooked: <p>Reminder: PyTorch Conference 2024 is 20 days away! We hope you’ll join us in San Francisco for talks on distributed training, responsible AI, edge and more.</p>
<p>Because you’re a part of our PyTorch Developer Community, The PyTorch Foundation is thrilled to <strong>offer a 30% discount</strong> for this year’s conference, ensuring everyone has a chance to join us! Use code PYTORCH when registering: <a href="https://hubs.la/Q02LQ1Np0" class="inline-onebox" rel="noopener nofollow ugc">Register | LF Events</a></p>
<p>See you there <img src="https://discuss.pytorch.org/images/emoji/apple/sparkles.png?v=12" title=":sparkles:" class="emoji" alt=":sparkles:" loading="lazy" width="20" height="20"></p>

username: ptrblck
action_code: pinned_globally.enabled
Uncontrolled growth of parameter values leads to NaN loss

username: mattiacampana
cooked: <p>I am conducting some Federated Learning experiments using a simple fully-connected PyTorch model for classification, with <code>CrossEntropyLoss()</code> as the loss function. Instead of averaging the model parameters as in FedAVG, I’m summing them at each training round. After a certain number of rounds, the loss value starts to become NaN, and the accuracy drops sharply from 0.9 to 0.2, with no recovery. However, when I restrict the parameter values to the range [−1,1] using <code>torch.clamp(params, min=-1, max=1)</code> at each iteration, the model continues to improve as expected.</p>
<p>Is this issue related to a limitation in PyTorch, or is it a theoretical problem (such as exploding gradients) that I might be overlooking?</p>

username: Soumya_Kundu
cooked: <p>I suspect it’s mostly exploding gradients as you said.</p>
<p>As the gradients increase, they might start throwing nans. Can also occur by exceeding the data type limits (although not sure on this).</p>
<p>Clamping essentially prevents this by sticking to a range.</p>

username: mattiacampana
cooked: <p>Thank you for confirming my suspects.</p>
Tried to allocate 784.00 MiB (GPU 0; 23.99 GiB total capacity; 7.15 GiB already allocated; 13.69 GiB free; 7.99 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation

username: Siladittya_Manna
cooked: <p>I have come across several posts which discuss the same error, but in all posts I have seen that the free memory is less than that PyTorch is trying to allocate.</p>
<p>In my case the free memory is showing more than 13 GB, while it is trying to allocate 784MB.</p>
<p>I have tried using a lower batch size (128) down from 256, but I received the following error</p>
<p><code>torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 23.99 GiB total capacity; 10.11 GiB already allocated; 11.45 GiB free; 10.23 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>I don’t seem to understand the main issue here.</p>
<p>UPDATE:: Decreasing the number of workers from 8 to 1 solved the issue. Later increased to 4, and still working.</p>
<p>Later increased to 8, which worked this time.</p>
<p>Again failed to run in a later run. Had to decrease the number of workers again.</p>

username: Siladittya_Manna
cooked: <p>Decreasing the number of workers did not work today!</p>

username: Soumya_Kundu
cooked: <p>Can you make a mock simulation code while tracking your memory using nvidia-smi/pytorch profiler/nvitop.</p>
<p>Maybe something in the background is getting generated and not being removed or something completely external.</p>
Can some please provide exemplary PyTorch transformer to consume sequences of variable lengths?

username: JimW
cooked: <p>I have some audio sequence with 6 channels. I hope to predict one class for each input sequence.<br>
The shape of the input sequences looks like [batch_size,  6, Length], where “Length” can vary from 20 to 200.</p>
<p>I am new to transformers. Could someone please provide an exemplary PyTorch definition of transformer to consume such data?  Thanks in advance.</p>
Torch shm manager blowing up my computer

username: Rishi_Malhotra
cooked: <p>I am training CNNs, which are a few layers deep. My laptop has ten cpu cores, verified using os.cpu_count(), and I’m using ten workers for my dataloader which could be the issue.</p>
<p>I run my training through the command line with<br>
<code>python train.py</code></p>
<p>When i hit ctrl+z on the command line, to halt a training run, there were approx three thousand “torch shm manager” processes on the activity monitor on my mac, each consuming 0.5% CPU. With great lag, I managed to force quit all of them. When I use a keyboard interrupt, ctrl+c, to exit the training, this issue doesn’t occur.</p>
<p>Last night and the night before, i wasn’t able to fight the lag, causing my m1 pro mac display to freeze followed by a shut down. Is there a way for me to use ctrl+z and not blow up my computer?</p>

username: Nahal_Mirzaie
cooked: <p>I have the same problem with m3 and running a pytorch code with vscode on it. Did you find any solution?</p>
How to compute covariance with torch.cov?

username: Brando_Miranda
cooked: <p>I noticed that the numpy and torch interface are not the same:</p>
<p><a href="https://numpy.org/doc/stable/reference/generated/numpy.cov.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://numpy.org/doc/stable/reference/generated/numpy.cov.html</a></p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.cov.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.cov.html</a></p>
<pre><code class="lang-auto">torch.cov(input, *, correction=1, fweights=None, aweights=None) → Tensor
</code></pre>
<pre><code class="lang-auto">numpy.cov(m, y=None, rowvar=True, bias=False, ddof=None, fweights=None, aweights=None, *, dtype=None)[source]
</code></pre>
<p>Which means I can’t easily compute the covariance of two batches…since torch.conv does not take y. Is this a mistake?</p>

username: eqy
cooked: <p>Per the numpy docs, passing y separately should be equivalent to stacking it to “m” or (cat’ing in torch parlance):<br>
<a href="https://numpy.org/doc/stable/reference/generated/numpy.cov.html" rel="noopener nofollow ugc">numpy.cov — NumPy v1.21 Manual</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.randn(128,128,dtype=torch.double)
&gt;&gt;&gt; y = torch.randn(128,128,dtype=torch.double)
&gt;&gt;&gt; X = torch.cat((x, y), axis=0)
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; np_res = np.cov(x.numpy(), y.numpy())
&gt;&gt;&gt; res = torch.cov(X)
&gt;&gt;&gt; (res.shape, torch.tensor(np_res).shape)
(torch.Size([256, 256]), torch.Size([256, 256]))
&gt;&gt;&gt; torch.allclose(res, torch.tensor(np_res))
True
&gt;&gt;&gt;
</code></pre>

username: Brando_Miranda
cooked: <p>I just did this:</p>
<pre><code class="lang-auto">def cov(x: Tensor, y: Optional[Tensor] = None) -&gt; Tensor:
    """
    Compute covariance of input

    :param x: [M, D]
    :param y: [M, D]
    :return:
    """
    if y is not None:
        y = x
    else:
        assert x.size(0) == y.size(0)
    # - center first
    x = center(x, dim=0)
    y = center(y, dim=0)
    # - conv = E[XY] is outer product of X^T Y or X Y^T depending on shapes
    sigma_xy: Tensor = x.T @ y
    return sigma_xy
</code></pre>
<p>should be the same no?</p>

username: hiyyg
cooked: <p>Are there any efficient ways to compute the covariance for a large number of vectors? <code>torch.cov</code> will result in OOM in such case.</p>
Using gradient optimization within larger gradient optimization training pipeline

username: nchen9191
cooked: <p>Hi,</p>
<p>I had a question on how to implement a training pipeline in torch. I was wondering how I could implement running gradient descent in a single module thats part of a larger multi-module training pipeline. In other words, I want the inner gradient descent steps as a part of the larger computational graph.</p>
<p>For example, I have a training pipeline consisting of 5 modules:</p>
<ol>
<li>preprocess input batch (x, y_ground_truth)</li>
<li>perform N steps of grad descent to find v, s.t. minimize [v + A - NN(v)] where A is a constant.</li>
<li>postprocess(v) → y_pred</li>
<li>compute loss L = mean((y_pred - y_ground_truth)^2)</li>
<li>backprop to train NN with L.backward(), optimizer.step()</li>
</ol>
<p>In this setup, you can see that module 2 uses the NN and in order to do grad descent, I would need to compute gradients that depend on the NN. Essentially, I want to know how to build module 2. Any help would be greatly appreciated, thanks!</p>
How to Control ATen Ops for `sdpa` in PyTorch Export

username: siahuat0727
cooked: <p>I would like to know how to control which ATen operation is used during <code>torch.export</code> when exporting the <code>scaled_dot_product_attention</code> operation. It seems that sometimes I get <code>aten._scaled_dot_product_flash_attention_for_cpu</code>, while other times I get <code>aten.scaled_dot_product_attention</code>. I’m not sure what factors cause this difference.</p>
<p>Could someone please help point me to the related documentation? Thank you.</p>

username: ptrblck
cooked: <p>You could try to select the desired algorithm via <a href="https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.sdp_kernel"><code>torch.backends.cuda.sdp_kernel</code></a> but unsure if this would work with <code>export</code> and for your use case.</p>
How to prevent memory oscillation when using Pytorch?

username: Vedant_Roy
cooked: <p>If I use too much memory in Pytorch, I notice thrashing. E.g, the GPU memory usage fluctuates when calling <code>nvidia-smi</code> or <code>gpustat</code>. Ideally, I’d force Pytorch to never de-allocate memory, since this kills performance. Is there a way I can do this?</p>

username: ptrblck
cooked: <p>You could try to enable <code>expandable_segments</code> as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf">here</a>.</p>
Custom Dataset Memory Issues

username: mfc2496
cooked: <p>I have a custom dataset that I use to load multi-modal npz files and create samples for my model. The way I do it currently is:</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
While True: 
   source = self.data[idx]
   for _ in 10: 
      target = select_target(self.data) 
     if (diff(source,target) &lt; .10):
        continue
     else:
       sample = {}
       if self.use_m1:                      
          sample['m1'] = torch.stack([
              to_tensor(normalize_images(np.transpose(cv2.resize(read_npz(m1_input), dsize=(256, 256), interpolation=cv2.INTER_AREA)[..., :3], (2, 0, 1)), max=255)) for m1_input in m1_dirs
          ], dim=0)
       if self.m2:     
          sample['m2'] = torch.stack([
              to_tensor(normalize_images(np.expand_dims(cv2.resize(sem_image, dsize=(256, 256), interpolation=cv2.INTER_NEAREST), axis=0), max=40)) for m2_input in m2_dirs
          ], dim=0)
                          
       if self.config["transform"] and random.random()&lt;0.5 and self.train:
        sample = self.transformations(sample)
       else: 
         if self.m1:
            sample['m1'] = torch.cat([m1_tensor for m1_tensor in sample['m1']], dim=0)
        if self.m2: 
            sample['m2'] = torch.cat([m2_tensor for m2_tensor in sample['m2']], dim=0)
                    
  return sample

</code></pre>
<p>I apply transformations as follows:</p>
<pre><code class="lang-auto">def transformations(self,input):
      i,j,h,w = self.crop.get_params(input['target_material'][0], scale=(0.7, 1.0), ratio=(1.0, 1.0))
      if self.use_modality1:
                input["m1"] = torch.cat([self.color_jitter(TF.resized_crop(sample, i, j, h, w, size=(256, 256))) for sample in input['m1']], dim=0)
      if self.use_semantic:
                input["m2"] = torch.cat([TF.resized_crop(sample, i, j, h,w,size=(self.img_size, self.img_size), interpolation=TF.InterpolationMode.NEAREST) for sample in input['m2']], dim=0)
       return input
</code></pre>
<p>I have a list of directories that I read from in getitem. I have multiple modalities to load up (6 to be exact) and each modality has a total of 4 images associated with it. I keep getting OOM errors around the 5-6 epochs and I cannot figure out where the issue is. I don’t get CUDA OOM - just seems like memory becomes an issue with this custom dataset.</p>
<p>Any help would be appreciated.</p>
<p>Edit: I’ve done some memory tracing and figured out that the issue is indeed in the dataloader. Memory keeps increasing with every getitem() call.</p>
Finen tuning Llama with using pytorch in colab

username: merto58
cooked: <p>Hello. Im having an issue with fine tuning Llama 3.1. Google cloab crashes whenever I try to train on my custom dataset.  I think its about pytorch and tensorflow versions. Im tried to use gpu and tpu both gave me error. And the memory not working when I start the train my model.Already thx a lot…</p>

username: Soumya_Kundu
cooked: <p>Could you provide the exact error if possible? Maybe a colab timeout.</p>
Constraining parameter vector to unit sphere

username: dherrera1911
cooked: <p>I am creating a class for my model using nn.Module. It’s a simple model with only a set of linear filters as parameters. The parameters are <code>self.f</code>, defined by <code>self.f = nn.Parameter(f_init)</code> where <code>self.f</code> has shape <code>(n_filters, n_dims)</code>.</p>
<p>The issue is that I want to constrain the filters to have unit norm. So far I am doing this with the wonderful <code>geotorch</code> <a href="https://github.com/lezcano/geotorch" rel="noopener nofollow ugc">package</a>, using the call <code>geotorch.sphere(self, "f")</code>. Then I don’t have to think of this constraint when doing back-propagation.</p>
<p>My question is whether there is a simple way of implementing this constraint without <code>geotorch</code>, since it seems simple enough, and to avoid an extra required package that is not in pip. Naively, it occurs to me that I could just divide <code>self.f</code> by its norm after every update step. It is not obvious to me though whether this would mess with my gradient and learning somehow, although I don’t see why they should.</p>
<p>So, does it sound right to impose the unit-length constrain by dividing the vectors after each update step?</p>

username: ptrblck
cooked: <p>A quick check of the code indicates <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html"><code>torch.nn.utils.parametrize.register_parametrization</code></a> is used which is then registering <a href="https://github.com/lezcano/geotorch/blob/ba38d406c245d609fee4b4dac3f6427bf6d73a8e/geotorch/sphere.py#L109"><code>Sphere</code></a> on the parameter, so you could try to use the same approach.</p>

username: dherrera1911
cooked: <p>I solved the problem using the parametrizations tool in Pytorch. I wrote a blogpost about how you’d implement the unit vector constrain I mentioned, and also how to constrain a matrix to be symmetric positive definite <a href="https://dherrera1911.github.io/posts/2024/08/constrained-learning-with-pytorch-parametrizations/" class="inline-onebox" rel="noopener nofollow ugc">Easy constrained optimization in Pytorch with Parametrizations - Daniel Herrera-Esposito</a></p>

username: ptrblck
cooked: <p>Really cool post! Thanks for sharing it here.</p>
Error with distributed training

username: rschireman
cooked: <p>Hi all,</p>
<p>I’m attempting to distribute the training of my network with the distributed data parallel strategy. However, I’m getting a runtime error:</p>
<p>RuntimeError: The size of tensor a (3) must match the size of tensor b (0) at non-singleton dimension 1</p>
<p>I found the function that raised the error ( _distributed_broadcast_coalesced(self, tensors, buffer_size, authoritative_rank=0)) and printed the shapes of the tensors going into the function and obtained this output:</p>
<p>torch.Size([30, 240])<br>
torch.Size([30])<br>
torch.Size([30, 30])<br>
torch.Size([30])<br>
torch.Size([30, 30])<br>
torch.Size([30])<br>
torch.Size([1, 30])<br>
torch.Size([1])<br>
torch.Size([30, 240])<br>
torch.Size([30])<br>
torch.Size([30, 30])<br>
torch.Size([30])<br>
torch.Size([30, 30])<br>
torch.Size([30])<br>
torch.Size([1, 30])<br>
torch.Size([1])<br>
torch.Size([30, 240])<br>
torch.Size([30])<br>
torch.Size([30, 30])<br>
torch.Size([30])<br>
torch.Size([30, 30])<br>
torch.Size([30])<br>
torch.Size([1, 30])<br>
torch.Size([1])<br>
torch.Size([1, 1])<br>
torch.Size([1, 16])<br>
torch.Size([1, 1, 1, 1])<br>
torch.Size([1, 1, 1, 1])<br>
torch.Size([1, 1, 4, 1])<br>
torch.Size([1, 1, 1, 8])<br>
torch.Size([3, 3])<br>
torch.Size([3, 3])<br>
torch.Size([0, 3])</p>
<p>I’m not really sure where the error is coming from</p>

username: normster
cooked: <p>Not sure if you’ve solved this already but I was running into the same error. The issue for me was a parameter with an empty dimension. It looks like your last parameter also has the same issue here. I believe the DDP constructor coalesces tensors in the same order that is returned by model.named_parameters() so you could use that to figure out which parameter specifically is empty.</p>

username: rschireman
cooked: <p>Hi Norman,</p>
<p>Thanks for your response! I haven’t solved this issue, but it does work with Pytorch Lightning’s implementation of the ‘Bagua’ distributed training strategy. In my model, it looks like my MSE loss has 0 paramaters - could this be causing the issue?</p>
<h2>
<a name="name-type-params-1" class="anchor" href="#name-type-params-1"></a>| Name      | Type       | Params</h2>
<h2>
<a name="h-0-h_network-sequential-91-k-1-c_network-sequential-91-k-2-s_network-sequential-91-k-3-mse-mseloss-0-4-nn-animodel-274-k-5-model-sequential-274-k-2" class="anchor" href="#h-0-h_network-sequential-91-k-1-c_network-sequential-91-k-2-s_network-sequential-91-k-3-mse-mseloss-0-4-nn-animodel-274-k-5-model-sequential-274-k-2"></a>0 | H_network | Sequential | 9.1 K<br>
1 | C_network | Sequential | 9.1 K<br>
2 | S_network | Sequential | 9.1 K<br>
3 | mse       | MSELoss    | 0<br>
4 | nn        | ANIModel   | 27.4 K<br>
5 | model     | Sequential | 27.4 K</h2>

username: Qun_Su
cooked: <p>I’m having the same problem, how did you solve it?</p>
Unsupported: setattr when exporting the WavLM model

username: vitreo12
cooked: <p>Hello everyone!</p>
<p>I have been trying to export the <a href="https://github.com/microsoft/unilm/blob/master/wavlm/WavLM.py" rel="noopener nofollow ugc">WavLM</a> model using <code>torch.export</code> but I’m hitting quite a lengthy error that ends with a</p>
<pre><code class="lang-auto">torch._dynamo.exc.Unsupported: setattr(UnspecializedNNModuleVariable(Conv1d), weight, ...)
</code></pre>
<p>This is apparently due to the <code>torch.nn.utils.weight_norm</code> operator at <a href="https://github.com/microsoft/unilm/blob/master/wavlm/WavLM.py#L526" rel="noopener nofollow ugc">this line</a>. I tried to workaround it using <code>torch.nn.utils.parametrizations.weight_norm</code> instead, which seems to work. However, this operator is not yet supported by <code>iree-turbine</code>, which is actually what I want to use to compile this model ultimately. So I’m wondering if there are other workarounds to use <code>torch.nn.utils.weight_norm</code> instead, or if this is a known issue that I should raise a GH issue for. I couldn’t find anything mentioning this specific error.</p>
<p>Thanks a lot!</p>
Large differences in training and inference results on two different machines

username: Wannes
cooked: <p>Hey y’all, I have seen some posts discussing this issue but none of the proposed solutions have worked for me thus far.</p>
<p>I trained a model (<a href="https://pytorch.org/vision/main/models/video_s3d.html" rel="noopener nofollow ugc">S3D</a>) and saved a checkpoint locally on machine 1, the final accuracy is supposedly 0.84 on my validation set, great! Now when using the same codebase, same seeds, and the same data to train the same model on machine 2 I get drastically different results: an accuracy of about 0.44, the learning curve went from a nice slope to a flatline, indicating that the model is just randomly guessing and not learning anything on machine 2.</p>
<p>I decided to debug the model by loading in my checkpoint on both machines and checking the predictions on both machines. I see slightly different numbers, possibly due to different rounding errors or FP precision. I am aware that the type of GPU differs between machines and that this might induce slight changes in FP operations.</p>
<pre><code class="lang-auto">Machine 1: array([7.2582060e-07, 9.9972206e-01, 1.5989746e-04, 2.1518445e-06,
       5.8096668e-07, 1.0367761e-06, 7.5574033e-05, 3.3640531e-06,
       1.9060394e-06, 1.7345132e-05, 1.1253899e-05, 3.9614688e-06],
      dtype=float32)

Machine 2: array([7.2582617e-07, 9.9972206e-01, 1.5989838e-04, 2.1518545e-06,
       5.8097061e-07, 1.0367791e-06, 7.5574528e-05, 3.3640756e-06,
       1.9060523e-06, 1.7345230e-05, 1.1253975e-05, 3.9614915e-06],
      dtype=float32)
</code></pre>
<p>This is the code I am running to verify the results. Could such tiny differences result in such a diverging training behavior? This seems unlikely to me. However, I feel like I’ve ticked all the boxes to make the training as reproducible as possible. Any ideas or advice would be greatly appreciated!</p>
<pre><code class="lang-auto">    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

    val_dataset = CVBDataset(
        annotations_path=val_annot_path,
        transform=test_transform,
    )

    pretrain_checkpoint = torch.load("./checkpoints/pretrain/s3d_20240826_1CBA9A/s3d_20240826_1CBA9A_seed43_best.pth", map_location="cpu")
    model.load_state_dict(pretrain_checkpoint["model"])
    model.to(device)

    model.eval()
    subset_dataset = data.Subset(val_dataset, [1,2,3])

    dataloader = data.DataLoader(
        subset_dataset,
        batch_size=3,
        shuffle=False,
        num_workers=1,
    )

    probabilities = []
    with torch.no_grad():
        for batch_idx, batch in enumerate(dataloader):
            videos, _ = batch
            videos = videos.to(device)
            outputs = model(videos)
            
            outputs = F.softmax(outputs, dim=1)
            outputs = outputs.cpu().numpy()

            print(outputs)
</code></pre>