akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory size of all tensors referenced by autograd graph?

vadimkantorov: <p>Is there a way to get statistics of memory used by activations/intermediate tensors in the autograd graph (after the forward pass completed loss computation)?</p>
<p>This is useful for debugging insufficient memory savings from autocast.</p>

ptrblck: <p>You could use forward hooks to check the size of the intermediate forward activations.</p>

vadimkantorov: <p>But it won’t take into account certain tensors being garbage-collected, inplace ops and some intermediate tensor saved_for_backward within the C++ ops, right?</p>

vadimkantorov: <p>Basically, the question is still on not easily observable memory consumption in combination with unclearness what gets actually stored as fp16 as a single copy and how/where casts are done and whether they are done on the fly or not. Autocast is being too “auto” and too magical <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>It might be good to somehow have tracing mode to understand at what point are casts executed (while in eager mode).</p>

ptrblck: <p>To check for casts and track operations you could use something like this:</p>
<pre><code class="lang-python">import torch
from torch.testing._internal.logging_tensor import LoggingTensorMode, capture_logs

model = torch.nn.Linear(10, 10).cuda()
x = torch.randn(1, 10).cuda()

with capture_logs(is_mode=True) as logs, LoggingTensorMode():
    with torch.autocast(device_type='cuda'):
        out = model(x)

for l in logs:
    print(l)
</code></pre>
<p>which should reduce the “magic”.<br>
I’m not aware of another way to check for intermediates created in the backend.</p>

vadimkantorov: <p>Curious, how will it show “cached” casts in autocast? What casts are “cached” in autocast? Casts of model parameters from fp32 to fp16, right? At what moments is “cache” invalidated or flushed?</p>
<p>Is autocast implemented as some sort of wrapper for function calls? E.g. if I call several functions on activation of a previous module, will there be two redundant different casts fp32-&gt;fp16 for two branches of computation? magic <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

vadimkantorov: <p>This logging might also be much better if these extensions were implemented: <a href="https://github.com/albanD/subclass_zoo/blob/main/logging_mode.py" class="inline-onebox" rel="noopener nofollow ugc">subclass_zoo/logging_mode.py at main · albanD/subclass_zoo · GitHub</a> (adding shape and dtype to the text trace)</p>
<p>Related issue: <a href="https://github.com/pytorch/pytorch/issues/81750" class="inline-onebox" rel="noopener nofollow ugc">Modernize logging tensor in torch.testing._internal · Issue #81750 · pytorch/pytorch · GitHub</a></p>
How to add attributes to a Tensor?

keyboardAnt: <p>What’s the most Pytorch-ich way to add an attribute to a Tensor? Is it necessary to implement a new class that inherits from the Tensor class? I would appreciate an example.</p>
<p>In the end, the goal is to allow access as follows:</p>
<pre><code class="lang-python">t: Tensor
attr_to_set: Any
t.attr = attr_to_set
attr: Any = t.attr
</code></pre>
<hr>
<p>So far, I’ve only found these two posts.</p>
<ol>
<li>A <a href="https://discuss.pytorch.org/t/subclassing-torch-tensor/23754/4">suggestion</a> to directly inherit from the Tensor class. The problem is that it seems like such a new subclass should implement methods like <code>__new__</code>, <code>clone</code>, and <code>to</code>, rather than only <code>__init__</code>. Since there’s no official answer in that post, I’m not sure that there aren’t additional functions that require handling. I don’t want to create a new Tensor subclass that breaks the Tensor interface.</li>
<li>A <a href="https://discuss.pytorch.org/t/typeerror-expected-cpu-got-cuda-when-sublcassing-inheriting-torch-tensor-class/110489/4">suggestion</a> to use the <code>tensor.as_subclass</code> method. The problem is that I’m not sure how to use it to set a new attribute at the initialization or afterward.  Therefore, I think that it doesn’t solve our case here.</li>
</ol>
<hr>
<p>Thanks</p>

vaezmarzieh: <p>Hello, do you have a final solution for this? Thanks</p>

keyboardAnt: <p>The new Tensorclass may be helpful:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.linkedin.com/posts/vincent-moens-9bb91972_tensordict-machinelearning-pythonprogramming-activity-7016132418881376256-S0Z7">
  <header class="source">
      <img src="https://static.licdn.com/sc/h/al2o9zrvru7aqj8e1x2rzsrca" class="site-icon" width="64" height="64">

      <a href="https://www.linkedin.com/posts/vincent-moens-9bb91972_tensordict-machinelearning-pythonprogramming-activity-7016132418881376256-S0Z7" target="_blank" rel="noopener nofollow ugc">linkedin.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:514/500;"><img src="https://media.licdn.com/dms/image/C4E22AQHxpxJrWOYBog/feedshare-shrink_800/0/1672776320645?e=1675900800&amp;v=beta&amp;t=SNZMyc8_Ix-9YJExhdKS2k87pB731w4fziOiqaEt7lc" class="thumbnail" width="514" height="500"></div>

<h3><a href="https://www.linkedin.com/posts/vincent-moens-9bb91972_tensordict-machinelearning-pythonprogramming-activity-7016132418881376256-S0Z7" target="_blank" rel="noopener nofollow ugc">Vincent Moens on LinkedIn: #tensordict #machinelearning #pythonprogramming...</a></h3>

  <p>Fancy a specialized dataclass to carry and manipulate tensors?

We've opensourced a new PyTorch tool under the #tensordict repo: tensorclass
In a few words…</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Applying nn.multiheadattention on CNN (resnet)

amtashfeen: <p>Hi,<br>
I am implementing nn.multiheadattention on resnet18 architecture.</p>
<p>My idea is to apply attenetion after every resnet block.</p>
<p>For Example:<br>
after ResBlock1 my output is a tensor of size [64,64,64,64] (B,C,H,W)<br>
I want to apply nn.multiheadattention on this output.</p>
<p>I am confused about multiheadattention part. Previously I was initializing nn.multiheadattention where embed_dim is equal to HxW of the output of resnet layer.</p>
<p>I was told to initialize embed_dim to be equal to the size of Channel dim of the resnet output.</p>
<p>Which one is the right approach?</p>
<p>one a side note, I get better accuracy when I use HxW as embed_dim in nn.multiheadattention.</p>
<p>Here is my code:</p>
<p>self.mha1 = nn.MultiheadAttention(embed_dim=4096, num_heads=8)<br>
self.mha2 = nn.MultiheadAttention(embed_dim=1024, num_heads=8)<br>
self.mha3 = nn.MultiheadAttention(embed_dim=256, num_heads=8)<br>
self.mha4 = nn.MultiheadAttention(embed_dim=64, num_heads=8)</p>
<p>def forward(self,x):<br>
x = self.conv1(x)<br>
x = self.bn1(x)<br>
x = self.relu(x)<br>
x = self.maxpool(x)</p>
<pre><code>    x = self.layer1(x) # [B, 64, 64, 64] [B,C,H,W]
    y = x.clone()
    y = y.view(-1, 64,64*64)
    y, _ = self.mha1(y,y,y)
    y = y.view(-1, 64,64,64)
    x = x + y

    x = self.layer2(x) # [B, 128, 32, 32] [B,C,H,W]
    y = x.clone()
    y = y.view(-1, 128,32*32)
    y, _ = self.mha2(y,y,y)
    y = y.view(-1, 128,32,32)
    x = x + y

    x = self.layer3(x) # [B, 256, 16, 16] [B,C,H,W]
    y = x.clone()
    y = y.view(-1, 256,16*16)
    y, _ = self.mha3(y,y,y)
    y = y.view(-1, 256,16,16)
    x = x + y

    x = self.layer4(x) # [B, 512, 8, 8] [B,C,H,W]
    y = x.clone()
    y = y.view(-1, 512,8*8)
    y, _ = self.mha4(y,y,y)
    y = y.view(-1, 512,8,8)
    x = x + y

    x = self.avgpool(x)
    x = torch.flatten(x, 1)
    x = self.fc(x)

    return x
</code></pre>
Large image loading is very slow, any way I can speed up things?

dazzle-me: <p>I have a very specific problem in which I have to use large images (resolution 2k x 3k) and relatively large batches (16-32), I’m totally CPU bottlenecked (it’s either transfer to GPU or image loading)</p>
<p>Images are stored in uint8 PNG format, I’m thinking about converting them to JPEG and use NVIDIA DALI library for loading/decoding using GPU, in my past it gave good speed-ups, however this library is so clumsy, is there an easier solution? I want fully utilize my GPU, it’s mostly idle now</p>
<p>Machine specs:<br>
Ryzen 9 5950x (16 cores, multithreading)<br>
128Gb RAM<br>
NVME m2 SSD<br>
4090 gpu</p>

eqy: <p>It would be useful to understand if it is truly data transfer or loading/decoding. Have you tried comparing the performance of:<br>
(1) loading + transfer + inference<br>
(2) load data once, transfer the same data multiple times + run inference multiples times<br>
(3) load data once, transfer once, run inference multiple times on the same data<br>
to isolate the cause?</p>

dazzle-me: <p>p.s. it’s during training<br>
i’m very sure 4090 can handle third step relatively easy,<br>
since neural network is very small (resnet-18-ish)</p>
<p>I’ll check the timings of batch generation and comeback</p>
Dimension error when multiplying weights and logits

haj2022: <p>I am implementing the paper<br>
<a href="https://arxiv.org/pdf/1808.08280.pdf" rel="noopener nofollow ugc">Deep multiscale convolutional feature learning for weakly supervised localization of chest pathologies in X-ray images</a></p>
<p>According to my understanding the layer relevance weights belong to the last layer of each dense block, with the dimension(N, 128,3,3) and logits from FC layer have dimension (1,1).</p>
<p>So Far i have implemented the code below:</p>
<pre><code class="lang-auto"> def weight_constraints(self):

        weights= {'feat1': self.model.features.denseblock2.denselayer12.conv2.weight.data,
            'feat2':self.model.features.denseblock3.denselayer24.conv2.weight.data,
            'feat3':self.model.features.denseblock4.denselayer16.conv2.weight.data}

        sum(weights.values()) == 1

        for i in weights.keys():
            w = weights[i]    
            w1 = w.clamp(min= 0)
            w = w1
        return weights


 weights= self.weight_constraints()
        for i in weights.keys():
            w = weights[i]
            l = logits[i]
            p = torch.matmul(w , l[0])
            sum = sum + p 

</code></pre>
<p>and</p>
<blockquote>
<p>logits = {‘feat1’: [tensor([[-0.0630]], …ackward0&gt;)], ‘feat2’: [tensor([[-0.0323]], …ackward0&gt;)], ‘feat3’: [tensor([[-8.2897e-06…ackward0&gt;)]}</p>
</blockquote>
<p>i get the following error:</p>
<blockquote>
<p>mat1 and mat2 shapes cannot be multiplied (12288x3 and 1x1)</p>
</blockquote>
<p>Please guide me if i am using the right approach and what is causing the issue.</p>

haj2022: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  <a class="mention" href="/u/smth">@smth</a><br>
looking forward to your response</p>
CUBLAS_STATUS_NOT_SUPPORTED for BF16 (Cuda11.6, Pytorch)

StevenTsai: <p>Hi,<br>
I just got the following error when I trained my Pytorch model with bfloat16 parameters</p>
<p>File “/opt/conda/envs/XXX/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 114, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasLtMatmul with transpose_mat1 1 transpose_mat2 0 m 768 n 2304 k 768 mat1_ld 768 mat2_ld 768 result_ld 768 abcType 14 computeType 68 scaleType 0</p>
<p>The types of input, self.weight, self.bias are all bfloat16 and the shapes are (9, 256, 768), (768, 768), (768, ), respectively.</p>
<p>My Pytorch version is <strong>1.14.0.dev20221213+cu116</strong>, and my python version is 3.8.15.<br>
Besides, I used <strong>8 A100 GPUs (80 GB)</strong>.</p>
<p>I ran “torch.cuda.is_bf16_supported()”, and got “True”.</p>
<p>Actually, I have tried different models with BF16 parameters but did not got the same error. However, for some reason, I could not share my model to you. Please tell me if you have any idea about the error. Thanks.</p>

StevenTsai: <p>Also, I tried different Pytorch and CUDA version.</p>
<pre><code class="lang-auto">Pytorch1.12.1, CUDA11.6
</code></pre>
<p><em>conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge</em></p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling <code>cublasLtMatmul( ltHandle, computeDesc.descriptor(), &amp;alpha_val, mat1_ptr, Adesc.descriptor(), mat2_ptr, Bdesc.descriptor(), &amp;beta_val, result_ptr, Cdesc.descriptor(), result_ptr, Cdesc.descriptor(), &amp;heuristicResult.algo, workspace.data_ptr(), workspaceSize, at::cuda::getCurrentCUDAStream())</code></p>
<pre><code class="lang-auto">Pytorch1.12.1, CUDA11.3
</code></pre>
<p>conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch</p>
<p>No error messages</p>

ptrblck: <p>Could you post a minimal, executable code snippet please, as  I cannot reproduce the issue using your posted shapes on a current master build on an A100 ?</p>
How to use the same batch to train the model?

msr3384: <p>I want to train my model using the same batch with DataLoader, but when I assigned train_loader = DataLoader(batch_size = any_size) and use iter() and next() on train_loader then for different epoch it takes the different batches as expected. Now if I want to fixed the same for the iterative epoch how to fixed batch to run everytime. I checked with small batch size and since the dataset is much bigger, as I printed the batch size input data it prints different batches until the data ends. How to fixed the batch for training?</p>

nwn: <p>Hello Muhammad,<br>
try to disable shuffling, i.e. DataLoader(dataset, batch_size=64, shuffle=False). Alternatively, you could try to set the random seed(s) to a fixed value which would make your experiment reproducible.</p>
<p>Best,<br>
nwn</p>

ginofft: <p>Luis solution is fine,</p>
<p>For even more customization, consider looking up on how to write a custom <a href="https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler" rel="noopener nofollow ugc">Sampler</a>.</p>
How to modify the output of the model

bgsuello: <p>Hello, I need help on modifying the output of the model.</p>
<p>I’m using YOLOX as the model, and it has the ff. output: <code>labels, dets</code>.</p>
<p>It should be straight forward on how to interpret the output but I am constrained on a embedded device that only accepts these kind of output: <code>locations, classes, scores, num_of_dets</code> kind of similar to <a href="https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2" rel="noopener nofollow ugc"><strong>SSD MobileNetv1</strong></a>.</p>
<p>I also tried doing the SSD model but the accuracy is low compared to the YOLOX trained with same dataset.</p>
Different Results Same Checkpoint

Dicko: <p>Hey folks, I trained a segmentation model on a GPU, saved the checkpoint and inference works great.</p>
<p>I use the same model and checkpoint on my PC using the CPU and the results (segmentation masks) are a lot poorer than they were in the GPU.</p>
<p>Any ideas what could be going on here and how to resolve?</p>

eqy: <p>Generally, some numerical differences with different hardware are expected as different implementations will incur some differences due to operation reordering. Could you try inference on the same data (e.g., the same image) and check how much the resulting values differ between CPU and GPU?</p>
Absent ReLU Layers in Pretrained GoogleNet?

Luke_Bun: <p>Hello,</p>
<p>I want to use the pretrained GoogleNet, but when I compared it to the original paper cited in the PyTorch, I noticed that it was missing ReLU layers.</p>
<p>According to the paper, “All the convolutions, including those inside the Inception modules, use rectified linear activation.” But when I run the following commands and view the output:</p>
<pre><code class="lang-auto">cnn = models.googlenet(pretrained=True)
print(cnn)
</code></pre>
<p>I get (only first 3 convolutional layers shown):</p>
<pre><code class="lang-auto">GoogLeNet(
  (conv1): BasicConv2d(
    (conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
  (maxpool1): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=True)
  (conv2): BasicConv2d(
    (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv3): BasicConv2d(
    (conv): Conv2d(64, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
</code></pre>
<p>When I run the equivalent commands for AlexNet and look at the architecture, I can see dedicated ReLU layers (only first 3 convolutional layers shown).</p>
<pre><code class="lang-auto">AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
</code></pre>
<p>Would anyone be able to tell me why these ReLU layers are absent or if I am doing something wrong? If it’s not ReLU, then what is the activation function being used and how is it achieved?</p>
<p>Thank you in advance!</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/vision/blob/32d254bbfcf14975f846765775584e61ef25a5bc/torchvision/models/googlenet.py#L184-L275">the code</a> the functional API is used via <code>F.relu</code> which is why these modules do not show up in <code>print(model)</code>.</p>

Luke_Bun: <p>I see, that makes sense. Thank you so much!</p>
How could I broadcast 1D uncertain length tensor

JuyiLin: <p>I have a 1D <code>torch.tensor</code>, which length is uncertain. How I could broadcast it?<br>
Currently, I propose one way, is it suitable?</p>
<pre><code class="lang-python">def run(rank, world_size):
            if rank == 0:
                n_id = uncertain length 1D tensor
            else:
                n_id = torch.zeros(maxlength of n_id, dtype=torch.int64).to(rank)
            dist.broadcast(n_id, src=0)
n_id = n_id[n_id.nonzero()]
</code></pre>

Yanli_Zhao: <p>if you know the max length of n_id, then you can pad n_id to be the max length with 0 elements</p>
Train model with use subset of data for subset of epoch

test_project: <p>I’m trying to train my model with subset of data for each 10 epoch and I write this code but unfortunately it doesn’t work how can I change this code?<br>
Num_data=25000<br>
Num_epoch=50000<br>
Num_sub_data=1000</p>
<pre><code class="lang-auto">
class DataModule(pl.LightningDataModule):

  def __init__(self, train_dataset):

    super(DataModule, self).__init__()
    self.train_dataset = train_dataset
  def train_dataloader(self):
    return DataLoader(self.train_dataset,2, shuffle = True)

Num_data=25000
Num_epoch=50000
Num_sub_epoch=10
Num_sub_data=1000
for i in range(int(Num_epoch/Num_sub_epoch)):
  i=np.random.choice(datamodule.train_dataloader(),Num_sub_data)
  x=x[i]
  y=y[i]
  for _ in range(Num_sub_epoch):
    trainer.fit(model, datamodule)
</code></pre>
<p>That error is:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
mtrand.pyx in numpy.random.mtrand.RandomState.choice()

TypeError: 'DataLoader' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
&lt;ipython-input-52-0b8908d20663&gt; in &lt;module&gt;
      2 ne=10
      3 for i in range(int(epochs/ne)):
----&gt; 4   i=np.random.choice(datamodule.train_dataloader(),10)
      5   x=x[i]
      6   y=y[i]

mtrand.pyx in numpy.random.mtrand.RandomState.choice()

ValueError: a must be 1-dimensional or an integer
</code></pre>
<p>If this code is error how can I change dataloder to could be use only 1000 data for each 10 epoch?</p>

nivek: <aside class="quote no-group" data-username="test_project" data-post="1" data-topic="169186">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/test_project/48/46868_2.png" class="avatar"> test_project:</div>
<blockquote>
<pre><code class="lang-auto">i=np.random.choice(datamodule.train_dataloader(),Num_sub_data)
  x=x[i]
</code></pre>
</blockquote>
</aside>
<p>You can have an inner <code>for</code> loop that get samples from DataLoader up to the desired number while enabling shuffling to get different samples during each epoch.</p>
How to organize with lots of related data?

tomwhiteman: <p>Hey guys,</p>
<p>For a university project I’m trying to see the relation oil production/consumption and crude oil price have on certain oil stocks, and I’m a bit confused about how to sort this data.</p>
<p>I basically have 4 datasets-<br>
Oil production<br>
Oil consumption<br>
Crude oil price<br>
Historical price of certain oil company stock</p>
<p>If I am trying to find a way these 4 tables relate, what is the recommended way of organizing the data? Should I manually combine all this data to a single Excel sheet (seems like the most straight-forward way) or is there a more efficient way to go about this.</p>
<p>I am brand new to PyTorch and data, so I apologise if this is a very basic question. Also, the data can basically get infinitely larger, by adding data from additional countries, other stock indexes, etc. So is there a way I can organize the data so it’s easy to add additional related data?</p>
<p>Finally, I have the month-to-month values for certain data (eg: oil production), and day-to-day values for other data (eg: oil price). What is the best way I can adjust the data to make up for this discrepancy?</p>
<p>Thanks in advance!</p>

nivek: <p>Hi <a class="mention" href="/u/tomwhiteman">@tomwhiteman</a>,</p>
<p>Thanks for posting here. These questions do not relate to PyTorch and you will likely find better responses elsewhere, such as on Stack Overflow.</p>
<p>I will provide a brief answer here as I have worked quite a bit on similar topics before:</p>
<blockquote>
<p>What is the recommended way of organizing the data?</p>
</blockquote>
<p>As long as you can easily read the data into Python (e.g. with <code>pandas</code>), it should be fine. Keep in mind that you are working with time series data and representation should reflect that. You can decide what to do with them after reading it into Python.</p>
<blockquote>
<p>Finally, I have the month-to-month values for certain data (eg: oil production), and day-to-day values for other data (eg: oil price). What is the best way I can adjust the data to make up for this discrepancy?</p>
</blockquote>
<p>Handling data of different frequencies is a common issue with time series data. You can either increase or decrease the frequency of certain data to match the other ones. The exact transformation (e.g. aggregating, averaging, picking first/last) should be chosen based on the economic context of the data.</p>
Are torchdata datapipes re-created at the start of each epoch?

Vedant_Roy: <p>I have a custom datapipe for sharding that I use instead of the default <code>sharding_filter</code>, etc. It roughly looks like this:</p>
<pre><code class="lang-auto">    def __iter__(self):
        from distributed_ml.logging.logger import logger

        l = logger.bind(tag="FullyShardedChunkWrapper")
        l.debug("iter_start", epochs=self.epochs)

        self.epochs += 1
        rng = random.Random(self.epochs)

        all_chunks = copy.deepcopy(self.chunks)
        if self.global_chunk_shuffle:
            rng.shuffle(all_chunks)
        world_info = self.get_world_info()
        cur_rank_chunks = partition(all_chunks, world_info.size, world_info.rank)

        worker_id, total_workers = self.get_worker_info()
        cur_worker_chunks = partition(cur_rank_chunks, total_workers, worker_id)
        rng.shuffle(cur_worker_chunks)

        total = len(cur_worker_chunks)
        for idx, chunk in enumerate(cur_worker_chunks):
            l.debug("iter_yield", idx=idx, left=total - idx - 1)
            yield chunk
</code></pre>
<p>I recently had the discovery that the my datapipe was not actually shuffling the data when a new epoch rolled around because <code>self.epochs</code> was never incremented.</p>
<p>My guess here is that, when a new epoch happens, the dataloader re-creates the datapipe from scratch, so maintaining / incrementing instance variables is a bad idea.</p>
<p>But, I wanted to confirm this intuition.<br>
<a class="mention" href="/u/ejguan">@ejguan</a></p>

ejguan: <aside class="quote no-group" data-username="Vedant_Roy" data-post="1" data-topic="169269">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vedant_roy/48/51087_2.png" class="avatar"> Vedant_Roy:</div>
<blockquote>
<p>My guess here is that, when a new epoch happens, the dataloader re-creates the datapipe from scratch</p>
</blockquote>
</aside>
<p>I don’t think we re-create <code>datapipe</code> at the start of each epoch. <code>DataLoader2</code> would only do a <code>deepcopy</code> during initialization.</p>
<p>cc: <a class="mention" href="/u/nivek">@nivek</a></p>

nivek: <p>It will at most re-create the iterator of the DataPipe (and reset), but it should not re-create the DataPipe object itself, unless you have a custom DataPipe that specifically do that (and probably should not).</p>
How do I create a dataloder with list of tensors in different shape

Netanel_Fried: <p>let say I have two 3d tensors. each tensor contain an entire batch (batch size = 64) and I want to stack them together in a data-loader object so that when I iterate over the data-loader with Batch size = 64 will first get the first object and the section…<br>
( I don’t need the shuffle so shuffle=False)<br>
code e.g:</p>
<pre><code class="lang-auto">a_torch.shape
Out[97]: torch.Size([64, 347, 5])
b_torch.shape
Out[98]: torch.Size([64, 151, 5])
t_iter = DataLoader([a_torch, b_torch], batch_size=64, shuffle=False)
for data in t_iter:
    print(type(data))
    print(data.shape)

Out[98]:torch.Tensor
Out[99]:torch.Size([64, 347, 5])
Out[100]:torch.Tensor
Out[101]:torch.Size([64, 151, 5])
</code></pre>
<p>I tried to set my own <code>collate_fn</code> but it did not work.</p>
<p>Thank you</p>
Is there anything similar to empty_cache for host memory?

rinkujadhav2013: <p>I’m running some code in a loop. I expect the memory to refresh after each run in the loop but it doesn’t happen automatically and deleting the tensor variables followed by a call to the garbage collector doesn’t do anything either. So, is there any function similar to <code>torch.cuda.empty_cache</code> for RAM? Or something else I can do to give pytorch a fresh start for each run?</p>

ptrblck: <p>Could you describe what “refresh” the memory would mean in this context? I don’t see any value in reusing host memory (besides page locked memory maybe) so could you also explain your expectations?</p>

rinkujadhav2013: <p>By “refresh”, I meant that I expect Python’s garbage collector to deallocate intermediate buffers that are no longer required. For instance, shouldn’t the layer’s output tensor (if not used anywhere else) be deallocated once it goes out of context (like in the 2nd iteration of a loop)?</p>

ptrblck: <p>I don’t think the GC <em>should</em> directly deallocate objects as I guess Python authors have thought about a proper way to balance too aggressive freeing vs. too much memory usage.<br>
If you want to force the GC to free objects directly, you might need to use the <a href="https://docs.python.org/3/library/gc.html">GC interface</a> and play around with some options.<br>
Note that I have never touched this interface as it was never creating an issue for me.</p>

rinkujadhav2013: <p>I’ve tried using GC already. My current code looks like this:</p>
<pre><code class="lang-auto">    for some loop:  # I loop over different configs
        layer = torch.nn.Conv2d(32, 64, k, stride=s, padding=p, dilation=d, bias=False, device=host_device, dtype=torch.float32)
        X = torch.rand(1, 32, sz, sz, requires_grad=True, dtype=torch.float32).to(memory_format=torch.channels_last).detach()
        X.requires_grad_()
        cpu_start = time.time()
        tmp = layer(X)
        cpu_fwd_time = time.time() - cpu_start
        del layer, X, tmp
        gc.collect()
</code></pre>
<p>Memory usage in one iteration affects the second iteration</p>
(Vision Transformer) On using subsets of a "large" PositionEmbedding matrix for smaller images

pt2: <p>Hello!</p>
<p>I am modifying the <a href="https://arxiv.org/pdf/2010.11929.pdf" rel="noopener nofollow ugc">VisionTransformer</a> implementation in <a href="https://github.com/rwightman/pytorch-image-models" rel="noopener nofollow ugc"><code>timm</code></a> to preserve the aspect ratios of input images (as best as possible).</p>
<p>The original ViT (as implemented in <code>timm</code> and in the original JAX code) expects square images. But you can still generate patch embeddings for an image of arbitrary size.</p>
<p>What remains is to add Position Embeddings to each of these patches before passing to the Transformer Encoder.</p>
<p>There is a maximum aspect ratio that I work with (say <code>1:2 :: h:w</code>). At the moment, I initialize the position embeddings for the <em>largest possible image</em>, and use the top-n embeddings based on the <code>n</code> patches that the input image generates.</p>
<p>I feel though that this approach is flawed.</p>
<p>From the ViT paper, it seems that the patches are unrolled left-to-right, top-to-bottom. IMO, this doesn’t account for any 2D positioning of the patches, but rather imposes an <em>arbitrary but consistent</em> ordering (i.e. left-to-right, top-to-bottom). This logic lends some sense to my approach.</p>
<p>At the same time, if the <code>k</code>-th embedding represents the top-right corner patch of the largest image, then this <code>k</code>-th embedding might be re-purposed to represent a center patch of a smaller image.</p>
<p>Was wondering what people make of this.</p>
<p>Thanks!</p>
From GPU to CPU Torch not compiled with Cuda enabled

Dicko: <p>Hello folks, I have a model that trained beautifully using cuda, the predictions look good.<br>
When I try to run on my local PC I get the error ‘AssertionError: Torch not compiled with CUDA enabled’ .<br>
I’ve tried with device = torch.device(‘cpu’), but still get the same error. Any ideas?</p>

ptrblck: <p>Which line of code raises this issue? PyTorch does not automatically assume you have a GPU so I assume you are explicitly trying to execute device code somewhere.</p>

Dicko: <p>Thank you for your reply. I managed to solve it. The code was referring to a file that had cuda, fixed it to cpu and all working fine now.</p>
Time series classification per sample (event detection)

Andrey: <p>Hi all,<br>
Please help me with the following.</p>
<p>I want to train a model to detect events (posture change, can last up to several seconds) from acc data.<br>
My input is multi-dimentional numeric data + binary labels (only for training, of course).<br>
The model should then identify variable-duration episodes of posture transition in the given data.<br>
It doesn’t have to be real-time → the model can look into the past and future samples.</p>
<p>I thought that RNN\LSTM can be a suitable option here, but can someone point to a good example or a comparable project?<br>
I only see either future time-series prediction, or total time-series classification examples. Sometimes also anomaly detection (unsupervised, from data itself, without labels).</p>
<p>In my case I have long recordings likely containing many short events and mostly other activities.<br>
Maybe it is not a good approach if I can’t find evidence that people use it?</p>
Skip connections in INet

Raphael_Olipitz: <p>Hello there,</p>
<p>I am trying to implement an Inet and a DenseINet but there are special skip connections mentioned in the original paper from Wang et al. ( <a href="https://doi.org/10.1109/access.2021.3053408" rel="noopener nofollow ugc">10.1109/ACCESS.2021.3053408</a> called “CUSTOMIZED RESIDUAL SHORTCUT” and “CONVOLUTIONAL INDEX” shown in the following image. I know how to implement the normal skip function for ResNet but not for this kind of connection in between blocks. Thx for any advice.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8b911083ec0b39566097bbbb2a80e42882dd354.jpeg" alt="access-gagraphic-3053408" data-base62-sha1="xcLh8kr7jEKOy5647iR9BOtkduI" width="660" height="295"></p>
Training function good practices

ginofft: <p>Hi, after reading through multiple Pytorch implementations, I noted that there really isn’t a standard on the function call for the train() and test() function.</p>
<p>On one extreme, most of the time, I’ve seen people simply put the train() and test() function inside main.py. Then initialize training using some form of:<br>
<code>python main.py --mode=train ...</code><br>
In which, the train() only take a single parameter (usually epoch) or none as all.<br>
Basically, model, dataset, DataLoader, loss and optimizer are global variables in the scope of  <code>main.py</code><br>
If I follow this convention, for my case; In which I compared the performance of two different custom loss → which requires two different custom Dataloaders (and Samplers)<br>
Which meant I have to write two different train() functions.</p>
<p>On the other extreme, if I pass all parameters into train(), the arguments would be: epoch, model, train_set(or dataloader), loss, optimizer.<br>
Which meant the train() function would be extremely bloated.</p>
<p>As such, I wanna ask you guys - What is your preference for the train functions ? train(epoch) or train(epoch, model, dataloader,…) ? Or is it some middle ground between these two extreme.</p>

shivammehta007: <blockquote>
<p>On the other extreme, if I pass all parameters into train(), the arguments would be: epoch, model, train_set(or dataloader), loss, optimizer.<br>
Which meant the train() function would be extremely bloated.</p>
</blockquote>
<p>I would like to think the other way actually since train is a function it is expected to perform one function/task i.e to train the model with the given parameters therefore, I would suggest passing everything as a parameter instead of having global references (which is also the base of functional programming).</p>
<p>Also, generally, I tend to use <a href="https://www.pytorchlightning.ai/" rel="noopener nofollow ugc">PyTorch lightning</a> to manage this boilerplate for me. To do this I define the <code>LightningModule</code> and <code>LightningDataModules</code> separately and then depending on the choice of dataset and hyperparameters I pass them to the Lighting Trainer accordingly.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
RuntimeError: Function 'ConvolutionBackward0' returned nan values in its 0th output

Oguzhan: <p>I am implementing visual transformer and when I set the ligthning atribute  as detect_anomaly=True I got the error in the title. Could it be related with <code>tfout_ =  einops.rearrange(tfout,"b n e -&gt;b e n")</code> .Can changing the dimension order cause the gradients to break?</p>
<pre><code class="lang-auto">class DotProductAttention(nn.Module):
    def __init__(self,in_channels:int):
        super().__init__()
        self.softmax = nn.Softmax(dim = 1)
        self.conv1 = nn.Conv1d(in_channels,in_channels,1)
        self.conv2 = nn.Conv1d(in_channels,in_channels,1)
        self.relu = nn.ReLU()
    def forward(self,query,key,embed):
        
        tfout = torch.matmul(self.softmax(torch.matmul(query,key)),embed) + embed   
        tfout = einops.rearrange(tfout,"b n e -&gt;b e n") + self.conv2(self.relu(self.conv1(einops.rearrange(tfout,"b n e -&gt;b e n"))))
        tfout_ = einops.rearrange(tfout,"b n e -&gt;b e n")

        return tfout_
</code></pre>
<p>First epoch was okey but at the second epoch, it broken.</p>
<p>It doesn’t actually say that the error is in the part of the code I shared. For this, I am adding the things I used in the model training.</p>
<p>Last layer of network → LogSoftmax<br>
Loss function → NLLLoss<br>
Optimizer → Rmsprop<br>
SWA used<br>
Lr Scheduler → <code>optimizer.param_groups[0]['lr'] = optimizer.param_groups[0]['lr'] *0.9875</code></p>

Oguzhan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> do you have an idea?</p>

T_Friedrich: <p>Hi <a class="mention" href="/u/oguzhan">@Oguzhan</a> , have you solved that issue?</p>

Oguzhan: <p><a class="mention" href="/u/t_friedrich">@T_Friedrich</a>  Yes but I can not remember the solution, sorry for that. I looked  the code again but I did not change this class.</p>

T_Friedrich: <p>That’s unfortunate, then I have to continue to debug that issue <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=12" title=":frowning_face:" class="emoji" alt=":frowning_face:" loading="lazy" width="20" height="20"><br>
Thank you anyway <a class="mention" href="/u/oguzhan">@Oguzhan</a></p>
Node regression on graphs with different sizes - Pytorch Geometric

chris: <p>I have a dataset of graphs, several of them have different target feature dimensions, e.g.</p>
<p>Graph1 = Data(edge_attr=[9468], edge_index=[2, 9468], x=[299, 21], y=[299])<br>
Graph2 = Data(edge_attr=[9622], edge_index=[2, 9622], x=[309, 21], y=[309])</p>
<p>As you see they contain a different amount of nodes, 299 vs 309, each target y is a continuous variable that I want to predict. The problem is since they are different sizes I would need a neural network with a dynamic output dimension, which is problematic. My solution would be to “pad” the graphs with nodes that have features equal to 0 and a target equal to 0 with no edge connections to make the graphs have the same number of nodes and targets, but I am not sure if this is a good solution. Do any of you have an idea of how to solve this problem? I am quite new to the field of geometric deep learning.</p>
<p>Thanks!</p>

chris: <p>I solved it by padding and then making a mask for the values I want to use for training. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

looc: <p>I can see that padding solves it for graphs of similar sizes. But what can I do if my graph sizes differ in magnitudes, say, from 100 nodes to 10000 nodes? Especially if I don’t know if the graphs presented in inference exceed the maximum size of training graphs? Padding seems not to be a solution for this case.<br>
If you have, for example, public transport networks for a couple of cities, and want to predict a certain property for each station. Training examples have up to N stations, but I want to apply the trained model to a city with 2*N stations.<br>
I only recently started with GNNs, so perhaps I am missing an obvious way to apply them to a task like this.</p>
Error importing Torchaudio

Alessandro_Proverbio: <p>Hi all,</p>
<p>I can’t import Torchaudio, this is my setup:</p>
<ul>
<li>macOS Monterey 12.4 (Intel Mac)</li>
<li>Python 3.10.4 (virtual env, Python has been installed with the installer from the website, no Conda or similar)</li>
<li>Torch 1.11.0 (installed with Pip)</li>
<li>TorchAudio 0.11.0 (installed with Pip)</li>
</ul>
<p>This is the error I get when I try to import Torchaudio with “import torchaudio”:</p>
<pre><code class="lang-auto">OSError: dlopen(/Users/ ... path to my env ... /ap_venv/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so, 0x0006): Symbol not found: __ZN2at8internal15invoke_parallelExxxRKNSt3__18functionIFvxxEEE
  Referenced from: /Users/... path to my env ... /ap_venv/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so
  Expected in: /Users/ ... path to my env ... /ap_venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.dylib
</code></pre>
<p>(I’ve removed from the error the part of the path that contains personal information)</p>
<p>I’ve checked and the file “libtorch_cpu.dylib” is at the path specified in the error. I’ve tried to install and import Torchaudio also in the main Python installation (so not in a virtual env) and the error is the same (of course the path is different).</p>
<p>Torch alone works without issues, only Torchaudio raise this error.</p>
<p>What can I do? Thank you for your support.</p>

robcamp-code: <p>I am also struggling with this same exact issue (same pytorch version) did you find a solution?</p>

fedorovgv: <p>I faced with the same problem. Change python in venv from 3.10 to latest 3.8.10 works for me.</p>
<p>With pyenv:</p>
<pre><code class="lang-auto">pyenv install 3.8.10
~/.pyenv/versions/3.8.10/bin/python -m venv venv_name
</code></pre>

AJDERS: <p>I am also experiencing this issue, but i do not have the option of downgrading python to 3.8, anybody have a solution for 3.10?</p>

AJDERS: <p>I am still experiencing this issue… Anybody have any insights?</p>
Error when building from source M1 Mac

gianni: <p>Hi, is there an updated tutorial for building/installing pytorch from source for Mac OS X M1? Every guide iv’e found (including the guide on the pytorch repo) leads to an explosion of errors and the build fails. Mainly surrounding the build step <code>FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/autograd/generated/python_functions_3.cpp.o</code></p>
<p>Any help would be much appreciated. Thanks!</p>

damian0815: <p>i’m on M1 and was able to build following the official guide but with a couple of modifications. wrote up my experiences here: <a href="https://medium.com/@damian0815/how-to-debug-native-mps-operations-in-pytorch-f094b7f4e8f0" class="inline-onebox" rel="noopener nofollow ugc">Debugging PyTorch MPS native code | by d; DROP TABLE `tweet | Jan, 2023 | Medium</a></p>
How to combine gradients from different devices into different weights

Hyeonuk_Woo: <p>As far as I understand, in DDP context, when <code>backward</code> is called, gradients from every device are gathered and the averaged gradient(averaged by number of device) is applied to <code>optimizer.step</code>.<br>
Due to the characteristic of my data, a different number of samples are created for deep learning to learn from each batch of data. In other words, when I call <code>backward()</code>, The gradients coming from each device are gradients calculated from different numbers of samples.<br>
When gather the gradients backward, I want to find the average of the gradients according to the number of samples, not the average according to the number of devices.<br>
How can I do this?<br>
Multiplying the gradients coming from each device by different weights (the number of samples in my case) can be a solution?<br>
Below is my code snippet.</p>
<pre><code class="lang-auto">def train_epoch(...):
     for i, batch in enumerate(train_dataloader):
        with torch.cuda.amp.autocast(enabled=args.amp):
            pred = model(...)
            loss=loss_fn(...)         
        grad_scaler.scale(loss).backward()
        # gradient accumulation
        if (i + 1) % args.accumulate_grad_batches == 0 or (i + 1) == len(train_dataloader):
            if args.gradient_clip:
                grad_scaler.unscale_(optimizer)
                torch.nn.utils.clip_grad_norm_(model.parameters(), args.gradient_clip)
        grad_scaler.step(optimizer)
        grad_scaler.update()
        optimizer.zero_grad()
</code></pre>

MitchellKT: <p>An easy workaround would be to use <code>loss_fn</code> which computes the sum of the losses per instance (i.e sum reduction in CE).<br>
Then, multiply this loss with <code>n_gpus/tot_num_samples</code> where <code>tot_num_samples</code> is the total amount of samples and <code>n_gpus</code> is the amount of gpus used.<br>
Then, the final gradients will be of the correct loss.</p>
Modify the dataset

msr3384: <p>I have a data directory with images and using different way I can convert my data into an array or list dataset with label for tensorflow to train a model with my data set. But I am doing clustering using torch with tensor dataset for MNIST data set. Now to use my model I want to convert my collections of images( for examples dogs and cats dataset) into a data set like MNIST built in datasets. How can I do that?</p>

ptrblck: <p>If your data and targets are already tensors you could just pass them to <code>torch.utils.data.TensorDataset</code> and iterate it directly or via a <code>DataLoader</code> (which would batch the samples for you, shuffle them etc.).<br>
On the other hand if you want to apply transformation to each sample and e.g. lazily load the data you could write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>
Vanishing Gradients and how to fix?

sardanian: <p>I believe I am running into vanishing gradients and was hoping to see if anyone could help me either fix this or revive the model.</p>
<p>I am use a recreation of alphazero’s model but made in pytorch.</p>
<p>This started showing up a while ago in the iterations but I never caught it early enough and am looking to either fix it running forward and if needed to modify some weights now.</p>
<p>I noticed it when I was compiling my model into a tensorrt model via torch_tensorrt. It was giving me warnings that my model has weights smaller than float16 that would get converted to the lowest float16 value.</p>
<p>Upon inspection of my weights I have noticed that about 90% or more of them on my layers are near 0, thus my issue with vanishing weights.</p>
<p>From my understanding of vanishing weights, my model shouldn’t have vanishing weights, but it seem that it does.</p>
<ol>
<li>Can I fix this running forward? Anything wrong here?</li>
<li>Can I just set all of my vanishing weights to a minimum value to jumpstart them? or maybe a range of minimum values? I understand this could temporarily mess up the model but I would hope that after a few iterations it could recover?</li>
</ol>

sardanian: <p>I recreated my first iteration to see if it was a problem early on or if it happened later and I noticed that when I use Adam optimizer with learning rate of .001 then I get vanishing gradients after the first pass.</p>
<p>When I used SGD with .2 learning rate then I do not get vanishing gradients on the first pass. I am not sure if this will hold up.</p>
<p>I initially had started with Adam but have recently changed to SGD (I read SGD generalizes better) which would explain why I now have vanished weights.</p>
<p>So additional questions now:<br>
Can I just train on SGD and will that then recover the vanished weights?<br>
Also, I’m curious why Adam would have caused this to be an issue to start with.</p>
<p>Is there an easy function to set all near zero weights to a range of small numbers?<br>
Something like:<br>
<code>torch.nn.init.uniform_(policy.conv.conv1.weight, -.01, .01)</code><br>
but only for near zero weights</p>
Didn't get any gpu using sbatch when submitting a job script through slurm

ximing: <p>Here is my slurm job script. I requested 4 gpus and 1 computing node. My script is as follows:</p>
<pre><code class="lang-auto">#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --gres=gpu:4
#SBATCH --ntasks-per-gpu=12
#SBATCH --mem-per-gpu=40G
#SBATCH --time=0:15:00

module use /ifs/opt_cuda/modulefiles
module load python/gcc/3.10
module load cuda11.1/toolkit cuda11.1/blas cuda11.1/fft cudnn8.0-cuda11.1 tensorrt-cuda11.1/7.2.3.4

# activate TF venv
source /ifs/groups/rweberGrp/venvs/py310-tf210/bin/activate


python -c "import torch;print(torch.cuda.device_count())"
</code></pre>
<p>so the torch.cuda.device_count() should give me 4 but actually the output is 0</p>
<pre><code class="lang-auto">0
</code></pre>
<p>I have no idea why this is happening. Anyone has any idea? Thanks</p>

ximing: <p>solved should use sbatch instead of bash</p>