akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Implement focal loss in train_one_epoch_function

Adrian_Contreras_Gar: <p>Hi,</p>
<p>I’m currently training models for a highly imbalanced medical dataset, so I’m trying to implement the Focal Loss function in order to compensate this.</p>
<p>For my training pipeline, I’m using the <strong>train_one_epoch</strong> function, and I don´t know how to code this on the function. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d8c290e575ca2562e07f6216283c51232877d1a.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d8c290e575ca2562e07f6216283c51232877d1a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d8c290e575ca2562e07f6216283c51232877d1a.png" alt="image" data-base62-sha1="8MtpUt6fAZSK376sRimDrFhJQDg" width="541" height="500" data-dominant-color="EFEFF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">787×727 37.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Call `step()` and `zero_grad()` for each layer?

Mahdi_Zaferanchi: <p>Normally, <code>optimizer.step()</code> and <code>optimizer.zero_grad()</code> are called after the backward pass for the entire model is done. This means that all the optimizer state will be in memory at the same time.</p>
<p>My question is, if we have a sequential stack of modules, is it possible to have a separate optimizer for each module and call <code>step()</code> and <code>zero_grad(set_to_none=True)</code> immediately after the backward pass is done for each of them?</p>
<p>Seems like this should decrease memory usage since all the gradients and optimizer statistics won’t need to be held in memory at once.</p>
<p>In other words, why not perform the weight update and free the memory taken by gradients as soon as possible?</p>

ptrblck: <p>You can take a look at <a href="https://pytorch.org/tutorials/intermediate/optimizer_step_in_backward_tutorial.html">this tutorial</a>.</p>
CNN model configuration: advice

K_M23: <p>Hi,<br>
Assume that a CNN model is to be developed to recognize commercial domestic planes flying in the sky. The training data should include images of flying domestic planes for true positives. Additionally, it should encompass other types of aircrafts, such as private jets and helicopters. Should the training data also include instances with no flying aircraft so that the output layer has two outputs: Domestic plane and Non-domestic plane? Or would it be better to have three outputs in the output layer: Commercial domestic plane, Non-commercial domestic, Non flying aircraft?</p>
<p>Cheers</p>
Processing large dataset

Edan_Patt: <p>Hello,<br>
I’m working on creating a Dataset on roughly 100,000 graph samples I’ve created using pytorch geometric’s libraries.<br>
I roughly followed the guide here:<br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html#creating-in-memory-datasets" class="inline-onebox" rel="noopener nofollow ugc">Creating Your Own Datasets — pytorch_geometric 2.0.4 documentation</a>.<br>
My data in total, after preprocessing, weighs roughly 14G so I think this is probably too large to be an InMemoryDataset so I opted to work on it as a regular dataset.<br>
So my processed folder now contains 100k samples, each sample saved in its own file. Now this was obviously a bad choice. To iterate over my whole dataset is extremely costly as I have to open and close 100k files but I’m unsure of how to batch my data up into larger files while also somehow keeping track of what file contains what indices. Even if I did do such a batching how would my “get” function know to keep these files open until I’m done with the one file? Obviously there are many issues with what I’m suggesting which is why I never went through with it.</p>
<p>So does anyone have any ideas of how I can deal with this issue?</p>
<p>Thanks</p>

ejguan: <p>Could you load data once per iteration rather than loading everything in to memory when creating Dataset.<br>
We have a new repo to construct iterDataPipe to advocate iterator-style data loading methodology. <a href="https://github.com/pytorch/data" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/data: A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.</a><br>
Let us know if this is going to help you</p>

Animesh_Basak_Chowdh: <p>What I did was passing a train.csv containing the list of files I want to train with and returning this list in processed_file_names. In get(), I am iterating through the list and reading each of the .pt files. In my case, <span class="hashtag">#samples</span> are 63k graphs and each .pt file can be as big as 5MB. So, I zipped each of the pt files having the ‘data’ instance.</p>
<p>I tried InMemory dataset too, but no luck. The training has been extremely slow due to dataloading (~40 mins per epoch, same on RTX 3070 and RTX A4000) for batchsize of 64 and 27k training samples.</p>

hugo_prudent: <p>Hello Edan_Patt,</p>
<p>Did you find a way for not storing that many files in the end? I found <a href="https://github.com/pyg-team/pytorch_geometric/issues/4543" rel="noopener nofollow ugc">this discussion</a> where people talked about implementing a solution but I don’t think that it led to any release yet.</p>
<p>I’m experiencing the same issue whith a Dataset containing roughly one million graphs. So even the checking of filenames when I’m creating the Dataset object (with the .pt files already stored) is taking way too much time.</p>
<p>Thanks</p>
Torch.expand_as() not working for tensors of different shape

aquorio15: <p>Hi everyone<br>
I am trying to calculate the top-n accuracy of a VQA model but every time it throwing an error regarding incompatible shape. I got the top-n code from here <a href="https://github.com/pprp/timm/blob/master/timm/utils/metrics.py" rel="noopener nofollow ugc">https://github.com/pprp/timm/blob/master/timm/utils/metrics.py</a></p>
<pre><code class="lang-auto">def accuracy(output, target, topk=(1,)):
    maxk = min(max(topk), output.size()[1])
    batch_size = target.size(0)
    _, pred = output.topk(maxk, 1, True, True)
    pred = pred.t()
    print(pred.shape)
    correct = pred.eq(target.reshape(1, -1).expand_as(pred))
    return [correct[:min(k, maxk)].reshape(-1).float().sum(0) * 100. / batch_size for k in topk]
</code></pre>
<p>Is there any way I can mitigate the issue?</p>

ptrblck: <p>Could you post the full error message as well as the tensor shapes causing it?</p>

aquorio15: <p>Sorry for not giving them in my previous post.</p>
<p>The tensor shapes are for a batch of data</p>
<pre><code class="lang-auto">pred = torch.size(64, 830)
target = torch.size(64, 830)
</code></pre>
<p>After getting the top-k predictions, I am getting the pred shape as follows torch.size(10, 64). After which I am getting a runtime error</p>
<pre><code class="lang-auto">RuntimeError: The expanded size of the tensor (64) must match the existing size (53120) at non-singleton dimension 1.  Target sizes: [10, 64].  Tensor sizes: [1, 53120]
</code></pre>
What does the function wrapper @once_differentiable do?

jxgu1016: <p>I find some codes in <a href="https://github.com/facebookresearch/maskrcnn-benchmark/blob/55d3ab44aa9ff5261afb8244ab5e265266d8436f/maskrcnn_benchmark/layers/roi_align.py#L25" rel="nofollow noopener">this</a> and become curious about the function of <code>@once_differentiable</code>.</p>

albanD: <p>If you wrap your <code>Function</code>'s <code>backward</code> method with this wrapper, the Tensors that you will get as input will never require gradients and you don’t have to write a backward function that computes the gradients in a differentiable manner. For example, you can use other libraries to do the computation.<br>
If you try to backward through the backward pass of such Function, an error will be raised stating that this Function is only differentiable once.</p>

jxgu1016: <p>Thx for your reply but it’s a bit confusing…Is that meaning we cannot compute high-order gradients from this function?</p>

albanD: <p>Yes.<br>
You should use it when you write a backward that cannot be used for high-order gradients.</p>

jxgu1016: <p>Got it!<br>
Very helpful!</p>

Fei_Liu: <p>I don’t think the decorator <code>once_differentiable</code> does anything here. The decorator checks whether the input requires_grad (and err’s if yes). However the grad computation logic has been fully encapsulated in the backward function body (with context from forward), and the only part exposed to the outside is the <code>grad_output</code>, which will always not require grad in higher order derivative computations, hence making the decorator pretty much useless.</p>
<p>I highly doubt if there is a way to trigger a double backward for functions like this. Can someone give an example where it may happen?</p>
What is the point of calling `reset_parameters()` when initializing a model on a meta device?

divinho: <p>I’m specifically referring to <a href="https://github.com/pytorch/pytorch/blame/main/torch/distributed/fsdp/_init_utils.py#L885" rel="noopener nofollow ugc">this code</a>:</p>
<p>Why is what’s in a module’s <code>__init__</code> not enough?</p>
Pytorch keypoint RCNN Model in Android deployment error due to torchvision::nms

yogesh_vasu: <p>I developed a Keypoint RCNN Detection model, but when deploying it on Android, I encountered this error:</p>
<pre><code class="lang-auto">Process: com.example.java_pytorch, PID: 5028
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.java_pytorch/com.example.java_pytorch.MainActivity}: com.facebook.jni.CppException: 
Unknown builtin op: torchvision::nms.
Could not find any similar ops to torchvision::nms. This op may not exist or may not be currently supported in TorchScript.
:
  File "code/__torch__/torchvision/models/detection/keypoint_rcnn/___torch_mangle_501.py", line 783
          _360 = torch.index(boxes4, _359)
          _361 = annotate(List[Optional[Tensor]], [curr_indices])
          curr_keep_indices = ops.torchvision.nms(_360, torch.index(scores1, _361), 0.69999999999999996)
                              ~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
          _362 = annotate(List[Optional[Tensor]], [curr_keep_indices])
          _363 = torch.index(curr_indices, _362)
</code></pre>
<p>I’ve already included these dependencies:</p>
<p>implementation ‘org.pytorch:pytorch_android:1.13.0’<br>
implementation ‘org.pytorch:pytorch_android_torchvision:1.13.0’<br>
implementation ‘org.pytorch:torchvision_ops:0.13.0’<br>
implementation ‘com.facebook.soloader:nativeloader:0.8.0’</p>
<p>Can anyone help me understand and resolve this error?</p>
2x slowdown when directly operating on parameter tensor in forward function

jrh: <p>Hi all,</p>
<p>I have a linear layer that I initialize in the model class called <code>self.X</code></p>
<p>Previously in my <code>forward</code> function, I was using a reshaped version of this layer:</p>
<p><code>X_nt = self.X.weight.expand(1,-1,-1)</code></p>
<p>This layer scales quadratically with my input dimensionality, so to scale this up, I figure I would rather directly operate on the original <code>self.X</code> instead of copying this into a new variable <code>X_nt</code></p>
<p>These two variables point to different memory addresses, so it seems this is a worthy cause to free up memory (though I could be mistaken here).</p>
<p>However, when I delete <code>X_nt</code> and directly substitute in its expanded version, my model suddenly becomes a little over twice as slow with much worse performance. I figured there’s something wrong with the way I’m storing gradients.</p>
<p>I turned <code>optimizer.zero_grad</code> into <code>model.zero_grad</code> and put this at the end of my training loop to no avail. My forward function is a bit unconventional in that it returns a loss since there’s a time dependency I need to track per forward pass, but the algorithm works fine before this change. Here is the training loop I’m using:</p>
<pre><code class="lang-auto">    def train_loop(self, dataloader, optimizer, clip_value=-1):
        losses = []
        self.train()
        for batch, data in enumerate(dataloader):
            # optimizer.zero_grad()
            loss = self(data)
            loss.backward()
            # default clip value is -1, so we don't clip unless explicitly specified
            if clip_value &gt; 0:
                torch.nn.utils.clip_grad_norm_(self.parameters(), max_norm=clip_value)
            optimizer.step()
            losses.append(loss.item())
            self.zero_grad()

        avg_epoch_loss = np.mean(losses)

        return avg_epoch_loss
</code></pre>
<p>Any help would be greatly appreciated!</p>

ptrblck: <p>I don’t know how you’ve profiled your code but if you are using your GPU, note that CUDA operations are executed asynchronously and you would thus need to synchronize your code before starting the stopping host timers. Otherwise your profile might be misleading. If that’s not an issue, check if the same kernels are called in both versions. If not, try to narrow down which operation or layer changes.</p>

jrh: <p>Thanks a ton for your reply! I will look into this and update my post later in case anyone else has the same issue</p>
What is the output of a TopKPooling layer?

David_Ireland: <p>I am trying to implement a <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.pool.TopKPooling" rel="noopener nofollow ugc">TopKPooling</a> layer but the output is not documented. I am get a tuple of 6 objects – the first two items look to be the new node feature matrix and edge indices, the third is a None type (probably because I am doing standard TopKPooling), the fourth looks to be the new batch tensor which indicates which of the nodes belong to which graph. The fifth and sixth I am less sure on: they are both 1D tensors of length |V’| where V’ is the new coarsened node set – the fifth looks to be strictly integers with no gradient attached so it could be some kind of labelling and the sixth looks to be floats with a gradient.</p>
<p>could some please help me understand what these outputs are, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The code I am using is as follows:</p>
<pre><code class="lang-auto">class KPooling(nn.Module):

    def __init__(self, ratio, input_size, hidden_size, output_size):

        super(KPooling, self).__init__()

        self.gcn1 = SAGEConv(input_size, hidden_size)
        self.first_pool_layer = TopKPooling(hidden_size, ratio)

    def forward(self, data):
        x, edge_index, batch = data.x, data.edge_index, data.batch

        x = self.gcn1(x, edge_index)
        x, edge_index, _, batch, ?, ? = self.first_pool_layer(x, edge_index, batch=batch)
</code></pre>
<p>where the <code>?</code> I’ve added to show that I am unsure on what these two tensor outputs are.</p>

GiorgioMB: <p>I know this is very late and possibly may only be used by future readers.</p>
<p>As I had the same question, here’s what I can infer from my testing:<br>
the ordering is the following:</p>
<pre data-code-wrap="python"><code class="lang-python">class YourModule(torch.nn.Module):
    def __init__(self, dropout, input_size, hidden_size, output_size):
        super(YourModule, self).__init__()
        self.gcn1 = GCN(input_size, hidden_size)
        self.topkpool1 = TopKPooling(16 * num_heads, ratio = 1 - dropout)
        

    def forward(self, data1):
        x1 = self.gcn1(data1.x, data1.edge_index))
        x1, edge_index, edge_attributes, batch, idx_of_nodes_retained, score_of_nodes = self.topkpool1(x = x1, edge_index = data1.edge_index, edge_attr = data1.edge_attr, batch = data1.batch)
</code></pre>
<p>Where:</p>
<ul>
<li><code>x1</code> is the smaller set of nodes,</li>
<li><code>edge_index</code> is the index of the edges retained,</li>
<li><code>edge_attributes</code> are the attributes of the retained edges</li>
<li><code>batch</code> the new, reduced batch</li>
<li><code>idx_of_nodes_retained</code> is a list of the nodes being retained (if node1, node2, node4 were retained, this would have [0,1,3])</li>
<li><code>score</code> is the score of the node, calculated as shown in the documentation of the layer</li>
</ul>
Abnormal forward time when increase batch size

tztztztztz: <p>When I increase batch size, the forward time increases unlinearly at some point.<br>
For example, the forward time for bs size 32, 64, 96 is 0.002, 0.003, 0.005s. but the forward time increase to 0.042s when I set batch size to 128.</p>
<p>I don’t understand why this happened. Could someone help me?</p>
<p>result screenshot<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/6468805e3887e87a3e1bbe097ba92700861b1731.jpeg" alt="image" data-base62-sha1="ekfDHtzarbTu1zkUtbhLfoffWaB" width="482" height="374"></p>
<p>reimplement code:</p>
<pre><code class="lang-auto">import torch 
from contextlib import contextmanager
from torch import nn

@contextmanager
def measure_time(label):
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    yield
    end.record()
    torch.cuda.synchronize()
    elapsed_time = start.elapsed_time(end) / 1000  
    print(f'{label} time: {elapsed_time:.3f}s')


test_bs = [2, 4, 8, 16, 32, 64, 96, 108, 128]
model = nn.Conv2d(384, 128, 3, 1, 1)
model.to('cuda')

for bs in test_bs:
    x = torch.randn(bs, 384, 32, 32).to('cuda')
    with measure_time(f'bs:{bs}'):
        y = model(x)

        

</code></pre>

tztztztztz: <p>BTW, torch version is 1.11.0+cu113</p>

ptrblck: <p>You could inspect the visual profile, e.g. from Nsight Systems, and check if lower batch size workloads are CPU-limited. The kernels should then show gaps between them and the GPU utilization would go down in these parts. If so, a larger batch size would increase the computation workload and would be visible in the end2end run.</p>

tztztztztz: <p>I don’t understand why this is related with CPU, as shown in the code, the model is just a simple Conv2d, and I just use random small tensor.</p>
--index-url to install pytorch

JuanFMontesinos: <p>Hi everyone,<br>
I was wondering why pytorch is installed relying on custom repositories with pip.<br>
Pytorch must be installed as:</p>
<pre><code class="lang-auto">pip install torch --index-url https://download.pytorch.org/whl/cu121
</code></pre>
<p>There different repositories for different cuda versions or cpu.<br>
This is a nightmare when specifiying pytorch as a requirement in python packages. Pretty much impossible to define dynamically (i.e. let the user choose a cuda version or cpu).</p>
<p>Jax on contrary nicely solves this enabling cuda as an extra: <code>pip  install jax[cuda_12]</code></p>

ptrblck: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="1" data-topic="198253">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>Pytorch must be installed as:</p>
<pre><code class="lang-auto">pip install torch --index-url https://download.pytorch.org/whl/cu121
</code></pre>
</blockquote>
</aside>
<p>That’s not true as <code>pip install torch</code> will install the latest stable PyTorch wheel with CUDA 12.1 dependencies.<br>
Nightly binaries, wheels with older (or newer) CUDA runtimes, non-Linux, etc. are hosted on custom S3 buckets.</p>

JuanFMontesinos: <p>Indeed, but that’s the case only for linux.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c382128643e50a31324d0d36bbc3c6fc9845be7.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c382128643e50a31324d0d36bbc3c6fc9845be7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c382128643e50a31324d0d36bbc3c6fc9845be7_2_690x285.png" alt="image" data-base62-sha1="d9ObqaSR6DiKg4GYXDILdeBbirt" width="690" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c382128643e50a31324d0d36bbc3c6fc9845be7_2_690x285.png, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c382128643e50a31324d0d36bbc3c6fc9845be7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c382128643e50a31324d0d36bbc3c6fc9845be7.png 2x" data-dominant-color="F6E1DE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">930×385 23.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
For windows it’s the other way around and default is cpu.<br>
And still offering the possibility of using cpu or gpu+cuda is really hard using index-url when making packages.</p>

Ronica_Yu: <p>Encountered the same issue, when we have to build for both cpu &amp; gpu+cuda.<br>
I like the old way of using different package names…</p>
Training multiple heads in a hierarchy

volkan: <p>Hi!<br>
I am trying to train a multiple heads using a ResNet backbone. Currently I am training the ResNet + Head0 first then (using outputs of ResNet backbone) training the other head, Head1. I am doing it like this because I do not want Head1 to affect the backbone at all, however, as I am unable to train them simultaneously this is taking longer and I am unable to observe the convergence characteristics of Head1 when the backbone has not converged fully.<br>
I wanna train (Backbone+Head0) and (Head1) as asynchronously as possible, with Head1 not causing Backbone’s parameters to be updated but only accessing those to read and perform inference to use as the input of its training.<br>
Even though the problem is easy to define in terms of its mathematics, I do not even know where to start.<br>
Can you please help me?</p>

ptrblck: <aside class="quote no-group" data-username="volkan" data-post="1" data-topic="201855">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/94ad74/48.png" class="avatar"> volkan:</div>
<blockquote>
<p>with Head1 not causing Backbone’s parameters to be updated</p>
</blockquote>
</aside>
<p>You can <code>.detach()</code> the output of the backbone before passing it to <code>Head1</code> and can directly pass it to <code>Head0</code>.</p>
Compute gradient with respect to the input for high-dimensional output

Fadia_Bkf: <p>Hello,<br>
I am training  a model on sequential labelling task and I want to compute the gradient of the output with respect to input. I want to do that only for specific positions and not all the sequence, how can I do this in the most efficient way ?</p>

soulitzer: <p>You can use the <code>inputs=</code> arguments to backward, or use <code>.grad()</code>  <a href="https://pytorch.org/docs/stable/generated/torch.autograd.grad.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.grad — PyTorch 2.3 documentation</a> to specify which inputs you would like to compute derivatives with respect to.</p>
How to make the first linear layer compatible with the last pruned convolutional layer?

DrGlitch: <p>I have pruned the conv layers of a CNN. Now I need to make the last conv layer compatible with the fist linear layer. How to do it?</p>

ptrblck: <p>Assuming you have reduced the number of output channels of the conv layer you would also need to reduce the <code>in_features</code> by the same factor. Note that activations are usually flattened before being passed to the first linear layer from <code>[batch_size, channels, height, width]</code> to <code>[batch_size, channels*height*width]</code>, so the channels are a multiplicative factor.</p>
What does Conv4_x, Conv8_x means in SSD

palguna_gopireddy: <p>In the <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.detection.ssd300_vgg16.html" rel="noopener nofollow ugc">SSD model</a> presented in the <a href="https://arxiv.org/abs/1512.02325" rel="noopener nofollow ugc">paper</a>, it is said that the base network is considered as <a href="https://arxiv.org/abs/1409.1556" rel="noopener nofollow ugc">VGG16</a> and the extra feature layers are added at the end of it that allows feature maps to be produced at different scales and aspect ratios.</p>
<p><strong>My question is</strong> that in the architecture shown in Fig.2 (shown below) in the <a href="https://arxiv.org/abs/1512.02325" rel="noopener nofollow ugc">SSD paper</a>, the convolution layers have notations shown like <code>Conv5_3</code> , <code>Conv4_3</code>, for the base network <code>Conv8_2</code>,  <code>Conv9_2</code>,  <code>Conv10_2</code> for the added features layers.</p>
<p>What does this <strong>_2, _3</strong> notation means in this?</p>
<p>I have seen the same notations being used in the <a href="https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/" rel="noopener nofollow ugc">SSD model description page</a>, where base network VGG16 change to ResNET50 and used notations like <strong>Conv5_x, Conv4_x</strong>.</p>
<p>What does this <strong>_x</strong> means for the convolution layer notation?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc2ed7dea08841d8ecbf15af4ca8b87bdf4490e.png" data-download-href="https://discuss.pytorch.org/uploads/default/1bc2ed7dea08841d8ecbf15af4ca8b87bdf4490e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc2ed7dea08841d8ecbf15af4ca8b87bdf4490e.png" alt="image" data-base62-sha1="3XAwo1tO295R8GhEf5P2fYHcPQq" width="690" height="197" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1067×306 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Why there is no support for communicating between linux and macos

xuanhua: <p>Hi, guys</p>
<p>I have a m1-ultra mac studio and a linux box and I want to use both of them for distributed model training. But I found that gloo does not support communicating between linux and macos. I want to the reason and how much effort I need to make this work.</p>
<p>Expect your help.</p>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.PngImagePlugin.PngImageFile'>

Priyank_Raj_Sharma: <p>trans = transforms.Compose([<br>
transforms.Resize((224, 224)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) # imagenet<br>
])</p>
<p>data = datasets.VOCSegmentation(root=‘data’, year=‘2012’, image_set=‘train’, download=True, transform=trans)<br>
data_loader = DataLoader(data, batch_size=8, shuffle=True)<br>
for x, y in data_loader:<br>
print(x.shape)<br>
print(y.shape)<br>
break</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in &lt;cell line: 1&gt;()<br>
----&gt; 1 for x, y in data_loader:<br>
2     print(x.shape)<br>
3     print(y.shape)<br>
4     break</p>
<p>6 frames<br>
/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/collate.py in collate(batch, collate_fn_map)<br>
150                 return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed]<br>
151<br>
 → 152     raise TypeError(default_collate_err_msg_format.format(elem_type))<br>
153<br>
154</p>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.PngImagePlugin.PngImageFile’&gt;</p>
<p>I am using ToTensor but still getting error</p>

Priyank_Raj_Sharma: <p>Found it , The problem is because of target_transform. here is the sample code which works</p>
<p>trans = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Resize((572, 572))<br>
])</p>
<p>trans_target = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Resize((388, 388))<br>
])</p>
<p>data = datasets.VOCSegmentation(root=‘data’, year=‘2012’, image_set=‘train’, transform=trans, target_transform= trans_target)</p>
Getting DDP outputs with example ID

gagan: <p>I have a list of queries for which I’m trying to get the embeddings using DDP. Currently, the way I get that is by collecting the <code>(example_id, embedding)</code> on each device and then writing them to separate files with the name `{gpu_id}_output.txt’.</p>
<p>Is there any better way to gather the <code>(example_id, embedding)</code> file with DDP? I can think of the following ways:</p>
<ol>
<li>
<p>Do <code>torch.distributed.all_gather(example_id)</code> and <code>torch.distributed.all_gather(embedding)</code> and then write to <code>output.txt</code> only on GPU 0. However, my concern is that <code>all_gather</code> is not guaranteed to gather the example_ids and embeddings in the same order. For instance, the gathered <code>example_id</code> could be from GPUs [3,0,1,2] while gathered <code>embedding</code> could be from GPUs [0,1,2,3]. Is there a way to reconcile which example_id corresponds to which embedding when doing all_gather?</p>
</li>
<li>
<p>The other approach I can think of is by creating tuple of (example_id.numpy(), embedding.numpy()) and then using <code>torch.distributed.all_gather_object</code>.</p>
</li>
</ol>
<p>What is the recommended way to handle this?</p>
<p>Thanks!</p>
ToDtype from float to uint8 behaves unexpected

Erik_Grosskopf: <p>Hello,</p>
<p>I want to transform a float16 tensor to a uint8 tensor and scale the values from 0…1 to 0…255 to save it as an image. When using float16 a perfectly even 1.0 is transformed to a 0? Maybe a float thingy overflow? Or am I doing something wrong?<br>
Or does scale work differently when applied for float16?<br>
When using float32 the transform works as expected and results a 255.</p>
<pre><code class="lang-auto">v2.ToDtype(torch.uint8, scale=True)(torch.tensor(1., dtype=torch.float16))
Out[7]: tensor(0, dtype=torch.uint8)

v2.ToDtype(torch.uint8, scale=True)(torch.tensor(1., dtype=torch.float32))
Out[8]: tensor(255, dtype=torch.uint8)
</code></pre>
FCNN seems like not optimize

BWW: <p>I have two measured signals called ‘Cobs’ and ‘kernel’, the ‘kernel’ is system response and I want to get ‘Ctrue’ (original signal). I build a FCNN to solve this problem</p>
<p>I have a simple fully connected NN:</p>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
<p>class DeconvolutionNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(DeconvolutionNN, self).<strong>init</strong>()<br>
self.layer1 = nn.Linear(149, 256)<br>
self.layer2 = nn.Linear(256, 128)<br>
self.layer3 = nn.Linear(128, 149)<br>
self.relu = nn.ReLU()</p>
<pre><code>def forward(self, x):
    x = self.relu(self.layer1(x))
    x = self.relu(self.layer2(x))
    x = self.layer3(x)  
    return x
</code></pre>
<p>From my understanding, the input will be ‘Cobs’ and the DeconvolutionNN will give me a random guess ‘Ctrue’. After that, I need to compute the loss between ‘Ctrue’ and ‘Cobs’, so I defined a personal L2 loss function, I called conv1d since the relationship between ‘Cture’, ‘Cobs’ and ‘kernel’ is: Cobs = Ctrue conv kernel:</p>
<p>I defined the Loss function like this:</p>
<p>def compute_loss(model, y_true, y_pred, kernel, device):</p>
<pre><code>kernel_tensor = torch.tensor(kernel, dtype=torch.float32).view(1, 1, -1).to(device)
y_pred = y_pred.view(y_pred.shape[0], 1, -1)
y_conv = F.conv1d(y_pred, kernel_tensor, padding='same').view(y_pred.shape[0], -1)

return torch.mean((y_true - y_conv) ** 2)
</code></pre>
<p>After this, I initialize the model and trying to optimize the model:</p>
<p>Cobs = torch.tensor(twilite, dtype=torch.float32).to(device)</p>
<p>model = DeconvolutionNN().to(device)<br>
optimizer = optim.Adam(model.parameters(), lr=0.01)<br>
Cturehat = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
train_loss = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>epochs = 3000<br>
for epoch in range(epochs):<br>
model.train()<br>
optimizer.zero_grad()<br>
Ctrue_hat = model(Cobs)<br>
loss = compute_loss(model, Cobs, Ctrue_hat, kernel, device)<br>
loss.backward()<br>
optimizer.step()<br>
Cturehat.append(Ctrue_hat.detach().numpy())</p>
<pre><code>if (epoch + 1) % 100 == 0:
    print(f'Epoch [{epoch+1}/{epochs}], Loss: {loss.item():.4f}')
    train_loss.append(loss.item())
</code></pre>
<p>The Loss I got is nearly 47112. I tried different things like increasing the training epochs, increase or decrease the learning rate but the Loss doesn’t change. I don’t know if my logic here is making sense or any posibile explaination that why this doesn’t work? Thanks</p>
Minmax confusion on VS2022 & build

QFuser: <p>Okay my problem is since pytorch 2.1 before the were no problem . Since then all  max(…) function get automatically assumed to be std::max(…) . It is easy to suppress the problem with define NOMINMAX when I load the pre build binaries . Someone deleted properly half a line of code there somewhere.</p>
<p>My problem now is I try to build myself the c++ backend with DNN and MPI so I run into this</p>
<p>E:\pytorch\third_party\gloo\gloo\mpi\context.cc(43): warning C4297: “gloo::mpi::MPIScope::~MPIScope”: The function throws an unexpected exception<br>
E:\pytorch\third_party\gloo\gloo\mpi\context.cc(43): note: Destructor or deallocator has an (possibly implicit) non-throwing exception specification<br>
E:\pytorch\third_party\gloo\gloo\mpi\context.cc(105): error C2672: “std::max”: No matching overloaded function found</p>
<p>So I’m not sure where I can add my NOMINMAX</p>

QFuser: <p>Okay I read it gloo doesn’t work with windows USE_GLOO=0</p>
<p>but now I have this error</p>
<blockquote>
<p>FAILED: bin/torch_cuda.dll lib/torch_cuda.lib C:\WINDOWS\system32\cmd.exe /C “cd . &amp;&amp; “C:\Program Files\CMake\bin\cmake.exe” -E vs_link_dll --intdir=caffe2\CMakeFiles\torch_cuda.dir --rc=C:\PROGRA<del>2\WI3CF2</del>1\10\bin\100226<del>1.0\x64\rc.exe --mt=C:\PROGRA</del>2\WI3CF2<del>1\10\bin\100226</del>1.0\x64\mt.exe --manifests – C:\PROGRA<del>1\MICROS</del>2\2022\COMMUN<del>1\VC\Tools\MSVC\1439</del>1.335\bin\Hostx64\x64\link.exe /nologo <span class="mention">@CMakeFiles</span>\torch_cuda.rsp /out:bin\torch_cuda.dll /implib:lib\torch_cuda.lib /pdb:bin\torch_cuda.pdb /dll /version:0.0 /machine:x64 /ignore:4049 /ignore:4217 /ignore:4099 /INCREMENTAL:NO &amp;&amp; cd .” LINK: command “C:\PROGRA<del>1\MICROS</del>2\2022\COMMUN<del>1\VC\Tools\MSVC\1439</del>1.335\bin\Hostx64\x64\link.exe /nologo <span class="mention">@CMakeFiles</span>\torch_cuda.rsp /out:bin\torch_cuda.dll /implib:lib\torch_cuda.lib /pdb:bin\torch_cuda.pdb /dll /version:0.0 /machine:x64 /ignore:4049 /ignore:4217 /ignore:4099 /INCREMENTAL:NO /MANIFEST:EMBED,ID=2” failed (exit code 1120) with the following output: Creating library “lib\torch_cuda.lib” and object “lib\torch_cuda.exp”. LINK : warning LNK4098: Default library “LIBCMT” conflicts with other libraries; use /NODEFAULTLIB:library.</p>
</blockquote>
<p>then the linker failed on magma couldn’t resolve also it seems not to resolve the intel MKL correctly</p>
Coming up example either specificy abcbType, scaleType and ComputeType

jury_duty: <p>i am looking for any python code examples, either of  following.</p>
<ul>
<li>model training library which I can specifcy abcType and computeType or scaleType all defined in cuda framework header files.</li>
<li>or simply gemm3 example which also can specify these 3 entities.<br>
So far I am able to do simple example for gemm3 but not sure how to specify abcType, computeType or scaleType differently in the example.</li>
</ul>
<p>i cam up following snippet of code but not sure where or how to specify those:<br>
I put this post in NLP, reason being usually in NLP tasks, speciflally inferencing these types are specified for lesser accuracy types to speed up inference.</p>
<p>import torch<br>
import numpy as np</p>
<p>cuda = torch.device(‘cuda’)</p>
<p>m = 1024<br>
n = 1024<br>
k = 11024</p>
<p>t1 = torch.randn([m,n], dtype=torch.bfloat16, device=cuda)<br>
t2 = torch.randn([n,k], dtype=torch.bfloat16, device=cuda)<br>
print("t1, t2: ", t1.shape, t2.shape)<br>
t3 = torch.mm(t1, t2)<br>
<span class="hashtag-raw">#print</span>(np.array(t3).shape)<br>
print(t3.shape)</p>
Does torch.compile has garbage collection mechanism for intermediate tensors inside a compiled subgraph?

littleq: <p>Hi, The following fx graph is after backend compile, and there are two fused partitions.</p>
<pre><code class="lang-auto">def forward(self, arg0_1: "f32[16, 16, 1, 1]", arg1_1: "f32[16, 16, 1, 1]", arg2_1: "f32[1, 16, 8, 8]"):
    fused_1: "f32[1, 16, 8, 8]" = self.fused_1(arg2_1, arg0_1, arg1_1);  arg2_1 = arg1_1 = None
    _to_copy: "f32[1, 16, 8, 8]" = torch.ops.aten._to_copy.default(fused_1, …, device = device(type='cpu'));  fused_1 = None
    relu: "f32[1, 16, 8, 8]" = torch.ops.aten.relu.default(_to_copy);  _to_copy = None
    _to_copy_1: "f32[1, 16, 8, 8]" = torch.ops.aten._to_copy.default(relu, …, device = device(type='hpu', index=0));  relu = None
    fused_0: "f32[1, 16, 8, 8]" = self.fused_0(_to_copy_1, arg0_1);  _to_copy_1 = arg0_1 = None
    _to_copy_2: "f32[1, 16, 8, 8]" = torch.ops.aten._to_copy.default(fused_0, …, device = device(type='cpu'));  fused_0 = None
    relu_1: "f32[1, 16, 8, 8]" = torch.ops.aten.relu.default(_to_copy_2);  _to_copy_2 = None
    _to_copy_3: "f32[1, 16, 8, 8]" = torch.ops.aten._to_copy.default(relu_1, …, device = device(type='hpu', index=0));  relu_1 = None
    return (_to_copy_3,)
</code></pre>
<p>When I run the compiled module, I saw that:</p>
<ul>
<li>fused_1 is freed before allocating _to_copy_1</li>
<li>fused_0 is freed before allocating _to_copy_3</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2ce473f2f94c5b7d9c3a7b87ed11911610967f59.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ce473f2f94c5b7d9c3a7b87ed11911610967f59" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce473f2f94c5b7d9c3a7b87ed11911610967f59_2_690x278.png" alt="image" data-base62-sha1="6p8sWwilgjpMxuKw5kuBRbe4Neh" width="690" height="278" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce473f2f94c5b7d9c3a7b87ed11911610967f59_2_690x278.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce473f2f94c5b7d9c3a7b87ed11911610967f59_2_1035x417.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2ce473f2f94c5b7d9c3a7b87ed11911610967f59_2_1380x556.png 2x" data-dominant-color="E1D3B8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1950×788 64.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems that pytorch has automatically freed the fused_1 and fused_0 after its last use. So my question is where this optimization happens? could anyone help to answer the question? Thanks~</p>
Backprop through indices based on some operations

Rahul_Vashisht: <p>Consider three tensors <code>a = 5*torch.rand(5)</code> , <code>b = torch.zeros(5,5)</code> and <code>c = torch.zeros(5)</code> . I just realized that <code>b[a.long(),a.long()] = c</code> disconnects <code>a</code> from the computation graph due to the casting operation (<code>a.long()</code> ). In my case, <code>a</code> has history and <code>a.requires_grad=True</code> . How can I avoid it getting disconnected during this operation?</p>

ptrblck: <p>Integer tensors are not usefully differentiable as described <a href="https://discuss.pytorch.org/t/attributeerror-nonetype-object-has-no-attribute-data/56732/31">here</a>.</p>

Rahul_Vashisht: <p>Is there a way to backpropagate through indices (selected based on some parameters).</p>

ptrblck: <p>You could write a custom <code>autograd.Function</code> and return a zero gradient everywhere besides the rounding points, but I wouldn’t know how this would be useful in any way.</p>
Setting num_workers>0 increase GPU memory of only 1 GPU

TeDataPro: <p>Hello,</p>
<p>I am training a model on 2 GPUs. In the <em>get_item</em> of my custom dataloader, pre-processing is done on GPU to increase speed.</p>
<p>When I set num_workers to 0, the memory usage of my 2 GPUs are the same : 22k/32k.</p>
<p>However, When I set num_workers=2 with the following code:</p>
<pre><code class="lang-auto">trainloader = DataLoader(train, batch_size=4, shuffle=True, num_workers=2, persistent_workers=True)
</code></pre>
<p>The memory usage of my first GPU becomes 31k/32k, while the memory usage of my second GPU remains 22k/32k.</p>
<p>Please, do you have any idea ?</p>
<p>EDIT : When I display the device used in the dataloader, only GPU 0 is used. GPU 1 is never used.</p>
<pre><code class="lang-auto">array = np.load(path)
tensor = torch.from_numpy(tensor).to(self.device)
print(tensor.device) # shows only cuda:0
</code></pre>
<p>where self.device is initialized in <em>init</em> such as :</p>
<pre><code class="lang-auto">self.device = torch.device('cuda')
</code></pre>

ptrblck: <p>You are using the default CUDA device in each worker so the memory increase on the default GPU is expected. Besides creating additional CUDA contexts the actual data loading will also use memory.</p>

TeDataPro: <p>Thank you for your answer.</p>
<p>Sorry I’m new to parallelism so I’m not sure to understand, what can I change from my implementation to solve this issue ?</p>
<p>Thanks in advance!</p>

ptrblck: <p>If you want to use a worker per GPU, you could use the worker ID and call <code>torch.cuda.set_device</code> with it to load the sample on the specified device.<br>
The common approach is to let the CPU load the samples and to move the batch to the GPU inside the training loop.</p>
How to download EMINST by using Pytorch?

nova001: <p>I ask you  a question. until 2023 December, I could download EMINST by using this code at google Colaboratory</p>
<pre><code class="lang-auto">import torch
import torchvision

emnist_data = torchvision.datasets.EMNIST(
    './EMNIST', 
    split='balanced',
    train=True, download=True, transform=torchvision.transforms.ToTensor())

data_loader = torch.utils.data.DataLoader(emnist_data, batch_size=100, shuffle=True)
</code></pre>
<p>but three days ago, I found that I could not download EMINST at Google Colaboratory.</p>
<p>error is here<br>
<code>File not found or corrupted</code></p>
<p>If you know something about EMNIST,could you tell me why I cannot  download it?</p>

ptrblck: <p>The download seems to be broken again as also mentioned in the <a href="https://github.com/pytorch/vision/issues/5662">corresponding issue</a>. CC <a class="mention" href="/u/pmeier">@pmeier</a></p>

nova001: <p>Thank you for replying!</p>

Tony-Y: <p>Try <a href="https://github.com/Tony-Y/pytorch_warmup/blob/master/examples/emnist/download.py" rel="noopener nofollow ugc">download.py</a></p>