akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does every image range from [0,255]

HaydenL: <p>I am somewhat new to image preprocessing and CNNs and was wondering why this is the case. I was testing several random gray images where I believed there did not exist a pure black or pure white portion of the image, though I still got 0 as the min pixel and 255 as the max pixel.</p>
<p>It seems that all images have 0 as the min and 255 as the max even if the image does not display a pure white or black in a grayscale setting.</p>
<p>I tested by the PIL library like this:</p>
<pre><code class="lang-auto">x = numpy.asarray(PIL.Image.open('gray.webp'))
print(x.min())
print(x.max())
</code></pre>
<p>Is pil manipulating the values so that it fits into a [0,255] range, or is something else going on behind the scenes that I do not understand.</p>

HaydenL: <p>Never mind, I have found an image that does produce a range that is not [0,255], thought it is very specific. It is a picture of pure gray image with no “objects”.</p>
Does pytorch have a tensor split method that does interleaving subslicing?

Fei_Liu: <p>E.g. my input tensor is <code>tensor([0, 1, 2, 3, 4, 5, 6])</code> , to be split into 3 chunks, where the intended outputs are:</p>
<pre><code class="lang-auto">tensor([0, 3, 6])
tensor([1, 4])
tensor([2, 5])
</code></pre>
<p>The tensor methods <code>chunk</code>, <code>split</code>, <code>tensor_split</code> all split the tensor into contiguous sub-tensors. I didn’t find one to split in the interleaving way above. Do we have any?</p>

eqy: <p>I think you can get partially there with a combination of <code>reshape</code> and <code>chunk</code> (might be missing a <code>squeeze</code> depending on your use-case):</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t = torch.arange(9)
&gt;&gt;&gt; t = t.reshape(3,3)
&gt;&gt;&gt; t
tensor([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])
&gt;&gt;&gt; t.chunk(3, dim=1)
(tensor([[0],
        [3],
        [6]]), tensor([[1],
        [4],
        [7]]), tensor([[2],
        [5],
        [8]]))
&gt;&gt;&gt;
</code></pre>
Build PyTorch from source code for old GPU on Windows

SirLance: <p>Hello all!<br>
So, I have an old GPU - GTX780. If I correctly understand, last version CUDA Toolkit for it is 11.4.4 and version of driver is 474.30, CUDA_ARCH_LIST=“3.5”<br>
Is it possible to build PyTorch v.2.0.0 from source or maybe other last versions?<br>
I didn’t find MAGMA for CUDA 11.4 for building and looks like it’s not exists.<br>
I will to try build from source on Windows 10.</p>

eqy: <p>It is definitely possible to build from source:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/pytorch#from-source">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch#from-source" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/a31414a8267d404e5a32f93deb14a7a2c7b859ac87ff4ffb0e6b80844303c3cd/pytorch/pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/pytorch#from-source" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with...</a></h3>

  <p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If you are not using MAGMA it is also possible to remove it from the build e.g., via the <code>USE_MAGMA=0</code> environment variable when running the build command.</p>
<p>Alternatively, if you are OK with using a somewhat older version to avoid building from source, it looks like <code>sm_35</code> should be supported by default in the published binaries to at least 1.8.1+, according to: <a href="https://discuss.pytorch.org/t/gpu-compute-capability-support-for-each-pytorch-version/62434/7" class="inline-onebox">GPU compute capability support for each pytorch version - #7 by moi90</a></p>
Failed to import `Ray` on jupyter notebook

Jaeseon: <p>While working on my school’s Linux server, I encountered an issue while attempting to install and import Ray in my Jupyter Notebook. I successfully installed the package <code>ray==2.4.0</code>, but encountered an error when trying to import it, specifically stating “no module named ray.__raylet”. However, I installed the same version of Ray on my local machine and it was installed and imported without any problems.</p>
<p>I’m seeking guidance on how to address this problem.</p>
Sizes do not match in scaled_dot_product_attention

Gary_W: <p>The following code is expected to work, but it is throwing an error.</p>
<pre><code class="lang-auto">q, k, v = q.reshape(-1, seq_len, self.num_attention_heads, self.attention_head_size).transpose(1, 2), \
      k.reshape(-1, seq_len, self.num_attention_heads, self.attention_head_size).transpose(1, 2), \
      v.reshape(-1, seq_len, self.num_attention_heads, self.attention_head_size).transpose(1, 2)

print(q.shape, k.shape, v.shape)

output = nn.functional.scaled_dot_product_attention(
   q, k, v, attn_mask, self.config.attention_probs_dropout_prob
)
</code></pre>
<pre><code class="lang-auto">torch.Size([80, 8, 128, 64]) torch.Size([80, 8, 128, 64]) torch.Size([80, 8, 128, 64])

  File "", line 81, in forward
    output = nn.functional.scaled_dot_product_attention(
RuntimeError: The size of tensor a (128) must match the size of tensor b (80) at non-singleton dimension 2
</code></pre>
<p>I am using pytorch 2.0.1.</p>

eqy: <p>Could you post more details about your setup such as the specific GPU you are using and a runnable reproducible example? I could not reproduce the error with:</p>
<pre><code class="lang-auto">import torch
from torch import nn

q = torch.randn(80, 128, 8, 64, device='cuda', dtype=torch.half).transpose(1, 2)
k = torch.randn(80, 128, 8, 64, device='cuda', dtype=torch.half).transpose(1, 2)
v = torch.randn(80, 128, 8, 64, device='cuda', dtype=torch.half).transpose(1, 2)

print(q.shape, k.shape, v.shape)

attn_mask = torch.ones(128, 128, device='cuda', dtype=torch.half)

output = nn.functional.scaled_dot_product_attention(
   q, k, v, attn_mask=attn_mask)
</code></pre>
<pre><code class="lang-auto">torch.Size([80, 8, 128, 64]) torch.Size([80, 8, 128, 64]) torch.Size([80, 8, 128, 64])
</code></pre>
<p>EDIT: could it be that your attention mask also needs to be [128, 128] when one of the dimensions is incorrectly set to 80?</p>
Loss.grad = None

Aml_Hassan: <p>Hi there,</p>
<p>I tried to build my own loss function, but I’m facing a problem as the loss.grad =None, although the loss.requires_grad = True.<br>
here’s my loss function:</p>
<pre><code class="lang-auto">def recall_prec(y_true, y_pred, gamma=1,beta=1):
    #y_pred_bin = torch.where(y_pred &gt;= 0.5, torch.ones_like(y_pred), torch.zeros_like(y_pred))
    #y_pred.requires_grad=True
    masked_pred = y_true * y_pred
    true_pos = torch.sum(masked_pred)
    false_negative = torch.sum(y_true) - true_pos
    false_pos = torch.sum(y_pred - masked_pred)
    
    #what if we reached a case where true_pos + false_pos =0 or true_pos + false_negative, then we're dividing by zero, so to be saver I added 1, it won't affect much 
    prec = true_pos / (true_pos + false_pos+1)
    recall = true_pos / (true_pos + false_negative+1)
    
    #print("prec", prec, "recall", recall)
    loss = ( ( (1-recall)*gamma + (1-prec)*beta )**2  )
    #loss.requires_grad = True
    print(loss.grad,loss.requires_grad)
    return loss
</code></pre>
<p>I’m using SGD as an optimizer, I tried to use Adam but still faced the same problem.<br>
Can you please help me with that?</p>

ptrblck: <p>Gradients will be calculated during the backward pass while your code does not show any <code>backward</code> calls. Thus seeing a <code>None</code> <code>.grad</code> attribute would be expected.<br>
Also, in case you are calling <code>backward</code> on the <code>loss</code> tensor you might need to call <code>retrain_grad()</code> in order to print the gradient afterwards.</p>

Aml_Hassan: <p>Thank you very much for your reply.<br>
I tried and printed loss.grad after doing loss.backward and still got NAN</p>

ptrblck: <p><code>NaN</code> is corresponding to “Not a number” while <code>None</code> is indicating the attribute wasn’t set so you need to be clear what exactly is returned.</p>
<p>It also works for me using <code>retain_grad</code> in this small code snippet:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
x = torch.randn(1, 10)
out = lin(x)

loss = out.mean()
loss.retain_grad()
loss.backward()
print(loss.grad)
# tensor(1.)
</code></pre>

Aml_Hassan: <p>I’m so sorry for that mistake I meant None.<br>
I’ll try the retain_grad and give you an updates.</p>

Aml_Hassan: <p>So quick update:</p>
<p>The loss.grad doesn’t give me None any more - thank you very much for your help -</p>
<p>It gives me :  tensor (1.)  And it keept giving me the same results for over 20 epochs now, the loss is changing but the loss.grad doesn’t change. Is that normal  ? Or do I need to do something differently?<br>
I’m using SGD as an optimizer.</p>

ptrblck: <p>Yes, this is expected, since you are passing an implicit <code>1</code> to the <code>backward</code> calls since <code>dLoss/dLoss = 1</code>.</p>

Aml_Hassan: <p>Thank you very much for the info <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to extract target from a dataset made of dictionary

peony: <p>I am using a dictionary for my dataset, here is two of the items:</p>
<pre><code class="lang-auto">metaitem {'imname': 'voc_dataset\\VOCdevkit\\VOC2007\\JPEGImages\\000003.jpg', 'anno_id': 2, 'impath': 'voc_dataset\\VOCdevkit\\VOC2007\\JPEGImages\\000003.jpg', 'xml_parsed': {'annotation': {'folder': 'VOC2007', 'filename': '000003.jpg', 'source': {'database': 'The VOC2007 Database', 'annotation': 'PASCAL VOC2007', 'image': 'flickr', 'flickrid': '138563409'}, 'owner': {'flickrid': 'RandomEvent101', 'name': '?'}, 'size': {'width': '500', 'height': '375', 'depth': '3'}, 'segmented': '0', 'object': [{'name': 'sofa', 'pose': 'Unspecified', 'truncated': '0', 'difficult': '0', 'bndbox': {'xmin': '123', 'ymin': '155', 'xmax': '215', 'ymax': '195'}}, {'name': 'chair', 'pose': 'Left', 'truncated': '0', 'difficult': '0', 'bndbox': {'xmin': '239', 'ymin': '156', 'xmax': '307', 'ymax': '205'}}]}}}
(375, 500, 3)
</code></pre>
<pre><code class="lang-auto">metaitem {'imname': 'voc_dataset\\VOCdevkit\\VOC2007\\JPEGImages\\000002.jpg', 'anno_id': 1, 'impath': 'voc_dataset\\VOCdevkit\\VOC2007\\JPEGImages\\000002.jpg', 'xml_parsed': {'annotation': {'folder': 'VOC2007', 'filename': '000002.jpg', 'source': {'database': 'The VOC2007 Database', 'annotation': 'PASCAL VOC2007', 'image': 'flickr', 'flickrid': '329145082'}, 'owner': {'flickrid': 'hiromori2', 'name': 'Hiroyuki Mori'}, 'size': {'width': '335', 'height': '500', 'depth': '3'}, 'segmented': '0', 'object': [{'name': 'train', 'pose': 'Unspecified', 'truncated': '0', 'difficult': '0', 'bndbox': {'xmin': '139', 'ymin': '200', 'xmax': '207', 'ymax': '301'}}]}}}
(500, 335, 3)
</code></pre>
<p>I am looping through the dataset items like this:</p>
<pre><code class="lang-auto">for imname, metaitem in metadata_test.items():
</code></pre>
<p>but this approach doesn’t take the ‘name’ key or label which is what I am trying to predict for training/eval. I believe I would need to convert these names to indices too.</p>
<p>Any idea how to extract the label from this dictionary to be use in the for loop above? I am sorry to ask its my first time dealing with dictionaries.</p>

Karthik_Ganesan: <p>Based on just what you have posted these are nested dictionaries but no actual images or labels, which is what you need for training.</p>
<p>From the name <code>metadata</code> it seems like you need to use these to get the actual images and labels for training.</p>
<p>This looks to be part of some larger code base. Can you share the rest of it so it’s easier to understand what exactly you are trying to do.</p>
Quantization Float to Int8

WilsonNeto: <p>Hi there! Well I am trying to persist 500 million Torch vectors (1024 dim float dtype). Since it take up too much memory space, I was suggested to do a quantization (from Float to Int8). Being really sincere, I do not know how to do that and as usual I will keep researching. In the meanwhile some nice help would be really appreciated. Thanks so much</p>

Karthik_Ganesan: <p>How much are you hoping to reduce storage by? Converting to <code>int8</code> will cause you to lose information. Maybe you could try converting to <code>float16</code> instead of <code>float32</code> and see if that is enough first? That is likely to lose less information than converting to <code>int8</code>. For a tensor <code>x</code>, you can convert it to use <code>float16</code> using <code>x.half()</code>.</p>
How can I ignore the first few items in the dataset and exclude them in the dataloader?

Daniel_Sabzi: <p>Hi everyone.</p>
<p>I have an issue which removing the first few datapoints in the dataset, which couldn’t find a solution for it in the forum.</p>
<p>I have made a dataset for a time series and had to pad the first 25 items in the dataset.<br>
Now I want to remove the padded datapoints but couldn’t find a way to do that. is there a way I can do this?</p>
<p>Cheers,</p>

eqy: <p>You could probably subclass the <code>Dataset</code> class you are using and override the <code>__len__</code> (to be 25 shorter than the actual length) and <code>__getitem__</code> functions (possibly to index into the parent class’s <code>__getitem__</code> with an offset)</p>

Daniel_Sabzi: <p><a class="mention" href="/u/eqy">@eqy</a><br>
Thanks a lot for your help.</p>
What is it exactly that DataLoaders do in DDP?

Clion: <p>So I have recently moved from a single GPU setting to a multi GPU setting. To date I have been using a custom class for loading data during nn training - will refer to this class as DM, and to Torch’s standard DataLoaders as DL. I do this because I see about a 450-550% speedup in all data loading/formatting ops when using DM as opposed to DL, which radically reduces my training time. In the single-GPU setting, this has 0 impact on the performance of the trained model or any loss statistics etc - in other words, it does nothing but cut down my training time by a factor of 4-5 for no cost to model performance.<br>
But since switching to a distributed setting, this is no longer the case. I have modified my DM class to (apparently naively) handle the distributed case, with each process started by DDP having its own DMₙ on GPU rank n. DMₙ will yield batches of data as tensors on device <code>torch.device(f"cuda:{n}")</code> for the forward pass executed by the DDP process with rank n. Very simple…or so I thought.</p>
<p>After training in the distributed setting like this for a while, I tested the model’s performance and it’s not just simply <em>not learning</em>, but is actually <em>actively getting worse</em> the more it’s trained.</p>
<p>But the point of this post isn’t a deep diagnosis of the problem…it’s too involved. All I am asking is this: My assumption is that the problem in overall model performance arises from the fact that the only part of my distributed setting which differs from the stock-standard pytorch ddp pipeline is the change from DL to DM, so is there anyone out there that can give me some idea of what else Torch’s DataLoaders do when using a <code>DistributedSampler</code>? Is there something else going on here other than partitioning the dataset into n subsets for n GPUs and sending the data to the right GPU at the right time?</p>
<p>I also note some distinct and regular differences in patterns displayed by GPU utilization shown in nvitop when I use Torch’s DL vs my own DM…but I won’t get into those just yet. I’m currently working my way through Torch’s DataLoader/DistributedSampler code to see if I can figure out what exactly is happening here myself, but figured I’d make this post to see if anyone out there can give me some high-level understanding of what else these things are doing outside of simply moving data to the right place at the right time.</p>

J_Johnson: <p>It doesn’t sound like an issue with your dataloader. But one way you could quickly test is by passing in a torch.rand() dataset of, say, 100 samples into the vanilla DataLoader and see if your model learns from it. But if you still have the same issue, you would at least know it is not due to your custom dataloader.</p>
Torch.empty() returns nan

leah: <p>Hi there,</p>
<p>I’m new to pytorch. I wanna create an uninitialized tensor as one of parameters in my model. I haven’t gone to the process of training or backpropagation yet. I just simply created a tensor with torch.empty() but got nan values by chance.</p>
<p>Here is the code and result. I sometimes got nan values inside the tensor sometimes not. It seems that the higher dimensions of size, the more likely to get nan.</p>
<hr>
<p>u = torch.empty(size=(15,15,15))</p>
<p>torch.sum(u.isnan()==True)<br>
Out[243]: tensor(6)</p>
<p>u = torch.empty(size=(15,15,15))</p>
<p>torch.sum(u.isnan()==True)<br>
Out[245]: tensor(0)</p>
<hr>
<p>Use torch.sum(u.isnan()==True) to check the tensor, you can see that the first time u has 6 nan values, the second time it doesn’t have nan values at all.</p>
<p>I wonder what is going on there. how can I completely avoid getting nan by creating an uninitialized tensor. If I replace those nan values with a specific value (e.g. zero), will it change the property of “uninitialized tensor” so to affect my model’s training process?</p>
<p>THANKS A LOT!!</p>

ptrblck: <p>This is expected since you are using <em>uninitialized memory</em> for the tensor which may contain any value including NaN.<br>
Initialize tensors via e.g. <code>torch.randn</code> to draw samples from a Normal distribution.</p>

leah: <p>Thank you so much!<br>
My purpose of creating this tensor data is to train it as a parameter (like a weight matrix) in my neural network, I’m still not sure if using torch.randn to initialize the tensor would be OK to train the parameter. (since i saw other people noramlly do torch.empty).<br>
I’ve read the other comments from you about the difference between initialized and unintialized tensor, but still I can’t figure out which function or type of data is proper for my situation.<br>
sorry if it sounds like a stupid question, as I’m new to pytorch and neural network modeling…</p>

ptrblck: <p>You should never use uninitialized memory as it can contain invalid data as already seen in your example. <code>torch.empty</code> is a fast way to initialize memory but requires you to fill it afterwards. E.g. it can be useful to preallocate a tensor which the result will be written to later. You should however never use it without a proper initialization.<br>
A trainable parameter is initialized randomly before it gets trained. <code>torch.randn</code> would be one proper way but you can also use any other random initialization method from e.g. <code>torch.nn.init</code>. Note the difference between random initialization and uninitialized memory.</p>

leah: <p>you’re right. A trainable parameter is initialized randomly before it gets trained. In that sense torch.randn would be a proper way to do the work.<br>
Truly appreciate your help!</p>
Torch_geometric: How can I create a lot of graphs efficiently?

mzimmer: <p>Hey! Not sure if this is the right place to ask regarding torch_geometric, but will try anyway.</p>
<p>I am working with a large number of graphs that all have the same number of nodes. In all examples, the graphs are initialized using the <a href="https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.data.Data.html#torch_geometric.data.Data" rel="noopener nofollow ugc">Data</a> structure. In my case, I often have a large number of graphs encoded as adjacency matrices in the format <code>(batch_size, n, n)</code>, however I am open to more memory efficient ways.</p>
<p>It seems like to only way to turn this into Data objects is to iterate over the batch dimension and create every Data object in a for-loop. This seems to be incredibly inefficient. How can I initialize all these graphs in a vectorized ways? It seems to be a waste to already have a large tensor and then iterating over it with a for-loop.</p>
<p>Any help is appreciated! Thanks!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/mzimmer">@mzimmer</a>,</p>
<p>Have you look at <code>torch.func.vmap</code>? I’m not sure how it will interact with <code>torch_geometric</code>, but there’s no harm in trying!</p>
<p>For example,</p>
<pre><code class="lang-auto">from torch.func import vmap

def f(x):
  return x.pow(2) #example function

x = torch.randn(1000,6,6) #1000s 6x6 matrices

vectorized_output = vmap(f, in_dims=(0))(x) 
#if f is nn.Module object, wrap the function in a torch.func.functional_call
</code></pre>

mzimmer: <p>Thanks for the fast reply, I have considered using <code>vmap</code> and will try. Still, I expected that there is some native way of doing so, since this seems to be a classical use case - I thought I might be missing something.</p>
<p>A bit of context: I am trying to generate a lot of graphs repeatedly using a VGAE. When sampling from the latent distribution and decoding, torch_geometric will give me either the edge list or adjacency matrix (in probabilistic form). Clearly, I can decode a lot of graphs simultaneously using batched tensors. However, turning these into Graphs again seems to require calling <code>Data</code> for each batch index, which seems like a complete bottleneck to my code.</p>

AlphaBetaGamma96: <p>One thing that comes to mind is if you can define your <code>data</code> class as an object that takes the edges as an input rather than within the constructor.</p>
<p>For example, let’s say you had 10 <code>nn.Module</code> objects with different sets of parameters. You can wrap your network in <code>torch.func.functional_call</code> to pass the parameters as an input, then just vmap over all sets. For all of the edges you have, are they all the same shape?</p>
Can I define n+1 outputs for n class classification?

bornexmachina: <p>I am sitting on an anomaly detection task and I had the following idea.</p>
<p>Assume I have 3 classes and each of them could have an anomalous sample. Then, I would train with 4 classes, the last one reserved for the anomaly. But in my training data I would never actually have any data on the class 4.</p>
<p>Now during inference, I would try to classify the data and if it is an anomaly it should have low scores for class 1 to 3 and thus be labeled as class 4.</p>
<p>Would this approach even compile? So expecting 4 class weights but having labels only for 3 classes?</p>

ptrblck: <aside class="quote no-group" data-username="bornexmachina" data-post="1" data-topic="181405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bornexmachina/48/58727_2.png" class="avatar"> bornexmachina:</div>
<blockquote>
<p>Now during inference, I would try to classify the data and if it is an anomaly it should have low scores for class 1 to 3 and thus be labeled as class 4.</p>
</blockquote>
</aside>
<p>This might be a good idea but I don’t think the model will automatically classify every “unknown” as class4 assuming it was never trained to output this label.</p>
<aside class="quote no-group" data-username="bornexmachina" data-post="1" data-topic="181405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bornexmachina/48/58727_2.png" class="avatar"> bornexmachina:</div>
<blockquote>
<p>Would this approach even compile?</p>
</blockquote>
</aside>
<p>Yes, it would technically work since you don’t need to use all logit outputs.</p>

bornexmachina: <p>technically correct is the best form of correct. Thanks!</p>
iOS: Could not run 'aten::as_strided' with arguments from the 'Metal' backend

Vij_Flex: <p>Hello! I am trying to run a model on iOS using the metal backend - it works completely fine on iOS using the CPU backend, even if it is quantized; however using the metal backend does not work even if it is unquantized.</p>
<p>The error I am receiving is:</p>
<p><code>Could not run 'aten::as_strided' with arguments from the 'Metal' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::as_strided' is only available for these backends: [CPU, QuantizedCPU, BackendSelect, Functionalize, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy].</code></p>
<p>I am using PyTorch Nightly. I assume that the error means that one of the operations is not supported on the Metal backend - is there any workaround for this?</p>
<p>Thanks!</p>
Error in transformer

chuck: <p>在Jupyter Notebook上运行Transformer时出现了一个错误：<br>
The computed hash &lt;&gt; don’t matched with the expected hash, delete the file and retry。</p>

ptrblck: <p>From Google Translate:</p>
<blockquote>
<p>I got an error when running Transformer on Jupyter Notebook:<br>
The computed hash &lt;&gt; don’t matched with the expected hash, delete the file and retry。</p>
</blockquote>
Model converge at high loss (good fit bad performance?)

Gears_Gears: <p>Hi!</p>
<p>a: My model converges and looks like a good fit (training and validation loss follow each other closely)<br>
The problem is, they stop (go horizontal) at a value (0.15 something) that’s higher than I would like (0.05!)<br>
Since the loss converged, I can’t reduce it by training more (e.g. if I increase patience of early stopping, that just lets the model overfit)<br>
b: my model is a good fit only in cross-validation. But when I go to testing, the model overfits (5 to 10 times the loss)<br>
I’m confused about this, since the point of cross-validation is to make sure that the model generalizes well. In this case, cross-validation tells me the model generalizes well, but the testing tells me the model does not generalize well.</p>
<p>For problems ‘a’ and ‘b’ above, what should I try?</p>
<p>Many Thanks!</p>
How to interpret torch summary output

leila000: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6a29b450f20ed9ced4448b0a9d78a9da7218bb9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b6a29b450f20ed9ced4448b0a9d78a9da7218bb9" title="Screenshot_20230604_134955_Samsung Internet"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6a29b450f20ed9ced4448b0a9d78a9da7218bb9_2_665x500.jpeg" alt="Screenshot_20230604_134955_Samsung Internet" data-base62-sha1="q3FhvT9qrsEGWhWNJ6EeHvO4lTP" width="665" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6a29b450f20ed9ced4448b0a9d78a9da7218bb9_2_665x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6a29b450f20ed9ced4448b0a9d78a9da7218bb9.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6a29b450f20ed9ced4448b0a9d78a9da7218bb9.jpeg 2x" data-dominant-color="EAEBED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20230604_134955_Samsung Internet</span><span class="informations">937×704 88.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i want to know how to interpret these output<br>
what does mult-adds mean?<br>
does it mean total computations to calculate one image?</p>
<p>and what does total estimated size means? does it means i need at least 602mb to run a image on model?</p>
<p>or is there any way to calculate how much memory do i need to run a image for a model?</p>

eqy: <p>Multiply-adds are used as shorthand for the total number of operations in the model as popular layers such as convolution and linear layers <em>multiply</em> weights with inputs and then <em>add</em> the results of the multiplication (possibly with a bias). A single multi-add would be e.g., a scalar <code>wx + b</code> and would correspond to <em>two</em> floating point operations. So the mult-adds would roughly be 1/2 the total number of floating point calculations per image.</p>
<p>Given that the total size is roughly the sum of the params + forward/backward pass size. I assume that this is roughly the total amount of memory needed to train the model.</p>
<p>However, the real number in practice is a bit tricker to obtain as there could be effects such as memory fragmentation that increase the total required memory needed to train the model. Also note that if you are just running inference without training you shouldn’t need the “total size” worth of memory as you can wrap the model execution with <code>torch.no_grad():</code> to discard intermediate activations during the forward pass.</p>
Train two models together problem: stop one and keep training the other?

Gears_Gears: <p>Hi! I am trying to train two models. The problem is a regression; setup is as follows:</p>
<p>network F and G. F turn input x into feature vector F(x), G takes F(x) and give G(F(x))<br>
This leads to some problems:<br>
1 since G’s input = F’s output, when F converges, I need to stop F and keep training G<br>
the way I ‘lock’ F is: put the loss.backward and optimizer.step under an if() statement<br>
Does this do what I think it does? (locks F, keep training G)<br>
2 following the rule ‘total parameters = 2 * number of samples + dimension of input’, should ‘total parameters’ apply to ‘parameters of F + parameter of G’ or F and G separately? (i.e. if total parameters = 10, should F and G have 10 params each, or should their number of params add up to 10?)</p>
<p>Any answers, critiques, and questions would be helpful!<br>
Many Thanks!</p>

eqy: <p>Re: 1 Yes, that should work, but you should also call <code>model.eval()</code> to prevent e.g., any normalization layers from having their running stats updated and wrap the frozen model’s forward execution in <code>with torch.no_grad():</code> to save memory.</p>
<p>Re 2: I’m not sure there are any hard and fast scaling rules like that, especially without knowledge of what domain you are currently working in. In general, most deep learning models these days err on the side of overparameterization.</p>

Gears_Gears: <p>Hi eqy,</p>
<p>Thanks for your help!<br>
1 I am adding these (no_grad() and eval()) to my code!</p>
<p>2 My input is a very long vector; my output is one number. Currently my F network has more than enough parameters, but my G network is ‘under-parametrized’. Increasing G’s parameters doesn’t seem to improve the results</p>
Initializing two networks from the same class with BatchNorm

cimuletz: <p>Hi, I’m a begginer and I’m attempting a student-teacher architecture by initializing two networks with different hidden sizes from the same class. It seems the first network initialization influences the second one ( I get different losses on the student network when initializing the teacher network first, even though I’m training the student network independently of the teacher). I am using BatchNorm after a Linear layer in my network class. So I’m guessing this is what causes the first initialization to influence the second, either the BatchNorm layer or Linear layer is keeping some running statistics from the first initialization. Any ideas how to solve this? Thanks.</p>

eqy: <p>Are you saying that the initial loss is different, or the network(s) is/are converging differently when one is initialized after the other. The former is expected as the random state has changed, the latter would be unexpected and shouldn’t be caused by the order of initialization.</p>
<p>If your issue is the former, you can try using <code>torch.manual_seed(seed)</code> before each initialization to prevent one from “influencing” the other. See also: <a href="https://pytorch.org/docs/stable/notes/randomness.html#controlling-sources-of-randomness" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 2.0 documentation</a></p>
Pass multiple parameters into DistributedDataParallel

MLAlex1: <p>What is the right way to use <code>DistributedDataParallel</code> when we have several things to optimize?<br>
I have a model that I am wrapping into <code>DistributedDataParallel</code><br>
But I also have another class with some parameters to optimize during training. Do I simply wrap it as well into <code>DistributedDataParallel</code>? So that final script looks like:</p>
<pre><code class="lang-auto">if distributed:
   model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[device])
   other_class = torch.nn.parallel.DistributedDataParallel(other_class , device_ids=[device])

</code></pre>

ptrblck: <p><code>DistributedDataParallel</code> expects an <code>nn.Module</code> as its first argument so you might need to create a custom module containing your parameters if you want to use them in DDP.</p>

MLAlex1: <p>Thanks for your reply!<br>
Yes that makes sense - I have indeed created two classes that inherit from <code>nn.Module</code>:</p>
<p>One for the model</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        # some code here
</code></pre>
<p>and one for multi-task losses</p>
<pre><code class="lang-auto">class Weightloss(nn.Module):

    def __init__(self):
        super(Weightloss, self).__init__()
        # some code here

    def forward(self, x):
        # some code here
</code></pre>
<p>So if I then pass them both in two different <code>torch.nn.parallel.DistributedDataParallel</code> like below:</p>
<pre><code class="lang-auto">model = MyModel()
w_loss = Weightloss()
if distributed:
   model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[device])
   w_loss = torch.nn.parallel.DistributedDataParallel(w_loss , device_ids=[device])
</code></pre>
<p>Is it the right way of doing it and it will be properly synchronized across all the GPUs? Or these classes should be passed somehow to only one <code>torch.nn.parallel.DistributedDataParallel</code>? Something like:</p>
<pre><code class="lang-auto">if distributed:
   model, w_loss = torch.nn.parallel.DistributedDataParallel(model, w_loss, device_ids=[device])
</code></pre>
<p>(which I guess will throw some error if I initialize it this way)</p>

ptrblck: <aside class="quote no-group" data-username="MLAlex1" data-post="3" data-topic="181338">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlalex1/48/59670_2.png" class="avatar"> MLAlex1:</div>
<blockquote>
<p>(which I guess will throw some error if I initialize it this way)</p>
</blockquote>
</aside>
<p>Yes, I think you are right as <code>DDP</code> expects a single module as its input argument.</p>
<p>Probably the safest approach would be to initialize <code>Weightloss</code> inside your <code>MyModel</code> which would allow you to only pass the <code>MyModel</code> object to <code>DDP</code>.<br>
If that’s not possible you could try to use two separate DDP objects, which I would assume should also work as they should be executed sequentially.</p>
How could I manage tensor memory in C++

ZHAN_LU: <p>I am trying to create tensors in python side and manage the memory in C++ side.</p>
<p>For example, I have the following C++ code trying to maintain the memory information.</p>
<pre data-code-wrap="C++"><code class="lang-plaintext">void RegisterTensor(const std::uint32_t tensor_id, torch::Tensor&amp; buffer){
   tensor_to_id_.insert(std::make_pair(&amp;buffer, tensor_id)); // tensor_to_id_ is an unordered_map, I want to change the tensor in the future in C++ side, so I maintain this information
   cout &lt;&lt; "tesor id " &lt;&lt; id &lt;&lt; " address " &lt;&lt; (void*)&amp;buffer; 
}
</code></pre>
<p>The code is built as a torch extension which can be loaded in python.<br>
However, when I run the following code, something got wrong.</p>
<pre><code class="lang-python">tensor1 = torch.randn(10, 10).pin_memory()
tensor2 = torch.randn(10, 10).pin_memory()
print("tensor1 address:", hex(id(tensor1)))
print("tensor2 address:", hex(id(tensor2)))
lib.register(tensor1, 1)
lib.register(tensor2, 2)
</code></pre>
<p>I expect to get the different address of the tensor, but the output is not as expected.</p>
<pre><code class="lang-auto">tensor1 address: 0x7f1eedb2e930
tensor2 address: 0x7f1eedb2ea20
tensor id 1 address 0x7fff27333998
tensor id 2 address 0x7fff27333998
</code></pre>
<p>Could anybody explain to me why the memory address <strong>are always the same</strong> in C++ side and how could I properly manage the memory in C++ side? Thank you!</p>

ZHAN_LU: <p>Does anyone have any suggestions?</p>

eqy: <p>I’m not sure you should use <code>id()</code> if your goal is to inspect memory addresses. Could you check if the <code>.data_ptr()</code> method returns different addresses for your tensors as expected?</p>
Next() behaviour with a list of DataLoaders of different length

bornexmachina: <p>My data has several conditions A, B, C. I would like to do the following.</p>
<ul>
<li>Draw a sample for each condition</li>
<li>Draw a random sample from the full data set</li>
<li>Some training magic</li>
</ul>
<p>Thus, I would have in one batch something like</p>
<pre><code class="lang-auto">[condition_A, condition_B, condition_C, random_sample]
</code></pre>
<p>I have created a dictionary of the form</p>
<pre><code class="lang-auto">loader_dict = {
cond_A : DataLoader(...Subset Magic...), 
cond_B : DataLoader(...Subset Magic...),
cond_C : DataLoader(...Subset Magic...)
}

train_loader = DataLoader(...full dataset...)
</code></pre>
<p>Now during each epoch I would like to</p>
<ol>
<li>Get a batch from each of the 4 loaders</li>
<li>Process them in some net shenanigans</li>
</ol>
<p>Currently, I am a bit stuck on the 1st point.</p>
<p>My approach so far is</p>
<pre><code class="lang-auto"># get a list of form [loader_A, loader_B, loader_C]
train_loaders = list(zip(*loader_dict.values()))

for batch_idx, batch in enumerate(tqdm(train_loader)):
    condit_sample = [next(loader) for loader in train_loaders]
    
    # do something with torch.cat([batch, condit_sample])
</code></pre>
<p>Now I am not sure - will the <code>next()</code> call actually always just pick the first batch of the conditions loaders (<strong>not desired</strong>) or will it actually iterate through the samples of the conditions?</p>
<p>Also, my data has something like <code>50% condition_A, 35% condition_B, 15% condition_C</code></p>
<p>Thus, I wonder, whether my code would run e.g. through all 100 batches of the full dataset and repeat condition_A twice, condition_B nearly 3 times and condition_C 6 times? Or will the code just run through all samples of condition C and break down?</p>
<p>Currently, the multiple cycling through the conditional samples would suffice.</p>
<p>For later purposes, I would like to consider the following:</p>
<ul>
<li>just pick a really random sample (in each epoch something different) from the full dataset</li>
<li>cycle through all the conditional loader samples</li>
<li>terminate the epoch whenever the smallest condition sample is “cycled through”</li>
</ul>
How can multiple processes write to a tensor at the same time?

Sam_Lerman: <p>Can I use _share_memory() and just write to different indices:</p>
<p><code>tensor = torch.randn([5,5])._share_memory()</code></p>
<p>And then from multiple processes:</p>
<p><code>tensor[process_id] = process_id</code></p>
<p>Is there a better way involving all_reduce or something? I want non-blocking writes, as in just hogwild writes from multiple processes in parallel.</p>
Different behaviour for jacwd and grad. For grad, RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TanhBackward

ylz: <p>Hi, I’m getting this error and I know I have in-place operations in my code. However, the wired thing is that “jacfwd” works, but grad does not. I don’t know why this is happening. Any help would be appreciated. Next is an simplified example.</p>
<pre><code class="lang-auto"> import torch
 from torch import nn
 import numpy as np
 from torch.func import vmap
 from torch.func import jacfwd,grad
 def test(cart):
     sph=cart.new_ones(10,cart.shape[0])
     for i in range(1,10):
          sph[i]=cart[i]*sph[i-1]
     return torch.sum(sph)
 
 cart=torch.randn((10))
 jac=jacfwd(test)
 print(jac(cart))
 gradient=grad(test)
 print(gradient(cart)) 
</code></pre>

KFrank: <p>Hi yl!</p>
<aside class="quote no-group" data-username="ylz" data-post="1" data-topic="181013" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ylz/48/36367_2.png" class="avatar"> ylz:</div>
<blockquote>
<p>“jacfwd” works, but grad does not.</p>
</blockquote>
</aside>
<p><code>jacfwd()</code> use forward-mode auto-differentiation, so, unlike <code>grad()</code>, it doesn’t<br>
have inplace-modification issues.  (Just for fun, you might try <code>jacrev()</code>, which<br>
uses “conventional” backward-mode auto-differentiation.)</p>
<p>Best.</p>
<p>K. Frank</p>

ylz: <p>Thank you for your answer!</p>
I get "RuntimeError: number of dims don't match in permute", when I try to solve it, a new runtime error "RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 500, 353, 3] to have 3 channels, but got 500 channels instead" occurs

peony: <p>I am converting NHWC to NCHW format because I receive the following error:</p>
<pre><code class="lang-auto">
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 500, 353, 3] to have 3 channels, but got 500 channels instead
</code></pre>
<p>My solution is to use permute like so:</p>
<pre><code class="lang-auto">
 torch.permute(imgdata, (0, 3, 1, 2))
</code></pre>
<p>But I get this error:</p>
<pre><code class="lang-auto">
RuntimeError: number of dims don't match in permute
</code></pre>
<p>I tried to edit the permute code like this:</p>
<p><code> torch.permute(imgdata, (0, 2, 1))</code></p>
<p>but it just goes back to the first error <code>RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 500, 353, 3] to have 3 channels, but got 500 channels instead</code></p>
<p>Here is part of my code that is relevant to the errors:</p>
<pre><code class="lang-auto">for imname, metaitem in  metadata_test.items():
  
        metaitem = metadata_test[imname]
        impath = metaitem['impath']
        impath = cv2.imread(impath)
        
        imgdata = torch.Tensor(impath)

        torch.permute(imgdata, (0, 3, 1, 2))
        pred = model(imgdata)
</code></pre>
<p>Any ideas for me? Thank you in advance</p>

ptrblck: <p>Your first approach works for me:</p>
<pre><code class="lang-python">img = torch.randn(1, 500, 353, 3)
img = img.permute(0, 3, 1, 2)
print(img.shape)
# torch.Size([1, 3, 500, 353])
</code></pre>
<p>so check why the <code>imgdata</code> tensor has only 3 dimensions and if e.g. the batch dimension might be missing assuming you are trying to <code>permute</code> the tensor inside the <code>Dataset.__getitem__</code> method. If so, just <code>permute</code> the <code>HWC</code> dims to <code>CHW</code>.</p>

peony: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="181382">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>Dataset.__getitem__</code></p>
</blockquote>
</aside>
<p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> , I don’t have batch size, I am loading one sample at a time from a subset of a dataset called metadata_test.items()</p>
<p>I tried unsqueezing like this</p>
<p><code>imgdata = torch.unsqueeze(imgdata, 0)</code></p>
<p>but it doesn’t work. Should I create a class like <code>torch.utils.data.Dataset</code> ? I am also not using dataloader where I would define batch_size</p>

peony: <p>Oh it worked now with this code instead of permute:</p>
<p><code>imgdata = torch.einsum('n h w c -&gt; n c h w', imgdata)</code></p>
<p>but permute should also work, I will try to know why it doesn’t.<br>
Thanks for helping me <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Dataloader for custom dataset in Hierarchical Deep CNN

Sumedh: <p>Hello Community, I want to implement hierarchical deep CNN with a custom dataset. How do I write a data loader for the same? I want to use both coarse as well as fine labels. The model will predict the coarse label first and then the fine label based on the class of the coarse label.</p>
<p>I would appreciate it if you help me write the code. I am new to Pytorch.</p>
<p>Thanks &amp; Regards.</p>
Trained model returning green image

loualcala: <p>Hello community, I was working on my own model, but when I run this script to test it, the result is a green image, as I will show you bellow my script:</p>
<p>This is my script:</p>
<pre><code class="lang-auto">import torch
from torchvision.transforms import ToTensor
from PIL import Image
from generator_model import Generator

# Cargar el modelo del generador
generator = Generator(in_channels=3)
generator.load_state_dict(torch.load('models/generator_model.pth'))
generator.eval()

# Cargar una nueva imagen para la prueba
image_path = 'watermarks/ex1/ex1-wm.jpeg'  # Ruta de la imagen que deseas probar
image = Image.open(image_path).convert('RGB')
transform = ToTensor()
input_image = transform(image).unsqueeze(0)  # Agregar una dimensión adicional para el batch

# Generar una nueva imagen utilizando el modelo
with torch.no_grad():
    generated_image = generator(input_image)

# Guardar la imagen generada
output_image = generated_image.squeeze(0).permute(1, 2, 0).clamp(0, 1).numpy()
output_image = (output_image * 255).astype('uint8')
output_image = Image.fromarray(output_image)
output_image.save('results/generated_image.jpg')

# Mostrar la imagen generada
output_image.show()
</code></pre>
<p>and this is the result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00d4f87d597b26fabb04f16720f797d80e8644e3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/00d4f87d597b26fabb04f16720f797d80e8644e3" title="generated_image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00d4f87d597b26fabb04f16720f797d80e8644e3_2_690x403.jpeg" alt="generated_image" data-base62-sha1="7mhPiVS6pO0WIqrhE77gUqZQk3" width="690" height="403" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00d4f87d597b26fabb04f16720f797d80e8644e3_2_690x403.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00d4f87d597b26fabb04f16720f797d80e8644e3_2_1035x604.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00d4f87d597b26fabb04f16720f797d80e8644e3_2_1380x806.jpeg 2x" data-dominant-color="00221B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">generated_image</span><span class="informations">1502×878 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would appreciate it if you help me debug my script and see how to correct it. I am new to using PyTorch.</p>