akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Stacked images look different converted from Tensor in dataloader

username: Gautier30
cooked: <p>Hi community,</p>
<p>I am training a custom network (with 3D conv layers) which takes a bunch of stacks of 3 images as input. Loaded from .npy files the image files have a shape of (depth=3, height=1280, width=1500, channels=3). I have a dataset class which converts the files to tensors (float32 type) and permutes the dimensions to have (3, 3, 1280, 1500) shape instead. I wanted to display example images from the validation set dataloader but the images look nothing like the originals once I revert the conversion to tensor. Here is my code:</p>
<pre><code class="lang-auto">def normalize8(I):
  mn = I.min()
  mx = I.max()
  mx -= mn
  I = ((I - mn)/mx) * 255
  return I.astype(np.uint8)

samples = next(iter(val_dataset))
images, labels = samples
image_permuted = images.permute(0, 2, 3, 1)
image_detached = image_permuted.detach().cpu().numpy()
image_unit8 = normalize8(image_detached)
plt.imshow(image_unit8[0]) # First image in stack
plt.show()
</code></pre>
<p>I am really not sure what I am doing wrong. Any help would be much appreciated! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Brock_Brown
cooked: <p>So your numpy array is in the format <code>D x H x W x C</code>, and you want it to be <code>C x D x H x W</code> for the PyTorch standard. I think that means your permute should be <code>images.permute(3, 0, 1, 2)</code>. The permute you wrote should give <code>D x W x C x H</code>.</p>

username: Gautier30
cooked: <p>Thanks a ton, I was misplacing the Depth and Channel dimensions indeed. You saved me from a headache <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>

username: Brock_Brown
cooked: <p>Not a problem, have fun. <img src="https://discuss.pytorch.org/images/emoji/apple/sunglasses.png?v=12" title=":sunglasses:" class="emoji" alt=":sunglasses:" loading="lazy" width="20" height="20"></p>
Numpy and pytorch backpropagation

username: AMIRA_BEDOUI
cooked: <p>Hello everyone,</p>
<p>I have a question regarding backpropagation. If I have a tensor <code>X</code> that I’m using for backpropagation, and there’s a constant <code>c</code>, if during an operation I compute <code>np.sqrt (c) × X</code> , will this cause any issue with the backpropagation process?</p>
<p>Thanks in advance,</p>

username: ptrblck
cooked: <p>I assume you are wrapping this <code>numpy * torch</code> operation into <code>torch.compile</code> which will translate it to pure PyTorch ops? If so, multiplying with a constant won’t cause any issues during the backpropagation.</p>
Uninstalling CUDA 11.8

username: plumberpt
cooked: <p>I have 3 CUDA 11.8, 12.4 and 12.5 installed. I will be using mainly 12.4 for Pytorch and 12.5 for Davinci Resolve Studio 19.</p>
<p>Is it ok to uninstall cuda 11.8 and not affecting Pytorch-Cuda 12.4? I want to save ~ 2 GB space.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>The PyTorch binaries will use their own CUDA runtime dependencies as explained in your <a href="https://discuss.pytorch.org/t/when-will-cuda-12-4-be-compatible-with-pytorch-2-3/205847/6">other thread</a> and your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension.</p>

username: plumberpt
cooked: <p>Yes, I understand that. My fear is by uninstalling 11.8, it may uninstall parts which are shared among the 3 CUDA. From your reply, I gather they are all totally independence.</p>
<p>Good. I will uninstall 11.8 to save some space. Thanks.</p>
Testing model scores vs. labels for COCO images

username: Magik2024
cooked: <p>First time posting so here goes.</p>
<p>During a test loop using the COCO dataset, I came across an interesting problem.  How do I verify that the ground truth labels in an image match the scores?  Now the model gives back a dictionary like predictions[‘scores’] and this is sorted from highest to lowest.  The problem happens when computing the accuracy score and verifying that the scores as sorted with the associated labels in predictions[‘labels’] are actually matching the ground truth.  A quick example, if the ground truth image contains person, dog, cat but the scores returns horse, sheep, person due to the CNN matching then the accuracy might be 0.33 or 0.0.  Its 0.0 if we do predictions[‘labels’] == ground_truth_labels because person or 1 didn’t match.  I fixed this using IoU and comparing all the rectangles to make sure that the label and rectangle match to get a True-True result.  But is this correct?</p>
Torch fails to find cpp_extension

username: nicoscr
cooked: <p>I am building a docker image to run machine learning models inside, specifically grounding DINO, which uses some C++ modules to optimise the attention module. However, when I run the docker image, the Grounding DINO install always falls back to CPU mode having failed to build this C++ code:</p>
<pre><code class="lang-auto">try:
    from groundingdino import _C
except:
    warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
</code></pre>
<p>I have done some digging in the container command line, and noticed that Torch and specifically the cpp_extension function from utils are not set up correctly, despite the versions being correct AFAICT:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.__version__
'2.3.0+cu121'
&gt;&gt;&gt; torch.utils.get_cpp_extension()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: module 'torch.utils' has no attribute 'get_cpp_extension'
</code></pre>
<p>I have no clue why torch.utils doesnt include this function, as the version <a href="https://pytorch.org/docs/2.3/cpp_extension.html" rel="noopener nofollow ugc">looks correct</a>?</p>

username: ptrblck
cooked: <p>The linked docs do not show a <code>get_cpp_extension()</code> call or am I missing it?<br>
Are you looking for <code>torch.utils.cpp_extension.CppExtension</code>?</p>
Question about bfloat16 operations in AMP and cuda

username: deepfailure
cooked: <p>Hello, when doing AMP with bfloat16 in cuda what are the operations autocasted?</p>
<p>In the documentation I only see bfloat16 for CPU and float16 for GPU.</p>
<p><a href="https://pytorch.org/docs/stable/amp.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/amp.html</a></p>
<p>I just want to know which operation run in bfloat16 and which in float32. I have a model which has conv2d layers, ReLU activation functions and a tanh activation function.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>Yes, the CUDA backend supports <code>bfloat16</code> for Ampere or newer GPUs.</p>

username: deepfailure
cooked: <p>Hello, thank you for your answer.</p>
<p>Sorry, I think I didn’t answer formulate my original question correctly.</p>
<p>I know that the CUDA backend supports bfloat16 in, for instance, the A100 GPU.</p>
<p>However, I want to know which operations AMP supports in bfloat16 and CUDA.</p>
<p><a href="https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float16" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float16</a></p>
<p>Are those the same ones as for float16?</p>
<p>Final question: how has it been decided that some operations can be cast to either float16 or bfloat16?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="deepfailure" data-post="3" data-topic="206078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a3d4f5/48.png" class="avatar"> deepfailure:</div>
<blockquote>
<p>Are those the same ones as for float16?</p>
</blockquote>
</aside>
<p>Yes, this should be the case as the lower precision ops are defined <a href="https://github.com/pytorch/pytorch/blob/68ad3eb722a429db738f93c73af82f700bc89ef5/aten/src/ATen/autocast_mode.h#L750">here</a> which should use both dtypes, if I’m not mistaken.</p>
Torch.distributed.send/recv not working

username: MaCasK
cooked: <p>I was trying to do a DDP transformer training on two machines (called machine 1 and 2) when I found the whole script stuck at dist.send and dist.recv. This script could work well when all processes are on machine 1 so I was confusing.</p>
<p>First, I simplified the code to a single send/recv</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.distributed as dist

# For machine1
dist.init_process_group(backend="nccl", init_method="tcp://machine1:port", rank=0, world_size=2)
a = torch.zeros(1, device="cuda")
dist.recv(a, src=1)

# For machine2
dist.init_process_group(backend="nccl", init_method="tcp://machine1:port", rank=1, world_size=2)
a = torch.tensor([1], device="cuda")
dist.send(a, dst=0)
</code></pre>
<p>This didn’t change anything, dist.init_process_group can work, and both process can get information from things like dist.get_rank(), but they still stuck at send/recv</p>
<p>Next, I tested the network connection between machines. Ping, telnet and nc all worked, so I believe it’s not a network connection or firewall issue.</p>
<p>Then I used another machine (machine 3) which also passed network test instead of machine 2, nothing changed.</p>
<p>I also searched for this issue, and found solutions like setting NCCL_IB_DISABLE=1 (from <a href="https://discuss.pytorch.org/t/torch-distributed-not-working-on-two-machines-nccl-backend/87659" class="inline-onebox">Torch distributed not working on two machines [nccl backend]</a>), do not work.</p>
<p>I tried to change to gloo backend, but this time I got a address family mismatch error while dist.init_process_group, and I also can’t solve it.</p>
<p>All my machines are Ubuntu, python 3.8.18 and pytorch 1.8.2, cuda 11.1 nccl 2708. I’m not very familiar to pytorch and cuda so I’m not sure if more information or more check is needed. If so, please tell me and I’m willing to provide any information.</p>

username: MaCasK
cooked: <p>Problem solved.<br>
It seems that nccl will find random usable port for each process, and send/recv try to directly connect to that random port, which is blocked by firewall.</p>
<p>However, I am still wondering if this port can be limited to some range, or fixed port? Open all ports on firewall seems not a good idea.</p>

username: GuWei007
cooked: <p>What changes did you make that resolved the issue?</p>

username: MaCasK
cooked: <p>Open all ports on firewall…<br>
A pretty bad solution, right? However I only found nccl using a random port everytime, and I dont know if I can restrict that choice.</p>
Questions about scaling of input and output when calculating gradients

username: alighato
cooked: <p>Hi all,</p>
<p>I am trying to implement a PINN for Fick’s law that looks like this:</p>
<pre><code class="lang-auto">dc/dt = r^-2 * d/dr * (r^2 * D * dc/dr)
</code></pre>
<p>r and t are inputs to the network and are min-max scaled between 0 and 1 (Sobol sampled).<br>
I also normalized my outputs when calculating training loss and boundary losses. What else do I need to consider regarding the scaling? The double derivative calculation and multiplying r^2, r^-2, and D (non-scaled) is kind off confusing me. What do I need to consider?</p>
<p>Thank you for answering my question!</p>
3D UNet Input channel for voxel categorisation

username: SystemLord
cooked: <p>Hi,</p>
<p>I have a 128x128x128 input 3D image where each voxel belongs to one category (basically a segmentation) out of ~100 total categories.</p>
<p>I’m aware that CNNs work best with one channel per category, and the channel has 1s where the voxel is in that category and 0 otherwise.</p>
<p>However, that is far too many channels.  Is having one channel with values between 0 and 1 for each category (say 0.01 for category 1, 0.02 for category 2, etc) a complete non-starter?</p>
<p>Are there any standard ways of dealing with this issue?</p>
<p>Thank you!</p>
Export onnx and channel_last

username: LLtomato
cooked: <p>i am using torch.onnx.export to export an onnx model, and it works. but my depolyment enviroment need a channel last model. i can only export channel first model now. Is there some ways to get a channel last model ? thanks for any replies !</p>

username: marksaroufim
cooked: <p>Does this work</p>
<p><code>model = model.to(memory_format=torch.channels_last)</code> and then run the export?</p>

username: vadimkantorov
cooked: <p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> But is the <code>memory_format</code> semantic faithfuly represented by ONNX format and implemented by ONNX runtime?</p>

username: Varun2
cooked: <p>Did you find out more about this? I am also trying to truly convert from channels first to channels last instead of just changing strides like the current solution do</p>
Understanding torch.fx.traceback.preserve_node_meta()

username: trusira
cooked: <p>Hi,<br>
I’m trying to understand the scope of  <code>torch.fx.traceback.preserve_node_meta()</code>. Can someone please explain <code>torch.fx.Node</code>’s meta preservation in the context of the following example?</p>
<pre><code class="lang-auto"># A simple torch module
class SimpleAdd(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x, y):
        return x + y

class SimpleTransform(torch.fx.Transformer):
    '''
    A simple FX transformer
    '''
    def __init__(self, module):
        super().__init__(module)
    
    def run_node(self, n: torch.fx.Node):
        '''
       Traverse FX graph and add a custom meta field for each node
        '''
        n.meta["node_op"] = n.op
        return super().run_node(n)
    
model = SimpleAdd()

with torch.fx.traceback.preserve_node_meta():
    fxg = torch.fx.symbolic_trace(model)

    # node meta prior to transform
    for node in fxg.graph.nodes:
        """
        Current node meta
        x {'seq_nr': -1}
        y {'seq_nr': -1} 
        add {'seq_nr': -1}
        output {'seq_nr': -1}
        """
        print(node, node.meta)

    # node traversal to add new meta
    SimpleTransform(fxg).run()

    for node in fxg.graph.nodes:
        """
        Updated meta. Previous meta is preserved.
        x {'seq_nr': -1, 'node_op': 'placeholder'}
        y {'seq_nr': -1, 'node_op': 'placeholder'}
        add {'seq_nr': -1, 'node_op': 'call_function'}
        output {'seq_nr': -1, 'node_op': 'output'}
        """
        print(node, node.meta)

    # No real transformation, call super().transform()
    fxgn = SimpleTransform(fxg).transform()

    # node meta after transform
    for node in fxgn.graph.nodes:
        """
        Node meta after calling transform().
        x {}
        y {}
        add {'from_node': [('add', &lt;built-in function add&gt;)], 'seq_nr': -1}
        output {}
        """
        print(node, node.meta)

</code></pre>
<p><strong>Question</strong>: Is this the correct behavior of <code>preserve_node_meta()</code>? Given that <code>transform()</code> returns a new graph, should it still preserve node meta from the old graph?</p>
<p>The example code was run against Pytorch commit <a href="https://github.com/pytorch/pytorch/commit/eff01bce21c862a3e39e0d55b47eef8f33e41ef0" rel="noopener nofollow ugc">eff01bc</a>.</p>
<p>Thanks!</p>

username: trusira
cooked: <p>Bump! for visibility.</p>
Initialize tensor ahead of time, write calculations straight into it

username: argg
cooked: <p>I have a for-loop that performs a lot of iterations, and in each one, I perform certain calculations in “extern inline” functions, and then set the result to a new torch::Tensor.<br>
I’m attempting to optimize my program, so I would like to initialize memory for the tensor-variables used in the for-loop ahead of time (the same variable has the same size in each iteration), and then have the operations performed in the “extern inline” functions write their results straight into that pre-initialized memory.<br>
Would this be possible somehow? Note that the extern inline functions are each only one line long (just a bunch of tensor operations), and I can replace them with <span class="hashtag-raw">#defines</span> if necessary.</p>
<p>On a somewhat unrelated side-note, when working with small matrices &amp; vectors, is it normal to experience a 100x (10000%) slowdown when switching from Eigen to LibTorch, or is that a sign that I’m doing something very wrong (I’ve already add no_grad guards and enabled inference mode)?</p>
Correct way of aggregating scalar tensors into a single tensor without losing gradients?

username: michaelala
cooked: <p>Hey all! Let’s suppose I have a non-jagged array of scalar tensors each with gradient information (e.g. the losses of a couple of different models). Is there an intended way of aggregating these together into a single tensor without losing gradient information?</p>
<p>Here’s a minimal example to demonstrate:</p>
<pre><code class="lang-auto">import torch
x1, x2, x3, x4 = torch.tensor([1., 2., 3., 4.])
w = torch.nn.Parameter(torch.tensor(5.), requires_grad = True)

l1, l2, l3, l4 = (x * w for x in [x1, x2, x3, x4])
l = [
    [l1, l2],
    [l3, l4]
]
assert torch.tensor(l).grad_fn is None
</code></pre>
<p>Currently to get around this I’m doing a double <code>torch.stack</code>, but I wonder if there’s a cleaner solution? Could get especially nasty if you’re working with an arbitrarily shaped <code>l</code>.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Using <code>torch.stack</code> or <code>torch.cat</code> to create a single tensor in a differentiable way is the right approach.</p>
Bidirectional encoder and decoder with 2 layers in Seq2Seq model

username: Sai1
cooked: <p>The output of my encoder is assume (2, 64, 256) so 2 layers of LSTM with a hidden size of 128 and I concatenate both directions. My question is, Say I initialized the decoder LSTM with same setup as encoder i.e., 2 layers bidirectional and hidden_dim of 256. Now the decoder expects a 4, 64, 256 tensor. How to init the tensor. Should I just keep the layers and directions separate and pass as it is or Is there a correct way to do this ? Each row in the encoder hidden tensor which is of shape 2, 64, 256 is the final hidden state of each layer in encoder. Doesn’t the decoder only expect the last hidden_state of encoder which means should I only pass -1, 64, 256  from encoder and initialize the rest with zeros ?</p>

username: Farshid1
cooked: <p>see with this code is your problem solved?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

encoder_hidden_state = torch.randn(4, 64, 128)  # (num_layers * num_directions, batch_size, hidden_size)
encoder_cell_state = torch.randn(4, 64, 128)

num_layers = 2
num_directions = 2
hidden_size = 128
batch_size = 64

decoder_initial_h_0 = encoder_hidden_state  # Shape should be (num_layers * num_directions, batch_size, hidden_size)
decoder_initial_c_0 = encoder_cell_state    # Same shape as h_0

decoder_lstm = nn.LSTM(input_size=256, hidden_size=hidden_size, num_layers=num_layers, bidirectional=True)

decoder_input = torch.randn(10, batch_size, 256)  # Assuming 10 time steps

decoder_output, (h_n, c_n) = decoder_lstm(decoder_input, (decoder_initial_h_0, decoder_initial_c_0))

print(f"Decoder output shape: {decoder_output.shape}")
print(f"Decoder hidden state shape: {h_n.shape}")
print(f"Decoder cell state shape: {c_n.shape}")
</code></pre>

username: Sai1
cooked: <p>Thanks for the reply farshid, The way you described is passing the hidden state of encoder as it is to the decoder. I was wondering if it should be done in the way you mentioned. I feel like passing the forward final hidden state of encoder to forward decoder and backward to backward is not the correct way. But, I might be wrong. I wanted to know if there’s a correct way to do this.</p>
Weighted Random Sampling with Inverse Class Frequency Weights

username: Hanna_Ribas_Peeters
cooked: <p>Hi everyone,</p>
<p>I am working on a multilabel classification problem. There is a very high class imbalance in the training dataset, where some classes have thousands of occurrences and others have less than 10. Therefore I am trying different methods to handle the class imbalance.<br>
Among them, I tried WeightedRandomSampler where the weights are the sqrt of the inverse class frequency (+ a regularization term). Like this I expect the rare classes to be oversampled and the common classes to be undersampled. The final size of the dataset is the same as the original.</p>
<p>I group the classes according to their frequency (i.e. classes that have 0-10 occurrences, 10-20 occurrences, 20-100, etc). And I look at the AUROC and F1 averaged per group to see if my method improves performance on minority classes.</p>
<p><strong>Surprisingly, upweighting the minority classes yields a substantial decrease in performance globally and in partiuclar for the minority class groups. Can someone explain why this happens?</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d5b63437878903cd667eb794b42117302950105.png" alt="image" data-base62-sha1="ms2L8eTdsC101gZTOQzIt2pjob3" width="618" height="219"></p>
<p>In the plot, light blue is the baseline and red and purple are different WeightedRandomSampling strategies.</p>

username: Narayanan_R
cooked: <p>Could  you post the code that you used to implement the sampler ?</p>
Error for not having the same size in the input and output in the seqtoseq algorithm

username: Farshid1
cooked: <p>hi everyone<br>
why if I write this code I will get this error</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

d_model = 4
nhead = 2
decoder_layer = nn.TransformerDecoderLayer(d_model=d_model, nhead=nhead)
transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)
memory = torch.rand(1, 808, d_model)
tgt = torch.rand(1, 810, d_model)

print(f"memory shape: {memory.shape}")
print(f"tgt shape: {tgt.shape}")

try:
    out = transformer_decoder(tgt, memory)
    print(f"output shape: {out.shape}")
except RuntimeError as e:
    print(f"RuntimeError: {e}")
</code></pre>
<blockquote>
<p>RuntimeError: shape ‘[1, 1620, 2]’ is invalid for input of size 3232</p>
</blockquote>
<p>it is related to seq length in input (808) and output(810)<br>
if we change the input length to 810 everythink will be ok<br>
in my project (seqToseq with transformer) my input and output has the same size but we should [SOS] and [EOS] to the beginning and at the end of our output seq that leads to my len(output)=len(input)+2</p>
<p>this is my own code for decoder</p>
<pre><code class="lang-auto">class TransformerDecoderLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config=config
        self.attention = MultiHeadAttention(config)
        self.feed_forward = FeedForward(config)
        self.addnorm = AddNorm(config)
        self.encoder_decoder_attention = MultiHeadAttention(config)
    def forward(self,encoder_output,outputs):
        # y=self.embedding_layer_output(outputs)
        y2 =self.attention(hidden_state=outputs,mask=True)
        y3=self.addnorm(outputs,y2)
        y4 =self.encoder_decoder_attention(encoder_output,y3)
        y5=self.addnorm(y3,y4)
        y6 =self.feed_forward(y5)
        y7=self.addnorm(y5,y6)
        return y7

class TransformerDecoder(nn.Module):
    def __init__(self,config):
        super().__init__()
        self.num_layer=config['num_layers']
        self.decoderLayers = nn.ModuleList([TransformerDecoderLayer(config) for _ in range(self.num_layer)])
    def forward(self,memory,tgt,mask=None):
        output=tgt
        for layer in self.decoderLayers:
            output=layer(memory,output)
        return output
</code></pre>
<p>if I run the example above with my code we have no error</p>
<pre><code class="lang-auto">decoder_layer = TransformerDecoderLayer(config)
transformer_decoder = TransformerDecoder(config)
memory = torch.rand(1, 808, 768)
tgt = torch.rand(1, 810, 768)

print(f"memory shape: {memory.shape}")
print(f"tgt shape: {tgt.shape}")

try:
    out = transformer_decoder(memory,tgt)
    print(f"output shape: {out.shape}")
except RuntimeError as e:
    print(f"RuntimeError: {e}")
</code></pre>
<blockquote>
<p>memory shape: torch.Size([1, 808, 768])<br>
tgt shape: torch.Size([1, 810, 768])<br>
output shape: torch.Size([1, 810, 768])</p>
</blockquote>
How to predict a robot's coordinates

username: bengao
cooked: <p>Hello everyone,</p>
<p>I have a problem and I don’t know how to solve it using AI.</p>
<p>Imagine a robot on a chessboard. It knows it’s on D4, for example. It also knows that if it performs a certain command, it will reach E6. So, generally speaking, it knows the positions it obtains as a result of its commands. We can learn from this. Because this can be repeated ad vitaem.</p>
<p>Now, what I want at the end of the learning process is for it to make a prediction: the robot is in F2, it performs a command: what position will it arrive in? This is prediction. In fact, I don’t want it to tell me that it will be in F4 at 60% (softmax), but I do want it to directly predict the most probable solution dex coordinates (x and y). (In real life, there are an infinite number of possible positions).</p>
<p>I don’t know how to do it, do you have an idea? or a link?</p>
<p>Thanks in advance</p>
<p>Ben</p>

username: bengao
cooked: <p>No link to propose ?</p>
Dataparallel gets stuck in AWS

username: something_random
cooked: <p>I was testing DataParallel in g5.12xlarge amazon linux AWS instance using <a href="https://github.com/pytorch/tutorials/blob/main/beginner_source/blitz/data_parallel_tutorial.py" rel="noopener nofollow ugc">this</a> code. The code works perfectly if I just make one GPU visible, however it gets stuck when multiple GPUs are used.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/1644e2a3fea92700456cd7f5f4876a4bbf7a0244.png" data-download-href="https://discuss.pytorch.org/uploads/default/1644e2a3fea92700456cd7f5f4876a4bbf7a0244" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1644e2a3fea92700456cd7f5f4876a4bbf7a0244_2_690x132.png" alt="image" data-base62-sha1="3b05BIgPI3uqxfygGAK4kaUZ9Iw" width="690" height="132" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1644e2a3fea92700456cd7f5f4876a4bbf7a0244_2_690x132.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1644e2a3fea92700456cd7f5f4876a4bbf7a0244_2_1035x198.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1644e2a3fea92700456cd7f5f4876a4bbf7a0244_2_1380x264.png 2x" data-dominant-color="322323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2070×396 42.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>System details:<br>
torch version: 2.3.1<br>
torch cuda version: 12.1<br>
NVIDIA-SMI 555.42.06              Driver Version: 555.42.06      CUDA Version: 12.5</p>
Unrecognized attribute: axes for operator ReduceMean

username: Abhijit_Deo
cooked: <p>I have a pytorch model which i want to export to onnx with opset version more than 18.</p>
<p>But im facing issue while exporting, I am attaching minimalistic script to reproduce error.</p>
<pre data-code-wrap="python"><code class="lang-python">
import torch 
import numpy as np
import torch.nn as nn

class MinimalExample(nn.Module):
    def __init__(self, ):
        super().__init__()
        self.linear1 = nn.Linear(100,32)
    def forward(self, x):
        x = self.linear1(x)
        return torch.mean(x, dim=(-1,-2))

torch_model = MinimalExample()
batch_size=1
    
x = torch.randn(1, 3,100, 100, requires_grad=True)
torch_out = torch_model(x)

torch.onnx.export(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "minimalist.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=18,          # the ONNX version to export the model to
                  do_constant_folding=False,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                'output' : {0 : 'batch_size'}})

import onnx
onnx_model = onnx.load("minimalist.onnx")
onnx.checker.check_model(onnx_model)
import onnxruntime

ort_session = onnxruntime.InferenceSession("minimalist.onnx", providers=["CPUExecutionProvider",]
                                               )

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

    # compute ONNX Runtime output prediction``
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(x)}
ort_outs = ort_session.run(None, ort_inputs)

try:
    np.testing.assert_allclose(to_numpy(torch_out), ort_outs[0], rtol=1e-03, atol=1e-05)
except:
    print(print(torch_out,"\n", ort_outs[0]))
print(f"Exported model has been tested with ONNXRuntime, and the result looks good on CPU!")


</code></pre>
<p>The code works for opset version 17 but fails on 18. Is there any way i can replace the mean function and get everything working ?</p>
1dCNN not able to detect major features

username: L_Z
cooked: <p>I think my 1d CNN is really bad either at detecting major features even though they are really obvious features visually or the model is overfitting. And switching to 2d CNN, it seems that major features can be extracted but not that good enough. Anyone know the reason why?</p>
<p>Any suggestions on how to improve my model?</p>
'is_sparse_any' from 'torch._cannot import namesubclasses.meta_utils'

username: Farshid1
cooked: <p>this is my code:</p>
<pre><code class="lang-auto">class Seq2Seq(nn.Module):
    def __init__(self,encoder,decoder,config):
        super().__init__()
        self.encoder = encoder(config)
        self.decoder = decoder(config)
    def forward(self,src,tgt):
        encoder_output = self.encoder(src)
        decoder_output = self.decoder(encoder_output, tgt)
        return decoder_output
        
model=Seq2Seq(TransformerEncoderLayer,TransformerDecoderLayer,config)
optimizer = torch.optim.Adam(model.parameters())```
 

 

but in the last line I have this error: ImportError: cannot import name 'is_sparse_any' from 'torch._subclasses.meta_utils' can anyone help me to solve this issue? apparently it's related to environmentvariable but I don't know how to fix it</code></pre>

username: TheEliasBe
cooked: <p>Can you share please your pytorch version and the import statement that you are using?</p>

username: Farshid1
cooked: <p>hi<br>
(torch version is ‘2.3.1+cpu’)<br>
by the way, I don’t know why but I could run my code in in a new session!</p>
Torch_geometric - Calculate Dirichlet energy at each layer

username: camille-004
cooked: <p>Hi, I’m pretty new to GNNs and PyTorch Geometric. I want to calculate the Dirichlet energy at each layer and plot them, so I can visualize the over-smoothing problem. I have looked into several ways to accomplish this, but none of them seem to work, as I have to compute the product between the features and graph Laplacian. I am not sure how to get the Laplacian when the hidden layer changes the dimension of the node features, as the dimensions of the new features and Laplacian computed off the edge_index would be completely different. Any tips pointing me in the right direction would be helpful!</p>
<p>I have a pretty standard GCN defined here:</p>
<pre><code class="lang-auto">class GCN(nn.Module):
    def __init__(
        self,
        n_hidden,
        n_features,
        n_classes,
        hidden_dim,
        dropout_rate,
        activation=F.relu,
        use_linear=False,
    ):
        super().__init__()
        self.n_hidden = n_hidden
        self.hidden_dim = hidden_dim
        self.activation = activation
        self.use_linear = use_linear

        self.in_conv = GCNConv(n_features, self.hidden_dim)
        self.hidden_convs = nn.ModuleList(
            [
                GCNConv(self.hidden_dim, self.hidden_dim)
                for _ in range(self.n_hidden)
            ]
        )
        self.out_conv = GCNConv(self.hidden_dim, n_classes)
        self.dropout = Dropout(dropout_rate)

    def forward(self, x, edge_index):
        x = self.dropout(x)
        x = self.activation(self.in_conv(x, edge_index))

        for i in range(self.n_hidden):
            x = self.activation(self.hidden_convs[i](x, edge_index))
            if self.use_linear:
                linear = Linear(self.hidden_dim, self.hidden_dim)(x)
                x = x + linear

        x = self.activation(self.out_conv(x, edge_index))
        x = self.dropout(x)
</code></pre>
<p>I would also like to accomplish this with a GAT and GIN as well.</p>

username: Debangan_Mishra
cooked: <p>I know I am late to this, but have you found any solutions to the problem? I wish to do the same as you for my experiment.</p>
Torch parameter gradient is None

username: Ido_Avrahami
cooked: <p>I’m trying to optimize a torch parameter using a pre-trained network.<br>
This is my code.</p>
<pre><code class="lang-auto">single_frame = image[0,:,:,:].to(device)
modif = torch.Tensor(1, 2, 34, 34).fill_(1).to(device)
modifier = torch.nn.Parameter(modif, requires_grad=True)
optimizer = torch.optim.Adam([modifier], lr=learning_rate)
criterion = torch.nn.CrossEntropyLoss()
iters = 10
for i in range(iters):

    # single_frame = single_frame.to(device)
    optimizer.zero_grad()
    modified_frame = single_frame + modifier
    if label.dim() == 0:
        label = label.unsqueeze(0).to(device)
    label = label.to(device)
    with torch.no_grad():
        output = model(modified_frame)
    output = output.requires_grad_(True)
    loss = criterion(output,label)
    loss.backward()
    optimizer.step()
    print(modifier.grad)
</code></pre>
<p>The output shows that the modifier does not change, and the print shows that modifier’s grad is None.<br>
What might be the cause?</p>

username: ptrblck
cooked: <p>You are performing the forward pass of your model in a <code>no_grad()</code> context and are afterwards setting the <code>.requires_grad</code> attribute of the output to <code>True</code>, which won’t attach it to the computation graph. Perform the forward pass in the global context and it should work.</p>
How to continue training with DDP when loss is nan causing by amp

username: proshm
cooked: <p>I tried to clear loss tenser to continue, but it didn’t work. It reports “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”:</p>
<blockquote>
<pre><code>    with torch.cuda.amp.autocast():
        loss, _, _ = model(samples, mask_ratio=args.mask_ratio)

    loss_value = loss.item()
    
    if not math.isfinite(loss_value):
        print(f"Loss is {loss_value} in iter {data_iter_step}, continue training")
        print(loss.shape)
        loss = torch.zeros_like(loss)
        loss_value = 0
    
    loss /= accum_iter
    loss_scaler(loss, optimizer, parameters=model.parameters(),
                update_grad=(data_iter_step + 1) % accum_iter == 0)
    if (data_iter_step + 1) % accum_iter == 0:
        optimizer.zero_grad()

    torch.cuda.synchronize()

    metric_logger.update(loss=loss_value)

    lr = optimizer.param_groups[0]["lr"]
    metric_logger.update(lr=lr)

    loss_value_reduce = misc.all_reduce_mean(loss_value)
    if log_writer is not None and (data_iter_step + 1) % accum_iter == 0:
        """ We use epoch_1000x as the x-axis in tensorboard.
        This calibrates different curves when batch size changes.
        """
        epoch_1000x = int((data_iter_step / len(data_loader) + epoch) * 1000)
        log_writer.add_scalar('train_loss', loss_value_reduce, epoch_1000x)
        log_writer.add_scalar('lr', lr, epoch_1000x)
</code></pre>
</blockquote>

username: ptrblck
cooked: <p>You should not expect to see NaN values in the loss calculation as it could indicate your model is overflowing internally.<br>
If you still call <code>backward</code> on it, the gradients will also be NaNs and the <code>GradScaler</code> will skip the parameter update. Since nothing in the model changed, the next forward pass could still create the NaN loss and you should thus consider debugging which operation is causing this invalid value.</p>
<p>Skipping iterations in DDP might not be possible without a <code>no_sync</code> context manager as DDP expects to allreduce the gradient buckets.</p>
Why is there a discrepency between nvidia-smi reported vram vs the one from torch.cuda.mem_get_info()?

username: Shisho_Sama
cooked: <p>Hello everyone,<br>
Hope youa re having a great day.<br>
I was playing a bit with memory stats in pytorch today that I noticed the numbers reported by pytorch for the gpu vram doesnt match that of nvidia-smi.</p>
<p>I have a RTX3080 10G which shows up as <strong>10240</strong>MiB in <code>nvidia-smi</code> but when I try <code>torch.cuda.mem_get_info(0)</code> to get vram amount, I get <strong>10,492,772,352</strong> bytes which translates to <strong>10,006</strong>MiB !<br>
<strong>10240 MiB</strong> is <strong>10,737,418,240</strong> bytes which means <strong>244,645,888</strong> bytes are missing/not accounted for. This is around <strong>233.3125</strong>MiB!</p>
<p>where does this discrepancy come from? wheres the other <strong>244,645,888</strong> bytes?</p>

username: Brock_Brown
cooked: <p>I think you’re looking for <a href="https://discuss.pytorch.org/t/pytorchs-torch-cuda-max-memory-allocated-showing-different-results-from-nvidia-smi/165706/2">this</a>. Basically, there’s some memory set aside to speed up future allocations/garbage collection. See <a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_reserved.html#torch.cuda.max_memory_reserved" rel="noopener nofollow ugc">torch.cuda.max_memory_reserved</a> for a value that will be closer to the nvidia-smi output.</p>

username: Shisho_Sama
cooked: <p>Thanks but it doesn’t make sense, this is not <strong>allocated</strong> memory, this is the <strong>total amount</strong> of memory. Whether its being used up by my processes or some of cuda’s or something else is another story.<br>
Why shouldnt the total amount be different?</p>
Request for Participation in a Survey on Non-Determinism Factors of Deep Learning Models

username: Marta_Catillo
cooked: <p>Dear ML/DL expert,</p>
<p>We are contacting you as an experienced deep learning practitioner,<br>
on behalf of a research group from the University of Sannio (Italy).</p>
<p>Our research activity concerns reproducibility of deep learning-intensive programs.</p>
<p>The focus of our research is on the presence of non-determinism factors<br>
in training deep learning models. As part of our research, we are conducting a survey to<br>
investigate the awareness and the state of practice on non-determinism factors of<br>
deep learning programs, by analyzing the perspective of the developers.</p>
<p>Participating in the survey is engaging and easy, and should take approximately 5 minutes.</p>
<p>All responses will be kept strictly anonymous. Analysis and reporting will be based<br>
on the aggregate responses only; individual responses will never be shared with<br>
any third parties.</p>
<p>Please use this opportunity to share your expertise and make sure that<br>
your view is included in decision-making about the future deep learning research.</p>
<p>To participate, simply click on the link below:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://forms.gle/YtDRhnMEqHGP1bPZ9">
  <header class="source">
      <img src="https://ssl.gstatic.com/docs/spreadsheets/forms/favicon_qp2.png" class="site-icon" width="16" height="16">

      <a href="https://forms.gle/YtDRhnMEqHGP1bPZ9" target="_blank" rel="noopener nofollow ugc">Google Docs</a>
  </header>

  <article class="onebox-body">
    <img width="200" height="200" src="https://lh6.googleusercontent.com/jK_Hb-Utf0cDxOH88Fpjt3hzjvdaxeeix2b56pqEz6ynm0mFsOgzPfFPyl7as5WdDMotGUDU7RA=w1200-h630-p" class="thumbnail onebox-avatar">

<h3><a href="https://forms.gle/YtDRhnMEqHGP1bPZ9" target="_blank" rel="noopener nofollow ugc">Survey on Non-Determinism Factors of Deep Learning Models</a></h3>

  <p>Welcome to this survey, which aims to investigate the awareness and the state of practice on non-determinism factors of deep learning models (training/evaluation).

Important Terms. A deep learning model is deterministic, if under the same training...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If you have any questions or concerns about the survey, or would like your email<br>
to be removed from the survey, please do not hesitate to contact us<br>
at marta.catillo@unisannio.it</p>
<p>We look forward to hearing from you soon.</p>
<p>Kind regards</p>
Does the order of with torch.no_grad(): and model.eval() matter?

username: iftg
cooked: <p>Does the order of with torch.no_grad(): and model.eval() matter?<br>
Let models be saved in a list model_list.</p>
<pre><code class="lang-auto">for m in model_list:
    m.eval()
    with torch.no_grad():
        netout = m(x)
        print(netout)
</code></pre>
<p>OR</p>
<pre><code class="lang-auto">with torch.no_grad():
    for m in model_list:
        m.eval()
        netout = m(x)
        print(netout)
</code></pre>
<p>In the latter example we avoid multiple calls of <code>with torch.no_grad()</code>.</p>

username: ptrblck
cooked: <p>No, the order would not matter here and both codes should yield the same results.<br>
Personally, I would move the <code>no_grad()</code> to the top to indicate that the following code won’t track any gradients.</p>
How can i plot loss funstions in my main script?

username: Hassan_Ghavidel
cooked: <p>i have a plotting.py script located in my auxiliary folder. two loss function defined in it as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a8a5b4b011f196fa724f16d73fcc97fe4b1a9e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a8a5b4b011f196fa724f16d73fcc97fe4b1a9e7" title="Screenshot from 2024-07-06 17-18-23"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0a8a5b4b011f196fa724f16d73fcc97fe4b1a9e7_2_474x500.png" alt="Screenshot from 2024-07-06 17-18-23" data-base62-sha1="1vfcxX6RWzJ8Fi5O0ITdQs2SIbt" width="474" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0a8a5b4b011f196fa724f16d73fcc97fe4b1a9e7_2_474x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a8a5b4b011f196fa724f16d73fcc97fe4b1a9e7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a8a5b4b011f196fa724f16d73fcc97fe4b1a9e7.png 2x" data-dominant-color="242525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-07-06 17-18-23</span><span class="informations">695×732 74.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i want to use them to plot my losses in results and use them in my main script:<br>
how can i do that?<br>
for example the code used this method below to plot plot_example_augmentations function:<br>
plotting.plot_example_augmentations(moving_image, fixed_image, os.path.join(path_saving, ‘augmentations_example_img.png’))</p>

username: Brock_Brown
cooked: <p>You should consider checking out TensorBoard. In my opinion it’s the best way to plot loss values in realtime. <a href="https://youtu.be/VJW9wU-1n18?si=b1J4erhsmCsGgvtG" rel="noopener nofollow ugc">https://youtu.be/VJW9wU-1n18?si=b1J4erhsmCsGgvtG</a></p>

username: Eduardo_Lawson
cooked: <p>Another great option is using <a href="https://docs.wandb.ai/" rel="noopener nofollow ugc">Weight and Biases</a>. It is really easy and quick to track your metrics and create reports aswell.</p>
RuntimeError: required rank 4 tensor to use channels_last format

username: maxrivera
cooked: <p>My transformer training loop seems to work correctly when I train it on the CPU, but when I switch to MPS, I get the below error when computing <code>loss.backward()</code> for Cross Entropy loss. I am doing machine translation.<br>
Dimensions:</p>
<pre><code class="lang-auto">model output: (batch_size, seq_len, target_vocab_size)
model output reshaped: (batch_size * seq_len, target_vocab_size)
target: (batch_size, seq_len)
target reshaped: (batch_size * seq_len) 
</code></pre>
<p>Terminal output:</p>
<pre><code class="lang-auto">out:  torch.Size([64, 60, 1446])
trg:  torch.Size([64, 60])
out reshaped:  torch.Size([3840, 1446])
trg reshaped:  torch.Size([3840])
Traceback (most recent call last):
  File "~/transformer_training.py", line 198, in &lt;module&gt;
    train_loss.backward()
  File "~/myenv2/lib/python3.9/site-packages/torch/_tensor.py", line 401, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "~/myenv2/lib/python3.9/site-packages/torch/autograd/__init__.py", line 191, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: required rank 4 tensor to use channels_last format
</code></pre>
<p>Here is my code for reference:</p>
<pre><code class="lang-auto">mask = torch.tril(torch.ones((MAX_LENGTH, MAX_LENGTH))).to(DEVICE)

# optimization loop 
best_loss = 1e5
optimizer=torch.optim.Adam(params=model.parameters(),lr=1e-3)
loss_fn = torch.nn.CrossEntropyLoss() 
train_losses = []
val_losses = []
for epoch in range(1,EPOCHS+1):

    # train loop 
    for i, (src,trg) in enumerate(train_data):

        # place tensors to device 
        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(trg).to(DEVICE).long()

        # forward pass 
        out = model(src,trg, mask)
        print('out: ', out.size())
        print('trg: ', trg.size())
        print('out reshaped: ', out.view(-1, tgt_vocab).size())
        print('trg reshaped: ', trg.view(-1).size())


        # compute loss 
        train_loss = loss_fn(out.view(-1,tgt_vocab), trg.view(-1))
        
        # backprop 
        optimizer.zero_grad()
        train_loss.backward()

        # update weights 
        optimizer.step()
</code></pre>

username: razbolt
cooked: <p>Hey, I got the same problem.</p>
<p>I believe it is about mps compatibility problem. i switched to “cpu” and it is working fine !</p>

username: Elvis_Sun
cooked: <p>Hi,did you solve the problem?  im having the same issue, looking for a solution.</p>

username: fatemeTaha
cooked: <p>Hi, have you managed to solve the problem? I’m having the same issue <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>