akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
A basic convolution implementation

Georges_Leukic: <p>Hello, I’m new to pytorch and I’d like to start with small steps by reproducing the following conv2d example found <a href="https://stackoverflow.com/questions/52923062/tensorflow-compute-tf-nn-conv2d" rel="noopener nofollow ugc">here</a>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/653c9eadcbb001aaee597795a4b8edb848c0c6b8.png" alt="convo2d" data-base62-sha1="erA6jMWdFMjaACDnB04tRj8avYA" width="506" height="284"></p>
<p>I would like to correctly implement the convolution of one image of size MxNx1 with c number of filters with size mxnx1. I suppose stacking filters into one filter tensor of size mxnxc, with the right reshaping, should produce stacked convolved images with last dimension of c and spatial dimensions depending on strides and convention (full, equal or valid).</p>

ptrblck: <p>You can directly create the input as well as weight tensors and directly apply <code>F.conv2d</code>:</p>
<pre><code class="lang-python">x = torch.tensor([[2,5,3],
                  [3,4,2],
                  [4,1,1]]).float().unsqueeze(0).unsqueeze(0)

weight = torch.tensor([[[2,1],
                        [3,4]],
                       [[4,1],
                        [1,2]]]).float().unsqueeze(1)

out = F.conv2d(x, weight)
print(out)
# tensor([[[[34., 33.],
#           [26., 17.]],

#          [[24., 31.],
#           [22., 21.]]]])
</code></pre>

Georges_Leukic: <p>Thank you, this really boosted me up. Since I don’t intend to build any neural network model, are there anything to worry about gradients or any unwanted calculations? My image is of the order 1e4x1e4, while kernels are of the order 500x500 - 100x100, and I only need to do some mathematical preprocessing after convolving. Among such operations I’d have to interpolate the convolved tensor along an arbitrary path (a large number of random paths, instead of one only) as found <a href="https://stackoverflow.com/questions/7878398/how-to-extract-an-arbitrary-line-of-values-from-a-numpy-array" rel="noopener nofollow ugc">here</a>. I expect lifting this up to fully tensor notation using pytorch should not be that difficult. The convolution part I tested on a server with 80 cpus and 512 GB, and it works like a charm, much faster when compared to scipy and numpy and slightly outperforms tensorflow.</p>

ptrblck: <p>No, you wouldn’t need to care about gradients etc. as in my example I did not initialize the <code>weight</code> tensor as a trainable <code>nn.Parameter</code> and also didn’t set its <code>requires_grad</code> attribute to <code>True</code> so that Autograd will ignore it and won’t save any intermediates needed for the gradient calculation.</p>
Is torch.randperm persitence in validation and testing

Leonnguyen: <p>Hi, I would like to have a question on torch.randperm. I used randperm in my model to reshuffle layers in the torch tensor because ChannelShuffle was not implemented in cuda.</p>
<p>My questions are:</p>
<ol>
<li>Does this function persist in torch.no_grad()? Does torch.randperm generate a new permutation list in validation and testing?</li>
<li>Does the torch.randperm generates a new permutation list for each batch in training</li>
</ol>
<p>This is a block in the model that I am implementing</p>
<p>class Mix_block(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Mix_block,self).<strong>init</strong>()</p>
<pre><code>def mix_func(self, tensor1, tensor2,  tensor3,  tensor4):
    input = torch.cat((tensor1, tensor2,  tensor3,  tensor4), 1)
    permute = torch.randperm(256, requires_grad=True)
    out = input[:, permute]
    # channel_shuffle = nn.ChannelShuffle(4)
    # out = channel_shuffle(input)
    return out

def forward(self, tensor1, tensor2,  tensor3,  tensor4):
    out = self.mix_func(tensor1, tensor2,  tensor3,  tensor4)
    return out
</code></pre>

ptrblck: <p><code>torch.randperm</code> returns a random permutation of integers in <code>[0, n-1]</code> and your code snippet won’t work:</p>
<pre><code class="lang-python">permute = torch.randperm(256, requires_grad=True)
# RuntimeError: Only Tensors of floating point and complex dtype can require gradients
</code></pre>
<ol>
<li>The function can be used in a <code>no_grad</code> context, but I don’t know what “persist” means here. <code>torch.randperm</code> is not a module and doesn’t use the <code>self.training</code> argument. It’s thus not influenced by <code>model.train()</code> or <code>model.eval()</code> calls.</li>
<li><code>torch.randperm</code> will shuffle the indices whenever called. If you are calling it for each batch, then a new output will be generated in each call.</li>
</ol>

Leonnguyen: <p>Thank ptrblck, I had the the same error when testing my model. Removing require_grad=True solved the error.</p>
<p>Let me explain my problem a bit:</p>
<p>I used  torch.randperm to do channel shuffle randomly. So, I want the input channels to be shuffled during the training phase, and then the model will remember the new position of channels from the permutation like a parameter. During the evaluation and the testing phases, the shuffle would stop, and the model would use the random permutation generated during the training phase.</p>
<p>I guess from your answer that my implementation would not work.</p>

ptrblck: <p>Based on your description it seems you could store the randomly shuffled indices and just reuse them without calling into <code>randperm</code> again.</p>
Why indices tensor behavior in cuda environment is weird?

daovietanh190499: <p>When I use indices tensor in cuda environment, it took about 0.077 second to finish. But when adding “print(x)”, it cost only 7*10^-5 second? Please help me, Thank you so much<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcdacbbc9b4900d2edeeed46e7adccfe6f820ba4.png" alt="indices tensor behavior" data-base62-sha1="vvLOjathUSC6oixGpG7YnC78LvS" width="427" height="437"></p>

ptrblck: <p>CUDA operations are executed asynchronously so you would need to synchronize your code via <code>torch.cuda.synchronize()</code> before starting and stopping host timers. Your current profiling is thus invalid and synchronizing operations, such as <code>print</code> statements of device tensors, will accumulate the running time of all launched and queued kernels.</p>
Help with Multi Domain Learning Implementation

GilKreisler: <p>Hello!</p>
<p>I’m trying to implement multi domain learning using pytorch. The problem is that I need that all the samples within a batch to be of the same domain. I will have a csv file containing the information about the domain of each sample.</p>
<p>Is there a way to select only the samples of the same domain to create a batch?</p>

yiftach: <p>There are more choices you will need to make, e.g. what happens if each domain has a different number of samples, or the number of samples is not divisible by the batch size. I would subclass <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="noopener nofollow ugc">torch.utils.data.Sampler</a> to write custom logic for creating batches that conform to your logic.</p>
<p>As an alternative, you could construct a dataset per each domain, have a loader per domain, and juggle between these dataloaders (e.g. use round-robin).</p>
Is it possible to get the value of f(x) while using torch.func.jacrev?

Yanze_Wang: <p>I would like to get the value of f(x) and its first derivative simultaneously. Is there any way to get that now?</p>

Yanze_Wang: <p>Otherwise, the model has to inference twice to get the value of f(x) and first derivative…</p>

J_Johnson: <p>Normally, the derivative is calculated after the loss is.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

model = nn.Linear(3, 1)

inputs = torch.rand(3)
targets = torch.rand(1)

outputs = model(inputs)

criterion = nn.MSELoss()

loss = criterion(outputs, targets)
loss.backward()
print(model.weight.grad)
print(model.bias.grad)
</code></pre>

Yanze_Wang: <p>Hi, thanks for reply, but I would like to calculate the jacob of a function f(x) by <code>torch.func.jacrev</code> which is not the loss.<br>
As I can find, <code>torch.func.jacrev</code> can only return the gradients.<br>
Another solution I can find is to use <code>torch.autograd.grad</code> instead for each row of <code>f(x)</code>. but it seems a bit slower than <code>jacrev</code></p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/yanze_wang">@Yanze_Wang</a>,</p>
<p>This is possible within the <code>torch.func</code> namespace, here’s a minimal reproducible example,</p>
<pre><code class="lang-auto">x = torch.tensor(5.) #dummy input

def f(x): #our function
  return x**2

def func_with_aux(x): #same *args as f(x)
  out = f(x)
  return out, out #return 'out' twice

from torch.func import jacrev
gradient, output = jacrev(func_with_aux, argnums=(0), has_aux=True)(x)
print(gradient, output) # returns 10, 25
</code></pre>

Yanze_Wang: <p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> ,<br>
Thanks, that was really helpful.<br>
I encounter a new issue, that I also need to jit script my model, but I got an error below when jitting my model involving <code>jacrev</code>.</p>
<pre><code class="lang-auto">torch.jit.frontend.NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults:
  File "/home/gridsan/ywang3/.conda/envs/rid_openmm/lib/python3.11/site-packages/torch/_functorch/eager_transforms.py", line 354
@exposed_in("torch.func")
def jacrev(func: Callable, argnums: Union[int, Tuple[int]] = 0, *, has_aux=False,
                                                                           ~~~~~ &lt;--- HERE
           chunk_size: Optional[int] = None,
           _preallocate_and_copy=False):
</code></pre>
<p>It’s prabobly related to the arg of jacrev functions, but could you please explain a bit this error? Thanlks!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/yanze_wang">@Yanze_Wang</a>,</p>
<p>I don’t believe <code>torch.func</code> supports <code>torch.script.jit</code>, so you’ll have to find a work-around for this. Why do you need to jit your model?</p>
Transformer architecture in PyTorch using Mac M1 GPU via MPS

radicalprotns: <p>I have a Mac M1 GPU (macOS 13.5.1, PyTorch 2.0.1) and I’m new to using the M1 GPU for deep learning. I’ve been playing around with the <a href="https://github.com/zhouhaoyi/Informer2020" rel="noopener nofollow ugc">Informer</a> architecture which is the transformer architecture applied to time series forecasting. However, upon changing the device from ‘cuda’ to ‘mps’ in the code, I cannot replicate the example provided by the authors in this <a href="https://colab.research.google.com/drive/1_X7O2BkFLvqyCdZzDZvV2MB0aAvYALLC" rel="noopener nofollow ugc">google colab notebook</a>. I obtain the same results running with CPU but not with MPS.</p>
<p>As the informer code is more involved, I looked for a simpler example that I’d have problems reconciling with the M1 GPU. The image classification tutorial on PyTorch did not pose an issue. However, <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">this example on sequence to sequence network and attention</a> fails on my M1 GPU. In particular, I get the following result while training the transformer model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93fb285d4619731f1ee84e7f04736f7994a1a5bd.png" data-download-href="https://discuss.pytorch.org/uploads/default/93fb285d4619731f1ee84e7f04736f7994a1a5bd" title="Screenshot 2023-09-16 at 12.27.46 AM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93fb285d4619731f1ee84e7f04736f7994a1a5bd.png" alt="Screenshot 2023-09-16 at 12.27.46 AM" data-base62-sha1="l76i2ySILSPQHQdArFM3qgqlNBb" width="470" height="500" data-dominant-color="232424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-09-16 at 12.27.46 AM</span><span class="informations">595×632 43.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I get nan’s for the loss while the loss gets down to 0.03 in the notebook. (The speed is another issue, unfortunately). The notebook shows a run time of around 6 minutes (CUDA) while it takes 40 minutes on my M1 GPU).</p>
<p>To those who have Mac M1 GPU’s, can you replicate the results in the PyTorch tutorial above? Does anyone know if there is an element of the transformer architecture which is not supported by MPS yet? It doesn’t throw off any errors but I suspect that adjustments have to be made?</p>
<p>Help appreciated!</p>

radicalprotns: <p>To anyone wondering, apparently, there are problems with GRU on mps. See <a href="https://github.com/pytorch/pytorch/issues/94691" rel="noopener nofollow ugc">this link</a></p>
DataLoader batches but also give you the original dataset indices of that batch

gandolfin69420: <p>Not sure how to formulate the question to search whether it has been already asked.<br>
So I have a DataLoader that is also shuffled, so for each batch of size batch_size, I want a index/tensor of length batch_size containing the indices the samples have had in the original dataset.<br>
So if for example batch_size=2 the DataLoader randomly chose the 100th and 420th sample of the dataset, the batch should instead be a 2-tuple of (index_or_tensor(100,420), tensor(sample_100,sample_420)).</p>
<p>Thanks for your answer</p>

ptrblck: <p>You can return the indices in the <code>Dataset.__getitem__</code> method including the samples by creating a custom <code>Dataset</code>.</p>

yiftach: <p>Alternatively, <code>TensorDataset</code> should work without requiring a custom class.<br>
It accepts tensors as input, and returns a tuple:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import TensorDataset

dummy_data = torch.arange(100, 1000) # replace with your data

dataset = TensorDataset(torch.arange(len(dummy_data)), dummy_data)

dataset[[100, 420]] # gives (tensor([100, 420]), tensor([200, 520]))
</code></pre>
Best practice for loss for batch size > 1: sum vs average

yiftach: <p>Hi,<br>
Multiple papers about scaling up training to large batch sizes usually discuss how as the batch size increases, the learning rate should increase by the same factor (I can find links if needed, but I think it’s a known phenomenon).<br>
As far as I can understand, this assumes the final loss is calculated as the <strong>sum</strong> of the loss of each entry in the batch.<br>
Lately, I’ve discovered that when using DDP with multiple workers (say, on a single node), their gradients are <strong>averaged</strong>, which to my understanding does not require any modification to the learning rate, which seems simpler.<br>
Should it be considered best practice, then, to always calculate the loss as the average of all entries in the batch, which would allow to not have to scale the learning rate? Or am I missing something?</p>
<p>Thanks,<br>
Yiftach</p>
Optimize over some columns of the input tensor

Zinuo_Yi: <p>Hi.</p>
<p>Suppose I have input tensor <code>x</code> in shape of [B, n_features]. Instead of optimize the parameters of nn, I freeze the nn and want to optimize only several columns of the input. Conceptually like this:</p>
<pre><code class="lang-python">x = torch.rand(batch_size, n_features)
x.requires_grad_()
col = [0, 2] # pick out several specifical columns
opt = torch.optim.Adam(params = [x[:, col]], lr = 0.01)

for epoch in range(epoches):
        opt.zero_grad()
        y = model(x)
        loss = Loss(x, y)
        loss.backward()
        opt.step()
</code></pre>
<p>If I directly run like above, the optimizer complains like:</p>
<pre><code class="lang-bash">ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>Is there a canonical way to do this?<br>
Thanks in advance.</p>

KFrank: <p>Hi Yi!</p>
<aside class="quote no-group" data-username="Zinuo_Yi" data-post="1" data-topic="188452" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zinuo_yi/48/63972_2.png" class="avatar"> Zinuo_Yi:</div>
<blockquote>
<p>If I directly run like above, the optimizer complains like:</p>
<pre data-code-wrap="bash"><code class="lang-bash">ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>Is there a canonical way to do this?</p>
</blockquote>
</aside>
<p>Because pytorch operates on entire tensors, there’s no simple efficient way to<br>
optimize just a portion of your input tensor.</p>
<p>The cleanest general way to do what you want is to let pytorch optimize your<br>
entire <code>x</code> and then restore the original values of the columns of <code>x</code> that you didn’t<br>
want to optimize.</p>
<p>Something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.rand(batch_size, n_features)
col = [0, 2]   # pick out several specified columns

x_frozen = x[:, col].clone()   # save values of columns to freeze

x.requires_grad_()
opt = torch.optim.Adam (params = [x], lr = 0.01)   # optimize all of x

for epoch in range(epoches):
        opt.zero_grad()
        y = model(x)
        loss = Loss(x, y)
        loss.backward()
        opt.step()
        with torch.no_grad():
                x[:, cols] = x_frozen   # restore values of frozen columns
</code></pre>
<p>(As an aside, you might want to turn off <code>requires_grad</code> for your <code>model</code><br>
parameters.  As it stands, your model won’t be optimized, but gradients<br>
for its parameters will be accumulated by every call to <code>loss.backward()</code>.<br>
This costs time in the backward pass and memory in the computation graph<br>
(which may or may not matter, depending on your use case).  This also could<br>
hypothetically lead to overflow in the <code>.grad</code>s for your model parameters if you<br>
train for many iterations.  If this is a concern, you could call <code>model.zero_grad()</code>.</p>
<p>Best.</p>
<p>K. Frank.</p>
Validation acuraccy decrease interpretation

rgbarriada: <p>Hi,</p>
<p>I am working on an image classification problem with CNN. Specifically, I am using a pre-trained VGG16 (ImageNet) with a relatively small dataset (1400 images). After about 500 iterations, it seems that the model converges correctly.</p>
<p>The problem is that, although the accuracy in train increases correctly, the accuracy of the validation set decreases almost from the beginning of the process.</p>
<p>You can see in the images below different independent configurations for train/test, but all with similar behavior.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/234dd4372fd25282d89fc2585c2ba198f589ac16.png" data-download-href="https://discuss.pytorch.org/uploads/default/234dd4372fd25282d89fc2585c2ba198f589ac16" title="imagen"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234dd4372fd25282d89fc2585c2ba198f589ac16_2_690x128.png" alt="imagen" data-base62-sha1="52jstochZuQM4SK7D9ZQuuZanz0" width="690" height="128" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234dd4372fd25282d89fc2585c2ba198f589ac16_2_690x128.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/234dd4372fd25282d89fc2585c2ba198f589ac16_2_1035x192.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/234dd4372fd25282d89fc2585c2ba198f589ac16.png 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagen</span><span class="informations">1225×229 37.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>graph increasing (model over train set) | graph decreasing (model over validation set)</p>
</blockquote>
<p>Looking at the graphs, I would like to know if it is possible to interpret an overfitting on the train data. Also, Why is the model performing better on the validation set with the pre-trained network, than after the train process?</p>
<p>Thanks for your help!</p>
How to find codegen CUDAFunctor_add kernel code?

shizao: <p>I’m wondering about the exactly code of the  CUDAFunctor_add kernel .<br>
But I can’t find any in pytorch source code.<br>
So I guess maybe  it’s generated by codegen feature.<br>
But how to print out the codegen kernel code ?<br>
Any help would be appreciated.</p>
Questions about batch normalization layer in EfficientNet (timm) with gradient checkpointing

thinhbang: <p>Hi,</p>
<p>I have been reading the code of EfficietNet from timm, and it has gradient checkpointing supported. From the tutorial: <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb" rel="noopener nofollow ugc">https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb</a>, what I understand is that we need to deal with batch normalization layers. Therefore, I have two questions as follows:</p>
<ol>
<li>If the number of checkpoints is n, does it mean the runing_mean has to be updated n-th times, with the momentum value of nth_root(m)</li>
<li>How does timm’s EfficietNet deal with batch normalization layers when gradient checkpointing is enabled?</li>
</ol>
How to remove diagnostic message while using torch.onnx.export

ashmallow: <p>Hi,</p>
<p>I am wondering if there is a way to remove the console message below while using torch.onnx.export() function.</p>
<pre><code class="lang-auto">================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR =============
</code></pre>
<p>We are automating a process to create hundreds of onnx models in a row and these messages can make the log cluttered and hard to read. I understand that an older version of PyTorch (torch 1.13.1) did not have this behavior, so I’m wondering if there is a way to remove it in version 2.0.1</p>
<p>Thank you.</p>

ptrblck: <p>I don’t know which exact commit changed the printing behavior, but <a class="mention" href="/u/bowenbao">@BowenBao</a> might know.</p>

Liphos: <p>Hi do we have an update about it ?</p>
<p>Thank you</p>

ashmallow: <p>We decided to leave the diagnostic messages as is given the side effects of the two solutions below:</p>
<ol>
<li>We could run the script like this: <code>python3 onnx_export.py &gt;&gt; /dev/null 2&gt;&amp;1 </code>. All stdout output will be dumped into <code>/dev/null</code> which discards it, leaving only <code>stderr</code> showing. This has the side effect of getting rid of all print statements in the script, which we don’t want.</li>
<li>We could also use an earlier version of PyTorch such as 1.13.1, but the side effect being the potential negative effects of not using the latest version of PyTorch.</li>
</ol>

Andrew_Lavin_Phantom: <p>The accepted answer does not solve the problem. Users would like to avoid this output clutter.</p>
<p>If there are no errors to report, then by default there should be no console message. Users who wish the “Diagnostic” summary even when there are no errors should enable it explicitly.</p>
Bidirectional lstm batched padded sequence

sumeet_baghel: <p>I’m passing a pack_padded_sequence to the bidirectional lstm and i want to get the foward and reverse lstm outputs, concatenate both of them and pass it to the fully connected layer.Also when i use pad_packed_sequence on the ‘h_t’ output of the lstm it gives the original padded sequence which has pads in it.</p>
<p>How should i proceed after retrieving the ‘h_t’ from the lstm?</p>

ptrblck: <p>I’m not sure to completely understand your use case, but I usually refer to <a class="mention" href="/u/vdw">@vdw</a>’s <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">post</a> to get a quick recap how to slice which output.</p>

sumeet_baghel: <p>Thanks for replying, i figured it out.</p>

MagedSaeed: <p>Can you please mention how?</p>
Any / Oneof patterns for SubgraphMatcher

jeremalloch: <p>I’m using the <a href="https://github.com/pytorch/pytorch/blob/aed9bee0413dac190452fbfa9ab2a44b6e6843f5/torch/fx/passes/utils/matcher_utils.py#L51" rel="noopener nofollow ugc">SubgraphMatcher</a>, and I’d like to be able to create patterns for a set of nodes.<br>
For example, matching a convolution → batch norm → activation function (where that could be a ReLU, GeLU, etc).  I see there once were <a href="https://github.com/pytorch/pytorch/pull/82853" rel="noopener nofollow ugc">Any and Oneof patterns</a>, but they were quickly reverted, and I don’t see any similar functionality.  Is there another way to implement this sort of functionality?</p>
Issue with R2 Score returning strange values

Caue_Evangelista: <p>Hello, dear community!</p>
<p>I’ve built an AutoEncoder and would like to obtain a metric that assesses the similarity between my outputs and inputs(since using just the loss it’s not as much straightforward).</p>
<p>My data has dimensions <code>[batch_size, channels, columns] = [40, 1, 22]</code>, which means the model processes 40 lines, each consisting of 22 positive values, with 1 channel.</p>
<p>While researching how to obtain a metric similar to accuracy for a regression task, I came across the R2 Score, which ranges from 0 to 1. However, I’m encountering some unusual results during preliminary tests, as sometimes the R2 score turns out to be negative.</p>
<p>I’ve read <a href="https://discuss.pytorch.org/t/r2-score-return-inf/181348">another post on the PyTorch Forum</a>, but I’m still confused about whether I can effectively use the R2 score or if I should consider an alternative approach.</p>
<p>Here are my tests and the results:</p>
<pre><code class="lang-auto"># PyTorch
import torch

# Metrics
from torcheval.metrics import R2Score
from sklearn.metrics import r2_score

input1 = torch.tensor([0, 2, 1, 3])
target1 = torch.tensor([0, 1, 2, 3])

input2 = torch.Tensor(4).random_(0, 3)
target2 = torch.Tensor(4).random_(0, 3)

input3 = torch.tensor([[0, 2, 1, 3], [0, 2, 1, 3]])
target3 = torch.tensor([[0, 1, 2, 3], [0, 2, 1, 3]])

metric = R2Score()

print("--------------------------------------")
print("PYTORCH")
metric.update(input1, target1)
print("R2Score for input1 and target1: ", round(metric.compute().item(), 2))
metric.update(input2, target2)
print("R2Score for input2 and target2: ", round(metric.compute().item(), 2))
metric.update(input3, target3)
print("R2Score for input3 and target3: ", round(metric.compute().item(), 2))

metric = R2Score()
batch_size = 40
input4 = torch.Tensor(batch_size, 22).random_(0, 10)
target4 = torch.Tensor(batch_size, 22).random_(0, 10)
metric.update(input4, target4)
print("R2Score for input4 and target: ", round(metric.compute().item(), 2))

print("--------------------------------------")
print("SKLEARN")
print("R2Score for input1 and target1: ", round(r2_score(input1, target1), 2))
print("R2Score for input2 and target2: ", round(r2_score(input2, target2), 2))
print("R2Score for input3 and target3: ", round(r2_score(input3, target3), 2))
print("R2Score for input4 and target4: ", round(r2_score(input4, target4), 2))
print("--------------------------------------")
</code></pre>
<pre><code class="lang-auto">--------------------------------------
PYTORCH
R2Score for input1 and target1:  0.6
R2Score for input2 and target2:  0.47
R2Score for input3 and target3:  nan
R2Score for input4 and target:  -1.01
--------------------------------------
SKLEARN
R2Score for input1 and target1:  0.6
R2Score for input2 and target2:  -1.0
R2Score for input3 and target3:  0.5
R2Score for input4 and target4:  -1.22
--------------------------------------
</code></pre>
<p>I really apreciate any insights! Thank you in advance!</p>
Graph execution in Pytorch 2.x

trusira: <p>Hi,<br>
I’m trying to understand the graph execution of Pytorch 2. I came across <a href="https://pytorch.org/blog/how-computational-graphs-are-executed-in-pytorch/" rel="noopener nofollow ugc">this</a> blog series, but looks like it was written for PT 1.x. I would like to know if the graph execution mechanics have changed since and if so, where I should look instead.<br>
Thanks,</p>
Unable to access CUDA using torch on aarch64 architecture

dagrawal: <p>I have a custom yocto build with aarch64 architecture running Linux. I have installed pytorch using Python. When I check torch.cuda.is_available(), it always returns false even though I have a CUDA capable GPU (nvidia A6000).<br>
Also, I am unable to install libtorch on it, I get an error saying “wrong file format”.</p>

ptrblck: <p>The PyTorch ARM binaries do not support CUDA at the moment as they are built for Mac.<br>
You could use the <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">PyTorch NGC container</a> which is built for x86 and ARM and supports CUDA.</p>

dagrawal: <p>Oh okay. Does installing pytorch from source support cuda? Is there any container for libtorch as well?</p>

ptrblck: <p>Yes, our source builds support CUDA as mentioned before. <code>libtorch</code> is also built and used from the Python API. To build a standalone <code>libtorch</code> application you can use</p>
<blockquote>
<p>cmake -DCMAKE_PREFIX_PATH=<code>python -c 'import torch;print(torch.utils.cmake_prefix_path)'</code></p>
</blockquote>
<p>which would find the right libraries.</p>

dagrawal: <p>Okay, I will try this. Thank you for your help!</p>

dagrawal: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, I am building pytorch from source on arm64 but I do not have a conda environment, however I have a python virtual environment. Do you guys have any documentation for a non-conda installation?</p>
<p>Instructions at <a href="https://github.com/pytorch/pytorch/#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a> need conda.</p>

ptrblck: <p>You don’t need conda and can install <code>magma-cuda</code> from source (if needed) e.g. via cloning the repository and running:</p>
<pre><code class="lang-auto">cmake -H. -Bbuild -DUSE_FORTRAN=OFF -DGPU_TARGET="All" -DBUILD_SHARED_LIBS=OFF -DCMAKE_CXX_FLAGS="-fPIC" -DCMAKE_C_FLAGS="-fPIC" -DCUDA_NVCC_FLAGS="-Xfatbin;-compress-all;-DHAVE_CUBLAS;-std=c++11;--threads=0;" -GNinja
cmake --build build --target install
</code></pre>
<p>and could also install MKL from PyPI although it’s irrelevant for ARM.</p>

dagrawal: <p>I got this error</p>
<pre><code class="lang-auto">CMake Error at cmake/Dependencies.cmake:1454 (add_dependencies):
  Cannot add target-level dependencies to non-existent target "gloo_cuda".

  The add_dependencies works for top-level logical targets created by the
  add_executable, add_library, or add_custom_target commands.  If you want to
  add file-level dependencies see the DEPENDS option of the add_custom_target
  and add_custom_command commands.
Call Stack (most recent call first):
  CMakeLists.txt:728 (include)
</code></pre>
<p>I have also attached the <a href="https://drive.google.com/file/d/1_WVo5QM_JHz6QDcPL7ISS6BeQ3B8rN3y/view?usp=sharing" rel="noopener nofollow ugc">CMakeError.log</a></p>
What are the "correct" CMake settings to compile the PyTorch?

pholonow: <p>Hi Everyone!<br>
I would like to compile a custom version of the PyTorch. Could someone kindly tell me, what are the “safe” CMake settings that guarantee a successful compilation? In particular, we would like to use CUDA 12.1.<br>
Thanks!<br>
Piotr</p>

ptrblck: <p>You can just stick to the build instructions on GitHub, which should not use any custom cmake settings.</p>

kenkit1: <p>I use this and it works right will download torch automatically</p>
<pre><code class="lang-auto">
set(Torch_DIR ${TORCH_INSTALL_DIR}/share/cmake/Torch)
Set(FETCHCONTENT_QUIET FALSE)
FetchContent_Declare(
  Torch
  URL       ${TORCH_URL}
  URL_HASH   MD5=${URL_HASH}
  SOURCE_DIR ${TORCH_INSTALL_DIR}
  OVERRIDE_FIND_PACKAGE
  DOWNLOAD_EXTRACT_TIMESTAMP
)
FetchContent_Populate(Torch)


find_package(Torch REQUIRED)
</code></pre>
Dataset/Dataloader class from cosmos database

mikey_t: <p>Hi,<br>
I’m trying to load an image dataset when the images are stored in an Azure Cosmos Database. I am deploying a model on an Azure instance, and have the images in blob storage in a Cosmos DB.</p>
<p>Is there any way to do this directly? (Or has anyone else tried this?) An alternative approach would be to pull all of the images from the DB into the instance, and then load the data like I normally would. But I assume this would be slower and I’d like to avoid the extra step if there is a work around.</p>
<p>Thanks in advance.</p>
How to get images idx=1,222,3333,55555 with a DataLoader?

wiiiktor: <p>From a training dataset for CIFAR100 (but this question is rather general) I want to load images with specific indexes, like for example 1, 222, 3333, 55555 (so, this would be a batch_size=4). How can I do it? Efficiency here would be cool, obviously, but currently I looking for any method.</p>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn (But I already caculate the grad_fn!))

yerongguang.2000: <p>When I run:</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn<br>
from torch.autograd import Variable<br>
class ParetoSetModel(torch.nn.Module):<br>
def <strong>init</strong>(self, n_dim, n_obj, problem):<br>
super(ParetoSetModel, self).<strong>init</strong>()<br>
self.n_dim = n_dim<br>
self.n_obj = n_obj<br>
self.problem = problem<br>
self.fc1 = nn.Linear(self.n_obj, 256)<br>
self.fc2 = nn.Linear(256, 256)<br>
self.fc3 = nn.Linear(256, self.n_dim*5)<br>
self.shape = (10, n_dim, 5)<br>
# self.fc3 = nn.Linear(256, self.n_dim)<br>
def forward(self, pref):</p>
<pre><code>    x = torch.relu(self.fc1(pref))
    x = torch.relu(self.fc2(x))
    x = self.fc3(x)
    x = x.view(self.shape)
    x = torch.softmax(x, dim=2)
    x = torch.argmax(x,dim=2)
    return x
</code></pre>
<blockquote>
<p>psmodel(pref_vec).backward(tch_grad)</p>
</blockquote>
</blockquote>
<p><em><strong>I get the error: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</strong></em></p>
<p>In my code, the psmodel(pref_vec) variable is <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Out[1]:<br>
tensor([[3, 0, 4, 1, 0, 4],<br>
[4, 4, 4, 0, 0, 4],<br>
[3, 0, 4, 1, 0, 4],<br>
[4, 4, 4, 0, 0, 4],<br>
[4, 4, 4, 0, 0, 4],<br>
[4, 4, 4, 0, 0, 4],<br>
[4, 0, 4, 0, 0, 4],<br>
[4, 4, 4, 0, 0, 4],<br>
[3, 0, 4, 1, 0, 4],<br>
[3, 0, 4, 1, 0, 4]])</p>
<p>tch_grad variable is:<br>
Out[2]:<br>
tensor([[-5.7283e-01,  7.1886e-01, -1.8480e-02, -3.8009e-01, -1.0146e-01,<br>
-8.2423e-04],<br>
[-2.0862e-01,  6.5151e-01, -7.2875e-01, -3.0513e-02, -1.4981e-03,<br>
-3.9053e-04],<br>
[-5.7287e-01,  7.1879e-01, -1.8521e-02, -3.8016e-01, -1.0147e-01,<br>
-8.8013e-04],<br>
[-2.0861e-01,  6.5153e-01, -7.2873e-01, -3.0524e-02, -1.4975e-03,<br>
-4.4674e-04],<br>
[-2.0861e-01,  6.5153e-01, -7.2873e-01, -3.0526e-02, -1.4974e-03,<br>
-4.5370e-04],<br>
[-2.0861e-01,  6.5154e-01, -7.2873e-01, -3.0530e-02, -1.4972e-03,<br>
-4.7680e-04],<br>
[ 2.9420e-01,  2.7046e-01, -8.6619e-01, -3.0001e-01,  9.6886e-04,<br>
-1.8034e-03],<br>
[-2.0863e-01,  6.5151e-01, -7.2875e-01, -3.0512e-02, -1.4981e-03,<br>
-3.8361e-04],<br>
[-5.7287e-01,  7.1880e-01, -1.8519e-02, -3.8015e-01, -1.0147e-01,<br>
-8.7734e-04],<br>
[-5.7288e-01,  7.1877e-01, -1.8532e-02, -3.8017e-01, -1.0147e-01,<br>
-8.9424e-04]], dtype=torch.float64, grad_fn=)</p>

ptrblck: <p><code>torch.argmax</code> is not differentiable so remove it from the model.</p>
Better stack trace for jit error messages

Ritesh_Khurana: <p>Recently while converting the model to onnx format I got this stack trace</p>
<pre><code class="lang-auto">[W lower_tuples.cpp:251] Warning: tuple appears in the op outputs, but this op does not forward tuples, unsupported kind: prim::unchecked_cast (function flattenOutputs)
Traceback (most recent call last):
  File "tools/export_as_trt.py", line 153, in &lt;module&gt;
    main()
  File "tools/export_as_trt.py", line 138, in main
    torch.onnx.export(
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/__init__.py", line 350, in export
    return utils.export(
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 163, in export
    _export(
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 1074, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 731, in _model_to_graph
    graph = _optimize_graph(
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 234, in _optimize_graph
    _C._jit_pass_lower_all_tuples(graph)
RuntimeError: prim::TupleUnpack not matched to tuple construct
</code></pre>
<p>Though the issue mentions an mismatch of operation with tuple output. It is not clear as to which part of the code throws this error and hence it is difficult to debug.</p>
<p>Any thoughts on how to go about this?</p>

Yue_Wang1: <p>I have the same issue. Did you solve it?</p>
Contrast-limited adaptive histogram equalization (CLAHE)

Pandelani: <p>I wanna do a Contrast-limited adaptive histogram equalization (CLAHE) as one of my custom transforms, how should i go about adding this in my transforms?</p>

vmirly1: <p>You can define a function that performs the sequence of operations for CLAHE on a single image in NumPy array or Torch tensor format.  Then, write a Dataset class, and in your <code>__getitem__</code> function call that function for CLAHE and pass the image tensor to do the job.</p>

lavender99: <p>did you find the answer for this question ?<br>
if so can you please share it</p>

voidpunk: <p>Since I needed it too, I wrote mine. Here’s my basic solution (no randomization since I need to apply it every time with the same parameters, but you can add it easily).</p>
<pre><code class="lang-auto">import torch
import kornia
from typing import Tuple


class Clahe(torch.nn.Module):

    def __init__(self, clip_limit: int | float = 40, grid_size: Tuple[int, int] = (8, 8)) -&gt; None:
        super().__init__()
        self.clip_limit, self.grid_size = float(clip_limit), grid_size

    def forward(self, img: torch.Tensor) -&gt; torch.Tensor:
        return kornia.enhance.equalize_clahe(img, self.clip_limit, self.grid_size)

    def __repr__(self) -&gt; str:
        return "{}(clip_limit={}, tile_grid_size={})".format(
            self.__class__.__name__,
            self.clip_limit,
            self.grid_size
        )
    
</code></pre>
<p>I hope you’ll find it useful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Modify Loaded TorchScript Model

Orcun_Deniz: <p>I have a torchscript model that I lost its source code, so I cant create the architecture from scratch. I want to retrain this model. A classic train loop is actually working fine but the model was exported in eval mode which causes batchnorms not to track activation stats through the training (their weights and biases are still being optimezed wrt to the loss) .</p>
<p>I am trying to instantiate new batchnorm layers instead of scripted ones as below but still no luck.</p>
<pre><code class="lang-auto">model = torch.jit.load("scripted_model.pt")

def replace_bn(m):
    if m.original_name == "BatchNorm2d":
        m = torch.nn.BatchNorm2d(module.weight.shape[0])

model.apply(replace_bn)
</code></pre>
<p>How can I have a fully functioning training?</p>
Allocation of Tensor on CUDA fails

skaldesh: <p>Hi, want to create a tensor from an existing image buffer copied directly to the GPU during creation,</p>
<p>The model was loaded like this:<br>
<code>mod_ = torch::jit::load("/data/deeplabv3-mobilenetv3-large.pt", torch::Device(torch::kCUDA, 0));</code></p>
<p>To create the tensor:<br>
<code>torch::from_blob((void*)buffer, {1, 3, 520, 520}, torch::TensorOptions().dtype(torch::kFloat32).device(torch::Device(torch::kCUDA, 0))));</code></p>
<p>But this fails with this error:<br>
<code>Specified device cuda:0 does not match device of data cuda:-2</code></p>
<p>I do not understand where this <code>cuda:-2</code> comes from.<br>
A workaround I found is to create the tensor on CPU and then use <code>tensor.to(cudadevice)</code>. But I want to understand why the first one fails.</p>

hyhuang00: <p>Did you managed to solve the problem? I met the same error!</p>

skaldesh: <p>No, so far I am still using my workaround</p>

Amit_Gupta: <p>I am also running in same issue. And problem with using <code>to(device)</code> is that it creates a non_leaf tensor, which gives further headache. How did you get the gradients of tensors moved via <code>.to(device)</code> semantics?</p>

ptrblck: <p>You could create the tensor directly on the right device, if possible or you could call <code>.retain_grad()</code> on the intermediate tensors.</p>

Shobhit_Narayanan: <p>How? This is still an issue.</p>

Jack_Huang1: <p>I also met the same error, so the solution is just use <code>.to(device)</code> to instead?</p>
Low Accuracy with MPS

KMT62: <p>This is a follow up to <a href="https://discuss.pytorch.org/t/really-low-accuraccy-of-0/152085/1">this topic</a>, in which I posted the following:<br>
"A similar issue is found when executing the sample code here: <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" rel="noopener nofollow ugc">Quickstart — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
<p>Specifically in function test(), line:</p>
<pre><code class="lang-auto">correct += (pred.argmax(1) == y).type(torch.float).sum().item()
</code></pre>
<p>When device = ‘mps’ it always results in 10% accuracy. when device = ‘cpu’, the accuracy is as expected."</p>
<p>I am creating a new topic as the old one doesn’t seem to be getting any traction, and this issue may be a different bug.</p>

Mah_Neh: <p>I’ve been training in MPS, use exactly that line as well.</p>
<p>No accuracy issues. Not sure how to help without mor info.</p>
<p>Maybe you can add the torch, torchvision and python versions. There could be also problems with some specific operation, so maybe the NN arch is useful as well.</p>
<p>Actually I think instead of <code>type(torch.float())</code> I use just  <code>float()</code></p>

KMT62: <p>Thanks for the response.<br>
I tried changing <code>type(torch.float())</code> to <code>float()</code>, but that didn’t change anything; still got an accuracy of 10% with MPS on all epochs.</p>
<p>I am using the following:<br>
Python: 3.11.4<br>
PyTorch: 2.0.1<br>
Torchvision: 0.15.2</p>
<p>I am using an Intel-based Mac (older AMD GPU, macOS 12.6.9) if that makes any difference. I believe it supports Metal v2.0, not 3.0…</p>
Unknown CMake command "cuda_select_nvcc_arch_flags"

dagrawal: <p>I am using cmake in my application and it results in the following errors:</p>
<pre><code class="lang-auto">CMake Error at /home/divyansh/Documents/libtorch-cxx11-abi-shared-with-deps-2.0.1+cu118/libtorch/share/cmake/Caffe2/public/utils.cmake:401 (cuda_select_nvcc_arch_flags):
  Unknown CMake command "cuda_select_nvcc_arch_flags".
Call Stack (most recent call first):
  /home/divyansh/Documents/libtorch-cxx11-abi-shared-with-deps-2.0.1+cu118/libtorch/share/cmake/Caffe2/public/cuda.cmake:362 (torch_cuda_get_nvcc_gencode_flag)
  /home/divyansh/Documents/libtorch-cxx11-abi-shared-with-deps-2.0.1+cu118/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/divyansh/.local/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:290 (find_package)
</code></pre>
<p>My cmake version is 3.27.0</p>

ptrblck: <p>Double post from <a href="https://github.com/pytorch/pytorch/issues/8269">here</a>.</p>

dagrawal: <p>That issue is 5 years old, the assignees might not respond, so I opened an issue here. Sorry for the repetition.</p>

ptrblck: <p>Based on a few related issues, such as <a href="https://github.com/pytorch/pytorch/issues/8249">this one</a>, it seems updating <code>cmake</code> to &gt;=3.9 fixes the issue. However, I’m unsure why it’s not working with your newer version, as I’m also using 3.27.4 and haven’t seen the error.</p>

dagrawal: <p>Thank you for your response! Do you have any suggestions on how I can know what the issue is?</p>

dagrawal: <p>Update: Specifying the path of the torch configuration files using -DCMAKE_PREFIX_PATH resolved the issue.</p>
Simple transformer

kenkit1: <p>Hi,<br>
I’m trying to create a simple transformer.<br>
Why can’t this code work I get an error here</p>
<pre><code class="lang-auto">// Define the Transformer model
#ifndef TRANSFORMER_MODEL_H
#define TRANSFORMER_MODEL_H

#include &lt;torch/torch.h&gt;
struct TransformerModel : torch::nn::Module
{
    TransformerModel(int input_size, int num_heads, int num_layers, int hidden_dim)
    {
        // Define the input and output embeddings
        embed = register_module("embed", torch::nn::Linear(input_size, hidden_dim));

        // Create a multi-head self-attention layer
        self_attn = register_module("self_attn", torch::nn::MultiheadAttention(hidden_dim, num_heads));

        // Create a feedforward neural network
        feedforward = register_module("feedforward", torch::nn::Sequential(
                                                         torch::nn::Linear(hidden_dim, hidden_dim),
                                                         torch::nn::ReLU(),
                                                         torch::nn::Linear(hidden_dim, hidden_dim)));

        // Create the transformer layers
        transformer_layers = register_module("transformer_layers", torch::nn::TransformerEncoderLayer(hidden_dim, num_heads));

        // Create the transformer encoder
        transformer_encoder = register_module("transformer_encoder", torch::nn::TransformerEncoder(transformer_layers, num_layers));

        // Define the output layer
        output_layer = register_module("output_layer", torch::nn::Linear(hidden_dim, 1));
    }

    torch::Tensor forward(torch::Tensor x)
    {
        // Embed the input data
        x = embed-&gt;forward(x);


        // Perform multi-head self-attention
       //ERROR IS HERE
///(38,9): error C2679: binary '=': no operator found which takes a right-hand operand of type 'std::tuple&lt;at::Tensor,at::Tensor&gt;' (or there is no acceptable conversion) 
        x = self_attn-&gt;forward(x, x, x);

        // Apply feedforward network
        x = feedforward-&gt;forward(x);

        // Apply the transformer encoder
        x = transformer_encoder-&gt;forward(x);

        // Apply the output layer
        x = output_layer-&gt;forward(x);

        return x;
    }

    torch::nn::Linear embed{nullptr};
    torch::nn::MultiheadAttention self_attn = nullptr;

    torch::nn::Sequential feedforward{nullptr};
    torch::nn::TransformerEncoderLayer transformer_layers{nullptr};
    torch::nn::TransformerEncoder transformer_encoder{nullptr};
    torch::nn::Linear output_layer{nullptr};
};
#endif
</code></pre>
<p>Thanks in advance</p>