akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Architecture suggestion for time series data

Subhankar_Ghosh: <p>I have a time series dataset (spatiotemporal, but not an image/video). The dataset is in 3D, where each (x,y,t) coordinate has a numeric value (such as the elevation of the sea at that location and at that specific point in time). So we can think of it as a matrix with a temporal component.</p>
<p>I need to predict/forecast the future (next few time steps) values for the whole region (i.e. all x,y coordinates in the dataset). I was thinking of using a ConvLSTM or CNN-LSTM but most of the posts online seem to be applied to video frame prediction. Since it’s not a video, I only have 1 channel for each time instance.</p>
<p>Can you all suggest an architecture that would be a good fit for my purpose? Thanks!</p>
Developing a Pytorch model to determine whether one image appears in another. 

Reem_Alharthi: <p>Hello everyone,<br>
I have a dataset with a collection of images called “outfits,” and for each outfit image, there is a list of images for the items that make up the outfit. Given two photos, for example, image X and image a, the model should recognize the two images and predict whether the second image is part of the outfit or not. In addition, if the second image appears in the first image, the model should predict the bounding box for this object. I’m new to Pytroch and have no idea how to develop such a model.<br>
My initial attempt is to use one of Pytorch’s pretrained models to obtain the two image embeddings (such as X and an or X and j), then concatenate the two embeddings and output the predictions and bounding box using a head. The model, however, never converges, and I get no results. I’m now stuck at this stage. Any advice on how to put this idea into practice, as well as any materials on an issue or implementation that is comparable to mine, would be really appreciated. Also, any advice on how to train such a model and if it is a difficult or simple challenge to train a network to predict such an output would be greatly appreciated. Thank you ahead of time.Thank you in advance.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c981e0b54d06c46d997ae243c492bc135629d5cc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c981e0b54d06c46d997ae243c492bc135629d5cc" title="data"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c981e0b54d06c46d997ae243c492bc135629d5cc_2_690x388.jpeg" alt="data" data-base62-sha1="sKCet9JnhTHGT99SDwHrbaHEsRm" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c981e0b54d06c46d997ae243c492bc135629d5cc_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c981e0b54d06c46d997ae243c492bc135629d5cc_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c981e0b54d06c46d997ae243c492bc135629d5cc.jpeg 2x" data-dominant-color="D9DADD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">data</span><span class="informations">1280×720 82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Error using MSELoss with CNN: Using a target size that is different to the input size

YPBS: <p>Hi,</p>
<p>I am running a CNN experiment where my <strong>y</strong> shape is <strong>(32, 56, 56)</strong> and <strong>y_pred</strong> is <strong>(32, 313, 56, 56)</strong>, 32 is batch size and images are 56x56. For classification when I use nn.CrossEntropyLoss() it accepts these tensors of different shapes but I also need to get results for MSELoss to show that MSE is not the better metric. I get following error:</p>
<pre><code class="lang-auto">loss_fn = nn.MSELoss(reduction='mean')

UserWarning: Using a target size (torch.Size([32, 56, 56])) that is different to the input size (torch.Size([32, 313, 56, 56])). 
This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<p>Can you please let me know how use MSELoss with different tensor shapes. I checked the parameters in nn.MSELoss() page but did not find anything I can use.</p>
<p>Thank You,</p>

ptrblck: <aside class="quote no-group" data-username="YPBS" data-post="1" data-topic="177794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/d2c977/48.png" class="avatar"> YPBS:</div>
<blockquote>
<p>For classification when I use nn.CrossEntropyLoss() it accepts these tensors of different shapes</p>
</blockquote>
</aside>
<p>That’s true because the target is expected to contain class indices to index the class dimension of the model’s output tensor.</p>
<aside class="quote no-group" data-username="YPBS" data-post="1" data-topic="177794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/d2c977/48.png" class="avatar"> YPBS:</div>
<blockquote>
<p>but I also need to get results for MSELoss to show that MSE is not the better metric.</p>
</blockquote>
</aside>
<p>You won’t be able to directly compare class indices with logits which your model returns.<br>
Instead you might need to one-hot encode the target tensor via <code>F.one_hot</code> and apply a <code>softmax</code> on your model output to create probabilities.<br>
I would expect this use case to perform worse than the previous one using <code>nnCrossEntropyLoss</code>.</p>
Trouble running CUDA

Asasafuchi: <p>So as the title suggest I’m having trouble running cuda, this is the output I get when i run nvcc --version</p>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2022 NVIDIA Corporation<br>
Built on Tue_Mar__8_18:36:24_Pacific_Standard_Time_2022<br>
Cuda compilation tools, release 11.6, V11.6.124<br>
Build cuda_11.6.r11.6/compiler.31057947_0</p>
<p>And I also have cuda 11.6.1 installed on pytorch but when I run the command ‘torch.cuda.is_available()’ I get the following error: ‘torch.cuda.is_available’ is not recognized as an internal or external command, operable program or batch file.</p>

ptrblck: <aside class="quote no-group" data-username="Asasafuchi" data-post="1" data-topic="177765">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asasafuchi/48/59606_2.png" class="avatar"> Asasafuchi:</div>
<blockquote>
<p>but when I run the command ‘torch.cuda.is_available()’ I get the following error: ‘torch.cuda.is_available’ is not recognized as an internal or external command, operable program or batch file.</p>
</blockquote>
</aside>
<p>Are you trying to execute this command directly in your terminal?<br>
Note that this line of code is Python code so you should run it in a Python script instead.</p>
<p>The PyTorch binaries ship with their own CUDA runtime and other CUDA libs. Your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension.<br>
Could you post the output of <code>python -m torch.utils.collect_env</code>?</p>
Training Pytorch model takes a long time to run

WirryWoo: <p>Hello, Pytorch newbie here. I am currently working on training a model that maps sequential data (of varying length) with dimensions <code>(batch_size, seq_len, num_features)</code> to a scalar <code>(batch_size, 1)</code> and I am unsure why my training code is running extremely slowly.</p>
<pre><code class="lang-auto">import torch.nn as nn

num_epochs = 100

class DifficultyModel(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.lstm = nn.LSTM(input_dim, output_dim = 1)

    def forward(self, x):
        output, (hidden, cell) = self.lstm(x)
        return torch.sum(output, dim=1).squeeze(1)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

num_features = 6
model = DifficultyModel(num_features).to(device)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())

# train_dataloader contains 6 features and automatically generates batches in batch_size = 16 but this can be configured.
for epoch in range(num_epochs):
    for i, (X, y) in enumerate(train_dataloader):
        X = X.to(device)
        Y = y.to(device)

        optimizer.zero_grad()

        output = model(X)
        loss = criterion(output, Y)

        loss.backward()
        optimizer.step()

    if epoch % 5 == 0:
        print(f'Epoch {epoch}, Loss {loss.item()}')
</code></pre>
<p>Also, the training output after 3 hours of running the code in Google Colab’s GPU runtime yielded the following:</p>
<pre><code class="lang-auto">Epoch 0, Loss 7248.39453125
Epoch 5, Loss 340.68792724609375
Epoch 10, Loss 320.6564636230469
Epoch 15, Loss 42.046302795410156
Epoch 20, Loss 62.36985778808594
Epoch 25, Loss 23.50816535949707
Epoch 30, Loss 55.912776947021484
Epoch 35, Loss 43.66672134399414
Epoch 40, Loss 27.032011032104492
Epoch 45, Loss 13.70750617980957
</code></pre>
<p>How can I be certain that the model is appropriately learning? Any help would be greatly appreciated.</p>

srishti-git1110: <p>Hi, LSTMs rely on sequential processing of data so the slow training might be expected.</p>
<p>The loss seems to be decreasing very rapidly so the model is indeed learning. You might want to compare the loss value to a baseline loss to see how well the model can learn.</p>
Pretty weird score on test and train set

meyssank: <p>Can someone explain to me how to interpret this behavior. Is the model overfitting and underfitting at each epoch ?<br>
I work with a dataset of size ~60</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38c832af2ebf932ce250d327ca7a08f8eed72930.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/38c832af2ebf932ce250d327ca7a08f8eed72930" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38c832af2ebf932ce250d327ca7a08f8eed72930.jpeg" alt="image" data-base62-sha1="86jEX9OAAMHKE59rsM8hbNdPlmM" width="667" height="500" data-dominant-color="D1DAE3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">674×505 81.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Recursively transforming Pytorch code to JIT script?

netty: <p>Many of the Pytorch tutorials seem to show a combination of tracing (<code>torch.jit.trace</code>) and scripting (<code>torch.jit.script</code>)</p>
<p>However it seems like if all the code in a given <code>nn.Module</code> (even if it contains other modules) is compatible with Torchscript, then do you only need a single line to convert, ie:  <code>torch.jit.script(my_net_instance)</code> ?</p>
<p>See this example:</p>
<pre><code class="lang-auto">import torch

class MyDecisionGate(torch.nn.Module):
    def forward(self, x):
        if x.flatten().sum() &gt; 0:
            return x + 1
        else:
            return -x

class MyCell(torch.nn.Module):
    def __init__(self, dg):
        super(MyCell, self).__init__()
        self.dg = dg
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h: int):
        new_h = torch.tanh(self.dg(self.linear(x)) + h)
        return new_h, new_h
    
class MyOuterModule(torch.nn.Module):
    def __init__(self, my_cell):
        super(MyOuterModule, self).__init__()
        self.my_cell = my_cell

    def forward(self, x):
        new_h, _ = self.my_cell.forward(x, 1)
        return new_h / new_h.sum()

# normal invocation
x = torch.rand(3, 4)
net = MyOuterModule(MyCell(MyDecisionGate()))
original_output = net(x)
print(original_output)

# use the script compiler to recursely turn into
# torchscript
scripted_net = torch.jit.script(net)
torchscript_output = scripted_net(x)
print(torchscript_output)

# check if they are close enough!
assert torch.allclose(original_output, torchscript_output), \
    "torchscript output does not match original"
</code></pre>
<p>Is this the correct way to think about things?</p>
<p>I am trying to understand better how Torchscript generation works.</p>

ptrblck: <p>Yes, you wouldn’t need to script submodules assuming no special conditions or tracing has to be used and should directly be able to script the model via <code>torch.jit.script</code>.</p>

netty: <p>Great, thanks for confirming!</p>
Does PyTorch support conda-CUDA-nvcc?

songyuc: <p>Hi, guys!<br>
It has been found that CUDA-nvcc can be installed via conda. But <code>CUDA_HOME</code> could find its path and return <code>None</code>.<br>
So, could you please add support for <a href="https://anaconda.org/nvidia/cuda-nvcc" rel="noopener nofollow ugc">conda-CUDA-nvcc</a>?</p>
<p>Your answer and guidance will be appreciated!</p>

ptrblck: <p>No, PyTorch does not support <code>nvcc</code> installed via <code>conda</code>. To build PyTorch from source or custom CUDA extensions you would need to install a full CUDA toolkit locally.</p>

songyuc: <p>Thank you very much.</p>
Mac M2 MPS: failed assertion `destination kernel width and filter kernel width mismatch'

JXuan: <p>Hi,</p>
<p>I am training an adversarial autoencoder using PyTorch 2.0.0 and pytorch lightning 2.0.1.post0 on Apple M2 (Ventura 13.1), with conda 23.1.0 as manager.</p>
<p>I encountered this error:</p>
<pre><code class="lang-auto">/AppleInternal/Library/BuildRoots/5b8a32f9-5db2-11ed-8aeb-7ef33c48bc85/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayConvolutionA14.mm:3967: failed assertion `destination kernel width and filter kernel width mismatch
/Users/vk/miniconda3/envs/betavae/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
</code></pre>
<p>To my knowledge, the code broke down when running <code>self.manual_backward(loss["g_loss"])</code> this block:</p>
<pre><code class="lang-auto">g_opt.zero_grad()
self.manual_backward(loss["g_loss"])
g_opt.step()
</code></pre>
<p>The same code run without problems on linux distribution.</p>
<p>Any thoughts on how to fix it are highly appreciated!</p>
Encode time as input for CNN

HelixPiano: <p>Hello everyone,<br>
I am currently looking into how to tell the CNN from which month an image is. I want to use one of the resnet networks for image classification. I know that for my case the month is a very important feature.<br>
I came across an interesting <a href="https://developer.nvidia.com/blog/three-approaches-to-encoding-time-information-as-features-for-ml-models/" rel="noopener nofollow ugc">article</a> on the nvidia website.<br>
Especially the third approach looks interesting, but it results in 12 time feature points per day.<br>
My original idea was to create an additional channel that takes the month as input (in case of a single value I wanted simply resize an array to match the image shape and have that value at every point), but I am not sure how I would do that with 12 values.<br>
Does anyone have an idea?</p>
Learning rate warm-up with SGD

gamebo5000: <p>I am looking for a way to do Epoch warm-ups/ learning rate warmups with SGD, but I can’t find anything useful. The best thing I could find was this site: <a href="https://github.com/Tony-Y/pytorch_warmup" rel="nofollow noopener">https://github.com/Tony-Y/pytorch_warmup</a> but this is for ADAM and I am looking for a way to do this SGD. can someone point me in the right direction?</p>

ptrblck: <p>Have you tried using this package on other optimizers?<br>
And if so, do you get an error? From the docs I cannot see that it’s only limited to <code>Adam</code> or any other optimizer besides <a href="https://github.com/Tony-Y/pytorch_warmup#radam-warmup">this special warmup for RAdam</a>.</p>

slyviacassell: <p>Hi, I think you may miss this description of <a href="https://github.com/Tony-Y/pytorch_warmup#radam-warmup" rel="noopener nofollow ugc">pytorch_warmup</a> in the docs:</p>
<pre><code class="lang-auto">The warmup factor depends on Adam's `beta2` parameter for `RAdamWarmup` . Please see the original paper for the details.
</code></pre>
<p>The author says that the implementation depends on the parameter of Adam. So I think <a href="https://github.com/Tony-Y/pytorch_warmup#radam-warmup" rel="noopener nofollow ugc">pytorch_warmup</a> cannot be used for SGD optimizer. Maybe I will check the code for more details when I am free.</p>

developer0hye: <p>Refer my project to easily use learning rate warmup method!</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/developer0hye/Torch-Warmup">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/developer0hye/Torch-Warmup" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/361ac3a23419cf56399ea6405627ebeda08a66a89d97e7238a88aeba44109bd5/developer0hye/Torch-Warmup" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/developer0hye/Torch-Warmup" target="_blank" rel="noopener nofollow ugc">GitHub - developer0hye/Torch-Warmup: Easiest way to use learning rate warmup...</a></h3>

  <p>Easiest way to use learning rate warmup method on PyTorch - GitHub - developer0hye/Torch-Warmup: Easiest way to use learning rate warmup method on PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Mole_m7b5: <p>The Untuned warmup methods rely on Adam’s beta parameter. But if you just use the standard linear or exponential warmup, where you specify the number of warmup sets, I believe this can be used with any optimiser.</p>
Sample integers greater than 1e19 (`torch.long` overflow)

leonardopetrini: <p>I need to sample <code>N</code> random integers between 0 and <code>N_max</code>, with  <code>N_max &gt; 10^19</code>. If I try,</p>
<pre><code class="lang-auto">N_max = int(1e20)
torch.randperm(N_max)[:N]
</code></pre>
<p>I get overflow:</p>
<pre><code class="lang-auto">RuntimeError: Overflow when unpacking long
</code></pre>
<p>Is there a way to do it?</p>

ConvolutionalAtom: <p>I receive the same error with code below</p>
<pre><code class="lang-auto">N_max = int(1e20)
torch.tensor([N_max], dtype=torch.long)
</code></pre>
<p>So, I think it comes down to <code>torch.long</code> type not being able to handle integers that cannot fit into 64 whereas python int automatically adapts bit size whenever <code>u64</code> bit range is exceeded.<br>
After my research, I was not able to see anything about torch.long (or any other type in torch) dedicated to handling integers that cannot fit into <code>u64</code> range.</p>
Load different sections of data at every epoch using data loader

ays: <p>A ran a pretrained model on a bunch of data and stored the result path in a csv file. the csv file currently contains the following columns; <strong>epoch, index, input_data, result</strong>. Now in my main model I want train using this result from the pretrained model and for 500 epochs. This means I need to retrieve the result for a particular epoch from the csv file at every epoch of my training (so for epoch 1, I will get all data where epoch =1 from the csv file and pass it using dataloader and so on for all 500 epochs), is there a way of doing this using dataloader?</p>

RohitRathore1: <p>You can achieve this by creating a custom dataset class that inherits from <code>torch.utils.data.Dataset</code> and implements the necessary methods. Inside this custom dataset, you can load the data for the specific epoch you’re interested in. Then, you can use a DataLoader to load the data in batches for training.</p>
<pre><code class="lang-python">#Create a custom dataset class
import pandas as pd
import torch
from torch.utils.data import Dataset

class CustomDataset(Dataset):
    def __init__(self, csv_file, epoch, transform=None):
        self.data = pd.read_csv(csv_file)
        self.epoch_data = self.data[self.data['epoch'] == epoch]
        self.transform = transform

    def __len__(self):
        return len(self.epoch_data)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        
        input_data = self.epoch_data.iloc[idx]['input_data']
        result = self.epoch_data.iloc[idx]['result']
        
        if self.transform:
            input_data = self.transform(input_data)

        return input_data, result

#Create a DataLoader for each epoch and use it in your training loop
from torch.utils.data import DataLoader

csv_file = 'your_csv_file.csv'
num_epochs = 500
batch_size = 64

for epoch in range(1, num_epochs + 1):
    print(f"Training for epoch {epoch}")
    
    # Create the custom dataset for the current epoch
    dataset = CustomDataset(csv_file, epoch)
    
    # Create a DataLoader to load the data in batches
    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=0)
    
    # Training loop
    for i, (input_data, result) in enumerate(dataloader):
        # Perform your training step here
        pass
</code></pre>
<p>Above code defines a custom dataset class that reads the data from a CSV file for a specific epoch. In the main training loop, the DataLoader is created for each epoch using the custom dataset. This way, you can load different sections of data for each epoch as required.</p>
<p>but remember that you should preprocess the <code>input_data</code> and <code>result</code> as needed to convert them into the appropriate format for your model.</p>

ays: <p>This makes sense. Thanks a lot!. Will the overall operation be computationally expensive though? seeing as I’m loading data at every epoch</p>
How to get output shape of each node from a loaded jit traced model?

chenzhengda: <p>If a model is traced and sve by torch.jit.trace.Then I load the model just before,the output shapes of nodes in graph are lost, how to get these output shapes of nodes?<br>
Here is an example code:</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch._C import _propagate_and_assign_input_shapes
def _model_to_graph(model, args):
    if isinstance(args, torch.Tensor):
        args = (args, )
    graph = model.forward.graph
    method_graph, params = torch._C._jit_pass_lower_graph(graph, model._c)
    in_vars, in_desc = torch.jit._flatten(tuple(args) + tuple(params))
    graph = _propagate_and_assign_input_shapes(  method_graph, tuple(in_vars), False, False)
    return graph
traced_model_savepath = 'traced.pt'
model = torchvision.models.mobilenet.mobilenet_v2(pretrained=True)
dummy_input = torch.rand((1,3,224,224))
traced_model = torch.jit.trace(model, dummy_input)
traced_model.save(traced_model_savepath)
# load the saved traced model from disk
loaded_traced_model = torch.jit.load(traced_model_savepath)
graph= _model_to_graph(loaded_traced_model,  dummy_input)
print(graph)
</code></pre>
<p>the output shape of nodes are lost, for example:</p>
<pre><code class="lang-auto">  %input.6 : Tensor = aten::_convolution(%626, %251, %276, %627, %628, %629, %277, %630, %274, %277, %277, %278) # /data0/shareVR/pytorch/learn/torch/nn/modules/conv.py:416:0
  %input.104 : Tensor = aten::batch_norm(%input.6, %252, %253, %254, %255, %278, %280, %279, %278) # /data0/shareVR/pytorch/learn/torch/nn/functional.py:2051:0
</code></pre>
How to roll some rows of a matrix

LeanderK: <p>I have a tensor <code>mxm</code> and I want to roll each row according to a specific, computed amount (for an easy example: <code>(row_index // 3)</code>). I have tried using <code>torch.roll</code>, but it appears to roll only across a complete axis and not along a specific dimension.<br>
Any ideas?</p>

lmnt: <p>Do you need to roll every row (as per your post) or some rows (as per your title?)<br>
If every row:</p>
<pre><code class="lang-auto">dim = 0 # all the rows
output = list(map(torch.roll, torch.unbind(mxm, dim), list_of_computed_amounts))
output = torch.stack(output, dim)
</code></pre>
<p>Where <a href="https://pytorch.org/docs/stable/torch.html#torch.unbind" rel="nofollow noopener"><code>torch.unbind</code></a> returns a tuple of all slices along a given dimension. You apply the torch.roll on each of these rows,<br>
<code>computed_amount</code> should be your shifts, equaling the number of rows.<br>
<code>torch.stack</code> just puts it back into one tensor.</p>
<p>Otherwise if only some rows, loop through the indices and computed amounts with torch.roll and set that row to the new rolled values.</p>
<p>Obviously not super efficient, but something that might work.</p>

LeanderK: <p>Sry for being vague. I need to roll every row.</p>

Yochanan_Scharf: <p>Another way to roll every row according to a list.</p>
<p>This answer is based on <a href="https://stackoverflow.com/a/57272600/3957794" rel="noopener nofollow ugc">an answer in <code>numpy</code></a> converted to <code>torch</code>.</p>
<pre><code class="lang-auto">vec = torch(vec)
rolls = torch([list_rolls])
vec[(torch.arange(len(vec))[:,None]+rolls) % len(vec)]


</code></pre>
Calculate the hessian trace for the intermediate inputs

chenqb1989: <p><a href="https://github.com/amirgholami/PyHessian/blob/master/pyhessian/hessian.py" rel="noopener nofollow ugc">This library</a> provides a method to calculate parameters’ Hessian trace using the Hutchinson’s method. In this method, you need to get the <strong>params</strong> and <strong>grads</strong> and then calculate them using torch.autograd.grad(gradsH, params).  The code is as below</p>
<pre><code class="lang-auto">def trace(self, maxIter=100, tol=1e-3):
        """
        compute the trace of hessian using Hutchinson's method
        maxIter: maximum iterations used to compute trace
        tol: the relative tolerance
        """
  
        device = self.device
        trace_vhv = []
        trace = 0.
  
        for i in range(maxIter):
            self.model.zero_grad()
            v = [
                torch.randint_like(p, high=2, device=device)
                for p in self.params
            ]
            # generate Rademacher random variables
            for v_i in v:
                v_i[v_i == 0] = -1
  
            if self.full_dataset:
                _, Hv = self.dataloader_hv_product(v)
            else:
                Hv = hessian_vector_product(self.gradsH, self.params, v)
            trace_vhv.append(group_product(Hv, v).cpu().item())
            if abs(np.mean(trace_vhv) - trace) / (abs(trace) + 1e-6) &lt; tol:
                return trace_vhv
            else:
                trace = np.mean(trace_vhv)
  
        return trace_vhv



def hessian_vector_product(gradsH, params, v):
    """
    compute the hessian vector product of Hv, where
    gradsH is the gradient at the current point,
    params is the corresponding variables,
    v is the vector.
    """
    hv = torch.autograd.grad(gradsH,
                             params,
                             grad_outputs=v,
                             only_inputs=True,
                             retain_graph=True)
    return hv

</code></pre>
<p>But when calculating the Hessian trace of intermediate inputs, I can obtain intermediate <strong>inputs</strong> and corresponding grads <strong>input_grads</strong> by hook,  but I cannot use torch.autograd.grad(input_grads, inputs), for <strong>input_grads</strong> don’t have grad_fn. Is there any way to use torch.autograd.grad for intermediate inputs?</p>
Inconsistent issue during installation

Nothing6_Yin: <p>When I tried to install Pytorch through Anaconda, I ran into the inconsistent issue. Why does it happen and what is the solution?</p>
<pre><code class="lang-auto">(base) C:\Users\Mono&gt;conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
Collecting package metadata (current_repodata.json): done
Solving environment: \
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - pytorch/win-64::torchaudio==2.0.0=py39_cpu
  - pytorch/win-64::torchvision==0.15.0=py39_cpu
done

## Package Plan ##

  environment location: D:\Coding\Anaconda

  added / updated specs:
    - pytorch
    - pytorch-cuda=11.7
    - torchaudio
    - torchvision


The following NEW packages will be INSTALLED:

  pytorch            pytorch/win-64::pytorch-2.0.0-py3.9_cpu_0


Proceed ([y]/n)?
</code></pre>
ValueError: outputs must be a Tensor or an iterable of Tensors

mobassir94: <p>i was trying a multi label classification problem using mixed precision,if i use nn.BCEWithLogitsLoss() then it works fine but when i try to use below custom loss :</p>
<pre><code class="lang-auto">def _check_input_type(x, y, loss):
    if loss in ['MSELoss', 'MAELoss', 'huber']:
        if x.shape[-1] == 1:
            return x.squeeze(), y.float()
        else:
            return x, y.float()
    elif loss in ['CrossEntropy']:
        return x, y.long()
    else:
        return x, y
    
def soft_cross_entropy_loss(logits, targets, weights=1, reduction='none'):
    if len(targets.shape) == 1 or targets.shape[1] == 1:
        onehot_targets = torch.eye(logits.shape[1])[targets].to(logits.device)
    else:
        onehot_targets = targets
    loss = -torch.sum(onehot_targets * F.log_softmax(logits, 1), 1)
    if reduction == 'none':
        return loss
    elif reduction == 'sum':
        return loss.sum()
    elif reduction == 'mean':
        return loss.mean()
    
def ousm(logits, targets, indices=None):
    logits, targets = _check_input_type(logits, targets, 'CrossEntropy')
    bs = logits.shape[0]
    k = 5
    if bs - k &gt; 0:
      losses = soft_cross_entropy_loss(logits, targets)
      if len(losses.shape) == 2:
          losses = losses.mean(1)
      _, idxs = losses.topk(bs-k, largest=False)
      losses = losses.index_select(0, idxs)

      return losses.mean()
</code></pre>
<pre><code class="lang-auto">it gives the following error: 

0%|          | 0/12040 [00:06&lt;?, ?it/s]
Traceback (most recent call last):

  File "C:\Users\Mobassir\ranzr.py", line 660, in &lt;module&gt;
    loss_train = train_func(train_loader,scheduler = scheduler)

  File "C:\Users\Mobassir\ranzr.py", line 288, in train_func
    scaler.scale(loss).backward()

  File "C:\Users\Mobassir\anaconda3\envs\kaggle\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 188, in scale
    return apply_scale(outputs)

  File "C:\Users\Mobassir\anaconda3\envs\kaggle\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 186, in apply_scale
    raise ValueError("outputs must be a Tensor or an iterable of Tensors")

ValueError: outputs must be a Tensor or an iterable of Tensors
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Gxy0221: <p>Did the author solve it? How was it solved?</p>

ptrblck: <p>The error can be raised in an invalid object is passed to <code>scale</code> as seen here via a plain Python <code>int</code> value:</p>
<pre><code class="lang-python">scaler.scale(100).backward()
ValueError: outputs must be a Tensor or an iterable of Tensors
</code></pre>
<p>However, based on the previously posted code it wasn’t clear how the <code>scaler</code> was used.</p>
The problem of calculating second-order derivatice with torch.autograd.grad

BaeHann: <p>It’s obvious that we can calculate high-order derivative with the help of torch.autograd.grad.<br>
But in the following code, the error has occured.</p>
<blockquote>
<p>Blockquote<br>
a = torch.randn(2,3).requires_grad_(True)<br>
b = a*3<br>
b = torch.sum(b)<br>
d = torch.autograd.grad(b,a,retain_graph=True,create_graph=True)[0]<br>
d_2 = torch.autograd.grad(d,a)[0]</p>
</blockquote>
<p>when calculating <em>d_2</em>, the console reads “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”. However, many example codes online have shown that setting create_graph=True will make <em>d</em> a tensor that requires grad. I am really confused.</p>

srishti-git1110: <p>Hey, please refer to <a href="https://discuss.pytorch.org/t/second-order-differentiation-doesnt-work-for-x-1/177459">this post</a> to understand why. Feel free to add follow up questions there.</p>

BaeHann: <p>Thank you very much!</p>

BaeHann: <p>I am really sorry about the comments I leaved in the posts.(My original comment has been modified by the administrator and the part involving personal comments is now invisible.)<br>
My original intention is absolutely to express my gratitude and thankfulness. As a beginner in Pytorch and also in western culture, I was enlightened by the textbook about the habit of compliment and praise which is positive in the western culture. However, I fail to pay attention to the etiquette of avoiding personal comments in the context of a techinal forum. If anyone has felt uncomfortable about my comments, please forgive my rudeness and thoughtlessness.</p>
Benchmarking PyTorch Tensor vs. Numpy Array Train Batch Loading

JakeColor: <p><strong>Question</strong><br>
Why is loading PyTorch tensors (stored as .pt files) from disk faster than loading Numpy arrays (stored as binary .npy files) when using a Dataset class, but <em>slower</em> when using a DataLoader?  Quantified results are below.</p>
<p><strong>Context</strong><br>
My company currently loads train batch files from disk, and asked me to profile the performance of loading data stored in numpy (.npy) vs. pytorch tensor (.pt) file formats</p>
<p><strong>Experiment Details</strong></p>
<ul>
<li>
<p>1000 batch files, 10000 rows x 200 columns, generated randomly and saved to disk in npy, parquet, and pt formats</p>
</li>
<li>
<p><strong>Round 1:</strong> Iterate through (load + convert to PyTorch <code>tensor</code>) each set of 1000 batch files 30 times (<code>num_epochs=30</code>) using a custom Dataset class. Measure the average per-epoch load time</p>
</li>
<li>
<p><strong>Round 2:</strong> Repeat round 1 using a Pytorch dataloader with default settings (<code>num_workers=0</code>)</p>
</li>
</ul>
<p><strong>Results:</strong></p>
<pre><code class="lang-auto">testing file format 'numpy'...
(Dataset) per-Epoch Avg: 7.84 (std: 0.1635)
(Dataloader) per-Epoch Avg: 8.3208 (std: 0.1341)

testing file format 'tensor'...
(Dataset) per-Epoch Avg: 6.7154 (std: 0.0997)
(Dataloader) per-Epoch Avg: 8.5963 (std: 0.0877)
</code></pre>
<p>We thought the significant overhead associated with the tensor/Dataloader combination might be attributable to pinning the loaded tensors to our GPU memory, but that action is disabled by default.</p>
<p><strong>Code</strong></p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/JakeColor/bb403a096f3c198f31c5a55f420ea738" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/JakeColor/bb403a096f3c198f31c5a55f420ea738" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/JakeColor/bb403a096f3c198f31c5a55f420ea738</a></h4>
<h5>benchmark_iteration_speed.py</h5>
<pre><code class="Python">def run_one_epoch(data, conn):
    epoch_start = time.time()
    for batch in data:
        pass
    epoch_end = time.time()

    conn.send(round(epoch_end - epoch_start, 6))


def measure_data_load_speed(data, num_epochs):</code></pre>
This file has been truncated. <a href="https://gist.github.com/JakeColor/bb403a096f3c198f31c5a55f420ea738" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Le_Tr_ng_Giang: <p>When you use Dataloader with default collate_fn, it will convert every numpy array to tensor before return to you. So I think that’s why Dataloader a bit slower</p>
Saliency analysis for RNN predictions

aabati: <p>I am working with a timeseries dataset in which I want to identify if a specific event occured (for each timeseries sequence the event only happens once). I trained a recurrent neural network model (with LSTM layers) that gives me for each input frame (sample) a binary output indicating if the RNN thinks the event happened or not.</p>
<p>Now I want to do a saliency analysis to find out how far in advance the trained model uses the input frames to make a positive prediction.</p>
<p>I came up with the following code, but I am not sure if my implementation is correct:</p>
<pre><code class="lang-auto">data, pred_label = get_data_and_model_prediction()
event_happened_at = np.argmax(pred_label)

# Zero-out gradients
for param in model.parameters():
    param.grad = None

# Start from zero state for each sequence
h = model.get_new_empty_state(requires_grad=True)

# Step 1: Forward propagation
subsequence = data[None, 0:event_happened_at + 1, :]  # Example shape: 1 x 100 x 2 (N x T x output units)
subsequence = torch.from_numpy(subsequence).to(device=device).float()
subsequence.requires_grad = True
prediction, new_state = model(subsequence, h)
probability = F.softmax(prediction, dim=2)

# Step 2: Backward propagation (the output layer has 2 units, where the unit at index 1 indicates if the event happened)
probability[0, :, 1].backward(torch.ones(subsequence.size(1), device=device))
gradients = subsequence.grad.detach().cpu().numpy()
</code></pre>
<p>For me, the tricky part is the first line in step 2. I would like to backpropagate through time to find the influence of each feature from all prior input frames on that particular prediction that the event happened. But I have the feeling that I am not using the backward function correctly.</p>
How to proper save embeddings?

marcomixer99: <p>I have a big-image-encoder and a lot of (more than 200K) high resolution images. I want to save all the image-embeddings. I tried with TensorDataset but when i check if it was the same to load a saved-pre-calculated embedding or forrward the same image i see a little difference in the representation. In particular:<br>
torch.max(saved_emb - same_image_forwarded)<br>
Out[8]: tensor(3.1292e-07)<br>
torch.sum(saved_emb - same_image_forwarded)<br>
Out[9]: tensor(-7.6951e-06)</p>
<p>There is something i’m missing, or i have just a bug in my code and i have to find it.</p>
<p>Thanks in advance and sorry for my english <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I don’t know how you are computing these embeddings, but I guess these are output features of a model.<br>
If so, you are most likely seeing the expected small errors caused by the limited floating point precision and a different order of operations.<br>
If you need more precision you might need to use a wider type, such as <code>float64</code>, for a performance penalty.</p>

marcomixer99: <p>Thanks for the quick response! Yes, these are output features of a model.<br>
I thought that putting the model in evaluation mode and setting a seed would let me get the same exactly features embeddings.<br>
what do you mean with “a different order of operations”?</p>

marcomixer99: <p>I’m computing these emebeddings because the image-encoder is only my freezed backbone. I just want to fine tune a simple adapetr on it. So i would like to store these feature embeddings to train only the adapter, cause i want the backbone freezed</p>

ptrblck: <aside class="quote no-group" data-username="marcomixer99" data-post="3" data-topic="177698">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marcomixer99/48/59347_2.png" class="avatar"> marcomixer99:</div>
<blockquote>
<p>what do you mean with “a different order of operations”?</p>
</blockquote>
</aside>
<p>A different order of operations can cause small, but expected, numerical differences as seen e.g. in this small example using <code>sum</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum()
print((s1 - s2).abs().max())
# tensor(1.5259e-05)
</code></pre>
<p>If you want to get deterministic results you would have to check the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>.<br>
Note that these small errors are usually harmless and in case your model or use case is sensitive to small errors in the ~1e-5 range you might need to use a wider <code>dtype</code> as already mentioned.</p>

marcomixer99: <p>Perfect! Thank you very much!</p>
It is possible to convert a pretrained pytorch model to a Tensorrt channel last model?

Salvador_Pedraza: <p>I have a fine-tuned pretrained classification vision model that I want to execute in a Jetson module.<br>
Once the model is trained in my custom dtadaset, I convert to ONNX and then in the Jetson module I convert to TensorRT engine with trtexec.<br>
I collect the image for inference from a USB camera in jpg format that I convert to a cuda zero copy memory array for later inference. This image is in memory in HWC (channel last) format and is the only format supported by the jpg decoder library.<br>
I used to have Tensorflow-&gt;ONNX-&gt;TensorRT models that I can use directly with the memory images.<br>
However in Pytorch the model is expecting CHW (channel first) images.<br>
I understand from <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="noopener nofollow ugc">this</a> that is more efficient to use channel-last models and I have the input already in channle-last format.<br>
I would like to (some how) convert my pytorch model to TensorRT engine that accept channel-last images so I dont need to add extra preprocessing to my images and also get performance benefits.<br>
Is that possible? How?</p>
<p>I have tried to do:<br>
model = model.to(memory_format=torch.channels_last)</p>
<p>before export to ONNX with torch.onnx.export, but editing the ONNX model still image is having type: <code>float32[batch_size,3,img_wheight,img_width]</code>, so I suppose once I convert to TensorRT engine still will expect the image in channel-first format.</p>
<p>What is the best possible solution?</p>
Running resnet50 with cuda out of memory in colab

S_M: <p>Hi all, when I trained renset50 with cross entropy loss in Google Colab I got this error:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 14.75 GiB total capacity; 13.25 GiB already allocated; 126.81 MiB free; 13.51 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
<p>how can I run resnet50 without any problem on Pytorch using GPU?<br>
any insight is very appreciated.</p>

ptrblck: <p>You might need to reduce the batch size of the training, use mixed-precision training, apply checkpointing etc. to lower the memory requirements of your training.</p>

S_M: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, my batch size is 128 and I used checkpoint but I don’t know about mixed-precision training. Could you please explain more?</p>

ptrblck: <p>Take a look at <a href="https://pytorch.org/docs/stable/notes/amp_examples.html">these docs</a> to learn more about how to use <code>torch.amp</code>.</p>
Optimizer with different learning rates, compatible with GPU

Lukas_Schroth: <p>Hi all,</p>
<p>I am currently facing the following problem:<br>
Given a list of inital values and a list of corresponding learning rates (sizes not know before), I want to pass the values to an optimizer with the corresponding lr specified in the lists. I cannot just create a single tensor and slice it when passing to the optimizer, as it would not be a leaf tensor anymore. Therefore i am working with a list of scalar torch.tensor. Then i use torch.stack to get a single tensor with which i can work. However I really need the code to be runnably on GPU efficienty, and I think the usage of lists leads to data tranfer between GPU and CPU, which makes the code slow (not sure, not an expert). also the usage of torch.stack in every iteration seems quite expensive. Any ideas how i can realize this to be fast on GPU?</p>
<p>Here are some code snippets:<br>
<code><br>
self.lagrange_lambda = [torch.tensor(lagrange_lambda_init_list[i], requires_grad =True, device= self.device).float() for i in range(self.num_constr)]<br>
self.lambda_optimizer = optim.Adam([{‘params’: [torch.tensor(self.lagrange_lambda[i])], ‘lr’:learning_rate_lambda_list[i]} for i in range(self.num_constr)])<br>
</code></p>
<p>and then in the loop:<br>
<code><br>
lagrange_lambda = torch.stack(self.lagrange_lambda)<br>
</code></p>
<p>Thanks for any help!</p>

ptrblck: <aside class="quote no-group" data-username="Lukas_Schroth" data-post="1" data-topic="177655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukas_schroth/48/59548_2.png" class="avatar"> Lukas_Schroth:</div>
<blockquote>
<p>and I think the usage of lists leads to data tranfer between GPU and CPU, which makes the code slow (not sure, not an expert)</p>
</blockquote>
</aside>
<p>That’s not the case since the <code>list</code> would only store the tensor with its metadata while the actual internal data of the tensor would still be on the GPU.</p>
<p>I also don’t fully understand your use case as it seems you would initialize the optimizer only once, so even a slow loop shouldn’t matter compared to the overall training duration.</p>
<p>This line of code:</p>
<pre data-code-wrap="python"><code class="lang-python">self.lambda_optimizer = optim.Adam([{‘params’: [torch.tensor(self.lagrange_lambda[i])], ‘lr’:learning_rate_lambda_list[i]} for i in range(self.num_constr)])
</code></pre>
<p>also creates new tensors, so <code>self.lagrange_lambda</code> will never be optimized.</p>