akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Les gammes musicales

Toumic: <p>Je travaille sur un code Python qui a pour résultat, un jeu de gammes chromatiques(12 1/2 tons), heptatoniques(7 1/2 tons), et tétracordiques(4 1/2 tons), basé sur une seule octave (de 12 1/2 tons).</p>
<p>Comme je débute ici avec PyTorch, j’aimerais savoir quel tutoriel aborder.<br>
Afin de développer un calculateur intelligent gammologique.</p>
<p>Si vous ne comprenez pas les notes, en gros il s’agit d’un espace de 12 1/2 tons. Sur lequel viennent s’accorder une grande séries de gammes variées.</p>
<p>EDIT by <code>ptrblck</code>:<br>
From Google translate:<br>
I’m working on Python code that results in a set of chromatic (12 1/2 tones), heptatonic (7 1/2 tones), and tetrachord (4 1/2 tones) scales, based on a single octave (from 12 1/2 tones).</p>
<p>As I am new here with PyTorch, I would like to know which tutorial to approach.<br>
In order to develop an intelligent gammological calculator.</p>
<p>If you don’t understand the notes, basically it’s a 12 1/2 tone space. On which come to agree a large series of varied scales.</p>
Checkpointing is not compatible with .grad()

ksmdanl: <p>I have two segmentation networks and need to obtain the gradients of the output with respect to the input. It works on one model. On the second model, however, it throws a checkpointing error, which is interesting since there’s no checkpoint performed.</p>
<p><code>RuntimeError: Checkpointing is not compatible with .grad() or when an </code>inputs<code>parameter is passed to .backward(). Please use .backward() and do not pass its</code>inputs<code> argument.</code></p>
<p>I added a minimal code snippet to make the case more clearer. <code>net_a</code> and <code>net_b</code> are the segmentation networks.</p>
<pre><code class="lang-auto">net_a = NetA()
#net_b = NetB()
net_a.eval()

with torch.no_grad():
    x = torch.rand(1, 3, 768, 768)
    y = torch.LongTensor(1,768,768).random_(0, 19)

loss = nn.CrossEntropyLoss()
x = x.clone().detach()
y = y.clone().detach()

x_ = x.clone().detach()
x_.requires_grad = True
y_ = net_a(x_)
y_ = y_['logits']

loss = loss(y_, y)
print(loss.requires_grad, x_.requires_grad)
grad_ = torch.autograd.grad(loss, x_, retain_graph=False, create_graph=False)[0]
</code></pre>
<p>What might cause this behaviour? Both nets are structured very similarly hence the confusion up until now. Looking forward for a discussion!</p>

ptrblck: <p>Could you make the code snippet executable by adding the missing pieces so that we could reproduce and bug it, please?</p>
Quantization config

AndreiMoraru123: <p>Hi. I am having a hard time wrapping my head around quantizing models. To the point, I have a basic ResNet model that I want to optimize:</p>
<pre><code class="lang-auto">encoder = EncoderCNN()
encoder.load_state_dict(torch.load(os.path.join('models', encoder_file)))

encoder.eval()
encoder.to(device)

dummy_input = torch.randn(1, 3, 480, 480, device='cuda')
with torch.jit.optimized_execution(True):
    encoder = torch.jit.trace(encoder, dummy_input)
    encoder.save("models/encoder.pt")

encoder = torch.jit.load(os.path.join('models', 'encoder.pt'), map_location=torch.device('cuda'))
</code></pre>
<p>And this is how I figured I would quantize it:</p>
<pre><code class="lang-auto">if use_fbgemm:
    quantization_config = torch.quantization.get_default_qconfig('fbgemm')
    torch.backends.quantized.engine = 'fbgemm'

else:
    quantization_config = torch.quantization.get_default_qconfig('qnnpack')
    torch.backends.quantized.engine = 'qnnpack'

quantization_config.quant_min = 0.0
quantization_config.quant_max = 1.0
encoder.qconfig = quantization_config

torch.quantization.prepare(encoder, inplace=True)
torch.quantization.convert(encoder, inplace=True)   # This line gets a warning
</code></pre>
<p>But the last line throws the warning</p>
<blockquote>
<p>UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch."</p>
</blockquote>
<p>This is located in <code>torch\ao\quantization\observer.py:216</code> and the following lines don’t help:</p>
<pre><code class="lang-auto">quantization_config.quant_min = 0.0
quantization_config.quant_max = 1.0
</code></pre>
<p>The <a href="https://pytorch.org/docs/stable/generated/torch.quantization.convert.html" rel="noopener nofollow ugc">doc</a> on this was not very clear to me, and I tried to change it to:</p>
<pre><code class="lang-auto">torch.quantization.prepare(encoder, inplace=True)
torch.quantization.convert(encoder, inplace=True,
                           convert_custom_config_dict={'_custom_module_class':
                                                           {'EncoderCNN': encoder}  # I want this as my Custom Module?
                                                       }
                           )
</code></pre>
<p>But the warning persists. And perhaps more importantly, I am also not making my model any faster judging by FPS, so I suppose I have set this up wrong to begin with.</p>
<p>If you could point me to any docs or examples that go through something like this, I would be thankful.</p>
<p>I know there already are <a href="https://pytorch.org/vision/main/models/resnet_quant.html" rel="noopener nofollow ugc">quantized ResNet models</a>  available, but it’s important for me that I can apply this to a custom network with modified layers.</p>
<p>Thank you</p>
TypeError: expected np.ndarray (got NoneType)

Resalat007: <p>I am using PPO algorithm for my project. I am getting this error. I have used the following line of code:<br>
state = torch.from_numpy(state).float().unsqueeze(0)</p>
<p>I also changed to state = torch.tensor(state).float().unsqueeze(0) then another error arises. It called :<br>
RuntimeError: Could not infer dtype of NoneType</p>

ptrblck: <p><code>state</code> seems to be <code>None</code> as seen here:</p>
<pre><code class="lang-python">state = None
state = torch.from_numpy(state).float().unsqueeze(0)
# TypeError: expected np.ndarray (got NoneType)
state = torch.tensor(state).float().unsqueeze(0) 
# RuntimeError: Could not infer dtype of NoneType

state = np.random.randn(10)
state = torch.from_numpy(state).float().unsqueeze(0) # works
</code></pre>
<p>so make sure it’s a valid numpy array and the code should work.</p>

Resalat007: <p>Thank you so much for your answer!</p>
Wrong matmul benchmark on GPU

Ronan_Gautier: <p>I wanted to run a simple matmul benchmark on GPU. Following the <a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html" rel="noopener nofollow ugc">PyTorch Benchmark</a> tutorial, I have written the following code:</p>
<pre><code class="lang-python">import torch
import torch.utils.benchmark as benchmark

# Prepare matrices
N = 10000
A = torch.randn(N, N)
B = torch.randn(N, N)

# Send to device
device = torch.device("cuda")
A = A.to(device)
B = B.to(device)

# Benchmark
n_threads = 1
t = benchmark.Timer(
    stmt="A @ B",
    globals={"A": A, "B": B},
    num_threads=n_threads,
)
m = t.blocked_autorange(min_run_time=3)

# Print results
print(m)
print(f"Mean:  {m.mean * 1e3:6.2f} ms"
      + f" | First: {m._sorted_times[0] *1e3:6.2f} ms"
      + f" | Median: {m.median *1e3:6.2f} ms"
      + f" | Last: {m._sorted_times[-1] *1e3:6.2f} ms.")
</code></pre>
<p>which prints out</p>
<pre><code class="lang-auto">&lt;torch.utils.benchmark.utils.common.Measurement object at 0x7ff2e2546250&gt;
A @ B
  Median: 25.13 us
  IQR:    0.50 us (24.92 to 25.42)
  541 measurements, 1 runs per measurement, 1 thread
Mean:    5.56 ms | First:   0.02 ms | Median:   0.03 ms | Last:  99.97 ms.
</code></pre>
<p>As you can see, there’s a huge gap in runtime. The first few hundred runs take only ~20us, while the rest take ~100ms. I have a hard time believing that a matmul over such big matrices can be run in ~20us. What is happening here? Have I done something wrong in the benchmark?</p>
Batched addition over contiguous indices

Ronan_Gautier: <p>I am trying to multiply a matrix and a vector together, but only over a certain range of indices that depends on the matrix row. I can write it with a for loop, but this is not efficient. Is there a way to vectorize/broadcast this operation in an efficient gpu-readable manner?</p>
<p>Here is the example and what I am trying to achieve:</p>
<pre><code class="lang-python">N = 5
D = torch.stack([torch.cat((torch.arange(1, N), torch.zeros(1))), 
                 torch.cat((torch.zeros(2), torch.arange(2, N)))])
offset = torch.tensor([-1, 2])
i0 = torch.tensor([0, 2])
i1 = torch.tensor([N-1, N])
X = torch.arange(N).to(torch.float)

# Compute the element-wise multiplication over each row of D
Y = torch.zeros_like(X)
for k in range(2):
    Y[(i0[k]-offset[k]):(i1[k]-offset[k])] += D[k, i0[k]:i1[k]] * X[i0[k]:i1[k]]

# Expected results
print(D)
print(X)
print(Y)
# D = tensor([[1., 2., 3., 4., 0.],
#             [0., 0., 2., 3., 4.]])
# X  = tensor([0., 1., 2., 3., 4.])
# Y  = tensor([4., 9., 18., 6., 12.])
</code></pre>
<p>Essentially, I just want to get rid of the for loop.</p>
<p>[Edit: added the <code>offset</code> tensor, otherwise the problem is trivial]</p>

Ronan_Gautier: <p>I have managed to simplify the expression, with the following:</p>
<pre><code class="lang-python">Y = torch.zeros_like(X)
index = torch.tensor([[1,2,3,4,0],[3,4,0,1,2]])
for k in range(2):
    Y.index_add_(0, index[k, :], D[k, :] * X)
print(Y)
# Y  = tensor([4., 9., 18., 6., 12.])
</code></pre>
<p>Not sure how to get rid of the for loop however.</p>

ptrblck: <p>You can avoid the loop by flattening the index and value tensors:</p>
<pre><code class="lang-python">out = torch.zeros_like(X)
value = D * X
out.index_add_(0, index.view(-1), value.view(-1))
print(out)
# tensor([ 4.,  9., 18.,  6., 12.])
</code></pre>

Ronan_Gautier: <p>Thanks for your answer ! This is indeed an elegent way of doing this.</p>
<p>However, benchmarking for larger tensors, this achieves worse speeds on GPU than my initial for loop without <code>index_add_</code>. Typically, I have <code>D.size() = (M, N)</code> and <code>X.size() = (N)</code> with <code>N = 100_000_000</code> and <code>M = 10</code>. I’m guessing that this is due to the fact that <code>index_add_</code> does not use the fact that I’m summing over <strong>contiguous</strong> indices. Is there an alternative batched version that would make use of this contiguous-ness in the sum?</p>
<p>Thank you for your help.</p>
Weight freezing

pynoob1: <p>Hello everyone,</p>
<p>I am new to pytorch and have a query about weight freezing. Suppose I have one linear layer, I want to initialize my weight matrix as a lower triangular matrix. Then during training I would like only the lower triangular part to get updated and the upper triangular to remain zero. I have created a lower triangular mask of the same size as the matrix and have tried using detach and requires_grad = False for the indices where the mask is 0. Like so:<br>
model.weight.data[mask == 0].requires_grad = False</p>
<p>model.weight.data[mask == 0] = model.weight.data[mask == 0].detach()</p>
<p>However both these instances freezes the entire layer and it does not update. I have also tried manually providing elements at these indices to the optimizer but that also doesn’t seem to work.</p>
<p>Any help would be greatly appreciated. Thanks in advance!</p>

InnovArul: <p>You could directly multiply the mask with the weight.<br>
Since the multiplication with mask acts as a multiplication with a constant, it would be factored into gradient as well and the gradient will be 0 for the positions where <code>mask = 0</code>.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class MyClass(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.weight = nn.Parameter(torch.randn(5,5))
        self.mask = torch.triu(torch.ones_like(self.weight))

    def forward(self, x):
        out = (self.weight * self.mask).matmul(x)
        return out

if __name__ == '__main__':
    data = torch.randn(5,4)
    model = MyClass()
    
    out = model(data)
    out.sum().backward()
    print(model.weight.grad)
</code></pre>

pynoob1: <p>This is a really great and simple solution, thank you very much. I am still not yet able to understand why this works as I am new to pytorch, specifically why you call out.sum() and then backward() on it.<br>
I presume if I do use this model in a training loop with an optimizer and call optimizer.step() it would only change the weights of the upper triangular part as the mask created is an upper triangular matrix.<br>
Just out of curiosity would providing only a slice of the weight matrix to the optimizer only update that slice? Thanks again.</p>
CUDNN_STATUS_NOT_INITIALIZED when using benchmark.timeit

XuRuihan: <p>I use <code>Benchmark.Timer</code> with the official docker<br>
<a href="https://hub.docker.com/layers/pytorch/pytorch/1.13.1-cuda11.6-cudnn8-runtime/images/sha256-1e26efd426b0fecbfe7cf3d3ae5003fada6ac5a76eddc1e042857f5d049605ee?context=explore" rel="noopener nofollow ugc">Image Layer Details - pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime | Docker Hub</a><br>
and I ran into CUDNN_STATUS_NOT_INITIALIZED error. I found that the nvidia-smi shows cuda version 10.2 in the docker, so I wonder if this problem was caused by a version mismatch between cuda driver and runtime?</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>
Why does the PyTorch Optimiser takes a iterable parameter instead of a single high-dimensional tensor?

OrangeMatcha: <p>I am new to PyTorch and still learning it. While I am trying to use the PyTorch optimiser, I noticed optimiser only accept iterable parameters.</p>
<p>I am wondering what is the reason behind to design PyTorch optimiser in this way? Why not just accept a single tensor with one added dimension?</p>
<hr>
<p>Here is my weight:</p>
<pre><code class="lang-auto">w = torch.tensor(0, dtype = torch.float32, requires_grad = True)
optimiser = torch.optim.SGD(w,lr = learning_rate)
</code></pre>
<p>It raise the below error:<br>
<code>TypeError: params argument given to the optimizer should be an iterable of Tensors or dicts, but got torch.FloatTensor</code></p>
<p>When I pass the <code>w</code> as a list into the optimiser, everything works great.</p>

ptrblck: <p>My guess would be that <code>iterable</code>s are expected to be compatible with the return value of <code>model.parameters()</code> in the initial design.<br>
I don’t know if there are any other technical reasons.</p>
"What is torch.nn really?" removal of activation function

samuel_xu: <p>In the section where we replace the activation function with the built-in cross entropy function, why do we remove the activation function from the calculation of the prediction accuracy? I had thought that the activation function was still needed for calculating the predictions of the neural network. The screenshot is included below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/692cc7effcf7479abe4821d56402ec4a923fce06.png" data-download-href="https://discuss.pytorch.org/uploads/default/692cc7effcf7479abe4821d56402ec4a923fce06" title="screenshot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/692cc7effcf7479abe4821d56402ec4a923fce06_2_690x381.png" alt="screenshot" data-base62-sha1="f0q4MZ3RBtm08g0LmTDnammhWVU" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/692cc7effcf7479abe4821d56402ec4a923fce06_2_690x381.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/692cc7effcf7479abe4821d56402ec4a923fce06_2_1035x571.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/692cc7effcf7479abe4821d56402ec4a923fce06_2_1380x762.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">screenshot</span><span class="informations">1506×832 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Peter_Vennerstrom: <p>F.<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">cross_entropy</a> is a function containing both the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc">softmax activation</a> and the <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">negative log likelihood loss</a>.</p>
<p>The tutorial is pointing out the log_softmax function doesn’t need to be called in the model, because it will be called in the newly added F.cross_entropy function instead.</p>
How are fp 32 weights converted to fp16 post training?

Basu_Jindal: <p>Can the weights of a model trained in full precision be converted to half precision post-training, with or without loss of accuracy? If so, what is the maths involved behind the conversion?<br>
Thanks</p>

jerryzh168: <p>you can do <code>model.half()</code>, and it will be lossy I think. it will just convert the weights, and the operators will dispatch to a fp16 implementation as well, you’ll also need to feed the model a fp16 input.</p>

Basu_Jindal: <p>Thank you for the reply.</p>
Is it good practice to assign module.weight.grad.data?

arunppsg: <p>Here is a small sample of code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

def func(m: nn.Module):
    p_grad = m.weight.grad.data
    return None
</code></pre>
<p>I am wondering, is it good to assign <code>m.weight.grad.data</code> to a variable and use it later? The reason is <code>m.weight.grad</code> can also be <code>None</code>. In pytorch 1.13.0, mypy throws <code>error: Item "None" of "Union[Tensor, None, Module]" has no attribute "data"</code>.</p>
<p>I use the <code>data</code> attribute to implement a custom optimizer.</p>

ptrblck: <p>No, it’s not because the usage of the <code>.data</code> attribute is deprecated and you shouldn’t use it and (as you can already see) the <code>.grad</code> attribute is not necessarily defined.<br>
Add checks to the <code>.grad</code> usage and make sure it was created before trying to assign or use it.</p>
Why I can't change value of tensor with this way?

Hyeonuk_Woo: <p>Hi I have a question about change the tensor value.</p>
<p>If I change the value of tensor this way, works well.</p>
<pre><code class="lang-auto">a=torch.zeros(5,3,2)
b=torch.Tensor([True,False,True,False,True]).bool()
a[b]=2 #shape 3,3,2
print(a)
###tensor([[[2., 2.],
         [2., 2.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[2., 2.],
         [2., 2.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[2., 2.],
         [2., 2.],
         [2., 2.]]])
</code></pre>
<p>However,why I cant change the value of tensor in this way?</p>
<pre><code class="lang-auto">a=torch.zeros(5,3,2)
b=torch.Tensor([True,False,True,False,True]).bool()
a[b][:,2,:]=2
print(a)
###
tensor([[[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]]])
</code></pre>
<p>My desired output is</p>
<pre><code class="lang-auto">Tensor([[[0., 0.],
         [0., 0.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [2., 2.]]])
</code></pre>
<p>Thanks in advance.</p>

Taejune: <p>Hi, if you do <code>a[b]</code>,  the <code>b</code> will only indicate <code>a</code>’s first dimension.<br>
So if you want to change the last rows of batch 0, 2, 4 of <code>a</code>, what you want to do is:<br>
<code>a[b, 2, :] = 2</code>.</p>
<p>On the other hand, if you want to make it to your desired output,<br>
you don’t have to use <code>b</code> and just have to do <code>a[:, 2, :] = 2</code>.</p>

Hyeonuk_Woo: <p>Sorry to miswrite desired output. I want.</p>
<pre><code class="lang-auto">Tensor([[[0., 0.],
         [0., 0.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.],
         [2., 2.]],

        [[0., 0.],
         [0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.],
         [2., 2.]]])
</code></pre>
<p>Thanks for kind explanation!</p>
Difference between torch.cuda.synchronize() and dist.barrier()

Taejune: <p>Hi, I’m wondering what is the main difference between <code>torch.cuda.synchronize()</code> and <code>dist.barrier()</code>.<br>
I know that the former prevents CPU thread from proceeding until the previous works are done,<br>
and the latter makes processes wait until every process reaches <code>dist.barrier()</code>.</p>
<p>So I think those two have the same purpose… what’s the difference?</p>
<p>If I was wrong, please feel free to convince me that I was wrong.</p>
<p>Thanks!</p>

ptrblck: <p><code>torch.cuda.synchronize()</code> synchronizes the current device and waits until all GPU work is finished thus blocking the host from advancing.<br>
<code>dist.barrier</code> is used in a distributed setup and synchronizes all processes until the group enters this function. Even if all GPU work is already done in one process it would still wait for all other processes until they reach the barrier before advancing.</p>

Taejune: <p>Thanks for the quick and clear answer!<br>
So then if I use <code>dist.barrier</code> for the distributed setup, does it also act like <code>torch.cuda.synchronize()</code> ?<br>
What is the right usage of each function?</p>
<p>Thanks a lot.</p>
Optimizer step failing with opacus

hungryc8rpillar: <p>“Per sample gradient is not initialized. Not updated in backward pass?”</p>
<p>Hi I get the above error message at optimizer.step() when training with opacus. I have already read other forum entries with the same or similar error messages. Usually the problem seemed to be that the models parameters were reinitialized (e.g. with ModuleValidator.fix(net)) after the creation of the optimizer, leading to a mismatch of net and optimizers parameters that causes the problem.<br>
To my understanding, I do not have such a missmatch in my code and I am already trying for days to figure out the problem. Would really appreciate if someone has an idea about what causes this problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> thanks</p>
<p>Here is the code of my train function:</p>
<p>def train(net, trainloader, epochs: int, verbose=True):</p>
<pre><code>net = ModuleValidator.fix(net)

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr = 0.001)

privacy_engine = opacus.PrivacyEngine()

net, optimizer, trainloader = privacy_engine.make_private(
    module=net,
    optimizer=optimizer,
    data_loader=trainloader,
    noise_multiplier=1.1,
    max_grad_norm=1.1,
)
for epoch in range(epochs):
    correct, total, epoch_loss = 0, 0, 0.0
    
    with BatchMemoryManager(
    data_loader=trainloader, 
    max_physical_batch_size=8, 
    optimizer=optimizer
    ) as memory_safe_data_loader:
            
        for images, labels in memory_safe_data_loader:
            images, labels = images.to(device), labels.to(device)
            optimizer.zero_grad()
            net.train()
            outputs = net(images)
            loss = criterion(net(images), labels)
            loss.backward()
            optimizer.step()
            epoch_loss += loss
            total += labels.size(0)
            correct += (torch.max(outputs.data, 1)[1] == labels).sum().item()
        epoch_loss /= len(memory_safe_data_loader.dataset)
        epoch_acc = correct / total
        if verbose:
            print(f"epoch {epoch+1}: train loss {epoch_loss}, accuracy {epoch_acc}")
</code></pre>

ffuuugor: <p>Hey - I’ve noticed you run forward twice:</p>
<pre><code class="lang-auto">outputs = net(images)
loss = criterion(net(images), labels)
</code></pre>
<p>with one node being detached from the loss.</p>
<p>If you fix that to one forward pass only, it should work fine</p>
<pre><code class="lang-auto">outputs = net(images)
loss = criterion(outputs, labels)
</code></pre>

hungryc8rpillar: <p>Great observation! Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to pad a set of tensors to a specific height

peony: <p>I am doing action recognition with mediapipe keypoints. These are the shapes of some of my tensors:</p>
<blockquote>
<p>torch.Size([3, 3, 75]) torch.Size([3, 6, 75]) torch.Size([3, 10, 75]) torch.Size([3, 11, 75]) torch.Size([3, 9, 75]) torch.Size([3, 4, 75]) torch.Size([3, 21, 75])</p>
</blockquote>
<p>The height of each tensor varies as they refer to the number of frames for each sample.</p>
<p>I have decided that I want to consider 8 frames for each sample. I understand I have to do padding and truncate (for heights above 8), but somehow just doing the padding worked, or so it seems. I wish to understand how my code worked.</p>
<pre><code class="lang-auto">if height &lt; 8:
            source_pad = F.pad(tensor1, pad=(0, 0, 0, 8 - height))
        else:
            source_pad = F.pad(tensor1, pad=(0,0, 0, 8 - height))
</code></pre>

ptrblck: <p>Negative padding seems to slice the tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(3, 3, 75)
height = x.size(1)
source_pad = F.pad(x, pad=(0, 0, 0, 8 - height))
print(source_pad.shape)
# torch.Size([3, 8, 75])

x = torch.randn(3, 9, 75)
height = x.size(1)
source_pad = F.pad(x, pad=(0, 0, 0, 8 - height))
print(source_pad.shape)
# torch.Size([3, 8, 75])
print((x[:, :8, :] - source_pad).abs().sum())
# tensor(0.)
</code></pre>

peony: <p>Does slicing have the same effect as truncating? Thank you for explaining.</p>

ptrblck: <p>I’m unsure if our definition of “truncating” a tensor is the same, but “slicing” in this case you mean you are indexing the dimension from <code>0</code> to <code>7</code> (including) as seen in my comparison between the original tensor and the output:</p>
<pre><code class="lang-python">print((x[:, :8, :] - source_pad).abs().sum())
</code></pre>
Usage issue with torchvision built-in MViT model

clai: <p>I am having trouble with the video MViT models built in torchvision 0.14. <a href="https://pytorch.org/vision/stable/models/video_mvit.html?highlight=mvit" rel="noopener nofollow ugc">Video MViT — Torchvision main documentation (pytorch.org)</a></p>
<p>Here is my code:</p>
<pre><code class="lang-auto">import torch
from torchvision.models.video import mvit_v2_s, MViT_V2_S_Weights

model = mvit_v2_s(weights="DEFAULT")
model.eval()
transforms = MViT_V2_S_Weights.KINETICS400_V1.transforms()
input = transforms(torch.rand(1, 32, 3, 800, 600))
output = model(input)
</code></pre>
<p>But I got errors as follows:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "c:\Users\clai\projects\DL_VA_Prediction\src\lib\models\mvit_model.py", line 8, in &lt;module&gt;
    output = model(input)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torchvision\models\video\mvit.py", line 558, in forward
    x, thw = block(x, thw)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torchvision\models\video\mvit.py", line 384, in forward
    x_attn, thw_new = self.attn(x_norm1, thw)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torchvision\models\video\mvit.py", line 293, in forward
    k, k_thw = self.pool_k(k, thw)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\clai\miniconda3\envs\torch_HCM_py39\lib\site-packages\torchvision\models\video\mvit.py", line 89, in forward
    x = x.reshape((B * N, C) + thw).contiguous()
RuntimeError: shape '[1, 96, 8, 56, 56]' is invalid for input of size 4816896
</code></pre>
<p>Any idea how I can make it work?</p>
<p>Thanks</p>

Peter_Vennerstrom: <p>The weights are loaded into a model initialized with <a href="https://github.com/pytorch/vision/blob/78ffda7eb952571df728e2ae49c2aca788596138/torchvision/models/video/mvit.py#L886" rel="noopener nofollow ugc">temporal length set to 16</a>.  Looks like the model does not handle dynamic temporal length input.  It could be retrained with a longer temporal length if the dataset supports it.</p>
Mario RL example with newer versions of libraries

rohitpid: <p>I’m working on the mario RL tutorial. I’ve found that it’s a bit out of date with the latest versions of the libraries. I’ve worked around a few issues like updating to the new env API where done is split into truncated or terminated.</p>
<pre><code class="lang-auto">next_state, reward, truncated, terminated, info = env.step(action)
done = truncated or terminated
</code></pre>
<p>I’m looking at this part where LazyFrames are converted to pytorch tensors and am getting errors.</p>
<pre><code class="lang-auto">state = torch.FloatTensor(state).cuda() if self.use_cuda else torch.FloatTensor(state)
</code></pre>
<pre><code class="lang-auto">expected sequence of length 4 at dim 1 (got 0)
  File "/home/user/stats/mario_rl/MadMario/mario_pytorch_example.py", line 342, in cache
    state = torch.FloatTensor(state).cuda() if self.use_cuda else torch.FloatTensor(state)
  File "/home/user/stats/mario_rl/MadMario/mario_pytorch_example.py", line 726, in &lt;module&gt;
    mario.cache(state, next_state, action, reward, done)
</code></pre>
<p>Can anyone suggest how to proceed with the conversion? I’ve tried suggestions to cast state using <code>np.ndarray(state)</code> but that didn’t work. Any suggestions? Thanks in advance.</p>

rohitpid: <p>Seems like the states that are returned by the environment have changed. They are now tuples with an extra info object.</p>
<pre><code class="lang-auto">state = self.observation(None), info
</code></pre>
<p>I was able to change the code to use just the observation part of the state to get things running. I will have to read more about what the info object contains. Seems like things are running. After I clean things up, is there any interest in an MR with the changes for the latest versions of things. I have also set things up in a poetry environment. The conda environment from the linked github was not working for me as I think some of the packages are missing/out of date.</p>

rohitpid: <p>PR is here: <a href="https://github.com/yfeng997/MadMario/pull/16" class="inline-onebox" rel="noopener nofollow ugc">fix: update versions and readme by rohitpid · Pull Request #16 · yfeng997/MadMario · GitHub</a></p>
Why am I getting a value greater than 1 as dice score

ays: <p>I have two segmented images, and I’ve computed the dice score using the formula below, however, I keep getting values greater than 1 (like 11.8, 12.8) as a dice score. is there a reason why? or is my approach for computing the dice score wrong?</p>
<pre><code class="lang-auto">
def dice(X, Y): 
        intersection = (X * Y).sum()
        union = X.sum() + Y.sum()
        
        return ((2. * intersection) / (union))

</code></pre>

ptrblck: <p>Did you verify that <code>X</code> and <code>Y</code> are probabilities in the range <code>[0, 1</code>]? If not, this could explain the unexpected values.</p>

ays: <p>Yes, X and Y are normalised between 0 and 1 when the data is loaded. I checked the min and Max of X and Y after training and I see that its 0 and 24 respectively. I think this is due to some operations that I added during the training process. is It wise to <strong>renormalise</strong> my image to range between 0 and 1 (after training) before passing it to the dice function to calculate the score?</p>

ptrblck: <p>It depends what these values now represent and which of tour operations applied during the training process change the range of these tensors.<br>
E.g. if <code>X</code> represents raw logits (in which case any value in <code>[-Inf, +Inf]</code> would be valid) you could create probabilities using <code>sigmoid</code> or <code>softmax</code> (it would depend on the actual use case).<br>
The target values should not be changed during training as these represent the ground truth values.</p>

ays: <p>Alright, thanks let me check out the output from each operation</p>
Building a custom operator using two pytorch ops

thyeros: <p>I have the following code in my nn.Module.</p>
<p>x = torch.cdist(a,b)<br>
y = torch.softmax(x)</p>
<p>These two are differentiable, but due to the size of x, x/y need a lot of GPU memory, causing a OOM during backprop. Since I’m not sure if the checkpoint can help this (getting NaN with ddp) and x/y are very sparse (many near zero values), I was thinking of a custom operator in the following fashion, but cannot find a doc on how to call backward of existing pytorch constructs in a custom operator. Can anyone point me out some examples on how to complete this?</p>
<pre><code class="lang-auto">class MyFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx, a, b):        
        x = torch.cdist(a,b)
        ctx.save_for_backward(a,b, sparse(x))
        return torch.softmax(x)

    @staticmethod
    def backward(ctx, g):
     ??
</code></pre>

soulitzer: <p>Have you looked into saved tensor hooks? Saved tensor hooks let you interpose behavior when the tensor is “packed” to be saved for backward and when it is “unpacked” to be used during the backward pass. Since GPU memory is your  issue, you can consider using the context manager <code>torch.autograd.graph.save_on_cpu</code> (which utilizes saved tensor hooks in its implementation). In this context all intermediary results are moved to cpu, and copied back to the original device when it is used during the backward pass.</p>
<p><a href="https://pytorch.org/docs/stable/autograd.html?highlight=saved%20tensor%20hooks#torch.autograd.graph.saved_tensors_hook" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html?highlight=saved%20tensor%20hooks#torch.autograd.graph.saved_tensors_hook</a></p>
<p>To directly answer your question though, I believe some of the backwards are available in the <code>torch.ops.aten</code> namespace, e.g., <code>torch.ops.aten._softmax_backward_data</code>, <code>torch.ops.aten._cdist_backward</code>. The full list is available in the Pytorch source at <code>aten/src/ATen/native/native_functions.yaml</code>. This is NOT intended to be public api though, and is subject to change.</p>

thyeros: <p>Actually, the 1st option looks better and more sustainable than the 2nd. I will try it out and report back here for future references, thanks!</p>

thyeros: <p>I gave a shot to the 1st option (save_on_cpu), and it works but with huge throughput penalty from cpu-gpu tensor copy (on pcie). I will look into the saved tensor hook.</p>

thyeros: <p>I eventually used autograph.graph to implement custom save_tensor_hook with some sort of grad-checkpoint based on the application specifics. It worked very well now, and thanks for the tip on the autograd.graph!</p>

gosiqueira: <p>Waiting for <a class="mention" href="/u/ptrblck">@ptrblck</a> response…</p>

Tong_Zhou: <p>Great you found a solution. Just curious, what is <code>sparse(x)</code> here? Converting <code>x</code> to a sparse matrix? Why is <code>x</code> sparse though?</p>
Is there any good documentation about MPS?

TachikakaMin: <p>Apple’s documentation is too bad(<a href="https://developer.apple.com/documentation/metalperformanceshaders?language=objc" class="inline-onebox" rel="noopener nofollow ugc">Apple Developer Documentation</a>)</p>
<p>and it only has API definition without any description.</p>
<p>It’s hard for me to use these APIs in PyTorch MPS backend.</p>
[Solved] How to get DataLaoder to batch over the second dimension and not the first

John_Palmer: <p>I’m using sequence data for input into a LSTMCell. As per the LSTMCell documentation the input shape should be (time_index, batch, n_features). I reshaped my training data into a similar shape of (time_index, n_sequences_in_training_set, n_features) for the input data X_train and (time_index, n_sequences_in_training_set, 1) for the target data y_train.</p>
<p>When I iterate over a data loader created with a data it batches over the time_index (1st) dimension which is what I would expect. However, in this case I need to batch over the second dimension and I can’t figure how to do that. I have read that this can be done using a custom collate_fn, but when I looked at this it appeared that the data is already passed batched along the first dimension into the collate_fn (is this correct?). Is it it possible to return batched data over the second dimension?</p>

nivek: <p>I believe the input to <code>collate_fn</code> is a list of Tensors. You should be able to have a custom collate function to do what you would like.</p>

John_Palmer: <p>Thanks Kevin, what you described helped me solve the problem I was seeing. The issue was actually in the CustomDataset class I was using and specifically in the <span class="bbcode-i"><strong>getitem</strong></span> method as this was passing the list of Tensors indexed over the first dimension into the collate_fn (Shows why I should have posted a code snippe!t). Below is a code snipped showing the solution I came to.</p>
<p>I wonder if it is possible to stack the Tensors more efficiently in the for loop in the collate_fn, but I couldn’t figure a way of doing this without unpacking the X and y tensors first using the loop.</p>
<pre><code class="lang-auto">
from torch.utils.data import Dataset, DataLoader
import numpy as np
import torch

class CustomDataSet(Dataset):

    def __init__(self, X, y):
        self.X = X
        self.y = y

    def __len__(self):
        return self.X.shape[1]

    def __getitem__(self, idx):

        X = self.X[:, idx, :]
        y = self.y[:, idx, :]
        return X, y

def my_collate(data_in):

    X_data = []
    y_data = []

    for seq in data_in:
        X_data.append(seq[0])
        y_data.append(seq[1])

    X_data = torch.stack(X_data, dim=1)
    y_data = torch.stack(y_data, dim=1)

    return X_data, y_data

# X shape = (time_step, n_samples, n_features)
# y shape = (time_step, n_samples, 1)

X = torch.Tensor(np.zeros((4, 10, 6)))
y = torch.Tensor(np.zeros((4, 10, 1)))

for i in range(X.shape[0]):
    X[i, :, :] = i
    y[i, :, :] = i

dataset = CustomDataSet(X=X, y=y)

loader = DataLoader(dataset, batch_size=3, shuffle=False, num_workers=1, collate_fn=my_collate)

for i, batch in enumerate(loader):
    print(batch[0].shape, batch[1].shape)
</code></pre>

nivek: <p>Glad you are able to resolve the issue! It might be marginally faster if you use list comprehension instead of <code>.append</code> but it shouldn’t matter too much.</p>
Regression ANN for percentage values

Duks: <p>Hi,<br>
in ‘classical statistics’ I would possibly use a glm (family = negative binomial) for a regression with tabular data (x = count values, categorial values etc.) on percentage values as output (y). This time however, I would like to fit a multilayer ‘multivariate’ ANN (with PyTorch) for this task.<br>
I found some threads, were the output (which means percentage values) is just treated as continuous (like % of fat in milk) and a regression ANN is run anyway, e.g.:</p>
<pre><code class="lang-auto">class multiple_regression_ANN(nn.Module):
      def __init__(self, num_features):
         super(multiple_regression_ANN, self).__init__()

         self.layer_1 = nn.Linear(number_of_features, some_number)
         self.layer_2 = nn.Linear(some_number, some_number)
              ...
         self.layer_n = nn.Linear(some_number, some_number)
         self.layer_out = nn.Linear(some_number, 1)
        
         self.relu = nn.ReLU()
</code></pre>
<p>Is that mathematically acceptable to run a regression ANN on percentage values, if I do not want to extrapolate with my ANN outside of my bounds (e.g. lower = 3.9%, upper = 30.2%)? Or how could I change the network in the hidden and output layer to be suitable for predicting percentage values?</p>
<p>Any help, code suggestions, alternative PyTorch ANN for this task or link is highly appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Error loading checkpoints using the sample code online

paulamartingonzalez: <p>Hi! I’ve been trying to use the code provided <a href="https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html" rel="noopener nofollow ugc">here</a> to save and load model checkpoints:</p>
<blockquote>
<p>torch.save({<br>
‘epoch’: epoch,<br>
‘model_state_dict’: model.state_dict(),<br>
‘optimizer_state_dict’: optimizer.state_dict(),<br>
‘loss’: train_loss,<br>
}, PATH)</p>
</blockquote>
<p>But I am getting the following error at loading:</p>
<blockquote>
<p>checkpoint = torch.load(DATA_ROOT_DIR + “/” + file_name)</p>
<hr>
<p>UnpicklingError                           Traceback (most recent call last)<br>
Input In [10], in &lt;cell line: 3&gt;()<br>
1 model = JARVISTransformer_channelInt(d_model = 4 + 12, nhead = 2, dim_feedforward = 64, num_layers = 2,dropout = 0.5,extra_feats = 12)<br>
----&gt; 3 checkpoint = torch.load(DATA_ROOT_DIR + “/” + file_name)<br>
4 model.load_state_dict(checkpoint[‘model_state_dict’])<br>
6 model.eval()</p>
<p>File ~/.conda/envs/cgrds/lib/python3.10/site-packages/torch/serialization.py:795, in load(f, map_location, pickle_module, weights_only, **pickle_load_args)<br>
793     except RuntimeError as e:<br>
794         raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None<br>
 → 795 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)</p>
<p>File ~/.conda/envs/cgrds/lib/python3.10/site-packages/torch/serialization.py:1002, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)<br>
996 if not hasattr(f, ‘readinto’) and (3, 8, 0) &lt;= sys.version_info &lt; (3, 8, 2):<br>
997     raise RuntimeError(<br>
998         "torch.load does not work with file-like objects that do not implement readinto on Python 3.8.0 and 3.8.1. "<br>
999         f"Received object of type "{type(f)}". Please update to Python 3.8.2 or newer to restore this "<br>
1000         “functionality.”)<br>
 → 1002 magic_number = pickle_module.load(f, **pickle_load_args)<br>
1003 if magic_number != MAGIC_NUMBER:<br>
1004     raise RuntimeError(“Invalid magic number; corrupt file?”)</p>
<p>UnpicklingError: invalid load key, ‘{’.</p>
</blockquote>
<p>I haven’t seen any similar issue online, have any of you found something similar? Is it possible to load the model at all? And where’s the error in saving? Thanks!</p>

ptrblck: <p>I’m not exactly sure why the error is raised in your code but <code>pickle</code> would fail to load the file if it was stored in another format.<br>
E.g. this small example shows the same error:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
sd = model.state_dict()

with open("test.pt", "w") as f:
    f.write("{'model_state_dict': sd}")
    
torch.load("test.pt")
# UnpicklingError: invalid load key, '{'.
</code></pre>
<p>and as you can see I’ve stored the <code>state_dict</code> as a string.<br>
Could you check if the file you are trying to load was indeed created via <code>torch.save</code>?</p>
Inputs with M>3 channels to a CNN

moth: <p>Greetings,</p>
<p>I have input objects with size (10, 1024, 1024). I wish to pass these objects to a CNN to generate 1-dimensional embeddings. Is it ok to pass these objects as they are or does something need to be changed in the CNN architecture?</p>

JuanFMontesinos: <p>Actually, it’s relevant to know what these “objects” are.<br>
It’s not like you can pass tensors with whichever meaning and it will work <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"><br>
Perhaps you should take a foundations course in ML or some pre-existing tool for the object type you are using.</p>
Batched affine transormation

Ryan_Keathley1: <p>I have a batch of images/observations and I want to apply an affine transformation to each of them as a batch, with angle, shear, translation etc provided as a tensor.<br>
Obviously I could so this with python iteration, but I’m trying to make this as performant as possible. Minimal example of what I’ve tried:</p>
<pre><code class="lang-auto">def affine (
    img,
    angle,
):
    return torchvision.transforms.functional.affine(
        img=img,
        angle=angle,
        translate=(0,0),
        scale=1,
        shear=1,
    )
batch_affine = vmap(affine)
batch_size = 10
x = obs(batch_size, 3, 3)
y = batch_affine(
    x,
    torch.rand((batch_size,), dtype=torch.float),
)
</code></pre>
<p>This above example doesn’t work, as functional.affine expects float args. Alas, you can’t use item() in vmap, so I’m not sure what to do.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/ryan_keathley1">@Ryan_Keathley1</a>,</p>
<p>Does the .item() error occur within <code>torchvision.transforms.functional.affine</code>? If so, open an issue on their github: <a href="https://github.com/pytorch/functorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/functorch · GitHub</a></p>

Ryan_Keathley1: <p>No, it’s in the vmap call. It gives a specific warning that it’s not supported.</p>

AlphaBetaGamma96: <p>If that function isn’t supported by <code>vmap</code>, you’ll need to open an issue (link above) and they’ll add it to their list of operation to vectorize!</p>
Does model weight affect batch size?

chopby: <p>I am using the same model but using different pretrained weights.<br>
When A weight is used, the batch size is 48, whereas when B weight is used, out-of-memory occurs when 4 or more is used. Except for torch.load, the other code is exactly the same.<br>
Why is this problem occurring?</p>

ptrblck: <p>No, this should not happen since each parameter would still take the same amount of memory.<br>
Would it be possible to share a minimal and executable code snippet to reproduce the issue?</p>
TorchData custom COCO dataset

austinmw: <p>Hi, if I have a custom COCO dataset and want to create a DataPipe, how can I best subclass/reuse the <code>torchvision.prototype.datasets.Coco</code> class?</p>
<p>For example:</p>
<pre><code class="lang-auto">data/vision/tiny_coco
├── instances_train2017_small.json
└── train_2017_small
    ├── 000000005802.jpg
    ├── 000000060623.jpg
    ├── 000000118113.jpg
    ├── 000000184613.jpg
    ├── 000000193271.jpg
    ├── 000000222564.jpg
    ├── 000000224736.jpg
    ├── 000000309022.jpg
    ├── 000000318219.jpg
    ├── 000000374628.jpg
    ├── 000000391895.jpg
    ├── 000000403013.jpg
    ├── 000000483108.jpg
    ├── 000000522418.jpg
    ├── 000000554625.jpg
    └── 000000574769.jpg
</code></pre>
<p>Any help is greatly appreciated!</p>