akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Mixed precision and r1 regularization

Andreas: <p>Hi</p>
<p>I’m currently trying to train a GAN on images with mixed precision. On important part in this GAN training is R1 regularization (<a href="https://arxiv.org/pdf/1801.04406.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1801.04406.pdf</a>, paper implementation: <a href="https://github.com/LMescheder/GAN_stability/blob/master/gan_training/train.py#L53" class="inline-onebox" rel="noopener nofollow ugc">GAN_stability/train.py at master · LMescheder/GAN_stability · GitHub</a>).</p>
<p>When I train the model using amp the gradient scales scale value gets halved every step and quickly ends up at a scale of 0.<br>
The model runs fine without mixed precision as well as with mixed precision and R1 disabled.</p>
<p>I oriented my implementation on the gradient penalty example from the AMP documentation since its quite similar (<a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-penalty" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision examples — PyTorch 1.11.0 documentation</a>).</p>
<p>This is the code block that is causing the issue when AMP is enabled:</p>
<pre><code class="lang-auto">real_loss = ...
real_logits = ...

if r1_reg:
  # Scales the loss for autograd.grad's backward pass, producing scaled_grads
  scaled_real_logits = scaler.scale(real_logits.sum())
  scaled_r1_grads = torch.autograd.grad(outputs=[scaled_real_logits], inputs=[real], create_graph=True)[0]
  
  # Creates unscaled grads before computing the penalty. scaled_grads are
  # not owned by any optimizer, so ordinary division is used instead of scaler.unscale_:
  inv_scale = 1. / scaler.get_scale()
  r1_grads = scaled_r1_grads * inv_scale
  
  with torch.cuda.amp.autocast():
      r1_penalty = r1_grads.square().sum([1, 2, 3])
      loss_r1 = torch.mean(r1_penalty * (r1_gamma / 2))
      
      real_loss += loss_r1

scaler.scale(real_loss).backward()
</code></pre>
<p>thanks in advance for any clue!</p>
<p>Best regards,<br>
Andreas</p>
Pytorch Training Time Estimator

Almaro_De_Villiers: <p>Good day, I would like to know if there is a way to write code that can measure one epoch or from the second one onwards to perhaps predict the time it would take to train a whole model. For instance, the code must be able to determine if a training model is going to take an hour, 6 hours, 10 days, and so on?</p>

InnovArul: <p>Take a look at the famous <a href="https://github.com/tqdm/tqdm"><code>tqdm</code></a> module.<br>
It would work to satisfy your requirements.</p>
<p>You can use as follows:</p>
<pre><code class="lang-auto">for index, data in enumerate(tqdm(dataloader)):
    ...
</code></pre>
<p>It will create a progress bar and show the remaining time.</p>

Almaro_De_Villiers: <p>Thank you <a class="mention" href="/u/innovarul">@InnovArul</a></p>
Loss function contains gradient w.r.t. input variables

Jiawei_Zhuang: <p>I am trying to implement a model in this <a href="https://becominghuman.ai/neural-networks-for-solving-differential-equations-fa230ac5e04c" rel="nofollow noopener">article</a>, where the loss function contains gradient w.r.t. to the input data <em>x</em>. Basically we want the <strong>gradient</strong> of the NN to approximate a certain function.</p>
<p>In <a href="https://github.com/HIPS/autograd" rel="nofollow noopener">HIPS/autograd</a>, it would be something like</p>
<ol>
<li>Define the forward path, with input data <code>x</code> and weights <code>w</code>:<br>
<code>f(x; w) = ...</code>
</li>
<li>Take gradient w.r.t. to <code>x</code>:<br>
<code>dfdx = grad(f, x)</code>
</li>
<li>Use such gradient to construct the loss function. We want <code>dfdx</code> to be close enough to our desired <code>dfdx_true</code>, so:<br>
<code>loss = sum_of_error(dfdx, dfdx_true)</code>
</li>
<li>Take gradient of the loss w.r.t. to <code>w</code>, just like in any neural network<br>
<code>lossgradient = grad(loss, w)</code>
</li>
<li>Use any normal training method, for exmple<br>
<code>w - lr*lossgradient(w, x_data)</code>
</li>
</ol>
<p>In pytorch, before I call <code>loss.backward()</code>, how can I let <code>loss</code> contain the gradient w.r.t. <code>x</code>?  Notice that the final result would be  a mixed derivative to <code>x</code> and <code>w</code>, not a second-order derivative to <code>w</code>.</p>

albanD: <p>In pytorch, the corresponding functions are:</p>
<ol>
<li><code>f(x; w) = ...</code></li>
<li><code>dfdx = autograd.grad(f, x, df, create_graph=True)</code></li>
<li><code>loss = sum_of_error(dfdx, dfdx_true)</code></li>
<li>
<code>optimizer.zero_grad()</code> then <code>loss.backward()</code>
</li>
<li><code>optimizer.ste()</code></li>
</ol>

ppower1: <p>What is df in this context? Thanks</p>

eva.die: <p>Is there a way of saving dfdx as a function and just evaluate it for given inputs x?</p>

InnovArul: <p>I think, if <code>f(x)</code> is a linear function of <code>x</code>, it would be possible. However, if there are multi-level hidden features with non-linearities involved, backpropagation needs intermediate feature values to perform a backward pass. Thus, a forward pass is needed. Don’t you think?</p>
CTC loss error after first epoch

Mohamed_Nabih: <p>I am training a character recognizer based on the Libri-speech dataset.</p>
<pre><code class="lang-auto">The input is the Mel-spectrogram shape --&gt; tensor of size [10,1,512, 2000]
The labels shape --&gt; tensor of size [10, 300]
The output shape --&gt; tensor of size [1000, 10, 29]
input_lengths = [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000]
target_lengths = [300, 300, 300, 300, 300, 300, 300, 300, 300, 300]
</code></pre>
<p>Following a snapshot of the code</p>
<pre><code class="lang-auto">criterion_back = torch.nn.CTCLoss(blank=28)
output back_model(enhanced)
output = F.log_softmax(output, dim=2)
output = output.transpose(0, 1)
back_loss = criterion_back(output, label, input_lengths, target_lengths)
</code></pre>
<p>The problem is that at the beginning of the second epoch. I got this error:<br>
Hint: The first epoch was done smoothly without any error this error happens when it starts the second epoch.</p>
<pre><code class="lang-auto">File "train.py", line 95, in &lt;module&gt;
    back_loss = criterion_back(output, label, input_length, target_length)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1714, in forward
    return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2559, in ctc_loss
    return torch.ctc_loss(
RuntimeError: input_lengths must be of size batch_size
</code></pre>

nateanl: <p>Hi <a class="mention" href="/u/mohamed_nabih">@Mohamed_Nabih</a>, according to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html" rel="noopener nofollow ugc">CTCLoss</a> documentation, the shape of output is [time, batch, num_class]. In your case, the shape of output is [1000, 10, 29]. 1000 is the number of frame and 10 is the batch size, so you don’t need to apply <code>output = output.transpose(0, 1)</code> after log_softmax.</p>
<p>Can you try with removing the transpose line?</p>
Differences between nn.Embedding and nn.Parameters?

ahhhh: <p>Hi there,</p>
<p>What’s the differences between nn.Embedding and nn.Parameters when I only need to access the the word embedding by indices?</p>
<p>Suppose:</p>
<ol>
<li>the word indices are W = [1, 2, 3, 4, 5, 0], which is a LongTensor.</li>
<li>E = nn.Embedding(10, 2)</li>
<li>P = torch.Tensor(10, 2).uniform_(-0.1, 0.1).requires_grad_()</li>
</ol>
<p>If I want to access the embedding of the corresponding words. what’s the differences between the following two ways:</p>
<p>E(W)</p>
<p>and</p>
<p>P[W, :]</p>
<p>?</p>
<p>Any efficiency differences?</p>

Carsten_Ditzel: <aside class="quote no-group" data-username="ahhhh" data-post="1" data-topic="17629">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahhhh/48/1769_2.png" class="avatar"> ahhhh:</div>
<blockquote>
<p>What’s the differences between nn.Embedding and nn.Parameters when I only need to access the the word embedding by indices?</p>
<p>Suppose:</p>
<ol>
<li>the word indices are W = [1, 2, 3, 4, 5, 0], which is a LongTensor.</li>
<li>E = nn.Embedding(10, 2)</li>
<li>P = torch.Tensor(10, 2).uniform_(-0.1, 0.1).requires_grad_()</li>
</ol>
<p>If I want to access the embedding of the corresponding words. what’s the differences between the following two ways:</p>
<p>E(W)</p>
<p>and</p>
<p>P[W, :]</p>
</blockquote>
</aside>
<p>I also would like to know the difference</p>

Aray: <p><a href="https://github.com/lucidrains/vit-pytorch/issues/60" rel="noopener nofollow ugc">Here</a> is a nice explanation:</p>
<blockquote>
<p>they are actually all the same underneath, just a trainable matrix (linear comes with an extra bias tensor). however, they have wrappers that allow them to behave differently when you give it an input.</p>
<ol>
<li><code>nn.Embedding</code> selects the rows of the given matrix, given a list of integers</li>
<li><code>nn.Linear</code> does the einsum operation <code>...d, d e -&gt; ...e</code></li>
<li><code>nn.Parameter</code> basically just makes a tensor trainable (receive gradients and updates on step). this is the lowest level you can go, so actually, you can define your entire deep neural network with just nn.Parameters and manually do all the above with gathers and matrix multiplies or einsums</li>
</ol>
</blockquote>
Can't backward the loss

Elad_Cohen: <p>I;m trying to perform gradient descent on a point, but fail in loss.backward() because requires_grad is False.</p>
<p>My code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, p, P1, P2):
        super(Net, self).__init__()
        self.p = torch.tensor(p, dtype=torch.float32, requires_grad=True)
        self.P1 = torch.tensor(P1, dtype=torch.float32)
        self.P2 = torch.tensor(P2, dtype=torch.float32)
        self.one = torch.tensor([1], dtype=torch.float32)

    def forward(self):
        p = torch.cat([self.p, self.one])
        u1_h = torch.matmul(self.P1, p)
        u2_h = torch.matmul(self.P2, p)
        u1 = u1_h[:2] / u1_h[2]
        u2 = u2_h[:2] / u2_h[2]
        return torch.cat([u1, u2])

net = Net(out_3d[landmark_idx, :], P1, P2)
net.train()
criterion = nn.MSELoss()
optimizer = optim.SGD([net.p], lr=0.001, momentum=0.9)
label = torch.Tensor([[u1, v1, u2, v2]])
for iter in range(10):
    optimizer.zero_grad()
    outputs = net()
    loss = criterion(outputs, label)
    print(f'landmark #{landmark_idx}: {iter} - {loss}')
    loss.backward()
    optimizer.step()

</code></pre>
<p>Apparantly, after <em>p = torch.cat([self.p, self.one])</em> p.requires_grad is already False</p>
<p>Any ideas why?</p>

ptrblck: <p>I don’t know which shapes you’ve used and how the undefined variables are used, but this code snippet shows that <code>output</code> has a valid <code>grad_fn</code> and also its <code>.requires_grad</code> attribute is <code>True</code>:</p>
<pre><code class="lang-python">b = torch.randn(1, 1)
c = torch.randn(3, 2)
net = Net(b, c, c)

outputs = net()
print(outputs.grad_fn)
# &lt;CatBackward0 object at 0x7f552df74f40&gt;
print(outputs.requires_grad)
# True

outputs.mean().backward() # works
</code></pre>

Elad_Cohen: <p>First of all: Thanks a lot for the quick reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Second: You’re right of course. I didn’t run the snippet as standalone, and in the code I have above it, there’s a <strong>with torch.no_grad():</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12" title=":man_facepalming:" class="emoji" alt=":man_facepalming:" loading="lazy" width="20" height="20"><br>
Sorry…</p>
Onnx mixed precision slow

xyz: <p>Hi, I am trying to export a mixed precision model to onnx. Sadly, the model is much slower when I’m running it in the onnxruntime:</p>
<pre><code class="lang-python">import onnxruntime as ort
from functools import partial
import onnx
import time
import timeit
import torch
import torch.nn as nn
from torchvision.models import resnet18
import torch.utils.benchmark as benchmark


class Net(nn.Module):
    def __init__(self, amp=False):
        super().__init__()
        self.amp = amp
        self._model = resnet18()
        return

    def forward(self, x):
        if self.amp:
            with torch.cuda.amp.autocast():
                x = self._model(x)
        else:
            x = self._model(x)
        return x


def bench_torch(net, x, N):
    with torch.inference_mode():
        _ = net(x)
        torch.cuda.synchronize()
        t0 = time.time()
        for _ in range(N):
            torch.cuda.synchronize()
            _ = net(x)
            torch.cuda.synchronize()
        t1 = time.time()
        print(f"{((t1-t0)/N)*1000:6.2f}")
    return


def bench_onnx(name, x, N):
    session = ort.InferenceSession(name, providers=['CUDAExecutionProvider'])
    _ = session.run(None, {'input':x})[0]
    t0 = time.time()
    for _ in range(N):
        _ = session.run(None, {'input':x})[0]
    t1 = time.time()
    print(f"{((t1-t0)/N)*1000:6.2f}")
    return

def export(net, name, x):
    torch.onnx.export(net, (x,), name, input_names=['input'], output_names=['logits'], dynamic_axes={'input':{0:'batch'}}, opset_version=15)
    assert onnx.checker.check_model(name, full_check=True) is None, "Check failed"

    print("exported model", name)
    return

net = Net(False).eval().cuda()
net_amp = Net(True).eval().cuda()
x = torch.randn(500, 3, 150, 150).cuda()
xn = x.cpu().numpy()

export(net, 'net.onnx', x)
export(net_amp, 'net_amp.onnx', x)



N = 10
bench_torch(net, x, N) # 117.28
bench_torch(net_amp, x, N) # 56.61
bench_onnx('net.onnx', xn, N) # 134.93
bench_onnx('net_amp.onnx', xn, N) # 2762.18
</code></pre>
<p>Is there anything wrong in my method?</p>
<p>System:</p>
<ul>
<li>windows 10</li>
<li>rtx 2080</li>
<li>pytorch 1.11.0</li>
<li>onnxruntime-gpu 1.10.0</li>
</ul>
<p>Thanks</p>

xyz: <p>This is quite important for me. Has really no one an opinion on this?</p>
Pytorch gpu memory management

oracal: <p>I tried to measure the gpu memory occupation when launching a DL model process. When I launched a process in conda env1(cuda10, pytorch 1.7), I observed that total 880MB memory was occupied by <strong>nvidia-smi</strong> while it became 1912MB when I measured in conda env2(cuda11, pytorch 1.11). I was a little confused and will someone give some further explanations ? Thanks so much</p>

JuanFMontesinos: <p>I don’t now in-depth details. Simply just to mention that initializing cuda requires different amounts of memory depending on the version.<br>
Note that if they add more functionalities (for example complex support) they need to add a bunch of headers and files for it.</p>
<p>So basically it will depend on the gpu you use and the pytorch/cuda/cudnn versions that it uses.<br>
Anyway I find 1912 Mb excessive.</p>
iNaturalist download crashes towards the end

Sam_Lerman: <p><em>INaturalist</em> download proceeds fine up to the end, then crashes with this error:</p>
<pre><code class="lang-auto">File "/scratch/user/myenvs/env/lib/python3.8/site-packages/torchvision/datasets/utils.py", line 152, in download_url
    raise RuntimeError("File not found or corrupted.")
</code></pre>
<p>Using torchvision automatic downloading of the <code>2021_train</code> dataset.</p>

nivek: <p>What version of the library are you running? You may want to upgrade to the latest version.</p>
<p>The error is raised when the <a href="https://github.com/pytorch/vision/blob/467b841ac2650c962952bbee5a542edb835c98bc/torchvision/datasets/inaturalist.py#L83-L84" rel="noopener nofollow ugc">integrity check fails</a>. You can re-try while setting <code>download=True</code> to see if it works or if you get another error message.</p>

Sam_Lerman: <p><code>torchvision==0.11.3</code></p>
<p>And <code>download=True</code> is enabled <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

Sam_Lerman: <p>It gets through most of the download and then fails a the end. I tested it across multiple systems even.</p>

nivek: <p>Can you verify the files are at the root location and intact? If not, you can remove them and re-download them again?</p>

Sam_Lerman: <p>Well, there’s a <code>INaturalist_Train/2021_train.tgz</code>. But I deleted it and tried re-downloading, and the same error occurred at the end of the download. When I try running, I get:</p>
<pre><code class="lang-auto">  File "/u/slerman/miniconda3/envs/env/lib/python3.8/site-packages/torchvision/datasets/utils.py", line 152, in download_url
    raise RuntimeError("File not found or corrupted.")
</code></pre>

pmeier: <p>Thanks for the report <a class="mention" href="/u/sam_lerman">@Sam_Lerman</a>. I’ve opened an issue <a href="https://github.com/pytorch/vision/issues/5817" rel="noopener nofollow ugc">here</a> to track it.</p>

pmeier: <p>I downloaded the archive and computed the MD5 checksum:</p>
<pre><code class="lang-auto">e0526d53c7f7b2e3167b2b43bb2690ed
</code></pre>
<p><a class="mention" href="/u/sam_lerman">@Sam_Lerman</a> can you confirm that? If yes, the one <code>torchvision</code> has on record is wrong and needs to be fixed.</p>

Sam_Lerman: <p>What do I need to do to confirm it?</p>

Sam_Lerman: <p>Yes, the output is <code>e0526d53c7f7b2e3167b2b43bb2690ed</code>.</p>

pmeier: <p>This was fixed in <a href="https://github.com/pytorch/vision/pull/5844" class="inline-onebox" rel="noopener nofollow ugc">fix INaturalist 2021_train checksum by pmeier · Pull Request #5844 · pytorch/vision · GitHub</a>. If you are using nightly builds, the fix will be available later today. Otherwise, you can follow the instructions in <a href="https://github.com/pytorch/vision/issues/5817#issuecomment-1102145706" class="inline-onebox" rel="noopener nofollow ugc">INaturalist download is broken for version="2021_train" · Issue #5817 · pytorch/vision · GitHub</a> for a workaround until the next release.</p>
Alternatives to torch.mode?

Yuki-F: <p>I would really like know if there are any possible workarounds to <code>torch.mode</code>, other than combining <code>torch.unique(input, return_counts=False)</code> and <code>torch.argmax()</code>. Obviously, simply iterating over the input tensor values and counting the number of times each value appear takes too much time when the tensor gets longer…</p>
<p>A slight background into why I want to do this, is because I want to convert a pretrained PyTorch model into a CoreML format. The details of CoreML is irrelevant here, but the converter doesn’t seem to support either <code>torch.mode</code> or <code>torch.unique</code>, so I was wondering how PyTorch actually execute the <code>torch.mode</code> function and if there are any other functions that can similarly do the job.</p>
Problem with running the validation tests

Luca_Pamparana: <p>I am using pytorch lightning and have defined my training epoch end and validation epoch end as follows:</p>
<pre><code class="lang-auto">def training_step(self, batch, batch_index):
        x, y, w = batch
        return {"loss": some_loss}

def training_epoch_end(self, outputs: EPOCH_OUTPUT) -&gt; None:
        loss = torch.stack([x["loss"] for x in outputs]).mean()
        self.log("train/loss", loss)

def validation_step(self, batch, batch_index):
        x, y, w = batch
        return {"val/loss": some_loss}

def validation_epoch_end(self, outputs: EPOCH_OUTPUT) -&gt; None:
        loss = torch.stack([x["val/loss"] for x in outputs]).mean()
        self.log("val/loss", loss)
</code></pre>
<p>Now, I have data loaders like:</p>
<pre><code class="lang-auto">def train_dataloader(self):
        weighted_sampler = WeightedRandomSampler(...)
        return DataLoader(
            self.data_train,
            shuffle=False,
            sampler=weighted_sampler,
            batch_size=self.batch_size,   #64
            num_workers=self.num_workers,
            pin_memory=True,
            persistent_workers=True,
        )

def val_dataloader(self):
        if self.data_val is None:
            return None

        weighted_sampler = WeightedRandomSampler(...)

        return DataLoader(
            self.data_val,
            shuffle=False,
            sampler=weighted_sampler,
            batch_size=self.batch_size,  #64
            num_workers=self.num_workers,
            pin_memory=True,
            persistent_workers=True,
        )
</code></pre>
<p>The issue is the training loop is fine but when I run the validation loop, it comes back with:</p>
<pre><code class="lang-auto">loss = torch.stack([x["loss"] for x in outputs]).mean()
RuntimeError: stack expects each tensor to be equal size, but got [64, 1] at entry 0 and [35, 1] at entry 1658
</code></pre>
<p>Do you think this is because of the <code>batch_size</code> which is set to 64 and maybe gets truncated at the last slice? How can I solve this?</p>

ptrblck: <p>Yes, I think your explanation makes sense and you could use <code>drop_last=True</code> in the <code>DataLoader</code> initialization to drop the last (smaller) batch.</p>
Why is 'sampler.set_epoch(epoch)' needed for DistributedSampler?

mesllo: <p>I’ve seen various examples using <code>DistributedDataParallel</code> where some implement the <code>DistributedSampler</code> <strong>and</strong> also set <code>sampler.set_epoch(epoch)</code> for every epoch in the train loop, and some that just skip this entirely. Why is this, and is it really needed for the distributed training to execute correctly?</p>

ptrblck: <p>Based on the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler">docs</a> it’s necessary to use <code>set_epoch</code> to guarantee a different shuffling order:</p>
<blockquote>
<p>In distributed mode, calling the <code>set_epoch()</code> method at the beginning of each epoch <strong>before</strong> creating the <code>DataLoader</code> iterator is necessary to make shuffling work properly across multiple epochs. Otherwise, the same ordering will be always used.</p>
</blockquote>
next(iter(DataLoader)) get stuck nd does not return

bach97: <p>Hi, I am using a custom <code>DataLoader</code> to iterate a custom dataset. It seemed working for a while, but suddendly when calling <code>next(iter(DataLoaderCustom))</code>, the program stops and wait forever. The same if I call <code>DatasetCustom[0]</code>.</p>
<p>Has anyone some ideas? Thank you!</p>

JuanFMontesinos: <p>I’d say you coded some sort of infinite loop in which your loader gets trapped.</p>
Use certain layers of a custom model

TeDataPro: <p>Hello !</p>
<p><em>My situation</em> : From a personal NLP model used for <strong>text classification with BERT</strong>, which was <strong>already pre-trained</strong> on a corpus (by myself). I want to <strong>remove the last layers</strong> (classification layers) and <strong>add new final ones for an other task</strong>. This to <strong>use the retained layers to create this other model, for a similar task, so as not to re-train everything.</strong></p>
<p>From this topic, I’ve been able to extract layers which interest me (These layers are BERT Embedding) : <a href="https://discuss.pytorch.org/t/how-to-delete-layer-in-pretrained-model/17648" class="inline-onebox">How to delete layer in pretrained model?</a></p>
<p>Here’s my code :</p>
<pre><code class="lang-auto">import torch.nn as nn

# Remove classification layer from my model
sub_model = nn.Sequential(*list(model.camembert.children())[:-1])
</code></pre>
<p><em>My problem</em> : How do I use my new object <strong>sub_model</strong> to re-build the same architecture of model with a different final layer ? Since sub_model is now a nn.Sequential object and model.camembert was a camemBERT transformer.</p>
<p>I tried to litteraly <strong>replace model.camembert by sub_model</strong> but it doesn’t work. Here’s the error :</p>
<pre><code class="lang-auto"># Original model : works fine
logits = model.camembert(input_ids, attention_mask=attention_mask, labels=None, output_hidden_states=False).logits

# New model : doesn't work
import torch.nn as nn
sub_model = nn.Sequential(*list(model.camembert.children())[:-1])
logits = sub_model(input_ids, attention_mask=attention_mask, labels=None, output_hidden_states=False).logits
</code></pre>
<p>The error is :</p>
<pre><code class="lang-auto">TypeError: forward() got an unexpected keyword argument 'attention_mask'
</code></pre>
<p>I’m pretty new to PyTorch so thank you very much for your help ! Have a nice day !</p>

JuanFMontesinos: <p>I don’t understand at all.<br>
The sequential network only digest 1 argument and it basically created like a chain.<br>
x-&gt;sub1-&gt;sub2-&gt;sub3<br>
so the output of 1 goes to 2 and so on.<br>
Either you recode evberything to work with dictionaries or you rewrite the nn.Sequential as a custom nn.Module</p>
Finding the latest nightly wheels without pip

rehno: <p>Hi,</p>
<p>I’m trying to create an experimental nix (the package manager) channel that tracks the latest nightly wheels from <a href="https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html</a></p>
<p>Is there any way to resolve the wheels without using pip?</p>
<p>I noticed that <a href="https://github.com/pytorch/pytorch/issues/23656" class="inline-onebox" rel="noopener nofollow ugc">Official and nightly wheel structure plan · Issue #23656 · pytorch/pytorch · GitHub</a>  mentions a possible <code> https://download.pytorch.org/whl_redirect?....</code> url structure that resolves to a download url. By any chance, is this something that’s been implemented?</p>
<p>(Pypi also publishes json e.g. <code>https://pypi.org/pypi/torch/json</code>, but unfortunately this doesn’t help with resolving nightly releases)</p>
Compute the gradient of the loss function

periklis17: <p>How i can compute the gradient of the loss with respect to the parameters of the network?</p>

KFrank: <p>Hi Periklis!</p>
<aside class="quote no-group" data-username="periklis17" data-post="1" data-topic="149503" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ac91a4/48.png" class="avatar"> periklis17:</div>
<blockquote>
<p>How i can compute the gradient of the loss with respect to the parameters of the network?</p>
</blockquote>
</aside>
<p>Autograd to the RESCUE!</p>
<p>Specifically, compute a <code>loss</code> that depends on your network, call<br>
<code>loss.backward()</code>, and autograd will do the rest.  You can examine the<br>
<code>.grad</code> properties of your network parameters.  (This requires that your<br>
network parameters have <code>requires_grad = True</code> and that you compute<br>
your loss using (usefully differentiable) pytorch tensor operations.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; network = torch.nn.Linear (2, 3, bias = False)
&gt;&gt;&gt; network.weight
Parameter containing:
tensor([[-0.1474,  0.5967],
        [ 0.3660, -0.1681],
        [-0.6700, -0.1989]], requires_grad=True)
&gt;&gt;&gt; input = torch.arange (10.).reshape (5, 2)
&gt;&gt;&gt; input
tensor([[0., 1.],
        [2., 3.],
        [4., 5.],
        [6., 7.],
        [8., 9.]])
&gt;&gt;&gt; loss = (network (input)).sum()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; network.weight.grad
tensor([[20., 25.],
        [20., 25.],
        [20., 25.]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

periklis17: <p>Thank you so much for your answer, however my model has no this .weight .<br>
On the other hand i tried to do something like<br>
for param in model.parameters():<br>
grads.append(param.grad.view(-1))<br>
My goal is to compute the \nabla_{\theta} l(f(x;\theta),y)<br>
where l: is the loss function<br>
f(x;\theta): is the prediction of the model for the input x parametrized by parameters \theta</p>
Training 3DU-Net in multiple GPUs

Yichong_Wang: <p>3DU-net Code:</p>
<pre><code class="lang-auto">class UNet(nn.Module):
    def __init__(self):
        super(UNet, self).__init__()
        self.c1 = convBlock(1, 64).to('cuda:0')
        self.d1 = downSample(64).to('cuda:0')
        self.c2 = convBlock(64, 128).to('cuda:0')
        self.d2 = downSample(128).to('cuda:0')
        self.c3 = convBlock(128, 256).to('cuda:0')
        self.d3 = downSample(256).to('cuda:1')
        self.c4 = convBlock(256, 512).to('cuda:1')
        self.d4 = downSample(512).to('cuda:1')
        self.c5 = convBlock(512, 1024).to('cuda:1')
        self.u1 = upSample(1024).to('cuda:1')
        self.c6 = convBlock(1024, 512).to('cuda:1')
        self.u2 = upSample(512).to('cuda:1')
        self.c7 = convBlock(512, 256).to('cuda:1')
        self.u3 = upSample(256).to('cuda:1')
        self.c8 = convBlock(256, 128).to('cuda:1')
        self.u4 = upSample(128).to('cuda:0')
        self.c9 = convBlock(128, 64).to('cuda:0')
        self.out = nn.Conv3d(64, 1, 3, 1, 1).to('cuda:0')
        self.th = nn.Sigmoid().to('cuda:0')

    def forward(self, x):
        L1 = self.c1(x.to('cuda:0'))
        L2 = self.c2(self.d1(L1.to('cuda:0')).to('cuda:0'))
        L3 = self.c3(self.d2(L2.to('cuda:0')).to('cuda:0'))
        L4 = self.c4(self.d3(L3.to('cuda:1')).to('cuda:1'))
        L5 = self.c5(self.d4(L4.to('cuda:1')).to('cuda:1'))
        R4 = self.c6(self.u1(L5.to('cuda:1'), L4.to('cuda:1')).to('cuda:1'))
        R3 = self.c7(self.u2(R4.to('cuda:1'), L3.to('cuda:1')).to('cuda:1'))
        R2 = self.c8(self.u3(R3.to('cuda:1'), L2.to('cuda:1')).to('cuda:1'))
        R1 = self.c9(self.u4(R2.to('cuda:0'), L1.to('cuda:0')).to('cuda:0'))

        return self.th(self.out(R1.to('cuda:0')).to('cuda:0'))
</code></pre>
<p>convBlock, downSample, upSample is layer in my own code.</p>
<p>I want to train 3DU-Net, but the GPU memory is not enough, so I want to use multiple GPUs to train this model.</p>
<p>I assign different U-net layers to different GPUs.</p>
<p>I want to ask if this is the correct way to use different GPUs to train models? And what’s the best way to run multiple GPU training python scripts using the PyTorch module in the Linux server?</p>

JuanFMontesinos: <aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/transformers/performance">
  <header class="source">

      <a href="https://huggingface.co/docs/transformers/performance" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/transformers.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/transformers/performance" target="_blank" rel="noopener nofollow ugc">Performance and Scalability: How To Fit a Bigger Model and Train It Faster</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
LSTM Gain Not Captured

Andre_Amaral_IST: <p>Hey,</p>
<p>I am training a LSTM for circuit behavioral modeling.<br>
Looking at this picture:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b424cb48279f74b3a7abc6150b0686caf1cdc53.png" data-download-href="https://discuss.pytorch.org/uploads/default/8b424cb48279f74b3a7abc6150b0686caf1cdc53" title="mis"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b424cb48279f74b3a7abc6150b0686caf1cdc53_2_590x412.png" alt="mis" data-base62-sha1="jRWpI50cuqAwiov8L3SP1gsGeJ5" width="590" height="412" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b424cb48279f74b3a7abc6150b0686caf1cdc53_2_590x412.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b424cb48279f74b3a7abc6150b0686caf1cdc53_2_885x618.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b424cb48279f74b3a7abc6150b0686caf1cdc53.png 2x" data-dominant-color="F2F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mis</span><span class="informations">1122×671 224 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It shows that the model is not capturing the amplitude changes correctly.<br>
What hyperparameters need to be changed or improved to solve this problem?<br>
<strong>Note:</strong> I feed the model with batches since i have a lot of data, i normalize the data using MinMaxScaler.</p>
<p>Regards,</p>
<p>André</p>
SyncBatchNorm and multiple forward passes

kimihailv: <p>Hello, I use DDP for training on multiple gpus. My model has several BatchNorm layers and in train loop I need to make forward pass two times on different batches. But if I make two forward passes, torch can not make backward pass due to inplace change of batchnorm stats. One of possible solutions is to disable synchronization of buffers, but it does not suit me as I want to update batchnorm stats with bigger batches. Is it possible to make two independent forward passes with batchnorm stats synchronization?</p>

ptrblck: <aside class="quote no-group" data-username="kimihailv" data-post="1" data-topic="149452">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimihailv/48/45534_2.png" class="avatar"> kimihailv:</div>
<blockquote>
<p>But if I make two forward passes, torch can not make backward pass due to inplace change of batchnorm stats.</p>
</blockquote>
</aside>
<p>This sounds strange, as batchnorm stats are buffers and will thus not get a gradient and won’t be updated by the optimizer. Instead, in each forward pass the running stats will be updated using a running average.<br>
Are you sure the batchnorm stats cause the error? If so, did you somehow manipulate them to get gradients and be trainable?</p>

kimihailv: <p>I understand that batchnorm stats are buffers. Actually I ran into the problem like in this <a href="https://discuss.pytorch.org/t/ddp-sync-batch-norm-gradient-computation-modified/82847/5">post</a>. In answers you suggested disabling buffers synchronization. But it means that each model on each gpu will receives stats from small batch (is it?), and I want to update batchnorm with stats using batches from all gpus.</p>

ptrblck: <p>Thanks for linking to this older post. Could you verify that setting this argument indeed solves the issue?</p>

kimihailv: <p>Yes, setting this argument allows to call forward passes multiple times, but I’m not sure that batchnorm stats are collected from all gpus</p>
Locating the x,y coordinate of tensors for each batch

chandlerbing65nm: <p>Please help. I want to find the x,y coordinate of certain values in a tensor with size <code>N, C, H, W</code></p>
<p>For example, I have a tensor as shown below:</p>
<pre><code class="lang-auto">N, C, H, W = 51, 1, 7, 7
x = torch.randn(N, C, H, W)
x_mean = x.view(N, -1).mean(1, keepdim=True) #calculating the mean for every batch
</code></pre>
<p>Now, I want to find the (x,y) locations in the <code>1, 7, 7</code> array where the values are greater than the <code>x_mean</code>, for each batch.</p>

AlphaBetaGamma96: <p>This can be solved via broadcasting,</p>
<pre><code class="lang-auto">N, C, H, W = 51, 1, 7, 7
x = torch.randn(N, C, H, W)
x_mean = x.view(N, -1).mean(1, keepdim=True)

x_mean = x_mean.unsqueeze(2).unsqueeze(3) #unsqueeze to match dims
xy_locations = x &gt; x_mean #boolean Tensor of which locations have a value greater than x_mean
</code></pre>

chandlerbing65nm: <p>I add <code>xy = (xy_locations == True).nonzero(as_tuple=False)</code> . Thank you.</p>
Missing library

Enrique_Pedroza: <p>I have an AMD RX 6800XT and using the current version of rocm, when i excecute my model i have the following error :</p>
<p>rocBLAS warning: No paths matched ~/.local/lib/python3.8/site-packages/torch/lib/library/<em>gfx1030</em>co. Make sure that ROCBLAS_TENSILE_LIBPATH is set correctly.</p>
<p>As it says, it seems to be missing a library inside of torch files …</p>
<p>my system is linux 64 bits and current version of pytorch is → 1.11.0+rocm4.5.2</p>
Multiple input model + transfer learning

Anna_Inberg: <p>Good afternoon!</p>
<p>My question is as follows: I’m building a binary classification model that combines 2 types of input (images and associated numerical data). There are many examples of such mixed input models: a few layers for type 1 input processing, a few layers for type 2 input processing, then both concatenate and final layers are applied to the concatenated input.</p>
<p>However, I’ve stumbled across a problem with how to incorporate transfer learning in such a model.<br>
i.e. I would like to use Resnet to classify the images instead of some custom layers. Is there any way to do it?</p>
How to normalize data with libtorch?

duola: <p>hello, everyone. i have a question about normalization with libtorch. In Pytorch help document, there shows<br>
"<br>
<code>torch.nn.functional.</code> <code>normalize</code> ( <em>input</em> ,  <em>p=2</em> ,  <em>dim=1</em> ,  <em>eps=1e-12</em> ,  <em>out=None</em> )</p>
<p>but in libotrch, it shows there is no function in libtorch.</p>
<p><code>auto datases_input_normalize = torch::nn::functional::normalize(datasets_input, 2, 1,1e-12, None);</code></p>
<p>how to solve this problem? Or is there any function can be used to normalize in libtorch?</p>
<p>Thanks very much!</p>

ptrblck: <p>Which PyTorch version are you using?<br>
This dummy code snippet works in the latest stable release:</p>
<pre><code class="lang-python">cpp_source = """
    torch::Tensor my_norm(torch::Tensor input){
    
      auto out = torch::nn::functional::normalize(input);
      return out;
    }
    
"""

module = torch.utils.cpp_extension.load_inline(
    name="my_norm",
    cpp_sources=cpp_source,
    functions="my_norm",
    verbose=True,
)

x = torch.randn(10, 10)
out = module.my_norm(x)
</code></pre>

duola: <p>Thanks for your reply, but i have problem with<br>
“auto out = torch::nn::functional::normalize(input);”</p>
<p>it shows no normalize in torch::nn::functional.</p>
<p>Do you know why?</p>

jy_mo: <p>Have you solved this problem?</p>

ximitiejiang: <pre><code class="lang-auto">auto mean = torch::tensor({ 0.485, 0.456, 0.406 }, torch::kFloat32).unsqueeze(1).unsqueeze(2).to(device);
auto std = torch::tensor({ 0.229, 0.224, 0.225 }, torch::kFloat32).unsqueeze(1).unsqueeze(2).to(device);
image_tensor = image_tensor.sub(mean).div(std);
</code></pre>
Resize image without changing prediction

JMunz: <p>I have a pretty challenging issue that I have been running into the past week. I have to get my images in the format of a numpy array. However, my model takes in images of different sizes, meaning the dimensions are different. I would like to get everything in the numpy array without changing the predictions. Right now, I can resize all images with a transformation to the same size and add to the numpy array and this works, but it changes the predictions of some of the images. I have also tried adding whitespace to the area when expanding a photo, but this also changed the predictions. Any ideas for how I can go about this?</p>

JMunz: <p>I have tried padding tensors as well to be the same size. This does not give an error, but it goes to the largest image’s dimensions, which I found that past a certain point, it completely changes the prediction of the image because there is too much white space and every prediction is the same.</p>

eqy: <p>Depending on the type of the model, you may not need to resize the images if you are OK with running it witch batch size 1. For example, many architectures use adaptive average pooling before fully connected layers which means that different resolutions are usable as input if they are not batched together.</p>
Could in-place op be fused

highfly22: <p>I try to figure out why the following code doesn’t fuse at all. I notice that the IrParser::registerJitOperator registers only the normal tensor op. Does it mean in-place op cannot be fused?</p>
<pre><code class="lang-auto">@torch.jit.script
def calc(exp_avg : Tensor, 
         exp_avg_sq : Tensor, 
         beta1 : float, 
         beta2 : float, 
         grad : Tensor):
    exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
    exp_avg_sq.mul_(beta2).addcmul_(grad, grad.conj(), value=1 - beta2)
</code></pre>
Torch.onnx customized op inputs unexpected

jinfagang: <p>I have a customized op like this:</p>
<pre><code class="lang-auto">class AtomFakeQuantizeLearnablePertensorAffine(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, scale, zero_point, quant_min, quant_max, grad_factor):
        return torch._fake_quantize_learnable_per_tensor_affine(
            x, scale, zero_point, quant_min, quant_max, grad_factor
        )

    @staticmethod
    def symbolic(g, x, scale, zero_point, quant_min, quant_max, grad_factor):
        logger.debug("symbolic for: AtomFakeQuantizeLearnablePertensorAffine")
        return g.op(
            "::AtomFakeQuantizeLearnablePertensorAffine",
            x,
            scale,
            zero_point,
            quant_min_i=quant_min,
            quant_max_i=quant_max,
        )
</code></pre>
<p>this defined inputs to be 5, but the exported onnx model make quant_min and quant_max as attributes.</p>
<p>this is unexpected, how to avoid it?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/3305132752e65fba18e2c8b8ff3cf708d57929af.png" data-download-href="https://discuss.pytorch.org/uploads/default/3305132752e65fba18e2c8b8ff3cf708d57929af" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/3305132752e65fba18e2c8b8ff3cf708d57929af.png" alt="image" data-base62-sha1="7hldS3YZLlHOtj8UlYont8WMrdZ" width="690" height="226" data-dominant-color="CDCDCD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">748×246 6.64 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Could I use torch.cuda.manual_seed() with Multiple GPUs to to keep the deterministic?

lcrun: <p>Hi experts, I am training the Vision model with multiple GPUs (8 GPUs), And I lunched the job with "mpirun --npernode 8 ",  which means I use 8 process for 8 GPUs (1 process each GPUs), So I currently initialize with this code:</p>
<pre><code class="lang-auto">if torch.cuda.is_available():
            self.seed = int(self.seed)
            random.seed(self.seed)
            np.random.seed(self.seed)
            torch.manual_seed(self.seed)

            # ddp: only set seed on GPU associated with this process
            torch.cuda.manual_seed(self.seed)

            torch.backends.cudnn.deterministic = True
            torch.backends.cudnn.benchmark = False
</code></pre>
<ol>
<li>
<p>To keep the deterministic,  Must I use torch.cuda.manual_seed_all() ? It seems torch.cuda.manual_seed() with same seed for all 8 process will also keep the deterministic for my job right?</p>
</li>
<li>
<p>Similar to the question above. I save the checkpoint (including the random_state blow) of above job</p>
</li>
</ol>
<pre><code class="lang-auto">              random_state = {'random': random.getstate(),
                                'numpy_random': np.random.get_state(),
                                'torch_random': torch.get_rng_state(),
                                'torch_cuda_random': torch.cuda.get_rng_state(device=self.opt['device']) if self.opt['CUDA'] else None
                                }
</code></pre>
<p>and try to load it  by:</p>
<pre><code class="lang-auto">        random.setstate(random_state['random'])
        np.random.set_state(random_state['numpy_random'])
        torch.set_rng_state(random_state['torch_random'])
        if self.opt['CUDA']:
            torch.cuda.set_rng_state(random_state['torch_cuda_random'], device=self.opt['device'])
</code></pre>
<p>Should I used</p>
<pre><code class="lang-auto">torch.cuda.get_rng_state_all() 
torch.cuda.set_rng_state_all () 
</code></pre>
<p>instead if</p>
<pre><code class="lang-auto">torch.cuda.get_rng_state() 
torch.cuda.set_rng_state() 
</code></pre>
<p>to keep deterministic ?</p>
<ol start="3">
<li>What is your suggestion ( <span class="hashtag">#a</span> or <span class="hashtag">#b</span> below ) is better?  :<br>
<span class="hashtag">#a</span>.  torch.cuda.manual_seed() + torch.cuda.get_rng_state()  +  torch.cuda.set_rng_state()<br>
<span class="hashtag">#b</span>.  torch.cuda.manual_seed_all() + torch.cuda.get_rng_state_all()  +  torch.cuda.set_rng_state_all()</li>
</ol>

ptrblck: <p><code>torch.manual_seed</code> should already seed the CPU and GPU(s) as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html#reproducibility">Reproducibility docs</a>.<br>
Since you are using a single process per GPU, <code>torch.cuda.manual_seed_all</code> (or the other <code>_all</code> calls) should not be necessary.</p>