akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
AMP GradScale multiple losses independently

tjoseph: <p>Hello,<br>
I am just wondering whether I am missunderstanding <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision examples — PyTorch 1.10.0 documentation</a> or whether the use case there is useless.</p>
<p>GradScaler has a single <code>scale</code> parameter. Instead of scaling two times like in the example, I could just add the losses and then scale.<br>
However, the use case that matters in my case is that the two losses could be very different in magnitude and thus, different scaling parameters could make sense. Should I just use two GradScalers then?</p>
<p>Something like:</p>
<pre><code class="lang-auto">grad_scaler_1 = GradScaler(enabled=True)
grad_scaler_2 = GradScaler(enabled=True)

loss_1_scaled = grad_scaler_1.scale(loss_1)
loss_2_scaled = grad_scaler_2.scale(loss_2)

loss = loss_1_scaled + loss_2_scaled
loss.backward()

...
</code></pre>
<p>Best,<br>
Tim</p>

ptrblck: <p>I wouldn’t call the example useless, but yes you can use multiple <code>GradScaler</code>s if that fits your use case better.</p>
Cannot load state_dict in Docker

venkat.r: <p>I have a weird problem. I can load the state_dict in my machine which was trained with Pytorch lightning on the GPU. However if i push it to docker it does not load. The model is a Pytorch Lightning module.</p>
<h1>
<a name="on-machine-1" class="anchor" href="#on-machine-1"></a>On Machine</h1>
<p>loaded_model = LitModel()<br>
loaded_model.load_state_dict(torch.load(‘models/trained-model-2021.pt’), strict=False)</p>
<p>Parameter containing:<br>
tensor([[ 0.0817, -0.2579, -0.0080,  …,  0.1619, -0.0783, -0.0087],<br>
[ 0.0866, -0.2708,  0.2528,  …,  0.0203, -0.2319,  0.1011],<br>
[ 0.0564,  0.0195, -0.2858,  …, -0.1899,  0.1004, -0.1013],<br>
…,<br>
[ 0.2584, -0.2400, -0.0358,  …, -0.1391,  0.2278,  0.0264],<br>
[ 0.1901, -0.3056, -0.2537,  …, -0.0747,  0.2703,  0.0803],<br>
[-0.2639, -0.0381,  0.0538,  …,  0.2504,  0.2152,  0.1084]],<br>
requires_grad=True)</p>
<h1>
<a name="on-docker-2" class="anchor" href="#on-docker-2"></a>On Docker</h1>
<p>device = torch.device(‘cpu’)<br>
loaded_model = LitModel()<br>
loaded_model.load_state_dict(torch.load(‘models/trained-model-2021.pt’, map_location=device), strict=False)</p>
<p>loaded_model.load_state_dict(torch.load(‘src/trained-model-2021-11-23–23:08-hs.pt’, map_location=device), strict=True)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 1407, in load_state_dict<br>
self.<strong>class</strong>.<strong>name</strong>, “\n\t”.join(error_msgs)))<br>
RuntimeError: Error(s) in loading state_dict for LitModel:<br>
Missing key(s) in state_dict: “layer_1.weight”, “layer_1.bias”, “layer_3.weight”, “layer_3.bias”.<br>
Unexpected key(s) in state_dict: “model”.</p>
<p>Any ideas what could be wrong here.</p>

ptrblck: <p>Based on the error message it seems you are not loading a <code>state_dict</code>, but a custom <code>dict</code> class containing the <code>'model'</code> key, so you might want to access it first.<br>
Something like this could work:</p>
<pre><code class="lang-python">loaded_model.load_state_dict(torch.load(‘src/trained-model-2021-11-23–23:08-hs.pt’, map_location=device)['model'])
</code></pre>

venkat.r: <p>Thanks. Thank works.</p>
Writing backward hook to change the gradients computations

Ameen_Ali: <p>Hello,<br>
Let’s say I have a model of the following layers :<br>
Linear Layer → Softmax.<br>
how can I write a backward hook in order to ignore the gradient computation of the softmax layer?</p>

ptrblck: <p>You could use use <code>register_full_backward_hook</code> and return the <code>grad_ouput</code>:</p>
<pre><code class="lang-python">def hook(module, grad_input, grad_output):
    return grad_output

x = torch.randn(1, 3, requires_grad=True)
lin = nn.Linear(3, 3, bias=False)
lin.register_full_backward_hook(hook)

out = lin(x)
out.mean().backward()
print(x.grad)
</code></pre>
Loss Variable grad_fn

DSL: <p>Hello together,</p>
<p>i try to combine a c++ programm with my neuronal network for training. Therefore i try to set a loss manually, but when i do this and print it out i get different results.</p>
<pre><code class="lang-auto">diff = part1 + part2
loss = diff.mean()
tensor(0.2783, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)

a = torch.Tensor([123]).to("cuda") - torch.Tensor([1]).to("cuda")
tensor([122.], device='cuda:0')
</code></pre>
<p>I know that the programm runs through without an error but i dont know how important this grad_fn is.<br>
Can somebody tell me more about this?</p>
<pre><code class="lang-auto">grad_fn = &lt;MeanBackward0&gt;
</code></pre>
<p>Best regards</p>

ptrblck: <p>The <code>grad_fn</code> is used during the <code>backward()</code> operation for the gradient calculation.<br>
In the first example, at least one of the input tensors (<code>part1</code> or <code>part2</code> or both) are attached to a computation graph. Since the <code>loss</code> tensor is calculated from a <code>mean()</code> operation, the <code>grad_fn</code> will point to <code>MeanBackward</code>.<br>
Your second example uses two plain tensors, which do not require gradients, so no computation graph will be created, and thus the <code>grad_fn</code> will be set to <code>None</code>.</p>

DSL: <p>Hey,<br>
thanks for the quick reply. I have looked at the computational graph again.<br>
I have the following problem and I am developing a program in C++ and I want to develop a self-supervised net. However, since I have already pre-trained the net in Python, I wrote an interface using Embedded Python. I am now wondering if I can use C++ to calculate the loss and then convert it to Python and then convert it to a torch variable and then use it to call my backward function. Would this have a negative impact on my training?</p>

ptrblck: <p>I haven’t tried to call Python from C++ and don’t know if this approach would work.<br>
The easier approach might be to script the Python model, load it in your <code>libtorch</code> C++ application and use plain C++ for further processing without going back to Python.</p>

DSL: <p>Hey,</p>
<p>yes i thought so too, however i can convert the neural net to c++ with torchscript but not further train it. If I rebuild the nn in c++, then the loading of the weights should not work.<br>
Do you know more about this?<br>
I would otherwise calculate everything necessary in c++ (without torch) and then convert to python and then to pytorch.</p>
Should I write a ParameterDeque?

Thomas_Ahle: <p>I have a module that stores a list of <code>nn.Parameter</code>s (old gradients - it’s an optimizer).</p>
<p>I only store a fixed number, say 100, so after a while I want to kick out the oldest.<br>
My natural instinct is to use a python <code>deque</code>, but of course that won’t work, since I need to store my parameters in a <code>ParameterList</code> or <code>ParameterDict</code> for stuff like <code>.cuda()</code> or <code>.double()</code> to work.</p>
<p>Now I wonder if I should just bite the (perhaps not that great) performance bullet of using a <code>ParameterList</code> as a deque, just calling <code>del list[0]</code> at each step.<br>
Or if I should clone <code>nn.ParameterList</code> and make my own implementation using <code>deque</code>?<br>
I suppose I could also use a <code>ParameterDict</code> with consecutive integers as keys…</p>
<p>None of the options seem ideal. Is there a best practice I should refer to?</p>

Thomas_Ahle: <p>I ended up creating this: <a href="https://gist.github.com/thomasahle/6e124a4e9b9f4903bf001e37cdeeea65" class="inline-onebox" rel="noopener nofollow ugc">ParameterDeque.py · GitHub</a></p>
<p>Still interested in ideas for other ways of doing this.</p>
Dataloader with sub-folder

mojzaar: <p>I have a dataset that the training data are split into 4 folders with the same size of the data in each (3000 data). I wanted to create a data loader that loads only 3000 data but instead of sampling from one folder, it randomly samples from one of the sub-folders. So, I have a Dataset object that in its <strong>getitem</strong> function, I have:</p>
<pre><code class="lang-auto"> def __getitem__(self, idx):
        iq = int(torch.randint(0, len(self.img_address), (1,)))
        img_address_ = self.img_address[iq]
        ref_address_ = self.ref_address[iq]

        img_address = img_address_[idx]
        ref_address = ref_address_[idx]
        img = self.loader(img_address)
        ref = self.loader(ref_address)
       return img, ref
def __len__(self):
        return 3000
</code></pre>
<p>However, when I sample from the data loader, I get 4 samples (number of sub-folders) instead of batch size! I tested the dataset.<strong>getitem</strong>(N), and it works fine. The problem starts when I wrap the dataset in a Dataloader! I changed the number of workers, batch size, shuffle, and … but neither of them worked!</p>
<p>Can you please help me to find where I am doing wrong?</p>

ptrblck: <p>Your code works for me using random data:</p>
<pre><code class="lang-python">class MyDataset(torch.utils.data.Dataset):
    def __init__(self):
        self.datasets = [torch.arange(i+i*100, i+i*100+100) for i in range(4)]

    def __getitem__(self, idx):
        iq = int(torch.randint(0, len(self.datasets), (1,)))
        dataset = self.datasets[iq]

        data = dataset[idx]
        return data
   
    def __len__(self):
        return len(self.datasets[0])

dataset = MyDataset()
print(dataset.datasets)

loader = torch.utils.data.DataLoader(dataset, batch_size=10)
for data in loader:
    print(data)
</code></pre>
<p>As you can see the <code>batch_size</code> is used and the samples are drawn randomly from the internal datasets.</p>

mojzaar: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="137535">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">loader = torch.utils.data.DataLoader(dataset, batch_size=10)
for data in loader:
    print(data)
</code></pre>
</blockquote>
</aside>
<p>Thank you for your help. I am having this problem when I set pin_memory=True on my data loader! I am not sure why this is happening!</p>

ptrblck: <p>Are you seeing the same issue using my code snippet with <code>pin_memory=True</code>?</p>

mojzaar: <p>No, I actually found a bug in my code that caused the problem! Sorry for bothering you and do appreciate your prompt answers!</p>
Difference between NVIDIA's pytorch-quantization and PyTorch's QAT process

MrOCW: <p>Hi, after doing some searching and readings, I notice that NVIDIA’s QAT process is different from PyTorch’s.<br>
NVIDIA seems to first calibrate the model offline, then train(QAT) the calibrated model<br>
Whereas in PyTorch, we fuse, prepare qat, enable observer and fake quant, train (QAT) then disables observers and freeze batchnorm stats after a few epochs.<br>
Is the enabling and disabling of the observers equivalent to NVIDIA’s offline calibration? Just that in PyTorch it is done during training?</p>
How to check if my code is currently JITed?

pkubik: <p>I don’t use JIT for the training of my model, but only during exporting. I have shape asserts and random sampling within my code so JIT complains that:</p>
<ul>
<li>assert, boolean conversion is not supported</li>
<li>the model outputs does not match over two multiple runs - because of randomness</li>
</ul>
<p>I would like to change the behavior of my forwards depending on whether they are exported or not. Is there a method or global variable that allows me to do this kind of check? I imagine something like:</p>
<pre><code class="lang-auto">if not torch.jit.is_jit_active():
    assert x.shape == (2, 2)
</code></pre>
<p>I would use something similar for sampling since I’m only interested in modes of the distribution during exporting.</p>

pkubik: <p>Just accidentally stumbled on the right function <code>torch.jit.is_tracing()</code>.</p>
<p>My linter says it’s not exported, so I’m not sure whether it’s safe to use but it says the same for <code>nn.Parameter</code> so there is probably nothing to worry about. Linters never liked Pytorch anyway.</p>
Indexing with sampled categorical

Sam_Lerman: <p>I might be doing something wrong, but when I try to index a tensor of <code>logits</code> with a <code>sample</code> from a <code>Categorical</code> of those logits, I expect <code>sample.shape == logits[sample].shape[:-1]</code>, but this is not the case.</p>
<pre><code class="lang-auto">dist = Categorical(logits=logits)
sample = dist.sample()
indexed_logits = logits[sample]
assert indexed_logits.shape == logits.shape[:-1]
</code></pre>
<p>Basically, I want to treat the categorical sample as indices.</p>
Replica of Harmonic Convolutions

fcola: <p>Hi everybody!</p>
<p>I am trying to replicate the Harmonic Convolution introduced in <a href="https://openreview.net/pdf?id=rygjHxrYDB" rel="noopener nofollow ugc">https://openreview.net/pdf?id=rygjHxrYDB</a>.<br>
Basically, I need a convolution that given a frequency bin omega, convolves normally over the time dimension (think image width) while processing the frequency bins (think image rows) given by (omega*k)n, where k and n are indexes that go from 1 to 7. As you can see, the concept is similar to dilated convolutions but spaced according to the harmonics of the frequency bin.<br>
According to the paper, fractional frequencies should be handled by bilinear interpolation. My dumb question is, thus:<br>
how should this be handled? Naively, I know I should use nn.interpolate, but I am not sure how to select the correct fractional bins from there. If it helps, the authors mentioned that their implementation is based on deformable convolutions.<br>
Thanks!</p>
Problem with Tutorial: "SPEECH COMMAND CLASSIFICATION WITH TORCHAUDIO"

ArcherX: <p>Hi, I am brand new to PyTorch and am currently working on the tutorials. I am currently working on this one: <a href="https://pytorch.org/tutorials/intermediate/speech_command_classification_with_torchaudio_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Speech Command Classification with torchaudio — PyTorch Tutorials 1.10.0+cu102 documentation</a></p>
<p>Basically, I am trying to get the model to predict spoken words, however, all the predictions are wrong. What exactly am I doing wrong? I have run the model for 20 epochs and also tried to run it for 30 epochs, the results are the same. I used Google Colab to run the tutorial:</p>
<p><a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/63ef278c9730746362d08162a440df77/speech_command_classification_with_torchaudio_tutorial.ipynb#scrollTo=IvRnJfS4-4Mb" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a>.</p>
<p>My Output is the following:</p>
<p>Expected: backward. Predicted: right.<br>
Expected: bed. Predicted: backward.<br>
Expected: bird. Predicted: backward.<br>
Expected: cat. Predicted: backward.<br>
Expected: dog. Predicted: backward.<br>
Expected: down. Predicted: backward.<br>
Expected: eight. Predicted: backward.<br>
Expected: five. Predicted: backward.<br>
Expected: follow. Predicted: backward.<br>
Expected: forward. Predicted: backward.<br>
Expected: four. Predicted: backward.<br>
Expected: go. Predicted: backward.<br>
Expected: happy. Predicted: backward.<br>
Expected: house. Predicted: one.<br>
Expected: learn. Predicted: backward.<br>
Expected: left. Predicted: backward.<br>
Expected: marvin. Predicted: one.<br>
Expected: nine. Predicted: backward.<br>
Expected: no. Predicted: backward.<br>
Expected: off. Predicted: backward.<br>
Expected: on. Predicted: backward.<br>
Expected: one. Predicted: backward.<br>
Expected: right. Predicted: backward.<br>
Expected: seven. Predicted: backward.<br>
Expected: sheila. Predicted: backward.<br>
Expected: six. Predicted: backward.<br>
Expected: stop. Predicted: backward.<br>
Expected: three. Predicted: backward.<br>
Expected: tree. Predicted: backward.<br>
Expected: two. Predicted: backward.<br>
Expected: up. Predicted: backward.<br>
Expected: visual. Predicted: backward.<br>
Expected: wow. Predicted: backward.<br>
Expected: yes. Predicted: backward.<br>
Expected: zero. Predicted: backward.</p>
Time taken for forward propagation and backpropagation in PyTorch

Sohan_Patnaik: <p>Is there any time complexity analysis depending on number of parameters for forward propagation and backpropagation? In simpler terms, given an architecture, can we comment whether forward propagation takes larger amount of time or backpropagation takes larger amount of time in PyTorch implementation?</p>

JuanFMontesinos: <p>Number of parameters is not determinant.<br>
You can re-use a network as many times as u want, which implies a more complex gradient computation.<br>
Besides gradient computation depends on the function used. There are gradients cheaper to compute than others.</p>
Torch distributed data-parallel vs Apex distributed data-parallel

GeorgeQ-Q: <p>My question is<br>
What are the differences between Apex distributed data-parallel with torch ddp since mixed precision training is also inside pytorch now is there any specific reason we use apex ddp in stead of torch ddp?</p>

ptrblck: <p>The apex implementations are deprecated, since they are now supported in PyTorch via their native implementations, so you should not use apex/DDP or apex/AMP anymore.<br>
<a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">This post</a> explains it in more detail.</p>

c_cj: <p>Hi, I am using pytorch ‘1.10.0+cu113’.<br>
The DDP model can work properly in apex DDP, but it causes deadlock in torch DDP after I save the model.<br>
I have ensured that the model only is saved by local=0.</p>

ptrblck: <aside class="quote no-group" data-username="c_cj" data-post="3" data-topic="121472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/c_cj/48/43936_2.png" class="avatar"> c_cj:</div>
<blockquote>
<p>but it causes deadlock in torch DDP after I save the model.</p>
</blockquote>
</aside>
<p>Does this mean that you are able to successfully save the model, but the next iteration hangs?<br>
Could you add print statements to the code to check where exactly the script is hanging?</p>

c_cj: <p>Thanks for your reply. I have solved this problem. It is caused that I have run a partial dataloader only in local_rank=0 for a temporary evaluation. It seems that all dataloaders in different processes must be in the same state.</p>
Libtorch bmm return 0s on device

3yuan: <p>When using bmm, it can output right result on cpu, but always 0s on device, which is not expected. the code is as follows:<br>
void test_bmm() {<br>
auto t1 = torch::randn({1, 3, 4});<br>
auto t2 = torch::randn({1, 4, 2});<br>
auto r = torch::bmm(t1, t2);<br>
std::cout &lt;&lt; r &lt;&lt; std::endl; // Right</p>
<p>t1 = t1.cuda();<br>
t2 = t2.cuda();<br>
auto r2 = torch::bmm(t1, t2);<br>
std::cout &lt;&lt; r2 &lt;&lt; std::endl; // Always 0s<br>
/*<br>
(1,.,.) =<br>
0  0<br>
0  0<br>
0  0<br>
[ CUDAFloatType{1,3,2} ]<br>
*/<br>
}</p>
<p>I want to know what’s wrong with it?<br>
Thanks</p>

ptrblck: <p>Were you able to use the GPU before in <code>libtorch</code> and if so, did you change anything in your setup (e.g. a driver update etc.)?<br>
Also, are you able to properly use the Python frontend or does it also return invalid values?<br>
Which PyTorch version and device are you using?</p>

3yuan: <p>I can use GPU, such as vpf and gpu path tracing (with libtorch, they are in the same project). The driver isn’t updated.<br>
It is in c++ environment, so I didn’t use the Python front end.<br>
libtorch version: 1.8.1, device: 1080Ti.<br>
And also, bmm is ok in another environment, which has a different driver version and doesn’t include gpu path tracing.<br>
It also doesn’t work for tensor add etc.</p>

ptrblck: <p>Thanks for the update.<br>
Given that you are using <code>1.8.1</code> and a 1080Ti (sm_61), I think you are hitting <a href="https://github.com/pytorch/pytorch/issues/56747">this issue</a>, which is already fixed in later releases. Could you update <code>libtorch</code> and rerun your workload?</p>

3yuan: <p>It works with libtorch 1.9.0 now. Thanks</p>
How can I obtain GPU utilization/memory metrics during training in a fast way?

dazzle-me: <p>Hello everyone, I’ve been thinking on how to obtain GPU performance metrics in the fast way, however, I couldn’t find a way to do so (I’m training ResNet-50 on CIFAR-10, nothing fancy)</p>
<p>The function I use to get GPU utilization and memory utilization values is the following :</p>
<pre><code class="lang-auto">def collect_gpu_statistics():
    gpu_statistics = !nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv
    gpu_util, memory_util = [int(x) for x in gpu_statistics[1].split(' ')[::2]]
    return gpu_util, memory_util
</code></pre>
<p>and I call it like that in the training loop :</p>
<pre><code class="lang-auto">for train_step, (image, label) in enumerate(train_dataloader):
        optimizer.zero_grad()
        image = image.cuda()
        label = label.cuda()
        
        prediction = model(image)
        loss = criterion(prediction, label.squeeze())
        loss.backward()
        optimizer.step()
        scheduler.step()
        gpu_util, memory_util = collect_gpu_statistics()
        ### do something with these metrics
        
        ## since the reduction is happening in CrossEntropyLoss itself
        average_meter.update(loss.item(), 1)
</code></pre>
<p>However, it slows down my code from 16s per epoch to &gt;40s per epoch, which is unacceptable<br>
Any best practices on this topic?</p>
<p>P.S.<br>
I’m using jupyter-notebook for the training, that’s why you see “!nvidia-smi …” in collect_gpu_statistics() function</p>

vadimkantorov: <p>Memory stats are available from <a href="https://github.com/pytorch/pytorch/pull/58635" class="inline-onebox" rel="noopener nofollow ugc">Expose `cudaMemGetInfo` by coreylammie · Pull Request #58635 · pytorch/pytorch · GitHub</a></p>
Am I doing torch.save wrongly the whole time?

Hongnan_G: <p>I have a long PyTorch custom pipeline, where I have been using for a year-ish. However, recently, I noticed that I called <code>self.model.eval()</code> everytime I save a model to state dict. This does not seem to be at all common and I am worried if this is wrong.</p>
<p>Snippet of code:</p>
<pre><code class="lang-python">    def save_model_artifacts(
        self,
        path: str,
        valid_trues: torch.Tensor,
        valid_logits: torch.Tensor,
        valid_preds: torch.Tensor,
        valid_probs: torch.Tensor,
    ) -&gt; None:
        """Save the weight for the best evaluation metric and also the OOF scores."""
        # self.model.eval() # I been calling this .

        torch.save(
            {
                "model_state_dict": self.model.state_dict(),
                "optimizer_state_dict": self.optimizer.state_dict(),
                "scheduler_state_dict": self.scheduler.state_dict(),
                "oof_trues": valid_trues,
                "oof_logits": valid_logits,
                "oof_preds": valid_preds,
                "oof_probs": valid_probs,
            },
            path,
        )
      def fit(...):
          train
          valid
          save
</code></pre>
<p>I did it for very long and did not notice inference issues (results look ok), but Just want to be sure if this is a potential issue and how to avoid it.</p>

JayHan: <p>calling eval may change the forward function behavior in the model, but not network parameter.<br>
If you appropriately call .train() or .eval() after loading the model parameter, it will be okay. However, seems that it is not usual to call eval() before saving the parameters.</p>
Spectral and inter-channel networks

kBodolai: <p>Hi everybody!<br>
I start this topic with the hope of having and enriching conversation about possible architectures, specially thinking about remote sensing segmentation applications on medium resolution, here’s the framing:</p>
<p>In many settings (say, mapping forests at 20-30m resolution) in remote sensing, the spatial context appears to be not-so-relevant, and the spectral information (pixel values) becomes much more important. So is the case, that it’s very common in remote sensing to use vegetation and other derived indices (such as NDVI). Such features are usually extracted beforehand.</p>
<p>It is also common at these resolutions to use multispectral instruments with ~12 bands, and even combine different sensors.</p>
<p>We could think intuitively then, that a network that could convolve across the channel dimensions to exploit some of the inter-band relationships, but 3D convolutions are usually expensive.</p>
<p>So here comes the point: what are good ways of exploiting this spectral information better, and efficiently?</p>
<p>I’m currently doing a literature review on this, and will be happy to post my findings as I advance if there’s interest, but would really love to hear people’s thoughts on this.</p>
<p>Cheers,<br>
K.</p>
Export metadata in torchscript model

Oli: <p>Hello! <img src="https://discuss.pytorch.org/images/emoji/apple/surfing_man.png?v=10" title=":surfing_man:" class="emoji" alt=":surfing_man:"></p>
<p>I was thinking that it could be useful to save metadata about a model when you export it via torchscript. Example of such metadata could e.g. be the git-hash that the model was trained under.</p>
<p>Some example code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Mymodel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(10, 10, 1, 1, 0, bias=True)
        self.accuracy = 0

    @torch.jit.export
    def model_info(self):
        return {
            "git_hash": "*git-hash-info*",
            "created": "*timestamp*",
            "accuracy": f"{self.accuracy}",
            "pytorch_version": "*version*",
        }

    def forward(self, x):
        x = self.conv(x)
        return x


model = Mymodel()
model.accuracy = 10
scripted_model = torch.jit.script(model)

model_path = 'testmodel.pt'
scripted_model.save(model_path)

loaded_model = torch.jit.load(model_path)
print(loaded_model)
print(loaded_model.model_info())
</code></pre>
<p>I think the practice could really help track your models in many systems. I haven’t seen the technique used before and would love some arguments against it or suggestion improvements.</p>
<p>What kind of information would be useful to embed in a torchscript model apart from what I’ve written above?</p>
Different learning rate of the same parameter

yuri: <p>I assign a different learning rate to the last conv2d layer of my model. the model_base parameters should be learned with the default lr. the tricky part is that the output of the model base will be used as a kernel to the last cov2d layer. since this output is updated with default lr I am afraid that this same lr will be inherited to the kernels of last conv2d layer and not the assigned one. the problem is that during training there is no way to consult the lr of a given parameter. I hope the sketching below could clarify better the idea.<br>
would the kernel inhirit the new lr assigned to the con2d layer ? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f0f67d98eb618a3d61c5206d7f0028a83565ace.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4f0f67d98eb618a3d61c5206d7f0028a83565ace" title="258860049_369171231561752_5616252379956804720_n"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f0f67d98eb618a3d61c5206d7f0028a83565ace_2_690x328.jpeg" alt="258860049_369171231561752_5616252379956804720_n" data-base62-sha1="bhoJ3elUGXKRSp4pRD9xfcchKO2" width="690" height="328" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f0f67d98eb618a3d61c5206d7f0028a83565ace_2_690x328.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f0f67d98eb618a3d61c5206d7f0028a83565ace_2_1035x492.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f0f67d98eb618a3d61c5206d7f0028a83565ace_2_1380x656.jpeg 2x" data-dominant-color="9F9E9B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">258860049_369171231561752_5616252379956804720_n</span><span class="informations">1646×784 168 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

my3bikaht: <p>Learning rate is just a value which sets limits on how significantly should you adjust the weights during training step. It doesn’t intersect with kernel values.</p>
Issue with Pytorch-Geometric (clustering) and Batches

Hello_World: <p>Hi *,</p>
<p>I’m a bit new to pytorch-geometric and it seems that there is something I’m missing when processing a batch using one the methods in torch-cluster.</p>
<p>My tensor shape is  [batch_size, 10, 8] so 10 data points with 8 features per batch (it actually post the embedding stage do these are embedding vectors).</p>
<p>Next, I wish to try and find the closest ones in this embedded space so I could focus the training on these. But, I cannot succeed in running over the batched tensor.</p>
<p>I’m using</p>
<pre><code class="lang-auto">from torch_cluster import radius_graph

x = model(input_data)
# [x] = [batch_size, 10, 8]
radius_graph(x, r=1) --&gt; Fails

radius_graph(x[0], r=1) --&gt; Works
</code></pre>
<p>I cannot understand the batch parameter the function is asking for, and its always asserting over the dimensions of the input tensor.</p>
<p>I would expect to get a returned value of</p>
<pre><code class="lang-auto">[batch_size, 2, n_matched]
</code></pre>
<p>Any help would be appreciated. Thank you</p>
The loss of vgg reaches the ln(1/n) threshold in one round, how to solve it

misszhucanjie: <ol>
<li>
<p>When dealing with face classification tasks, prepare to use vggnet to implement face classification. The vggface data set is used. First, one hundred categories are classified, and the official vggnet model is used. Only the number of classifications in the last layer and the loss used are changed. The function is the cross-entropy loss function. The training is started, but after several trainings. The result is that the loss value directly reaches the threshold of ln(1/n), and the model will not continue to train.</p>
</li>
<li>
<p>The second question is, when you use vgg to implement migration learning, what is the effect of using the parameters that have been given by the official? I use migration learning to freeze the previous parameters and modify only the last layer, and find the recognition accuracy in the back The improvement is not very obvious. Finally, it has been over-fitted, but after the test set accuracy rate is 7%</p>
</li>
</ol>

my3bikaht: <p>The idea behind transfer learning for vision tasks is to assume that convolution filters are more or less universally applicable to different tasks or different distributions. So you freeze everything related to convolutional part and keep training just fully connected part. VGG has multiple dense layers, you need to let them train too.</p>
Loss function for multi-label classification with missing values

ortasa: <p>I am training a network to identify topics in text. Text can contain several topics. In some of the texts, I have missing labels for some of the topics (I do not have information if the text contains or not contains the topic ).</p>
<p>I use  MultiLabelSoftMarginLoss() as loss function.</p>
<p>I try to indicate the missing values with a special label and get bad results.</p>
<p>Do you have a recommendation for another loss function or a reference to a custom function that someone wrote? before I am trying to invent the wheel myself and write a custom function?</p>
<p>Thanks,<br>
Ortal</p>

ptrblck: <p>I don’t know, if there is a good implementation of this approach for this loss function, but instead of using a new “missing” label, you could also ignore the missing label.<br>
To do so, you could calculate the unreduced loss first with any label (including a missing label) via <code>reduction='none'</code> while creating the loss function. Afterwards you could multiply this unreduced loss with a mask to set the missing losses to zero, and reduce it e.g. via <code>mean()</code>.</p>

SofiaCP: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="114346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>uction='none'</code> while creating the loss function. Afterwards you could multiply this unreduced loss with a mask to set the missing losses to zero, and reduce it e.g. vi</p>
</blockquote>
</aside>
<p>Hi. I am doing exactly this for my application: calculating the unreduced loss and applying a mask to zero-out some of the values. The values I want to mask depend on the set of grount-truth labels, meaning that the masked values are different in each sample of a batch. Is calling .mean() on the masked tensor or should I somehow ignore the zero values when calculating the mean? I tried doing this</p>
<p><code>https://discuss.pytorch.org/t/use-tensor-mean-method-but-ignore-0-values/60170</code></p>
<p>but got worse results. Thanks!</p>
Train different model after each epoch for evaluation

pintonos: <p>Hi,<br>
I am trying to train a model that generates embeddings. I want to evaluate this model every epoch based on its output (embeddings). So I create a simple DNN which takes these embeddings as input and performs classification (also trains several epochs every evaluation step).</p>
<p>However, it seems PyTorch is not able to train a decoupled model during the training loop itself. I am always getting an <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code> error when training the simple DNN for evaluation. Training the DNN decoupled of the outer training loop for the embeddings works fine!</p>
<p>Is there a trick to train 2 models simultaneously, but with different computational graphs?</p>

pintonos: <p>–Closed–</p>
<p>Found a different solution.</p>
The propagation problem of the gradient encountered when tensor is partitioned and indexed

wyz: <p>I want to run the following demo to verify my idea, but a bug appeared during the operation, and the prompt said"leaf variable has been moved into the graph interior".Because the loaded image occupies a lot, I need to use the intermediate variable recon to accept the data that has been calculated.I don’t know how to correct this error so that the program can run and the gradient can be passed correctly. I hope you can help me, thank you.</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

img = Variable(torch.rand(4, 4).float(), requires_grad=True)
recon = Variable(torch.zeros(4, 4).float(), requires_grad=True)
kernel = Variable(torch.rand(2, 2).float(), requires_grad=True)

for i in range(2):
	for j in range(2):
		recon[i*2:(i+1)*2, j*2:(j+1)*2] = kernel * (img[i*2:(i+1)*2, j*2:(j+1)*2])

recon.backward(torch.ones_like(recon))
print(kernel.grad)
</code></pre>
What is the underlying implementation process of 2D convolution in libtorch?

dwwcqu: <p>I want to add other operations (which form is ciphertext) to convolution process. To do this, i need implement the underlying operations in 2D convolution. Such as, how to construct the data structure of input and output, how to deal with padding to get correct output?<br>
If anybody have direct and simple method or ideas, thank you very much for your messages.</p>

ptrblck: <p>To write a manual convolution you could use <code>nn.Unfold</code> and apply a matrix multiplication.<br>
The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html?highlight=unfold#torch.nn.Unfold"><code>nn.Unfold</code> docs</a> give you an example.</p>
How to know input shape of loaded model(weights)?

Sherzod_Bek: <p>I did several training with different input size [224,512,…].<br>
In inference I want to know input shape of each loaded model(weights) to get more accurate result.<br>
is it possible to know input shape of loaded(saved) model?</p>
<p>inference works with any shape but if I use exact input shape performance will be better. (in inference I don’t know which weights trained with which input shape)</p>

ptrblck: <p>There isn’t a way to check the used shapes during training if the model is able to accept variable input shapes. You could store the input shapes e.g. in the checkpoint along with the <code>state_dict</code> or store it as an attribute in the model code instead.</p>
NotImplementedError in forward while implementing NAS using nn.ModuleList()

bishalu: <p>Hello, I could really use some expertise.</p>
<p>I am getting a NotImplementedError with this code:</p>
<pre><code class="lang-auto">class Network(nn.Module):

    def __init__(self, layers):

        super(Network, self).__init__()
        
        model = nn.ModuleList()
        
        # NAS SETUP
        for layer in layers:
            if len(model) == 0:
                input_dim = 4
            model.append(nn.Conv2d(input_dim, layer['output_dim'], kernel_size=layer['kernel']))
            model.append(nn.SiLU(inplace=True))
            model.append(nn.MaxPool2d(4))
            model.append(nn.BatchNorm2d(layer['output_dim']))
            input_dim = layer['output_dim']
        
        #TODO: fully connected layer
        #model.append(nn.Flatten())
        #model.append(nn.Linear(1408, 5))

        self.model = model


    def forward(self, flow):
        output = self.model(flow)
        return output
</code></pre>
<p>I have no idea why. I could not find any support topics online to help either.</p>

ptrblck: <p><code>nn.ModuleList</code> acts like a Python <code>list</code> with the advantage to properly register all modules inside the parent model.<br>
However, you can’t call it directly and might want to use <code>nn.Sequential</code> instead or index the modules:</p>
<pre><code class="lang-python">for m in self.model:
    x = m(x)
</code></pre>
Big diff between quantized model and model with fake_quants's

bigtree: <p>After the QAT is done I saved the model with fake_quant (torch.quantization.convert(model_fp32_prepared)) and the quantized model (after torch.quantization.convert(model_fp32_prepared))</p>
<p>when doing inference, I notice a big difference between these two models (see below pic, left:model with fake_quant, right: quantized model ).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/3118b4ea159ff695407b59906222558715a1e3e3.png" alt="image" data-base62-sha1="70kks9NWaxxhwqgNsaQlKx4dvb5" width="496" height="76"><br>
Is there a way to reduce the diff?</p>

HDCharles: <p>can you give a repro? the issue could be any number of things, without more context its impossible to tell.</p>

bigtree: <p>Sorry. For some reason, I can’t share the repo.<br>
One thing is suspected is I replaced all the activation functions with RELU.<br>
Could that be an issue?</p>

HDCharles: <p>it certainly sounds like it, without a way to reproduce your result though its difficult to say much more.</p>
Is it possible to quantize only activations when using QAT?

chatterboy: <p>Hi, I’m trying to quantize only the activations of a model not its weights. I followed the quantization documentation for QAT provided by PyTorch. It is difficult to find a content what I want in the documentation.</p>
<p>Thanks.</p>

chatterboy: <p>PyTorch is supporting it as <code>default_activation_only_qconfig</code> in <code>qconfig.py</code>. I’m using PyTorch 1.6.0.</p>

HDCharles: <p>To add to that,</p>
<p>I’m unsure about the goal here, pretty much the only reason to do QAT is to learn the best weights when weight quantization is applied. QAT’s central thesis is about how you can propagate gradients across a discontinuous function that quantizes teh weights, so doing QAT without weight quantization is a bit of a mismatch.</p>
<p>If you only want to quantize the activations, you may want to consider using something more suited to it.</p>
Which activation suggested for QAT

bigtree: <p>For QAT, which kind of activation function is suggested to use? (RElu, Relu6, Hardthanh)</p>

HDCharles: <p>if you are going to use QAT and need an activation, its probably a good idea to fuse your op if possible. A list of fusions can be found here: <a href="https://github.com/pytorch/pytorch/blob/10411e356160e1d0f406a9cee435e51f95fa90fa/torch/ao/quantization/fx/fusion_patterns.py#L28" class="inline-onebox" rel="noopener nofollow ugc">pytorch/fusion_patterns.py at 10411e356160e1d0f406a9cee435e51f95fa90fa · pytorch/pytorch · GitHub</a></p>
Understanding torch.nn.LayerNorm in nlp

Wang-Yu-Qing: <p>I’m trying to understanding how <code>torch.nn.LayerNorm</code> works in a nlp model. Asuming the input data is a batch of sequence of word embeddings:</p>
<pre><code class="lang-auto">batch_size, seq_size, dim = 2, 3, 4
embedding = torch.randn(batch_size, seq_size, dim)
print("x: ", embedding)

layer_norm = torch.nn.LayerNorm(dim)
print("y: ", layer_norm(embedding))

# outputs:
"""
x:  tensor([[[ 0.5909,  0.1326,  0.8100,  0.7631],
         [ 0.5831, -1.7923, -0.1453, -0.6882],
         [ 1.1280,  1.6121, -1.2383,  0.2150]],

        [[-0.2128, -0.5246, -0.0511,  0.2798],
         [ 0.8254,  1.2262, -0.0252, -1.9972],
         [-0.6092, -0.4709, -0.8038, -1.2711]]])
y:  tensor([[[ 0.0626, -1.6495,  0.8810,  0.7060],
         [ 1.2621, -1.4789,  0.4216, -0.2048],
         [ 0.6437,  1.0897, -1.5360, -0.1973]],

        [[-0.2950, -1.3698,  0.2621,  1.4027],
         [ 0.6585,  0.9811, -0.0262, -1.6134],
         [ 0.5934,  1.0505, -0.0497, -1.5942]]],
       grad_fn=&lt;NativeLayerNormBackward0&gt;)
"""
</code></pre>
<p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html" rel="noopener nofollow ugc">document’s description</a>, my understanding is that the mean and std are computed by all embedding values per sample. So I try to compute <code>y[0, 0, :]</code> manually:</p>
<pre><code class="lang-auto">mean = torch.mean(embedding[0, :, :])
std = torch.std(embedding[0, :, :])
print((embedding[0, 0, :] - mean) / std)
</code></pre>
<p>which gives <code>tensor([ 0.4310, -0.0319,  0.6523,  0.6050])</code> and that’s not the right output. I want to know what is the right way to compute <code>y[0, 0, :]</code> ?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">batch_size, seq_size, dim = 2, 3, 4
embedding = torch.randn(batch_size, seq_size, dim)
print("x: ", embedding)

layer_norm = torch.nn.LayerNorm(dim)
y = layer_norm(embedding)
print("y: ", y)

out = (embedding - torch.mean(embedding, dim=2, keepdims=True)) / torch.sqrt(torch.var(embedding, dim=2, keepdims=True, unbiased=False) + layer_norm.eps)

print((out - y).abs().max())
# &gt; tensor(1.1921e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Wang-Yu-Qing: <p>I found the problem. I need to add <code>unbiased=False</code> in <code>torch.std(embedding[0, :, :])</code></p>
Image classification with new class

Myailab: <p>Hi there,<br>
I trained a ResNet34 model to classify the given images into three classes. Now I want to add a new class into my model. Do I need to train my whole model again on all four classes or is there any way I can just train my model on new class and get the predictions for all four classes?</p>
<p>Thanks</p>

KFrank: <p>Hi My!</p>
<aside class="quote no-group" data-username="Myailab" data-post="1" data-topic="137521" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/9de053/48.png" class="avatar"> Myailab:</div>
<blockquote>
<p>is there any way I can just train my model on new class and get the predictions for all four classes?</p>
</blockquote>
</aside>
<p>If you literally mean train your model with batches that consist<br>
solely of samples of the new class, then, no, this won’t work.  As<br>
it trains, your model will simply learn to predict your new class<br>
all of the time, regardless of the input, and, with respect to your<br>
single-class training set, it will always be right.</p>
<p>If you are asking something  else, please clarify.</p>
<p>Best.</p>
<p>K. Frank</p>

Myailab: <p>Thanks for the reply. Yes, I mean training the model with batches of samples of new class.<br>
I am wondering if there is any way to train the existing model (already trained on three classes) <em>further</em> on new class samples, and finally getting the model trained for all four classes.</p>
Julia in python pytorch

randino: <p>I installed julia in my computer and in google colab. but</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/824482f87c1b7922a118e5bf360f867f20e033f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/824482f87c1b7922a118e5bf360f867f20e033f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/824482f87c1b7922a118e5bf360f867f20e033f3.png" alt="image" data-base62-sha1="iAoR6K87D0G3DgLznl8hkNWrd2r" width="690" height="70" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">759×78 3.36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import Shearlab
reload("Shearlab")
n = 512;

#data = rand(n,n);

#sizeX = size(data,1);
#sizeY = size(data,2);

# Set the variables for the Shearlet trasform
#rows = sizeX;
#cols = sizeY;
#X = data; 
#nScales = 2;

#shearletSystem = Shearlab.SLgetShearletSystem2D(rows,cols,nScales);

#coeffs = Shearlab.SLsheardec2D(X,shearletSystem);

mat_fname_gt = pjoin('./sample_data/', 'barbara.jpg')
data_nopar = Shearlab.load_image(name, n);
data_par = Shearlab.load_image(name, n,n,1);
data_nopar = data_nopar[:,:,1];
data_par = data_par[:,:,1]
Shearlab.imageplot(data_nopar);

</code></pre>
<p>error : ModuleNotFoundError: No module named ‘Shearlab’</p>

ptrblck: <p>I’m not sure how the <code>julia</code> package relates to PyTorch. Could you explain your use case a bit more and how PyTorch interacts with it? Based on your code it seems you are not using PyTorch at all or am I missing something?</p>

randino: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I will decomposed image with shearlet in julia.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/JuliaPy/pyjulia">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/JuliaPy/pyjulia" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/011d91ae23ef3131558d6c2b93b0ce155c21809567a170588d681fc2112798ed/JuliaPy/pyjulia" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/JuliaPy/pyjulia" target="_blank" rel="noopener nofollow ugc">GitHub - JuliaPy/pyjulia: python interface to julia</a></h3>

  <p>python interface to julia. Contribute to JuliaPy/pyjulia development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>I would recommend to create an issue in the linked GitHub repository or ask the package authors in their discussion board, if available, as your use case is unrelated to PyTorch.</p>