akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Train loss decreases very slowly with BCELoss()

goksu: <p>Hello!</p>
<p>I am trying to train a multi-label image classification model with an SGD optimizer, and my learning rate 0.0005. As you can see from the train/val loss figure below, the model’s losses are decreasing, but precision and accuracy scores are really bad, between 0.3-0.10.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/7047026f034dba3db639aee493fb95a4f0f96c7b.png" alt="loss_mlc_base" data-base62-sha1="g1fC3F6NL3bTD82iDA8lMnY0AmT" width="618" height="428"></p>
<p>To tackle the issue, I changed with different models (deeper and shallower) and various learning rates, and now train and val losses decrease very slowly. What is your opinion, or what should I do about it?</p>
<p>Thank you very much in advance,<br>
Best,</p>
<p>Oz</p>

fllci: <p>Hi Goksu,</p>
<p>Making the network deeper and learning rate smaller increases the training time and causes the model to make very small gradient updates which the model can stuck at a local minima. Making the network shallower and increasing the learning rate typically shortens the training time and your model makes larger gradient updates which it causes the model to have a general knowledge about the domain you are training with your model. You should find a good balance between shallow and deeper network. Accordingly, you may want to use Adam or AdamW as it is less sensitive to learning rate parameter.<br>
You can also incorporate skip connections as it diminish the vanishing gradient problem and I use this strategy a lot in fact. It helped me a lot. You may want to try new activation functions such as Mish etc. They demonstrated really good results compared to other functions e.g. ReLu.</p>

goksu: <p>Thank you very much for your help; I will try those technics.</p>
<p>All the best,<br>
Oz.</p>
The training speed becomes slower as the replay memory of transitions grows

Jemmie_Wang: <p>My DQN program runs slower as more transitions are stored in the replay buffer.</p>
<ol>
<li>It is not because of the sampling speed from the replay buffer, and it is the training speed that becomes slower, which involves intensive tensor calculations</li>
<li>For instance, when the system memory used for the thread is only 10% with still 50% unused memory (32G memory), the time for training increases by around 50%.</li>
<li>The problem exists on both CPU and GPU platforms.</li>
<li>The replay buffer only stores Numpy arrays</li>
<li>The training time will stop increasing once the replay buffer is full and starts to discard old transitions.<br>
Is there anyone meeting the problem before? I don’t find any solution to the problem on Google. Thank you!</li>
</ol>

vmoens: <p>Hi Jemmie</p>
<p>Sorry but without having a look at the code it is hard to tell anything. Is the code multi-threaded / multi-processed? Perhaps you could try to use a replay buffer where things are stored contiguously, I’ve found that this can help in the past (see our <a href="https://github.com/pytorch/rl/tree/main/torchrl/data/replay_buffers" rel="noopener nofollow ugc"><code>ReplayBuffer</code> in torchrl</a> which works with contiguous memory allocations through <a href="https://github.com/pytorch/rl/blob/417817abc12240cb8c93d767445fdec5f6c7050c/torchrl/data/replay_buffers/storages.py#L154-L212" rel="noopener nofollow ugc"><code>LazyMemmapStorage</code> and <code>LazyTensorStorage</code></a>.</p>

Andrew_Edmondson: <p>Oh, I hadn’t noticed this. That would have saved me some time! Is there also an equivalent lib in c++?</p>

vmoens: <p>Not yet, feel free to submit an issue with the description of what you’d like to see and we’ll give it a shot if we can!</p>
How to calculate diagonal elements of a hessian matrix in pytorch

Mrinmay_sen: <p>Hi,<br>
I am trying to calculate diagonal elements of a hessian matrix. I used the below code:</p>
<p>g—&gt;&gt; 1D tensor of gradients.<br>
diag=[]<br>
for i in range(len(g)):<br>
cnn.zero_grad()<br>
g[i].backward(retain_graph=True, create_graph=False, inputs=p)<br>
ele=p[i].grad<br>
diag.append(ele)</p>
<p>But it is taking long time.</p>
<p>Any suggestions how to calculate diagonal elements of a hessian in an efficient way.<br>
Thanks in advance</p>
Somebody help me to get the gradient values for each iteration , while training using SGD?

Sumesh_Sankar: <p>i need to get the gradient value of a loss function with respect to the weight(s) in a neural network while training it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65ed1407a9b74d127a9624d6e96b489280c3a8fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/65ed1407a9b74d127a9624d6e96b489280c3a8fd" title="query_pytorchforum"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65ed1407a9b74d127a9624d6e96b489280c3a8fd_2_690x341.png" alt="query_pytorchforum" data-base62-sha1="exGa3U5sS8keKT7DJVPtr3MTL0p" width="690" height="341" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65ed1407a9b74d127a9624d6e96b489280c3a8fd_2_690x341.png, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65ed1407a9b74d127a9624d6e96b489280c3a8fd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65ed1407a9b74d127a9624d6e96b489280c3a8fd.png 2x" data-dominant-color="FAFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">query_pytorchforum</span><span class="informations">1031×510 67.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

srishti-git1110: <p>Hi,<br>
After <code>loss.backward()</code>, you could loop over the <code>net</code> instance’s parameters and extract their <code>grad</code> attribute, like so:</p>
<pre><code class="lang-auto">for param in net.parameters():
    print(param.grad)
</code></pre>
Using Pretrained Swin v2 model

BRIJESH_SINGH: <p>I tried following code to import <strong><a href="https://pytorch.org/vision/main/models/generated/torchvision.models.swin_v2_s.html#torchvision.models.swin_v2_s" rel="noopener nofollow ugc">swin v2 small</a></strong> :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/608d312f82f49ba449be2a3d60df401a2febd095.png" alt="image" data-base62-sha1="dM8l4BRkh4FB57yNme9fX1h2KrP" width="348" height="64"></p>
<p>I got following error:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2556b25299991646bccb4c5bad1f7892e06e272.png" alt="image" data-base62-sha1="na4meo3qspzNExFQQuUp607W57c" width="671" height="60"></p>
<p>I tried same approach for <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.swin_s.html#torchvision.models.swin_s" rel="noopener nofollow ugc">swin_s</a>, which works out of the box. But when I am trying for <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.swin_v2_s.html#torchvision.models.swin_v2_s" rel="noopener nofollow ugc">swin_v2_s</a> I get above error.</p>

ptrblck: <p><code>SwinV2</code> was adde ~2 months ago in <a href="https://github.com/pytorch/vision/pull/6246">this PR</a> so you might need to update to a recent nightly binary in order to use it.</p>

BRIJESH_SINGH: <p>I installed nightly version using: <code>pip3 install --pre torch torchvision torchaudio</code>, which was listed <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">here</a> but still got the same error.<br>
Could you please suggest any specific nightly binary version?</p>

ptrblck: <p>It works for me with the latest nightly:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; torch.__version__
'1.13.0.dev20221003+cu117'
&gt;&gt;&gt; torchvision.__version__
'0.14.0.dev20221003+cu117'
&gt;&gt;&gt; torchvision.models.swin_
torchvision.models.swin_b(           torchvision.models.swin_transformer  torchvision.models.swin_v2_t(
torchvision.models.swin_s(           torchvision.models.swin_v2_b(        
torchvision.models.swin_t(           torchvision.models.swin_v2_s(        
&gt;&gt;&gt; torchvision.models.swin_v2_s
&lt;function swin_v2_s at 0x7f3274c30160&gt;
&gt;&gt;&gt; model = torchvision.models.swin_v2_s()
&gt;&gt;&gt; model
SwinTransformer(
  (features): Sequential(
    (0): Sequential(
      (0): Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4))
      (1): Permute()
      (2): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
    )
    ...
</code></pre>

BRIJESH_SINGH: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a> . Now, its also working for me.</p>
Index fill on 2d tensor, index and fill value

Fst_Msn: <p>Given a 2d zeros tensor to fill:</p>
<pre><code class="lang-auto">res = torch.zeros((4, 3))
</code></pre>
<p>a 2d tensor with the values to fill and the same dim 0 of res (e.g. 4, 5):</p>
<pre><code class="lang-auto">fill = torch.tensor([
[0.1, 0.2, 0.3, 0.4, 0.5],
[0.6, 0.7, 0.8, 0.9, 1.0],
[1.1, 1.2, 1.3, 1.4, 1.5],
[1.6, 1.7, 1.8, 1.9, 2.0],
])
</code></pre>
<p>and an index tensor of the same shape of fill (4, 5) and with index values in the range of dim 1 of res (0_2 in this case); with the max value (2) that can repeat more times per row and the other values that can appear 0 or 1 times per row:</p>
<pre><code class="lang-auto">index = torch.tensor([
[0, 2, 2, 2, 2],
[0, 1, 2, 2, 2],
[1, 2, 2, 2, 0],
[2, 2, 1, 2, 2],
], dtype=torch.int64)
</code></pre>
<p>I would like to fill res by row, given fill and index. I don’t care what values go into index 2 of res.</p>
<pre><code class="lang-auto">final_res = [
[0.1, 0.0, 0.3],
[0.6, 0.7, 0.9],
[1.5, 1.1, 1.4],
[0.0, 1.8, 1.6]
]
</code></pre>

ptrblck: <p><code>scatter_</code> should work:</p>
<pre><code class="lang-python">res = torch.zeros((4, 3))

fill = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5],
                     [0.6, 0.7, 0.8, 0.9, 1.0],
                     [1.1, 1.2, 1.3, 1.4, 1.5],
                     [1.6, 1.7, 1.8, 1.9, 2.0],
                     ])

index = torch.tensor([[0, 2, 2, 2, 2],
                      [0, 1, 2, 2, 2],
                      [1, 2, 2, 2, 0],
                      [2, 2, 1, 2, 2],
                      ], dtype=torch.int64)

res[index] = fill
res.scatter_(1, index, fill)
# tensor([[0.1000, 0.0000, 0.5000],
        # [0.6000, 0.7000, 1.0000],
        # [1.5000, 1.1000, 1.4000],
        # [0.0000, 1.8000, 2.0000]])
</code></pre>
Intermediate data processing for consecutive networks and backwarding

YajinX: <p>Hello to all! I have encountered a tricky problem.</p>
<p>I have two networks n1 and n2 to be trained. The structure of my model is something like</p>
<p>def Model():<br>
def forward(self):<br>
output1 = n1(input1)  # feed forward of network 1<br>
input2 = function()  # preparing input for network 2 containing iterations, which depends on output1<br>
output2 = net2(input2)  # feed forward of network 2</p>
<pre><code>def function(self):
    # iteration using self.output1
    # takes relatively long to run (a few minutes)

def backward(self):
    self.loss = loss1 + loss2  # losses based on output1 and output2
    self.loss.backward()
</code></pre>
<p>The net1 was pre-trained for a few epoches to ensure meaningful output, and everything was correct and running well. Then net2 was added, and two networks are supposed to be trained simultaneously. The observation is that loss.backward() takes too long (around 20 minutes). I assume it has something to do with the function() in the forward() method, because when I changed the iteration to a smaller size, the backwarding was much faster. However, for the complete model, I cannot decrease the iteration size.</p>
<p>So my question is, does the loss.backward() method somehow “re-runs” the function() method in forward()? If it is true, any suggestions to avoid this problem, e.g. changing the structure of my Model?</p>
<p>Many thanks for any help!</p>
Nested loop over two dataloaders

shashank_gupta1: <p>Hi,</p>
<p>I am training a model with two dataloaders in an alternate optimization fashion.</p>
<p>The algorithm looks like this:</p>
<pre><code class="lang-auto">def train(K: number of steps for loss2):
    for batch in dataloader1:
          optimize(loss1)  
           for step, batch in enumerate(dataloader2):
                  optimize(loss2)
                  if step &gt; K:
                     break
</code></pre>
<p>Now even with k=1, the training is very slow. I have checked the training time by removing the inner loop, and it finishes relatively quickly. Roughly, without a nested loop, it takes around 7 minutes to finish, and with a nested loop, it takes around 45 mins to finish. With higher k values, it’s much slower. I have tried restricting the number of workers on dataloader2 to 1, but it doesn’t make much of a difference.</p>
<p>Any suggestion on optimizing the runtime in such a setup?</p>

ptrblck: <aside class="quote no-group" data-username="shashank_gupta1" data-post="1" data-topic="160659">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shashank_gupta1/48/19577_2.png" class="avatar"> shashank_gupta1:</div>
<blockquote>
<p>Roughly, without a nested loop, it takes around 7 minutes to finish, and with a nested loop, it takes around 45 mins to finish. With higher k values, it’s much slower.</p>
</blockquote>
</aside>
<p>Wouldn’t this be expected assuming both <code>DataLoader</code>s have approx. the same number of elements?<br>
For each batch in <code>dataloader1</code> you are iterating the complete <code>dataloader2</code>.<br>
Assuming both <code>DataLoader</code>s have 7 batches (each one takes ~1min to load and process) you would use <code>(1+7)*7=56</code> iterations, thus ~56mins.</p>

shashank_gupta1: <p>Thanks for your response.</p>
<p>Sorry, I was a bit unclear in the question.</p>
<p>For each batch of dataloader1, I am sampling K steps (K varies from [1, 20]) only from dataloader2 and optimizing a loss.</p>
<p>Even for K=1, i.e. one batch from dataloader1 and one batch from dataloader2, it is taking a lot of time.</p>
Bucket Versioning using S3FileLoader

Dor_Biton: <p>Hi guys<br>
I couldnt find any support for versioning in this function.I gave a URI of s3 bucket, s3://mybucket.myjson to S3 as following:<br>
<code>S3FileLoader(['s3://mybucket.myjson'], region='eu-central-1', multi_part_download=None)</code><br>
and only the latest version of my object was streamed.</p>
<p>Is there a way to fix it? Or I need to open feature request?<br>
Thanks!</p>

nivek: <p>I do not believe it is possible right now. Let me open issues on GitHub to track this. Thanks for bringing this up.</p>

Dor_Biton: <p>Thank you, please tag me in order to track his issue. My github user is dorbittonn</p>
Different inference time for the same model, trained on different datasets

sam-motamed: <p>I have two (‘.pkl’) instances of a model, each trained on a different dataset. Inference time of one however is considerably (X15) higher than the other one. Any suggestions as to how this could be explained?</p>

sam-motamed: <p>I thought I found the issue but still no luck</p>

ptrblck: <p>In case you are using the CPU, try to use <code>torch.set_flush_denormal(True)</code> to check if this would change anything.</p>

sam-motamed: <p>Thanks for the suggestion. I am indeed running on GPU. Here is a bit more detail:<br>
I tested the model on two different machines, same GPU + environment. On one machine, it works as expected, with runtime of both saved models the same. On the other machine however, one program is much slower that the other!</p>
Getting the Individual Gradients of each Loss Function for Multiple Loss Functions

ryoukai: <p>I have a vector of different loss functions [loss1(model, inputs), …, lossN(model, inputs)]</p>
<p>I want to get the gradients of each loss function separately w.r.t. model parameters.</p>
<p>From looking at similar questions, I see that calling lossN.backward(), storing the gradients, and doing the same again, once for each loss function, is a possibility.</p>
<p>The problem is, my training approach relies on minibatch updates i.e. split a batch of data into several minibatches and for each minibatch, compute the loss, compute the gradients, update, rinse-repeat. The final gradients for a given loss function are the result of processing all minibatches.</p>
<p>Which means I’d have to do the whole minibatch process repeatedly, once for each loss function, which is inefficient, so I wanted to see if there was a better way.</p>
<p>Thanks!</p>
`flip` coordinates in F.grid_sample

qiminchen: <p>Hi, I happened to find this repo (<a href="https://github.com/ChrisWu1997/SingleShapeGen/issues/4" class="inline-onebox" rel="noopener nofollow ugc">`flip` when sampling features from plane · Issue #4 · ChrisWu1997/SingleShapeGen · GitHub</a>) and was having questions about flipping the coordinates in F.grid_sample. Here is the code snippet from the issue. Is this a bug or am I missing something? Why <code>flip(-1)</code> solve the issue in this case?</p>
<pre><code class="lang-auto">def make_coord(H: int, W: int, device="cpu", normalize=True):
    xs = torch.arange(H, device=device).float() 
    ys = torch.arange(W, device=device).float()
    if normalize:
        xs = xs / (H - 1) * 2 - 1 # (-1, 1)
        ys = ys / (W - 1) * 2 - 1 # (-1, 1)
    coords = torch.stack(torch.meshgrid(xs, ys), dim=-1)
    return coords

coords = make_coord(3, 4)
coords = coords.unsqueeze(0)
print(coords.shape)

arr = torch.zeros((3, 4))
arr[0, 1] = 1
arr = arr.unsqueeze(0).unsqueeze(0)
print(arr)
# tensor([[[[0., 1., 0., 0.],
#           [0., 0., 0., 0.],
#           [0., 0., 0., 0.]]]])

samples = F.grid_sample(arr, coords, align_corners=True)
print(samples)
# tensor([[[[0.0000, 0.0000, 0.0000, 0.0000],
#           [0.5000, 0.1667, 0.0000, 0.0000],
#           [0.0000, 0.0000, 0.0000, 0.0000]]]])
# wrong!

samples = F.grid_sample(arr, coords.flip(-1), align_corners=True)
print(samples)
# tensor([[[[0., 1., 0., 0.],
#           [0., 0., 0., 0.],
#           [0., 0., 0., 0.]]]])
# correct!
</code></pre>

ptrblck: <p>I guess the confusion might arise from the standard “x/y” coordinate system in image processing and the “row/column” layout used for general tensors/matrices as explained here: <a href="https://discuss.pytorch.org/t/solved-torch-grid-sample/51662/13" class="inline-onebox">[SOLVED]Torch.grid_sample? - #13 by ptrblck</a></p>
Debugging memory (RAM) buildup during training

kheyer: <p>Looking for help debugging RAM memory buildup during training.</p>
<p>Current situation:<br>
My dataset is a very large CSV<br>
I load 5m rows at a time<br>
I create a new dataset and dataloader<br>
I train on the dataloader<br>
After going through the data, I delete the loaded dataframe, the dataset and the dataloader and run garbage collection</p>
<p>System memory usage starts out at ~5GB and after a day and a half has risen to 90GB/128GB. I’d like this training loop to run for several days, so I’m concerned about the memory buildup.</p>
<p>I’m aware of <a href="https://github.com/pytorch/pytorch/issues/20433" rel="noopener nofollow ugc">previous posts on the topic of dataloader memory buildup</a>. I’ve tried passing my main inputs (a list of strings) as a numpy array, and I’m still seeing the buildup.</p>
<p>I’m also interested in figuring out why I get the buildup given that after each 5m data chunk, I explicitly delete the dataframe, dataset and dataloader. When this happens, there’s a noticeable memory release in the system, but the “reset” memory usage is still higher than the original memory usage, leading to the gradual buildup.</p>
<p>Is there a persistent multiprocessing Pool object that survives between dataloaders? I know those won’t release memory until the pool is explicitly closed.</p>
<p>Any thoughts on addressing this issue?</p>
How to set weight/bias in nn.quantized.Conv2d?

ZoltAndr: <p>After executing this code</p>
<pre><code class="lang-auto">conv = torch.nn.quantized.Conv2d(3, 64, (3, 3), dtype=torch.quint8)
weight = torch.randn(64, 3, 3, 3)
weight = torch.quantize_per_channel(weight, torch.ones(64)/200, torch.zeros(64), axis=0, dtype=torch.quint8)
conv.set_weight_bias(weight, torch.zeros(64))
</code></pre>
<p>the error occurs:<br>
<code>RuntimeError: Unsupported qscheme: per_channel_affine_float_qparams</code></p>

jerryzh168: <p>the api call is correct, but you’ll need to set the dtype for zero_point to int64 I think</p>
<pre><code class="lang-auto">weight = torch.quantize_per_channel(weight, torch.ones(64)/200, torch.zeros(64, dtype=torch.int64), axis=0, dtype=torch.quint8)
</code></pre>

ZoltAndr: <p>thank you. it solves the problem.</p>
Loss for sparse tensor

Max_Unhold: <p>Hello, I would like to create a simple loss function for two <a href="https://pytorch.org/docs/stable/sparse.html" rel="noopener nofollow ugc">sparse<br>
tensors</a></p>
<pre><code class="lang-python">def criterion_sparse(x,y):
      return torch.sparse.sum(
                      torch.pow(torch.subtract(x,y).coalesce(),2)
)
</code></pre>
<p>I get the error:<br>
NotImplementedError: Could not run ‘aten::is_coalesced’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend.</p>
<p>although all functions are listed as compatible with sparse tensors.<br>
What can I do?</p>
<p>Another broader question is: There is basically nothing implemented for the <code>torch.sparse</code> module. Neither <em>activation functions</em>, <em>loss functions</em>, a broad range of autograd supporting Pytorch functions, nor <em>loss</em> or linear layers. Does it make sense to work with <code>torch.sparse</code>, is there any other framework for pytorch or do we have to stick with tensorflow?</p>

KFrank: <p>Hi Max!</p>
<aside class="quote no-group" data-username="Max_Unhold" data-post="1" data-topic="162689" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/max_unhold/48/31337_2.png" class="avatar"> Max_Unhold:</div>
<blockquote>
<p>NotImplementedError: Could not run ‘aten::is_coalesced’ with arguments from the ‘CPU’ backend.</p>
</blockquote>
</aside>
<p>What version of pytorch are you using?  I cannot reproduce your specific<br>
issue on 1.12.0:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; x = torch.randn (3, 5).to_sparse().requires_grad_ (True)
&gt;&gt;&gt; y = torch.randn (3, 5).to_sparse().requires_grad_ (True)
&gt;&gt;&gt; torch.sparse.sum (torch.pow (torch.subtract (x, y).coalesce(), 2))
tensor(29.7927, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>Another broader question is:<br>
…<br>
Does it make sense to work with <code>torch.sparse</code></p>
</blockquote>
</aside>
<p><code>torch.sparse</code> is a work in progress with a number of things that are<br>
not (yet) implemented (although <code>torch.sparse</code> does more than your<br>
blanket statement would seem to suggest).</p>
<p>In general, I would not expect <code>torch.sparse</code> to work on “real”<br>
neural-network problems.  If your use case could potentially benefit<br>
from <code>torch.sparse</code>, it could be worth trying, but as you note, many<br>
features won’t work.</p>
<p>Best.</p>
<p>K. Frank</p>
Inference of pytorch model

dneprdva: <p>Hi guys,<br>
I tried to find solution by myself but not enough experience.<br>
I have such inference code for prediction:</p>
<pre><code class="lang-auto">test_loader = DataLoader(dataset=test_dataset, batch_size=1)

y_pred_list = []

with torch.inference_mode():
    model.eval()
    for X_batch in test_loader:
        X_batch = X_batch.to(device)
        y_test_pred = model(X_batch)        
        _, y_pred_tags = torch.max(y_test_pred, dim = 1)        
        y_pred_list.append(y_pred_tags.cpu().numpy())

y_pred_list = [a.squeeze().tolist() for a in y_pred_list]
</code></pre>
<p>The inference ‘y_test_pred’ gives tensor with 6 possibilities and torch.max takes max out of them.  If i put batch_size=1 everything is working great but very slow since the data is very huge.<br>
If I put batch_size=32 it is working fast but the inference ‘y_test_pred’  comes with additional dimension(32) and I can’t understand how to squeeze it or maybe to make torch.max later. Please, help.</p>

srishti-git1110: <p>What’s the error that you are facing when doing torch.max on a batch of tensors?</p>
<p>Also, please post the exact shape of <code>y_test_pred</code>.</p>

dneprdva: <p>There is no error. If I put batch_size=1, the inference shape is [1,6] and ‘torch.max’ takes max value out of 6. But if I put batch_size=32, the inference shape is already [32,6] and I don’t understand how to make ‘torch.max’ from this shape.</p>

srishti-git1110: <p>See if this helps -</p>
<pre><code class="lang-auto">x = torch.randn(32, 6) 
max_values, max_indices = torch.max(x, dim=1) # takes the maximum from each batch
print(max_values.shape, max_indices.shape)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">(torch.Size([32]), torch.Size([32]))
</code></pre>

dneprdva: <p>I don’t understand. I redid everything step by step and it started to work even with batch 32. Magic. Thank you very much. It is second time you help me.</p>
Model accuracy is different when loaded on different hardware

sparshgarg23: <p>I am currently doing a classification task and I noticed that if  I load my model on GPU my test accuracy is around 0.7982.<br>
When I load that same model on a cpu,the accuracy comes out to be 0.7894.<br>
Although the difference is about 8.8e-4,I would still like to know as to why this difference is coming.<br>
I had also included some code related to setting manual seed.As shown below.Does this have any impact on the changing accuracy?</p>
<pre><code class="lang-auto">  random_seed=42
    torch.manual_seed(random_seed)
    if torch.cuda.is_available():
        #set these parameters to ensure that model accuracy and loss remains same 
        #even when the same code is run multiple times.
        torch.cuda.manual_seed(random_seed)
        torch.backends.cudnn.deterministic=True
        torch.backends.cudnn.benchmark=False
    np.random.seed(random_seed)
</code></pre>
<p>I don’t think it should since those statements will only be executed if cuda device is available.<br>
Note on colab pytorch version is  torch==1.12.1+cu113<br>
But on my own system(CPU ) pytorch is 1.8.1</p>

ptrblck: <p>Small differences in the model output are expected between different platforms due to the limited floating point precision and the usage of different algorithms. Seeding the code wouldn’t help since different devices could use a different pseudorandom number generator.</p>

Ritvick_Gupta: <p>I have been using the torch 1.8.1 cuda 11.1 and the accuracy for googlenet was 72.78 but now migrating to newer version of torch 1.12.1 cuda 11.6 the accuracy is changing to 72.76.<br>
Can you help me figuring why this difference is coming?</p>

ptrblck: <p>You could try to narrow down which image prediction(s) are different between both models (in <code>eval()</code> mode) and then compare the intermediate outputs of the model as well as the model output to narrow down where the difference might be coming from.</p>
(Toy) Meta learning example not working

David_Ireland: <p>I am trying to implement a toy meta learning example. Below is code that demonstrates what I am trying to do:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

torch.random.manual_seed(1)


net = nn.Linear(1, 1, bias=False)
# print("net parameter is ", list(net.parameters())[0])
optim = torch.optim.SGD(net.parameters(), lr=1e-3)
meta_net = nn.Linear(1, 1, bias=False)
# print("meta net parameter is ", list(meta_net.parameters())[0])
meta_optim = torch.optim.SGD(meta_net.parameters(), lr=1e-3)

x = torch.randn((1, 1))
y = net.forward(x)
z = meta_net.forward(x)

optim.zero_grad()
loss = y * z
loss.backward(retain_graph=True)
for p in meta_net.parameters():
    print(p.grad)
optim.step()
# print("net parameter is ", list(net.parameters())[0])

x = torch.randn((1, 1))
z = net.forward(x)
meta_optim.zero_grad()
for p in meta_net.parameters():
    print(p.grad)
loss = z
loss.backward(retain_graph=True)
for p in meta_net.parameters():
    print(p.grad)
</code></pre>
<p>If we define the weight of <code>net</code> to be w1 and the weight of <code>meta_net</code> to be w2, then the first parameter update should make update the w1 parameter to be <code>w1' = w1 - lr * w2 * x^2</code>, i.e. a function of <code>w2</code>. Now, when I use the updated <code>net</code> parameter with weight <code>w1'</code> to obtain <code>z</code>, <code>z</code> should be a function of <code>w2</code>. However, when I back propagate through <code>z</code> the gradient of <code>w2</code> is 0. (the print functions confirm this). What is going wrong here?</p>
<p>Note that I was initially trying with <code>retain_graph=False</code> in the <code>.backward()</code> call and thought this may have been what was causing the issue, but alas it still does not work.</p>
How to apply softmax function on two tensors from two streams of convolutional layers to fuse the features?

Muhaddisa: <blockquote>
<p>I want to compute weighted sum on the feature vectors out of two convolutional layers to perform fusion of the features. Let F_1 and F_2 be the tensors of feature maps each of size (batch_size, 256, 4,4). I have to compute F = a_1 * F_1 + a_2 * F_2, where a_k is:</p>
</blockquote>
<p>ak = exp(w’_k * f_k)/exp(w’_1*f_1 +w’_2 * f_2)</p>
<blockquote>
<p>where w_k is the column scanned vector of W_k( the attention weighting modality for k_th modality), f_k is the column scanned vector for the vector F_k.</p>
</blockquote>