akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Performance issues in Octave Convolution ResNet

username: miguelvr
cooked: <p>Hello pytorch community,</p>
<p>I’ve been working on an <a href="https://github.com/braincreators/octconv" rel="nofollow noopener">implementation</a> of <a href="https://arxiv.org/pdf/1904.05049v2.pdf" rel="nofollow noopener">Octave Convolution</a> in pytorch, and now I’m trying to benchmark it.</p>
<p>This new octave convolution layer is supposed to be a “drop-in” replacement for nn.Conv2d and according to the paper it should have better performance, both in accuracy and speed, than its vanilla counterpart.</p>
<p>In my implementation, I benchmarked the convolutions individually and the <code>OctConv2d</code> is indeed faster than <code>nn.Conv2d</code>. However, the same is not true for the <a href="https://github.com/braincreators/octconv/blob/oct-resnet152/benchmarks/models/resnets.py" rel="nofollow noopener">ResNet implementation</a>, which I modified from the original <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="nofollow noopener">torchvision implementation</a>.</p>
<p>The benchmark code can be found <a href="https://github.com/braincreators/octconv/blob/oct-resnet152/benchmarks/benchmark.py" rel="nofollow noopener">here</a></p>
<p>Any clue of why this may be?</p>
<p>Thanks!<br>
Miguel</p>

username: mehtadushy
cooked: <p>Hi</p>
<p>Were you able to resolve this?</p>
<p>I think with CUDA, the timings being measured by your code might not be accurate:<br>
(<a href="https://github.com/braincreators/octconv/blob/oct-resnet152/benchmarks/benchmark.py#L49" rel="nofollow noopener">https://github.com/braincreators/octconv/blob/oct-resnet152/benchmarks/benchmark.py#L49</a>)</p>
<p>You would either need cuda synchronize events or the autograd profiler (<a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964/2" class="inline-onebox">How to measure time in PyTorch</a>)</p>

username: kakarla_yochana
cooked: <p>Hii<br>
Can u please mention the challenges phased by octave convolution</p>

username: kakarla_yochana
cooked: <p>Hii<br>
can u please mention what is the methodology used for implementing octave convolution</p>

username: miguelvr
cooked: <p>You can check the implementation in the repository linked in the first post</p>

username: kakarla_yochana
cooked: <p>Can u share any proposed diagram for octave convolution</p>

username: miguelvr
cooked: <p>it is also in the repository, but you should read the paper for that.</p>

username: kakarla_yochana
cooked: <p>u didn’t mentioned any proposed diagram in the link</p>

username: kakarla_yochana
cooked: <p>Can u please send the  paper regarding octaveconvolution</p>
Exception: must run observer before calling calculate_qparams!

username: srohit0
cooked: <p>Running the tutorial at <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a> raises exception</p>
<blockquote>
<p>/usr/local/lib/python3.6/dist-packages/torch/quantization/observer.py in calculate_qparams(self)<br>
44         n_levels = 255.0<br>
45         if self.max_val is None or self.min_val is None:<br>
—&gt; 46             raise Exception(‘must run observer before calling calculate_qparams!’)<br>
47         max_val, min_val = self.max_val.item(), self.min_val.item()<br>
48         if max_val == min_val:</p>
<p>Exception: must run observer before calling calculate_qparams!</p>
</blockquote>

username: hx89
cooked: <p>Which version of PyTorch are you using? Seems there are some differences between the code you pasted and the master: <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py</a></p>
Non differentiable loss function in CNN

username: netaglazer
cooked: <p>I have a costume loss function:</p>
<pre><code class="lang-auto">def loss(p, y):
    fa= -np.dot(1-y), list(np.log(1-,p))
    l = (y * list(np.log(p))
    prod = (1-y) -  l 
    loss = fa+ np.min(prod)
    return loss
</code></pre>
<p>p - prediction<br>
y - target</p>
<p>i have a multi label problem, and i want to minimize the even when the prediction was right in only one label. (hence the minimun in my loss)<br>
the problem is, that this function is non differentiable</p>
<p>how can i use this function anyway?</p>

username: netaglazer
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
can you help me with this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Why is this non-differentiable? You don’t seem to be using non-differentiable operations?<br>
You want to reimplement it though so that it uses torch Tensors and only torch functions so that it can use the autograd engine.</p>

username: netaglazer
cooked: <p>i thought that the minimum operation is not differentiable.<br>
isnt it true?</p>

username: albanD
cooked: <p>The <code>argmin</code> operation is non-differentiable (it returns an integer number). But the operation that returns the min value is. The gradient is just 1 for the value that was selected and 0 for all the others.</p>

username: Mertens
cooked: <p>There’s a distinction here between mathematically differentiable and differentiable wrt. autograd’s internals, right?</p>
<p>That might be a small point of confusion.</p>

username: netaglazer
cooked: <p>i think this is actually the confusing point.</p>
<p>can you explain a little bit more?</p>

username: Mertens
cooked: <p>I can only hypothesize as to what autograd is doing in the background, but one can get a sense of why such a distinction exists by looking at a simpler example: relu.</p>
<p>One way to define relu is relu(x) = max(x, 0). This function isn’t analytically differentiable. However, at every point except 0, it is. In practice, for the purpose of gradient descent, it works well enough to treat the function as if it were differentiable. You’ll rarely be computing the gradient at precisely 0, and even if you do, it’s sufficient to handle things via a special case.</p>
<p>If you’re okay with the behavior of relu, it’s not too hard to generalize things to fit your min function. In general, you won’t be doing computation at the precisely the points that are problematic, and provided your underlying functions are differentiable, the max/min of the functions should be differentiable ‘enough’ to allow for optimization.</p>
<p>There’s some fancier math you could throw in to explain why this works, and I’m sure I could be explaining things better, but that’s the intuition.</p>
Get result after each node?

username: flynntax
cooked: <p>Correct me if this isn’t a JIT problem but I would like to see the resulting output after each node in a graph (traced graph specifically). Is there any way to do this easily? My current approach (which would only look at layers) is to modify the model to save each layer’s output and return that in the forward function but this isn’t ideal as I would like to look at each node and ideally not modify the model.</p>

username: driazati
cooked: <p>There is no way to do this today, we’ve had <a href="https://github.com/pytorch/pytorch/issues/21064">this issue</a> open for a while which describes what you want. Until we implement something like that storing the results manually is the only way to go. A cleaner way than adding a giant return to your <code>forward</code> may be to store intermediate results as attributes on the module, something like</p>
<pre><code class="lang-python">class X(nn.Module):
  layer1: torch.Tensor
  layer2: torch.Tensor

  def __init__(self):
    self.layer1 = None
    self.layer2 = None
    self.fc1 = nn.Linear(10, 10)
  
  def forward(self, x):
    self.layer1 = self.fc1(x)
    self.layer2 = self.fc1(self.layer1)

   return self.layer2

torch.jit.script(X())
</code></pre>
<p><code>torchvision</code> has a similar problem, they use <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/_utils.py">this class</a> as a workaround.</p>
Accessing gradients directly from the loss

username: TinfoilHat0
cooked: <p>Is there a way to access the gradients directly from the loss, without using an optimizer?</p>
<p>For example,</p>
<blockquote>
<p>inputs, labels = inputs.half().to(device=self.args.device, non_blocking=True), labels.to(device=self.args.device, non_blocking=True)<br>
outputs = self.model(inputs)<br>
minibatch_loss = self.criterion(outputs, labels)<br>
minibatch_loss.backward()</p>
</blockquote>
<p>Can I access the gradients from</p>
<blockquote>
<p>minibatch_loss</p>
</blockquote>
<p>somehow ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you use <code>autograd.backward()</code>, it populates the <code>.grad</code> field of all the leaf Tensors (all tensors created with requires_grad=True and <code>nn.Parameter</code>).<br>
So since you use an nn.Module, you can do:</p>
<pre><code class="lang-python">for p in self.model.parameters():
  print(p.grad)
</code></pre>
<p>Otherwise, you can use <code>autograd.grad</code> to get a list of Tensor for your inputs by calling <code>grads = autograd.grad(minibatch_loss, self.model.parameters())</code>.</p>
Mkl-dnn problem with pytorch. I run on a university cluster, so not able to change the mkl-dnn version

username: Ahammed_Sidheek
cooked: <p>python3: /pytorch/third_party/ideep/mkl-dnn/src/common/scratchpad.cpp:63: mkldnn::impl::global_scratchpad_t::global_scratchpad_t(size_t): Assertion `scratchpad_ != nullptr’ failed.<br>
./setup_venv_on_cluster.sh: line 21: 19312 Aborted                 (core dumped)</p>
Multi-dimension torch.take (tf.gather_nd)

username: ding3820
cooked: <p>Hi all,<br>
I tried some of the available API in Pytorch but I think none of them meet my requirement.<br>
The most similar API in Pytorch is torch.take but the input is in 1D. However, tf.gather_nd probably could work but I couldn’t find the same function in Pytorch.</p>
<p>Here’s the task:<br>
I have a 3D tensor in shape (B, H, K) as indices.<br>
I would like to extract tensor from a map in shape (B, N, M) with the indices above.<br>
The desired output should be in shape (B, H, KM).<br>
All the values in the indices tensor are within [0, N].<br>
That means every element in the indices is corresponding to one vector with shape (M) from the map and I want to query it for H * K times.<br>
The operation should run in batch mode. So the final output should be in shape (B, H, KM).<br>
The easiest way to solve this task is to use double for-loop. I’m wondering if there’s another way to solve it without for-loop.<br>
Here’s the example:</p>
<pre><code class="lang-auto"># indices (B, H, K)
# map (B, N, M)
# for H
for j in range(H):
    k_block = []

    # for K
    for i in range(K):
        # b, 1
        index = indices[:, j, i]
        k_block.append(map[range(B), index])

    # b, KM
    k_block = torch.cat(k_block, dim=-1) 
    ...
</code></pre>

username: ptrblck
cooked: <p>Indexing the tensor might work.<br>
I assume the initialization of <code>k_block</code> should be outside the first loop?<br>
Otherwise, you would throw away the first results.</p>
<p>Based on this assumption, this code should work:</p>
<pre><code class="lang-python"># Setup
B, H, K = 2, 3, 4
N, M = 5, 6

indices = torch.randint(0, N, (B, H, K))
x = torch.randn(B, N, M)

# Your approach
k_block = []
for j in range(H):
    

    # for K
    for i in range(K):
        # b, 1
        index = indices[:, j, i]
        k_block.append(x[range(B), index])

    # b, KM
k_block = torch.cat(k_block, dim=-1) 

# Indexing approach
x1 = x.clone()
res = x1[torch.arange(B).unsqueeze(1).unsqueeze(2), indices].view(B, -1)

# Test
print((k_block - res).abs().max())
&gt; tensor(0.)
</code></pre>

username: ding3820
cooked: <p>Hi ptrblck,</p>
<p>Thank you for your fast reply!<br>
It did solve my problem. Didn’t expect it can be such easy!<br>
Thanks again.</p>
History of gradients in pytorch

username: AghaRezaey
cooked: <p>Hi<br>
I am doing some task very similar to translation</p>
<p>there is two nested for loop in picture below<br>
in j = 0<br>
I assigned some thing to h<br>
and doing some thing in inner loop<br>
then<br>
in in j = 1<br>
I assigned some thing else to h , and …</p>
<p>doesn’t j=1 step remove the history of gradients for j=0 (for back propagation time )??</p>
<p>if it is the case , how can i fix it ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9fe898e0aa36beb71c86119a96123c01dd13e90.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a9fe898e0aa36beb71c86119a96123c01dd13e90" title="Screenshot (108)_LI"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9fe898e0aa36beb71c86119a96123c01dd13e90_2_690x231.jpeg" alt="Screenshot (108)_LI" data-base62-sha1="ofQ2fb3YsD3sBWDIHfAxwUApkdy" width="690" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9fe898e0aa36beb71c86119a96123c01dd13e90_2_690x231.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9fe898e0aa36beb71c86119a96123c01dd13e90_2_1035x346.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9fe898e0aa36beb71c86119a96123c01dd13e90_2_1380x462.jpeg 2x" data-dominant-color="EBEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (108)_LI</span><span class="informations">1755×588 183 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>When you do <code>h = xxx</code> in python, you associate the python object <code>xxx</code> to the name <code>h</code>. And the old python object that <code>h</code> was pointing to is discarded.</p>
<p>That being said. The gradient computation for the different Tensors will still be computed properly. What is the exact issue you’re seeing here?</p>
Missing class in attention.pyi: MultiHeadAttention

username: Emmanuelle_Renauld
cooked: <p>Working with python3.7</p>
<p>In the terminal I can do: from torch.nn.modules.activation import MultiheadAttention</p>
<p>But in Pycharm, it says "cannot reference in ‘activation.pyi’. Indeed, I checked and MultiheadAttention exists in activation.py but not in activation.pyi.</p>
<p>Is that an error that should be reported?</p>

username: zhangguanheng66
cooked: <p>Seem like a general <a href="https://github.com/pytorch/pytorch/issues/15609" rel="nofollow noopener">issue</a>. Feel free to open an issue or comment under the existing issue. If you open a new issue, please include a code snippet to reproduce the error.</p>
Regarding the LSUN Dataset

username: justanhduc
cooked: <p>Could you please explain why we should write data to buffer like <a href="https://github.com/pytorch/vision/blob/9a22a8a750cbfecfae44adfb1c6e72991637f35f/torchvision/datasets/lsun.py#L46" rel="nofollow noopener">this</a>? I fail to see the advantage of doing so.<br>
Thanks</p>

username: ptrblck
cooked: <p>My best guess is, that this is one valid approach to create <code>PIL.Images</code> using the <code>lmdb</code> format of the LSUN dataset.</p>

username: justanhduc
cooked: <p>I think you are right. Thanks!</p>
Runtime Error in .backward()

username: fsattler
cooked: <p>Hi! The following code:</p>
<pre><code class="lang-auto">import torch
h_ = torch.nn.Parameter(torch.ones([1,1]))
h = h_**2
optimizer = torch.optim.SGD([h_], lr=0.1)

for _ in range(2):
  optimizer.zero_grad()
  loss = h.sum()
  loss.backward()
  optimizer.step()
</code></pre>
<p>throws a</p>
<p>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<p>Can anyone tell me whats the issue? Thanks!</p>

username: anantguptadbl
cooked: <p>Try this code</p>
<pre><code class="lang-auto">import torch

h_ = nn.Parameter(torch.ones([1,1]),requires_grad=True)
optimizer = torch.optim.SGD([h_], lr=0.01)

for _ in range(10):
    optimizer.zero_grad()
    hOut=h_**2 - h_*10
    loss = hOut.sum()
    loss.backward()
    optimizer.step()
    print(loss.item())
</code></pre>
<p>What I have done is inserted the calculation of hOut within the loop itself. Let us take a look at the changes in the graphical manner</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f09442a8df4711185cb81be5931d0b1b6c758eac.png" data-download-href="https://discuss.pytorch.org/uploads/default/f09442a8df4711185cb81be5931d0b1b6c758eac" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f09442a8df4711185cb81be5931d0b1b6c758eac_2_690x483.png" alt="image" data-base62-sha1="ykgePYhTj3wF9uBkzrW4Z34mQuE" width="690" height="483" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f09442a8df4711185cb81be5931d0b1b6c758eac_2_690x483.png, https://discuss.pytorch.org/uploads/default/original/3X/f/0/f09442a8df4711185cb81be5931d0b1b6c758eac.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/0/f09442a8df4711185cb81be5931d0b1b6c758eac.png 2x" data-dominant-color="E4EAF5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">783×549 18.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Since you were not recalculating H within each loop, it freed the buffer after the first LOSS Bakcward. The next loop it threw and error<br>
We included the the recalculation of H within the loop so that the buffer is refilled. Hope this helps.</p>

username: fsattler
cooked: <p>I see. That fixes the problem. Thanks alot! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Softmax output has zero grad

username: Tony_Lee
cooked: <p>I want to multiply two vectors <code>a</code> and <code>b</code> with different dimensions, and then send the product  vector <code>c</code>  into the objective function.<br>
For example, the demo code is as follows:</p>
<pre><code class="lang-auto">import torch

a=torch.rand(2,requires_grad=True)
b=torch.rand(4,requires_grad=True)
c=torch.cat((a*b[:2], b[4:]), dim=0)

d = torch.nn.functional.softmax(c, dim=0)
d.sum().backward(retain_graph=True)
print(a.grad)
print(b.grad)

c.sum().backward()
print(a.grad)
print(b.grad)
</code></pre>
<p>The outputs are as follows:</p>
<pre><code class="lang-auto">tensor([0., 0.])
tensor([0., 0., 0., 0.])
tensor([0.4431, 0.1512])
tensor([0.2158, 0.2565, 1.0000, 1.0000])
</code></pre>
<p>I don’t know why the grad of <code>a</code> and <code>b</code> are both zero if there is a softmax function. As shown above,  when I remove the softmax function, the grad of <code>a</code> and <code>b</code> are both correct.</p>

username: tom
cooked: <p>Because d is the output of a softmax, the sum of d is always 1 by definition no matter what you input. The gradient of a constant function is 0. You could try taking <code>c[0].backward()</code> for something more interesting.</p>
<p>Best regards</p>
<p>Thomas</p>
Convert model With torchScript differs from python version

username: erict
cooked: <p>Hi,</p>
<p>i’am trying to convert my model with torchscript and i get the following warning:</p>
<blockquote>
<p>TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:<br>
Not within tolerance rtol=1e-05 atol=1e-05 at input[4, 0, 35, 33] (0.406830757856369 vs. 0.40629732608795166) and 6114 other locations (18.00%)<br>
check_tolerance, _force_outplace, True, _module_class)</p>
</blockquote>
<p>The code is this</p>
<pre><code class="lang-auto">model=torch.load("...\\Model\\version1\\train0\\gray\\ConvAuto750.pth")
model.eval()
model.cuda()
example = torch.rand(8, 1, 64, 64).cuda()
traced_script_module = torch.jit.trace(model, example)
</code></pre>
<p>I want to use a batch size greater than one to speed up inference.</p>
<p>The strange thing is that if i use the model with default weights initialization parameters the above warning not appear even with batch_size greater than one.</p>
<p>This is the code:</p>
<pre><code class="lang-auto">model= simple_autoPrelu()
model.eval()
model.cuda()
example = torch.rand(8, 1, 64, 64).cuda()
traced_script_module = torch.jit.trace(model, example)
</code></pre>
<p>of course the model that i load has the same architecture as the one below. The only difference is that i have trained it for 750 epochs.</p>
<p>If i change the batch size from 8 to 1 the warning not appears.</p>
<p>Could someone provide me some explanations or some way on how to solve this warning?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Did you save the <code>state_dict</code> by chance?<br>
If not, could you load your model using your current script, store it via <code>torch.save(model.state_dict(), PATH)</code> and change your script to initialize the model and load the <code>state_dict</code>?</p>

username: erict
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>thanks for the response. I have tried but i got the same warning. Maybe i have some problems with my custom model, but is a simple Convolutional Autoencoder and i cannot understand what can cause this warning after doing the training. If you have other hints or tell me what can cause this warning, i can try to do some tests.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Would it be possible to post a code snippet to reproduce this error?</p>

username: erict
cooked: <p>Unfortunately i cannot share the code because it is proprietary. I have checked the inference accuracy of the two models converted with batch size = 1 which gives no warning and batch size = 8 the result is quite similar. I have checked the output pixel’s value in both cases. In the next days i will try to build a different version of the code with no proprietary restriction and if the error persists i will share the code.</p>
<p>Thanks.</p>
Cannot iterate through a dataloader

username: Geoffrey_Payne
cooked: <p>I get this error message: “Unable to open ‘hashtable_class_helper.pxi’: Unable to read file (Error: File not found (d:*\pandas_libs\hashtable_class_helper.pxi)).”<br>
The code is as follows;</p>
<blockquote>
<p>train_data_params = {‘batch_size’:512, ‘shuffle’:True, ‘num_workers’:0, ‘pin_memory’: True}<br>
train_dataset = data.DataLoader(data_batch.train_dataset, **train_data_params)<br>
for iter, batch in enumerate(train_dataset)</p>
</blockquote>

username: ptrblck
cooked: <p>This seems to be an error message from <code>pandas</code>. Is this the full error message?<br>
Could you iterate your <code>Dataset</code> (not <code>DataLoader</code>) once and make sure all keys are valid in your pandas dataframe?</p>
Why first forward pass takes very long time for nn.DataParallel model

username: Dhanachandra
cooked: <p>pytorch Version: 1.3.1<br>
System configuration: p3.8xlarge aws instance</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import time

x = torch.randn(2048, 2048).to("cuda")

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(2048, 1024)

    def forward(self, x):
        x = self.fc1(x)
        return x
#training:
net = Net()
net = nn.DataParallel(net, device_ids=[0, 1, 2, 3])
net.cuda()
for _ in range(10):
    net.train()
    tis =time.time()
    x = x.to("cuda")
    net(x)
    print(time.time() - tis)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">12.42794680595398
0.0023696422576904297
0.0019919872283935547
0.0018835067749023438
0.001916646957397461
0.001863718032836914
0.001901865005493164
0.001911163330078125
0.0018074512481689453
0.0018219947814941406
</code></pre>
How to apply L1 and l2 for PyTorch library for overfitting problem

username: AbuOmair
cooked: <p>Hi to all,<br>
I am a beginner and trying to learn PyTorch, I am facing that my training overfitting and I have tried once the L2 by applying it with optimizer SGD weight_decay = 0.4 and the I used Dropout with my model and I still face the accuracy too bad I don’t know why its bad.</p>
<p>Once I found that should apply the weights of each class. can anyone help me for that.?</p>
<p>Note I have 6 classes which are (0, 1, 2, 3, 6, 7) and<br>
class 0: 14626 samples<br>
class 1: 13010 samples<br>
class 2: 15902 samples<br>
class 3: 13337 samples<br>
class 6: 15079 samples<br>
class 7: 16403 samples</p>
<p>how can I apply the weights?</p>
<p>this is my training code</p>
<pre><code class="lang-auto">model = Model()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum = 0.9, weight_decay = 0.4)
criterion = torch.nn.CrossEntropyLoss()
num_epochs = 25

model.train()
for epoch in range(num_epochs):
    correct = 0
    for inputs,labels in train_loader:

        inputs = inputs.float()
        labels = labels.long()
        
        # Feed Forward
        outputs = model(inputs)
                
        # Loss Calculation
        loss_train = criterion(outputs, labels)
        L1_lambda = 0.5

        for name, param,*_ in model.parameters():# Applying for L1 Regularization
                    if 'weight' in name:
                        L1_1 = Variable(param, requires_grad=True)
                        L1_2 = torch.norm(L1_1, 6)
                        L1_3 = L1_lambda * L1_2
                        loss_train = loss_train + L1_3
        # Clear the gradient buffer (we don't want to accumulate gradients)
        optimizer.zero_grad()      

        # Backpropagation 
        loss_train.backward() 
        
        # Weight Update: w &lt;-- w - lr * gradient
        optimizer.step()
        
        #Accuracy
        _, predicted = torch.max(outputs, 1)
        labels1 = labels.long()
        correct += (predicted == labels1).float().sum()
        accuracy = (100 * correct / len(train_dataset))
     
    # Print statistics 
    print("Epoch {}/{}, Loss: {:.3f}, Accuracy: {:.3f}".format(epoch+1,num_epochs, loss_train, accuracy))
</code></pre>
<p>Thanks in advance.</p>
Can we save computation for some entries in a tensor?

username: zchen
cooked: <p>Some operations are extremely costly, such as nn.BiLinear.</p>
<p>Can we have a mask for the input tensors where some of their entries get calculated and some does not?</p>
<p>For example,</p>
<pre><code>input_a = input_b = torch.ones(A, B, ..., Z)
net     = nn.BiLinear(Z, Z, Z)
mask    = torch.rand(A, B, ..., Y, 1) &gt; 0.9 # only a fraction needs to be calculated.
output  = net(input_a, input_b, mask)
# unmasked entries may yield 0 or any other numbers, it's okay.

output.sum().backward()
# (mask xor input_a.grad) will show the pattern
</code></pre>
<p>I can now only think of a wrapper function getting scatter and gather involved.</p>
<p>Thanks in advance!</p>
How do you implement such multiplication

username: Mosquito_ya
cooked: <p>hello，</p>
<pre><code class="lang-auto">&gt;&gt;&gt; b = torch.tensor([1,2,3])
&gt;&gt;&gt; a = torch.tensor([1,2,3,4,5,6,7,8,9])
</code></pre>
<p>what should I do to make c = a* b  = [1,4,9,4,10,18,7,16,27]??<br>
Thanks!</p>

username: zhl515
cooked: <p>Maybe you should read tensor matrix operation first</p>
<pre><code class="lang-auto">c = (a.reshape(-1, b.shape[0])*b).flatten()
</code></pre>

username: Mosquito_ya
cooked: <p>Thanks！I do it in this way</p>
<pre><code class="lang-auto"> b.repeat(3).mul(a)
</code></pre>
<p>=-=</p>

username: V_Deamo
cooked: <pre><code class="lang-python">c = a.reshape(-1, b.shape[0]) * b
print(c.flatten())
</code></pre>
<p>I’d like to use the broadcast mechanic instead of extend the tensor manually, for I <strong>guess</strong> the broadcast implement in the torch or numpy will be more efficient.</p>
How to swap tensor elements and keep gradient

username: zchen
cooked: <p>Hi, I am new to PyTorch.</p>
<p>I want to swap elements in a 3D tensor which contains scattered non-zeros elements in shape (batch_size, seq_len, elem_dim), while keeping the gradient tracked. Given an example,</p>
<p>inputs = [[<strong>1 0 2</strong>]<br>
[<strong>0 3 0</strong>]<br>
[<strong>0 0 4</strong>]] # each bold integer represents a vector</p>
<p>outputs = func(inputs)<br>
assert outputs == [[<strong>1 2</strong>] [<strong>3 0</strong>] [<strong>4 0</strong>]]<br>
outputs.sum().backward # raise no errors</p>
<p>how to implement this function?</p>
<p>I come from TensorFlow knowing that this is somehow inconvenient and ineffective to do, but TensorFlow allows gradient tracking with scatter_fn and gather_fn.</p>
<p>If some swap function is provided in newest version of PyTorch, I think I do this.</p>

username: JuanFMontesinos
cooked: <p>I would say you can do that by modifiying tensor.data<br>
like:</p>
<pre><code class="lang-auto">import torch

a=torch.rand(10).requires_grad_(True)
dummy = torch.Tensor(list(range(10)))
optim = torch.optim.SGD([a],lr=1)

optim.zero_grad()

b=(a*a).mean()
print('FAKE A')
print(b.grad_fn)
a.data=dummy
print(b.grad_fn)
b.backward()

print(a.grad)
print('REAL A')
optim.zero_grad()
b=(a*a).mean()
b.backward()
print(a.grad)
</code></pre>
<pre><code class="lang-auto">FAKE A
&lt;MeanBackward0 object at 0x7f5d5c729710&gt;
&lt;MeanBackward0 object at 0x7f5d5c729710&gt;
tensor([0.1948, 0.1893, 0.0677, 0.0387, 0.0062, 0.1082, 0.1482, 0.1386, 0.0594,
        0.0293])
REAL A
tensor([0.0000, 0.2000, 0.4000, 0.6000, 0.8000, 1.0000, 1.2000, 1.4000, 1.6000,
        1.8000])
</code></pre>

username: zchen
cooked: <p>Sorry for the late reply. I tried scatter and confirmed that it is the function I was looking for.<br>
I was misled by the scatter_nd in TensorFlow which has a more scattered capacity.<br>
However, torch.scatter is just simple and handy with the gradients tracked.</p>
<p>Thank you!</p>
Convolution vs sum - floating point error

username: apisarek
cooked: <p>Hi all,</p>
<p>I am working with reimplementing a model in PyTorch and reusing pretrained weights. I have encountered errors which I thought are coming from floating point related problems.</p>
<p>I am providing a simple example which shows difference between convolution over input of ones vs summation of filter weights.</p>
<pre><code class="lang-auto">import torch
from torch.nn import functional as F

torch.backends.cudnn.deterministic = True 
torch.backends.cudnn.benchmark = False

torch.manual_seed(0)

inputs = torch.ones((1, 2, 2, 2))
filters = torch.randn((1, 2, 2, 2))

res1 = F.conv2d(inputs, filters).numpy()[0, 0, 0, 0]
res2 = filters.sum().numpy()

print(res1, res2, res1 - res2)

# -1.6045358 -1.6045356 -2.3841858e-07
</code></pre>
<p>Is it something expected or is the error too big for such a small operation?</p>

username: ptrblck
cooked: <p>The internal order of operations will most likely be different and e.g. a simple sum can also yield these floating point precision errors:</p>
<pre><code class="lang-python">filters = torch.randn((1, 2, 2, 2))
res1 = filters.sum()
res2 = filters.sum(0).sum(0).sum(0).sum(0)
print(res1 - res2)
&gt; tensor(2.3842e-07)
</code></pre>
Guided backprop - single ReLU module

username: Clundin
cooked: <p>Hi, I want to implement guided backpropagation and my model is currently using torch.nn.functional.relu. It looks like it is not easy to add a hook to this, so I am wondering if replacing the calls to F.relu could be replaced by calls to a single nn.ReLU() module for every layer. The backprop code would be implemented by:</p>
<pre><code class="lang-python">self.model.zero_grad()

    for module in self.model.modules():
        if type(module) == nn.ReLU:
            module.register_backward_hook((torch.clamp(grad_in[0], min=0.0), ))
</code></pre>
<p>Would using only one nn.ReLU() module that is reused at every layer be a problem? It seems like it would be fine to me, but I wanted to ask as I’m not sure how I’d discover if it was behaving as expected.</p>
<p>Thanks</p>

username: yyzharry
cooked: <p>Hi, I also encounter the problem. Have you found a way to do guided backprop with F.relu? Any suggestions or examples would be helpful.</p>
<p>Thanks</p>
Binary classification for single class

username: kunasiramesh
cooked: <p>Hi,<br>
I want to detect a cat image. Rest all images are i am treating as a non-cat. Do i need to treat this as a two class problem(should i use two nodes at the end?)?  Can I use only one node at the end and after model trained, use threshold to differentiate two whether image is cat or non-cat?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>To classify a cat, one node at the end of your model is enough, because 0 can represent non-cat and 1 for cats. To achieve this, you can add a <code>Sigmoid</code> layer at the end of your model, which will work as you explained about using the threshold for classification.</p>

username: kunasiramesh
cooked: <p>Hi Nikronik,</p>
<p>If I use single layer, I should depend upon threshold. Threshold tuning will consume lot of time.</p>

username: netty
cooked: <p>The difference between 2-node and 1-node has no “trainable” parameters. So to suggest that you’ll need to spend more time in the 1-node case tuning doesn’t really make sense.</p>
<p>With 2-nodes at the end, you can think of getting 2 sigmoid-ed scores, and what you do with those is up to you. A popular way is softmax of course, but if you have precision/recall constraints it seems to me you’re going to end up doing tuning either choice of architecture.</p>
MultiLabelMarginLoss max operation

username: netaglazer
cooked: <p>i see that the MultiLabelMarginLoss loss function has a max operation:</p>
<blockquote>
<p>loss(x, y) = sum_ij(max(0, 1 - (x[y[j]] - x[i]))) / x.size(0)</p>
</blockquote>
<p>how is it possible when max operation isnt derivative?</p>

username: beaupreda
cooked: <p>I believe this thread answers your question (<code>max</code> is an identity function for the maximum element).<br>
<a href="https://discuss.pytorch.org/t/confused-about-torch-max-and-gradient/14283">Confused about torch.max() and gradient</a></p>
Diagnosing bias / variance issues with dropout

username: netty
cooked: <p>In creating and running a few networks with Torch now, one issue I run into is that networks with Dropout tend to have test/validation accuracies higher than train.</p>
<p>I suppose this is normal, we are handicapping the network with a few layers that are unreliable, but this does make it harder to diagnose if problems in performance in training over time are due to 1) bias (underfitting), or 2) variance (overfitting).</p>
<p>Does anyone have recommendations for dealing with this?</p>
<p>One solution I might have would be after each epoch to not only evaluate the model (in <code>model.eval()</code> mode) the validation score, but also the training score, so that Dropout isn’t applied, but we see the “true” score on the training set.</p>
<p>Anyone have suggestions for such situations? Thanks!</p>
Layernorm, implementation?

username: gslaller
cooked: <p>Should not the following code be <code>y1==y2</code>?</p>
<pre><code class="lang-auto">x = torch.rand(64, 256)
model = nn.LayerNorm(256, elementwise_affine = False)

y1 = model(x)

mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True)
y2 = (x-mean)/torch.sqrt(var+model.eps)
</code></pre>

username: gslaller
cooked: <p>So the answer is:</p>
<pre><code class="lang-auto">...
var = torch.mean((x-mean)**2, -1, keepdim = True)
...
</code></pre>
<p>I got the variance wrong.</p>
Create multiple training dataloaders while one batch from the same dataloader

username: HomerNee
cooked: <p>Hi, everyone!</p>
<p>Supposed I have three kinds of datasets, I want to use all of them for the model training.</p>
<p>Now I hope that for a given batch, its data should be from the <strong>same</strong> dataset. But different batches can be randomly from different datasets.</p>
<p>For example, the first batch is from dataset A and the second batch is from dataset B, and so on.</p>
<p>Does anyone have some ideas to implement it?</p>
<p>Thanks for your time!</p>
Training siamese and triplet networks: stacking vs multi pass

username: dkajtoch
cooked: <p>Hi,<br>
I am starting to explore the topic of siamese and triplet networks, where the same model appears in the loss at least two time. Think about siamese network and data loader that produces two images. Each image should be passed through the same network and loss is e.g. the euclidean distance between embeddings. Initially, I thought that If I have two images I can stack them (to form larger batch) and pass it through a single model and then at the end recover separate outputs. Right now I am not pretty sure if backprop would work correctly since I do not see how my setup knows that there are two models (gradient is definitely different). My question is: In the case of siamese and triplet network do I need to treat pair/triple images separately and pass them 2/3 times through the model or will stacking actually work? Second part: if stacking does not work, then what I would have to modify so that the equations match (it is desirable to pass single batch and then form e.g. triplets after the pass in online semi-hard triplet mining). Tensorflow example does not seem to care that there are 3 networks in triplet problem (<a href="https://www.tensorflow.org/addons/tutorials/losses_triplet" rel="nofollow noopener">https://www.tensorflow.org/addons/tutorials/losses_triplet</a>)</p>

username: Yaroslav_Bulatov
cooked: <p>PyTorch keeps track of activations – you can reuse the same network for different inputs and/or losses and things work as expected. For instance, <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">official GAN example</a> does this</p>
<p>One thing to keep in mind is that layer.weight.grad accumulates gradients it has seen, but this is normally what you want. You can use hooks to print what autograd sees during interleaved forward/backward passes – <a href="https://colab.research.google.com/drive/1xC0Wx4D9FUuz-Wo3uEuAz40s1gPgRihd#scrollTo=Z88C83mAjx05" rel="nofollow noopener">example colab</a></p>
Something is slowing down the iteration?

username: Huimin_ZENG
cooked: <p>Hi, I got confused by observing some inconsistency regarding the time cost due to certain operations.</p>
<p>Case 1:</p>
<pre><code class="lang-auto">acc_start = time.time()
acc = torch.mean((predicted_labels==targets).float()).item()
train_acc_his.append(acc)
print("acc calculating time: ", time.time() - acc_start)
</code></pre>
<p>This returns <code>1.739304780960083</code>.<br>
However, I also printed the time for an entire iteration of mini batch like this:</p>
<pre><code class="lang-auto">for i, (inputs, targets) in enumerate(train_loader):
            iter_start = time.time()
            ...
           print("iter time: ", time.time()-iter_start)
</code></pre>
<p>, which returns around <code>2.210217237472534</code></p>
<p>Therefore, if I comment out the part for calculating the <code>acc</code>, the time for iteration should be <code>0.7</code>, right? However, I still got the time like <code>2.210217237472534</code>.</p>
<p>Is it wrong to test time like this?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>If you are using the GPU, note that CUDA operations are asynchronous, so that you should call <code>torch.cuda.synchronize()</code> before starting and stopping the timer.</p>

username: Huimin_ZENG
cooked: <p>Ah! I see! Thank you so much!</p>
Inference with alexnet and cv::imread image

username: spiovesan
cooked: <p>I am trying to infer with a C++ application an image classification task using an alexnet pre-trained net. I have successfully inferred a dog image loading the net with python:</p>
<pre><code class="lang-auto">alexnet = torchvision.models.alexnet(pretrained=True)
img = Image.open("dog.jpg")
transform = transforms.Compose([
 transforms.Resize(256),                
 transforms.CenterCrop(224),        
 transforms.ToTensor(),                  
 transforms.Normalize(                   
 mean=[0.485, 0.456, 0.406],         
 std=[0.229, 0.224, 0.225]              
 )])
img_t = transform(img)
batch_t = torch.unsqueeze(img_t, 0)
alexnet.forward(batch_t)
_, index = torch.max(out, 1)
</code></pre>
<p>Index is 208, Labrador_retriever, that looks good.<br>
Then I save the net to be loaded from a c++ application</p>
<pre><code class="lang-auto">example = torch.rand(1, 3, 224, 224)
traced_script_module_alex = torch.jit.trace(alexnet, example)
traced_script_module.save("alexnet.pt")
</code></pre>
<p>When I  load to C++, I get the wrong result:</p>
<pre><code class="lang-auto">cv::Mat img = cv::imread("dog.jpg");
cv::resize(img, img, cv::Size(224, 224), cv::INTER_CUBIC);

// Convert the image and label to a tensor.
torch::Tensor img_tensor = torch::from_blob(img.data, { 1, img.rows, img.cols, 3 }, torch::kByte);
img_tensor = img_tensor.permute({ 0, 3, 1, 2 }); // convert to CxHxW
img_tensor = img_tensor.to(torch::kFloat);
std::vector&lt;torch::jit::IValue&gt; input;
input.push_back(img_tensor);
torch::jit::script::Module  module = torch::jit::load("alexnet.pt");
at::Tensor output = module.forward(input).toTensor();
std::cout &lt;&lt; output.argmax(1) &lt;&lt; '\n';

</code></pre>
<p>the max is 463, bucket.<br>
I think the normalization is bad: how I can normalize the image to get the same 0…1 range as the transform does in the python version? Dividing by 255 gives the same result…</p>

username: ptrblck
cooked: <p><code>ToTensor</code> returns a tensor with values in the range <code>[0, 1]</code>, while <code>transforms.Normalize</code> will subtract the mean and divide by the standard deviation.<br>
Note that OpenCV will read the image channels as BGR, while PIL uses RGB, so you might need to convert it additionally.</p>

username: spiovesan
cooked: <p>Shame on me <img src="https://discuss.pytorch.org/images/emoji/apple/hot_face.png?v=9" title=":hot_face:" class="emoji" alt=":hot_face:">: I saved <code>traced_script_module</code> (a previous trained <code>resnet18</code> model in the same notebook) instead of<code> traced_script_module_alex</code> as <code>alexnet.pt</code>. Now it works…!</p>
Fast implementation of Gaussian Kernel

username: kolorado
cooked: <p>Hi,</p>
<p>I have a non-vectorized implementation of 2d gaussian:</p>
<pre><code class="lang-auto">def fillWithGaussian(inp,sigma,offset=0):
    center = ((inp.shape[1]-2)+offset)/2.
    inp[:,:,:].fill_(0)
    for i_ in range(inp.shape[1]):
        for j_ in range(inp.shape[2]):
            inp[:,i_, j_]+= 1 / 2 / np.pi / (sigma ** 2) * torch.exp(
            -1 / 2 * ((i_ - center - 0.5) ** 2 + (j_ - center - 0.5) ** 2) / (sigma ** 2))
</code></pre>
<p>But although it is differentiable and I want it to remain autograd compatible, it is really slow. Is there any suggestion?</p>

username: kolorado
cooked: <p>I Found something myself:</p>
<pre><code class="lang-auto">def return2DGaussian(resolution,sigma,offset=0):
    kernel_size = resolution

    # Create a x, y coordinate grid of shape (kernel_size, kernel_size, 2)
    x_cord = torch.arange(kernel_size).to(sigma.device)
    x_grid = x_cord.repeat(kernel_size).view(kernel_size, kernel_size)
    y_grid = x_grid.t()
    xy_grid = torch.stack([x_grid, y_grid], dim=-1)

    mean = ((kernel_size - 1)+offset)/2.
    variance = sigma**2.

    # Calculate the 2-dimensional gaussian kernel which is
    # the product of two gaussian distributions for two different
    # variables (in this case called x and y)
    gaussian_kernel = (1./(2.*math.pi*variance)) *\
                      torch.exp(
                          -torch.sum((xy_grid - mean)**2., dim=-1) /\
                          (2*variance)
                      )
    # Make sure sum of values in gaussian kernel equals 1.
    gaussian_kernel = gaussian_kernel / torch.sum(gaussian_kernel)

    # Reshape to 2d depthwise convolutional weight
    return gaussian_kernel.view( 1, 1, kernel_size, kernel_size)
</code></pre>
How to apply modified gradient to optimizer

username: tailintalent
cooked: <p>I want to add some Gaussian noise to the gradient, so that the gradient applied is the sum of the gradient computed by the graph, and my Gaussian noise. I tried using register_hooks, as the test code below:</p>
<pre><code>X = Variable(torch.FloatTensor([4]))
a = Variable(torch.FloatTensor([2]), requires_grad = True)
y = a * X
criterion = nn.MSELoss(size_average = True)
optimizer = torch.optim.Adam([a], lr = 1e-2)

h = a.register_hook(lambda grad: grad * 10)
optimizer.zero_grad()
loss = criterion(y, Variable(torch.FloatTensor([7])))
loss.backward()
optimizer.step()
print(a)
print(a.grad)
</code></pre>
<p>But I find that although I can modified the gradient at will, the optimizer always update the gradient using the original gradient, not the one after register_hook. Does anyone know how can we apply the modified gradient with the optimizer?</p>
<p>Thanks!</p>

username: smth
cooked: <p>if you modify the gradient, you have to <code>return</code> it from the hook. The returned gradient is used further. If you dont <code>return</code> a modified gradient, then the original gradient is used.</p>

username: SKYHOWIE25
cooked: <p>Hi</p>
<p>What do you mean by return it from the hook?</p>

username: smth
cooked: <aside class="quote no-group" data-username="tailintalent" data-post="1" data-topic="7728">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/7cd45c/48.png" class="avatar"> tailintalent:</div>
<blockquote>
<p>lambda grad: grad * 10</p>
</blockquote>
</aside>
<p>lambda grad: return grad * 10</p>

username: SKYHOWIE25
cooked: <p>But the lambda automatically return the expression right? And the lambda cannot contain statements like return.</p>

username: XavierXiao
cooked: <p>Try to use SGD. Adam has momemtum so you may need to run several iterations to see the difference</p>
Unresolved attribute reference 'backward' for class 'int' less

username: oasjd7
cooked: <p>Hi, all</p>
<p>How to solve this warning?</p>
<pre><code class="lang-auto">loss = torch.mean(torch.sum(pred* torch.log(pred)),1))
loss.backward()
</code></pre>
<p><code>Unresolved attribute reference 'backward' for class 'int' less</code></p>

username: albanD
cooked: <p>Hi,</p>
<p>What is the full warning, it seems to be truncated? I have never seen it before. Do you have a small code sample that reproduces it?</p>

username: oasjd7
cooked: <pre><code class="lang-auto">...
loss = 0
if cond1:
    loss += torch.mean(torch.sum(pred1* torch.log(pred1)),1))
if cond2:
    loss += torch.mean(torch.sum(pred2* torch.log(pred2)),1))
loss.backward()
</code></pre>
<p>As far as I know this is because the loss is initialized with <code>int</code>.</p>
<pre><code class="lang-auto">Unresolved attribute reference 'backward' for class 'int' less... (Ctrl+F1) 
Inspection info: This inspection detects names that should resolve but don't. Due to dynamic dispatch and duck typing, this is possible in a limited but useful number of cases. Top-level and class-level items are supported better than instance items.

</code></pre>

username: albanD
cooked: <p>Ho, it is a warning from the type checker? Because you define <code>loss=0</code> and if both conditions are False, you would call <code>.backward()</code> on an int.<br>
I guess you will need to either define loss as a Tensor to begin with, or only call backward if either cond1 or cond2 is met.</p>
Visualizing images using torchvision SummaryWriter

username: Israel_Abebe
cooked: <p>foloowing examples <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">here</a> encountered an error while doing</p>
<pre><code class="lang-auto"># log embeddings
features = images.view(-1, 28 * 28)
writer.add_embedding(features,
                    metadata=class_labels,
                    label_img=images.unsqueeze(1))
writer.close()
</code></pre>
<p>ERROR</p>
<p>AttributeError: module ‘tensorflow_core._api.v2.io.gfile’ has no attribute ‘get_filesystem’</p>