akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training Loss is fluctuating

danishnazir: <p>I am using CosineEmbeddingLoss with a margin= 0.35. My loss just keep fluctuating until 500 epochs and it does not smoothen out.</p>
<p>Here are my parameters</p>
<p>Adam: Momentum = 0.9,Weight Decay = 0.01, LR = 0.000001.</p>
<p>Batch Size = 64</p>
<p>The network is a simple Auto Encoder.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dcee0a6b88c636b33202540df339c1981fb6ce8.png" alt="Capture" data-base62-sha1="mw2c9fo30Cr2Laj01cahWgLHHaU" width="532" height="406"></p>
Train and test loss deviates from each other

jzy95310: <p>Hi there,</p>
<p>I’m doing a regression task with ResNet50 by feeding in 200x200 dimension images. However, when I plot out the loss, I see a really strange pattern because the train loss keeps decreasing while the test loss keeps fluctuating and sometimes increasing, which suggests my model generalizes very poorly (as shown in the figure below). Usually, I expect the test loss to at least go down a little bit and then go up, but here it just goes straight up. Did anyone observe this behavior before?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/7219b0804b82c96a2e68d420ca96eb497666dd84.jpeg" alt="Capture" data-base62-sha1="ghntcG5YVK4QZ8Q5h4nP7qL2Ooc" width="640" height="450"></p>
Change weights and back-propagation

oasjd7: <pre><code class="lang-auto">[Forward process]
model_1, model_2 = Model_1(), Model_2()
model_1_weights = model_1.conv1[0].weight.data # get weights from model_1
model_2_weights = model_2.conv1[0].weight.data # get weights from model_2

new_weights = model_1_weights * 0.9 + model_2_weights * 0.1

model_1.conv1[0].weight.data = new_weights # change weights 
model_2.conv1[0].weight.data = new_weights # change weights 

model_1.forward()
model_2.forward()
...
[Backward process]
'model_1 ' only affected by 0.9 from 'new_weights'?
'model_2 ' only affected by 0.1 from 'new_weights'?
If not, how can I implement this ?
</code></pre>
How to access gradients wrt each element in latent space vector z?

Nymue: <p>Hello, I’m building a VAE and I am trying to implement a way to compute the gradients of my encoder network activations wrt each element in the latent vector z. My encoder network outputs a mean and standard deviation of size [1,latent_size] for each image in the batch, which I then reparameterize to a vector z for each image. I’ve implemented hook functions that save the gradients for each layer, which have of course the feature dimension of that particular layer. Now I want to know how I can get from this to the gradients for each element in the latent vector z. I’ve looked at many repositories online, but they always compute the gradients wrt the full vector z, so I’m a bit lost on how to get to where I want.</p>
<p>Could someone maybe help me out? I’m quite new to pytorch so if I’m not clear or if there’s information missing, please let me know!</p>
Weird behaviour after resume training

Jia_Ming: <p>I encounter a weird situation when resume training. Let’s say I have 10 epochs. From 1st epoch, the train loss decrease steadily whereas the valid loss is floating around 1.8-2.0. At 5th epoch, the training loss is 0.95 with validation loss of 1.7. I save this epoch with model state dict and optimizer state dict. The first epoch when I resume training is that the training loss rise a bit, to 1.05, whereas the validation loss drop to 1.1.  After that, the training loss continuously drop, whereas the validation loss constantly rise to 1.6.</p>
<p>My question is, I have save the state dict of model and optimizer, why would the valid loss is not following the trend of the previous when I resume training. My expectation of the valid loss after resume training is that it will drop from 1.7, but the fact is it drop to 1.1 and rise again back to 1.6.</p>
<p>I use adam optimizer with cross entropy loss. What would the issues causing this problem?</p>

CedricLy: <p>I had similara issues:<br>
<a href="https://discuss.pytorch.org/t/loaded-model-is-not-the-same/107339">Loaded model is not the same - PyTorch Forums</a></p>
<p>So what you can check, if you use a scheduler which also contains the information of last_epoch.</p>
<p>Otherwise you can check if you calculate your loss values in model.eval(). This makes a difference especially if you have dropout layers.</p>
<p>If everything is fine, than there might also be other non-deterministic reason. You can read this up here:<br>
<a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.7.0 documentation</a></p>

Jia_Ming: <p>The valid loss is calculated after calling the model.eval() everytime. After the calculation, the parameters are saved. In theorectical view, by loading these parameters to calculate valid loss in model.eval, there should not be a big gap of losses…</p>
Basic "manual two" 2 layers

Brent1: <p>In the below code; building the <code>a1</code> tensor seems to “lose” the gradients - what would be the correct way to use <code>l11</code> and <code>l12</code> in further computations that would be included in the backward call?</p>
<pre><code class="lang-auto">x = torch.tensor([3.0,5.0], requires_grad=True)
w11 = torch.tensor([5.0,2.0], requires_grad=True)
w12 = torch.tensor([1.5,3.5], requires_grad=True)

w21 = torch.tensor([-0.5,2.0], requires_grad=True)

l11 = x@w11
l12 = x@w12
a1 = torch.tensor([l11, l12], requires_grad=True)
l21 = w21@a1

l21.backward()

print("x")
print(x.grad)
print("w11")
print(w11.grad)
print("w12")
print(w12.grad)
print("w21")
print(w21.grad)
print("a1")
print(a1.grad)
print("l21")
print(l21.grad)
</code></pre>
<p>Outputs:</p>
<pre><code class="lang-auto">
x
None
w11
None
w12
None
w21
tensor([25., 22.])
a1
tensor([-0.5000,  2.0000])
l21
None
</code></pre>

CedricLy: <p>I think replacing this line</p>
<aside class="quote no-group" data-username="Brent1" data-post="1" data-topic="109111">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brent1/48/33505_2.png" class="avatar"> Brent1:</div>
<blockquote>
<p><code>a1 = torch.tensor([l11, l12], requires_grad=True)</code></p>
</blockquote>
</aside>
<p>with</p>
<pre><code class="lang-auto">a1 = torch.cat((l11, l12), 1)
</code></pre>

Brent1: <p>Ah right; thanks for the pointer; it errors on <code>RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated</code> and first google gives a few hits which I need to try wrap my head around; but do think that is in the right direction</p>

Nikronic: <p>Hi,</p>
<p><a class="mention" href="/u/cedricly">@CedricLy</a> 's suggestion is the correct way of doing it.</p>
<p>Note that <code>torch.tensor(another_tensor)</code> works like <code>tensor.clone().detach()</code> which causes the trouble you have mentioned.<br>
Also, you are printing <code>a1.grad</code> which gives you a tensor instead of None because it should not be a leaf tensor but because of the <code>torch.tensor(tensor, requires_grad=True)</code>, it creates a leaf tensor.</p>
<p>About the second issue, <code>l11</code> and <code>l12</code> are scalars but they should be at least 1D tensors which you achieve it by using <code>tensor.unsqueeze(dim)</code>.</p>
<p>But as a general tip, if you use any functions other than methods provided by PyTorch, you are breaking the computational graph. So, always try to use PyTorch semantics as it also help you to find bugs later on.<br>
And, following tutorials can help you a lot as they are very well written (and of course documentations).</p>
<p>Bests</p>
Forward path in nn.Linear

Duy_Phan1: <p>torch.manual_seed(0)<br>
input = torch.tensor([[1.,1.,1.]])<br>
print(torch.nn.Linear(3, 1).weight)<br>
print(torch.nn.Linear(3, 1).bias)<br>
print(list(torch.nn.Linear(3, 1).parameters()))<br>
torch.nn.Linear(3, 1)(input)</p>
<p>I got the result:<br>
Parameter containing:<br>
tensor([[-0.0043,  0.3097, -0.4752]], requires_grad=True)<br>
Parameter containing:<br>
tensor([0.4578], requires_grad=True)<br>
[Parameter containing:<br>
tensor([[-0.0512,  0.1528, -0.1745]], requires_grad=True), Parameter containing:<br>
tensor([-0.1135], requires_grad=True)]<br>
tensor([[-1.1505]], grad_fn=)<br>
I calculated the result by hand using weight and bias, and using parameters, but I get the two different numbers, not the one above -1.1505. Why is it?</p>

CedricLy: <p>Try use this:</p>
<pre><code class="lang-auto">torch.manual_seed(0)
input = torch.tensor([[1.,1.,1.]])
a = torch.nn.Linear(3, 1)
print(a.weight)
print(a.bias)
print(list(a.parameters()))
a(input)
</code></pre>
<p>Otherwise you will always create a new layer with different weights and biases.<br>
Line <code>print(list(a.parameters()))</code> needs to contain the lines <code>print(a.weight) and print(a.bias)</code>.</p>
Create a sliding mask around diagonals

talipturkmen: <p>Hello,</p>
<p>Firstly I’m very sorry with this newbie question. I couldn’t find an efficient answer. I want to create a mask with zeros and ones for the n by n matrix. For value each i-th row I want to set the columns between i-a and i+a to 1 and rest of them to 0.<br>
As an example of the values<br>
n = 4 (4x4 matrix)<br>
a = 1<br>
I want to get a matrix as follows.<br>
[<br>
[1,1,0,0],<br>
[1,1,1,0],<br>
[0,1,1,1],<br>
[0,0,1,1]<br>
]</p>
<p>How can I implement this in the best way in Pytorch? Thanks in advance…</p>

pascal_notsawo: <p>This piece of code can do the job. But it’s not very optimal.</p>
<pre><code class="lang-auto">import torch

n = 4
a = 1

A = torch.zeros((n, n))

for i in range(n) :
    for j in range(max(i - a, 0), min(i + a + 1, n)) :
        A[i][j] = 1.0
</code></pre>

talipturkmen: <p>Thank you I am using in this way but I want to do it without a loop.</p>

KFrank: <p>Hi Talip!</p>
<aside class="quote no-group" data-username="talipturkmen" data-post="1" data-topic="109035" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5e9695/48.png" class="avatar"> talipturkmen:</div>
<blockquote>
<p>As an example of the values<br>
n = 4 (4x4 matrix)<br>
a = 1<br>
I want to get a matrix as follows.<br>
[<br>
[1,1,0,0],<br>
[1,1,1,0],<br>
[0,1,1,1],<br>
[0,0,1,1]<br>
]</p>
</blockquote>
</aside>
<p>I think that <a href="https://pytorch.org/docs/1.7.0/generated/torch.tril.html#torch.tril" rel="noopener nofollow ugc">torch.tril()</a> and/or <a href="https://pytorch.org/docs/1.7.0/generated/torch.triu.html#torch.triu" rel="noopener nofollow ugc">torch.triu()</a> will be the key to what you want.</p>
<p>Here is one specific approach:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt; n = 4
&gt;&gt;&gt; a = 1
&gt;&gt;&gt; torch.tril (torch.ones ((n, n)), a) * torch.triu (torch.ones ((n, n)), -a)
tensor([[1., 1., 0., 0.],
        [1., 1., 1., 0.],
        [0., 1., 1., 1.],
        [0., 0., 1., 1.]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

talipturkmen: <p>thank you very much Frank!</p>
Encounter a confusing RNN architecture

Gideon: <ol>
<li>
<p>import torch.nn as nn</p>
</li>
<li>
<p>from torch.autograd import Variable</p>
</li>
<li>
<p>class RNN(nn.Module):</p>
</li>
<li>
<pre><code>def __init__(self, input_size, hidden_size, output_size):
</code></pre>
</li>
<li>
<pre><code>    super(RNN, self).__init__()
</code></pre>
</li>
<li>
<pre><code>    self.hidden_size = hidden_size
</code></pre>
</li>
<li>
<pre><code>    self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
</code></pre>
</li>
<li>
<pre><code>    self.i2o = nn.Linear(input_size + hidden_size, output_size)
</code></pre>
</li>
<li>
<pre><code>    self.softmax = nn.LogSoftmax(dim = 1)
</code></pre>
</li>
<li>
<pre><code>def forward(self, input, hidden):
</code></pre>
</li>
<li>
<pre><code>    combined = torch.cat((input, hidden), 1)
</code></pre>
</li>
<li>
<pre><code>    hidden = self.i2h(combined)
</code></pre>
</li>
<li>
<pre><code>    output = self.i2o(combined)
</code></pre>
</li>
<li>
<pre><code>    output = self.softmax(output)
</code></pre>
</li>
<li>
<pre><code>    return output, hidden
</code></pre>
</li>
<li>
<pre><code>def initHidden(self):
</code></pre>
</li>
<li>
<pre><code>    return Variable(torch.zeros(1, self.hidden_size))
</code></pre>
</li>
</ol>
<p>I see it in a book named   (Vishnu).<br>
Are there any errors in Line 13th? I mean that ‘combined’ should be replaced by ‘hidden’?<br>
Correspondingly, does the Line 8th should be changed to ’self.h2o = nn.Linear(hidden_size, output_size)‘?<br>
My another reference book gives the similar codes with , so I was very confused.</p>

Abhilash_Srivastava: <p>The code above assumes that the input to <code>self.i2h</code> takes in both the input and the output from the last hidden layer. Similarly, <code>self.i2o</code> takes in both the input and the output from the last hidden layer.</p>
<p>While the code above will still work. I prefer what you mentioned below. In theory, <code>i2o</code> should not be dependent on the input, instead, it should depend on just the output of the current <code>hidden</code>.</p>
<aside class="quote no-group" data-username="Gideon" data-post="1" data-topic="109090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gideon/48/33010_2.png" class="avatar"> Gideon:</div>
<blockquote>
<p>Are there any errors in Line 13th? I mean that ‘combined’ should be replaced by ‘hidden’?<br>
Correspondingly, does the Line 8th should be changed to ’self.h2o = nn.Linear(hidden_size, output_size)‘?</p>
</blockquote>
</aside>
Facing problem in creating confusion matrix

krishna511: <p>I am using following function to predict  and eventually plotting confusion matrix. Here I am using CroosEntropyLoss criterion. The model is returning to values, which are tuples, So when I m using this function after saving the training model. It throws the typeError :exp(): argument ‘input’ (position 1) must be Tensor, not tuple. In bold line. What I am grasping is in exp function i need a scaler number and this a tuple, I took the max value of tuple using max() method , but still its giving same error.<br>
Please suggest a solution</p>
<pre><code class="lang-auto">def predict_sm(model=None, dataloader=None, device=None, path=None):
    if path:
        model.load_state_dict(torch.load(path))

    model.eval()
    model.to(device)

    y_pred = []
    
    for X, y,y1 in dataloader:
        X = X.to(device)
        with torch.no_grad():
            logProb,_ = model(X)
            
        **Prob = torch.exp(logProb)**
        Pred = torch.argmax(Prob, dim=1).long()
        y_pred.append(list(Pred.detach().cpu().numpy()))
    print(sum(y_pred,[]))
    return sum(y_pred,[])
</code></pre>

ptrblck: <p>As you’ve described, the error is raised, since a tensor instead of a <code>tuple</code> is expected.<br>
Could you explain a bit, what the model is returning and what the <code>tuple</code> contains?<br>
I guess you could index it to solve this error, but it’s difficult to recommend the right approach without knowing what the <code>tuple</code> represents.</p>

krishna511: <p>Thanks for reply and sorry I forgot it, Actually I was working in colab, and probably the files are not saved immediately there, so t was not working then, After sometime,on  reloading the page I got it. My model has returned two values ,the last layer before output was encoder layer, so I took the mean std dev and finally concatenated them, then these two values are passed through the final linear layer.</p>
Performance drop between two devices

bernhard2202: <p>Hello,</p>
<p>I am loading a resnet model and running images through that model. I am using forward_hooks to extract the internal activations from the model and then calculating a score with those. All this works unsupervised with out any training and quite well. Now I am trying to execute the same code locally on my laptop and the performance drops significantely. I copied the entire conda environment from the server and serialized the model to disk and loaded it locally to make sure I am not using different python versions or models. I also triple checked to use the same data locally. Now the only thing that is different any more (imho) is the hardware this model is executed. Can this lead to different activations? Is this possible?</p>
<p>Best, Bernhard</p>

ptrblck: <p>I assume you are referring to the “model performance”, i.e. a model metric, not the speed of the model.<br>
In that case, how large are the differences? Different hardware can yield absolute errors due to the limited floating point precision, but should not introduce larger differences.</p>
Constrained parameters in torch

vr308: <p>The way to specify a constrained parameter in pyro:</p>
<p>weights = pyro.param(‘weights’, torch.randn(K), constraint = constraints.simplex)</p>
<ul>
<li>whats the way to do it with torch.nn.Parameter?</li>
</ul>

googlebot: <p>IIRC pyro just imports constraints from pytorch.</p>
<p>Check <a href="https://pytorch.org/docs/master/distributions.html#module-torch.distributions.constraint_registry" rel="noopener nofollow ugc">constraint_registry</a> doc, there is a code example. Alternatively, you can just manually transform parameters, e.g. scale = F.softplus(self.raw_scale)</p>
TypeError: cannot unpack non-iterable NoneType object

aatiibutt: <p>I am training CNNs through transfer learning, following pytorch transfer learning tutorial. As shown below, after completion of training, type error occurs.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3c9fa326fec82edd9f7729f8fbe00f26b32d75b.png" alt="image" data-base62-sha1="udztNOZK95hWZisaNQQzELbblgL" width="645" height="123"><br>
I shall be grateful, If anybody help me out.</p>

CedricLy: <p>Can you further show, how the function train_model() look like.</p>

aatiibutt: <p>Here is the train code.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b45557f890d7b35d48a18a944775fce7fe715f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b45557f890d7b35d48a18a944775fce7fe715f8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b45557f890d7b35d48a18a944775fce7fe715f8_2_552x500.png" alt="image" data-base62-sha1="m9AyquYjPhHfC67VsC3hUC9EwXu" width="552" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b45557f890d7b35d48a18a944775fce7fe715f8_2_552x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b45557f890d7b35d48a18a944775fce7fe715f8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b45557f890d7b35d48a18a944775fce7fe715f8.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">575×520 67.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

CedricLy: <p>looks alright to me.<br>
What are the last few printouts you get?<br>
Please also share the return line of the function, maybe there is some issues?<br>
I don’t know what the return hist would be.</p>

ptrblck: <p>The error is raised, since <code>train_model</code> doesn’t return anything, so you might want to add a <code>return</code> statement. This code snippet raises the same error:</p>
<pre><code class="lang-python">def fun():
    print("don't return anything")

fun() # works
a, b = fun() # fails
&gt; TypeError: cannot unpack non-iterable NoneType object
</code></pre>
Reconstruction a neuron synapse

Alex_NG: <p>Dear all,<br>
I want to reconstruct a synapse in the first layer of full-connection, which shows in Fig. Could  you help me.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a18d7d386dfbbb362d8c5509cad959ffc4d92e0f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a18d7d386dfbbb362d8c5509cad959ffc4d92e0f" title="RECONS_NN"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18d7d386dfbbb362d8c5509cad959ffc4d92e0f_2_690x388.jpeg" alt="RECONS_NN" data-base62-sha1="n3a0JtNOq8G6DkPSfNu7znnQhj1" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18d7d386dfbbb362d8c5509cad959ffc4d92e0f_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a18d7d386dfbbb362d8c5509cad959ffc4d92e0f_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a18d7d386dfbbb362d8c5509cad959ffc4d92e0f.jpeg 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RECONS_NN</span><span class="informations">1280×720 43.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If I understand your use case correctly, you would like to transform the left layer to the right one.<br>
In that case I assume the left layer is defined as <code>nn.Linear(784, 1, bias=False)</code> and you could replace it with:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1a = nn.Linear(392, 1, bias=False)
        self.fc1b = nn.Linear(392, 1, bias=False)
        self.fc2 = nn.Linear(2, 1, bias=False)
        
    def forward(self, xa, xb):
        xa = self.fc1a(xa)
        xb = self.fc1b(xb)
        x = torch.cat((xa, xb), dim=1)
        x = self.fc2(x)
        return x

model = MyModel()
batch_size = 2
xa = torch.randn(batch_size, 392)
xb = torch.randn(batch_size, 392)
out = model(xa, xb)
</code></pre>
<p>Note that I haven’t added any activation functions etc. as it’s unclear from the figure, if this is desired.</p>
Combine trained models

julioeu99: <p>Hello! Anyone knows if it is possible to combine two differents models? E.g. CNN with AE.</p>

ptrblck: <p>It depends a bit how you would like to combine these models. There are no specific limitations to call modules e.g. in a sequential way as each “model” would be treated in the same way as a “layer” (e.g. <code>nn.Conv2d</code>). Could you explain your use case a bit more and what you’ve tried so far?</p>
Synchronisation after Allreduce

vineeths: <p>I know that a <code>torch.cuda.synchronize()</code> waits until all operations on GPUs are completed. Does that also include communication operations?</p>
<p>More specifically, does calling <code>torch.cuda.synchronize()</code> just after an allreduce operation waits till the allreduce operation gets completed?</p>
<pre><code class="lang-auto">tensor_reduce_op = torch.distributed.all_reduce(tensor=tensor async_op=True)
torch.cuda.synchronize()
</code></pre>
Weight and Parameters in nn.linear

Duy_Phan: <p>print(torch.nn.Linear(3, 1).weight)<br>
print(torch.nn.Linear(3, 1).bias)<br>
print(list(torch.nn.Linear(3, 1).parameters()))</p>
<p>Results:<br>
Parameter containing:<br>
tensor([[0.4428, 0.3657, 0.0139]], requires_grad=True)<br>
Parameter containing:<br>
tensor([0.3234], requires_grad=True)<br>
[Parameter containing:<br>
tensor([[ 0.3756,  0.2845, -0.3558]], requires_grad=True), Parameter containing:<br>
tensor([-0.4788], requires_grad=True)]</p>
<p>Can you explain the difference between weight, bias and parameters in nn.linear?<br>
What does it choose to calculate the forward path?</p>

pascal_notsawo: <p>I couldn’t do better with these equations.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7cb1e403cfa46ba85d1fdd55484ffb4fb1a8bbd.png" data-download-href="https://discuss.pytorch.org/uploads/default/c7cb1e403cfa46ba85d1fdd55484ffb4fb1a8bbd" title="linear"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7cb1e403cfa46ba85d1fdd55484ffb4fb1a8bbd.png" alt="linear" data-base62-sha1="svsc6GgRn1xWJfkRTyIJsxFdeNv" width="690" height="228" data-dominant-color="404041"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">linear</span><span class="informations">1054×349 44.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(See <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">documentation</a> for more information)<br>
(See <a href="https://en.wikipedia.org/wiki/Random_seed" rel="noopener nofollow ugc">wiki/Random_seed</a> for more information about random seed).</p>
Unable to load the state dict model

IamSparky: <p>I have saved the model using this code:<br>
<code>torch.save(best_model.state_dict(),'./leaf_Classsification_GPU_CutMix_EfficientNet-B4.pt')</code></p>
<p>then when I tried to load the model using this:<br>
model = Net()</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load("leaf_Classsification_GPU_CutMix_EfficientNet-B4.pt"))
model.eval()
</code></pre>
<p>it is giving me error:<br>
AttributeError: ‘_IncompatibleKeys’ object has no attribute ‘eval’</p>
<p>Please suggest.</p>
<p><a href="https://drive.google.com/file/d/1260gSjyRIjepZgOJqCIdrrvYnYJoXYBa/view?usp=sharing" rel="noopener nofollow ugc">THIS IS THE MODEL WHICH IS TO BE LOADED</a> for evaluation</p>

pascal_notsawo: <p>Can you show me the complete code of your <code>Net</code> class?</p>

IamSparky: <p>Yes why not , here’s the link to my <a href="https://www.kaggle.com/soumochatterjee/inference-cassava-leaf-detection" rel="noopener nofollow ugc">notebook</a> .</p>
<p>and here’s my model class:</p>
<pre><code class="lang-auto">class EfficientNet_b4(nn.Module):
    def __init__(self):
        super(EfficientNet_b3, self).__init__()
        self.model = efficientnet_pytorch.EfficientNet.from_pretrained('efficientnet-b4')
        self.dropout = nn.Dropout(0.1)
        self.final_layer = nn.Linear(1792 , 5)
        
    def forward(self, inputs):
        batch_size, _, _, _ = inputs.shape
        
        x = self.model.extract_features(inputs)

        # Pooling and final linear layer
        x = self.model._avg_pooling(x)
        
        x = F.adaptive_avg_pool2d(x, 1).reshape(batch_size, -1)
        outputs = self.final_layer(self.dropout(x))

        return outputs
    
model = efficientnet_pytorch.EfficientNet.from_pretrained('efficientnet-b4')
</code></pre>
Loss function returning NaN Loss

Bhavishya_Pandit: <p>Description:<br>
I have been trying to build a simple linear regression model with the neural network with 4 features and one output. The loss function used is mse loss. It is returning loss as Nan. Learning rate is 1e-3.<br>
I trying tuning the lr but I didn’t see any change in it. Would appreciate your help in the same.<br>
Thanks in advance!</p>
<p>Training Loop</p>
<pre><code class="lang-auto">epochs=100
for epoch in range(epochs):
    for inputs,targets in training_data:
        y_pred=model(inputs.float())
        y_pred=y_pred.squeeze(1)
        #print(y_pred.shape)
        #print(inputs.dtype)
        #print(targets.dtype)
        #print(y_pred.dtype)
        y_pred=y_pred.double()
        loss=mse(y_pred,targets)
        loss.backward() #computing gradients
        opt.step() #updating parameters
        opt.zero_grad() #setting the grads back to zero
    if epoch%10==0:
        print(f'Epoch:{epoch}/{epochs} | Loss:{loss.item()}')
</code></pre>
<p><strong>Output</strong></p>
<pre><code class="lang-auto">Epoch:0/100 | Loss:nan
Epoch:10/100 | Loss:nan
Epoch:20/100 | Loss:nan
Epoch:30/100 | Loss:nan
Epoch:40/100 | Loss:nan
Epoch:50/100 | Loss:nan
Epoch:60/100 | Loss:nan
Epoch:70/100 | Loss:nan
Epoch:80/100 | Loss:nan
Epoch:90/100 | Loss:nan
</code></pre>

Nikronic: <p>Hi,</p>
<p>You are using MSE then the only reason that can cause this is either your model definition is wrong or your targets are not correct numbers.</p>
<p>The issue is not related to learning rate or other hyperparamters as your first iteration is also producing <code>nan</code>. Simply test your network with a random input and see if it’s producing a proper output and also verify your targets.</p>
<p>Bests</p>
Stack expects each tensor to be equal size, but got [200, 200, 3] at entry 0 and [200, 200] at entry 10

Hacking_Pirate: <p>Hi,<br>
I am trying to make an image classification model. To open the image I am using <code>Image.open(img_path)</code>.<br>
But I am getting this error:</p>
<p><code>stack expects each tensor to be equal size, but got [200, 200, 3] at entry 0 and [200, 200] at entry 10</code>.</p>
<p>But when I use CV2 to open the image it works fine, like this:</p>
<pre><code class="lang-auto">img = cv2.imread(img_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
</code></pre>
<p>Why isn’t it working with PIL backend?<br>
Here is my Dataset class:</p>
<pre><code class="lang-auto">class HolidayDataset(Dataset):
    def __init__(self, train_path, df, transform=None):
        self.train_path = train_path
        self.df = df
        self.transform = transform
        
    def __len__(self):
        return len(self.df)
    
    def __getitem__(self, idx):
        img_name = str(self.df.Image[idx])
        target = str(self.df.Class[idx])
        img_path = self.train_path + img_name
        img = Image.open(img_path)
        img = np.array(img)/255
        if target == "Miscellaneous":
            target = 0
        elif target == "Christmas_Tree":
            target = 1
        elif target == "Jacket":
            target = 2
        elif target == "Candle":
            target = 3
        elif target == "Airplane":
            target = 4
        elif target == "Snowman":
            target = 5
        if self.transform:
            img = self.transform(image=img)["image"]
        return {
            "img": torch.tensor(img, dtype=torch.float),
            "target": torch.tensor(target, dtype=torch.long)
        }
</code></pre>
<p>It would be great if someone could help!<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>PIL</code> would check the image format, in particular the number of channels, and will return 3 channels for RGB images and will remove the channel dimension for grayscale images, which is the case for the image in the shape <code>[200, 200]</code>.<br>
You could use <code>Image.open(path).convert('RGB')</code> to transform the grayscale images into the RGB format.</p>

Hacking_Pirate: <p>Thanks a lot, got it!</p>
Debugging torch.onnx.export

fteufel: <p>Hi,<br>
I’m trying to export a model via ONNX. The export is failing due to an operator not being supported.</p>
<pre><code class="lang-auto">RuntimeError: Exporting the operator __derive_index to ONNX opset version 11 is not supported. Please open a bug to request ONNX export support for the missing operator.
</code></pre>
<p>Implementing the missing operator is beyond my means, so I want to rewrite this part of the model.<br>
I’m now trying to figure out which line of my model is causing the issue - the ONNX traceback has no info on this at all. I tried putting print statements in-between as recommended elsewhere, but they all get printed until the forward pass end, and then the error appears.</p>
<p>Is there any way to find the problematic line? I can’t make sense of the operator name itself.</p>

ptrblck: <p>Also tracked <a href="https://github.com/pytorch/pytorch/issues/50125">here</a> with reference to an open <a href="https://github.com/pytorch/pytorch/pull/49514">PR</a>.</p>
Output of affine grid

sait: <p>In spatial transformers, we multiply grid coordinates with matrix to get new co-ordinates. So for example if size of image is 28*28. Then grid co-ordinates would be (x,y) where 0 $0 \leq x \leq 27$ and $0 \leq y \leq 27$. Then after multiplication we would get different co-ordinates which could be fractional. But output of affine grid seems to be always in 0,1. Does any how to interpret the output of affine grid and how it is used in conjunction in grid sample.</p>

ptrblck: <p><code>affine_grid</code> should mostly return grid locations ins <code>[-1, 1]</code>, which represent the sampling pixel locations normalized by the image dimension. The value <code>[-1, -1]</code> represents the left-top pixel, while <code>[1, 1]</code> is the right-bottom one. Values out of this range will be using the <code>padding</code> mode in <code>grid_sample</code>.</p>
How could a BatchNorm layer generate nan in backward process?

JiaRu2016: <p>I’m training a customized Transformer modle with my customized loss function, i found loss value is nan after some training steps, so I set  <code>torch.autograd.set_detect_anomaly(True)</code> try to figure out what happend. after adding this line, It gives this error:</p>
<pre><code class="lang-auto">[W python_anomaly_mode.cpp:104] Warning: Error detected in CudnnBatchNormBackward. Traceback of forward call that caused the error:

.... ommited ...

  File "/home/rjia/dl/seq_release/research/seq_models.py", line 389, in forward
    x_preprocessed = self.preprocess_module(X)
  File "/home/rjia/anaconda3/envs/cuda102/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rjia/dl/seq_release/research/seq_models.py", line 915, in forward
    rtn = self.batchnorm(X.view(bz * seq_len, m)).view(bz, seq_len, m)
  File "/home/rjia/anaconda3/envs/cuda102/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rjia/anaconda3/envs/cuda102/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 131, in forward
    return F.batch_norm(
  File "/home/rjia/anaconda3/envs/cuda102/lib/python3.8/site-packages/torch/nn/functional.py", line 2056, in batch_norm
    return torch.batch_norm(
</code></pre>
<p>and finally raise <code>RuntimeError("Function 'CudnnBatchNormBackward' returned nan values in its 1th output.") </code></p>
<p>I’v serarch in google and pytorch forum, eg.   .  some people memtioned that “if all inputs are same , or exist nan, or batch_size == 1, then batchnorm will generate nan value in backward process” but I’v tried to feed all-zero inputs to <code>BatchNorm1d()</code> layer and backward() works well in this case:</p>
<pre><code class="lang-python">import torch
torch.autograd.set_detect_anomaly(True)

m = 5
n = 8

bn = torch.nn.BatchNorm1d(num_features=m)

y = torch.zeros(n, 1)
x = torch.zeros(n, m)  # &lt;----- all inputs are same do not cause batchnorm raise Error
w = torch.nn.Parameter(data=torch.rand(1, n))

for i in range(1000):
    out = bn(x)
    yh = torch.matmul(w, x)
    loss = ((yh - y) ** 2).sum()
    loss.backward()
    
    assert ~torch.isnan(bn.running_mean).any()
    assert ~torch.isnan(bn.running_var).any()
</code></pre>
<p>So my question is: In which case could BatchNorm generate nan values?</p>

ptrblck: <p>BatchNorm layers shouldn’t return NaN values, if the input is well defined.<br>
You could add assert statements to the input and output of this batchnorm layer and check if all values are finite via <code>torch.isfinite(tensor)</code>.</p>
Pytorch reserves only 1GB on the GPU

a_d: <p>Hello all,<br>
I get the Cuda OOM error when I try to run the forward pass on the GPU, with the following as the error message -</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 5.96 GiB (GPU 0; 7.80 GiB total capacity; 1.18 GiB already allocated; 5.68 GiB free; 1.21 GiB reserved in total by PyTorch)
</code></pre>
<p>Firstly, why is pytorch reserving so less memory when my graphics card is of 8GB and second is instead of pytorch reserving memory, is there any way we can tell pytorch to use all the space available.</p>
<p>The model I a trying to run is essentially a retinanet with resnet50 backbone.</p>
<p>I built pytorch from source from the master branch and my <code>torch.__version__</code> is <code>'1.8.0a0+46b8321' </code></p>

ptrblck: <aside class="quote no-group" data-username="a_d" data-post="1" data-topic="107946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e19b73/48.png" class="avatar"> a_d:</div>
<blockquote>
<p>Firstly, why is pytorch reserving so less memory when my graphics card is of 8GB and second is instead of pytorch reserving memory, is there any way we can tell pytorch to use all the space available.</p>
</blockquote>
</aside>
<p>When the error is raised, <code>1.18GB</code> are already allocated from the previously executed code and PyTorch tries to allocate <code>5.96GB</code>, which doesn’t fit into the available <code>5.68GB</code>.<br>
The rest of the device would most likely be used by the CUDA context (to store the kernels etc.).</p>
Something wrong with this notebook?

Hacking_Pirate: <p>Hi, I am participating in a competition on hacker earth. But when I create my submission file all of the predicted classes are coming same. I would be very grateful if someone could have a look at my notebook and figure out if something is wrong?</p>
<p>Here is the link:<br>
<a href="https://www.kaggle.com/hackingpirate/notebookb658beec12" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.kaggle.com/hackingpirate/notebookb658beec12</a></p>
<p>Thanks!</p>

ptrblck: <p>The notebook doesn’t seem to exist anymore.</p>
Is this overfitting?

lhkhiem28: <p>This is my training and validation loss on 5-fold CV for a classification task. Is this overfitting ?</p>
<p>I’m using Focal Loss, AdamW(lr=1e-3, weight_decay=0.01), and ReduceLROnPlateau(factor=0.5, patience=3). What I can do to make validation loss more stable (another optimizer or lr_cheduler or anything) ?<br>
Thank you !</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abc6dcb3d5a76978f04cfaf0225df711b1085050.png" alt="loss_fold1" data-base62-sha1="ovBHRUca4qNetU0c1WR5MOT1aLu" width="320" height="240"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/456d2f1718ec621f0d404265d92f2c19bd278503.png" alt="loss_fold2" data-base62-sha1="9UaS04ybiKyWeWYkEXRsWTmmz0D" width="320" height="240"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcdd44bc94cc68546c72bf5769040cc28c93fbcb.png" alt="loss_fold3" data-base62-sha1="vvR6LiB5wKK2jYBH2Pc9WE0q9Pt" width="320" height="240"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/174c67966a7e1be823823d0ca7f1e886ef3b93a2.png" alt="loss_fold4" data-base62-sha1="3k6G2foyAAHPeBQY81AqMnC8lgK" width="320" height="240"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd59cf52abec6876ee645a1db5c64e86d83eaf40.png" alt="loss_fold5" data-base62-sha1="tiCiua7ToZSIlzbpXCNHCyD94SQ" width="320" height="240"></p>

ptrblck: <p>The loss curves indicate overfitting, but it also seems that not many samples might be used since the curves are showing some “discrete” steps.</p>
<p>In order to counter overfitting you could add more regularization to the model, e.g. through dropout, weight decay etc.</p>
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450

Rokia: <p>torch.bmm raise this error , how could I solve this problem please</p>

ptrblck: <p>Could you post an executable code snippet as well as your setup (PyTorch, CUDA versions, used GPU etc.) so that we can take a look and try to reproduce this issue?</p>
Transfer Semantic Segmentation result to CPU is slow

tominator: <p>I trained a semantic segmentation model (FCN). Now I am writing an API where I can call the model and get back the segmentation result as an array. The inference of the model on the GPU is really fast (0,02 seconds). The problem is  when I want to bring the output back to the CPU: it takes 3,8 seconds. My image resolution is really high (2048x1536). But even with a smaller resolution (250x1024, I cropped  the input images) it takes 0.45 seconds.  Is there a way to reduce this time or another way to improve performance?</p>
<pre><code class="lang-auto">#Inference
start = time.time()
with torch.no_grad():
    output = eval_model(input_batch)['out'][0]
torch.cuda.synchronize()
end = time.time()

#Tensor to CPU
start2 = time.time()
output_prediction = output.argmax(0).cpu().byte().numpy()
torch.cuda.synchronize()
end2 = time.time()

print(str(end-start)) #inference time
print(str(end2-start2)) #transfer time
</code></pre>

ptrblck: <p>I get a transfer time of approx. 700ms for an output of <code>[1, 64, 2048, 1536]</code> on my laptop, which shouldn’t have the fastest GPU connection compared to a server or workstation using:</p>
<pre><code class="lang-python">x = torch.randn(1, 64, 2048, 1536, device='cuda')
nb_iters = 100

# warmup
for _ in range(10):
    y = x.cpu()

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    y = x.cpu()
torch.cuda.synchronize()
t1 = time.time()

print('{:.3f}ms'.format((t1 - t0)/nb_iters*1000))
</code></pre>

tominator: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you very much. I found my mistake.</p>
How to disable AMP in some layers?

fangwei123456: <p>Hi everyone, I want to disable AMP for all BatchNorm2d layers in my models because running_var is prone to cause overflow when converting from float32 to float16. How can I implement this idea?</p>

ptrblck: <p>You can use nested context managers via <code>with torch.cuda.amp.autocast</code> and disable it for certain layers.</p>
Char-level RNN sampling

neuralpat: <p>Hi,</p>
<p>I built a character-level RNN that generates headlines. It has clearly learned, as it gives me mostly correct words and headlines when I sample it.<br>
My problem is though, that it mostly generates headlines from the training data.<br>
I’m using np.random.choice() to sample the next character based on the probability distribution the network outputs.</p>
<p>Is this kind of network ever going to output truly novel headlines?<br>
What could I do differently ?</p>
<p>Here’s the network:</p>
<pre><code class="lang-auto">class HeadlineModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(HeadlineModel, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.fc1 = nn.Linear(hidden_size, output_size)

    def forward(self, data, hidden, cell):
        x, (h, c) = self.lstm(data, (hidden, cell))
        x = self.fc1(x)
        y_hat = x  # torch.softmax(x, dim=2)

        return y_hat, h, c
</code></pre>
How can I write the CUDA code to support FP16 calculation?

cb_zhang: <p>I want to write a custom layer using CUDA. However, It fails when I use NVIDIA-Apex to train the model with mixed-precision. What should I do? Is there any example of FP16 cuda layer?</p>

ptrblck: <p>You could use <code>AT_DISPATCH_FLOATING_TYPES_AND_HALF</code> to dispatch the code for the <code>float16</code> type and use <code>scalar_t</code> in the code (similar to e.g. <a href="https://github.com/pytorch/pytorch/blob/74dcb6d363824a35060bc0546854530daa92c8fc/aten/src/ATen/native/cuda/BinaryMiscOpsKernels.cu#L13-L21">this code</a>).<br>
Also note, that we recommend to use the native mixed-precision training utility via <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples"><code>torch.cuda.amp</code></a> instead of <code>apex/amp</code> now.</p>

fangwei123456: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/fangwei123456/spikingjelly/blob/master/spikingjelly/cext/csrc/neuron/neuron_forward_kernel.cu" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/fangwei123456/spikingjelly/blob/master/spikingjelly/cext/csrc/neuron/neuron_forward_kernel.cu" target="_blank" rel="noopener nofollow ugc">fangwei123456/spikingjelly/blob/master/spikingjelly/cext/csrc/neuron/neuron_forward_kernel.cu</a></h4>
<pre><code class="lang-cu">#include &lt;cuda.h&gt;
#include &lt;cuda_runtime.h&gt;
#include &lt;cuda_fp16.h&gt;
#include &lt;torch/extension.h&gt;
#include &lt;math.h&gt;
#include &lt;stdio.h&gt;
#include "neuron_def.h"
__forceinline__  __device__ float grad_atan(const float &amp; alpha, const float &amp; x)
{
  const float M_PI_2__alpha__x = (float) M_PI_2 * alpha * x;
  return alpha / 2.0f / (1.0f + M_PI_2__alpha__x * M_PI_2__alpha__x);
}

__forceinline__  __device__ float grad_sigmoid(const float &amp; alpha, const float &amp; x)
{
  const float sigmoid_ax = 1.0f / (1.0f + expf(- alpha * x));
  return (1.0f - sigmoid_ax) * sigmoid_ax * alpha;
}

typedef float (*grad_surrogate_function) (const float &amp;, const float &amp;);
</code></pre>

  This file has been truncated. <a href="https://github.com/fangwei123456/spikingjelly/blob/master/spikingjelly/cext/csrc/neuron/neuron_forward_kernel.cu" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
These codes may help.</p>
Pytorch dataloader and collate_fn

Ohm: <p>Hi,<br>
“Batch sizes” in “train loader” should be changed during the training based on the result of some other function. Maybe collate_fn in “train loader” can be useful. Could you give an example to change Batch_size(for example based on the following simple function):</p>
<pre><code class="lang-auto">if Sth%2==0:
   Batch_size += 1 
</code></pre>
PyTorch with C++11 ABI

Duncan_Freeman: <p>Is there a pre-built version of PyTorch that uses the C++11 ABI? I need to be able to use TorchScript exported from Python in <a href="https://github.com/LaurentMazare/tch-rs" rel="noopener nofollow ugc">tch-rs</a> which only works with C++11 ABI version of libtorch, but the pip version of PyTorch does not use the C++11 ABI (calling <a href="https://pytorch.org/docs/stable/generated/torch.compiled_with_cxx11_abi.html" rel="noopener nofollow ugc">this</a> returns <code>False</code>).</p>
<p>EDIT: After inspecting <code>torch-sys</code>'s <code>build.rs</code>, apparently there are environment variables one can set to use the older ABI:</p>
<pre><code class="lang-auto">export TORCH_CUDA_VERSION=10.2
export LIBTORCH_CXX11_ABI=0
</code></pre>
Step Activation Function

Ayman_Al_Jabri: <p>Is there a step activation function in pytorch? One that returns -1 for values &lt; 0 and 1 for values &gt; 0</p>

ziqipang: <p>You may define this activation function on your own. I think there is a great tutorial on pytorch website.</p>
<p>However, I think your function is not differentiable, so you might have to be careful on using this function.</p>

KFrank: <p>Hello Ziqi and Ayman!</p>
<aside class="quote no-group" data-username="ziqipang" data-post="2" data-topic="71791" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/6bbea6/48.png" class="avatar"> ziqipang:</div>
<blockquote>
<p>However, I think your function is not differentiable, so you might have to be careful on using this function.</p>
</blockquote>
</aside>
<p><code>tanh()</code> is a commonly-used differentiable approximation to the<br>
step function, and is sometimes used as an activation function.<br>
(We often call these differentiable approximations “soft” versions<br>
of the functions they approximate.)</p>
<p>Best.</p>
<p>K. Frank</p>

ziqipang: <p>Yeah I think using tanh makes sense.</p>