akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Input Shape for LSTM

username: scrungus
cooked: <p>I have a LSTM defined in PyTorch as:</p>
<p><code>self.actor = nn.LSTM(input_size=101, hidden_size=4, batch_first=True)</code></p>
<p>I then have a <code>deque</code> object of length 4, full of a history of states (each a 1D tensor of size 101) from the environment. I reshape this and pass it to my agent:</p>
<p><code>self.agent(torch.stack(list(self.state))[None,...])</code></p>
<p>so that it has shape <code>[1,4,101]</code>.</p>
<p>Essentially I have an environment which I believe that, given more history, agents should be able to predict what will happen next better. I am trying to use an LSTM to acheive this. The problem is, when I pass this through my model:</p>
<pre><code class="lang-auto">def forward(self, x):
        logits = self.actor(x)
...
</code></pre>
<p>I get the error:<br>
<code>RuntimeError: input.size(-1) must be equal to input_size. Expected 404, got 101</code></p>
<p>Why is this happening? As I understand it, the LSTM should expect a sequence of observations of size 101 to be passed to it, not a tensor of shape <code>[hidden_size*obs_size]</code>. Otherwise how does it figure out how big a feature is?</p>

username: Andrei_Cristea
cooked: <p>Could you provide a reproducible bit of code that raises that RuntimeError? When I try to reproduce your code, it runs fine for me, so I suspect the input you’re passing it isn’t actually shaped like (1, 4, 101).</p>
<pre><code class="lang-auto">import torch
from torch import nn
actor = nn.LSTM(input_size=101, hidden_size=4, batch_first=True)
x = torch.randn(1, 4, 101)
actor(x)

Output:
(tensor([[[ 1.7885e-05,  3.4918e-01,  3.8907e-01,  2.3808e-01],
          [ 4.0551e-02,  6.3810e-02,  6.0021e-01,  2.8658e-03],
          [ 1.1655e-01,  2.2330e-01, -2.9728e-02,  1.0425e-02],
          [ 5.7080e-02, -3.9743e-01, -5.7700e-01,  7.0268e-02]]],
        grad_fn=&lt;TransposeBackward0&gt;),
 (tensor([[[ 0.0571, -0.3974, -0.5770,  0.0703]]], grad_fn=&lt;StackBackward0&gt;),
  tensor([[[ 0.9932, -0.4577, -1.0284,  0.1895]]], grad_fn=&lt;StackBackward0&gt;)))
</code></pre>

username: scrungus
cooked: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> I will try to get a minimal example working, but bear in mind that I get the same error when I run it like this:</p>
<pre><code class="lang-auto">class ActorNet(nn.Module):
    def __init__(self, obs_size, n_actions, depth, hidden_size = 64):
        super().__init__()

        print(obs_size)
        self.actor = nn.Sequential(nn.LSTM(input_size=obs_size, hidden_size=4, batch_first=True))

    def forward(self, x):
        x = torch.randn(1,4,101)
        logits = self.actor(x)
        logits = torch.nan_to_num(logits)
        dist = Categorical(logits=logits)
        action = dist.sample()

        return dist, action
</code></pre>

username: scrungus
cooked: <p>Not to worry, there was a very silly mistake. You can delete this post now <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
Adam works well but SGD is not working at all

username: Shen
cooked: <p>Hi I’m working on the binarized neural network proposed here <a href="https://arxiv.org/pdf/1602.02830.pdf" rel="nofollow noopener">binarynet</a> using Pytorch. I saw there’s already a version available on Github <a href="https://github.com/itayhubara/BinaryNet.pytorch" rel="nofollow noopener">BinarynetPytorch</a>, where it used Adam as the optimizer. I tried to change it to SGD optimizer, however, the network then was not training at all.</p>
<p>I changed the initial learning rate for Adam (i.e., 5e-3) to 5e-1 for SGD. I also played with momentum various momentum and batch size. Unfortunately none of them worked. The loss value just fluctuated around one value, but never got dropped.</p>
<p>Are there anything I should do or check?</p>
<p>Thanks!</p>

username: maroov
cooked: <blockquote>
<p>I changed the initial learning rate for Adam (i.e., 5e-3) to 5e-1 for SGD</p>
</blockquote>
<p>This always depends on the experiment, but 5e-1 seems really high for a learning rate. Have you tried with smaller values?</p>

username: Shen
cooked: <p>Yes, I’ve been playing with various ranges of learning rate. However, none of them worked for SGD. I wonder if the way binarized neural net was implemented does not support SGD?</p>

username: Cames
cooked: <p>Did you ever find the solution to the problem of SGD not working when Adam does, even after experimenting with the variables?</p>

username: tom
cooked: <p>One aspect of Adam vs SGD is thatthe former “normalizes” the pointwise sizeof gradients, i.e. if you scale gradients by a fixed number, it will (up to those epsilon regularizations) not change Adam’s behaviour, while the steps will be scaled if you use SGD with the same learning rate as before.<br>
Now if you have different scales of gradients (happens easily, e.g. for weights vs biases, we have an example in ch 5 of our book even for linear regression), this can impede convergence for SGD while Adam does fine. Some models (eg StyleGAN, to randomly pick a model I reimplemented and remember this) explicitly address this to work well with SGD.</p>
<p>Best regards</p>
<p>Thomas</p>
What causes this error?？mat1 and mat2 shapes cannot be multiplied

username: YuhskeHujisaki
cooked: <p>What causes this error?<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (322560x1 and 46080x7)<br>
x = torch.ones(128,1,50,432)<br>
class Net(nn.Module):</p>
<pre><code>def __init__(self):

    super(Net, self).__init__()

    self.conv1 = nn.Conv2d(1, 60,kernel_size=(1, 15),stride=(1,3))

    self.conv2 = nn.Conv2d(60, 60, kernel_size=(1, 4),stride=(1,2))

    self.conv3 = nn.Conv2d(60, 60, kernel_size=(30,1),stride=(1,3))

    self.conv4 = nn.Conv2d(60, 90, kernel_size=(1, 3),stride=(1,1))

    self.conv5 = nn.Conv2d(90, 120, kernel_size=(1, 1),stride=(1,1))

    self.pool = nn.MaxPool2d(kernel_size=(1, 2),stride=(1,2))

    self.soft = nn.Softmax()

    self.li=nn.Linear(128*120*3,7)

def forward(self, x):

  x = self.conv1(x)

  print(x.shape)

  x = self.pool(x)

  print(x.shape)

  x = self.conv2(x)

  print(x.shape)

  x = self.pool(x)

  print(x.shape)

  x = self.conv3(x)

  print(x.shape)

  x = self.conv4(x)

  print(x.shape)

  x = self.pool(x)

  print(x.shape)

  x = self.conv5(x)

  print(x.shape)

  x = self.pool(x)

  print(x.shape)

  x = self.li(x)

  print(x.shape)

  x = self.soft(x)

  return x</code></pre>

username: Xanthan
cooked: <p>U have to mention the dimension parameter in the <code>nn.softmax</code></p>

username: Andrei_Cristea
cooked: <p>You should flatten your input before you pass it to the linear layer, and make sure that the linear layer input size is <em>not</em> the minibatch size, which it seems like it might be in your implementation.</p>
<p>in other words, if you change the linear layer to this:<br>
<code>self.li=nn.Linear(120*21,7)  # the 128 from before looks like the minibatch size, which doesn't belong here</code></p>
<p>And add a flattening step before the linear layer in your forward pass, like so:</p>
<pre><code class="lang-auto">x = nn.Flatten(1, -1)(x)
x = self.li(x)
</code></pre>
<p>Then your code will run.</p>
Multiplying two tensors returns first tensor

username: Cornelius_Denninger
cooked: <p>Hello,</p>
<p>I’m trying to multiply two tensors to mask out certain values.</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.tensor([[1,2],[3,4]])
b = torch.tensor([[1,0],[0,1]])
c = a*b
</code></pre>
<p>Where c should return tensor([[1, 0],[0, 4]]). This works perfectly in my python notebook.</p>
<p>However, in my train loop, I’m multiplying X_pos * y which outputs the X_pos tensor rather than the multiplication of the two tensors.</p>
<pre><code class="lang-auto">for batch in dataloader:
    X, y = batch[0][:], batch[1][:]

    X_pos = X[:,1,:]
    Mask_X = X[:,2,:]
    X,y = X[:,0,:],y[:,0,:]

    sequence_length = y.size(1)
    tgt_mask = model.get_tgt_mask(sequence_length).to(self.device)
    print(X_pos.size(),X_pos)
    print(y.size(),y)
    print(X_pos*y)
    pred = model(X, X_pos*y,mask=Mask_X, tgt_mask=tgt_mask)

    pred = pred.permute(1,2,0)

    loss = loss_fn(pred, y.type(torch.LongTensor).to(self.device))

    opt.zero_grad()
    loss.backward()
    opt.step()
    total_loss += loss.detach().item()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcc052db374bf6e413e5a757459399976c274de1.png" alt="image" data-base62-sha1="A3WnbZRJSgK54dmzPQWN7wGOphf" width="602" height="478"></p>
<p>I was not able to reproduce this behaviour.</p>

username: ptrblck
cooked: <p>Your current screenshot shows the expected output since the values at <code>y</code> where <code>x</code> contains ones are also ones.</p>
Keras to Pytorch for Autoencoder

username: anuj_singh1
cooked: <p>Hi could anyone please convert below Keras code to Pytorch one?</p>
<h2>
<a name="input-layer-1" class="anchor" href="#input-layer-1"></a>input layer</h2>
<p>input_layer = Input(shape=(X.shape[1],))</p>
<h2>
<a name="encoding-part-2" class="anchor" href="#encoding-part-2"></a>encoding part</h2>
<p>encoded = Dense(100, activation=‘tanh’, activity_regularizer=regularizers.l1(10e-5))(input_layer)<br>
encoded = Dense(50, activation=‘relu’)(encoded)</p>
<h2>
<a name="decoding-part-3" class="anchor" href="#decoding-part-3"></a>decoding part</h2>
<p>decoded = Dense(50, activation=‘tanh’)(encoded)<br>
decoded = Dense(100, activation=‘tanh’)(decoded)</p>
<h2>
<a name="output-layer-4" class="anchor" href="#output-layer-4"></a>output layer</h2>
<p>output_layer = Dense(X.shape[1], activation=‘relu’)(decoded)</p>
<p>autoencoder = Model(input_layer, output_layer)<br>
autoencoder.compile(optimizer=“adadelta”, loss=“mse”)</p>
<p>x = data.drop([“Class”], axis=1)<br>
y = data[“Class”].values</p>
<p>x_scale = preprocessing.MinMaxScaler().fit_transform(x.values)<br>
x_norm, x_fraud = x_scale[y == 0], x_scale[y == 1]</p>
<p>autoencoder.fit(x_norm[0:2000], x_norm[0:2000], batch_size = 256, epochs = 10, shuffle = True, validation_split = 0.20)</p>
<p>hidden_representation = Sequential()<br>
hidden_representation.add(autoencoder.layers[0])<br>
hidden_representation.add(autoencoder.layers[1])<br>
hidden_representation.add(autoencoder.layers[2]).</p>
<p>=======here I am not getting specially t his part:<br>
hidden_representation = Sequential()<br>
hidden_representation.add(autoencoder.layers[0])<br>
hidden_representation.add(autoencoder.layers[1])<br>
hidden_representation.add(autoencoder.layers[2]).</p>
How to view a tensor as an image?

username: gokulp01
cooked: <p>Hi, I was working on a project where I have a tensor output. How do I view it is an image?</p>
<p>What I’ve tried so far:</p>
<pre><code class="lang-auto">arr_ = np.squeeze(out_p)
plt.imshow(arr_)
plt.show()
</code></pre>
<p>The error:</p>
<pre><code class="lang-auto">RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
</code></pre>
<p>I tried some of the answers mentioned on this forum and on stackoverflow, all to vain.<br>
If anyone help, I would be very grateful<br>
~Thanks!</p>

username: ptrblck
cooked: <p>Did you try to add the suggested <code>tensor.detach().numpy()</code> operation from the error message?<br>
If so, what was the new error?</p>

username: gokulp01
cooked: <p>I was wondering if I am doing it wrong?<br>
This is what I’ve done so far:</p>
<pre><code class="lang-auto">out_img2=out_img.detach().numpy()
arr_ = np.squeeze(out_img2)
plt.imshow(arr_)
plt.show()
</code></pre>
<p>The error is: <code>TypeError: Invalid shape (3, 512, 512) for image data</code></p>
<p>And when I tried:</p>
<pre><code class="lang-auto">out_img2=out_img.detach().numpy()
arr_ = out_img2.squeeze
plt.imshow(arr_)
plt.show()
</code></pre>
<p>The error is: <code>TypeError: Image data of dtype object cannot be converted to float</code></p>
<p><strong>Edit: Solved it.I had forgotten to permute!</strong></p>
Can pretrained model use different loss function?

username: toppyishappy
cooked: <p>I want to research various loss function with the model but I don’t sure. Is it possible to apply another loss with pretrained model.</p>

username: InnovArul
cooked: <p>It is technically possible to apply any loss function on pretrained models.</p>
<p>You may need to give more information on what kinda task are you going to apply it on, what are tasks the pretaining models trained on, etc. The answers to this type of question will be subjective and may not be concrete.</p>

username: toppyishappy
cooked: <p>I have loaded vggface2 pretrained model for image similarity task and curious about using different loss function. The repo use euclid distance to distinguish between images but I want to use triplet loss. Is it possible?</p>

username: InnovArul
cooked: <p>In my understanding, Euclidean distance is a generic distance measure. It can be used internally in Triplet loss as well (For example, the distance between the anchor and positive can be calculated by Euclidean distance).</p>
<p>Maybe you are thinking about siamese loss function vs. triplet loss function?<br>
If so, yes. you can apply it of course. You might be aware of <a href="https://arxiv.org/abs/1503.03832">FaceNet</a> paper which applied it.</p>

username: toppyishappy
cooked: <p>It means if I use another pretrained model such as resnet15. I can apply triplet loss as well. Thanks for your answers.</p>
Does redundancy in ground truth labels hurt model performance

username: sparshgarg23
cooked: <p>Let’s say I am training an object detection algorithm where the goal is to identify the class as well as the bounding box .</p>
<p>Now if in my dataset if a majority of  the ground truth bounding boxes have the same values,then will this in any way impact the model’s learning capability.</p>
<p>For example while training a basic object detection model on the above mentioned dataset,I noticed that when the model was given a new unseen images,it was unable to correctly predict the bounding box.</p>
<p>Preview of dataset is shown below</p>
<pre><code class="lang-auto">Class 1
image 1 bbox coords 192 63 242 243
image 2 bbox coords 192 63 242 243
.
.
.
.
.image 19
</code></pre>
<p>in each image,object is either zoomed in or being viewed from different angle.<br>
(Also please note that the bounding box coordinates are not an exact representation of my dataset and were included to give an idea of what my dataset looks like).<br>
I tried computing the R2 score between the predicted bounding box and actual bounding box,and found that for unseen examples the R2 score came out to be highly negative.</p>

username: ptrblck
cooked: <p>My guess would be that the model might learn to predict the “majority” bounding box coordinates in the same way it would learn to predict the majority class index in an imbalanced classification use case.</p>

username: sparshgarg23
cooked: <p>So in  a sense ,can we assume that there might be some overfitting,as such would you recommend making changes in the dataset so that the bounding box distribution is properly captured.<br>
I tried to use weighted sampling during training process but that didn’t give me any significant improvements.</p>
Merge two pretrained weights from 2 neural networks

username: Z_Huang
cooked: <p>I am not sure how to make this happen.</p>
<p>For example, I have 2 neural networks. Each neural network’s first 3 layers are identical.</p>
<p>If I want to merge the pretrained weights/biases of these 2 sets of 3 layers together. How may I do that?</p>
<p>I am thinking the following logic:</p>
<ul>
<li>(layer 1 of 1st NN + layer 1 of 2nd NN) / 2</li>
<li>(layer 2 of 1st NN + layer 2 of 2nd NN) / 2</li>
<li>(layer 3 of 1st NN + layer 3 of 2nd NN) / 2</li>
</ul>
<p>Does it make any sense?</p>

username: InnovArul
cooked: <p>In my opinion, if the models are independently trained, it is not meaningful to combine the neural networks this way, as there are no correlation / common reference points between them. i.e., it’s not guaranteed that the 1st layer in 1st NN learns similar features in the same order as the 1st layer in 2nd NN.</p>
<p>It is technically possible to do it as you know and may possibly give weights better than random initialization.</p>
Pytorch's autograd issue with joblib

username: Sam2022
cooked: <p>There seems to be a problem mixing pytorch’s autograd with joblib. I need to get gradient in parallel for a lot of samples. Joblib works fine with other aspects of pytorch, however, when mixing with autograd it gives errors. I made a very small example which shows serial version works fine but the parallel version crashes.</p>
<pre><code class="lang-auto">from joblib import Parallel, delayed
import numpy as np
torch.autograd.set_detect_anomaly(True)
tt = lambda x, grad=True: torch.tensor(x, requires_grad=grad)

def Grad(X, Out):
    return autograd.grad(Out, X, create_graph=True, allow_unused=True)[0]

xs, ys = [], []
for i in range(10):
    xi = tt(np.random.rand()).float()    
    yi = xi * xi
    xs += [xi]
    ys += [yi]


Grads_serial = [Grad(x, y) for x, y in zip(xs, ys)]
print("Grads_serial", Grads_serial)
Grads_parallel = Parallel(n_jobs=2)([delayed(Grad)(x, y) for x, y in zip(xs, ys)])
print("Grads_parallel", Grads_parallel)
</code></pre>
<p>The error message is not very helpful as well:</p>
<pre><code class="lang-auto">RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
Segmentation and number of classes

username: Chris_XU
cooked: <p>Hi, I am reading the <a href="https://github.com/pytorch/vision/tree/main/references/segmentation" rel="noopener nofollow ugc">torchvision reference for segmentation model</a>. What I am trying to understand is, since the pretraining classes is 21, Should the target mask be integers from 0 to 21, where 0 is the background?</p>
<p>If so, my next question is do we need to compute loss over 0 (background)? I think generally background pixels is the majority, will it hurt the model so that the model would always predict 0 (background)? Or should we ignore background label by passing <code>ignore_index=0</code> in cross entropy loss? I see that <a href="https://github.com/pytorch/vision/blob/main/references/segmentation/train.py#L51" rel="noopener nofollow ugc">here</a> we are ignoring index 255, but I am not sure where 255 comes from.</p>
<p>Thanks! Any input is highly appreciated!</p>
Convert non-image data into image-like data to use CNN

username: Aaron_Tseng
cooked: <p>I have 3 kinds of spatial data to use as inputs. I want to simulate them as image data with 3 channels. Those data are all 5x35 size with 5000 samples.</p>
<pre><code class="lang-auto">for i in range(5000):
    in_x.append([x[i].reshape(35,5), delta[i].reshape(35,5), np.pad(z[i], (0,170),'wrap').reshape(35,5)])
</code></pre>
<p>The three types of data are x, delta and z. They were input seperately with sample size 5000.</p>
<pre><code class="lang-auto">delta=np.loadtxt('./input_delta4.out')
x=np.loadtxt('./input_x4.out')
z=np.loadtxt('./input_z4.out')
</code></pre>
<p>But I find my transfering method to be wrong.</p>
<pre><code class="lang-auto">dataiter = iter(dataloaders['train'])
images, labels = dataiter.next()
len(images)
</code></pre>
<p>This outputs a length of 5000. But it should be 3, right?<br>
How can I fix it?</p>

username: ptrblck
cooked: <p>I don’t know how you are exactly creating the <code>Dataset</code> or <code>DataLoader</code> but this small example shows that the batch size is used as expected:</p>
<pre><code class="lang-python">dataset = torch.utils.data.TensorDataset(torch.randn(5000, 35, 5), torch.randn(5000, 35, 5), torch.randn(5000, 35, 5))
loader = torch.utils.data.DataLoader(dataset, batch_size=20)

a, b, c = next(iter(loader))

print(a.shape)
# torch.Size([20, 35, 5])
print(b.shape)
# torch.Size([20, 35, 5])
print(c.shape)
# torch.Size([20, 35, 5])
</code></pre>

username: Aaron_Tseng
cooked: <p>Thanks!!!</p>
<p>I have figured it out</p>
Autograd does not work for torch.stack on complex tensor

username: minzhao.liu
cooked: <p>It is pretty easy to use unsqueeze and cat as workarounds, but stack does not work and throws the following error:</p>
<pre><code class="lang-bash">RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
</code></pre>
<p>The minimal code to reproduce the error:</p>
<pre><code class="lang-python">params = torch.rand(1, requires_grad=True)
out = torch.stack([params, 1j*params], 1)
loss = out.sum()
print(loss)
loss.backward()
</code></pre>
<p>However, the following two run fine. First, if the parameters are initialized as cfloat (of course this is not the same behavior and is not a valid solution), it’s fine:</p>
<pre><code class="lang-python">params = torch.rand(1, dtype=torch.cfloat, requires_grad=True)
</code></pre>
<p>Or, if we don’t use stack, it’s fine:</p>
<pre><code class="lang-python">out = torch.cat([params.unsqueeze(1), 1j*params.unsqueeze(1)], 1)
</code></pre>

username: AlphaBetaGamma96
cooked: <p>If you’d want PyTorch to add this behavior (i.e. autocasting floats to cfloats when stacking real/complex datatypes), you can open an issue on PyTorch Github, and see if they’ll add it.</p>

username: tom
cooked: <p>Hi,</p>
<p>thank you for sending this! It seems like a perfectly good bug report with reproducing code.<br>
This is a shortcoming of the backward of torch.stack. If you ping me (t-vi on github) on an issue, I’ll send a PR.</p>
<p>Best regards</p>
<p>Thomas</p>

username: minzhao.liu
cooked: <p>I did not realize casting is the issue but now I see it. The problem is that it DID work with cat (what I meant was that it works when I initialize <code>params</code> to <code>cfloat</code>, or I initialize to <code>float</code> but use <code>cat</code>), which really confused me. I spent more time writing this bug report than coming up with my workaround, so not sure why it’s lazy.</p>

username: minzhao.liu
cooked: <aside class="quote no-group" data-username="minzhao.liu" data-post="1" data-topic="148734">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minzhao.liu/48/46136_2.png" class="avatar"> minzhao.liu:</div>
<blockquote>
<p><code>out = torch.cat([params.unsqueeze(1), 1j*params.unsqueeze(1)], 1)</code></p>
</blockquote>
</aside>
<p>Hi Thomas! Thanks for offering to take a look. I have created an issue <a href="https://github.com/pytorch/pytorch/issues/75852#issue-1205101624" class="inline-onebox" rel="noopener nofollow ugc">torch.stack backward does not automatically cast real float to complex float · Issue #75852 · pytorch/pytorch · GitHub</a> and tagged you in the comment.</p>
Cannot Build PyTorch from source in dev mode

username: Hannah_Salameh
cooked: <p>Hello! I was trying to build PyTorch from source in development mode and I keep coming across the following error. For context, I am running this on MacOS.</p>
<p>Python 3.9.7<br>
CMake 3.19.6</p>
<pre><code class="lang-auto">[3/4] Generating ATen headers
[1346/5995] Linking CXX static library lib/libprotobuf-lite.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libprotobuf-lite.a(io_win32.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libprotobuf-lite.a(io_win32.cc.o) has no symbols
[1488/5995] Generating src/x86_64-fma/2d-fourier-8x8.py.o
FAILED: confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o 
cd "/Users/hannahsalameh/Desktop/Junior Year/EECS481/pytorch/build/confu-deps/NNPACK" &amp;&amp; "PYTHONPATH=/Users/hannahsalameh/Desktop/Junior Year/EECS481/pytorch/third_party/python-six:/Users/hannahsalameh/Desktop/Junior Year/EECS481/pytorch/third_party/python-peachpy" /Users/hannahsalameh/opt/anaconda3/bin/python -m peachpy.x86_64 -mabi=sysv -g4 -mimage-format=mach-o -I/Users/hannahsalameh/Desktop/Junior\ Year/EECS481/pytorch/third_party/NNPACK/src -I/Users/hannahsalameh/Desktop/Junior\ Year/EECS481/pytorch/third_party/NNPACK/src/x86_64-fma -I/Users/hannahsalameh/Desktop/Junior\ Year/EECS481/pytorch/third_party/FP16/include -o /Users/hannahsalameh/Desktop/Junior\ Year/EECS481/pytorch/build/confu-deps/NNPACK/src/x86_64-fma/2d-fourier-8x8.py.o /Users/hannahsalameh/Desktop/Junior\ Year/EECS481/pytorch/third_party/NNPACK/src/x86_64-fma/2d-fourier-8x8.py
/bin/sh: PYTHONPATH=/Users/hannahsalameh/Desktop/Junior Year/EECS481/pytorch/third_party/python-six:/Users/hannahsalameh/Desktop/Junior Year/EECS481/pytorch/third_party/python-peachpy: No such file or directory
[1501/5995] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/command_line_interface.cc.o
ninja: build stopped: subcommand failed.
</code></pre>
Loss gets saturated after some epochs irrespective of architecture and data

username: Gaurav_Sharma
cooked: <p>Hi there!</p>
<p>I am working on an image to text seq2seq model, where am using CNN as an encoder and LSTM as a decoder. I am facing one issue that I tried to solve but couldn’t get any success. While running my model, after a few epochs my training loss either gets saturated or the rate of change becomes really less. It takes almost 100 epochs to go from ~1.68 to ~1.55 (training loss). And these loss numbers are still too large to deal with. While on the other hand, my validation loss increases and kind of gets saturated or will be fluctuating in a particular range, as shown in the figure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/825bdd69821bfcd915905404c301a4f6695667c9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/825bdd69821bfcd915905404c301a4f6695667c9" title="Screen Shot 2022-04-09 at 11.11.58 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/825bdd69821bfcd915905404c301a4f6695667c9_2_618x500.jpeg" alt="Screen Shot 2022-04-09 at 11.11.58 PM" data-base62-sha1="iBcTe9OD8fOMDCEb3A89GEzxvlf" width="618" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/825bdd69821bfcd915905404c301a4f6695667c9_2_618x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/825bdd69821bfcd915905404c301a4f6695667c9_2_927x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/825bdd69821bfcd915905404c301a4f6695667c9_2_1236x1000.jpeg 2x" data-dominant-color="EFE8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-04-09 at 11.11.58 PM</span><span class="informations">1830×1480 93.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried the following things to improve the model’s performance but nothing worked:</p>
<ol>
<li>I increased the size of the dataset from 50K to 100K</li>
<li>I preprocessed it differently to reduce the output dimensionality i.e. size of the vocab. The current size is ~630 tokens. (I am using torchtext and bucketiterator for preprocessing.)</li>
<li>increase and decrease the size of both encoder and decoder. Tried adding and removing some layers from CNN and LSTM both.</li>
<li>Tried changing the hidden size, and learning rate but nothing worked.</li>
<li>Tested for a higher number of epochs.</li>
<li>As I am using cross-entropy loss, I tried initializing weights at the beginning of each and every batch.</li>
<li>changed my training style from epoch to step training.</li>
<li>even tried using different datasets, the resulting plot had the same behavior just with a different magnitude.<br>
May I request you to please help me? Any suggestions will be appreciated!<br>
Thank you!</li>
</ol>
After some changes, my pytorch RL self driving car game refused to learn

username: PringleOrange
cooked: <p>So previously, the AI would do several laps about 50 or 60 generations in and then after 80 generations it’d spin in circles. I managed to fix this by changing self.epsilon to be equal to 80-self.n_games*0.01 rather than without the *0.01. In this process though, i had changed several functions and variables to get there including:</p>
<p>get_action<br>
_move<br>
play_step<br>
model.py init and forward<br>
self.gamma</p>
<p>and after changing all of these, the AI would not learn anymore and it would not perform several laps as it used to (even though it wouldnt go in circles anymore after 80 generations). So therefore i tried to change them all back but it still wouldnt learn anymore and wouldnt lap around the racetrack or survive long at all.</p>
<p>My question is, what is wrong with this AI and what do i need to fix for it to be able to learn properly again whilst not spinning in circles?</p>
<p>The AI project:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/PringleOrange/self-driving-car">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/PringleOrange/self-driving-car" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/722b8273435260ea3924012b60fa51abd06739d810989ff9ad89e4d396255003/PringleOrange/self-driving-car" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/PringleOrange/self-driving-car" target="_blank" rel="noopener nofollow ugc">GitHub - PringleOrange/self-driving-car</a></h3>

  <p>Contribute to PringleOrange/self-driving-car development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How can I keep the batch_size per gpu in DDP?

username: 111344
cooked: <p>If my model original batch-size is 32, when I use two gpus ,one node per gpu, I use batch-size 16(32/ngpu), but if the number of gpus is 3 or any odd number, we should keep the size like 32/3=10 or limit the number of gpus to 2?<br>
Any help is welcome.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/111344">@111344</a>, if you are looking for mathematical equivalence, you will need at least two things:</p>
<ol>
<li>each DDP process processes <code>batch_size / num_gpu</code> samples: this allows DDP collectively process the same amount of inputs as local training.</li>
<li>
<code>loss_fn(model([sample1, sample2])) == (loss_fn(model([sample1])) + loss_fn(model[sample2])) /2</code>: this is because DDP uses AllReduce to compute the average gradients across all processes. If the above condition is not met, then average gradients across all DDP processes are not equivalent the the local training gradients.</li>
</ol>
<p>However, practically, applications usually do <strong>not</strong> have to satisfy the above conditions. Did you see any training accuracy degradation when scale up to 3 GPUs?</p>

username: 111344
cooked: <p>I haven’t tested GPU = 3, but I will take the time to verify it and then reply to you</p>

username: 111344
cooked: <p>I think there is no difference between gpu=2 or 3.<br>
In my experiment:<br>
batch-size=8  gpu=2  --&gt;batch_size=4 for single gpu<br>
batch-size=8  gpu=3  --&gt;batch_size=2 for single gpu(so total batch_size is 6)<br>
batch-size=8 or 6, under normal circumstances, it does not have much impact on performance<br>
For some task which are very sensitive to batch_size may need to take it into account</p>

username: ishaan-mehta
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>When you say each process processes <code>batch_size / num_gpu</code> samples, does that mean each GPU is assigned one process, which will each have their own <code>DataLoader</code> instance with <code>batch_size</code> equal to whatever you set in your code? So each GPU (<strong>not</strong> each node) will process an equal split of the programmer-set batch size?</p>
<p>And then when you say DDP computes the average gradient across all processes, does that mean it divides by the number of GPUs (<strong>not</strong> the number of nodes)?</p>
<p>Thanks for the clarification!</p>
How to check the bandwidth usage of nccl backend?

username: Maxwell_Albert
cooked: <p>I am using nccl backend to send and recv pytorch tensor on my server.<br>
And I am trying to use these command to check the bandwidth usage, is it corrent?<br>
Fisrt I use</p>
<pre><code class="lang-auto">'tcp://127.0.0.1:1224'
</code></pre>
<p>as my dist utils.<br>
And then I print</p>
<pre><code class="lang-auto">export NCCL_SOCKET_IFNAME=lo
</code></pre>
<p>lo is the network interface used for loop.<br>
Then I use</p>
<pre><code class="lang-auto">sudo iftop -i lo -n -P
</code></pre>
<p>But the bandwidth is too small.<br>
WHat should I do to check the bandwidth uses of dist.send and dist.recv?</p>

username: pbelevich
cooked: <p>bandwidth is a network interface term, if you want to measure <code>dist.send</code> and <code>dist.recv</code> speed I think you can send and receive some large tensors and measure latency in any suitable way like <code>time.time()</code>?</p>
Set_grad_enabled (True/False) for train/val predictions

username: Z_Huang
cooked: <p>In the tutorial (<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.11.0+cu102 documentation</a>), why do we set set_grad_enabled when we predict in train? When phase == “val”, the grad is disabled. My question is does the statement “_, preds = torch.max(outputs, 1)” generate different results mathematically or logically? Why do I need to disable or enable grad when predicting in train/val phases?</p>
<pre><code class="lang-auto">with torch.set_grad_enabled(phase == 'train'):
    outputs = model(inputs)
    _, preds = torch.max(outputs, 1)
    loss = criterion(outputs, labels)

    # backward + optimize only if in training phase
    if phase == 'train':
        loss.backward()
        optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>You would disable the gradient computation during the validation phase to save memory by avoiding storing the intermediate forward activations, which would be needed to compute the gradients.<br>
A more aggressive optimization might be achieved via <code>with torch.inference_mode()</code> which would disable the view tracking and version counter bumps additionally.</p>

username: Z_Huang
cooked: <p>I am not sure if it makes sense to disable grad when pred results in train to safe memory? I mean during the train session, only disable grad when pred. Does it impact train pred results?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Z_Huang" data-post="3" data-topic="149055">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/z_huang/48/8201_2.png" class="avatar"> Z_Huang:</div>
<blockquote>
<p>I mean during the train session, only disable grad when pred. Does it impact train pred results?</p>
</blockquote>
</aside>
<p>Do you mean something like this?</p>
<pre data-code-wrap="python"><code class="lang-python">outputs = model(inputs)
with torch.no_grad():
    _, preds = torch.max(outputs, 1)
loss = criterion(outputs, labels)
</code></pre>
<p>I.e. wrapping the <code>max</code> operation only into the <code>no_grad</code> guard?<br>
If so, then I would probably <code>.detach()</code> the output instead as the code looks a bit cleaner to me, but the <code>no_grad</code> guard should also work.<br>
The <code>pred</code> output will not be changed as explained before, as disabling the grad calculation will only save resources.</p>

username: Z_Huang
cooked: <p>Thanks, I mean the following logic. Since torch.no_grad() is to save memory, I was wondering if it makes sense to save extra memory in train.</p>
<pre><code class="lang-auto">mode.train()
outputs = model(inputs)
model.eval()
with torch.no_grad():
    _, preds = torch.max(outputs, 1)
model.train()
loss = criterion(outputs, labels)
</code></pre>

username: ptrblck
cooked: <p>Your code snippet fits my example, so yes, you can <code>detach()</code> the tensor or use the guard.</p>
Cudatoolkit 10.0 not found when installing PyTorch via conda

username: erobertc
cooked: <p>Hi,</p>
<p>I’m trying to follow the instructions at <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">https://pytorch.org/get-started/locally/</a> to install PyTorch on my Ubuntu (16.04) machine. I already have the NVIDIA drivers and CUDA 10.0 set up.</p>
<p>When I attempt to run the command from that page</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
</code></pre>
<p>in a Conda environment, Conda says “PackageNotFoundError: Package missing in current linux-64 channels: - cudatoolkit 10.0*”. What’s wrong?</p>

username: ptrblck
cooked: <p>I just created a new conda env and this command runs successfully on my machine (Ubuntu 18.04).<br>
Could you try to update conda using:<br>
<code>conda update conda</code><br>
and run the command again?</p>

username: erobertc
cooked: <p>That updated me from conda 4.3.17 to 4.3.30 and seems to have fixed it, thanks.</p>

username: Vikram_M
cooked: <p>Updating conda worked, but I got this</p>
<p>Fetching package metadata …<br>
Solving package specifications: .</p>
<p>And screen froze.</p>

username: ptrblck
cooked: <p><code>conda</code> can be quite slow when it’s trying to solve the env dependencies, so you could either wait for a bit longer or maybe creating a new virtual conda env and installing PyTorch there could help.</p>
PyTorch model size in MBs

username: Amur_Saqib_Pal
cooked: <p>Hi, I am working with different quantized implementations of the same model, the main difference being the precision of the weights, biases, and activations. So I’d like to know how I can find the difference between the size of a model in MBs that’s in say 32-bit floating point, and one that’s in int8. I have the models saved in .PTH format, any suggestions will be great. Thanks in advance.</p>

username: huahuanZ
cooked: <p>The function torch.finfo (torch.iinfo for integer) would be useful</p>
<pre><code class="lang-auto">import torch
model = torch.nn.Linear(2, 3)
model.bias.data = torch.tensor([1, 2, 3], dtype=torch.int8)
size_model = 0
for param in model.parameters():
    if param.data.is_floating_point():
        size_model += param.numel() * torch.finfo(param.data.dtype).bits
    else:
        size_model += param.numel() * torch.iinfo(param.data.dtype).bits
print(f"model size: {size_model} / bit | {size_model / 8e6:.2f} / MB")
</code></pre>
<p>Have a look at this <a href="https://pytorch.org/docs/stable/type_info.html?highlight=info" rel="noopener nofollow ugc">Type Info — PyTorch 1.11.0 documentation</a></p>

username: Amur_Saqib_Pal
cooked: <p>Thank you for replying. I had done almost exactly the same thing. However, I’m using a special quantization library which applies the quantizations and saves the model in a .pt form. Can you tell me how the same thing can be done using a .pt file, since the model loaded by a .pt doesn’t have a “parameters” attribute?</p>

username: huahuanZ
cooked: <p>use torch.load to load the parameters in a dict, then do it the same way</p>
<pre><code class="lang-auto">import torch
# checkpoint should be an OrderedDict type,
# like {'weight': Tensor(...)}
checkpoint = torch.load("yourfile.pth")
size_model = 0
for param in checkpoint.values():
    if param.is_floating_point():
        size_model += param.numel() * torch.finfo(param.dtype).bits
    else:
        size_model += param.numel() * torch.iinfo(param.dtype).bits
print(f"model size: {size_model} / bit | {size_model / 8e6:.2f} / MB")

</code></pre>

username: Amur_Saqib_Pal
cooked: <p>This is what I get after running the script…<br>
<strong>AttributeError: ‘collections.OrderedDict’ object has no attribute 'is_floating_point’</strong></p>

username: huahuanZ
cooked: <p>My example only works if you save the .pth file with an object returned by model.state_dict(). In practical, this depends on how you save your model .pth file.<br>
You could have a look at the loaded object checkpoint. Whatever it is, just get the parameters to count the size of model.</p>
optim.lr_scheduler.ReduceLROnPlateau minimum learning rate

username: Z_Huang
cooked: <p>I am running my CNN model using ReduceLROnPlateau to manage my learning rate and factor=0.1/patience=5. However, after 100 epochs, the learning rate stuck at 1.0000000000000005e-08 and it won’t go any lower. Any idea why?</p>

username: mxahan
cooked: <p>This behavior is expected as stated in the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#torch.optim.lr_scheduler.ReduceLROnPlateau" rel="noopener nofollow ugc">documentation</a>:  “If the difference between new and old lr is smaller than eps, the update is ignored. Default: 1e-8.”</p>
<p>You can provide the “eps” value and may observe the change.</p>

username: Z_Huang
cooked: <p>Got it, I am using the default eps 1e-8. I will make it to 1e-10 or even smaller then. Thanks!</p>
NNAPI invalid dimensions error on Android

username: kmitsunami
cooked: <p>Hi, I am trying to run a small UNet-like Image2Image model on Android using NNAPI, I have successfully converted it with NNAPI, but when I run it on my android app, I am getting the following error. What’s the cause of this?</p>
<blockquote>
<p>E/CpuExecutor: Invalid dimensions for model operand: packages/modules/NeuralNetworks/common/TypeUtils.cpp:198: Incompatible dimensions: [1, 128, 55, 55] and [1, 128, 56, 55]<br>
E/CpuExecutor: STRIDED_SLICE failed.<br>
E/AndroidRuntime: FATAL EXCEPTION: Thread-4</p>
</blockquote>
<p>Without NNAPI the model works as expected, but after NNAPI conversion the model has the error. This is probably due to ConvTranspose2d. I am using it with the following parameters. Are there any workarounds?<br>
<code>torch.nn.ConvTranspose2d(in, out, kernel_size=3, padding=1, output_padding=1, stride=2)</code></p>

username: kmitsunami
cooked: <p>I figured out the cause. The cause was not ConvTranspose2d, but the use of slices as shown below:<br>
<code>dec = self.dec(enc)[:, :, :prev_enc.size(2), :prev_enc.size(3)]</code><br>
Looks like it caused the incompatible dimensions error.</p>
Equivalent to tf.data.Dataset.zip in pytorch?

username: ChrisLR
cooked: <p>Hi, i am moving from tensorflow to pytorch and i am looking for an equivalent to <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset#zip" rel="noopener nofollow ugc">tf.data.Dataset.zip</a> in pytorch.</p>
<p>Use case:</p>
<ul>
<li>
<p>i have multiple quantities to precompute for a generated dataset dso (using sampling).</p>
</li>
<li>
<p>several datasets have to be created by mapping functions f1, f2, …fn: dsi = fi(dso) which can be scalars, vectors, matrices;</p>
</li>
<li>
<p>i want to be able to batch iterate all these datasets (including original one) simultaneously after an identical shuffle.</p>
</li>
</ul>
<p>Is there an (elegant) to do this in pytorch ?</p>
<p>Thanks for any help.</p>

username: nivek
cooked: <p>If it is feasible for you to use <a href="https://pytorch.org/data/beta/torchdata.datapipes.iter.html" rel="noopener nofollow ugc"><code>IterDataPipe</code></a> (it is a replacement for <code>IterableDataset</code>) from <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc"><code>torchdata</code></a>, you can use <a href="https://pytorch.org/data/beta/generated/torchdata.datapipes.iter.Zipper.html#torchdata.datapipes.iter.Zipper" rel="noopener nofollow ugc"><code>Zipper</code></a>.</p>

username: ChrisLR
cooked: <p>Thank you for your good pointers.<br>
I think Zipper is suitable for me.<br>
I have never dive into this part of the documentation <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Best,<br>
Christophe.</p>
Implementing weighted poisson_nll_loss

username: EvanZ
cooked: <p>I am using <code>torch.nn.functional.poisson_nll_loss</code> as a loss function. Now I would like to take into account the weights for each instance (for example, exposure in a risk application). I am thinking all I need to do is set <code>reduction='none'</code> and then dot the vector of losses with the weights column. Does that sound right?</p>

username: ptrblck
cooked: <p>Yes, your idea sounds reasonable. I’m not familiar with your exact use case, but you should consider normalizing the reduced mean with the used weight as seen <a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">here</a> to avoid creating a data-dependent loss spike.</p>

username: EvanZ
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>! Very helpful.</p>
How to get output parameter from ScriptModules

username: MartinZhang
cooked: <p>Hello everyone, I want to get output parameter from ScriptModules layer.</p>
<p>But</p>
<pre><code class="lang-auto">RuntimeError: register_forward_hook is not supported on ScriptModules
</code></pre>
<p>Is there any other way to achieve it?</p>
<p>thx you.</p>