akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ONNX convert BERT-pytorch

username: jbm
cooked: <p>I’ve posted this question on SO (<a href="https://stackoverflow.com/questions/56962116/bert-pytorch-to-onnx-conversion-lambda-error" rel="nofollow noopener">https://stackoverflow.com/questions/56962116/bert-pytorch-to-onnx-conversion-lambda-error</a>), but I thought this might be a better place for it. I’m trying to convert the PyTorch BERT implementation from <a href="https://github.com/codertimo/BERT-pytorch" rel="nofollow noopener">https://github.com/codertimo/BERT-pytorch</a>, but I’m hitting an error with the TransformerBlock:</p>
<pre><code class="lang-auto">builtins.ValueError: Auto nesting doesn't know how to process an input object of type bert_pytorch.model.transformer.TransformerBlock.forward.&lt;locals&gt;.&lt;lambda&gt;. Accepted types: Tensors, or lists/tuples of them
</code></pre>
<p>The code for the TransformerBlock is:</p>
<pre><code class="lang-auto">class TransformerBlock(nn.Module):
    """
    Bidirectional Encoder = Transformer (self-attention)
    Transformer = MultiHead_Attention + Feed_Forward with sublayer connection
    """

    def __init__(self, hidden, attn_heads, feed_forward_hidden, dropout):
        """
        :param hidden: hidden size of transformer
        :param attn_heads: head sizes of multi-head attention
        :param feed_forward_hidden: feed_forward_hidden, usually 4*hidden_size
        :param dropout: dropout rate
        """

        super().__init__()
        self.attention = MultiHeadedAttention(h=attn_heads, d_model=hidden)
        self.feed_forward = PositionwiseFeedForward(d_model=hidden, d_ff=feed_forward_hidden, dropout=dropout)
        self.input_sublayer = SublayerConnection(size=hidden, dropout=dropout)
        self.output_sublayer = SublayerConnection(size=hidden, dropout=dropout)
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, x, mask):
        x = self.input_sublayer(x, lambda _x: self.attention.forward(_x, _x, _x, mask=mask)) // &lt;-- Error!
        x = self.output_sublayer(x, self.feed_forward)
        return self.dropout(x)
</code></pre>
<p>Any workarounds?</p>
ConvNet+LSTM video classification: use of GPUs

username: sigma_x
cooked: <p>I’m training a classifier for video data (2 classes, n videos, each is represented with a sequence of K frames). I create a dataset:</p>
<p><code>train_data = torch.utils.data.DataLoader(dataset, **params)</code></p>
<p>where <code>dataset</code> instance has <code>__getitem__</code> that returns <code>idx, X,y</code>: idx is the index of the video. X has dimensions <code>(n, K, 3, H, W)</code> and y <code>(n,1)</code></p>
<p>I don’t want to put the whole dataset on GPU, as it constrains the number of frames and the frame size. So I want to loop through each video, put it on GPU, run it throgh ResNet+LSTM, get the loss, proceed to the next video. After the sample’s finishes, I sum the losses and backprop:</p>
<pre><code class="lang-auto">for e in range(epochs):
     total_loss = 0
     optimizer.zero_grad()
     for batch_idx, X,y in train_data:
          for num,id in enumerate(batch_idx):
               X_data,y_label = X[num, :,:,:,:].to("cuda").unsqueeze_(0) ,y[num,:].to("cuda").unsqueeze_(0)
               output = lstm(convnet(X_data))
               loss = binary_loss(sigmoid(output), y_label)
               total_loss += loss
           total_loss.backward()
           optimizer.step()
</code></pre>
<p>And yet every time I put a datapoint on CUDA, additional 1.8Gb of VRAM gets used, and after a few feedforward operations GPU runs out of memory. The models are parallelized (I used nn.DataParallel).</p>
<p>Any suggestions on how to optimize the GPU use are very welcome. Most of all, I don’t understand why putting the whole dataset on GPU takes less VRAM than putting its subsets.</p>
Getting problem to load numpy file in pytorch

username: mchowdhu
cooked: <p>Hi,<br>
I am very new in the Deep learning and Pytorch platform. I have started a project related to image classification with CNN , for this reason I would like to load my custom data set( a folder which contain 4000 numpy files) In Pytorch. I used <a href="https://discuss.pytorch.org/t/loading-npy-files-using-torchvision/28481">DatasetFolder</a> as a reference. But I am unable to load my dataset.<br>
Here is my code</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms


import matplotlib.pyplot as plt
import numpy as np


DATA_TRAIN = '/home/mchow/Downloads/homework/data/pp'



def npy_loader(path):
    sample = torch.from_numpy(np.load(path))
    return sample

#
dataset = datasets.DatasetFolder(
    root='DATA_TRAIN',

    loader=npy_loader,
    extensions=['.npy']
)
</code></pre>
<p>But I got the following errors.I could not solve this problem.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;input&gt;", line 1, in &lt;module&gt;
  File "/home/mchow/Downloads/pycharm-professional-2019.1.3/pycharm-2019.1.3/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/mchow/Downloads/pycharm-professional-2019.1.3/pycharm-2019.1.3/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/mchow/Downloads/Assignment 3.py", line 29, in &lt;module&gt;
    extensions=['.npy']
  File "/home/mchow/miniconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 75, in __init__
    classes, class_to_idx = find_classes(root)
  File "/home/mchow/miniconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 23, in find_classes
    classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
FileNotFoundError: [Errno 2] No such file or directory: 'DATA_TRAIN'

</code></pre>
<p>I will be very grateful,if any one help me out .</p>

username: ptrblck
cooked: <p>Try to set <code>root=DATA_TRAIN</code> instead of the string <code>'DATA_TRAIN'</code>.</p>

username: mchowdhu
cooked: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a> , Now this code worked.</p>
Embeddings from SNLI with transitions produce additional tensor

username: pkhdipraja
cooked: <p>Hello all, I’m trying to create embeddings from SNLI datasets with shift-reduce parser, however it seems when creating the iterator using data.BucketIterator.splits() from torchtext, it created additional tensor compared to the ones without transitions in premise and hypothesis. What is this tensor for? Is it useful for creating embeddings?</p>
<p>Steps to reproduce:</p>
<pre><code class="lang-auto">import os 
import torch
from torchtext import data
from torchtext import datasets

# With transitions
inputs = datasets.nli.ParsedTextField(lower=True)
transitions = datasets.nli.ShiftReduceField()
answers = data.Field(sequential=False)

train, dev, test = datasets.SNLI.splits(inputs, answers, transitions)

# Without transitions
inputs = data.Field(lower=True, tokenize='spacy')
answers = data.Field(sequential=False)
train, dev, test = datasets.SNLI.splits(inputs, answers)

# =====================================================================
device = torch.device('cuda')

inputs.build_vocab(train, dev, test)
if os.path.isfile(os.path.join(os.getcwd(), '.vector_cache/input_vectors.pt')):
    inputs.vocab.vectors = torch.load(os.path.join(os.getcwd(), '.vector_cache/input_vectors.pt'))
else:
    inputs.vocab.load_vectors('glove.42B.300d')
    os.makedirs(os.path.dirname(os.path.join(os.getcwd(), '.vector_cache/input_vectors.pt')), exist_ok=True)
    torch.save(inputs.vocab.vectors, os.path.join(os.getcwd(), '.vector_cache/input_vectors.pt'))
answers.build_vocab(train)

train_iter, dev_iter, test_iter = data.BucketIterator.splits(
    (train, dev, test), batch_size=128, device=device)

batch = next(iter(train_iter))
print(batch)
</code></pre>
Confusion with dataloader

username: chaslie
cooked: <p>Morning all,</p>
<p>I am reading in a 2048x420000 numpy array, this array is 420000 samples each 2048 long.<br>
I convert this np array into a tensor using torch.Tensor and then set up a single tensor 1x2048 long for labels.<br>
Using TensorDataset(X_train, y_train) to combine (I think as i understand???) this is then passed into DataLoader with a batch sie of 32.</p>
<p>This then gives me the error</p>
<p>RuntimeError: Expected 3-dimensional input for 3-dimensional weight 16 1, but got 2-dimensional input of size [32, 42000] instead</p>
<p>I’m confused, why isn’t the tensor generated by dataload [32 , 2048]? it is using the size of the row (42000) and not the columns (2048).</p>
<p>chaslie</p>

username: Deepali
cooked: <p>The error indicates that the supplied input dimension and weight dimension doe not match.<br>
It is expecting a 3 dimensional input.</p>

username: chaslie
cooked: <p>Further to this. Looking at the input vector, it has shape 32x42000, and not 32x3048 as intended, how do I change this?</p>
Mean is not a valid value for reduction

username: wxystudio
cooked: <p>I run the deeplabv3+ on cityscapes dataset, in the training I run the following code:</p>
<p>for cur_step, (images, labels) in enumerate( train_loader ):<br>
if scheduler is not None:<br>
scheduler.step()<br>
<span class="hashtag-raw">#images</span> is ([8,3,512,512]) tensor, labels is  ([8,512,512]) tensor, 8 is the batch_size<br>
images = images.to(device, dtype=torch.float32)<br>
labels = labels.to(device, dtype=torch.long)<br>
print( np.unique(labels.cpu().numpy()) )<br>
# N, C, H, W<br>
optim.zero_grad()<br>
<span class="hashtag-raw">#outputs</span> is ([8,20,512,512]) tensor, 20 is class num<br>
outputs = model(images)<br>
<span class="hashtag-raw">#criterion</span> is Cross Entropy Loss<br>
loss = criterion(outputs, labels)<br>
loss.backward()<br>
optim.step()</p>
<p>in the above code, I got the error<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c171fcbbfe8c280227694df97385f43b44541ca4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c171fcbbfe8c280227694df97385f43b44541ca4" title="IMG_0118"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c171fcbbfe8c280227694df97385f43b44541ca4_2_666x500.jpeg" alt="IMG_0118" data-base62-sha1="rBimPTF3oGgXWGP7kOBahNtso4Y" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c171fcbbfe8c280227694df97385f43b44541ca4_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c171fcbbfe8c280227694df97385f43b44541ca4_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c171fcbbfe8c280227694df97385f43b44541ca4_2_1332x1000.jpeg 2x" data-dominant-color="4B4E51"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_0118</span><span class="informations">3264×2448 2.31 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think the bug is in the " loss = criterion(outputs, labels)", but I dont know how to fix it</p>

username: ptrblck
cooked: <p>How did you initialize the criterion?<br>
Note that <code>'mean'</code> should be passed as a string.<br>
Also, in older PyTorch versions (pre <code>1.0</code>), you had to use <code>'elementwise_mean'</code> as far as I remember.<br>
If you are using one of the older versions, I would strongly recommend to update.</p>
Raise TypeError("Cannot handle this data type") TypeError: Cannot handle this data type

username: madhavkhoslaa
cooked: <p>torchvision.utils.save_image(, file_name) throws this error. while saving outputs from the network.</p>

username: ptrblck
cooked: <p>What are you passing to this method?<br>
Could you print the type of the object so that we could debug it?</p>
Problem with LSTM Time Prediction of a sinus curve

username: N3m0
cooked: <p>Hi Everyone,<br>
i really hope you guys can help me out with my problem. For my Master Thesis i want to program a LSTM that predicts the remaining Life Time of a mechanical part, by using time/vibration signals as input. To get a feeling for LSTM’s i’m trying to train one to predict a sine curve.</p>
<p>This here is <strong><a href="https://github.com/n3iii0/LSTM-Time-Prediction/blob/master/LSTM_0.7.1.py" rel="noopener nofollow ugc">My Code</a></strong>. My model doesn’t train very well hence the predictions are bad as well. The loss starts at 0.5 and decreases really fast after just 3 epochs. I tried tried the multi-one approach, where my input is a seq_len tensor and my output is the one next value after the sequence.</p>
<p>I’d tried so many configurations; Added LSTM layers, changed the hidden size in a range of 2-500,<br>
tried a lr from e-06 - e-03, tried different Sequence length and batch sizes, used different optimizer, but it seems like nothing really works.</p>
<p>Does anyone have a idea what i can try, or does anyone see a mistake in my code? I would be really happy if someone could help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>My latest tries look like this:<br>
Output while training<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9bf0dd9e593f1eb668f1e183622c4a1395316977.png" alt="Training%20outputs" data-base62-sha1="mfw3JMjR4Efh2QpPTGCV6Vhi3oX" width="640" height="480"><br>
Prediction<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/debe8a9319dfd152f4bec0855ef2d5dd134b041f.png" alt="Prediction" data-base62-sha1="vMuel91o0MLSpHIzBWbVhVp4X4H" width="640" height="480"></p>

username: progerz
cooked: <p>Hello there)</p>
<p>Don’t know, if its still important, but I’ve noticed, that you’ve used shuffle:</p>
<pre><code class="lang-auto">    trainloader = DataLoader(Trainingset, batch_size, shuffle = True, drop_last=True, num_workers=0)
    validloader = DataLoader(Validationset, batch_size, shuffle = True, drop_last=True, num_workers=0)
</code></pre>
<p>For LSTM order is important, and by using shuffle you’ve confused LSTM. I’ve changed shuffle to False and it works okay.</p>
Where can I find backward implementation of existing Function in pytorch

username: march1905
cooked: <p>How can I found the backward implementation of a function in the codebase? For example, <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Linear.cpp" rel="nofollow noopener">here</a> I can find the forward step of <code>Linear</code> layer but there is no backward function existing over there.</p>
Terminate called after throwing an instance of 'c10::Error' what(): isTensor() for LSTM

username: wuclddwf
cooked: <p><em>[ Variable[CPUType]{12,64} ]</em><br>
<em>terminate called after throwing an instance of ‘c10::Error’</em><br>
<em>what():  isTensor() ASSERT FAILED at /export/users/long/gits/Pytorch/libtorch-1.0.1/libtorch/include/ATen/core/ivalue.h:205, please report a bug to PyTorch. (toTensor at /export/users/ong/gits/Pytorch/libtorch-1.0.1/libtorch/include/ATen/core/ivalue.h:205)</em><br>
<em>frame <span class="hashtag">#0:</span> std::function&lt;std::string ()&gt;::operator()() const + 0x11 (0x7fe9e2979441 in /export/users/long/gits/Pytorch/libtorch-1.0.1/libtorch/lib/libc10.so)</em><br>
<em>frame <span class="hashtag">#1:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x2a (0x7fe9e2978d7a in /export/users/long/gits/Pytorch/libtorch-1.0.1/libtorch/lib/libc10.so)</em><br>
<em>frame <span class="hashtag">#2:</span> c10::IValue::toTensor() &amp;&amp; + 0xa6 (0x4489de in ./dcgan)</em><br>
<em>frame <span class="hashtag">#3:</span> main + 0x526 (0x4455a0 in ./dcgan)</em><br>
<em>frame <span class="hashtag">#4:</span> __libc_start_main + 0xf5 (0x7fe9e1508c05 in /usr/lib64/libc.so.6)</em><br>
<em>frame <span class="hashtag">#5:</span> ./dcgan() [0x443f9c]</em></p>
<p><strong>Python input</strong><br>
model = torch.load("./model.12.1L.pt", map_location=lambda storage, loc: storage)<br>
hidden = model.init_hidden(args.batch_size)<br>
examples = torch.ones(12, 64).type(torch.LongTensor)<br>
hidden=(torch.zeros(1, 64, 200), torch.zeros(1, 64, 200))<br>
output, hidden = model(examples.to(“cpu”), hidden)</p>
<p><strong>C++ input</strong><br>
torch::Tensor indata1 = torch::ones({12,64}, torch::kLong);<br>
torch::Tensor h0 = torch::from_blob(std::vector(1* 64 * 200, 0.0).data(), {1, 64, 200});<br>
torch::Tensor c0 = torch::from_blob(std::vector(1* 64 * 200, 0.0).data(), {1, 64, 200});<br>
torch::jit::IValue tuple = torch::ivalue::Tuple::create({h0, c0});<br>
at::Tensor output = module-&gt;forward({indata1, tuple}).toTensor();</p>
<p>Trouble:<br>
Python input code can output right  results,  when I transfer to libtorch the problem occurs. anyone kowns how to solve ? Thanks.</p>

username: wuclddwf
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> hi，could you help me?</p>

username: tom
cooked: <p>Comparing</p>
<pre><code class="lang-auto">output, hidden = model(examples.to(“cpu”), hidden)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">at::Tensor output = module-&gt;forward({indata1, tuple}).toTensor();
</code></pre>
<p>you would not expect the <code>.toTensor()</code> on the output to succeed, but you probably have a tuple that you need to unpack. This is why <code>isTensor</code> fails.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: While you might have preferred ptrblck to answer, it’s generally preferred to not tag people to not discourage anyone who wants to contribute from chiming in.</p>

username: wuclddwf
cooked: <p><a class="mention" href="/u/tom">@tom</a><br>
Hi，Thomas. Thanks for your time. I‘ve fixed the problems with following your proposal.</p>
Element 0 of tensors does not require grad?

username: raymongL
cooked: <p>Hey guys, i’m having a problem about the tensors not having gradients? Here is my code. Embeddings is the direct output of the pretrained resnet50, cls is the labels given by dataloader. So i’m trying to copy the value of embeddings and cls to update some no-grad values in tcl, and then use these no-grad values to calculate loss for embeddings. In my design, loss_group[i] contains a single-value tensor  loss for embeddings[i], so it could do backward() to update embeddings.</p>
<pre><code># my code
tmp_embeddings = embeddings.data.cpu().numpy()
tmp_cls = cls.cpu().numpy()
tcl.updateCenters(tmp_embeddings, tmp_cls)

loss_group = []
for i in range(len(tmp_cls)):
    cur_cls = tmp_cls[i]
    cur_center = tcl.get_center(cur_cls)
    cur_center = cur_center.cuda()
    critirion = torch.nn.MSELoss()
    loss1 = critirion(embeddings[i], cur_center)

    diff_center = tcl.search_min_center(tmp_embeddings[i])
    diff_center = diff_center.cuda()
    loss2 = critirion(embeddings[i], diff_center)
    loss_group.append(loss1+margin-loss2)

    loss_group = torch.FloatTensor(loss_group)
    loss = F.relu(loss_group)
    loss = loss.mean()

loss.backward()
optimizer.step()
</code></pre>
<p>It comes with the error : RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.<br>
So i tried ‘print(loss_group[0].requires_grad)’ and it says True<br>
Kinda loss here, any help? muchas gracias!</p>

username: ptrblck
cooked: <p>Could you try to use <code>torch.stack</code> instead of <code>torch.FloatTensor</code> to create your <code>loss_group</code>:</p>
<pre><code class="lang-python">loss_group = torch.stack(loss_group)
</code></pre>
<p>If you are creating a new tensor e.g. by using <code>FloatTensor</code>, you will detach the input from the computation graph.</p>

username: raymongL
cooked: <p>Oh, it works! Thanks !</p>
Do I need embedding in LSTM if I have 4D array (DataType: double)

username: Sherzod_Bek
cooked: <p>I want to implement LSTM<br>
input_size:   10x512x7x7   (DataType: double)<br>
output_size: 512x7x7   (DataType: double)</p>
<p>Do I need embedding layer, I heard embedding is used to convert word_to_number…<br>
Could you explain more about embedding layer in LSTM…?</p>
CPU support for LibTorch

username: pratyush911
cooked: <p>Could someone provide a minimal code to use LibTorch on a machine that does not have CUDA? I was running into errors trying the same.</p>

username: jinfagang
cooked: <p>it is already supported with CPU in libtorch. You simply needs convert your tensor to device CPU</p>

username: pratyush911
cooked: <p>Okay I realized that there is a different download option on the website for that. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
<a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-latest.zip" class="onebox" target="_blank" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-latest.zip</a></p>
How to use custom word2vec embedding in torchtext.vocab Vectors

username: Sirui_Li
cooked: <p>I train a custom word2vec embedding file named “word2vec.txt” and I would like to use it in <code>TEXT.build_vocab(train_data, vectors=Vectors("word2vec.txt"))</code> where train_data is my training data in torchtext Dataset.</p>
<p>But I got this issue:</p>
<blockquote>
<p>Vector for token b’\xc2\xa0’ has 301 dimensions, but previously read vectors have 300 dimensions. All vectors must have the same number of dimensions.</p>
</blockquote>
<p>I have checked my embedding file. All vectors are 300 dimensions. If I change the embedding file to pre-trained glove file, it works without any issue.</p>

username: Sirui_Li
cooked: <p>I resolve it by reading embedding file  in “rb” and search the non-break space (b’\xc2\xa0’ ) to delete them. I think the most effective way is to preprocess the training data for word2vec file.</p>
Confusion with regards to Generator ConvTranspose2d and input/output size

username: MasayoMusic
cooked: <p>I am going through the tutorial below, but I am confused as to how the right image shape is created via the generator.</p>
<p><a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a></p>
<p>It seems the latent vector created is a one dimensional vector of size 100:</p>
<p>We are feeding that into ConvTranspose2D as getting back a  512, 8x8?</p>
<p><code>ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)</code></p>
<p>How exactly is this happening?</p>
<p>In the diagram shown within tutorial, 100z is being projected into a 1024, 4, 4 vector then converted to<br>
512,8,8 via ConvTranspore.</p>
<p>Are we skipping this step?  I am assuming the projection would take place using a fc layer and then reshape?</p>
<p>So what exactly is happening here:</p>
<p><code>ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)</code></p>

username: ptrblck
cooked: <p>By feeding the noise tensor of <code>[batch_size, nz, 1, 1]</code> into the first transposed conv layer, the spatial size will be increased.<br>
Let’s have a look at a simple use case using just a single pixel value:</p>
<pre><code class="lang-python">b = 1
nz = 1
noise = torch.ones(b, nz, 1, 1)
conv = nn.ConvTranspose2d( nz, nz, 4, 1, 0, bias=False)

output = conv(noise)
print(output.shape)
&gt; torch.Size([1, 1, 4, 4])
print(conv.weight)
&gt; Parameter containing:
tensor([[[[-0.0689, -0.0482,  0.1806,  0.0298],
          [-0.1211,  0.1254, -0.1988, -0.1285],
          [ 0.0184,  0.1757,  0.1835, -0.0602],
          [ 0.0267, -0.0453, -0.0595,  0.0140]]]], requires_grad=True)
print(output)
&gt; tensor([[[[-0.0689, -0.0482,  0.1806,  0.0298],
            [-0.1211,  0.1254, -0.1988, -0.1285],
            [ 0.0184,  0.1757,  0.1835, -0.0602],
            [ 0.0267, -0.0453, -0.0595,  0.0140]]]],
         grad_fn=&lt;ConvTranspose2DBackward&gt;)
</code></pre>
<p>As you can see in this example the kernel will just be multiplied with the single input value.<br>
Since I’ve initialized it as a 1, you’ll get the same output value.</p>
<p>Note that if you are using more than a single input channel, the output will be the sum over the input channels of the weight kernels in this example.</p>

username: MasayoMusic
cooked: <p>Thank you so much for this example. I forgot I could just do tests on my own especially since Pytorch graphs are dynamic. I will play around with strides as well.</p>
Running network twice with different set of parameters in single epoch/optimizer step

username: Toshi
cooked: <p>I’m trying to implement a network in which I am trying to run the network twice in one epoch with different sets of parameters. The workflow goes like the following (G being the network and P1 and P2 being subset of parameters of G):</p>
<p>G(with P1 set of parameters) --&gt; Loss Calculation --&gt; G.backwards() --&gt; G(exchange P1 with P2) --&gt; Loss Calculation --&gt; G.backwards() --&gt; optimizer.step()</p>
<p>P1 and P2 are mutually exclusive and get updates by the different set of gradients while rest of the parameters get updates by the combined backprop. How should I go about it?</p>
Combine mask and selection by dimension

username: LeanderK
cooked: <p>I currently have a lot of code that looks like:</p>
<p><code>tensor[mask, :][..., i]</code></p>
<p>because apparently, I can’t mix masks and selection by dimensions. <code>tensor[mask, i]</code> complains about dimensions not working out, but the mask has the shape of the tensor up to the last dimension.</p>
<p>Any better way to achieve what I want to do?</p>

username: ptrblck
cooked: <p>Is <code>mask</code> a <code>uint8</code> or <code>bool</code> tensor?<br>
If so, could you post a small example of what you would like to achieve?</p>
Loss does not change all the time

username: se7en
cooked: <pre><code class="lang-auto">Training...
LR 1e-05
Train Epoch: 0 [0/1 (0%)] Loss 2.4733 (2.4733) 
LR 1e-05
Train Epoch: 1 [0/1 (0%)] Loss 2.4733 (2.4733) 
LR 1e-05
Train Epoch: 2 [0/1 (0%)] Loss 2.4733 (2.4733) 
LR 1e-05
Train Epoch: 3 [0/1 (0%)] Loss 2.4733 (2.4733) 
LR 1e-05
Train Epoch: 4 [0/1 (0%)] Loss 2.4733 (2.4733) 
LR 1e-05
</code></pre>
<p>Any idea?</p>

username: Prerna_Dhareshwar
cooked: <p>Could you post more of your code? Looks like there is something wrong, loss should decrease.</p>

username: se7en
cooked: <p>Thanks for your response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  problem solved now.</p>
Code handwritting recognition

username: buinhatquan
cooked: <p>Excuse me! I dont understand this code, what is the “data” . I have seen in website but still not understand. Please help me. Thanks you so much .</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/34b31b9072c068b0e8ebd3cd945a42b6491c0ef4.png" alt="1233213" data-base62-sha1="7wcz1pm8bBFQEgjV8r83WW50ZJa" width="665" height="241"></p>

username: sonnguyen
cooked: <p><em>(data, target)</em> are whatever you return from your <em>train_data_loader</em>.<br>
You better come back to the part code of <em>train_data_loader</em> to understand what does it return.</p>
Calculate mean absolute deviation in PyTorch

username: blackbirdbarber
cooked: <p>Is there a single function for this:</p>
<pre><code class="lang-auto">t = torch.rand(3) #tensor
m = t.mean() #mean
print(t)
print(m)
ma = (t-m) # mean absolute
print(ma)
ama = ma.abs() # make it absolute
print(ama)
mama = ama.mean() # take the mean
print(mama)

# tensor([0.3468, 0.4841, 0.3940])
# tensor(0.4083)
# tensor([-0.0615,  0.0758, -0.0143])
# tensor([0.0615, 0.0758, 0.0143])
# tensor(0.0505)
</code></pre>
Loading model and weight each time in the for loop?

username: Saewon_Yang
cooked: <blockquote>
<p>all_prediction = np.zeros((len(test_dataset), num_classes))</p>
<p>for i, weight in enumerate(weight_list):</p>
<pre><code>for _ in range(tta):
    
    model = ...

    model.cuda()
    model.load_state_dict(torch.load(weights_path / weight))

    model.eval()
    
    prediction = np.zeros((len(test_dataset), num_classes)) # num_classes=196
    with torch.no_grad():
        for i, images in enumerate(test_loader):
            images = images.cuda()

            preds = model(images).detach()
            prediction[i * batch_size: (i+1) * batch_size] = preds.cpu().numpy()
            all_prediction = all_prediction + prediction
</code></pre>
<p>all_prediction /= total_num_models</p>
</blockquote>
<p>as you can see in my code. I’m calling model and weight in the for loop.<br>
Do I have to call  model everytime I load my weight? or could I just load model once and override weight?</p>

username: Tahir
cooked: <p>No, you dont have to load the weights each iteration. You normally load it once at the start of your program.</p>

username: Saewon_Yang
cooked: <p>but as you can see in my code. I have different weight lists trained with different folds. meaning that I will get different predictions for each weights.</p>

username: Tahir
cooked: <p>Ok.</p>
<p>If your question is should you call the model constructor every time to create a new object in the model variable to load the weight into, then no you dont have to. You can just load the weight each time but create the model object at start once, IMO.</p>
Parallelize inputs on CPU

username: matthieuheitz
cooked: <p>I have a learning algorithm that tries to reconstruct the input data using a particular model.<br>
My forward problem is to reconstruct the 10 input data, and compute the sum of the losses between reconstructions and inputs.<br>
I would like the 10 reconstructions to be done in parallel on the CPU, with 10 processes.<br>
I don’t know if it’s even possible, meaning if autograd can be used in different processes.</p>
<p>I have tried so many approaches, mainly with <code>torch.multiprocessing.spawn</code> and there always seems to be a problem with different things.</p>
<p>What is the best way to do this ?<br>
Here is the single-process code:</p>
<pre><code class="lang-auto">import numpy as np
import torch


def forward_routine(leaf):

    xi = lambda x: apply_kernel(leaf,x,extra_param)
    
    err = np.zeros([S,L])
    for j in range(S):      # S = 10
        # xi is a function that uses the leaf variable, so what it returns requires grad.
        # The function 'reconstruct' uses xi many times.
        # rec is the reconstruction, that requires grad. 
        # P is a parameter that is constant for each reconstruction
        # err is a numpy array.
        rec, err[j] = reconstruct(P, w[j], xi)
        
        loss_cur = loss_func(rec, obs[j])
        loss = loss + loss_cur

    return loss


if __name__ == "__main__":

    # x0 is a numpy array, let's say of size (D,N)
    # I give this functor to the scipy's LBFGS routine.
    
    def torch_func(x0):
        leaf = torch.from_numpy(x0, requires_grad=True)
        loss = forward_routine(leaf)
        loss.backward()
        
        grad = leaf.grad
        return loss.numpy(), grad.numpy()
    
    # Read input data
    # Run LBFGS, with torch_func as the function to minimize.
</code></pre>
<p>I know that lambda functions can’t be pickled so I wrote a function object for xi, but I didn’t add it to be short.</p>
How to change a loaded model to evaluation mode in C++

username: Panpan_CAI
cooked: <p>I loaded a model in my C++ code in this way:</p>
<p><code>std::shared_ptr&lt;torch::jit::script::Module&gt; model = torch::jit::load("model.pt");</code></p>
<p>Is there an equivalent of the python model.eval() to turn it into evaluation mode in the C++ frontend?</p>

username: pszabo
cooked: <p>Yes, just call  model.eval()</p>
<p>related doc:<br>
<a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html</a></p>

username: Panpan_CAI
cooked: <p>When I use model.eval(), the compiler says:</p>
<blockquote>
<p>error: ‘struct torch::jit::script::Module’ has no member named ‘eval’<br>
net-&gt;eval();</p>
</blockquote>
<p>Do I need to include some additional header file?</p>

username: pszabo
cooked: <p>You are right, unfortunately it is not that easy. The main problem is that jit::scriptmodule and nn::module do not have the same set of methods. I do not know the solution, but in this post they write that eval is not yet implemented in scriptmodule, but they suggest a workaround and there seems to be a brand new fix.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/13991" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/13991</a></p>

username: yf225
cooked: <p><a class="mention" href="/u/panpan_cai">@Panpan_CAI</a> calling <code>model.eval()</code> on a <code>torch::jit::script::Module</code> already works. Example:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/a3346e100e7f4e7ec90f18b7befcccc47d5a1c82/test/cpp/jit/test_misc.h#L445-L450" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/a3346e100e7f4e7ec90f18b7befcccc47d5a1c82/test/cpp/jit/test_misc.h#L445-L450" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/a3346e100e7f4e7ec90f18b7befcccc47d5a1c82/test/cpp/jit/test_misc.h#L445-L450</a></h4>
<pre class="onebox"><code class="lang-h"><ol class="start lines" start="445" style="counter-reset: li-counter 444 ;">
<li>torch::jit::script::Module module = torch::jit::load(module_path);</li>
<li>AT_ASSERT(module.get_module("dropout").is_training());</li>
<li>module.eval();</li>
<li>AT_ASSERT(!module.get_module("dropout").is_training());</li>
<li>module.train();</li>
<li>AT_ASSERT(module.get_module("dropout").is_training());</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Getting nan after the first backward pass even after clipping the gradients

username: sthitap2
cooked: <p>I have two outputs of my model of the shape [1,1024] and want to compute the cosine similarity loss between the vectors to update the weights.<br>
Even after gradient value clipping I get nan after the backward pass.</p>
<p>Please help</p>
<pre><code class="lang-auto">epochs=100
batch_size=128
clip=1

model=ReverseDictionaryUnifiedPipeline(d_model).cuda()

for p in model.parameters():
    if p.dim() &gt;1:
        nn.init.xavier_uniform_(p)
        


optim=torch.optim.Adam(model.parameters(),lr=0.00001,betas=(0.9, 0.98),eps=1e-9)
loss_func=nn.CosineSimilarity(dim=1,eps=1e-6)
for i in range(0,epochs):
    for ind in range(0,(len(X)//batch_size)+1):
        X_train=X[ind*batch_size:(ind+1)*batch_size]
        Y_train=torch.from_numpy(Y_grnd[ind*batch_size:(ind+1)*batch_size]).cuda(0)
        optim.zero_grad()
        output=model(X_train)
        loss1=1-loss_func(output[0][:,0,:],Y_train[:,0,:])
        loss2=1-loss_func(output[1][:,0,:],Y_train[:,1,:])
        print((loss1+loss2))
        loss=torch.sum(loss1+loss2,dim=0)
        print(loss)
        loss.backward()
        nn.utils.clip_grad_value_(model.parameters(), clip)
        optim.step()
        torch.cuda.empty_cache()
        print("Batch "+str(ind)+" loss:",loss)
    print("Epoch "+str(i)+" loss:",loss)  
</code></pre>

username: Tahir
cooked: <p>Its difficult to judge this without looking at your model definition. Can you post the model?</p>
2D Matrix Convolution Question

username: isalirezag
cooked: <p>This is probably very silly question. However, I could not find an answer for it.</p>
<p>given that I have Matrix A (with the size of NxN), and Kernel K (with the size of MxM) how I can get the output B, where: B = A*K? where * is the 2d-convolution sign</p>
<p>P.S. I did looked at torch.nn.functional.conv2d, but im not sure how we can define the kernel in that. I am not even sure if it is doing what I need…</p>

username: richard
cooked: <p>I’m not sure if <code>conv2d</code> is what you’re looking for, but here’s how to adapt it to take your inputs.</p>
<p>Try this:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
from torch.autograd import Variable
N = 4
M = 3
A = Variable(torch.randn(1, 1, N, N))
M = Variable(torch.ones(1, 1, M, M))
output = F.conv2d(A, M)
</code></pre>
<p>You can set <code>A</code> to be any <code>N x N</code> matrix with <code>A[0,0,:] = A2</code>, where <code>A2</code> is <code>N x N</code>. Same with <code>M[0, 0, :] = M2</code> where <code>M2</code> is <code>M by M</code>.</p>

username: isalirezag
cooked: <p>Yeah, Can you please tell me how would you do 2d convolution in pytorch if you dont want to use conv2d? or if you wanna use another way other than conv2…<br>
I mean is there any other way to do it here in pytorch?</p>
<p>Thanks</p>

username: smth
cooked: <p>with the functional.conv2d, the kernel is the <code>weight</code> argument</p>

username: isalirezag
cooked: <p>sorry for retrieving this question.<br>
lets say i wanna do convolution with a specific filter on tensors.<br>
e.g. my kernel is:<br>
<code>H = torch.Tensor([[1 ,0, -1],[2, 0 ,-2], [1, 0 ,-1]])</code>, so it is <code>3x3</code> originally, but i can expand it in a size that is needed.<br>
If my tensor T, has size of <code>BxCxDxD</code>, What should be the size and shape of my kernel so i can do <code>conv2d(T,H)</code> and get the output of <code>BxCxDxD</code></p>

username: rotem_tu
cooked: <p>what if i have RGB image (3xNxN) and 2D kernel (MxM)?</p>
Training slows down and stops very quick

username: Milky
cooked: <p>Hello,</p>
<p>I have DQN training that I’m doing, and I’ve messed around with the hyperparameters a lot and the best results I have so far, it learns a bit until around 10k-20k episodes, then it stagnates, and the average top return it gets starts to stay the same. The settings I have are:</p>
<p>Batch size = 128<br>
gamma = 0.999<br>
eps start = 0.99<br>
eps end  = 0.05<br>
eps decay = 100000<br>
target update = 500<br>
Learning rate = 0.0005</p>
<p>I’ve tried many settings, decaying till later, different learning rates, keeping exploration rate constant. I realize the exploration rate is still low at 10k episodes if it decays at 100k, however it is weird that after around 10k and 20k, suddenly, it stays same and the average scores keep start to go up and down a lot, here’s a screenshot:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8ae91f7396f754f8eb973e22a5dd907b82441bed.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ae91f7396f754f8eb973e22a5dd907b82441bed" title="500%20target%20update"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8ae91f7396f754f8eb973e22a5dd907b82441bed_2_690x367.png" alt="500%20target%20update" data-base62-sha1="jORlY5iYk5C6SaHMVvqEAH3Ofa5" width="690" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8ae91f7396f754f8eb973e22a5dd907b82441bed_2_690x367.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8ae91f7396f754f8eb973e22a5dd907b82441bed_2_1035x550.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8ae91f7396f754f8eb973e22a5dd907b82441bed_2_1380x734.png 2x" data-dominant-color="C7DBE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">500%20target%20update</span><span class="informations">1920×1023 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any ideas?<br>
So it seems to me like it already starts converging into something very quick even though it has high exploration rate still</p>