akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Nan gradient with MultiHeadAttention when passing a key mask

EtienneT: <p>While using MultiHeadAttention, I got the following error message when using with autograd.detect_anomaly():</p>
<p><code>Function 'PowBackward0' returned nan values in its 0th output.</code></p>
<p>The error seemed related to the mask I was passing.  I was calculating the mask like so:</p>
<pre><code class="lang-auto">mask = torch.arange(question_mids.shape[1]).repeat((question_mids.shape[0],1)).to(children_lengths.device) &lt; children_lengths
mask = ~mask

question_mids = question_mids.permute(1,0,2)
question_mids, attention = self.attn(question_mids, question_mids, question_mids)
</code></pre>
<p>I finally figured out that the problem was related to ~mask and I simply changed the line before to avoid having to use the negation sign.  But I was wondering why this negation might cause gradient problems.</p>
<p>Thanks,</p>
Large runing time difference between P100 and V100

taod: <p>I am writing a script to register two images using PyTorch and I test my code on both V100 and P100 GPU. It turns out P100 (running time is about 30 seconds ) is much slower than V100 (2-3 seconds). Usually V100 gives a 2x or 3x speedup over the P100. Does anyone has an opinion on what causes this large difference?</p>
<p>Environment:</p>
<pre><code class="lang-auto">Ubuntu: 16.04
PyTorch: 1.1.0
CUDA: 9.2
</code></pre>
<p>My code is like:</p>
<pre><code class="lang-python">fixed = torch.from_numpy(fixed).float().cuda()
moving = torch.from_numpy(moving).float().cuda()
theta = torch.eye(3,4, requires_grad=Ture).unsqueeze(0).float().cuda()
for i in range(max_iteration):
    grid = torcn.nn.functional.affine_grid(theta, fixed.size())
    output = torch.nn.functional.grid_sample(moving, grid)
    optim.zero_grad()
    loss = loss_fn(fixed, output)
    loss.backward()
    optim.step()
</code></pre>

ptrblck: <p>Are you measuring the time for a specific operation or the forward/backward pass?<br>
If so, note that you would need to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>, since CUDA operations are asynchronous.</p>

taod: <p>Thanks for your reply. I am measuring the time for registration process (for loop) and data loading step is excluded. I have added <code>torch.cuda.synchronize()</code> before starting and stopping the timer and the difference still exists.</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/issues/34576">Related upstream issue</a>.<br>
Let’s continue the discussion there.</p>
Indexing in C++

Suyash458: <p>I’m trying to replicate the following Python code in the PyTorch C++ frontend:</p>
<pre><code class="lang-auto">value = torch::rand({2, 2, 4});
value = value[0, :1, ...]
</code></pre>
<p>I’m not sure how this’d look like if I’m using <code>tensor.narrow()</code> or <code>tensor.select()</code></p>

LeviViana: <p>The <code>0</code> in <code>value[0, :1, ...]</code> is equivalent to <code>value.select(0, 0)</code>, and the <code>:1</code> is equivalent to <code>.narrow(0, 0, 1)</code>, then:</p>
<pre><code class="lang-auto">value.select(0, 0).narrow(0, 0, 1) == value[0, :1, ...]
</code></pre>
<p>Pay attention to the little catch: you can be tempted to do <code>narrow(1, 0, 1)</code> since you are narrowing the second dimension, but you will actually be narrowing the tensor after the selection, which will make first dimension disappear. What I mean is:</p>
<pre><code class="lang-auto">value.select(0, 0).narrow(0, 0, 1) == value.narrow(1, 0, 1).select(0, 0)
</code></pre>

Suyash458: <p>Thanks for answering Levi! So to generalize things<br>
<code>value[a, :b, ...]</code> is equivalent to <code>value.select(a, 0).narrow(0, 0, b)</code> and <code>value[:b, a, ...]</code> is equivalent to <code>value.narrow(1, 0, b).select(a, 0)</code>?</p>

LeviViana: <p>You can check it by yourself <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> !</p>
<pre><code class="lang-python">import torch
value = torch.rand(3, 4, 5, 6, 7)
a = 2
b = 1
value.select(0, a).narrow(0, 0, b).equal(value[a, :b, ...]) # True
value.narrow(1, 0, b).select(0, a).equal(value[a, :b, ...]) # True
</code></pre>

yf225: <p>Starting from the current nightly build (and PyTorch 1.5 soon), for</p>
<pre><code class="lang-auto">value = value[0, :1, ...]
</code></pre>
<p>we can write</p>
<pre><code class="lang-auto">using namespace torch::indexing;
value = value.index({0, Slice(None, 1), "..."});
</code></pre>
<p>Here is the general translation for <code>Tensor::index</code> and <code>Tensor::index_put_</code> functions:</p>
<pre><code class="lang-auto">Python             C++ (assuming `using namespace torch::indexing`)
-------------------------------------------------------------------
0                  0
None               None
...                "..." or Ellipsis
:                  Slice()
start:stop:step    Slice(start, stop, step)
True / False       true / false
[[1, 2]]           torch::tensor({{1, 2}})
</code></pre>
Prediction using traced torch model

Terbinos_Getachew: <p>How can I make a prediction? using traced torch model( the model is suppose to identify image similarity model , not image classification ) I need help please because am about to make an android app</p>

driazati: <p>You should be able to call it just like you call the model before you trace it. If you are looking for an Android specific example, check out <a href="https://github.com/pytorch/java-demo">this repo</a>.</p>

Terbinos_Getachew: <p>I did implement it on the android but the tensor that I am getting after prediction on my android app is different that of on the desktop</p>

driazati: <p>Can you post a repro that shows how you are calling it on desktop vs andriod?</p>
PyTorch do not load weights if size mismatch

HaziqRazali: <p>I can ignore missing keys when loading weights by setting <code>strict=False</code>. What about weights where there is a size mismatch? How can I ignore it i.e. not load it at all?</p>

ptrblck: <p>You could try to filter out the wrongly shaped parameters from the <code>state_dict</code> and try to load it using <code>strict=False</code> afterwards.</p>
DataParallel with manual scatter

Patrick_Dallaire: <p>With DataParallel, how can we assign exemples to GPUs manually while iterating on data loader?</p>
<p>My dataset contains images of highly variable sizes and we chose to use a batchsize of 1. The automatic scatter in dataParallel will use the batchsize dimension to realize the scatter and will only assign to 1 GPU in this case.</p>
<p>Is there a way to compute compute the backward in multi-GPU fashion in this context?</p>

Yanli_Zhao: <p>Do you want to try to use DistributedDataParallel API, where you can spawn each process running on one GPU</p>
U-net connected to camera for live segmentation

Diauro: <p>I recenlty managed to train (not amazingly) an multiclass u-net module. I was looking around github but I could not find a relatively straightforward example of how to feed a tranined model (saved as a ‘pt’ file) to a live camera through for instance opencv.  I was having a look to [DISCOGAN] (<a href="https://github.com/ptrblck/DiscoGAN/blob/master/discogan/run_inference.py" rel="nofollow noopener">https://github.com/ptrblck/DiscoGAN/blob/master/discogan/run_inference.py</a>) but I must admit that not all the code is not entirely clear to me.</p>
<p>What could be ideal just out of curiosity  to start is to open opencv and feed the live frames as as a sort of (very naively):</p>
<pre><code class="lang-auto">prediction= model(inputs)

pred = prediction.data.cpu().numpy()
</code></pre>
<p>where I guess the inputs should be the frame currently on the live-video and as pred I should get the probability mask for the different classes.</p>
<p>If somebody could direct me torward an example or show a snippet of code would be amazing.<br>
Thanks in advance!</p>

ptrblck: <p>My dummy example uses OpenCV to get the current frame, detect faces in the image, and crop it based on the faces. For visualization I played around with PyGame, as it seemed to work OK and I wanted to take a look at it.</p>
<p>Could you point me to specific lines of code, which are unclear?</p>

Diauro: <p>After looking more in detail to your code now is much more clear and I managed to modify it with my needings. Sorry for the bothering :).</p>
<p>I noticed that however the model used from here <a href="https://github.com/usuyama/pytorch-unet/blob/master/pytorch_unet.py" rel="nofollow noopener">U-net</a> is extremely big and therefore I cannot have a “live segmentation” because it’s too slow to process the current frame. I guess the only issue in this would be to reduce the size of the network. Is there a simple way to reduce the size of this U-net and see with training if its still work in a decent way?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Good to hear, you figured it out! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The simplest way would be to reduce the channels by a constant factor and check, if the model is still training and gets a speedup.</p>

Diauro: <p>Could you show me a sort of example or code snippet please? I could not find much and it’s not completely clear how to proceed.</p>
<p>Thanks a lot!</p>

ptrblck: <p>The UNet definition you’ve linked uses a defined number of input and output channels for the convolution blocks. You could simply divide all in and out channels by 2 to reduce the number of parameters.</p>
AttributeError: 'module' object has no attribute 'define_optimizers'

M11: <p>The following error came out when implementing with reference to fineGAN. I can’t solve it after trying a lot.<br>
Is there any good way?</p>
<p>Ubuntu18.04<br>
python2.7.17<br>
conda 4.8.2<br>
pytorch 0.41</p>
<pre><code class="lang-auto">    def train(self):
        self.netG, self.netsD, self.num_Ds, start_count = load_network(self.gpus)
        avg_param_G = copy_G_params(self.netG)
        self.optimizerG,self.optimizersD = define_optimizers(self.netG,self.netsD)

        self.criterion = nn.BCELoss(reduce=False)
        self.criterion_one = nn.BCELoss()
        self.criterion_class = nn.CrossEntropyLoss()

        self.real_labels = \
            Variable(torch.FloatTensor(self.batch_size).fill_(1))
        self.fake_labels = \
            Variable(torch.FloatTensor(self.batch_size).fill_(0))

        nz = cfg.GAN.Z_DIM
        noise = Variable(torch.FloatTensor(self.batch_size, nz))
        fixed_noise = \
            Variable(torch.FloatTensor(self.batch_size, nz).normal_(0, 1))
        hard_noise = \
            Variable(torch.FloatTensor(self.batch_size, nz).normal_(0, 1)).cuda()

        self.patch_stride = float(4)    # Receptive field stride given the current discriminator architecture for background stage
        self.n_out = 24                 # Output size of the discriminator at the background stage; N X N where N = 24
        self.recp_field = 34            # Receptive field of each of the member of N X N

</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 106, in &lt;module&gt;
    algo.train()
  File "/home/hogehoge/finegan-master/code/trainer.py", line 360, in train
    self.optimizerG,self.optimizersD = define_optimizers(self.netG,self.netsD)
  File "/home/hogehoge/finegan-master/code/trainer.py", line 109, in define_optimizers
    betas=(0.5, 0.999))
  File "/home/hogehoge/anaconda3/envs/finegan-master/lib/python2.7/site-packages/torch/optim/adam.py", line 41, in __init__
    super(Adam, self).__init__(params, defaults)
  File "/home/hogehoge/anaconda3/envs/finegan-master/lib/python2.7/site-packages/torch/optim/optimizer.py", line 38, in __init__
    raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list

</code></pre>
<p>thank you</p>

IliasPap: <aside class="quote no-group" data-username="M11" data-post="1" data-topic="72717">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/m11/48/21500_2.png" class="avatar"> M11:</div>
<blockquote>
<p>self.netG, self.netsD, self.num_Ds, start_count = load_network(self.gpus)</p>
</blockquote>
</aside>
<p>Check if you are not returning the models correctly</p>

M11: <p>True, there were other code mistakes. We are fixing it now. Thank you so much.</p>
PCA on hyperspecrtral data (Multi channel array)

pedram1: <p>Hi everyone</p>
<p>I have a multi channels image of the size (65000,16,2), 65000 is number of my samples , 16 in number of channel as 2 is length of my third dimension. I want to apply a PCA on this data set to see which channel is more important?<br>
Dose anyone have any idea?</p>
<p>Thank you all</p>
Seq2seq for NMT why the decoder keeps predicting repeated tokens?

Yingqiang_Gao: <p>I have build a encoder decoder architecture for machine translation. During the inference, I found that the decoder is generating repeated words, something like:</p>
<pre><code class="lang-auto">tensor([[ 2,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  9, 11,  4,  6, 13,
          5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,
          6, 13,  5,  4,  6, 13,  5, 12,  5,  4,  6, 13,  5,  4,  9, 11,  4,  6,
         13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,
          4,  6, 13,  5, 12,  5,  4,  0],
        [ 2,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  9, 11,  4,  6, 13,
          5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,
          6, 13,  5,  4,  6, 13,  5, 12,  5,  4,  6, 13,  5,  4,  9, 11,  4,  6,
         13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,
          4,  6, 13,  5,  4,  6, 13,  0],
        [ 2,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  9, 11,  4,  6, 13,
          5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,
          6, 13,  5,  4,  6, 13,  5, 12,  5,  4,  6, 13,  5,  4,  9, 11,  4,  6,
         13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,  4,  6, 13,  5,
          4,  6, 13,  5,  4,  9, 10,  0]], device='cuda:0')
</code></pre>
<p>I set the beam size to be 3 in this case. The model takes sequence of characters as input and the decoder predicts at each time step a character. I don’t know why this is happening. Any help will be great!</p>

shahensha: <p><a class="mention" href="/u/yingqiang_gao">@Yingqiang_Gao</a> Were you able to figure out the reason?</p>

zhangguanheng66: <p>Do you use transformer architecture?</p>
nn.Conv2d output wrong in iOS?

111255: <p>I try to run libtorch in iOS, but get a weird output. I use the below python code to export the model to file. When load in python it works well, But when load in ios the output looks weird.<br>
export code:</p>
<pre><code class="lang-auto">class TestModule(nn.Module):
  def __init__(self):
    super(TestModule, self).__init__()
    self.op = nn.Sequential(
      nn.Conv2d(1, 2, 3, padding=1),
    )

    self._init_weights()

  def _init_weights(self):
    for m in self.modules():
      if isinstance(m, nn.ConvTranspose2d) or isinstance(m, nn.Conv2d):
        nn.init.kaiming_normal_(m.weight.data)
        m.bias.data.zero_()

  def forward(self, x):
    return self.op(x)

down = TestModule()

script_model = torch.jit.trace(down, torch.rand(1, 1, 4, 4))

print(script_model)

script_model.save('model/down_4.pt')
</code></pre>
<p>Load in iOS:</p>
<pre><code class="lang-auto">    auto module_ = torch::jit::load(modulePath.UTF8String);
    
    at::Tensor input = at::ones({1, 1, 4, 4}, at::kFloat);
    
    torch::autograd::AutoGradMode guard(false);

    auto output = module_.forward({input}).toTensor();
    
    std::cout &lt;&lt; output &lt;&lt; "\n";
</code></pre>
<p>iOS output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e2a4df65cedfb9580a2c6e6933201148714735b.png" alt="截屏2020-03-0301.13.21" data-base62-sha1="6AoBf5LiE8mas62Dxp2OQQk66Qr" width="636" height="390"></p>
<p>libtorch version: 1.4.0</p>
Pytorch docs website really slow and lags computer

Laksh_Aithani: <p>Title says it all.</p>
<p>When I type in e.g torch.nn.LSTM on google, I need to go to the torch.nn page, do ctrl+find LSTM, and finally get to it, somewhere in the middle of this massive torch.nn page.</p>
<p>Furthermore, it’s really laggy scrolling through it.</p>
<p>Also, following a tutorial on the pytorch website is also laggy.</p>
<p>Dell XPS 13, i7-8550u, 16gb ram…</p>

FruitVinegar: <p>Did you checked your internet speed?<br>
I have 2 computers in my home(fast Internet) and office(slow Internet).<br>
There’s no problem accessing Pytorch Docs website when I tried at home, but very slow in my office and even sometimes lose connection.<br>
But however, It seems - Docs website is always slower than Discuss website(I think) cause I can access Discuss site in my office.</p>

PersonalFinance004: <p>I have the same problem. This is a very slow website.</p>

amalik: <p>Blazing fast on my side <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> No clue</p>
Data read from text

sandeep1: <p>In the dataset the Last column is Y_data and other remaining columns are x_train_data<br>
While slicing and converting it into tensor</p>
<p>Question: What does xy[:, 0:-1] or xy[:, [-1]] mean?</p>
<p>from torch import nn, optim, from_numpy<br>
import numpy as np</p>
<p>xy = np.loadtxt(’./data/diabetes.csv.gz’, delimiter=’,’, dtype=np.float32)<br>
x_data = from_numpy(xy[:, 0:-1])<br>
y_data = from_numpy(xy[:, [-1]])<br>
print(f’X’s shape: {x_data.shape} | Y’s shape: {y_data.shape}’)</p>

yash1994: <p>It’s just NumPy array slicing. You get a slice of NumPy array by passing different values arr[v1, v2, v3, …] according to the dimension of your data. Here colon means all values across that dimensions. Consider rows as first dimension and columns as the second dimension in your data:</p>
<ol>
<li>xy[:, 0:-1]: will get you all rows (colon) and all columns except the last column (0:-1).</li>
<li>xy[:, [-1]]: will get you all rows (colon) and last columns ([-1]).</li>
</ol>
<p>I hope it helps.</p>
How to extract CNN features from video frames using pre-trained models? No splitting of data in train and test is recommended

Maheshpal: <p><strong>How to extract CNN features from video frames using pre-trained models?</strong> No splitting of data in train and test is recommended. I have an IACC.3 dataset keyframes, I need to extract visual features using pre-trained network models such as VggNet, ResNet, GoogleNet etc. and store the features in the binary format .bin. I am thinking to use pytorch to write my code.</p>

IliasPap: <p>Check this post on how to extract 2D CNN features  from a video <a href="https://discuss.pytorch.org/t/cnn-lstm-implementation-for-video-classification/52018/7" class="inline-onebox">CNN LSTM implementation for video classification</a></p>
<p>To save them  in binary format I think you can use this function torch.save and io.BytesIO()</p>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.save" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.save</a></p>
How to batch videos into PackedSequences

andresc: <p>Hi! I’ve been learning how to use PyTorch for a month, and I have achieved to process video input data to classify videos or regress a vector from them.</p>
<p>I’ve done this by overfitting data with this structure:</p>
<p>(PackedSequence with ALL the videos) -&gt; (Conv2D layer process all the frames) -&gt; (new PackedSequence with the features extracted from frames) -&gt; (LSTM) -&gt; (Linear layer takes the output of last frame) -&gt; (Final output)</p>
<p>I’ve been able to load the videos in batches of PackedSequence’s manually, but now I want to make use of torch.utils.data.Dataset and I don’t know how to implement it, or if it’s even possible.</p>
<p>Should <code>__getitem__</code> return a PackedSequence? Should I stop using PackedSequence?</p>
<p>Thanks in advance.</p>
Concatenate two tensors in a "Scissors" manner

barakb: <p>Hi I have 2 tensors, let’s say Image with size (batch,3,224,224) each, lets name it T1 and T2.<br>
I want to concatenate the tensor in the channels dimension, means an output of  (batch,6,224,224).<br>
Now I want the output tensor will look like this (In the channel dimension): (1st channel of T1,1st channel of T2,2nd channel of T1,2nd channel of T2,3rd channel of T1,3rd channel of T2)</p>
<p>Instead of the regular concatenate that will give:<br>
(1st channel of T1,2nd channel of T1,3rd channel of T1,1st channel of T2,2nd channel of T2,3rd channel of T2)</p>
<p>What is the fastest way to achieve this output? any Idea?</p>
<p>Thanks!</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">a = torch.arange(0, 3).unsqueeze(0).unsqueeze(2).expand(2, -1, 4)
b = torch.arange(3, 6).unsqueeze(0).unsqueeze(2).expand(2, -1, 4)

c = torch.stack((a, b), dim=2)
print(c.view(2, 6, 4))
</code></pre>
<p>Note that I initialized each  channel with a certain value, so that you can check the interleaved result.</p>

barakb: <p>Very Interesting!<br>
Thanks!</p>

vishnu_vardhan1: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , but what if I have 2 tensors , let’s say Image with size (batch,55,224,224) and (batch,34,224,224), lets name it T1 and T2.<br>
I want to concatenate the tensor in the channels dimension, means an output of (batch,89,224,224).<br>
And it should be a regular concatenation .</p>
<p>I tried torch.cat((t1,t2),3) and got runtime error:invalid argument 0: Sizes of tensors must match except in dimension 3. Got 55 and 34 in dimension 1 at /pytorch/aten/src/TH/generic/THTensor.cpp:612</p>
<p>Please help me with the above problem!<br>
thanks!</p>

ptrblck: <p><code>torch.cat</code> uses the <code>dim</code> argument to define in which dimension the tensors should be concatenated.<br>
In your case, you should use <code>dim=1</code>, since you want to concatenate in the channel dimension.</p>

vishnu_vardhan1: <p>thank you very much  <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

vishnu_vardhan1: <p>what if we want to do the same as per <a class="mention" href="/u/barakb">@barakb</a>, where (1st channel of T1,1st channel of T2,2nd channel of T1,2nd channel of T2,3rd channel of T1,3rd channel of T2) , instead of regular concatenation for different channels like T1 containing 55 channels and T2 containing only 34 channels?</p>
<p>How to solve this problem?<br>
Thanks!</p>

ptrblck: <p>Would you like to use the first 34 channels only in this use case or how would you like to deal with the missing channels in <code>T2</code>?</p>

vishnu_vardhan1: <p>I would like to use the first 34 channels of T1 and second 34 channels of T2 to stack upon each other, 1 channel of T1 and 1 channel of T2 at a time.<br>
after that the rest 21 channels of T1[which has 55 channels] should get concatenated to the resultant<br>
T3 = concat(([T1 of 34 channels ,T2 of 34 channels] one channel at a time), 21 channels  remaining in T1)</p>
Torch.no_grad() doesn't result in speed boost

minhduc0711: <p>I thought with gradient calculations involved, the inference time must be fairly faster. Am I misunderstanding something here?</p>
<pre><code class="lang-auto">import torch
from torchvision.models import vgg16

vgg = vgg16(pretrained=True)
x = torch.randn(1, 3, 480, 640)

%timeit vgg(x)
# 1.36 s ± 93.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
with torch.no_grad():
    vgg(x)
# 1.88 s ± 294 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>

DAlolicorn: <p>I think in your test, the forward loop does not involve gradient calculations.<br>
Thus the speed does not change much if no backward() is called, it may just save you some memory of the gradient placeholders.</p>

minhduc0711: <p>that sounds reasonable, thanks</p>
_pickle.UnpicklingError: pickle data was truncated?

ky_Pa: <p>What is the mistake?I checked and found no path errors. How to modify it? Thank you!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/5974e7296f0d8adba799ab785c81fc85e2d3cb20.png" data-download-href="https://discuss.pytorch.org/uploads/default/5974e7296f0d8adba799ab785c81fc85e2d3cb20" title="微信图片_20200311142103"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/5974e7296f0d8adba799ab785c81fc85e2d3cb20.png" alt="微信图片_20200311142103" data-base62-sha1="cLmXpRsNAeVD0VxVvmRXATGfrAQ" width="690" height="269" data-dominant-color="32332D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20200311142103</span><span class="informations">1909×745 56.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">for step, (testdata, testlabel) in enumerate(src_data_loader):
            
            testdata = make_variable(testdata)
            testlabel = make_variable(testlabel.squeeze_())


</code></pre>
<pre><code>src_dataset = scipy.io.loadmat('E:\\ADDA\\adda-lab-mat\\datasets\\lab_mat\\test_target_domain_sandy')
testdata = src_dataset['testdata'] 
testlabel = src_dataset['testlabel'] 

testdata = torch.from_numpy(testdata).float()
testlabel = torch.from_numpy(testlabel).long() 

src_dataloader = torch.utils.data.DataLoader(
    dataset=src_dataset,
    batch_size=batch_size,
    shuffle=True,
    drop_last=True,
    num_workers=1)

return src_dataloader
</code></pre>

ptrblck: <p>Could you redownload or recreate the data as it seems the file is not complete (truncated).</p>

ky_Pa: <p>Is there any other possible cause of the error? The error persists after I re-download the file.</p>

ptrblck: <p>Oh, it seems you are trying to pass a <code>mat</code> to the <code>DataLoader</code>.<br>
Could you try to create a <code>TensorDataset</code> using the <code>testdata</code> and <code>testlabels</code> and pass this dataset to the <code>DataLoader</code>?</p>

ky_Pa: <p>This time it works, thank you very much! Does the Dataloader have to load a dataset? What if I need to load testdata and testlabel separately?</p>

ptrblck: <p><code>DataLoader</code> accepts <code>Dataset</code>s and should also accept tensors.<br>
However, I would recommend to wrap even tensors into a <code>TensorDataset</code>.<br>
As long as your <code>testdata</code> and <code>testlabel</code> are tensors, it should be fine.</p>

ky_Pa: <p>OK, thank you for your patience.</p>
Pad_sequence DataLoader for batches

ishkeesh12: <p>Getting the following error. Not sure what’s happening given I only give in two inputs?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-70-faa5cb9b2378&gt; in &lt;module&gt;()
----&gt; 1 for batch, target in train_loader:
      2   print(batch)

3 frames
&lt;ipython-input-62-a9f98bef985f&gt; in pad_collate(batch)
      1 def pad_collate(batch):
----&gt; 2   (xx, yy) = zip(*batch)
      3   x_len = [len(x) for x in xx] # length of each tweet
      4 
      5   xx_pad = pad_sequence(xx, batch_first=True, padding_value=0)

ValueError: too many values to unpack (expected 2)
</code></pre>
<pre><code class="lang-auto">X = [torch.tensor(tweet) for tweet in X]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

def pad_collate(batch):
  (xx, yy) = zip(*batch)
  x_len = [len(x) for x in xx] # length of each tweet

  xx_pad = pad_sequence(xx, batch_first=True, padding_value=0)

  return xx_pad, yy, x_len

batch_size = 200

train = (X_train, y_train)

test = (X_test, y_test)

train_loader = DataLoader(train, batch_size=batch_size, shuffle=True, collate_fn=pad_collate)

test_loader = DataLoader(test, batch_size=batch_size, shuffle=True, collate_fn=pad_collate)


for batch, target in train_loader:
  print(batch)
</code></pre>
Unreferrenced function in Pytorch official code

John_Deterious: <p>I’m reading ResNet code from official website, however, I see this norm_layer used without ever defined:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/2611f5cc79230f3729393279093775d08bc52718/torchvision/models/resnet.py#L50" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/2611f5cc79230f3729393279093775d08bc52718/torchvision/models/resnet.py#L50" target="_blank" rel="nofollow noopener">pytorch/vision/blob/2611f5cc79230f3729393279093775d08bc52718/torchvision/models/resnet.py#L50</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
<li>             base_width=64, dilation=1, norm_layer=None):</li>
<li>    super(BasicBlock, self).__init__()</li>
<li>    if norm_layer is None:</li>
<li>        norm_layer = nn.BatchNorm2d</li>
<li>    if groups != 1 or base_width != 64:</li>
<li>        raise ValueError('BasicBlock only supports groups=1 and base_width=64')</li>
<li>    if dilation &gt; 1:</li>
<li>        raise NotImplementedError("Dilation &gt; 1 not supported in BasicBlock")</li>
<li>    # Both self.conv1 and self.downsample layers downsample the input when stride != 1</li>
<li>    self.conv1 = conv3x3(inplanes, planes, stride)</li>
<li class="selected">    self.bn1 = norm_layer(planes)</li>
<li>    self.relu = nn.ReLU(inplace=True)</li>
<li>    self.conv2 = conv3x3(planes, planes)</li>
<li>    self.bn2 = norm_layer(planes)</li>
<li>    self.downsample = downsample</li>
<li>    self.stride = stride</li>
<li>
</li>
<li>def forward(self, x):</li>
<li>    identity = x</li>
<li>
</li>
<li>    out = self.conv1(x)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


IliasPap: <p>It is defined in line 43 otherwise it is None</p>

ptrblck: <p><code>norm_layer</code> is defines as <code>nn.BatchNorm2d</code> by default <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L126-L127">here</a> and passed to the blocks <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L187-L193">here</a>.</p>
Linear regression Basic dobt

sandeep1: <p>I have some basic doubts.</p>
<ol>
<li>If loss.backward() is done, there shouldn’t be requires_grad=True while creating x_data or y_data?</li>
<li>In line y_pred = model(x_data), How does this call forward function, In <strong>init</strong>  , x_data is not taken as parameter or forward function is not called. How come is it is returning predicted y?</li>
<li>What does model(hour_var).data[0][0].item() Mean in the last statement?<br>
Thanks</li>
</ol>
<p>from torch import nn<br>
import torch<br>
from torch import tensor</p>
<p>x_data = tensor([[1.0], [2.0], [3.0]])<br>
y_data = tensor([[2.0], [4.0], [6.0]])</p>
<p>class Model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Model, self).<strong>init</strong>()<br>
self.linear = torch.nn.Linear(1, 1)  # One in and one out</p>
<pre><code>def forward(self, x):
    y_pred = self.linear(x)
    return y_pred
</code></pre>
<p>model = Model()</p>
<p>criterion = torch.nn.MSELoss(reduction=‘sum’)<br>
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)</p>
<p>for epoch in range(500):<br>
y_pred = model(x_data)<br>
loss = criterion(y_pred, y_data)<br>
print(f’Epoch: {epoch} | Loss: {loss.item()} ')<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<h1>After training</h1>
<p>hour_var = tensor([[4.0]])<br>
y_pred = model(hour_var)<br>
print(“Prediction (after training)”,  4, model(hour_var).data[0][0].item())</p>

ptrblck: <ol>
<li>
<p>Usually you don’t care about the gradients in the input tensor or the target, so for most use cases you don’t need to set <code>requires_grad=True</code> for these tensors.</p>
</li>
<li>
<p><code>nn.Module.__call__</code> is called, registers some hooks and calls <code>forward</code> as seen <a href="https://github.com/pytorch/pytorch/blob/8d84c5f1c7a62e9a103a56ee7192eb50570d4045/torch/nn/modules/module.py#L540-L568">here</a>.</p>
</li>
<li>
<p>Don’t use the <code>.data</code> attribute, as it might yield unwanted side effects. The <code>item()</code> call returns a Python literal, e.g. which can be used easily for logging purposes.</p>
</li>
</ol>

sandeep1: <p>Thanks for your beautiful reply,<br>
Can you just explain further, why don’t we care about the gradients in the input tensor or the target?</p>

ptrblck: <p>You could use gradients in the input tensor to update the input directly, e.g. for an adversarial attack.<br>
I’m not sure, when gradients in the target are useful (as you could potentially update the target), but there are most likely use cases I’m not aware of.</p>
How to average tensor elements via an index vector

AkashGanesan: <p>Hi,</p>
<p>I have an index matrix that is of the form</p>
<pre><code class="lang-auto">idx = [[-1,2,3,4],                   # Agent 1
          [0, -1, 2,3],                 # Agent 2
          [1, 1, -1, 2]                 # Agent 3
          [1, 0, 0, -1]                 # Agent 4
]```

Now, I have a bunch of tensors (shape is (#Agents, hidden_layer_size)).  I need a function that averages the agents based on the index for every agent.

So, for example, for agent #2, I need to average hidden layer values from 1 and 2.  -1 represents itself and I won't pool its layer and agent #4 will be as is in #2.  I have the following pooled output tensor of size (#Agents, #Sectors, hidden_layer_size). I have used loops and a dictionary to do them but how would I go about doing this?

Thank you.</code></pre>

abhirut: <p>Did you figure out a way to do this?</p>
Strategy for loading in multiple 3D volumes as input [Image Registration]

batlike: <p>Hello! I’m wondering if there’s a “field” standard for registering 3D volumes. Namely, I have an input volume image (128 x 128 x 128), and I have a deformation field in each of the 3 axis (dx = (128 x 128 x 128), dy = (128 x 128 x 128), dz = (128 x 128 x 128)) that registers this image to a different time point.</p>
<p>My goal is for a network to learn to “apply” the deformation field to “warp” the input volume into a desired output.  As a consequence, my input layer current needs to accommodate four (128 x 128 x 128) volumes, and flattening this results in a massive input layer! I find that downsampling my input volumes doesn’t work well, as it has to be scaled down to such a small 3D volume it is no longer recognizable.</p>
<p>My questions are:</p>
<ol>
<li>What are common strategies for feeding <em>multiple</em> 3D volumes?</li>
<li>What are strategies to accommodate large, but flat input layers?</li>
</ol>
<p>-B</p>
How to normalize data without creating a transform.compose block

copperwiring: <p>This is what I have-</p>
<pre><code>    images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
    masks = Variable(torch.from_numpy(y)).to(device) 
    # print("images",type(images)) # output: tensor class
                    
    images = Normalize(images,(0, 0, 0, 0, 0), (0, 0, 0,0, 0))
    masks =  Normalize(masks, (0.0), (0.0))
    
    optim.zero_grad()        
    outputs = model(images)
    loss = criterion(outputs, masks)
    loss.backward() 
</code></pre>
<p>I get the error <code>TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not Normalize</code><br>
To the best of my understanding, <code>model</code> needs a tensor. How should I give it that?</p>

ptrblck: <p><code>torchvision.transforms.Normalize</code> is a class so you would need to create an instance and apply it with your tensor afterwards:</p>
<pre><code class="lang-python">norm = Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
x = norm(x)
</code></pre>
How to read data from two dataloader at same time

Kapil_Rana: <p>train_dataset = torchvision.datasets.ImageFolder(<br>
root=data_path,<br>
transform=transform1<br>
)</p>
<p>train_loader = torch.utils.data.DataLoader(<br>
train_dataset,<br>
batch_size=batch_size,<br>
num_workers=0,<br>
shuffle=False<br>
)</p>
<p>train_dataset2 = torchvision.datasets.ImageFolder(<br>
root=data_path,<br>
transform=transform1<br>
)</p>
<p>train_loader2 = torch.utils.data.DataLoader(<br>
train_dataset2,<br>
batch_size=batch_size,<br>
num_workers=0,<br>
shuffle=False<br>
)</p>
<p>I want to use image data from these two dataloaders.</p>

Kushaj: <p>How do you want to use the data? You can call the two dataloaders independently in the same iteration.</p>
<pre><code class="lang-auto">train_iter = iter(train_loader)
train_iter2 = iter(train_loader2)
for epoch in range(10):
    for iter in in range(10000): 
        data1 = train_iter.next()
        data2 = train_iter2.next()</code></pre>

Kapil_Rana: <p>I am feeding data to one single model for image classification which has 4 classes. I need data and targets from the dataloaders.</p>

Kushaj: <p>Did the above method work for you?</p>

Kapil_Rana: <p>No,<br>
Actually I am using ImageFolderLoader.</p>
Sparse matrix modification

jojono: <p>Hi,<br>
I am working with sparse matrix and I very frequently modify my matrix by adding new elements or changing its size. How to do it efficiently ?<br>
Because, at the moment, each time I do a modification I use again the constructor torch.sparse.FloatTensor(indices, values, size) and I am also obliged to use the method coalesce (because I am using indices() and values() ) even if I know that my matrix is already coalesced.</p>
<p>How can I simplify and accelerate the modification of my sparse matrix ?<br>
Thank you</p>
Optimizing Custom Hartley Pooling Layer

Jack_Rolph: <p>I have implemented an ND Hartley Pooling function (see <a href="https://arxiv.org/pdf/1810.04028.pdf" rel="nofollow noopener">this paper</a> for more information) for Pytorch, using  <code>cupy</code>  and  <code>sigpy</code> , as shown below.</p>
<p>While this function seems to work fine, is there a way to further optimize this code to make it faster?</p>
<p>I am interested in up to four-dimensional data, so I have tried to use N-dimensionality wherever possible. Sadly, Pytorch does not seem to have a 4-dimensional FFT function, so I have instead used other GPU libraries to perform the FFT on the GPU.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.autograd import Function
import math
import operator
import cupy as cp
import sigpy as sp


def _spectral_crop(array, array_shape, bounding_shape):
    start = tuple(map(lambda a, da: (a-da)//2, array_shape, bounding_shape))
    end = tuple(map(operator.add, start, bounding_shape))
    slices = tuple(map(slice, start, end))
    return array[slices]

def _spectral_pad(array, array_shape, bounding_shape):
    out = cp.zeros(bounding_shape)
    start = tuple(map(lambda a, da: (a-da)//2, bounding_shape, array_shape))
    end = tuple(map(operator.add, start, array_shape))
    slices = tuple(map(slice, start, end))
    out[slices] = array
    return out

def DiscreteHartleyTransform(input):
    N = input.ndim
    axes_n = np.arange(2,N)
    fft = sp.fft(input, axes=axes_n)
    H = fft.real - fft.imag
    return H

def CropForward(input, return_shape):

    output_shape = np.zeros(input.ndim).astype(int)
    output_shape[0] = input.shape[0]
    output_shape[1] = input.shape[1]
    output_shape[2:] = np.asarray(return_shape).astype(int)

    dht = DiscreteHartleyTransform(input)
    dht = _spectral_crop(dht, dht.shape, output_shape)
    dht = DiscreteHartleyTransform(dht)

    return dht

def PadBackward(grad_output, input_shape):
    dht = DiscreteHartleyTransform(grad_output)
    dht = _spectral_pad(dht, dht.shape, input_shape)
    dht = DiscreteHartleyTransform(dht)

    return dht


class SpectralPoolingFunction(Function):
    @staticmethod
    def forward(ctx, input, return_shape):
         input = sp.from_pytorch(input)
         ctx.input_shape = input.shape
         output = CropForward(input, return_shape)
         output = sp.to_pytorch(output)
         output = output.float()
         return output

    @staticmethod
    def backward(ctx, grad_output):
        grad_output = sp.from_pytorch(grad_output)
        grad_input = PadBackward(grad_output, ctx.input_shape)
        grad_input = sp.to_pytorch(grad_input)
        grad_input = grad_input.float()
       return grad_input, None, None


class SpectralPoolNd(nn.Module):
    def __init__(self, return_shape):
        super(SpectralPoolNd, self).__init__()
        self.return_shape = return_shape

    def forward(self, input):
        return SpectralPoolingFunction.apply(input, self.return_shape)


</code></pre>
Differentiating through a gradient

Arthur_B: <p>I have a model f parametrized by <code>w</code>, input data <code>x</code> and labelled output <code>y</code>. I’m searching for a synthetic input and output pair <code>(x0, y0)</code>, and a step-size e such that – letting <code>w' = w - e d(loss(f(x0, w), y0))/dw</code> – <code>loss(f(w', x), y)</code> is minimized. To find those through gradient descent, I need to compute the gradient of <code>loss(f(w', x), y)</code> with respect to <code>x0</code>, <code>y0</code>, and <code>e</code>.</p>
<p>When <code>f</code> is a relatively simple neural network I can compute the update directly fairly easily, but I would like to use torch to compute those gradients directly. Is it possible to differentiate through the taking of a gradient like this? I’m new to Torch, and I understand this might be possible by using <code>keep_graph = True</code>. However, I’ve pained a bit to make it work.</p>
<p>Concretely, if <code>f</code> is a <code>nn.Sequential</code> model piling on various layers, how would one write the update to the parameters so that I can then take a gradient with respect to <code>x0</code> and <code>e</code>. It seems straightforward to update the parameters of the various layers found in <code>nn.</code> when using an off-the-shelf optimizer but this requires something a bit less standard. Is it still possible?</p>
<p>The general idea is to greedily construct a synthetic “curriculum” which can be used to train a network from scratch quickly. It might be possible to devise a curriculum on a small network and use it to quickly initialize a much larger network for instance. It helps if you think as <code>x</code> and <code>y</code> as the whole dataset, and <code>x0</code> and <code>y0</code> as a synthetic mini-batch, typically with only one sample.</p>
Building a loss function without neural nets

Tiger_Dai: <p>Hi, I’m applying pytorch to a character-based language model. I have to redefine the loss function according to the customized model. The consequence is the optimization becomes extremely slow, especially in the multiple embedded for loops here. I wonder if there’s any way to convert the <code>NLL()</code> and <code>coemitNumerators()</code> here to the built-in NLLloss function in pytorch? I look forward to hearing any thoughts, and love to share the full code if anyone is interested!</p>
<pre><code class="lang-auto">def coemitNumerators(word, ix_inventory, automata):
    states = [0] * len(automata)

    numerators = {}
    
    for pos, someSeg in enumerate(word):
        for seg in ix_inventory:
            numerators[pos, seg] = torch.tensor(1.) 
    # In each pos of a word, set the transition prob of a phoneme to 1.

    for somePos, someSeg in enumerate(word):
        for i, M in enumerate(automata):
            for seg in ix_inventory:
                numerators[somePos, seg] = numerators[somePos, seg] * M.tProb[(states[i], seg)]
            states[i] = M.D[(states[i], someSeg.item())]
    return numerators
</code></pre>
<pre><code class="lang-auto">def NLL(wlist, automata, inventory):
    nll = 0
    for w in wlist:
        numerators = coemitNumerators(w, inventory, automata)
        for realpos, realseg in enumerate(w):
            nll = nll - torch.log(numerators[realpos, realseg.item()] / sum(numerators[realpos, seg] for seg in inventory))
    return nll
</code></pre>
<pre><code class="lang-auto">def train_lm(dataset, dev, params, automata, ix_inventory, phone2ix, ix2phone, prev_perplexity = 1e10):

    variables = []
    for automaton in automata:
        variables += automaton.tProb.values()
    optimizer = torch.optim.Adam(variables, params['learning_rate'])
    num_examples, seq_len = dataset.size()
    batches = [
        (start, start + params['batch_size'])
        for start in range(0, num_examples, params['batch_size'])
    ]

    for epoch in range(params['epochs']):
        ep_loss = 0.
        start_time = time.time()
        random.shuffle(batches)
        for b_idx, (start, end) in enumerate(batches):
            loss = NLL(dataset[start:end], automata, ix_inventory)
            optimizer.zero_grad()
            
            loss.backward()
            optimizer.step()
            with torch.no_grad():
                for v in variables:
                    v.clamp_(0, 1)
            ep_loss += loss.detach()
</code></pre>
Why pytorch can use gpu calculation without cuda installed on my PC

KingOfPuu: <p>CUDA has never been installed on my computer. Is it possible that NVIDIA’s graphics driver has integrated CUDA</p>

theairbend3r: <p>I think when you install PyTorch, cuda and other software come bundled with it. So, you don’t have to manually install cuda to use PyTorch. The only catch is that your driver must be updated to whatever version is required by cuda.</p>
<p>Need confirmation on this tho.</p>

ptrblck: <p>That is correct. The binaries ship with additional libs such as CUDA, cudnn etc. so that you would have to make sure the GPU can be detected and has sufficiently new drivers.</p>
Converting Con2d result to Linear

uhotspot4: <p>I’m trying to implement a convolutional autoencoder and when I flatten the tensor resulting from Con2D layer to feed into the Linear layer, I realize that the tensor is not from one but many samples. What is the correct way to make this conversion?</p>

ilhamfp: <p>Hi <a class="mention" href="/u/uhotspot4">@uhotspot4</a>,</p>
<p>I believe that’s to be expected if you feed the network in batch! So flatten the tensor resulting from Conv2D by batch too. You can do this using <code>.view(-1, &lt;the expected value of flattened conv2d&gt;)</code>. The <code>-1</code> means it will automatically adjust to your batch size.</p>
<p>Take a look at this sample code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))

        x = x.view(-1, 320) # this is the flatten part

        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)
</code></pre>
<p>Please correct me if I’m wrong <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>