akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Return the data according to the torch.tensor

username: shangguan91
cooked: <p>i have a datalist[0:59]. i used k-folder cross validation, split the data:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aaac8fbd19d6a3b51ac0c0af3b08e0629fbe526.png" data-download-href="https://discuss.pytorch.org/uploads/default/2aaac8fbd19d6a3b51ac0c0af3b08e0629fbe526" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aaac8fbd19d6a3b51ac0c0af3b08e0629fbe526.png" alt="image" data-base62-sha1="65rXsTZb5VNXEprkPXk9p8rDwPk" width="690" height="337" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">932×456 17.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .</p>
<p>But when i try to return the first split for my data, i get error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15ce0f044085556be627bf13a2f6a2eb439903fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/15ce0f044085556be627bf13a2f6a2eb439903fd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15ce0f044085556be627bf13a2f6a2eb439903fd.png" alt="image" data-base62-sha1="36TvldrwVdQLaXD8356naRSI5qZ" width="690" height="182" data-dominant-color="F4F4F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×225 8.12 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .</p>
<p>How can i get the data_list from my tensor</p>

username: ptrblck
cooked: <p>Of yhat types are <code>data_list</code> and <code>train_indices</code>? The first posted <code>list(k)</code> doesn’t seem to be used in the second code snippet.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: shangguan91
cooked: <p>solved by:  train_dataset =  [data_list[index] for index in train_idx]</p>
XNNPACK engine not utilized

username: JustasZ
cooked: <p>Hello,</p>
<p>If I understand correctly, XNNPACK has become the default computation engine for Pytorch mobile after the 1.6 release. I’ve built Pytorch mobile from source according to the following tutorial: <a href="https://pytorch.org/mobile/android/#building-pytorch-android-from-source" rel="nofollow noopener">https://pytorch.org/mobile/android/#building-pytorch-android-from-source</a> with -DUSE_XNNPACK=ON, but XNNPACK engine is still not being used (verified by putting some print statements in aten/src/ATen/native/xnnpack/Convolution.cpp). Any help on how to resolve this issue would be appreciated.</p>
Make batch with different size tensors

username: 032june
cooked: <p>Let we have four images per a batch. (4 x 3 x W x H)<br>
During the loop, I extract features from the images using CNN. (4 x C x w x h)</p>
<p>Here, for every image, we have pre-defined mask with different size. (Let, M1 ~ M4)<br>
I want to select and concatenate the features at the given positions only.<br>
So, now the features are C x M1 ~ C X M4.</p>
<p>As you can see, this process makes the selected features to have different size.<br>
So I failed to make them as a batch.</p>
<p>I want to put the features to another CNN based network.<br>
Of course I can put them separately, but this took really long time.<br>
How can I do this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid you won’t be able to create a single Tensor with different sizes on a single dim.<br>
You might be able to create a Tensor with the largest size though and copy all the elements in it. Padding with 0s for the Tensors that are smaller than the full size.</p>

username: 032june
cooked: <p>Hi,</p>
<p>I already considered your idea, but it requires additional dummy memory for zeros…<br>
Anyway thank you for your response.</p>
When auto a = tensor, what happened?

username: tgsmdww
cooked: <p>In ConstantPadNd.cpp, I find this:</p>
<pre><code class="lang-auto">    auto output = at::empty(new_shape, self.options());
    output.fill_(value);

    auto c_output = output;
    for (int i = l_diff; i &lt; l_inp; i++) {
        auto pad_idx = 2 * (l_inp - i - 1);
        if (pad[pad_idx] &gt; 0) {
            c_output = c_output.narrow(i, pad[pad_idx], c_output.size(i) - pad[pad_idx]);
        }
        if (pad[pad_idx + 1] &gt; 0) {
            c_output = c_output.narrow(i, 0, c_output.size(i) - pad[pad_idx + 1]);
        }
    }
    c_output.copy_(c_input);
    return output;
}
</code></pre>
<p>Does c_output share tensorImpl with output?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/tgsmdww">@tgsmdww</a><br>
auto output = at::empty(…) -&gt; output is a newly allocated tensor<br>
auto c_output = output -&gt; c_output and output share the same underlying storage<br>
c_output = c_output.narrow -&gt; c_output’s storage is not change.<br>
c_output.copy_(c_input) -&gt; c_output copy c_input into its storage</p>
<p>return output -&gt; output shared storage with c_output due to above.</p>

username: tgsmdww
cooked: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:">!!!</p>
How to find "weight" of nn.module?

username: luckydog1996
cooked: <p>I am new to pytorch, and I am reading a piece of code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">import torch.nn.init as init
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                init.orthogonal_(m.weight)
                if m.bias is not None:
                    init.constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                init.constant_(m.weight, 1)
                init.constant_(m.bias, 0)
</code></pre>
<p>I read the document of nn.module and do not find any detail about this weight attribute, so please help me to find out where does this attribute defined. How to know what attribute or method I have when I use a nn.module?</p>

username: mariosasko
cooked: <p><code>nn.Module</code> is a base class for other modules and its purpose is not only to store the learnable parameters (weights/biases) but hooks, buffers as well. In PyTorch, modules are not only the network layers (<code>nn.Conv2d</code>, <code>nn.BatchNorm</code>), but loss functions (<code>nn.MSELoss</code>) or containers (<code>nn.Sequential</code>), to name a few. So it’s not necessary for each module to store the weights. You can check the Variables section in the docs to find out what parameters the module has (e.g. <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="nofollow noopener">for <code>nn.Conv2d</code></a>).</p>
LSTM time sequence GAN implementation

username: Topsoil
cooked: <p>If i want to build a basic LSTM GAN, is that a proper way of implementing it:</p>
<ol>
<li>Take conditioning vector</li>
<li>
<code>.cat()</code> it with a previous output, if it is a first time step, <code>.cat()</code> it with zeros or random tensor</li>
<li>Pass it as input to lstm along with previous hidden, if it is a first time step, init h_0 and c_0 to zeros</li>
<li>Save hidden state (h_n, c_n) into a variable for next time step, take h_n as output of next time step prediction</li>
<li>Go to 1, repeat until desired sequence length is reached</li>
</ol>
<p>Is that a right algorithm?</p>
Rectangular data set best practice

username: chaslie
cooked: <p>Morning,</p>
<p>I have an input array which by its nature is rectangular [-1,1,2a,a]. The questions are:</p>
<p>is it better to keep this shape ratio through the convolutional layers using rectangular filters?<br>
square it up in the first layer and then use square filters in the remaining convolutional layer?<br>
square it up in pre-processing?</p>
<p>chaslie</p>
Spawn, torch.multiprocessing

username: AghaRezaey
cooked: <p>Hello all</p>
<p>We have developed a multilingual TTS service, and we have got several DL models to run at test time , and those models can be run in parallel because they don’t  have any dependencies to each other (trying to get lower runtime, better performance)<br>
We do that on a GPU<br>
but I ran into several problems</p>
<p>A simpler version of it is declared by below codes :</p>
<pre><code>import torch.multiprocessing as mp

def train1():
    print("\nx")
    q5 = np.random.randint(2,size=(4,2))
    q5_targ = torch.tensor(q5).to(torch.device("cuda"))

def train2():
    print("\ny")
    g5 = np.random.randint(2,size=(4,2))
    g5_targ = torch.tensor(g5).to(torch.device("cuda"))

if __name__ == '__main__':
    p1 = mp.Process(target=train1, args=())
    p2 = mp.Process(target=train2, args=())

    p1.start()
    p2.start()
    p1.join()
    p2.join()
</code></pre>
<p>when I run above codes, I get output of</p>
<blockquote>
<p>x<br>
y<br>
.<br>
.<br>
.<br>
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
</blockquote>
<p>and when I run</p>
<blockquote>
<p>from torch.multiprocessing import set_start_method<br>
set_start_method(‘spawn’)</p>
</blockquote>
<p>before those , I would see nothing as my output !<br>
(means it doesn’t even enter target functions )</p>
<p>I am on colab now , and using Tesla T4</p>
<p>Do you know that <a class="mention" href="/u/alband">@albanD</a> ?</p>

username: albanD
cooked: <p>Note that if you spawn the processes <strong>before</strong> doing anything cuda related, you won’t see “RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method” even if you use fork.</p>
<p>I am not sure why you have this behavior with spawn. But multiprocessing in a notebook might be tricky. And you never really kill the python process…</p>

username: AghaRezaey
cooked: <p>I tried running below codes as .py file on our own Linux server and it worked OK<br>
Thanks!<br>
maybe the problem is with .ipynb</p>
<pre><code class="lang-auto">import torch
import numpy as np
import torch.multiprocessing as mp
from torch.multiprocessing import set_start_method

def train1():
    print("\nx")
    q5 = np.random.randint(2,size=(4,2))
    q5_targ = torch.tensor(q5).to(torch.device("cuda"))

def train2():
    print("\ny")
    g5 = np.random.randint(2,size=(4,2))
    g5_targ = torch.tensor(g5).to(torch.device("cuda"))

if __name__ == '__main__':
    set_start_method('spawn')
    p1 = mp.Process(target=train1, args=())
    p2 = mp.Process(target=train2, args=())

    p1.start()
    p2.start()

    p1.join()
    p2.join()
</code></pre>
Visualize class activation maps in C++

username: Pak_Lau
cooked: <p>Hello,<br>
Is it possible in C++?</p><aside class="quote quote-modified" data-post="2" data-topic="29597">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/visualize-feature-map/29597/2">Visualize feature map</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    You can just use a plot library like matplotlib to visualize the output. 


Sure! You could use some loss function like nn.BCELoss as your criterion to reconstruct the images. 


Forward hooks are a good choice to get the activation map for a certain input. 


Here is a small code example as a starter: 
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
import torchv…
  </blockquote>
</aside>

<p>Thanks,</p>

username: Shisho_Sama
cooked: <p>You might want to have a look here <a href="https://github.com/lava/matplotlib-cpp" rel="nofollow noopener">matplotlib-cpp</a></p>

username: Pak_Lau
cooked: <p>This does not solv the problem of creating the actual hook to generate the maps.</p>
Cannot find CUB include files when building from source

username: litxio
cooked: <p>I’m trying to build pytorch on Arch Linux with OpenBLAS.  I’m encountering errors about NVCC not being able to find the CUB include files:</p>
<pre><code class="lang-auto">[3861/5000] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/operators/torch_cuda_generated_affine_channel_op.cu.o
FAILED: caffe2/CMakeFiles/torch_cuda.dir/operators/torch_cuda_generated_affine_channel_op.cu.o 
cd /home/richard/workspace/third_party/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/operators &amp;&amp; /usr/bin/cmake -E make_directory /home/richard/workspace/third_party/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/operators/. &amp;&amp; /usr/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/home/richard/workspace/third_party/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/operators/./torch_cuda_generated_affine_channel_op.cu.o -D generated_cubin_file:STRING=/home/richard/workspace/third_party/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/operators/./torch_cuda_generated_affine_channel_op.cu.o.cubin.txt -P /home/richard/workspace/third_party/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/operators/torch_cuda_generated_affine_channel_op.cu.o.Release.cmake
/home/richard/workspace/third_party/pytorch/caffe2/operators/affine_channel_op.cu:3:10: fatal error: cub/block/block_reduce.cuh: No such file or directory
 #include &lt;cub/block/block_reduce.cuh&gt;
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMake Error at torch_cuda_generated_affine_channel_op.cu.o.Release.cmake:221 (message):
  Error generating
  /home/richard/workspace/third_party/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/operators/./torch_cuda_generated_affine_channel_op.cu.o

</code></pre>
<p>And more errors of this sort from the <code>build/caffe2/CMakeFiles/torch_cuda.dir/operators</code> directory.  This occurs both on master and on 1.6.0.  I can see the needed file is present at third_party/cub/cub/block/block_reduce.cuh.  My build command is:</p>
<pre><code class="lang-auto">CUDA_HOME=/opt/cuda-10.1 CC=/usr/bin/gcc-8 CXX=$(which c++-8)  CMAKE_PREFIX_PATH=~/.local/lib/python3.8 BLAS=OpenBLAS python setup.py build
</code></pre>
CPUAllocator running out of memory while assigning complex tensors

username: a_d
cooked: <p>Hello all,<br>
I just got started with complex tensors and tried assigning a complex tensor of shape (2,2) and it runs out of memory. My code is -</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

a = torch.rand(size=(2,2), dtype=torch.complex32)
print(a)
</code></pre>
<p>and the output error is  -</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/atharva/PycharmProjects/DeepComplexNetwork/main.py", line 4, in &lt;module&gt;
    a = torch.rand(size=(2,2), dtype=torch.complex32)
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 34359738368 bytes. Error code 12 (Cannot allocate memory)
</code></pre>
<p>I was just wondering if complex tensor of type <code>complex32</code> even of size (2,2)  <code>34359738368 bytes</code> big ?<br>
TIA…</p>

username: a_d
cooked: <p>I just realized that it works with complex128 and comples64 but not with complex32… complex32 tensors are bigger?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Thanks for the report! This definitely looks like a bug: <a href="https://github.com/pytorch/pytorch/issues/43143">https://github.com/pytorch/pytorch/issues/43143</a><br>
Note that complex32 is most likely not ready for general use.</p>
<p>Also could you share on the issue your pytorch version and how you installed it please?</p>

username: a_d
cooked: <p>Hi<br>
Sorry for the late response… pytorch version is 1.6.0 and it was installed simply through pip3 - pip3 install torch torchvision</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes we were able to reproduce it and it should be fixed now if you use the nightly build <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: a_d
cooked: <p>Thanks a lot <a class="mention" href="/u/alband">@albanD</a></p>
Issues using Data Parallelism: DataParallel object has no attribute 'xxxxxxxx'

username: add023
cooked: <p>I have an NLP model and I need to use data parallelism due to large batch data size. I wrapped my model using nn.DataParallel. I need the model attributes else where during training, validation, logging etc. Here’s how I’m doing it:</p>
<pre><code class="lang-auto">class Trainer():
    def __init__(self, model, config):
        self.model = model
        if torch.cuda.device_count() &gt; 1:
            print("Let's use", torch.cuda.device_count(), "GPUs!")
            self.model = nn.DataParallel(model)
       
        self.txt_property = self.model.txt_property
        # many more properties

    def do_something(self):
        param = self.model.param
</code></pre>
<p><code>AttributeError: 'DataParallel' object has no attribute 'txt_property'</code></p>
<p>What is the best practice to encapsulate model with <code>nn.DataParallel</code>?</p>

username: ayalaa2
cooked: <p>Since you wrapped it inside DataParallel, those attributes are no longer available. You should be able to do something like <code>self.model.module.txt_property</code> to access those variables.</p>
<p><a href="https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html" rel="noopener nofollow ugc">Be careful with altering these values though</a>:</p>
<blockquote>
<p>In each forward,  <code>module</code>  is  <strong>replicated</strong>  on each device, so any updates to the running module in  <code>forward</code>  will be lost. For example, if  <code>module</code>  has a counter attribute that is incremented in each  <code>forward</code> , it will always stay at the initial value because the update is done on the replicas which are destroyed after  <code>forward</code> .</p>
</blockquote>

username: add023
cooked: <p>Thank you for the reply. Understood about accessing the values. Thanks for the link!</p>
Difference between Train loss and Accuracy in CNN

username: shangguan91
cooked: <p>why isnt Train loss=1- accuracy?<br>
how is train loss calculated?</p>

username: ayalaa2
cooked: <p>Assuming we’re talking about a classification task, a model doesn’t just take in an image and say “it belongs to class A”. Instead, it output some number of values (called logits) which essentially is how confident the model is in some image being class A, B, C, and so on.</p>
<p>For example, say that for some image (that belongs to class A), the model outputs the logits 0.5, 0.4, and 0.1. What we want the model to instead output is something like 1.0, 0.0, 0.0. As in, be very confident that the image belongs to class A and be confident that it doesn’t belong to class B or C. If we only take our loss to be <code>1 - accuracy</code>, this information isn’t conveyed to the model. We want to instead take our loss to be something like <code>cross entropy loss</code>. This is the typical way loss is calculated in classification networks. I would suggest reading more into that loss to get more of an intuition on what’s happening.</p>

username: shangguan91
cooked: <p>Thank you for elaborating. So, accuracy is only be calcualted as correctness/len(data). However, loss is something like probablity. could you pls explain more, how loss is accumulated in first epoch, suppose it has 20 batches. and how could i print(test loss) according to the code below.</p>
<p>`</p>
<pre><code>    def train(epoch):
    model.train()
    for data in train_loader:
    data = data.to(device)
    optimizer.zero_grad()
    loss = F.nll_loss(model(data), data.y)
    loss.backward()
    optimizer.step()

def test(loader):
model.eval()

correct = 0
for data in loader:
    data = data.to(device)

    with torch.no_grad():
        pred = model(data).max(1)[1]
    correct += pred.eq(data.y).sum().item()
return correct / len(loader.dataset)`</code></pre>

username: ayalaa2
cooked: <p>Sure, so I posed it as the model outputting probabilities, but in practice the model actually just outputs a value for each class (e.g, [20, -15, 1]), where we do something like soft-max to convert the logits into probabilities.</p>
<p>In your example, we’re using the negative log-likelihood which is actually what cross entropy loss ends up using. Keep in mind that NLL needs you to first use log softmax on the output of the model (if this isn’t already happening inside the model). See the example <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html" rel="nofollow noopener">here</a> for more details.</p>
<p>So essentially, what NLL will do for each datapoint is consider if that datapoint belongs to some class. If it does, we want the model to be as close to 1.0 as possible. Otherwise, want it as close to 0.0 as possible. This is all done in a differential way so we can compute our gradients where the model understands how to push the value closer to 1.0 or 0.0 depending on the label.</p>
Custom weighted NLL_loss

username: erap129
cooked: <p>Hello,<br>
I want to develop my own NLL_loss which will penalize certain errors by their severity (I want some errors to be punished more than others).<br>
To this end I need to alter the source code of pytorch’s NLL_loss, but it is written in C and I can’t seem to find it.<br>
I found an implementation of CrossEntropyLoss here:<br>
<a href="https://gist.github.com/mjdietzx/50d3c26f1fd543f1808ffffacc987cbf" rel="nofollow noopener">https://gist.github.com/mjdietzx/50d3c26f1fd543f1808ffffacc987cbf</a><br>
and when I test it against the CrossEntropyLoss in Pytorch it gives the same output, although when I use weighted samples the output is different, thus my starting point for altering the code is bad.</p>
<p>My main points are:</p>
<ol>
<li>Is there any out of the box functionality in Pytorch to penalize certain mistakes over others?</li>
<li>If not, I can implement this myself, but I don’t understand how NLL_loss in Pytorch deals with sample weights, which prevents me from achieving my first goal of creating the exact same loss function with my own tweakable implementation.</li>
</ol>
<p>Thanks all.</p>

username: mariosasko
cooked: <p>This is the <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/LossNLL.cpp" rel="nofollow noopener">source code</a> (and <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/LossNLL2d.cpp" rel="nofollow noopener">2D version</a>).</p>
<p>The weights in <code>NLLLoss</code> are used to compute the weighted average (one weight per target class). I think the <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html" rel="nofollow noopener">docs</a> explain it clearly.</p>
What backend for linear algebra does the precompiled Pytorch use?

username: Anders_G
cooked: <p>When using the precompiled Pytorch (on windows) from the official homepage, what backend is then used for linear algebra on GPU? Is it Magma or something else? As I have understood/misunderstood it one can choose this when compiling.</p>
<p>Background: I want to do batched lu and cholesky factorization and it is a bit slow when applyed to many small matrices (200 x 200). For instance, the same operation is twice as fast in Tensorflow. And when looking at some research it seems like implementation can matter a lot. Up to a factor 10.</p>

username: albanD
cooked: <p>Hi,</p>
<p>IIRC the binaries are shipped with mkl for cpu and magma for GPU.</p>

username: Anders_G
cooked: <p>Thanks! I had actually hoped not. Magma is as far as I understand really good, and still in LU factorisation I only get about 30GFlop/s on a card that can do 10TFlop/s.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Note that many small matrices is usually the worst case scenario in terms of performance for a GPU. So I wouldn’t be very surprised not to be anywhere near to theoretical maximum card throughput.<br>
Is it better with larger matrices?</p>
Understanding Masks in Transformer

username: Sedherthe
cooked: <p>So, I have a time series data, where my input sequences are of different lengths. For example, suppose I have a batch of three sequences of sizes [400, 39], [500,49], [600,39]. I want to use a transformer.Encoder model for training. Thus, to feed into the model I will pad the sequences to size 600 and now my batch shape would be of shape: [600, 3, 39] (here 39 is like embedding size). Next, I want the model to train in such a way that ith position should be able to attend to both the before and the after time instants but not the ones which are padded. Now, I guess this is where the src_key_padding_mask comes into the picture. Also, I do not want the padded ones(the ones which I padded to make it to 600) to affect any of the values and also not calculate any attention as well. So, is the src_mask intended for this? So, do I need to use src_mask for this?<br>
I also wanted to know, whether I should initialize the padded values to 0 so that it works this way or any number is fine, that is for the first sequence which is of length 400 I would have to add 200 more of [1,39] vectors to make it to size 600. Here, do I have to add all zeros?</p>
Are image tensors automatically computed in cuda?

username: Valerio_Biscione
cooked: <p>As far as I understand, PyTorch is pretty conservative in terms of what goes executed on the GPU: you need to explicitly call cuda(), otherwise they will be executed on the CPU.</p>
<p>Is this true from the images loaded and transformed by the <code>torchvision.datasets</code> ? Let’s say that after having transformed the PIL image to tensor, I want to apply several transformation. Are these going to be executed automatically in cuda if available, or should we manually run a <code>cuda()</code> command  in the dataset? Would this work nicely with the dataloader?</p>
<p>Thanks for the clarification.</p>

username: albanD
cooked: <p>Hi,</p>
<p>They will run on the CPU, unless you send them on the GPU explicitly yes.</p>
<p>Note that in this case, you most likely don’t want to do that. Indeed the main purpose of the dataloader (with multiple workers) is to use many cpu cores to load and pre-process data on the CPU so that the GPU can be fully used to perform the forward/backward which is the non-parallelizeable part.</p>
Error With Sequential Sampler

username: Jaideep_Valani
cooked: <p>When dataloader iterates through the below loader ,i get an error saying<br>
<code>index 307 is out of bounds for axis 0 with size 307</code><br>
but with RandomSampler i dont get any error. size of valid set array is 307 .</p>
<pre><code class="lang-auto">for tr_idx, val_idx in tqdm(kf.split(z)):

    train_dataset = DatasetRetriever(
        train_arrays=z[tr_idx],#.index.values,
        targets= y[tr_idx]

    )

    valid_dataset = DatasetRetriever(
        train_arrays=z[val_idx],#.index.values,
        targets= y[val_idx]

    )

valid_loader = torch.utils.data.DataLoader(
        valid_dataset,
        batch_size=128,
        sampler=SequentialSampler(train_dataset),
        #sampler=RandomSampler(train_dataset),
        pin_memory=False,
        drop_last=False,
        num_workers=1
        #collate_fn=collate_fn,
    )
</code></pre>
<p>Below is result of print statement from Dataset Get</p>
<pre><code class="lang-auto">0 (307,10)
...
...
301 (307, 10)
302 (307, 10)
303 (307, 10)
304 (307, 10)
305 (307, 10)
306 (307, 10)
307 (307, 10)
384 (307, 10)
512 (307, 10)
</code></pre>
<p>i dont know how 512 ,384 indices are coming up here above.</p>

username: BaruchG
cooked: <aside class="quote no-group" data-username="Jaideep_Valani" data-post="1" data-topic="94088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaideep_valani/48/8482_2.png" class="avatar"> Jaideep_Valani:</div>
<blockquote>
<p><code>sampler=SequentialSampler(train_dataset),</code></p>
</blockquote>
</aside>
<p>I’m guessing that <code>train_dataset</code> should be <code>valid_dataset</code> since that is your sampler is for the validation set.  That index is likely one that is only applicable to the training dataset which is larger.</p>
Tying weights for between two Linear layers

username: John_Watkins
cooked: <p>I have quick question about weight sharing/tying. Suppose I have two Linear modules in an encoder-decoder framework:<br>
layer_e = torch.nn.Linear(20, 50)<br>
layer_d = torch.nn.Linear(50, 20)<br>
And I wish for the weights of the two modules to be tied. How would I go bout doing this? Specifically, the weight of layer_e and layer_d must be tied for both initialization and backpropagation. So after training the entire framework, the weights of layer_e and layer_d must still be the same.</p>
<p>Previous posts about potential solution to this problem seems to have some flaws. For example:</p>
<ol>
<li>
<p>layer_d.weights = layer_e.weights.T<br>
This does not work, as transpose of any variant (.T, .t(), .transpose(0, 1)) all changes the weights from Parameter class into a Tensor. This assignment raises an error.</p>
</li>
<li>
<p>layer_d.weights = torch.nn.parameter.Parameter(layer_e.weights.T)<br>
This method creates an entirely new set of parameters for layer_d. While the initial value is a copy of the layer_e.weights. It is not tied in backpropagation, so layer_d.weights and layer_e.weights will be different after training.</p>
</li>
<li>
<p>layer_d = torch.nn.functional.linear(input, layer_e.weights.T)<br>
This reassigns the entire layer_d. This may work if layer_e.weight.T is returning the original weights. However, this changes the layer_d from a Module to a function, which is really inconvenient when considered with respect to the existing codebase.</p>
</li>
</ol>
<p>Any help is appreciated.</p>

username: ayalaa2
cooked: <p>This works for me: <code>layer_e.weight[:] = layer_d.weight.T[:]</code>.</p>

username: John_Watkins
cooked: <p>Thank you for your response.</p>
<p>However, while the assignment works, when passing the parameters to initialize the optimizer, the following error is returned.</p>
<p>ValueError: can’t optimize a non-leaf Tensor</p>
<p>I am wondering what is the reason behind this, and how to solve the error.</p>

username: ayalaa2
cooked: <p>I would honestly just write your own module for this. You can initialize a tensor that is to be shared and inside of its forward method, use those weights in a transposed and non-transposed way. Perhaps have it as a flag that’s passed in through the forward method. I think you’ll get long-term weirdness when trying to strap two linear layers together.</p>
Transfer learning - Question about frozen layers

username: rubenIng93
cooked: <p>Hi everyone!<br>
I’m trying some experiments in transfer learning using the Alexnet, in particular I tried 3 different modalities for my problem, all of the following have of course the last fully connected layer changed with respect to the number of classes required:</p>
<ul>
<li>I trained the net with the weight from ImageNet without freezing any layer;</li>
<li>Same as before, but with the convolutional layers frozen, keeping only the classifier part with <code>requires_grad=True</code>;</li>
<li>Same as first row, but with the fully connected layers frozen, keeping only the feature part with <code>requires_grad=True</code>.</li>
</ul>
<p>Well, to the best of my knowledge, the first point can involve to a previous information loss and achieve a poor result; the second point should be the raccomended situation, in which exploiting the knowledge obtained with ImageNet for the feature extraction, my net learns only the classification weights. Instead, freezing the FCs I guess that again it can involve to a loss of knowledge and the classifier part won’t be well suitable for my task.</p>
<p>Now, the question is… Why do I get almost equal results for both accuracy and loss in the 3 different cases?</p>

username: ayalaa2
cooked: <p>It really depends on the task. Your model may just be at the point where it’s already able to do the task without adjusting much weights (hence the frozen components don’t matter). It can also be that the unfrozen components can each still adapt on their own and do just fine. It’s hard to say without knowing the transfer learning task and taking a deep look at how the model is training.</p>
<p>I would agree that your second option is the most traditional way of performing transfer learning. I don’t think the first option is all that bad, but if you have confidence that the features learned by the model is sufficient, then it makes sense to just freeze those layers (and probably gain a training speed-up).</p>

username: rubenIng93
cooked: <p>Many thanks for the reply!<br>
So, if I load a net with the ImageNet’s weights is not automatic that training again over a small dataset, will involve to a loss of the previous knowledge, is it right?</p>

username: ayalaa2
cooked: <p>To clarify, you’re asking that if we get a pretrained network (without freezing any layers) and train it on a transfer task, will the network get worse (or at least, is it possible)?</p>
<p>I would say not necessarily. I would suspect that early on in training, since the last layer was just freshly initialized, that the pretrained network may get “worst”. But I would imagine this would only be early in training. This is just coming from my personal experiences, so take it as you will.</p>

username: rubenIng93
cooked: <blockquote>
<p>To clarify, you’re asking that if we get a pretrained network (without freezing any layers) and train it on a transfer task, will the network get worse (or at least, is it possible)?</p>
</blockquote>
<p>Yes this is what I mean.</p>
<p>Ok, thanks again I understand.</p>
PyTorch core & tooling needs

username: maxiluk
cooked: <p>Hi, I am a product manager at Microsoft working on PyTorch. We are looking to expand our PyTorch contributions and would like to hear from community to help determine direction of our investments. I would like to have 30-45 min discussion with you to understand how you use PyTorch and what are its current painpoints and limitations.</p>
<p>If this is something that interests you please grab some time on my calendar: <a href="https://outlook.office365.com/owa/calendar/Microsoft1994249@microsoft.onmicrosoft.com/bookings/" rel="nofollow noopener">https://outlook.office365.com/owa/calendar/Microsoft1994249@microsoft.onmicrosoft.com/bookings/</a></p>
Weight update using gradient as a loss function

username: KAISER1997
cooked: <p>Assume an image classification task using a neural net where loss function is defined as L .Consider an image x ,while performing gradient descent on the loss, the gradient of L with respect to x is dL/dx . Now about my doubt. After this I want to calculate the gradient of norm of dL/dx wrt to weights of neural net. how do I do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can do the following:</p>
<pre><code class="lang-python">x = torch.rand()
output = net(x)
L = crit(output, label)

dLdx = autograd.grad(L, x, create_graph=True)[0]

dLdx.backward()
# Now you have the grads as usual in the .grad fields
</code></pre>

username: KAISER1997
cooked: <p>Thanks,this works perfectly</p>
How to generate nn.Transformer output using only src (no tgt) sequence?)

username: Past-Future-AI
cooked: <p>I have trained an nn.Transformer model using required src and tgt sequences</p>
<p>Training</p>
<pre><code class="lang-auto">out = transformer(src, tgt)
</code></pre>
<p>and now I want to use the model to generate sequences using a src sequence only but it’s not obvious how to do this.</p>
<p>Desired Eval:</p>
<pre><code class="lang-auto">out = transformer(src)
</code></pre>
<p>Any ideas?</p>
How to insure same initialization? is there a provision to declare seed value to assure reproducability?

username: Mayank_Jha
cooked: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/how-are-layer-weights-and-biases-initialized-by-default/13073">How are layer weights and biases initialized by default?</a>:</p>

username: RaLo4
cooked: <p>You might wanna check out <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">this section of the documentation</a></p>
Module prune has no effect on speed and memory consumption

username: Mohamed209
cooked: <p>I am developing gpu friendly app , to do so I aimed for pruning the model weights to reduce gpu memory usage and increase speed</p>
<p>pruning strategy</p>
<p><code>parameters_to_prune = ( (generator.dense_motion_network.hourglass.encoder.down_blocks[0].conv,'weight'), (generator.dense_motion_network.hourglass.encoder.down_blocks[1].conv, 'weight'), (generator.dense_motion_network.hourglass.encoder.down_blocks[2].conv, 'weight'), (generator.dense_motion_network.hourglass.encoder.down_blocks[3].conv, 'weight'), (generator.dense_motion_network.hourglass.encoder.down_blocks[4].conv, 'weight'), (generator.dense_motion_network.hourglass.decoder.up_blocks[0].conv,'weight'), (generator.dense_motion_network.hourglass.decoder.up_blocks[1].conv,'weight'), (generator.dense_motion_network.hourglass.decoder.up_blocks[2].conv,'weight'), (generator.dense_motion_network.hourglass.decoder.up_blocks[3].conv,'weight'), (generator.dense_motion_network.hourglass.decoder.up_blocks[4].conv,'weight'), (generator.first.conv,'weight'), (generator.down_blocks[0].conv,'weight'), (generator.down_blocks[1].conv,'weight'), (generator.up_blocks[0].conv,'weight'), (generator.up_blocks[1].conv,'weight') )</code></p>
<p><code>prune.global_unstructured( parameters_to_prune, pruning_method=prune.L1Unstructured, amount=0.2, )</code></p>
<p>by varying amount i got different final results , that means that pruning is done correctly , but gpu usage and inference time are the same when amount for example is 0.1 and 0.9</p>
Pytorch's Tutorial on Neural Style Transfer has drastically different results than the original paper

username: Nick-Morgan
cooked: <p>I also posted this on <a href="https://github.com/pytorch/pytorch/issues/43457" rel="nofollow noopener">github</a>.</p>
<p>Following the <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" rel="nofollow noopener">official tutorial</a> on van Gogh’s Starry Night produces the results below. I’ve included the results from the <a href="https://arxiv.org/abs/1508.06576" rel="nofollow noopener">original paper</a> for comparison. The notebook that generated these plots can be viewed <a href="https://colab.research.google.com/drive/1W-oboiUmziK7N6CBKp1y6uQISgukfiZP?usp=sharing" rel="nofollow noopener">here</a>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/23083692/90962458-c7a4e300-e475-11ea-9fc0-4b590a50b8ae.png" title="image" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/23083692/90962458-c7a4e300-e475-11ea-9fc0-4b590a50b8ae.png" alt="image" width="222" height="500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">image</span><span class="informations">780×1753</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>
<p>The original paper used Caffe’s pre-trained VGG19 network, which has a few differences:</p>
<ol>
<li>The pixels are in the range of <code>[0, 255]</code> (Pytorch’s was trained in the range of <code>[0, 1]</code>)</li>
<li>The color channels are in the order <code>BGR</code> (Pytorch’s is in the order of <code>RGB</code>)</li>
<li>The method of normalization is different.</li>
</ol>
<p>Despite these differences, I would expect to see results that are closer to the original paper. What else could be causing such a large difference in results?</p>

username: AladdinPerzon
cooked: <p>I would assume it has to do with hyperparameters and most importantly the learning rate and following the notation from the original paper the alpha and beta parameter to emphasize how much of the total loss comes from the style loss vs the content loss. I also think you could perhaps try out initializing with the content image (instead of a random image) and see if that helps. Let us know how it goes</p>

username: Nick-Morgan
cooked: <p>The images above come from Figure 1 of the original paper, which applied back-propagation to a noisy image to capture the style alone (the content loss weight being effectively 0). I have also tried learning rates in the range [1e-05, 1e03] and gotten similar results.</p>
<p>If I load the caffe model, switch the method of normalization, and follow this same process, I get results that are comparable to the original paper.</p>
PyTorch standard Build Configuration (CI version)

username: Philipp_Fischer
cooked: <p>I am trying to build pytorch from source exactly as is done for the official releases.<br>
I found this repo: <a href="https://github.com/pytorch/builder" rel="nofollow noopener">https://github.com/pytorch/builder</a><br>
Apparently, it contains all the scripts needed for CI, building and testing, however it’s missing an entry point for newbies like me.</p>
<p>Which script is the right one to run? Which arguments? Do I need to set up one of the docker containers first?<br>
Please help with this.</p>
<p><strong>EDIT</strong>: When trying to follow the instructions found here (<a href="https://github.com/pytorch/builder/blob/master/manywheel/README.md" rel="nofollow noopener">https://github.com/pytorch/builder/blob/master/manywheel/README.md</a>), I realized that the prebuilt containers by soumith do not have a version for cuda 10.2 and cuda 11.0.<br>
Which Dockerfile was used to create those, so that I can build an updated version?</p>
<p><strong>EDIT 2</strong>: I haven’t verified, but now my guess is that the prebuilt docker containers from soumith are built from the Dockerfile at <a href="https://github.com/pytorch/builder/blob/master/manywheel/Dockerfile" rel="nofollow noopener">https://github.com/pytorch/builder/blob/master/manywheel/Dockerfile</a><br>
It remains unclear to me though, why the README tells to run them with “bash” once, because the seem to be pulled by the build_docker.sh script anyway</p>
Speeding up the trainning - RNN with LSTM

username: learningpytorch
cooked: <p>Dear all, I am trying to train a LSTM for energy demand forecast but it takes too long. I do not understand why because the model looks “simple” and there is no much data. Might it be because I am not using the DataLoader? How could I use it with RNN since I have a sequence?</p>
<p>Complete code is in Colab: <a href="https://colab.research.google.com/drive/130rG8_j1Lf8RQoVRrfXCeo5h_CcC5NU6?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/130rG8_j1Lf8RQoVRrfXCeo5h_CcC5NU6?usp=sharing</a></p>
<p>The interesting part to be improved may be this:</p>
<pre><code class="lang-auto">for seq, y_train in train_data:

         optimizer.zero_grad()

         model.hidden = (torch.zeros(1,1,model.hidden_size), 

                         torch.zeros(1,1,model.hidden_size)) 
      
         y_pred = model(seq) 

         loss = criterion(y_pred, y_train) 

         loss.backward() 

         optimizer.step() 

Thanks in advance to anyone helping me.</code></pre>
Torch.nn.utils.prune errors

username: serereuk
cooked: <p>Hi, I’m working with your tutorial for Pruning custom methods  that use the abstract class <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="nofollow noopener">link</a></p>
<p>However, when I use torch version == <strong>1.6.0</strong>, there are some problems with module._parameters.</p>
<p>As I understand from the source codes from <code>BasePruningMethod</code>, it uses apply function to use compute_mask function.</p>
<p>However, when I call like  <code>Prunemethod.apply(module, name)</code>, then there is an error</p>
<p><code>cannot assign 'torch.floattensor' object to parameter 'weight_orig' (torch.nn.parameter or none required)</code></p>
<p>so I tried to inherit the apply function and change it into like this</p>
<pre><code class="lang-auto"># original line was at 164 in the prune.py github
# module.register_parameter(name + "_orig", orig)
module.register_parameter(name + "_orig", torch.nn.Parameter(orig))
</code></pre>
<p>But, after I changed it, there is another error with <code>del module._parameters[name]</code>, there is no ‘weight’ attribute.</p>
<p>Is there any better idea than downgrading the torch vesion?</p>
<p>When I downgrade torch to 1.4.0, there is a problem with compute_mask paramter which doesn’t have ‘dim’ arguments.</p>
<p>Thanks in advance</p>
How to reshape tensors for LSTM input

username: ygl
cooked: <p>Hi,  I have the data like below,<br>
txt_features = np.array([[1,2,3], [4,5,6]])<br>
num_features = np.array([[1,2,4], [4,5,7]])<br>
I want to pass it into a lstm. So how do I reshape the data to c, which is shown below, so that I can feed it into a lstm<br>
c = [<br>
[[1,2,3], [1,2,4]],<br>
[[4,5,6], [4,5,7]]<br>
]</p>

username: ptrblck
cooked: <p>You could use <code>torch.stack</code> as seen here:</p>
<pre><code class="lang-python">txt_features = torch.tensor([[1,2,3], [4,5,6]])
num_features = torch.tensor([[1,2,4], [4,5,7]])
c = torch.stack((txt_features, num_features), dim=1)
</code></pre>
Issue of over fitting with Wide ResNet

username: Prasann
cooked: <p>I’m trying to build a Wide ResNet (WRN - 16 - 4) on CIFAR10 from scratch. I encounter over fitting problem with training accuracy at 91% and validation accuracy at 25%. The following are measures I tried but couldn’t get a solution,</p>
<ol>
<li>Drop out - 40%</li>
<li>Data augmentation and normalization</li>
<li>Xavier Uniform weight initialization</li>
<li>Batch size</li>
</ol>
<p>Kindly help me with ideas I can try to solve this issue.</p>
Open source project to easily manage datasets, models and optimizers in pytorch

username: graykode
cooked: <p>Hello, pytorch forum! How are you managing your dataset or models? Introducing a project called matorage that can easily solve these problems.</p>
<ul>
<li>
<p>Github : <a href="https://github.com/graykode/matorage" rel="nofollow noopener">https://github.com/graykode/matorage</a></p>
</li>
<li>
<p>Document : <a href="https://matorage.readthedocs.io" rel="nofollow noopener">https://matorage.readthedocs.io</a></p>
</li>
<li>
<p>Tutorial: <a href="https://github.com/graykode/matorage#quick-start-with-pytorch-example" rel="nofollow noopener">https://github.com/graykode/matorage#quick-start-with-pytorch-example</a></p>
</li>
</ul>