akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Convert lstp neuro from pythorch to libtorch

puxoper: <p>Hi, i am to try studying the libtorch and want to write the programm which will make predict the value of time sequence with using lstp neuro net. I found the example of lstp rnn on python <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html:" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html:</a></p>
<pre><code class="lang-auto">lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3
inputs = [torch.randn(1, 3) for _ in range(5)]  # make a sequence of length 5

# initialize the hidden state.
hidden = (torch.randn(1, 1, 3),
          torch.randn(1, 1, 3))
for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)

# alternatively, we can do the entire sequence all at once.
# the first value returned by LSTM is all of the hidden states throughout
# the sequence. the second is just the most recent hidden state
# (compare the last slice of "out" with "hidden" below, they are the same)
# The reason for this is that:
# "out" will give you access to all hidden states in the sequence
# "hidden" will allow you to continue the sequence and backpropagate,
# by passing it as an argument  to the lstm at a later time
# Add the extra 2nd dimension
inputs = torch.cat(inputs).view(len(inputs), 1, -1)
hidden = (torch.randn(1, 1, 3), torch.randn(1, 1, 3))  # clean out hidden state
out, hidden = lstm(inputs, hidden)
print(out)
print(hidden)
</code></pre>
<p>and trying to rewrite this on c++, but i could not translate the line of code:<br>
<strong>out, hidden = lstm(i.view(1, 1, -1), hidden)</strong><br>
Can anybody prompt to me how can i do it?<br>
My c++ code look that:</p>
<pre><code class="lang-auto">#include &lt;vector&gt;
#include &lt;iostream&gt;

int main(int argc, char *argv[])
{
    // lstm - long short-time memory
    torch::nn::LSTM lstm = torch::nn::LSTM(3,3); //dimension of in 3, dimension of out 3
    std::cout &lt;&lt; lstm &lt;&lt; '\n';

    // initialize a source data for training of lstm nn
    std::vector&lt;torch::Tensor&gt; inputs(5);

    for (torch::Tensor &amp;tr : inputs) {
        tr = torch::randn({1,3});
    }

    // matrix of hidden layer
    std::vector&lt;torch::Tensor&gt; hidden{torch::randn({1,1,3}),torch::randn({1,1,3})};

    // output of nn
    torch::Tensor out;
    for (torch::Tensor &amp;tr : inputs) {
        out = hidden = lstm(tr.view({1,1,-1}),hidden); // there is error, i dont know how do it on c++
    }

    return 0;
}
</code></pre>
<p>Thank you all for your attention!</p>
Any Idea why is it returning zero

dcl: <pre><code class="lang-auto">import torch

from torch import nn

from torch import functional as F

class Model(nn.Module):

    def __init__(self):

        super(Model, self).__init__()

        self.conv = [nn.Conv2d(1, 20, 5)]

        self.conv.append(nn.Conv2d(20, 20, 5))

    def forward(self, x):

        x = F.relu(self.conv[0])

        return F.relu(self.conv[1])

p = Model()

len([i for i in p.parameters()])  #====&gt; This is turning out to be zero
</code></pre>
<p>I have a problem where I am not sure how many layers I end up with before hand to name them individually, So I am maintaining a list of conv layers then iterating over them in the forward pass to use them. But</p>
<pre><code class="lang-auto">model = Model() 

model.parameters()
</code></pre>
<p>is turning out to be <em><strong>0</strong></em>. How to get rid of this problem. Can anyone please explain.</p>

ptrblck: <p>You would have to use <code>nn.ModuleList</code> instead of a Python <code>list</code> to make sure the submodules are properly registered inside the <code>Model</code>.</p>
Related to pertrained models

sripathisony_sony: <p>Can we train one dataset for some time and then freeze the layers and then use another dataset and freeze other layers in xlnet pertained model?</p>
<p>Is it possible to freeze layers for pertained models?</p>

ptrblck: <p>Yes, you can freeze the parameters of a specific layer by setting their <code>.requires_grad</code> attribute to <code>False</code>:</p>
<pre><code class="lang-python">model = ... # model initialization
# freeze parameters of my_layer
for param in model.my_layer.parameters():
    param.requires_grad_(False)
</code></pre>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you, I will try</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I want to load dataset 1 with freeze 6-11 layers and load other dataset2 with freeze 0-5 layers my code for one dataset1:<br>
for epoch in range(EPOCHS):</p>
<pre><code>       print(f'Epoch {epoch + 1}/{EPOCHS}')
       print('-' * 10)
       if epoch==0:
          for name, param in list(xlnet_model.named_parameters())[104:206]: 
                    print('I will be frozen: {}'.format(name))
                    #print(param)
                    param.requires_grad = False
       

       train_acc, train_loss = training(xlnet_model,train_data_loader,optimizer, device, scheduler, len(train_data))

       print(f'Train loss {train_loss} Train accuracy {train_acc}')
</code></pre>
<p>will this statement  "for name, param in list(xlnet_model.named_parameters())[104:206]: " freeze only 6-11 layer and othe weight will update?( i am using pertrained model)</p>

ptrblck: <p>I don’t know, if these indices would correspond to layers 6 to 11 and you would have to verify it.<br>
Maybe a safer approach would be to create a list with the layer names and to compare the submodule and/or the parameter names with this list. If it’s matching a name in this list, you could freeze it.</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , Thank you, its working</p>
To conv2d or to conv3d, that is the question?

chaslie: <p>I was thinking, if i have an input of 128x128x128, this can be represented as 128 128x128 2d arrays or a single 128x128x128 3D array, so which is most effiecent in terms of memory usage and speed on a single GPU?</p>

ptrblck: <p>You could profile both workloads using e.g. <code>torch.utils.benchmark</code> in the nightly release or profile it manually. I guess the 2D layer would be faster, but ymmv depending on the used hardware and system setup.</p>

chaslie: <p>Hi Ptrblck</p>
<p>I assumed that a 2d would be quicker, but then i got thinking about the impact of the number of channels. I you use 3D, then you would start with 1 and increase to ‘x’, however if you use 2D, then you would start with 128 and have to increase to a suitable number, thus making the number of channels become very large, very quickly, unless you adopted the inception net approach…</p>
<p>This then made me wonder, what is going to affect the efficency of the network more, the extra dimension or the number of channels.</p>
<p>I’ll have a look at running some benchmark tests and get back to you…</p>
<p>Cheers</p>
<p>Chaslie</p>
Build level optimization and selective compilation

hlahkar: <p>I was exploring the code for the mobile optimizer. The <a href="https://pytorch.org/mobile/home/" class="inline-onebox" rel="noopener nofollow ugc">Home | PyTorch</a> ; mentions that PyTorch mobile optimizer supports build level optimization and selective compilation depending on the operators needed for user applications.</p>
<p>On exploring the code I could find the code for Operator Fusion and Folding of prepacked ops; however, I could not understand how the build level optimization and selective compilation is happening. It would be great if someone can help me understand this part.</p>
<p>Thanking in Advance<br>
Himangshu</p>
nn.DataParallel degrades the prediction performance

soroosh_99: <p>nn.DataParallel degrades the prediction performance when trying to localize text in images.<br>
The below image is when I comment nn.DataParallel and it gives me the text files just fine.<br>
<a href="https://ibb.co/dm8B994" rel="noopener nofollow ugc">Bad Detection</a></p>
<p>However when I parallelize the model the performance degrades and only predicts tiny boxes below the image.<br>
<a href="https://ibb.co/fpgq6fD" rel="noopener nofollow ugc">Good Detection</a></p>
<p>Commenting the 3rd line does this effects.</p>
<pre><code class="lang-python">       self.model = BasicModel(args)
        # for loading models
        self.model = parallelize(self.model, distributed, local_rank)
        self.criterion = SegDetectorLossBuilder(
            args['loss_class'], *args.get('loss_args', []), **args.get('loss_kwargs', {})).build()
        self.criterion = parallelize(self.criterion, distributed, local_rank)
        self.device = device
        self.to(self.device)
</code></pre>
<p>parallelize function looks like the following:</p>
<pre><code class="lang-auto">def parallelize(model, distributed, local_rank):
    if distributed:
        return nn.parallel.DistributedDataParallel(
            model,
            device_ids=[local_rank],
            output_device=[local_rank],
            find_unused_parameters=True)
    else:
        return nn.DataParallel(model)
</code></pre>
<p>Something similar also happens when I train my model on GPU and try to use it on CPU but I am not sure why.</p>

ptrblck: <p>Are you using <code>model.eval()</code> before executing the prediction step?<br>
If not, then note that the prediction results could depend on the batch size, which might be different on the single-GPU vs. data parallel setup, e.g. due to batchnorm layers.</p>

soroosh_99: <p>Yes, I use model.eval().</p>
<p>I solved the issue. The problem was that the model was trained using DataParallel and when I was trying to load the state dictionary there was a mismatch between the dictionary keys of the model and the network even though I set the strict=False.</p>
<p>Once I remove the .module from the state_dict keys it solved the issue.</p>

ptrblck: <p>Good to hear, you’ve figured it out!</p>
<aside class="quote no-group" data-username="soroosh_99" data-post="3" data-topic="111387">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soroosh_99/48/34474_2.png" class="avatar"> soroosh_99:</div>
<blockquote>
<p>there was a mismatch between the dictionary keys of the model and the network even though I set the strict=False.</p>
</blockquote>
</aside>
<p>I would not recommend to use <code>strict=False</code> in case you are running into unexpected mismatches while loading the <code>state_dict</code>, as this will ignore the keys.</p>
Swap array positions / values

cndcnd: <p>Is there a way to perform this operation more efficiently?</p>
<pre><code class="lang-auto">   sort1 = torch.sort(abs(u.grad))[1]
   srtcoe= u.grad.clone()
   for li in range(z.lrsize[lr]):
      for pi in range(z.lrsize[lr-1]):
         srtcoe[li][sort1[li][pi]] = pi /z.lrsize[pr]
</code></pre>
About elementwise_kernel

mahmoodn: <p>Hi<br>
In the output of the profiler, I see that <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/CUDALoops.cuh#L84" rel="noopener nofollow ugc">unrolled_elementwise_kernel</a> takes some GPU time (second kernel with highest GPU time). However, when I looked at the source code, this kernel simply calls another kernel</p>
<pre><code class="lang-auto">__global__ void unrolled_elementwise_kernel(int N, func_t f, array_t data,
                                            inp_calc_t ic, out_calc_t oc, loader_t l, storer_t s)
{
  int remaining = N - block_work_size * blockIdx.x;
  auto policy = memory::policies::unroll&lt;array_t, inp_calc_t, out_calc_t, loader_t, storer_t&gt;(data, remaining, ic, oc, l, s);
  elementwise_kernel_helper(f, policy);
}
</code></pre>
<p>So, I wonder why that is shown in the output of profiler? I mean what can be understand from that?</p>

ptrblck: <p>Could you post the profiling output you were creating as well as the current setup (CUDA, PyTorch version, used GPU etc.)?</p>
<aside class="quote no-group" data-username="mahmoodn" data-post="1" data-topic="111333">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahmoodn/48/13302_2.png" class="avatar"> mahmoodn:</div>
<blockquote>
<p>So, I wonder why that is shown in the output of profiler? I mean what can be understand from that?</p>
</blockquote>
</aside>
<p>The profiler shows which kernel spends what amount of time and could thus point towards the most expensive calls and bottlenecks in your code.</p>
How to limit the weight and bias of nn.Conv3d as non-negative?

amyxwgu: <p>Hi all,<br>
I am new to Pytorch and I am not familiar with it yet. I run the experiment of my paper on Pytorch using Conv3d (the 3D convolution module) as one of the major layers. According to the source codes, the self.weight and self.bias parameters of Conv3d are initialized uniformly within a range of (-bound, bound). I would like to limit the two parameters to non-negative values. How could I make it? Thanks in advance.</p>
<p>Regards,<br>
Amy</p>

ptrblck: <p>You could use a method from <a href="https://pytorch.org/docs/stable/nn.init.html"><code>torch.nn.init</code></a> and apply it on the parameters as e.g. seen <a href="https://github.com/pytorch/examples/blob/36441a83b6595524a538e342594ee6482754f374/dcgan/main.py#L113-L120">here</a> and used <a href="https://github.com/pytorch/examples/blob/36441a83b6595524a538e342594ee6482754f374/dcgan/main.py#L159">here</a>.<br>
Alternatively, you could directly copy the desired initial values into the parameter by using:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layer.weight.copy_(your_values)
</code></pre>
Customizing the batch with specific elements

dato: <p>Hey, I am a fresh starter with pytorch. Strangley I cannot find anything related to this, although it seems rather simple.</p>
<p>I want to structure my batch  with specific examples, like all examples per batch having the same label or just fill the batch with examples of just 2 classes.</p>
<p>How would I do that? For me it seems the right place within the dataloader and not in the dataset? As the dataloader is responsible for the batches and not the dataset?</p>
<p>Is there simple minimal example?</p>

ptrblck: <p>You could write a custom <code>sampler</code> and could use the current <a href="https://github.com/pytorch/pytorch/blob/215d9daceb5482d2d5cceaaf44cb3f2aa38faaaf/torch/utils/data/sampler.py">implementations</a> as the base class.<br>
The <code>sampler</code> is responsible to create the indices, which are then passed to the <code>Dataset,__getitem__</code>.<br>
You could thus use the <code>target</code> tensor and create batches of indices using your custom sample logic.<br>
In case you would like to use weighted sampling, you could use <code>WeightedRandomSampler</code> instead.<br>
Note however, that this sampler will not guarantee to sample a specific number of classes in each batch.</p>

dato: <p>What you mean by target tensor?</p>

ptrblck: <p>The tensor returned by the <code>Dataset</code> containing the target values.<br>
E.g. for a multi-class classification it would contain the class indices, which would be passed to <code>nn.CrossEntropyLoss</code>.</p>

dato: <p>I created a sampler and use it in the dataloader as a batch_sampler argument. No I need to have a <strong>len</strong> method. What is it? The length of all datapoints or the number of batches the iterator returns?</p>
<p>That is not clear and confusing and in the link mentioned above not well documented…</p>

ptrblck: <p>The <code>Dataset.__len__</code> method returns the number of samples, which should be drawn from the <code>Dataset</code>.</p>

dato: <p>Sorry, but we are talking about sampler, and in the code aboce in the link you see that Samplers have a len emthod. I get an error because I need len(dataloader)…</p>
<p>Why are you talking about the Dataset now? pytorch is really confusing…</p>
Resize 96*96*3 image in STL10 dataset into 48*48*3 image

Hong_Cheng: <p>I use the following codes, but it doesn’t work.</p>
<pre><code class="lang-auto">dataset0 = torchvision.datasets.STL10('./Drive/training/stl10', split='train+unlabeled', download=True,
                                     transform=torchvision.transforms.Compose([
                                         torchvision.transforms.ToTensor(),
                                            torchvision.transforms.Resize(64),
                                     ]))
</code></pre>
<p>also tried<br>
<code> torchvision.transforms.Resize(int(64)),</code></p>
<p><strong>I find that there is another parameter</strong>,<br>
<strong>Is there anyone can give me some suggestions?</strong></p>
<p><code>torchvision.transforms.</code> <code>Resize</code> (<em>size</em>, <em>interpolation=2</em> )</p>
<p><strong>interpolation</strong> (<a href="https://docs.python.org/3/library/functions.html#int" rel="noopener nofollow ugc"><em>int</em></a><em>,</em> <em>optional</em>) – Desired interpolation enum defined by <a href="https://pillow.readthedocs.io/en/latest/handbook/concepts.html#filters" rel="noopener nofollow ugc">filters</a>. Default is <code>PIL.Image.BILINEAR</code>. If input is Tensor, only <code>PIL.Image.NEAREST</code>, <code>PIL.Image.BILINEAR</code> and <code>PIL.Image.BICUBIC</code> are supported.</p>

Dwight_Foster: <p>You need to put the resize before the ToTensor() transform.</p>

Hong_Cheng: <p>Thank you for your declearation</p>
Exporting LSTM to ONNX model at Operator level

jwlee: <p>Dear all,</p>
<p>I have a question regarding exporting the PyTorch model to ONNX at the operator level.</p>
<p>Regrading LSTM Cell, LSTM is layer level and matmul, add inside LSTM is operator level.</p>
<p>Currently, nn.LSTM is exported at the layer level, but is it possible to export operator level?</p>
<p>Any comments will be helpful.</p>
<p>Thanks,<br>
Jaewook</p>
Port is still listening after rpc shutdown

Frank_Leeeee: <p>Hi,</p>
<p>I created a rpc network on localhost:29500.</p>
<p>When I execute rpc.shutdown(), I can see that port 29500 is still listening. Is there a way to shut down the port completely?</p>
<p>Thank you.</p>

osalpekar: <p>Are you using the TensorPipe RPC Backend? From what I understand calling shutdown should mean that process stops listening on the port - is this correct <a class="mention" href="/u/lcw">@lcw</a> <a class="mention" href="/u/mrshenli">@mrshenli</a> ?</p>

lcw: <p>I believe that port is the one used by the TCP store, for rendezvous. It’s not the first time I hear this, but I never understood if the store outlives the RPC agent on purpose or if it’s due to some sort of leak…</p>

Frank_Leeeee: <p>Hi, I am using the ProcessGroup backend.</p>
Identity element for stack operator `torch.stack([emtpty,x]) = x` (empty tensor exists?)

Brando_Miranda: <p>I want to stack tensors as I collect them in a loop and start with an empty tensor. Usually, when collecting elements in python we start with an empty list and then append. But when I try to stack with an “empty tensor” (with no dimensions) I get an error. Check the error and code:</p>
<pre><code class="lang-auto">import torch

empty = torch.tensor([])
x = torch.randn(3, 5, 7)
print(torch.stack([empty, x], dim=0).size())
</code></pre>
<p>error</p>
<pre><code class="lang-auto">import torch
empty = torch.tensor([])
x = torch.randn(3, 5, 7)
print(torch.stack([empty, x], dim=0).size())
Traceback (most recent call last):
  File "/Users/brando/anaconda3/envs/automl-meta-learning/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-9-59f445556977&gt;", line 5, in &lt;module&gt;
    print(torch.stack([empty, x], dim=0).size())
RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [3, 5, 7] at entry 1
</code></pre>
<p>how do we do torch.stack such that: <code>torch.stack([emtpty,x]) = x</code>? where x is a torch tensor.</p>
<hr>
<h1>Update</h1>
<p>hmmm perhaps what I need is torch.cat actually</p>
<pre><code class="lang-auto">import torch

empty = torch.tensor([])
x = torch.randn(3, 5, 7)

print(torch.cat([empty, x], dim=0).size())
</code></pre>
<p>since this adds new values at the front of the “list” but I guess I don’t really need to distinguish each append in my special case…probably still useful to have an identity for stack though.</p>
<pre><code class="lang-auto"># https://discuss.pytorch.org/t/identity-element-for-stack-operator-torch-stack-emtpty-x-x-empty-tensor-exists/111459
import torch
empty = torch.tensor([])
x = torch.randn(3, 5, 7)
print(torch.cat([empty, x], dim=0).size())
# print(torch.stack([empty, x], dim=0).size())
torch.Size([3, 5, 7])
</code></pre>
How to show image from tensor?

Samuel_Bachorik: <p>Hello ! I have tensor with shape of torch.Size([1, 2, 1080, 1920]) and i want to see this as image.<br>
How can i do it please ?</p>

Dwight_Foster: <p>You can use torchvision.utils.save_image from <a href="https://pytorch.org/vision/stable/utils.html" rel="noopener nofollow ugc">here</a></p>

Samuel_Bachorik: <p>I cant realise what put into fp parameter. Can you give me example ?</p>
<p>torchvision.utils.save_image(y_pred, example) ?</p>

Dwight_Foster: <p>Sure</p>
<pre><code class="lang-auto">torchvision.utils.save_image(IMAGE_TENSOR, fp=FILEPATH)
</code></pre>
<p>fp just is the filepath that you want to put the image. So you could do something like output.png</p>
Not able to detach and clone the model

MrPositron: <p>I want to clone the model. However, PyTorch fails to detach and clone the model.</p>
<pre><code class="lang-auto">temp_enc = model.encoder_net.detach().clone()
</code></pre>
<p>Error I get</p>
<pre><code class="lang-auto">torch.nn.modules.module.ModuleAttributeError: 'Encoder' object has no attribute 'detach'
</code></pre>
<p>When I clone without detaching I receive similar error:</p>
<pre><code class="lang-auto">torch.nn.modules.module.ModuleAttributeError: 'Encoder' object has no attribute 'clone'
</code></pre>

albanD: <p>Hi,</p>
<p>Both clone and detach are constructs on Tensors, not nn.Modules.<br>
If you want to make an independent copy of a model, you can use the regular python deepcopy.</p>

MrPositron: <p>Thanks! Stupid mistake. My bad.</p>
<p>Unfortunately, deepcopy throws the following error:</p>
<pre><code class="lang-auto">“Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment”
</code></pre>
<p>UPD: Actually <code>deepcopy</code> on <code>model.state_dict()</code> works fine! Thanks anyway.</p>
Problems using Dataloader for dask/xarray/netCDF data

fermat97: <p>Using multiple worker s(<code>num_workers &gt; 0</code>) for loading the data with <code>DataLoader</code>, results in a strange behavior. Here is a sample code snippet:</p>
<pre><code class="lang-auto">import xarray as xr
class mydataset(torch.utils.data.Dataset):
    def __init__(self,datapath):
        self.ds = xr.open_dataset(datapath)
        self.labels = # read the lables from some dimensions of self.ds
    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        return #choose some value from the dimensions of self.ds


</code></pre>
<p>Using this method for loading the data I face <code>NaN</code>s or very extreme values which are not expected at all. There is another <a href="https://discuss.pytorch.org/t/deadlock-with-dataloader-and-xarray-dask/9387">issue</a> which might be also related.</p>

ptrblck: <p>This behavior might point towards missing synchronizations.<br>
I’m not familiar with the implementation and usage of <code>xarray</code>, so could you post an example using random data, which shows this behavior?</p>

ghiggi: <p>I experienced similar problems.<br>
Are you reading the dataset from netCDF or GRIB files right? I guess is something related to the fact that netCDF I/O is not thread-safe and there is some locking going on.<br>
If you chunk your dataset with<code> ds=ds.chunk(&lt;your chunk options&gt;)</code> , you save the dataset to disk in zarr format (<code>ds.to_zarr()</code> ) and reopen the dataset with <code>xr.open_zarr()</code> then everything should work fine.<br>
I additionally suggest to set <code>dask.config.set(scheduler='synchronous')</code> to speed up the data loading if <code>num_workers &gt; 0</code> in your DataLoader.</p>
Deadlock with DataLoader and xarray/dask

cbarrick: <p><a href="http://xarray.pydata.org/" rel="nofollow noopener">xarray</a> is a library working with high-dimensional datasets. It uses <a>dask</a> to stream data from disk when it doesn’t fit in main memory. When I try to wrap an xarray <code>DataArray</code> in a torch <code>DataLoader</code>, I the program stalls. It appears to be a deadlock.</p>
<p>I’ve tried setting <code>lock=False</code> in <code>xr.open_mfdataset</code> which is passed to <code>dask.array.from_array</code>, but it’s not helping.</p>
<p>Does anyone here have experience combining xarray or dask with PyTorch?</p>
<p>Here’s a minimal example of what I’m talking about:</p>
<pre><code class="lang-python">from torch.utils.data import Dataset, DataLoader
import xarray as xr

class XarrayDataset(Dataset):
    '''A simple torch Dataset wrapping xr.DataArray'''
    def __init__(self, ar, batch_dim):
        self.ar = ar
        self.batch_dim = batch_dim

    def __len__(self):
        return len(self.ar[self.batch_dim])

    def __getitem__(self, idx):
        return self.ar[{self.batch_dim: idx}].values

class XarrayDataLoader(DataLoader):
    '''A simple torch DataLoader wrapping xr.DataArray'''
    def __init__(self, ar, batch_dim, **kwargs):
        ar = XarrayDataset(ar, batch_dim)
        super().__init__(ar, **kwargs)

# Open the dataset with xarray and select the feature to work with.
ds = xr.open_mfdataset(['NAM-NMM/nam.20170115/nam.t00z.awphys.tm00.nc', 'NAM-NMM/nam.20170115/nam.t12z.awphys.tm00.nc'])
ar = ds['TMP_SFC']

# If I load the data into main memory, the deadlock does not happen
#ar.load()

# Trying to load the data with num_workers &gt; 0 results in deadlock.
dl = XarrayDataLoader(ar, batch_dim='reftime', num_workers=1)
i = iter(dl)
next(i)
</code></pre>

ghiggi: <p>I experienced similar problems.<br>
Is something related to the fact that netCDF I/O is not thread-safe and there is some locking going on.<br>
If you chunk your dataset with<code> ds=ds.chunk(&lt;your  chunk options&gt;)</code>, you save the dataset to disk in zarr format (<code>ds.to_zarr()</code>) and reopen the dataset with <code>xr.open_zarr()</code> then everything works fine <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
I additionally suggest to set <code>dask.config.set(scheduler='synchronous')</code> to speed up the data loading if <code>num_workers &gt; 0</code> in your DataLoader.</p>
Multiple losses, ignored when multiplied by scalar

TeaWaterSleep: <p>Hi,<br>
I try to implement a stabilized attack method, which consist of a classical iterative fast gradient sign method (FGSM) plus a stabilization term.</p>
<p>Here is the definition of the loss stabilized (W is gaussian kernel, eta the sign grad through iteration) :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f283df17d7bcfc7557521a114ff5dcac87a1ace.png" alt="smia_attack" data-base62-sha1="i8SR7ojE2gT744yFrxvGYxwzLhI" width="518" height="92"></p>
<p>Below my implementation</p>
<pre><code class="lang-auto">def smia_attack(image, epsilon, alpha, model, target, n_step):

    attack_image = image.detach().requires_grad_()

    L_dev = F.nll_loss(model(attack_image), target)
    model.zero_grad()
    L_dev.backward()
    grad_image = attack_image.grad.data
    sign_grad_image = grad_image.sign()
    
    eta = epsilon*sign_grad_image
    W = gaussianBlur(10, 3, 1).cuda()
    for i in range(n_step):
        with torch.no_grad():
            attack_image.add_(eta)
            attack_image = attack_image.clamp(0, 1).detach().requires_grad_()
        model.zero_grad()
        L_smia = F.nll_loss(model(attack_image), target) - alpha * F.nll_loss(model(attack_image), model(attack_image + W(eta) - eta).argmax().unsqueeze(0))
        L_smia.backward()
        eta = epsilon * attack_image.grad.data.sign()
    return attack_image + eta
</code></pre>
<p>When I generate perturbed images, Pytorch seems to ignore the  right parrt of L_smia, namely : -alpha *F.nll_loss (…)  because if I remove it, nothing changes, and I have the exact same result without, ie, it’s doing vanilla FGSM.<br>
However, if I remove the alpha term, then the result is pretty different, Pytorch seems to make us of it.</p>
<p>Any idea why ?</p>

albanD: <p>Hi,</p>
<p>Note that it would be better to replace:</p>
<pre><code class="lang-python">        L_smia.backward()
        eta = epsilon * attack_image.grad.data.sign()
</code></pre>
<p>by</p>
<pre><code class="lang-python">        grad, = autograd.grad(L_smia, attack_image)
        eta = epsilon * grad
</code></pre>
<p>For your issue, what is the value of alpha?<br>
Also you can check the gradients flowing back by using hooks: on any Tensor, you can set <code>t.register_hook(your_fn)</code>. So that your_fn will be called for the gradient for that Tensor. That way you can check what are the gradient values flowing on the right part of your model.</p>
How to efficiently resize tensors?

tmaric: <p>I’m using PyTorch in a setting where the size of the training data can change from one learning task to the next one, so I need to check if the number of items in the tensor is different from the size of the training data, something like</p>
<pre><code class="lang-auto">    if (points_.sizes()[0] != mesh.nCells())
    {
        points_ = torch::zeros({mesh.nCells(),3});
        values_ = torch::zeros({mesh.nCells(),1});
    }
</code></pre>
<p>The mesh can change in terms of the number of cells (never mind what those are), and I would not like to perform <code>points_=</code> and <code>values_=</code> if this is not necessary. The number of <code>points_</code> equals the number of <code>values_</code>, which in turn equals to <code>mesh.nCells()</code>.</p>
Reduce GPU memory cost in building custom Convolution layer

ZifanZeng: <p>Dear all,</p>
<p>I’m building a custom convolution layer but have the problem of too much GPU memory used.</p>
<p>The idea of my custom conv operation is to add a weight “s” in the normal convolutional operation.</p>
<p>For example, I have an input of 16 , 1 , 28 , 28 and my custom conv kernel 32 , 1 , 3 , 3, with stride = 1, padding = 1.</p>
<p>Then in each input image (1 , 1 , 28 , 28) I slide the kernel throughout the input image and thus do 28 * 28 custom conv operations. In each custom conv operation I do <code>output = torch.sum(input * weight * s)</code> where s is calculated by input and weight in some ways.</p>
<p>So my way of implementing my custom conv layer’s forward function is:</p>
<ol>
<li>padding and unfolding the input into a Tensor of (16 , 1 , 784 (784 = 28 * 28) , 9 (9 = 1 * 3 * 3))</li>
<li>reshape the weight into Tensor of (32 , 1 , 9)</li>
<li>calculate the additional weight ‘s’ using the unfolded input and reshaped kernel weight, ‘s’ is a Tensor of (16 , 32 , 784 , 9)</li>
<li>point-wise multiplication between inputs (16 , 1 , 784 , 9), weight (32 , 1 , 9), and ‘s’ (16, 32, 784, 9), get the outputs (16, 32, 784, 9)</li>
<li>sum out the last dimension, and fold the outputs in the third dimension (784) into two dimensions (28, 28 which are the width and height of the ouput images.) Now I get the output I want as a Tensor of (16, 32, 28, 28)</li>
</ol>
<p>My custom conv layer has the desired performance as we expected, but the time cost and GPU memory cost are super high. I tried to use checkpointing to reduce the GPU memory cost, but it’s still super high.</p>
<p>So does anyone have suggestions about optimizing my implementations or helping me solve my problems? Also I’m interested in extending my custon conv in cpp, but I cannot find where the regular convolution operation is implemented. (I know <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" rel="noopener nofollow ugc">this file</a> but there seems to be no details about how conv operation is done. What I can find finally that is they used the <code>thnn_conv2d_forward</code> function.)</p>
<p>Sorry for my redundant English :P, and any infos are welcomed! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thank you!</p>

ptrblck: <p>The native implementations are mentioned in <a href="https://discuss.pytorch.org/t/source-code-f-conv2d-definition-location/81569/4">this post</a>, which might be helpful in writing your custom implementation.</p>

ZifanZeng: <p>Thank you Ptrblck! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Does list comprehension break the DAG for backprop?

Carsten_Ditzel: <p>Trying to combine several loss function in a dict called loss like so</p>
<p><code>err = torch.sum(torch.stack([l(input, label) for l in loss.values()]))</code></p>
<p>I am wondering if this breaks the connection for back prop</p>
<p>debugging tells me</p>
<blockquote>
<p>&lt;bound method Tensor.backward of tensor(17353.5000, device=‘cuda:0’, grad_fn=)&gt;</p>
</blockquote>
<p>which look ok but I am not certain if using an intermediate list followed by torch.stack succeeds in finalizing the computational graph</p>
<p>maybe someone knows the answer?</p>

ptrblck: <p>Your approach seems to work and I get valid gradients:</p>
<pre><code class="lang-python">output = torch.randn(1, 1, requires_grad=True)
target = torch.randn(1, 1)

loss = {i: nn.MSELoss() for i in range(10)}
err = torch.sum(torch.stack([l(output, target) for l in loss.values()]))
err.backward()
print(output.grad)
&gt; tensor([[-6.1927]])
</code></pre>

Carsten_Ditzel: <p>thank you Patrick. That was my observation too.<br>
However I was caught by surprise since there seems to be open issues regarding comprehension syntaxes cf. <a href="https://github.com/pytorch/pytorch/issues/48153" class="inline-onebox" rel="noopener nofollow ugc">More robust list comprehension · Issue #48153 · pytorch/pytorch · GitHub</a></p>
<p>But maybe that only relates to jit…</p>

ptrblck: <p>This issue seems to be specific to the JIT, but let me know, if you see any issues using the eager approach (e.g. <code>None</code> gradients where you would expect valid gradients).</p>
PyTorch Geometric Creating your own dataset

Saguaro: <p>Hi!</p>
<p>I am new to PyTorch and I have one task: my objective is to upload the personally collected data to the PyTorch. I am working with the PyTorch Geometric library extension.</p>
<p>So I have some problems with understanding the following code:</p>
<pre><code class="lang-auto">import os.path as osp

import torch
from torch_geometric.data import Dataset


class MyOwnDataset(Dataset):
    def __init__(self, root, transform=None, pre_transform=None):
        super(MyOwnDataset, self).__init__(root, transform, pre_transform)

    @property
    def raw_file_names(self): #Point 1
        return ['some_file_1', 'some_file_2', ...]

    @property
    def processed_file_names(self): #Point 2
        return ['data_1.pt', 'data_2.pt', ...]

    def download(self): #Point 3
        # Download to `self.raw_dir`.

    def process(self): #Point 4
        i = 0
        for raw_path in self.raw_paths:
            # Read data from `raw_path`.
            data = Data(...)

            if self.pre_filter is not None and not self.pre_filter(data):
                continue

            if self.pre_transform is not None:
                data = self.pre_transform(data)

            torch.save(data, osp.join(self.processed_dir, 'data_{}.pt'.format(i)))
            i += 1

    def len(self):
        return len(self.processed_file_names)

     def get(self, idx):
         data = torch.load(osp.join(self.processed_dir, 'data_{}.pt'.format(idx)))
         return data
</code></pre>
<p>I have marked the points I would like to discuss. So Point 1 and Point 2 are the names of the files I want to upload and then obtain.<br>
But what exactly are we doing at the Point 3, Pont 4 and after them? I have been trying to understand it also by looking at the similar source codes, but I cannot get it.<br>
Could somebody explain it to me in the pain words? With some examples it would be perfect.</p>
<p>Thank you in advance!</p>
<p>Regards</p>

ptrblck: <p>I’m not familiar with the real implementation of this dataset, but based on the function names I would assume this:</p>
<ul>
<li>
<code>raw_file_names</code>: returns the file names as a list of “raw” input data, which is not yet processed in any way</li>
<li>
<code>processed_file_names</code>: returns the file names of already processed names. Based on the file extension I assume these processed files are stored as PyTorch tensors (or at least via PyTorch).</li>
<li>
<code>download</code>: downloads the raw files to the <code>self.raw_dir</code> most likely from a remote server.</li>
<li>
<code>process</code>: iterates all raw files, loads and processes them. Afterwards it’s storing the processes data tensors to the <code>selfprocessed_dir</code>.</li>
<li>
<code>len</code>: returns the number of processed files</li>
<li>
<code>get</code>: loads a processed file from the <code>self.processed_dir</code> using the passed <code>idx</code> and returns it.</li>
</ul>
Model Summary from Pytorch

pgomez: <p>I am trying to obtain in Pytorch something similar as a Model.Summary from Keras. I am using the following code but I need to obtain the summary as well</p>
<p>class MaskCnnModel(ImageClassificationBase):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()  #<span class="hashtag">#Se</span> utiliza para no tener que volver a escribir el codigo de la clase anterior, en este caso la de ImageClassificationBase<br>
self.network = nn.Sequential(<br>
nn.Conv2d(3, 128, kernel_size=3, padding=1),<br>
nn.ReLU(),<br>
nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),<br>
nn.ReLU(),<br>
nn.MaxPool2d(2, 2), # output: 256 x 64 x 64</p>
<pre><code>        nn.Conv2d(256, 384, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.Conv2d(384, 384, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.MaxPool2d(2, 2), # output: 384 x 32 x 32

        nn.Conv2d(384, 512, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),
        nn.ReLU(),
        nn.MaxPool2d(2, 2), # output: 512 x 16 x 16
        
        nn.Flatten(), # toma el batch size, en este caso 344x10x10 y lo ordenamos en 2048 outputs y de ahi reducimos
        nn.Linear(512*16*16, 2048),  #hasta 2, que son las clases que tenemos
        nn.ReLU(),
        nn.Linear(2048, 1024),
        nn.ReLU(),
        nn.Linear(1024, 2))
    
def forward(self, xb):
    return self.network(xb)</code></pre>

ptrblck: <p>You could have a look at e.g. <a href="https://github.com/TylerYep/torchinfo">torchinfo</a>.</p>
Indexing tensor on cuda

eghbalhosseini: <p>Hi,</p>
<p>I am trying to understand how indexing in cuda works, I noticed that if I have a tensor in CUDA, and then use index_select  depending on whether the indices are sorted or unsorted I get different results. if indices are sorted, then I get the correct values. however if they are not, I tend to get  zero for some of the values. The code setting is as follows :</p>
<p>device =torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
torch.cuda.get_device_name(0)<br>
torch.backends.cudnn.deterministic = True<br>
torch.set_deterministic(True)</p>
<p>The original tensor is created inside a function. I have noticed that if I instead not use a function, this doesn’t occur</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce this issue as well as information about your current setup (PyTorch, CUDA version, used GPU, etc.)?</p>
Load multiple batched npz files or huge data files for asynchronous loading

Rohit_Modee: <p>I have a dataset of about 40 Gb. I have saved this as .npz files (150 files) <code>batch0.npz to batch150.npz</code>. Each contains 64 data points (batch size = 64). Need to create a balance between I/O reads hence saved 64 data points but I want to train my model with batch size = 32. with the shuffle. How should I do this? Below is the code I have right now.</p>
<pre><code class="lang-auto">ai = np.random.randn(64,70,128)
aj = np.random.randn(64,70,18,128)
ak = tnp.random.randn(64,70,50,128)
al = torch.rand(64,70,60,128)

sample = {ai, aj, ak, al}

np.savez("batch0.npz", "ai"=ai, "aj"=aj, "ak"=ak, "al"=al)
</code></pre>
<p>I create a dataset as follows</p>
<pre><code class="lang-auto">class load_batch_files(Dataset):
    def __init__(self):
        self.data_files = os.listdir("batch_data/")
        self.data_files.sort()

    def __getitem__(self, idx):
        # pdb.set_trace()
        return np.load(self.data_files[idx])

    def __len__(self):
        return len(self.data_files)

def somecollate(batch):
    # pdb.set_trace()
    batch = batch[0]
    ai = torch.tensor(batch["ai"]).float().to(device)
    aj = torch.tensor(batch["aj"]).float().to(device)
    ak = torch.tensor(batch["ak"]).float().to(device)
    al = torch.tensor(batch["al"]).float().to(device)
    batch = {"ai": ai, "aj": aj, "ak": ak, "al": al}
    return batch

desc_train = load_batch_files()
trainloader = DataLoader(desc_train, batch_size=32, collate_fn=somecollate)
</code></pre>
<p>This does not give me data with a batch size of 32. even shuffle does not work. what am I doing wrong?<br>
This is what I have tried <a href="https://discuss.pytorch.org/t/loading-huge-data-functionality/346/3">loading-huge-data-functionality</a></p>
GCN Input shape increase

szahan: <p>Hi I am trying to design a GCN encoder - decoder network. My input shape was 3<em>300</em>25 and after some GCN layers followed by average pooling the encoded shape is 256<em>1</em>1. I want my decoder to produce the input shape, so that I can compare the original and reproduced data. But I am lost how can I do it.<br>
My Encoder:<br>
input = [N, 3, 300, 25]<br>
h8 = [N, 256, 75, 25]<br>
encode = F.avg_pool2d(h8, h8.size()[2:])   <span class="hashtag">#shape</span>  [N, 256, 1, 1]</p>
<p>How can I use this “encode” to increase to original shape.</p>
<p>I am using AS-GCN blocks based on ST-GCN from this repo</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/limaosen0/AS-GCN" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/26040429?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/limaosen0/AS-GCN" target="_blank" rel="noopener nofollow ugc">limaosen0/AS-GCN</a></h3>


  <p><span class="label1">The model architecture of AS-GCN (for human action recognition)</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Question on nn.MultiheadAttention

weerapatp: <p>Hello everyone, I’m trying to code the Transformer for the first time using nn.MultiheadAttention. However, I’m very confused. Could someone help me with these two questions? <a href="https://pytorch.org/docs/master/nn.html#multiheadattention" rel="nofollow noopener">Reference to the documentation of nn.MultiheadAttention</a></p>
<ol>
<li>
<p>Why is the batch size of query not the first dimension? (The size of query for nn.MultiheadAttention is [target length, batch size, embed dim]). I wish it is [batch size, target length, embed dim] instead. I thought that torch.utils.data.DataLoader always put batch size to be the first dimension, and most other neural networks also put batch size in the first dimension. What is the reason behind this? I would like to know in case I’m missing something.</p>
</li>
<li>
<p>nn.MultiheadAttention returns attn_output and attn_output_weights. I would like to know why it returns attn_output_weights. Wouldn’t it be sufficient to just use the output “attn_output”? Do I need to use attn_output_weights somewhere in the Transformer?</p>
</li>
</ol>

andreabac3: <p>UP! I am also interested on that.</p>

googlebot: <ol>
<li>I think this order is needed for internal bmm operation, that does the main job. It is also consistent with RNNs’ default.</li>
<li>as need_weights argument may suggest, weights can be ignored.</li>
</ol>

veritas: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hello. I am also very interested in this question. Setting the sequence dimension to the first dimension would make the matrix multiply extremely inefficient due to memory access issues.<br>
Is this really because of compatibility with RNNs?<br>
If completely designed from scratch, the most reasonable order would appear to be (N, E, S) instead of the current (S, N, E) order. This would allow the sequence length axes, which are the ones being multiplied in the (self-attention matrix @ value matrix) operation, to be contiguous. As this is the most computationally intensive, this operation should be the most heavily optimized.<br>
cuDNN and cuBLAS optimizations mean that (N, S, E) order, which is called <code>batch_first </code>order in torchtext, will probably have similar performance, though this should be tested.</p>
Multi workers specified by num_workers load samples to form a batch, or each worker load a batch respectively in DataLoader?

yangwenhuan: <p>I mean, supposed batch_size=4 and num_workers=2, which of the following may match the runtime case?</p>
<p><span class="hashtag">#1</span><br>
worker1: load sample1, sample2<br>
worker2: load sample3, sample4<br>
batch1 contains sample1, sample2, sample3, sample4</p>
<p><span class="hashtag">#2</span><br>
worker1: load batch1<br>
worker2: load batch2</p>

alex.veuthey: <p>Apparently, it would be case <span class="hashtag">#2</span>. See <a href="https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/2">this answer</a>.</p>

yangwenhuan: <p>I have read the mentioned post, but how can I proof it? or any PyTorch source code show the fact?</p>

alex.veuthey: <p>The source code for the DataLoader logic is too complex for me, but I see two reasons why <span class="hashtag">#2</span> would be the approach used in practice:</p>
<ol>
<li>
<a class="mention" href="/u/apaszke">@apaszke</a> said it,</li>
<li>creating a batch from the two workers would require waiting for them to finish their task (getting data) before returning the batch, which is inefficient and (I guess depending on the architecture, data structure, storage etc…) useless compared to a single process loading sample1, sample2, sample3 and sample4 and then returning the batch.</li>
</ol>
<p>What is your use case?</p>

yangwenhuan: <p>for your first point, <a class="mention" href="/u/apaszke">@apaszke</a> only showed the result, but I want to know the reason.<br>
for the second, multi workers handle a batch may not be slower than single worker.</p>

Howl: <p>In my test case, I use 200 samples and set batch size is equal to 50. The result shows only 4 subprocess works even if i set num_workers=16(I have 16 CPUs). So I think <a href="https://discuss.pytorch.org/u/alex.veuthey">alex.veuthey</a> is right.</p>