akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Problem with reading pfm image

jj0mst: <p>I’m trying to load disparity images from <a href="https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html" rel="nofollow noopener">https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html</a> into pytorch tensors for my network.<br>
They are stored in PFM format and this code <a href="https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlow/assets/code/python_pfm.py" rel="nofollow noopener">https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlow/assets/code/python_pfm.py</a> is provided for reading them.<br>
It returns a numpy array, so i just used torch.from_numpy(), but i get</p>
<p>RuntimeError: some of the strides of a given numpy array are negative. This is currently not supported, but will be added in future releases.</p>
<p>Does anyone know a workaround for this or an ETA for negative strides support?</p>

jekbradbury: <p>The workaround would be to call <code>.as_c_contiguous_array()</code> (I think that’s the method name – maybe you need <code>.clone()</code> instead though) in numpy immediately before giving it to <code>.from_numpy()</code></p>

jj0mst: <p>Thanks, numpy.ascontiguousarray() worked.</p>
<p>I also discovered that the negative strides are caused by an upside-down flipping of the array (in the code used to read the pfm image), so deleting that line and flipping the pytorch tensor with an index_select() works too.</p>

111340: <p>a = readPFM(file)<br>
a = np.ascontiguousarray(a)<br>
a = torch.from_numpy(a)</p>
<p>still doesn’t work？</p>
What is the difference among padding_mode 'reflect', 'replicate' and 'circular'?

hanbit: <p>Hi, I’m using<code> nn.Conv1d</code> to handle sequential data, and want to use <em>mirror padding</em> not to focus out the boundary.</p>
<p>I found that in PyTorch, there are three types of padding except <code>'zeros', 'reflect', 'replicate', 'circular'</code> and what’s the different among them? I searched the docs, comments of source code, but there isn’t any comment about the difference.</p>
<p>Thanks!</p>
Pytorch (pth) to libtorch (pt) conversion

wolfviking0: <p>Hi,</p>
<p>I am currently working with deepspeech model, I was able to convert the model into libtorch, and also to try out the model in c++, but I am facing an issue, I feel it could be a lack of understanding when I convert the model, so maybe someone can guide me.</p>
<p>When converting the pth to pt from python you need to pass the tensor input example, but my understanding is then the pt file contains some folder Constants matching the exact input, then when I call the forward from C++ its seems using the exact same value of the input used when doing the conversion.</p>
<ol>
<li>
<p>Do I forgot something when converting the model ? Is there anyway to not generate this Constants ?</p>
</li>
<li>
<p>I can unzip the pt file and I would like to add some print or something in order to debug the different value during the forward, is it possible to do that ? I tried a print in python but not working.</p>
</li>
</ol>
<p>Thanks for feedback.</p>
Turn a model from double into float

ziqipang: <p>Dear all, Hi!</p>
<p>I am recently managing to use float32 as the precision of my model. The pytorch program on my computer seems to use “double” precision by default. I find it straight forward to cast a tensor from double to float, but how to define the precision of a model?</p>
<p>Thank you for your time and help!</p>

Nikronic: <p>Hi,</p>
<p>It will do the job:</p>
<pre><code class="lang-auto">model = model.to(torch.float)
</code></pre>
<p>Bests</p>

ziqipang: <p>Thank you! It works!</p>
Train directories multiple times

visionDose: <p>My data directory is as follows:</p>
<p>data/<br>
|–folder1/<br>
|–folder2/<br>
|–folder3/</p>
<p>I wanted to train multiple models in which test split contain folder1 and train split contains and folder2 and folder3</p>
<p>second, test contains folder2 and train contains folder1 and folder2 and so on…</p>
<p>A simple way is I can permutate folders and in a for loop use data loader and train and save weights for each model. Is there an efficient way to do this?</p>
Efficient way of doing multiple MLPS

hij: <p>I want to create multiple independent MLPs. For example, my input has 1000 dimensions, I want to split it up into 10 groups of 100. I want 10 separate MLPs (no shared weights), each for one of the groups. Is there a way to do tis efficiently? Right now I am just creating 10 different MLPs, and looping through them in the forward call. But it seems to be quite slow.</p>
<p>I tried implementing it via a 1x1 convolution. It is much faster but I do not want the weight sharing.</p>
Larger batch size didn't bring shorter training time

111338: <p>I’m new to semantic segmentation and distribute dataparallel of PyTorch.<br>
And I run this <a href="https://github.com/hszhao/semseg" rel="nofollow noopener">pspnet</a> code on my machine (2 x P100).<br>
To my knowledge, larger batch size usually brings shorter training time. But it didn’t happen this time.<br>
On one GPU, the training time with <code>batch_size=8</code> is actually the same as (even a little longer) than <code>batch_size=2</code>. Specifically, the training time for 10 iterations with <code>bs=8</code> is 4 times of that <code>bs=2</code>.<br>
Did I make some mistake here?</p>

Kiuk_Chung: <p>GPU kernels are typically launched with block batch sizes (typically the data batch size is some multiple of warp size - 32 - since CUDA runs instructions in SIMD fashion in warps). So it is possible that between 8 and 2 the underlying CUDA kernel is actually launched with the same batch size (say 32) in which case the other rows in the batch are typically zero-filled or just discarded from the result, and you’ll see roughly the same runtime.</p>
<p>Can you share the runtime differences for iteration 1,2,3,…, 10,11, …? Its possible there are other operations in the pipeline that is causing the inverted performance numbers.</p>
Dimension out of range

Ernst_Oberortner: <p>Dear all,</p>
<p>we’re experimenting with Transformers for deep-learning the translation from Protein to DNA sequences. Here’s an excerpt of our code:</p>
<pre><code>##
## ITERATOR
##
BATCH_SIZE = 10
train_iterator, test_iterator = BucketIterator.splits(
    (train_data, test_data), 
    batch_size = BATCH_SIZE,
    device = device, sort = False)

##
## TRANSFORMER
##
model = nn.Transformer(d_model=1, nhead=1)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1)
for i, batch in enumerate(train_iterator):
    
    lbl = batch.seq_id
    src = batch.prot_seq
    trg = batch.dna_seq

    optimizer.zero_grad()
    model.train()

    output = model(src, trg)
    print(model)
</code></pre>
<p>However, this code produces the following error:</p>
<p>Traceback (most recent call last):<br>
File “learner.py”, line 195, in <br>
output = model(src, trg)<br>
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/modules/transformer.py”, line 117, in forward<br>
print(f’src.size(2) --&gt; {src.size(2)}’)<br>
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)</p>
<p>src and trg both are mxn matrices, where m the length of the protein and DNA sequence respectively and n is the BATCH_SIZE. According to the documentation (<a href="https://pytorch.org/docs/master/generated/torch.nn.Transformer.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.nn.Transformer.html</a>), the third dimensions of src and trg must be the the feature number E. What does this exactly mean? What’s the required form of src and trg? Also, how can I add E to the src and the trg matrices?</p>
<p>Thanks for shedding light into my confusion!</p>
How to draw graph in tensorboardX with a two inputs model?

Mata_Fu: <p>The inputs of model are like this</p>
<blockquote>
<p>output = model(input1, input2)</p>
</blockquote>
<p>How can I use function <code>writer.add_graph(model, input)</code> ?</p>

Jaja: <p><strong>Here is an example:</strong></p>
<p>import torch<br>
import torch.nn as nn<br>
from tensorboardX import SummaryWriter</p>
<p>input1 = torch.ones(1, 2)<br>
input2 = torch.randn(1, 2)<br>
<strong><em>batch_input = torch.cat((input1, input2), 0)    # Two input</em></strong></p>
<p>class SimpleLinear(nn.Module):</p>
<pre><code>def __init__(self):
    super(SimpleLinear, self).__init__()
    self.slin = nn.Linear(2, 4)

def forward(self, x):
    return self.slin(x)
</code></pre>
<p>with SummaryWriter(‘runs/graph2’, comment=‘SimpleLinear’) as w:</p>
<blockquote>
<p>w.add_graph(SimpleLinear(), batch_input, True)</p>
</blockquote>

palimboa: <p>Hi,</p>
<p>you can do:</p>
<p><code>writer.add_graph(model, (input1, input2) )</code></p>
<p>Here is an <a href="https://github.com/lanpa/tensorboardX/blob/master/examples/demo_graph.py" rel="nofollow noopener">example</a></p>

0bff83efac608c536648: <p>hi I got the error:  Type ‘Tuple[Tensor, Dict[str, List[int]]]’ cannot be traced. Only Tensors and (possibly nested) Lists, Dicts, and Tuples of Tensors can be traced<br>
The code is below:<br>
writer.add_graph(flow_net, (torch.DoubleTensor(dataset.datas_vali[:2,:]), idx))</p>
Combine ImageFolder dataset with custom dataset

CopyOfA: <p>I am loading data from multiple datasets. I have some images stored in properly labeled folders (e.g., \0 and \1), and in those cases I can use torch.utils.data.ConcatDataset after loading the lists, for example (where trans is a set of pre-defined Pytorch transformations):</p>
<pre><code class="lang-auto">l = []
l.append(datasets.ImageFolder(file_path, trans))
l.append(datasets.ImageFolder(file_path2, trans))
image_datasets = torch.utils.data.ConcatDataset(l)

img_datasets = dict()
img_datasets['train'], img_datasets['val'] = torch.utils.data.random_split(image_datasets, (round(0.8*len(image_datasets)), round(0.2*len(image_datasets)) ))
</code></pre>
<p>However, I am also loading images from other disparate locations using a csv file. The process here looks like this:</p>
<pre><code class="lang-auto">class MyData(Dataset):
  def __init__(self, df):
      self.df = df

  def __len__(self):
      return self.df.shape[0]

  def __getitem__(self, index):
      image = trans(PIL.Image.open(self.df.file_path[index]))
      label = self.df.label[index]

      return image, label


df = pd.read_csv(image_file_paths), names=["file_path", "label"])
mydata = MyData(df)

my_datasets = dict()
my_datasets['train'], my_datasets['val'] = torch.utils.data.random_split(mydata, (round(0.8*len(mydata)), round(0.2*len(mydata))))
</code></pre>
<p>So I’d like to be able to combine these datasets into a single dataloader. Any ideas for how I should go about doing this? Thanks!</p>

CopyOfA: <p>Found the solution; just need to use multiple passes of ConcatDataset:</p>
<pre><code class="lang-auto">l = []
l.append(datasets.ImageFolder(file_path, trans))
l.append(datasets.ImageFolder(file_path2, trans))
image_datasets = torch.utils.data.ConcatDataset(l)

df = pd.read_csv(image_file_paths), names=["file_path", "label"])
mydata = MyData(df)

image_datasets = torch.utils.data.ConcatDataset([image_datasets, mydata])

img_datasets = dict()
img_datasets['train'], img_datasets['val'] = torch.utils.data.random_split(image_datasets, (round(0.8*len(image_datasets)), round(0.2*len(image_datasets))))
</code></pre>
<p>Good to go from there.</p>
High CPU usage when running my code on GPU

yamand16: <p>Hello everyone,</p>
<p>Although I did lots of research about my question, I could not solve the problem yet. I am new in PyTorch and I am trying to run basic code from GitHub for training GAN. Although all processes in the code are working on GPU, the CPU usage is 100% (even more) during training.</p>
<p>Here is the original code:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/wgan_gp/wgan_gp.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/wgan_gp/wgan_gp.py" target="_blank" rel="nofollow noopener">eriklindernoren/PyTorch-GAN/blob/master/implementations/wgan_gp/wgan_gp.py</a></h4>
<pre><code class="lang-py">import argparse
import os
import numpy as np
import math
import sys

import torchvision.transforms as transforms
from torchvision.utils import save_image

from torch.utils.data import DataLoader
from torchvision import datasets
from torch.autograd import Variable

import torch.nn as nn
import torch.nn.functional as F
import torch.autograd as autograd
import torch

os.makedirs("images", exist_ok=True)

</code></pre>

  This file has been truncated. <a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/wgan_gp/wgan_gp.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I updated this code and I changed the generator and discriminator. Besides, in order to use my data, I added the following data loading code that I checked on PyTorch official documentation.</p>
<pre><code class="lang-auto">def Read_LMDB(root, classes):
    d1 = torchvision.datasets.LSUN(root=root, classes=classes, transform=transforms.Compose(
                [transforms.Resize((opt.img_size, opt.img_size)), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))
    dataloader = torch.utils.data.DataLoader(d1, batch_size=opt.batch_size, shuffle=True)
    return dataloader

def Read_from_folder(folder_name):
    imagefolder = datasets.ImageFolder(folder_name,transform=transforms.Compose(
        [transforms.Resize(opt.img_size), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))
    dataloader = torch.utils.data.DataLoader(imagefolder,batch_size=opt.batch_size,shuffle=True)
    return dataloader
</code></pre>
<p>However, still the same problem. It does not matter which functions I used, the CPU usage is 100%. I think using such high CPU usage is not normal since the training is performed on GPU (and there is no problem with GPU usage. The GPU memory usage, utility, and the training speed are okay).</p>
<p>Could anyone make a suggestion to me that makes CPU usage less? Thank you!</p>
<p>Best.</p>
How to save training iterations output?

shwe87: <p>Hello,</p>
<p>So I have a training loop and I am printing the following:</p>
<pre><code class="lang-auto">Train Epoch: 1 [0/8812 (0%)]	Loss: 0.509592	Valid Loss: 0.463734
Train Epoch: 1 [1200/8812 (36%)]	Loss: 0.600266	Valid Loss: 0.387744
Train Epoch: 1 [2400/8812 (72%)]	Loss: 0.351144	Valid Loss: 0.368207
Epoch 1 took 595.4694926738739 seconds
Train Epoch: 2 [0/8812 (0%)]	Loss: 0.179442	Valid Loss: 0.367970
Train Epoch: 2 [1200/8812 (36%)]	Loss: 0.178893	Valid Loss: 0.362563
Train Epoch: 2 [2400/8812 (72%)]	Loss: 0.000000	Valid Loss: 0.372528
Epoch 2 took 535.0275015830994 seconds
Train Epoch: 3 [0/8812 (0%)]	Loss: 0.276426	Valid Loss: 0.365664
Train Epoch: 3 [1200/8812 (36%)]	Loss: 0.267340	Valid Loss: 0.367112
Train Epoch: 3 [2400/8812 (72%)]	Loss: 0.270740	Valid Loss: 0.384545
Epoch 3 took 548.3842833042145 seconds
Train Epoch: 4 [0/8812 (0%)]	Loss: 0.364331	Valid Loss: 0.365495
Train Epoch: 4 [1200/8812 (36%)]	Loss: 0.300914	Valid Loss: 0.363899
Train Epoch: 4 [2400/8812 (72%)]	Loss: 0.441467	Valid Loss: 0.360874
Epoch 4 took 539.1664731502533 seconds
Train Epoch: 5 [0/8812 (0%)]	Loss: 0.599065	Valid Loss: 0.358953
Train Epoch: 5 [1200/8812 (36%)]	Loss: 0.374723	Valid Loss: 0.358028
Train Epoch: 5 [2400/8812 (72%)]	Loss: 0.263893	Valid Loss: 0.361765
</code></pre>
<p>Is there any way to store this in a file (but also printing) or in a table or something?</p>
<p>I want to use these as results for a research…</p>
<p>Thanks.<br>
BR,<br>
Shweta.</p>

Chirath_Pansilu: <p>You can use pickle library to do that.</p>
<p>First you need to create a empty <code>list</code> for storing results. Then you can append the needed infomation (epoch, loss , valid_loss) into that  list in each training loop. It will  be easy to append these information as a tuple in each iteration.</p>
<p>Then after training you can save this <code>list</code> to a file via pickle library. Which is a file you can load into python later,  as the saved <code>list</code></p>
<p>As a example–&gt;</p>
<pre><code class="lang-auto">import pickle as pkl

results = []
.
.
for e in range(epochs):
             .
             .
             .
             .
             results.append((epoch, loss.item(), valid_loss.item()
             .
             .

with open('results.pkl', 'wb') as f:       #this will save the list as "results.pkl" which you can load in later 
             pkl.dump(results, f)          #as a list to python  
</code></pre>
<p>Loading the saved list ----&gt;</p>
<pre><code class="lang-auto">with open('results.pkl', 'rb') as f:
    results = pkl.load(f)
</code></pre>
<p>Hope this helps</p>
Prepare sqrt (N )/sqrt(N) square image from N tiles

Jaideep_Valani: <p>How can we make square image from  given N tiles that is perfect square . Say N=36<br>
each tile is 3 * H * W dimensions</p>
<pre><code class="lang-auto">N=25
Ng=5
for i in range(N//Ng):
            imgs_tmp=[imgs[i*Ng+j] for j in range(Ng) if i*Ng+j &lt;= len(imgs)]
          
            combine_tensor_list.append(torch.cat(imgs_tmp,0))
        
    
    img=torch.cat(combine_tensor_list,0).view(3,Ng*imgs[0].size(1),Ng*imgs[0].size(2))
</code></pre>
<p>I am using above logic but gives a very weird image. dsnt looks to be right one wrt to view of single tile image.</p>

chetan_patil: <p><a class="mention" href="/u/jaideep_valani">@Jaideep_Valani</a>  Could you please elaborate the question a little bit more  ?</p>

Jaideep_Valani: <p>i have got  16 small rgb  patches of a big image…<br>
I want to combine them into 4/4 square grid to make a single sq size image out of that</p>

chetan_patil: <p>Your problems seems interesting, just trying to solve it.<br>
One question, does your rgb patches come in this format (N, 3, h,w) ?</p>

arogozhnikov: <p>assuming your stack is 16x3xhxw this will give you an image in HxWx3:</p>
<pre><code class="lang-auto">import einops
einops.rearrange(stack, '(b1 b2) c h w -&gt; (b1 h) (b2 w) c', b1=4)
</code></pre>

Jaideep_Valani: <p>thanku what are arguements b1,b2<br>
suppose  i have 16 image tiles i need 4*4  image from this ,what would be b1,b2</p>

arogozhnikov: <blockquote>
<p>thanku what are arguements b1,b2<br>
suppose i have 16 image tiles i need 4*4 image from this ,what would be b1,b2</p>
</blockquote>
<p>b1=4, b2=4 (but you can specify only one, second will be guessed since 16 / 4 = 4)</p>
Batched sequence indexing

Tin: <p>Hi,</p>
<p>I have a tensor which represents a batch of same-length sequences. Each item in the sequence is a vector of probabilities across the item space.<br>
The tensor is of dimension <code>(batch_size, seq_length, num_items)</code>.<br>
I want to index it with an index tensor of dimension <code>(batch_size, variable_length)</code>, where, for each batch item, I would index a variable number of items, across the item space, and in the same way for each element of the sequence at this batch index.<br>
The goal is to set each of these indexed elements to the same scalar <code>x</code>.</p>
<p>I tried a simple <code>my_tensor[index] = x </code>, however this attempts indexing <code>my_tensor</code> over the batch dimension. I am not sure how to achieve a ‘repeated’ indexing across the sequence length.</p>

ptrblck: <p>Hi,</p>
<p>could you post a minimal (slow) code snippet, which shows the desired results using loops?</p>

Tin: <p>This is the desired behaviour from python lists.</p>
<pre><code class="lang-auto"># my_batch is of dimension (batch_size, seq_length, num_items)
# indices is of dimension (batch_size, num_items)
# x is some value

for i in range(batch_size):
  index = indices[i]
  for j in seq_length:
    for index in indices:
      my_batch[i, j, index] = x
</code></pre>
<p>I solved it using <code>scatter_</code>:</p>
<pre><code class="lang-auto"># make indices same dimension as my_batch
indices = indices.unsqueeze(1).repeat(1, seq_length, 1)
my_batch = my_batch.scatter_(dim=2, index=indices, src=x)
</code></pre>
Transfrom Compositions

bluesky314: <p>Hey is there a way to do compose multiple sets of transforms like:?</p>
<pre><code class="lang-auto">        OneOf([IAAAdditiveGaussianNoise(),
            GaussNoise(),
        ], p=0.2),
        OneOf([
            MotionBlur(p=.2),
            MedianBlur(blur_limit=3, p=0.1),
            Blur(blur_limit=3, p=0.1),
        ], p=0.2),
</code></pre>
Convolution output different result

Eta_C: <p>Here’s my code.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


x = torch.rand(3, 3, 5).float()
weight = torch.rand(6, 3, 1).float()
bias = torch.rand(6).float()

net = nn.Conv1d(3, 6, 1, bias=True)
net.weight = nn.Parameter(weight.clone(), requires_grad=True)
net.bias = nn.Parameter(bias.clone(), requires_grad=True)
y_src = net(x.clone())

net_no_bias = nn.Conv1d(3, 6, 1, bias=False)
net_no_bias.weight = nn.Parameter(weight.clone(), requires_grad=True)
y_dst = net_no_bias(x) + bias.view(1, -1, 1)

print((y_src - y_dst).abs().max())  # tensor(2.3842e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Why? I run this code on CPU and expect to get 0. What’s the different between them?</p>

Eta_C: <p>BTW, if the values in <code>weight</code> and <code>bias</code> are large, the deviation will be amplified to <code>0.5</code>.</p>

ptrblck: <p>These errors are most likely due to the limited floating point precision:</p>
<pre><code class="lang-python">torch.manual_seed(2809)

x = torch.randn(100, 100, 100)
y = torch.randn(100, 100, 100)

x_sum1 = x.sum(0).sum(0).sum(0)
x_sum2 = x.sum(2).sum(1).sum(0)
y_sum1 = y.sum(0).sum(0).sum(0)
y_sum2 = y.sum(2).sum(1).sum(0)

res1 = x_sum1 * y_sum1
res2 = x_sum2 * y_sum2 

print(x_sum1-x_sum2)
&gt; tensor(-0.0002)
print(y_sum1-y_sum2)
&gt; tensor(-0.0006)
print(res1-res2)
tensor(0.6250)
</code></pre>
<p>If you need more precision, you could use <code>.double()</code> precision.</p>

Eta_C: <p>Is there any other way to make <code>y_src - y_dst</code> equal to 0? I want to replace each convolution of my network to convolution_without_bias+bias_add. As the depth of the network increases, this phenomenon is amplified.</p>
<p>Convert all data to <code>double</code> would works, but waste too many memory.<img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>

ptrblck: <p>I don’t think you can force the order of operation to be constant, at least I’m not aware of any way.<br>
How large are your activation values, when you have the large error?<br>
I assume you would like to get rid of the <code>0.5</code> in absolute difference.</p>

Eta_C: <p>Yes I want to get rid of the <code>0.5</code> in absolute difference.<br>
I will try to use this result for subsequent operations. If the end result is acceptable, I think this phenomenon could be ignored.<br>
God bless me.</p>
PyTorch slicing

IGero228: <p>Have tensor like</p>
<pre><code class="lang-auto">tensor =  [1,2,3,4,5,6,7,8,9,10].
</code></pre>
<p>How I can get tensor like this:</p>
<pre><code class="lang-auto">[1,2,5,6,9,10]
</code></pre>
<p><strong>take 2 elements with step 2(::2)</strong></p>
<p>I try something like  <code>tensor[:2::2]</code> .it’s not work</p>

albanD: <p>Hi,</p>
<p><code>[:2::2]</code> is not a valid indexing syntax in python.<br>
And just like for a regular python list, you cannot do what you want with just indexing.<br>
You can either take the slices by hand and <code>torch.cat()</code> then to get the final result.<br>
Or generate the indices you want to keep and use torch.index_select or torch.gather to get the values.</p>
Backward implementation of a new activation function

rohithkrn: <p>I am experimenting with implementing a custom activation function. To start with, I tried to mimic the behavior of <code>relu</code>. So, I added the custom op <code>my_relu</code> in <code>ATen\native\native_functions.yaml</code> which dispatches <code>relu</code>.</p>
<pre><code class="lang-auto">- func: my_relu(Tensor self) -&gt; Tensor
 use_c10_dispatcher: full
 variants: function, method
 dispatch:
   CPU: relu
   CUDA: relu
   MkldnnCPU: mkldnn_relu
   QuantizedCPU: quantized_relu
</code></pre>
<p>I built pytorch from source after this change and I could run the forward pass successfully.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.randn(5, device='cuda')
&gt;&gt;&gt; x.requires_grad_()
tensor([ 1.8718,  0.7903, -0.6581,  1.3783, -1.1062], device='cuda:0',
       requires_grad=True)
&gt;&gt;&gt; x
tensor([ 1.8718,  0.7903, -0.6581,  1.3783, -1.1062], device='cuda:0',
       requires_grad=True)
&gt;&gt;&gt; y = x.relu()
&gt;&gt;&gt; y
tensor([1.8718, 0.7903, 0.0000, 1.3783, 0.0000], device='cuda:0',
       grad_fn=&lt;ReluBackward0&gt;)
&gt;&gt;&gt; z = x.my_relu()
&gt;&gt;&gt; z
tensor([1.8718, 0.7903, 0.0000, 1.3783, 0.0000], device='cuda:0',
       grad_fn=&lt;NotImplemented&gt;)
</code></pre>
<p>As you can see, <code>z</code> (<code>my_relu(x)</code>) has <code>grad_fn=&lt;NotImplemented</code>. Could some one help with where the backward function should be added. I tried to follow <code>relu</code>'s implementation but couldn’t find the answer.</p>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>You will need to specify it in <code>tools/autograd/derivatives.yaml</code> then recompile.<br>
You can check the entry for relu there and the comment at the beginning of the file for more details.</p>
Assigining torch.Tensor(x).cpu() to cuda in runtime

barani_prashanth: <p>I’ve a C++ application with main and worker threads calling the python modules for the inference.</p>
<p>So with the main thread I can execute the below lines</p>
<blockquote>
<p>img =   torch.Tensor(x).cpu().unsqueeze(0)<br>
img = img.cuda()</p>
</blockquote>
<p>The tensor are switching to cuda tensor.</p>
<p>But the above line img.cuda()  will crash when calling from the worker thread.<br>
It is understood that worker thread wont have access to create a cuda memory so cant switch from cpu to cuda tensor.<br>
Anyway to solve this problem switiching from cpu() to cuda() in the worker thread?</p>

albanD: <p>Hi,</p>
<p>All the threads should have the same access to cuda. So it should not be an issue.<br>
What is the exact error message you’re seeing?</p>
Pytorch nested nn modules

witha: <p>Hi everyone!</p>
<p>I’m new to PyTorch and I have a problem regarding nested PyTorch nn modules. The problem as follows.</p>
<p><strong>I have a main nn model lets say model A. Which in turn uses two objects created from another nn class inside a for loop that contains lstm cells(single-cell)</strong>.</p>
<p>First, the autograd complains that the graphs need to be retained because backward is going through multiple times over and over. After making retain_graph = True it gives the following error.**</p>
<p>**RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2048]] is at version 2; expected version 1 instead. **</p>
<p>By the way, it does the first forward pass successfully. It would be greatly helpful if one of you guys could show me a consistent way to implement a subset of neural agent objects inside a for loop in the main nn.module class .</p>
<pre><code>class ModelA(nn.Module):
   def __init__():
      super(ModelA, self).__init__()
      self.agent1 = Agent(#initialising parameters)
      self.agent2 = Agent(#initialising parrameters)
      out1 = torch.zeros(#required dimension)
      out2 = torch.zeros(#required dimension)

   def forward(self, inputs):
      for i in range(25):
         out1 = self.agent1(out2, #for loop ends = false)
         out2 = self.agent2(out1, #for loop ends = false)
      y1 = self.agent1(out2, # for loop ends = True)
      x1 = self.agent2(out1, # for loop ends = True)
      return x1, y1

class Agent(nn.Module):
    def __init__():
    super(Agent, self).__init__()
    # initialize all the layers and recurrent cells here

    def forward(#inputs, #whether for loop ends)
       if for_loop_ends = True
          return #result1.
       else:
          return #result2</code></pre>

albanD: <p>Hi,</p>
<p>Do you properly re-initialize the hidden state to make sure it does not backrprop through the previous iterations. Otherwise you will get the error that you see above with trying to backward though the graph multiple times.</p>

witha: <p>Hi i add re-initialisation at the if loop where it checks the condition for loop end. And it worked like a charm. Thanks a lot.</p>

witha: <p>Hi alban,<br>
Just another question. According to my structure are these agents sharing any parameters?.</p>

albanD: <p>If each of them is creating it’s own <code>nn.Parameter</code> then they won’t share no.</p>
GAN works differently in fully convolutional mode

edyuan: <p>I’m running a pix2pix model trained on 256x256 images, which are crops of a larger 1024x512 image (no rescaling going on here).  I then try to test my pix2pix model on the full 1024x512, which should yield identical results as the entire model is fully convolutional.  However, the resultant images are quite different.</p>
<p>Pix2pix testing on 256x256 patches:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7ebbf224ad79474dd5f788f6b2b0a3bc35476c7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d7ebbf224ad79474dd5f788f6b2b0a3bc35476c7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7ebbf224ad79474dd5f788f6b2b0a3bc35476c7_2_690x214.jpeg" alt="image" data-base62-sha1="uO7JWbIrqJqnpEWP1nHlMpbWtcH" width="690" height="214" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7ebbf224ad79474dd5f788f6b2b0a3bc35476c7_2_690x214.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7ebbf224ad79474dd5f788f6b2b0a3bc35476c7.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7ebbf224ad79474dd5f788f6b2b0a3bc35476c7.jpeg 2x" data-dominant-color="75607A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1010×314 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Pix2pix testing on full 1024x512 image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75a0cd784d57610654521382573d6d24eedba4c5.jpeg" alt="image" data-base62-sha1="gMAnGDGwGqyEIxkuWGLNSRgohwx" width="556" height="333"></p>
<p>The most noticeable difference is the large white patch in the 1024x512 image.  Any ideas what is causing such an artifact?</p>
<p>My initial thoughts are that it could be an unusual edge effect do the black areas in the larger image.  The receptive field of generator in the pix2pix model is enormous - 744x744 pixels, which is mostly wasted in the case of the 256x256 model.</p>
<p>The model is running batchnorm in eval model during test time - using the running mean and std.  So I don’t think this is causing the problem.</p>
<p>Does anyone have ideas on causes of/how to address such a problem?  Thanks!</p>
How to generate a Tensor from a cv::cuda::GpuMat directly?

PistonY: <p>Normally we could generate a <code>Tensor</code> from <code>cv::Mat</code> by:</p>
<pre><code class="lang-auto">cv::Mat img = ....
img.convertTo(img, CV_32FC3, 1.0f / 255.0f);
at::Tensor tensor = torch::from_blob(img.data, {img.rows, img.cols, 3});
</code></pre>
<p>But when this img is a <code>cv::cuda::GpuMat</code> the <code>torch::from_blob</code> can’t work.</p>
<pre><code class="lang-auto">Signal: SIGSEGV (Segmentation fault)
</code></pre>
<p>I hope the tensor is same device with this <code>cv::cuda::GpuMat</code>,Does libtorch support this?</p>

PistonY: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/13898" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/13898" target="_blank" rel="nofollow noopener">Importing an opencv GpuMat</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-11-13" data-time="15:16:45" data-timezone="UTC">03:16PM - 13 Nov 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-11-26" data-time="21:12:34" data-timezone="UTC">09:12PM - 26 Nov 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/leconteur" target="_blank" rel="nofollow noopener">
          <img alt="leconteur" src="https://avatars3.githubusercontent.com/u/1876772?v=4" class="onebox-avatar-inline" width="20" height="20">
          leconteur
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Contrary to opencv Mat objects, GpuMat are strided in memory. It seems impossible to tell the torch::from_blob method from the...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to script a model using c++ extension? I met this error

Aiden: <p>How to script a model using c++ extension? I met this error</p>
<pre><code class="lang-auto">RuntimeError: 
Could not export Python function call '_DCNv2'. Remove calls to Python functions before export. Did you forget add @script or @script_method annotation? If this is a nn.ModuleList
</code></pre>

G.M: <p>Can u show the part of code that produced this error?</p>

Aiden: <p>I try to use “torch.jit.trace” to script EDVR(<a href="https://github.com/xinntao/EDVR" rel="nofollow noopener">https://github.com/xinntao/EDVR</a>) model as follows:</p>
<pre><code class="lang-auto">import models.modules.EDVR_arch as EDVR_arch
device = torch.device('cuda')

model_path = '../experiments/pretrained_models/train_perceptual_loss_420600_G.pth'
N_in = 5
predeblur, HR_in = True, True
model = EDVR_arch.EDVR(64, N_in, 8, 5, 10, predeblur=predeblur, HR_in=HR_in, w_TSA=False) 
model.load_state_dict(torch.load(model_path), strict=True)
model.eval()
model = model.to(device)
example = torch.rand((1, 5, 3, 512, 512), device='cuda:0')
torch_script_model = torch.jit.trace(model, example)
torch_script_model.save('edvr.pt')
</code></pre>
<p>EDVR  use c++ extension to implement deformable convolutional networks</p>

G.M: <p>I think <code>trace</code> is having trouble with the c++ extensions. To make <code>jit.trace</code> work with C++ extension, those C++ operations should be registered to jit. See this <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="nofollow noopener">link</a> for more information.</p>

Aiden: <p>Thand you! I will try, and then feedback here.</p>

Kiki_Rizki_Arpiandi: <p>I face the same problem. any update on this?</p>
Half Precision BackProp Taking up More RAM than Float

Henry_Chinner: <p>I have a model that takes approx 8gb RAM. When I convert the model to half via model.half() the RAM blows up to 48GB during backward().</p>
<p>Has anyone encountered a similar problem?</p>

albanD: <p>Hi,</p>
<p>Do you run this on CPU?<br>
I don’t think the CPU actually has half precision implementation and so it falls back to single precision float. This might explain the memory increase <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Henry_Chinner: <p>Hi,</p>
<p>It is running on GPU Tesla V100. It is running fine as float. Get’s in just under 16GB. But when I switch to half it blows up during the backward pass. The memory goes up to 48GB then.</p>

albanD: <p>The CPU memory or the GPU memory? V100 don’t have 48GB of memory right?</p>

Henry_Chinner: <p>GPU memory. I get an errror message saying the GPU tried to allocate 48GB RAM and then crashes as it only has 16GB.</p>

ptrblck: <p>Could you post your model definition here so that we could reproduce this behavior, please?</p>