akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Imagenet Standard Normalization/Standardization

seer_mer: <p>Can someone point out which <strong>reliable source</strong> does the “mean = [0.485, 0.456, 0.406]” and “std = [0.229, 0.224, 0.225]” normalization for imagenet come from?</p>
<p>After researching and reading some articles, whenever someone asks this, it seems like all articles I read are just referring to non-academic or unreliable sources like blogs, discussions, or even “by conventions”.</p>
<p>The reason I have this question is that in TensorFlow/Keras application implementations (for example, EfficientNet), they use [0.229, 0.224, 0.225] as <strong>variance</strong> to standardize the images. However, Pytorch official document uses the exact same value as <strong>standard deviation</strong> to standardize images.</p>
<p>This causes difference since normalized = (data - mean) / std = (data - mean) / sqrt(variance),<br>
which therefore causes difference in resulting standardized data, the pytorch version results in data ranging in approximately (-2.1, 2.2), while tensorflow version results in data ranging in approximately (-1.0, 1.1).</p>
<p>From what I learned, standardizing data to approx. the unit length seems to make more sense, but using [0.229, 0.224, 0.225] as std seems to be a more common approach. Can anyone explain which is correct and is there any paper/academic article on it. Thanks</p>

ptrblck: <aside class="quote no-group" data-username="seer_mer" data-post="1" data-topic="126430">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seer_mer/48/40173_2.png" class="avatar"> seer_mer:</div>
<blockquote>
<p>This causes difference since normalized = (data - mean) / std = (data - mean) / sqrt(variance),<br>
which therefore causes difference in resulting standardized data, the pytorch version results in data ranging in approximately (-2.1, 2.2), while tensorflow version results in data ranging in approximately (-1.0, 1.1).</p>
</blockquote>
</aside>
<p>You could check the stddev and variance of the resulting tensor and make sure it has a unit variance (=1), which is the purpose of this normalization/standardization step.<br>
I’m not familiar with the TF implementation and don’t know what internally is used.</p>
Convention for: import torch as

mhalsharif: <p>When reading through pytorch documentation/codes, one can quickly notice the habit of typing ‘torch’ in full many times instead of using a 2-letter abbreviation such as np for numpy or tf for tensorflow. Of course, you’re free to use whatever abbreviation you want or none at all, but I think it would be better if the pytorch community agreed on a standard convention that is used and seen across all codes/documents. Two possibilities come to mind are:</p>
<ul>
<li>import torch as pt       # pt for pytorch</li>
<li>import torch as tc       # tc for torch, using the 1st and 4th letters from torch, similar to np for numpy</li>
</ul>
<p>Any thoughts?</p>

ajbrock: <p>Just “torch” is fine–it’s not like you’re typing “TensorFlow,” all PyTorch code I’ve ever seen uses it (tutorials included), and switching everyone would be an inordinately huge hassle just to save two characters here and there.</p>

wizardforcel: <p>Why not be simpler? just</p>
<pre><code class="lang-auto">import torch as t
</code></pre>

codingpy: <p>I like <code>import torch as th</code>, but few people do so.</p>
Received an invalid combination of arguments

anoud_abdalziz: <p>return _ssim(img1, img2, window, self.window_size, channel, self.size_average)</p>
<p>File “C:\Users\anoud\anaconda3\envs\env1\lib\site-packages\pytorch_ssim_<em>init</em>_.py”, line 18, in _ssim<br>
mu1 = F.conv2d(img1, window, padding = window_size/2, groups = channel)</p>
<p>TypeError: conv2d() received an invalid combination of arguments - got (Tensor, Tensor, padding=float, groups=int), but expected one of:</p>
<ul>
<li>(Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)</li>
<li>(Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)</li>
</ul>

ptrblck: <p>Base in the error message you are passing a <code>float</code> value as the <code>padding</code> argument, which is not supported.<br>
Transform <code>window_size/2</code> to an <code>int</code> and it should work.</p>
Loading extension module _prroi_pooling Segmentation fault

InternalInstance: <p>Sometimes an error occurs when running object tracking on Jetson XAVIER.<br>
It happens after the loading of the extension prroi_pooling.</p>
<pre><code class="lang-auto">Detected CUDA files, patching ldflags
Emitting ninja build file /home/roboneers/.cache/torch_extensions/_prroi_pooling/build.ninja...
Building extension module _prroi_pooling...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module _prroi_pooling...
Segmentation fault
</code></pre>
<p>Maybe someone has encountered this problem and knows how to solve it.</p>
<p>Python: 3.6<br>
Torch: 1.8.0<br>
Numpy: 1.18.5</p>

ptrblck: <p>Could you update to the latest PyTorch release (<code>1.9.0</code>) and check, if you are still seeing this issue?<br>
If so, could you post an executable code snippet to reproduce this error?</p>
UserWarning: An output with one or more elements was resized since it had shape

Roqyah_bdeen: <p>This part of code was working properly with old torch versions but it raises this error when I switched to torch 1.9</p>
<pre><code class="lang-auto">torch.index_select(v_flat, dim=0, index=self._i00.view(-1), out=self._v00)
        torch.index_select(v_flat, dim=0, index=self._i01.view(-1), out=self._v01)
        torch.index_select(v_flat, dim=0, index=self._i10.view(-1), out=self._v10)
        torch.index_select(v_flat, dim=0, index=self._i11.view(-1), out=self._v11)
</code></pre>
<p>it raises this warning</p>
<pre><code class="lang-auto">UserWarning: An output with one or more elements was resized since it had shape [638976, 1], which does not match the required output shape [638976, 3].This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448265233/work/aten/src/ATen/native/Resize.cpp:23.)
  torch.index_select(v_flat, dim=0, index=self._i00.view(-1), out=self._v00)

</code></pre>
<p>how could I resolve it ? thank you</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/typeerror-tuple-object-is-not-callable/55075/8">here</a> with follow up.</p>
Is the gradient produced by batch loss equal to the average gradient produced by corresponding samples

XJ_LI: <p>I wonder the two types of gradients is equal or not. I use SGD optimizer.<br>
The first gradient is that, I use a batch and calculate the average_loss, then get gradient by average_loss.backward().<br>
The second is I calculate loss of each sample and then get gradient, and average the gradients.</p>
How to visualize semantic segmentation maps

drydenwiebe: <p>Hello!</p>
<p>I have the output of a semantic segmentation network of the shape [21, 144, 256] and I would like to change it to an image of shape [3, 144, 256] to visualize it with a colour for each class.</p>
<p>The way that images are saved <a href="https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/" rel="noopener nofollow ugc">here</a> works, but I would like to have the visualized image as a tensor so I can concatenate it with the ground truth or torchvision.utils.save_image a whole batch.</p>
<p>Thank you!</p>

ptrblck: <p>Assuming you are dealing with 21 classes and the current output contains the logits, you could get the predicted class indices via <code>preds = torch.argmax(output, dim=0)</code>, where <code>output</code> would have the shape <code>[21, 144, 256]</code>. Once this is done you could then map the class indices to colors and transform the prediction to an RGB image.</p>
Pickle error when saving pandas dataframe

SU801T: <p>Hi,</p>
<p>I get this error:</p>
<p>RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device(‘cpu’) to map your storages to the CPU.</p>
<p>I’m not sure if I am saving my outputs incorrectly, however, I changed the last line in the below code from <code>df.to_csv</code> to <code>df.to_pickle</code>, in order to save the predictions in a different format. When saving into a csv file I do not get errors, however would prefer to pickle/ dump into an npy file. Any suggestions on what I should do to resolve this error?</p>
<pre><code class="lang-auto">predictions = []
imgpaths = []
probabilities = []
true_labels = [] 

with torch.no_grad():
     for i,(inputs10x, paths10x, labels) in enumerate(dataloaders_dict['val']):

            test_inputs10x = inputs10x.to(device)
            test_labels = labels.to(device)
            test_paths = paths10x

            test_x10 = densenet(test_inputs10x)

            outputs = classifier(test_x10)
            test_loss += criterion(outputs, test_labels).item()
            _, pred = torch.max(outputs, 1) #pred = torch.max(outputs, 1)[1]

            prob = F.softmax(outputs, dim=1)
            top_p, top_class = prob.topk(1, dim = 1)

            for q in top_p:
                probabilities.append(q)

            for x in pred:
                predictions.append(x)
            
            for z in test_paths:
                imgpaths.append(z)
            
            for l in test_labels:
                true_labels.append(l)


b = {'paths': imgpaths, 'predicted_labels': predictions, 'true_labels': true_labels, 'probabilities': probabilities}
df = pd.DataFrame.from_dict(b)
df.to_pickle(args.npy_file  + '.pickle')
</code></pre>

ptrblck: <p>I don’t quite understand why this error is raised in the posted code, as it’s pointing to a deserialization in a non-CUDA installation.<br>
Based on the code snippet I assume you might be using CUDA tensors? If so, could you push them to the CPU before trying to store them via <code>pandas</code>?</p>
How to assign each layer its following layer but not make the following layer as a child of current layer?

deJQK: <p>Suppose I have one model of l1 → l2 → l3 … I want to make l1 knows l2 is its following layer, and l2 know l3. If I set l1.following_layer = l2 after defining l2 or the whole model, the model will become very cumbersome because l2 will become a child of l1. Moreover, when I reset parameters during initialization, l2’s weight might also change, which might not be desired. I only want l1 know l2 is following it.</p>
<p>Thanks.</p>

ptrblck: <p>Could you explain the use case a bit?<br>
<code>.following_layer</code> isn’t a default attribute so I assume you are trying to create a manual graph somehow?</p>

deJQK: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> . I found setting the following layer into dict solves this, i.e., l1.following_layer[‘following’] = l2. I need this because I need to use weights from l2 for l1.</p>

deJQK: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> It seems that this method does not work if I would like to distribute the model on multiple GPUs, as the parameters in the module will be on different devices.</p>
<p>For example, suppose my module is something like this:</p>
<pre><code class="lang-auto">class A(nn.Module):
    def __init__(self, m):
       super(A, self).__init__()
       self.l_d = {'m': m}

    def forward(self, x):
        m = self.l_d['m']
        if m is not None:
            res = m.weight * self.weight * x
        else:
            res = self.weight * x
        return res

class M(nn.Module):
    def __init__(self):
        super(M, self).__init__()
        self.a = A(None)
        self.b = A(self.a)
</code></pre>
<p>If I use <code>m = M(); m = nn.DataParallel(m)</code>, it will report that <code>self.l_d['m'].weight</code> is not on the same device as <code>m.weight</code>, because that module is not parallelized.</p>
<p>If I use <code>self.l_d = nn.ModuleDict()</code> in the initialization of the class <code>A</code>, during <code>nn.DataParallel</code> it will report bug of <code>Recursion Error: maximum recursion depth exceeded</code>.</p>
<p>BTW the native dictionary works well for one device.</p>
Pytorch doesen't use GPU

Yamolda: <p>PyTorch does not using GPU on full power.<br>
model and tensors are in GPU and PyTorch use GPU</p>
<pre><code class="lang-auto">    print('The train is continue!')
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0) / 1024 ** 2, 1), 'MB')
    print('Cached:   ', round(torch.cuda.memory_reserved(0) / 1024 ** 2, 1), 'MB')
    double_barrier_net = double_barrier_net.double().to(get_device())

</code></pre>
<p>This is the console logs</p>
<blockquote>
<p>cuda:0<br>
The train is continue!<br>
NVIDIA GeForce RTX 2070 Super<br>
Memory Usage:<br>
Allocated: 14.5 MB<br>
Cached:    22.0 MB</p>
</blockquote>
<p>but training is going too slow and on my time_waste_test we could see that train on GPU even slowly than on CPU.<br>
link to logs</p><aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1RxSd2oLEsGytjNeudIs0hUUSIbV_sX8u/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1RxSd2oLEsGytjNeudIs0hUUSIbV_sX8u/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1RxSd2oLEsGytjNeudIs0hUUSIbV_sX8u/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1RxSd2oLEsGytjNeudIs0hUUSIbV_sX8u/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">cuda_bug.PNG</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have no idea how to fix this</p>
How to make a dataLoader with reqiures_grad=True from a dasaset and not revert reqiures_grad to false?

mohale08: <p>I am trying to create a Dataloader using the built-in DataLoader class from a dataset created using the built-in Dataset class, the problem is that the tensor in the DataLoader reverts to reqiures_grad=False. I think the DataLoader class makes a copy or something, what might be the best way of going around this?.</p>
<p>_data=torch.tensor(newData).float();<br>
lebels=torch.tensor(y_train).float();<br>
data=torch.utils.data.TensorDataset(_data,lebels)<br>
print("data shape: ", _data.shape);<br>
print("Lebels shape : ", lebels.shape);<br>
<em>data=torch.utils.data.TensorDataset(<em>data.requires_grad</em>()<br>
, lebels.requires_grad</em>());</p>
<p>_data[0][0].requires_grad <span class="hashtag">#outputs</span> true</p>
<p>trainDataLoader = DataLoader(dataset=_data, batch_size=15, shuffle=True)<br>
it = iter(trainDataLoader)<br>
first = next(it)<br>
first[0][0].requires_grad <span class="hashtag">#outputs</span> False</p>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p>I cannot reproduce the issue using:</p>
<pre><code class="lang-python">_data = torch.randn(100, 1)
lebels = torch.randn(100, 1)
data = torch.utils.data.TensorDataset(_data,lebels)
print("data shape: ", _data.shape);
print("Lebels shape : ", lebels.shape);
data = torch.utils.data.TensorDataset(_data.requires_grad_(), lebels.requires_grad_())

print(_data[0][0].requires_grad)
&gt; True

x, y = data[0]
print(x.requires_grad)
&gt; True
print(y.requires_grad)
&gt; True

trainDataLoader = torch.utils.data.DataLoader(dataset=data, batch_size=15, shuffle=True)
it = iter(trainDataLoader)
first = next(it)
first[0][0].requires_grad
&gt; True
</code></pre>
<p>In your current code snippet you are trying to “call” the <code>requires_grad</code> attribute, which is a <code>bool</code> value and which would raise an error:</p>
<pre><code class="lang-python">data=torch.utils.data.TensorDataset(data.requires_grad(), lebels.requires_grad())
</code></pre>
<p>so I changed it to the inplace ops: <code>tensor.requires_grad_()</code>.<br>
Also, you are passing the <code>_data</code> tensor (not the <code>data</code> <code>Dataset</code>) to the <code>DataLoader</code>, which you also might want to change.</p>
Multiply each 2D matrix in 3D tensor to another 3D tensor

abinashsinha330: <pre><code>    tensor_a is of shape (256, 4, 64) and tensor_b is of shape (256, 64, 1)
    res = None
    for u in range(256):
        temp = torch.matmul(tensor_a[u, :, :], tensor_b)
        temp = temp.sum().unsqueeze(-1)
        if u == 0:
            res = temp
        else:
            res = torch.cat((res, temp), -1)

    res = res.unsqueeze(-1)
</code></pre>
<p>Can anyone get the result using torch functions rather than use looping?</p>

ptrblck: <p>You could broadcast one tensor, which would use more memory for a potential performance gain:</p>
<pre><code class="lang-python">out = torch.matmul(tensor_a.unsqueeze(1), tensor_b)
out = out.sum([1, 2])
print((out - res).abs().max())
&gt; tensor(2.5580e-13, dtype=torch.float64)
</code></pre>
<p>Note that I used <code>DoubleTensor</code>s to check the error, as the numerical errors could be accumulating quite a lot.</p>
untimeError: input image (T: 1 H: 7 W: 7) smaller than kernel size (kT: 64 kH: 7 kW: 7)

Mai_Magdy: <p>input<br>
torch.Size([64, 3, 224, 224])<br>
target<br>
torch.Size([64])<br>
inputforward167<br>
after expand<br>
torch.Size([1, 64, 3, 224, 224])<br>
input185after permute<br>
torch.Size([64, 3, 1, 224, 224])</p>
<p>this is the code<br>
def forward(self, input):</p>
<pre><code>    print("inputforward167")

    print (input.shape)

    #print(input)

    #sample_len = (3 if self.modality == "RGB" else 2) * self.new_length

    if self.modality == 'RGBDiff':

        sample_len = 3 * self.new_length

        input = self._get_diff(input)

        print("input157")

        print(input.shape)

    if not self.slow_testing:#self.test_mode:

        input=input.view((-1,self.new_length,3)+ input.size()[-2:])

        print("input181")

        print(input.shape)

        input=input.permute(1,2,0,3,4).contiguous()

        print("input185")

        print(input.shape)



    else:

         input=input.view((-1,3,self.new_length)+ input.size()[-2:])

         print("inputbase")

         print(input.shape)

  #  input = input.view(input.size(0), -1)

   # print (input.shape)

    base_out = self.base_model(input)

    print("base193")

    print(base_out.shape)

    #base_out=F.dropout(base_out,p=0.5)

    if self.gtsn:

        base_out=base_out.view(base_out.shape[0]/self.num_segments,-1)

        print("base198")

        print(base_out.shape)

    if self.dropout &gt; 0:

        base_out = self.new_fc(base_out)

        print("base205")

        print(base_out.shape)

            

    if not self.gtsn:

        if self.reshape:

            base_out = base_out.view((-1, self.num_segments) + base_out.size()[1:])

            print("base213")

            print(base_out.shape)

        base_out = self.consensus(base_out).squeeze(1)

        print("base220")

        print(base_out.shape)

        if  self.slow_testing:#apply_softmax:

            base_out = self.softmax(base_out)

        print("base_out")

        print(base_out.shape)

        return base_out#.squeeze(1)

    else:

        return base_out
</code></pre>
<p>and i’m using resnet50 as base model</p>
<p>self.base_model= I3ResNet(resnet50, self.new_length,num_class,test_mode=test_mode,num_segments=self.num_segments,fast_implementation=fast_implementation)<br>
self.new_fc1 = nn.Linear(3<em>224</em>224, 2048)<br>
self.new_fc=nn.Linear(2048,num_class)</p>

ptrblck: <p>The error is raised, if an intermediate activation’s spatial size is too small for a particular conv layer, so you would either have to increase the spatial size of the input, or reduce the pooling in the model.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Torchvision ToTensor MUCH slower than manually normalizing

IsCoelacanth: <p>I’ve been seeing VERY high CPU utilization when using ToTensor in my transforms and its easily the slowest operation in my transformation compositions.<br>
system:<br>
torch: 1.8.1<br>
torchvision: 0.9.1</p>
<p>been facing the same issue since 1.4.0, even tried building from source (tried 1.6.0 and 1.8.0)</p>
<pre><code class="lang-python">def to_tensor(img): 
        img = np.array(img) 
        img = torch.from_numpy(img).float().permute(2,0,1) 
        img = img / 255.0 
        return img
</code></pre>
<pre><code class="lang-auto">In [12]: %%timeit 
    ...: to_tensor(img)                                                                                                                                                                                                                                                          
826 us +- 43.3 us per loop (mean +- std. dev. of 7 runs, 1000 loops each)
</code></pre>
<p>vs torchvision</p>
<pre><code class="lang-auto">    ...: tfunc.to_tensor(img)                                                                                                                                                                                                                                                         
1.49 ms +- 73.5 us per loop (mean +- std. dev. of 7 runs, 1000 loops each)
</code></pre>
<p>img is a 512x512 RGB PIL Image.<br>
any particular reason this may be happening?</p>

ptrblck: <p>Besides the additional checks for numpy array inputs, number of channels etc. in the <code>torchvision</code> implementations, you are not creating a contiguous output (<code>TF.to_tensor</code> does it <a href="https://github.com/pytorch/vision/blob/bf2fe567c80e61e7adee94163f04bfd346892393/torchvision/transforms/functional.py#L136">here</a>) and would also see a performance penalty due to the needed copy after the <code>permute</code> operation.<br>
You can check it via:</p>
<pre><code class="lang-python">out = to_tensor(img)
print(out.is_contiguous())
&gt; False
</code></pre>
<p>Next operations, which might require a contiguous input, would then trigger the copy and you would slow them down instead.</p>
AttributeError: module 'torch.linalg' has no attribute 'cholesky'

jks12: <p>Running the example from the pytorch website gives me this error. see: <a href="https://pytorch.org/docs/stable/generated/torch.linalg.cholesky.html#torch.linalg.cholesky" class="inline-onebox" rel="noopener nofollow ugc">torch.linalg.cholesky — PyTorch 1.9.0 documentation</a></p>
<p>I’ve checked that all packages are updated.</p>
<p>!pip install torch -U --user<br>
import torch</p>
<p>a = torch.randn(2, 2, dtype=torch.complex128)<br>
a = a @ a.t().conj()  + torch.eye(2) # creates a Hermitian positive-definite matrix<br>
l = torch.linalg.cholesky(a)<br>
a<br>
l<br>
l @ l.t().conj()</p>
<hr>
<p>AttributeError                            Traceback (most recent call last)<br>
 in <br>
1 a = torch.randn(2, 2, dtype=torch.complex128)<br>
2 a = a @ a.t().conj()  + torch.eye(2) # creates a Hermitian positive-definite matrix<br>
----&gt; 3 l = torch.linalg.cholesky(a)<br>
4 a<br>
5 l</p>
<p>AttributeError: module ‘torch.linalg’ has no attribute ‘cholesky’</p>

ptrblck: <p>I cannot reproduce the issue. Could you check the installed PyTorch version via <code>torch.__version__</code>, please?</p>
Loading a shared C/C++ library depending on libtorch immediately before importing torch

Joseph_Gardi: <p>I have a shared library named liblodepng.dylib.<br>
depending on libtorch. It is mixed C and C++ and depends on libtorch. The following code crashes python:</p>
<p>import ctypes<br>
lib_projn = ctypes.CDLL( “/build/liblodepng.dylib”)<br>
import torch</p>
<p>Changing the order like this gives me an error message:</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
lib_projn = ctypes.CDLL(<br>
…     “/build/liblodepng.dylib”<br>
… )<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/<strong>init</strong>.py”, line 374, in <strong>init</strong><br>
self._handle = _dlopen(self._name, mode)<br>
OSError: dlopen(/build/liblodepng.dylib, 6): Symbol not found: __ZNK3c1010TensorImpl20is_contiguous_customENS_12MemoryFormatE<br>
Referenced from: /usr/local/opt/libtorch/lib/libtorch_cpu.dylib<br>
Expected in: /usr/local/lib/python3.9/site-packages/torch/lib/libc10.dylib<br>
in /usr/local/opt/libtorch/lib/libtorch_cpu.dylib</p>
</blockquote>
</blockquote>
</blockquote>
How to get a NN to move a Mouse

Lamb_Sauce: <p>Hi Everyone!<br>
I was running through some ideas for a school project and wanted to create a bot for some popular games, the performance would really matter but the concept seems pretty cool. It is a PC game and we are looking at Overwatch or Valorant (other suggestions accepted;)). We just wanted to what would be the best way to record and play through (the Neural Network) the mouse movements. We plan to use RL (like the OpenAI team did for Dota II) but getting a computer to move a mouse would be an interesting task.<br>
<strong>The question is: How do we record and get a AI to move a mouse in the most efficient way possible?</strong></p>
Tensor from list of tensors

subhashnerella: <p>I have a tensor of shape 12X9X1024. I have created 9 individual fully connected layers which take 12 X 1024 shape array as input. Here 12 is the batch size and 1024 is the input dimension to the fully connected layer.</p>
<p>I planned to append the output of each FCN to a list. Then convert the list of tensors to a tensor using <code>torch.cat</code>. When the code is at <code>out.append(f(tensors[:,i,:]).view(bs,-1,2))</code><br>
I get this error <code>RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)</code></p>
<pre><code class="lang-auto">class Classifier(nn.Module):

    def __init__(self,hidden_dim):
        super().__init__()
        _,centers = config.get_inds()
        _,counts = np.unique(np.array(centers, dtype= np.int32),return_counts=True)
        self.fcn = []
        self.classes = sum(counts)
        for j in counts:     
            self.fcn.extend([nn.Linear(hidden_dim,2*j)])
    def forward(self,tensors):
        """
            tensor: n_batch X n_center X hidden_dim
        """
        bs = tensors.shape[0]
        out = []
        for i,f in enumerate(self.fcn):
            out.append(f(tensors[:,i,:]).view(bs,-1,2))
        out_tensor = torch.Tensor((bs,self.classes,2),device=tensors.device,requires_grad=True)
        torch.cat(out,dim=1,out=out_tensor)
        out_tensor = F.log_softmax(out_tensor, dim=1)
        return out_tensor
</code></pre>
<p>How to make sure the list of tensors are on GPU and has requires_grad=True?</p>

subhashnerella: <p><code>self.fcn = torch.nn.ModuleList(self.fcn)</code> fixed the issue.</p>
3D capsule neural networks: any way to use nn.Conv3d? Or should I vectorize all convolutions?

arman.avesta: <p>Hi gang,</p>
<p>I’m trying to code convolutions in a 3D capsule neural network (the 3D version of this: <a href="https://openreview.net/pdf?id=HJWLfGWRb" rel="noopener nofollow ugc">https://openreview.net/pdf?id=HJWLfGWRb</a>). I tried using nn.Conv3d, which expects the input to be in the shape: input.shape = (batch, channels, depth, height, width). In a normal convolutional network, each element of this input is a number:</p>
<p>input[batch, channel, depth, height, width] = a number.</p>
<p>In a <strong>capsule network</strong>, however, each element of this input is a 4x4 pose matrix P:</p>
<p>input[batch, channel, depth, height, width] = P = 4x4 matrix</p>
<p>Each element of the kernel is also a 4x4 matrix (e.g. a 3x3x3 kernel contains 3x3x3x4x4 numbers). At each point in a normal convolutional network, the input value is multiplied by the kernel element and then they’re added together. At each point in a capsule network, the input pose matrix (4x4 matrix) undergoes dot product with the kernel element (also a 4x4 matrix), and then results are added together to give the 4x4 pose matrix of the next capsule layer at that point (with some details omitted for brevity).</p>
<p>I could not find any way to use nn.Conv3d to convolute over 4x4 matrices (instead of simple numbers). Therefore, I’m being forced to vectorize all convolutions in my network using matrix multiplications instead of convolutions (<a href="https://arxiv.org/pdf/1501.07338.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1501.07338.pdf</a>).</p>
<p>I just wanted to make sure I’m not missing something; that there’s no way to use nn.Conv3d to achieve my goal of convoluting over 4x4 matrices! I would greatly appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Arman</p>
Load CRAFT .pth

Jack7: <p>Error loading a pretrained model, e.g. Craft .pth or TPS-ResNet-BiLSTM-Attn-case-sensitive.pth with:<br>
module = torch::jit::load(“C:…/craft_mlt_25k.pth”);</p>
<blockquote>
<p>Blockquote<br>
“<code>torch::jit::load()</code> received a file from <code>torch.save()</code>, but <code>torch::jit::load()</code> can only load files produced by <code>torch.jit.save()</code>”<br>
“Exception raised from load at …..\torch\csrc\jit\serialization\import.cpp:388 (most recent call first):<br>
… c10.dll!c10::detail::LogAPIUsageFakeReturn [ @ ]<br>
… c10.dll!c10::detail::LogAPIUsageFakeReturn [ @ &lt;unknown line …”<br>
Blockquote</p>
</blockquote>
<p>do i need to convert to caffe or onnnx? isnt .pth Pytorch? How can i load that?</p>

Jack7: <p>now i know, but to save it from python i need to know all the layers, how i know that, arent they saved in the file?</p>
For DataParallel model, during loss.backward(), where is the reduce done?

BruceDai003: <p>As documented in DataParallel(torch/nn/parallel/data_parallel.py), during the backwards pass, gradients from each replica are summed into the original module. So ASAIK, this is a reduce op, right?<br>
Now I’m trying to do some debug into the C++ source code for this loss.backward(), but can’t find the part related to this “reduce” operation, could you please tell me where is this part done?</p>

rvarm1: <p>DataParallel works via scatter/gather, so the gathering of scattered gradients is implemented in the backwards function of these operators here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/_functions.py#L52" class="inline-onebox" rel="noopener nofollow ugc">pytorch/_functions.py at master · pytorch/pytorch · GitHub</a></p>
What is the difference between torchvision.models.resnet50 and torchvision.models.__dict__['resnet50']?

nikiguo93: <p>Hi I intend to use MoCo and I have to modify to make it output results of each layer. Should I modify the forward function of Resnet50? Because the base encoder in MoCo is resnet50. However, when I modify the resnet.py and use it as the base encoder, I got error like “type error, forward function has unexpected argument 'num__classes”.<br>
In the original code of MoCo, they use base_encoder= torchvision.models.<strong>dict</strong>[‘resnet50’]to get the base encoder. However, when I modify resnet.py I got the error mentioned above. Also when I didn’t modify the original code and use base_encoder =  torchvision.models.resnet50, I got the similar error. Can somebody tell me how to figure out this problem?</p>
How do I know which spectrogram frames belong to which audio samples?

DarQ: <p>I’ve been using this script:</p>
<pre><code class="lang-auto">spgram = torchaudio.transforms.Spectrogram(512, hop_length=32)
audio = spgram(audio)
</code></pre>
<p>to get the spectrogram of some stereo music audio. I expected that the resulting spectrogram has the shape [2, 257, audio.shape[1]/32] However, that’s not the case. For examples, an audio clip with size [2, 199488] (with sr=24576) yields a spectrogram with size [2, 257, 6241] (note that 199488/32=6234). Why is that? and how can I convert from frame location to sample location?</p>
A paper reading session conflict

OuisYasser: <p>hello everyone, I started reading about adversarial attacks, and then I passed by this paper:</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/2105.00203.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/2105.00203.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/2105.00203.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/2105.00203.pdf" target="_blank" rel="noopener nofollow ugc">2105.00203.pdf</a></h3>

  <p class="filesize">17.14 MB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I understand the work done until figure number 08 where I saw that he used the layer number 27 in order to predict the saliency region !<br>
but I haven’t find the reason why choosing 27th and not another ?<br>
thank you in advance.</p>
Implementing second order derivative for grid_sample

AliaksandrSiarohin: <p>When you are doing backward of grad of grid_sample, you will get an error: <code>derivative for cudnn_grid_sampler_backward is not implemented</code>.</p>
<p>I can compute second order derivative in pure pytorch, but have no idea how to register it, so that it invoked when backward of cudnn_grid_sampler_backward is called.</p>
<p>What is the easiest way to register it?</p>
How does nn.CrossEntropyLoss() work under the hood?

zamannafi: <p>Hi there,</p>
<p>I would like to know how nn.CrossEntropyLoss() works under the hood. I implemented 2 toy examples one where there is only 1 class/target and a prediction of shape (1, 3) where 1 is the batch size and 3 is the number of classes. I followed <a href="https://www.programmersought.com/article/77987121448/" rel="noopener nofollow ugc">this</a> example which shows how to do this when dealing with the above mentioned variables. However I would like to know how the loss is computed when there are multiple targets and the prediction is in the form of say (2, 3) (batch size = 2, number of classes = 3). How is the loss calculated then?</p>
<p>Following the instructions in the link above I did this:</p>
<pre><code class="lang-auto">torch.manual_seed(0)

target = torch.randint(0, 3, (1, 1)).squeeze(0)
preds = torch.randn(1, 3)

print("target:")
print(target)

print()
print("preds:")
print(preds)

loss_func = nn.CrossEntropyLoss()
loss = loss_func(preds, target)

print()
print("loss:", loss.item())

print()
print("my loss:---------------------------.")
print(-preds + torch.log(torch.sum(torch.exp(preds))))
</code></pre>
<p>But I wasn’t able to understand how PyTorch calculates the loss with the following variables:</p>
<pre><code class="lang-auto">torch.manual_seed(0)

target = torch.randint(0, 3, (1, 2)).squeeze(0)
preds = torch.randn(2, 3)

print("target:")
print(target)

print()
print("preds:")
print(preds)

loss = loss_func(preds, target)

print()
print("loss:", loss.item())
</code></pre>
<p>Any help is appreciated thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

zamannafi: <p>OK so I figured it out. A huge shout out to <a href="https://discuss.pytorch.org/t/understanding-nllloss-function/23702/3">this</a> thread. Basically the predictions that you get from the model are run through a log softmax function. Then the results from that are used to get the values associated with corresponding class index. And then finally, those values are summed and divided by the number of classes or the shape of the target tensor.  <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>And here is the code:</p>
<pre><code class="lang-auto">preds = torch.randn(3, 5)
print("preds:")
print(preds)

targets = torch.randint(0, 5, (1, 3)).squeeze(0)
print()
print("targets:")
print(targets)

loss_func = nn.CrossEntropyLoss()
loss = loss_func(preds, targets)
print()
print("PyTorch loss:", loss)

preds_log_softmax = F.log_softmax(preds, 1)
print()
print("preds_log_softmax:")
print(preds_log_softmax)

my_loss = 0
for i in range(targets.shape[0]):
    my_loss += preds_log_softmax[i, targets[i].item()]
    
my_loss *= -1
my_loss /= targets.shape[-1] 
print()
print("my loss:", my_loss)
</code></pre>
Build Model from Submodels

blueeagle: <p>Hi everyone,<br>
i am trying to implement a model that consists of multiple encoders and one classifier.<br>
Therefore I already implemented an Encoder as a PyTorch Model (a Class that inherits from nn.Module). I now want to implement my “Main-Model”, i.e. a model that consists of multiple Encoders and a classifier. In order to achieve this, I implemented a second class that inherits from nn.Module and passed the encoders to it as member-variables. However, if I call my_model.to(device), the encoders won’t be passed to the GPU. I think there must be a better way of assembling multiple “submodels” to one big model.<br>
Can you tell me, what the best practice is for my problem?<br>
Thank you!</p>

blueeagle: <p>I think I found the solution by myself.<br>
For everyone struggling with the same problem:<br>
You can use <a href="https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html" rel="noopener nofollow ugc">ModuleList</a>. I my example, I can just append each encoder and the classifier to the ModuleList. Using this class, my “Main-Model” is aware of its submodels and for example the number of parameters is calculated correctly. I think there is a pretty good explanation of the concept <a href="https://discuss.pytorch.org/t/when-should-i-use-nn-modulelist-and-when-should-i-use-nn-sequential/5463">here</a>.</p>
Switching between batches from two different bucket iterator objects

Souro: <p>I have two below <code>BucketIterator</code> objects:<br>
<code>Iterator1</code> and <code>Iterator2</code></p>
<p>For training I have to use two Iterators (Mixing is not possible in my case)<br>
Now while training I am using the batches from these two iterators<br>
say <code>batch1</code> from <code>Iterator1</code> and <code>batch2</code> from <code>Iterator2</code> …</p>
<p>I want to use these batches(<code>batch1</code> and <code>batch2</code>) alternatively while training … But I am not able to do that … because what I learned till now about bucket iterator is … there is no <code>next()</code> and it can only iterates through <code>for(itr in iterator1)</code> …</p>
<p>That is why <strong>currently I am using in pipeline basis</strong> … first completing training with <code>iterator1</code> and then <code>iterator2</code> …</p>
<p>can you please let me know <strong>how I can use these interchangeably</strong>?<br>
Thank you…</p>
Is CTC Loss doing it's job?

Sanbot-kk: <p>Hello,<br>
So my understanding of CTC is that it instructs the model to insert a blank token whenever it has to wait for the next acceptable target.<br>
But I am not sure if the CTCLoss function in PyTorch is doing that?<br>
This is an image of the posterior probabilities at each timestep for all the classes. The bottom most target ‘_’ is my blank class.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e9569c6ec953e3060ebf96a3cad4db97407f469.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e9569c6ec953e3060ebf96a3cad4db97407f469" title="posterior"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e9569c6ec953e3060ebf96a3cad4db97407f469_2_690x266.png" alt="posterior" data-base62-sha1="4myriTFNqmDLkEZVsdYhhNsQUMF" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e9569c6ec953e3060ebf96a3cad4db97407f469_2_690x266.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e9569c6ec953e3060ebf96a3cad4db97407f469_2_1035x399.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e9569c6ec953e3060ebf96a3cad4db97407f469_2_1380x532.png 2x" data-dominant-color="BD3D61"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">posterior</span><span class="informations">5120×1977 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In my understanding, the probability for the blank class should be high more often than it is right now, in order to accept that the model is learning to use the blank token properly and CTCLoss is helping the model to learn to wait. Please tell me if this seems like CTC is working properly?</p>
How to use RECORD_FUNCTION macro

MichaelHsu170: <p>For measure performance data of a customized C++ function, should RECORD_FUNCTION macro be used or follow steps in Dispatcher::callWithDispatchKeySlowPath function?</p>
<ol>
<li>It looks like it is better to use at::shouldRunRecordFunction(&amp;pre_sampled) to determine whether to add a RecordFunction into execution to reduce overhead in non-profiling scenario. Is there any reason at::shouldRunRecordFunction(&amp;pre_sampled) is not invoked in the macro?</li>
<li>pre_sampled parameter seems to be not working in RECORD_FUNCTION. Will pre_sampled bring benefits to performance?</li>
<li>How RECORD_FUNCTION macro deal with op nesting invocation scenario?</li>
</ol>
Intermediate activation Values keep changing when I execute the hook

thesekyi: <p>I am extracting the features of an image from an intermediate network using this function</p>
<pre><code class="lang-auto">def get_layer_activation(model, name, layer, transform, path):
    activation = {}
    # model = model
    def get_activation(name):
        def hook(model, input, output):
            activation[name] = output.detach()

        return hook

    layer.register_forward_hook(get_activation(name))
    img = tensorify_img(path, transform)
    output = model(img[None, ...].float())  # .to(device)

    return activation[name].reshape(-1)
</code></pre>
<p>When I run the function above, it returns different values each time.</p>
<pre><code class="lang-auto">_test_transform = transforms.Compose(
        [
            transforms.ToTensor(),
        ]
    )
act = get_layer_activation(
    network_saved, "latent", network_saved.fc1, _test_transform, single_image)
</code></pre>
<pre><code class="lang-auto">network_saved is the loaded pretrained network from this architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 3)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)
</code></pre>
<p>Is this an expected behavior? If yes, why?</p>

ptrblck: <p>You are using an <code>nn.Dropout</code> layer in your model, which will randomly drop the activations during training, so the different outputs are expected.<br>
Call <code>model.eval()</code> and it should work (unless you are using other random operations).</p>

thesekyi: <p>Yes, I later realized. This solution works perfectly.</p>
Question about batch in enumerate(dataloader)

sfyzsr: <p>Hello, sir. I am running a multiclass classification model on pytorch by using my customize dataset. The size of my dataset is 1000, and I use 750 for training. My model can run successfully, but there will be a problem when displaying the number. I think the bug may appear here <code>for batch, (data, label) in enumerate(dataloader)</code>, I printed the batch but it is always 0.</p>
<p>Here is part of the code:</p>
<pre><code class="lang-python">
def train_loop(dataloader, model, loss_fn, optimizer):

    size = len(dataloader.dataset)

    for batch, (data, label) in enumerate(dataloader):

        data = data.to(device)

        label = label.to(device)

        # Compute prediction and loss

        output = model(data)

        label = label.squeeze(1)

        

        loss = loss_fn(output, label)

        # Backpropagation

        optimizer.zero_grad()

        loss.backward()

        optimizer.step()

        if batch % 100 == 0:

            # print(batch)

            loss, current = loss.item(), batch * len(data)

            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")

    schedulerR.step(loss)

</code></pre>
<p>Here is some output:</p>
<pre><code class="lang-auto">
dataset: 1000

Epoch 1

loss: 4.898516  [    0/  750]

Test Error: 

 Accuracy: 21.2%, Avg loss: 2.147434

Epoch 2

-------------------------------

loss: 2.177498  [    0/  750]

Test Error: 

 Accuracy: 18.8%, Avg loss: 2.567600

Epoch 3

-------------------------------

loss: 2.530193  [    0/  750]

Test Error: 

 Accuracy: 19.2%, Avg loss: 2.736725

</code></pre>
<p>I think in the normal circumstances, 0 will continue to grow. Could you tell me how can I fix it?</p>
<p>Thanks for reading.</p>

ptrblck: <p>Based on the output it seems that you are printing the <code>loss</code> only once per epoch, so when <code>batch = 0</code>.<br>
I don’t know how large the <code>batch_size</code> is, but I assume <code>batch=100</code> won’t be reached for the second output in an epoch.</p>

sfyzsr: <p>Thank you for reply. I change it to the batch &gt; 0, then it works well. Thank you so much.</p>
Performing in-place operations with autograd

Hmrishav_Bandyopadhy: <p>I have been trying to perform in-place operations on tensors and am facing an issue. When i do:</p>
<pre><code class="lang-auto">model=torchvision.models.resnet50()
k=torch.sigmoid(model(torch.rand(1,3,64,64)))
k[k&gt;=0.5]=0.9
loss=nn.BCELoss()(k,torch.rand(1000))
loss.backward()
</code></pre>
<p>I get an error that my gradients cannot be calculated. However when i do:</p>
<pre><code class="lang-auto">model=torchvision.models.resnet50()
k=(model(torch.rand(1,3,64,64)))
k[k&gt;=0.5]=0.9
k_out=torch.sigmoid(k)
loss=nn.BCELoss()(k_out,torch.rand(1000))
loss.backward() 
</code></pre>
<p>The gradient calculation goes fine. I am a bit confused by why this happens.</p>

googlebot: <p>autograd functions (sigmoid here) lock either inputs or output tensors, a bit arbitrarily (as sigmoid is invertible, one can store either tensor). Here I think that simpler derivative formula was the deciding factor to choose to store output.</p>