akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
GIL error when importing torch in file with pybind11 function

InternalInstance: <p>I wrote a script where I call pybind11 function. Everything works fine until I import torch in the same file.</p>
<pre><code class="lang-auto">import example_cpp
import torch

if __name__ == "__main__":
    example_cpp.add(1, 2)
</code></pre>
<pre><code class="lang-auto">RuntimeError: The autograd engine was called while holding the GIL. If you are using the C++ API, the autograd engine is an expensive operation that does not require the GIL to be held so you should release it with 'pybind11::gil_scoped_release no_gil;'. If you are not using the C++ API, please report a bug to the pytorch team.
</code></pre>
Selectively add along axes without using a loop

ganindu: <p>Hi,<br>
let’s say I have a Tensor <code>X</code> with dimensions <code>[batch, channels, H, W]</code><br>
then I have another tensor b that holds bias values for each channel which has dims <code>[channels,]</code></p>
<p>I want <code>y = x + b </code></p>
<p>Is there a nice way to broadcast this over H and W for each channel for each sample in the batch without using a loop.</p>
<p>If i’m convolving I know I can use the bias field in the function to achieve this, but I’m just wondering if this can be achieved just with primitive ops (not using explicit looping)</p>

InnovArul: <p>If you can make <code>b</code> to be of size <code>1 x channels x 1 x 1</code>, it is possible.<br>
You can do something like below:</p>
<pre><code class="lang-auto">y = x + b[None, :, None, None]
</code></pre>

ganindu: <p>Thanks <a class="mention" href="/u/innovarul">@InnovArul</a></p>
<p>I tried the opposite <code>y = x[None, :, None, None] + b</code> and gave up haha.</p>
<p>Basically the trick is to expand ‘b’ to match ‘x’ axis template</p>
<p>Thanks a lot for the answer</p>

InnovArul: <p>Ah ok.<br>
basically, we use <code>None</code> in the array index to introduce a singleton dimension. (similar to <code>tensor.unsqueeze(dim=N)</code> for some <code>N</code>).</p>
<p>When you use <code>x[None,:, None, None]</code>, three more singleton dimensions will be introduced in <code>x</code> and it will make the operation as an addition between the <code>7-</code>dimensional tensor and <code>1-D</code> tensor.</p>
Can there be an infinite amount of output neurones in a neural network?

Pytorch_developer: <p>Am I right into thinking that one output neurone can only equal one thing like dog or the number 7 for example. If yes, then how will I be able to have an infinite amount of output neurones in my neural network.</p>

ptrblck: <p>The meaning of the output of a neuron is defined by you (the developer) as well as the use case.<br>
In a common multi-class classification scenario, each neuron will represent the logits of the corresponding class. I.e. to predict 10 classes the last linear layer would have 10 outputs (neurons).<br>
However, you could also claim that a single output could work and see it as a regression case (I don’t think it would perform well, but in the end nothing stops you from experimenting with different approaches).</p>
<aside class="quote no-group" data-username="Pytorch_developer" data-post="1" data-topic="153926">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/3d9bf3/48.png" class="avatar"> Pytorch_developer:</div>
<blockquote>
<p>If yes, then how will I be able to have an infinite amount of output neurones in my neural network.</p>
</blockquote>
</aside>
<p>You won’t be able to use an infinite amount of neurons, as you cannot store an infinite amount of data on any system so your idea sounds like a theoretical experiment.</p>

Pytorch_developer: <p>Thank you for your help.</p>
Global contrast normalization in Pytorch

S_M: <p>Hi all, how can I implement global contrast normalization in the form of ‘transforms.Compose’? any help would be appreciated.</p>

harpone: <p>You probably want just the standard ImageNet style RMS normalization? I.e. <code>transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))</code> or similar…</p>

S_M: <p>I want to apply the global contrast normalization to the CIFAR-10 dataset. This type of normalization was investigated by “Goodfellow, Ian J, Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. arXiv preprint arXiv:1302.4389, 2013.”</p>
Would like some suggestions for C++ related web service deployment?

ettzzz: <p>Hi forum,</p>
<p>I’m basically completely new to C++. I would like to deploy a pytorch-trained model as a continuously running web service in production. It should accept http request with one or more images attached and it will respond with inferenced results. If this demand shall be fulfilled by python, I would take Django/Flask as web service framework but I would like to dive into C++ runtime environment for less inference latency.  Can anyone share me some rough ideas on what should I do to achieve the same goal? Like should I search for a similar web service framework written in C++ or should I head for another direction?</p>
<p>Any ideas would be appreciated:)</p>

ptrblck: <p>I would check some available solutions such as <a href="https://pytorch.org/serve/"><code>torchserve</code></a> or <a href="https://developer.nvidia.com/nvidia-triton-inference-server">Triton</a>.</p>

ettzzz: <p>Thank you! I will go and do some search about torchserve:)</p>
Pth to onnx : _convolution_mode error

Vishak_Raj: <p>Hi,</p>
<p>I am trying to convert .pth to .onnx file,</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        cnn = nn.Sequential()
        cnn.add_module('c1', nn.Conv2d(3, 32, 3, 1, 1))
        cnn.add_module('r1', nn.ReLU())
        .....
        .....
        .....
        cnn.add_module('l3', nn.Linear(512, 2))
        cnn.add_module('ls', nn.LogSoftmax())

        self.cnn = cnn

    def forward(self, x):
        output = self.cnn(x)
        return output

sample_batch_image = torch.zeros(64, 3, 224, 224).to(device)

torch.onnx.export(model, sample_batch_image, 'output.onnx', verbose=True, opset_version=13, export_params=True, training=torch.onnx.TrainingMode.EVAL, do_constant_folding=True, input_names = ['cnn.c1'], output_names=['cnn.ls'], dynamic_axes={'cnn.c1': {0: 'batch'}, 'cnn.ls': {0: 'batch'} ,})

</code></pre>
<p>I got this error,</p>
<blockquote>
<p>RuntimeError: Exporting the operator _convolution_mode to ONNX opset version 13 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.</p>
</blockquote>

Matt2: <p>I got the same error, did you solve it?</p>

Vishak_Raj: <p><a class="mention" href="/u/matt2">@Matt2</a> , yes, I update the pytorch version to 1.11, the issue is solved and changed the padding to integer</p>

yanw: <p>I’ve updated pytorch version to 1.11.0-cpu, but I still get the same error.</p>

Vishak_Raj: <p><a class="mention" href="/u/yanw">@yanw</a>,</p>
<p>Then, you can replace the <code>nn.Conv2d</code> with this</p>
<pre><code class="lang-auto">from functools import reduce
from operator import __add__
class Conv2dSamePadding(nn.Conv2d):
    def __init__(self,*args,**kwargs):
        super(Conv2dSamePadding, self).__init__(*args, **kwargs)
        self.zero_pad_2d = nn.ZeroPad2d(reduce(__add__,
            [(k // 2 + (k - 2 * (k // 2)) - 1, k // 2) for k in self.kernel_size[::-1]]))

    def forward(self, input):
        return  self._conv_forward(self.zero_pad_2d(input), self.weight, self.bias)
</code></pre>
<p>For example,<br>
<code>cnn.add_module('c1', Conv2dSamePadding(3, 64, 2, 1))</code></p>
<p>I used this custom Conv. layer, which works same as <code>nn.Conv2d</code> and it wont make issue when converting the model to .onnx file</p>
<p>Happy Coding</p>
[GRU] Different results between batched and one by one inference

docteur_saoul: <p>Hi,</p>
<p>I’m using Pytorch 1.7.1 on CPU and I’m getting inconsistent results during inference over the same data.<br>
It seems that the GRU implementation gives slightly different results for a sample by sample prediction vs batched prediction.</p>
<p>Here is a code to reproduce the problem:<br>
<code><br>
import torch<br>
a = torch.randn((128, 500, 4))<br>
layer = torch.nn.GRU(4, 4, num_layers=2, dropout=0.0, bidirectional=True, batch_first=True)<br>
layer.eval()<br>
batched, _ = layer(a)<br>
obo = torch.zeros(batched.shape)<br>
for i in range(128):<br>
 obo[i] = layer(a[[i]])[0][0]</code></p><code>
</code><p><code>batched == obo<br>
</code></p>
<p>I was expecting this code to return only True but instead returns a mix of True and False. Is it normal or is it a pytorch problem ?</p>
<p>Thank you for your time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>If the relative error is in the range <code>~1e-6</code> the difference is most likely due to the limited precision used in <code>float32</code> and you should get a lower error using e.g. <code>flaot64</code>.<br>
The better way to compare floating point numbers is to use <code>torch.allclose</code>, which uses a small <code>eps</code> value to compare both tensors.</p>

docteur_saoul: <p>Thank you for the answer !</p>
<p>Error seems to be in the 1e-6 range right after the RNN layers indeed.<br>
Are you aware of any way to attenuate this behaviour ? In my actual code, after a few layers, this error is more in the 1e-4 range, which causes non reproducibility problems during inference</p>

ptrblck: <p>Are you seeing non deterministic behavior using one approach and following the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a> or just if you compare the batched vs. single sample approach?</p>

docteur_saoul: <p>I have already tried changing the seed and setting pytorch in deterministic mode to no avail.<br>
I find a dirty hack: using quantization aware training seems to attenuate the negative impact of this.</p>

qingqingxu2020: <p>hello, I suffered the same problem.<br>
In my case, after 28 layers, this error is more in 1e-2 range.<br>
So, how did you finally solve it? Thank you very much.</p>
Compiling pytorch handler file to run on windows app

Aravind_Kota: <p>I am trying to compile the inference code used in handler file and run the torchserve to load the handler file along with compiled code. But I am not able to load the compiled code after creating the MAR file. Anyway to load the compiled inference file in the handler file. This is to integrate the entire training and inferencing code with a windows based application for remote serving purpose.</p>

dbish: <p>Have you followed the process here for using torchserve on windows: <a href="https://pytorch.org/serve/torchserve_on_win_native.html" class="inline-onebox" rel="noopener nofollow ugc">14. TorchServe on Windows — PyTorch/Serve master documentation</a>?</p>
Can we compile cpp torch into pytorch for debugging purpose?

Ziyu_Huang: <p>Hi! I am interested in some function in cpp torch, and I think, if I can debug cpp torch, I can understand it better! So can I change some code in cpp version, and then somehow run the code, maybe by compiling it into python?<br>
Thank you!!!</p>

ptrblck: <p>Please correct me if I’m misunderstanding your use case, but in case you would like to change code in some C++ file in the PyTorch backend you could rebuild it and should see the changes.</p>

Ziyu_Huang: <p>yes!!! That is what I want! How to build it? Sorry, I am not familiar with cpp torch install. Oh, maybe install cpp torch must need to build and then install. So I just need to change it, and then just use cpp version and I can see the difference!</p>

ptrblck: <p>Follow <a href="https://github.com/pytorch/pytorch#from-source">these instructions</a> to build PyTorch from source.</p>
Inplace Operations, huge Tensors and leaves

LeanderK: <p>I have the following problem. I have a huge tensor that is used and modified throughout the model. Something like:</p>
<pre><code class="lang-auto">Init -&gt; Modification -&gt; Modification -&gt; Modification -&gt; Modification -&gt; Modification -&gt; (...) -&gt; Loss
                     |                                                |            
                     -&gt; Loss                                          -&gt; Loss
</code></pre>
<p>(I hope the ascii-art translates)</p>
<p>The Tensors are quite huge, multiple gigabytes, so I really can’t keep more than a few copies around. I run into problems quickly.</p>
<p>I have ideas how to implement the modifications via inplace operations, but due to the diverging computations I can’t always express this. But since the nodes where the graph splits are directly going into the loss, I wonder if I can take these computations and directly compute the gradient and therefore eliminate the path from the autograd engine? As I understood it, the problem for inplace-operations is the autograd engine.</p>
Retraining pytorch model (augmented learning)

norhther: <p>I have the following code:</p>
<pre><code>import torch
from facenet_pytorch import InceptionResnetV1, MTCNN
from torch.utils.data import DataLoader
from torchvision import datasets
import numpy as np
import pandas as pd
import os


workers = 0 if os.name == 'nt' else 4
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('Running on device: {}'.format(device))

mtcnn = MTCNN(
    image_size=160, margin=0, min_face_size=20,
    thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True,
    device=device
)

def collate_fn(x):
    return x[0]

dataset = datasets.ImageFolder('data/images/')
dataset.idx_to_class = {i:c for c, i in dataset.class_to_idx.items()}
loader = DataLoader(dataset, collate_fn=collate_fn, num_workers=workers)
#print(dataset.idx_to_class)

aligned = []
names = []
i = 0
for x, y in loader:
    x_aligned, prob = mtcnn(x, return_prob=True)
    if x_aligned is not None:
        print('Face detected with probability: {:8f}'.format(prob))
        aligned.append(x_aligned)
        names.append(dataset.idx_to_class[y])
        i += 1
#print(i)

for name, param in mtcnn.named_parameters(): #Freezing everything but last layer
    #print(name)
    if name != "onet.dense6_3.bias":
        param.require_grad = False
    else:
        param.require_grad = True
</code></pre>
<p>And now I would like to retrain this model to predict three classes (Now it only predicts the probability of a face). Let say that I have inside <code>data/images/</code> three folders, <code>faces1</code>, <code>faces2</code> and <code>faces3</code>. How could I retrain this model with these three folders? I would like to have a tensor like <code>[prob1, prob2, prob3]</code> with the probability of an image for each class. Thanks.</p>
Question: building from source with CUDA, requires cuda 10.2 or above?

tk0320: <p>i have read readme in torch repo, and found that:</p>
<pre><code class="lang-auto">If you want to compile with CUDA support, install the following (note that CUDA is not supported on macOS)

* [NVIDIA CUDA](https://developer.nvidia.com/cuda-downloads) 10.2 or above
* [NVIDIA cuDNN](https://developer.nvidia.com/cudnn) v7 or above
* [Compiler](https://gist.github.com/ax3l/9489132) compatible with CUDA
</code></pre>
<p>**is that means cuda 11 or later not supported? **</p>

AlphaBetaGamma96: <p>You’re building from source on macOS, macOS doesn’t support CUDA.</p>

ptrblck: <aside class="quote no-group" data-username="tk0320" data-post="1" data-topic="153891">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tk0320/48/14342_2.png" class="avatar"> tk0320:</div>
<blockquote>
<p>**is that means cuda 11 or later not supported? **</p>
</blockquote>
</aside>
<p>No, to build PyTorch from source with CUDA support you need at least CUDA 10.2 or a newer release as given in:</p>
<blockquote>
<p>10.2 or above</p>
</blockquote>
<p>(Note the <strong>above</strong> phrase)</p>

tk0320: <p>It seems that your documents generally use <code>later</code> more often. excuse my poor english and thanks for help</p>

tk0320: <p>macos is my local env thanks</p>
How to translate this pytorch code to c++(libtorch)?

cjpurackal: <p>I’m unable to figure out how to overide these functions in c++. What’s the right way to translate this?</p>
<pre><code class="lang-auto">class DenseLayer(nn.Linear):
    def __init__(self, in_dim: int, out_dim: int, activation: str = "relu", *args, **kwargs) -&gt; None:
        self.activation = activation
        super().__init__(in_dim, out_dim, *args, **kwargs)

    def reset_parameters(self) -&gt; None:
        torch.nn.init.xavier_uniform_(
            self.weight, gain=torch.nn.init.calculate_gain(self.activation))
        if self.bias is not None:
            torch.nn.init.zeros_(self.bias)
</code></pre>

AidanShipperley: <p>If you’re just looking for a function to init an inputted linear layer with custom weight and bias initialization methods, you can do so like this with a function:</p>
<pre><code class="lang-auto">torch::nn::Linear reset_parameters(torch::nn::Linear layer) {

    torch::NoGradGuard noGrad;
    torch::nn::init::xavier_uniform_(layer-&gt;weight, torch::nn::init::calculate_gain(torch::enumtype::kReLU));
    torch::nn::init::zeros_(layer-&gt;bias);

    return layer;

}
</code></pre>
<p>You can see documentation for libtorch’s <code>torch::nn::init</code> functions here:<a href="https://pytorch.org/cppdocs/api/namespace_torch__nn__init.html" rel="noopener nofollow ugc">Torch::nn::Init Documentation</a></p>
<p>I use <code>torch::NoGradGuard noGrad</code> to initialize the layer’s weights and biases, the C++ equivalent to Pytorch’s <code>with torch.no_grad():</code>, as according to this documentation: <a href="https://pytorch.org/docs/stable/notes/autograd.html#no-grad-mode" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.11.0 documentation</a>, The implementations in <a href="https://pytorch.org/docs/stable/nn.init.html#nn-init-doc" rel="noopener nofollow ugc">torch.nn.init</a> also rely on no-grad mode when initializing the parameters as to avoid autograd tracking when updating the initialized parameters in-place. You can remove that line if you know there’s no need for it.</p>
<p>Hopefully, this helps.</p>
Deterministic behavior using bilinear2d

Hatem: <p>I get the following error</p>
<p>RuntimeError: upsample_bilinear2d_backward_out_cuda does not have a deterministic implementation</p>
<p>Is there a workaround?</p>

ptrblck: <p>The workaround would be to use the CPU implementation, i.e. push the data to the CPU before applying this method and back to the GPU afterwards for now assuming that you need the deterministic outputs for debugging (as it would slow down your code).</p>

heitorrapela: <p>Thanks! Nice solution, I just run both model and input on cpu to test if everything was reproducible, so after run twice on cpu and have the same result. Now, I will move back to gpu <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Good to hear this workaround seems to work fine. However, note that deterministic methods might have been already implemented in the last months for more methods, so you could install the current nightly release and check if that’s the case for your operations.</p>

heitorrapela: <p>I changed the F.interpolate of the models that I am using, following this issue (<a href="https://github.com/pytorch/pytorch/issues/7068#issuecomment-716719820" class="inline-onebox" rel="noopener nofollow ugc">can't reproduce results even set all random seeds · Issue #7068 · pytorch/pytorch · GitHub</a>), and it seems to be working on gpu <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
A question about splitting training dataset when do training

602389789: <p>Suppose I have training dataset with the size of 50 million, I would like to train it with a batch_size and epoch.</p>
<p>If I separate the dataset into 5 subset with 10 million each one. and I train my model like this: first I train my model with subset_1 under the same batch_size and epoch as I mentioned before, and get Model_1. then I train Model_1 with subset_2 under the same batch_size and epoch, repeat this loop until I get the final Model.</p>
<p>Will this influence the precision of my final Model compared to training on the whole dataset at each epoch with the same batch_size.</p>

ptrblck: <p>Your workflow of splitting the data into non-overlapping parts and then training the model on each part in a sequential manner would be equal to training the model on the entire dataset.<br>
Of course you can’t expect to see bitwise-identical values as the pseudorandom number generator calls would be different, but besides that there shouldn’t be any difference (assuming the data is properly shuffled and the same transformations are used).</p>
Input shape of pretrained 3D resnets

SungmanHong: <p>I’m using pretrained pytorch video classification model, as,<br>
<a href="https://pytorch.org/vision/stable/models.html#video-classification" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/models.html#video-classification</a></p>
<p>I loaded r3d_18 model, but I can’t found the dimension of inputs.<br>
It requires 5 inputs so I tried [batch X frames X filters(R,G,B) X W X H] but not worked… And the docs do not shows the input shape.</p>
<p>Anybody knows the input shape of r3d_18?(Resnet 3d - 18 layers)</p>

rusifan: <p>I was having the same issue. The error shows that its [batch x channels x frames x w x h].</p>
len of Predicted bounding boxes are not equal to len of true bounding boxes

sajad_Zarei: <p>I am trying to to evaluate a model that I have already trained for two classes in pytorch. I have the checkpoint and extracted its values. However when I try to predict, the length of predicted bounding boxes are more than the length true bounding boxes. I have tried different ways but they were not helpful.</p>
<p>Here is my code,</p>
<pre><code class="lang-auto">checkpoint = torch.load(checkpoint_path,map_location=torch.device('cpu'))
model = torchvision.models.detection.retinanet_resnet50_fpn(
        pretrained=False, num_classes=2
        ).to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']
</code></pre>
<p>after loading the data, I have</p>
<pre><code class="lang-auto">with torch.no_grad():
    for i,(images, targets) in enumerate(tqdm(test_loader)):
        if i ==0: # try the first iteration just for test

            outputs = model(images,targets) # get the predictions on the image
            #print(outputs[0].keys())
             
            # predictions
            scores = outputs[0]['scores'].cpu().numpy()
            pboxes = outputs[0]['boxes'].cpu().numpy()
            plabels = outputs[0]['labels'].cpu().numpy()
            
            # true BB and labels
            oboxes = targets[0]['boxes'].cpu().numpy()
            olabels = targets[0]['labels'].cpu().numpy()
            

            print(' len oboxes--&gt;',len(oboxes)) #56
            print(' len olabels--&gt;',len(olabels)) #56
            

            print(' len scores--&gt;',len(scores)) #231
            print(' len pboxes--&gt;',len(pboxes))#231
            print(' len plabels--&gt;',len(plabels))#231
</code></pre>
<p>predictions and true values dont have same length. Any help is appreciated.</p>
Is needed to use Image.Antialias in Resize?

Miguel_Campos: <p>Hello guys.</p>
<p>In a transformation of a Pil Image (1200x1200) to a Tensor like this.</p>
<pre><code class="lang-auto">T.Compose([T.Resize((224, 224), Image.ANTIALIAS), 
                    T.ToTensor(),
                    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
</code></pre>
<p>has <strong>Image.ANTIALIAS</strong> sense ? as far as I know, antialias flag (in resize) is <strong>True</strong> by default if the Image is PIL</p>
<p>I usually use something like InterpolationMode.BILINEAR or InterpolationMode.BICUBIC lit this:</p>
<pre><code class="lang-auto">T.Compose([T.Resize((224, 224), InterpolationMode.BILINEAR), 
                    T.ToTensor(),
                    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
</code></pre>
<p>but sometimes, I see people using Image.ANTIALIAS and that confuses me.</p>
<p>Is there any reason for using Image.ANTIALIAS?.</p>
<p>Thanks</p>
How to read memory like pytorch? Even after transpose?

Ziyu_Huang: <p>Hi! I find out pytorch can read memory differently than cuda, like:<br>
a = torch.tensor([1, 2, 3])<br>
b = a.T</p>
<p>If we use this b, it is really transposed. But if we read it in another self defined cuda kernel, it will still read it like a. in a’s shape!</p>
<p>How can I package my kernel like pytorch? Sounds like, like aten? Any suggestions? Thank you!!!</p>

ptrblck: <p>I’m not sure what the code snippet should do since <code>a</code> is a 1-dim tensor with a size of 3 and <code>a.T</code> won’t change anything. Reading the memory of <code>a</code> and <code>b</code> should thus yield the same values.</p>
<aside class="quote no-group" data-username="Ziyu_Huang" data-post="1" data-topic="153739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/48/14158_2.png" class="avatar"> Ziyu_Huang:</div>
<blockquote>
<p>I find out pytorch can read memory differently than cuda</p>
</blockquote>
</aside>
<p>Could you also explain this sentence a bit more? PyTorch uses C++ and CUDA in its backend, so I’m not sure how else PyTorch “can read memory”.</p>

Ziyu_Huang: <p>Oh, yes. Thank you for your reply! My question is more precisely decribed here: <a href="https://discuss.pytorch.org/t/how-does-nn-linear-work-in-cpp-for-multi-dimension-input-torch-c-nn-linear/153735" class="inline-onebox">How does nn.linear work in cpp for multi dimension input? (torch._C._nn.linear)</a><br>
If you could answer my further question, it will be highly appreciated!!!</p>

Karthik_Ganesan: <p>I think what you mean is when you do <code>a.t()</code> and then read that memory in your CUDA kernel, a is not actually transposed? I believe this is because <code>a.t()</code> does not actually tranpose the data in memory but it just returns a view of the data. But in your CUDA kernel, when your read <code>b</code>, you are still reading <code>a</code> not the transposed version of <code>a</code>.</p>
<p>I had the same thing when I was trying to implement a fully connected layer in CUDA. I provided a link to my implementation as a reply to your other thread <a href="https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/6">here</a>. What I did was to flatten the array before I called the CUDA kernel so that <code>b</code> would be contiguous and correctly transposed in the CUDA kernel.</p>
CosineAnnealingWarmRestarts t_0

spyroot: <p>Hi Folks,</p>
<p>I just confirmed my understanding related to T_0 argument.<br>
Let’s say I have 500 epochs and data loader len 97</p>
<pre><code class="lang-auto">loader_data_size = 97
for epoch in epochs:
   self.state.epoch  = epoch # in my case it different place so I track epoch in state.
    for batch_idx, batch in enumerate(self._train_loader):    
        # I took same calculation from example. 
        next_step = self.state.epoch + batch_idx / loader_data_size
        scheduler.step(next_step)
</code></pre>
<p>It will anneal every batch at the end of the batch ( since I set T_0 = 97, it will restart back) if I understand semantics. If that is the case, it implies that every example in batch ( the last example will always get the lowest LR). If my understanding is correct, do I need to shift the cosine function somehow and slide it to fix the behavior? or it will slide by itself. i.e for epoch 0 lower LR at batch_idx = 96 at epoch=1 at batch_idx = 96 LR going to be the same at in epoch=0 or it will slide ?</p>
<p>Thank you,<br>
Mus</p>
How to make sense of Pytorch linear layer shape?

Archit_Srivastava: <p>I am training FFNN for MNIST with a batch size of 32. My first linear layer has 100 neurons, defined as nn.linear(784,100). When I check the shape of the layer using model[0].weight.shape, I get [100,784]. My input is of the shape [32,784]. It was my understanding that there are matrix multiplication Weights with the input, however, I cannot see how to do that between the weight tensor of shape [100,784] and input tensor of shape [32,784]. Shouldn’t the weight tensor be [784,100]? Or does PyTorch transpose the tensor before multiplication?</p>
<p>Thanks!!</p>

ptrblck: <aside class="quote no-group" data-username="Archit_Srivastava" data-post="1" data-topic="153796">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/archit_srivastava/48/50428_2.png" class="avatar"> Archit_Srivastava:</div>
<blockquote>
<p>Or does PyTorch transpose the tensor before multiplication?</p>
</blockquote>
</aside>
<p>Yes, the <code>weight</code> is transposed as seen e.g. <a href="https://github.com/pytorch/pytorch/blob/4a15f4a902c5640f3be4d18027db4316dc11d6d9/aten/src/ATen/native/Linear.cpp#L33-L40">here</a>.</p>

Archit_Srivastava: <p>Thanks!!<br>
I have a follow-up question<br>
The gradients of the parameters of the weights of for the above-mentioned layer<br>
model[0].weight.grad, after J.backward() has been called is of the shape [100,784].<br>
The shape of model[0].weight.grad is the transpose of the shape of the parameters of the layer that is used during forward propagation- model[0].weight<br>
I am using SGD optimizer with just the learning rate hyperparameter to update the parameters. Am I correct to assume that the weights for the layer are updated as follows?<br>
model[0].weight+=(learning_rate)model[0].weight.grad.<br>
And this works out because PyTorch stores model[0].weight in the same shape as model[0].weight.grad?</p>

ptrblck: <p>Yes, the parameters and corresponding gradients are stored in the same shape and layout and thus can be subtracted from each other without any manipulations/transposes etc.</p>

Archit_Srivastava: <p>That makes so much sense!! Thank you so much!!!</p>
How to load large dataset using torchtext?

dipta007: <p>I couldn’t find any efficient or conventional way to use <code>torchtext.data.Dataset</code> with large dataset. We can use the large file automatically in case of the original <code>Dataset</code> from pytorch. But in the case of <code>torchtext</code> , it’s not that simple.</p>
<p>Is there any way we can use <code>Generator</code> with <code>torchtext</code>?<br>
Or is there any other way?</p>
Assign color to scalars when logging to tensorboard logger

pumplerod: <p>I’ve recently begun to convert my models over to pytorch-lightning and am trying to take advantage of the logger (default: tensorboard).</p>
<p>The logger seems to randomly assign colors to the scalars for every run which becomes awful messy when comparing various metrics and runs.  I can go through and set them by hand after they show up in tensorboard, but I would really like to assign them at the time I’m logging the value.</p>
<p>I would like to be able to assign all train metrics one color and all validation metrics a different color, for example.  So it’s clear from chart to chart which one I’m looking at.  Then I could alter those for different model runs so that even with several runs showing in one chart I could clearly see each model’s train/validation metric curves.</p>
<p>is there a way to do this?  I can’t seem to find any indication in the documents.</p>
How to implement a custom layer

Fabrice_Auzanneau: <p>Hi<br>
I’d like to test a custom layer. I wrote the init and forward methods, but when I test it on a MNIST testcase, the training loss doesn’t change. What did I miss? Should I also write a backward method ?<br>
Is there any good tutorial with examples somewhere ?</p>
<p>Thanks for your help</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/examples_nn/polynomial_module.html">This tutorial</a> might be a good starter. Generally you don’t need to write the <code>backward</code> method if you stick to differentiable PyTorch operations.</p>

Fabrice_Auzanneau: <p>Thanks a lot, I have seen this tutorial, but didn’t understand it.<br>
If my loss doesn’t change on 20 epochs, I guess it means that my gradients are zero or near zero. So is this a symptom saying that I should write a backward method ?</p>

ptrblck: <p>Not necessarily and you would have to debug the issue first.<br>
Check if the <code>.grad</code> attributes of used parameters are showing a valid gradient after the first <code>backward()</code> call or if they show <code>None</code> without calling <code>optimizer.zero_grad()</code>.<br>
In the first case, the gradients are properly calculated but might be small, in the latter case the computation graph might be detached and you would need to check why that’s the case (e.g. by re-wrapping a tensor, calling <code>.detach()</code> on it, using a 3rd party library etc.).</p>

Fabrice_Auzanneau: <p>I made it again from scratch, and now it works perfectly. Don’t know why…<br>
Thanks for your help anyways.</p>
Cannot construct tensor directly on GPU in torch 1.10.1

Fst_Msn: <p>torch version == 1.10.1</p>
<p>Given:</p>
<pre><code class="lang-auto">device = 'cuda:0'
</code></pre>
<p>If I try:</p>
<pre><code class="lang-auto">x = torch.Tensor([1.0], device=device)
</code></pre>
<p>I got the following error:</p>
<blockquote>
<p>RuntimeError: legacy constructor expects device type: cpu but device type: cuda was passed</p>
</blockquote>
<p>What am I doing wrong ?</p>
<p>I solved this by doing:</p>
<pre><code class="lang-auto">x = torch.Tensor([1.0]).to(device)
</code></pre>
<p>which works, but I would like to know why I can directly initialize the Tensor on GPU device.</p>
<p>Additional Information:</p>
<p>This error occurs only with torch.Tensor<br>
The following works:</p>
<pre><code class="lang-auto">x = torch.zeros(1, device=device)
y = torch.ones(1, device=device)
</code></pre>

ptrblck: <p>Use <code>torch.tensor([1.0], device='cuda')</code> (lowercase <code>t</code>) and it should work.<br>
I would not recommend to use the legacy <code>torch.Tensor</code>, <code>torch.FloatTensor</code> etc. constructors.</p>

Fst_Msn: <p>Thank you!</p>
<p>Can you also please explain me what is the difference between the two ? Or post a link to the explanation ?</p>

ptrblck: <p>The main difference is that <code>torch.tensor</code> is the supported and right way and <code>torch.Tensor</code> the legacy and unsupported.<br>
The <a href="https://pytorch.org/docs/stable/generated/torch.tensor.html#torch.tensor">docs for <code>torch.tensor</code></a> show all input arguments while the <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor">docs for <code>torch.Tensor</code></a> point to ways to initialize tensors.</p>
Where is the nn.Linear cuda implementation

cheng_chen2: <p>I cannot find the cuda implementation of nn.Linear, for forward implementation and backward implementation. Can anyone helps me with this?</p>

ptrblck: <p><code>nn.Linear</code> is using <code>F.linear</code>, which will call into <code>addmm</code> or <code>matmul</code> <a href="https://github.com/pytorch/pytorch/blob/4a15f4a902c5640f3be4d18027db4316dc11d6d9/aten/src/ATen/native/Linear.cpp#L33-L40">here</a> and then deploy to the cublas method <a href="https://github.com/pytorch/pytorch/blob/c1c9be16c4d0648fc134d04f30c8463575df7ada/aten/src/ATen/native/cuda/Blas.cpp">here</a>.</p>

cheng_chen2: <p>Thanks for the answer. And btw, what is the backward function/implementation for the F.linear? I see many Ops defined their related backward functions in  native_functions.yaml file, but for thoes not defined , how do we check their backward implmenetation? Is there a common way?</p>

ptrblck: <p>The backward methods of the aforementioned functions are defined in the <code>derivatives.yaml</code> as seen <a href="https://github.com/pytorch/pytorch/blob/d431c77d76b8c92dfddfdfec2ce29b2e46c441c2/tools/autograd/derivatives.yaml#L204-L226">here</a>.</p>

Ziyu_Huang: <p>Thank you very much for your answer!!! But I still have some small questions:<br>
in <a href="https://github.com/pytorch/pytorch/blob/c1c9be16c4d0648fc134d04f30c8463575df7ada/aten/src/ATen/native/cuda/Blas.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/c1c9be16c4d0648fc134d04f30c8463575df7ada/aten/src/ATen/native/cuda/Blas.cpp</a>  I can not find <strong>function named addmm, or matmul</strong>, only can find something like, addmm_out_cuda_impl. I know they should be, close, but how they are related together?</p>
<p>Given that I am implementing linear layer myself, I also wonder, <strong>how can I use</strong> this in my code, which is, can I really include and run them?</p>
<p>Thank you!!!</p>

Karthik_Ganesan: <p>Hi <a class="mention" href="/u/ziyu_huang">@Ziyu_Huang</a> , I had the same issue so I wrote code to implement the Conv2D and Linear layers using native CUDA code <a href="https://github.com/KarthikGanesan88/pytorch-manual-layers" rel="noopener nofollow ugc">here</a>. Hopefully it will be useful for you! I only wrote code for the forward pass in CUDA but hopefully you can use that as a base to write code for backward if you need that.</p>
Writing an autograd.Function which uses Pytorch and custom backward functionality simultaneously

Nerrror: <p>Hello,</p>
<p>I’m trying to implement a new Pytorch autograd function which should use a custom-derived gradient and combine it with a gradient obtained by the autograd engine. In essence I would like to do the following:</p>
<pre><code class="lang-auto">class some_funny_function(torch.autograd.Function):

     @staticmethod
     def forward(ctx, input):
          output1 = some_custom_function(input)
          input = torch.tensor(input, requires_grad=True)
          output2 = some_pytorch_function(input)
          ctx.save_for_backward(input, output1, output2)
          return output1, output2

     @staticmethod
     def backward(ctx, grad_input1, grad_input2):
          input, output1, output2 = ctx.saved_tensors()
          grad_output1 = custom_gradient(output1, grad_input1)
          output2.backward(grad_input2)
          grad_output2 = input.grad
          return grad_output1 + grad_output2
</code></pre>
<p>I have some input that is processed by two different functions, one I can directly write with PyTorch functions. The other can’t be done with PyTorch because it involves calling third-party software. Fortunately, I have an analytic expression for this gradient which I would like to seamlessly integrate into a PyTorch module.</p>
<p>Since the computation that I abbreviated ba some_pytorch_function is rather lengthy and the gradient complicated to derive, I would much rather use the autograd engine to do the hard work for me.<br>
Unfortunately, ctx seems not to be able to do the trick for me since requires_grad seems to be disabled in a custom autograd.Function.</p>
<p>Do you have any suggestions on how to combine autograd and manual gradients in a nice way? Any help would be much appreciated!</p>
<p>Thanks</p>
<p>Nerrror</p>
How to improve inference timing?

iamexperimentingnow: <p>Hi Everyone,</p>
<p>Can anyone please recommend to me what is an efficient way to reduce the inference timing? Currently, my model takes 50sec to do the inference.</p>
<p>Objective: I have built the flask app and it takes 50sec to do the inference. So, I would like to reduce the inference timing and return the result from the flask app.</p>
<p>Steps were already taken:</p>
<ol>
<li>attached model to GPU</li>
</ol>
<p>Note:<br>
Framework: PyTorch<br>
Application: Object detection</p>
<p>It would be really helpful if anyone recommends an efficient way to optimize.</p>

Karthik_Ganesan: <p>There’s very little info here that anyone can help you with. What are you system configs? What GPU are you using? Is it for batched inference? or for a single input at a time? What is the flask app?</p>
Accessing weights and bias values of a pretrained and quantized model

nivedita_shrivastava: <p>How can we access all weights and biases for a pretrained and quantized model of a neural network?<br>
I have downloaded the model using the following command:</p>
<p>model = models.quantization.resnet18(pretrained=True, quantize=True)<br>
for param_tensor in model.state_dict():<br>
print(’ For param_tensor is ',param_tensor)<br>
if model.state_dict()[param_tensor].dtype==torch.qint8:<br>
double_x=(torch.int_repr(model.state_dict()[param_tensor]).numpy())<br>
else:<br>
double_x=(model.state_dict()[param_tensor]).detach().numpy()</p>
<p>This leads to the following error:<br>
For param_tensor is  fc._packed_params.dtype<br>
Traceback (most recent call last):<br>
File “compression_quantize_test.py”, line 63, in <br>
if model.state_dict()[param_tensor].dtype==torch.qint8:<br>
AttributeError: ‘torch.dtype’ object has no attribute ‘dtype’</p>
<p>Can anyone let me know the correct way to access all the weights and biases?</p>

Karthik_Ganesan: <p>You are seeing this because the quantized model saves the weights for fully connected layers using this odd <code>_packed_params</code> object. That is what is actually stored in the state_dict.</p>
<p>Try to just print the objects in the state_dict to see. And then check what is in the <code>_packed_params</code> object. That way you’ll know how to access weights for FC layers.</p>
[Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-1318227.write-test'

jiapei100: <p>Hi, morning everyone:</p>
<p>I’m trying to build <strong>PyTorch from souce</strong> again, but got the <strong>ERROR</strong> message as titled:</p>
<pre><code class="lang-auto">running develop
~/.local/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
~/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-1318227.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/local/lib/python3.10/dist-packages/
</code></pre>
<p>My ENV (Ubuntu 22.04 + Cuda 11.6.2)</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ uname -r
5.15.0-35-generic
➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy
➜  ~ python --version
Python 3.10.4
➜  ~ gcc --version
gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</code></pre>
<p>My questions are:</p>
<ul>
<li>What is the right way to build PyTorch with the <strong>pip</strong> command ?</li>
<li>And, how can I specify to just <strong>build</strong> but <strong>NOT install</strong> ?</li>
<li>Finally, how can I specify the default installation path (–target ??? ) to <code>~</code> with permission, instead of <code>/usr/local</code> without permission?</li>
</ul>
<p>Thank you</p>
Unknown builtin op: torch_cluster::fps

YoPit: <p>Hello,</p>
<p>I’d like to load a TorchScript model trained with Python and which uses 4 libs: <code>pytorch-geometric</code>, <code>pytorch-scatter</code>, <code>pytorch-sparse</code> and <code>pytorch-cluster</code>. The export part is ok and now I want to load the .pt model on C++.  The problem is that some operators are unknown. I’ve managed to install C++ APIs of pytorch-scatter, pytorch-sparse and pytorch-cluster and I want to link these libs to my C++ project. I succeed for pytorch-scatter, pytorch-sparse and naively though that I should proceed similarly for pytorch-cluster. But this doesn’t work and I don’t know how to proceed.</p>
<p>Here are the CMakeLists.txt and my application code.</p>
<h3>
<a name="cmakeliststxt-1" class="anchor" href="#cmakeliststxt-1"></a>CMakeLists.txt</h3>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)

find_package(TorchSparse REQUIRED)
find_package(TorchScatter REQUIRED)
find_package(TorchCluster REQUIRED)

add_executable(example-app example-app.cpp)

target_compile_features(example-app PUBLIC cxx_range_for)
target_link_libraries(example-app TorchSparse::TorchSparse)
target_link_libraries(example-app TorchScatter::TorchScatter)
target_link_libraries(example-app TorchCluster::TorchCluster)

set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

</code></pre>
<h3>
<a name="example-appcpp-2" class="anchor" href="#example-appcpp-2"></a>example-app.cpp</h3>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.
#include &lt;torchsparse/sparse.h&gt;
#include &lt;torchscatter/scatter.h&gt;
#include &lt;torchcluster/cluster.h&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }


  torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error&amp; e) {
      std::cerr &lt;&lt; e.what();
    std::cerr &lt;&lt; "error loading the model\n";
    return -1;
  }

  std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>When I execute the application I got the following error:</p>
<pre><code class="lang-auto">Unknown builtin op: torch_cluster::fps.
Could not find any similar ops to torch_cluster::fps. This op may not exist or may not be currently supported in TorchScript.
</code></pre>
<p>Do you have any ideas about how I can resolve my problem please?</p>
Change `inspect` Signature of Modules

rambo: <p>Consider a nn.Module-derived class <code>A</code>.<br>
Then using <code>signature</code> from the Python <code>inspect</code> module</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(inspect.signature(A))
(*input, **kwargs)
</code></pre>
<p>appropriately produces the signature of the the <code>forward</code> method of <code>torch.nn.Module</code>.<br>
Is it possible to change this behavior to more accurately reflect the signature of the callable <code>A</code>?<br>
E.g., if the <code>forward</code>-method of <code>A</code> reads</p>
<pre><code class="lang-auto">def forward(a, b, c)
    ...
</code></pre>
<p>is it somehow possible to get</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(inspect.signature(A))
(a, b, c)
</code></pre>
<p>?</p>
<p>Thanks!</p>