akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using Adam optimizer for custom Model/Variables?

soulslicer: <p>Have the following model as follows:</p>
<pre><code class="lang-auto">import torch.nn as nn
class DefaultModel(nn.Module):
    def __init__(self, guess, K):
        super(DefaultModel, self).__init__()

        #guess = [-0.7541, -0.044,  0.0916,  1.5914, -0.0017,  1.4991]
        self.T = torch.tensor(guess).unsqueeze(0)
        self.T.requires_grad = True

        self.K = K

    def forward(self, datapoints, draw):
        Mat = pose_vec2mat(self.T).squeeze(0)
        loss = stuff(datapoints)
        return loss
</code></pre>
<p>Normally with hand coded gradient descent I would do something like this:</p>
<pre><code class="lang-auto">model = DefaultModel(guess, K)
gamma = torch.tensor([5e-5, 5e-5, 5e-5, 2e-5, 2e-5, 2e-5])
for i in range(0, 10000):
    
    loss = model(datapoints, draw=[1, 0,5,6])
    loss.backward()

    with torch.no_grad():
        model.T = model.T - gamma * model.T.grad
    model.T.requires_grad = True
</code></pre>
<p>However, if I want to do this using Adam Optimizer:</p>
<pre><code class="lang-auto">model = DefaultModel(guess, K)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
</code></pre>
<p>It crashes with <code>ValueError: optimizer got an empty parameter list</code> How can I get my variable T which requires grad in the parameter list?</p>

ptrblck: <p>You would have to register <code>self.T</code> as an <code>nn.Parameter</code>:</p>
<pre><code class="lang-python">self.T = nn.Parameter(torch.tensor(guess).unsqueeze(0))
</code></pre>
<p>which makes sure it’s returned in <code>model.parameters()</code>.</p>

soulslicer: <p>how do i assign a value to self.T</p>
<p>eg. This doesnt work</p>
<p>self.T = self.T - self.T</p>

ptrblck: <p>Manipulating an <code>nn.Parameter</code> creates a non-leaf tensor as seen here:</p>
<pre><code class="lang-python">T = nn.Parameter(torch.randn(1))
print(T.is_leaf)
&gt; True
T = T - T
print(T.is_leaf)
&gt; False
</code></pre>
<p>so you would either have to manipulate it inplace (e.g. via <code>T.copy_()</code>) or create a new <code>nn.Parameter()</code> (but make sure to add it to the <code>optimizer</code> afterwards, if necessary).</p>
Folder of images to a tensor

Gabriella_m: <p>Hi, I made algorithm that loads images from a folder as numpy arrays or PIL images. Now I am not sure how to fit these images to a tensor of a shape</p>
<pre><code class="lang-auto">torch.Size([64, 1, 28, 28])
</code></pre>
<p>by the way they are MNIST images, I want to make my own loader</p>
<p>Right now i have my 64 images as numpy arrays forms. In array.</p>
<p>How to fit them into torch.Size([64, 1, 28, 28]) ?</p>
<p>EDIT: For clear info, i created normal python array and every image as numpy array form iam apending to this array.<br>
Now i want to make from this  torch.Size([64, 1, 28, 28])</p>
<p>EDIT2: My images have shape - (28x28)</p>

Vrushank98: <p>You can use,</p>
<blockquote>
<p>torchvision.transforms.ToTensor()</p>
</blockquote>

JosueCom: <p>You can also convert each image before it goes to the array to a tensor via <code>imgs.append(torch.from_numpy(img))</code>, then use <code>torch.stack(imgs)</code> to turn the array into a tensor.</p>
Validation loss is same, but validation metric is increasing

Myailab: <p>Hi</p>
<p>Validation loss stayed same from 20th to 30th epoch, while the validation F1 score metric changed from 0.79 to 0.82.</p>
<ol>
<li>Can I please get some help in interpreting this?</li>
<li>Also is it suggested to train it further in hope to get better validation metric score!?</li>
</ol>
<p>Thanks</p>

gphilip: <p>Without knowing how the validation loss is defined, it is not possible to answer your first question.</p>
<p>Without knowing the problem or the inputs, it is not possible to answer your second question.</p>
ATen > `at::_fft_r2c` Defined in File Function.h

eduardo4jesus: <p>I am looking at the <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1aaea819b1367e99c6ef062ac8335edba2.html#function-documentation" rel="noopener nofollow ugc">documentation page</a> for the definition of <code>at::_fft_r2c</code>. In that page it says the <code>at::_fft_r2c</code> is defined in the <code>Function.h</code> file. Where is this file located? I can’t find it.</p>

albanD: <p>Hi,</p>
<p>This file is automatically generated during compilation.<br>
The specifications of these generated files can be found here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml" class="inline-onebox">pytorch/native_functions.yaml at master · pytorch/pytorch · GitHub</a><br>
And from there, you can see which function is being called on CPU, CUDA, etc</p>

eduardo4jesus: <p>Thank you so much <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>Despite having a considerable experience with programming, I’ve always struggled understanding code from frameworks (such as PyTorch, Numpy, Scipy). They are on another level of organization that I didn’t quite learn from school, university courses, and most of the projects I worked in the past. Actually, I never even heard about them until I start to dig into these frameworks. Is there a place that explains the organization of these frameworks? (“Organization” meaning all architecture design, patterns and tools that tie all together)</p>

albanD: <p>Ed has a set of blogposts that are quite nice on that topic: <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" class="inline-onebox">PyTorch internals : ezyang’s blog</a></p>

eduardo4jesus: <p><a class="mention" href="/u/alband">@albanD</a>, once again thank you so much. This is a very valuable resource. If I could, I would add it to the end of the <a href="https://github.com/pytorch/pytorch/blob/master/README.md" rel="noopener nofollow ugc">README.md</a> file.</p>
How to transfer a grad attribute from tensor to tensor?

tani: <p>Hello,</p>
<p>I would like to know if it is possible to transfer the <code>.grad</code> attribute of a tensor, e.g, <strong>y</strong> to a <strong>z</strong> tensor which does not have one.</p>
<p>To simplify my problem:</p>
<blockquote>
<p>import torch<br>
x = torch.ones(2, 2, requires_grad=True)<br>
<strong>y</strong> = x + 2<br>
<strong>z</strong> = tensor([[3, 3], [3, 3]])</p>
</blockquote>
<p>I would like then to transfer <strong>y</strong>.grad_fn to <strong>z</strong>. More specifically I want to, in a way, save grad_fn of a tensor and give it later to a new tensor.</p>
<p>Thank you in advance.<br>
Best regards,</p>

ptrblck: <p>You can set the <code>.grad</code> attribute to a specific value, but I don’t think you can add a <code>.grad_fn</code> to a tensor directly.</p>
Concatenate a tensor to sub-dimensions

VERSPD0: <p>For a tensor <code>a</code> in shape of 1 x n, and <code>b</code> in batch x m x n, is there any elegant way to concatenate <code>a</code> to the second dimension of <code>b</code> so that the result should in shape of batch x (m + 1) x n. Meanwhile, the gradient in different batch can effect the same origin <code>a</code> by <code>loss.backward()</code>.</p>

tom: <p>You can decompose cat into allocating an empty tensor and assigning slices (which broadcasts)</p>
<pre><code class="lang-python">res = torch.empty((b, m + 1, n), device=a.device, dtype=a.dtype)
res[:, :1] = a[None]
res[:, 1:] = b[:, :, None]
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
Struggling with debugging. Meaningless error messages, can't inspect tensors, etc

robclouth: <p>I’m currently trying to port a model that works fine in python to c++ but I’m having trouble with the backwards() of a custom autograd function and I’m struggling to make progress.</p>
<p>I’m just getting a “std::system_error at memory location 0x000000FD532FE7C0.” error when calling torch::autograd::grad.</p>
<p>I probably would have solved it by now by comparing to the python version, but debugging the python backward method is difficult because it’s called from C.</p>
<p>Although this post is primarily about this specific error, in general I’m finding the Libtorch dev experience to be a PITA. Can anyone suggest any tips to get:</p>
<ul>
<li>Useful error messages</li>
<li>Useful inspection of tensors (without having to litter couts everywhere in the code)</li>
</ul>
<p>Thanks</p>

ptrblck: <p>You could try to use <code>export TORCH_SHOW_CPP_STACKTRACES=1</code> to (hopefully) get a better stacktrace. Besides that you could also use <code>gdb</code> to try to isolate the issue.</p>
Why is nms so slow?

sunshine1: <p>When the object confidence threshold is set to 0.01,the nms is extremely slow. But when it is set to 0.5 in the case where most object  are filtered out, nms become very fast,but the recall become slow. The nms  code  is as follows which is referenced from github.</p>
<p>def non_max_suppression(prediction, conf_thres=0.5, nms_thres=0.4):<br>
“”"<br>
Removes detections with lower object confidence score than ‘conf_thres’ and performs<br>
Non-Maximum Suppression to further filter detections.<br>
Returns detections with shape:<br>
(x1, y1, x2, y2, object_conf, class_score, class_pred)<br>
“”"</p>
<pre><code># From (center x, center y, width, height) to (x1, y1, x2, y2)
prediction[..., :4] = xywh2xyxy(prediction[..., :4])
output = [None for _ in range(len(prediction))]
for image_i, image_pred in enumerate(prediction):
    # Filter out confidence scores below threshold
    image_pred = image_pred[image_pred[:, 4] &gt;= conf_thres]
    # If none are remaining =&gt; process next image
    if not image_pred.size(0):
        continue
    # Object confidence times class confidence
    score = image_pred[:, 4] * image_pred[:, 5:].max(1)[0]
    # Sort by it
    image_pred = image_pred[(-score).argsort()]
    class_confs, class_preds = image_pred[:, 5:].max(1, keepdim=True)
    detections = torch.cat((image_pred[:, :5], class_confs.float(), class_preds.float()), 1)
    # Perform non-maximum suppression
    keep_boxes = []
    while detections.size(0):
        large_overlap = bbox_iou(detections[0, :4].unsqueeze(0), detections[:, :4]) &gt; nms_thres
        label_match = detections[0, -1] == detections[:, -1]
        # Indices of boxes with lower confidence scores, large IOUs and matching labels
        invalid = large_overlap &amp; label_match
        weights = detections[invalid, 4:5]
        # Merge overlapping bboxes by order of confidence
        detections[0, :4] = (weights * detections[invalid, :4]).sum(0) / weights.sum()
        keep_boxes += [detections[0]]
        detections = detections[~invalid]
    if keep_boxes:
        output[image_i] = torch.stack(keep_boxes)

return output</code></pre>

ptrblck: <p>I guess the internal loops would be called more so I would expect to see an increase in the runtime. You could check it by e.g. printing the number of invocations inside the loop and compare both approaches.</p>
Question for torch.nn.functional.interpolate

pm415: <p>hi, why does linear interpolation not preserve the points themselves?</p>
<p>import torch<br>
import torch.nn.functional as F<br>
torch.manual_seed(0)</p>
<p>a = torch.randn((16,64,8))<br>
b = F.interpolate(a, (a.size(2)*2), mode=‘linear’)</p>
<blockquote>
<blockquote>
<blockquote>
<p>a[0][0]<br>
tensor([-1.1258, -1.1524, -0.2506, -0.4339,  0.8487,  0.6920, -0.3160, -2.1152])<br>
b[0][0]<br>
tensor([-1.1258, -1.1325, -1.1457, -0.9269, -0.4760, -0.2964, -0.3881, -0.1132,<br>
0.5281,  0.8095,  0.7312,  0.4400, -0.0640, -0.7658, -1.6654, -2.1152])</p>
</blockquote>
</blockquote>
</blockquote>
<p>It seems only the corner points are perserved but other points are changed. Is there a way to preserve the rest of points?</p>

ptrblck: <p>That’s not possible out of the box, as it wouldn’t be a linear interpolation anymore.<br>
For your use case you could have a look at the indices, which would be created:</p>
<pre><code class="lang-python">xs = torch.linspace(0, 63, 128)
print(xs)
ys = torch.linspace(0, 7, 16)
print(ys)
</code></pre>
<p>which do not contain integer values between the start and end point (close to some, but not exactly equal).</p>

rraayyii: <p>Hi! I am experiencing a similar issue. What exactly do you mean by ‘as it wouldn’t be a linear interpolation anymore’? In my use case, I compared the pytorch interpolation result to the scipy.zoom(order=1) result, and they are very different. Specifically, the pytorch interpolation function extended the intensity distribution of the original volume, which I really don’t understand since I was just using trilinear interpolation. The range of the interpolated values should be confined within the range of the original value, right?</p>

ptrblck: <aside class="quote no-group" data-username="rraayyii" data-post="3" data-topic="84068">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rraayyii/48/41138_2.png" class="avatar"> rraayyii:</div>
<blockquote>
<p>What exactly do you mean by ‘as it wouldn’t be a linear interpolation anymore’?</p>
</blockquote>
</aside>
<p>A linear interpolation can read values “between” or indices, so forcing to use discrete indices only would not be a linear interpolation anymore in my opinion.</p>
<p>Could you post an executable code snippet for your issue?</p>
Grid sampler equivalent annotation (polygon) transformation

gabrieldernbach: <p>I am using the awesome grid sampler to apply affine transformations to my images.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html#torch.nn.functional.grid_sample" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html#torch.nn.functional.grid_sample</a></p>
<p>For this, I generate a mesh grid over the [-1, 1] square which I then transform with the desired affine matrix. This is the grid I hand to the grid_smaple method. Scaling differences and interpolation are worked out internally by the sampler.</p>
<p>Now I would like to also transform my segmentation masks that for matter of precision are stored as polygons (list of x,y coordinates). Although I can create a binary mask and apply the grid_sample method to it, this means a loss of precision (polygons sometimes overlap, and I don’t want to create unnecessary loops).</p>
<p>How should the polygons be formatted in order to conform with the grid_samplers internal resizing?</p>
<p>I tried the sequence of<br>
a) resize the polygon such that they are relative to [-1, 1] on the source image dimensions<br>
b) apply affine transformation matrix to polygon coordinates<br>
c) resize the polygon to live in the target image dimensions.</p>
<p>This unfortunately was not successful.</p>
Make predictions on CPU (CPU Usage too high)

thalia27: <p>Hello Everyone,</p>
<p>I trained the model on GPU and saved the model. But, I want to load this model and make the predictions on CPU. Unfortunately, the CPU usage is too high as you can see below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19fb0527881f7fde97f4fa64cf621d32bdad37e8.png" data-download-href="https://discuss.pytorch.org/uploads/default/19fb0527881f7fde97f4fa64cf621d32bdad37e8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19fb0527881f7fde97f4fa64cf621d32bdad37e8_2_690x334.png" alt="image" data-base62-sha1="3HPKbt2Lke4icgrzVTXXKU7L8MM" width="690" height="334" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19fb0527881f7fde97f4fa64cf621d32bdad37e8_2_690x334.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19fb0527881f7fde97f4fa64cf621d32bdad37e8_2_1035x501.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19fb0527881f7fde97f4fa64cf621d32bdad37e8_2_1380x668.png 2x" data-dominant-color="F3F8FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1455×705 50.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">img = Image.open(filePath)
        tfms = transforms.Compose([
            transforms.Resize((config["Img_Height_Pred"], config["Img_Width_Pred"])),
            transforms.ToTensor(),
            transforms.Normalize([0.0], [1.0])])
        model.eval()
        img_tensor = tfms(img).unsqueeze(0)
        with torch.no_grad():
            preds = torch.sigmoid(model(img_tensor))
            preds = (preds &gt; 0.5).float()
            tensor_to_pil = transforms.ToPILImage()(preds.squeeze_(0))

</code></pre>
<p>Is there anything I can do to lower the CPU usage?</p>
<p>Best,</p>
<p>Thalia</p>

gphilip: <p>Try limiting the number of threads that PyTorch uses, using <a href="https://pytorch.org/docs/stable/generated/torch.set_num_threads.html#torch.set_num_threads" rel="noopener nofollow ugc">torch.set_num_threads()</a>.</p>

thalia27: <p>Thank you! now the CPU usage is lower!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89a2d985930f50aff35fe1fb55acbb74bac823f6.png" alt="MicrosoftTeams-image (12)" data-base62-sha1="jDAjFKvlvOpCyf6cOeeD0Akc8ku" width="406" height="320"></p>
How to find data_index from ConcatDataset?

oasjd7: <pre><code class="lang-auto">Here is my codes.

trains = []
trains.append(datasets.ImageFolder(Data1_path)
trains.append(datasets.ImageFolder(Data2_path)
datasets = torch.utils.data.ConcatDataset(trains)
train_loader = torchdata.DataLoader(datasets , batch_size=args.batch_size, shuffle=True, num_workers=args.workers, drop_last=True) # Shuffle here.
for i, 'data' in enumerate(train_loader):
    train with 'data'
-&gt; I want to know where this 'data' comes from... 
-&gt; Data1 or Data2 ? How can I know this?
...
</code></pre>
<pre><code class="lang-auto">Here is data structure.
Data1_path: 
/dataset/data1/cat
/dataset/data1/dog
, ...

Data2_path: 
/dataset/data2/cat
/dataset/data2/dog
, ...
</code></pre>
<p>I want to know where this ‘data’ comes from… Data1 or Data2 ? How can I know this?</p>

gphilip: <p>As per the <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#ConcatDataset" rel="noopener nofollow ugc">source code for ConcatDataSet</a>, your <code>datasets</code> object has an attribute called <code>cumulative_sizes</code> that contains information which can help you figure this out. Try querying <code>cumulative_sizes</code> and see if it gives you something useful.</p>
Use of tensor.sum() function

sandeep1: <p>In the code as follows :-<br>
<code>acc = (prediction_cls.eq(y_test)).sum() / float (y_test.shape[0])</code></p>
<p>Here the output of  <code>print(prediction_cls.eq(y_test))</code> is [[ True],[ True],[ True],[ True],[False]…]</p>
<p>Which is a list of boolean values . How can we sum the boolean values using .sum?<br>
Does .sum() function also performs Boolean additon?</p>

pascal_notsawo: <p>Just consider <code>True</code> as <code>1</code> and <code>False</code> as <code>0</code>.</p>
<p>For example <code>torch.tensor([True, True]).sum()</code> is equal to <code>tensor(2)</code>.</p>
Try to understand instance normalization, weight normalization, layer normalizationand and group normalizationinstead of batch normalization

PytorchLearning: <p>Hi all,</p>
<p>I have a question concerning how to use instance normalization, weight norm, layer norm and group norm instead of batch normalization.</p>
<p>Can someone explain to me please how to replace the batchnorm by the others normalization in the following example, just to understand better how it works.</p>
<p>Thanks a lot for your help.</p>
<p>Here is the example :</p>
<pre><code>class ModelExample(torch.nn.Module):

def __init__(self):
super(ModelExample, self).__init__()

self.linear = torch.nn.Linear(100, 1024*4*4)

self.conv1 = nn.Sequential(
                           nn.ConvTranspose2d(in_channels=1024, out_channels=128, kernel_size=4, stride=2, padding=1, bias=False)
                           ,nn.BatchNorm2d(128)
                           ,nn.ReLU(inplace=True))
                          
self.conv2 = nn.Sequential(
                           nn.ConvTranspose2d(in_channels=128, out_channels=3, kernel_size=4, stride=2, padding=1, bias=False)
                          )
self.out = torch.nn.Tanh()

def forward(self, x):

x = self.linear(x)
x = x.view(x.shape[0], 1024, 4, 4)

x = self.conv1(x)
x = self.conv2(x)

x = self.out(x)

return x</code></pre>
Pytorch how to stack a torch tensor

nikiguo93: <p>I use resnet to predict EEG response to image data. And the input data is a matrix x_i with size of 1 * 3 * 500 * 500. Then the output is a vector y_i with size of  1 * 1 * 17 * 100. Let’s call the output EEG response. there is a loop like the following:<br>
for i in range(number of input data):<br>
y_i = model(x_i)</p>
<p>if max_i=200, I would to stack all the y_i to create a matrix with size of 200 * 1 * 17 * 100. How to realize that?</p>

gphilip: <p>See <a href="https://pytorch.org/docs/stable/generated/torch.stack.html" rel="noopener nofollow ugc">torch.stack()</a>.</p>

KFrank: <p>Hi Nikiguo (and gphilip)!</p>
<aside class="quote no-group" data-username="gphilip" data-post="2" data-topic="129022" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/858c86/48.png" class="avatar"> gphilip:</div>
<blockquote>
<p>See <a href="https://pytorch.org/docs/stable/generated/torch.stack.html" rel="noopener nofollow ugc">torch.stack()</a>.</p>
</blockquote>
</aside>
<p>For the specific use case you detail, you will prefer <code>torch.cat()</code>.<br>
(<code>torch.stack()</code> will create a new dimension along which to stack.)</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; y_0 = torch.arange (5).unsqueeze (0)
&gt;&gt;&gt; y_1 = torch.arange (5).unsqueeze (0) + 10
&gt;&gt;&gt; y_2 = torch.arange (5).unsqueeze (0) + 20
&gt;&gt;&gt; y_0.shape
torch.Size([1, 5])
&gt;&gt;&gt; y_list = [y_0, y_1, y_2]
&gt;&gt;&gt; torch.cat (y_list).shape
torch.Size([3, 5])
&gt;&gt;&gt; torch.cat (y_list)
tensor([[ 0,  1,  2,  3,  4],
        [10, 11, 12, 13, 14],
        [20, 21, 22, 23, 24]])
&gt;&gt;&gt; torch.stack (y_list).shape
torch.Size([3, 1, 5])
&gt;&gt;&gt; torch.stack (y_list)
tensor([[[ 0,  1,  2,  3,  4]],

        [[10, 11, 12, 13, 14]],

        [[20, 21, 22, 23, 24]]])
</code></pre>
<p>(In your case, you could use <code>stack()</code> followed by <code>squeeze()</code> to<br>
eliminate one of the singleton dimensions.)</p>
<p>Best.</p>
<p>K. Frank</p>

nikiguo93: <p>Hi thank you for your help, gphilip. It can work but would increase the dimension of the data. But this can be solved by using squeeze().</p>

nikiguo93: <p>aha thanks a lot Frank. That is helpful and it works well for me.</p>
Testing if mathutils works with torch

sidu28: <p>I had a question.  In general, what’s the best way to test if autograd works with certain external packages?  That is, the gradient is being properly calculated…</p>
<p>In my case I am coding a GAN - the generator outputs some values but before I pass them into the discriminator, I have to perform some transformations (this is odometry data).  Those transformations involve this package, <a href="https://docs.blender.org/api/current/mathutils.html" rel="noopener nofollow ugc">mathutils</a>. Here’s an example of a function I apply to the output of the generator (Note that R and T and rotation and translation values that determine a 6dof pose)</p>
<pre><code class="lang-auto">
def to_rotation_matrix(R, T):
    if isinstance(R, torch.Tensor):
        R = quat2mat(R)
        T = tvector2mat(T)
        RT = torch.mm(T, R)
    else:
        R = R.to_matrix()
        R.resize_4x4()
        T = mathutils.Matrix.Translation(T)
        RT = T@R
    return RT
</code></pre>
<p>I want to ensure that using these sorts of custome functions isn’t going to mess with the autograd and that all the gradients will be taken correctly. I’m trying to find the best way to test whether or not autograd is correctly calculating the gradients, i.e., that it’s account for the transformations and rotations that I am doing.  Is there a good way/simple way to test this?  Thanks</p>

albanD: <blockquote>
<p>I had a question. In general, what’s the best way to test if autograd works with certain external packages? That is, the gradient is being properly calculated…</p>
</blockquote>
<p>In general, if a package is not using torch.Tensor, then it cannot be supported by autograd. So mathutils in this case won’t work as I don’t think you can give torch.Tensors as input.</p>
<p>Note that if you really need such function, you can always use it and use a custom autograd Function (doc here <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox">Extending PyTorch — PyTorch 2.1 documentation</a>) to teach the autograd what is the backward pass for it. That way you will be able to use it within pytorch autograd.</p>
Backpropagation through a layer without updating its parameters

TomerF: <p>Hello,</p>
<p>I consider a model consists of 3 fully-connected layers: Layer1, Layer2 and Layer3.<br>
In addition, I would like to minimize 2 losses: LossAll and LossPartial.<br>
Regarding to LossAll, I would like to update the weights of all layers.<br>
Regarding to LossPartial, I would like to update the weights of Layer1 and Layer3 only. The gradients should backpropagate through Layer2, but should not update its weights.<br>
In addition, I use a single optimizer, which optimizes the parameters of all layers, i.e. the step function is called once for optimizing both LossAll and LossPartial concurrently.</p>
<p>How can it be implemented?</p>
<p>Is it acceptable to set Layer2.requires_grad_(False) while I compute the forward pass for LossPartial and then to set Layer2.requires_grad_(True) for computing the forward pass for LossAll?</p>
<p>Thank you,<br>
TomerF</p>

albanD: <aside class="quote no-group" data-username="TomerF" data-post="1" data-topic="128946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tomerf/48/31750_2.png" class="avatar"> TomerF:</div>
<blockquote>
<p>Is it acceptable to set Layer2.requires_grad_(False) while I compute the forward pass for LossPartial and then to set Layer2.requires_grad_(True) for computing the forward pass for LossAll?</p>
</blockquote>
</aside>
<p>Yes it is <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> That should work as you expect.</p>
Seemingly there is a mistake with the docs of KLDivLoss?

kabu1204: <p>docs:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html?highlight=kl#torch.nn.KLDivLoss" rel="noopener nofollow ugc">torch.nn.KLDivLoss</a><br>
which I think should be:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf2c3768d78a10f732482c3f8eeef1b725696ace.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf2c3768d78a10f732482c3f8eeef1b725696ace" title="图片"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf2c3768d78a10f732482c3f8eeef1b725696ace_2_690x87.png" alt="图片" data-base62-sha1="tyJzjPNZSBfSDLwRCNgM30n5Ed0" width="690" height="87" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf2c3768d78a10f732482c3f8eeef1b725696ace_2_690x87.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf2c3768d78a10f732482c3f8eeef1b725696ace_2_1035x130.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf2c3768d78a10f732482c3f8eeef1b725696ace_2_1380x174.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片</span><span class="informations">3550×452 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Kabu!</p>
<aside class="quote no-group" data-username="kabu1204" data-post="1" data-topic="128999" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kabu1204/48/41118_2.png" class="avatar"> kabu1204:</div>
<blockquote>
<p>which I think should be:</p>
</blockquote>
</aside>
<p>No, the documentation for <code>KLDivLoss</code> is correct.</p>
<p>The equation you posted an image of contains the term:</p>
<pre><code class="lang-plaintext">y_n (log y_n - x_n)
</code></pre>
<p>You are expecting <code>log x_n</code> rather than just <code>x_n</code>.  But <code>KLDivLoss</code><br>
expects <code>x_n</code> to be already a log-probability.  Quoting from the<br>
documentation you linked to:</p>
<blockquote>
<p>As with <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a>, the input given is expected to contain <em>log-probabilities</em> and is not restricted to a 2D Tensor. The targets are interpreted as <em>probabilities</em> by default, but could be considered as <em>log-probabilities</em> with <code>log_target</code> set to <code>True</code>.</p>
</blockquote>
<p>Best.</p>
<p>K. Frank</p>
But what is linear layer?

Samuel_Bachorik: <p>Hi I am trying to understand fundamentals of PyTorch and ML and I want to ask you.<br>
For example lets think code like this -<br>
I create x and put that x into linear layer. What is happening with this x ?<br>
When I print y it is equal to some random numbers.<br>
Can someone give me hint about this ? And maybe when should I use lienar layer ?</p>
<pre><code class="lang-auto">x = torch.randn((2, 2))
in_features = 2
out_features = 16

m = torch.nn.Linear(in_features, out_features)

y = m(x)
</code></pre>

arya47: <p>A linear layer is a MLP layer. (Multi layer perceptron)</p>
<p>Have a look at this video:</p><div class="youtube-onebox lazy-video-container" data-video-id="aircAruvnKk" data-video-title="But what is a neural network? | Chapter 1, Deep learning" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=aircAruvnKk" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/aircAruvnKk/maxresdefault.jpg" title="But what is a neural network? | Chapter 1, Deep learning" width="690" height="388">
  </a>
</div>


Samuel_Bachorik: <p>Can you give example on what kind of problem would you use linear layer ? Only one linear layer. No network. Some kind of problem and its solution with linear layer.</p>
<p>EDIT: This video is nice for sure but it do not answer question I think.</p>

arya47: <p>Currently i’m using a linear layer in a project for classification with a CNN.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33335670986081b5c1ed1d0ede3ed777d75dd4f3.png" alt="Screenshot from 2021-08-10 21-20-52" data-base62-sha1="7iWlaocweIouAIKCDJsB6uN3dPJ" width="492" height="196"></p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/2007.13224.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/2007.13224.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/2007.13224.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/2007.13224.pdf" target="_blank" rel="noopener nofollow ugc">2007.13224.pdf</a></h3>

  <p class="filesize">1016.26 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
This one use case where linear layers are used for classification after feature extraction. There may be more problems in which linear layers can be used but it depends on what you’re working on.</p>
Unable to calculate differential due to zero value in A2C actor-network

111119: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c563325434b5aacb396e1484df3b33f9d7b1be0b.png" alt="image" data-base62-sha1="saaAxGtlEAdOspueYvDKUx24Brd" width="466" height="357"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f7f78a36029c720a4f3ae2ffdebc3346c261ea5.png" alt="image" data-base62-sha1="dCOsRECjz2bHQ0rvFWqFDjVPezr" width="606" height="90"></p>
<p>Hi, I’m trying to make A2C(Advantage Actor-Critic) network for my project<br>
The problem is occured by actor-network</p>
<p>when actor produce definitive probablity(like [0.00, 1.00]),<br>
by actor’s loss function,<br>
it cause log(0) = inf-value and then i can’t update actor-network.</p>
<p>i can’t any solution about this problem.</p>
<p>Do i have to modify that prob arbitrarily??<br>
(I’m using separated two-network structure model, not one model structure with saperated output nodes)</p>
<p>P.S) There are two choices of actions.(ACTION_DIM)</p>

111119: <p>I solve it.<br>
The problem is in computational method.<br>
I used softmax at my actor-network’s final activation function.<br>
After that, The output(softmax_probability) is used to calculate log(torch.log10)<br>
But I changed softmax to log_softmax function and then success.</p>
Hdf5 with num_workers > 0

xhtsansiro: <p>Hello, my hdf5 version is 1.12.0. I open the hdf5 file by using hf5 = h5py(‘path’, r), and give this class as an argument to my Dataset. so that i dont need to open hdf5 file every time in <strong>getitem</strong>(). I searched something online, So, it is possible now  that the multi-processing read the same hdf5 file (no change, only read mode)?  but i get a warning at the end of one epoch: Leaking Caffee2 thread-pool after fork. (function pthreadpool).  my dataloader setting is num_worker =16, pin_memory = true. This warning has nothing to do with hdf5 and i can just ignore this warning, right ?  what is the cause of this warning ?</p>
<p>Thanks</p>
Best way to load and organize a 2D CNN that takes in 12 images (ECG leads) per one instance of a class

Jussi_Boersma: <p>So I am making a 2D CNN which takes in 12 images of individual ECG leads per one training example. So my network will basically concatenate 12 individual subnetworks into one output. My data set consists of heart arrythmia positive or negative patients. The folder structure is like this:</p>
<pre><code class="lang-auto">-Images
    -negative
        -patient0
            0.jpg
            ...
            12.jpg
        -patient1
            0.jpg
            ...
            12.jpg
        ...
    -positive
        -patient150
        ...

</code></pre>
<p>When I use datasets.ImageFolder I get one long list of individual images instead of grouped per group of 12. This way I cannot apply K-fold cross validation or shuffling of the dataset as the corresponding 12 lead images for one patients ECG are now no longer banded together.<br>
So I’m wondering what would be a smart approach.</p>
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 11.17 GiB total capacity; 10.49 GiB already allocated; 46.44 MiB free; 10.63 GiB reserved in total by PyTorch)

Niranj_Jyothish: <p>Hi everyone,<br>
Im on this issue for a long time and tried out all possible solutions found online.<br>
I created a new aws instance. While, training a pretrained Pegasus Pytorch model on CUDA on it, I get this error within seconds when I run the command.<br>
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 11.17 GiB total capacity; 10.49 GiB already allocated; 46.44 MiB free; 10.63 GiB reserved in total by PyTorch)<br>
The following are the things I tried but didn’t worked:</p>
<ol>
<li>torch.cuda.empty_cache()</li>
<li>gc.collect() to remove unsed variables.</li>
<li>Reset the GPU with nvidia-smi gpu reset</li>
<li>Rebooting the instance.</li>
<li>Reducing the batch size to 5 (news articles in this case)</li>
<li>Interestingly, the nvidia-smi gives no running process on GPU. Thus, there is nothing to kill.<br>
Please, let me know if anyone has got an idea to tackle this issue.</li>
</ol>

ptrblck: <p>As the error message claims, you are running out of memory and would need to reduce the memory usage in your script by e.g. reducing the batch size or using <code>torch.utils.checkpoint</code> to trade compute for memory.<br>
The not reported allocated memory would be used by the CUDA context. If <code>nvidia-smi</code> doesn’t report any processes, it might be a permission issue, but the GPU usage should nevertheless be shown.</p>

gphilip: <p>One more thing you can try: <code>del variable</code> once <code>variable</code> is not required. Examples: batch data at the end of each training loop, models once they are not needed anymore, datasets which are no longer used, and so on.</p>
Is there a way to handle batch_index-dependent data in PyTorch DataParallel

Randool: <p>My PyTorch model has the input contains a <code>dict</code> . That is:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):

    def __init__(self):
        ...

    def forward(self, input1: Tensor, input2: dict):
        ...
</code></pre>
<p>I know in <code>DataParallel</code> or <code>DistributedDataParallel</code>, a function called <code>scatter_kwargs</code> will split inputs of type <code>Tensor</code> and replicate other types of data.</p>
<p>However, the keys of <code>input2</code> are associated with the batch index, e.g. <code>b[0]</code> contains the data of the first dimension, which results in the original batch index no longer corresponding to the keys in <code>input2</code> after calling <code>scatter_kwargs</code>.</p>
<p>So is there an easy way to divide the data in the dictionary along with the data in the dictionary?</p>

mrshenli: <p>Hey <a class="mention" href="/u/randool">@Randool</a>, as of PyTorch v1.9, <code>DistributedDataParallel</code> only supports Single-Program Single-Device mode, where each process exclusively works on a single model replica. In this case, <code>DistributedDataParallel</code> will not divide forward inputs, and instead will directly pass it to the wrapped local model. Will this be sufficient for your use case?</p>

Randool: <p><a class="mention" href="/u/mrshenli">@mrshenli</a>  Thanks! Looks like a good method, I will try it.</p>
Help with building faster images loader

Gabriella_m: <p>Hi, just for education purposes I downloaded MNIST dataset in zip. version as jpg. files.  And I made my own images loader for these images. I am loading these images from folders with PIL library, then covert them to Numpy and then convert them to a torch.tensor. Of shape (64,1,28,28)</p>
<p>Then I created normal python list and this list have these tensors in it.<br>
(very long array of torch.tensors of shape - (64,1,28,28))</p>
<p>Then iam looping inside this array and iam pushing these tensors to model.</p>
<p>I want to ask you if you have any idea where can I store this tensors if I don’t want to store them in that array ? Any better way ? Or any tips how to speed up this images loader ?</p>

gphilip: <p>Using <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" rel="noopener nofollow ugc">Datasets and DataLoaders</a> is the way to do this in PyTorch. An <a href="https://pytorch.org/vision/stable/datasets.html#mnist" rel="noopener nofollow ugc">MNIST DataSet</a> is part of torchvision.</p>
Problem with input shape, please help

Gabriella_m: <p>Hi, till now I was using small model for MNIST like this -<br>
Input tensor was (64x1x28x28) and I used this for flatten images to vector. So it was (64x784)</p>
<pre><code class="lang-auto">images = images.view(images.shape[0], -1)
</code></pre>
<pre><code class="lang-auto">input_size = 784
hidden_sizes = [128, 64]
output_size = 10

model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.LogSoftmax(dim=1))
print(model)
</code></pre>
<p>I changed model to this bigger model That starts with these layers.<br>
How should I change input?<br>
<strong>I have (64x1x28x28), how should i transofrm it for this model ?</strong></p>
<pre><code class="lang-auto">        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=2, padding=1)
        self.activation1 = nn.ReLU()

        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1)
        self.activation2 = nn.ReLU()
ETC.......
AT THE END I HAVE
       self.linear1 = nn.Linear(128 * 5 * 5, 10)
</code></pre>

gphilip: <p>Did you try feeding the input without transforming it in any way?</p>

Gabriella_m: <p>Yes you are right is helped ! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
Build Classifier Head Properly

AlperEkmekci: <p>Hello,</p>
<p>I’m tuning an encoder-decoder model in order to classify images. However I’m a little bit confused about how to build a classifier head. Currently I’m using only an activation function and a <code>Linear</code> layer in the classifier head but I can’t be sure that It’s the way I can only do. How should I build the <em>classifier head</em> properly and how can I be sure about the <code>Linear</code> layer number that I put the classifier head in.</p>
<p>Best<br>
Alper</p>
Loss function - multi target

Naima_Bessaoud: <p>Hello,</p>
<p>I built a classification model which takes a vector as input and which gives me 3 labels for example [score_1, score_2, score_3] = [3, 7, 9] with the score 3,7,9 it is a score chosen from 11 possible values</p>
<p>3 \ in {0,…, 11}<br>
7 \ in {0,…, 11}<br>
9 \ in {0,…, 11}</p>
<p>I calculated the loss function losse as follows:</p>
<p>loss = criterion (pred, y.long ()) with criterion = nn.CrossEntropyLoss (), but I get an error in the code</p>
<p>multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15</p>
<p>i dont know how to solve it. Could you please help me?</p>

ptrblck: <p>Assuming that all 3 “scored” correspond to active classes, the use case sounds like a multi-class classification and you would use <code>nn.BCEWithLogitsLoss</code> as the criterion.</p>

Naima_Bessaoud: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="128281">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>nn.BCEWithLogitsLoss</p>
</blockquote>
</aside>
<p>Hello,</p>
<p>Thank you for your answer. May with this error function I have a dimension error</p>
<p>“Target size (torch.Size([200, 3])) must be the same as input size (torch.Size([200, 11]))”</p>
<p>I did not understand the usefulness of this function in my case?</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> expects the model output and target to have the same shape, so you would need to provide a model output in <code>[batch_size, nb_classes]</code> (same as the target) where a high value indicates an active class and a low one an inactive.</p>
when call torch.onnx.export(), the graph is pruned by default ? how to cancel pruning

liym27: <p>For example:</p>
<pre><code class="lang-python">import torch

hidden_dim1 = 10
hidden_dim2 = 5
tagset_size = 2

class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.line1 = torch.nn.Linear(hidden_dim1, hidden_dim2)
        self.line2 = torch.nn.Linear(hidden_dim2, tagset_size)

    def forward(self, x, y):
        out1 = self.line1(x)
        out2 = self.line2(y)
        return out1

X = torch.randn(20, hidden_dim1)
Y = torch.randn(hidden_dim1, hidden_dim2)
inputs = (X, Y)

model = MyModel()
f = './model.onnx'
torch.onnx.export(model, inputs, f,
                    opset_version=9,
                    example_outputs=None,
                    input_names=["X"], output_names=["Y"],verbose=True)
</code></pre>
<pre><code class="lang-bash">graph(%X : Float(20, 10, strides=[10, 1], requires_grad=0, device=cpu),
      %line1.weight : Float(5, 10, strides=[10, 1], requires_grad=1, device=cpu),
      %line1.bias : Float(5, strides=[1], requires_grad=1, device=cpu)):
  %Y : Float(20, 5, strides=[5, 1], requires_grad=1, device=cpu) = onnx::Gemm[alpha=1., beta=1., transB=1](%X, %line1.weight, %line1.bias) # /root/.conda/envs/torch1.9/lib/python3.6/site-packages/torch/nn/functional.py:1847:0
  return (%Y)
</code></pre>
<h4>
<a name="how-every-the-exported-graph-doesnt-contain-line2-maybe-because-the-output-of-mymodel-is-not-depend-on-out2-selfline2y-i-guess-the-graph-is-pruned-by-default-1" class="anchor" href="#how-every-the-exported-graph-doesnt-contain-line2-maybe-because-the-output-of-mymodel-is-not-depend-on-out2-selfline2y-i-guess-the-graph-is-pruned-by-default-1"></a>How every, the exported graph doesn’t contain <code>line2</code> , maybe because the  output of MyModel  is not depend on <code>out2 = self.line2(y)</code> ? I guess the graph is pruned by default.</h4>
<p><strong>What should I do if I want to not do pruning?</strong></p>
<h2>
<a name="motivation-2" class="anchor" href="#motivation-2"></a>Motivation</h2>
<p>I want to do something for <code>self.named_parameters()</code> in <code>model.forward()</code>, eg.</p>
<pre><code class="lang-python">def check_parameters():
  # do something for parameters by calling 
  # some ops including OP1, OP2 and so on
  return

class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.line = torch.nn.Linear(hidden_dim1, hidden_dim2)

    def forward(self, x, y):
        out = self.line1(x)
        check_parameters()
        return out
</code></pre>
<p>How every, the exported graph doesn’t contain <code>OP1, OP2</code> , maybe because the  output of MyModel  is not depend on <code>check_parameters()</code> ? I guess the graph is pruned by default.</p>
How to prevent some buffers from being broadcasted, while having broadcast_buffers=true?

yxchng: <p>I have some buffers defined using <code>self.register_buffer('x', ...)</code> and I need it to be not broadcasted. However, I do not want to set <code>broadcast_buffers=False</code> because I want other behaviours to remain the same (e.g. running stats in BN being broadcasted).</p>
<p>Or should I define <code>x</code> in other ways?</p>

pbelevich: <p>Could you please elaborate on why you need to register buffer but not broadcast it?</p>

yxchng: <p>I want to accumulate results (in different processes) every n forward passes, aggregate, do something and repeat.</p>
<p>Let say I store the result in self.x. If I broadcast self.x at the end of a forward pass. In the second pass, when I broadcast again, there will be double counting.</p>