akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Predicting float values based off image input

jmax: <p>My goal is as above, to train a model to take images as input and predict a variable (ideally off a regression) based on the image. I previously used ResNet for classification, but my understanding is that the best approach for this would be to design a Sequential NN from scratch, but I don’t know where to start. What would be the best model structure for this that would allow me to load the variable as well as images? Also, would using an ImageFolder work for leading data or is that oriented towards classification?</p>

ptrblck: <p>I’m unsure why a ResNet wouldn’t work as you could still replace the last linear layer (the original classifier) and add a new linear layer returning the regression outputs.</p>
<aside class="quote no-group" data-username="jmax" data-post="1" data-topic="185104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/839c29/48.png" class="avatar"> jmax:</div>
<blockquote>
<p>that would allow me to load the variable as well as images?</p>
</blockquote>
</aside>
<p>Why would you want to pass the regression target to the model instead of treating it as a target and pass it to the loss function?</p>
<aside class="quote no-group" data-username="jmax" data-post="1" data-topic="185104">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/839c29/48.png" class="avatar"> jmax:</div>
<blockquote>
<p>Also, would using an ImageFolder work for leading data or is that oriented towards classification?</p>
</blockquote>
</aside>
<p><code>ImageFolder</code> is used for classification use cases as it will assign a class label to each folder.</p>

jmax: <p>I attempted that originally, but I may not have implemented it properly. Here is what I tried</p>
<pre><code class="lang-auto">model.fc = nn.Linear(model.fc.in_features, 1)
</code></pre>
<p>But the model was giving me very poor accuracy.</p>
<p>Also, I was a little confused about the model structure, and yes, there is no need to load the variable directly. I asked that because I was writing a custom data loader to normalize the variable, but that is all resolved now.</p>
Pytorch replication in c++

Swapnil_Baad: <p>I want to replicate the below code for inference of a neural network model in C++:</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize((224, 224)),  # Resize images to 224x224
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))  # Normalize the image tensors
])
</code></pre>
<p>The C++ code that I am using:</p>
<pre><code class="lang-auto">
    // convert from BGR to RGB
    cv::cvtColor(image_org, image, cv::COLOR_GRAY2RGB);

    // resize
    cv::resize(image, image, cv::Size(sizeX, sizeY));

    // reshape to 1D
    image = image.reshape(1, 1);

    // Convert the image to floating-point type and  Normalize the image
    cv::Mat floatImage;
    image.convertTo(floatImage, CV_32F, 1.0 / 255.0);

    // Define mean and standard deviation values
    cv::Scalar mean(0.5, 0.5, 0.5);
    cv::Scalar stdDev(0.5, 0.5, 0.5);

    // Subtract mean and divide by standard deviation***
    floatImage -= mean;
    floatImage /= stdDev;

</code></pre>
<p>But the accuracy differs around 5% when I execute the code with pytorch and C++.</p>

ptrblck: <p>If you think the preprocessing step is causing differences in your workflow, you could process a specific image in both codes, and compare their output to check the error.</p>

Swapnil_Baad: <p>Yes, I have comapared and it is not the same</p>

spiegelball: <p>This could be interesting: <a href="https://zuru.tech/blog/the-dangers-behind-image-resizing" class="inline-onebox" rel="noopener nofollow ugc">The dangers behind image resizing</a></p>
Faster block diagonal attention mask

RomanCast: <p>Hi,</p>
<p>I am trying to implement a block diagonal attention mask. My input is the size of each block, and I have different sizes depending on the sample in the batch. Usually the blocks are small (size between 1 and 10). My current solution is to <strong>loop through the batch to create the block diagonal masks</strong>, and then concatenate them:</p>
<pre><code class="lang-python">all_sizes = [[2, 0], [1, 1]]

all_masks = []
for sizes in all_sizes:
    mask = torch.block_diag(*[torch.ones((s, s)) for s in sizes])
    all_masks.append(mask)

attention_mask = torch.stack(all_masks, dim=0)
# Output:
# [[[1.0, 1.0], 
#  [1.0, 1.0]], 
#
# [[1.0, 0.0], 
#  [0.0, 1.0]]]
</code></pre>
<p>However, since my batch sizes may get large, this results in a lot of calls to <code>torch.block_diag</code> which takes too much time. I have tried a few different solutions but none is faster than the previous one.</p>
<p>Do you have any idea how I could write this operation to speed it up? In particular given the fact that I am not creating “any” block diagonal tensor, but an attention mask that only has ones on the diagonal?</p>
<p>Thanks a lot !</p>
Pytorch DataLoader switch item to List and cause several errors

Peter_Phan: <p>My source code is using pytorch and like this:</p>
<pre><code class="lang-auto">def Embed(sequenceSet):
  output = []
  for s in sequenceSet:
    PseDNCSequence = Embedding.PseDNC(str(s))
    ANFSequence = Embedding.ANF(str(s))
    EIIPSequence = Embedding.EIIP(str(s))
    emdbededSequence = PseDNCSequence+ANFSequence+EIIPSequence
    output.append(emdbededSequence)
  return np.array(output)

text = file.read()
lines = text.strip().split('\n')
embededS = Embed(lines)
embeddedSequences = torch.tensor(embededS)
my_dataset = TensorDataset(embeddedSequences)
loader = data.DataLoader(my_dataset, batch_size=batch_size, shuffle=True)

for epoch in range(num_epochs):
   for training_sample in loader:
      training_sample = training_sample.view(-1, sequenceLength)
      batch_size = training_sample.shape[0]
      ....
</code></pre>
<p>It is getting error " AttributeError: ‘list’ object has no attribute ‘view’ " at the line:</p>
<pre><code class="lang-auto">training_sample = training_sample.view(-1, sequenceLength)
</code></pre>
<p>I tried to change it into:</p>
<pre><code class="lang-auto">training_sample = torch.tensor(training_sample)
training_sample = training_sample.view(-1, sequenceLength)
</code></pre>
<p>… then I  I received another error “ValueError: only one element tensors can be converted to Python scalars”</p>
<p>I already checked the other places and make sure that the input data to the loader is tensor. Can anybody help me how to solve it</p>
<p>Thank you so much !!!</p>

ptrblck: <p>Use <code>torch.stack</code> or <code>torch.cat</code> depending on the desired output shape instead of <code>torch.tensor(training_sample)</code>.</p>

Peter_Phan: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>After tried it, but seem not work. For instead, I add one more fake column like this:</p>
<p>y = torch.ones(len(embededSequence), 1).float()<br>
dataset = TensorDataset(embededSequence, y)</p>
<p>then it work</p>
How can I load/read only part of a model checkpoint?

Adrian_Postelnicu: <p>Hello I am trying to do inference with a large model which can not fit into my CPU RAM.<br>
Is there any way I can load only a part of the model checkpoint ? Is it possible to load only the layer names from a model and later the weights of specified layers?<br>
For example I am trying to do something like this:</p>
<pre><code class="lang-auto">import torch 

class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.linear = torch.nn.Linear(10, 10)
        self.act = torch.nn.ReLU()
        self.linear2 = torch.nn.Linear(10, 10)
        self.act2 = torch.nn.Sigmoid()
    
    def forward(self, x):
        x = self.linear(x)
        x = self.act(x)
        x = self.linear2(x)
        x = self.act2(x)
        return x

class MyLoader():
    """
    This class is a custom loader for PyTorch models
    that can be used to load only specific layers. 
    """
    def __init__(self, pth_file):
        self.pth_file = pth_file

    def get_layer_names(self):
        ...

    def load_layer_by_name(self, names):
        out = {}
        for name in names:
            out[name] = self.load_layer(name)
            return out 

    def load_layer(self, name):
        ...


if __name__ == "__main__":
    m = MyModel()
    torch.save(m.state_dict(), 'example.pth')

    # This will fail with OOM error
    #model_dict = torch.load( 'example.pth', torch.device('cpu') )
    #print(model_dict)

    # Custom loader call
    loader = MyLoader('example.pth')
    n = loader.get_layer_names()
    print(n)
    loader.load_layer_by_name(['linear.weight', 'linear.bias'])
</code></pre>
<p>So far I have tried understanding how models are loaded by trying to debug torch.load function but many functions from pickle module are not exposed.</p>
<p>Thank you for your help ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I’m not sure I completely understand your use case but it seems you would like to load only a subset of the model at once and delete the rest since your host RAM is limited?<br>
You could technically only load a specific layer from the <code>state_dict</code> by accessing its keys, but you would then need to delete the rest of the model, which could generally cause a large bottleneck since you would be loading from your drive in every iteration.</p>

Adrian_Postelnicu: <p>Yes, I want to load only a subset of the model, but torch.load will generate a OOM error for very large models. Another way of looking at it is that I want to load just a shard from HDD to RAM and run that.<br>
I understand that there could be a bottleneck, but that could be alleviated if the subsets are executed in a network made computers with less RAM than model size.</p>
Why F.SoftMax gives exact numbers of a matrix across rows

Omnia_Al-wazzan: <p>Hi all!</p>
<p>I have a question regarding Softmax.</p>
<p>suppose I have two tensors with a batch size like this</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

A = torch.Tensor([[1]
                ,[2]
                ,[3]]).float()[None] , # tensor shape &gt;&gt; (1,3,1)

B = torch.Tensor([[5],
                  [2],
                  [6]]).float()[None] # tensor shape &gt;&gt; (1,3,1)

outer_product =A.view(A.shape[0], A.shape[1], -1) * B.view(B.shape[0],-1, B.shape[1])

#outer_product: 
#tensor([[[ 5.,  2.,  6.],
#         [10.,  4., 12.],
#         [15.,  6., 18.]]])

outpro_SM = F.softmax(out_prod, dim=-1)
# output of outpro_SM is:
#tensor([[[2.6539e-01, 1.3213e-02, 7.2140e-01],
#         [1.1917e-01, 2.9539e-04, 8.8054e-01],
#         [4.7426e-02, 5.8528e-06, 9.5257e-01]]])

</code></pre>
<p>Until this point everything seems to work well, though, I when I change the operation of the outer product and do for example outer addition like this</p>
<pre><code class="lang-auto">out_add =A.view(A.shape[0], A.shape[1], -1) + B.view(B.shape[0],-1, B.shape[1])
# output of out_add is:
# tensor([[[6., 3., 7.],
#         [7., 4., 8.],
#         [8., 5., 9.]]])

outadd_SM = F.softmax(out_add , dim=-1)
# output of outadd_SM is 
#tensor([[[0.2654, 0.0132, 0.7214],
#         [0.2654, 0.0132, 0.7214],
#         [0.2654, 0.0132, 0.7214]]])

</code></pre>
<p>I am very confused why the softmax of the out_add gives a matrix of similar numbers for each row. Am I doing somthing wrong here?</p>
<p>Any help is greatly appreciated</p>

KFrank: <p>Hi Omnia!</p>
<aside class="quote no-group" data-username="Omnia_Al-wazzan" data-post="1" data-topic="185094" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omnia_al-wazzan/48/49966_2.png" class="avatar"> Omnia_Al-wazzan:</div>
<blockquote>
<p>I am very confused why the softmax of the out_add gives a matrix of similar numbers for each row.</p>
</blockquote>
</aside>
<p>This is to be expected.</p>
<p><code>softmax()</code> converts “unnormalized” log-probabilities to probabilities, and<br>
in the process normalizes them.  That is why <code>softmax()</code> can map different<br>
inputs to the same outputs.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.0'
&gt;&gt;&gt; t1 = torch.arange (5.)
&gt;&gt;&gt; t2 = t1 - 1.5
&gt;&gt;&gt; t1
tensor([0., 1., 2., 3., 4.])
&gt;&gt;&gt; t2
tensor([-1.5000, -0.5000,  0.5000,  1.5000,  2.5000])
&gt;&gt;&gt; t1.softmax (dim = 0)
tensor([0.0117, 0.0317, 0.0861, 0.2341, 0.6364])
&gt;&gt;&gt; t2.softmax (dim = 0)
tensor([0.0117, 0.0317, 0.0861, 0.2341, 0.6364])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Omnia_Al-wazzan: <p>Really interesting! That makes sense…</p>
<p>Thanks a lot Frank!</p>
<p>Regards,<br>
Omnia</p>
Torch not Cuda enabled

Erkan: <p>Hi. I am master student so I have a thesis. I have a issue with loading pytorch. I send you with image. If you can help me I will be very grateful.<br>
Thank you</p>

ptrblck: <p>I don’t know what exactly fails since you are not sharing any information, but stick to the install instructions from <a href="https://pytorch.org/get-started/locally/">here</a> and it should work.</p>
Multi GPU training vs Training on GPUs separately

Ahmad_Omar_Ahsan: <p>I am training a model on miniImageNet and have access to a machine with two GPUs.</p>
<p>Should I develop a script allowing me to train on two GPUs or train on each GPU separately?  My options are to train on a single model using multi-GPU training or train different models on different GPUs in parallel.</p>
<p>Regarding training in parallel with two GPUs how do I select the devices such that Model A will train on GPU 0 and Model B will train on GPU 1?</p>

Marandil: <p>I recommend starting with reading <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" rel="noopener nofollow ugc">this article</a> from PyTorch docs, as it gives insight into what’s available. I personally prefer the “Data Parallel Training” approach, but also tried splitting the model between devices.</p>
<p>When dealing with a machine having multiple GPUs you select them either with <code>device='cuda:0'</code>, <code>'cuda:1'</code>, etc. or <code>torch.device('cuda', index=0)</code> (I prefer the latter, but YMMV).</p>
Rotated image distorted

Bishwa_Karki: <p>I tried rotating image but the <strong>rotated images looks distorted.</strong> Why is the rotated image distorted?</p>
<pre><code>import torchvision.transforms.functional as TF
image = TF.rotate(img, angle)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/842d7417e3f45ca9539bb53b9331109c8fd7984c.png" alt="image" data-base62-sha1="iRipp5fBLlmub5V0YFXrSUkK1TC" width="464" height="244"></p>

InnovArul: <p>It will be great if you could give a complete executable example to debug in your question.<br>
There are some missing pieces (how the image is read, angle = ?). Overall, it will help if you give full examples that can be executed readily.</p>
Problem with mps in Jupyter notebook

sag21: <p>When I verify <code>mps</code> support using a recommended Python script:</p>
<p>import torch<br>
if torch.backends.mps.is_available():<br>
mps_device = torch.device(“mps”)<br>
x = torch.ones(1, device=mps_device)<br>
print (x)<br>
else:<br>
print (“MPS device not found.”)</p>
<p>I get output the following output:<br>
tensor([1.], device=‘mps:0’)</p>
<p>But when I use the same script in Jupyter notebook, I get:<br>
AttributeError                            Traceback (most recent call last)<br>
Input In [1], in &lt;cell line: 2&gt;()<br>
1 import torch<br>
----&gt; 2 if torch.backends.mps.is_available():<br>
3     mps_device = torch.device(“mps”)<br>
4     x = torch.ones(1, device=mps_device)</p>
<p>AttributeError: module ‘torch.backends’ has no attribute ‘mps’</p>

ptrblck: <p>You are most likely using a different virtual environment in your Jupyter notebook, so make sure the same env is used with the same PyTorch binary.<br>
You can check the installed PyTorch release e.g. via <code>print(torch.__version__)</code>.</p>

sag21: <p>Yes, you are absolutely right! Thanks a lot!</p>
Submit a PyTorch Function Request

ptuser1: <p>Hi, where can I submit a request to  PyTorch  for a PyTorch library implementation of interpolation that works exactly like <a href="https://numpy.org/doc/stable/reference/generated/numpy.interp.html" rel="noopener nofollow ugc">numpy.interpolate</a>?</p>

ptrblck: <p>You can leave a comment in <a href="https://github.com/pytorch/pytorch/issues/50334">this feature request</a>.</p>
RuntimeError: Expected pytree structure of cotangents to be the same as pytree structure of outputs to the function

AlphaBetaGamma96: <p><strong>TL;DR</strong> - <code>torch.func.vjp</code> fails with functionalized <code>nn.Module</code> objects in the <code>torch.func</code> namespace due to mis-matched pytree structure.</p>
<pre><code class="lang-auto">Pytorch version: 2.1.0.dev20230519
CUDA version: 11.8
</code></pre>
<p>Hi All,</p>
<p>I’ve been trying to compute jacobian-vector and vector-jacobian products within the <code>torch.func</code> namespace, however, when using <code>torch.func.vjp</code> I seem to get the following error. I know that <code>vjp</code> works for single tensor functions (see here: <a href="https://pytorch.org/tutorials/intermediate/jacobians_hessians.html" class="inline-onebox" rel="noopener nofollow ugc">Jacobians, Hessians, hvp, vhp, and more: composing function transforms — PyTorch Tutorials 2.0.1+cu117 documentation</a>)</p>
<pre><code class="lang-auto">RuntimeError: Expected pytree structure of cotangents to be the same as pytree structure of outputs to the function. cotangents: {'fc1.weight': *, 'fc1.bias': *, 'fc2.weight': *, 'fc2.bias': *}, primal output: *
</code></pre>
<p>A minimal reproducible example is below,</p>
<pre><code class="lang-auto">import torch
_ = torch.manual_seed(0)
from torch import nn, Tensor
from torch.func import vjp, jvp, functional_call, vmap
from functools import partial

class NET(nn.Module):

  def __init__(self, num_input: int, num_hidden: int, num_output: int) -&gt; None:
    super(NET, self).__init__()
    
    self.fc1 = nn.Linear(num_input, num_hidden)
    self.fc2 = nn.Linear(num_hidden, num_output)
    
    self.af = nn.Tanh()
    
  def forward(self, x: Tensor) -&gt; Tensor:
    x = self.fc1(x)
    x = self.af(x)
    x = self.fc2(x)
    return x

#==============================================================================#

batch_size=100
num_input = 4
num_hidden = 64
num_output = 1

net = NET(num_input=num_input,
          num_hidden=num_hidden,
          num_output=num_output)

#define functional call here
def fcall(params, x):
  return functional_call(net, params, x)

#functional inputs 
params = dict(net.named_parameters()) 
x = torch.randn(batch_size, num_input)

#generate random vectors for jacobian-vector and vector-jacobian products
tangent_params = {k: torch.randn_like(p) for k, p in params.items()} #random tangent vector
tangent_x = torch.randn_like(x)

#generate functions for jacobian-vector and vector-jacobian products respectively
jvp_fn = lambda params, x, tangent_params, tangent_x: jvp(fcall, (params, x), (tangent_params, tangent_x))
_, vjp_fn = vjp(fcall, params, x)

out, jvp_out = vmap(jvp_fn, in_dims=(None,0,None,0))(params, x, tangent_params, tangent_x) #OK
print("jvp_out: ",jvp_out.shape)
out, vjp_out = vmap(vjp_fn, in_dims=(None,0))(tangent_params, tangent_x) # &lt;&lt;&lt; FAILS HERE
print("vjp_out: ",vjp_out.shape)
</code></pre>
How to use torch.ceil strategy when process PTSQ

ChenGongfu: <p>i want to use the round ceil strategy in quantization.which options shall I set?</p>

HDCharles: <p>we don’t have anything that does ceiling rounding by default. You could probably just add .5 to all the weights and do normal quantization though, or something along those lines?</p>

jerryzh168: <p>this is not configurable right now, the current rounding strategy (used in quantize ops etc.) is the same as <a href="https://pytorch.org/docs/stable/generated/torch.round.html?highlight=torch+round#torch.round" class="inline-onebox" rel="noopener nofollow ugc">torch.round — PyTorch 2.0 documentation</a></p>

ChenGongfu: <p>Thanks,i want to verify whether the results of my own Winograd convolution calculation match those of Pytorch quantization convolution calculation, and I found that the round strategy was not taken correctly.</p>
<p>What is the specific source code corresponding to the implementation of the round using the FBGEMM backend for int8 operations？I will try using Python to implement it.</p>

jerryzh168: <aside class="quote no-group" data-username="ChenGongfu" data-post="4" data-topic="184464">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chengongfu/48/62180_2.png" class="avatar"> ChenGongfu:</div>
<blockquote>
<p>What is the specific source code corresponding to the implementation of the round using the FBGEMM backend for int8 operations？I will try using Python to implement it.</p>
</blockquote>
</aside>
<p>I think it’s calling this <a href="https://github.com/pytorch/FBGEMM/blob/main/src/FbgemmConv.cc#L119" rel="noopener nofollow ugc">https://github.com/pytorch/FBGEMM/blob/main/src/FbgemmConv.cc#L119</a>, but you’ll need to dig through the code to see what it is actually doing</p>
Exception When Using torch::nn::Sequential

funkypickle: <p>I registered a sequential variable and saved the result into a member variable called toPatchEmbedding.</p>
<p><code>toPatchEmbedding = register_module("toPatchEmbedding", torch::nn::Sequential(torch::nn::LayerNorm(torch::nn::LayerNormOptions({256})), torch::nn::Linear(256, 512), torch::nn::LayerNorm(torch::nn::LayerNormOptions({512})))));</code></p>
<p>In the forward function later on I tried to use <code>torch::Tensor x = toPatchEmbedding-&gt;forward(input);</code> where input is a tensor with shape (16, 14, 14, 256).</p>
<p>For some reason the forward call to layerNorm throws an exception.</p>
<p>“Exception: Exception 0xe06d7363 encountered at address 0x7ffd8e384b2c” and I don’t know how to debug it.</p>
<p>According to the documentation for the python function at: <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html" class="inline-onebox" rel="noopener nofollow ugc">LayerNorm — PyTorch 2.1 documentation</a></p>
<p>the normalizedShape applies the function on the last n dimensions of the tensor. Where n is the size of the normalizedShape list. So the size of the input tensor matches but it is still not working.</p>
<p>I also noticed that when I run the program through the debugger, when the sequential variable is initialized if you look at the member variables of the module some variables have a value of so I think that the problem is with initializing the sequential variable.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3aecf4d33673a91cd6f659d2e7d3ed9ca193309.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3aecf4d33673a91cd6f659d2e7d3ed9ca193309" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3aecf4d33673a91cd6f659d2e7d3ed9ca193309_2_690x243.png" alt="image" data-base62-sha1="ucDh14Haey43IBcGw9s7CmKoYYN" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3aecf4d33673a91cd6f659d2e7d3ed9ca193309_2_690x243.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3aecf4d33673a91cd6f659d2e7d3ed9ca193309_2_1035x364.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3aecf4d33673a91cd6f659d2e7d3ed9ca193309_2_1380x486.png 2x" data-dominant-color="353638"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1957×691 178 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If this is relevant it is on the GPU.</p>
<p>Thank you for helping <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to calculate HVP in neural networks

ksnxr: <p>What is the recommended way of calculating the Hessian vector product of a neural network function? Torch.func seems to provide the related functions, but I’m not sure how to pass in the vector.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/ksnxr">@ksnxr</a>,</p>
<p>There’s an example within the documentation here: <a href="https://pytorch.org/tutorials/intermediate/jacobians_hessians.html" class="inline-onebox" rel="noopener nofollow ugc">Jacobians, Hessians, hvp, vhp, and more: composing function transforms — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
<p>If you have a minimal reproducible example, please do share it.</p>
Google TFT prediction on training / validation

El-Hassan-Hajbi: <p>Hello all, I got some questions about Google TFT implementation on <strong>pytorch</strong></p>
<p>1 - If I understood well, the <strong>predict</strong> method of tft model, returns prediction of the last <strong>max_prediction_length</strong> values. how to predict on whole training / validation set using tft model? is there any built-in function for that ?<br>
2 - The plots I get have time index on xaxis, is there any way to show date (which is a feature on my dataset) on xaxis ?<br>
3 - Since the model takes min, max encoder / prediction <em>length</em> which one does it use on training / prediction ? max ?</p>
<p>Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Feeding mini-batches of variable input image sizes

Sherine_Brahma: <p>Hi,</p>
<p>I have a dataset comprising variable sizes of noisy 3D input volumes. I would like to feed them into the network simultaneously in mini-batches to smooth out the gradient during backpropagation. I would also like to avoid loops and input padding as much as possible. In principle, this sounds feasible, but I do not know if this is implemented in PyTorch. Please let me know if you are aware of a method to accomplish this.</p>
<p>Update: Running models with different input sizes in parallel could be an option. I know Pytorch allows us to do this on multiple GPUs. Is there a way for running two models with two different input sizes (mini-batch size of 2) on a single GPU, and then share the gradients?</p>
Does using torch.where cause the model's parameter gradients to become zero?

abc50111: <p>Here is the <code>forward()</code> method of my pytorch model:</p>
<pre><code class="lang-python">    def forward(self, x, output_type, *unused_args, **unused_kwargs):
        gru_output, gru_hn = self.gru(x)
        # Decoder (Graph Adjacency Reconstruction)
        for data_batch_idx in range(x.shape[0]):
            pred = self.decoder(gru_output[data_batch_idx, -1, :])  # gru_output[-1] =&gt; only take last time-step
            pred_graph_adj = pred.reshape(1, -1) if data_batch_idx == 0 else torch.cat((pred_graph_adj, pred.reshape(1, -1)), dim=0)
        if output_type == "discretize":
            bins = torch.tensor(self.model_cfg['output_bins']).reshape(-1, 1)
            num_bins = len(bins)-1
            bins = torch.concat((bins[:-1], bins[1:]), dim=1)
            discretize_values = np.linspace(-1, 1, num_bins)
            for lower, upper, discretize_value in zip(bins[:, 0], bins[:, 1], discretize_values):
                pred_graph_adj = torch.where((pred_graph_adj &lt;= upper) &amp; (pred_graph_adj &gt; lower), discretize_value, pred_graph_adj)
            pred_graph_adj = torch.where(pred_graph_adj &lt; bins.min(), bins.min(), pred_graph_adj)

        return pred_graph_adj
</code></pre>
<p>And here is the snippet of training:</p>
<pre><code class="lang-python">                pred = self.forward(x, output_type=self.model_cfg['output_type'])
                batch_loss = self.loss_fn(pred, y)
                self.optimizer.zero_grad()
                batch_loss.backward()
                self.optimizer.step()
                self.scheduler.step()
</code></pre>
<ol>
<li>When <code>output_type</code> is not  <code>"discretize"</code> (not using <code>torch.where</code>), <code>sum([p.grad.sum() for p in self.decoder.parameters()])</code> will be non-zero.
<ul>
<li>But When <code>output_type</code> is  <code>"discretize"</code> (using <code>torch.where</code>), <code>sum([p.grad.sum() for p in self.decoder.parameters()])</code> will be zero.</li>
</ul>
</li>
<li>I’ve check the <code>batch_loss</code>, it’s not zero.</li>
<li>I’ve check all the <code>require_grad</code> of weight of model, they are True.</li>
<li>I’ve check computational graph, <code>pred</code> and <code>batch_loss</code> are connect to model’s weight.</li>
</ol>
<p>My questions are:</p>
<ol>
<li>Does using <code>torch.where</code> cause the model’s parameter gradients to become zero?</li>
<li>If <code>torch.where</code> won’t cause that, what’s other possible reasons?</li>
</ol>
<p>PS. I ask the same question on stackoverflow: <a href="https://stackoverflow.com/questions/76769248/does-using-torch-where-cause-the-models-parameter-gradients-to-become-zero" class="inline-onebox" rel="noopener nofollow ugc">python - Does using torch.where cause the model's parameter gradients to become zero? - Stack Overflow</a></p>
Replacing CrossEntropyLoss() with nn.BCEWithLogitsLoss()

Soumya_Kundu: <p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>In this tutorial, what changes should one make to use nn.BCEWithLogitsLoss() instead of nn.CrossEntropyLoss() ?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> is used for a multi-class classification (or segmentation) use case, where each sample (or pixel) belongs to exactly one class. <code>nn.BCEWithLogitsLoss</code> is used for a binary or multi-label classification (or segmentation) where each sample (or pixel) belongs to zero, one, or multiple classes.<br>
Depending on your actual use case (e.g. binary vs. multi-label) you would need to change the model output shape and the targets. E.g. for a binary classification your model would output logits in the shape <code>[batch_size, 1]</code> and the target would contain values in <code>[0, 1]</code> in the same shape.</p>

Soumya_Kundu: <p>Thank you very much, that cleared my doubt!</p>