akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Supporting new devices, where to start

username: nscotto
cooked: <p>Hi,</p>
<p>where to start if I want to add support for a new parallel computing device to pytorch?</p>

username: nscotto
cooked: <p>Found it here <a href="https://pytorch.org/tutorials/advanced/extend_dispatcher.html" class="inline-onebox" rel="noopener nofollow ugc">Extending dispatcher for a new backend in C++ — PyTorch Tutorials 1.10.1+cu102 documentation</a></p>
LTC: C++ bindings?/Why is this in torch/csrc not ATen?

username: rnett
cooked: <p>I’ve been looking into the upcoming lazy tensor support, and I don’t understand why it’s in <code>torch/csrc</code> rather than ATen.  I’ve noticed there’s a few other components that seem to do this (distribution and jit), is there a reason or heuristic behind it?</p>
<p>The reason this came up was because I was (very tentatively) looking at creating Java bindings for it for a project of mine, and had planed on using ATen as the base.  Using <code>torch/csrc</code> as a binding source is likely not possible (I don’t want to include python), is there any alternative?</p>
How does it makes sense to apply fft on features?

username: seyeeet
cooked: <p>I am making it super simplified, I wonder if it would make sense to apply fft to the features of a 1D signal.<br>
Lets say I have a batch of inputs and the extracted features looks like this <code>x.shape = BxCxL=10x64x100</code>.<br>
I was wondering if it makes sense if I apply fft to the features and I was wondering what the fft features would represent in that case.<br>
I am confused what dimension I should apply the fft and how should apply it in this scenario.<br>
Any suggestion would be really appreciated!</p>

username: mMagmer
cooked: <p>this may help</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html">
  <header class="source">

      <a href="https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 21 April 2016">Haytham Fayek – 21 Apr 16</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html" target="_blank" rel="noopener nofollow ugc">Speech Processing for Machine Learning: Filter banks, Mel-Frequency Cepstral...</a></h3>

  <p>Understanding and computing filter banks and MFCCs and a discussion on why are filter banks becoming increasingly popular.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Pretrained VGG16 giving Nan output

username: Flock1
cooked: <p>Hi,</p>
<p>I am working with a pretrained VGG16 model to classify images. I have added my own layers after the model like this:</p>
<pre><code class="lang-auto">model = torchvision.models.vgg16(pretrained=True)
model.classifier.add_module('7', nn.Linear(in_features=1000, out_features=500,bias=True))
model.classifier.add_module('8', nn.ReLU())
model.classifier.add_module('9', nn.Linear(in_features=500, out_features=100,bias=True))
model.classifier.add_module('10', nn.ReLU())
model.classifier.add_module('11', nn.Linear(in_features=100, out_features=67,bias=True))
</code></pre>
<p>When I start the training, I print the output of the model and I see it’s <code>nan</code>. I don’t know what going on? I’ve ensured my input doesn’t contain <code>nan</code> and this is my dataloader:</p>
<pre><code class="lang-auto">class dataload(Dataset):

    def __init__(self, x, transform=None):
        self.data = x

    def __len__(self):
        return len(self.data)

    def __getitem__(self, i):

        img = mpimg.imread(self.data[i])/255.0
        img = img.transpose((2, 0, 1))
        img = torch.from_numpy(img).float()
        tmp = np.int32(filenames[i].split('/')[-1].split('_')[0][1])
        label = np.zeros(67)
        label[tmp] = 1
        label = torch.from_numpy(label).float()

        # if self.transform:
        #     sample = self.transform(sample)

        return img,label
</code></pre>
<p>What could be going on?</p>

username: ptrblck
cooked: <p>I don’t know what might be causing the issue, as I’m not able to reproduce the NaN outputs using your code and random inputs:</p>
<pre><code class="lang-python">model = models.vgg16(pretrained=True)
model.classifier.add_module('7', nn.Linear(in_features=1000, out_features=500,bias=True))
model.classifier.add_module('8', nn.ReLU())
model.classifier.add_module('9', nn.Linear(in_features=500, out_features=100,bias=True))
model.classifier.add_module('10', nn.ReLU())
model.classifier.add_module('11', nn.Linear(in_features=100, out_features=67,bias=True))

for _ in range(100):
    x = torch.randn(1, 3, 224, 224)
    out = model(x)
    print(torch.isnan(out).any())
</code></pre>

username: Flock1
cooked: <p>Even I don’t know. When I was trying, I was getting <code>Nan</code>. But then, I broke it down. I checked if just the pretrained model was giving <code>nan</code>. Sometimes it did and sometimes it didn’t. Mostly it didn’t.</p>
<p>Then after adding each fc layer I checked the output and again the same. Sometimes it would give nan but mostly, it would give some result.</p>
<p>I don’t know if there’s some dedicated debugging for this. What do you suggest? Maybe pytorch needs a <code>nan</code> detector.</p>
How to code Actor Critic Reinforcement Learning with continuous action ouput

username: Khabbab_Zakaria
cooked: <p>I am working on an Actor-Critic Model with Continuous action output. The output should be any number from 0 to 1. In the Actor part, I am calculating the mu using Sigmoid (to have output from 0 to 1) and variance using ReLU (to always have a positive value).</p>
<p>The code is as follows:</p>
<pre><code class="lang-auto">class Policy(nn.Module):
    """
    implements both actor and critic in one model
    """
    def __init__(self):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(state_size, 128)

        self.fc2 = nn.Linear(128, 64)

        # actor's layer
        self.action_head = nn.Linear(64, action_size)
        self.mu = nn.Sigmoid()
        self.var = nn.ReLU()

        # critic's layer
        self.value_head = nn.Linear(64, 1)

    def forward(self, x):
        """
        forward of both actor and critic
        """
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))

        # actor: choses action to take from state s_t 
        # by returning probability of each action
        action_prob = self.action_head(x)
        mu = self.mu(action_prob)
        var = self.var(action_prob)

        # critic: evaluates being in the state s_t
        state_values = self.value_head(x)

        return mu, var, state_values

class Agent():
    def __init__(self, model, is_eval=False, model_name=""):
      self.model_name = model_name
      self.is_eval = is_eval
      self.model = load_model(model_name) if is_eval else model

    def act(self, state):
      mu, var, state_value = self.model(state)
      mu = mu.data.cpu().numpy()
      sigma = torch.sqrt(var).data.cpu().numpy()
      actions = np.random.normal(mu, sigma)
      actions = np.clip(actions, 0, 1) #to have output ranging from 0 to 1
      
      actions = torch.from_numpy(actions)

      return actions, state_value
</code></pre>
<p>Now, lets say action_size = 1.</p>
<p>One output for print(mu, var) is: tensor([0.5266], grad_fn=), tensor([0.7478], grad_fn=)</p>
<p>The action, which is the normal distribution, is 0 (actually it was a negative number before the clipping step).</p>
<p>Now, my question is if the logic is correct to have an Actor-Critic model with Continuous action output.</p>
Adding subvolumes to tensor without for-loop

username: ASpeiser
cooked: <p>Hey,<br>
I have this procedure, where I place a number 3D volumes within a larger Volume at different positions.<br>
Due to the for-loop this is currently a bottleneck on speed. I feel like there has to be a way to do vectorize this but I can’t quite get it to work.</p>
<p>(I took out any consideration of edge cases to make the example more concise).</p>
<pre><code class="lang-auto"># Tensor containing N different 3D volumes. We want to place these at certain spots in a bigger volume.
N, psf_h, psf_w, psf_d = sub_vols.shape

# b, ch, x, y, z are lists that specify where each sub volume should be placed.
for idx in range(N):
    big_volume[b[idx], ch[idx],
    z[idx] : z[idx]+psf_h,
    y[idx] : y[idx]+psf_w,
    x[idx] : x[idx]+psf_d] += sub_vols[idx]
</code></pre>
NotImplementedError('wrap must be called at the top level of a module')

username: MartinZhang
cooked: <p><strong>NotImplementedError(‘wrap must be called at the top level of a module’)</strong></p>
<p>First, I define an operator in <strong>add.cpp</strong></p>
<pre><code class="lang-auto">torch::Tensor add_ops(torch::Tensor x, torch::Tensor y){
    torch::Tensor z  = x + y;
    return z;
}

PYBIND11_MODULE(TORCH_EXENSION_NAME, m){
    m.def("add_ops", &amp;add_ops, "add ops");
}
</code></pre>
<p>then ,I use <code>python setup.py build_ext --inplace</code> to change add.cpp into add.os and use it.</p>
<p><strong>pyadd.py</strong></p>
<pre><code class="lang-auto">import torch
import add_ops
from torch.fx import wrap

add = add.add_ops
wrap("aa")

def pyadd(n1, n2):
    return add(n1, n2)

</code></pre>
<p>I still use <code>python setup.py build_ext --inplace</code> to convert pyadd.py to pyadd.os</p>
<p>but, when I use the pyadd.os such as <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_head_bandage.png?v=10" title=":face_with_head_bandage:" class="emoji" alt=":face_with_head_bandage:"></p>
<pre><code class="lang-auto">from pyadd import pyadd

pyadd(torch.randn(10), torch.randn(10))
</code></pre>
<p>Then , there is an error about wrap must be called at the top level of a module</p>
<p>why? please help me thanks.</p>
Here is a question about fuse torch.nn.BatchNorm2d

username: MartinZhang
cooked: <p>This a function about fuse torch.nn.BatchNorm2d by myself <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_head_bandage.png?v=10" title=":face_with_head_bandage:" class="emoji" alt=":face_with_head_bandage:"></p>
<pre><code class="lang-auto">import torch.nn as nn

class FuseBN(nn.Module):
    def __init__(self, layer):
        super().__init__()
        eps = layer.eps
        mean = layer.running_mean
        var = layer.running_var
        weight = layer.weight
        bias = layer.bias

        bias = bias - (weight*mean)/torch.sqrt(var + eps)
        weight = weight / torch.sqrt(var + eps)
        
        self.weight  = weight.reshape(1, -1, 1, 1)
        self.bias = bias.reshape(1, -1, 1, 1)
       
    def forward(self, x):
        out = self.weight * x + self.bias
        return out
</code></pre>
<p>When I try to compare the results of BN and FuseBN</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

data = torch.randn(1, 3, 224, 224)
bn = nn.BatchNorm2d(3)
fuse_bn = FuseBN(bn)

bn_result = bn(data)
fuse_bn_result = fuse_bn(data)

compare_value = torch.max(torch.abs(bn_result - fuse_bn_result))
</code></pre>
<p>In theory, the difference should be about 1e-5， but I get the <code>compare_value</code> is 0.143. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"></p>
<p>I don’t know, why? please help me, thanks a lot. <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>

username: mMagmer
cooked: <p>See <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d" rel="noopener nofollow ugc">batchnorm2d</a><br>
when you forward data from the batch norm layer, you are changing running averages.<br>
Use <code>bn.eval()</code> to freeze these buffers.</p>

username: ptrblck
cooked: <p>You are comparing the native batchnorm layer in training mode with your <code>FuseBN</code> layer, which uses the eval logic.<br>
Also, after initializing the batchnorm layer the running mean would be all zeros and running_var all ones so you might want to train it for a few steps so that both layers would indeed normalize the data with running stats.<br>
This should work:</p>
<pre><code class="lang-python">data = torch.randn(1, 3, 224, 224) * 10 + 5
bn = nn.BatchNorm2d(3)

for _ in range(100):
    out = bn(data)

print(bn.running_mean)
print(bn.running_var)
bn.eval()

fuse_bn = FuseBN(bn)
bn_result = bn(data)
fuse_bn_result = fuse_bn(data)

compare_value = torch.max(torch.abs(bn_result - fuse_bn_result))
print(compare_value)
&gt; tensor(4.7684e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>PS: <a class="mention" href="/u/mmagmer">@mMagmer</a> was a bit faster, but just posting it for the sake of completeness with the code.</p>

username: MartinZhang
cooked: <p>awsome. thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  <a class="mention" href="/u/mmagmer">@mMagmer</a></p>
conv1D parameters

username: user36
cooked: <p>Hi! I am trying to use 1D convolution in order to classify a set of time signals. Every data unit I need to classify is made out of 65 different time series, each one contains 50 time samples, so if I write:</p>
<pre><code class="lang-auto">dataset = MyDataset(train,y_train_one_hot)
a,b = dataset[1]
print(a.shape
</code></pre>
<p>I will get: <code>[56,50]</code>.</p>
<p>I want to run 1D convolutional filters on each one of the channels. Problem is I cant get right the inputs of the first nn.Conv1d layer - right now I am using:</p>
<pre><code class="lang-auto">self.c1 = nn.Conv1d(in_channels=56 , out_channels=100 , kernel_size=ks1)
</code></pre>
<p>but when I run the model with a batch size of 100, the input becomes of the shape <code>[100,56,50]</code> and I get only one prediction for a batch size of 100 (instead of 100X3). Can anyone help with the right syntax? Thank you very much!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="user36" data-post="1" data-topic="139753">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/user36/48/45075_2.png" class="avatar"> user36:</div>
<blockquote>
<p>and I get only one prediction for a batch size of 100 (instead of 100X3)</p>
</blockquote>
</aside>
<p>I guess you are reshaping the activation in the <code>forward</code> method of your model and might change the batch size to 1. If you are using <code>x = x.view(-1, SHAPE)</code> to flatten the tensor, then replace it with <code>x = x.view(x.size(0), -1)</code> which will keep the batch size and might raise a valid shape mismatch error in another layer.</p>

username: user36
cooked: <p>That was indeed the case, I wasn’t using <code>reshape</code> the right way (didn’t know about <code>view</code>).<br>
Thank you very much!</p>
Normalizing a multi dimensional positive tensor so that it sums to 1 along the last dimension

username: Soumya_Jain
cooked: <p>Hii,<br>
I have a positive tensor of shape (bsz , num_heads , tgt _len , src_len). I  need that it sums to 1 along the last dimension (i.e src_len dimension). I don’t want to use softmax. I want to take sum of elements along the last dimension and then divide the elements by that sum.<br>
How should I do this? Can someone help?</p>
<p>Thanks in advance.</p>

username: AlphaBetaGamma96
cooked: <p>Try this?</p>
<pre><code class="lang-auto">X = torch.randn(bsz, num_heads, tgt_len, src_len)
s = X.sum(keepdim=True, dim=-1)
X_norm = X / s
</code></pre>

username: Soumya_Jain
cooked: <p>Worked. Thanks. keepdim = True was important.</p>

username: AlphaBetaGamma96
cooked: <p>You need <code>keepdim=True</code> in order for the division to broadcast correctly.</p>
Search architecture to find objects in the image

username: ll2pakll
cooked: <p>Hi. I’m making a tool to find detached faces in images. I plan that it will work individually for each project, that is, if there is a need to find such faces, the user will have to mark a small number of frames, and the rest of the frames will find the neural network itself after training. An image is fed to the input, the output is 4 coordinates. Now I am faced with the question of neural network architecture. I am a beginner, I have looked through several available courses and tutorials on PyTorch website. From these tutorials I have written myself a few simple neural networks and they work, but very badly and not accurately. I’ve tried to search already available architectures, but what I found is very complicated and very far from what pyTorch site teaches (Yolox for example). I’m asking for help in finding an architecture that will fit my needs, it doesn’t have to match the inputs and outputs exactly, those parameters I can edit, but it has to have a standard architecture, otherwise I won’t be able to edit and use it:</p>
<pre><code>class NeuralNetwork(nn.Module):
    def __init__(self):
        ...

    def forward(self, x):
        ...
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9f73b7e4e5f8004055f2011e9e756b7ea5a13bf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d9f73b7e4e5f8004055f2011e9e756b7ea5a13bf" title="Снимок экрана 2021-12-20 111541"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9f73b7e4e5f8004055f2011e9e756b7ea5a13bf_2_690x401.jpeg" alt="Снимок экрана 2021-12-20 111541" data-base62-sha1="v6diQMlyTsSywjqeMCXcWrJ0XbV" width="690" height="401" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9f73b7e4e5f8004055f2011e9e756b7ea5a13bf_2_690x401.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9f73b7e4e5f8004055f2011e9e756b7ea5a13bf_2_1035x601.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9f73b7e4e5f8004055f2011e9e756b7ea5a13bf_2_1380x802.jpeg 2x" data-dominant-color="433B2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Снимок экрана 2021-12-20 111541</span><span class="informations">3358×1956 465 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Install packages failed

username: DoveEchoing
cooked: <p>I have installed Anaconda and created torch’s virtual environment.I want to use torch in jupyter notebook,so I must install nb_conda.However, the package “nb_conda_kernels” installed failed(I had installed nb_conda successfully a few days ago,but for some reasons,I uninstalled Anaconda. When I installed nb_conda today,it occurred error.The error is as follows.)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb6ffba38558d398d53791231daf2a16a56b8699.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb6ffba38558d398d53791231daf2a16a56b8699" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb6ffba38558d398d53791231daf2a16a56b8699.png" alt="捕获" data-base62-sha1="xAM7VPRmkDMSHKptxTmwwLIcz2h" width="690" height="463" data-dominant-color="070707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">1092×733 12.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Environment</strong>:</p>
<ul>
<li>Client OS: Windows10</li>
<li>Python version:3.8.5</li>
<li>PyTorch/TensorFlow version:torch 1.10.0（python version is 3.6 in torch environment）</li>
</ul>
Extract the corresponding columns of multiple tensors

username: yipliu
cooked: <p>Hi,</p>
<p>I know how to extract rows from multiple tensors to form a list of new tensors. However, I haven’t found a good way to do that for columns, so what I’ve done so far is put the tensor forward transpose.</p>
<p>E.g:</p>
<pre><code class="lang-python">a = torch.randn(3,5)
b = torch.randn(3,5)
c = torch.randn(3,5)
orig = [a, b, c]
</code></pre>
<p>For  rows:</p>
<pre><code class="lang-python">list_rows = list(zip(*orig))
</code></pre>
<p>Now, this is what I did with the columns</p>
<pre><code class="lang-python">list(zip(*[torch.transpose(m, 0, 1) for m in orig]))
</code></pre>
<p>Is there a cleaner and more straightforward way to handle the column case?</p>
Feature extraction from trained neural network for my new test data

username: Gaurav_Prasad
cooked: <h2>
<a name="i-trained-the-following-architecture-i-need-to-extract-the-intermediate-layer-output-from-label-predictor-layer-before-the-fully-connected-output-which-is-supposed-to-be-of-size-64-i-do-not-want-to-retrain-the-model-using-forward-hook-is-it-possible-to-get-the-intermediate-layer-output-for-new-test-samples-1" class="anchor" href="#i-trained-the-following-architecture-i-need-to-extract-the-intermediate-layer-output-from-label-predictor-layer-before-the-fully-connected-output-which-is-supposed-to-be-of-size-64-i-do-not-want-to-retrain-the-model-using-forward-hook-is-it-possible-to-get-the-intermediate-layer-output-for-new-test-samples-1"></a>I trained  the following architecture, i need to extract the intermediate layer output from label predictor layer before the fully connected output, which is supposed to be of size 64. I do not want to retrain the model using forward hook. Is it possible to get the intermediate layer output for new test samples.</h2>
<p>class DANN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.num_features = num_features</p>
<pre><code>self.feature_extractor = nn.Sequential(

       nn.Linear(self.num_features,32), nn.ReLU(True),     
       nn.Linear(32,64), nn.ReLU(True),        
       nn.Linear(64,128), nn.ReLU(True),
       nn.Linear(128,128), nn.ReLU(True) )

 

self.label_predictor = nn.Sequential(nn.Linear(128,128),

                                     nn.ReLU(True),
                                     nn.Linear(128,64),
                                     nn.ReLU(True) ,nn.Linear(64,3) )

                              

self.domain_classifier = nn.Sequential(

       nn.Linear(128,256),  nn.ReLU(True),
       nn.Linear(256,64),  nn.ReLU(True),
       nn.Linear(64,2),
       nn.LogSoftmax(dim=1) )
</code></pre>
<p>def forward(self,x, grl_lambda = 1.0):<br>
x = x.expand(x.data.shape[0], num_features)</p>
<pre><code>  features = self.feature_extractor(x)
  features_grl = GradientReversalFn.apply(features, grl_lambda)
  label_pred = self.label_predictor(features)
  domain_pred = self.domain_classifier(features_grl)
 
  return label_pred, domain_pred</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Gaurav_Prasad" data-post="1" data-topic="139725">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gaurav_prasad/48/45062_2.png" class="avatar"> Gaurav_Prasad:</div>
<blockquote>
<p>I do not want to retrain the model using forward hook.</p>
</blockquote>
</aside>
<p>I’m not sure how “retrain” is related to forward hooks, but in case you don’t want to use hooks you could write a custom <code>nn.Module</code> and override the <code>forward</code> method with your logic (i.e. return the desired intermediate tensors additionally yo the model output).</p>
Loading specific optimizer tensors ont different devices using torch.load

username: sjscotti
cooked: <p>Hi<br>
I have a neural net model with optimizer state data saved on a pickled file (excuse if my terminology is imprecise) at a checkpoint.  The tensors for the model and the optimizer were all saved from the GPU, and when the checkpoint is loaded using torch.load, they are loaded on the GPU again.  However, when I load from the checkpoint, I would like some specific optimizer state tensors (e.g., the exponential moving average momentum tensor of the Adam optimizer, for example) to be loaded into CPU instead of GPU.  I have found very little documentation on the map_location option in torch.load to explain how to write a function that does this.  Can you  please provide an example for how to do this for one or more tensors?  You can assume that the optimizer tensors to be loaded into CPU are state[‘exp_avg’] and state[‘exp_avg_sq’] for definiteness.<br>
Thanks!</p>

username: ptrblck
cooked: <p>You can pass <code>map_location='cpu'</code> to <code>torch.load</code> to load the object to the CPU.<br>
Here is a small example showing the usage:</p>
<pre><code class="lang-python"># save optimizer.state_dict() with CUDATensors
model = models.resnet18().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1.)
out = model(torch.randn(1, 3, 224, 224).cuda())
out.mean().backward()
optimizer.step()

torch.save(optimizer.state_dict(), 'opt.pth')
optimizer.state_dict()

# in a new script make sure the GPU is not visible and load the state_dict to the CPU
import os
os.environ['CUDA_VISIBLE_DEVICES'] = ""
import torch
import torchvision.models as models

torch.cuda.get_device_name()
# &gt; RuntimeError: No CUDA GPUs are available

model = models.resnet18()
optimizer = torch.optim.Adam(model.parameters(), lr=1.)
optimizer.load_state_dict(torch.load('opt.pth'))
# &gt; RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
optimizer.load_state_dict(torch.load('opt.pth', map_location='cpu')) # works
</code></pre>
Torch still uses memory after function executed

username: hadaev8
cooked: <pre><code class="lang-auto">def process_images():
    import clip
    import glob
    clip_model, preprocess = clip.load("ViT-B/32")
    clip_model.eval()
    files = glob.glob('./**/*.jpg', recursive=True)
    batch_size = 64
    with torch.inference_mode():
        for i in tqdm(range(0, len(files), 64)):
            batch_files = files[i:i+batch_size]
            images = torch.stack([preprocess(Image.open(file)) for file in batch_files], dim=0).cuda()
            features = clip_model.encode_image(images)
            features /= features.norm(dim=-1, keepdim=True)
            features = features.detach().cpu().clone().numpy()
            for feature, file in zip(features, batch_files):
                np.save(file.replace('.jpg', '.npy'), feature)

process_images()
torch.cuda.empty_cache()
</code></pre>
<p>I have processing like this and torch still uses memory after execution while I expect memory to be freed, how should I fix it?</p>

username: ptrblck
cooked: <p>I don’t know which memory you are measuring, but in case you are checking the GPU memory, I can’t reproduce the issue and get zero allocated memory after <code>process_image</code> and zero reserved memory after <code>empty_cache()</code> as expected using a <code>resnet18</code>:</p>
<pre><code class="lang-python">def process_images():
    clip_model = models.resnet18().cuda()    
    clip_model.eval()
    with torch.inference_mode():
        for i in range(0, 10):
            images = torch.randn(64, 3, 224, 224).cuda()
            features = clip_model(images)
            features /= features.norm(dim=-1, keepdim=True)
            features = features.detach().cpu().clone().numpy()

print(torch.cuda.memory_allocated())
# &gt; 0
print(torch.cuda.memory_reserved())
# &gt; 0
process_images()
print(torch.cuda.memory_allocated())
# &gt; 0
print(torch.cuda.memory_reserved())
# &gt; 723517440
torch.cuda.empty_cache()
print(torch.cuda.memory_allocated())
# &gt; 0
print(torch.cuda.memory_reserved())
# &gt; 0
</code></pre>
Differences between .ones_like() and .new_ones()

username: CDhere
cooked: <p>Hi! I was wondering what’s the differences between two tensor operations, namely <code>x.ones_like()</code> and <code>x.new_ones()</code>? Looks like they are all intended for creating tensors filled with ones and inheriting from the properties of the calling tensor <code>x</code>?</p>
<p>Thank you!</p>

username: anantguptadbl
cooked: <aside class="quote no-group quote-modified" data-username="CDhere" data-post="1" data-topic="139768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdhere/48/29866_2.png" class="avatar"> CDhere:</div>
<blockquote></blockquote>
</aside>
<p><strong>ones_like</strong><br>
.ones_like() takes tensor as an input argument</p>
<pre><code class="lang-auto">input = torch.empty(2, 3)
torch.ones_like(input)
</code></pre>
<p><strong>new_ones</strong><br>
new_ones is a member function of the tensor object</p>
<pre><code class="lang-auto">
&gt;&gt;&gt; tensor = torch.tensor((), dtype=torch.int32)
&gt;&gt;&gt; tensor.new_ones((2, 3))
tensor([[ 1,  1,  1],
        [ 1,  1,  1]], dtype=torch.int32)
</code></pre>

username: tom
cooked: <p>The obvious difference is that you pass a shape into <code>t.new_ones</code> but not into <code>torch.ones_like</code>.</p>
<p>If I remember correctly, the <code>torch.ones_like</code> is an equivalent of the numpy API and <code>t.new_ones</code> is a “torch-legacy-idiomatic” replacement of a long-deprecated 
<code>t.new</code>
 method (which incidentally still seems to exist e.g. here <a href="https://github.com/pytorch/pytorch/blame/b199e3c8427a9648b9a76830ed9898ab4603288c/torch/nn/init.py#L465" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/init.py at b199e3c8427a9648b9a76830ed9898ab4603288c · pytorch/pytorch · GitHub</a> ) which is of similar nature as the deprecated  
<code>Tensor(...)</code>
 and 
<code>FloatTensor(...)</code>
 constructors.</p>
<p>Of course, part of the attraction of <code>t.new_ones</code> is that in contrast to C++ with <code>t.options()</code>, you have the “tensor properties” spread over several keyword arguments in Python (dtype, device, memory layout, …), but I tend to prefer <code>torch.ones</code> and <code>torch.ones_like</code> over the <code>t.new_ones</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
How to switch variable between state_dict and parameter_dict

username: fschiffers
cooked: <p>Hello,</p>
<p>I have understood difference between register_buffer() and register_parameter/nn.Parameter.</p>
<p>In my models I initialize my variable either into the buffer or as a parameter similar to this code:</p>
<pre><code class="lang-auto">if self.is_param == True:
  self.register_buffer("_variable", variable)    
elif self.is_param == False:
  self.variable= torch.nn.Parameter(variable)      

# This is the function call I want to implement (but I have no idea how)
def change_variable_state(flag_param : bool = True):
     if flag_param == True:
            # make the variable a parameter
     elif flag_param == False:
            # make the variable a normal buffer in the state_dict
</code></pre>
<p>However, now I would like to be able to move this variable from a Parameter to a "normal buffered variable and vice-versa. I.e. I would look to transformed a buffered variable into a parameter.</p>
<p>In other words: I want to dynamically change the “buffer” state of my variable so that I can see it either in the function named_parameters() or not.</p>
<p>Any help is much appreciated.</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>The simplest way I can think of to do that is remove it and add it again:</p>
<pre><code class="lang-python"># from param to buffer
var = self.variable
del self.variable
self.register_buffer("variable", var)

</code></pre>

username: fschiffers
cooked: <p>Clean and simple. Thanks</p>

username: anantguptadbl
cooked: <p>In case you want to do something more elaborate, you can directly access the underlying dict. But <a class="mention" href="/u/alband">@albanD</a> 's solution is the best</p>
<pre><code class="lang-auto">import torch
model.register_buffer("_variable", torch.FloatTensor([0]))    
model.register_parameter("_parameter", torch.nn.Parameter(torch.FloatTensor([1])))
model._buffers['_parameter'] = model._parameters['_parameter'].cpu().detach()
model._parameters.pop('_parameter')
</code></pre>
Error when trying to use _VF.vf.lsm()

username: Desblancs_Mika
cooked: <p>Hello, I am trying to use a functional lstm for a few shot learning project. I saw from this post (<a href="https://discuss.pytorch.org/t/torch-nn-functional-lstm/32266" class="inline-onebox">Torch.nn.functional.lstm</a>) that we could use _VF.lstm(). However, when trying to run it with the following parameters I get an error.</p>
<p>I am not sure how tuple of Tenors is different from a tuple containing tensors. Furthermore, from the source code found here (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/rnn.py at master · pytorch/pytorch · GitHub</a>), setting hx=None for the parameter right after the input is the default value. However, I get an error telling me I forgot a parameter when I don’t set it.</p>
<pre><code class="lang-auto">data = train_loader.__iter__().next()
inputs, labels = data

#get model parameters
fast_weights = dict((name, param) for (name, param) in model.named_parameters())

#get the lstm specific parameters
lstm_params = [tensor for key, tensor in fast_weights.items() if key not in ['embedding.weight', 'linear.weight', 'linear.bias']]
lstm_params = tuple(lstm_params)

#try to run _VF.vf.lstm()
lstm = torch.nn.modules.rnn._VF.vf.lstm(inputs, None, lstm_params, True, 2, 0.2, True, True, True)
</code></pre>
<p>Error:<br>
TypeError: lstm() received an invalid combination of arguments - got (Tensor, NoneType, tuple, bool, int, float, bool, bool, bool), but expected one of:</p>
<ul>
<li>(Tensor data, Tensor batch_sizes, tuple of Tensors hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional)<br>
didn’t match because some of the arguments have invalid types: (Tensor, !NoneType!, !tuple!, !bool!, !int!, !float!, !bool!, bool, bool)</li>
<li>(Tensor input, tuple of Tensors hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional, bool batch_first)<br>
didn’t match because some of the arguments have invalid types: (Tensor, !NoneType!, !tuple!, bool, int, float, bool, bool, bool)</li>
</ul>
How to get 2d sub tensor with specific value condition from an original tensor in pytorch?

username: Razieh_Baradaran
cooked: <p>I want to copy a 2-d torch tensor to a destination tensor containing only values until the first occurrence of 202 value and zero for the rest of the items like this:</p>
<pre><code class="lang-auto">source_t=tensor[[101,2001,2034,1045,202,3454,3453,1234,202]
         ,[101,1999,2808,202,17658,3454,202,0,0]
         ,[101,2012,3832,4027,3454,202,3454,9987,202]]

destination_t=tensor[[101,2001,2034,1045,202,0,0,0,0]
                    ,[101,1999,2808,202,0,0,0,0,0]
                    ,[101,2012,3832,4027,3454,202,0,0,0]]
</code></pre>
<p>how can i do it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not sure if it is the most efficient, but the following works:</p>
<pre><code class="lang-python">mask = source_t.eq(202).flip(1).cumsum(dim=1).flip(1)
mask = mask.eq(mask.select(1, 0).unsqueeze(1))
print(mask) # mask of the source values to keep

res = mask * source_t
print(res)
</code></pre>
Access a registered buffer directly by name

username: MasterSkepticista
cooked: <p>I’ve registered a couple of buffers using <code>model.register_buffer(...)</code> method. The model here is an <code>nn.Module</code> object with defined architecture and forward pass.</p>
<p>Question is, how do I <strong>directly</strong> access a buffer known its name? Iterating over <code>named_buffers</code> is not efficient for my use case, and methods like <code>module.get_buffer()</code> shown <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" rel="noopener nofollow ugc">here</a> return <code>... object has no attribute get_buffer</code>.</p>
<p>Example to reproduce:</p>
<pre><code class="lang-python">import torch, torch.nn as nn
fc = nn.Linear(512, 10)
fc.register_buffer('buffer', None)

print(fc._buffers['buffer']) # This works, but doesn't look like I should, because its a private entity
print(fc.get_buffer('buffer')) # Doesn't work

# How do I access 'buffer' without iterator?
</code></pre>
<p><strong>Edit:</strong> Just got it resolved. <code>getattr(fc, 'buffer')</code> or <code>fc.buffer</code> gives the expected behaviour.</p>

username: albanD
cooked: <p>Hi,</p>
<p>After you <code>fc.register_buffer('buffer', None)</code>, you can access it at the attribute with that name: <code>fc.buffer</code>.</p>
Gathering in multidimensional tensors: weird brodcasting

username: treepleks
cooked: <p>I have a 5 dimensional tensor <code>S</code> of shape<code> (bs,a,b,c,d)</code> which can be seen as a batch of (a x b) 2D matrices of (c x d) 2D matrices. Let say<code> bs =1</code> and <code>a = b = c = d = 3</code> for simplicity.</p>
<p>I have a shape (b) vector (let say <code>ind = [2,0,1]</code>) that contains indices to columns of the inner c x d matrices. For each row i and column j of the outer a x b matrix, I need to gather the column ind[j] of the c x d matrix in an (a x b x c) tensor.</p>
<p>Example with <code>S</code></p>
<pre><code class="lang-auto">tensor([[[[[   0,    0,    0],
           [   0,    0,    0],
           [   0,    0,    0]],

          [[1211, 1212, 1213],
           [1221, 1222, 1223],
           [1231, 1232, 1233]],

          [[1311, 1312, 1313],
           [1321, 1322, 1323],
           [1331, 1332, 1333]]],


         [[[1211, 1221, 1231],
           [1212, 1222, 1232],
           [1213, 1223, 1233]],

          [[   0,    0,    0],
           [   0,    0,    0],
           [   0,    0,    0]],

          [[2311, 2312, 2313],
           [2321, 2322, 2323],
           [2331, 2332, 2333]]],


         [[[1311, 1321, 1331],
           [1312, 1322, 1332],
           [1313, 1323, 1333]],

          [[2311, 2321, 2331],
           [2312, 2322, 2332],
           [2313, 2323, 2333]],

          [[   0,    0,    0],
           [   0,    0,    0],
           [   0,    0,    0]]]]])
</code></pre>
<p>I found that</p>
<p><code>S[:,torch.arange(3)[None,:],torch.arange(3)[:,None],ind[None,:]]</code></p>
<p>does the job for this specific case</p>
<pre><code class="lang-auto">tensor([[[[   0,    0,    0],
          [1211, 1221, 1231],
          [1312, 1322, 1332]],

         [[1231, 1232, 1233],
          [   0,    0,    0],
          [2312, 2322, 2332]],

         [[1331, 1332, 1333],
          [2311, 2312, 2313],
          [   0,    0,    0]]]])
</code></pre>
<p>does the job but I don’t get why and how broadcasting works here. My indices have all shape <code>(1,3)</code> or <code>(3,1)</code> and should broadcast to a <code>(3,3)</code> shape but the result has shape <code>(3,3,3)</code> as required. I don’t get why.</p>
<p>What would be a general solution for arbitrary <code>bs,a,b,c,d</code>?</p>
<p>T.</p>
How load neurnet? I need args and kwargs

username: MACHOCAPTCHA
cooked: <p>I need args and kwargs for load model. But how can I know these parameters? Sorry if this is a strange question. But I don’t know the answer. I am downloading a MIDAS model from this source <a href="https://github.com/isl-org/MiDaS" rel="noopener nofollow ugc">MiDaS</a>. And I save it to a file. But to get a model, I need to create a model with parameters that I don’t know. Help?!)</p>
<p>update!</p>
<p>I looked at the parameters in</p>
<pre><code class="lang-auto">class MidasNet(BaseModel):
    def __init__(self, path=None, features=256, non_negative=True):
</code></pre>
<p>And indicated them when creating the model. In this model, I want to load a saved network. the path was not copied.</p>
<p>But I got an error.</p>
<blockquote>
<p>Error (s) in loading state_dict for Model:<br>
Unexpected key (s) in state_dict: “pretrained.model.cls_token”, “pretrained.model.pos_embed”, …</p>
</blockquote>
<p>And then there are a lot of, as I understand it, network settings that I did not specify. What should I do? Can you tell me ?!)</p>

username: Martyn
cooked: <p>Hey, I’m just a programming enthusiast, but, from what I know, **kwargs usually is an optional argument passed in a dictionary form(the **dictionary is a short way of saying “for each A key in dictionary, use its values”). For example, you can use <code>function(args, dict(learning_rate=3))</code>. Usually, you won’t be using **kwargs that much and your code will still be fine with default parameters.<br>
Args is the arguments which are mandatory for that function.</p>
<p>Example: <a href="https://pytorch.org/docs/stable/generated/torch.save.html" rel="noopener nofollow ugc">In the torch.save documentation</a>, you can see that the function is <code>torch.save(obj , f , pickle_module=pickle , pickle_protocol=DEFAULT_PROTOCOL , _use_new_zipfile_serialization=True)</code>. The args, in this case, are the object and f, which you MUST pass into the function. The other arguments are optional.</p>
<p>You probably need to check the code used to load the model in your class MidasNet, so you can see how it works and which args are mandatory.</p>
<p>From what I see in both torch.save and torch.load functions docs and from my experience, the settings are already included in your state_dict(which is a dictionary with parameters and weights for your model, saved when you use torch.save())</p>
Loss modules and DistributedDataParallel

username: vadimkantorov
cooked: <p>Why are loss modules (<code>criterion</code>) often not wrapped by DDP?</p>
<p>Is it because loss modules typically have no learned parameters?</p>
<p>(and thus if they have, they should be part of DDP?)</p>

username: ptrblck
cooked: <p>The loss calculation is performed of each node and doesn’t need any communication. In case your loss needs to communicate some gradients, I guess it could be used as part of the model (and would thus be treated as any <code>nn.Module</code>).</p>
How to write pytorch code for one encoder with two decoders?

username: Dalal
cooked: <p>Hello all,<br>
I want to implement in PyTorch a generator with encoder-two decoders architecture and then pass the outputs of the decoders (images) to a discriminator. First of all, is that feasible? If yes, how to do it?<br>
Thanks in advance.</p>

username: ptrblck
cooked: <p>I don’t think I understand the proposed model architectures completely, but would claim your idea is feasible since you can combine models as you wish. As long as you don’t detach the computation graph (e.g. via rewrapping a tensor, using another library such as numpy etc.) it should work.</p>
Actor-Critic with discrete Actions

username: codeflux
cooked: <p>Hi,</p>
<p>I have a problem where the action space constists of one continous value and the other is a selection out of three options.</p>
<p>For the continious action, it is simply the output of my network (the mean value for a gaussian distribution).<br>
While I also want to use a Gaussian distribution for the policy of my discrete decision.</p>
<p>My question is now, how ´would I transform the continious output of my network, to a discrete decision?</p>
<p>I thought about calculating two boundary values, so if the output of the network is lower than bound 1, take action 1, in between bound 1 and bound 2 = action 2 and higher than bound 2 =  action3.</p>
<p>Is there better way to do this?</p>
Does embed dimemsion need to be divisible by num of heads in MultiheadAttention just because of parallel work?

username: laro
cooked: <p>When using <code>nn.</code> <code>Transformer</code> the size of</p>
<blockquote>
<p>d_model</p>
</blockquote>
<p>must be divvided by</p>
<blockquote>
<p>nhead</p>
</blockquote>
<ol>
<li>What is the reason for this restriction ?</li>
<li>Is the reason is a parallel work ? [The sequence will be split along the <code>emb_dim</code> dimension ?]</li>
</ol>

username: my3bikaht
cooked: <p>Yes, inputs to MHA are split along the hidden dimension, which is d_model due to d_model → d_model linear projection.</p>
<p>But now I wonder if having d_model as hidden dimension is too restrictive?</p>
How can I multiply a vector along all dimensions of a 3+D tensor at once?

username: ethan-r-gallup
cooked: <p>The title really says it all…<br>
How can I multiply along all dimensions of a tensor when the way code is written forces me to imply a certain order. There may be other problems that I have not encountered yet as well.</p>
<p>Thanks in advance.</p>

username: ethan-r-gallup
cooked: <p>I have found the answer.</p>
<p>you can use torch.einsum() to do pretty much anything.</p>
torch.nn.modules.module.ModuleAttributeError: 'Tudui' object has no attribute 'conv1'

username: hfyt
cooked: <h1>
<a name="dataset-1" class="anchor" href="#dataset-1"></a>利用dataset数据集进行卷积</h1>
<p>import torchvision<br>
from torch import nn<br>
from torch.nn import Conv2d<br>
from torch.utils.data import DataLoader</p>
<p>dataset_transform = torchvision.transforms.Compose([<br>
torchvision.transforms.ToTensor()<br>
])</p>
<h1>
<a name="h-2" class="anchor" href="#h-2"></a>获取数据集（标准数据集）</h1>
<p>test_data = torchvision.datasets.CIFAR10(root=“PointCloud_data/data_3”, train=False,<br>
transform=dataset_transform, download=True)</p>
<h1>
<a name="h-3" class="anchor" href="#h-3"></a>加载数据</h1>
<p>test_loader = DataLoader(dataset=test_data, batch_size=4, shuffle=True, num_workers=0, drop_last=False)</p>
<h1>
<a name="h-4" class="anchor" href="#h-4"></a>搭载神经网络：对数据集进行操作</h1>
<p>class Tudui(nn.Module):<br>
def <strong>int</strong>(self):<br>
super(Tudui, self).<strong>int</strong>()<br>
self.conv1 = Conv2d(in_channels=3, out_channels=6, kernel_size=3, stride=1, padding=0)</p>
<pre><code>def forward(self, _input):
    _output = self.conv1(_input)  # x输入到该卷积层
    return _output
</code></pre>
<h1>
<a name="h-5" class="anchor" href="#h-5"></a>声明类</h1>
<p>tudui = Tudui()</p>
<p>for data in test_loader:<br>
imgs, targets = data<br>
output = tudui(imgs)<br>
print(imgs.shape)</p>

username: ptrblck
cooked: <p>It seems you have a typo in the <code>__init__</code> function and call it <code>__int__</code>.</p>

username: hfyt
cooked: <p>Thank you for your reply</p>
Why does the indices tensor have to be Long dtype?

username: YangChenyuan
cooked: <p>Hi there, I am curious about why the indices tensor must be <code>Long (torch.int64)</code> dtype, like the indices parameter in <code>torch.gather</code> and <code>torch.take_along_dim</code>. I think it should accept all Integer dtype.</p>

username: tom
cooked: <p>There are two parts:</p>
<ul>
<li>There may actually be cases with very large tensors where you need 64 bit indexing.</li>
<li>The dtypes essentially figures into the signature (i.e. the formal parameter types) of the computational kernels, so you need to compile separate kernels for all (combinations of) argument dtypes. This is quite an issue as is with the support of many floating point types, but it would grow completely out of control if we had arbitrary index types for everything.</li>
</ul>
<p>So even if we might prefer to use 32 bit indexing if we can (which we do for selected GPU kernels because it has a large speed impact, but not on CPUs in general I guess), it would significantly grow the size of PyTorch and we’re aching from the size as is.</p>
<p>Best regards</p>
<p>Thomas</p>

username: YangChenyuan
cooked: <p>Thank you very much for your detailed explanation!</p>