akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model generalization?

username: davidlee
cooked: <p>Hi there,</p>
<p>I’m trying to check if the trained model is more generalized version than the previous models.<br>
Are there any techniques, or Tools for seeing if the model is enhanced in terms of generalization?<br>
<strong>(Model generalizability for image classification)</strong></p>
<p>Thank you in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The usual way to do this is to keep a subset of your data for validation: you train both model on a training subset of your data. After training, you evaluate both on this held out validation set to see how they generalize to this unseen data.</p>
Extracting trained model hyperparameters

username: sam101
cooked: <p>how to extract the  model hyperparameters from the save trained pytorch model(model.pth)</p>

username: Eta_C
cooked: <p><code>torch.load("model.pth")</code>.<br>
What you extract depends on what its author save.</p>

username: sam101
cooked: <p>i have used torch.save(net.state_dict()). How can i get the hyperparameters used in this model. Like optimizers</p>

username: Eta_C
cooked: <pre><code class="lang-python">torch.save(
    dict(
        model=your_state_dict,
        optimizer=your_optimizer,
        epoch=your_epoch,
        other hyperparameters,
    ),
    save_file
)
</code></pre>

username: sam101
cooked: <p>Thank you for the answer.</p>
LSTM in Pytorch: how to add/change sequence length dimension?

username: TTT
cooked: <p>Can someone help me to modify the code to add a sequence dimension? The code and description are provided <a href="https://stackoverflow.com/questions/59381695/lstm-in-pytorch-how-to-add-change-sequence-length-dimension" rel="nofollow noopener">here</a>. Thank you.</p>

username: Debasish_Jyotishi
cooked: <p>In PyTorch LSTM model, you need not/can’t specify sequence length. Here sequence length of the model depends on the input data. As in your code you have specified ‘batch_first=True’, the data dimension should be (batch size, seq length, input feature dimension ). In your case the data given to LSTM model should have a shape of (200,4,16839)  .</p>

username: TTT
cooked: <p>What would be the shape of my target label, (200, 4, 3)?<br>
I think dataloader takes care of the batch size. What shapes do I supply for the train and target to the dataloader then?</p>
Adding mask to LSTM with attention causes gradient computation exception

username: Dylan_Yung
cooked: <blockquote>
<p>def forward(self, input, hidden, encoder_outputs, mask):</p>
<pre><code>    """

    Run LSTM through 1 time step

    SHAPE REQUIREMENT

    - input: &lt;1 x batch_size x N_LETTER&gt;

    - hidden: (&lt;num_layer x batch_size x hidden_size&gt;, &lt;num_layer x batch_size x hidden_size&gt;)

    - lstm_out: &lt;1 x batch_size x N_LETTER&gt;

    """

    # Incorporate attention to LSTM input

    hidden_cat = torch.cat((hidden[0], hidden[1]), dim=2)

    # attn_weights is 1 x batch_sz x MAX_NAME_LEN

    attn_weights = F.softmax(self.attn(torch.cat((input, hidden_cat), 2)), dim=2)

    # Set all pad characters to negative infinity

    attn_weights[mask] = float('-inf')

    # Softmax to re-adjust weights so pad chars have no weight, in torch dimension correlated to name, dim=2

    attn_weights = torch.softmax(attn_weights, dim=2)

    attn_applied = torch.bmm(attn_weights.transpose(0,1),encoder_outputs.transpose(0,1)).transpose(0,1)

    attn_output = torch.cat((input, attn_applied), 2)

    attn_output = F.relu(self.attn_combine(attn_output))

    # Run LSTM

    lstm_out, hidden = self.lstm(attn_output, hidden)

    lstm_out = self.fc1(lstm_out)

    lstm_out = self.softmax(lstm_out)

    return lstm_out, hidden
</code></pre>
</blockquote>
<p>This is my forward function as you can see I’m passing in a mask that sets all characters that are pad characters in the inputs to negative infinity then I’m applying a softmax over the inputs. I read in a blog this is how you’re supposed to apply masking to attention, but when I try to compute loss I get this exception.</p>
<p>“Exception has occurred: RuntimeError<br>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 2048, 40]], which is output 0 of SoftmaxBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</p>

username: tom
cooked: <p>You cannot use indexed assignment here, as the backward of the previous operation wants to use the calculation result you are overwriting.<br>
The typical solution is to use <code>torch.where(mask, attn_weights, neginf)</code> where neginf is <code>torch.tensor(float('-inf'), device=device)</code> (I would recommend precomputing that).<br>
I’m not sure your double-softmax makes much sense to me but that’s probably my limited imagination.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Dylan_Yung
cooked: <p>Don’t I need to do the softmax again after applying the mask in order to get the new distribution of weights? Cause won’t doing batch matrix multiplication with negative infinity values really mess with it? Unless maybe the relu layer adjusts for that?</p>

username: tom
cooked: <p>Well, maybe masking things to -inf before the first softmax is a better strategy. But I wouldn’t know. In my experience, two softmaxes in a row usually are not a good idea.</p>
Is is possible to mix "user define weights" with "torch.nn.Module"

username: Ren_yf
cooked: <p>Thanks for you attention. I am not quite familiar with PyTorch and wondering is it possible to mix weights defined by my self (not weights like that in <code>torch.nn.conv2d</code>, but like <code>self.filter</code> in the below example) into <code>torch.nn.Module</code>.</p>
<p>If that is possible. I have two questions:</p>
<ol>
<li>How can I bring it to the model.parameter and bring it to cuda?</li>
<li>How can I add it to optimizer?</li>
</ol>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
<p>A sample,</p>
<pre><code class="lang-python">class myModel(torch.nn.Module):
    def __init__(self):
        super(myModel, self).__init__()
        self.filter = torch.randn(8, 4, 3, 3)
    def forward(self, x):
        return torch.nn.functional.conv2d(x, self.filter)

inputs = torch.randn(1, 4, 5, 5).cuda()
model = myModel().cuda() # it doesn't work (1)
optimizer = torch.optim.SGD([model.filter], lr=0.01, momentum=0.9) # will it work? (2)

# result
&gt;&gt;&gt; [i for i in model.parameters()]
[]
</code></pre>

username: Eta_C
cooked: <p><a href="https://pytorch.org/docs/stable/nn.html?highlight=parameter#torch.nn.Parameter" rel="nofollow noopener">torch.nn.Parameter</a></p>

username: Ren_yf
cooked: <p>Coooool! That works.</p>
<p>here’s my testing code</p>
<pre><code class="lang-python">class myModel(torch.nn.Module):
    def __init__(self):
        super(myModel, self).__init__()
        self.filter = torch.nn.Parameter(torch.randn(8, 4, 3, 3))

    def forward(self, x):
        return torch.nn.functional.conv2d(x, self.filter)

inputs = torch.randn(1, 4, 5, 5).cuda()
model = myModel().cuda()

gt = torch.randn(1, 8, 3, 3).cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

for _ in range(10):
    optimizer.zero_grad()
    result = model(inputs)
    loss = torch.nn.functional.mse_loss(result, gt)
    print(loss)
    loss.backward(retain_graph=True)
    optimizer.step()
</code></pre>
Convert MultiArray to Image(delete batch size)

username: MarcSteven
cooked: <p>I have a PyTorch model and then I convert it to CoreML model. I convert input type(multiArray) to image, the multiArray is 1x3x224x224, I delete batch size in input when convert it to Image. I have a doubt, does it influence the result?</p>

username: ptrblck
cooked: <p>Your PyTorch model should raise an error, if you try to pass the image tensor without a batch dimension. I don’t know, how CoreML handles it and you might get a better answer in their repository or discussion board, if it exists.</p>

username: MarcSteven
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> How to make the input and output  both are image in PyTorch</p>

username: MarcSteven
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi can I add scaler into the model of PyTorch, and then add preprocess_args to my mlmodel file. And then how to change the input type of multiArray to image. I want to do an image_classifier.</p>
Understanding last_epoch usage in optim.lr_scheduler.CyclicLR

username: delip
cooked: <p>I am trying to use CyclicLR but I’m confused by the documentation, which seems to stress <code>last_epoch</code> is more like <code>num_batches_seen</code> as opposed to the last epoch#. I don’t see this in other classes that inherit from <code>_LRScheduler</code>.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/3150266b3ca38e8b40ee99a704a937295fc10192.png" alt="image" data-base62-sha1="72f7eX50Zd7qx6pDLTJXDtkXmtI" width="671" height="141"></p>
<p>Following <a href="https://github.com/pytorch/pytorch/blob/8a0503b3552671a5e4b566e5a3cdb7a4d5e6e3cb/torch/optim/lr_scheduler.py#L668" rel="noopener nofollow ugc">the code</a>, it’s not clear to me if <code>last_epoch</code> here is actually the number of batches computed.</p>

username: ptrblck
cooked: <p>My best guess is that the argument was just reused to be compatible with the base class, which uses <code>self.last_epoch</code>.<br>
Since other schedulers are supposed to be called once per epoch, I assume this name was used initially.<br>
Once <code>CyclicLR</code>, which is supposed to be called after each batch, was written, the ship has sailed and this argument name was reused.</p>
<p>I haven’t worked on the implementation, so I might be completely wrong, but that’s my understanding based on the code.</p>
FFT of an image using torch.rfft

username: Petro_Key
cooked: <p>Hi, thank you always for your support.<br>
I am trying to Fourier transform an image using the torch.rfft, but I cannot guess how I can get a frequency map and retrieve the original image by ifft.<br>
The output frequency map has actually the column size half of the original map and 2 elements are real and imaginary number I suppose. How can I visualize the frequency map and calculate the energy of the image?</p>
<p>Thank you in advance:)</p>
<p>The code is as follows:<br>
from PIL import Image<br>
import torchvision.transforms.functional as TF<br>
…<br>
file_path = “test_image.jpg”<br>
image = TF.to_tensor(Image.open(file_path)).unsqueeze_(0)<br>
# The size of “image_ch” is torch.Size([128, 128])<br>
image_ch = torch.squeeze(image[:,0:1,:,:], 0)<br>
spatial_map = torch.squeeze(image_ch, 0)<br>
# The size of “fft_map” is torch.Size([128, 64, 2])<br>
fft_map = torch.rfft(spatial_map, signal_ndim=2)<br>
# The size of “spatial_map_retrieved” is torch.Size([128, 64, 2])<br>
spatial_map_retrieved = torch.ifft(fft_map, signal_ndim=2)</p>
Libtorch: different batchsize should trace different model?

username: dancedpipi
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49e173932a79da0b0748ceb7f5c527751b1dcc09.png" data-download-href="https://discuss.pytorch.org/uploads/default/49e173932a79da0b0748ceb7f5c527751b1dcc09" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49e173932a79da0b0748ceb7f5c527751b1dcc09.png" alt="image" data-base62-sha1="axzSJMp2A68wMSAVIfWUX7j6MXv" width="690" height="147" data-dominant-color="0A0909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">696×149 2.29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
If I trace the model with batchsize 1, libtorch can’t run with batchsize=2.<br>
If I trace the model with batchsize 2, libtorch can’t run with batchsize=1.</p>
<p>I have set libtorch batchsize qual to batchsize in trace?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b9527fa27b7adfba161aeff4966dfa86683d0f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/3b9527fa27b7adfba161aeff4966dfa86683d0f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b9527fa27b7adfba161aeff4966dfa86683d0f3.png" alt="image" data-base62-sha1="8v5JEyxG7L1B1o5YbE2Zodsbq15" width="689" height="173" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1292×325 4.38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Based on the error message it looks like the error is raised from a mismatch between dim1 of <code>mat1</code> and dim0 of <code>mat2</code>, while I assume dim0 in <code>mat1</code> is representing the batch size.<br>
Are you initializing these matrices manually based on the batch size?<br>
If not, could you post a minimal code snippet to reproduce this error?</p>
How to solve this error missing_keys=[]?

username: Zhaoyi-Yan
cooked: <p><strong>IncompatibleKeys(missing_keys=[], unexpected_keys=[])</strong></p>
<p>When I tried to use IBN-net as an feature extractor, I met an error.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/XingangPan/IBN-Net" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/13579537?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/XingangPan/IBN-Net" target="_blank" rel="nofollow noopener">XingangPan/IBN-Net</a></h3>

<p>Instance-Batch Normalization Networks (ECCV2018). Contribute to XingangPan/IBN-Net development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<pre><code class="lang-python">class LargeModel(nn.Module):
    def __init__(self, pretrain=True):
        super(LargeModel, self).__init__()

        # define ibn_model and initilize it with pretrained model.
        IBN_model_name = 'resnet50_ibn_a.pth.tar'
        IBN_model = resnet50_ibn_a(pretrained=False)
        pretrained_model_weight = torch.load(IBN_model_name)['state_dict']
        new_state_dict = OrderedDict()
        for k, v in pretrained_model_weight.items():
            name = k[7:] # remove `module.`
            new_state_dict[name] = v
     #   for k, v in new_state_dict.items():
     #       print(k)
     #   print('----------')
     #   for k in IBN_model.state_dict().keys():
     #       print(k)

        self.ibn_res50 = IBN_model.load_state_dict(new_state_dict)
        print(self.ibn_res50)
         # Define Extra layers and initilize them with uniform distribution
</code></pre>
<p>When I try to print <code>self.ibn_res50</code>, it gives such error.<br>
In fact, when I try to print <code>new_state_dict</code> and <code>IBN_model.state_dict().keys()</code>, both of them return the same keys.</p>

username: ptrblck
cooked: <p>This is not an error, since no incompatible keys were found.<br>
Since this message was quite misleading, it was removed already and you shouldn’t see it in the current stable release (<code>1.3.1</code>).</p>
How to train each batch to have the same class

username: 94Andrew
cooked: <p>I want to train each batch to have the same class. For example, when batch size is 32, update once with 32 data of class 0, and then learn with 32 data of randomly selected class.<br>
If you learn all data in class 0 and then learn about class 1, it can’t be learning.</p>

username: Eta_C
cooked: <aside class="quote" data-post="1" data-topic="65198">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mika_spilner/48/19158_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/load-the-same-number-of-data-per-class/65198">Load the same number of data per class</a> 
  </div>
  <blockquote>
    I know weighted sampler can solve imbalanced data problem. 
However, I wonder is there a way to load exactly the same number of data per class ? 
What I need now is, for example, a batch of 10 samples from class A, 10 from class B, 10 from class C, ETC…( I mean “not probablistically” but deterministically make sure to load 10 sample per class. ) 
I also want to know how to combine that solution with torch.utils.data.DataLoader
  </blockquote>
</aside>

<aside class="quote quote-modified" data-post="2" data-topic="65198">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/load-the-same-number-of-data-per-class/65198/2">Load the same number of data per class</a> 
  </div>
  <blockquote>
    I had the same problem when I re-implement <a href="https://arxiv.org/abs/1904.09658" rel="noopener nofollow ugc">PFE</a>, which needs 64 classes, which have 4  data in each class, as an input. 
Here’s my solution (not very good but useful for me). 
#class Dataset
def __init__(self):
    self.label_to_index = {'A': [1, 3, 4], 'B': [0, 2, 5], 'C': [6, 7, 8]}

def __getitem__(self, item):
    images = np.empty((1, *image_shape))
    labels = np.empty((1,))
    for i, (label, indices) in enumerate(self.label_to_index.items()):
        index = np.random.choice(indices, 1, …
  </blockquote>
</aside>


username: Mika_Spilner
cooked: <p>You can make custom Sampler based on <code>torch.utils.data.WeightedRandomSampler</code>.<br>
I think you can set a class probability as 1 and rest of the classes 0, then in next iteration you can choose another class and set its probability 1 and rests 0, and so on.</p>
Why training loss floats so much

username: DoubtWang
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21d7a88e59627c1633238a70ca8b6f6c7c932607.png" data-download-href="https://discuss.pytorch.org/uploads/default/21d7a88e59627c1633238a70ca8b6f6c7c932607" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21d7a88e59627c1633238a70ca8b6f6c7c932607_2_690x270.png" alt="image" data-base62-sha1="4PnNL3hfor16jTbPWnwK4pnWrWf" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21d7a88e59627c1633238a70ca8b6f6c7c932607_2_690x270.png, https://discuss.pytorch.org/uploads/default/original/3X/2/1/21d7a88e59627c1633238a70ca8b6f6c7c932607.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/1/21d7a88e59627c1633238a70ca8b6f6c7c932607.png 2x" data-dominant-color="FDEEE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1007×395 49.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Smoothing=0.65<br>
batch size=40<br>
gpu num=2<br>
learning rate = 3e-5<br>
The model  is based on bert and is used to complete simple binary classification task.</p>
<p>In previous experiments, I set batch size to 16,  and the situation is similar.<br>
Does anyone have a good suggestion to change this situation?</p>
Logistic Regression Tutorial without MNIST (or any other image)

username: titoniubo
cooked: <p>Hello,</p>
<p>I’d be interested to know if there is any Pytorch Logistic Regression tutorial that does not use images.</p>
<p>I would like to build a LogReg using n variables (columns) to have a muliple class output (i.e 5)</p>
<p>Thanks,</p>
<p>Josep Maria</p>

username: KFrank
cooked: <p>Hi Josep!</p>
<aside class="quote no-group" data-username="titoniubo" data-post="1" data-topic="66032" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/titoniubo/48/19669_2.png" class="avatar"> titoniubo:</div>
<blockquote>
<p>I’d be interested to know if there is any Pytorch Logistic Regression tutorial that does not use images.</p>
</blockquote>
</aside>
<p>See my response to the similar question you asked in your earlier<br>
thread:</p>
<aside class="quote quote-modified" data-post="4" data-topic="66012">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/input-size-dimensions-when-not-use-images-on-nn-for-classification/66012/4">Input size dimensions (when not use images) on nn for classification</a> 
  </div>
  <blockquote>
    Hi Josep! 

I don’t know offhand of a tutorial that is pytorch and logistic regression, 
but doesn’t use the MNIST dataset. 
But it doesn’t matter.  If you look at one of your logistic-regression 
tutorials (if it’s done right) you will see that it ignores the spatial 
structure of the MNIST image.  You just need to use your 20 variables 
in lieu of the ~1000 variable (pixels) in the MNIST image (and adjust 
the number of input variable in your model accordingly). 

Just to be clear, you will be…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
How to change the last layer(s) of a traced torchscript model

username: jhhurwitz
cooked: <p>Hello, I am wondering if it is possible to change the last layer (or more) of a loaded torchscript model?</p>
<p>This would be useful for changing the number of categories a torchscript model could predict after training again.</p>
<p>Right now I get an odd error when trying to overwrite the last layer manually.</p>
<pre><code class="lang-python">&gt;&gt; model = torch.jit.load("model_cpu.pth")
&gt;&gt; model.last_layer

RecursiveScriptModule(original_name=Linear)

&gt;&gt; new_last_layer = torch.jit.script(torch.nn.Linear(a, b))
&gt;&gt; new_last_layer

RecursiveScriptModule(original_name=Linear)

&gt;&gt; model.last_layer = new_last_layer
&gt;&gt; a.forward(torch.rand([x, y, z ... ]))

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: isTensor() INTERNAL ASSERT FAILED at ../aten/src/ATen/core/ivalue_inl.h:86, please report a bug to PyTorch. Expected Tensor but got Bool
The above operation failed in interpreter.
Traceback (most recent call last):
/Users/****/venv/lib/python3.7/site-packages/torch/nn/functional.py(1370): linear
/Users/****/venv/lib/python3.7/site-packages/torch/nn/modules/linear.py(87): forward
/Users/****/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(525): _slow_forward
/Users/****/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(539): __call__
/Users/****/*****/ops/models.py(277): forward
/Users/****/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(525): _slow_forward
/Users/****/venv/lib/python3.7/site-packages/torch/nn/modules/module.py(539): __call__
/Users/****/venv/lib/python3.7/site-packages/torch/jit/__init__.py(997): trace_module
/Users/****/venv/lib/python3.7/site-packages/torch/jit/__init__.py(858): trace
convert.py(91): main
convert.py(110): &lt;module&gt;
Serialized   File "code/__torch__/ops/models.py", line 1052
    input153 = torch.flatten(x31, 1, -1)
    input154 = torch.dropout(input153, 0.5, False)
    base_out = torch.addmm(bias52, input154, torch.t(weight105), beta=1, alpha=1)
                                             ~~~~~~~ &lt;--- HERE
    _512 = ops.prim.NumToTensor(torch.size(base_out, 1))
    input_tensor = torch.view(base_out, [-1, 8, int(_512)])
</code></pre>

username: eellison
cooked: <p><a class="mention" href="/u/jhhurwitz">@jhhurwitz</a> I don’t think there’s an easy way of doing that currently. See this issue for more context: <a href="https://github.com/pytorch/pytorch/issues/21064" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/21064</a></p>

username: jhhurwitz
cooked: <p><a class="mention" href="/u/eellison">@eellison</a> Is it on the roadmap to add an easy way to do this? Looks like the issue has stalled out a little bit.</p>
<p>In the meantime, even an example of the hard way to do this would be useful if you happened to know of any.</p>
<p>Thanks!</p>
Struggling to Understand F.grid_sample for this example

username: soulslicer
cooked: <p>I have a input tensor of size [1,32,296,400]</p>
<p>and I have a pixel set of [1, 56000, 400, 2]</p>
<p>After applying grid_sample with mode=‘bilinear’ I have [1, 32, 56000, 400]</p>
<p>Can I know what exactly happened here?</p>

username: el_samou_samou
cooked: <p>Hi,</p>
<p>All first dimensions (dim=0) correspond to the batch. Then dimension 1 of the input tensor is the number of channels. Then the rest of the input is a grid of size (296 x 400). The pixel set corresponds to another grid of size (56000 x 400). This resulting grid will be populated using information located in the input grid. So the last 2 channels are coordinates where to find the information to populate the result table.<br>
The mode=“bilinear” says that is you have non-discrete coordinates, it linearly interpolates the resulting vector (of size 32) base on the neighboring vertices in the input grid.</p>
<p>Hence you obtain a result of dim (1 x 32 x 56000 x 400) because your batch is one and you sampled a 32 x 296 x 400 grid at 56000 x 400 locations.</p>
<p>Hope this helps.</p>
Controlling whether pytorch uses nnpack or thnn?

username: solvingPuzzles
cooked: <p>Let’s say you build PyTorch with NNPACK. Now, when doing inference on an ARM CPU, how do you tell PyTorch whether to use THNN or NNPACK for tensor computations?</p>
<p>And, to head off some likely questions:</p>
<ul>
<li>Yes, I am using PyTorch on an ARM device; no, it’s not a smartphone</li>
<li>yes, I know I can also use Caffe2 on ARM devices; no my model isn’t currently compatible with onnx or caffe2</li>
</ul>

username: solvingPuzzles
cooked: <p>figured it out here: <a href="https://github.com/pytorch/pytorch/issues/30622" class="inline-onebox" rel="noopener nofollow ugc">Upgrade pytorch to use XNNPACK instead of NNPACK for android · Issue #30622 · pytorch/pytorch · GitHub</a></p>
<blockquote>
<p>Assuming that you are indeed compiling PyTorch with NNPACK enabled … you are effectively at the mercy of the the checks in aten/src/ATen/native/Convolution.cpp:_convolution() for convolutions (and equivalent checks for other operators) which do their best to pick the most efficient code path based on a multitude of conditions, the core of which usually boil down to testing whether an efficient implementation indeed exists for that particular configuration or not.  Your best bet then, would be to either put a breakpoint in that function or printf-debug it to see why a more efficient code path is not taken, and then modify your network to meet those requirements.  It might also be the case that you are not compiling with NNPACK enabled.</p>
</blockquote>
Pytorch (CPU) vs. keras(CPU) speed comparison

username: 111219
cooked: <p>I know Pytorch is faster than Keras.</p>
<p>Is the <strong>Pytorch on cpu</strong> also faster than Keras?</p>
<p>If you have information or Document of proof comparing the speed of Pytorch (cpu, gpu) and Keras, please share it.</p>
<p>Thank you.</p>

username: albanD
cooked: <p>I wouldn’t say unconditionally that “Pytorch is faster than Keras.”<br>
You would  need to specify what is the model you want to run, which device you want to run it on, how did you installed both, which backend you use for Keras etc</p>
Bitbake recipe for libtorch

username: platisd
cooked: <p>Hi all,</p>
<p>We are trying to include <code>libtorch</code> to our Yocto-based distribution. Unfortunately, compiling from source has been a hurdle. Specifically, the build system does not really play well with this CMake - Python combo. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"></p>
<p>Any chance there’s a <code>pytorch</code> or <code>libtorch</code> available by the community already?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I’m not sure, what Yocto is and if it helps, but you can download <code>libtorch</code> (with different CUDA libs) from the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">website</a> without building it from source.</p>

username: platisd
cooked: <p>Thanks for your response. Yeah, this is the way I am going for at the moment but it’d be cool to have it being built from source. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>I’m not sure I understand the issue.<br>
Would you like to build from source yourself and find the build process too tedious?</p>
<p>Based on this:</p>
<aside class="quote no-group" data-username="platisd" data-post="1" data-topic="54432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/platisd/48/15814_2.png" class="avatar"> platisd:</div>
<blockquote>
<p>Any chance there’s a <code>pytorch</code> or <code>libtorch</code> available by the community already?</p>
</blockquote>
</aside>
<p>I thought you are looking for a pre-built package.</p>

username: platisd
cooked: <p>My apologies, I took too much domain-specific (embedded Linux) knowledge for granted. Let me rephrase:<br>
I am looking for a pytorch/libtorch bitbake recipe which would compile libtorch from source and allow me to include it in our (embedded Linux) distribution.</p>
<p>Additionally, such a recipe would allow libtorch to be available across all platforms/hardware and architectures supported by the Yocto build system, thus facilitating the deployment of, pytorch, edge computing applications.</p>

username: Oli
cooked: <p>Looking for something similar</p>

username: platisd
cooked: <p>I eventually resorted in compiling libtorch myself and using it as a pre-built shared library.<br>
Of course I was lucky that our platform was an x86. I wouldn’t want to think what would happen were we using an ARM or something instead. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=9" title=":sweat:" class="emoji" alt=":sweat:"></p>
Freezing/unfreezing model before/after mounting on gpu

username: Upgrade_Yourself
cooked: <p>Must freezing/unfreezing model be done necessarily before mounting it on gpu or can it be done whenever?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Depends how you freeze your model. But I would recommend doing it last. As in general, moving to the GPU will create new Tensors and some of the metadata can be lost.</p>

username: Upgrade_Yourself
cooked: <p>Sorry <a class="mention" href="/u/alband">@albanD</a> , I didn’t quite understand. Is mounting on gpu recommended to be done last or the contrary?</p>

username: albanD
cooked: <p>Moving to the gpu is recommended to be done first.<br>
For example, it should be done before passing the parameters to the optimizer as well.</p>

username: Upgrade_Yourself
cooked: <p>I’d like to ask what you meant by “it depends on how the model is frozen”.</p>

username: albanD
cooked: <p>You can freeze the model in multiple ways:</p>
<ul>
<li>set the requires_grad field to False</li>
<li>run in no_grad mode because you don’t need gradients to flow through</li>
<li>just never call optimizer.step()<br>
etc</li>
</ul>
<p>The ones that rely on the parameters not to change might not work if you do them before sending the model to cuda. The ones that do not touch the parameters (2 and 3 above), will work either way <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Can every model on pytorch be parallelised?

username: bing
cooked: <p>I am trying to run a deep learning model for the segmentation task.<br>
I have 2 gpus.<br>
My doubt is whether can I parallelise my model or not?<br>
My batch size is 1 and can’t change it (maybe change it but I have to figure that out) because of some image transformations in the code.</p>

username: ptrblck
cooked: <p>You could most likey use <code>nn.DistributedDataParallel</code> to use both GPUs each using a batch size of 1. I assume your model fits on a single GPU and you don’t need to split it onto two devices.<br>
Would that work for you?</p>

username: bing
cooked: <p>Thanks.<br>
Yes, that would work for me.I’ll try and will post here it again if it doesn’t work.</p>

username: bing
cooked: <p>okay, I tried wrapping my head around <code>nn.DistributedDataParallel</code>, following this link -<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel</a></p>
<p>but I still not figure out how to do it as tutorial just tells to include following lines for single process multi-gpu.</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend="nccl")
model = DistributedDataParallel(model)
</code></pre>
Problems on getting the hessian vector product of a network

username: phantom90
cooked: <p>Hi, I’m trying to get a hessian vector product of a network.</p>
<p>I try to follow the hvp implemented in Tensorflow. But the following codes don’t work as expected. Does anybody know how to solve it?</p>
<p>Thank you in advance.</p>
<pre><code class="lang-auto">
input = torch.randn(1, 3, 32, 32)  # Batch_size is 1.
out = net(input).sum()  # net is a neural network

para_list = [x for x in net.parameters()]

grads = autograd.grad([out], para_list, retain_graph=True, create_graph=True)

elem_prod = [g * v for g, v in zip(grads, list_of_v)]  # Here list_of_v is a list of vector v. Each v is corresponding to a parameter in para_list.

hvps = autograd.grad(elem_prod, para_list, create_graph=True)

</code></pre>
<p>The error says:</p>
<p>RuntimeError: grad can be implicitly created only for scalar outputs.</p>
<p>If my implementation is totally wrong, what is the correct way of doing this?</p>
<p>Thanks again.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the problem is that you do an element-wise product when you do <code>g * v</code> instead of a dot product.<br>
Does <code>elem_prod = sum([(g * v).sum() for g, v in zip(grads, list_of_v)])</code> compute what you want?</p>
Torch.gather along one axis

username: Vannila
cooked: <p>This function is a bit confusing to me. Is it similar to normal indexing if we run along a single axis only?</p>
<pre><code class="lang-auto">values = torch.rand(2, 16, 4)
index = torch.tensor([3, 6, 7])
</code></pre>
<pre><code class="lang-auto">index = index.view(1, -1, 1).expand(values.shape[0], -1, values.shape[2])
values = torch.gather(values, 1, index)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">values = values[:, index, :]
</code></pre>
<p>Are they exactly the same function?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes they are. A simpler version is <code>values.index_select(1, index)</code>.</p>
Block Matmul with different column size

username: ThyrixYang
cooked: <p>Hi,</p>
<p>I have many matrix w1, w2, w3…wn with shape (k*n1, k*n2, k*n3…k*nn) and x1, x2, x3…xn with shape (n1*m, n2*m, n3*m…nn*m), I want to get w1@x1, w2@x2, w3@x3 repectively.</p>
<p>The resulting matrix is multiple k*m matrices and can be concatenated into a large matrix with shape ((k*n)*m).</p>
<p>I didn’t find any efficient solution to vectorize this, simply multiply the matrix (w1;w2; …;wn) and (x1; x2; …xn)^T will yield a matrix with shape (k*m).</p>
<p>Edit: I’m dealing with graph neural networks, where there are a different number of nodes in each graph thus the n1, n2, n3 may be different.</p>
Working with IPython.core.display.Image

username: Upgrade_Yourself
cooked: <p>I compared different methods of loading image from url. And using IPython.display.Image(url) is the fastest. So Is it possible to work with IPython.core.display.Image in dataset?</p>
<pre><code class="lang-auto">from IPython.display import Image
from torchvision.transforms import Resize
import torch 
start=time.time()
a = IPython.display.Image(url)

a = Resize(512)(a)
stop=time.time()

print("durée: {:.2} min {:.4} sec".format((stop - start) // 60,(stop - start) % 60))

</code></pre>
<p>I got this error:<br>
<code>img should be PIL Image. Got &lt;class 'IPython.core.display.Image</code></p>

username: G.M
cooked: <p>Is this error anything related to pytorch?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: Upgrade_Yourself
cooked: <p>The error was generated when using Resize transform  from torchvision</p>

username: G.M
cooked: <p>I expect a pytorch function wouldn’t expect a IPython image, try changing it into Tensor.</p>

username: Upgrade_Yourself
cooked: <p>It gives an error</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-79-f4dfad644a8d&gt; in &lt;module&gt;()
      3 import torch
      4 start=time.time()
----&gt; 5 a = torch.Tensor(IPython.display.Image(url))
      6 
      7 a = Resize(512)(a)

TypeError: new(): data must be a sequence (got Image)
</code></pre>

username: G.M
cooked: <p>I think it is preferred to use torch.tensor instead of using torch.Tensor directly. Tho i’m not familiar with IPython image, but there must some way that can convert it into 2/3 D array that can be further use to create tensors. PS: look at the <a href="https://pytorch.org/docs/" rel="nofollow noopener">Docs</a> before asking <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>

username: ptrblck
cooked: <p><code>torchvision.transforms</code> work currently with <code>PIL.Images</code> or tensors, so you would have to convert <code>IPython.display.Image</code> to a <code>PIL.Image</code> somehow to use the transformations.</p>

username: G.M
cooked: <p>thx for correcting me</p>

username: Upgrade_Yourself
cooked: <p>I see thank you for the info!</p>
Debugging loss plot

username: Upgrade_Yourself
cooked: <p>I’m using a densenet121 for multilabel image classification.<br>
Loss function is beta_score.<br>
I used iterative stratification to balance distributions of multilabel data across splits for cross validation.<br>
In the plot, the validation loss is lower than train loss.<br>
I saw this link: <a href="https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/" rel="noopener nofollow ugc">https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/</a><br>
They gave many reasons. But I’m still unsure.<br>
And I guess I also don’t know if it’s something to be corrected.</p>
<p>I saw in <a href="https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/" rel="noopener nofollow ugc">https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/</a>  that if training loss is still decreasing than it means the model is underfitting.<br>
I just want to make sure that this still holds true in my case (loss_train &gt; loss_val)<br>
This is a snapshot of the loss plot where blue is train and red is val.<br>
I smoothed the plot (0,999).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96ef34ed6e32c8e7a476a8018dfe65985126da85.png" alt="smoothed_losses_densenet121" data-base62-sha1="lxe7m8gAzctQ0cz1jEArAzvR0xv" width="433" height="240"></p>
<p>EDIT:<br>
I deleted Dropout and added more epochs. But the plot shows the peak in the beginning. And I have no idea what it means.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96624b52bb41220db6b66c5e38d3d9e6355839f4.png" alt="loss_no_dropout_more_epochs" data-base62-sha1="lsmdkpfUkHCh2oMvEgDifxEplyc" width="604" height="354"></p>

username: ptrblck
cooked: <p>Did you check the data splits between the training and validation sets?<br>
Could maybe “simpler” samples have landed in the validation set or are you using aggressive data augmentation techniques during training?</p>

username: Upgrade_Yourself
cooked: <p>I used iterative stratification to balance distributions of multilabel data across splits for cross validation. And then I chose 1 fold/5folds  to be the validation fold. So, normally there is no data leakage. Now, concerning the “simpler” samples, I have no idea. Do people normally go through the samples one by one to see?</p>
Different uses of torch.save

username: Upgrade_Yourself
cooked: <p>How can we benefit afterwards from saving both model and [args.arch, model.state_dict()] ?</p>
<pre><code class="lang-auto"># Final model
    torch.save(model, MODEL_DIR / f"final_{args.fold}.pth")
    # Failover (args + state dict)
    torch.save(
        [args.arch, model.state_dict()],
        MODEL_DIR / f"failover_{args.arch}_{args.fold}.pth"
</code></pre>

username: ptrblck
cooked: <p>I would recommend to save the <code>state_dict</code> instead of the model directly as explained <a href="https://pytorch.org/docs/stable/notes/serialization.html">here</a>.<br>
I’m not sure, what <code>args.arch</code> is defined, but I guess it would be a string to a pretrained <code>torchvision</code> model?</p>

username: Upgrade_Yourself
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="65366">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>args.arch</p>
</blockquote>
</aside>
<p>Yes it is a string of the name of the architecture, like “densenet121” for example.</p>
Is FFT also optimized by torch.backends.cudnn.enabled = True?

username: veritas
cooked: <p>Hello. I am have had an occasion to use FFTs extensively in my projects.<br>
However, I used torch.backends.cudnn.enabled=True to speed up my code.<br>
In doing so, I wonder if the Fourier Transforms are also sped up by this setting or only the convolutions.<br>
After all, the FFTs are in the cuFFT library, not the cuDNN library.<br>
However, I would be most grateful to know if this is so.<br>
This is because the data being Fourier transformed is of different sizes while the image data is obtained by cropping the Fourier Transformed data to a fixed size.<br>
Thank you in advance for anyone who can help out.</p>

username: ptrblck
cooked: <p>cuFFT should be called by <a href="https://github.com/pytorch/pytorch/blob/a561a8448b6b7bb8f6068178fe7bc9e2bf348304/aten/src/ATen/native/SpectralOps.cpp#L22"><code>_fft</code></a>, if you are running your code on a GPU.</p>
<p>EDIT: Sorry, I misunderstood the question (or rather skipped the cudnn part).<br>
The cuFFT methods should not depend on cudnn, as you already mentioned. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: veritas
cooked: <p>Thank you! You have saved me from quite a lot of worry.</p>
Aggregating Dataloader : how?

username: elTaan
cooked: <p>Hello,</p>
<p>I am trying to find a proper solution to create a custom DataLoader able to handle multiple datasets with a specific strategy when yielding data.</p>
<p>More specifically, given a list of Datasets, I want the loader to take all of them in account and consider that a batch of data is actually a concatenation of a batch of each of those datasets.<br>
Ideally I would like to handle samplers for each of those datasets too.</p>
<p>The problem is that a Dataloader only consider one dataset, and writing a custom Dataset doesn’t allow me to implement the fetching strategy that I want (batch/sampling) because this logic is in the Dataloader.</p>
<p>How could I pull through this issue ? Technically, rewriting another Dataloader may work but it’s kind of low-level so not the preferred solution at all.</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
RNN with truncated BPTT

username: Gilles_Jack
cooked: <p>I implemented a basic RNN network from scratch on MNIST. The sequence X1,X2,…X28 is made of the 28 rows of a MNIST digit. The features are the 28 points in a row.</p>
<p>It works fine. Now, I would like to add the truncated backpropagation through time feature to it. I know detach() should be used somewhere but I can’t figure out where and how.</p>
<p>I am pretty sure it should be in the forward proparagation function so please find this function below :</p>
<pre><code class="lang-auto">from torch import tanh, softmax, matmul as m
...
Wxh = torch.empty(i_r, h_r, device=device, requires_grad=True)
Whh = torch.empty(h_r, h_r, device=device, requires_grad=True)
Why = torch.empty(h_r, o_r, device=device, requires_grad=True)
Bh = torch.empty(h_r, device=device, requires_grad=True)
Bhy = torch.empty(o_r, device=device, requires_grad=True)
pars = [Wxh, Whh, Why, Bh, Bhy]
for par in pars:
    torch.nn.init.normal_(par, mean=0, std=std)
...
def forward_prop(X):
    X = X.permute(1, 0, 2)    # (seq, batch, feature)
    ss, bs, _ = X.shape
    H = torch.zeros(bs, h_r, device=device)
    for k in range(ss):
        H = tanh(m(H, Whh) + m(X[k], Wxh) + Bh)
    Z = m(H, Why) + Bhy
    Y = softmax(Z, dim=1)
    return Y
</code></pre>
<p>How should I modify forward_prop() to add truncated BPTT?<br>
Thanks in advance</p>
Load parameters only from an entirely saved model

username: xinqi
cooked: <p>Hi, I have trained a neural network for Purpose 1 and I mistakenly use <code>torch.save(model, PATH)</code>, which saves the entire model rather than only weights.</p>
<p>As the performance is very good, I would like to plug this model into another model to assist Purpose 2…</p>
<p>I have created the exact model object…<br>
Then, I try to load the parameters by</p>
<pre><code class="lang-auto">checkpoint = torch.load("./models/xxxnet_weights.pk", map_location=torch.device('cpu'))
model.load_state_dict(checkpoint['model_state_dict'])
</code></pre>
<p>But, the <code>torch.load</code> gives an error like this</p>
<pre><code class="lang-auto">AttributeError: Can't get attribute 'xxxNet' on &lt;module 'models' from '/home/xxx/Projects/xxxx/models/__init__.py'&gt;
</code></pre>
<p>(As stated in the official tutorial <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict</a>) I suspect this is due to the <code>torch.save</code> saves the directory information, which is different now…</p>
<p>Is there a way that I can still load these parameters?</p>
<p>Many thanks</p>

username: xinqi
cooked: <p>I think I can provide a solution for myself.</p>
<p>Load the model in the original project folder, save using <code>torch.save(model.state_dict(), PATH)</code> with parameters only again to another checkpoint. Then, load this new one in the new project.</p>