akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Copy.deepcopy or clone() cause UserWarning

username: Y_Y
cooked: <p>Hi all,<br>
I got the error message <code>opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/ATen/native/cudnn/RNN.cpp:1266: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage.</code> when I tried to run the following code to store the weights inside optimizer class.</p>
<pre><code class="lang-auto">for p in group['params']:
    # p_data = p.data.clone() 
    p_data = copy.deepcopy(p.data)
</code></pre>
<p>I noticed people said this message is related to <code>nn.Parallel</code>, but I am not using parallel run. I tested if I remove clone or deepcopy, there won’t be warning message. Do you know how to fix this?<br>
Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Have you tried using <code>module.flatten_parameters()</code> on your RNN module to make sure the weights are nicely layed out in memory again?</p>

username: Y_Y
cooked: <p>Hi albanD,<br>
Thank you. I created this module by combining <code>nn.RNN</code> and <code>nn.Linear</code>, say I name it rnn_model(). But <code>rnn_model.flatten_parameters()</code>  gives object has no attribute ‘flatten_parameters’. Where should we call <code>flatten_parameters</code> to make sure the weights are nicely laid out in memory again in this case? Thank you for your time.</p>

username: albanD
cooked: <p>Hey,</p>
<p>I am not sure, that depends on your network structure.<br>
Is your rnn_model wrapped into a Sequential or some container? Make sure to call that method on the rnn model itself.</p>
How to construct plot for loss

username: MLCharlotte
cooked: <p>I am trying to record the loss over 5 epochs, I have tried to store the value of the loss variable in a list for plotting but I get an error. This is weird to me because I am able to print the value of the loss variable after each step but I cannot append those results into a new list.</p>
<p>Here is my training cell:</p>
<blockquote>
<p>encoder.train()<br>
decoder.train()<br>
loss1 = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
total_step = len(train_loader)<br>
for epoch in range(num_epochs):<br>
for i, (images, captions, lengths) in enumerate(train_loader):</p>
</blockquote>
<pre><code>    # Set mini-batch dataset
    images = images.to(device)
    captions = captions.to(device)

    # Packed as well as we'll compare to the decoder outputs
    targets = pack_padded_sequence(captions, lengths, batch_first=True)[0]

    # Forward, backward and optimize
    features = encoder(images)
    outputs = decoder(features, captions, lengths)

    loss = criterion(outputs, targets)
    
    # Zero gradients for both networks
    decoder.zero_grad()
    encoder.zero_grad()

    loss.backward()
    optimizer.step()

    # Print log info
    if i % log_step == 0:
        print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
              .format(epoch, num_epochs, i, total_step, loss.item())) 

   
loss1.append(loss.item)
</code></pre>

username: Tapan_Patel
cooked: <p>The last <code>loss1.append(loss.item)</code> should be<code> loss1.append(loss.item())</code>.</p>
Apply 1D CNN to time-series data + Temporal Adaptive Normalization

username: hfdp
cooked: <p>Hello,<br>
I am quite new to python/pytorch and I would like to implement a ‘Temporal Adaptive Batch Normalization’ as described in the picture.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8042cf199a0ac137d9f691bc243cceb1e939ca2.png" data-download-href="https://discuss.pytorch.org/uploads/default/b8042cf199a0ac137d9f691bc243cceb1e939ca2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8042cf199a0ac137d9f691bc243cceb1e939ca2_2_690x402.png" alt="image" data-base62-sha1="qfSNDK9LiDHH5Og9iwEkGDtJz0e" width="690" height="402" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8042cf199a0ac137d9f691bc243cceb1e939ca2_2_690x402.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8042cf199a0ac137d9f691bc243cceb1e939ca2_2_1035x603.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8042cf199a0ac137d9f691bc243cceb1e939ca2.png 2x" data-dominant-color="D1D8E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1174×685 208 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can someone help me? This is an example of a code below:</p>
<p>import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(model, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv1d(1, 18, kernel_size=3)<br>
self.conv2 = nn.Conv1d(18, 36, kernel_size=3)<br>
self.conv2_drop = nn.Dropout2d() <span class="hashtag-raw">#dropout</span><br>
self.fc1 = nn.Linear(1536, 72) <span class="hashtag-raw">#Fully-connected</span> classifier layer<br>
self.fc2 = nn.Linear(72, 19) <span class="hashtag-raw">#Fully-connected</span> classifier layer</p>
<pre><code>def forward(self, x):
    x = x.unsqueeze(1) # create an additional channel dimension (only 1 channel here)
    x = F.relu(F.max_pool1d(self.conv1(x), 2))
    print(x.shape)
    x = F.relu(F.max_pool1d(self.conv2_drop(self.conv2(x)),2))
    print(x.shape)
    return x
</code></pre>
Efficient operation for subtraction all rows of one matrix from the other?

username: dvirginz
cooked: <p>I have two vectors</p>
<pre><code class="lang-auto">a.shape = [4,6000,1000]
b.shape = [4,6000,1000]
</code></pre>
<p>And I would like to preform subtraction between all row-pairs.<br>
I.e for</p>
<pre><code class="lang-auto">[[1,2,3]     -  [[3,4,5]
 [3,4,5]]        [6,7,8]]

[[-2,-2,-2],[-5,-5,-5]
 [0,0,0],[-3,-3,-3]]
</code></pre>
<p>Using unsqueeze I was able to do so, with the following scheme:<br>
<code>a.unsqueeze(1) - b.unsqueeze(2)</code><br>
And it works as expected.<br>
Unfortunately, with the specified shapes, I have<br>
<code>CUDA out of memory. Tried to allocate 724.37 GiB</code></p>
<p>Is there a way to do it, even if slower, in a vectorized manner?</p>

username: LeviViana
cooked: <p>If <code>a</code> and <code>b</code> are of the shape  <code>(k,n,m)</code> then the shape of your output is <code>(k, n, n, m)</code>. This means that, if you are using <code>int32</code>, you’ll need to allocate at least <code>4*k*n*n*m</code> bytes. In your case, it counts up to <code>4 * 4 * 6000 * 6000 * 1000</code> bytes, or <strong>536 Gb</strong>, it seems a little too much for me. So maybe you should implement some lazy calculation of your output in order to fit it in any modern GPU.</p>
Imbalance data and custom loss function

username: netaglazer
cooked: <p>hi, I created a custom loss function:</p>
<pre><code class="lang-auto">def new_loss_function(p, y, b):
    eps = 0.000000000000000000000000001
    losses = 0
    k = len(y[0])
    ones = torch.ones(1, 1).expand(b, k).cuda()
    loss1 = -((ones-y)*(((ones-p)+(eps)).log())).sum(dim=1)
    prod = (ones-y)*k - y*((p+ eps).log())
    loss2 = torch.min(prod, dim=1)[0]
    losses = (loss1 + loss2).sum()
    return losses / b
</code></pre>
<p>and I have imbalanced data. how can I use my class weights vector in this case?<br>
thanks</p>

username: netaglazer
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> maybe you can help ?</p>
Compare hyperparameters in Tensorboard

username: whoab
cooked: <p>I want to be able to group runs by hyperparameter. For example, I should be able to select “lr” in tensorboard and see the plot for the loss where each line is the mean of all runs with the same learning rate (lr).</p>
<p>This seems to be a very common way of analyzing training plots for hyperparameters, but it’s missing.</p>
<p>*** You could potentially do this with regex and putting the hyperparameter names as the run directory name… but a folder name is limited at 255 characters!!! It’s not a good solution.</p>

username: whoab
cooked: <p>Bumping this. I think people may have hyperparameter tuning pipelines that must work better than putting the parameters in the folder titles and regexing in tensorboard?</p>
Deploying Pytorch model to production in Mobile

username: braindotai
cooked: <p>Hi there, I’m completely new to the mobile app development and would like to deploy my model trained using pytorch to both android and ios. Now I know one way for this is to use the <strong>Pytorch Mobile</strong>, but for that, I’d have to build my app with different code bases for both android and ios (which I’m not worried that much though). I’m just wondering is there a way for deploying pytorch models for both android and ios using the same codebase.</p>
<p>Hence I’m really interested in Flutter, but there’s no as such official way of deploying a pytorch model to flutter. (Some packages are there but they aren’t updated since like last year)</p>
<p>One way I can think of is by deploying my model using Flask or Django as a REST API, and then use the JSON file to in order to connect it to Flutter. One con of this approach is that model wouldn’t run on the device, so I’d have to deploy the API to the server, which of course could cost money for better inference.</p>
<p>So is there any better way for the same. Again, I’ve no problem for learning different things for different os, just wondering if such an approach exists.</p>
Data parallelism error for pretrained model

username: SU801T
cooked: <p>Hi,</p>
<p>I am trying to run my model on multiple GPUs for data parallelism but receiving this error:</p>
<pre><code class="lang-auto">AttributeError: 'DataParallel' object has no attribute 'fc'
</code></pre>
<p>I have defined the following pretrained model :</p>
<pre><code class="lang-auto">def resnet50(num_classes, device):
    model = models.resnet50(pretrained=True)
    model = torch.nn.DataParallel(model)
    for p in model.parameters():
        p.requires_grad = False
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, num_classes)

    model = model.to(device)
    return model
</code></pre>
<p>It’s unclear to me where I can add “module”</p>
<p>Cheers</p>

username: mrshenli
cooked: <p>If you are trying to access the <code>fc</code> layer in the resnet50 wrapped by the DataParallel model, you can use <code>model.module.fc</code>, as <code>DataParallel</code> stores the provided model as <code>self.module</code>:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L131" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L131" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L131</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="121" style="counter-reset: li-counter 120 ;">
<li>        self.module = module</li>
<li>        self.device_ids = []</li>
<li>        return</li>
<li>
</li>
<li>    if device_ids is None:</li>
<li>        device_ids = list(range(torch.cuda.device_count()))</li>
<li>    if output_device is None:</li>
<li>        output_device = device_ids[0]</li>
<li>
</li>
<li>    self.dim = dim</li>
<li class="selected">    self.module = module</li>
<li>    self.device_ids = list(map(lambda x: _get_device_index(x, True), device_ids))</li>
<li>    self.output_device = _get_device_index(output_device, True)</li>
<li>    self.src_device_obj = torch.device("cuda:{}".format(self.device_ids[0]))</li>
<li>
</li>
<li>    _check_balance(self.device_ids)</li>
<li>
</li>
<li>    if len(self.device_ids) == 1:</li>
<li>        self.module.cuda(device_ids[0])</li>
<li>
</li>
<li>def forward(self, *inputs, **kwargs):</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: SU801T
cooked: <p>Great, thanks. I added .module to everything before .fc including the optimizer.</p>
Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))

username: IrfanUmar
cooked: <p>I am using <code>models.vgg16(pretrained=True)</code> model for image classification, where number of classes = 3.</p>
<p>Batch size is 12 <code>trainloader = torch.utils.data.DataLoader(train_data, batch_size=12, shuffle=True)</code> since error says <code>Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))</code></p>
<p>I have changed last fc layer parameters and got last FC layer as  <code>Linear(in_features=1000, out_features=3, bias=True)</code></p>
<p>Loss function is <code>BCEWithLogitsLoss()</code></p>
<pre><code class="lang-python">criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(vgg16.parameters(), lr=0.001, momentum=0.9)
</code></pre>
<p>Training code is</p>
<pre><code class="lang-python">        # zero the parameter gradients
        optimizer.zero_grad()
        outputs = vgg16(inputs)               #----&gt; forward pass
        loss = criterion(outputs, labels)   #----&gt; compute loss
        loss.backward()                     #----&gt; backward pass
        optimizer.step()                    #----&gt; weights update
</code></pre>
<p>While computing loss, I get this error <code>Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))</code></p>

username: ptrblck
cooked: <p>Your output looks like a an output for</p>
<ul>
<li>a multi-class classification use case, where you could keep the shapes and use <code>nn.CrossEntropyLoss</code> as the criterion (if <code>target</code> contains the class indices)</li>
<li>or a multi-label classification (zero, one or more samples can be valid in each sample), in which case you could keep the criterion, but would have to provide a target tensor in the same shape as the model output.</li>
</ul>
Smart indexing question

username: Zhenya_Brichkova
cooked: <p>Hi, guys,</p>
<p>Let us have two tensors A and B. I want to obtain the tensor C of indices, containing the indices in A for the elements in B. The shape of C is the same as B.</p>
<p>For instance:</p>
<pre><code class="lang-auto">A = t.Tensor([1,2])
B = t.Tensor([[1,1,2],[2,2,1]])
C = (A == B)
</code></pre>
<p>I expect the result to be:</p>
<pre><code class="lang-auto">C = t.Tensor([[0,0,1],[1,1,0]])
</code></pre>

username: acarlier
cooked: <p>Hey Zhenya,</p>
<p>I think something like this will do:</p>
<pre><code class="lang-auto">A = t.Tensor([1,2])
B = t.Tensor([[1,1,2],[2,2,1]])

C = torch.max(A.view(-1, 1, 1) == B.unsqueeze(0), dim=0).indices
</code></pre>
<p>Tell if that helped <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Zhenya_Brichkova
cooked: <p>It helped, thank you!</p>
0 Memory allocated on GPU

username: John_Minou
cooked: <p>Hello,</p>
<p>I am doing the Udemy course on Pytorch and trying to run the MNIST example on the GPU. Even though I activated CUDA, training is slow and GPU usage is at 0%.</p>
<p>Here is my code :</p>
<pre><code class="lang-auto">device = 'cuda'
model.to(device);
</code></pre>
<pre><code class="lang-auto"># number of epochs to train the model
n_epochs = 30  # suggest training between 20-50 epochs


model.train() # prep model for training

for epoch in range(n_epochs):
    # monitor training loss
    train_loss = 0.0
    
    ###################
    # train the model #
    ###################
    for data, target in train_loader:
        data, target = data.to(device), target.to(device)
        start = time.time()
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # calculate the loss
        loss = criterion(output, target)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()

        # update running training loss
        train_loss += loss.item()*data.size(0)
    print(f"Device = {device}; Time per batch: {(time.time() - start)/3:.3f} seconds")
    print(' ')    
    # print training statistics 
    # calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.sampler)

    print('Epoch: {} \tTraining Loss: {:.6f}'.format(
        epoch+1, 
        train_loss
        ))
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')
    print(' ')
</code></pre>
<p>Should I add other lines of code? Thank you</p>

username: ptrblck
cooked: <p>Could you return the <code>round</code> operation and check the memory again?<br>
If the GPU is not being used, then</p>
<pre><code class="lang-python">device = 'cuda'
model.to(device)
</code></pre>
<p>should raise an error.</p>
<p>For small models and workloads, you might see a low GPU utilization.</p>

username: John_Minou
cooked: <p>Thank you for your answer, as you said for small models there isn’t a big difference.</p>
Time Series prediction help

username: 1337mathster
cooked: <p>I wish to predict values for an arbitrary function.  ( Like sinx + sin2x). I’ve searched a lot for tutorials but can find any that explains how to code up such a network in pytorch. I have a decent understanding of the concept behind RNNs and LSTMs.</p>
<p>If anyone can direct me to a place where a simple tutorial or lecture is available I’d greatly appreciate it.</p>
Torchtext - Loading vectors from zip archive

username: 1991sig
cooked: <p>I am working on a remote server where there is a large collection of different pre-trained word embedding vectors available in zip archives, on a shared location.</p>
<p>As my personal storage space is limited, I cannot store much more than one pre-trained model on my personal user at a time.</p>
<p>My problem is as follows:<br>
I would like to access the zip archives on the shared space at the server, and load them into torchtext without having to unzip them and store them in my personal storage.</p>
<p>I currently do:</p>
<pre><code class="lang-auto">import zipfile
import torchtext

target = "/path/to/shared/storage/model.zip"
with zipfile.ZipFile(target, "r") as archive:
    archive.extract("model.txt")

vectors = torchtext.vocab.Vectors("model.txt")
</code></pre>
<p>I would like to do something along the lines of:</p>
<pre><code class="lang-auto">import zipfile
import torchtext

target = "/path/to/shared/storage/model.zip"
with zipfile.ZipFile(target, "r") as archive:
    vectors = torchtext.vocab.Vectors(archive.open("model.txt"))
</code></pre>
<p>(this example does not work of course, since <code>torchtext.vocab.Vectors</code> expects a path to <code>model.txt</code>)</p>
<p>Is there a way to achieve this in torchtext?</p>
<p>I am grateful for any suggestions you may have</p>
How to set this loss?

username: smani
cooked: <p>I am not sure whether this is the appropriate place to ask.</p>
<p>My loss has two parts, say L1 and L2. I want to minimize both, and at the same time I need to constrain that L1 should be always greater than L2 (L1&gt;L2). Is the following correct?</p>
<p>loss = L2 - L1</p>
<p>Thank you in advance.</p>

username: BarkenBark
cooked: <p>By minimizing loss = L2 - L1 you are only really optimizing for making L1 greater than L2 rather than minimizing both loss components. For example, parameters such that (L1=1000, L2 = 990) is just as good of a solution as parameters such that (L1=10, L2=0).</p>
<p>Another loss function which is more in line with your problem description could be:<br>
loss = L1 + L2 + H(L1, L2) where H is a very large number if L2&gt;L1 and 0 otherwise.<br>
To make H differentiable, you can approximate it with a continuous function taking the difference L1-L2 which increases rapidly at 0 (note that you are relaxing the inequality constrain by doing so).</p>
<p>I am not entirely sure about this reasoning though. Please correct me if I’m wrong.</p>

username: googlebot
cooked: <p>You could downscale L2 like so: loss=L1+L2*min(1, (L1/L2).detach().cpu().item()), assuming L1 and L2 have same sign. Kinda weird optimization constraint though…</p>
Concatenate two tensors

username: Tapan_Patel
cooked: <p>I have two tensors say x and y:</p>
<p>x has shape:  [21314, 3, 128, 128]<br>
y has shape:  [21314]</p>
<p>Can I get new tensor of shape : [[21314, 3, 128, 128], [21314]], basically of shape 2</p>
Converting intermediate variable to ONNX graph input

username: s1ddok
cooked: <p>Hello!</p>
<p>I’m currently trying to export a model to ONNX from PyTorch. I have some layers inside a model that are not supported by the ONNX and I don’t really want them to be inside ONNX graph anyway, so I’m trying to use <code>keep_initializers_as_inputs=True</code> in order to fool tracer that my variables are actually inputs and inside custom nn.Module implementation’s <code>forward</code>, I’ve added this:</p>
<pre><code class="lang-auto">        if onnx.is_in_onnx_export():
            gamma = torch.zeros([int(x.shape[2]), int(x.shape[3])]).clone().detach().requires_grad_(True)
            beta = torch.zeros([int(x.shape[2]), int(x.shape[3])]).clone().detach().requires_grad_(True)
        else:
</code></pre>
<p>but those initializers aren’t getting traced as graph inputs, but rather as constants and then being baked into a graph. I’ve tried various approach but all of them failed. Can somebody hint on how can one convert intermediate variables to graph inputs?</p>
Rotations and bounding box transformation for 3d voxels

username: chinmay5
cooked: <p>If I am performing a rotation on the voxel grid (C, W, H, D) and the associated bounding box is (N,6) where N is the number of instances, how would I handle the transforms for rotating the scene about the Height ie dim=2. It seems a simple rotation of the scene is not too complex and one can use <code>permute()</code> but I am not sure how to handle the changes in the Bounding boxes for the same</p>
"attaching an optimizer" to the model

username: John_J_Watson
cooked: <p>I am quite new to pytorch and I am looking at this piece of code:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/WillKoehrsen/pytorch_challenge/blob/master/Transfer%20Learning%20in%20PyTorch.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/WillKoehrsen/pytorch_challenge/blob/master/Transfer%20Learning%20in%20PyTorch.ipynb" target="_blank" rel="nofollow noopener">WillKoehrsen/pytorch_challenge/blob/master/Transfer Learning in PyTorch.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction: Transfer Learning in PyTorch\n",
    "\n",
    "In this notebook, we'll see how to use PyTorch to train a classifier to identify 101 different objects in images. Through this project, we'll get familiar with the basics of transfer learning, PyTorch, convolutional neural networks, and inference. \n",
    "\n",
    "## Dataset: CalTech 101\n",
    "\n",
    "The dataset is the [CalTech 101 collection](http://www.vision.caltech.edu/Image_Datasets/Caltech101/) of images from 101 categories. There are about 40 to 800 images in each category. I've reduced the number of categories to 100 and arranged them in three sets: train, valid, test with a split of 50%, 25%, and 25%. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
</code></pre>

  This file has been truncated. <a href="https://github.com/WillKoehrsen/pytorch_challenge/blob/master/Transfer%20Learning%20in%20PyTorch.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I see that the author does this (see in In[30] for example):</p>
<pre><code># Attach the optimizer
model.optimizer = optimizer
</code></pre>
<p>What is the point of doing this? The examples shown on PyTorch’s website for training, never uses this <code>attaching</code> concept.</p>
<p>Am perplexed as to why one needs this. Any pointers would be useful…</p>
How to concatenate a tensor WITHIN axis=1?

username: Umair_Javaid
cooked: <p>I have a tensor of shape<code> (2,2,2,2)</code>:</p>
<pre><code class="lang-auto">tensor([[[[   5.,    5.],
          [   5.,    5.]],

         [[  10.,   10.],
          [  10.,   10.]]],


        [[[ 100.,  100.],
          [ 100.,  100.]],

         [[1000., 1000.],
          [1000., 1000.]]]], device='cuda:0')
</code></pre>
<p>I want to transform it such that the tensor along axis=1 are repeated 3 times. And after applying <code>.view(-1)</code> to that I get a 1D resultant tensor as:</p>
<pre><code class="lang-auto">tensor([   5.,    5.,    5.,    5.,   5.,    5.,    5.,    5.,   5.,    5.,    5.,    5.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,  100.,  100.,  100.,  100.,  100.,  100., 100.,  100.,  100.,  100., 100.,  100.,  100.,  100.,  100.,  100., 1000., 1000., 1000., 1000.  1000., 1000., 1000., 1000. 1000., 1000., 1000., 1000. 1000., 1000., 1000., 1000.], device='cuda:0')
</code></pre>
<p>How to do this?</p>
user_suspended: yes

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">x = x.repeat(1, 1, 3, 1)
print(x.view(-1))
&gt; tensor([   5.,    5.,    5.,    5.,    5.,    5.,    5.,    5.,    5.,    5.,
           5.,    5.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,
          10.,   10.,   10.,   10.,  100.,  100.,  100.,  100.,  100.,  100.,
         100.,  100.,  100.,  100.,  100.,  100., 1000., 1000., 1000., 1000.,
        1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.])
</code></pre>

username: Umair_Javaid
cooked: <p>Thank you so much. Your solution works fine but for a larger tensor such as <code>(50,512,224,224)</code> my system crashes(All available ram used). Is there a  time efficient work around that?</p>
user_suspended: yes

username: ptrblck
cooked: <p>I don’t think so.<br>
Even if you could create a tensor containing single elements so that you could use <code>expand</code>, you would still have to call <code>contiguous()</code> before the <code>view()</code> call, which would increase the memory again:</p>
<pre><code class="lang-python">x = x.expand(-1, -1, 6, 2)
print(x.contiguous().view(-1))
</code></pre>

username: Umair_Javaid
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="79049">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>x = x.expand(-1, -1, 6, 2)</p>
</blockquote>
</aside>
<p>RuntimeError: The expanded size of the tensor (6) must match the existing size (2) at non-singleton dimension 2.  Target sizes: [-1, -1, 6, 2].  Tensor sizes: [2, 2, 2, 2]<br>
getting this error when using <code>x = x.expand(-1, -1, 6, 2)</code></p>
user_suspended: yes

username: ptrblck
cooked: <p>As explained, for this you would need to create a tensor containing a single element for each value.<br>
But even then, the <code>contiguous</code> call will use the memory.</p>
Applying linear layer but with weights transposed?

username: Sam_Lerman
cooked: <p>I want to reuse the weights in my linear layer but with the weights transposed. Unfortunately, it seems that using a functional linear layer and passing in the weights during the forward call of my module is inefficient. Is there another way to do it?</p>
<p>Specifically, what I’ve done so far looks like:</p>
<pre><code class="lang-auto">class Bla(nn.Module):
    def __init__(self, args):
        super(Bla, self).__init__()
        self.fc = nn.Linear(100, 256) 

    def forward(x):
        return F.linear(x, self.fc.weight)  #  F.linear appears slow
</code></pre>
<p>I tried einsum as well (since x is a batch of items of shape self.fc.weight.shape each) but didn’t get a performance improvement.</p>
<p>Note: I still want gradients to propagate through this operation! And I don’t want to use self.fc’s biases.</p>

username: ptrblck
cooked: <p>The <code>nn.Linear</code> module calls into <code>F.linear</code> in its <code>forward</code> pass as seen <a href="https://github.com/pytorch/pytorch/blob/69e2f1aaff9614bd801b34b0c9b4cc8305ee9a61/torch/nn/modules/linear.py#L87">here</a>.<br>
How did you profile it to come to the conclusion the functional call is slower than the modules?<br>
Usually all modules call into their functional equivalent.</p>

username: Sam_Lerman
cooked: <p>You’re right. I was too quick to post this. The slowdown was coming from something else. Thanks!</p>
Best way to load sparse data in Pytorch

username: Bert84258836
cooked: <p>Hi all I am working with some of bag of word features, my dataset is n x p where n is the number of documents and p is the vocab size. p is quite large - 16 million but it is sparse.</p>
<p>The feature matrix is represented as a scipy sparse (CSR format) which I build a custom Pytorch Dataset around. I then write my own batching mechanism w/o the use of Python DataLoader which takes the  Pytorch Dataset (<strong>getitem</strong> gives us a scipy sparse X and a numpy y (0,1) ) and converts X into Pytorch Sparse Tensor and y into a simple FloatTensor and puts them in the following format:<br>
[ [X_b1, y_b1], [X_b2, y_b2], … ] which I then use as normal in training. I am getting decent results with this in terms of speed.</p>
<p>I did try using normal Python dataloaders, however this was a lot slower then the custom batching I did above for some reason.</p>
<p>I am looking for ways to speed up my code, I believe there should be faster ways to load and batch sparse inputs in PyTorch and looking for code samples and/or recommendations here?</p>
<p>Thanks</p>
Get a dataloader of resized images from a single dataset

username: R_B
cooked: <p>Hi, I would like to train a net with images of different resolutions (from cifar10).  My strategy was the following :</p>
<ul>
<li>download the cifar dataset with resolution = 224, with</li>
</ul>
<pre><code class="lang-auto">transform = transforms.Compose([transforms.Resize(size=(224, 224)), transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))])

trainset = torchvision.datasets.CIFAR10(root= path, train=True, transform=transform, download=True)
</code></pre>
<ul>
<li>then from this trainset, get a dataloader with images of the desired resolution, by downsampling.</li>
</ul>
<p>I don’t know how to do this… As I’m going to work with different resolutions, I do not want to download the cifar dataset every time.</p>

username: ptrblck
cooked: <p>If you are using <code>torchvision.datasets.CIFAR10</code>, the dataset will be downloaded once and stored in the location passed via <code>root</code>.<br>
Note that you are not downloading the CIFAR10 dataset in a resolution of <code>224x224</code>, but you are resizing each image to this resolution.<br>
CIFAR10 contains RGB images with the resolution <code>32x32</code>.<br>
If you want to apply different resolutions, I would recommend to create different datasets passing the transformation with the desired shape.</p>

username: R_B
cooked: <p>Finally I changed my method : every time I call <code>net(image)</code>, I make sure to interpolate the image to the desired size with <code>torch.nn.functionnal.interpolate</code>. This way I only have one dataset.<br>
Thank you !<br>
`</p>
Loss in not decreasing for fusion netwok

username: Surekha_Gaikwad
cooked: <p>Hi, I am creating network consisting of two modules where each of them have their own separate classifier. At the end I am adding up the weighted loss of both the modules and finally using it for backpropagation.<br>
But when I am training the model I could observe that the loss for only one module i.e. L1 is decreasing but not for the other one. I tried updating the learning rate also but it did not help much.<br>
I am not sure what is the issue behind it but I would like to know whether the approach that I am following is correct or not. Please provide your suggestion and advice.</p>
<p>Below is the high level snaphsot for architecture I am creating.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b63dbe4c31e8bcd0b3550fe0c694657525fa7f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b63dbe4c31e8bcd0b3550fe0c694657525fa7f0" title="Screen Shot 2020-04-21 at 8.26.36 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b63dbe4c31e8bcd0b3550fe0c694657525fa7f0_2_690x282.png" alt="Screen Shot 2020-04-21 at 8.26.36 PM" data-base62-sha1="hByFTnZtGBu5dFpQXDxn6XCAEyQ" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b63dbe4c31e8bcd0b3550fe0c694657525fa7f0_2_690x282.png, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b63dbe4c31e8bcd0b3550fe0c694657525fa7f0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b63dbe4c31e8bcd0b3550fe0c694657525fa7f0.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-04-21 at 8.26.36 PM</span><span class="informations">853×349 9.91 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Surekha_Gaikwad
cooked: <p>This is with reference to previous question.</p>
<p>I am trying to train the entire netork which consists of two modules with their own classifiers and I observed that both of them have their own convergence speed. Moreover they also need different learning rates because I tried with different possible combinations still the loss of only first module is decreasing but not the other. I wanted to know how will I find common learning rate in such case? Is there any existing way to do so?</p>
torch::Tensor nonzero process time is too long?what is the reason?

username: Wenzhe_Li
cooked: <p>currently i run the torch::Tensor nonzero to implement the superpoint feature extraction algorithm.the nonzero function consume about 30% time,640X360 image consume about 11ms time,what’s the reason?</p>

username: albanD
cooked: <p>Are you running on CPU or GPU?<br>
The main problem with this function is that it does not know in advance how many non-zero elements there are and so it is tricky to allocate the output properly in advance.</p>

username: Wenzhe_Li
cooked: <p>How can i reduce the processing time about torch::Tensor nonzero about gpu</p>

username: albanD
cooked: <p>Can you give more information about how you use it and what is the general goal around using it? Maybe you can do without?</p>
Caffe 2 C++ Error: google:protobuf::internal does not name a type?

username: bukenhatice
cooked: <p>I compiled pytorch in ubuntu. I followed these steps to install Caffe2:</p>
<ol>
<li>cd pytorch</li>
<li>mkdir release &amp;&amp; cd release</li>
<li>cmake …</li>
<li>make -j8</li>
<li>sudo make install</li>
</ol>
<p>Qt Creator .pro file:<br>
INCLUDEPATH + = / usr / local / include / caffe2<br>
LIBS + = -L / usr / local / lib -lcaffe2_detectron_ops_gpu -lcaffe2_nvrtc -lcaffe2_observers</p>
<p>INCLUDEPATH + = / usr / local / include / google / protobuf<br>
LIBS + = -L / usr / local / lib -lprotobuf</p>
<p>Following code</p>
<p>main.cpp file:<br>
<span class="hashtag">#include</span> &lt;caffe2 / core / blob.h&gt;<br>
<span class="hashtag">#incude</span> &lt;google / protobuf / any.h&gt;</p>
<p>Bugs: no matching function for call to ‘google :: protobuf :: internal: InternalMetadata :: unknown_fields’ or’ google :: protobuf :: internal ‘does not name a type’</p>
<p>Any ideas? I am happy to provide any more details or share the project.</p>
<p>Thanks,</p>
MaskRCNN crashes silently when predicting

username: Nuno-Mota
cooked: <p>Hi</p>
<p>When running Pytorch’s MaskRCNN implementation in eval mode the program crashes ‘silently’ (error: ‘zsh: killed python &lt;program&gt; &lt;args&gt;’.</p>
<p>I’ve traced the problem back to torchvision.models.detection.roi_heads on function paste_masks_in_image(masks, boxes, img_shape, padding=1):</p>
<pre><code class="lang-auto">def paste_masks_in_image(masks, boxes, img_shape, padding=1):
    # type: (Tensor, Tensor, Tuple[int, int], int)
    masks, scale = expand_masks(masks, padding=padding)
    boxes = expand_boxes(boxes, scale).to(dtype=torch.int64)
    im_h, im_w = img_shape

    if torchvision._is_tracing():
        return _onnx_paste_masks_in_image_loop(masks, boxes,
                                               torch.scalar_tensor(im_h, dtype=torch.int64),
                                               torch.scalar_tensor(im_w, dtype=torch.int64))[:, None]
    res = [
        paste_mask_in_image(m[0], b, im_h, im_w)
        for m, b in zip(masks, boxes)
    ]
    if len(res) &gt; 0:
        ret = torch.stack(res, dim=0)[:, None]
    else:
        ret = masks.new_empty((0, 1, im_h, im_w))
    return ret
</code></pre>
<p>In particular the problem seems to be that when res has a lot of elements, the whole program crashes due to lack of memory:</p>
<pre><code class="lang-auto">res = [
    paste_mask_in_image(m[0], b, im_h, im_w)
    for m, b in zip(masks, boxes)
]
if len(res) &gt; 0:
    ret = torch.stack(res, dim=0)[:, None]
</code></pre>
<p>For example, artificially reducing the number of results, by replacing the last line, torch.stack(…) by the following solves the problem (although obviously yielding incomplete results):</p>
<pre><code class="lang-auto">res = [
    paste_mask_in_image(m[0], b, im_h, im_w)
    for m, b in zip(masks, boxes)
]
if len(res) &gt; 0:
    ret = torch.stack(res[:2], dim=0)[:, None]
</code></pre>
<p>My main point is that it would be nice to have some kind of verbose error, if at all possible.</p>
<p>In my case, having 16GB of RAM in my machine, but with images images having a size 2054x2456 and quite a lot of memory consumption by other programs, I believe the best solution is to simply reduce the value of the MaskRCNN’s box_detections_per_img parameter.</p>
<p>Maybe someone knows of some other possible workaround?</p>
<p>Cheers</p>
Partial derivatives like tf.gradients stop_gradients parameter

username: Y_Y
cooked: <p>Hi all,<br>
Is there a way to calculate partial derivatives instead of total derivative like tf.gradients with parameter stop_gradients <a href="https://www.tensorflow.org/api_docs/python/tf/gradients" rel="nofollow noopener">https://www.tensorflow.org/api_docs/python/tf/gradients</a></p>
<pre><code class="lang-auto">a = torch.tensor(1., requires_grad=True)
b = 2 * a
c = a + b
print(torch.autograd.grad(c, a, retain_graph=True))
</code></pre>
<p>Output <code>(tensor(3.),)</code>, Expected output: 1 for partial derivative.<br>
I know we can use <code>b = 2*(a.detach())</code>. But what if <code>d=2*c</code>. In that case, I want to keep calculating the total derivative <code>dd/da</code>. I want to get <code>dd/da=6</code> without <code>a.detach</code> instead of 2.</p>
<p>Background: I am trying to calculate the last hidden layer partial gradient w.r.t the the first hidden in RNN. Then I still need to calculate <code>loss.backward()</code> where I don’t want to detach the hidden layer.</p>
<p>Thank you all.</p>

username: ShengweiAn
cooked: <p>I think the reason TensorFlow provides the <code>stop_gradients</code> for <code>tf.gradients</code> is mainly to <a href="https://github.com/tensorflow/tensorflow/blob/23c218785eac5bfe737eec4f8081fd0ef8e0684d/tensorflow/python/ops/gradients_impl.py#L457-L461" rel="nofollow noopener">“provides a way of stopping gradient after the graph has already been constructed”</a>.</p>
<p>However, PyTorch builds the graph dynamically. But I think this <code>stop_gradients</code> feature is useful when we want to change the retained graph.</p>
<p>Regarding your scenario, I think something like this may work:</p>
<pre><code class="lang-auto">import torch

a = torch.tensor(1., requires_grad=True)
b = 2 * a
c = a + b
c_detach = a + b.detach()
d = 2 * c
loss = d.sum()

print(torch.autograd.grad(d, a, retain_graph=True))
print(torch.autograd.grad(c, a, retain_graph=True))
print(torch.autograd.grad(c_detach, a, retain_graph=True))
print(a.grad)
loss.backward()
print(a.grad)
</code></pre>

username: Y_Y
cooked: <p>Hi Shengwei,<br>
Thank you for the answer. But I think the method of creating an extra <code>c_detach</code> variable is only work for this simple case. For Pytorch built in <code>nn.RNN</code>, we can’t do this to hidden states unless we build the structure from scratch.</p>

username: ShengweiAn
cooked: <p>Yes. But we might be able to add a hook to the hidden layer to do that.</p>

username: Y_Y
cooked: <p>Oh yeah. I didn’t investigate hook yet. I will look at it. Thank you.</p>
Input normalisation issue

username: Angry_potato
cooked: <p>Is it possible to un-normalized inputs to petrained VGG-16.because if i pass unnormalized images the lower level features are not detected properly.But it is also recommended in neural style transfer to clamp data between 0,255 for better image quality.Is it possible to pass the image without normalization and achieve exication of lower level features.</p>
<p>Help !!!</p>

username: ptrblck
cooked: <p>This could be possible, if you retrain the model (which might be trickier using unnormalized inputs).<br>
Note that the pretrained models were trained using normalized inputs. Unnormalized inputs are “new” to the model and thus the extracted features might be way off.</p>

username: Angry_potato
cooked: <p>Sorry i didnt retain model part</p>
Implementing own "batching" method

username: JanoschMenke
cooked: <p>Hi,</p>
<p>I was wondering how do I change the way batches are created by the dataloader. Many tutorials show how I make my own Dataset class which I can then feed into the dataloader. However, I also want to change how batches are created. For example when the matrices have to be combined in a specific way rather than just stacked onto each other.</p>
<p>Is there something like a tutorial out there?</p>

username: JanoschMenke
cooked: <aside class="quote quote-modified" data-post="1" data-topic="8278">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zchrissirhcz/48/1878_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278">How to create a dataloader with variable-size input</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi, I’d like to create a dataloader with different size input images, but don’t know how to do that. 
I’ve read the official tutorial on loading custum data (<a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">http://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a>), however in the tutorial, all the input images are rescaled to 256x256 and randomly cropped to 224*224. 
I’d like to do another image classification task, and in my task I want to: keep image their original size, and the batch size should be greater than 1. 
I noticed that th…
  </blockquote>
</aside>
Use CUDA to train autoencode

username: Tony_Chang
cooked: <p>Hi guys,<br>
I’m trying to use CUDA to train my autoencoder, however, I get the error like this. My code is as followed. Could someone help me to figure out the problem?<br>
AttributeError: ‘numpy.ndarray’ object has no attribute ‘dim’</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

### Change dataset to tensor
train_target = torch.tensor(y_train.values.astype(np.float32)).to(device) 
train = torch.tensor(X_train.values.astype(np.float32)).to(device)
train_tensor = data_utils.TensorDataset(train, train_target)

train_loader = Data.DataLoader(dataset = train_tensor, batch_size = BATCH_SIZE, shuffle = True)

class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder, self).__init__()

        self.encoder = nn.Sequential(
            nn.Linear(1*input_layer, 256),
            nn.ReLU(True),
            nn.Linear(256, 64),
            nn.ReLU(True),
            nn.Linear(64, 16),
            nn.ReLU(True),
            nn.Linear(16, 2),   
        )
        self.decoder = nn.Sequential(
            nn.Linear(2, 16),
            nn.ReLU(True),
            nn.Linear(16, 64),
            nn.ReLU(True),
            nn.Linear(64, 256),
            nn.ReLU(True),
            nn.Linear(256, 1*input_layer),
            nn.Sigmoid(),       # compress to a range (0, 1)
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded

autoencoder = AutoEncoder().to(device) 
params = list(autoencoder.parameters()) 
optimizer = torch.optim.SGD(params, lr=LR)
loss_func = nn.MSELoss()

for epoch in range(EPOCH):
    for step, (x, b_label) in enumerate(train_loader):
        batch_x = x.view(-1, 1*input_layer).cpu().numpy()   
        batch_y = x.view(-1, 1*input_layer).cpu().numpy()  

        encoded, decoded = autoencoder(batch_x)

        loss = loss_func(decoded, batch_y)      
        optimizer.zero_grad()               
        loss.backward()                     
        optimizer.step()                    

        if step % 100 == 0:
            print('Epoch: ', epoch, '| train loss: %.4f' % loss.data.numpy())
</code></pre>

username: Sunshine352
cooked: <p>You shouldn’t transform tensor to numpy like this:<br>
batch_x = x.view(-1, 1<em>input_layer).cpu().numpy()<br>
batch_y = x.view(-1, 1</em>input_layer).cpu().numpy()<br>
but write  as follows<br>
batch_x = x.view(-1, 1<em>input_layer).to(device)<br>
batch_y = x.view(-1, 1</em>input_layer).to(device)</p>

username: Tony_Chang
cooked: <p>Thanks a lot !!! I just solved the problem.</p>