akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model Parallelism and NVIDIA NVLINK

username: Hector_Corrales
cooked: <p>As I understand, in model parallelism, you divide a model and train each part separately in different GPUs.<br>
An example code I’ve found is this:</p>
<blockquote>
<p>class Network(nn.Module):<br>
def <strong>init</strong>(self, split_gpus):<br>
self.module1 = (some layers)<br>
self.module2 = (some layers)</p>
<pre><code>    self.split_gpus = split_gpus
    if self.split_gpus:
        self.module1.device("cuda:0")
        self.module2.device("cuda:1")

def forward(self, x):
    x = self.module1(x)
    if self.split_gpus:
        x = x.device("cuda:1") 
    return self.module2(x)
</code></pre>
</blockquote>
<p>My question is, is there a way to join 2 GPUs to be seen as a single GPU with double memory and not having to split the model?<br>
Does NVIDIA NVLINK do this?<br>
If not, what does NVLINK do?</p>
<p>Thanks!</p>

username: samster25
cooked: <p>It sounds like what you want is Data Parallelism. Where you have the model replicated on each GPU and they each work on different data. This works if you can do a batch size &gt; 1 on a single one of your GPUs.</p>
<p>NVLINK provides a faster interconnect compared to PCIe. This allows faster GPU&lt;&gt;GPU communication for either data or model parallelism.</p>

username: Rubeen_Mohammad
cooked: <aside class="quote no-group" data-username="samster25" data-post="2" data-topic="32569">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/858c86/48.png" class="avatar"> samster25:</div>
<blockquote>
<p>you want is Data Parallelism.</p>
</blockquote>
</aside>
<p>Hi, I’m very new to Nvlink and before starting up, do we need write any code to enable the usage of nvlink for model parallelism. Could you please give me any example code for running a model on nvlink, if you have any.<br>
Thank you</p>
Pytorch 1.4 Tracer warning help

username: Dmitry_Temnov
cooked: <p>Hi!<br>
I’ experimenting with converting Pytorch models to ONNX using this repository(<a href="https://github.com/amdegroot/ssd.pytorch" rel="nofollow noopener">repo</a>) - It is basically implementation of SSD.</p>
<p>Tracer argues at this script that computes coordinates for prior boxes.(<a href="https://github.com/amdegroot/ssd.pytorch/blob/master/layers/functions/prior_box.py" rel="nofollow noopener">script</a>)<br>
Specifically this code - where tensor is converted into python range:</p>
<pre><code class="lang-auto"> for i, j in product(range(f), repeat=2):
                f_k = self.image_size / self.steps[k]
                # unit center x,y
                cx = (j + 0.5) / f_k
                cy = (i + 0.5) / f_k
</code></pre>
<p>After that the whole process hangs. Any idea what might cause this or suggestions how to rewrite this part?</p>
<p>Thanks a lot for your help!</p>

username: ptrblck
cooked: <p>What kind of warning do you get?<br>
Is the warning thrown right before the hang occurs?</p>

username: Dmitry_Temnov
cooked: <p>Sorry, the warning is this:</p>
<pre><code class="lang-auto">TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
</code></pre>
How to use input in RNN model

username: 111273
cooked: <p>I have some rnn model</p>
<pre><code class="lang-auto">   class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
# some code #
def forward(self, x):
x = self.rnn(x)
output = my_function(# here is a code to get input #)
return x, output
</code></pre>
<p>And I need use input in my function in this model. Сan i do it somehow</p>

username: charan_Vjy
cooked: <p>It is not clear to me as to what you are trying to do with myfunc(). Is <code>x</code> not the input? More info would be much appreciated. If possible, do print out <code>myfunc</code> and clearly state it’s functionality and how it differs from <code>x</code>.</p>
Residual connection with multi-head attetnion

username: pouria_babvey
cooked: <p>Is there any simple example of implementing a residual connection with <code>nn.MultiheadAttentiont</code> .<br>
I am asking because I wonder what happens with masked elements. For example if an element is masked in a multi-head attention layer then it comes back in the residual connection after and that element information is not blocked anymore.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8407c56f734ab9cb342701f5915b7d965fcd16a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c8407c56f734ab9cb342701f5915b7d965fcd16a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8407c56f734ab9cb342701f5915b7d965fcd16a_2_410x500.jpeg" alt="image" data-base62-sha1="szvEyeZlS7M5CSuR7d2ZjduEXWO" width="410" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8407c56f734ab9cb342701f5915b7d965fcd16a_2_410x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8407c56f734ab9cb342701f5915b7d965fcd16a_2_615x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8407c56f734ab9cb342701f5915b7d965fcd16a_2_820x1000.jpeg 2x" data-dominant-color="E9E8E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1440×1752 257 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Adaptive Computation Time help

username: jafioti
cooked: <p>Hi everyone,</p>
<p>I am working on implementing a system that uses Adaptive Computation Time as outlined <a href="https://arxiv.org/abs/1603.08983" rel="nofollow noopener">here</a> I am building a basic example where the input is a random number 1-9 and the output should always be 10. Instead of directly choosing the output, the network runs in a loop and each loop it adds 1 to the input. The network has a halting neuron that decides when to stop running. The hope is that the network learns to stop running when the input equals 10, at which point it will return the input.</p>
<p>The problem I am facing is that the network always chooses to do the maximum number of loops, regardless of the input. Since the output is a weighted sum of the inputs depending on the output of the halting neuron at that time step, it is able to “cheat” in a way and simply use a precise halting output at the last time step to get roughly close to 10 and decrease the loss. To stop the network from always using the max loops, I tried to implement a “ponder loss” which penalizes the network for each loop it uses. I am certian that I am doing this wrong, as I adjusted the weight on the ponder loss up very high believing the network would only do one loop, but it still does the max number of loops.</p>
<p>Here is the full implementation of the model:</p>
<pre><code class="lang-auto">class DynamicNet(nn.Module):
    def __init__(self, hiddenSize, epsilon, maxSteps=20):
        super(DynamicNet, self).__init__()
        self.haltingNeuron = nn.Sequential(nn.Linear(1, hiddenSize), nn.Linear(hiddenSize, 1))
        self.epsilon = epsilon
        self.maxSteps = maxSteps

    def forward(self, x):
        haltTotal = torch.Tensor([0]).to(device)
        computePenalty = torch.Tensor([0]).to(device)
        computePenalty.requires_grad = True
        currentOutput = x
        currentOutput.requires_grad = True
        finalOutput = torch.zeros(self.maxSteps, requires_grad=True).to(device)

        for step in range(self.maxSteps):
            currentOutput = currentOutput + 1

            haltingOutput = torch.sigmoid(self.haltingNeuron(currentOutput))
            computePenalty[0] = computePenalty[0] + (1 - haltingOutput)

            if haltTotal[0] &gt; 1 - self.epsilon:
                finalOutput[step] = (currentOutput * ((1 - self.epsilon) - haltTotal[0])) # Finish final total with the remainder of halt total
                return(currentOutput * ((1 - self.epsilon) - haltTotal[0]), computePenalty[0] * 5, step)
            else:
                finalOutput[step] = (currentOutput * haltingOutput[0])

            haltTotal[0] = haltTotal[0] + haltingOutput

        return(currentOutput * ((1 - self.epsilon) - haltTotal[0]), computePenalty[0] * 5, self.maxSteps)
</code></pre>
<p>Please let me know if you need to provide additional information! Thanks in advance!</p>
Good way to copy 2 tensors with different shape but same number of elements

username: Kenisy
cooked: <p>For example I have 2 tensor:<br>
x = torch.randn(4)<br>
y = torch.randn(2, 2)<br>
If I want to copy y to x I have to do this:<br>
x.copy_(y.view(x.shape))<br>
Is there any more efficient ways to do it ? I notice that torch7 is able to do x:copy(y) but pytorch can’t.</p>

username: ptrblck
cooked: <p>The <code>view</code> operation will change some meta data of the tensor and will not copy <code>y</code>, so you shouldn’t see any performance hits.</p>
Module.to('cuda') and optimizer initialization

username: AreTor
cooked: <p>From the PyTorch 1.4.0 docs in the <code>torch.optim</code> page:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd3a275188854243fc8d79fc5779b4da33563465.png" data-download-href="https://discuss.pytorch.org/uploads/default/bd3a275188854243fc8d79fc5779b4da33563465" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd3a275188854243fc8d79fc5779b4da33563465.png" alt="image" data-base62-sha1="qZYPNhJSBPrzdtDmPGQLcPGQRXD" width="690" height="130" data-dominant-color="2D3A3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×161 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Does this happen also when using <code>.to('cuda')</code>?</p>

username: ptrblck
cooked: <p>Yes, <code>to('cuda')</code> and <code>.cuda()</code> are equivalent calls.<br>
Make sure to push the model to the appropriate device before creating the optimizer.</p>
TopK pooling layer giving return error

username: AsmaAdil
cooked: <p>I am following this tutorial</p><aside class="onebox githubblob" data-onebox-src="https://github.com/khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb">
  <header class="source">

      <a href="https://github.com/khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb" target="_blank" rel="noopener nofollow ugc">khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import csv\n",
    "import os\n",
    "import torch\n",
    "from torch_geometric.data import Data\n",
    "from tqdm import tqdm\n",
    "np.random.seed(42)"
   ]
  },
  {
</code></pre>



  This file has been truncated. <a href="https://github.com/khuangaf/PyTorch-Geometric-YooChoose/blob/master/YooChooseBuy.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
line no 157<br>
x, edge_index, _, batch, _ = self.pool1(x, edge_index, None, batch)<br>
giving error. I thought may be it returns four tensors as output instead of five, even that also giving error<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9456e244fcef7c2e4034b87232d893bb70a05d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9456e244fcef7c2e4034b87232d893bb70a05d1" title="err1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9456e244fcef7c2e4034b87232d893bb70a05d1.png" alt="err1" data-base62-sha1="o9rrER4XBfv0jfS2psN0scCfnJD" width="690" height="201" data-dominant-color="EFEDF5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">err1</span><span class="informations">1062×310 12.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I guess the note book might not have been updated to the latest PyTorch Geometric version, as <code>TopKPooling</code> returns not 6 values as seen in <a href="https://github.com/rusty1s/pytorch_geometric/blob/61865eeddc22b0dfbb310c9816622da7f01403f5/torch_geometric/nn/pool/topk_pool.py#L160">this line of code</a>.</p>
Multiclass Classification with sub-classes

username: haris
cooked: <p>Hello folks</p>
<p>I have a multi-class classification problem, where the number of classes is, say 10000 (or a very large number). The usual way of calculating loss is to use first take the softmax of 10000-dim raw scores (to turn them into a probability) and then take negative log i.e. the multi-class cross entropy loss.</p>
<p>Learning a 10000-way classifier obviously needs a lot of training samples. Now if I know that each training sample belongs to a (different) subset of these 10000 classes, how can I use this information to my advantage. Essentially it would involve taking a softmax over only that subset. This is different from the class imbalance problem because each sample belongs to a different subset.</p>
<p>I thought of using a boolean mask but that would skew the computation of denominator of softmax by having terms of exp(0).</p>
<p>Any pointers will be helpful. Oh and I don’t want to lose too much on speed <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks!</p>
Cross entropy loss for sentence classification

username: sajastu
cooked: <p>I have a tensor in shape of [ <span class="hashtag">#batch_size</span>, <span class="hashtag">#n_sentences</span>, <span class="hashtag">#scores</span>]. <span class="hashtag">#scores</span> are calculated for each fixed class. To clarify, suppose we have batch size of 1, with 31 sentences and 5 classes that sentences have been assigned to. So the tensor would have the shape of [1, 31, 5]. Now as my target (i.e., true section labels of each 31 sentences), I’d have a tensor in shape of [1, 31]. containing 31 numbers in the range of 0-4. I’m wondering how I can compute cross entropy loss for these (inputs and targets). Any ideas?</p>

username: sajastu
cooked: <p>I used a method similar to what discussed in <a href="https://github.com/pytorch/pytorch/issues/3556#issuecomment-388276442" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/3556#issuecomment-388276442</a> I think this works well. Hope it helps anyone facing the same problem!</p>

username: charan_Vjy
cooked: <p>You don’t really need to view the tensor in a different way. Cross entropy is now equipped to handle multiple dimensions. The code below demonstrates its working.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
criterion = nn.CrossEntropyLoss()
batch_size = 16
no_of_classes = 5
input = torch.randn(batch_size,no_of_classes,31)
target = torch.randint(0,4,(batch_size,31))
criterion(input,target)
</code></pre>

username: sajastu
cooked: <p>Thank you for the response. Which version of pytorch is this feature comes along? I’m using 1.1.0 and it gives me the error: ValueError: Expected target size (1, 5), got torch.Size([1, 31])…</p>

username: charan_Vjy
cooked: <p>I’m using 1.4.0. However, I don’t think that is the problem. This feature has been available for as long as I can remember. Going through your question, your tensor is of shape [1,31,5]. I would suggest changing its shape to [1,5,31] before you go ahead. You can do that with <code>x = x.permute(0,2,1)</code> before providing it to cross entropy loss.</p>

username: sajastu
cooked: <p>This works perfectly! Now I can figure out how pytorch handles such cases! Thank you <a class="mention" href="/u/charan_vjy">@charan_Vjy</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
I used cuda but the project doesn't seem to run on my GPU

username: Jim_Yang
cooked: <p>I am trying to run my cnn project on GPU. I tried to change the tensor type:</p>
<pre><code class="lang-auto">def _evaluate_epoch(axes, tr_loader, val_loader, model, criterion, epoch, stats):
    torch.set_default_tensor_type('torch.cuda.FloatTensor')
</code></pre>
<p>and in main function I added</p>
<pre><code class="lang-auto">    model = model.cuda()
</code></pre>
<p>after the model was created. But the project still doesn’t seem to run on GPU. Is there anything wrong? I am pretty new to this. Thanks!</p>

username: jmandivarapu1
cooked: <p>can you try <code>model.to('cuda')</code></p>

username: ptrblck
cooked: <p>Are you seeing some memory consumption as well as some GPU-Util on the device via <code>nvidia-smi</code>?</p>

username: Jim_Yang
cooked: <p>I did. GPU-uil is about 10 - 20 percent on Nvidia-smi and around 8% on task manager.</p>

username: Jim_Yang
cooked: <p>Isn’ that the same as <code>model = model.cuda()</code>? I mean I have tried to print out all the tensors device, they are all on cuda:0, but the GPU usage was pretty low and the speed didn’t improve really much as I used cpu</p>

username: jmandivarapu1
cooked: <aside class="quote no-group" data-username="Jim_Yang" data-post="4" data-topic="75273">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jim_yang/48/22313_2.png" class="avatar"> Jim_Yang:</div>
<blockquote>
<p>10 - 20 percent on Nvidia-smi and around 8% on task</p>
</blockquote>
</aside>
<p>may be try <code>watch nvidia-smi</code> then run the run the code in another terminal to see the usage realtime and I think there is always maximum speedup which you can achieve. Try increasing the batch size which will fit your GPU.</p>

username: ptrblck
cooked: <p>The Windows task manager doesn’t show the “compute” tab by default, so either add it or use <code>nvidia-smi</code>.<br>
Since the GPU Util is &gt;0%, the GPU is used.<br>
You might suffer from other bottlenecks, e.g. data loading, so make sure to store the data on a local SSD.<br>
If that doesn’t help, you would have to profile the code and check, which parts uses most of the runtime.</p>

username: Jim_Yang
cooked: <p>Thank you! I guess I was expecting it to be super fast when GPU is being used, cuz I heard there should be a huge boost up between gpu and cpu. But when I improve my CNN’s layers and batch size, the GPU usage was still pretty low and the project wasn’t really fast.</p>
Class in cpp accessing torch classes and methods

username: am0
cooked: <p>Hi,</p>
<p>I’m interested in combining python code with cpp code.<br>
I read about custom cpp extension, extending torchscript, and c++ frontend.</p>
<p>I would like to create a class written in cpp and use it from python code. The cpp class must be able to access tensor objects and methods (as done in <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>), while still being used as done in (<a href="https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html</a>).<br>
The first link I added shows how to create methods in c++ that access torch classes and methods and the code is compiled using setup.py script, the second link shows how to use a class in c++ that does not use torch classes and methods (the example is a stack class) and is it compiled using cmake.</p>
<p>Is it possible to sort of combine the two? I want to create a class in cpp and be able to access torch classes and methods, then load the class and use it from python code.</p>
<p>Thanks,<br>
Alex</p>

username: tom
cooked: <p>You should absolutely be able to use e.g. <code>torch::Tensor</code> instead of <code>std::string</code> for the method. Note that the “modern” way of extending PyTorch is <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="nofollow noopener">CustomOperators</a> rather than “vanilla” C++ extensions.</p>
<p>Best regards</p>
<p>Thomas</p>

username: am0
cooked: <p>Thank you for your answer, but that is not exactly what I’m asking.<br>
What I’m asking is:<br>
In this example - <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>, we see an example of LLTM class. In this example “forward” and “backward” methods are implemented in cpp and called from python “forward” and “backward” methods, instead of the python implementation.<br>
Is it possible to write the whole LLTM class in cpp and use it from python, instead of just the methods?</p>
ERROR! Session/line number was not unique in database. History logging moved to new session 64

username: hunar
cooked: <p>hi , i want to select randomly faces from test dataset anfter trained training folder of faces<br>
i use facenet-pytorch library</p>
<pre><code>optimizer = optim.Adam(resnet.parameters(), lr=0.001)
scheduler = MultiStepLR(optimizer, [5, 10])

trans = transforms.Compose([
   np.float32,
   transforms.ToTensor(),
   fixed_image_standardization
]) 
dataset = datasets.ImageFolder(data_dir, transform=trans)
img_inds = np.arange(len(dataset)) 
np.random.shuffle(img_inds)
train_inds = img_inds[:int(0.8 * len(img_inds))]
val_inds = img_inds[int(0.8 * len(img_inds)):]

train_loader = DataLoader(
   dataset,
   num_workers=workers,
   batch_size=batch_size,
   sampler=SubsetRandomSampler(train_inds)
)
val_loader = DataLoader(
val_inds,
batch_size=batch_size,
shuffle = True,
num_workers=workers,
)
</code></pre>
<p>how to select a face randomly in test data set to make prediction ?<br>
the result of the code provide this error</p>
<blockquote>
<p>val_inds,<br>
^<br>
SyntaxError: positional argument follows keyword argument</p>
</blockquote>
<p>thanks</p>

username: JuanFMontesinos
cooked: <aside class="quote no-group quote-modified" data-username="hunar" data-post="1" data-topic="75420">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c0e974/48.png" class="avatar"> hunar:</div>
<blockquote>
<p>val_loader = DataLoader( dataset, num_workers=workers, batch_size=batch_size, val_inds, shuffle = True,</p>
</blockquote>
</aside>
<pre><code class="lang-auto">val_loader = DataLoader(
dataset,
num_workers=workers,
batch_size=batch_size,
val_inds,&lt;-------------------------------------------------------
shuffle = True,
</code></pre>

username: hunar
cooked: <p>so whats wrong ?is there something i’ve missed</p>

username: JuanFMontesinos
cooked: <p>In python there are two type or function arguments, args and kwargs. Args are those which are identified by their position when you call the function. Kwargs are those which are identified by the name you pass.<br>
In python you must past those without name and later those with name. so val inds needs something like: indices=val_inds (whatever the name of the argument is in the function)</p>
Jupyter notebook timeout on remote server

username: lycaenidae
cooked: <p>Hi,</p>
<p>I am using jupyter notebook to train my code on the remote server and I am forwarding jupyter notebook port to my local machine so that I can see on the browser.</p>
<p>On the server side:<br>
I am using tmux and detaching session and not getting timeout</p>
<p>On the local machine side:<br>
I am running that command<br>
<code>ssh -N -f -L localhost:2000:localhost:2000 username@&lt;remotemachine&gt;</code><br>
but I am getting timeout after 10epocs so my train never finished on jupyter notebook:( Because when I forward port again, I already lost the progress.</p>
<p>What do you use to solve that ssh timeout issue while training on the remote machine?</p>
<p>Thanks a lot</p>

username: jmandivarapu1
cooked: <p>I use like below and everything works for me without issues. Just give a try for luck (Make sure you type below in the terminal/cmd not in the IDE’s terminal like vscode inbuilt terminal)</p>
<pre><code class="lang-auto">ssh -L 2000:localhost:200 usrname@server
</code></pre>

username: lycaenidae
cooked: <p>Hi <a class="mention" href="/u/jmandivarapu1">@jmandivarapu1</a>, you’re right, I realized that this command works as long as my local pc has internet connection. I got this ssh timeout because wifi connection dropped. Thanks a lot for clarification:)</p>
Loss Remains Constant

username: Angry_potato
cooked: <p>HI,<br>
I am training a model for neural style transfer,but the loss remains constant.</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt
import numpy as np
import cv2
import torchvision

class Vgg16(torch.nn.Module):
    def __init__(self):
        super(Vgg16, self).__init__()
        features = list(vgg16(pretrained = True).features.eval())[:23]
        self.features = nn.ModuleList(features).eval() 
        
    def forward(self, x):
        results = []
        for ii,model in enumerate(self.features):
            x = model(x)
            if ii in {3,8,15,22}:
                results.append(x)
        return results

net=Vgg16()
net=net.float()
print(net)

def prod(x):
    j=1
    for i in x:
        j=j*i
    return j

def feature_loss(pred,target):
    loss=(1/prod(pred.shape))*(pred-target).pow(2).sum()
    return loss

def Gram_matrix(inpu):
    p,x,y,z=inpu.shape
    mat=inpu.view(x,y*z)
    G=torch.mm(mat,mat.t())
    return G/prod(inpu.shape)
def style_loss(pred,target):
    loss=(pred-target).pow(2).sum()
    return loss
content_im=cv2.imread('/home/jatin/ML_intern/challenge-master/japanese_garden.jpg')
content_im=cv2.resize(content_im,(224,224))
content_im=np.reshape(content_im,(-1,3,224,224))
content_im=torch.from_numpy(content_im)
content_im=content_im.float()
style_im=cv2.imread('/home/jatin/ML_intern/challenge-master/picasso_selfportrait.jpg')
style_im=cv2.resize(style_im,(224,224))
style_im=np.reshape(style_im,(-1,3,224,224))
style_im=torch.from_numpy(style_im)
style_im=style_im.float()
print(style_im.shape)
inp=torch.randn(1,3,224,224,requires_grad=True)
def get_input_optimizer(input_img):
    optimizer = torch.optim.LBFGS([input_img.requires_grad_()])
    return optimizer
for i in range(10000):
    optimizer=get_input_optimizer(inp)
    def closure():
        inp.data.clamp_(0, 1)
        y_pred=net(inp)
        optimizer.zero_grad()
        loss=0
        for j in range(len(y_pred)):
            if j==1:
                loss=loss+feature_loss(y_pred[j],y_content[j])+style_loss(Gram_matrix(y_pred[j]),Gram_matrix(y_style[j]))
            else:
                loss=loss+style_loss(Gram_matrix(y_pred[j]),Gram_matrix(y_style[j]))
        loss.backward(retain_graph=True)
        return loss
    print('loss '+str(i)+'th iteration=',loss)
    optimizer.step(closure)
</code></pre>

username: ptrblck
cooked: <p>Could you post dummy tensors to debug the code, please?</p>

username: Angry_potato
cooked: <p>Thanks for the reply. I ran the code once again from the beginning then I didn’t encounter this error. Since then the code runs as intended</p>
How do I pass numpy array to conv2d weight for initialization?

username: han324solo
cooked: <p>How do I pass numpy array to conv2d weight for initialization?</p>
<p>I tried using fill_ but apprarently it only support for 0-dimension value.<br>
My numpy_data is 4-dimension array.</p>
<p>Here’s what I tried:</p>
<blockquote>
<p>myModel = Net()<br>
layers = [x.data for x in myModel.parameters()]<br>
layers[idx].data.fill_( numpy_data )</p>
</blockquote>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/han324solo">@han324solo</a>,</p>
<p>You can simply do this:</p>
<pre><code class="lang-auto">layers[idx].data = torch.FloatTensor(numpy_data)
</code></pre>

username: ptrblck
cooked: <p>You could use <code>copy_</code> instead of <code>fill_</code> or assign an <code>nn.Parameter</code>.<br>
Also, the usage of the <code>.data</code> attribute is discouraged. Wrap the assignment in a <code>torch.no_grad()</code> block instead:</p>
<pre><code class="lang-python">numpy_data= np.random.randn(6, 1, 3, 3)

conv = nn.Conv2d(1, 6, 3, 1, 1, bias=False)
with torch.no_grad():
    conv.weight = nn.Parameter(torch.from_numpy(numpy_data).float())
    # or
    conv.weight.copy_(torch.from_numpy(numpy_data).float())
</code></pre>

username: han324solo
cooked: <p>Thanks. This is exactly what I needed.</p>

username: wenlongzhao094
cooked: <p>What happens if we don’t wrap with torch.no_grad()? Thanks!</p>

username: ptrblck
cooked: <p>Then the intermediate activations, which are needed to calculate the gradients during the backward pass will be stored and will use memory, which is not needed during inference/testing.</p>
Can I train a asynchronous actor critic model without keeping a copy of the model?

username: urw7rs
cooked: <p>I’m editing code from <a href="https://github.com/facebookresearch/torchbeast" rel="nofollow noopener">torchbeast</a> to fit my use case. I want to use a discriminator for calculating the reward at a certain state.</p>
<p>I noticed that torchbeast uses a copy of the model for inference called actor_model and a the original model for learning.</p>
<p>This is part of polybeast.py:</p>
<pre><code class="lang-auto">def learn(
    ...
    for tensors in learner_queue:
        ...
        optimizer.zero_grad()
        total_loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), flags.grad_norm_clipping)
        optimizer.step()
        scheduler.step()

        actor_model.load_state_dict(model.state_dict())
        ...
</code></pre>
<p>I tried the same thing with the discriminator but I ran out of memory and had to use minibatches. I thought of using locks so only one thread is using the discriminator. Is there any other solution to this problem? Is it impossible?</p>
Literature on NNs with multiple objectives

username: Felipe_Mello
cooked: <p>Hi guys,</p>
<p>I have a RNN with tabular data that needs to predict over 30 outputs at each time step. What I have observed is that the prediction of one output damages the other. Can you guys recommend some literature on how to specialize your NN to predict different outputs? I know that, for autonomous driving, you have many outputs being generated at the same time.</p>
<p>Thanks in advance!</p>

username: charan_Vjy
cooked: <blockquote>
<p>What I have observed is that the prediction of one output damages the other?</p>
</blockquote>
<p>I’m assuming your problem is a multi task, multi class problem. This is the essence of Multi Task learning in general. You utilize inductive bias when training on mulitple tasks, thus allowing your model to generalize better to all the tasks. The roots of this idea trace way back to 1993: <a href="https://link.springer.com/article/10.1023/A:1007379606734" class="inline-onebox" rel="noopener nofollow ugc">Multitask Learning | Machine Learning</a>. I am not an expert in MTL for NLP. However, I just went through this blog: <a href="https://ruder.io" rel="noopener nofollow ugc">https://ruder.io</a>./multi-task-learning-nlp/ . This is a good primer for MTL in NLP. Do go through the citations to get an idea of the basics of MTL fot NLP. In general, the relatability of tasks seems to be very important to achieve generalization. I would suggest going through the latest work in ACL, EMNLP or NIPS to get a better idea about the cutting edge in MTL for NLP. Do let me know if you come across a good paper yourself.</p>
LSTM loss keeps fluctuating

username: hs271095
cooked: <p>Hello I am a beginner and I am using pytorch to implement LSTM model for classification. when running the model the loss values keep changing (Increase and decrease randomly). shown by the following graph. When I try to implement it by other frameworks like Keras using the same data the results are correct and the loss decreases with time.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6fd9220f58fb40f9465dd61a6249b8cfa1429cb.png" alt="image" data-base62-sha1="uFTs074xIgdueAEYPuAwG240ksP" width="384" height="248"></p>
<p>I am using MSELoss function and following is my code for the LSTM class</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, input_dim=272, hidden_layer_size=64, output_size=1,batch=64,n_layer=1):
        super().__init__()
        
        self.hidden_layer_size = hidden_layer_size
        self.lstm = nn.LSTM(input_dim, hidden_layer_size,batch_first=True)
        self.linear1 = nn.Linear(hidden_layer_size, 32)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(32, output_size)
        self.hidden_cell = (torch.randn(n_layer,batch,self.hidden_layer_size).cuda(), torch.randn(n_layer,batch,self.hidden_layer_size).cuda())
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, input_seq):
        lstm_out, self.hidden_cell = self.lstm(input_seq, self.hidden_cell)
        #print(lstm_out[:,-1,:].shape)
        linear1_out = self.linear1(lstm_out[:,-1,:])
        relu=self.relu(linear1_out)
        linear2_out = self.linear2(relu)
        prediction=self.sigmoid(linear2_out)
        return prediction
    
    def backward(self,g0):
        raise RuntimeError("Some Error in Backward")
        return g0.clone()
</code></pre>

username: charan_Vjy
cooked: <ul>
<li>Why are you using MSELoss if it is a classification task (you have used a sigmoid function at the end of network squishing the value between 0 and 1)? Was this intentional? Did you use the same methodology in Keras?</li>
<li>Did you set the same learning rates in both the models? Can you also print out the Keras code? Both the codes could then be crosschecked.</li>
</ul>
Different outputs for same input

username: sait
cooked: <p>I have alread trained model. but same input is producing different outputs.Note that I have put model on eval() phase so that dropout is not enabled. But still model is producing different outputs for same input.Can someone verify what is wrong ?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40940d9c56aed22e5e61460fa6f8640e4bf66878.png" alt="Screenshot_2020-04-05 Jacobain saliency Attack - Jupyter Notebook" data-base62-sha1="9dhL9zHYV58OLMQajWc6QedXswg" width="609" height="183"></p>

username: charan_Vjy
cooked: <p>Can you print out the model “teacher”?</p>
NVIDIA Quadro P4000(PCIe/8GB)

username: sai_m
cooked: <p>I have an NVIDIA Quadro P2000(PCIe/5GB) on my PC which is not supporting CUDA and I have an option to upgrade to P4000(PCIe/8GB). does  NVIDIA Quadro P4000(PCIe/8GB) support CUDA? If so is it a good option to start with? I have spent some time on google, but didn’t find answer.</p>
<p>Thanks in advance.</p>

username: albanD
cooked: <p>What do you mean by “not supporting CUDA”. Quadro cards like these ones definitely are in the list of supported cards (see <a href="https://en.wikipedia.org/wiki/CUDA">https://en.wikipedia.org/wiki/CUDA</a> for example).</p>

username: sai_m
cooked: <p>with P2000, when I run torch.cuda.is_available(), its returning False. Do I have to do any pre setup before running cuda.is_available()?</p>
<p>Thanks alban</p>

username: Ram_Mettu
cooked: <p>may be you have pytorch cpu only version or you didn’t install cudatoolkit.<br>
Following command install latest pytorch with cudatoolkit.<br>
<strong>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</strong></p>

username: sai_m
cooked: <p>Thank you Ram.<br>
I had cudatoolkit installed. Now,I have done uninstalling and installing pytorch , cudatoolkit. No luck, still its returning false for torch.cuda.is_available()</p>

username: ptrblck
cooked: <p>Is the GPU properly recognized in <code>nvidia-smi</code>?<br>
Also, which NVIDIA driver are you using?</p>

username: sai_m
cooked: <p>I am using 369.49 version of drivers and when I type nvidia-smi -L its returning<br>
GPU 0: Quadro M2000</p>
<p>Really appreciate your help.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Thanks for the information!<br>
Based on <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#binary-compatibility">this list</a> you should be able to install the PyTorch binaries with CUDA9.2.<br>
If possible, I would recommend to update the driver and use the binaries with CUDA10.1.</p>

username: sai_m
cooked: <p>ok, will try with CUDA9.2</p>
<p>Thanks a lot !!</p>

username: Yan_Xu
cooked: <p>can you tell me if cuda 9.2 works for you?</p>
<p>i have same issue with P2000 on my Dell precision 5530</p>
GPU out of memory error on loss.backward() but plenty of memory available - ModelParallel

username: raybannerman
cooked: <p>I am training models on large volumetric images with a batch size of 1, where a single example is 402 x 420 x 420 pixels (so I cannot make the batch size any smaller as I am already at a batch size of 1. I also cannot make the training examples any smaller.) I am using two NVIDIA Titan Xp GPUs. I have trained many models on this data set successfully.</p>
<p>However, over the past week I have gotten weird out of memory errors at loss.backward() for some models which are almost identical to other models I’ve run, except for a couple extra layers which add a tiny number of additional parameters relative to the overall model size.</p>
<p>Here is the error - Pytorch claims that it can’t allocate 1.41 GiB even though it also says there is plenty of free memory available:</p>
<p>When I put everything on GPU 0 and leave GPU 1 empty:<br>
RuntimeError: CUDA out of memory. Tried to allocate 1.41 GiB (GPU 0; 11.91 GiB total capacity; 4.28 GiB already allocated; 6.42 GiB free; 675.20 MiB cached)</p>
<p>When I use ModelParallel and put part of the model on GPU 0 and the other part on GPU 1 (in different ways):<br>
RuntimeError: CUDA out of memory. Tried to allocate 1.41 GiB (GPU 1; 11.91 GiB total capacity; 4.27 GiB already allocated; 6.09 GiB free; 1016.44 MiB cached)<br>
RuntimeError: CUDA out of memory. Tried to allocate 1.41 GiB (GPU 1; 11.91 GiB total capacity; 270.62 MiB already allocated; 11.09 GiB free; 3.38 MiB cached)</p>
<p>I am using torch.cuda.empty_cache(), and I have a fixed size input and I am using torch.backends.cudnn.benchmark = True<br>
torch.backends.cudnn.enabled = True<br>
(although the error occurs regardless of how I set these flags).</p>
<p>I read that perhaps this error can happen from memory fragmentation but I don’t know what I can do to resolve the problem and I also don’t know why the memory fragmentation would be so bad because the models where this error occurs are extremely similar to other models that don’t cause the error.</p>
<p>I’ve been trying everything I can to solve this problem. If anyone has suggestions about how to troubleshoot it, I would be very grateful.</p>
Either too little or too many positional arguments when using nn.Sequential

username: Jan_Pisl
cooked: <p>Hi,</p>
<p>I am new to PyTorch, so please excuse my silly question.</p>
<p>I define a nn.Sequential in init of my Encoder object like this:</p>
<pre><code class="lang-auto">self.list_of_blocks = [EncoderBlock(n_features, n_heads, n_hidden, dropout) for _ in range(n_blocks)]
self.blocks = nn.Sequential(*self.list_of_blocks)
</code></pre>
<p>The forward of EncoderBlock looks like this</p>
<p><code>def forward(self, x, mask):</code></p>
<p>Then, in the forward() of my Encoder, I try to do:</p>
<p><code>z0 = self.blocks(z0, mask)</code></p>
<p>However, I get</p>
<p><code>TypeError: forward() takes 2 positional arguments but 3 were given</code></p>
<p>When I try:</p>
<p><code>z0 = self.blocks(z0)</code></p>
<p>I get (understandably):</p>
<p><code>TypeError: forward() takes 2 positional arguments but only 1 was given</code></p>
<p>When I do not use nn.Sequential and just execute one EncoderBlock after another, it works:</p>
<pre><code class="lang-auto">for i in range(self.n_blocks):
     z0 = self.list_of_blocks[i](z0, mask)
</code></pre>
<p>What am I doing wrong and how do I use nn.Sequential correctly in this case?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p><code>nn.Sequential</code> is used for very simple models and, if I’m not mistaken, doesn’t accept multiple inputs.<br>
You could try <a href="https://github.com/pytorch/pytorch/issues/19808#issuecomment-487291323">this approach</a> or write a custom model.</p>
Image looks different after torchvision.transforms

username: Xin_Ma
cooked: <p>I use cv2 to read and resize the image and the top image is the result. Then I want to use torchvision.transforms to do data augmentation for this image. I applied RandomHorizontalFlip(), but the result shows quite different with the top one.<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/adc485bf557de7f0df7302bac8a24667f6357d65.png" alt="Screen Shot 2020-04-04 at 6.38.58 PM" data-base62-sha1="oNdEhOJcbNVPW7eG7EBF0DPqpFj" width="198" height="288"><br>
Why do they looks like so different? How can I do data augmentation in Pytorch?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I’m not sure how you’ve visualized the images, but note that OpenCV uses BGR color format by default, while <code>PIL</code>, which is used in <code>torchvision.transforms</code> uses RGB.</p>
<p>That being said, did you try to permute the dimensions using <code>view</code> or <code>reshape</code>, as the image might also be interleaved. It’s a bit hard to tell, as the images are quite noisy.</p>
How to add intercept/bias to sigmoid function

username: 111273
cooked: <p>I need to use nn.Sigmoid(), but I need sigmoid growth when the number is greater than x, not zero.<br>
Thanks a lot.</p>

username: ptrblck
cooked: <p>Would adding or subtracting the desired constant to the input tensor work?<br>
You would then calculate <code>sigmoid(input-x)</code>, which will move the function on the x-axis.</p>
Adding fc layers to a model

username: cltexe
cooked: <p>I’m trying to extend this <code>VGG</code> network by adding 2 FC layer but seems like I failed at something. Original one:</p>
<pre><code class="lang-auto">class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 7)

    def forward(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        out = F.dropout(out, p=0.5, training=self.training)
        out = self.classifier(out)
        return out
</code></pre>
<p>What I did is:</p>
<pre><code class="lang-auto">class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.fc1 = nn.Linear(512, 4096)
        self.fc2 = nn.Linear(4096, 4096)
        self.classifier = nn.Linear(4096, 7)

    def forward(self, x):
        out = self.features(x)
        out = out.view(x.size(0), -1)
        out = F.relu(self.fc1(x))
        out = F.dropout(out, p=0.5, training=self.training)
        out = F.relu(self.fc2(x))
        out = F.dropout(out, p=0.5, training=self.training)
        out = self.classifier(x)
        return out
</code></pre>

username: ptrblck
cooked: <p>You are currently passing <code>x</code> instead of <code>out</code> to all linear layers.</p>

username: alanzhai219
cooked: <p>I think you should make sure of inputs and outputs of each layer. take your time.</p>
Unexpected out of memory during inference

username: entslscheia
cooked: <p>I have a model that need to do <code>log_softmax</code> on a tenser of shape <code>(batch_size, x, y, z)</code> during inference. And then I can do <code>gather</code> to get a tensor of shape <code>(batch_size, x, y, 1)</code> from it. For most of the cases, <code>x</code> is smaller than 2000 and it works fine, but when the model encounters an example with <code>x</code> around 3000, it reports <code>CUDA out of memory</code> during the computation of <code>log_softmax</code>, because <code>(batch_size, x, y, z)</code> is indeed a large tensor. So I try to split <code>x</code> into several shards for which each shard has a size no larger than 1000 then first do <code>log_softmax</code> and <code>gather</code> independently for each shard and finally <code>cat</code> the results from different shards. However, it still reports <code>CUDA out of memory</code> during the computation of <code>log_softmax</code>. I feel this unexpected because the computation for different shards are totally independent. In other words, when computing the next shard, the previous results of <code>log_softmax</code> can be totally released. Should I manually release the memory by calling <code>del</code> after the computation of each shard?<br>
Also, I am wondering why doing several computation of size  <code>(batch_size, 1000, y, z)</code> within one forward pass will result in <code>OOM</code> while doing many computation of size <code>(batch_size, 2000, y, z)</code> across different forward pass never lead to <code>OOM</code>, considering the computation in both scenario doesn’t really depends on the previous results of <code>log_softmax</code>? Does PyTorch have any memory releasing mechanism at the end of every <code>forward</code> pass?</p>

username: ptrblck
cooked: <p>Did you wrap the code in a <code>with torch.no_grad()</code> block, since you are dealing with an inference use case?<br>
This will make sure to avoid storing the intermediate tensors, which would be needed to calculate the gradients during the backward call.</p>
<p>If you’ve already done that, could you check how close you are to the memory limit on your device using <code>dim1=2000</code>? If you are close to an OOM, creating multiple instances of smaller tensors, could yield the OOM e.g. due to memory fragmentation.</p>
Changing the value of an entry in the forward pass

username: Kamil_Adamczewski
cooked: <p>There is a tensor which I want to learn, except for one variable in the tensor. In fact, I would like the entry of this tensor not to be a variable and, moreover, change the value of this entry in the forward pass so that I can take into account two passes with two different values of that entry into my loss function. Is this possible and if so, how to achieve it?</p>

username: ptrblck
cooked: <p>Would concatenating the input with another tensor during the forward pass work?</p>
<pre><code class="lang-python">torch.manual_seed(2809)

lin = nn.Linear(10, 1)

x = torch.randn(1, 9, requires_grad=True)
constant = torch.zeros(1, 1)
data = torch.cat((x, constant), dim=1)

out = lin(data)
out.mean().backward()

constant = torch.ones(1, 1)
data = torch.cat((x, constant), dim=1)
out = lin(data)
out.mean().backward()

print(x.grad)
print(lin.weight.grad)
</code></pre>