akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
GPU and its Memory usage is very low

aswamy: <p>GPU usage is very less(1% to 10%, sometimes 30%) and GPU memory usage is also very less(less than 10%).<br>
I am using transformer model with reinforcement learning to solve graph related problems. So my data is basically, list of nodes with its features.</p>
<p>I feel that resources are not being used efficiently, kindly help me resolve this.</p>
<p>Here is how my DataLoader look like:<br>
training_dataloader = DataLoader(training_dataset, batch_size=opts.batch_size, num_workers=32, pin_memory=True)<br>
my data is basically, random based generation like below,</p>
<p>def generate_instance(size, prize_type):<br>
# Details see paper<br>
MAX_LENGTHS = {<br>
20: 2.,<br>
50: 3.,<br>
100: 4.<br>
}</p>
<pre><code>loc = torch.FloatTensor(size, 2).uniform_(0, 1)
depot = torch.FloatTensor(2).uniform_(0, 1)
# Methods taken from Fischetti et al. 1998
if prize_type == 'const':
    prize = torch.ones(size)
elif prize_type == 'unif':
    prize = (1 + torch.randint(0, 100, size=(size, ))) / 100.
else:  # Based on distance to depot
    assert prize_type == 'dist'
    prize_ = (depot[None, :] - loc).norm(p=2, dim=-1)
    prize = (1 + (prize_ / prize_.max(dim=-1, keepdim=True)[0] * 99).int()).float() / 100.

return {
    'loc': loc,
    # Uniform 1 - 9, scaled by capacities
    'prize': prize,
    'depot': depot,
    'max_length': torch.tensor(MAX_LENGTHS[size])
}
</code></pre>
<p>Thanks!<br>
Anil</p>

Prerna_Dhareshwar: <p>What is your batch size? You could try increasing it and see what your GPU usage is after that.</p>

aswamy: <p>Batch size is 512 and I think it’s large enough. But I’ll try increasing</p>
Can't load weights to model, dict for model and weights are the same

marx_was_right: <p>I’m trying to load trained model weights and get this error message:</p>
<p><code>IncompatibleKeys(missing_keys=[], unexpected_keys=[])</code></p>
<p>I can’t figure out the issue because when I inspect <code>model.state_dict().keys()</code> and <code>state_dict.keys()</code> all of the keys are the same!</p>
<p><code>model.state_dict().keys() == state_dict.keys()</code> produces <code>True</code></p>
<p>What else can be preventing me from loading weights into my model?<br>
I’ve been scanning the internet but I can’t find someone with a similar error.</p>
<p>Appreciate the help in advance.</p>

ptrblck: <p>This message just states, that <strong>no</strong> incompatible keys were found.<br>
Since this message confused a lot of users, it was changed in <a href="https://github.com/pytorch/pytorch/pull/22119" rel="nofollow noopener">this PR</a>, which should make it available in the current nightly build.</p>

marx_was_right: <p>lmao damn, thought it was an error.</p>
<p>Thanks</p>
Do I have to explicitly call the dataloader in each epoch for training augmentation?

PapRazzi: <p>Hi. I’d like to (virtually) augment my data for training. I just want more random samples and to do this I want to train the network for a longer number of epochs. I’m wondering if for every epoch my dataset will be transformed differently or if I have to explicitly call the dataloader in each epoch</p>

ptrblck: <p>The transformations are set in the <code>Dataset</code> and are applied on the fly for each sample.<br>
If you are using random transformations, each sample will be randomly transformed in each epoch.</p>
<p>You don’t have to recreate the <code>DataLoader</code> in each epoch.</p>
From torch._C import* ImportError:libcurand.so.8.0

rock: <p>My previous installations of CUDA8.0, anaconda, python3.6, and pytorch 1.0 on ubuntu 16.04 worked fine.<br>
I upgraded CUDA8.0 to CUDA10.0 and pytorch 1.0 to pytorch1.1 on ubuntu 16.04 server, and the following errors occurred:<br>
file “/anaconda3/**/python3.6/site-packages/torch/<strong>init</strong>.py",line 79, in  from torch._C import* ImportError:libcurand.so.8.0:cannont open shared object file:No such file or directory.</p>
<p>How to solve this problem? Thank you.</p>

ptrblck: <p>Did you build PyTorch from source?<br>
If so, could you try to clean the build (<code>python setup.py clean</code>) and uninstall all old versions?</p>
<pre><code class="lang-auto">pip uninstall torch
pip uninstall torch
conda uninstall pytorch
</code></pre>
<p>(also *-nighly builds if previously installed).</p>
<p>However, if you didn’t build it from source, you don’t need to install CUDA on your system, as the PyTorch binaries will already ship with the CUDA and cudnn runtime libs.</p>

rock: <p>First of all, thank you very much for your answer! I’m pretty sure I’ve installed torch under anaconda, but when I import torch in python, the current python environment doesn’t know all the search paths for dependent dynamic libraries under the torch library.</p>

ptrblck: <p>Could you create a new and clean conda environment and install pytorch there again?<br>
E.g.:</p>
<pre><code class="lang-auto">conda create -n pytorch_env Python=3.7 Anaconda
...
conda activate pytorch_env
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
</code></pre>
Add new layer to model

slavavs: <p>I have a model that I trained on small data. Weight network I saved. Now I want to add a new layer to this model, how to do it? I tried to add a new layer with a new name, but I still have an error.<br>
Error(s) in loading state_dict for Net: Missing key(s) in state_dict: “fc4.weight”, “fc4.bias”, “bn5.running_mean”, “bn5.running_var”.</p>

RicCu: <p>Hi,<br>
Load your model using <code>your_new_model.load_state_dict(your_old_state_dict, strict=False)</code></p>
Fold(unfold(x)) != x

Nadav_Bhonker: <p><code>In[1]: x = torch.rand(32, 3, 16, 16)</code><br>
<code>In[2]: torch.equal(F.fold(F.unfold(x, 3), x.shape[2], 3), x)</code><br>
<code>Out[2]: False</code></p>
<p>Shouldn’t this return <code>True</code>? If not, How do I inverse this operation?<br>
I tried to run this on MNIST images, and visually I also see a difference at the edges of the digits.</p>

isalirezag: <p>any updates on this question?</p>

ptrblck: <p><code>F.fold</code> sums overlapping blocks, which is the case here.<br>
If you don’t have overlapping windows, it’ll work:</p>
<pre><code class="lang-python">x = torch.rand(32, 3, 16, 16)
torch.equal(F.fold(F.unfold(x, 2, stride=2), x.shape[2], 2, stride=2), x)
</code></pre>
Activation of every neuron at test time

shreyasbapat: <p>Can someone please tell an easy way to get activation matrix of every layer at test time?</p>

el_samou_samou: <p>Hi,<br>
I am not sure it is the best option but I would write a custom function of my new module class which outputs the output of each layer. So lets say you have 3 layers:</p>
<pre><code class="lang-auto">class custom_module(nn.Module):
    def __init__(self):
    ....
    def forward(self, x):
    ....


    def forward_test(self, x):
        res = {}
        x = self.layer1(x)
        res["layer_1"] = x
        x = self.layer2(x)
        res["layer_2"] = x
        x = self.layer3(x)
        res["layer_3"] = x
        return res
</code></pre>
<p>And at test time you call:</p>
<pre><code class="lang-auto">for batch in dataloader:
     with torch.no_grad():
         out = custom_module.forward_test(batch)
</code></pre>
Error on Batch Multiplication

watchdog: <p>I have two matrices, <code>A,B</code> .</p>
<p>A is a normal map, with shape: [b,h,w,3] and B is a transformation matrix with shape [b,4,4,3], where b is for batch_size and 3 for the number of channels.<br>
I want to implement the rendering equation by:  (A.T *B *A) . The output should be a matrix that should be multiplied afterwards with the texture(b,h,w,3) so it would have to be same or HDR. I tried doing the aforementioned multiplications via torch.bmm() but failed. Any idea?</p>
Torch c++ vs python about cost time

Ryan95: <p>Hi, I want to know whether libtorch must faster than python?  For example, if i run VGG16, Resnet50 on the CPU, how much faster libtorch can be than python?</p>
<p>when i run this experiment, i got a result that their time is very closed!. But when i test addmm function, libtorch can be 3~5x faster than python. Is this normal? why a pretrained model’s performance is very different from function’s？</p>

tom: <p>Well, so a cartoon is that Python introduces a roughly constant overhead per instruction. When you have a chain of relatively cheap instructions - say, like the lltm example in the C++ extension tutorial - the raw benefit of moving to C++ is about 10%. When you have more expensive operations - and I’d expect the typical convnets on not too small inputs to be so - the gain you might have is much smaller.</p>
<p>There are many factors and pitfalls to properly benchmarking and comparing things, so it’s hard to tell without a detailed description of what you did.</p>
<p>Don’t feel bad about it, though! Many of us will have forgotten a <code>torch.cuda.sync()</code> or accidentally used a debug build or I personally even just misread measurements late at night to drop what would have been a promising optimization…</p>
<p>Best regards</p>
<p>Thomas</p>
Abou the libtorch | torch.jit.trace

menguangwen: <p>I use torch.jit.trace as follow:</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>class simpleNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(simpleNet, self).<strong>init</strong>()<br>
self.layer1 = nn.Conv2d(3,16,3)<br>
self.layer2 = nn.Conv2d(16,32,3)<br>
self.layer3 = nn.Conv2d(32,64,3)</p>
<pre><code>def forward(self, x):
    x = self.layer1(x)
    x1 = self.layer2(x)
    x2 = self.layer3(x1)
    return x1,x2
</code></pre>
<p>img = torch.rand(1, 3, 416, 416)<br>
model = simpleNet()</p>
<p>traced_script_module = torch.jit.trace(model, img)<br>
traced_script_module.save(“val_libtorch.pt”)</p>
<p>But i think there may be something wrong because there are two tensor about the model out put…</p>
<p>and in c++ api :<br>
std::shared_ptr<a>torch::jit::script::Module</a> module = torch::jit::load(“D:\lib_test\tensor_libtorch\val_libtorch.pt”);<br>
assert(module != nullptr);<br>
std::vector<a>torch::jit::IValue</a> inputs;<br>
inputs.push_back(torch::ones({ 1, 3, 416, 416 }));<br>
auto out_put = module-&gt;forward(inputs).toTensor();<br>
…<br>
I will get  wrong</p>

ptrblck: <p>Could you return the outputs as a <code>tuple</code> and get it in C++ via:</p>
<pre><code class="lang-auto">auto outputs = module-&gt;forward(inputs).toTuple();
torch::Tensor out1 = outputs-&gt;elements()[0].toTensor();
torch::Tensor out2 = outputs-&gt;elements()[1].toTensor();
</code></pre>

menguangwen: <p>Thank you for your help.</p>
Training Multiple Bounding Boxes

ahmed: <p>I am trying to train a network using multiple bounding boxes. One image can have more than one bounding box associated with it. This is the form a bounding box looks like when there are 3 objects in the image:</p>
<pre><code class="lang-auto">tensor([[[444., 220.,  27.,  65.],
         [468., 220.,  26.,  66.],
         [415., 224.,  20.,  33.]]])
</code></pre>
<p>I am using a simple training loop while trying to get it to work:</p>
<pre><code class="lang-auto">for epoch in range(2):
    for i, data in enumerate(trainloader, 0):
        images, target = data
        box = target['nboxes'].view(1, -1)
        optimizer.zero_grad()
        out_box = net(images)
        loss = criterion_bbox(out_box, box)
        loss.backward()
        optimizer.step()
</code></pre>
<p>The <code>box = target['nboxes'].view(1, -1)</code> line outputs the boxes as the boxes with the shape: <code>torch.Size([1, 12])</code>. I am just wondering if this is the best/correct approach?</p>
<p>My network is:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, num_classes):
        super(Net, self).__init__()
        self.base = nn.Sequential(
        nn.Conv2d(3, 6, 5),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),
        nn.Conv2d(6, 16, 5),
        nn.ReLU(),
        Flatten(),
        nn.ReLU(),
        nn.Linear(1256000, 157),
        nn.Linear(157, 84),
        nn.Linear(84, 5)    
        )
        self.out_bbox = nn.Linear(5, 4)

    def forward(self, x):
        x = self.base(x)
        x = x.view(x.size(0), -1)
        x_bbox = self.out_bbox(x)
        return x_bbox

net = Net(num_classes=5)
x = torch.randn(1, 3, 512, 640)
output = net(x)

criterion_bbox = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
</code></pre>
How to plot a heatmap by pytorch

mshmoon: <p>I want to plot a heat map of features by pytorch ,but I  do not know how to do it.</p>

The_Rajnikanth: <p>One way would be to convert the tensor to an ndarray and use seaborn/matplotlib to plot the heatmap.</p>

mshmoon: <p>Thank you for you help</p>

solsol: <p>Hi,</p>
<p>I have a tensor with shape of [x, y, z, z]. First I converted it to numpy:<br>
Tensor_a = (Tensor_a).cpu().numpy()<br>
Then I tried to plot it as:<br>
Tensor_a= sns.heatmap(Tensor_a, linewidth=0.5)<br>
plt.show()</p>
<p>but i faced this error:<br>
raise ValueError(‘Must pass 2-d input’)</p>
<p>Any help please?</p>
Cuda() and DataParallel

jack1234: <p>Do we need to call <code>cuda()</code> for model and data if we use <code>DataParallel</code>?</p>
<p>Say we have four GPUs, specifically there are three questions:</p>
<p>a. If we do not call <code>cuda()</code>, the model and data is on CPU, will it be any time inefficiency when it is replicated to 4 GPUs?</p>
<p>b. If we call <code>cuda()</code>, the model and data is on GPU <span class="hashtag">#1</span>, will it be any space inefficiency in terms of replicate it again on GPU <span class="hashtag">#1</span>, or it won’t be replicate again if the model/data was there?</p>
<p>c. Overall, for time/space efficiency, should we call <code>cuda()</code> if we use <code>DataParallel</code>?</p>
PriorBox layer in libtorch

jinfagang: <p>In order to run SSD or Pelee or RFBNet so many one stage detector framework we gonna need PriorBox layer.</p>
<p>I can use PriorBox in CPU but that would be ineffient, does there any implementation of this layer in libtorch???</p>
<p>So many thanks to community if anyone could give a advice.</p>
Tensor.topk() so slow?

EthanZhangYi: <p>Hi, all<br>
I found that the speed of function <strong>tensor.topk()</strong> is very slow when evaluating a model for semantic segmentation. Below is the code snippet. output is <strong>2x5x297x817</strong> and it takes about <strong>3.6s</strong> to calculate top-1 on GPU. Is there any optimization for the code?</p>
<pre><code class="lang-auto">output_var = model(input_var)
output = output_var.data
torch.cuda.synchronize()
start = time.time()
_, pred = output.topk(1, 1, True, True)
torch.cuda.synchronize()
print("Top one Time :{}".format(time.time() - start))
</code></pre>

tom: <p>Would top-1 not be equivalent to max?</p>

EthanZhangYi: <p>Yes,top-1 is equivalent to max(), and max() is very fast. Sometimes I just want to get top-3, and topk() is very slow. Forward time of CNN is 0.27s, however topk() takes nearly 3s…</p>

jetcai1900: <p>how did you solve this problem? Thanks.</p>
What should I do if I want to add something in cell node in LSTM?

kangkang_li: <p>Just like below<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/03ebbc51876c6ab01e06a3652d1c435106d01476.png" alt="image" data-base62-sha1="yGuBzClzKecNSDelcyJB9EPa3c" width="527" height="233"><br>
The red one is what I want to add. But LSTMCell and LSTM in nn seems don’t support this modification. Should I have to realize the LSTM by myself?</p>
What does net.to(device) do in nn.DataParallel

tengerye: <p>The following code from the tutorial to <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">pytorch data paraleelism</a> reads strange to me:</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = Model(input_size, output_size)
if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)

model.to(device)
</code></pre>
<p>According to my best knowledge, <code>mode.to(device)</code> copy the data to GPU.</p>
<blockquote>
<p>DataParallel splits your data automatically and sends job orders to multiple models on several GPUs. After each model finishes their job, DataParallel collects and merges the results before returning it to you.</p>
</blockquote>
<p>If the <code>DataParallel</code> does the job of copying, what does the <code>to(device)</code> do here?</p>

pietern: <p>It moves the model weights to GPU.</p>

tengerye: <p>If so, what does <code>nn.DataParallel(model)</code> do then?</p>

pietern: <p>On calling <code>forward</code> it splits the input into multiple chunks (one chunk per GPU), replicates the underlying model to multiple GPUs, runs <code>forward</code> on each of them, and gathers the outputs.</p>

tengerye: <p>Thank you. I think I need to read more core code of pytorch to fully understand.</p>
How to displaying an image with 6 channels instead of 3

boostedwind: <p>I had created my own dataloader to work with the <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="nofollow noopener">CycleGan project</a>.</p>
<p>I generated a new DataSet that cat 2 images together to form an input with 6 channels. This also causes the output of the network to be 6 channels.</p>
<p>When I tried passing the output into Visdom to display, this was the error message i got</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 45, in &lt;module&gt;
    visualizer.display_current_results(model.get_current_visuals(), epoch, save_result)
  File "/home/stormaggedon/fyp/util/visualizer.py", line 109, in display_current_results
    padding=2, opts=dict(title=title + ' images'))
  File "/home/stormaggedon/pytorch-CycleGAN-and-pix2pix/env/lib/python3.7/site-packages/visdom/__init__.py", line 335                                                                                                                        , in wrapped_f
    return f(*args, **kwargs)
  File "/home/stormaggedon/pytorch-CycleGAN-and-pix2pix/env/lib/python3.7/site-packages/visdom/__init__.py", line 975                                                                                                                        , in images
    grid[:, h_start:h_end, w_start:w_end] = tensor[k]
ValueError: could not broadcast input array from shape (6,64,64) into shape (3,64,64)

</code></pre>
<p>Any suggestions on what I can do to solve this?</p>

PT_587: <p>I hope someone else can give a better answer. I had done something similar and found I needed to break it into two 3 channel images like img[:3] &amp; img[3:] if you want,  you can do something like a cv2.addWeighted to see all 6 channels in one image by combining the two 3 channel images back to one 3 channel image and showing that.</p>
Training CNNs using real deconvolution techniques

cxy: <p>The current way of training CNNs is likely to be inefficient and indirect. Can pytorch developers give a closer look to this issue?</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/22330" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/yechengxi" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/2723537?v=2&amp;s=96" class="thumbnail onebox-avatar" width="96" height="96">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/22330" target="_blank" rel="nofollow noopener">Issue: Training CNNs with deconvolution</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/yechengxi" target="_blank" rel="nofollow noopener">yechengxi</a>
	on <a href="https://github.com/pytorch/pytorch/issues/22330" target="_blank" rel="nofollow noopener">2019-06-28</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🚀 Feature
Implicitly calculate the scatter/covariance matrix of the im2col matrix.
def F(I):
""
Please help us to do the following implicitly:
"""
X= im2col(I)
Cov=X@X.t()/N
return Cov
Motivation
Neural networks...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: operators</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">needs research</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: convolution</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
What will parallel model do when calling the forward function

yangzhh: <p>I encountered a problem that it seems the data movement operation takes long time in parallel forward, but I can’t find the reason. The main codes are as follow,</p>
<pre><code class="lang-python">def main():
    model = nn.DataParallel(MyModel())
    model = model.cuda()

    before_time = time.time()
    logits = model(input)


class MyModel(nn.Module):
   def forward(self, input):
      start_time = time.time()
      xxx
      end_time = time.time()
      return out
</code></pre>
<p>I use 4 V100 GPU in parallel, and num_workers set to 0.</p>
<p>The <em>start_time - before_time</em> takes 0.6s, and the actual forward time <em>end_time-start_time</em> only takes 0.15s. Data batch is (512 * 3 * 224 * 224).</p>
<p>I am curious that the actual forward time is only 0.15 seconds, but what operations takes 0.6 seconds during before_time and start_time? Is it the data movement operation? I don’t think that such few data would cost 0.6 seconds because 0.6s is too long for moving the 77M data.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

pietern: <p>I suspect that calling <code>model.cuda()</code> on <code>nn.DataParallel</code> is causing trouble. Can you try creating the model first, then calling <code>model.cuda()</code>, and <em>then</em> wrapping it in <code>nn.DataParallel</code>? I think that in this example the entire model is copied to GPU every time you call <code>forward</code>.</p>

yangzhh: <p>Thank you! I will try it and then reply. The original style is inspired from <a href="https://github.com/quark0/darts/blob/master/cnn/train_imagenet.py#L92" rel="nofollow noopener">DARTS</a>.</p>
Random training runs don't converge

amirhf: <p>I’m training a basic VGG network on a vision dataset of 48x48 images. I run the same script multiple times to report the average but some runs randomly don’t converge. Meaning sometimes classification accuracy doesn’t increase and loss doesn’t decrease. This happens randomly (e.g. 1 run out of 5 runs). Where should I start to debug?! what are the possible culprits?</p>

krishnamurthy: <p>This is pretty common when training deep networks. A lot of the underlying processes (eg. initialization of network weights, convolutional backend, mini-batch sequencing, hyperparameters etc.) are randomized. Some configurations perform better/worse than others.</p>
<p>If you want your experiments to be perfectly repeatable, consider adding something like this to your main script.</p>
<pre><code class="lang-auto">randomseed = 12345 # Or your favourite integer
np.random.seed(randomseed) # If you are using numpy functions anywhere
torch.manual_seed(randomseed)
torch.cuda.manual_seed(randomseed) # If you are using pytorch on a GPU
# One other source of randomness, that most people ignore
torch.backends.cudnn.deterministic = True
# If you are using python's random module, also include this one
random.seed(randomseed)
</code></pre>

amirhf: <p>I agree. The randomness is intrinsic and one might prefer deterministic when the target is production. But when doing research most people average their results between runs and check against any modification that they might have done. How about that situation. Can it be for example due to the learning rate being on the borderline? and by this I mean learning rate sometimes causes the model to converge and sometimes it doesn’t.</p>

ptrblck: <p>This might be a reason, but I would rather suspect the parameter initializations to create these failing runs sometimes.<br>
Do you use any custom init functions or the default ones?</p>

amirhf: <p>Yes I am using the default VGG initializers:</p>
<pre><code class="lang-auto">def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.Linear):
                nn.init.normal_(m.weight, 0, 0.01)
                nn.init.constant_(m.bias, 0)
</code></pre>
<p>I am using a custom cross-entropy which is basically the same as the built-in function. I don’t think it is the reason though!</p>
<pre><code class="lang-auto">class Loss(torch.nn.Module):
    def __init__(self):
        super(Loss, self).__init__()
        self.epsilon = 1e-7
        self.log_softmax = nn.LogSoftmax(dim=1)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, prediction, target):
        clamped = torch.clamp(-self.log_softmax(prediction), self.epsilon, 1000)
        loss = torch.mean((torch.sum(target.float() * clamped, dim=1)))
        return loss
</code></pre>
Speed up a for loop in pytorch

Xiaokang_Wang: <p>Hi<br>
I have an input tensor of  n*p.  p is equal to k times q, which means in the p columns, every k columns are a group of features.</p>
<p>Meanwhile, I have a weight tensor of k*1. So I use a for loop to do multiplication between every k column of the input and the weight. It is slow.</p>
<p>So I am thinking it is possible to use vectorization to speed up the for loop. I got stuck on how to convert the weight tensor to a p*q tensor. The new weight tensor has a specific pattern (see the following image).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b24adbf340ed5bc9554177ca5b46757ddaa90b6d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b24adbf340ed5bc9554177ca5b46757ddaa90b6d" title="vectorization"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b24adbf340ed5bc9554177ca5b46757ddaa90b6d_2_666x500.jpeg" alt="vectorization" data-base62-sha1="prfneI17A7nB3dtJF0gJ92QRzUV" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b24adbf340ed5bc9554177ca5b46757ddaa90b6d_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/b/b24adbf340ed5bc9554177ca5b46757ddaa90b6d.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b24adbf340ed5bc9554177ca5b46757ddaa90b6d.jpeg 2x" data-dominant-color="F1F0E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">vectorization</span><span class="informations">960×720 46.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Xiaokang_Wang: <p>One alternative way is to reshape the input and keep the shape of the weight unchanged. But I need to reshape back the product. So I am still thinking about how to reorganize the weight. Maybe I should use a binary mask.</p>

styx97: <p>Does this solve your loop problem ? I guess if (w1,1) goes to (k*w1,k) (here k = 2), it’s hard to avoid a loop.</p>
<pre><code class="lang-auto">import numpy as np 

w1 = np.asarray([1,2,4])
w1.shape = (3,1)

new_w = np.zeros((2*w1.shape[0],2)) 
new_w[:w1.shape[0],0] = w1[:,0]
new_w[w1.shape[0]:,1] = w1[:,0]
</code></pre>

Xiaokang_Wang: <p>Thanks for your reply. In your solution, you created a new_w and assign the old way at specific locations. I can do it in this way.</p>
<p>Another concern is that whether I can use SGD to update the original weight after I created a new weight matrix from the original weight.</p>
Clonging the optimizer

blackbirdbarber: <p>Is there a way to clone the optimizer?</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=lr, nesterov=True, momentum=0.9)
</code></pre>
<p>I would like to use the same optimizer for multiple examples, but when I set the same, it already has some history.</p>
<p>The other option would be to clear the optimizer to defaults.</p>

blackbirdbarber: <p>Found it, model and optimizer need to go together, since optimizer uses what model has.</p>
<pre><code class="lang-auto">model = M()
optimizer = torch.optim.SGD(model.parameters(), lr=lr, nesterov=True, momentum=0.9)
</code></pre>
<p>So If I don’t create a new model I would still be on old model, which was a probelm I had.</p>
Manual back propagation in Neural network

Rajamannar_Krishnamo: <p>In  Neural network when performing back propagation we get change weights when we get these values we have to either add or subtract the change of weights from the existing weights. So my question is how do we get to know when to add and subtract when I am doing the back propagation(while doing manually) it’s not like we plot each iteration to check whether it’s increasing or decreasing.<br>
<a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noopener"> Step by step back propagation example</a></p>
<p>In the above post he has either adding or subtracting the change in weights randomly there was no explanation for it.</p>

ptrblck: <p>The vanilla gradient descent formula updates the parameters as:</p>
<pre><code class="lang-auto">new weights = old weights - learning_rate * gradient
</code></pre>
<p>In the linked tutorial the same formula is used as far as I can see:</p>
<blockquote>
<p>To decrease the error, we then subtract this value from the current weight (optionally multiplied by some learning rate, eta, which we’ll set to 0.5):</p>
</blockquote>

Chris_Gottbrath: <p>This might help</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#update-the-weights" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#update-the-weights</a></p>
Huge overhead when using torch.jit.trace

coderinuse: <p>Hi, I’m trying to serialize my model using <code>TorchScript</code>. Since there is no control flow in the module, I have decided to just <code>trace</code> the module. But when I profile the script I see the weird picture (Profiling is done in sync mode <code>CUDA_LAUNCH_BLOCKING=1</code>):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e98be0c025e6030c000e9f3530b0fb5b7f826fe1.png" data-download-href="https://discuss.pytorch.org/uploads/default/e98be0c025e6030c000e9f3530b0fb5b7f826fe1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e98be0c025e6030c000e9f3530b0fb5b7f826fe1_2_690x65.png" alt="image" data-base62-sha1="xk2W8wz7YGUarP573X4KpYL8WPL" width="690" height="65" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e98be0c025e6030c000e9f3530b0fb5b7f826fe1_2_690x65.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e98be0c025e6030c000e9f3530b0fb5b7f826fe1_2_1035x97.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e98be0c025e6030c000e9f3530b0fb5b7f826fe1_2_1380x130.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1701×162 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Overall running time has been increased and there is a huge overhead when calling the <code>__call__</code> method. What is the cause of the issue? In my big model I have only traced one module and there are subsequent modules following it.</p>
<p>The module that I’m trying to serialize is the Pose ResNet.</p>
Char RNN Not Outputting EOS Tokens

eholmdahl: <p>Hi All,</p>
<p>I’m trying to implement a character RNN for the purpose of spell correction and tokenization. The model is based on the practical pytorch GRU-RNN implementation (<a href="https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation-batched.py" rel="nofollow noopener">https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation-batched.py</a>) - the loss function is masked cross entropy like they use here and I’m using a 2 layer bidirectional GRU for my encoder/decoder. I am also using scheduled sampling with a teacher forcing ratio of 0.5.</p>
<p>The problem I’m having is that the system is struggling to output end tokens. Sentences such as ‘do3s t4is w0rk’ are being correctly changed to ‘does this work’, but then random tokens are outputted until either the system reaches it’s max output length or seemingly randomly puts down an EOS token, making the output something like ‘does this workkkkkk orwk EOS’.</p>
<p>Does anyone have any insight into this problem? I tried unmasking my loss function but this lead to the system outputting PAD tokens now. The system is able to overfit very easy sets so I don’t think there’s a bug in the training. Any input would be appreciated!</p>

JohnCwok: <p>Up !<br>
I have the same problem. Have you found the solution yet ?</p>
Is it possible to send a model to a cuda kernel?

ivan_rodriguez: <p>Hello,</p>
<p>I am new with pytorch and I am trying to implement pytorch c++ with a cuda/c++ project.</p>
<p>My question is the following, is it possible to do something like this:</p>
<p>//// Neural Net. definition ////</p>
<p>struct Net : torch::nn::Module<br>
{<br>
Net(int inputSize, int hiddenSize, int numClasses):<br>
fc1(register_module(“fc1”, torch::nn::Linear(inputSize, hiddenSize))),<br>
fc2(register_module(“fc2”, torch::nn::Linear(hiddenSize, numClasses))) {};</p>
<pre><code>torch::nn::Linear fc1;
torch::nn::Linear fc2;

torch::Tensor forward(torch::Tensor input) 
{
    auto out = fc1(input);
    out = torch::relu(out);
    out = fc2(out);
    
    return out;    

}
</code></pre>
<p>};</p>
<p>//// Use Neural Net in CUDA kernel</p>
<p><strong>global</strong> void testKernel (Net model)<br>
{<br>
… use model here …<br>
}</p>
<p>Thanks for any help!!</p>
RuntimeError when putting nn to gpu

JachinMa: <p>net is a neural network I desired,I want to put it to gpu</p>
<p>In the pytorch tutorial,the code is:</p>
<p>net.to(device)</p>
<p>but when I run this code,an error occurs:</p>
<p>CUDA error: unknown error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/035da6264168b96276199fbb4f37e7f0524bfe0a.png" data-download-href="https://discuss.pytorch.org/uploads/default/035da6264168b96276199fbb4f37e7f0524bfe0a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/035da6264168b96276199fbb4f37e7f0524bfe0a.png" alt="image" data-base62-sha1="tM4CqLHFbVNwLiSetNiAZVoZN8" width="380" height="499" data-dominant-color="F3F5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">636×835 15.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>by the way,my cuda is available</p>
<p>so if anyone could help me with this problem,very thx to you</p>

ptrblck: <p>Are you able to create CUDATensors:</p>
<pre><code class="lang-python">x = torch.randn(1, device='cuda')
</code></pre>
<p>Also, could you post your current setup, i.e. PyTorch, CUDA, and driver versions?</p>

JachinMa: <p>yes,I’m able to create CUDATensor,my CUDA is available.</p>
<p>my PyTorch version is 1.1，CUDA version is 10.2,my video card is GTX1060(if driver version means this).</p>
<p>strangely,this morning when I ran that code again,there was no error anymore,I am confused.</p>

ptrblck: <p>Oh, maybe your driver was in an invalid state and a restart helped?<br>
Anyway, let us know, if you encounter this error again.</p>

JachinMa: <p>ok,if this error occurs again,I will let you know</p>
<p>thx for help</p>
Why can‘t I import pandas in pytorch for deeplearning？

JachinMa: <p>I have installed pandas,if I create an ipynb file via python3<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e7ad03f1015b33552c3e3edecb6811377fa0406f.png" alt="image" data-base62-sha1="x3uYJ0tRFWpGfDhJGFSQQjWeEFh" width="213" height="74"><br>
I can import pandas in this file</p>
<p>but if I create an ipynb file via pytorch for deeplearning<br>
I can’t import pandas</p>
<p>my environment is win10</p>
<p>if anyone could help me with this problem?<br>
very thx</p>

ptrblck: <p>You can import <code>pandas</code> in an environment where PyTorch is also installed.<br>
Most likely <code>pandas</code> is not installed in the environment <code>pytorch for Deeplearning</code>.<br>
How did you install it?<br>
Could you try to activate the environment and install it via</p>
<pre><code class="lang-auto">conda install pandas
</code></pre>
<p>in your terminal?</p>

JachinMa: <p>very thx to you !</p>
<p>I tried to install pandas in PyTorch environment,and solved my problem.</p>
<p>although conda install pandas wasn’t useful,I tried pip install pandas,and sucessfully installed it.</p>
<p>now I can use pandas in pytorch for deeplearning</p>
<p>thx again !</p>
How can I handly write parallel code using PyTorch?

yangzhh: <p>I’m going to convolve kernel <code>K</code> with a list of input data <code>X = [x1, x2, x3]</code>. Using the code below could work but is slow for it runs in serial.</p>
<pre><code class="lang-python">for x in X:
  out = F.conv2d(x, K)
</code></pre>
<p>I wonder if I could handly write parallel code like parfor(Matlab) using the PyTorch lib?</p>
<pre><code class="lang-python">parfor x in X:
  out = F.conv2d(x, K)
</code></pre>
<p>Or are there other mechanisms for writing parallel code?</p>
<p>Thanks a lot!</p>

LeviViana: <p>Here is an example of how to spawn parallel processes using <code>torch.distributed.launch</code>. You should save this snippet as a python module (say  <code>torch_dist_tuto.py</code> ) then run  <code>python -m torch.distributed.launch --nproc_per_node=4 torch_dist_tuto.py</code> to launch 4 parallel processes. It sums all numbers from 0 to 9 in a parallel fashion.</p>
<pre><code class="lang-python">import torch
import argparse
import os
import torch.distributed as dist

parser = argparse.ArgumentParser()
parser.add_argument("--local_rank", type=int)
args = parser.parse_args()

rank = args.local_rank
size = int(os.environ["WORLD_SIZE"]) if "WORLD_SIZE" in os.environ else 1

dist.init_process_group("gloo", rank=rank, world_size=size)

x = torch.arange(10)[rank::size].sum()

print("Process rank {}, partial result {}".format(rank, x))

dist.reduce(x, dst=0)
if rank == 0: print("Final result:", x)
</code></pre>
<p>You can adapt this snippet to your code. The <code>rank</code> variable will allow you to control parallelism.</p>

yangzhh: <p>Thanks, <a class="mention" href="/u/leviviana">@LeviViana</a></p>
<p>I have written some MPI codes years ago, if I remember right, this is for distributing the parallel code, and then open many process to run. It seems much more like Distributed Parallel (not sure)? And I have only one computer with 4 cards, I want to parallel the forward function within the main process, but using 4 GPU for acceleration. I wonder if this could work too?</p>

LeviViana: <p><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">This</a> is a nice tutorial on how to do what you want using <code>torch.nn.DataParallel</code>.</p>

yangzhh: <p>Thank you! I will try it!</p>
Model.cuda() in pytorch

jack1234: <p>if I call <code>model.cuda()</code> in pytorch where model is where model is a subclass of <code>nn.Module</code>, and say if I have four GPUs, how it will utilize the GPUs and how do I know which GPUs that are using?</p>

ptrblck: <p><code>model.cuda()</code> will push the parameters to the default device.<br>
If you print a parameter’s <code>device</code>, you should see, which GPU is used:</p>
<pre><code class="lang-python">print(model.fc.weight.device)
&gt; device(type='cuda', index=0)
</code></pre>
<p>Also, have a look at <code>nvidia-smi</code> to see, which GPUs are used.</p>
<p>If you would like to use data parallel (copy the model onto all GPUs and send chunks of the batch to each GPU), have a look at <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">this tutorial</a>.</p>
How to use multiple parameters for optimizer

maris_ancans: <p>in pytorch it is possible to do:</p>
<pre><code class="lang-auto">params = list(fc1.parameters()) + list(fc2.parameters())
torch.optim.SGD(params, lr=0.01)
</code></pre>
<p>How can I do this with c++ API?<br>
Im not sure im doint it right like this:</p>
<pre><code class="lang-auto"> auto encoder = std::make_shared&lt;net::Encoder&gt;(3);
   auto decoder = std::make_shared&lt;net::Decoder&gt;(3);

   auto params = encoder-&gt;parameters(true);
   auto decParams = decoder-&gt;parameters(true);
   params.insert(params.begin(), decParams.begin(), decParams.end());
   auto adamOpts = torch::optim::AdamOptions(0.1);
   torch::optim::Adam optimizer(params, adamOpts);
</code></pre>

maris_ancans: <p>Looks like this method works!<br>
If anyone stumbles upon this, just concacenate network parameter vectors</p>
Embeddings shows index out of range if `padding_id` is -1

hiro-o918: <h2>Overview</h2>
<p>I want to use <code>-1</code> as an ignore index of an embedding layers.<br>
But it does not work.</p>
<h2>Environment</h2>
<p>Python: 3.7.1<br>
torch: 1.1.0</p>
<h2>To Reproduce</h2>
<pre><code class="lang-python">embedding = nn.Embedding(10, 3, padding_idx=-1)
tokens = torch.tensor([-1, 2]).long()
embedding(tokens)
</code></pre>
<p>The third line, in the above, shows</p>
<pre><code class="lang-auto">RuntimeError: index out of range at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:193
</code></pre>

Deepali: <pre><code class="lang-auto">&gt;&gt;&gt; embedding = nn.Embedding(10, 3, padding_idx=-1)
&gt;&gt;&gt; tokens = torch.tensor([1, 2]).long()   # positive indexes here
&gt;&gt;&gt; embedding(tokens)
tensor([[-0.6424, -1.2020,  0.4287],
        [ 0.5234,  1.2113,  0.8808]], grad_fn=&lt;EmbeddingBackward&gt;)
</code></pre>

hiro-o918: <p>Thank you for your reply.</p>
<p>I think <code>padding_idx=-1</code> means <code>-1</code> map to <code>[0., 0., 0.]</code>, as the example of <a href="https://pytorch.org/docs/stable/nn.html#embedding" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#embedding</a>, but is it not correct?</p>
<p>Probably, does <code>padding_idx=-1</code> mean there are no padding index?</p>

Deepali: <p>I think -ve padding means negative indexing.</p>

blackbirdbarber: <p>Just set</p>
<pre><code class="lang-auto">tokens = torch.tensor([3, 2]).long()
</code></pre>
<p>This may be the embedding index problem &lt;0.<br>
Negative is bad in here.<br>
Also, by default is long so you don’t need to use <code>long()</code></p>
Labeling class Name

Shakir_Khurshid: <p>**The model is  79.82897162437439 % certain that the image has a predicted class of  <strong>2</strong></p>
<p><em>How to make the output show the real name if the class instead of the index ?</em></p>
<pre><code class="lang-auto">def predict(image, model):
    # Pass the image through our model
    image = image.to(device)
    output = model.forward(image)
    
    # Reverse the log function in our output
    output = torch.exp(output)
    
    # Get the top predicted class, and the output percentage for
    # that class
    probs, classes = output.topk(1, dim=1)
    
   
    
    return probs.item(), classes.item()
</code></pre>

ptrblck: <p>You could create a mapping between the class indices and the corresponding names using a <code>dict</code>:</p>
<pre><code class="lang-python">idx_to_name = {
  0: 'class0',
  1: 'class1',
  ...
}

print(idx_to_name[target])
</code></pre>