akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How could I update attributes of a RRef?

Kunlin_Yang: <p>I am studying the tutorial on parameter server <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="nofollow noopener">parameter server</a>. I would like to update the grad of parameters during training (i.e. between <code>dist_autograd.backward(cid, [loss])</code> and <code>opt.step(cid)</code>). How could I achieve this?</p>

mrshenli: <p>Hey <a class="mention" href="/u/kunlin_yang">@Kunlin_Yang</a>, say, <code>param_rref</code> is the RRef of the corresponding parameter, you can do sth like the following:</p>
<pre><code class="lang-python">import torch.distributed.autograd as dist_autograd
...

def update_grad(param_rref, cid):
    grad = dist_autograd.get_gradients(cid)[param_rref.local_value()]
    # do inplace update on the grad
    with torch.no_grad():
        grad.add_(1)

# following is a toy training step
with dist_autograd.context() as cid:
    loss = model(inputs).sum()
    dist_autograd.backward(cid, [loss])
    rpc.rpc_sync(param_rref.owner(), update_grad, args=(param_rref, cid))
    opt.step(cid)
</code></pre>
<p>Curious, what is the reason to manually update the grad instead of using the grad given by the backward pass?</p>

Kunlin_Yang: <p>Thanks! That’s very helpful. I am doing federated learning research, so I need to add noise to gradients of local worker.</p>
Assigning value to variable

Vishu_Gupta: <p>I have been reading a few of the neural network codes and came across the following:<br>
<code>hidden_ele = [256]</code><br>
Is it just assigning the size to the variable or is it assigning the variable with the value 256? I was not exactly sure what it is doing.</p>

Isaac_Kargar: <p>it is definition of list <code>hidden_ele</code> with one value of <code>256</code>.</p>
Anomoly loss function

sukhadj: <p>Hey, I am working on a loss function that detects anomaly of a model with another model.<br>
Consider any model and loss function as this:</p>
<pre><code class="lang-auto">
def loss_anom(global_model, worker_model):
    global_model_copy = global_model.copy()
    
    loss_ = 0.0
    for p,q in zip(global_model_copy.parameters(), worker_model.parameters()):
        loss_ += torch.sum((p-q)**2)

    print(loss_.copy())
    return loss_**0.5
</code></pre>
<p>My optimizer works on worker_model parameters.<br>
Now problem, I am facing is that when I do loss.backward(), its setting the grads of parameters as <strong>NaN</strong>.<br>
Is there any workaround this? I am making any mistake here?</p>

KFrank: <p>Hi Sukhad!</p>
<aside class="quote no-group" data-username="sukhadj" data-post="1" data-topic="81507" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/278dde/48.png" class="avatar"> sukhadj:</div>
<blockquote>
<pre><code class="lang-auto">    global_model_copy = global_model.copy()
</code></pre>
</blockquote>
</aside>
<p>What is <code>.copy()</code>?  Is that a method you implemented yourself?</p>
<p>Could you post a complete, runnable mini-script that illustrates<br>
your <code>loss_anom</code> definition, your call to <code>loss_anom()</code> and your<br>
call to <code>loss.backward()</code>?  (Create some dummy data for your<br>
models, as appropriate, to keep your example script self-contained.)</p>
<p>Please also post the output of your mini-script, together with your<br>
expectations for what it should have produced.</p>
<p>Best.</p>
<p>K. Frank</p>
Parameter: weight decay- optimizer ADAM

Mike2004: <p>Hello, can someone explain me better, what the weight decay parameter in optimizer ADAM, does?<br>
Thank you.</p>

jmandivarapu1: <aside class="quote no-group" data-username="Mike2004" data-post="1" data-topic="81523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mike2004/48/24981_2.png" class="avatar"> Mike2004:</div>
<blockquote>
<p>someone explain me better, what the weight decay parameter in optimizer ADAM, does?<br>
Thank you.</p>
</blockquote>
</aside>
<p>The weight_decay parameter adds a L2 penalty to the cost which can effectively lead to to smaller model weights.</p>
<aside class="quote quote-modified" data-post="1" data-topic="33105">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ba9def/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-sgd-weight-decay-work/33105">How does SGD weight_decay work?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hello, 
i write a toy code to check SGD weight_decay. 
but it seems to have no effect to the gradient update. 
am i misunderstand the meaning of weight_decay? 
thank you very much. 
PyTorch 1.0 

import torch
import numpy as np

np.random.seed(123)
np.set_printoptions(8, suppress=True)

x_numpy = np.random.random((3, 4)).astype(np.double)
w_numpy = np.random.random((4, 5)).astype(np.double)
x_torch = torch.tensor(x_numpy, requires_grad=True)
w_torch = torch.tensor(w_numpy, requires_grad=True)

l…
  </blockquote>
</aside>

<aside class="onebox allowlistedgeneric" data-onebox-src="https://dejanbatanjac.github.io/2019/07/02/Impact-of-WD.html">
  <header class="source">

      <a href="https://dejanbatanjac.github.io/2019/07/02/Impact-of-WD.html" target="_blank" rel="noopener nofollow ugc">dejanbatanjac.github.io</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/360;"><img src="https://dejanbatanjac.github.io/images/monai.png" class="thumbnail" width="690" height="360"></div>

<h3><a href="https://dejanbatanjac.github.io/2019/07/02/Impact-of-WD.html" target="_blank" rel="noopener nofollow ugc">Impact of Weight Decay</a></h3>

  <p>Logistic Regression is a single linear layer (nn.Linear in PyTorch) neural network used with CrossEntropyLoss PyTorch criterion. We can write a custom module...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Mike2004: <p>Thank you for you time, now i understand!</p>
How to move a new defined class/struct into CUDA device?

jingyu_han: <p>I have created a struct named <code>TargetData</code> , which includes size, index, and intrinsics of each image.</p>
<pre><code class="lang-auto"> struct TargetData{

            int64_t img_idx;
            torch::Tensor tensor_pre;
            torch::Tensor tensor_cur;
            torch::Tensor tensor_next;

            int img_width;
            int img_height;
            torch::Tensor img_K;
            // torch::Tensor img_invK;
        };
</code></pre>
<p>when training my own model, one thing I need to do is moving this kind of<code> struct</code> into the CUDA device. how to move a personalized class/struct into the Cuda device? any references or cues would be highly appreciated.</p>
Pytorch: a shared conv layer

lifeblack: <p>Hi!<br>
I have two different networks. The parameters of one of the Conv layers are shared in every space it appears. what should I do? How could I use the parameters of the layer in one network to another network?</p>

jmandivarapu1: <p>you can look at this thread or you can copy the weights of convolutional layer from one to another whenever you make a change to any one of them like <code>model.conv1.weight=model2.conv1.weight</code></p>
<aside class="quote quote-modified" data-post="1" data-topic="66551">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wafaa_wardah/48/53794_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-run-2-parallel-cnn-on-same-data-then-combine-outputs-at-fc-layer/66551">How can I run 2 parallel CNN on same data then combine outputs at FC layer?</a> 
  </div>
  <blockquote>
    I want to have a CNN architecture with the below pipeline: 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b00a1b1b3e34bdc8be4471a57c20e658e7d5f91" title="DeepInsightArchitecture">[DeepInsightArchitecture]</a> 
I’ve done normal CNN using PyTorch before. Can someone please suggest if there’s a feature in PyTorch that let’s me easily split as above? 
Thanks so much in advance. 
<a class="mention" href="/u/ptrblck">@ptrblck</a> Please help <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title="pensive" alt="pensive" class="emoji">
  </blockquote>
</aside>
Transfer Learning With Resnet18 on CIFAR10: Poor Training Accuracy

fk4517: <p>Hi,</p>
<p>I am playing around with the Pytorch library and trying to use Transfer Learning.</p>
<p>My code is as follows:</p>
<pre><code class="lang-auto"># get the model with pre-trained weights
resnet18 = models.resnet18(pretrained=True)

# freeze all the layers
for param in resnet18.parameters():
    param.requires_grad = False
    
# print and check what the last FC layer is: 
# Linear(in_features=512, out_features=1000, bias=True)
print(resnet18)

# set the final FC layer to what we require for our problem
resnet18.fc = nn.Linear(512, 10)

# unfreeze the last layer so it learns on our dataset
for param in resnet18.fc.parameters():
  param.requires_grad = True
</code></pre>
<p>I set the optimizer as:</p>
<pre><code class="lang-auto"># set optimizer
lr = 1e-2
optimizer = torch.optim.SGD(resnet18.parameters(), lr=lr, momentum=0.5)
</code></pre>
<p>Training this model on CIFAR10 gives me a very poor training accuracy of 44%.</p>
<p>Am I doing transfer learning correctly here? I would have expected much better results.</p>
<p>I followed the tutorial here:</p>
<p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>

ptrblck: <p>Are you using the same transformations as used in the tutorial?<br>
Also, have you played around with some hyperparameters, e.g. did you try to lower the learning rate?</p>

fk4517: <p>Hi did lower the learning rate for the last layer but it didn’t perform much better. If I unfreeze the layers and set a very low learning rate the model performs much better. In terms of transformations I simply did normalization and toTensor.</p>
Adding nn.dataparallel to autoencoder

SU801T: <p>Hi,</p>
<p>I’m training a simple autoencoder over several GPUs (probably 4) with a batch size of 256 to 512. I have millions of examples to train. I want to make sure I am doing the right thing.</p>
<p>Below I have defined the autoencoder where I add <code>self.encoder= nn.DataParallel(self.encoder) </code>. Please see below:</p>
<pre><code class="lang-auto">class AutoEncoder(nn.Module):
    def __init__(self, n_embedded):
        super(AutoEncoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(6144, n_embedded))
        self.encoder= nn.DataParallel(self.encoder)
        self.decoder = nn.Sequential(nn.Linear(n_embedded, 6144))
       
    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded
</code></pre>
<p>I initiate my model before training by:</p>
<pre><code class="lang-auto">model = AutoEncoder(2048)
model= nn.DataParallel(model)
model.to(device)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),weight_decay=1e-5)

</code></pre>
<p>Would this get the most out of the GPUS?</p>

mrshenli: <p>Hey <a class="mention" href="/u/su801t">@SU801T</a> I think you need do call  <code>model.to(device)</code> before calling <code>DataParallel</code> ctor.</p>
<blockquote>
<p>Would this get the most out of the GPUS?</p>
</blockquote>
<p><code>DistributedDataParallel</code> is expected to be faster than <code>DataParallel</code>. See this <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">example</a></p>

SU801T: <p>Hi,</p>
<p>I have another issue.</p>
<p>So I have placed model.to(device) before calling nn.DataParallel. However, get this error:</p>
<pre><code class="lang-auto">RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1

</code></pre>
<p>I’m assuming everything needs to be on cuda0 before splitting to the other devices. How do I fix that?</p>
<p>Cheers,</p>
<p>Taran</p>

mrshenli: <p>The code below works for me. Your original code has a <code>DataParallel</code> submodule within <code>AutoEncoder</code> (commented out below)? Is that intentional?</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class AutoEncoder(nn.Module):
    def __init__(self, n_embedded):
        super(AutoEncoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(n_embedded, n_embedded))
        #self.encoder= nn.DataParallel(self.encoder)
        self.decoder = nn.Sequential(nn.Linear(n_embedded, n_embedded))

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded


n_embedded = 20
model = AutoEncoder(n_embedded)
model.to(0)
model= nn.DataParallel(model)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),weight_decay=1e-5)
loss = model(torch.ones(n_embedded, n_embedded))[0].sum()
loss.backward()
</code></pre>

SU801T: <p>Yes, apologies. I did uncomment that line in the autonencoder class. I thought perhaps it was also needed…</p>
Reduce number of operations in this PyTorch code

mohit117: <p>I have a input variable ‘fm’ which needs to be transformed into ‘fm1’ and ‘fm3’ to be processed by the system. However, in doing so I create a temporary variable ‘fm2’ which I do not require at all.</p>
<p>Since i want to reduce time and memory of my deep model and make it as fast as possible I suspect it should be possible to directly obtain ‘fm3’ from ‘fm1’ with less number of ‘view()’ and ‘permutations()’ and ‘fm2’ is not really required. Can someone please help me on this?</p>
<p>Alternatively obtaining ‘fm3’ directly from ‘fm’ will also work.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">h=16 # any number expressible as a power of 2
w=32 # any number expressible as a power of 2
fm = torch.arange(1*1*h*w).view(1,1,h,w)
print(fm)

b,c,h,w = fm.size()
R=2 
out_channel = c*(R**2)
out_h = h//R
out_w = w//R
fm1 = fm.view(b, c, out_h, R, out_w, R).permute(0,1,3,5,2,4).contiguous().view(b,out_channel, out_h, out_w)
print(fm1)

b,c,h,w = fm1.size()
r=4 
out_channel = c*(r**2)
out_h = h//r
out_w = w//r
fm2 = fm1.view(b, c, out_h, r, out_w, r).permute(0,1,3,5,2,4).contiguous().view(b,out_channel, out_h, out_w)
# This fm2 is temporary and is not required

b,c,h,w = fm2.size()
G=R**2
fm3 = fm2.view(b, G, c // G, h, w).permute(0, 2, 1, 3, 4).contiguous().view(b, c, h, w)
print(fm3)
</code></pre>
<p>The output is as follows (for brevity here h,w=8; R=2 and r=2),</p>
<pre><code class="lang-auto">fm: tensor([[[[ 0,  1,  2,  3,  4,  5,  6,  7],
          [ 8,  9, 10, 11, 12, 13, 14, 15],
          [16, 17, 18, 19, 20, 21, 22, 23],
          [24, 25, 26, 27, 28, 29, 30, 31],
          [32, 33, 34, 35, 36, 37, 38, 39],
          [40, 41, 42, 43, 44, 45, 46, 47],
          [48, 49, 50, 51, 52, 53, 54, 55],
          [56, 57, 58, 59, 60, 61, 62, 63]]]])

 fm1: tensor([[[[ 0,  2,  4,  6],
          [16, 18, 20, 22],
          [32, 34, 36, 38],
          [48, 50, 52, 54]],

         [[ 1,  3,  5,  7],
          [17, 19, 21, 23],
          [33, 35, 37, 39],
          [49, 51, 53, 55]],

         [[ 8, 10, 12, 14],
          [24, 26, 28, 30],
          [40, 42, 44, 46],
          [56, 58, 60, 62]],

         [[ 9, 11, 13, 15],
          [25, 27, 29, 31],
          [41, 43, 45, 47],
          [57, 59, 61, 63]]]])

 fm3: tensor([[[[ 0,  4],
          [32, 36]],

         [[ 1,  5],
          [33, 37]],

         [[ 8, 12],
          [40, 44]],

         [[ 9, 13],
          [41, 45]],

         [[ 2,  6],
          [34, 38]],

         [[ 3,  7],
          [35, 39]],

         [[10, 14],
          [42, 46]],

         [[11, 15],
          [43, 47]],

         [[16, 20],
          [48, 52]],

         [[17, 21],
          [49, 53]],

         [[24, 28],
          [56, 60]],

         [[25, 29],
          [57, 61]],

         [[18, 22],
          [50, 54]],

         [[19, 23],
          [51, 55]],

         [[26, 30],
          [58, 62]],

         [[27, 31],
          [59, 63]]]])
</code></pre>
Seeking NLP interviewees

A_G: <p>Hello,<br>
I am a MSc student and this semester I’m participating in a business plan seminar with a teammate. We wanted to ask you a few questions regarding your experience working and researching in the NLP field. Do you have time to speak with us sometime in the next week via Google Meet. We would only ask 5-10 minutes of your time.<br>
Best Regards</p>
Model.load_state_dict() returns NonType

Oktai15: <p>Hi there!</p>
<p>I successfully trained my model that was declared as:</p>
<pre><code class="lang-auto">model = ModelClass(...)
model = torch.nn.DataParallel(model, device_ids=[0, 1, 2])
</code></pre>
<p>During training, I saved my weights like this:</p>
<pre><code class="lang-auto">best_model = copy.deepcopy(model.state_dict())
torch.save(best_model, path)
</code></pre>
<p>And I could get my weights for test (it isn’t NoneType, I checked):</p>
<pre><code class="lang-auto">w = torch.load(path) # type(w) is &lt;class 'collections.OrderedDict'&gt;
</code></pre>
<p>But! When I try:</p>
<pre><code class="lang-auto">trained_model = model.load_state_dict(w)
</code></pre>
<p>I got NoneType! What has happened? Maybe it happens due to DataParallel? What should I do?</p>

ptrblck: <p>You don’t have to assign the <code>trained_model</code> to <code>load_state_dict</code>. <code>model</code> should have all weights loaded.<br>
Have a look at the <a href="http://pytorch.org/docs/master/notes/serialization.html">Serialization info</a>.</p>

Oktai15: <p>As far as I understood, all weights will be downloaded to <code>model</code> properly, right?</p>

ptrblck: <p>Yes, you could also verify it by printing some weights (<code>print(model.layer_name.weight)</code>) before and after loading.</p>

J_Liu: <p>You may try the procedure:<br>
define model=some_model(), and device=torch.device(‘cuda: 0,1,2’),<br>
and model=torch.nn.DataParallel(model, device_ids=[0,1,2]), model=model.to(device),<br>
then load the pretrained model:<br>
model.module.load_state_dict(torch.load(‘path/to/model.pth’)</p>

InnovArul: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="15904">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You don’t have to assign the <code>trained_model</code> to <code>load_state_dict</code></p>
</blockquote>
</aside>
<p>as mentioned in the answer, you do not need to assign the model to return value of <code>load_state_dict()</code>.</p>
<p>Use of the following is enough:</p>
<pre><code class="lang-auto">G.load_state_dict(torch.load('G.pth'))
...
img = G(z)
</code></pre>

mathematics: <p>oh my fault. It is solved. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .<br>
edit: I deleted my question .</p>
Join 4 outputs from different CNN into 1 output

pulpaul: <p>Hello there, I have the following architecture:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d6782d233e670dcbb280b01a3928beb2b47ddaa.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d6782d233e670dcbb280b01a3928beb2b47ddaa" title="Captura de pantalla de 2020-05-16 13-19-23"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d6782d233e670dcbb280b01a3928beb2b47ddaa_2_690x285.png" alt="Captura de pantalla de 2020-05-16 13-19-23" data-base62-sha1="kaV5RUGD4LSY5uZHCkIZtVER3tw" width="690" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d6782d233e670dcbb280b01a3928beb2b47ddaa_2_690x285.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d6782d233e670dcbb280b01a3928beb2b47ddaa_2_1035x427.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d6782d233e670dcbb280b01a3928beb2b47ddaa_2_1380x570.png 2x" data-dominant-color="E7E4E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Captura de pantalla de 2020-05-16 13-19-23</span><span class="informations">1489×617 241 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I train 4 differents CNN, with 4 different scales and N randoms crops of each. I have 8 classes in total. After training I have each CNN trained with 4 final outputs (4 outputs with 8 classes each). The 8 classes are the same. But when I have to validate I want to have only 1 output with 8 classes. I dont know how to join the 4 outputs (FC layers) into output (FC layer) of 8 classes, like in the picture.</p>
<p>The CNNs are ResNet50.</p>
<p>Thank you very much.</p>

Anshumaan_Dash: <p>If you are getting 4 class results (max_index) out of the 4 classifiers, just take majority voting. That would be the simplest way to ensemble the results.</p>

pulpaul: <p>Thanks <a class="mention" href="/u/anshumaan_dash">@Anshumaan_Dash</a>  I’ll try</p>
How to speed up if each instance in a batch needs to be processed separately?

xinyiding: <p>Hi,</p>
<p>I have this implementation for a LSTM network, I am iterating through each step using the LSTMCell. For each step, I have to transform the input data with dimension [batch_size, embedding_size] to [batch_size, hidden_size]. However, for each instance in this batch, I am applying a different Linear layer based on the actual instance. Thus, I can not just use batch operation and I have to process each instance individually. However, it is very slow. Wonder there is any way to speed this up? Thank you very much!</p>
<p>Below is my code:</p>
<p>self.m1 and self.m2 are shared by all instance, so this is not the problem. The problem is this self.encoders. It is an array of Linear layers. Based on the actual instance, the corresponding Linear layer has to be used.</p>
<pre><code class="lang-auto">        for i in range(self.num_steps):
            # modality 1
            input_data = input_1[:,i]
            m2_data = input_2[:, i]
            current_skills = routers_info[:, i]

            # Routing
            batch_size = 32
            # current_pred = []
            tmp_data = torch.FloatTensor(batch_size, self.hidden_size)
            tmp_data.zero_()
            for b in range(batch_size):

                out_1 = self.m1(input_data[b])
                out_2 = self.m2(m2_data[b])
                fused = torch.cat((out_1, out_2), 0)
                out = self.encoders[current_skills[b]](fused)
                tmp_data[b] = out

            input_data = tmp_data
            h_t, c_t = self.lstm1(input_data, (h_t, c_t))
</code></pre>

googlebot: <p>Sure, just get rid of “array of layers” (and then of loop). As linear layer only does fused.matmul(W) + B, create (batched) W and B instead - nn.Embedding may be appropriate for that.</p>
<p>I should add that matmul will probably need to be replaced with torch.bmm (shapes (B,1,in) @ (B,in,out) = (B,1,out))</p>

xinyiding: <p>Thanks Alex, I will definitely try that</p>
Startified Sampling for multiclass

James_W: <p>I am quite new to pytorch so please bear with me here. I have a CSV dataset which looks like so:</p>
<p>class_label,image_location<br>
1, /some/loc0<br>
2, /some/loc1<br>
0, /some/loc2<br>
1 /some/loc4</p>
<p>where the class_label is my <code>target</code> and the<code>image_location</code> is my input to NN.</p>
<p>I would like to use a dataloader to <code>somehow</code> split this into train and test sets, with a stratified sampling of each class in the CSV (20 classes, 4 <code>images</code> per class in the CSV).</p>
<p>When googling around, I see some pandas and scikit-learn solutions, so, beginning with something like:</p>
<pre><code class="lang-auto">sss = StratifiedShuffleSplit(df['event'], n_iter=1, test_size=0.2)
</code></pre>
<p>which I presume gives you a single train and test split which I can use with <code>dataSet</code> later batch with <code>DataLoader</code>. However, I am not sure if this is an elegant solution and I was wondering if someone could point me in the right direction to schieve this.</p>
Simulate data in IterableDataset

mertensu: <p>Hi there,</p>
<p>in my application, I continuously simulate the data that are fed into my NN. I would like to implement this using <code>IterableDataset</code>. All examples I found online assume all the training data to be already present. Can you give me a short example on how to continuously generate new batches of data. Imagine one training example consists of 10 random normal numbers (<code>torch.randn(10)</code>), so that a final batch should have shape (batch_size, 10).</p>
<p>Thanks in advance for your help!</p>
Where is the bias used in this example

Shekar_Neo: <p>I am bit confused and new to pytorch, i was going through pytorch with examples</p>
<p>y_pred  = inputs * weights + bias</p>
<p>but in the below example where did they used bias</p>
<pre><code class="lang-auto">import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h &lt; 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2```</code></pre>

Nikronic: <p>Hi,</p>
<p>If you are using <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="nofollow noopener">this page</a>, note that the main purpose is to demonstrate the convention of PyTorch and its similarities and differences with numpy, etc. So, they are not really focusing on teaching deep learning stuff.</p>
<p>On the other hand, <code>bias</code> is not mandatory in learning models and can be omitted. For instance, in another example on this page (<a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn" rel="nofollow noopener">this link</a>), it uses <code>torch.nn.Linear(D_in, H)</code> and in the comment it says that it calculates weights and biases. Although, if we see the documentation of <a href="https://pytorch.org/docs/master/generated/torch.nn.Linear.html" rel="nofollow noopener">nn.Linear</a>, we see that bias is optional argument.</p>
<p>bests</p>

Shekar_Neo: <p>thanks for the info, i was wrong with bias</p>
Bug in memory efficient DenseNet?

papyan: <p>Dear PyTorchers,</p>
<p>I am trying to train the densenet model, implemented in torchvision, in the memory efficient mode (memory efficiency is obtained by trading computational complexity for memory via gradient checkpointing).</p>
<p>However, the minimal example below shows that some of the weighs are not being updated. Namely, the weights of conv1 are unchanged whereas those of conv2 are changed.</p>
<p>Is this a bug? Has anyone encountered this?</p>
<p><strong>EDIT:</strong> I tried running test_checkpoint_module_list, defined in <a href="https://github.com/pytorch/pytorch/blob/master/test/test_utils.py" rel="nofollow noopener">pytorch/test/test_utils.py</a>, on densenet and the test <strong>fails</strong>.</p>
<p>Kind regards,<br>
Vardan</p>
<pre><code class="lang-python">import torch
from torchvision import datasets, transforms
from torchvision.models.densenet import _densenet

model = _densenet('densenet121', 32, (1, 1, 1, 1), 64,
                  pretrained=False, progress=True,
                  memory_efficient=True,
                  num_classes=10)

train_transforms = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.ToTensor(),
])

train_set = datasets.CIFAR10('data', train=True, download=True,
                             transform=train_transforms)

train_loader = torch.utils.data.DataLoader(train_set, batch_size=4)

optimizer = torch.optim.SGD(model.parameters(), lr=1)

for batch in train_loader: break

input, target = batch

output = model(input)

loss = torch.nn.functional.cross_entropy(output, target)

conv1 = list(model.features)[4].denselayer1.conv1.weight.clone()
conv2 = list(model.features)[4].denselayer1.conv2.weight.clone()

optimizer.zero_grad()
loss.backward()
optimizer.step()

new_conv1 = list(model.features)[4].denselayer1.conv1.weight.clone()
new_conv2 = list(model.features)[4].denselayer1.conv2.weight.clone()

print(torch.norm(conv1 - new_conv1))
print(torch.norm(conv2 - new_conv2))
</code></pre>

Marksein: <p>I found the performance is quite different between with and without memory efficient implement by checkpoint in training process.</p>
<p>Do you know anything wrong?</p>

rwightman: <p><a class="mention" href="/u/marksein">@Marksein</a> <a class="mention" href="/u/papyan">@papyan</a> I just ran into this issue, founding this looking for confirmation, and pretty sure I fixed the bug</p>
<p>change the original code here…</p>
<pre><code class="lang-auto">    def call_checkpoint_bottleneck(self, input):
        # type: (List[Tensor]) -&gt; Tensor
        def closure(*inputs):
            return self.bn_function(*inputs)

        return cp.checkpoint(closure, input)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">    def call_checkpoint_bottleneck(self, input):
        # type: (List[Tensor]) -&gt; Tensor
        def closure(*inputs):
            return self.bn_function(inputs) # remove *

        return cp.checkpoint(closure, *input) # add *
</code></pre>
<p>Very subtle!</p>
Error: ‘indexing’ is not a namespace-name

yyd: <p>While trying  <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html" rel="nofollow noopener">indexing with c++</a>  with the code below,the compiler reports  error: ‘indexing’ is not a namespace-name</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;vector&gt;
#include &lt;torch/nn/cloneable.h&gt;
#include &lt;torch/nn/pimpl.h&gt;
#include &lt;torch/types.h&gt;
#include &lt;ATen/ATen.h&gt;
#include &lt;torch/csrc/autograd/generated/Functions.h&gt;
#include &lt;cmath&gt;

using namespace torch::indexing;
int main()
{
            
        return 0;
}
</code></pre>

Hartbook: <p>You need to install the last stable release 1.5 to use this feature</p>
Cross entropy loss for sentence classification does not improve accuracy

sajastu: <p>I’m trying to use CrossEntropyLoss criterion for sentence classification. The task is basically a multi-classification problem where each sentence can be of only one class (given three classes). I’m having trouble knowing which approach I should follow when calculating the loss.</p>
<p>I’m using masking for the instances as well. The problem is that the loss is decreasing till some points, so I expect the accuracy gets elevated, but it does not in practice. I’m now wondering if there’s something wrong with my intuition (and approach) to calculate and backpropagate through the computed loss:</p>
<pre><code class="lang-auto">self.loss_sect = torch.nn.CrossEntropyLoss(reduction='none')
loss_sect = self.loss_sect(sent_sect_scores.permute(0, 2, 1), sent_sect_labels)
loss_sect = (loss_sect * mask.float()).sum(dim=1).mean()
loss_sect.backward()
</code></pre>
<p>Above, <code>sent_sect_scores</code> is a tensor in the shape of: <code>[batch_size, sentence_num, section_scores]</code> where <code>section_scores</code> are the outputs from the neural network with leakyReLu activation funtion. <code>sent_sect_labels</code> has the shape of <code>[batch_size, sentence_num]</code> containing the labels associated with each sentence (i.e., labels between 0 and 2). Is it a correct proactive to compute the loss? Then why isn’t the accuracy improving in my training?</p>
<p>PS: I also used the following snippet, but it didn’t help improving accuracy neither:</p>
<pre><code class="lang-auto">loss_sect = self.loss_sect(sent_sect_scores.permute(0, 2, 1), sent_sect_labels)
loss_sect = (loss_sect * mask.float()).sum(dim=1)
loss_sect = (loss_sect / torch.sum(mask, dim=1).float()).mean()
</code></pre>
<p>And this is the log report of some training steps:</p>
<blockquote>
<p>[2020-05-16 00:10:38,775 INFO] Step 50/152000; xent_sect: 9.72 (ACC: 0.4217)<br>
[2020-05-16 00:11:06,709 INFO] Step 100/152000; xent_sect: 9.00 (ACC: 0.3430)<br>
…</p>
<p>[2020-05-16 00:14:21,781 INFO] Step 450/152000; xent_sect: 6.10 (ACC: 0.2967)<br>
[2020-05-16 00:14:49,663 INFO] Step 500/152000; xent_sect: 6.68 (ACC: 0.3027)<br>
[2020-05-16 00:15:17,563 INFO] Step 550/152000; xent_sect: 6.55 (ACC: 0.3126)<br>
[2020-05-16 00:15:45,208 INFO] Step 600/152000; xent_sect: 6.26 (ACC: 0.3179)</p>
</blockquote>
Xid 79 GPU disappears after long training period

Albert_T: <p>I have a model that trains for &gt;8 hours, and completes without issue. However, immediately after training, the GPU goes into P0 instead of P8 (where it was before), and causes an ERR reading on fan speed and power usage (after reading ERR it goes into P5). I’m forced to reboot the system every time this happens (which is consistently happening now, which is quite annoying). The model training program itself doesn’t use anything fancy, except setting CUDA_VISIBLE DEVICES (<a href="https://discuss.pytorch.org/t/cuda-visible-devices-make-gpu-disappear/21439/16" class="inline-onebox">CUDA_VISIBLE_DEVICES make gpu disappear</a>), but that thread doesn’t appear to be the same problem. Running nvidia-bug-report.sh gives Xid79 and GPU has fallen off the bus, which indicates an overheating error. However, GPU temps are stable at 60-70C when training and 30C at idle.</p>
<p>Since I am currently working from home, I can’t physically go check in on the machine, so I’m guessing as to what the issue might be. Some things that I’ve considered 1) GPU failure 2) motherboard/PCIE bus failure (I’ve disabled ASPM) 3) machine PSU failure 4) Some pytorch thing.</p>
<p>Has anyone ever experienced something similar caused by Pytorch? Is there something I’m missing software wise that would cause this? Most likely it is a hardware issue, but since I don’t have physical access to the machine I’m hoping I can fix this.</p>
<p>The exact setup of the machine is a dual socket Haswell system, with one 2070 on the PCIE lanes for one of the CPU’s (CPU 1 in NUMA) and a 2070 and 1070 attached to the PCIE lanes for the other CPU (CPU0 in NUMA). The failure is happening on the 2070 that shares a CPU with the 1070. The 1070 is on bus 04:00.0 and the 2070 is on bus 05:00.0 for that socket, so I did consider if some PCIE error was causing all but the first bus to lose connection (since the 1070 and other 2070 are both on the first PCIE bus). The machine PSU is 1300W, which should be more than sufficient for such a system.</p>
<p>What’s odd is running GPU-burn for 1 hr doesn’t cause issues, and training some example pytorch models from online for a few hours doesn’t cause issues either.</p>

ptrblck: <p>Was the device working before or have you seen similar failures?<br>
You could check the drivers and update them, but besides that my best guess is that Xid79 might be hardware related (GPU, PCIe bus etc.).</p>

Albert_T: <p>Sorry for the late reply, the device was working before. I’ve concluded its probably a card issue and have just masked out that card for now until I can RMA it.</p>
Double* to tensor and back

teddykoker: <p>I have a model that excepts an input of shape <code>[1, 1, 1000]</code>. I am creating a real-time application in which I would like to feed in a buffer (of type <code>double*</code>) into the model, and then write the output back into the buffer. I am having trouble with the <code>double*</code> to <code>torch::Tensor</code> conversion and back.</p>
<p>Example <strong>Python</strong>:</p>
<pre><code class="lang-python">input = torch.rand(1, 1, 1000)
input = model(input)
</code></pre>
<p><strong>C++</strong>:</p>
<pre><code class="lang-cpp">double* input; // written to by some other function
int len = 1000;
torch::jit::script::Module model; // loaded in from torchscript
/* 
Now how do I convert input into a tensor of shape [1, 1, len],
pass it through the model, and convert it back into an array of
type double*
*/

// EDIT: figured out conversion to tensor
std::vector&lt;torch::jit::IValue&gt; input_tensors;
input_tensors.push_back(torch::from_blob(input, {1, 1, len}, torch::kFloat64));
torch::Tensor output_tensor = model.forward(input_tensors).toTensor();
// now how do copy back into double*
</code></pre>
<p>Thanks in advance!</p>

glaringlee: <p>I think you can use torch::from_blob.<br>
Take a look at the comment in this old issue<br>
‘<a href="https://github.com/pytorch/pytorch/issues/37201" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/37201</a>’<br>
You need to specify torch::kFloat64 as the input type in torch::from_blob in your case.</p>

teddykoker: <p>Thanks! Using <code>from_blob</code> I can create a tensor from <code>double*</code>, now how would I get it back into <code>double*</code> after passing through the model?</p>

teddykoker: <p>I think I figured it out: I can use <code>std::memcpy()</code>:</p>
<pre><code class="lang-cpp">double* input;  // address of input
double* output; // address of output
int len = 1000; // length of input and output buffers
torch::jit::script::Module model; // model loaded in from torchscript

// Convert input to tensor
std::vector&lt;torch::jit::IValue&gt; input_tensors;
input_tensors.push_back(torch::from_blob(input, {1, 1, len}, torch::kFloat64));

// Forward pass
torch::Tensor output_tensor = model.forward(input_tensors).toTensor();

// Copy into output
std::memcpy(output_tensor.data_ptr(), output, sizeof(double)*len);
</code></pre>
Error when torch.load() model, libcaffe2_nvrtc.so not found

sonnguyen: <p>Dear all,</p>
<p>I tried to load a pre-train model saved in *.pth by using torch.load().<br>
It always return <em>RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory</em><br>
I tried to export path as in <a href="https://github.com/pytorch/pytorch/issues/31554" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory · Issue #31554 · pytorch/pytorch · GitHub</a>, still the same error.<br>
Even I changed directory to the folder which has the file  <em>libcaffe2_nvrtc.so</em>, it still report same error.</p>
<blockquote>
<p>./               libcaffe2_detectron_ops_gpu.so*    libshm.so*<br>
…/              libcaffe2_module_test_dynamic.so*  libtorch.so*<br>
libc10.so*       libcaffe2_nvrtc.so*                libtorch_python.so*<br>
libc10_cuda.so*  libcaffe2_observers.so*</p>
</blockquote>
<p>I used torch version 1.3.1 installed in a linux conda environment.</p>

Alexandru_Mihaila: <p><a class="mention" href="/u/sonnguyen">@sonnguyen</a> have you find a solution?</p>

sonnguyen: <p>Yes, kind of. I upgraded it to PyTorch 1.4 and problem solved.</p>

chilango: <p>I’m having the same issue appearing under different circumstances: it happens only when I try to run a script from the terminal using CUDA. If I don’t the script runs fine.</p>
<p>I am running:</p>
<pre><code class="lang-auto">pytorch 1.4.0 py3.7_cuda10.1.243_cudnn7.6.3_0
Linux 5.6.13 
</code></pre>
Pytorch: tensor multiplication

lifeblack: <p>Hi!<br>
I want to code this matrix multiplication:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3ce46b9cab7be3b9b7bc6f5a93da9bfd8a9e565.png" alt="image" data-base62-sha1="udIGQK4MGU7d4HKU6SQTBhxWXDD" width="428" height="252"><br>
B=16, h=32, w=56, b*b=64, n=25<br>
my code:</p>
<pre><code class="lang-auto">torch.mul(torch.rand([16, 32, 56, 64]),torch.rand([16, 32, 56, 64,25]))
</code></pre>
<p>but I encountered in with this error:</p>
<pre><code class="lang-auto">The size of tensor a (64) must match the size of tensor b (25) at non-singleton dimension 4
</code></pre>
<p>Would you mind helping me to fix this problem?</p>

albanD: <p>Hi,</p>
<p>I think the simplest is to write it as a matrix matrix multiplication. And so add a temporary dimension of size 1 on the first Tensor:</p>
<pre><code class="lang-auto">torch.matmul(torch.rand([16, 32, 56, 1, 64]),torch.rand([16, 32, 56, 64,25])).squeeze(-2)
</code></pre>

lifeblack: <p>Thanks for your reply. The problem solved.<br>
Would you mind explaining it? I do not understand it.</p>

albanD: <p>It consider <code>Bxhxw</code> as batch dimensions. And <code>1x(bb)</code> and <code>(bb)xn</code> for the other dimensions of each other dimensions.<br>
Doing the matrix multiply between the two, will give you a Tensor with <code>1xn</code> and with the same batch size.<br>
The squeeze removes the dimension of size 1.</p>

lifeblack: <p>Thanks for your help dear <a class="mention" href="/u/alband">@albanD</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using all_reduce

Jeroen_Vermunt: <p>whenever I run my code using torch I get this error:</p>
<pre><code class="lang-auto">~\AppData\Local\Continuum\Anaconda3\envs\pytorch CVdevKit\lib\site-packages\CVdevKit\core\dist_utils.py in average_gradients(model)
     24     for param in model.parameters():
     25         if param.requires_grad and not (param.grad is None):
---&gt; 26             dist.all_reduce(param.grad.data)
     27 
     28 def broadcast_params(model):

AttributeError: module 'torch.distributed' has no attribute 'all_reduce'
</code></pre>
<p>Now I found out that all_reduce should be in distributed_c10d which is in the distibuted folder, so I tried running the following:</p>
<pre><code class="lang-auto">from torch.distributed import distributed_c10d

~\AppData\Local\Continuum\Anaconda3\envs\pytorch CVdevKit\lib\site-packages\torch\distributed\rendezvous.py in &lt;module&gt;
      5 
      6 import os
----&gt; 7 from . import FileStore, TCPStore
      8 
      9 

ImportError: cannot import name 'FileStore' from 'torch.distributed' (C:\Users\s158979\AppData\Local\Continuum\Anaconda3\envs\pytorch CVdevKit\lib\site-packages\torch\distributed\__init__.py)
</code></pre>
<p>It seems there is a whole bunch missing from my enviroment? Does anyone know how I can solve this? I’ve been stuck for a few hours already.</p>

mrshenli: <p>Looks like <code>torch.distributed</code> is not enabled in your env. Can you run <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.is_available" rel="nofollow noopener"><code>is_available</code></a> to find out.</p>
<p>Given the path in log, it seems you are using Windows? PyTorch distributed is not available on Windows yet. Please comment <a href="https://github.com/pytorch/pytorch/issues/37068" rel="nofollow noopener">here</a> if you need this feature.</p>