akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LSTM for images with more than one channel

Thabang_Lukhetho: <p>I’m trying to use a vanilla LSTM to model video frame sequences as a benchmark approach.</p>
<p>I have the following image data set with the following input dimensions (batch_size=256, channels=2, height=64, width=64), I want it to match the following LSTM input format (sequence_length, batch_size, input_size)</p>
<p>I wanted to pass in the input  images row by row</p>
<p>Here is my implementation of the LSTM i’m using<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74a7872ee8fd0559beb6452a37014f187c9ab5cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/74a7872ee8fd0559beb6452a37014f187c9ab5cb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74a7872ee8fd0559beb6452a37014f187c9ab5cb.png" alt="image" data-base62-sha1="gDYjsbAilPwfrGpbHB6Sdv2kJpF" width="560" height="500" data-dominant-color="222323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1024×913 47.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
The way to implement attention-mask/uni-direction attention in TransformerDecoder

AOZMH: <p>Hi guys,</p>
<p>I’m learning about nn.Transformer in pytorch these days and I’m a bit confused about the implementation of the attention mask in decoder. Say we’re doing a machine translation task using Transformer, when inferencing, the output of each time step can only “see” the tokens before it. However, when training, we simply feed the correct sequence to the decoder and thus a token in the decoded sequence can see the tokens both before and after it, which I guess is not suitable for a robust model.</p>
<p>I know that GPT adopted single directional attention when decoding (and I guess it’s suitable for a decoder), but I’m wondering if the APIs in nn.Transformer can realize such feature (e.g. using attention masks for different tokens to avoid it from attending on later tokens)? If so, what should I do?</p>
<p>Any help would be appreciated, thanks in advance!</p>

zhangguanheng66: <p>You should set up the attention mask for the decoder to mask the token from see the one before it.</p>

AOZMH: <p>Thanks for the reply! Could you please offer more concrete hints, e.g. ways to setup attention masks, sample codes or a link that provide these? Many thanks!</p>

zhangguanheng66: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/5551061414d3bcf202de520d20e8163f58eb664a/word_language_model/model.py#L126" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/5551061414d3bcf202de520d20e8163f58eb664a/word_language_model/model.py#L126" target="_blank" rel="nofollow noopener">pytorch/examples/blob/5551061414d3bcf202de520d20e8163f58eb664a/word_language_model/model.py#L126</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="116" style="counter-reset: li-counter 115 ;">
<li>    self.src_mask = None</li>
<li>    self.pos_encoder = PositionalEncoding(ninp, dropout)</li>
<li>    encoder_layers = TransformerEncoderLayer(ninp, nhead, nhid, dropout)</li>
<li>    self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)</li>
<li>    self.encoder = nn.Embedding(ntoken, ninp)</li>
<li>    self.ninp = ninp</li>
<li>    self.decoder = nn.Linear(ninp, ntoken)</li>
<li>
</li>
<li>    self.init_weights()</li>
<li>
</li>
<li class="selected">def _generate_square_subsequent_mask(self, sz):</li>
<li>    mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)</li>
<li>    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))</li>
<li>    return mask</li>
<li>
</li>
<li>def init_weights(self):</li>
<li>    initrange = 0.1</li>
<li>    self.encoder.weight.data.uniform_(-initrange, initrange)</li>
<li>    self.decoder.bias.data.zero_()</li>
<li>    self.decoder.weight.data.uniform_(-initrange, initrange)</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


AOZMH: <p>Many thanks for the code, it’s much clearer right now!</p>
How to process c++ forward multiple return values

111117: <p>my model return value like this , it is detection model<br>
output = (<br>
tensor1, # loc preds<br>
tensor2 # conf preds<br>
)</p>
<p>how to get return values in c++</p>
<p>i try this</p>
<blockquote>
<p>auto outputs = module-&gt;forward(inputs).toTuple();<br>
cout &lt;&lt; ouputs &lt;&lt; endl;</p>
</blockquote>
<p>the stdout is 1,it’s not ok.</p>

Edwinzero: <p><a class="mention" href="/u/111117">@111117</a> Hello, do you have a solution to this problem? I come across the same problem and cannot find any solution yet.</p>

Hartbook: <p>You could return one tensor containing all the information (like the concatenation of tensor1 and tensor2), and then use view or narrow to retrieve the relevant tensor.</p>

Edwinzero: <p>Hi <a class="mention" href="/u/hartbook">@Hartbook</a>, could you give an example?<br>
What if the shape of tensors are different. How to retrieve or partition the returned single tensor?<br>
Thank you very much.</p>
Problem with tensorboard epochs

mahsa: <p>I have a for loop and in every epoch, I add some scalars by using writer.add_scalar, but when I open tensorboard, the number of epochs drawn in tensorboard is not the same as loop epochs. For example, in a loop of 100 epochs, it draws 90 epochs. The add_scalar is run in every epoch (I printed the epoch number), I have absolutely no idea what I should check to find the problem.</p>
RNN learn within epoch but no between epochs

marco_zaror: <p>Hi,<br>
I’ve been working on an RNN model to predict the next word, but something weird is hapenning with the loss. It decreases within an epoch, but when a new epoch starts, it go back to the beginning. Here is my code:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
    current_loss = 0
    current_loss_test = 0
    random.shuffle(batches)
    for i,batch in enumerate(batches):
        loss = torch.zeros(1,dtype=torch.float,requires_grad=True)
        hidden_1 = rnn_1.initHidden()
        optimizer_1.zero_grad()
        for tensor in batch[0]: #For each word
            output_1, hidden_1 = rnn_1(tensor, hidden_1) 
        loss = two_dim_loss(output_1,output_2,batch[1], hidden_1,hidden_2)
        loss.backward()
        optimizer_1.step()
        hidden_1s.append(hidden_1)
        current_loss += loss.item()
</code></pre>
<p>The graph shows the effect that I’m talkign about:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44e134a26dd1182bb391f75ce662f3353a86654e.png" alt="loss" data-base62-sha1="9PkY3Xbm84jySbPeLE2d47A90js" width="350" height="250"></p>

ptrblck: <p>I guess it might come due to feeding the hidden output back to the model during the epoch, while you are reinitializing it for each new epoch.<br>
How were you planning on handling the hidden state during testing?</p>

marco_zaror: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for your answer.<br>
I’m feeding the hidden output back again to the model because the idea is use 4 words to predict the next one. For that, I pass the first word and a hidden with 0s, and then the hidden output and the next word and repeat that process 4 times. Is there a better way to do it?<br>
The problem is, I do that process (hidden output back again) for each batch within an epoch and the problem arises when I change epochs. The only thing that I do when changing epochs is set the current_loss to 0 (for printing reasons, nothing important). I really don’t know what can be. Do you have any idea?</p>
How to enable bfloat16 PyTorch/XLA on Google Colab

sakaia: <p>In PyTorch/XLA implementation, an environment vatiable XLA_USE_BF16 exists.<br>
Regrettably It cannot see the TF_LOG on Google Colab.<br>
(If some one know, how to set TF_LOG enable, please notice me.)<br>
Is there any way to enable bfloat16 for Pytorch/XLA on Google Colab?</p>
<p>Ref.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/xla/blob/master/torch_xla/csrc/tensor_util.cpp#L21" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/xla/blob/master/torch_xla/csrc/tensor_util.cpp#L21" target="_blank" rel="nofollow noopener">pytorch/xla/blob/master/torch_xla/csrc/tensor_util.cpp#L21</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="11" style="counter-reset: li-counter 10 ;">
<li>#include "tensorflow/compiler/xla/xla_client/sys_util.h"</li>
<li>#include "tensorflow/compiler/xla/xla_client/tf_logging.h"</li>
<li>#include "tensorflow/compiler/xla/xla_client/util.h"</li>
<li>#include "tensorflow/core/lib/bfloat16/bfloat16.h"</li>
<li>#include "torch_xla/csrc/helpers.h"</li>
<li>
</li>
<li>namespace torch_xla {</li>
<li>namespace {</li>
<li>
</li>
<li>bool ShouldUseBF16() {</li>
<li class="selected">  int use_fp16 = xla::sys_util::GetEnvInt("XLA_USE_BF16", 0);</li>
<li>  if (use_fp16 != 0) {</li>
<li>    TF_LOG(INFO) &lt;&lt; "Using BF16 data type for floating point values";</li>
<li>  }</li>
<li>  return use_fp16 != 0;</li>
<li>}</li>
<li>
</li>
<li>xla::PrimitiveType XlaTypeFromTensorType(at::ScalarType scalar_type,</li>
<li>                                         const Device&amp; device) {</li>
<li>  switch (scalar_type) {</li>
<li>    case at::ScalarType::Double:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

sieginglion: <p>try %env XLA_USE_BF16=1</p>
What is d_model in nn.Transformer? And Why does it have to be same for encoder and decoder?

Dylan_Yung: <p><a href="https://pytorch.org/docs/master/nn.html#torch.nn.Transformer" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/nn.html#torch.nn.Transformer</a></p>
<p>I don’t understand what d_model is? What if the vocab for input and target are different sizes? How do you use torch.nn.Transformer if you want to translate a sentence with a vocab size 10 to something with vocab size 8?</p>
Binary cross entroy loss and validation

ykukkim: <p>Hi all,</p>
<p>I have a quick question regards to binary cross-entropy classification. I am very new to this field.</p>
<p>At the moment I am working on time-series data fed into LSTM. However, I am not certain about my method of validation and loss as to what to expect.</p>
<p>Below is my code for loss and accuracy.</p>
<p>My true positive output happens once in every 100~120 frames, so I penalise with weight 100.</p>
<p>Should I expect my loss to be something like 0.00xxx if I do enough training? and does my accuracy make any sense to you all?</p>
<pre><code class="lang-auto">           self.optimizer = optim.Adam(self.model.parameters(), lr=lr)
           self.pos_weight_factor = torch.tensor(100)
           self.criterion = nn.BCELoss(reduction='none')
</code></pre>
<pre><code class="lang-auto">
            predictions = self.model(data.float())

            loss_get = self.criterion(predictions.float(), target.float())
            loss_flat = loss_get.flatten()
            target_flat = target.flatten()
            loss_flat[target_flat == 1] *= self.pos_weight_factor
            loss = loss_flat.mean()
            loss.backward()

            self.optimizer.step()

            pred = predictions.detach()
            pred[pred &gt;= 0.5] = 1
            pred[pred &lt; 0.5] = 0

            correct_indx = pred[target == 1]
            out_size = correct_indx[correct_indx == 1]
            real_indx = target[target == 1]
            accuracy = out_size.shape[0] / real_indx.shape[0]

</code></pre>
<p>Thanks</p>

ptrblck: <p>For an imbalances use case, the accuracy might be misleading due to the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy paradox</a>.<br>
In this case it would be better to observe the confusion matrix and other stats such as Sensitivity, Specificity, etc.</p>
Missing definition from Pytorch docs?

John_Deterious: <p>I’m reading RNN docs, I can only see hidden layer formula, but not the output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a254070db1c968b8ef32546e743fe73a7d03dfd9.png" data-download-href="https://discuss.pytorch.org/uploads/default/a254070db1c968b8ef32546e743fe73a7d03dfd9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a254070db1c968b8ef32546e743fe73a7d03dfd9_2_690x325.png" alt="image" data-base62-sha1="na1nsCGC5aoS0opsir5GtVsINn3" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a254070db1c968b8ef32546e743fe73a7d03dfd9_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a254070db1c968b8ef32546e743fe73a7d03dfd9_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a254070db1c968b8ef32546e743fe73a7d03dfd9.png 2x" data-dominant-color="F6F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1086×512 40.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Comparing to LSTM, all the equations are delineated. Am I missing something?</p>

G.M: <p>The hidden state is the output.</p>

John_Deterious: <p>If that’s the case, then what is the other output shown here?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29f6bff575288b72a8f2fba3df8d21beaa663f00.png" data-download-href="https://discuss.pytorch.org/uploads/default/29f6bff575288b72a8f2fba3df8d21beaa663f00" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29f6bff575288b72a8f2fba3df8d21beaa663f00_2_690x411.png" alt="image" data-base62-sha1="5ZeeANN0gsIP2gCXUkSo2Tdo11e" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29f6bff575288b72a8f2fba3df8d21beaa663f00_2_690x411.png, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29f6bff575288b72a8f2fba3df8d21beaa663f00.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29f6bff575288b72a8f2fba3df8d21beaa663f00.png 2x" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">941×561 58.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

G.M: <p>The output here is all the hidden states generated by the RNN; h_n is the last hidden state.</p>

John_Deterious: <p>Thanks. I didn’t find the docs extremely clear.</p>
Failed to install libtorch

wolfviking0: <p>Hi,</p>
<p>I am building from sources libtorch with support of GPU on OSX.</p>
<p>The build is fine everthing seems to be working but when I call make install I have this error</p>
<pre><code class="lang-auto">CMake Error at third_party/protobuf/cmake/cmake_install.cmake:80 (file):
  file cannot create directory:
  /Volumes/WORKSPACE/exp/pytorch/torch/include/google/protobuf.
  Maybe need administrative privileges.
Call Stack (most recent call first):
  cmake_install.cmake:95 (include)
</code></pre>
<p>Here the command I used to build:</p>
<ol>
<li>export CMAKE_PREFIX_PATH=/Volumes/WORKSPACE/exp/pytorch/install</li>
<li>MACOSX_DEPLOYMENT_TARGET=10.13 CC=clang CXX=clang++ python3 setup.py build --cmake-only</li>
<li>python3 setup.py install</li>
</ol>
<p>Is there something I am doing wrong ?<br>
Thanks</p>

albanD: <p>Did you properly updated all the submodules with <code>git submodule update --init --recursive</code>?<br>
Also make sure you don’t have permission issues in your folder. It complains that it cannot create directories.</p>

wolfviking0: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, sorry for my late response, It seems I initialize all the repository properly.</p>
<p>During the install it seems, torch is creating symlink, the google/protobuf is not a folder in my install but a symlink, is it normal ?</p>
<p>Thanks</p>
<p>Tony</p>

albanD: <p>In the third_party folder? It should be a submodule.</p>

wolfviking0: <p>I just delete everything and rebuild from scratch, it seems I am able to build and install Libtorch now.</p>
<p>It may be a trouble when I was doing the submodule update --init --recursive.</p>
<p>Thanks for you help.</p>
Simply adding an nn.Linear to a module changes SGD behavior

lastrasl: <p>I have observed what I regard as strange/undesirable behavior. This is observed in python 3.7.4; I have not verified on other versions.</p>
<p>In essence, simply adding an nn.Linear module which is not used anywhere in the network changes the behavior of SGD and Adam.</p>
<p>Here’s a simple example showing the behavior:</p>
<pre><code class="lang-auto">import torch
import torch.utils.data
from torch import nn, optim
import numpy as np
import torch.utils.data as data_utils
from torch.nn.parameter import Parameter

torch.manual_seed(0)
torch.cuda.manual_seed(0)
np.random.seed(0)

class test(nn.Module):
    def __init__(self):
        super(test, self).__init__()
        self.a = Parameter(torch.zeros(1,1),requires_grad=True)
#        self.nuisance = torch.nn.Linear(10,10,bias=True)

    def forward(self, x):
        return self.a*x

    def loss_function(self, x):
        return torch.sum(self.forward(x))


def get_loader():
    x_train = torch.rand(1024,1)
    y_train = torch.rand(1024,1)

    train = data_utils.TensorDataset(x_train, y_train)
    train_loader = data_utils.DataLoader(train, batch_size=128, shuffle=True)
    return train_loader

device = "cpu"
model = test().to(device)
train_loader = get_loader()
optimizer = optim.SGD(model.parameters(),lr=0.0001)

def train():

    # set the the model to train mode
    model.train()

    train_loss = 0

    for batch_idx, (data, _) in enumerate(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        loss_total = model.loss_function(data)
        loss_total.backward()
        optimizer.step()
        train_loss += loss_total.item()
        print(loss_total.item() / len(data) )

    train_loss /= len(train_loader.dataset)

    return train_loss

if __name__ == "__main__":
        train()

</code></pre>
<p>Commenting/uncommenting the “nuisance” Linear module results in a different printout. With “nuisance” commented out:</p>
<pre><code class="lang-auto">0.0
-0.002867907052859664
-0.006473415531218052
-0.008921311236917973
-0.01330609805881977
-0.01546912919729948
-0.019429275766015053
-0.023011282086372375
</code></pre>
<p>while with “nuisance” added to the network:</p>
<pre><code class="lang-auto">0.0
-0.003120225388556719
-0.006216045003384352
-0.008216088637709618
-0.01179362740367651
-0.01729675382375717
-0.020048711448907852
-0.022001465782523155
</code></pre>
<p>In my opinion, this is highly undesirable. In my original (much more complex) example, this behavior resulted in markedly different values for the optimized loss function.</p>
<p>What is happening here?</p>

albanD: <p>Hi,</p>
<p>This happens because when you add this module, it initialises its weights and so uses the random number generator. If you try to use the random number generator again afterward, you will get different results as more random numbers were drawn before.</p>

lastrasl: <p>Thanks!! Your diagnosis does make a lot of sense. I was working under the assumption that there was some kind of static graph analysis even before things like initialization ever took place. Learn something new every day!</p>
Give different lr for different parameters in the same layer

barakb: <p>Hi, let’s say I have a convolution layer with weights of size: [64,64,3,3]. now I want to give different lrs , acctualy I want to freeze part of the kernel (for pure research), I first tried:</p>
<pre><code class="lang-auto">model.layer_name.weight[1:,:,:,:].requires_grad  = False
</code></pre>
<p>This returned:</p>
<pre><code class="lang-auto">RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().     
</code></pre>
<p>Furthermore, I tried to think how to use the normal format I’m using to define <strong>different lr for different layers</strong>:</p>
<pre><code class="lang-auto">    parameters= []
    
    ft_module_names = ['layername']
    for k, v in model.named_parameters():
        for ft_module in ft_module_names:
            if ft_module == k:
               parameters.append({'params': v, 'lr':args.lr_new})
               break
        else:
             parameters.append({'params': v})
    
    optimizer = torch.optim.SGD(parameters, args.lr,
                                momentum=args.sgd_momentum,
                                weight_decay=args.weight_decay)
</code></pre>
<p>But I wasn’t able to think how to modify it so that it will be compatible for <strong>different lrs in the same layers</strong></p>
<p>I’ll be happy if you have any ideas!<br>
Thanks!</p>

barakb: <p>ping, any idea someone?</p>

ptrblck: <p>As the error suggests, you won’t be able to change the learning rate of a specific part of the parameter.<br>
You could register a hook to this parameter (<code>param.register_hook</code>) and manipulate the gradients by scaling them with your learning rates.</p>
Pytorch memory issue

kartheek: <p>Hai <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>PyTorch memory issue:</p>
<p>while I m running my PyTorch model after changing " to.device()"  it’s taking 1GB of GPU memory too high at the same time when I m running Keras model, its taking 93mb only.</p>
<p>can you please help me as soon as possible?</p>

ptrblck: <p>Are you seeing a large memory difference also during the actual training/inference?<br>
I’m not sure, how Keras/TF initializes the CUDA context, but the memory usage in PyTorch should be due to the creation of the CUDA context on the device.<br>
You can check it by comparing the memory usage in <code>nvidia-smi</code> to <code>print(torch.cuda.memory_allocated())</code>.</p>
How to distribute batches among gpus selectively

isalirezag: <p>I have two GPUs, one has 11GB and the other one 24GB.<br>
If I use batch size 8, both will have 4 batches.<br>
I want to do batch size 12 in a way that gpu 0 has 4 and gpu 1 has 8.<br>
usually, gpu 0  requires more memory due to backropagation, so i like put most of the batches on other gpus.<br>
any help?</p>
Noisy gradients occurring during training

Anubhab_Ghosh: <p>Hi everyone,</p>
<p>I am encountering a problem while training a pretty simple neural network where each layer is a triangular matrix weight matrix (alternating between lower and upper) and a SoftPlus activation applied at every layer, except the last layer. I was trying to see how the sum of the squares of each weight matrix gradient was varying over the number of iterations of training, and it was found to be pretty noisy with a large variance.</p>
<p>I was suspecting vanishing or exploding gradients so I used some solutions like gradient clipping using grad_clip_norm_ and grad_clip_value_ but of no use.  Also, I used batch normalization to solve the problem but get more noisy gradients.</p>
<p>Any suggestions would be appreciated.</p>

snxeno: <p>Probably we’ll need more information about what you’re trying to achieve. Can you reproduce the problem in code?</p>

Anubhab_Ghosh: <p>I am trying to implement a model for prob.density estimation that is going to map a particular distribution into normal distribution. So my loss is somewhat a negative log-likelihood function which I am trying to minimize.</p>
<p>My custom forward function for every layer looks something like:</p>
<p><code>def forward(self, input): </code><br>
<code>.    self.Y_l = input.mm(self.W)</code><br>
<code>.    self.Z_l = self.phi(self.Y_l)</code></p>
<p>Each self.phi is a <code>nn.Softplus()</code> function<br>
Now I basically have 5 or 7 such layers in the network and I try to minimise some log likelihood loss based on <code>self.W</code> (which is non-convex I think) using <code>torch.optim.adam</code> as the optimizer. I try to observe the mean of <code>self.W.grad.abs( )</code> for every layer as training progresses and it shows a noisy curve which I trying to figure out why.</p>
RuntimeError: CUDA error: out of memory in Pytorch

Ariyan_Hasan: <p>I am getting this error in the last batches while i am training a large speech dataset. I tried reducing batch sizes (16,8,4,2) but every time i got this error at the end of the epoch.<br>
can someone give me the solution?</p>
<h2>99%|█████████▉| 11706/11812 [29:32&lt;01:08,  1.55it/s, avg_loss=tensor(76.1413, device=‘cuda:0’), iter=11705, loss=tensor(117.4730, device=‘cuda:0’)]</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
3     start = time.time()<br>
4<br>
----&gt; 5     run_state = run_epoch(model, optimizer, train_ldr, *run_state)<br>
6<br>
7     msg = “Epoch {} completed in {:.2f} (s).”</p>
<p>~/Hasan/Project/SpeechRNNT/speech-master/train.py in run_epoch(model, optimizer, train_ldr, it, avg_loss)<br>
28         optimizer.zero_grad()<br>
29         loss = model.loss(batch)<br>
—&gt; 30         loss.backward()<br>
31<br>
32         grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 200)</p>
<p>~/miniconda3/envs/ariyan/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
91                 products. Defaults to <code>False</code>.<br>
92         “”"<br>
—&gt; 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
94<br>
95     def register_hook(self, hook):</p>
<p>~/miniconda3/envs/ariyan/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
88     Variable._execution_engine.run_backward(<br>
89         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 90         allow_unreachable=True)  # allow_unreachable flag<br>
91<br>
92</p>
<p>~/Hasan/Project/SpeechRNNT/speech-master/transducer/functions/transducer.py in backward(self, *args)<br>
78         grads = parent.backward(*args)[0]<br>
79         if self.size_average:<br>
—&gt; 80             grads = grads / grads.shape[0]<br>
81         return grads, None, None, None<br>
82</p>
<p>RuntimeError: CUDA error: out of memory</p>

Yimeng_Zhu: <p>Hi,</p>
<p>I’m wondering would you mind to share your implementation of transducer loss? Is it a python native code or did you used c/c++ extensions.</p>
<p>Best,</p>
How to set ouput size independetly from hidden size in any RNN?

John_Deterious: <p>In RNN, GRU and LSTM, I seem to only have control over the hidden state size, which will also dictate output size. Isn’t this a limitation? It seems to me that with this setup, the complexity of the NN is dictated by output size. My question is: is there any way to make things more flexible than those args:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/2635b7a0f28244f8e1d0202192a9b3beb4c1dd43.png" alt="image" data-base62-sha1="5s1eqlyCRUUwZKDNAllaiNShLlV" width="505" height="94"></p>

sharvil: <p>You can use a linear layer on the output of the RNN to get whatever output dimension you want. That’s exactly what the implementation of an LSTM or GRU cell would have done if it would have provided an <code>output_size</code> argument.</p>
torch.utils.data.distributed.DistributedSampler very slow over Lustre

DPC: <p>All,</p>
<p>I am attempting to troublehsoot poor performance of torch.utils.data.distributed.DistributedSampler when deployed onto a machine using Lustre for file serving.</p>
<p>The specific scenario is a resnet50 network, using imagenet for inputs.  My cluster uses Lustre for fileserving, and the node I am using has a pair of Volta V100 GPUs.  I expect a throughput of 600-1300 images/second per GPU depending on settings and configuration details.</p>
<p>My first attempt was reading directly from the mounted Lustre FS.  This had the obvious problem of being limited by the ability of the FS server to handle large numbers of small file requests.  Speed results were on the order of 100 img/s/GPU.</p>
<p>In order to isolate the performance bottleneck, I stripped out portions of the training loop until I was left with only the sampler being used each loop.  Performance did not improve.</p>
<p>I tried mounting a squashfs image containing the dataset, in order to limit fileserver metadata requests, but performance did not improve substantially.  I also tried an ext4 image which also did not improve results significantly.</p>
<p>An analogous benchmark in tensorflow achieves about 1100 imgs/sec/GPU on the same node, so I believe there is something specific to either Torch or the way that I am using Torch.</p>
<p>Can anyone suggest next steps for troubleshooting this?</p>
Data Transforms - RandomTranspose?

DJ_1992: <p>Hi,<br>
I would like to perform some data augmentation on the images with</p>
<p>transforms.RandomHorizontalFlip(),<br>
transforms.RandomVerticalFlip(),</p>
<p>I would also like to do another transform like a RandomTranspose()?<br>
Please help<br>
Thank You</p>

yash1994: <p>Hi,</p>
<p>There is no built-in function/transformation available in Pytorch for Random Transpose. But you can build a custom data loader and call transpose on image tensor randomly. Something like <a href="https://www.kaggle.com/mratsim/starting-kit-for-pytorch-deep-learning" rel="nofollow noopener">this</a>. In <code>__getitem__</code> function you have to set a flag true/false randomly and on the basis of that flag you can apply your Transpose transformation.</p>

DJ_1992: <p>Hey, Thanks alot Yash <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> It is working …</p>
Codebook learning

musgan: <p>Hello,  I need your help.<br>
I was read journal about codebook learning.  The journal say encoded feature by codebook learning.  Please help,  what is mean about encoded codebook learning ?</p>

kulikovv: <p>Codebook learning is a powerful technique of signal representation and encoding used in many ares of computer vision (image enhancement, denoising, compression etc). Google “K-SVD” for example of sparse coding.<br>
There are few approach that allow to use codebooks in DL frameworks for example: VQ-VAE <a href="https://arxiv.org/pdf/1711.00937.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1711.00937.pdf</a> , and Memory Networks <a href="https://arxiv.org/pdf/1410.3916.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1410.3916.pdf</a></p>
<p>Hope it will help.</p>
Reporting a taller stack for warnings in PyTorch

mortazavi: <p>How can we trigger the system to print a fuller stack whenever a “warning” log is printed in PyTorch . . . Under the default conditions, it seems to only provide the location issuing the “warning” not its stack . . .</p>

albanD: <p>Hi,</p>
<p>The warnings we through are python warnings. So you do that the same way you would do for any python warning (see <a href="https://stackoverflow.com/questions/22373927/get-traceback-of-warnings">https://stackoverflow.com/questions/22373927/get-traceback-of-warnings</a> for example).</p>
RuntimeError: builtin cannot be used as a value

slavavs: <p>What is the reason for this error?</p>
<pre><code class="lang-auto">class Network(ResNet):
    def __init__(self, num_actions):
        self.num_actions = num_actions
        super(Network, self).__init__(BasicBlock, [2, 2, 2, 2], num_classes=self.num_actions)
        self.conv1 = torch.nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    def forward(self, x):
        x = x.cuda()
        batch, _ = super(Network, self).forward(x).shape
        x = super(Network, self).forward(x).view(batch, self.num_actions)
        return x
</code></pre>
<pre><code class="lang-auto">net = Network(action_dim).half().to(device)
</code></pre>
<p><strong>fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(obj))</strong></p>
<p>RuntimeError:<br>
builtin cannot be used as a value:<br>
at /home/slavavs/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/_utils.py:14:56<br>
def zeros_like(tensor, dtype):<br>
# type: (Tensor, int) -&gt; Tensor<br>
return torch.zeros_like(tensor, dtype=dtype, layout=tensor.layout,<br>
~~~~~~~~~~~~~ &lt;— HERE<br>
device=tensor.device, pin_memory=tensor.is_pinned())<br>
‘zeros_like’ is being compiled since it was called from ‘<strong>torch</strong>.torchvision.models.detection._utils.BalancedPositiveNegativeSampler.<strong>call</strong>’<br>
at /home/slavavs/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/_utils.py:72:12</p>
<pre><code>        # randomly select positive and negative examples
        perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
        perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]

        pos_idx_per_image = positive[perm1]
        neg_idx_per_image = negative[perm2]

        # create binary mask from indices
        pos_idx_per_image_mask = zeros_like(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  &lt;--- HERE
            matched_idxs_per_image, dtype=torch.uint8
        )
        neg_idx_per_image_mask = zeros_like(
            matched_idxs_per_image, dtype=torch.uint8
        )

        pos_idx_per_image_mask[pos_idx_per_image] = torch.tensor(1, dtype=torch.uint8)
        neg_idx_per_image_mask[neg_idx_per_image] = torch.tensor(1, dtype=torch.uint8)</code></pre>
Optimizer.step() vs manual parameter update in RNN tutorial

anna_go: <p>hey,<br>
I’m learning PyTorch and just stumbled upon this in the official tutorial on RNN<br>
(<a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html#" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html#</a>):<br>
the parameter update is encoded in the train function manually, while in the older version of this tutorial on Sean’s github<br>
(<a href="https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb" rel="nofollow noopener">https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb</a>)<br>
the optimizer is defined in the usual way and the update is done by calling optimizer.step().</p>
<p>I haven’t found any comments on that in the tutorial. So why manual instead of optimizer?</p>

Ranahanocka: <p>Check out this post:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/spro/practical-pytorch/issues/21" target="_blank" rel="nofollow noopener">github.com/spro/practical-pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/WeiFoo" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/7039841?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/spro/practical-pytorch/issues/21" target="_blank" rel="nofollow noopener">Issue: weights update for "char-rnn-classification"</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/WeiFoo" target="_blank" rel="nofollow noopener">WeiFoo</a>
	on <a href="https://github.com/spro/practical-pytorch/issues/21" target="_blank" rel="nofollow noopener">2017-05-19</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/spro" target="_blank" rel="nofollow noopener">spro</a>
	on <a href="https://github.com/spro/practical-pytorch/issues/21" target="_blank" rel="nofollow noopener">2017-05-20</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">In the "char-ran-classification" tutorial, the weights are updated by the following code.
 for p in rnn.parameters():
 p.data.add_(-learning_rate, p.grad.data)
I was trying to...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Essentially, using the optimizer will redefine the RNN when it is called inside the training loop and then update the wrong network weights</p>

Aniketh_Suresh: <p>I spent the last three hours trying to figure out what was wrong. This was perfect thanks!</p>
1d conv primitive ops

harpone: <p>Regular <code>conv1d</code> is basically $a_i^c = \sum_{k, c’} weight[c, c’, k] x_{i+k}^c’$, where <code>x</code> is the input with <code>c'</code> the input channel dim, and <code>weight</code> has dimensions <code>[out_channel_dim, in_channel_dim, kernel_size]</code>.</p>
<p>The questions</p>
<ol>
<li>Is there a way to do the convolution operation alone, i.e. without the summation over the channel dim <code>c'</code>?</li>
<li>I guess I could use the <code>einsum</code> for this, but is it going to be fast enough (like cuDNN fast)?</li>
</ol>
<p><code>functional.conv1d</code> does the actual convolution, summation over the channel dim <em>and</em> adds the bias… I’d need some more primitive ops for my crazy experiments! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>You could use <code>unfold</code> as given in <a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">this example</a> to calculate your manual convolution.<br>
However, it should be slower, but could be a good enough for first exeriments.</p>

harpone: <p>Hmm yeah I guess that would work! <code>unfold</code> seems to indeed return a new tensor instead of a view, so I guess it can use quite a bit of memory (esp. for 2D and higher convolutions).</p>
<p>I’ll do some benchmarks vs. regular convolutions. Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to use a pretrain model to update part of the model in torchvison

isalirezag: <p>Im using FCN50+Resnet50 from torchvision, but it does not have pretrain model available now.<br>
Is there a way to use the pretraining weights from resnet50 to just update part of the FCN50 which belongs to Resnet?<br>
Here is what I mean</p>
<pre><code class="lang-auto">

model_1 = torchvision.models.segmentation.fcn_resnet50(pretrained=False, num_classes=21).to('cuda:0')
 resnet50 = models.resnet50(pretrained=True)
</code></pre>
<p>How can I just update part of model1 which is in common which model2?</p>

yash1994: <p>Hi,<br>
You just have to create new <code>state_dict</code> from resnet50 model which doesn’t contain last fully connected classification layer and load it to the <code>fcn_resnet50.backbone</code>.</p>
<pre><code class="lang-python">import torchvision
from collections import OrderedDict
m1 = torchvision.models.segmentation.fcn_resnet50(pretrained=False, num_classes=21)
m2 = torchvision.models.resnet50(pretrained=True)

new_state_dict = OrderedDict()
for k, v in m2.state_dict().items():
    if k != "fc.weight" and k != "fc.bias":
        new_state_dict[k] = v

m1.backbone.load_state_dict(new_state_dict)
</code></pre>
Load the model and tensor in C++ (libtorch) for inference

kl_divergence: <p>I’m new to C++ frontend. Here is my current code</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; 

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  
  torch::jit::script::Module module;
  try {
     module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model\n";
    return -1;
  }
  
    std::vector&lt;torch::Tensor&gt; sample_input;
    torch::load(sample_input, argv[2]);
    
    std::cout &lt;&lt; "Loaded Successfully\n";
    
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(sample_input);

    at::Tensor output = module.forward(inputs).toTensor();
    std::cout &lt;&lt; output &lt;&lt; '\n';
    
}
</code></pre>
<p>I want to run the executable as</p>
<pre><code class="lang-auto">./example-app ../model.pt ../sample_input.pth
</code></pre>
<p>The code doesn’t compile (<code>make</code>) .<br>
In the documentation, I saw usage of <code>torch::load</code> but during <code>make</code> I noticed the error message <code> ‘load’ is not a member of ‘torch’</code>.<br>
Also the inputs would not append the tensor, not sure what <code>IValue</code> is.</p>
<p>Could you please provide the fix?</p>

driazati: <p>You probably need to <code>#include &lt;torch/all.h&gt;</code> for it to pick up <code>torch::load</code>.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/20356#issuecomment-567663701">This comment</a> has more info about the save/load APIs.</p>
TorchScript function slower than non-JIT?

sharvil: <p>I have a simple function (LSTM layer) that I’m converting to TorchScript and executing. From some initial experiments, it looks like the JIT version runs slower than the non-JIT version on both CPU and GPU. The relevant code is listed here: <a href="https://github.com/lmnt-com/haste/blob/master/frameworks/pytorch/lstm.py#L31-L64" rel="nofollow noopener">https://github.com/lmnt-com/haste/blob/master/frameworks/pytorch/lstm.py#L31-L64</a>. Is this expected behavior?</p>
PyTorch and Jupyter notebook?

kaiseryet: <p>Hey guys, so I am using pytorch 1.4.0 on my jupyter through ipykernel. The kernel dies for no (unknown?) reason… and this often happens when I ran my specific notebook (sometimes it’s good though). is this a bug? Can I collect some info to help debug this? Thanks.</p>

ptrblck: <p>In my experience Jupyter notebooks are not the best in reporting the error message or showing the stack trace. I would recommend to save your notebook as a Python script (there is an export option for it) and run the script in your terminal to get the error message.</p>

kaiseryet: <p>I don’t think it’s a bug in my code though… I think it’s something between pytorch and jupyter notebook?</p>
1D Sparse Network - Using Conv1d

qdl: <p>Hello,</p>
<p>I am trying to implement and train a sparse network that looks like the following:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/626fcc792d51f331619dbed1e1c47b2b2f108a1e.png" alt="conv1d_flat-3" data-base62-sha1="e2OjPb4KAloJIvVrO9dwSOE4Rno" width="217" height="231"></p>
<p>My understanding was that it is very similar to a 1D convolutional network with a single channel.  So this is how I implemented it:</p>
<pre><code class="lang-auto">nn.Sequential(
    nn.Conv1d(in_channels=1, out_channels=1, kernel_size=3, bias=False),
    nn.Sigmoid(),
    nn.Linear(6, 1, bias=False),
    nn.Sigmoid()
)
</code></pre>
<p>Data looks like the following:</p>
<pre><code class="lang-auto">X = [
       [1, 0, 1, 1, 0, 1, 0, 0],
       [1, 1, 1, 1, 0, 0, 1, 1],
       ...
    ]

y = [
       [1],
       [0],
       ...
    ]
</code></pre>
<p>However, when I train it with simple inputs (like shown above) and examine the weights, they don’t look like I understood/expected them to be. So I reckoned perhaps that my architecture was incorrect. Can someone please let me know if this looks right?</p>
<p>PS: What I actually have is a hidden layer with the same number of dimensions as the input layer, so I have set <code>padding=1</code> in my actual code. The <code>padding=0</code> version was easier to show in a diagram.</p>

ptrblck: <p>The code looks alright.<br>
Note that you would have to add <code>nn.Flatten</code> before the linear layer in case your conv layer returns more then a single output channel.<br>
In the current setup it should be working. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>What do you mean by: “examine the weights, they don’t look like I understood/expected them to be”?</p>

qdl: <p>Thanks! I’ll try that.</p>
<p>What I meant is:</p>
<p>Suppose I want my network to learn a pattern such as 1101 (and say now the kernel size is 4), and my inputs were such as:</p>
<pre><code class="lang-auto">x_1 = 1, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0; y_1 = 1
...
x_k = 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0; y_k =  0
</code></pre>
<p>My understanding was that, after I achieved a predetermined acceptable level of accuracy, the learned weights would look something like <code>[0.987, 0.99, -0.0008, 0.959]</code>. Is that understanding incorrect?</p>

ptrblck: <p>Not necessarily, since you are using non linear activation functions (sigmoid), the parameters might have lower or higher values.</p>

qdl: <p>The reason I use sigmoid is to emulate a step function. I was hoping to emulate an <strong>OR</strong> (hid-&gt;out) of <strong>AND</strong>s (in-&gt;hid) with the synthetic dataset I made up. I’ll tinker around a bit more and get back, thanks!</p>
There is still memory occupied after the tensor is released

Yiman_von: <p>Hi, guys!<br>
I created a new tensor to observe the occupation of video memory. When I load it on the GPU, I can see a significant increase in video memory (by nvidia-smi). However, when I load it to the CPU and release it (I used <code>torch.cuda.empty.cache()</code>, I know that if this is not used, the video memory occupation may not be refreshed), but the video memory occupation does not return to 0, and It is 377M left (sometimes 367M). I don’t quite understand why is there still memory consumption? Why is it 377M? Is this the memory that PyTorch must occupy when calling CUDA?<br>
Is there any way to release this memory？</p>
<p>Here is my environment:</p>
<ul>
<li>
<p>PyTorch version:1.2.0</p>
</li>
<li>
<p>CUDA version: release 10.1, V10.1.243</p>
</li>
</ul>
<p>PS:I didn’t initialize any network parameters <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

ptrblck: <p>The memory should be used by the CUDA context, which is initialized with the first CUDA operation, e.g. creating a tensor.<br>
You cannot release it during the runtime of your script.</p>

Yiman_von: <p>Thank you for your reply！Have a good day！ <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Pytorch Errors with Titan RTX [Windows 10]

Ortiane: <p>I get errors after running for an arbitrary number of iterations for my attention model. (Either in the encoder or during the loss backwards). I’ve literally tried everything I could for this, but it simply just won’t work for this machine. This doesn’t seem to be an issue on my two other machines. I’ve tried with “pytorch=1.4.0=py3.7_cuda101_cudnn7_0” and that would just crash after 100 iterations; then I tried using “pytorch=1.2.0=py3.7_cuda100_cudnn7_1” and was working for a bit then after changing my code slightly caused it to crash with the following error:</p>
<p>“Exception has occurred: RuntimeError<br>
CUDA error: unspecified launch failure<br>
THCudaCheck FAIL file=…\aten\src\THC\THCCachingHostAllocator.cpp line=296 error=4 : unspecified launch failure”</p>
<p>Adding the following lines stops it from crashing:<br>
import os<br>
os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1”</p>
<p>But that severely impacts the training time. I’m at a loss on how to fix this error and haven’t found any good solutions as of yet from general searching. Can someone who has encountered this issue or seen a post describing a fix for this please help?</p>
<ul>
<li>I updated my GPU drivers to the current release; was I not supposed to do this?</li>
</ul>

peterjc123: <p>Would you please open a new issue at <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a> and post the code and your specs there?</p>