akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't get rid of <eos> & <sos> tokens in Torchtext Field

andcarnivorous: <p>I am following one of torchtext tutorials using the wikitext2 dataset and I cannot figure out why, even though I specify None as value for init and eos tokens I still get them in my vocabulary, also padding ones.</p>
<p>torchtext version: 0.4.0</p>
<p>This is part of my vocabulary:</p>
<pre><code class="lang-auto">            {'&lt;unk&gt;': 0,
             '&lt;pad&gt;': 1,
             'the': 2,
             ',': 3,
             '.': 4,
             'of': 5,
             'and': 6,
             'in': 7,
             'to': 8,
             '&lt;eos&gt;': 9,
             'a': 10,
             'was': 11,
</code></pre>
<p>This is instead my field set up:</p>
<pre><code class="lang-auto">TEXT = torchtext.data.Field(tokenize = get_tokenizer("basic_english"), 
                            init_token=None, 
                            eos_token=None, 
                            stop_words = ["=", "==", "@", "@@", "&lt;", "&gt;", "@-@"], 
                            lower=True)

train_txt, val_txt, test_txt = torchtext.datasets.WikiText2.splits(TEXT)
TEXT.build_vocab(train_txt)
</code></pre>
<p>I also get a  somewhere which I cannot really explain since padding is not enabled by default but only when specifying max_length.</p>
Differentiable indexing: grid states where to map pixel

Soham_Tamba: <p>I’m looking to perform the following operation in a differentiable manner:<br>
Input:<br>
In: dim N x C x H_in x W_in<br>
grid: dim N x H_in x H_out x 2</p>
<p>Output:<br>
Out: dim N x C x H_out x W_out</p>
<p>(N is batch_size, C is number of channels)</p>
<p>Code:</p>
<pre><code class="lang-auto">Out = torch.zeros((N, C, H_out, W_out))
for i in range(H_in):
    for j in range(W_in):
        for n in range(N):
            Out[n, :, grid[n, i, j, 0], grid[n, i, j, 1]] += In[n, :, i, j]
return Out
</code></pre>
<p>Basically grid[n, i, j, :] states where it maps input feature <strong>to</strong> output.<br>
I know in <a href="https://discuss.pytorch.org/t/differentiable-indexing/17647/3">F.grid_sample</a>, grid[n, i, j, :] where to map out <strong>from</strong>.</p>
Modifying parameters directly

theultimatecrouton: <p>Hi. I have been doing an implementation of Evolutionary Strategies (<a href="https://arxiv.org/abs/1703.03864" rel="nofollow noopener">https://arxiv.org/abs/1703.03864</a>) in PyTorch. This is a gradient-free optimisation approach to RL problems, in which neural net params are perturbed and a corresponding update is calculated based on the reward at these perturbations.</p>
<p>For the hacky example I have got working I have resorted to doing perturbations like:</p>
<pre><code class="lang-auto">layer.weight = torch.nn.Parameter(current_weight + eps_weight)
layer.bias = torch.nn.Parameter(current_bias + eps_bias)
</code></pre>
<p>where <code>layer</code> is, for example <code>nn.Linear</code>.</p>
<p>My problem is that for a complex net I can’t see a way to perturb all parameters in a nice way. Ideally I would iterate through <code>model.parameters()</code> and perturb each one in turn, but it seems <code>model.parameters()</code> does not return the actual modules in the model, just a copy of the parameters. Thus I cannot change the weights this way as far as I can tell.</p>
<p>Is there a clean way to update neural net parameters directly?</p>
<p>Thanks,<br>
Tom</p>

Saila_Maham: <p>Hi, were you ever able to resolve this issue ?</p>

theultimatecrouton: <p>Pretty much. The way I ended up doing it was to get the parameters into a <code>state_dict</code>, update them in the dict and the use <code>load_state_dict</code> to perform the update</p>

adarsh-kr: <p>So I am assuming your <code>eps_weight</code> is getting updated by backpropagation, right ? Can  you please share a snippet of your code to enable this.</p>
<p>RIght now, I was thinking of adding to <code>x.data</code> the <code>eps_weight</code>, however, that won’t let me backpropagate I guess to the <code>eps_weight</code></p>
How to optimize multiple variables with respect to multiple losses?

chausies: <p>I want different losses to have their gradients computed with respect to different variables, and those variables to then all step together.</p>
<p>Here’s a simple example demonstrating what I want:</p>
<pre><code class="lang-auto">import torch as T
x = T.randn(3, requires_grad = True)
y = T.randn(4, requires_grad = True)
z = T.randn(5, requires_grad = True)

x_opt = T.optim.Adadelta([x])
y_opt = T.optim.Adadelta([y])
z_opt = T.optim.Adadelta([z])

for i in range(n_iter):
  x_opt.zero_grad()
  y_opt.zero_grad()
  z_opt.zero_grad()

  shared_computation = foobar(x, y, z)

  x_loss = f(x, y, z, shared_computation)
  y_loss = g(x, y, z, shared_computation)
  z_loss = h(x, y, z, shared_computation)

  x_loss.backward_with_respect_to(x)
  y_loss.backward_with_respect_to(y)
  z_loss.backward_with_respect_to(z)

  x_opt.step()
  y_opt.step()
  z_opt.step()
</code></pre>
<p>My question is how do we do that <code>backward_with_respect_to</code> part in PyTorch? I only want <code>x</code>'s gradient w.r.t. <code>x_loss</code>, etc… And then I want all the optimizers to step together (based on the current values of <code>x</code>, <code>y</code>, and <code>z</code>).</p>

chausies: <p>I’ve written a function to do just this. The two key components are (1) using <code>retain_graph=True</code> for all but the final call to <code>.backward()</code> and (2) saving grads after each call to <code>.backward()</code>, and restoring them at the end before <code>.step()</code>ing.</p>
<pre><code class="lang-auto">def multi_step(losses, optms):
  # optimizers each take a step, with `optms[i]`'s variables being 
  # optimized w.r.t. `losses[i]`.
  grads = [None]*len(losses)
  for i, (loss, optm) in enumerate(zip(losses, optms)):
    retain_graph = i != (len(losses)-1)
    optm.zero_grad()
    loss.backward(retain_graph=retain_graph)
    grads[i] = [ 
          [ 
            p.grad+0 for p in group['params'] 
          ] for group in optm.param_groups
        ]
  for optm, grad in zip(optms, grads):
    for p_group, g_group in zip(optm.param_groups, grad):
      for p, g in zip(p_group['params'], g_group):
        p.grad = g
    optm.step()
</code></pre>
<p>In the example code stated in the question, <code>multi_step</code> would be used as follows:</p>
<pre><code class="lang-auto">for i in range(n_iter):
  shared_computation = foobar(x, y, z)

  x_loss = f(x, y, z, shared_computation)
  y_loss = g(x, y, z, shared_computation)
  z_loss = h(x, y, z, shared_computation)

  multi_step([x_loss, y_loss, z_loss], [x_opt, y_opt, z_opt])
</code></pre>
<p>If anyone could confirm that this is correct (and accounting for all cases), that would be great.</p>

ptrblck: <p>I haven’t verified your code, but would <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad"><code>torch.autograd.grad</code></a> work for your use case?<br>
This would compute the gradients w.r.t. to the passed input.</p>
Correct Input dimension of Images for Training?

sgaur: <p>I need to know what is the correct input dimension of Input images some discussion forum tell me its channel<em>Height</em>Width other telling me Height<em>Width</em>channel</p>
<p>Which one do I need to chose while doing training of Model and for Visualization.</p>

ptrblck: <p>PyTorch expects inputs as <code>[batch_size, channels, height, width]</code> for the “usual” layers in CNNs.</p>
<p>If you want to visualize numpy arrays via e.g. <code>matplotlib</code>, then you should remove the batch dimension and permute the array to a channels-last format.<br>
However, this is <code>matplotlib</code> specific and other visualization libraries might use other formats.</p>
Mini-batching if for-loop involved in forward operation

tyterry: <p>Hi all,</p>
<p>I am experimenting on a attention autoencoder which does the following:</p>
<ol>
<li>Each input is tuples of array (e.g. Tuples of length 10, among which each array has a size of 90)</li>
<li>The encoder encodes each array from size 90 to size 30</li>
<li>The encoded are concatenate together to form a array of size 10 x 30 = 300)</li>
<li>The attention layer compress the encoded_combined to a size of 30 and see which array (among the 10 arrays) to pay attention to</li>
<li>The decoder decode the array of size 30 to a array of 90</li>
</ol>
<p>Hence, in forward, a for-loop has to be incorporated the encoded of each of the array in the tuple::</p>
<pre><code class="lang-auto">    def forward(self, X):
        x = torch.from_numpy(X[0]).view(1,-1,self.in_features)
        encoded_combined = self.encoder(x)
        encoded_combined = torch.squeeze(encoded_combined)
        for arr in X[1:]:
            x = torch.from_numpy(arr).view(1,-1,self.in_features)
            tail = self.encoder(x)
            tail = torch.squeeze(tail)
            encoded_combined = torch.cat((encoded_combined,tail))
        attn_w = F.softmax(self.attn(encoded_combined), dim = 0)
        encoded_combined = encoded_combined.view(-1, 1)
        attn_applied = torch.mm(attn_w_expanded, encoded_combined)
        decoded = self.decoder(attn_applied.view(1,-1,30))
        return decoded, attn_w
</code></pre>
<p>The above forward operation only can take 1 sample at a time and the training process is very slow.</p>
<p>.Is there any way to achieve mini-batching / get rid of the for-loop for such dataset?</p>
<p>Currently I have a collate_fcn as follows which take 1 sample at a time so I do not need to disable automatic batching of the dataloader.</p>
<pre><code class="lang-auto">def collate_fcn_one_sample(data):
    data_batch = [batch[0] for batch in data]
    target_batch = [batch[1] for batch in data]
    for group in data_batch:
        for data in group:
            data = torch.from_numpy(data)
    for target in target_batch:
        target = torch.from_numpy(target)
    return group, target
</code></pre>
<p>Much obliged for your help!!!</p>
<p>Many thanks.</p>
Pytorch merging list of tensors together

Ameen_Ali: <p>Let’s say I have a list of tensors ([A , B , C ] where each tensor of is of shape [batch_size X 1024].<br>
I want to merge all the tensors into a single tensor in the following way :<br>
The first row in A is the first row in the new tensor, and the first row of B is the seocnd row in the new tensor, and the first row of C is the third row of the new tensor and so on and so forth.<br>
So far I did it with for loops and this is not effictive at all. Would love to hear about more efficent ways.<br>
Thanks</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">a = torch.arange(0, 6).view(3, 2)
b = torch.arange(6, 12).view(3, 2)
c = torch.arange(12, 18).view(3, 2)

ret = torch.cat((a.unsqueeze(1), b.unsqueeze(1), c.unsqueeze(1)), dim=1).view(9, 2)
</code></pre>
BrokenPipeError [Errno 32] Broken pipe

chillum1718: <p>when i define No. of workers 2 it gives me an error [broken pipe error]<br>
but when i define No. of workers 0 it works fine, but it takes loang time to run</p>
<p>Attachin screenshot also</p>
<p>iam using intel 9920x<br>
RTX 2080 TI 11GB<br>
windows 10 pro<br>
i have also uploaded the screenshot!<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d3f758f89f65ae8a8c35a57d627573d6d4cd541.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d3f758f89f65ae8a8c35a57d627573d6d4cd541" title="Capture2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d3f758f89f65ae8a8c35a57d627573d6d4cd541.png" alt="Capture2" data-base62-sha1="8JP5mk1fznUjXqAUYXHvebwluZb" width="690" height="336" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture2</span><span class="informations">991×483 14.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Maybe you are missing the if-clause protection as described in the <a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">Windows FAQ</a>.</p>
BrokenPipeError: [Errno 32] Broken pipe

chillum1718: <p>BrokenPipeError: [Errno 32] Broken pipe<br>
threads issue, unable to set threads<br>
data loader problem<br>
RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 11.00 GiB total capacity; 8.25 GiB already allocated; 70.74 MiB free; 8.31 GiB reserved in total by PyTorch)</p>

ptrblck: <p>You are running out of memory on your GPU.<br>
Try to decrease the batch size or alternatively try to trade compute for memory via <code>torch.utils.checkpoint</code>.</p>

chillum1718: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f763b49664bbb2fb77a71e0e6cc72015b0c8535.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f763b49664bbb2fb77a71e0e6cc72015b0c8535" title="Capture1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f763b49664bbb2fb77a71e0e6cc72015b0c8535.png" alt="Capture1" data-base62-sha1="ibzWP1vOoW31MpZ6WhQfUn2vtNr" width="690" height="296" data-dominant-color="2E2D28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture1</span><span class="informations">787×338 16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
when i define No. of workers 2 it gives me an error [broken pipe error]<br>
iam using intel 9920x<br>
RTX 2080 TI 11GB<br>
windows 10 pro<br>
i have also uploaded the screenshot</p>

ptrblck: <p>Have a look at the <a href="https://pytorch.org/docs/stable/notes/windows.html#usage-multiprocessing">Windows FAQ</a> for multiprocessing. Maybe you are missing the if-clause protection.</p>
Masked KL Divergence loss with label smoothing

sougata_saha: <p>Hi,<br>
I am trying to implement a masked KL divergence loss with label smoothing. Below is the code. Can anyone attest if this is the right way to go about it? Thanks!<br>
Context: Calculating loss in Sequence2Sequence architecture:<br>
Decoder output format: Batch x Sequence Length x Vocab size<br>
Input format: Batch x Sequence Length (Note: this is not one hot encoded)</p>
<pre><code class="lang-auto">class KLDivergenceLossWithMask(nn.Module):
  def __init__(self, label_smoothing, target_vocab, ignore_value=102):
    super().__init__()
    self.ignore_value = ignore_value
    self.label_smoothing = label_smoothing
    self.confidence = 1.0 - label_smoothing
    self.target_vocab = target_vocab
    smoothing_value = self.label_smoothing / (self.target_vocab - 2)
    self.one_hot = torch.full((self.target_vocab,), smoothing_value)
    self.one_hot = self.one_hot.unsqueeze(0)

  def forward(self, inp, tgt):
    model_prob = self.one_hot.repeat(tgt.size(0),tgt.size(1), 1)
    model_prob = model_prob.scatter_(2,tgt.unsqueeze(2), value=self.confidence)
    mask = (tgt != self.ignore_value)*1
    model_prob_masked = model_prob*mask.unsqueeze(2)
    inp = (inp*mask.unsqueeze(2))
    return nn.KLDivLoss(reduction = 'batchmean')(F.log_softmax(inp, dim=2), 
                                                 model_prob_masked)
</code></pre>
<pre><code class="lang-auto">inp = torch.randn(4,7,12000).float() #batch,seq_len,vocab
tgt = torch.LongTensor([[23,34,22,67,10122,11343,102],
                        [9999,999,99,9,12,344,6678],
                        [1133,9434,2234,102,102,102,102],
                        [0,1,2,3,102,102,102]])#batch,seq_len
print(inp.shape, tgt.shape)
</code></pre>
<pre><code class="lang-auto">label_smoothing = 0.2

target_vocab = 12000

ignore_value = 102

kl_loss_with_mask = KLDivergenceLossWithMask(label_smoothing, target_vocab, ignore_value)

loss = kl_loss_with_mask(inp, tgt)

print(loss)
</code></pre>
I still don't understand pytorch WeightedRandomSampler

NickE: <p>I have a dataset with 22 classes total.  21 classes have 90 images and 1 class has 11 images for a total of 1901 images.</p>
<p>Either I don’t understand the pytorch WeightedRandomSampler instructions, the instructions are wrong, or both.  I can certainly say that they are not nearly as clear as directions for other classes.</p>
<p>I used replacement=True throughout.  I created an ImageFolder instance and used it to create a DataLoader instance (along with the WeightedRandomSampler instance).  I set shuffle=False for the data loader as well.</p>
<p>From the instructions, I originally thought that the weights were just the weights associated with each class (22 numbers).  I also set the sample_size to the batch_size (8).  When I iterated through the data loader and calculated the total number of images selected per class, I got garbage:<br>
[1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]<br>
Where the numbers represent the number of images selected per class.  In other words, only one image was selected out of 1901 images.</p>
<p>I then assigned the weights to the class weights for all targets (1901 numbers).  The batch size is still 8.  I got something that looked more reasonable:<br>
[85 76 63 75 87 88 80 76 77 76 86 69 79 71 66 57 85 69 75 70 69 60]<br>
Although it looks reasonable, I think that there is something wrong because these numbers only sum to 1639, which means that about 14% of the images are being overlooked.  I realize that there may be a little slop in the total number of selected images because the sampler is sampling images at random, but a 14% discrepancy seems large.</p>
<p>If I do the same test with the batch_size=1, I get:<br>
[ 95  80  92  77  81  85  83  78 105  89  84  85  91  86  79  86 101  97 100  75  67  85]<br>
These numbers add to 1901, which is what I expected.</p>
<p>I would like to get 1901 images (or a number close to it) selected with a batch_size other than 1.  Is this possible?  Am I doing something wrong?</p>
DDP and iterable datasets

sharvil: <p>How are folks using iterable datasets with DDP? <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="nofollow noopener">The example for splitting</a> an <code>IterableDataset</code> across workers (or DDP processes) seems a little silly – if I had random access to my dataset (<code>iter_start</code>), I wouldn’t be using an iterable dataset in the first place.</p>
<p>Has anyone come across / built a better solution?</p>

mrshenli: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for data loader question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vincentqb: <p>This is a recurring requestm, e.g. <a href="https://github.com/pytorch/pytorch/pull/32420" rel="nofollow noopener">here</a> or <a href="https://github.com/pytorch/pytorch/pull/26547" rel="nofollow noopener">here</a>. Please feel free to suggest a mechanism <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Device to host transfer

almeetb: <p>When running a module on a device e.g. GPU or TPU does the output stay on device or is it always sent back to the host CPU by default?</p>

smth: <p>it is always on the device, and not transferred back to the CPU host memory unless you call <code>.cpu()</code> (or <code>.to(device='cpu')</code> on the output Tensor</p>
First level of dispatcher

almeetb: <p>How is the first level of tensor dispatch (sending tensors to devices/backends) decided and is this related to DispatchStub?</p>
<p>Related issue: <a href="https://discuss.pytorch.org/t/when-is-dispatchstub-called/79153" class="inline-onebox">When is DispatchStub called</a></p>
<p>Thanks in advance!</p>
Dealing with variable sized input signals for classification using FCN

BarkenBark: <p><strong>Background:</strong><br>
I am working on classification of 1D signal data using Fully Convolutional Networks acting directly on the signal data.</p>
<p>My dataset consists of signals of various lengths. In order to deal with this during training, I choose an appropriate signal length, cropped all signals longer than that, and padded all signals shorter than that with zeros.</p>
<p><strong>Problem:</strong><br>
After training models and inspecting the results, I’ve noticed that flat zero-valued regions in signals tend to bias the classification towards a few certain classes (The cause might be partly due to the length distribution of signals between classes). This is problematic when classifying shorter realizations of other classes which need to be padded, since even though the correct class activation is high for the original part of the signal, the flat region drastically decreases the activation for these classes.</p>
<p>In the end, since the FCN model can handle inputs of variable sizes, I think it would be ideal for me to train it on data without the padding causing the problems. But I don’t see how I could achieve that efficiently.</p>
<p><strong>Question:</strong><br>
How can efficiently deal with this problem?</p>
<p><strong>Ideas that I haven’t tried yet:</strong></p>
<ul>
<li>Pad signals by duplicating the original signal until target length is achieved rather than zero-pad</li>
<li>Zero-pad all signals excessively to hopefully make the resulting models agnostic to flat regions</li>
<li>Train the model using batch_size=1</li>
<li>Train the model on batches with variable sized inputs by accumulating gradients from each input separately</li>
</ul>
<p>I understand that this is a huge open-ended question, so any ideas or shared experiences are very appreciated. Thank you.</p>

googlebot: <p>Well, you can multiply intermediate layer outputs with binary mask, that would zero values computed from partially available inputs. You would start this mask shaped like input, with one channel, and do “min pooling” whenever data shape is changed by convolution or pooling operation. It is actually easier to do max pooling with inverse mask (1 = bad location), and invert it for multiplication.</p>
<p>correction: when doing convolution and pooling, different mask pooling modes should be used (i.e. data max-pooling output is good if ANY input is good, convolution - needs ALL inputs)</p>
Build Options on pytorch/CMakeLists.txt

Peter_Peter: <p>Hello guys,</p>
<p>I’m looking for the build option explanation. Where can I find more on those options for build PyTorch from a source such as USE_OPENCL, USE_OPENCV, USE_SYSTEM_ONNX, USE_NNAPI, USE_NNPACK, etc…</p>
<p>Thank you.</p>

richard: <p>The only thing I know about is at the top of the file here: <a href="https://github.com/pytorch/pytorch/blob/master/setup.py">https://github.com/pytorch/pytorch/blob/master/setup.py</a> . It’s not really detailed though.</p>
Loss function for a classifier with output and target shape: (batch_size, N, M)

Ali_Amiri: <p>Hi,<br>
I have a model with output shape: (batch_size, N, M) and target shape: (batch_size, N, M)<br>
this is a classifier.<br>
so I should use NLL or CrossEntropy as the loss function.<br>
but torch gives me <strong>1D target tensor expected, multi-target not supported</strong> error<br>
how can I fix this?</p>

KFrank: <p>Hi Ali!</p>
<aside class="quote no-group" data-username="Ali_Amiri" data-post="1" data-topic="79279" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_amiri/48/22823_2.png" class="avatar"> Ali_Amiri:</div>
<blockquote>
<p>I have a model with output shape: (batch_size, N, M) and target shape: (batch_size, N, M)<br>
this is a classifier.<br>
so I should use NLL or CrossEntropy as the loss function.</p>
</blockquote>
</aside>
<p>Well, you should tell us what “output” means, what “target” means,<br>
and what “N” and “M” are.  Otherwise, we will have to guess.</p>
<p>But let me guess a little bit, and make the following comment:</p>
<p>If you are performing a multi-class classification problem (but not<br>
a multi-label problem), then the shapes of the <code>input</code> (output of<br>
your model) and <code>target</code> passed to your <code>CrossEntropyLoss</code> loss<br>
function must match up in the following way:</p>
<p>Your <code>input</code> can have <code>shape</code> <code>[batch_size, nClass]</code>, and then your<br>
<code>target</code> must have shape <code>[batch_size]</code> and consist of integer (<code>long</code>)<br>
class labels that run from <code>0</code> to <code>nClass - 1</code>.</p>
<p>Or, in the “K-dimensional case,” your <code>input</code> could have <code>shape</code>, say,<br>
<code>[batch_size, nClass, height, width, depth]</code>, and your <code>target</code><br>
would have to have <code>shape</code> <code>[batch_size, height, width, depth]</code><br>
(and consist of values from <code>0</code> to <code>nClass - 1</code>).</p>
<p>Good luck.</p>
<p>K. Frank</p>

Ali_Amiri: <p>thanks Frank, that was helpful</p>
PyTorch implementation of normalisation layers for RNNs, LSTMs, GRUs

AliAb3d: <p>I am having a hard time finding a solid PyTorch implementation that adopts normalization layers for recurrent networks. Does this require changing implementations at LSTMCell level, for example in the case of LSTM layers? Is normalization necessarily required to be applied at each time-step separately or on entire output sequence at once?</p>
<p>I would also be very grateful if someone could share their experience on which normalization technique (e.g. batchnorm, layernorm, instancenorm) they found to work best for effectively normalising recurrent layer outputs in Pytorch.</p>

googlebot: <p>That’s a big topic, there is “vertical” and “horizontal” (h[t-1] -&gt; h[t]) normalization, and many approaches have been tried… But basically, simple vertical batchnorm is problematic, LayerNorm(num_features) works, but feature distortion it does may sometimes have adverse effects. You can check <a href="https://arxiv.org/abs/1607.06450" rel="nofollow noopener">Layer Normalization</a> paper, as it addresses rnn normalization. <a href="https://arxiv.org/abs/1603.09025" rel="nofollow noopener">Recurrent batch normalization</a> paper describes horizontal LSTM normalization.</p>
Circular convolution

papyan: <p>Dear pytorchers,</p>
<p>Is it possible to compute a Circular convolution in pytorch?</p>
<p>My initial thought was to use Circular padding followed by regular convolution with no padding but it seems that Circular padding does not exist.</p>

jpeg729: <p>You probably have to use a regular square convolution with a constraint on the weights to remove the corners of the convolution.</p>

papyan: <p>Could you elaborate on what do you mean by:</p>
<blockquote>
<p>“constrain on the weights to remove the corners”</p>
</blockquote>

jpeg729: <p>For example, a 2d convolution with kernel size 4 would have a 4x4 matrix of weights for each channel. Forcing the corners of this 4x4 matrix to be zero would give your convolution a nearly circular receptive field.</p>
<p>But maybe I have completely misunderstood what you mean by “circular convolution”.</p>

LambdaWill: <p>Maybe you can draw some ideas from here: <a href="https://github.com/DmitryUlyanov/texture_nets/blob/master/src/SpatialCircularPadding.lua" rel="nofollow noopener">https://github.com/DmitryUlyanov/texture_nets/blob/master/src/SpatialCircularPadding.lua</a> . It is easy to read.</p>

Liang: <p>You can use <code>F.conv2d(F.pad(input, pad=(5,5,5,5), mode='circular'), kernel, padding=0)</code> for circular convolution. Btw, you sometimes cannot find the mode <code>circular</code> for F.pad if you directly search in  the help doc, but your can go to <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=pad#torch.nn.functional.pad" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.functional.html?highlight=pad#torch.nn.functional.pad</a></p>
How to pass additional arguments to odeint(.) from torchdiffeq library

sayan_ghosal: <p>Hi,<br>
I have question about the pytorch library torchdiffeq (<a href="https://arxiv.org/pdf/1806.07366.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1806.07366.pdf</a>). This is a pytorch implementation of ordinary differential equations where the differential function is defined by a neural network. For example,<br>
dy/dt = f(t,y,\theta), here f(.) is a neural network. My question is if my \theta comes from another neural network and I want to pass it to odeint(.) from the library how should I do that?</p>
<p>Any help is really appreciated.</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>This is not a library provided by the core developers so I don’t know.<br>
Maybe the people that wrote that lib are reading this forum and will answer. Otherwise, maybe you should open an issue on the repo directly: <a href="https://github.com/rtqichen/torchdiffeq">https://github.com/rtqichen/torchdiffeq</a></p>
Loss stuck at the same value with pretrained model

rashmi_chaurasiya: <p>hello, i am new in pytorch. Here i am trying to update my existing model with some additional training, but failed to update the loss value. I couldn’t be able to find what is wrong here.</p>
<p>model= AttnVGG_before()<br>
model.load_state_dict(torch.load(’/content/AttStateDict.pth’))</p>
<p>def train(model, train_X):<br>
best_loss = 10000000<br>
for epoch in range(1000):<br>
train_loss = 0<br>
val_loss = 0<br>
adjust_learning_rate(optimizer, epoch)</p>
<pre><code>    model.train()  
    for data in trainloader:
        X, Y = data[0].to(device, non_blocking=True), data[1].to(device, non_blocking=True)
        optimizer.zero_grad()
        outputs = model(X)
        loss = criterion(outputs,Y)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()

    model.eval()  
    with torch.no_grad():
      for data in testloader:
          X, Y = data[0].to(device, non_blocking=True), data[1].to(device, non_blocking=True)
          outputs = model(X)
          loss = criterion(outputs,Y)
          val_loss += loss.item()
    
    print('[Epoch %d] train_loss: %.8f val_loss : %.8f' %
                  (epoch + 1, train_loss/len(trainloader), val_loss/len(testloader)))
    if val_loss &lt; best_loss:
      torch.save(model.state_dict(), 'AttStateDict.pth')
      best_loss = val_loss
      
print('Done Training')
</code></pre>
<p>model.to(device)<br>
train(model, trainloader)</p>
<p>torch.save(model.state_dict(), ‘AttentionState.pth’)</p>
<p>Output</p>
<p>[Epoch 1] train_loss: 0.01428753 val_loss : 0.00846462<br>
[Epoch 2] train_loss: 0.01428753 val_loss : 0.00846462<br>
[Epoch 3] train_loss: 0.01428753 val_loss : 0.00846462</p>

vmirly1: <p>The first thing I would check in this situation is if the optimizer is defined correctly with the parameters of the model:</p>
<pre><code class="lang-auto">optimize = optim.Adam(model.parameters(), ...)
</code></pre>
<p>It could be that there are two instances of model and optimizer is defined on one instance, while loss is computed on a different instance.</p>

rashmi_chaurasiya: <p>Thank you, it worked</p>
Tensorboard with multiprocessing

overfitted: <p>Hi all,</p>
<p>Do you know if I can run <code>torch.utils.tensorboard</code> along with multiprocessing? If yes can anybody provide some hints/code? I tried tensorboardX with multiprocessing and the problem that I had is that each spawned process was initialising its own SummaryWriter.  I know that  the solution is only the master process to initialise SummaryWriter and this should be used globally by all processes, but since Im not experienced with multiprocessing I dont know how to make this work. In tensorboardX, a recent commit implements GlobalSummaryWriter (<a href="https://github.com/lanpa/tensorboardX/blob/master/tensorboardX/global_writer.py" rel="nofollow noopener">link</a>) but I couldn’t have this running either, was getting errors, although following exactly the documentation from github on how to use it, its still in experimental mode so Im guessing  there are still bugs.</p>
<p>Thank you.</p>

zhuyi490: <p>Hi <a class="mention" href="/u/overfitted">@overfitted</a>, did you find a solution to this problem? Thank you.</p>

sharvil: <p>If you’re using <code>torch.multiprocessing</code> (which you probably should be doing), you’ll get the process index as the first parameter of your entry point function. You can consider index 0 to be your master process and do all of your summary writing in that process.</p>
<pre><code class="lang-python">def my_entry_point(index):
  if index == 0:
    writer = SummaryWriter(summary_dir)
    writer.add_scalar('test', 1.0, 1)

torch.multiprocessing.spawn(my_entry_point, args=(), nprocs=2)
</code></pre>

overfitted: <p>Hi,</p>
<p>Thanks for your answer! I will try and let you know. Cheers!</p>
When does pytorch use cuDNN for RNN inference?

Joshua_Gevirtz: <p>Hi everyone,</p>
<p>I tried running some LSTMs manually using cuDNN by calling directly out to the library.  Everything seems to work great, except the performance isn’t what I would expect when comparing against libtorch: basically my direct-to-cudnn code is slower by a factor of two compared to the identical calculation as performed through libtorch.</p>
<p>I was under the impression that if libtorch claims to have access to cuDNN features (which in my case it does), it will use cuDNN’s RNN implementation, so I expected performance to be identical.  Is it possible I’m under the wrong impression and that, in fact, libtorch is using custom kernels which outperform cuDNN, at least on my hardware?  Is there something else I am overlooking?</p>
<p>For context: to do the measurements I wrote two C++ programs, one links to libtorch and the other to libcudnn.  After setting up the LSTM weights/descriptors/etc I create a large input tensor and run it through the LSTM 10 times, keeping track of start and end time for the whole loop; the program prints the difference between these times and then exist.  I am running on a GTX 1080.</p>
<p>Any insight will be super helpful!  Thanks in advance.</p>

tom: <p>You can check whether CuDNN is used using e.g. nvprof to list the kernels that are being run. This might also give you insights what else is going on.<br>
Many people asking about performance oddities forget to synchronize the GPU before the measurement begins and ends, this may or may not be the case in your testing.</p>
<p>Best regards</p>
<p>Thomas</p>

Joshua_Gevirtz: <p>Thanks for your input, <a class="mention" href="/u/tom">@tom</a>.  Actually, after I posted I did a bit of inspecting (using oprofile because I had the logs sitting around so I didn’t have to run anything new) and indeed it looks like cudNN <em>is</em> being used my libtorch, which leaves me befuddled.  It isn’t a synchronization issue because, just in case, I do a GPU -&gt; CPU transfer, which is blocking.</p>
<p>I wonder if it might have to do with the way I build my cuDNN application vs. the way libtorch is built?  On that theory, I’ve tried both dynamic and static linking in my cuDNN test app without seeing any difference in terms of performance.</p>
How can I install a older pytorch cuda version(to be specific 1.4) on google colab?)

Lupos: <p>The newer PyTorch version 1.5 is now preinstalled on google colab. But my code is not working on the newer version, that’s why I want to install the PyTorch version 1.4.</p>
<p>But when using(i dont know if that breaks cuda on colab hopefully not):<br>
<code>!pip install pytorch==1.4</code><br>
I get the error message:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement pytorch==1.4 (from versions: 0.1.2, 1.0.2)
ERROR: No matching distribution found for pytorch==1.4
</code></pre>
<p>Can someone help me?</p>

albanD: <p>You can find the instructions on the website <a href="https://pytorch.org/get-started/previous-versions/#linux-and-windows-1">here</a>.<br>
In particular, you’re missing the <code>-f</code> part of the command.</p>