akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Segmentation fault with vocab.py

username: Aiman_Mutasem-bellh
cooked: <p>Helo dears,</p>
<p>I’m trying to load pre-training mode as bellow code:</p>
<pre><code class="lang-auto">url = 'https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.ar.vec'
SRC.build_vocab(train_data, vectors=Vectors('wiki.ar.vec', url=url), unk_init = torch.Tensor.normal_, min_freq = 2) 
TRG.build_vocab(train_data, min_freq = 2)
</code></pre>
<p>I get an unusual error like :</p>
<pre><code class="lang-auto">Fatal Python error: Segmentation fault

Current thread 0x00007f054de01740 (most recent call first):
File "/home/aiman/anaconda3/lib/python3.7/site-packages/torchtext/vocab.py", line 387 in cache
File "/home/aiman/anaconda3/lib/python3.7/site-packages/torchtext/vocab.py", line 323 in __init__
File "test.py", line 101 in &lt;module&gt;
</code></pre>
<p>I have update <code>torchtext</code> and <code>conda</code> but still not fix.</p>
<p>The environment as bellow:</p>
<pre><code class="lang-auto">pytorch-ignite==0.4.2
pytorch-nlp==0.5.0
torch==1.4.0
torchaudio==0.4.0a0+719bcc7
torchtext==0.6.0
torchvision==0.5.0
</code></pre>
<p>Any suggestions to fix this issue?</p>

username: Aiman_Mutasem-bellh
cooked: <p>Any support to fix this issue? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Could you update all libs to the latest stable release and retry the code?<br>
If you are still seeing the seg fault, please create an issue in the <code>torchtext</code> GitHub repository.</p>

username: Aiman_Mutasem-bellh
cooked: <p>The problem  with torchtext version 1.4.0. I have updated it to version 1.5.0. and the issue is fixed.</p>
How do forward hooks work with respect to gradient?

username: Glerk
cooked: <p>Summary: I want to do transfer learning where I use a pre-existing network as a backbone. I want to access intermediate layers instead of only the final layer. All the pre-trained networks I have seen don’t offer a simple model to load the weights so I believe I am going to need to access the layers using hooks.</p>
<p>I created a forward hook to access the output of an intermediate layer. My next question is, how does back propagation work with respect to this?</p>
<p>In particular, lets say I use the output of that intermediate layer as input to a final layer. How do I apply back propagation with the hook working?</p>
<ol>
<li>
<p>Can I lock the weights of the backbone and only update the weights of the layers I add?</p>
</li>
<li>
<p>Can I update both the weights of my added layers and the backbone? How does the backward pass work with respect to a forward hook? Does the hook get added to the computation graph automatically?</p>
</li>
</ol>
<p>Thanks</p>

username: ptrblck
cooked: <ol>
<li>Yes, you can freeze parameters by setting their <code>.requires_grad</code> attribute to <code>False</code>.</li>
<li>The forward hook would in the simple use case just return the intermediate activation, so Autograd will not record any operation and you can just pass the tensor to any layers.</li>
</ol>
Reading data from dataloader with offset index

username: wasabi
cooked: <p>Can we read data from dataloader with offset index like a list? For example, if I have data_list, I can read data like</p>
<pre><code class="lang-auto">for i in range(offset, something):
    x, y = data_list[i]
</code></pre>
<p>I’m wondering if this can be done with dataloader, thanks.</p>

username: ptrblck
cooked: <p>Yes, you could implement a custom <code>sampler</code>, which is responsible to create the indices and pass them to the <code>Dataset.__getitem__</code>.<br>
Alternatively, you could also add an offset in the <code>__getitem__</code> method directly, if this would fit your use case.</p>
Cuda memory is too small

username: timtaotao
cooked: <p>Hi there, I got this runtime error when I was running my code on CUDA,</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 33.22 MiB already allocated; 2.65 MiB free; 40.00 MiB reserved in total by PyTorch)</p>
<p>I notice that the memory reserved by PyTorch is extremely small, I’m using GTX 1050Ti with torch version 1.4.0, driver version 457.09 and CUDA version 11.1<br>
Is this an issue with my CUDA settings? Does anyone know how I can fix this?<br>
Cheers</p>

username: ptrblck
cooked: <p>Based on the error message it seems that your GPU might be used by other processes, so you should check its free memory via <code>nvidia-smi</code>.</p>

username: timtaotao
cooked: <p>Hi, thanks for your reply, since I am using Windows I can’t monitor gpu memory via nvidia-smi, I googled but can’t find a replacement.<br>
I do notice that the dedicated gpu memory usage was empty at the beginning, but increased to 3.7GB when I run my code, though according to the error message,  only 268 MiB memory was reserved by pytorch.</p>

username: timtaotao
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d39dec5e1f15358f13696f217ef9362d330ec5b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/d39dec5e1f15358f13696f217ef9362d330ec5b5" title="QQ图片20201116105945"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d39dec5e1f15358f13696f217ef9362d330ec5b5.png" alt="QQ图片20201116105945" data-base62-sha1="uc35UI8jfisJ0WFVZaYpPTdfMpL" width="491" height="500" data-dominant-color="29323C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ图片20201116105945</span><span class="informations">593×603 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>On Windows you should be able to find an <code>nvidia-smi.exe</code>, which would give you the memory usage.<br>
The PyTorch memory stats won’t show other processes.</p>

username: timtaotao
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for your help, I executed nvidia-smi on windows but I only got N/A for each process’ gpu usage, however, I do find the cause to my problem.<br>
Since I load data from tfrecord file, I import tensorflow to do data preprocessing, and tf takes up all the gpu memory by default. I flush CUDA after the preprocessing and everything works fine now!</p>
Change MSELoss to cross entropy

username: rui_zhou
cooked: <p>should I change MSELoss to cross entropy?</p>
<p>criterion = torch.nn.MSELoss(reduction=‘mean’)<br>
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4<br>
metrics = {‘f1_score’: f1_score, ‘auroc’: roc_auc_score}</p>
<p>I have the rgb image and mask of 32 bit, they are divided into two classes(background and the thing), why I got this one? how can I change the MSELoss to cross entropy?</p>
<p>File “/content/DeepLabv3FineTuning/trainer.py”, line 58, in train_model<br>
metric(y_true.astype(‘uint8’), y_pred))<br>
File “/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_ranking.py”, line 390, in roc_auc_score<br>
sample_weight=sample_weight)<br>
File “/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_base.py”, line 77, in _average_binary_score<br>
return binary_metric(y_true, y_score, sample_weight=sample_weight)<br>
File “/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_ranking.py”, line 221, in _binary_roc_auc_score<br>
raise ValueError("Only one class present in y_true. ROC AUC score "<br>
ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.</p>

username: ptrblck
cooked: <p>If you are working on a multi-class segmentation use case, <code>nn.CrossEntropyLoss</code> would be the preferred loss function.<br>
Your error is raised by a <code>sklearn</code> metric method and is unrelated to the criterion in PyTorch.</p>
(C++) Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. 'aten::empty_strided' is only available for these backends: [CPU, BackendSelect, Autograd, Profiler, Tracer]

username: adaber
cooked: <p>Hey everyone,</p>
<p>I am trying to copy a Sequential object consisting of the Conv2d, BatchNorm2d, LeakyReLU modules from the CPU to the GPU by doing object-&gt;to(…) using Pytorch  C++ 1.6 and I get the message that’s in the title of this thread.</p>
<p>The exact same code works fine using Pytorch 1.4.</p>
<p>Could anyone help me figure out what the problem is ?</p>
<p>Thanks</p>
<p>P.S. I am using Visual Studio 2017 and the operating system is Win 10.</p>
<p>P.S.1 I tried the following code:</p>
<pre><code>	auto test = torch::nn::Conv2d(torch::nn::Conv2dOptions(1, 1, 1));
	test-&gt;to(torch::Device(torch::kCUDA));
</code></pre>
<p>and I got a different message. It is “PyTorch is not linked with support for cuda devices”. I did link both torch_cuda.lib and c10_cuda.lib.</p>
<p>P.S.2. I’ve found a solution to this problem. I find it a bit strange that I have to force the linker to link against a library by directly adding a symbol to the symbol table. Microsoft do describe this linker option as a useful feature for including a library object that otherwise would not be linked to the program. I guess it’s just that I’ve never had to do this up until now.</p>

username: seunghan-go
cooked: <p>I solved this issue by adding ‘?warp_size@cuda@at@<span class="mention">@YAHXZ</span>’ to linker option of visual studio(2017).</p>
<ol>
<li>Add  /INCLUDE:?warp_size@cuda@at@<span class="mention">@YAHXZ</span>
</li>
<li>Add ‘torch_cuda.lib’</li>
</ol>
<p>ref. <a href="https://github.com/pytorch/pytorch/issues/33435#issuecomment-685241862" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/33435#issuecomment-685241862</a></p>
<p>Thanks.</p>

username: adaber
cooked: <p>Seunghan-go,</p>
<p>I solved the issue soon after I had made the post but thanks for the reply.</p>

username: OronG13
cooked: <p>hello,</p>
<p>Thanks for this issue I successfully fixed the same error on my Windows system.</p>
<p>But, I also have a Linux:</p>
<p>Ubuntu 18.04 - NVIDIA Jetson Xavier AGX with JetPack 4.4.</p>
<p>PyTorch 1.6 for ARM.</p>
<p>When I tried to used the same solution described above and added the</p>
<p>/INCLUDE:?warp_size@cuda@at@<span class="mention">@YAHXZ</span> to the linker option a build error was raised:</p>
<p>g++: error: /INCLUDE:?warp_size@cuda@at@<span class="mention">@YAHXZ</span>: No such file or directory</p>
<p>g++: error: /INCLUDE : error : No such file or directory</p>
<p>When I tried to change the /INCLUDE to -INCLUDE the build process was successfully completed but the original problem came bask and report on the aten::empty_strided problem as described above.</p>
<p>Can you please help me to understand how to use this flag on Linux system?</p>
<p>Thanks</p>
DataLoader not returning Tensor

username: nafizh1
cooked: <p>Here is my code:</p>
<pre><code class="lang-auto">import torch
class Dataset(torch.utils.data.Dataset):
  def __init__(self, labels):
        'Initialization'
        self.labels = labels
        
  def __len__(self):
        'Denotes the total number of samples'
        return len(self.labels)

  def __getitem__(self, index):
        X = torch.load('data/' + str(index) + '.pt')
        y = self.labels[index]
        return X, y

training_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size = 1)
</code></pre>
<p>But when I do a for loop with training_dataloader on this dataset class, it returns only the label y, not the features X, even though if I print X, I can see it being printed. What am I doing wrong here?</p>

username: Rohit_Modee
cooked: <p>Can you also post your training loop.</p>
Get output of intermediate layer of pre-trained model

username: Glerk
cooked: <p>I am interested in accessing intermediate layer outputs of the pre-trained models, ex, mobilenetV2. It’s not clear to me how to do this. I read something about hooks but from reading it seemed that wasn’t the correct way to do it.</p>
<p>Advice?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Forward hooks are the right approach as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
How to completely modify the classifier of a pretrained network to use KNN

username: Kaynor
cooked: <p>Hi, I’m currently using a pretrained MobileNetV2 model for my audio classification.<br>
As a beginner of using PyTorch and from reading other similar topics, I understand how to modify the first and the last layer:</p>
<pre><code class="lang-auto">from torchvision.models import mobilenet_v2
import torch
import torch.nn as nn
mobilenet_model = mobilenet_v2(pretrained=True)
mobilenet_model.features[0][0] = nn.Conv2d(1, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)  # Modify the first layer
mobilenet_model.classifier[1] = nn.Linear(1280, NUM_CLASSES, bias=True) # Modify the last layer
</code></pre>
<p>It works great, but now I want to experiment this model with different approaches by applying different classifiers (like KNN), but I’m not sure how to change the last layer to use, let’s say, KNN instead of linear transformation.</p>
<p>Does anyone know how to do this? Any resource is also fine!</p>

username: ptrblck
cooked: <p>If you want to freeze the model and use a KNN implementation from e.g. <code>sklearn</code>, you could replace the last linear layer with <code>nn.Identity</code> to get the features from the penultimate layer and feed them to <a href="https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html"><code>sklearn.neighbors.KNeightborsClassifier</code></a>.<br>
Alternatively, you could also implement KNN manually in PyTorch (you should be able to find some implementations in this forum or probably on GitHub).</p>
Efficient subset selection

username: jloverich
cooked: <p>I’m looking for the most efficient way to perform this operation</p>
<pre><code class="lang-auto">weight_range = []
for i in range(id_min.shape[0]):
    weight_range.append(torch.arange(id_min[i], id_max[i], device=device))
weight_range = torch.stack(weight_range).flatten()
</code></pre>
<p>Where id_min and id_max are 1D torch tensors representing the minimum and maximum range (the size id_min:id_max is always the same).  This operation is performed in the forward step and is very time consuming.  It’s basically grabbing a small subset of weights for each each batch and it will be a different subset depending on the input.</p>

username: googlebot
cooked: <p>(id_min.unsqueeze(-1) + torch.arange(size,device=id_min.device)).view(-1)</p>

username: jloverich
cooked: <p>Thanks! That’s what I’m looking for.</p>
Understanding of implementation of roi_pool and roi_aligh in torchvision

username: luciaL
cooked: <pre><code class="lang-auto">[docs]def roi_pool(
    input: Tensor,
    boxes: Tensor,
    output_size: BroadcastingList2[int],
    spatial_scale: float = 1.0,
) -&gt; Tensor:
    """
    Performs Region of Interest (RoI) Pool operator described in Fast R-CNN

    Arguments:
        input (Tensor[N, C, H, W]): input tensor
        boxes (Tensor[K, 5] or List[Tensor[L, 4]]): the box coordinates in (x1, y1, x2, y2)
            format where the regions will be taken from. If a single Tensor is passed,
            then the first column should contain the batch index. If a list of Tensors
            is passed, then each Tensor will correspond to the boxes for an element i
            in a batch
        output_size (int or Tuple[int, int]): the size of the output after the cropping
            is performed, as (height, width)
        spatial_scale (float): a scaling factor that maps the input coordinates to
            the box coordinates. Default: 1.0

    Returns:
        output (Tensor[K, C, output_size[0], output_size[1]])
    """
    _assert_has_ops()
    check_roi_boxes_shape(boxes)
    rois = boxes
    output_size = _pair(output_size)
    if not isinstance(rois, torch.Tensor):
        rois = convert_boxes_to_roi_format(rois)
    output, _ = torch.ops.torchvision.roi_pool(input, rois, spatial_scale,
                                               output_size[0], output_size[1])
    return output
</code></pre>
<p>here is the implementation of ROI_Pool in torchvision. I can’t understand how it works because I  didn’t find the to torch.ops.torchvision.roi_pool. please help me and explain how does this function work.</p>
Batching Question

username: jackeown
cooked: <p>I have a complicated graph-neural-network-like pytorch model which is very hard for me to vectorize.</p>
<p>I am looking forward to using the new vmap functionality and might even start trying it out from the master branch on github, but in the meantime, I’m trying to understand more about vectorization in general.</p>
<p>I made the following short script to test how important batching is for speed in PyTorch.<br>
When I use the batched version, it’s about 77 times faster.<br>
(I used input and output dimensions of 256 and number of vectors = 10000)</p>
<pre><code class="lang-python">import torch
import argparse
from time import time

parser = argparse.ArgumentParser(
 description="A short program to test how much batching helps in PyTorch"
)
parser.add_argument("i", type=int, help="input vector dimension")
parser.add_argument("o", type=int, help="output vector dimension")
parser.add_argument("n", type=int, 
  help="number of vectors to multiply matrix by"
)
parser.add_argument("--batched", action="store_true")
args = parser.parse_args()

layer = torch.nn.Linear(args.i, args.o)
xs = torch.randn(args.n, args.i, requires_grad=False)

print("start")
t1 = time()
if args.batched:
    ys = layer(xs)
else:
    for x in xs:
        y = layer(x)
t2 = time()
print("end")
print(f"time elapsed: {t2-t1}")
</code></pre>
<p>Basically, my question comes down to the following:<br>
What are all of the factors that contribute to making batched code take 1/77th the time? (even on CPU)</p>

username: ptrblck
cooked: <p>For loops add the Python overhead in each iteration and call into the dispatcher for a small workload.<br>
A lot of operations are using vectorization inside, which won’t be fully used.<br>
Note that this is not PyTorch-specific and can also be seen in e.g. numpy:</p>
<pre><code class="lang-python">def fun1(x):
    out = []
    for x_ in x:
        out.append(np.sqrt(x_))
    out = np.stack(out)
    return out

def fun2(x):
    out = np.sqrt(x)
    return out

x = np.random.rand(1000, 100)
%timeit out1 = fun1(x)
%timeit out2 = fun2(x)
</code></pre>

username: jackeown
cooked: <blockquote>
<p>For loops add the Python overhead in each iteration and call into the dispatcher for a small workload.</p>
</blockquote>
<p>The for loops add very little overhead as far as I could tell from profiling using line_profiler.<br>
Is the pytorch dispatch really enough to explain it being 77 times slower?</p>
<blockquote>
<p>A lot of operations are using vectorization inside, which won’t be fully used.</p>
</blockquote>
<p>What does vectorization really mean at a low level?<br>
Do you mean that pytorch uses avx2 style SIMD parallelism on CPU?</p>
Regarding batch loading in pytorch

username: peepeepoopoo
cooked: <p>Hi! just curious as to how Dataloader create batches in sequential mode. if I have a list of files in order of<br>
[“file1.txt”, “file2.txt”, “file3.txt”, “file4.txt”,“file5.txt”,“file6.txt”], would the output order be something like this (with a batch size of 2): [“file1.txt”, “file2.txt”], [ “file3.txt”, “file4.txt” ] , [“file5.txt”,“file6.txt”],</p>

username: ptrblck
cooked: <p>If you are storing the filenames in e.g. a <code>list</code> in the initially posted order, then you are correct in the batch outputs.<br>
If you are not using <code>shuffle=True</code>, the <code>Dataset.__getitem__</code> will get indices in a sequential order (i.e. <code>0, 1, 2, 3, ...</code>) and depending how you are storing and loading the data, these indices will just be used in the <code>__getitem__</code> method.</p>

username: peepeepoopoo
cooked: <p>Thanks! I can confirm this is the case observing the output.</p>
Can not move the tensor onto GPU

username: Yuchen_Mu
cooked: <p>Hi everyone, I am using PyTorch 1.7 and cuda 10.2, I found a strange thing, please see the following code and corresponding outputs:</p>
<pre><code class="lang-auto">test =torch.randn(3,3)
print(test.device)
test.to('cuda:0')
print(test.device)

test_2 = torch.randn(3,3,device='cuda:0')
print(test_2.device)
</code></pre>
<p>which has outputs:</p>
<pre><code class="lang-auto">cpu
cpu
cuda:0
</code></pre>
<p>It seems strange to me, since I can set its location onto gpu when initialize the tensor but cannot move it onto gpu if it was initialized on cpu, can anyone help me with this? Thanks in advance!</p>

username: Kirondem
cooked: <p>What you want:</p>
<p>device = torch.device(‘cuda:0’)<br>
test = test.to(device)</p>

username: Yuchen_Mu
cooked: <p>Thanks Kirondem! I just realized I forget to reassign the variable. (actually a little silly of me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: dhyey
cooked: <p>you have to pass additional argument device = cuda in torch.randn()</p>
How to serve same dataset to models on each gpu?

username: Trump
cooked: <p>In the same machine, there are four different models on each gpu.<br>
They are all use same dataset, but i don’t know how to serve it efficiently.</p>
<p>Naively I could make four copies of the dataset, and create four python process to train them.<br>
But Is it the best way? Are all four same dataset have to be duplicated on the memory?..</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="Trump" data-post="1" data-topic="101673">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/trump/48/12143_2.png" class="avatar"> Trump:</div>
<blockquote>
<p>Are all four same dataset have to be duplicated on the memory?</p>
</blockquote>
</aside>
<p>If those dataset won’t be mutated, then they don’t have to be duplicated.</p>
<blockquote>
<p>Naively I could make four copies of the dataset, and create four python process to train them.<br>
But Is it the best way?</p>
</blockquote>
<p>This approach has pros and cons. It will certainly use more memory on duplicated dataset and per-process CUDA context (500MB CUDA memory). But the benefit is there won’t be GIL contentions. Another option could be using multi-thread with multiple CUDA streams. In this way, data set and CUDA context can be shared, and multi-thread will also allow concurrent processing on CUDA devices. But the CPU computations from multiple threads will still compete to grab the GIL.</p>
Unable to cast Python instance to C++ type (Trying to script Yolov3)

username: Carmel_Barak
cooked: <p>Hi,<br>
I’m trying to use <code>torch.jit.script</code> to script a modified version of ultralytics yolov3<br>
I’ve modified it a bit to fix all kinds of errors during the scripting process and now I’m getting</p>
<pre><code class="lang-auto">  File "C:\ProgramData\Anaconda3\envs\yolo_pytorch\lib\site-packages\torch\jit\_recursive.py", line 325, in init_fn
    cpp_module.setattr(name, orig_value)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
</code></pre>
<p>It happens when trying to set the attribute for the module_def, which is of type <code>List[Dict[str, str]]</code><br>
I’ve modified the module_defs for all the keys &amp; values to be strings</p>
<p>I’m using torch 1.5.0 on Windows 10 64 bit</p>
<p>Any suggestions on how to tackle this? if more info is needed, let me know what</p>

username: ptrblck
cooked: <p>I would generally recommend to update to the latest stable version, as a lot was fixes/added into the JIT.</p>

username: Carmel_Barak
cooked: <p>Thanks,<br>
I will try with 1.7.0 and will update</p>

username: Carmel_Barak
cooked: <p>The problem still happened in 1.7.0, but then I iterated the list and found that in some cases, one of the values was initialized as <code>int(0)</code>. I changed that to <code>"0"</code> and it worked</p>
<p>thanks!</p>
How to stack more LSTMs?

username: suits_cloud
cooked: <p>I am trying to create an LSTM encoder decoder. The following code has  LSTM layers. How can I add more to it?</p>
<pre><code class="lang-auto">
class Encoder(nn.Module):

    def __init__(self, seq_len, n_features, embedding_dim=128):
        super(Encoder, self).__init__()
        self.seq_len, self.n_features = seq_len, n_features
        self.embedding_dim, self.hidden_dim = embedding_dim, 2 * embedding_dim

        self.rnn1 = nn.LSTM(
            input_size=n_features,
            hidden_size=self.hidden_dim,
            num_layers=1,
            batch_first=True)

        self.rnn2 = nn.LSTM(
            input_size=self.hidden_dim,
            hidden_size=embedding_dim,
            num_layers=1,
            batch_first=True)


    def forward(self, x):
        x = x.reshape((1, self.seq_len, self.n_features))
        x, (_, _) = self.rnn1(x)
        x, (hidden_n, _) = self.rnn2(x)

        return hidden_n.reshape((self.n_features, self.embedding_dim))



class Decoder(nn.Module):

    def __init__(self, seq_len, input_dim=64, n_features=1):
        super(Decoder, self).__init__()
        self.seq_len, self.input_dim = seq_len, input_dim
        self.hidden_dim, self.n_features = 2 * input_dim, n_features

        self.rnn1 = nn.LSTM(
            input_size=input_dim,
            hidden_size=input_dim,
            num_layers=1,
            batch_first=True)

        self.rnn2 = nn.LSTM(
            input_size=input_dim,
            hidden_size=self.hidden_dim,
            num_layers=1,
            batch_first=True)

        self.output_layer = nn.Linear(self.hidden_dim, n_features)

    def forward(self, x):
        x = x.repeat(self.seq_len, self.n_features)
        x = x.reshape((self.n_features, self.seq_len, self.input_dim))
        x, (hidden_n, cell_n) = self.rnn1(x)
        x, (hidden_n, cell_n) = self.rnn2(x)
        x = x.reshape((self.seq_len, self.hidden_dim))
        return self.output_layer(x)
</code></pre>

username: Kirondem
cooked: <p>You have 3 ways of approaching this</p>
<ol>
<li>
</li></ol>
<pre><code class="lang-auto">nn.LSTM(input_size, hidden_size, num_layers=2)
</code></pre>
<p><code>num_layers=2</code> would mean stacking two LSTMs together to form a stacked LSTM,</p>
<ol start="2">
<li>
</li></ol>
<pre><code class="lang-auto">self.rnn = nn.Sequential(OrderedDict([
    ('LSTM1', nn.LSTM(n_features, self.hidden_dim, 1),
    ('LSTM2', nn.LSTM(self.hidden_dim, embedding_dim, 1)
]))
</code></pre>
<ol start="3">
<li>
</li></ol>
<pre><code class="lang-auto">self.rnns = nn.ModuleList()
for i in range(nlayers):
    input_size = input_size if i == 0 else hidden_size
    rnns.append(nn.LSTM(input_size, hidden_size, 1))
</code></pre>
<p>Limitation of the first  2 approaches, you can’t get the hidden states of each individual layer.</p>
How to obtain gradients of a simple neural network?

username: ozanaygun
cooked: <p>Hello all,</p>
<p>I’ve been trying to train a federated learning scheme with 25 different workers, so I need to get the gradient vector of each model in my train method. Here is the code:</p>
<pre><code class="lang-auto">def train(model, device, train_loader, optimizer, epoch, num_epochs, worker):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data.float())
        loss = criterion(output, torch.max(target, 1)[1])
        loss.retain_grad()
        loss.backward()
        a = loss.grad
        print(a)
        optimizer.step()
        print ('Epoch %d/%d, Loss: %.4f' 
               %(epoch+1, num_epochs, loss.data))
    worker.train_loss.append(loss.data)
</code></pre>
<p>I am using cross entropy loss with adam optimizer. As an output, I always get 1 as the gradient of each network (losses belong to independent networks). When I remove “loss.retain_grad()”, I always see None as the gradient value.</p>
<pre><code class="lang-auto">tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3039
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3212
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3052
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3079
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3136
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.2929
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3002
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3018
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.2962
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3137
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3000
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3146
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3238
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3346
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3222
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3005
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3209
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.2871
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3237
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.2833
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3195
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3187
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3118
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3188
tensor(1., device='cuda:0')
Epoch 1/10, Loss: 2.3252
</code></pre>
<p>I want to get a gradient vector with a size same with my number of weights (which is 7850). What should I do? Thanks in advance.</p>
Problems with custom loss function

username: tim4ous
cooked: <p>Hello!</p>
<p>I’m writing custom loss function, which looks like this:</p>
<pre><code class="lang-python">class CustomLoss(object):
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta
        
    def calc(self, first, second, k):
        u = torch.exp(-self.beta * F.pairwise_distance(torch.reshape(first, (first.size(0), -1)), 
                                                       torch.reshape(second, (first.size(0), -1))))
        
        D1 = torch.sum(u * torch.log((self.alpha * u) / ((self.alpha - 1) * u + k)))
        D2 = k * torch.log((self.alpha * k) / ((self.alpha - 1) * k + u))
        D2[torch.isnan(D2)] = 0
        D2 = torch.sum(D2)
        
        return D1 + D2
</code></pre>
<p>However, after the first optimizer.step() the loss is equal to nan. Can you tell me what the problem might be?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>Could you check if <code>torch.log</code> get a zero or negative input and might thus return a <code>-Inf</code> or <code>NaN</code>, respectively?</p>

username: tim4ous
cooked: <p>Yes, maybe, that’s why I end up assigning 0 in the case of NaN:</p>
<pre><code class="lang-auto">D2[torch.isnan(D2)] = 0
D2 = torch.sum(D2)
</code></pre>
In the model training,should torch.zero_grad() be put after or before loss.backward()? And why are sometimes loss values fixed?

username: ZhangSheng
cooked: <p>In the model training,should torch.zero_grad() be put after or before loss.backward()? And why are sometimes loss values fixed?</p>

username: Henry_Chibueze
cooked: <p>It should be called b4 fitting the model with some<br>
training input data like this:</p>
<pre><code class="lang-auto">model.zero_grad()
pred_y = model(X)
loss = criterion (pred_y, y)
loss.backward()
optimizer.step()
</code></pre>
<p>As to y some loss values are fixed, it’s a normal thing as long as it’s not fixed through out training.<br>
U should probably change ur model architecture or check if ur datasets is scaled and normalized properly.</p>
<p>Also this is sth I’d like to say from the experience I’ve gotten over the past year: If u are working with structured data then neural networks might not just be the way forward so I suggest u use gradient boosters eg; random forest…</p>
<p>Neural networks are only better than other machine learning models when it comes to heavy lifting like Image data modeling or NLP and language modeling.</p>
Combine 2 channels of an image

username: smu226
cooked: <p>Hello! I have a 2 channel images, but the 2 channels come in different files, so I have 2 tensors of size 64 x 64 each. How can I combine them in a single tensor of size 2 x 64 x 64? I found some ways with view, but I am not totally sure if the resizing is done the way I want (it goes from 128 x 64 to 2 x 64 x 64).</p>

username: ptrblck
cooked: <p>You could call <code>torch.stack</code> on these tensors:</p>
<pre><code class="lang-python">x1 = torch.randn(64, 64)
x2 = torch.randn(64, 64)
x = torch.stack((x1, x2))
</code></pre>

username: moreshud
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I would be glad for your approach of stacking one-hot tensors to have the same dimension. For example. I have x1 = torch.randint(0, 2, size=(1, 3, 3)) -&gt;<br>
tensor([[[0, 1, 1],<br>
[0, 1, 0],<br>
[0, 1, 1]]])<br>
and x2 = tensor([[[1, 0, 0],<br>
[1, 0, 1],<br>
[1, 0, 0]]])<br>
I want to have x3 = tensor([[[1, 1, 1],<br>
[1, 1, 1],<br>
[1, 1, 1]]]).</p>
<p>And how can I have x2 using pytorch function?</p>

username: ptrblck
cooked: <p>I’m not sure if I understand the use case correctly, but you could use <code>OR</code> instead of any stacking operation:</p>
<pre><code class="lang-python">x1 = torch.tensor([[[0, 1, 1],
                    [0, 1, 0],
                    [0, 1, 1]]])
x2 = torch.tensor([[[1, 0, 0],
                    [1, 0, 1],
                    [1, 0, 0]]])
x3 = x1 | x2
print(x3)
&gt; tensor([[[1, 1, 1],
           [1, 1, 1],
           [1, 1, 1]]])
</code></pre>

username: moreshud
cooked: <p>Thanks for the suggestion and that what I need for x3. For x2, torch.where(x1 == 0, 1, x1) worked.</p>

username: moreshud
cooked: <p>I got this error RuntimeError: “bitwise_or_cpu” not implemented for ‘Float’. How can I fix this?</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? You might need to update it, if you are using an older version.</p>

username: moreshud
cooked: <p>The installed version is torch 1.7.0+cpu</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue on <code>1.7.0</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x1 = torch.tensor([[[0, 1, 1],
...                     [0, 1, 0],
...                     [0, 1, 1]]])
&gt;&gt;&gt; x2 = torch.tensor([[[1, 0, 0],
...                     [1, 0, 1],
...                     [1, 0, 0]]])
&gt;&gt;&gt; x3 = x1 | x2
&gt;&gt;&gt; print(x3)
tensor([[[1, 1, 1],
         [1, 1, 1],
         [1, 1, 1]]])
&gt;&gt;&gt; print(torch.__version__)
1.7.0
</code></pre>

username: moreshud
cooked: <p>II is because the dtype is torch.int64. If you set the dtype to torch.float, the error is reproduced</p>

username: ptrblck
cooked: <p>You are right. I didn’t run into this error, as I was using your originally posted tensors.<br>
For these bool operations, I would transform the tensors via <code>.long()</code> or <code>byte()</code> to get the right result or use <code>torch.logical_or(x1, x2)</code> alternatively.</p>

username: moreshud
cooked: <p>Thanks so much. The two approaches worked although torch.logical_or gives boolean.</p>
How to use Tencrop

username: oasjd7
cooked: <p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html?highlight=tencrop#torchvision.transforms.TenCrop" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/transforms.html?highlight=tencrop#torchvision.transforms.TenCrop</a></p>
<blockquote>
<blockquote>
<blockquote>
<p>transform = Compose([<br>
TenCrop(size), # this is a list of PIL Images<br>
Lambda(lambda crops: torch.stack([ToTensor()(crop) for crop in crops])) # returns a 4D tensor<br>
])<br>
<span class="hashtag-raw">#In</span> your test loop you can do the following:<br>
input, target = batch # input is a 5d tensor, target is 2d<br>
bs, ncrops, c, h, w = input.size()<br>
result = model(input.view(-1, c, h, w)) # fuse batch size and ncrops<br>
result_avg = result.view(bs, ncrops, -1).mean(1) # avg over crops</p>
</blockquote>
</blockquote>
</blockquote>
<p>If I want to use <code>result_max</code> instead of `result_avg, then</p>
<p><code>result_max = torch.max(result.view(`bs, ncrops, -1), 1)</code> is ok?</p>

username: ptrblck
cooked: <p><code>torch.max</code> will return the <code>values</code> and <code>indices</code> for the specified dimension, so you might want to access the desired attribute.<br>
Besides that you would get the result containing the max. pixel values from all crops.</p>
Sum() operation produces float32 under autocast(). Is it expected?

username: erogol
cooked: <p>I try to use native amp support and realized that under my model sum() operation produces float32 from f16 inputs whereas all the other operations run with float16. I’d like to create an issue but I am not sure if it is an expected situation.</p>

username: ptrblck
cooked: <p>Yes, this is expected as noted in the <a href="https://pytorch.org/docs/stable/amp.html#op-eligibility">op eligibility section</a>.</p>
Grid_sample padding_mode constant?

username: maaft
cooked: <p>Hi,</p>
<p>I have a use-case where I need grid_sample to use a constant padding_mode (1 instead of 0).<br>
Why is this functionality not included in grid_sample?</p>
<p>Is there any other way how I can achieve this?</p>

username: ptrblck
cooked: <p>Feel free to create a feature request with a description of the use case on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a>.<br>
Contributions are also always welcome. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Convolutions with different filter for each feature in the mini-batch wise

username: rmsgur
cooked: <p>I would like to have a 2d convolution with a dynamic filter.  I have a input data [B, C, H, W] and 1d kernel [B, H, W, k].  I treat each channel equally and apply the same 1d kernel to each of the channels. my aim is to generate [B, C, H, W] data.<br>
What is the correct way to do this?</p>
<p>Thanks a lot.</p>

username: ptrblck
cooked: <p>To apply a filter to a single input channel, you could use <code>groups=in_channels</code> when creating the <code>nn.Conv2d</code> layer.</p>
Is magma included in libtorch?

username: Mohammadreza_Nazari
cooked: <p>I was wondering if MAGMA is included in the prebuilt libtorch? Is it necessary to ensure faster performance in C++?</p>

username: Mohammadreza_Nazari
cooked: <p>Can anyone provide the cmake summary of the standard libtorch to know what packages are active?</p>

username: ptrblck
cooked: <p>I don’t know where <code>libtorch</code> is built exactly, but the build scripts should be included in <a href="https://github.com/pytorch/builder">pytorch/builder</a>.</p>
Is it possible to save modified data to a dataloader?

username: GeoffNN
cooked: <p>Suppose each of my batches are <code>(data, target, modified_variable)</code> where <code>modified_variable</code> has shape <code>(batch_size, *)</code> (each datapoint has an associated <code>mdofied_variable</code>). At a given epoch, I’d like to modify this extra variable, and save it to be accessed at the following epoch, when the same batch is loaded. How can this be done?<br>
As an example: modified_variable could be a perturbed version of data.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Assuming <code>modified_variable</code> is stored inside the <code>Dataset</code>, you could directly manipulate it after the epoch (if <code>persistent_workers=False</code> in your <code>DataLoader</code>) via <code>loader.dataset.modified_variable = ...</code>.</p>
Applying Transforms to a single image:

username: Debayon
cooked: <p>Following is my code:</p>
<pre><code class="lang-auto">from torchvision import datasets, models, transforms
import matplotlib.image as mpimg
import matplotlib.pyplot as plt

import torch



data_transforms = transforms.Compose([
        transforms.RandomResizedCrop(256),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

img = mpimg.imread('dataset/train/cats/cat.1.jpg')
print(type(img))
print(len(img))
# img = img.astype(float)
# img = img.float()
print(img)
img = data_transforms(img)
</code></pre>
<p>Error Message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Image_Preprocessing.py", line 47, in &lt;module&gt;
    img = data_transforms(img)
  File "/home/debayon/anaconda3/envs/ENV2/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/home/debayon/anaconda3/envs/ENV2/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 676, in __call__
    i, j, h, w = self.get_params(img, self.scale, self.ratio)
  File "/home/debayon/anaconda3/envs/ENV2/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 638, in get_params
    area = img.size[0] * img.size[1]
TypeError: 'int' object is not subscriptable
</code></pre>
<p>I am unable to understand why is that occuring and please tell how can I solve it?<br>
Thank You.</p>

username: ptrblck
cooked: <p>Your image seems to be a numpy array.<br>
<code>torchvision</code> transformations work on <code>PIL.Image</code>s, so either load the image directly via <code>Image.open</code> or convert it to a <code>PIL.Image</code> before passing it to the transformations.</p>

username: Debayon
cooked: <p>Oh. Thank You so much. Lesson Learnt: "<code>torchvision</code> transformations work on <code>PIL.Image</code> s "</p>

username: ptrblck
cooked: <p>To be a bit more specific: “a lot of” torchvision’s transformation use <code>PIL</code>, while e.g. <code>Normalize</code> will work on tensors. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: dennywangdev
cooked: <p>Just wonder why? Isn’t numpy image more common?</p>

username: ptrblck
cooked: <p>I think the major advantage is to reuse a common library for loading and processing images.<br>
<code>PIL</code> is one of these libs, but I don’t know which requirements were used to pick it.</p>
<p>Note however that as of <code>torchvision</code> <code>0.8</code> transformations are now supported on tensors and <code>torchvision</code> ships with a native image loading utility. More details can be found in the <a href="https://github.com/pytorch/vision/releases/tag/v0.8.0">release notes</a>.</p>

username: dennywangdev
cooked: <p>Thank you very much for quick responses, and background info is very helpful.</p>
<p>Still not direct support Numpy as input data format is a little bit strange for transform functions. For example, a project deal with both images and videos a lot, we use OpenCV for preprocessing.</p>
CRNN: one of the variables needed for gradient computation has been modified by an inplace operation

username: erikt516
cooked: <p>I am trying to create a CRNN that will feed the output of the CNN to an LSTM layer. However, I am running into an error that there is an inplace error somewhere and I don’t know where. Here is my error message and model “one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 64]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient.”</p>
<p>class CNN_RNN(nn.Module):</p>
<pre><code>def __init__(self):
    super(CNN_RNN, self).__init__()
    self.conv1 = nn.Conv2d(3,16,3)
    self.conv2 = nn.Conv2d(16,32,3)
    self.conv3 = nn.Conv2d(32,64,3)
    self.conv4 = nn.Conv2d(64,128,3)
    self.conv5 = nn.Conv2d(128,64,3)
    self.fc = nn.Linear(128, 16) # (hidden_size, input_size)
    self.pool1 = nn.MaxPool2d(2, 2)
    self.pool2 = nn.MaxPool2d(4, 4)
    self.lstm = nn.LSTM(input_size = 64, hidden_size=64, bidirectional=True, batch_first=True)


def forward(self, x, hidden):
    cnn = self.pool1(F.relu(self.conv1(x), inplace=False))
    cnn = self.pool1(F.relu(self.conv2(cnn), inplace=False))
    cnn = self.pool1(F.relu(self.conv3(cnn), inplace=False))
    cnn = self.pool2(F.relu(self.conv4(cnn), inplace=False))
    cnn = self.pool2(F.relu(self.conv5(cnn), inplace=False)) # outputs (batch, filters, width, height)

    cnn = torch.squeeze(cnn)
    cnn = torch.unsqueeze(cnn, 1)
    lstm, hidden = self.lstm(cnn, hidden)
    lstm = torch.squeeze(lstm)
    out = self.fc(lstm)
    out = F.softmax(out).clone()
    return out, hidden
</code></pre>
<p>model = CNN_RNN()<br>
criterion = nn.NLLLoss()<br>
optimizer = optim.Adam(model.parameters(), lr=learning_rate)</p>
<p>for epoch in range(max_epochs):<br>
for i, data in enumerate(dataloaders[“train”], 0):<br>
inputs, labels = data<br>
optimizer.zero_grad()<br>
outputs, hidden = model(inputs, hidden)<br>
loss = criterion(outputs, labels)<br>
loss.backward(retain_graph=True)<br>
optimizer.step()</p>

username: ptrblck
cooked: <p>I think the <code>backward</code> call might be raising this issue, since you are using <code>retain_graph=True</code> while also updating the parameters, which could be a similar issue to <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this one</a>.</p>
Inconsistent decreases in training loss across training runs

username: cosmosaa
cooked: <p>I have a computer vision problem, but it’s probably relevant to other deep learning models.<br>
Sometimes my model will immediately get stuck at a high training loss and won’t be able to improve from there. If I re-train the model several times, eventually one of the training runs results in learning. I’m assuming this has to do with bad initialization of weights, but I was wondering what all I should try to address this.</p>
<p>I have a small dataset (only ~100 positive examples). The problem is binary classification.</p>
<p>For reference here is the model:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, padding=1)
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)

        self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)
        self.conv4 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1)
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)

        # Input to classifier is input_dim / 4 because 2 max pool layers with kernel_size of 2
        final_dim = int(input_dim / 4)
        self.classifier = nn.Sequential(
            nn.Linear(in_features=64 * final_dim * final_dim, out_features=128),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(in_features=128, out_features=2)
        )

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool1(x)
        x = F.dropout(x, p=.25)

        x = self.conv3(x)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = self.pool2(x)
        x = F.dropout(x, p=.25)

        x = x.reshape(x.shape[0], -1)
        x = self.classifier(x)
        x = F.softmax(x, dim=-1)
        return x
</code></pre>
<p>Using:</p>
<ul>
<li>NLLLoss<br>
-Adam optimizer<br>
-lr 1e-3</li>
</ul>
<p>Data is normalized to .5 mean and .5 std and had range [0, 1]</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p><code>nn.NLLLoss</code> expects log probabilities so you should use <code>F.log_softmax</code> as the last activation instead of <code>F.softmax</code>. Could you change it and check, if your training performs better?</p>