akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is tokenize's type in Field

dedf21972b190932c756: <p>Hi, I am new to deep learning.<br>
I have tired to make sentiment analysis, but I have some problems.<br>
Usually, many people code like this:<br>
TEXT = data.Field(tokenize = ‘spacy’)<br>
LABEL = data.LabelField(dtype = torch.float)</p>
<p>But, I want to make Korean version.<br>
I can’t use spacy, because spacy doesn’t offer Korean.<br>
There is a my question.<br>
How to make custom tokenize class?<br>
I don’t know what type to return, list? str?<br>
What kind of return types does tokenize want in data.Field?<br>
Please answer.<br>
Thank you !</p>

JamesTrick: <p>Hiya!</p>
<p>I’m not familiar with the Korean language, so this answer will be a bit general.</p>
<p>With torchtext, you’re able to pass your own tokenizer to it. In a simple but extensible example check this out.</p>
<pre><code class="lang-auto">def tokenizer(text):
    """
    Function to tokenize given string of text.
    # Arguments:
        text: (str) String to be tokenized.
    # Returns:
        List of tokens.
   """
   return str.split() # This returns the list of tokens.
</code></pre>
<p>You can then pass it into torchtext using the following. <code>TEXT = data.Field(tokenize=tokenizer)</code></p>
<p>To extend it to Korean, you’ll need to find or develop a tokenizer function that works for Korean <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Hopefully that helps!</p>

dedf21972b190932c756: <p>Hi ~<br>
Thank you for comments!!!<br>
I found the Korean tokenizer and applied it ^^<br>
But, I am also curious about how to treat when text is NULL.<br>
I’d like to remove whole train data.<br>
Type is torchtext.data.example.Example.<br>
Do you know about that?</p>
Xstddef(372): error C 2338: The C++ Standard doesn't provide a hash for this type

TomerP: <p>Hi, I’m trying to compile Pytorch on Windows 10, since the precompiled versions on the website won’t support cuda computability 3.0 any more, but I ran into the following error during compilation:</p>
<p>I’d appreciate any help regarding this issue!</p>
<p>“D:\Portable\pytorch\build\INSTALL.vcxproj” (default target) (1) -&gt;<br>
“D:\Portable\pytorch\build\ALL_BUILD.vcxproj” (default target) (3) -&gt;<br>
“D:\Portable\pytorch\build\c10\test\c10_TypeTraits_test.vcxproj” (default target) (51) -&gt;<br>
(ClCompile target) -&gt;<br>
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.10.25017\include\xstddef(372): error C<br>
2338: The C++ Standard doesn’t provide a hash for this type. [D:\Portable\pytorch\build\c10\test\c10_TypeTraits_test.vc<br>
xproj]<br>
D:\Portable\pytorch\c10\test\util\TypeTraits_test.cpp(36): error C2338:  [D:\Portable\pytorch\build\c10\test\c10_Type<br>
Traits_test.vcxproj]</p>
<p>“D:\Portable\pytorch\build\INSTALL.vcxproj” (default target) (1) -&gt;<br>
“D:\Portable\pytorch\build\ALL_BUILD.vcxproj” (default target) (3) -&gt;<br>
“D:\Portable\pytorch\build\caffe2\torch\torch_python.vcxproj” (default target) (703) -&gt;<br>
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.10.25017\include\xstddef(372): error C<br>
2338: The C++ Standard doesn’t provide a hash for this type. (compiling source file D:\Portable\pytorch\torch\csrc\jit<br>
python_arg_flatten.cpp) [D:\Portable\pytorch\build\caffe2\torch\torch_python.vcxproj]<br>
D:\Portable\pytorch\torch/csrc/utils/hash.h(78): error C2668: ‘torch::_hash_detail::dispatch_hash’: ambiguous call to<br>
overloaded function (compiling source file D:\Portable\pytorch\torch\csrc\jit\python_arg_flatten.cpp) [D:\Portable\pyt<br>
orch\build\caffe2\torch\torch_python.vcxproj]</p>
<pre><code>1124 Warning(s)
4 Error(s)</code></pre>

ptrblck: <p>Are you using a script to build from source on Windows?<br>
Have you had a look at <a class="mention" href="/u/peterjc123">@peterjc123</a>’s <a href="https://github.com/peterjc123/pytorch-scripts">scripts</a>?<br>
You’ll also find some <a href="https://github.com/peterjc123/pytorch-scripts#compiling-examples">compiling examples</a> there.</p>

peterjc123: <p>Please update your VS to 15.3.</p>

dashesy: <p>I get this error with “Visual Studio 2017 Developer Command Prompt v15.9.11”</p>
<blockquote>
<p>C:\Users\tobai\AppData\Local\conda\conda\envs\IRIS30\lib\site-packages\torch\lib\include\torch\csrc\api\include\torch/torch.h(7): fatal error C1021: invalid preprocessor command 'warning’error: command ‘C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\HostX86\x64\cl.exe’ failed with exit status 2</p>
</blockquote>
Loss function for 2 Images

ATHARVA_BADVE: <p>Hi,<br>
I am want a loss function which will give me loss in terms of different image comparison methods such as SSD (sum of square differences).<br>
Is there any predefined function for that??<br>
Thank you in advance!!</p>

ptrblck: <p>You might use <code>nn.MSELoss(reduction='sum')</code>.</p>
Why the output of sigmoid always converge to 0.5?

John1231983: <p>I have a network likes</p>
<pre><code class="lang-auto">
input (64x16x16)---&gt; Global Average Pooling (1)--&gt; Conv (1x1)--&gt; ReLU--&gt; Sigmoid()--&gt;Output
</code></pre>
<p>During training, I saw that the output converge to <code>0.5</code>, it means the input of <code>Sigmoid</code> function become zero. What is happening in my network? How should I solve it? Thanks all</p>

dhpollack: <p>There really isn’t enough info in your post to give an extensive answer.  But generally, have you tried removing the sigmoid function and have you made sure that your loss function accepts inputs between 0 and 1?  Many of the loss functions require log probabilities.  Hopefully this helps but if not, you will probably have better luck if you post an example input and your code.</p>

Alvaro_Fernandez: <p>Same problem as you… How did you solve it??</p>
<p>Thanks in advance!<br>
Alvaro</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b7ef668e73ac24fcb89dc0f20ac47198f05395cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7ef668e73ac24fcb89dc0f20ac47198f05395cd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b7ef668e73ac24fcb89dc0f20ac47198f05395cd.png" alt="image" data-base62-sha1="qfai0ORPlgWuCc5NUd7DOEJJTat" width="690" height="411" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1208×721 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Sync and batch-size with multi gpus

ddeeppnneett: <p>In the ResNet, I want to crop the feature map(10X10, with larger input images) before the avgpool and fc. And the crop boxes(with fixed size, say 5x5) have different locations for each input image. The following code is for the single gpu.</p>
<pre><code class="lang-auto">class ResNet(nn.Module):
    def __init__(self,parameters):
        super(ResNet, self).__init__()
    def forward(self, x, crop_info):
        x = self.conv1(x)
        x = self.layer4(x)

        # start 2 crop
        bs,ch,hgh,wid = x.size()
        crop_feat_map = torch.zeros(bs,ch,5,5).type(torch.cuda.FloatTensor)
        # len(crop_info) == bs
        for itr in range(bs):
            x0 = crop_info[itr, 0]
            y0 = crop_info[itr, 1]
            crop_feat_map[itr] = x[itr, :, y0:y0+5, x0:x0+5]

        x = self.avgpool(crop_feat_map)
        x = x.reshape(x.size(0), -1)
        x = self.fc(x)
        return x
</code></pre>
<p>When I use it with torch.nn.parallel.DistributedDataParallel, it is quite slow. I am not sure it is a proper way to crop feat_map  with multi gpus,  since the batch-size is kind of a global variable, and gpus do not share infomation during the forward() part.</p>
<p>For example there are two gpus,  bs=8, with DDP. So each GPU has 4 input images. What would be the bs values(bs,ch,hgh,wid = x.size()) in the first GPU? If 4, everything is fine. If 8, without the extra  sync code, how could the single GPU get the whole data?</p>
<p>If the bs part of the code is correct, how could I speed it up?<br>
If the bs part of the code is wrong, what it the right way to do it?</p>
Trying to understand torch.transpose

isalirezag: <p>I am trying to go through the steps in (<a href="http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a>) regarding to training a classifier.<br>
In this tutorial we have <code>plt.imshow(np.transpose(npimg, (1, 2, 0)))</code> that show the npimg after transposing it.<br>
Can you explain how it exactly transpose it?</p>
<p>based on torch.transpose(<a href="http://pytorch.org/docs/master/torch.html" rel="nofollow noopener">http://pytorch.org/docs/master/torch.html</a>) , if (1,2,0) is corresponds to (dim0,dim1,dim2) what it exactly does?<br>
Sorry but cannot understand what is going on here…<br>
Thanks</p>

moskomule: <p>Hi, in PyTorch, the order of dimension is <code>channel*width*height</code> but in matplotlib it’s <code>width*height*channel</code>. That’s why the <code>transpose</code> is needed.</p>

Ketankvishwakarma: <p>Numpy and Matplotlib actually represents 3 dimension in x,y,z format. Whereas pytorch represents them as z,x,y. So with (np.transpose(npimg, (1, 2, 0)) we are telling numpy the mapping of the indexed. i.e.<br>
for a torch Image matrix with shape (3,32,32) where 3 is z (channel) 32 is x and 32 is y  (np.transpose(npimg, (1, 2, 0)) says transpose the given matrix npimg into numpy where value of x,y.z are at index 1,2,0 respectively.</p>
Regarding Graph Creation using Tensorboardx and pytorch1.1.0

bigmb: <p>Hello,</p>
<p>So I want to create a graph for my Unet model and with pytorch1.1.0 and tensorboardx, it shows me<br>
‘’‘pytorch version is too old, how about build by yourself?’’’ this remark and moves forward.</p>
<p>I have created model on tensorboardx with pytorch1.0.6 post2.</p>
<p>I tried to use torch.utils.tensorboard , but it gives me error too.</p>
<p>Here my summary writer code.</p>
<pre><code class="lang-auto">writer1 = SummaryWriter()
dummy_inp = torch.randn(1, 3, 128, 128)
model_test.to('cpu')
writer1.add_graph(model_test, model_test(torch.randn(3, 3, 128, 128, requires_grad=True)))
model_test.to(device)
</code></pre>
<p>If you have a working example of creating models with pytorch1.1.0 for skip connections one please refer me to that too.</p>
<p>Thank you.</p>
Resetting Optimizer Params without resetting learning rate etc

Chrixtar: <p>Hello there,</p>
<p>after I optimize an embedding for a while, I change the embedding (i.e. I change the dimensions and values of the embedding tensor by instantiating a new nn.Embedding).<br>
Do I have to reset the params of my optimizer to keep optimizing this updated embedding and if so, how can I do it without creating a whole new optimizer instance, which would reset the adaptive learning rate and the weight decay?</p>
<p>Thank you already for looking into this.</p>
<p>Best regards<br>
Chrixtar</p>

MariosOreo: <p>Hi,</p>
<p>You could replace old_embedding.parameters() in <code>optimizer.param_groups[0]['params']</code> by new_embedding.parameters() to fit your requirement.</p>

Chrixtar: <p>Yes, thank you, this is what I was looking for.</p>
Scatter_add gives different outputs at each run

el_samou_samou: <p>Hi,</p>
<p>If I run scatter_add twice in the exact same setting, I end up with a non-negligible gap between both results. This is weird as they should be exactly the same results since they correspond to the exact same operations. Here is a code to reproduce this strange behavior:</p>
<pre><code class="lang-auto">import torch


splat_1 = torch.zeros(1, 8, 80).cuda()
splat_2 = torch.zeros(1, 8, 80).cuda()


for i in range(100):
    indices = torch.randint(0, 80, (40000,)).unsqueeze(0).repeat(8, 1).cuda()

    feat = torch.rand(8, 40000).cuda()

    splat_1[0] = splat_1[0].scatter_add(1, indices, feat)
    splat_2[0] = splat_2[0].scatter_add(1, indices, feat)

print(((splat_1-splat_2)**2).sum())
</code></pre>
<p>I obtained on average 0.04 squared difference.</p>
<p>Any explanation on this? I assume it is due to some computation error accumulation but since these are the exact same operations, should not the error get accumulated in the exact same way both for splat_1 and splat_2?</p>
<p>Thanks in advance for your help.</p>
<p>Samuel</p>

ratishsp: <p>I think the reason for this is explained in <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a>.<br>
<code>There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular atomicAdd, where the order of parallel additions to the same value is undetermined and, for floating-point variables, a source of variance in the result. PyTorch functions that use atomicAdd in the forward include torch.Tensor.index_add_(), torch.Tensor.scatter_add_(), torch.bincount().</code><br>
An example of such addition is <code>1 + 2**100 - 2**100</code> which can be <code>1</code> or <code>0</code> depending on the order of operations.</p>
<p>One thing I am unsure about is why we don’t see similar behavior with  <code>torch.sum()</code></p>
Difference of a matrix

matliu: <p>Given a matrix with the shape (1,3, H, W), I wish to obtain its difference in x-direction by channels.  I use the function F.conv2d as follows.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np
import torch.nn.functional as F

# the shape of bdx3 is 1X3X3X3
bdx3 = torch.Tensor([[[[0.0,  0.0,  0.0], [0.0, 1.0, -1.0], [0.0, 0.0,  0.0]], 
					[[0.0,  0.0,  0.0], [0.0, 0.0, 0.0], [0.0, 0.0,  0.0]],
					[[0.0,  0.0,  0.0], [0.0, 0.0, 0.0], [0.0, 0.0,  0.0]]
					],
					[[[0.0,  0.0,  0.0], [0.0, 0.0, 0.0], [0.0, 0.0,  0.0]], 
					[[0.0,  0.0,  0.0], [0.0, 1.0, -1.0], [0.0, 0.0,  0.0]],
					[[0.0,  0.0,  0.0], [0.0, 0.0, 0.0], [0.0, 0.0,  0.0]]
					],
					[[[0.0,  0.0,  0.0], [0.0, 0.0, 0.0], [0.0, 0.0,  0.0]], 
					[[0.0,  0.0,  0.0], [0.0, 0.0, 0.0], [0.0, 0.0,  0.0]],
					[[0.0,  0.0,  0.0], [0.0, 1.0, -1.0], [0.0, 0.0,  0.0]]
					]
					])
class MyGradientx(nn.Module):
    def __init__(self):
        super(MyGradientx, self).__init__()

    def forward(self, input):
        output = F.conv2d(input, bdx30, padding=1)
        return output

H = 4
W = 5
u = np.arange(0, 3*H*W).astype(np.float32).reshape(1, 3, 4,5)
u = torch.from_numpy(u)
print("this is u: \n", u)

net1 = MyGradientx()
u1 = net1(u)

print("this is u1: \n", u1, "\n u1 shape", u1.shape)
</code></pre>
<p>I get the results:</p>
<pre><code class="lang-auto">this is u:
 tensor([[[[ 0.,  1.,  2.,  3.,  4.],
          [ 5.,  6.,  7.,  8.,  9.],
          [10., 11., 12., 13., 14.],
          [15., 16., 17., 18., 19.]],

         [[20., 21., 22., 23., 24.],
          [25., 26., 27., 28., 29.],
          [30., 31., 32., 33., 34.],
          [35., 36., 37., 38., 39.]],

         [[40., 41., 42., 43., 44.],
          [45., 46., 47., 48., 49.],
          [50., 51., 52., 53., 54.],
          [55., 56., 57., 58., 59.]]]])
this is u1:
 tensor([[[[-1., -1., -1., -1., 4.],
          [-1., -1., -1., -1., 9.],
          [-1., -1., -1., -1., 14.],
          [-1., -1., -1., -1., 19.]],

         [[-1., -1., -1., -1., 24.],
          [-1., -1., -1., -1., 29.],
          [-1., -1., -1., -1., 34.],
          [-1., -1., -1., -1., 39.]],

         [[-1., -1., -1., -1., 44.],
          [-1., -1., -1., -1., 49.],
          [-1., -1., -1., -1., 54.],
          [-1., -1., -1., -1., 59.]]]])
 u1 shape torch.Size([1, 3, 4, 5])
</code></pre>
<p>The first problem is if there are some functions in pytorch for the difference to a matrix give the same results?</p>
<p>Thank you!</p>
How to change the Adam Optimizer's parameters when adding neurons

Joe_Harrison: <p>I’ve written code that adds hidden neurons to each hidden layer during training. I’m using the Adam optimizer which maintains a learning-rate for each parameter. At the moment I create a new optimizer with the new model parameters whenever nodes are added to the hidden layers. By doing this I lose the learning-rates for each parameter that was present before the addition of neurons.</p>
<p>How do I keep the same learning-rates per parameter? Is the correct way of doing this adding rows and columns of zeros to the parameters, exp_avg_sq and exp_avg or is there an easier way?</p>

acobobby: <p>You can add new units to your optimizer and keep same hyperparameters without recreating it by using <code>add_param_group</code> like this:</p>
<pre><code class="lang-auto">optimizer.add_param_group({'params': [p for p in new_units.parameters()]})
</code></pre>
<p>where <code>new_units</code> in this case is a list of modules containing parameters.<br>
Look <a href="https://pytorch.org/docs/stable/optim.html?highlight=add_param_group#torch.optim.Optimizer.add_param_group" rel="nofollow noopener">here</a> for the official docs.</p>
<p>Hope this helps.</p>

Joe_Harrison: <aside class="quote no-group" data-username="acobobby" data-post="2" data-topic="44989">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f0a364/48.png" class="avatar"> acobobby:</div>
<blockquote>
<p>where <code>new_units</code> in this case is a list of modules containing parameters.</p>
</blockquote>
</aside>
<p>There are no new modules. The neurons added are to existing layers, not new layers. When I print the network parameters they show up in the same parameter group.</p>

MariosOreo: <p>Hello,</p>
<p>Did you get easier method rather than padding?</p>
How to use QNNPACK?

hossein1387: <p>I am trying to understand how to use <strong><a href="https://github.com/pytorch/QNNPACK" rel="nofollow noopener">QNNPACK</a></strong>. The repo lacks documentation (as it is already mentioned in issues raised in github repo). Can some one explain how to use it? In the repo desciption it is mentioned <em>" As of today, QNNPACK is integrated in <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">PyTorch 1.0</a> with Caffe2 graph representation."</em> but no where in the Pytorch documentation explains how it is integrated and more importantly, what are the hooks/funtions are availabe for Pytorch users to use this library.</p>

blacky_i: <p>I am also interested in how to use qnnpack. All I found in sources, that pytorch jit can <a href="https://github.com/pytorch/pytorch/commit/acbd9c49b0f3414b2a161731b0b46aa0712d78f7#diff-78aab5b4217aeaed920efab0c5f5f5c8R9396" rel="nofollow noopener">quantize linear layers</a>.</p>
<p>But I’m  thinking it is applicable for amd64 processors, not for arm… Trying to find some tests in sources to quantize model, but no luck</p>
Https://pytorch.org/tutorials/advanced/cpp_export.html

dinesh_mareedu: <p>While trying load model using c++ from pytorch.  I could not able do the trace, I got validation error.  To solve that error we used deepcopy method and then used trace method. So that,  my model size got increased from 2mb to 4mb. Can anyone show me the way to reduce the model size…</p>
When training FCN, GPU is lost

jiujing23333: <p>When I train FCN, after tens of iterations(multi-GPU), command nvidia-smi print “GPU is lost”. I wonder this problem never appeared when I use Tensorflow and caffe in the same GPU server. Besides, GPU is lost when the batch size equals to 16, if less than 16, for example it equals to 4, the phenomenon would not happen.</p>
<p>So, I think it’s some bug in Pytorch rather than my GPUs.</p>

Noirmist: <p>Hi, did you solve the issue? I get same problem when I’m using 4 gpu at once.<br>
Either 2 model using 2 gpu each and 1model using 4 gpu cause “GPU is lost”<br>
It’s very tricky. It is fine, when I’m using only 2 gpu at once.</p>
<p>Please share your experience about how to deal with your problem.</p>
Is there a way to load the initiated hidden states to the device with the model?

ChangGao: <p>Hi! I used to use “<code>model.to(device_gpu)</code>” to train my network. However, when the RNN is involved, this couldn’t load the initiated hidden states to the GPU. In that case, I needed to additionally load the initiated hidden states to the GPU inside the model:<br>
<code>hidden = torch.zeros(num_layers, batch_size, hidden_size).to(device_gpu)</code><br>
Is there a way to directly load the hidden states to the</p>
GPU memory increasement for multi-task learning

Vane: <p>Hello. I have some questions about my GPU memory usage when doing multi-task training. Suppose I have two tasks which share the shallow conv layers and then separated into two branches and calculate two losses. When the two losses are added and then backward, like loss = loss1 + loss2; loss.backward(), the GPU memory usage is about 7971MB, but when only one of the two losses is backward, like loss=loss1, loss.backward(), the GPU memory usage increases to 88xx MB.<br>
I cannot figure out the explicit reason why the memory usage increases a lot. The possible reason I thought is about the unreleased graph in loss2 part, but I still dont know exactly what are included in the increased memory. Please help me and thank you very much!</p>

cuixing158_1: <p>could you show your code?</p>

Vane: <p>Thanks for replying! It is a big project and the code is a bit complicated, so can you detail the part you want to check?</p>
<p>PS: If I use the form: loss = loss1 + 0*loss2, the memory usage is normal (7971MB).</p>
Gpu memory increase ,second forward use far more gpu memory

Wu_jiang: <p>when run code :</p>
<pre><code class="lang-auto">import torch
from torch import nn

class Test(nn.Module):
  def __init__(self, in_ch , n_classes):
    super(ResASPSPNetEncoder, self).__init__()
    self.conv1 = nn.Conv2d(1280,1024,1)
  def forward(self, x):
    x = self.conv1(x)
    return x

model = Test(in_ch=7,n_classes =3)
model = torch.nn.DataParallel(model,device_ids=list(range(2))).cuda()
x = torch.zeros((8,1280,256,256),dtype=torch.float)

for i in range(2):
  y = model(x) 
  print(i) 
  # set breakpoint，use nvidia-smi to watch GPU memory
</code></pre>
<p>when set breakpoint after line “y = model(x)”, and use nvidia-smi to watch GPU memory,<br>
second use far more gpu memory , why ？<br>
first forward :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/004d5b802bf0d56f94e9f663bef8a72a9270891d.jpeg" alt="image" data-base62-sha1="2FJJ15WpJxhIPMtdZwSNUZLxoN" width="678" height="210"><br>
second forward :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6126efaaeedc0866daec44dba1584436c2af62b6.jpeg" alt="image" data-base62-sha1="dRrJzzKLLyZXoW9KYV8zkdHvuUm" width="632" height="210"><br>
THX</p>

alex.veuthey: <p>Because your model is set to <code>train</code> mode by default, pytorch is probably keeping the forward flow of the network with the specific inputs for future gradient computations. If you only want to use your model for inference, you should try calling <code>model.eval()</code> first, and also use <code>with torch.no_grad():</code> before all calls that should not compute gradients.</p>
<p>If you want to train, however, you should try with an optimizer and loss function and follow the usual procedure, i.e:</p>
<pre><code class="lang-python">model.train()
for i in range(2):
    # forward
    y = model(x)
    # reset previous gradients (free memory)
    optimizer.zero_grad()
    # compute gradients across the network
    loss.backward()
    # update weights according to gradients
    optimizer.step()
</code></pre>
Torch.jit.trace model relates tensor to specified gpu_id and when using traced model in libtorch the tensor will move to the previous specified gpu_id which can not be changed?

cfanyyx: <p>torch.jit.trace(model, example, check_trace=False) relates tensor to specified gpu_id and when using traced model in libtorch the tensor will move to the previous specified gpu_id which can not be changed?</p>
Is it possible to fix the seed when the program is running on slurm cluster

coincheung: <p>Hi,</p>
<p>I add the line to fix the seed for random operations at the beginning of my code, like this:</p>
<pre><code class="lang-python">import model # definition of my model, containing randomly initializing the parameters
import components
import optim

torch.manual_seed(123)

def train():
 ....
</code></pre>
<p>The import is before the line of <code>torch.manual_seed(123)</code>, but the model is created after that line.</p>
<p>I run the same code on a 8-gpu machine for two times, and the result is quite near, with  difference to be less than 0.1, but after moving my code to some slurm cluster, the difference is enlarged up to 0.2-0.8. The difference of the two running is that slurm allocates my code to different nodes to execute each time I submit a task. But I dont have proof whether this has anything to do with the difference.</p>
<p>Does  <code>torch.manual_seed(123)</code> have different behaviors on different machines? How could I fix the random behavior please?</p>
Is dist.all_reduce block or not, or is there memory leakage or others

coincheung: <p>Hi,</p>
<p>thanks for developing pytorch for us!!</p>
<p>I am using two gpus to train my model, and I use <code>torch.distribtued</code> to manepulate them. After training, at the end of the program, I used a line of <code>dist.all_reduce(hist, dist.ReduceOp.SUM)</code> to merge the tensors on two gpus.</p>
<p>My problem is that: The program seems to continue executing after this line and I could see the following log messages from the <code>rank=0</code> process. But after the last log message is printed to the screen, the program does not cease. I can see from the <code>nvidia-smi</code> that the memory on the second gpu is released and the process on that gpu is cleaned, but the memory on the first gpu is not totally cleaned, with around 2G still remained in the gpu(the gpu usage ratio is 0%). The associated lines in the output of <code>ps aux</code> is like this:</p>
<pre><code class="lang-auto">yz    330577  0.0  0.0 2945348 51632 ?       S    May12   0:00 /mnt/ai-vision/home/yz/miniconda3/envs/py37/bin/python -m torch.distributed.launch --nproc_per_node=2 train.py
yz    330599  354  0.8 30957028 2308304 ?    Sl   May12 3176:02 /mnt/ai-vision/home//miniconda3/envs/py37/bin/python -u train.py --local_rank=0
</code></pre>
<p>What is the cause of this please?</p>
Loss Function not supported error

Codacus: <p>I am trying to create a custom loss function,</p>
<pre><code class="lang-auto">def loss(x,y):
    x_bb=x[:,:4]*224
    x_c=x[:,4:]
    y_bb=y[:,:4]*224
    y_c=y[:,4:]
    return F.l1_loss(x_bb.to(dtype=torch.long),y_bb.to(dtype=torch.long))+F.cross_entropy(x_c,y_c)*20
</code></pre>
<p>but it’s throwing me this error</p>
<pre><code class="lang-auto">RuntimeError: _thnn_l1_loss_forward not supported on CPUType for Long
</code></pre>
<p>I tried with different datatypes and with GPU as well, but got the same error, just instead of “CPUType” it’s showing the same error with “GPUType”</p>

iArunava: <p><code>.float()</code> is supported, just do that. It will work.</p>
About DataParallel

genous110: <p>The DataParallel function distributes the batch_size evenly to each GPU, and then each GPU processes the allocated data, but in my program, I need to re-integrate an intermediate output variable, that is, firstly， aggregate this variable from all GPUs. Then use the Tensor.view() function to re-integrate the data, and finally distribute the integrated data to each GPU again. Every time I do this, the program will report an error. Is there a solution?</p>
<p>x = x.view(self.batch_size * self.num_segments, 3, 224, 224)<br>
RuntimeError: shape ‘[12, 3, 224, 224]’ is invalid for input of size 602112</p>

JuanFMontesinos: <p>Hi, what you can do is to use dataparallel twice inside the nn.Module like this:</p>
<pre><code class="lang-auto">class model(Module):
def init():
   self.subnet1 = DataParallel(net1).to(device)
   self.subnet2 = DataParallel(net2).to(device)
def forward():
  x = self.subnet1(x)
  # Here x is gathered from all gpus
  x = self.subnet2(x) # DataParallel again
  return x
</code></pre>

genous110: <p>Thank you!   I know what you said. But in my program, the model is very big, and the x is also very big, the size of x is 32 * 32 * 96 * 28 * 28, type of x is torch.FloatTensor. If x is gathered to one GUP, the GPU would explode</p>

JuanFMontesinos: <p>If you let me know, what’s the point of gathering back everything?<br>
I mean,<br>
you have batches x segments x others in a per-gpu level</p>
<p>if you gather everything back you will get</p>
<p>N *batches x segments x others in a global level<br>
when you reshape you will get</p>
<p>N * batches * segments x others</p>
<p>But you are gonna get same ordering once you re-send everything to gpus.</p>
<p>Another problem I see is that if you have not enough memory to allocate everything in the same device, how do you expect to perform reshaping?</p>

genous110: <pre><code class="lang-auto">lstm = torch.nn.LSTM(10, 20,1)
lstm.state_dict().keys()
</code></pre>
<p>Output result：</p>
<pre><code class="lang-auto">Out[47]: odict_keys(['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0'])
</code></pre>
<p>According to the calculation process of LSTM, there should be only one bias. Why do we output two bias variables, that is,‘bias_ih_l0’and’bias_hh_l0’?</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/why-lstm-has-two-bias-parameters/45025/2">here</a>.</p>
Size mismatch while loading state dictionary

Anshumaan_Dash: <p>Hi</p>
<p>While loading a pre-trained model, I am getting the same runtime error for every layer.<br>
E.g. <code>size mismatch for conv_batchnorm.weight: copying a param with shape torch.Size([1, 32]) from checkpoint, the shape in current model is torch.Size([32])</code></p>
<p>Kindly help me figure out how to solve this problem.</p>
<p>Thanks</p>

ptrblck: <p>Is <code>conv_batchnorm</code> an <code>nn.BatchNorm</code> layer?<br>
Was the <code>state_dict</code> saved in another PyTorch version than the one you are using to reload it?</p>
Using elements from a tensor to build another tensor

cgo: <p>Hi all,<br>
not sure if I can formulate my problem in the right way, as I may not see the actual problem.<br>
Here is what I am trying to do:</p>
<ul>
<li>Network A is generating N numbers as output, in  a Nx1 tensor, t.</li>
<li>I want to use the elements of t to create a matrix M (a new tensor), and set some of the<br>
matrix entries to an element from t.</li>
<li>Then, I generate a sample grid g’ using torch.matmul(M, g), where g is just a grid<br>
mapping (i,j) to (i,j), the identity.</li>
<li>Then, I use g’ with torch.nn.functional.grid_sample() to resample an image.</li>
<li>I use the resampled image and the input image to compute a loss.</li>
</ul>
<p>My naive expectation was that the backward pass starting at the loss would compute<br>
gradients for the entries in t, and that they would be updated by the optimizer.step() function.<br>
However, that is not happening; t is not changing at all, while other parameters are.</p>
<p>Can anyone help me to find where I am making the mistake?</p>
<p>Thanks a lot!</p>
Output dimension after using padding with Conv2d

thyr: <p>Hi, I need to add padding to my input before the convolution layer. Earlier I was adding the padding manually myself but now I’m trying to add it using the padding attribute in nn.Conv2d and I’m getting some unexpected outputs.</p>
<p>Suppose my input is</p>
<blockquote>
<p>sample = torch.randn(1, 1, 11, 128)</p>
</blockquote>
<p>Now I need to do a convolution of window size 2 along the axis 2. Hence I did as below</p>
<blockquote>
<p>conv = nn.Conv2d(1, 3, (2, 128))<br>
conv(sample).size()<br>
torch.Size([1, 3, 10, 1])</p>
</blockquote>
<p>The output here made sense as there are 3 output channels and (11-2+1) = 10 is the dimension of axis 2</p>
<p>Now I’m trying to add a padding of size (1, 128) to the input before the convolution window. Hence I did as below</p>
<blockquote>
<p>conv2 = nn.Conv2d(1, 3, (2, 128), padding=(1,128))</p>
</blockquote>
<p>Now my understanding is that before convolution, padding of size (1,128) will be added on both sides. Hence new input size will be [1, 1, 11+1+1, 128]  And convolution of size 2 will give 13-2+1=12 values and 3rd dimension of the output will have size 1. But instead I’m getting a different value</p>
<blockquote>
<p>conv2(sample).size()<br>
torch.Size([1, 3, 12, 257])</p>
</blockquote>
<p>Why is the axis 3 size = 257 instead of 1? It will be great if someone can help me understand this.</p>
<p>Thank you</p>

kaixin: <p>You are padding 128 values to both sides of the axis 3 when setting <code>padding=(1, 128)</code>. To get an output of shape <code>(1, 3, 12, 1)</code>, you may set <code>padding=(1, 0)</code>.</p>
RuntimeError: cur_offset == offset ASSERT FAILED

Sais: <p>Hi all,</p>
<p>I try to utilize LSTM to classify the time series data. The program report this error.<br>
When I try to load another dataset. This program runs well.<br>
I don’t know what’s gonging wrong.</p>
<h2>Environment</h2>
<ul>
<li>PyTorch Version: 1.0.0</li>
<li>OS (e.g., Linux): Windows 10</li>
<li>How you installed PyTorch:  <code>conda</code>
</li>
<li>Python version: 3.6.5</li>
<li>CUDA/cuDNN version: 8.0</li>
<li>GPU models and configuration: Nvidia GeForce GTX 1060</li>
</ul>
<p>Here is the message:<br>
Traceback (most recent call last):<br>
File “test_version1.py”, line 236, in <br>
loss.backward()<br>
File “D:\ProgramFiles\Anaconda\lib\site-packages\torch\tensor.py”, line 102, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “D:\ProgramFiles\Anaconda\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 90, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: cur_offset == offset ASSERT FAILED at …\aten\src\ATen\native\cudnn\RNN.cpp:471, please report a bug to PyTorch. cur_offset = 1860; offset = 930</p>
<pre><code class="lang-auto">            input = input.cuda(async = True)
            target = target.cuda(async = True)
            # Forward
            output = model(input)
            loss = F.cross_entropy(output, target)
            losses["train"] += loss.item()
            # Compute accuracy
            _,pred = output.data.max(1)
            correct = pred.eq(target.data).sum().item()
            accuracy = correct/input.data.size(0)
            accuracies[split] += accuracy
            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
</code></pre>
<p>Thanks!</p>
What's wrong with the simple demo using fashion-mnist?

HX-idiot: <p>Hello guys, I just wanna have a try using pytorch on Fashion-Mnist, here is a demo I test, the NN structure is copied from a solution on Kaggle, but the accuracy is still low, I don’t know why…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/32b89a010bf67464c67fb600400266baf67e3bdc.png" data-download-href="https://discuss.pytorch.org/uploads/default/32b89a010bf67464c67fb600400266baf67e3bdc" title="09"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/32b89a010bf67464c67fb600400266baf67e3bdc_2_617x500.png" alt="09" data-base62-sha1="7eHnAWVW8TLbQUBFQTAxAdarw72" width="617" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/32b89a010bf67464c67fb600400266baf67e3bdc_2_617x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/32b89a010bf67464c67fb600400266baf67e3bdc_2_925x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/32b89a010bf67464c67fb600400266baf67e3bdc_2_1234x1000.png 2x" data-dominant-color="EEEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">09</span><span class="informations">2002×1622 472 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

HX-idiot: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2abefe5e30e22b8f0f6cd2d0e4765133c7fa5e61.png" data-download-href="https://discuss.pytorch.org/uploads/default/2abefe5e30e22b8f0f6cd2d0e4765133c7fa5e61" title="21"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2abefe5e30e22b8f0f6cd2d0e4765133c7fa5e61_2_556x500.png" alt="21" data-base62-sha1="669fQXtYJY6qqVfNr0xE4Y95DbP" width="556" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2abefe5e30e22b8f0f6cd2d0e4765133c7fa5e61_2_556x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2abefe5e30e22b8f0f6cd2d0e4765133c7fa5e61_2_834x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2abefe5e30e22b8f0f6cd2d0e4765133c7fa5e61_2_1112x1000.png 2x" data-dominant-color="EFEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">21</span><span class="informations">1584×1424 341 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

HX-idiot: <p>The acc is around 10% no matter how many epochs I train , that’s confused, I change the dataset to Mnist, it works well.</p>
What does 'pth.tar' extension mean?

nulledge: <p>In some projects, they called <code>torch.save</code> and <code>torch.load</code> with ‘pth.tar’ extension.<br>
I think ‘pth’ is for ‘<strong>P</strong>y<strong>T</strong>orc<strong>H</strong>’, but what is ‘tar’ for…?</p>
<p>Actually I had tried to open ‘pth.tar’ files with Archive Manager and failed <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Just for curious.<br>
Thanks in advance.</p>

ptrblck: <p><code>.tar</code> refers to tarball and is an archive file.<br>
As far as I know, the legacy serialization module used <code>tarfile</code>.</p>
<p>You can find more information about this file format <a href="https://en.wikipedia.org/wiki/Tar_(computing)" rel="nofollow noopener">in the Wikipedia article</a>.</p>

tom: <p>To add some advice you didn’t ask for: I’d recommend to just use <code>.pt</code> as the extension.<br>
As you rightly note it’s been long ago that this used to be a tar file, and pth isn’t particularly compatible with Python’s “.pth” path files (there is an open issue to standardize on <code>.pt</code> in the PyTorch examples).</p>
<p>Best regards</p>
<p>Thomas</p>

nulledge: <p>Thanks for your replies.</p>

jon: <p>I agree, since it doesn’t seem like we actually need to tar/untar files when we use Pytorch’s <code>save</code> and <code>load_state_dict</code> functions. Hence, it’s just unnecessary overhead to explicitly write out the tar</p>
Customized collate_fn gives False pin_memory in the official example

AlexisW: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L112" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L112" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L112</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="102" style="counter-reset: li-counter 101 ;">
<li>The default memory pinning logic only recognizes Tensors and maps and iterables</li>
<li>containg Tensors.  By default, if the pinning logic sees a batch that is a custom type</li>
<li>(which will occur if you have a ``collate_fn`` that returns a custom batch type),</li>
<li>or if each element of your batch is a custom type, the pinning logic will not</li>
<li>recognize them, and it will return that batch (or those elements)</li>
<li>without pinning the memory.  To enable memory pinning for custom batch or data types,</li>
<li>define a ``pin_memory`` method on your custom type(s).</li>
<li>
</li>
<li>Example::</li>
<li>
</li>
<li class="selected">    class SimpleCustomBatch:</li>
<li>        def __init__(self, data):</li>
<li>            transposed_data = list(zip(*data))</li>
<li>            self.inp = torch.stack(transposed_data[0], 0)</li>
<li>            self.tgt = torch.stack(transposed_data[1], 0)</li>
<li>
</li>
<li>        def pin_memory(self):</li>
<li>            self.inp = self.inp.pin_memory()</li>
<li>            self.tgt = self.tgt.pin_memory()</li>
<li>            return self</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Running this piece of code gives me</p>
<p>False<br>
False<br>
False<br>
False<br>
False<br>
False<br>
False<br>
False<br>
False<br>
False</p>
<p>Is that normal?</p>

SimonW: <p>it’s normal if you are using old pytorch or does not have cuda</p>
Multithread deadlock

pascal.soveaux: <p>Hello,</p>
<p>I am using v1.0.1/CUDA 10.1 on a  2080. glibc is 2.28.</p>
<p>I read that libtorch is multithread safe but I cannot make it work correctly in a MT C++ app. I have 2 trainers each one in its own thread, no global var, no static var, no shared var… but it always ends in a deadlock.</p>
<p>When deadlocked both threads are near cuLaunchKernel. Is there something I need to do ? Do I have to initialize a context or something?</p>
<p>I also tested with master and a nightly of libtorch but it ends the same.</p>
<p>Thanks,</p>
<p>Pascal</p>

pascal.soveaux: <p>Hello,</p>
<p>Great! it seems that PoolWindow makes it work correctly.</p>
<p>I run into a problem where available_handles (global) was destructed <em>before</em> PoolWindow (thread_local) (Valgrind told me).</p>
<pre><code class="lang-auto">available_handles[d_h.first].push_back(d_h.second); // SIGSEGV
</code></pre>
<p>I dont know how this can happen, I will try to reproduce.</p>
<p>Pascal</p>

yf225: <p><a class="mention" href="/u/pascal.soveaux">@pascal.soveaux</a> <a href="https://github.com/pytorch/pytorch/issues/19394" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/19394</a> is likely related. Would you like to comment in the issue about your findings? Thanks!</p>

pascal.soveaux: <p><a class="mention" href="/u/yf225">@yf225</a> Sorry for the late response. I found that this problem is totally not related with poolwindow. The thread is exited after the “main” (libtorch) thread because both are worker threads in fact.</p>
"WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode" while running a pytorch code

amirhf: <p>I am running a pytorch code in CPU mode, why am I getting a caffe2 warning!?</p>

amirhf: <p>I think it is because of the new tensorboard integration. So I have this line in my code:<br>
<code>from torch.utils.tensorboard import SummaryWriter</code></p>
Extraction of activations from a layer

muammar: <p>I have a model and would like to regularize the activations of one of its layers. I wondered if I had to do something “special” to get them or just returning the layer I need from <code>forward()</code> would work?</p>
<p>I would appreciate if anyone could give me a hint about it. I probably am overthinking this.</p>

LeviViana: <p>Doesn’t <code>torch.nn.BatchNorm</code> fit your request? Otherwise, you can always break your forward method into many other methods to get the intermediate results, and normalize them as you wish. The most complex case could come from modules that you didn’t implement yourself, in that case, you can always override this module’s forward method. For instance:</p>
<pre><code class="lang-python">import torch
from torchvision.models.resnet import ResNet, BasicBlock, resnet18
class NewResNet(ResNet):
    def __init__(self, **kw):
        super(NewResNet, self).__init__(**kw)
    def forward_1(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)
        x = self.layer1(x)
        return x
    def forward(self, x):
        x = self.forward_1(x)
        # Potentially a new normalization could happen here ...
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)
        return x

# A little check to validate that we didn't break the module
torch.manual_seed(2019)
net = NewResNet(block=BasicBlock, layers=[2, 2, 2, 2])
torch.manual_seed(2019)
net_check = resnet18(pretrained=False)

img = torch.rand((1, 3, 224, 224))
res1 = net_check(img).sum()
res2 = net(img).sum()

print(res1, res2) # Same results
</code></pre>
Downsampling tensors

jj0mst: <p>How can i downsample a tensor representing an image using Nearest/Bilinear interpolation?</p>
<p>I’ve tried using <code>torch.nn.Upsample</code> with a size smaller than the original one, my outputs seem fine and i don’t get any errors. Are there any problems i’m not seeing with this kind of usage of Upsample?</p>

IanYeung: <p>Same question here…</p>

jj0mst: <p>Finally it has been implemented by<br>
<a href="https://pytorch.org/docs/stable/nn.html?highlight=interpolate#torch.nn.functional.interpolate" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html?highlight=interpolate#torch.nn.functional.interpolate</a><br>
<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>

henrique: <p>Upsample uses F.interpolate as suggested.<br>
We can check the source to see what’s actually doing: <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#interpolate" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#interpolate</a><br>
The default mode ‘nearest’ doesn’t suit downscaling but ‘bilinear’ works fine (please correct me if I’m wrong!).<br>
Otherwise, just use mode ‘area’ or adaptive_avg_pool2d directly. However, the latter seems a bit slower than ‘bilinear’.</p>
Test Loss in not decreasing

Deb_Prakash_Chatterj: <p>I am making a four-class classifier and I have a dataset of four classes containing 10000 images in the training set and 8 in the validation set. So, I took 1000 images from the training set and 192 images from the training set, which are not used in training and add it to the valid set to make a proper ratio. Now when I had only 8 images in the validation set, it is giving me a good accuracy and valid loss was good too, but after adding 192 extra images, the loss decreasing has almost stopped and accuracy is about 55% max. I have tried with many learning rates, but nothing changing. Can you guys help me out to increasing the accuracy and decreasing the test loss?</p>