akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What does continuous mean, isn't any array is continuous?

CHENCHAO0526: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/af618208a667e3094bb390549f37cf94a4ecca8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/af618208a667e3094bb390549f37cf94a4ecca8a" title="Screenshot%20from%202019-09-17%2010-36-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/af618208a667e3094bb390549f37cf94a4ecca8a_2_690x394.png" alt="Screenshot%20from%202019-09-17%2010-36-58" data-base62-sha1="p1usXFD0jPkNAlEMF5YSov7tIWC" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/af618208a667e3094bb390549f37cf94a4ecca8a_2_690x394.png, https://discuss.pytorch.org/uploads/default/original/2X/a/af618208a667e3094bb390549f37cf94a4ecca8a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/af618208a667e3094bb390549f37cf94a4ecca8a.png 2x" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-17%2010-36-58</span><span class="informations">1006×575 60.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jrue1994: <p>Tensors are saved in a whole block in MEMORY,<br>
contiguous means adjacent elements are adjacent in memory. Some tenor operation may change the index of element, such as transpose, expand, at such situation, only the index of data is changed, the data in memory is not changed, thus<br>
adjacent elements are not adjacent in memory.</p>
Cross product for more than 3 dimensions

ttoosi: <p>Hi,<br>
I am trying to calculate the plane passing through 2 vectors which are both of 200 dimensions. I know if these vectors were 3-D, I could use <strong>cross product</strong>. I wonder if there are any functions for cross product generalization to more than 3d in pytorch?</p>
<p>Thanks,<br>
T</p>

dawenzi098: <p>Hi, I got same question. Have you found any solution yet?</p>
VGG Multi-Scale Training

daniel0: <p>The <a href="https://arxiv.org/abs/1409.1556" rel="nofollow noopener">VGG paper</a> describes “multi-scale training”, a procedure whereby input images are scaled to various sizes and cropped to the expected size for training the model.</p>
<p>Were the VGG models in <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">torchvision.models</a> trained with the same multi-scale procedure?</p>
<p>Thanks.</p>

solsol: <p>This is also my question</p>
Parallel execution of computation graph

vdantu: <p>Sorry for some basic questions, but interpreter in JIT seems to run all the operators in a sequence. Can it be assumed that JIT based execution of computation-graphs are sequential unless we use “torch.script._fork” and “future.wait()” primitives? Or am I missing something basic <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Michael_Suo: <p>Yes, that is correct</p>

vdantu: <p>Thanks for the response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Distributed training slow

as754770178: <p>I training my model on one GPU(v100), the speed as below:</p>
<pre><code class="lang-auto">2019-09-17 01:46:48,876 - INFO - [ 1022/10000]	lr: 0.000100	Time  1.773 ( 1.744)	Data  0.001 ( 0.001)	Loss 6341.771 (6945.944)
2019-09-17 01:46:50,593 - INFO - [ 1023/10000]	lr: 0.000100	Time  1.607 ( 1.722)	Data  0.001 ( 0.001)	Loss 7225.229 (6958.357)
2019-09-17 01:46:52,323 - INFO - [ 1024/10000]	lr: 0.000100	Time  1.717 ( 1.732)	Data  0.001 ( 0.001)	Loss 7218.038 (6929.233)

</code></pre>
<p>The format of time info, such as <code>Time 1.717 ( 1.732)</code>, that <code>1.717</code> is current batch, <code>1.732</code> is the one hundred batch recently cost time.</p>
<p>when I use 8GPU on one node, use <code>torch.nn.parallel.DistributedDataParallel</code>, <code>torch.nn.SyncBatchNorm.convert_sync_batchnorm</code> and <code>mp.spawn</code>, the speed as below:</p>
<pre><code class="lang-auto">019-09-16 06:06:40,619 - INFO - [   9/5000]    lr: 0.000036    Time  2.822 ( 4.896)    Data  0.001 ( 1.428)    Loss 307113.969 (331260.794)
2019-09-16 06:06:43,485 - INFO - [  10/5000]    lr: 0.000037    Time  3.419 ( 4.749)    Data  0.001 ( 0.001)    Loss 303037.688 (325792.062)
2019-09-16 06:06:46,120 - INFO - [  11/5000]    lr: 0.000037    Time  2.866 ( 2.943)    Data  0.001 ( 0.001)    Loss 296579.000 (320417.425)
2019-09-16 06:06:48,925 - INFO - [  12/5000]    lr: 0.000037    Time  2.634 ( 2.879)    Data  0.001 ( 0.001)    Loss 292080.625 (315081.881)
2019-09-16 06:06:51,671 - INFO - [  13/5000]    lr: 0.000038    Time  2.806 ( 2.847)    Data  0.001 ( 0.001)    Loss 286678.000 (309843.294)
</code></pre>
<p>The speedup ratio of 8GPU is about 0.6.</p>
How to Install pytorch 1.1.0 with cuda version 10.1

Ahmad_Khan: <p>How to install pytorch v1.1.0 with pip with the following configurations,<br>
nvidia -drivers version 418.87<br>
nvidia rtx 2080 ti gpu<br>
cuda version 10.1</p>

ptrblck: <p>CUDA10.1 isn’t included in the binaries yet, so you would need to build from source as explained <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">here</a>.</p>
PyTorch logo and documentation template usage in another library

szymonmaszke: <p>Hello,</p>
<p>I’m developing third party software with a goal to release it as FOSS (MIT license) on Github, strictly connected with PyTorch.</p>
<p>Is it okay if I use your documentation template provided <a href="https://github.com/pytorch/pytorch_sphinx_theme" rel="nofollow noopener">here</a>? I have changed the footer to <code>Built with Sphinx using PyTorch's theme provided originally by Read the Docs.</code> and removed any copyrights, stripped down some parts I will not use and changed the header a little bit, other than that it is almost the same.</p>
<p>Secondly, is it okay to use PyTorch’s new logo as part of this library/libraries? I would assume it is after asking permission (based on <a href="https://discuss.pytorch.org/t/pytorch-logo-copyright/12524">this thread</a>), but it’s got quite a few months, anything changed?</p>
<p>If so, do you want me to include any licenses or notices if you allow me to use any/all of the above?</p>

ptrblck: <p>Pinging <a class="mention" href="/u/smth">@smth</a> regarding copyright/license inquiries.</p>

smth: <p>yes you can use the documentation template.</p>
<p>Yes you can use the pytorch logo to represent pytorch as the product. As long as it isn’t misrepresented, we are fine.</p>

szymonmaszke: <p><a class="mention" href="/u/smth">@smth</a> Came just to ping you with the results <a href="https://github.com/szymonmaszke/torchdata" rel="nofollow noopener">torchdata</a> and <a href="https://szymonmaszke.github.io/torchdata/" rel="nofollow noopener">direct link to documentation itself</a>. If anything is wrong, hit me up, thanks for permission.</p>
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/

erikxiong: <p>I have two GPUs installed, with the same script, if I select to run on GTX2080ti, the error occurs, and if I run on GTX1080ti, it works right</p>
<pre><code class="lang-auto">~/Molecule_Optimizer.py in forward(self, atom_list, bond_list, atom_degree_list, bond_degree_list, atom_mask)
     33         atom_mask = atom_mask.unsqueeze(2)
     34         batch_size,mol_length,num_atom_feat = atom_list.size()
---&gt; 35         atom_feature = self.dropout(self.atom_fc(atom_list))
     36 
     37         bond_neighbor = [bond_list[i][bond_degree_list[i]] for i in range(batch_size)]

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--&gt; 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     53 
     54     def forward(self, input):
---&gt; 55         return F.linear(input, self.weight, self.bias)
     56 
     57     def extra_repr(self):

~/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1024         return torch.addmm(bias, input, weight.t())
   1025 
-&gt; 1026     output = input.matmul(weight.t())
   1027     if bias is not None:
   1028         output += bias

RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1533672544752/work/aten/src/THC/THCBlas.cu:249
</code></pre>

xujin: <p>I also encountered the same problem. How did you solve it?</p>

fortitude94deng: <p>I also occurred this question ,finally i find that it is fc</p>

AldrichZeng: <p>Excuse me, could you please tell me how to tackle this problem? I found that it will occur when I carry out the “loss.backward()”.</p>

SamB: <p>Is this resolved? It may be related to an issue I am having and cannot resolve. See <a href="https://discuss.pytorch.org/t/cublas-runtime-error-with-cuda-10-and-torch-1-2-0/56087">here</a></p>
Can one gradient update overpower another?

jchaykow: <p>Say I have a training situation where I have two loss functions defined and want to update my gradients by backward pass over both. If I simply specify:</p>
<pre><code class="lang-auto">err1 = criterion1(...)
err2 = criterion2(...)
err = err1 + err2
err.backward()
optimizer.step()
</code></pre>
<p>Is it possible that one gradient update will overpower the other so that, in effect, only one loss really gets optimized since the other gradient update is so much smaller?</p>
<p>If so, can I normalize the gradients of the two before combining them and make sure they have similar impact on my update?</p>

vabh: <p>The short answer is yes.</p>
<p>One way to balance the contribution of each loss is to weight them. In this way you can control for which loss you want to prioritise. Something similar happens when you use <code>weight_decay</code> in your optimiser, which essentially scales the loss by the given value.</p>

jchaykow: <p>Just a weighted sum, like:</p>
<p><code>k = 0.2</code><br>
<code>err = k * err1 + (1 - k) * err2</code>?</p>
<p>How do I determine a good weight_decay value? Is this based on the norm of the gradient updates of one compared to that of the other?</p>
<p>Actually, I’m realizing the <code>weight_decay</code> might not work because it will decay the weight for both losses, no?</p>

DerekGloudemans: <p>Yeah so weight_decay  is used to add a regularization penalty to model weights over time, to keep model weights small on average and prevent overfitting. In answer to your second question, yes, weight decay will generally tend to decay the weights for both losses, unless it was implemented in a custom manner to apply only to one loss. The importance here is that weight_decay is used to adjust values’ weightings <strong>over time</strong> during training.<br>
(see <a href="https://discuss.pytorch.org/t/how-does-sgd-weight-decay-work/33105" class="inline-onebox">How does SGD weight_decay work?</a>)</p>
<p>What you are likely looking for is instead a weighting function that combines two scalar values, probably the same way at all points during training.</p>
<blockquote>
<p>err = k * err1 + (1-k)* err2</p>
</blockquote>
<p>One way to do this is a simple weighted sum, as shown above. Practically, this weighting value k must be selected by trying a number of candidate values.There are other ways to combine these values as well, such as multiplication:</p>
<blockquote>
<p>err =  err1*err2</p>
</blockquote>
<p>or geometric mean (this is a commonly used technique for combining the error metrics of precision and recall into a single metric called F1 score).</p>
<blockquote>
<p>err =  (2* err1 *err2) / (err1 + err2)</p>
</blockquote>
<p>The short of it is, there’s not an easy way to see how different weightings of the different loss components will translate into performance gains, since the relationship is a function of your model. Try a range of values or functions to see what yields satisfactory results.</p>
Why doesn't script module serialization support returning multiple outputs?

vdantu: <p>Hi All,<br>
I am curious to understand the reasoning behind <a href="https://github.com/pytorch/pytorch/blob/v1.2.0/torch/csrc/jit/passes/python_print.cpp#L788" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/v1.2.0/torch/csrc/jit/passes/python_print.cpp#L788</a>? Why would jit module export not support multiple values?  I also see that the python-c++ interface is only returning at the last element in the stack vector (stack.back()) when we invoke a scriptmodule from python, but why this limitation?  Why not return all elements in the stack vector? I am surely missing more technical details and would love to understand this further.</p>
<p>Thanks,</p>

driazati: <p>We follow the same conventions as Python here, if a function has multiple return values they all get wrapped up into a tuple, so it’s actually only 1 return value.</p>
<p>You can have as many returns as you want, they will just be wrapped into a tuple that you will have to unpack.</p>

vdantu: <p>Thanks. I think I understand now. I will dig into the code to look at where this creation of tuples happens on the C++ side <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

driazati: <p>We have an open issue to document it better and make it a nicer experience: <a href="https://github.com/pytorch/pytorch/issues/17165" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17165</a></p>
SyncBatchNorm gives lower accuracy than BatchNorm

Beinan_Wang: <p>I ran 3 experiments:<br>
(1) BatchNorm2d, num_gpus = 1, batch_size = 2, learning_rate = 2e-4 -&gt; accuracy = a1<br>
(2) SyncBatchNorm2d, num_gpus = 2, batch_size = 1, learning_rate = 2e-4 -&gt; accuracy = a2<br>
(3) SyncBatchNorm2d, num_gpus = 2, batch_size = 1, learning_rate = 1e-4 -&gt; accuracy = a3</p>
<p>All other variables are the same for the 3 experiments. I expected to see either a1 = a2 or a1 = a3. But both a2 and a3 are lower than a1. Also, when using SyncBatchNorm2d, the network converges slower. Any idea what is happening?</p>
Runtime Problem with .cuda()

AikeAhrens: <p>Since I started to use the pytorch dataloader, I´ve got runtime problems with .cuda(). My dataset consists of 250000 .npy-files each containing a numpy array with the shape 33x27. I’m using the dataloader the following way:</p>
<pre><code class="lang-auto"># list containing all file paths
train_file_paths = getPaths(self.dir_training)
            
trainDataSet = IterDataset(feature_path=train_file_paths)
            
train_loader = utils.DataLoader(dataset=trainDataSet,batch_size=32,shuffle=False,num_workers=16,pin_memory=True)
</code></pre>
<p>My training loop calls each time a new batch (batch-size: 32) and stores it to the GPU via .cuda(). The model is stored to the GPU at the beginning of the script.</p>
<pre><code class="lang-auto">for i,(feature,labels) in enumerate(train_loader):

    feature = Variable(feature.cuda(), requires_grad=True)
    labels = Variable(labels.cuda(), requires_grad=True)

    outputs = model(feature.float())
        
    loss = criterion(outputs,labels.long())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>My Dataclass looks like the following (the first column of the ndarray is the label):</p>
<pre><code class="lang-auto">class IterDataset(Dataset):

    def __init__(self, feature_path):
        self.feature_path = feature_path

    def __len__(self):
        return len(self.feature_path)

    def __getitem__(self, index):

        feature = np.load(self.feature_path[index])

        X = feature[:,1:] 

        y = feature[0,0]   
        
        # checking for NAN
        if np.isnan(X).any():
            print('NAN'+ self.feature_path[index])

        return X, y
</code></pre>
<p>The cProfile for the first 1000 batches:</p>
<pre><code class="lang-auto"> Ordered by: internal time

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2023   13.745    0.007   13.745    0.007 {method 'cuda' of 'torch._C._TensorBase' objects}
1000    1.931    0.002    1.931    0.002 {method 'run_backward' of 'torch._C._EngineBase' objects}
</code></pre>
<p>Hardware / Software I’m using:</p>
<ul>
<li>Cuda Version: 10.1</li>
<li>GPU: 2x GeForce GTX 1080</li>
</ul>
<p>So if anybody has an idea why .cuda() takes so much time, I would appreciate it.</p>

ptrblck: <p>Since CUDA operations are asynchronous, the host to device copy via <code>.cuda()</code> could create a synchronization point and thus accumulate the timing from the actual forward and backward pass.<br>
If you would like to profile the code manually, you could add manual synchronization points via <code>torch.cuda.synchronize()</code>.</p>
Custom Relu function slower than normal?

pytorcher: <p>I am trying to make a Custom Relu function that just doesn’t apply the relu to the gradient.  The code is as follows:</p>
<blockquote>
<p>class reluForward(torch.autograd.Function):<br>
def forward(self, inp):<br>
<span class="hashtag-raw">#option1:</span><br>
<span class="hashtag-raw">#return</span> = inp * (inp&gt;0).float()<br>
<span class="hashtag-raw">#option2:</span><br>
<span class="hashtag-raw">#return</span> F.relu(inp).data<br>
<span class="hashtag-raw">#option3:</span><br>
return F.relu(inp)<br>
def backward(self, grad_out):<br>
return grad_out</p>
</blockquote>
<p>Option 1 and 2 both perform the correct function but are significantly slower than the built in nn.relu function (actually starting ok and then getting slower and slower as it runs) while option 3 tells me “data must be a Tensor.”  Looking for any ideas for the slowdown or a different way to do this!</p>
<p>Edit: I have also now tried with return inp.clamp(min=0) which also works with the slowdown.</p>

pytorcher: <p>Got it!  Don’t know why this would cause a problem, but I was declaring self.reluForwarder = reluForward() in the Module’s init.  Moving it to just be reluForwarder = reluForward() in the forward function seems to make it work at the same speed as regular relu.</p>

albanD: <p>To avoid this mistake, you should keep in mind that a <code>Function</code> is not an <code>nn.Module</code> and should be used only once.<br>
Also, for better performance, you should use the new style functions as follow:</p>
<pre><code class="lang-auto">class reluForward(torch.autograd.Function):
  @staticmethod
  def forward(self, inp):
    #option1:
    #return = inp * (inp&gt;0).float()
    #option2:
    #return F.relu(inp).data
    #option3:
    return F.relu(inp)

  @staticmethod
  def backward(self, grad_out):
    return grad_out

# To use it:
inp = Variable(torch.rand(10, 10))
out = reluForward.apply(inp) # Use the class here, not an instance of it !
</code></pre>
Data Loader giving segmentation error with more than 0 workers

ammalikwaterloo: <p>I am running NN on the Summit. I am using ImageNet. I am having problem with the data loader</p>
<p>DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,<br>
batch_sampler=None, num_workers=0, collate_fn=None,<br>
pin_memory=False, drop_last=False, timeout=0,<br>
worker_init_fn=None)</p>
<p>It works well with 0 workers. However, when I use more than zero workers, it gives me a segmentation error. I googled this problem. It seems that this works with many people but I see many posts which complain the same error which I am getting.</p>
ImportError: cannot import name 'Dataloader'

ammalikwaterloo: <p>I am using PyTorch 1.0. Can anybody comments on this problem.</p>
<p>I am just testing</p>
<p>from torch.utils.data import Dataloader<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
ImportError: cannot import name ‘Dataloader’</p>
<p>Thanks</p>

ammalikwaterloo: <p>I was following the instructions on</p>
<p><a href="https://pytorch.org/tutorials/beginner/ptcheat.html#" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/ptcheat.html#</a></p>
<p>There is a typo. “Dataloader” should be “DataLoader”.</p>

ptrblck: <p>Good catch! Would you mind creating an <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">issue</a> so that we can fix this typo? If you don’t want to fix it yourself, could you please tag me?</p>

ammalikwaterloo: <aside class="quote no-group" data-username="ammalikwaterloo" data-post="2" data-topic="40832">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/74df32/48.png" class="avatar"> ammalikwaterloo:</div>
<blockquote>
<p>I was following the instructions on</p>
<p><a href="https://pytorch.org/tutorials/beginner/ptcheat.html#" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/ptcheat.html# </a></p>
<p>There is a typo. “Dataloader” should be “DataLoader”.</p>
</blockquote>
</aside>
<p>Sorry for the delay… Just submitted. <a href="https://github.com/pytorch/pytorch/issues/26278" class="inline-onebox" rel="noopener nofollow ugc">ImportError: cannot import name ‘Dataloader’ · Issue #26278 · pytorch/pytorch · GitHub</a></p>
PackedSequence for non rnn model

Eyal_Orbach: <p>I have a custom transformer-like model which can receive a sequence of various length.<br>
I’d like to parralelize across gpu’s. so i wish to pack multiple sequences of various length in one tensor so i can use DataParallel.<br>
I am looking for something like PackedSequence but that i can conveniently unpack at the model, and get the sequences after the padding was removed.<br>
Any existing implementations or tips how to do this?</p>
Multiple Dataloader and source

thomas_liu: <p>Hi<br>
I have multiple data loader currently and I want to read from them. I concat the data loaders but I would like to know the data comes from which original data loader so I can apply different transform to them.<br>
Is there any way directly return the label?</p>
<p>Thanks</p>

Oli: <p>Hi Thomas,</p>
<p>Do you have multiple dataloaders or multiple datasets? Or both?<br>
If its datasets, you can return whatever you want from the datasets get_item() function. One idea would be to return a dataset descriptor / ID.</p>
<p>If its dataloaders, can you show me how you concat your loaders :)?</p>
Warning: scipy/ndimage/interpolation

hxtisme: <p>scipy/ndimage/interpolation.py:583: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.<br>
“the returned array has changed.”, UserWarning)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fd39134e1444f23e3c64c1e245a62d64608aefa5.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd39134e1444f23e3c64c1e245a62d64608aefa5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd39134e1444f23e3c64c1e245a62d64608aefa5_2_690x59.png" alt="image" data-base62-sha1="A875b4yqWYbgo41cENJP9hD78b3" width="690" height="59" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd39134e1444f23e3c64c1e245a62d64608aefa5_2_690x59.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd39134e1444f23e3c64c1e245a62d64608aefa5_2_1035x88.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fd39134e1444f23e3c64c1e245a62d64608aefa5_2_1380x118.png 2x" data-dominant-color="111111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1657×143 20.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
why this warning occur ? and how can I address it ?</p>

ptrblck: <p>This warning doesn’t seem to be related to PyTorch, at least I’m not aware of any usage of <code>scipy</code> in PyTorch.<br>
Somewhere in your core you are apparently using <code>zoom</code>, which throws this warning.</p>
Autograd optimize matrix neural network not working (please help)

apr: <p>I cannot get the gradient (SGD) to work in the below example. Similar posts and autograd documentation have not helped. In the below L.grad is always none. Your help is appreciated.</p>
<p>import numpy as np<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torch.optim as optim<br>
from torch.autograd import Variable</p>
<p>def NN( w=np.random.rand(135), x=np.random.rand(4) ):</p>
<pre><code>X = torch.Tensor(x.reshape(4,1))
M1 = torch.Tensor(np.array(w[0:36]).reshape(9,4))
b1 = torch.Tensor(np.array(w[36:45]).reshape(9,1))
M2 = torch.Tensor(np.array(w[45:126]).reshape(9,9))
b2 = torch.Tensor(np.array(w[126:135]).reshape(9,1))

nIn = 4
nH1 = 9
nH2 = 9	

W1 = Variable(M1, requires_grad=True)
B1 = Variable(b1, requires_grad=True)
Y1 = torch.mm(W1, X) + B1
Y1 = F.relu(Y1)

W2 = Variable(M2, requires_grad=True)
B2 = Variable(b2, requires_grad=True)
Y2 = torch.mm(W2, Y1) + B2
Y2 = F.relu(Y2)

print(Y2.shape)
print(Y2)

return [W1,B1,W2,B2], Y2
</code></pre>
<p>def loss(Y, T):</p>
<pre><code>with torch.enable_grad():
	diff = Y.reshape(-1)-T.reshape(-1)
	print('loss type', type(diff))
	return diff.dot(diff)
</code></pre>
<p>pars, Y = NN()<br>
T = torch.randn(9,1)<br>
optimizer = optim.SGD(pars, lr=0.1, momentum=0.9)</p>
<p>for j in range(1):<br>
optimizer.zero_grad()<br>
L = loss(Y, T)<br>
L.backward(retain_graph=True)<br>
print(‘grad’, L.grad)<br>
optimizer.step()</p>

ptrblck: <p>The gradient in the loss won’t be retained by default.<br>
By calling <code>loss.backward()</code> you are passing a gradient of 1 by default, since <code>dL/dL = 1</code>.<br>
To print the gradient in <code>L</code>, you can use the following code:</p>
<pre><code class="lang-python">L.retain_grad()
L.backward(retain_graph=True)
print('grad', L.grad)
</code></pre>

apr: <p>Thank you, that works. But I was looking for the derivative with respect to the weights dY/dw_i. I have since found the answer to this from another post I believe as follows:<br>
for f in pars[0]:<br>
print(‘data is’)<br>
print(f.data)<br>
print(‘grad is’)<br>
print(f.grad)</p>
<p>Please correct me if this is wrong…</p>

ptrblck: <p>If <code>f</code> is a parameter / leaf variable, the code looks fine.</p>
CUDA out of memory when optimizer.step()

shirui-japina: <p>I try to extract image features by InceptionA (part of GoogLeNet). When there is no <code>optimizer.step()</code>, it works even with the batch size 128. But when there is <code>optimizer.step()</code>, it will Error: <code>CUDA out of memory</code>.<br>
Here is the code:</p>
<blockquote>
<p>model = InceptionA(pool_features=2)<br>
model.to(device)<br>
optimizer = optim.Adam(model.parameters())<br>
criterion = nn.BCELoss(reduction=‘mean’)<br>
for epoch in range(100):<br>
&nbsp;&nbsp;&nbsp;&nbsp;for i, (batch_input, label) in enumerate(data_loader):<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;optimizer.zero_grad()<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;output = model(batch_input)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;loss = criterion(output, label)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;loss.backward()<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;optimizer.step() <em># Error here</em></p>
</blockquote>
<p>How can I fix this error?</p>

shirui-japina: <p>I have already solved the problem. The code is here:<br>
<code>1) optimizer = optim.SGD(model.parameters(), lr=0.0001)</code><br>
and<br>
<code>2) loss = criterion(torch.sigmoid(output), label)</code><br>
the reason of 2) is</p>
<blockquote>
<p>BCELoss accepts only inputs that have all elements in range [0; 1]</p>
</blockquote>
<p>But I don’t know why 1) have to use optim.SGD() and optim.Adam() can’t</p>

ptrblck: <p><code>Adam</code> uses internal running estimates and thus uses more memory than e.g. SGD.<br>
If your GPU is almost full and you call <code>step</code> on your <code>Adam</code> optimizer, these running estimates will be created and might thus yield an out of memory error.</p>

shirui-japina: <p>Is there any solution or PyTorch function to solve the problem? Even work at a slow speed.<br>
Or the only way to solve it is to use a better GPU or multiple GPUs, is that right?</p>

ptrblck: <p>The easiest way would be to lower your batch size. If that’s not possible (e.g. if your batch size is already 1), you could have a look at <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="nofollow noopener">torch.utils.checkpoint</a> to trade compute for memory.</p>

shirui-japina: <p>I got it. Thanks for your advising <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Accuracy not increasing with BERT Large model

Kalyan_Katikapalli: <p>I used both  <code>BERT_base_cased</code>  and  <code>BERT_large_Cased</code>  model for multi class text classification. With BERT_base_cased, I got satisfactory results. When I tried with BERT_large_cased model, the accuracy is same for all the epochs<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aa0327ade5d1ae8ff6d384388cd454fe962e842a.png" data-download-href="https://discuss.pytorch.org/uploads/default/aa0327ade5d1ae8ff6d384388cd454fe962e842a" title="bert-error1"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/aa0327ade5d1ae8ff6d384388cd454fe962e842a.png" alt="bert-error1" data-base62-sha1="ofZVD6eVa3VKKuYnLr8SsLj2DdE" width="690" height="473" data-dominant-color="454545"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bert-error1</span><span class="informations">926×636 34.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/977c5d42910db85f91e4108514ca2ed6ced722a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/977c5d42910db85f91e4108514ca2ed6ced722a4" title="bert-error2"><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/977c5d42910db85f91e4108514ca2ed6ced722a4.png" alt="bert-error2" data-base62-sha1="lC6xVSxGY65dVj7pEYx9VdFqKJC" width="690" height="474" data-dominant-color="454545"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bert-error2</span><span class="informations">925×636 36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With  <code>BERT_base_cased</code> , there is no such problem. But with  <code>BERT_large_cased</code> , why accuracy is same in all the epochs? Any help is really appreciated…</p>
Convert int * into tensor

Dan_Sagher: <p>Hi,<br>
I’m new in Pytorch and I would like to know if there’s a good way to convert an int pointer to a new tensor using Libtorch in c++.<br>
Thank you very much!</p>

LeviViana: <p>Yes there is a good way: use <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1aff6f8e6185457b2b67a1a9f292effe6b.html" rel="nofollow noopener">torch::from_blob</a>.</p>
Einsum problem in Pytorch 0.4

Zihang_Dai: <p>I encounter a problem when using the new <code>torch.einsum</code> in Pytorch 0.4.</p>
<p>What I want to do is simply using <code>torch.einsum</code> to implement dot-product attention as often used in seq2seq models. The following is a snippet I create.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

bsz, clen, qlen = 32, 20, 20
d_hid = 64

context = torch.rand(clen, bsz, d_hid, requires_grad=True)
query = torch.rand(qlen, bsz, d_hid, requires_grad=True)
key = val = context

att_score = torch.einsum('cbd,qbd-&gt;cqb', (key, query)) 
att_prob = torch.nn.functional.softmax(att_score, dim=0)
att_vec = torch.einsum('cbd,cqb-&gt;qbd', (val, att_prob))

att_vec.mean().backward()
</code></pre>
<p>However, the <code>backward</code> call raises the following runtime error</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.</p>
</blockquote>
<p>I’m not quite sure what’s going on here. Any idea?</p>

richard: <p>That warning is usually for when a tensor required for gradient computation is modified in-place. However, I don’t see any in-place modifications here, unless einsum does some internally…</p>

Ran: <p>Have you solved it?<br>
If I replace torch.einsum with att_vec = (val.unsqueeze(1)*att_prob.unsqueeze(-1)).sum(dim=0), will obtain the same error.</p>

xinwei_he: <p>what about cloning the input variable, is it a correct way to solve this problem?</p>

xinwei_he: <p>UPDATE: latest version of PyTorch has fixed this bug.  For 0.4, I have seen someone tries to fix it by clone the variables</p>
What's prim::Constant() mean in onnx model transformed from pytorch?

XiaXuehai: <p>I transformed the pytorch model to onnx :</p>
<pre><code class="lang-auto">  %79 : Tensor = onnx::Unsqueeze[axes=[0]](%75)
  %80 : Tensor = onnx::Unsqueeze[axes=[0]](%77)
  %81 : Tensor = onnx::Concat[axis=0](%78, %79, %80)
  %82 : Float(2, 1, 256) = onnx::ConstantOfShape[value={0}](%81), scope: CRNN/Sequential[rnn]/BidirectionalLSTM[0]/LSTM[rnn]
  %83 : Tensor? = prim::Constant(), scope: CRNN/Sequential[rnn]/BidirectionalLSTM[0]/LSTM[rnn]
</code></pre>
<p>Does  <code>%83:prim::Constant()</code>  belong to onnx? I did not find that op in <a href="https://github.com/onnx/onnx/blob/master/docs/Operators.md#Constant" rel="nofollow noopener">onnx</a>.</p>

tom: <p><code>Prim::Constant()</code> gives you a “typed None” here, so in PyTorch terms, this is a <code>Optional[Tensor]</code> (aka <code>Tensor?</code>) that is <code>None</code> / not present.<br>
One little known difference between Python and TorchScript – at least I didn’t know it until I submitted a PR that needed to be corrected to do the right thing – is that while <code>None</code> has its own type in Python, it is always typed (of some optional type) in TorchScript.</p>
<p>(For those pedantic about internals, there is a <code>NoneType</code> defined in <code>jit_types.h</code>, but it’s not for use “inside” TorchScript and a None passed in to a scripted function is converted to the corresponding typed None.)</p>
<p>Best regards</p>
<p>Thomas</p>

XiaXuehai: <p>Thanks, but I can’t convert the onnx model to Caffe2., how to fix it?</p>

lchia: <p>Hi, I’m also curious about prim:Constant() in onnx, how did you solve this problem?</p>

lchia: <p>Hi，did you solve this problem？</p>
How to apply a right-to-left seq2seq using Paytorch

Aiman_Mutasem-bellh: <p>Dear friends,</p>
<p>How I can apply a right-to-left seq2seq model and a left-to-right seq2seq model using Pytorch for GEC Task.</p>

DoubtWang: <p>u can reverse the target sequence .</p>

Aiman_Mutasem-bellh: <p>Please example bro <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

DoubtWang: <p>X = (x1, x2, x3, …, xn),  Y=(y1, y2, y3,…ym), ^Y=(ym, …, y3, y2, y1)<br>
seq2seq is left-to-right when u input the (X, Y) ;<br>
seq2seq is right-to-left when u input the (X, ^Y)  ;</p>

Aiman_Mutasem-bellh: <p>Thank you bro, I will try this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Control loading of pre-trained weights into a model with similar architecture

Pankaj_Bhambhani: <p>Hi,</p>
<p>I want to know if PyTorch allows us to customise/control the loading of weights from a pre-trained model into a new model?</p>
<p>I am implementing a Bayesian Resnet CNN model (similar to the one mentioned here - <a href="https://github.com/kumar-shridhar/PyTorch-BayesianCNN" rel="nofollow noopener">https://github.com/kumar-shridhar/PyTorch-BayesianCNN</a>) in PyTorch. In this model, for each weight, instead of having one parameter (the value of the weight), we have two parameters (the mean and standard deviations of the normal distribution defining the weight). Since the rest of the architecture is basically the same, I would like to be able to pre-populate the lower layers using weights from pre-trained frequentist model (setting mean equal to the weights and standard deviation to a small non-zero value).</p>
<p>I want to know if this is possible using PyTorch? If yes, could you point me to an example or a resource that helps me implement it? Thanks.</p>
Getting TypeError: img should be PIL Image. Got <class 'NoneType'> when I use a custom transformation

talhak: <p>I have been using my custom transformation namely <code>FaceDetection</code> which I added below. Now I am trying to apply it to another dataset which contains face images within the same format of the former (<code>.png</code>). What am I missing? When I do not use this transformation, I do not get any errors.</p>
<p>Your help is appreciated a lot.</p>
<p>Here is my custom transformation class, <code>FaceDetection</code>, based on <code>OpenCV</code> to detect the location of faces in the images:</p>
<pre><code class="lang-auto">class FaceDetection(object):

    def __init__(self):
        pass

    def __call__(self, frame):
        faceCascade = cv2.CascadeClassifier(cascPath)

        if frame is not None:
            gray = cv2.cvtColor(np.array(frame), cv2.COLOR_BGR2GRAY)

            faces = faceCascade.detectMultiScale(
                gray,
                scaleFactor=1.1,
                minNeighbors=5,
                minSize=(1, 1)
            )

            for (x, y, w, h) in faces:
                image_as_arr = np.array(frame)[y:y + w, x:x + h]
                return Image.fromarray(image_as_arr.astype('uint8'), 'RGB')
</code></pre>
<p>And here is the composition of my transformations:</p>
<pre><code class="lang-auto">data_transforms = transforms.Compose([
    FaceDetection(),
    transforms.Resize(48),
    transforms.Grayscale(num_output_channels=1),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5],
                         std=[0.5])
])
</code></pre>
<p>Finally, I apply it when I load the training dataset:<br>
<code>face_train_dataset = datasets.ImageFolder(root=DEST_PATH_TRAIN, transform=data_transforms)</code></p>

ptrblck: <p>It seems your custom transformation didn’t find any faces and thus didn’t return a valid <code>PIL.Image</code>.</p>

talhak: <p>I see but it works when I skip my custom transformation. So, the destination (<code>DEST_PATH_TRAIN</code>) should be a right path as the transformation does nothing related to the path of the images.</p>

ptrblck: <p>I think the face cascade doesn’t find any faces and this loop:</p>
<pre><code class="lang-python">for (x, y, w, h) in faces:
    image_as_arr = np.array(frame)[y:y + w, x:x + h]
    return Image.fromarray(image_as_arr.astype('uint8'), 'RGB')
</code></pre>
<p>will be empty.<br>
Since you don’t have any other return statement, your transformation will return <code>None</code> by default.<br>
If you add an assert before the loop and check <code>faces</code> for a valid entry, you should see an exception.</p>

talhak: <p>Just added a control in the case of not founding any faces in the image. Now works as expected. Thanks a lot for your guidance.</p>
PyTorch fp16 CUDA slow

JimFan: <p>I’m recently developing a new layer type with pytorch 1.1 CUDA extension. I’ve followed the official tutorial and used the macro  <code>AT_DISPATCH_FLOATING_TYPES_AND_HALF</code>  to generate support for fp16. Other than this, my code has no special treatment for fp16. I took care to cast all floating point constants in my code with  <code>static_cast&lt;T&gt;</code>  (where  <code>T</code>  is the template scalar type).</p>
<p>However, in all my benchmarks, fp16 is slower than fp32 for my kernel (&gt; 30%).<br>
FP16 should be faster in both GPU memory access and arithmetic, no?<br>
I’m quite new to CUDA programming and might have missed something simple. Can someone please point out any likely cause?</p>
<p>My test hardware is Titan Turing RTX 24G with CUDA10.</p>
Can using torch.clamp change the output of my model?

shresta: <p>Hi,<br>
Can torch.clamp() which I assume is some form of ReLU change the output of my NN classification?</p>
<p>I am running a celebrity face image on a pretrained VGG Face model. I set the model in eval mode and make a forward pass.</p>
<pre><code class="lang-auto">        image, label = image.to(device), label.to(device)  
        preds = F.softmax(vgg_model_instance(image.view(1, 3, 224, 224)), dim=1)
        values, indices = preds.max(-1)
</code></pre>
<p>I get a results. 87% accuracy.</p>
<p>Then i apply clamp on the input image.</p>
<pre><code class="lang-auto">        image, label = image.to(device), label.to(device) 
        **image = torch.clamp(image, 0, 1)**
        preds = F.softmax(vgg_model_instance(image.view(1, 3, 224, 224)), dim=1)
        values, indices = preds.max(-1)
</code></pre>
<p>The accuracy crashes like anything. Any idea what is going on.</p>
<p>With all other setting constant I have tried ranges (-1,1) and (0,1) for clamp. I also tried running the model through softmax, max and logsoftmax.</p>
<p>The reason I am trying this experiment for testing a FGSM attack. While I am not sure if clamping the tensor is necessary for the attack, all examples I follow seem to do it. Would really appreciate if one could explain why this is done too.</p>

ptrblck: <p>If you’ve normalized the input data, it’ll have a zero mean and unit variance.<br>
Clamping it to <code>[0, 1]</code> might delete a lot of important features, which could explain the bad accuracy.</p>
Slow distributed training

Boris_Lestsov: <p>My network is 1 Gbit ethernet and i am trying to use pytorch distributed training on two 8-gpu servers. Training procedure is simple classification objective with feed-forward network. I experience significant slowdown in comparison with single 8-gpu server training. Also “nload” tool shows full bandwidth usage even for small model (resnet18).</p>
<p>Is my network too slow for distributed training? If it is, what bandwidth (in Gbit/s) do I need to train heavy models like resnet101?</p>

Yaroslav_Bulatov: <p>You can figure it out from your gradient size (could just take size of your model checkpoint) + step time. For instance Resnet50 160ms per batch, 50MB checkpoint, therefore each worker needs to send and receive 50/.16 = 312 MB per second, means you need &gt;=2.5 Gbps</p>
<p>What matters here is the ratio of compute time to parameter size. If you double computation + double parameter size, the network requirement is unaffected. Conv nets have good ratio of compute/bandwidth, transformers will need more bandwidth because of matmuls</p>
Doubt about weight-sharing between two layers

mukulkhanna: <p>I am a beginner, I wanted to know how I could share weights between two conv layers in a model. I read some posts about this on the forum but coudn’t understand.</p>
<p>I found this example in the Pytorch tutorials -</p>
<p><a href="https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html</a></p>
<p>Here we can see that the <code>middle_linear</code> layer is being reused in the <code>forward()</code> method.</p>
<pre><code class="lang-auto">for _ in range(random.randint(0, 3)):
            h_relu = self.middle_linear(h_relu).clamp(min=0)
</code></pre>
<p>So does this mean that after backprop, the weights of all the copies of this layer (<code>middle_linear</code>) will have the same values.</p>
<p>If yes, then is it true that if i want two layers to share the weights, then I can just instantiate the layer once and no matter how many copies I make of that layer, their weights will be the same? For example</p>
<pre><code class="lang-auto">def __init__(self):
    self.conv_start = nn.Conv2d(3, 64, kernel_size=3, padding=1)
    self.conv_1 = nn.Conv2d(64,64,kernel_size=3, padding=1)

def forward(self, input):
    out = self.conv_start(input)
    out = self.conv_1(out)
    out = self.conv_1(out)
    out = self.conv_1(out)

return out
</code></pre>
<p>After backprop, are the weights for the 3 copies of <code>conv_1</code> layer differently updated? Or is it just one layer being used multiple times in the dynamic graph, and therefore sharing the weights?</p>
<p>Thank you.</p>

dejanbatanjac: <p>I think this was just :</p>
<blockquote>
<p>To showcase the power of PyTorch dynamic graphs</p>
</blockquote>
<p>In practice, this has no sense to me. The weights should be updated by the optimizer and typically they are different in each step.</p>

mukulkhanna: <p><a class="mention" href="/u/dejanbatanjac">@dejanbatanjac</a></p>
<p>No no. I know that the weights will be different at each time step, I want to know if the three copies of <code>conv_1</code> share the weights.</p>
Subsample class ballencing

brightstar: <p>Hi,</p>
<p>Sorry if a similar topic already exists. I am trying sample the data in each batch during training so that each class represented in the batch has at least n numbers of each class when possible. So say I have 1000 classes if the batch size is 16 and I want n to be 4 I want the batch to only contain 4 different classes in each batch</p>
Multi-optimizers performance

falmasri: <p>Does having Multiple optimizers assigned each for different module in the main model could have different learning path than having one optimizer for the full model ? for example Adam</p>

DoubtWang: <p>it’s a interesting problem. and  it need some experiments.<br>
i think the learning path maybe different, because the optimizers compute the different formula when updating the parameter</p>

spanev: <p><a class="mention" href="/u/falmasri">@falmasri</a> <a class="mention" href="/u/doubtwang">@DoubtWang</a><br>
If you use Adam optimizer with indentical hyperparameters and calling the optimization step at the same time you will not see any numerical difference.</p>
<p>Adam optimizations step is done per parameter (moving averages are per parameter/independent).</p>
<p>For each <a href="https://github.com/pytorch/pytorch/blob/1c81d9006a292aa5b5ec904533d6d5c63bb0d0cd/torch/optim/adam.py#L61" rel="nofollow noopener">parameter</a> we have <a href="https://github.com/pytorch/pytorch/blob/1c81d9006a292aa5b5ec904533d6d5c63bb0d0cd/torch/optim/adam.py#L69" rel="nofollow noopener">one state</a>.</p>

DoubtWang: <p><a class="mention" href="/u/spanev">@spanev</a>  sorry, I misread it. my understand is that multiple optimizers in text refer to the different optimizers, e.g., SGD and Adam.</p>

spanev: <p>Sure, the question is a little bit ambiguous but this:</p>
<aside class="quote no-group" data-username="falmasri" data-post="1" data-topic="55380">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falmasri/48/11265_2.png" class="avatar"> falmasri:</div>
<blockquote>
<p>for example Adam</p>
</blockquote>
</aside>
<p>lead me to think that <a class="mention" href="/u/falmasri">@falmasri</a> was referring to same optimizer <em>instances</em>.</p>
<p>Having different <em>mathematical</em> optimizers remains a interesting problem! And it should be addressed with empirical observations, as you said. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

falmasri: <p>this is good point about adam parameters I didn’t pay attention.</p>
<p>I initiated 3 Adam optimizers and assigned each of them to different module and training was sequentially. this is that first module was run first and the updated using its assigned optimizers then the second and the third. It achieved a little bit enhancement in overfitting model.</p>