akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch std::shared_ptr<torch:nn:Module>has no member named 'value'?

username: bukenhatice
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/Keson96/ResNet_LibTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/17428448?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Keson96/ResNet_LibTorch" target="_blank" rel="nofollow noopener">Keson96/ResNet_LibTorch</a></h3>

<p>A small example of using new PyTorch C++ frontend to implement ResNet - Keson96/ResNet_LibTorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>for(auto m: this-&gt;modules()){<br>
if (m.value.name() == “torch::nn::Conv2dImpl”){</p>
<p>error: ‘class std::shared_ptrtorch::nn::Module’ has no member named ‘value’</p>
<p>how can i solve ? I would be happy if you help.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/bukenhatice">@bukenhatice</a><br>
This is a 2 years old example. Many code changed during this period, if you want this example working.<br>
Please try older version of libtorch, eg. 1.4 or even older.</p>

username: bukenhatice
cooked: <p>Thank you,</p>
<p>I will install and try torch 1.4.</p>
nn.Linear with 3 or more d input

username: Caruso
cooked: <p>Hi guys,</p>
<p>I was toying around with nn.Linear and 3d tensor inputs. When I entered the 3d tensor the <code>grad_fn</code> function is named <code>&lt;UnsafeViewBackward&gt;</code>. With a manually flatten 2d tensor, which I viewed back to a 3d tensor the <code>grad_fn</code> function is <code>&lt;ViewBackward&gt;</code>, like expected. Now I’m curious why it’s called ‘unsafe’ and if i rather should manually view the 3d tensor to 2d or enter it as 3d a tensor?</p>
<p>Code (not sure if nessecary):</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn 

l1 = nn.Linear(4, 5, bias=False)
l1.weight = nn.Parameter(torch.ones(l1.weight.shape))

b, s, e = 2, 3, 4
a = torch.arange(0, b*s*e, dtype=torch.float).view(b, s, e)
print(a.shape)

out1 = l1(a)
out2 = l1(a.view(-1, e)).view(b, s, -1)

print(out1, out2, sep="\n")
print(torch.equal(out1, out2))
</code></pre>

username: ptrblck
cooked: <p>The <code>UnsafeViewBackward</code> shouldn’t be problematic as explained <a href="https://discuss.pytorch.org/t/unsafeviewbackward-in-grad-fn/25992/3">here</a>.</p>
<p>However, note that the linear layer will behave differently for a 2D and 3D input.<br>
While a 2D input should be the “standard” approach, a 3D input would be treated as a sequence (in dim1) or inputs.</p>
Strange question, number of filters

username: SandPhoenix
cooked: <p>When I use conv2d(1,13,5). I know that I am having as output 13 filters of size 5x5. The question, is that I need it to the output channel value to stay as it is but can I change the number of the resulting filters to 5 somehow. I do not see a solution but I want to do that</p>

username: ptrblck
cooked: <p>Each filter will create an output channel. If you want to use 5 filters only, you could repeat these filters, but you would of course get redundant output information.</p>
<p>Alternatively you could also use a <code>1x1</code> convolution after this layer and output 5 channels, if that would fit your use case.</p>
<p>Could you explain a bit, what you are trying to achieve? Maybe I’m misunderstanding the question.</p>
Two categories, after using torchtext to generate a dictionary, label.vocab becomes 3。

username: xiaomo
cooked: <p>I am doing a text binary classification problem, the original label has only two values, but after using the build_vocab built in torchtext.data to generate a dictionary, len (label.vocab) is 3. In addition, it is found that everyone in the final fully connected layer of the neural network basically uses out_dim = len (label.vocab). Isn’t this a three-category?</p>
<pre><code class="lang-auto">def get_dataset(csv_data, text_field, label_field, test=False):
    # id数据对训练在训练过程中没用，使用None指定其对应的field
    fields = [('request_url',text_field), ('is_malicious',label_field)]     
    examples = []

    if test:
        # 如果为测试集，则不加载label
        for text in tqdm(csv_data['request_url']):
            examples.append(data.Example.fromlist([text, None], fields))
    else:
        for text, label in tqdm(zip(csv_data['request_url'], csv_data['is_malicious'])):
            examples.append(data.Example.fromlist([text, label], fields))
    return examples, fields

text = data.Field(sequential=True, lower=True, tokenize=get3gram)
label = data.Field(sequential=False)
text.tokenize = get3gram
examples, fields = get_dataset(all_data, text, label) #注意更改es_data
    # new_data = data.Dataset(examples, fields=[('request_url',text), ('is_malicious',label)])
train,val = data.Dataset(examples, fields).split()
    # train,val = new_data.split()

text.build_vocab(train, val)
label.build_vocab(train, val)

label_num = len(label.vocab)

#Fully connected layer in cnn
self.linear = nn.Linear(len(filter_sizes)*filter_num, label_num)
</code></pre>
<p>It is found by printing that len (label.vocab) = 3, I do n’t understand why, it becomes a three-category, can you manually modify out_dim (the output dimension of the fully connected layer) to 2, will there be no problem in doing so ?</p>
Convert C++/CUDA extension from pytorch 0.4.1 to pytorch 1.x.x

username: Truong_Khang
cooked: <p>Hello,<br>
I have a file C++/CUDA test.cu written in torch 0.4.1 like this:</p>
<pre><code class="lang-auto">#include&lt;TH.h&gt;
#include&lt;THC.h&gt;

extern THCState *state;

__global__ void do_something(float *x, float *y) {
    ....
}
extern "C" void warping(THCudaTensor *tensor_x, THCudaTensor *tensor_x) {
    int a = 16;
    int step_h = THCudaTensor_size(state, tensor_x, 3);
    int step_w = THCudaTensor_size(state, tensor_x, 4);
    const dim3 block = dim3(1, a, a);
    const dim3 grid = dim3(1, step_h, step_w);
    float *x = THCudaTensor_data(state, tensor_x);
    float *y = THCudaTensor_data(state, tensor_y);

    cudaStream_t stream = THCState_getCurrentStream(state);
    do_something&lt;&lt;&lt;grid, block, 0, stream&gt;&gt;&gt;(x, y);
    cudaStreamSynchronize(stream);

    THCudaCheck(cudaGetLastError());
}
</code></pre>
<p>Now I want to use a higher pytorch version 1.x.x, so I have to change this file. I also read this tutorial from the official documentation <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="nofollow noopener">here</a>. I still got stuck when converting the code above.<br>
Any suggestions would be appreciated.</p>
<p>Best,<br>
Khang Truong.</p>
DDP -Sync Batch Norm - Gradient Computation Modified?

username: soulslicer
cooked: <p>I am using DDP, and I am usiong BatchNorm in my network. If I do not set track_running_stats=False, then I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)
</code></pre>
<p>I tried to switch it to SyncBatchNorm, and I still get the same error.</p>

username: ptrblck
cooked: <p>Could you post your model definition (and if possible an executable code snippet) so that we could have a look, please?</p>

username: soulslicer
cooked: <p>Okay, I have created a self runnable code:</p>
<p><a href="https://pastebin.com/5QKKsfe6" class="onebox" target="_blank" rel="nofollow noopener">https://pastebin.com/5QKKsfe6</a></p>
<p><a href="https://pastebin.com/kD0fgPve" class="onebox" target="_blank" rel="nofollow noopener">https://pastebin.com/kD0fgPve</a></p>
<p>You need to download both the files to run it</p>

username: soulslicer
cooked: <p>Okay, I realized if I remove the 2nd call, <code>output_right = model(input_right)</code>, then it no longer gives this error. How can I make SyncBatchNorm work for two inputs to the same network that I want to constrain??</p>
<p>Works</p>
<pre><code class="lang-auto">output_left = model(input_left)
loss = torch.sum(output_left["output"][0] - 0)
</code></pre>
<p>Doesnt Work</p>
<pre><code class="lang-auto">output_left = model(input_left)
output_right = model(input_right)
loss = torch.sum(output_left["output"][0] - 0) + torch.sum(output_right["output"][0] - 0)
</code></pre>

username: soulslicer
cooked: <p>This means I cannot call the model twice if I use DDP? I have to rewrite my code so that both input_left and input_right are passed into the model for computation</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/issues/22095#issuecomment-505099500">this comment</a> it seems that broadcasting might be seen as an inplace operation, so you might avoid this error by using <code>broadcast_buffers=False</code>.</p>
Difference between torch.tensor and torch.Tensor

username: AreTor
cooked: <p>In PyTorch docs it is written that <code>torch.Tensor</code> is an alias for <code>torch.FloatTensor</code>. This is not the case when using <code>torch.tensor</code>. For example:</p>
<pre><code>&gt;&gt;&gt; torch.Tensor([1,2,3]).dtype
torch.float32
&gt;&gt;&gt; torch.tensor([1, 2, 3]).dtype
Out[32]: torch.int64
&gt;&gt;&gt; torch.Tensor([True, False]).dtype
torch.float32
&gt;&gt;&gt; torch.tensor([True, False]).dtype
torch.uint8
</code></pre>
<p>Isn’t this unnecessarily confusing?</p>

username: ptrblck
cooked: <p><code>torch.tensor</code> infers the <code>dtype</code> automatically, while <code>torch.Tensor</code> returns a <code>torch.FloatTensor</code>.<br>
I would recommend to stick to <code>torch.tensor</code>, which also has arguments like <code>dtype</code>, if you would like to change the type.</p>

username: AreTor
cooked: <p>Agree, but my point is about the need of defining an alias <code>torch.FloatTensor</code>, which happens to be very similar in name to <code>torch.tensor</code> but has a very different behaviour and can generate confusion. Is it really needed?</p>

username: lazhd
cooked: <p>I agree, i costs me some time to find the problem caused by that.</p>
<p>The same here: <a href="https://stackoverflow.com/questions/48482787/pytorch-memory-model-torch-from-numpy-vs-torch-tensor" rel="nofollow noopener">https://stackoverflow.com/questions/48482787/pytorch-memory-model-torch-from-numpy-vs-torch-tensor</a></p>
<pre><code class="lang-auto">
</code></pre>
<p>arr = np.arange(10, dtype=np.float32).reshape(5, 2)</p>
<p>t0  = torch.Tensor(arr)<br>
t1  = torch.tensor(arr)<br>
t2  = torch.from_numpy(arr)</p>
<p>arr<br>
t0<br>
t1<br>
t2</p>
<p>t2[:, 1] = 23.0</p>
<p>arr<br>
t0<br>
t1<br>
t2</p>
<pre><code class="lang-auto">arr
array([[ 0., 23.],
       [ 2., 23.],
       [ 4., 23.],
       [ 6., 23.],
       [ 8., 23.]], dtype=float32)

t0
tensor([[ 0., 23.],
        [ 2., 23.],
        [ 4., 23.],
        [ 6., 23.],
        [ 8., 23.]])

t1
tensor([[0., 1.],
        [2., 3.],
        [4., 5.],
        [6., 7.],
        [8., 9.]])

t2
tensor([[ 0., 23.],
        [ 2., 23.],
        [ 4., 23.],
        [ 6., 23.],
        [ 8., 23.]])
</code></pre>
<p>It’s confusing when your wife’s name is Katja, but you call her Tanja right? Unclear side effects are possible in both cases …</p>
<p>Searching for “torch tensor” can lead to different results containing torch.tensor and torch.Tensor results as well…</p>

username: Yaroslav_Bulatov
cooked: <p>Agreed that it’s confusing, I get tripped up by this as well.</p>

username: lijunzh
cooked: <p>In addition to the confusion discussed here, it also makes pylint complaining.</p>
<p><code>a = torch.tensor([1,2,3])</code> results in  the following error:</p>
<pre><code class="lang-auto">[pylint E1102] torch.tensor is not callable (not-callable) [E]
</code></pre>
<p>but <code>a=torch.Tensor([1,2,3])</code> works fine.</p>
Licence for a wrapper

username: amandeep1991
cooked: <p>Hi PyTorch Team,</p>
<p>Thanks for building this wonderful library.</p>
<p>I have been using many open-source libraries for quite some time now; however, I am a newbie contributor in the same space, so doubtful about licensing.</p>
<p>I have a vision that I would be building at least my domain-specific 5-10 models  using pytorch library, which could be useful for the masses (just ready for deploy). Considering this, I thought to create a separate python library which would be having dependency of pytorch library.</p>
<p>Could you please clarify my following doubts, please?</p>
<ul>
<li>Am I allowed to do the same?</li>
<li>What licence I need to add to my github repository?</li>
</ul>
<p>Kindly let me know for any further information.</p>
<p>With regards,</p>
<p>Amandeep</p>
<p>+91-9971743335</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/smth">@smth</a> for licensing issues.</p>

username: amandeep1991
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p><a class="mention" href="/u/smth">@smth</a>: could you please provide your valuable inputs?</p>
<p>Thanks,<br>
Aman</p>

username: amandeep1991
cooked: <p>Dear Pytorch Community,</p>
<p>I have directly copied the LICENSE file from the pytorch library, I hope this would suffice the requirement of writing wrapper of pytorch.</p>
<p>If, however; my understanding needs to be corrected, please let me know.</p>
<p>Thanks,<br>
Aman</p>

username: smth
cooked: <p>PyTorch is BSD licensed.</p>
<p>If you write New Code in your own repository, you can license it under whatever license you want.<br>
If you are distributing PyTorch code as-is, you have to preserve PyTorch code license in the distribution.</p>

username: amandeep1991
cooked: <p>Thanks <a class="mention" href="/u/smth">@smth</a>, I would be creating my own neural network graphs and will do train and all, so will be using any other license.</p>
<p>Thanks again.</p>
<p>Cheers</p>
I have a question about how to optim an defined matrix by myself

username: mshmoon
cooked: <p>The sample code is blow:<br>
class Test(nn.Module):<br>
def <strong>init</strong>():<br>
super(Test,self).<strong>init</strong><br>
self.p=torch.randn(10).require_grad_(True)<br>
def forward(self,x):<br>
return self.p*x</p>
<p>I define a class like Test, now the question is how to optim self.p at the loss  backward stage, although I use the flag of True to make self.p can be autograd, but the value of self.p do not change afther every loss backward and optimizer.step(), so is there any experts can help me?</p>

username: mshmoon
cooked: <p>I know how to solve it</p>

username: albanD
cooked: <p>If you want it to be properly picked up when you do <code>model.parameter()</code>, you need to make it a nn.Parameter:</p>
<pre><code class="lang-auto">self.p = nn.Parameter(torch.randn(10))
</code></pre>

username: mshmoon
cooked: <p>Yes,it can be work,thank you for your help</p>
Middle layers not learning anything. What should I do?

username: KUSHAL_JAIN
cooked: <p>I am training a QA model, which is not giving satisfactory results. The metrics are subpar. Upon investigation, I tried plotting the layer gradients and found this. I have used the function from this <a href="https://discuss.pytorch.org/t/check-gradient-flow-in-network/15063/10">thread</a>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67290bc0b829b3d6a81662c7e7b68d1f55ceb7a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/67290bc0b829b3d6a81662c7e7b68d1f55ceb7a0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67290bc0b829b3d6a81662c7e7b68d1f55ceb7a0_2_463x500.png" alt="image" data-base62-sha1="eIB7qWi1WL0N3bBhK0mSM5xs00g" width="463" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67290bc0b829b3d6a81662c7e7b68d1f55ceb7a0_2_463x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67290bc0b829b3d6a81662c7e7b68d1f55ceb7a0_2_694x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/67290bc0b829b3d6a81662c7e7b68d1f55ceb7a0_2_926x1000.png 2x" data-dominant-color="BDDDDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1269×1369 325 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From the figure above, it seems as if the middle layers are not learning anything.<br>
What should I try doing in order to fix this?</p>
Creating MTGP constraints failed

username: jdhurwitz
cooked: <p>I’m training the weights of an embedding layer for POS tags (tagging done by nltk), and then appending this to my word embedding features before inputting the combined vector into an LSTM. Has anyone seen or dealt with the following issue before when working with embedding weight training?</p>
<pre><code class="lang-auto">  File "/home/software/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/dropout.py", line 39, in forward
    ctx.noise.bernoulli_(1 - ctx.p).div_(1 - ctx.p)

RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1518244507981/work/torch/lib/THC/THCTensorRandom.cu:33```</code></pre>

username: AotY
cooked: <p>I have the save probleam.</p>
<pre><code class="lang-auto">ctx.noise.bernoulli_(1 - ctx.p).div_(1 - ctx.p)
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1532581333611/work/aten/src/THC/THCTensorRandom.cu:34

</code></pre>

username: scuShanshanChen
cooked: <p>I have the save probleam． how to fix it？?？</p>
Working with videos of arbitrary length

username: Tom_Ginsberg
cooked: <p>Hi, I’m trying to implement a network that feeds each frame of an arbitrary length video into a CNN to generate a sequence of feature vectors, then feeds the sequence into an LSTM.</p>
<p>My first idea seems a bit hacky so I thought I’d ask for an opinion or if someone thinks there is a better way</p>
<pre><code class="lang-auto">import torch

# Two 1 channel videos of 3 and 4 frames respectively
vid1 = torch.ones(1,3,3,3)
vid2 = 2* torch.ones(1,4,3,3)

# A toy CNN that takes the average every 2x2 square in a frame
cnn = torch.nn.Conv2d(1,1, kernel_size=(2,2), bias=False)
cnn.load_state_dict({'weight': torch.ones(1,1,2,2) * 1/4})

# Now I can run each frame of a video through the CNN, for example
cnn(vid1.transpose(0,1)).flatten(1)
# [out] tensor([[1., 1., 1., 1.],
#               [1., 1., 1., 1.],
#               [1., 1., 1., 1.]])

# but to run a batch say containing vid1 and vid2
cnn(torch.cat([vid1, vid2],dim=1).transpose(0,1)).flatten(1)
# [out] tensor([[1., 1., 1., 1.],
#               [1., 1., 1., 1.],
#               [1., 1., 1., 1.],
#               [2., 2., 2., 2.],
#               [2., 2., 2., 2.],
#               [2., 2., 2., 2.],
#               [2., 2., 2., 2.]])

# Now, with some difficulty, 
# I could make a PackedSequence and feed that into an LSTM
</code></pre>
<p>This idea seems not very elegant so I’m open to suggestions. I’ve also thought about making the batch size 1, running one video at a time using the batch dimension for frames, but only step the optimizer every <code>N</code>  call to <code>loss.backward()</code> to create an effective batch.</p>
<p>If anyone has any thoughts on this I would greatly appreciate it. Thanks</p>
<p><a href="https://colab.research.google.com/drive/1DclRcnnNA3P17wJxmEIp-eUFyf3WnKAF?usp=sharing" rel="nofollow noopener">Here</a> is a collab notebook continuing the implementation</p>

username: Tom_Ginsberg
cooked: <p>I found a very similar post here that answers my questions</p><aside class="quote quote-modified" data-post="3" data-topic="82422">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yanweiw/48/24684_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-process-variable-length-sequence-of-images-with-cnn/82422/3">How to process variable length sequence of images with CNN</a> 
  </div>
  <blockquote>
    Sure, each data sequence is of format [image_0, action_1, image_1, action_2, image_2, action_3, image_3…], and the task is to predict action sequence [action_1, action_,…, action_k] given image sequence [image_0, image_1,…, image_k]. The image sequence of a variable length k+1 in a batch, so I pad each sequence with zero images until sequence length is max_seq_len. The batched input is thus of shape (B, max_seq_len, C, H, W). 
My network uses a CNN model to embed each image into a feature vector…
  </blockquote>
</aside>
How do I interpret the chrome trace from the profiler?

username: kmaeng
cooked: <p>This seems like a newbie question but couldn’t find any information that is detailed enough for me to understand.<br>
I am trying to understand how to interpret the chrome trace from the autograd profile.<br>
Below code generates a very simple chrome trace</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    with torch.autograd.profiler.profile(True, False) as prof:
        net = Net()
        optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
        optimizer.zero_grad()
        y = net(torch.tensor([1.0,2.0]))
        y.sum().backward()
        optimizer.step()
    print(prof.key_averages().table(sort_by="cpu_time_total"))
    prof.export_chrome_trace("test.json")
</code></pre>
<p>And this is the chrome trace I get:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41eaf90e4c93aad58caf77362f54e688a4869e7d.png" data-download-href="https://discuss.pytorch.org/uploads/default/41eaf90e4c93aad58caf77362f54e688a4869e7d" title="Screen Shot 2020-05-27 at 6.11.41 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41eaf90e4c93aad58caf77362f54e688a4869e7d_2_690x63.png" alt="Screen Shot 2020-05-27 at 6.11.41 PM" data-base62-sha1="9p8sGoheBlusBO9NkPL6WtI3jzL" width="690" height="63" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41eaf90e4c93aad58caf77362f54e688a4869e7d_2_690x63.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41eaf90e4c93aad58caf77362f54e688a4869e7d_2_1035x94.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41eaf90e4c93aad58caf77362f54e688a4869e7d_2_1380x126.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-05-27 at 6.11.41 PM</span><span class="informations">2828×262 19.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems like the three chunks roughly represents the forward, backward, and weight update.<br>
However, I don’t understand what is the two rows, shown as “Process CPU functions”. Are these two different processes? Or two different threads? Or two different CPUs?<br>
From looking at the JSON file it seems like it is two separate threads (because they are tid 1 and 2). However, I am not sure why only the backprop would run on a different thread. Is the two really two different thread?<br>
If so, is there any way to view process/CPU activities as well? Or can I only collect profile for a single process?</p>
<p>This seems like a very simple question, yet I cannot find a clear answer from the web. If there is any resource that I am not finding, please let me know.</p>
<p>Thank you!</p>
Is there a way to extract weight update matrix using PyTorch api?

username: csailnadi
cooked: <p>During training we are update weights inside the model.<br>
Say we take conv layers.<br>
How do I access the weight update matrix?</p>
<p>Is there an api for this sort of thing or do I need to save weights of interest myself and then find the difference of those?</p>
<p>Thank you</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you’re using the builtin optimizers from torch.optim, then not really. Because each optimizer is free to make the update as it sees fit. And some of them (life LBFGS) actually does multiple updates.</p>
<p>If you’re doing the update yourself by hand, then I guess you can do this  by making your update code return what you want.</p>

username: csailnadi
cooked: <p>I see, yeah I was asking in scope of torch optimizers. Thank you</p>
Suggestion for code optimisation

username: aleSuglia
cooked: <p>Hello everyone,</p>
<p>I’m creating a specific application in which there are 2 agents that send messages to each other. Each agent is modelled with a neural network designed using PyTorch. These agents generate predictions until the end of the episode. A possible interaction is the following:</p>
<ol>
<li>Agent 1: &lt;message 1&gt; -&gt; Agent 2: &lt;answer to message 1&gt;</li>
<li>Agent 1: &lt;message 2&gt; -&gt; Agent 2: &lt;answer to message 2&gt;</li>
<li>Agent 1: &lt;message 3&gt; -&gt; Agent 2: &lt;answer to message 3&gt;<br>
…<br>
</li>
</ol>
<p>After this step, I have to collect the experiences and run an optimisation step for Agent 1 and a possible third agent that is used in the process (Agent 2 is not optimised). I’m trying to find a way to optimise my code in such a way that the phase in which the two agents have to generate messages over multiple turns is minimised. In my current implementation, the models are using batch-wise operations on GPU to process multiple examples at the same time. Unfortunately, I have some operations that are using pure Python. How do you suggest I should proceed in optimising this code? Do you have any specific optimisation suggestion about this specific design?</p>
Weight_norm with mean-only batch_norm

username: nnss
cooked: <p>Does nn.utils.weight_norm apply mean-only batch norm too? I am trying to do semi-supervised learning with GANs on the Cifar-10 dataset and I’ve come across numerous resources that recommend applying weight_norm with mean-only batch norm for the task as it improves performance.</p>
<p>Additionally, I am facing issues when I try to deepcopy a network that uses nn.utils.weight_norm. I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
<p>Could you please advise a workaround this issue? I assume this is because, deepcopy cannot copy hooks of a module.</p>
How to test on the model using my images

username: Adon_Augustin
cooked: <p>Hello,<br>
I have referred this url :<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>and made a model for a dataset.My data set folder contain train,val and test folders.How can I test this model using my test images?</p>
<p>Thanks</p>

username: SANTOSH_S
cooked: <p>Hello <a class="mention" href="/u/adon_augustin">@Adon_Augustin</a><br>
Testing the model on images set follows the same process as validation as given in the url shared by you. All you need to do is create test DataLoader(if you have many images) and pass it to the model and obtain the output.<br>
The output from the model will have class probabilities  (batch_size x no.Classes) and using torch.max(model_output,1) returns the  max_probability for a class and index.<br>
Ex:<br>
<code>max_value, max_index = torch.max(model_output,1)</code></p>
<p>you can use the max_index to get the class name for particular input.<br>
<code>class_names[max_index]</code> where class_names is a list contains all class names.</p>
<p>For reference see the last  test code cell in this <a href="https://github.com/mukul54/A-Simple-Cat-vs-Dog-Classifier-in-Pytorch/blob/master/catVsDogv2.ipynb" rel="nofollow noopener">GitHub code link</a></p>
<p>Let me know if you face any issues!</p>

username: Adon_Augustin
cooked: <p>Thank you let me try that</p>

username: Adon_Augustin
cooked: <p>Thank you <a class="mention" href="/u/santosh_s">@SANTOSH_S</a><br>
It is working for me</p>
Weights for different datasets

username: Salim
cooked: <p>If you have 2 different datasets that you would like to upweight training samples from one more the than other, how would you go about that? This is similar to what we do in class imbalance but for datasets instead of classes.<br>
A small example would be great and appreciated!</p>

username: dunefox
cooked: <p>Cost functions like <a href="https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html#crossentropyloss" rel="nofollow noopener">CrossEntropyLoss</a> have a parameter <code>reduction</code> that you can set to none so that you get the loss per instance, as I understand it. You could maybe use that and weight the relevant instances yourself before combining all losses and calling backward.</p>
How can I concatenate tensors during the forward pass

username: Lanorius94
cooked: <p>This is what I have so far:</p>
<p>input_size_prot = 1024<br>
input_size_comp = 196<br>
hidden_size_prot = 32<br>
hidden_size_all = 20<br>
output_size = 1</p>
<p>batch_size = 80</p>
<p>class pcNet(nn.Module):</p>
<pre><code>def __init__(self, input_size_prot, input_size_comp, hidden_size_prot, hidden_size_all, output_size):
    super(pcNet, self).__init__()
    self.fc_prot = nn.Linear(input_size_prot, hidden_size_prot)
    self.fc_all  = nn.Linear(hidden_size_prot+input_size_comp, hidden_size_all)
    self.fc2     = nn.Linear(hidden_size_all, output_size)

def forward(self, x):
    out = F.leaky_relu(self.fc_prot(x[0]))
    out = torch.cat((out, x[1]), 0) #here lies the problem
    out = F.leaky_relu(self.fc_all(out)) 
    out = F.relu(self.fc2(out))
    return out
</code></pre>
<p>So x is a list of two tensors. The first has 1024 elements and gets compressed to 32 nodes, which I now try to combine with the second tensor in x. Currently I get the following error:</p>
<p>RuntimeError: Sizes of tensors must match except in dimension 0. Got 32 and 196 in dimension 1</p>

username: Nikronic
cooked: <p>Hi,</p>
<aside class="quote no-group" data-username="Lanorius94" data-post="1" data-topic="83091">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e19b73/48.png" class="avatar"> Lanorius94:</div>
<blockquote>
<p><code>out = torch.cat((out, x[1]), 0) #here lies the problem</code></p>
</blockquote>
</aside>
<p>Your input will be something like this for concatenation:</p>
<pre><code class="lang-auto">x = [torch.randn(1, 32), torch.randn(1, 196)]
</code></pre>
<p>So based on this line, I expect the desired out is <code>[1, 196+32]</code>.</p>
<aside class="quote no-group" data-username="Lanorius94" data-post="1" data-topic="83091">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e19b73/48.png" class="avatar"> Lanorius94:</div>
<blockquote>
<p><code>nn.Linear(hidden_size_prot+input_size_comp, hidden_size_all)</code></p>
</blockquote>
</aside>
<p>Based on this, you need to concatenate in second dimension not first. First one is for batch size.<br>
So, this will solve this issue:</p>
<pre><code class="lang-auto">out = torch.cat((out, x[1]), dim=1)  # note that dim=1 -&gt; [batch_size, 196], [batch_size, 32]
</code></pre>
<p>Bests</p>

username: Lanorius94
cooked: <p>Nicely explained and it works. Thanks!</p>
TORCH::FROM_BLOB memory types

username: solarflarefx
cooked: <p>So to my understanding, pytorch has the capability of doing direct GPU inference.  For example, if the inputs are already in the GPU I can tell the inference network the input location in GPU memory and also specify where to place the output in GPU memory.  The Torch::From_Blob method would be used for this.</p>
<p>My question now is how does the situation differ if the memory is not linear but instead a volume as a Cuda Array.  This type of structure doesn’t really have a pointer in the same sense and under normal circumstances one would have to use surf3Dwrite() and surf3Dread() in Cuda kernels.  If the input is in this format, what is the recommended way to run inference on it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortuatly, from_blob can only take raw memory pointer (and size/stride informations). So you won’t be able to wrap complex data types inside a torch Tensor.<br>
But if one copy is ok, you can dump it into a new contiguous memory buffer and use that.</p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Thanks for the clarification</p>
How to do ReLU 1 operation?

username: mohit117
cooked: <p>I want to use ReLU1 non-linear activation.  <em><strong>ReLU1 is linear in [0,1] but clamps values less than 0 to 0 and clamps values more than 1 to 1</strong></em> .</p>
<p>So which of the following will be the best way to do, causing no problem in backpropagation.</p>
<p><code>x.clamp_(min=0.0, max=1.0)</code> or <code>torch.nn.functional.hardtanh_(x, min_val=0.0, max_val=1.0)</code></p>
<p>This will be just used for the final layer. For other layers I shall just use LeakyReLu.</p>
<p>Thankyou</p>

username: albanD
cooked: <p>Hi,</p>
<p>Both should work fine.<br>
If you see issues with modifying inplace values needed for gradient computation, you can swap them to use the out of place versions.</p>

username: mohit117
cooked: <p>Oh so does this mean that inplace may cause some trouble in back-propagation? I ask this almost all codes of ResNet including the Pytorch official release always set the inplace flag as true, maybe to save memory usage.<br>
Thankyou</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can try the inplace, if you don’t get errors, then it’s fine.<br>
If you get an error, then you’ll have to remove the inplace.</p>
Straitified K-fold filling up my ram too fast

username: chetan06
cooked: <pre><code class="lang-auto">skf = StratifiedKFold(n_splits = 5)
skf.get_n_splits(train_csv)
i=0
for idx1, idx2 in skf.split(train_csv['ID'],train_csv['Class']):
    model = timm.create_model('efficientnet_b1', pretrained = True)
    model.train()
    model.classifier = nn.Linear(in_features=1280, out_features=3, bias=True)
    model.to(device)
    optimizer1 = optim.Adam(model.parameters(), lr = 0.001)
    optimizer2 = optim.SGD(model.parameters(), lr = 0.001)
    scheduler1 = lr_scheduler.ReduceLROnPlateau(optimizer1,factor=0.33, mode="min", patience=4)
    scheduler2 = lr_scheduler.ReduceLROnPlateau(optimizer2,factor=0.33, mode="min", patience=4)
    train_data,_= train_csv.iloc[idx1], _
    valid_data = train_csv.iloc[idx2]
    train_dataset = AgeDataset(train_data, IMG_DIR, train_transform())
    valid_dataset = AgeDataset(valid_data, IMG_DIR, valid_transform())
    train_loader = DataLoader(
    train_dataset,
    batch_size = 4,
    shuffle = True,
    num_workers = workers
    )
    valid_loader = DataLoader(
    valid_dataset,
    batch_size = 4,
    shuffle = False,
    num_workers = workers
    )
    model = train(model, train_loader, valid_loader, optimizer1, scheduler1, criterion, 5)
    model = train(model, train_loader, valid_loader, optimizer2, scheduler2, criterion, 5)
    i+=1
    torch.save(model, r"/content/drive/My Drive/Age Detection/effnet"+str(i)+".pth")
    print("Saved Model "+str(i))
</code></pre>
<p>This is how I implemented Stratified K-fold on my model my this implemetation is filling up my RAM too fast. Please suggest some changes or another efficient implementation</p>

username: chetan06
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a> and others please help</p>

username: ptrblck
cooked: <p>Could you explain a bit, how fast your RAM is filled and in which iteration?<br>
Are you seeing a contiguous increase in memory for each iteration?</p>

username: chetan06
cooked: <p>Actually I used 10 epochs for training and only after 5 epochs my ram was filled up. I am using the same code though I think that the code is not perfect <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: ptrblck
cooked: <p>Is this the entire code for the model training or are you storing some output tensors, losses or other objects somewhere in the code?<br>
Also, are you running out of GPU memory or system RAM?</p>

username: chetan06
cooked: <p>I’m running out of system RAM.<br>
In the code, at last I am saving the models</p>

username: ptrblck
cooked: <p>That should be alright, as you are storing them on your drive.</p>
<p>Could you post a code snippet to reproduce this issue, please?<br>
The model and the input data shapes would probably be enough so that we could run it locally.</p>

username: chetan06
cooked: <p>train_csv’s shape = (19906,2)<br>
model is efficientnet_b0 from timm</p>
Different learning rate from the same module

username: ryoma
cooked: <p>How can I assign forward-adaptive learning rates to modules in sequential?</p>
<p>I know in normal case, we can use optimizer and make this like <code>"params" : model.a.parameters(), 'lr' : lr / 100</code>, but I wanna use different learning-rate for loss1 and loss2 which went through the same module.</p>
<pre><code class="lang-auto">class X(nn.Module):
   def __init__(self):
      super().__init__()
      self.a = nn.Conv2d(1,1,1)
      self.b = nn.Conv2d(1,1,1)

   def forward(self, input):
       return self.b(self.a(input))


model = X()
output1 = model(input1)
output2 = model(input2)

loss1 = F.mse_loss(output1, label1)
loss2 = F.mse_loss(output2, label2)
</code></pre>
<p>And for <code>loss1</code>, I want both <code>a</code> and <code>b</code> parameters step-up with <code>lr</code>, but for <code>loss2</code>, <code>a</code> for <code>lr</code> and <code>b</code> for <code>-lr</code>.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>module</th>
<th>a</th>
<th>b</th>
</tr>
</thead>
<tbody>
<tr>
<td>loss1</td>
<td>lr</td>
<td>lr</td>
</tr>
<tr>
<td>loss2</td>
<td>lr</td>
<td>-lr</td>
</tr>
</tbody>
</table>
</div><p>because <code>loss1</code>, <code>loss2</code> backward through both <code>a</code> and <code>b</code> modules, we can’t make this just by <code>(- loss1).backward()</code> or like that.</p>
<p>can I assign some options when <code>model</code> forwards?</p>
<p>thanks,</p>

username: ptrblck
cooked: <p>The loss calculation is independent from the optimization step (parameter update) in general.<br>
You could create two optimizers with the specified learning rates and apply them sequentially, if that would fit your use case:</p>
<pre><code class="lang-python">optimizer1.zero_grad()
loss1.backward(retain_graph=True) # calculate gradients using loss1
optimizer1.step() # update parameters with lr, lr

optimizer2.zero_grad()
loss2.backward() # calculate gradients using loss2
optimizer2.step() # update parameters with lr, -lr
</code></pre>
<p>Would this work for you?</p>

username: ryoma
cooked: <p>Great! It seems to work well!<br>
Thank you for your quick and kind reply!</p>
More gpu accelerate the speed of dataloader?

username: XavierLinNow
cooked: <p>In my experiment, I test different num of gpus and num_wokers in dataloader. Here is the result</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>num gpu</th>
<th>batch size</th>
<th>num worker</th>
<th>one batch time(get data and forward backward)</th>
<th>one batch time(get data)</th>
</tr>
</thead>
<tbody>
<tr>
<td>2</td>
<td>512</td>
<td>128</td>
<td>0.8</td>
<td>0.3</td>
</tr>
<tr>
<td>2</td>
<td>512</td>
<td>32</td>
<td>0.8</td>
<td>0.3</td>
</tr>
<tr>
<td>2</td>
<td>512</td>
<td>16</td>
<td>0.8</td>
<td>0.3</td>
</tr>
<tr>
<td>2</td>
<td>512</td>
<td>8</td>
<td>0.8</td>
<td>0.3</td>
</tr>
<tr>
<td>2</td>
<td>512</td>
<td>2</td>
<td>0.8</td>
<td>0.3</td>
</tr>
<tr>
<td>2</td>
<td>512</td>
<td>1</td>
<td>3.7</td>
<td>3.2</td>
</tr>
<tr>
<td>8</td>
<td>512</td>
<td>32</td>
<td>0.3</td>
<td>0.08</td>
</tr>
<tr>
<td>8</td>
<td>1024</td>
<td>32</td>
<td>0.55</td>
<td>0.14</td>
</tr>
</tbody>
</table>
</div><p>There are two weird thing. 1) is more gpu accelerate the speedup of dataloader. 2) when increase the num_worker, it doesn’t accelerate the speed of getting data.</p>

username: SANTOSH_S
cooked: <p>Interesting. Are these experiments conducted with <code>pin_memory=True</code> in the DataLoaders ?</p>

username: ptrblck
cooked: <p>Could you share the code you’ve used to measure the data loading speed?<br>
Note that too many workers might degrade the performance as explained in <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>.</p>
Dataset on Local Computer

username: MUnique
cooked: <p>Hello all,<br>
I have a big dataset (for video classification) located on my local computer. Is it possible to train a PyTorch network on a remote GPU server while the dataset is located on local computer?</p>

username: ptrblck
cooked: <p>That would be possible, if you mount the local drive to the remote machine.<br>
Note however that you might see a pretty big performance hit, as the data loading would go through your network and will most likely be the bottleneck of your training.</p>
transforms.Resize

username: ky_Pa
cooked: <p>The default input image size of resnet network is 224 * 224, but the image size of my data set is 200 * 200. How should I adjust it during preprocessing? Is it possible to use transforms.Resize (224)?</p>

username: Rokuki
cooked: <pre><code class="lang-python">class mydataset(data.Dataset):
    def __init__(self,root,transforms=None):
        ...
        transforms = T.Compose([
                    T.Resize ((224,224)),
                    T.ToTensor(),
                    T.Normalize(mean = [0.485, 0.456, 0.406], 
                                     std = [0.229, 0.224, 0.225])
                    ]) 
    def __getitem__(self,index):
        ...
        data = self.transforms(data)
        return data, label

</code></pre>

username: ky_Pa
cooked: <p>Thank you very much! So is this to enlarge the 200 * 200 image to 224 * 224 first?</p>

username: SANTOSH_S
cooked: <p>Yes <a class="mention" href="/u/ky_pa">@ky_Pa</a>  you are right.<br>
In the above code you might need to write <code>T.Resize ((224,224))</code> ,<br>
If size is a sequence like (h, w), output size will be matched to this. If size is an int, smaller edge of the image will be matched to this number. i.e, if height &gt; width, then image will be rescaled to (size * height / width, size)</p>

username: ky_Pa
cooked: <p>Thanks for your answers!</p>
Multi-Task Classifier

username: Aki58
cooked: <p>I am working on a Multi-Task text classifier, I want to make it config based i.e. I have an x input but not fix set of tasks. tasks are decided based on the config.<br>
So how can I initialize different Classifiers based on config?</p>
<p>eg:- if my config has 2 tasks then 2 linear layers are initialized in model init and if my config has 5 tasks then 5 linear layers are initialized.</p>

username: ptrblck
cooked: <p>You could create a config file as e.g. a JSON file and read it in your script via <code>pandas</code>.<br>
Once you have the right keys from your config, you could use <code>nn.ModuleList</code> or <code>nn.Sequential</code> to create the number of layers based on the current config.</p>
<p>The <a href="https://pytorch.org/docs/stable/nn.html?highlight=modulelist#torch.nn.ModuleList">ModuleList docs</a> give you an example:</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])

    def forward(self, x):
        # ModuleList can act as an iterable, or be indexed using ints
        for i, l in enumerate(self.linears):
            x = self.linears[i // 2](x) + l(x)
        return x
</code></pre>
<p>Instead of the fixed <code>10</code> you could use the value passed in your config file.</p>

username: Aki58
cooked: <p>I am adding this layer on top of hugginging face transformers layers, and in params it is showing that these layers are added, training loss is reducing greatly but eval loss is very high and increasing.</p>
How to manually set which GPU to use?

username: Ratan
cooked: <p>Hi,<br>
I am trying to load my model to the gpu using model=model.cuda()<br>
I have 2 gpus in my system and one gpu has 10gb memory. How to load my model to gpu with more memory?</p>

username: SANTOSH_S
cooked: <p>Hi,<br>
<a class="mention" href="/u/ratan">@Ratan</a> you  can use <code>nvidia-smi</code> command to find out which GPU 0 or 1 has more memory. Accordingly you can follow one of the following ways to load the model to the manually to particular GPU:</p>
<p>Inside the python file you can add:<br>
<code>os.environ['CUDA_VISIBLE_DEVICES'] = "0"</code></p>
<p>In command line<br>
<code>CUDA_VISIBLE_DEVICES=0 python ...</code></p>
<p>In the python file you can add:<br>
<code>torch.cuda.set_device(0)</code></p>

username: Ratan
cooked: <p><a class="mention" href="/u/santosh_s">@SANTOSH_S</a> thank you</p>
Improving precision error in model parallel setting

username: sh0416
cooked: <p>Code:</p>
<pre><code class="lang-auto">model0 = nn.Linear(10, 10)
model1 = nn.Linear(10, 10)
model2 = nn.Linear(10, 10)
model3 = nn.Linear(10, 10)
input = torch.rand(128 ,10)
output = model3(model2(model1(model0(input))))
model0.to('cuda:0')
model1.to('cuda:1')
model2.to('cuda:2')
model3.to('cuda:3')
pred = model3(model2(model1(model0(input.to('cuda:0')).to('cuda:1')).to('cuda:2')).to('cuda:3')).cpu()
assert torch.allclose(output, pred) # Return False
</code></pre>
<p>How to fix this test? The maximum difference between output and pred is just 6.3330e-08 so that the implementation is not wrong but some system issue, I think.</p>
<p>Is there any possible option to make this thing right?</p>
<p>I don’t want to lose precision accuracy so that adding <code>atoi=1e-7</code> to <code>torch.allclose</code> is not I want.</p>
<p>Thanks,</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sh0416" data-post="1" data-topic="83041">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sh0416/48/38708_2.png" class="avatar"> sh0416:</div>
<blockquote>
<p>The maximum difference between output and pred is just 6.3330e-08 so that the implementation is not wrong but some system issue, I think.</p>
</blockquote>
</aside>
<p>This small difference is most likely a result of the limited floating point precision, which can be seen e.g. by changing the order of operations:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10, 10, 10)
sum1 = x.sum()
sum2 = x.sum(0).sum(0).sum(0)

print(sum1 - sum2)
&gt; tensor(-3.8147e-06)
</code></pre>
<p>Avoiding these differences is especially hard (or impossible) using different hardware.<br>
What is your current setup, i.e. which GPUs are you using?</p>

username: sh0416
cooked: <p>Wow… your testcase makes me creepy…<br>
Thanks, though. I’m using 4 Titan Xp and I just increase the <code>atol</code> to 1e-7 to make my testcase work.<br>
Is there any technical paper that the cascaded precision error incurs the performance degeneration in machine learning?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sh0416" data-post="3" data-topic="83041">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sh0416/48/38708_2.png" class="avatar"> sh0416:</div>
<blockquote>
<p>Is there any technical paper that the cascaded precision error incurs the performance degeneration in machine learning?</p>
</blockquote>
</aside>
<p>Not that I’m aware of and usually you should consider in your calculations the limited floating point precision. I.e. if you current use case needs more precision, you would have to use <code>float64</code>, which increases the precision (but is still limited).</p>