akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory usage, best practices

111439: <p>Good day,<br>
I am wondering if there are any best practices on how to use the cuda memory effectively or how to overcome the out of memory error if it is possible (except for reducing the batch size or the complexity of the model)?</p>
<p>The thing is, I am stacking the output of the neural network to put it to the consecutive layers and I receive ‘cuda out of memory’ error.<br>
My input size is [1,1,128,256,256], batch size=1. I have 40gb of VRAM. Without this stacking step the simpler model took about 26gb of VRAM. With the stacking step, the out of memory error occurs on the 91 iteration of this code</p>
<pre><code class="lang-auto">def stack(self, x):
    output = torch.zeros(x.shape,dtype=torch.float16)
    for i in range(x.shape[2]):
        print(i)
        output[:,:,i,:,:] = neural_net(x[:,:,i,:,:])
</code></pre>
CUDA 11.1 support for pytorch

bergentruckung: <p>Hello,</p>
<p>Is there a probable roadmap for when we can expect pytorch stable release to have support for CUDA 11.1? I have pytorch-1.7.1 (built for CUDA 11.0) installed on a box with CUDA 11.1, but I’m a bit paranoid if there could be potential bugs lurking underneath. This may just be a duplicate of <a href="https://github.com/pytorch/pytorch/issues/47109" class="inline-onebox" rel="noopener nofollow ugc">Pytorch 1.7/Cuda 11.1 binaries · Issue #47109 · pytorch/pytorch · GitHub</a>, but a rough timeline is much appreciated.</p>
<p>Thanks,<br>
bergentruckung</p>
Output shape of torch.count_nonzero

pdp: <p>After the execution of code mentioned below,The output shape of c1 variable after using torch.count_nonzero is torch.Size([]).</p>
<p>1.)  When “vb.backward(torch.ones(va.size()))”  is called, it gives following error as c1 shape is torch.Size([])</p>
<p><strong>RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([1, 1, 256, 256]) and output[0] has a shape of torch.Size([]).</strong></p>
<p>2.) If I replace “c1=torch.count_nonzero(input)” with “c1 = torch.tensor([1.1], dtype=torch.float)” , then it gives the following error.</p>
<p><strong>RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([1, 1, 256, 256]) and output[0] has a shape of torch.Size([1]).</strong></p>
<p>Code is as below.</p>
<p>class fun(Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(cxt,input):<br>
cxt.save_for_backward(input)<br>
c1=torch.count_nonzero(input)<br>
print(c1.shape)<br>
return c1<br>
<span class="mention">@staticmethod</span><br>
def backward(cxt, grad_output):<br>
input, = cxt.saved_tensors<br>
grad_input = grad_output.clone()<br>
grad_input[input &lt; 0] = 0<br>
return grad_input<br>
fun1 = fun.apply<br>
if <strong>name</strong> == “<strong>main</strong>”:<br>
from torch.autograd import Variable<br>
a=torch.rand(1,1,256,256)<br>
va = Variable(a, requires_grad=True)<br>
vb = fun1(va)<br>
vb.backward(torch.ones(va.size()))</p>

pdp: <aside class="quote no-group" data-username="pdp" data-post="1" data-topic="110836">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pdp/48/34217_2.png" class="avatar"> pdp:</div>
<blockquote>
<p>if <strong>name</strong> == “<strong>main</strong>”:</p>
</blockquote>
</aside>
<p>There is a correction here.</p>
<p>Its if <strong>__name == __main</strong></p>
How to add meaning node names to onnx model exported by torch.onnx.export?

Liming: <p>I have a PyTorch model of torch.jit.ScriptModule and have successfully converted it to onnx format. The problem is all the onnx nodes are named with sequential numbers. E.g., in the attached image below, the circled conv’s inputs and outputs are named with numbers (visualized with Netron), which is inconvenient for the following analysis if the network is large. How can I add more meaningful names to these intermediate onnx nodes? Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f34d5f7212f0f78b84063b2cdff32fdb5dd9193.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5f34d5f7212f0f78b84063b2cdff32fdb5dd9193" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f34d5f7212f0f78b84063b2cdff32fdb5dd9193_2_690x351.jpeg" alt="image" data-base62-sha1="dAeyIai5LgWNcOhDxmOWuvN53oL" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f34d5f7212f0f78b84063b2cdff32fdb5dd9193_2_690x351.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f34d5f7212f0f78b84063b2cdff32fdb5dd9193_2_1035x526.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f34d5f7212f0f78b84063b2cdff32fdb5dd9193_2_1380x702.jpeg 2x" data-dominant-color="3A393A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2030×1034 168 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

addisonklinke: <p>Hi <a class="mention" href="/u/liming">@Liming</a> did you end up finding a solution to annotate nodes with more meaningful names? I would be interested in your approach to help with debugging an invalid graph that PyTorch produced</p>
Use Tensorboard Writer to track multiple model graph at once

CaoHoangTung: <p>I have a model which was constructed by 2 sub-models. I tried to log the 2 models graph structure to Tensorboard using</p>
<pre><code class="lang-auto">writer.add_graph(model1, input1)
writer.add_graph(model2, input2)
</code></pre>
<p>I want to view the structure of both models, but Tensorboard will overwrite the first graph with the latter one.</p>
Sigmoid on a tensor returns [1.0 ...]

Brent1: <p>I did not expected the below output; as I thought that sigmoid would move the values of a tensor to be between 0 and 1?</p>
<p>Perhaps its related to handling finite precision of floats; but I didn’t manage to track down the relevant docs.</p>
<pre><code class="lang-auto">print(f"sig {torch.tensor([10.0,11.0,12.0]).sigmoid()}")
&gt; sig tensor([1.0000, 1.0000, 1.0000])
</code></pre>

MrPositron: <p>Yep. It just rounds them to 1. But it is in valid range, i.e. between 0 and 1. I also think that it just happens during printing.</p>
<pre><code class="lang-auto">val = torch.tensor([10.0,11.0,12.0]).sigmoid()
for i in range(val.shape[0]):
    print("{:.10}".format(val[i]))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">0.9999545813
0.9999833107
0.9999938011
</code></pre>
<p>Btw,  <code>torch.set_printoptions(precision=5)</code> function might help to view the output more nicely.</p>
Backpropagation with LSTM + Soft Attention mechanism

sigma_x: <p>Here X (last hidden state) and Y (all hidden states) are outputs of LSTM, F is attention computation and X’ is a copy of X, + is elementwise summation. My question is, how should I backprop in this case? Should I detach the output of F (context vector) before summation with X’? Also, is it better to <code>clone()</code> X or just create a new variable?</p>
<p>Why I’m concerned something’s not right is because for all inputs the softmax distribution is always the same: all 0s and the last one (dot-product of the last hidden state with itself) is 1.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dfd35cc8b404d01fc140acfbe0dd50498d0f349.png" alt="backprop" data-base62-sha1="fH0xLalY5j4UCY0BUZsMNgkehRv" width="379" height="261"></p>
Error: identifier "THCudaBlas_Sgemv" is undefined

gouthamvgk: <p>I am trying to compile DCNv2 layer from this repo <a href="https://github.com/CharlesShang/DCNv2" rel="nofollow noopener">https://github.com/CharlesShang/DCNv2</a>. While doing so my compilation fails with this error in stack trace “src/cuda/dcn_v2_cuda.cu(324): error: identifier “THCudaBlas_Sgemv” is undefined”. Any help??</p>

ptrblck: <p>It seems, the method was already updated as seen <a href="https://github.com/lbin/DCNv2/blob/c916fdf2ed69ebebc43cbe8a2cb5e7ce0dd71132/src/cuda/dcn_v2_cuda.cu#L327">here</a>.</p>

gouthamvgk: <p>Seems that THCudaBlas_Sgemv has been moved to at::cuda::blas::gemv. But after compiling the dcn with the new API linkage error shows up like undefined symbol: _ZN2at4cuda4blas4gemvIfEEvcllT_PKS3_lS5_lS3_PS3_l  . After going through torch repo i couldnt find the definition of at::cuda::blas::gemv anywhere. Am I missing something??</p>

ptrblck: <p>The method definition starts in <a href="https://github.com/pytorch/pytorch/blob/23174ca71b580b0a7f53e92fcf11bf3db1666938/aten/src/ATen/cuda/CUDABlas.cpp#L327">this line of code</a> and defines different <code>dtypes</code> etc.</p>
<p>Some examples on its usage are e.g. in <a href="https://github.com/pytorch/pytorch/blob/d7f7c290e3d76a1e3019166644baf78de0d95a31/aten/src/THCUNN/generic/SpatialConvolutionMM.cu#L506-L514">SpatialConvolutionMM.cu</a>.</p>

lbin: <p>Try this <a href="https://github.com/lbin/DCNv2/tree/pytorch_1.6" rel="nofollow noopener">https://github.com/lbin/DCNv2/tree/pytorch_1.6</a></p>

Felix_Lessange: <p>Using nvidia tensorrt 20.11 image and pytorch 1.7.1,<br>
Running this</p>
<pre><code class="lang-auto">find . -type f -exec sed -i 's/THCState_getCurrentStream(state)/c10::cuda::getCurrentCUDAStream()/g' {} \;
</code></pre>
<p>And replace external/DCNv2/src/cuda/dcn_v2_cuda.cu file by this version <a href="https://github.com/lbin/DCNv2/blob/c916fdf2ed69ebebc43cbe8a2cb5e7ce0dd71132/src/cuda/dcn_v2_cuda.cu#L327" class="inline-onebox" rel="noopener nofollow ugc">DCNv2/dcn_v2_cuda.cu at c916fdf2ed69ebebc43cbe8a2cb5e7ce0dd71132 · lbin/DCNv2 · GitHub</a></p>
<p>I manage to get a successful build, but I have the same issue mentionned above when importing dcn_v2 :</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "yolact/external/DCNv2/dcn_v2.py", line 13, in &lt;module&gt;
    import _ext as _backend
ImportError: yolact/external/DCNv2/_ext.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at4cuda4blas4gemvIfEEvcllT_PKS3_lS5_lS3_PS3_l

</code></pre>
Classifying buckets with documents

gangsongt: <p>I’m tackling the following problem – there are buckets of long text documents, each bucket contains 1…N documents, each bucket has 1 target label. First, I pretrain encoder – RoBERTa model with a masked language model objective on my corpus of texts (python code). Then I prepare buckets by splitting documents into chunks of length 512. Each document contains 1…M chunks. Usually, there are many of them.</p>
<p>I can’t process a batch of buckets with 1 GPU, so I go one bucket at a time. At first, I retrieve chunk embeddings to get an averaged document embedding. Then I average document embeddings to get a bucket embedding. Then I pass bucket embedding to a classification head to get logits by classes.</p>
<p>Pseudo-code:</p>
<pre><code class="lang-python">for bucket in train_dataset:
    document_embeddings = []
    for document in bucket:
        chunk_embeddings = []
        for chunk in document:
            chunk_embedding = encoder(**chunk) # (1)
            chunk_embeddings.append(chunk_embedding)
        document_embedding = torch.cat(chunk_embeddings, dim=0).mean(dim=1).unsqueeze(0)
        document_embeddings.append(document_embedding)
    bucket_embedding = torch.cat(document_embeddings, dim=0).mean(dim=1)
    logits = classification_head(bucket_embedding)
</code></pre>
<p>The problem is that I can’t get over the OOM error, because there are too many chunks that I pass through the pretrained encoder. This can be avoided by detaching retrieved embeddings (1) from the computational graph, but then optimization becomes tricky.</p>
<p>If I detach (1) embeddings in this algorithm, only the classification head will get trained. I’ve tried to use a shallow transformer encoder block to put averaged documents embeddings to get a bucket embedding for further classification. This seems to work only if the chunk encoder is well pretrained.</p>
<p>I’ve also tried to accumulate gradients after each chunk embedding retrieval (1) –– this reduces bucket classification to chunks average classification. Moreover, it doesn’t seem fair because the original setup calculates loss on averaged embeddings –– and it makes sense because documents have weighted contexts, so we often can’t say much about the whole bucket by looking at one document.</p>
<p>What else can I do? Is there a proper way to get all embeddings without losing their gradients?</p>
Best Practice for Conditionals In Forward Pass

Tom_Ginsberg: <p>Hi, I have a fairly simple question. Which would be the recommended implementation for a model that if passed <code>out_features=1</code> flattens the output after the forward pass. The first option uses an Identity placeholder and the second an if statement in the forward pass. I find this problem coming up a lot when trying to have models with lots of options to configure. Any advice would help guide my development. Thanks</p>
<pre><code class="lang-auto"># Option A: No conditions in forward pass, but a redundant call to nn.Identity()
class Model(Module):
    def __init__(self, out_features):
        self.tail = torch.nn.Identity()
        if out_features == 1:
             self.tail = torch.nn.Flatten()
    def forward(self, x):
        x = ...
        return sel.tail(x)        
</code></pre>
<pre><code class="lang-auto"># Option B: Conditions in forward pass
class Model(Module):
    def __init__(self, out_features):
        self.out_features = out_features
        
    def forward(self, x):
        x = ...
        if self.out_features == 1:
             return x.flatten()
        return x
</code></pre>

googlebot: <p>I wouldn’t worry about this too much performance-wise, but note that there is a way to omit runtime checks with JIT:</p>
<pre><code class="lang-auto">
class Model(Module):
  flag : jit.Final[bool]
  def __init__(self, out_features):
     ...
     self.flag = out_features &gt; 1
     self.tail = Flatten() if self.flag else None
  def forward(self,x):
     ...
     if self.flag:
       x=self.tail(x)
m = jit.script(Model(1))
</code></pre>
<p>this should totally remove ‘if’ code block. Even simpler, (tail is not None) check would also be statically resolved.</p>

Tom_Ginsberg: <p>cool, I’ll try it out</p>
Calculating the Gradients for WGAN-GP

israrbacha: <p>This sounds maybe a trivial question but I still don’t understand how the gradients are calculated for the applying gradient penalty.<br>
gradients = torch_grad(outputs=prob_interpolated, inputs=interpolated,<br>
grad_outputs=torch.ones(prob_interpolated.size()).cuda() if self.use_cuda else torch.ones( prob_interpolated.size()), create_graph=True, retain_graph=True)[0]</p>
<p>Why we need the value at index “[0]”?</p>
Onnx export node.scopeName() returns "" since pytorch 1.4 broke it

xvdp: <p>I need a means of connecting model.modules and exported jit graph.<br>
in torch 1.3  I used to be able to do</p>
<pre><code class="lang-python">import torchvision
import torch
from torch.onnx import utils
model = torchvision.models.resnet18()
tensor = torch.randn([1,3,224,224])
trace, out = torch.jit.get_trace_graph(model, tensor)
graph = trace.graph()
graph = utils._optimize_graph(graph, operator_export_type=torch._C._onnx.OperatorExportTypes.ONNX)
for i, node in enumerate(graph.nodes()):
    scopename = ".".join([x.split('[')[-1] for x in
                                  [s for s in node.scopeName().split("]")] if len(x.split('[')) &gt; 1])
    print("node.scopeName() &lt;%s&gt;  named_module:"%scopename, dict(model.named_modules())[scopename])
</code></pre>
<p>Which gives me a nice way to be able to inspect the latents with  _hooks at the same time that I know connectivity, hence can see properties like receptive filed and so on.</p>
<p>Since 1.4 scopeName() is empty. Looks like the pointer gets wiped out instead of stored in jit.trace</p>
<p>I just tried in 1.8 and it is still empty, I do realize that code should change to something like</p>
<pre><code class="lang-python">from torch.onnx import TrainingMode
from torch.onnx import utils
graph, dic, out = utils._model_to_graph(model, tensor, training=TrainingMode.TRAINING, _retain_param_name=True)
for i, node in enumerate(graph.nodes()):
   print(node.scopeName())
</code></pre>
<p>I know, I can “retain_names” and then i can grab the common prefix of the inputs that is not numeric, which will give me the scopeName of nodes with inputs. Conv yes, but not other nodes like ReLUs…</p>
<p>I also know that instead of <code>_model_to_graph()</code> I could do</p>
<pre><code class="lang-python">trace = torch.jit.trace(model, tensor)
graph = trace.inlined_graph
</code></pre>
<p>which keeps the dirtier version  out of which I can get the scopename, anyway, hacks.</p>
<ul>
<li>Is there a reason scopeName() gets cleared by jit?</li>
<li>Anyone know if there is an alternative to scopeName such that I can connect a graph to the trace of the graph, node by node?<br>
Or do I have to fix the code, probably around &lt;pytorch/torch/csrc/jit/ir/ir.h&gt;<br>
thanks/</li>
</ul>

xvdp: <p>where is my mind, i logged this as a bug one year ago…  i guess, last time i looked at this.</p>

xvdp: <p>the answer to previous query works for this one…  <a href="https://github.com/pytorch/pytorch/issues/33463" class="inline-onebox" rel="noopener nofollow ugc">torch._C.Node.scopeName() missing in pytorch 1.4 · Issue #33463 · pytorch/pytorch · GitHub</a></p>
Forward pass working in PyTorch but not ONNX

addisonklinke: <p>I am looking for generic troubleshooting advice others have found helpful when debugging PyTorch to ONNX exports.</p>
<p>Following the official PyTorch tutorial, I converted a 3rd party model to ONNX (see this Github <a href="https://github.com/rwightman/efficientdet-pytorch/discussions/184" rel="noopener nofollow ugc">discussion</a> for details and code to reproduce). The strange thing is, my dummy image batch works fine for forward prop in the PyTorch version of the model, but when I try to initialize an ONNXRuntime session, it gives the following error</p>
<pre><code class="lang-auto">ort_session = onnxruntime.InferenceSession(path)  &lt;-- raised by this Python line
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : 
FAIL : Load model from tf_efficientdet_lite0.onnx failed:Node (Concat_42) Op (Concat) 
[ShapeInferenceError] All inputs to Concat must have same rank
</code></pre>
<p>So far, I have used Netron’s visualization of the ONNX model to identify the <code>Concat_42</code> node in question. However, I am struggling to determine what line of code in the PyTorch implementation this node corresponds to. Has anyone solved a problem like this before? I imagine there are clues to look for in the verbose output of <code>torch.onnx.export</code> that would help identify the culprit in the PyTorch model.</p>
<p>On principle, it seems odd that forward prop can work in PyTorch but have shape compatibility issues in ONNX. Given this, I suspect there is a portion of the model definition (if-statements, loops, etc) that does not trace well, and therefore it breaks when accepting an input tensor other than the one that produced the trace. However, I use the same dummy tensor for the trace and ONNXRuntime session, so this explanation does not seem to hold.</p>
Optimizer initialization in Distributed Data Parallel

YutongZheng: <p>Hi, I am new to PyTorch’s DistributedDataParallel module. Now I want to convert my GAN model to DDP training, but I’m not very confident about what should I modify.</p>
<p>My original toy script is like:</p>
<pre><code class="lang-auto"># Initialization
G = Generator()
D = Discriminator()
G.cuda()
D.cuda()
opt_G = optim.SGD(G.parameters(), lr=0.001)
opt_D = optim.SGD(D.parameters(), lr=0.001)

G_train = GeneratorOperation(G, D) # a PyTorch module to calculate all training losses for G. 
D_train = DiscriminatorOperation(G, D) # a PyTorch module to calculate all training losses for D. 

# Training
for i in range(10000):
  loss_D = D_train()
  opt_D.zero_grad()
  loss_D.backward()
  opt_D.step()

  loss_G = G_train()
  opt_G.zero_grad()
  loss_G.backward()
  opt_G.step()
</code></pre>
<p>My question is, when I add DDP module to the above script, should I modify in</p>
<pre><code class="lang-auto">torch.cuda.set_device(local_rank)
G = Generator()
D = Discriminator()
G.cuda()
D.cuda()
G_ddp = DDP(G, device_ids=[local_rank], output_device=local_rank)
D_ddp = DDP(D, device_ids=[local_rank], output_device=local_rank)
opt_G = optim.SGD(G_ddp.parameters(), lr=0.001)
opt_D = optim.SGD(D_ddp.parameters(), lr=0.001)

G_train = GeneratorOperation(G_ddp, D_ddp)
D_train = DiscriminatorOperation(G_ddp, D_ddp)
</code></pre>
<p>or in</p>
<pre><code class="lang-auto">torch.cuda.set_device(local_rank)
G = Generator()
D = Discriminator()
G.cuda()
D.cuda()
opt_G = optim.SGD(G.parameters(), lr=0.001)
opt_D = optim.SGD(D.parameters(), lr=0.001)

G_train = DDP(GeneratorOperation(G, D), device_ids=[local_rank], output_device=local_rank)
D_train = DDP(DiscriminatorOperation(G, D), device_ids=[local_rank], output_device=local_rank)
</code></pre>
<p>Should I use one of the above over another, or they are the same? Appreciated if you can also explain in detail. Thanks!</p>

osalpekar: <p>Per the example here: <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/main.py" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples · GitHub</a>, the DDP model is created first and then it’s parameters are passed as an arg when creating the optimizer (like the first option shown).</p>
<p>A more specific answer would depend on what <code>GeneratorOperation</code> and <code>DiscriminatorOperation</code> do. If you want the functionality in these to be replicated across ranks and the corresponding gradients to be synchronized during the backward pass, then they should passed to DDP.</p>
How dose distributed sampler passes the value "epoch" to data loader?

haoshiming: <p>in ImageNet examples, the codes look like:</p>
<pre><code class="lang-auto">for epoch in range(all_epochs):
    train_sampler.set_epoch(epoch)
    train_one_epoch_with_train_loader()
</code></pre>
<p>how is the sampler.epoch <strong>inside</strong> train_loader set by the <strong>outside</strong> definition <code>train_sampler.set_epoch(epoch)</code>? Seems that the train_loader is not explicitly re-constructed in every epoch.<br>
Didn’t find much hints in /torch/utils/data<br>
Thanks!</p>

osalpekar: <p>The sampler is passed as an argument when initializing the DataLoader, so the train loader will have access to the sampler object. Neither the loader not the sampler need to be re-constructed every epoch.</p>
How to implement batch l2 normalization with pytorch

bannima: <p>hey guys, I’ m new to pytorch, I just want to know is there any pytorch API that<br>
can process the tensor with l2-normalization? In tensorflow, the corresponding API is tf.nn.l2_normalize.</p>

bannima: <p>I think I just got the answer.</p>
<ul>
<li>import torch.nn.functional as f</li>
<li>a = torch.randn(2,3)</li>
<li>norm_a = f.normalize(a,dim=0,p=2)</li>
</ul>
<p>where p=2 means the l2-normalization, and dim=0 means normalize tensor a with row.</p>

hoangcuong2011: <p>Thanks for the code. It should be -1 instead of 0 I think.<br>
<code>norm_a = f.normalize(a,dim=-1,p=2)</code></p>
Object expands in memory despite size remaining constant

humzaiqbal: <p>Hi, I’m running into the following issue. I have a list of tuples of the form</p>
<pre><code class="lang-auto">a = [(torch.Tensor(3), 4), (torch.Tensor(5), 6)]
</code></pre>
<p>and I want to add something of a similar form like</p>
<pre><code class="lang-auto">b= [(torch.Tensor(5), 6), (torch.Tensor(10), 3)]
</code></pre>
<p>I then want to add those two together to get something like</p>
<pre><code class="lang-auto">a = a + b
a = [(torch.Tensor(8), 10), (torch.Tensor(15), 9)]
</code></pre>
<p>when I do I notice that the memory of  <code>a</code> is larger than it was before despite the fact that its of the same size and I can’t figure out why it increases and what I can do to fix it (I’m doing this on a larger scale which leads to me running out of RAM)</p>
<p>One important thing to note that the torch tensors in questions here are the result of some forward passes in a network and as a result have <code>requires_grad</code> set to true. I suspect that the issue is related to the fact that additional memory is needed for the tensor to remember all the variables needed for it to compute gradients properly, does that seem right? If so is there a way I can mitigate increased memory usage or is it just inherent if I want to keep <code>requires_grad</code> to be true? (which is required for my problem</p>
<p>Any help would be appreciated, thanks!</p>

Reuben_Wenisch: <p>Can you print out the result of a. To me, it looks like the list will contain all 4 elements. You will have to add elementwise to get the result you need.</p>
load_state_dict(strict=False)

basingse: <p>Hello,<br>
I was experimenting with load_state_dict with strict = False. Before my asking my query, I will post my model code here</p>
<pre><code class="lang-auto">class AutoEncoder(nn.Module):
    def  __init__(self, input_dim):
        super(AutoEncoder, self).__init__()

        self.input_dim = input_dim
        # encoder
        self.batchnorm1 = nn.BatchNorm1d(self.input_dim)
        self.linear1 = nn.Linear(self.input_dim, 64)
        self.batchnorm2 = nn.BatchNorm1d(64)
        self.linear2 = nn.Linear(64, 32)
        # decoder
        self.batchnorm3 = nn.BatchNorm1d(32)
        self.linear3 = nn.Linear(32, 64)
        self.batchnorm4 = nn.BatchNorm1d(64)
        self.linear4 = nn.Linear(64, self.input_dim)
        
        # relu
        self.relu = nn.ReLU()
    
    def forward(self, h0):
        # encoder forward pass
        h1 = self.relu(self.linear1(self.batchnorm1(h0)))
        h2 = self.linear2(self.batchnorm2(h1))
        # decoder forward pass
        h3 = self.relu(self.linear3(self.batchnorm3(h2)))
        h4 = self.linear4(self.batchnorm4(h4))
        
        return h4
    
class Classifier(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(Classifier, self).__init__()

        self.input_dim = input_dim
        self.output_dim = output_dim
        # encoder
        self.batchnorm1 = nn.BatchNorm1d(self.input_dim)
        self.linear1 = nn.Linear(self.input_dim, 64)
        self.batchnorm2 = nn.BatchNorm1d(64)
        self.linear2 = nn.Linear(64, 32)
        
        self.batchnorm = nn.BatchNorm1d(32)
        self.classifer = nn.Linear(32, self.output_dim)
    
    def forward(self, h0):
        h1 = self.relu(self.linear1(self.batchnorm1(h0)))
        h2 = self.linear2(self.batchnorm2(h1))
        
        out = self.classifier(self.batchnorm(h2))
        return out
</code></pre>
<p>After this I intialized both the models as shown below</p>
<pre><code class="lang-auto">autoencoder_model = AutoEncoder(130)
classifier_model = Classifier(130, 1)
</code></pre>
<p>After this I tried this <code>classifier_model.load_state_dict(autoencoder_model.state_dict(), strict=False)</code>. That is it will load the encoder part of autoencoder weights to the classifier. This is the output I got (which shows the missing keys and unexpected keys)</p>
<pre><code class="lang-auto">_IncompatibleKeys(missing_keys=['batchnorm.weight', 'batchnorm.bias', 
'batchnorm.running_mean', 'batchnorm.running_var', 'classifer.weight', 'classifer.bias'],
 unexpected_keys=['batchnorm3.weight', 'batchnorm3.bias', 'batchnorm3.running_mean', 
'batchnorm3.running_var', 'batchnorm3.num_batches_tracked', 'linear3.weight', 
'linear3.bias', 'batchnorm4.weight', 'batchnorm4.bias', 'batchnorm4.running_mean',
 'batchnorm4.running_var', 'batchnorm4.num_batches_tracked', 'linear4.weight', 
'linear4.bias'])
</code></pre>
<p>The output also looks correct, but my query is <strong>why does “batchnorm.num_batches_tracked” didn’t come in missing keys</strong> ?</p>
Cyclic Learning rate - How to use

basingse: <p>I am using <code>torch.optim.lr_scheduler.CyclicLR</code> as shown below</p>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(),lr=1e-2,momentum=0.9)
optimizer.zero_grad()
scheduler = optim.lr_scheduler.CyclicLR(optimizer,base_lr=1e-3,max_lr=1e-2,step_size_up=2000)
for epoch in range(epochs):
       for batch in train_loader:
                X_train = inputs['image'].cuda()
                y_train = inputs['label'].cuda()
                y_pred = model.forward(X_train)
                loss = loss_fn(y_train,y_pred)
                with amp.scale_loss(loss,optimizer) as scaled_loss:
                    scaled_loss.backward()                         # backprop
                optimizer.step()
                optimizer.zero_grad()
                scheduler.step()
</code></pre>
<p>as cyclic lr updates learning rate after every batch, I have used scheduler.step inside the batch loop.  But when I do this I get this warning :</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:73: UserWarning: Seems like `optimizer.step()` has been overridden after learning rate scheduler initialization. Please, make sure to call `optimizer.step()` before `lr_scheduler.step()`. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
</code></pre>
<p>Can I know why this happens  ?</p>

ptrblck: <p>Your usage of the scheduler is generally fine and the warning is thrown, as we are patching the <code>optimizer.step</code> method in a similar way in <code>apex</code> as is done by the scheduler in <a href="https://github.com/pytorch/pytorch/blob/6ce6939be9b01965cc830f5d819010ac97bf2a1f/torch/optim/lr_scheduler.py#L40" rel="nofollow noopener">this line of code</a>.</p>
<p>To avoid this warning, initialize the scheduler after running <code>amp.initialize(model, optimizer, opt_level)</code>.</p>
<p>Also, if you want, you could also add this check to avoid changing the learning rate, if the optimization step was skipped due to a gradient overflow:</p>
<pre><code class="lang-python">    optimizer.step()
    if amp._amp_state.loss_scalers[0]._unskipped != 0: # assuming you are using a single optimizer
        scheduler.step()
</code></pre>

Mohammed_Awney: <p>I want to ask about tracking the progress of training, Are you plot loss value each epoch or each batch"iteration" ?</p>

basingse: <p>As far as I remember, I was tracking the loss value each epoch. Can I know how this is relevant to the query I asked.</p>
What is an efficient way to do a forward pass of a batch of data for an Graph Neural Network (GNN)?

Brando_Miranda: <p>I wanted to do a forward pass for a GNN (or Tree NN). I know that for vision one can process the entire batch in one go by having the batch be a dimension in the tensor. Is something like this possible for GNNs (Graph Neural Networks)? I am worried that having to do a forward pass for each example for my GNN might slow things down, but perhaps there is no other way.</p>
<p>I assume multithreading might be possible but I don’t know how that would work with multiple threads/processing using the same gpu…plus debugging sounds like a nightmare. But I am curious what is the standard practice for this since GNN research exists.</p>

Brando_Miranda: <p>these might be worth looking into:</p>
<ul>
<li><a href="https://github.com/rusty1s/pytorch_geometric" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rusty1s/pytorch_geometric: Geometric Deep Learning Extension Library for PyTorch</a></li>
<li><a href="https://www.dgl.ai/" rel="noopener nofollow ugc">https://www.dgl.ai/</a></li>
</ul>
<p>though I don’t know which one is the standard pytorch or recommended way to process graph data (or implement their forward passes). This feels like shouldn’t be something I need to implement from scratch.</p>
<hr>
<p>related: <a href="https://stackoverflow.com/questions/66052104/what-is-the-standard-way-to-batch-and-do-a-forward-pass-through-tree-structured" class="inline-onebox" rel="noopener nofollow ugc">python - What is the standard way to batch and do a forward pass through tree structured data (ASTs) in pytorch so to leverage the power of GPUs? - Stack Overflow</a></p>
Reset seed in pytorch

sadakmed: <p>I just set torch seed to 0 with <code>torch.manual_seed</code>, Is there any way to reset the seed.</p>
<p>my case was, I wanted to generate a fixed random tensor, without affecting the randomness of others(other tensors should be different each time).</p>
<p>I found a context manager in <code>torchfunc</code> module, but I don’t have access to it in HuggingFace</p>

sharvil: <p>You can use <code>torch.random.get_rng_state()</code> and <code>torch.random.set_rng_state(prev_state)</code>. See <a href="https://pytorch.org/docs/stable/random.html" class="inline-onebox" rel="noopener nofollow ugc">torch.random — PyTorch 1.7.0 documentation</a> for docs.</p>
<p>Another option is to use NumPy to generate your random fixed tensor.</p>
<pre><code class="lang-python">np.random.seed(0)
random_fixed_tensor = torch.as_tensor(np.random.normal(...))
</code></pre>
Pretrained Keras LSTM weights to PyTorch LSTM

cerebralnetworkdev: <p>Hello! A lot of posts on Keras → PyTorch LSTM seem to focus on retaining the model architecture, but I couldn’t find anything online here; the closest I’ve reached is this <a href="https://stackoverflow.com/questions/48361376/converting-state-parameters-of-pytorch-lstm-to-keras-lstm" rel="noopener nofollow ugc">SO post</a> but would love your help.</p>
<p>Essentially, I have a Keras LSTM that I want to migrate:</p>
<pre><code class="lang-auto"># Keras
model.add(
    Bidirectional(LSTM(256, dropout=0.15, recurrent_dropout=0.2, return_sequences=True), 
    merge_mode='concat')
)
o_keras = model(x)
</code></pre>
<p>In PyTorch, my implementation is simply</p>
<pre><code class="lang-auto">lstm = nn.LSTM(256, bidirectional = True, batch_first=True)
o_pytorch, (h, c) = lstm(x)
</code></pre>
<p>The output tensor <em>shapes</em> are identical, which is a good sign:</p>
<pre><code class="lang-auto">o_keras.shape == o_pytorch.shape # TRUE
</code></pre>
<p>However, I’m having trouble migrating weights and biases. Setting the <em>weights</em> is pretty straight forward as it’s essentially some matrix transpositions between Keras and PyTorch (e.g. Keras will have one set of weights that are (n_hidden_cells x num_positions), so the transpose is what you need for PyTorch).</p>
<p>The one that’s causing me some headache is that PyTorch has <em>two</em> bias terms but Keras has <strong>one</strong>.</p>
<p>Judging by the equation in the docs, my understanding is that we can set <code>bias_ih_l0</code> as the value from the pre-trained model and the corresponding <code>bias_hh_l0</code> to 0 as it’s a sum. However, adjusting the values (e.g. using something like this below) for the bias term gives slightly different results in the output.</p>
<pre><code class="lang-auto">alpha = [1.0, 0.9, 0.8 ... ]
lstm.bias_ih_l0 = keras_weights['bias'] * alpha
lstm.bias_hh_l0 = keras_weights['bias'] * (1-alpha)
</code></pre>
<p>I also want to ask if the <code>tanh</code> activation from Keras’ LSTM is something we have to call as a separate step, e.g.</p>
<pre><code class="lang-auto">o_pytorch = lstm(x)
# o_pytorch = nn.Tanh()(o_pytorch) # is this necessary?
</code></pre>
<p>Would really appreciate your help. Thanks everyone!</p>
Caffe framework code equivalent in PyTorch

lords: <p>Hello there.</p>
<p>I’m trying for a while now to implement  <a href="https://github.com/KamitaniLab/End2EndDeepImageReconstruction" rel="noopener nofollow ugc">this</a> Caffe code in PyTorch, since I am more familiar with the latter.</p>
<p>I am having trouble understanding this specific step in the training loop and how to implement it in pytorch.</p>
<p><code>generator.net.blobs['generated'].diff[...] = encoder.net.blobs['data'].diff + discriminator.net.blobs['data'].diff</code></p>
<p>The complete architecture consists of a generator a discriminator and a comparator.</p>
<p>I am more interested in understanding what this command does intuitively and what is the equivalent approach in pytorch, rather than the network’s specifics.</p>
<p>Thanks in advance</p>
Why do we use .pow(2) and not **2?

pinocchio: <p>I would have thought that pytorch overwrote such syntax <code>**2</code> to mean <code>.pw(2)</code> automatically. Is that not the case? Can I do <code>**2</code> and have things work correctly (gradients etc) in pytorch?</p>

tom: <p>I always use <code>**2</code> and have not run into problems and I would expect it to roughly have the same performance, too (for operators, python needs to figure out whose special function it should call). I suspect that people coming from other languages might not be used to the convenience of a power operator.</p>
<p>Best regards</p>
<p>Thomas</p>

pinocchio: <p>I assume it does the element wise square?</p>

colesbury: <p>Yes it’s element-wise.  In PyTorch, <code>x.pow(2)</code> and <code>x ** 2</code> are equivalent:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/6f6d70ffeda9b9918b8fcd02ffea28b126e283e3/torch/tensor.py#L320-L321" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6f6d70ffeda9b9918b8fcd02ffea28b126e283e3/torch/tensor.py#L320-L321" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/6f6d70ffeda9b9918b8fcd02ffea28b126e283e3/torch/tensor.py#L320-L321</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="320" style="counter-reset: li-counter 319 ;">
<li>def __pow__(self, other):</li>
<li>    return self.pow(other)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The <code>**</code> operator (<code>__pow__</code>) just calls <code>pow()</code>.</p>

kidd_lee: <p>In some cases, the member function (x.pow()) is more elegant than the expression mode (x**2).  Given a tensor <code>a</code> defined as:</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3]).
</code></pre>
<p>Calling a member function like</p>
<pre><code class="lang-auto">a.pow(2).mean()
</code></pre>
<p>should be cleaner than</p>
<pre><code class="lang-auto">(a**2).mean().
</code></pre>
<p>Calling member function is <strong>object oriented</strong> while bracket turn the expression it into a more <strong>functional style</strong> programming.<br>
Following case might be rare but shows the advantages more clearly:</p>
<pre><code class="lang-auto">((a**2).add(1)**2).mean()
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">a.pow(2).add(1).pow(2).mean()
</code></pre>
<p>As the expression gets more complicated, member function shows more advantages.</p>
How to quickly transfer data from cuda to cpu with libtorch?

cyanM: <p>I use a resnet to classify images, and find that it is slower to get the result back to the CPU than forward(), and far slower than put image data to CUDA as well.<br>
Does anyone know how to increase the speed of transferring data back to cpu?</p>
<p>os:<br>
windows 7<br>
cuda9.2<br>
libtorch1.0-nightly-release<br>
visual studio 2019</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/089a59bbbf6c21ad66049c4b5ea83ca9f6be1385.jpeg" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="1e6vSHmlcsFWxcLQ641nf46WsEl" width="300" height="66"></p>
<p>relative code is as follows:</p>
<p>t1 = std::chrono::steady_clock::now();</p>
<p>at::Tensor tmpData2 = torch::from_blob(tmp, { resNetParam.batchSize, resNetParam.roiSize,resNetParam.roiSize,resNetParam.imgDepth }, torch::kFloat).to(m_device);<br>
tmpData2 = tmpData2.permute({ 0, 3, 1, 2 });</p>
<p>t2 = std::chrono::steady_clock::now();<br>
time_used = std::chrono::duration_cast&lt;std::chrono::duration&gt;(t2 - t1) * 1000;<br>
printf(“putIntoCuda %.3f ms \n”, time_used);</p>
<p>t1 = std::chrono::steady_clock::now();</p>
<p>torch::Tensor out = m_model-&gt;forward({ tmpData2 }).toTensor();</p>
<p>t2 = std::chrono::steady_clock::now();<br>
time_used = std::chrono::duration_cast&lt;std::chrono::duration&gt;(t2 - t1) * 1000;<br>
printf(“forward %.3f ms \n”, time_used);</p>
<p>std::tuple&lt;torch::Tensor, torch::Tensor&gt; result = out.sort(-1, true);</p>
<p>t1 = std::chrono::steady_clock::now();</p>
<p>torch::Tensor sortedScores = std::get&lt;0&gt;(result).to(torch::kCPU);<br>
torch::Tensor sortedIdx = std::get&lt;1&gt;(result).toType(torch::kInt32).to(torch::kCPU);</p>
<p>t2 = std::chrono::steady_clock::now();<br>
time_used = std::chrono::duration_cast&lt;std::chrono::duration&gt;(t2 - t1) * 1000;<br>
printf(“toCPU %.3f ms \n”, time_used);</p>

tom: <p>These measurements are quite likely off because you don’t seem to synchronize between calls. That means that the part measured as “toCPU” likely involves bits from before.</p>
<p>Best regards</p>
<p>Thomas</p>

cyanM: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f5d2797e857814d1176637b96f740b69fa2d60e1.png" alt="13E67AA7-5DE6-40c9-A29C-6E61F6F93CD3" data-base62-sha1="z4DV2pBjnWgtuGPkkIoQNkNbxol" width="292" height="60"><br>
you’re right! thx</p>

EsdeathYZH: <p>I’m new to libtorch and what’s the meaning of “synchronize between calls”?<br>
Could you share the modified code? Thank you!</p>

cyanM: <p>Just add cudaDeviceSynchronize() after ‘forward()’.<br>
(according to nvidia doc, cudaDeviceSynchronize() could be used to wait for GPU compute to finish)<br>
However,  this command has no effect on the speed of the execution,just let cpu count the elapsed time correctly.</p>

kevin_wang: <p>The forward time is so slow?</p>

cyanM: <p>Generally ,yes. It depends on your net and input tensor’s size.</p>

FantasyJXF: <p>I met the same problem, <code>pytorch</code> code works fine but <code>libtorch</code> very slow, no matter whether I add <code>c10::cuda::CUDACachingAllocator::emptyCache();</code>  or not.</p>
Time-series predictions from a dataloader instead of list

krishnab: <p>I am using an LSTM for some time-series modelling. The model runs fine and I want to generate future predictions from my model. I have the code below that will generate N predictions into the future. Basically loop will take an input series, make a prediction on the t+1 step, and then append that step to the end of the input series. The loop will then include that prediction when prediction the t+2 step, and so forth, until I get to t+N steps.</p>
<p>The code looks like this, but it does not feel very polished from a numerical perspective. Since I am using a list to hold the input series, there are memory constraints on how far forward I can predict and how many series I can predict, etc. <strong>My question was, is there a way to write this code below using a DataLoader, instead of a list–meaning, can I dynamically add elements on to the end of a DataLoader?</strong> That way I imagine I can use an iterator instead of a list and hence not worry about memory, etc.</p>
<pre><code class="lang-auto">def generate_predictions(prediction_window: int, data: list, model: nn.Model):
    model.eval()
    for i in range(prediction_window):
      seq = torch.FloatTensor(data[-prediction_window:])
      seq = seq[np.newaxis, :, np.newaxis]
      #import pdb; pdb.set_trace()
      with torch.no_grad():
        seq = seq.to(device)
        data.append(model(seq).item())
    model.train()
</code></pre>

bkaankuguoglu: <p>I see it’s been quite some time since you posted this, but I’ve recently run into the same issue. Have you found a solution to it?</p>
Question about error

Hedi_Fekih: <p>i have this error, i tried a lot of times and i see all forum and not yet solved<br>
imporerror: cannot import dataloader</p>
<p>any help please,</p>

ptrblck: <p>Could you post the line of code, which raises the error as well as the PyTorch version you are using, please?</p>

Hedi_Fekih: <p>i solve this error by changing dataloader to DataLoader</p>
Embedding categorical similarities with continuous data

JoshP: <p>Hi, I am working on a project where I am initialising an LSTM with objects detected in an image:</p>
<p>For example I have categories: <code>dog, person, car, ... -&gt; [1,3,0,...]</code> where the value at each index corresponds to the numbers of detections for that category in the image (so 1 dog, 3 people, 0 cars …).</p>
<p>Currently I am just using this frequency count vector to initialise the LSTM hidden/cell state: <code>[1,3,0,...] -&gt; Linear layer -&gt; LSTM init</code></p>
<p>I am using a object detector with 1,600 categories, some of the categories are similar such as car, bus and truck. I am looking for a way to represent these similarities using <code>nn.Embeddings</code>. My plan is to use bert to encode object labels but I am wondering how I can use the bert embeddings with the objects frequency (<code>[1,3,0,...]</code>) to generate a 1D vector to initialise the LSTM.</p>
Tensor in float16 is transformed into float32 after torch.norm

wuchiz: <p>I trained the mode in mixed precision. in certain function, a input tensor in float16 is transformed into float32 after torch.norm<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa09a045a9caa08b97875665083f8e1a4e1c3e1e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/aa09a045a9caa08b97875665083f8e1a4e1c3e1e" title="截屏2021-02-04 下午4.36.25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa09a045a9caa08b97875665083f8e1a4e1c3e1e_2_690x487.jpeg" alt="截屏2021-02-04 下午4.36.25" data-base62-sha1="ogdNd3YpYTpD2bPqhHshE53Zw62" width="690" height="487" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa09a045a9caa08b97875665083f8e1a4e1c3e1e_2_690x487.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa09a045a9caa08b97875665083f8e1a4e1c3e1e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa09a045a9caa08b97875665083f8e1a4e1c3e1e.jpeg 2x" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-04 下午4.36.25</span><span class="informations">974×688 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
dis_vec (float16)  dis (float32)<br>
and if I use dis=torch.sqrt((dis_vec**2).sum(-1)), dis is also float32</p>

ptrblck: <p><code>autocast</code> uses an internal “allow-list” to cast tensors into <code>float16</code>, if the operation is considered save using this precision.<br>
The <a href="https://pytorch.org/docs/stable/amp.html#torch.cuda.amp.autocast"><code>autocast</code> docs</a> give you some more information.<br>
Are you seeing any <code>dtype</code> mismatch errors in your code (inside the <code>autocast</code> region) as this could be a bug?</p>

wuchiz: <p>no mismatch errors. in principle, op torch.norm will not change the dtype, so why do the output tensor become float32 even though my input tensor is float16?</p>

ptrblck: <p><code>torch.norm</code> is an op, which autocasts to <code>float32</code> as given in <a href="https://pytorch.org/docs/stable/amp.html#ops-that-can-autocast-to-float32">this list</a>.</p>

wuchiz: <p>Thanks! so if i want to continue the following computation with float16, I need set it float16 manually by dis.half()?</p>

ptrblck: <p>If you want to apply the <code>norm</code> using <code>float16</code> values and are sure that you won’t run into numerical issues, you could disable <code>autocast</code> for this operation and manually cast it to the desired type (you can use nested <code>autocast</code> decorators).</p>