akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pad_sequence failing troubleshoot "The expanded size of the tensor () must match the existing size () at non-singleton dimension 2."

Ajna_Ram: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/59719710/pad-sequence-failing-troubleshoot-the-expanded-size-of-the-tensor-must-match" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
    <h4>
  <a href="https://stackoverflow.com/questions/59719710/pad-sequence-failing-troubleshoot-the-expanded-size-of-the-tensor-must-match" target="_blank" rel="nofollow noopener">pad_sequence failing troubleshoot "The expanded size of the tensor () must match the existing size () at non-singleton dimension 2."</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/12683500/anjalla" target="_blank" rel="nofollow noopener">
    Anjalla
  </a>
  on <a href="https://stackoverflow.com/questions/59719710/pad-sequence-failing-troubleshoot-the-expanded-size-of-the-tensor-must-match" target="_blank" rel="nofollow noopener">03:37PM - 13 Jan 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>Hi,</p>
<p>According to the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence">doc</a>, pad sequence only performs padding on a single dimension (either first or second depending on the <code>batch_first</code> argument). All the other dimensions after this should have the same size.<br>
In your example, the third dimension is 1444 for one Tensor and 1936 for the other.</p>

Ajna_Ram: <p>Ahh that explains it! Thanks a lot.<br>
Does anyone what function would be more suited for what i’m looking for?(padding images to match their size)</p>

albanD: <p>I guess you can modify the original <code>pad_sequence</code> that you can find <a href="https://github.com/pytorch/pytorch/blob/8e93159fb65edba0934da76772ea4f77d3590156/torch/nn/utils/rnn.py#L325-L344">here</a> for your needs in the 2D case. By replacing things like <code>out_tensor[i, :length, ...] = tensor</code> by <code>out_tensor[i, :height, :width ...] = tensor</code>.</p>

Ajna_Ram: <p>I will try that! Thanks a lot!</p>
RuntimeError: The expanded size of the tensor (16320) must match the existing size (6375) at non-singleton dimension 1. Target sizes: [4, 16320, 4]. Tensor sizes: [4, 6375, 1]

111215: <p>Experts, I have checked some data, some people say it is the pytorch version of the problem, I am not very clear about this, may I ask what valuable advice?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/299feee04f14e218ae41646e3391252cbed843b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/299feee04f14e218ae41646e3391252cbed843b5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/299feee04f14e218ae41646e3391252cbed843b5_2_690x123.png" alt="image" data-base62-sha1="5WeeluiVeZViZaSyJxPYUBAlCh7" width="690" height="123" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/299feee04f14e218ae41646e3391252cbed843b5_2_690x123.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/299feee04f14e218ae41646e3391252cbed843b5_2_1035x184.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/299feee04f14e218ae41646e3391252cbed843b5_2_1380x246.png 2x" data-dominant-color="F6F3F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2717×486 95.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>It looks like the Tensor given to the criterion have sizes that do not match.<br>
What is your criterion? What are the size of the Tensors that should be given to it?</p>
CUDA error when changing the machine

smu226: <p>Hello! I have these lines of code (part of a larger code, but this is where I am getting the error):</p>
<p><code>from torch.distributions import MultivariateNormal</code><br>
<code>prior = MultivariateNormal(torch.zeros(data.size(1)).cuda(), torch.eye(data.size(1)).cuda())</code></p>
<p>and I am getting this error (for the second line):</p>
<pre><code class="lang-auto">File "/path/to/lib/python3.6/site-packages/torch/distributions/multivariate_normal.py", line 149, in __i\
nit__
    self._unbroadcasted_scale_tril = torch.cholesky(covariance_matrix)
RuntimeError: CUDA error: no kernel image is available for execution on the device
</code></pre>
<p>I ran the exactly same code on a different machine, and it works fine there, so I am not sure what is wrong. Can someone help me? Thank you!</p>

albanD: <p>Hi,</p>
<p>This happens because the pytorch you’re running does not have the code to run on this GPU.<br>
If you compiled from source, you want to recompile while this GPU is visible (or specify the correct architecture with this option: <a href="https://github.com/pytorch/pytorch/blob/67c1d930eb51626d282e93818200977acc6945a9/setup.py#L90">https://github.com/pytorch/pytorch/blob/67c1d930eb51626d282e93818200977acc6945a9/setup.py#L90</a>).<br>
If you use a binary, make sure that you have the right binary for your cuda version.</p>

smu226: <p>Thank you for your reply! I am not compiling anything. I am just using a python code from GitHub and running it the way it is, with the exception of adding .cuda() to some of the parameters. Here is the link: <a href="https://github.com/tonyduan/normalizing-flows" rel="nofollow noopener">https://github.com/tonyduan/normalizing-flows</a> As I said, adding .cuda() woks just fine on the other machine, but not on the one I am using right now. However the machine I am using right now, handles .cuda() for other scripts (I ran lots of code on GPU on this machine), I am not sure what is wrong with this particular one.</p>

albanD: <p>How did you install pytorch on that particular machine?<br>
Can you try in the command line to create a Tensor, send it to the GPU and try to pass it to <a href="https://pytorch.org/docs/stable/torch.html#torch.cholesky"><code>torch.cholesky</code></a> ? To make sure this is the problematic function.</p>
Access z-axis from tensor 4d

RicLaGra: <p>Hello everyone!<br>
How can i get values from z-axis of a tensor 4d (example 32,128,4,4).<br>
In this case i would like to get 128 value (planes) from the first 4x4 matrix and so on.<br>
Anyone known the right operations to do that?</p>
<p>Thanks in advance!</p>

ptrblck: <p>I’m not sure I understand the use case correctly.<br>
To access values in a tensor, you can use indexing as:</p>
<pre><code class="lang-python">x = torch.randn(32, 128, 4, 4)
print(x[0, 0])
</code></pre>
<p>This would print the “first” <code>4x4</code> tensor.</p>
<p>Could you give an example, how your output should look like, please?</p>

RicLaGra: <p>Ok, i try to explain myself correctly.<br>
In the image below, i would like to extract a tensor from each N plane (red points) starting from the first column/rown and so on. So, in this way i have to obtain X tensors composed by each planes (conv2). Is it possible with pytorch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0ba18f30d905aa73173eac107cdd382729b306d.png" data-download-href="https://discuss.pytorch.org/uploads/default/c0ba18f30d905aa73173eac107cdd382729b306d" title="LeNet-5-1998"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0ba18f30d905aa73173eac107cdd382729b306d_2_690x264.png" alt="LeNet-5-1998" data-base62-sha1="ruWnWUlHGYAFaWqDNiVAD8QsWL3" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0ba18f30d905aa73173eac107cdd382729b306d_2_690x264.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0ba18f30d905aa73173eac107cdd382729b306d_2_1035x396.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0ba18f30d905aa73173eac107cdd382729b306d_2_1380x528.png 2x" data-dominant-color="EBF2F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">LeNet-5-1998</span><span class="informations">1552×595 41.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nicolalandro: <p>This is a sample:</p>
<pre><code class="lang-python">import torch

x = torch.tensor(
    [
        [
            [[1, 2], [3, 4], ],
            [[5, 6], [7, 8], ],
            [[9, 10], [11, 12], ],
        ],
        [
            [[13, 14], [15, 16], ],
            [[17, 18], [19, 20], ],
            [[21, 22], [23, 24], ],
        ]
    ]
)

print(x.shape)
# torch.Size([2, 3, 2, 2])

print(x.reshape(2, 3, 4).transpose(2, 1).reshape(8, 3))
# tensor([[ 1,  5,  9],
#         [ 2,  6, 10],
#        [ 3,  7, 11],
#        [ 4,  8, 12],
#       [13, 17, 21],
#      [14, 18, 22],
#     [15, 19, 23],
#    [16, 20, 24]])

</code></pre>
<p>generally:</p>
<pre><code class="lang-python">a, b, c, d = x.shape
x.reshape(a, b, c*d).transpose(2, 1).reshape(a*c*d, b)
</code></pre>

RicLaGra: <p>perfect, thanks a lot!</p>
Is there any essential difference between * and + for Pytorch autograd?

entslscheia: <p>I was trying to understand the autograd mechanism in more depth. To test my understanding, I tried to write the following code which I expected to produce an error (i.e., <strong>Trying to backward through the graph a second time</strong>).</p>
<pre><code class="lang-auto">b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b + a
    c = b*a
    c.backward()
</code></pre>
<p>Apparently, it should report an error when <code>c.backward()</code> is called for the second time in the for loop, because the history of b has been freed， however, nothing happens.</p>
<p>But when I tried to change b + a to b * a as follows,</p>
<pre><code class="lang-auto">b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b * a
    c = b*a
    c.backward()
</code></pre>
<p>It did report the error I was expecting.<br>
This looks pretty weird to me. I don’t understand why there is no error evoked for the former case, and why it makes a difference to change from + to *.</p>

tom: <p>Don’t use <code>Tensor(...)</code> or <code>.data</code> any more! They have been deprecated since PyTorch 0.4.</p>
<p>the reason you are not getting an error for the first is that the <code>+</code> backward doesn’t need any saved tensors while the multiplication needs the other argument to compute the input gradient from the output gradient. These are the ones getting freed in the backpropagation and so you get the error.</p>
<p>Best regards</p>
<p>Thomas</p>
What is @staticmethod?

f3ba: <p>In the docs it is recommended that when implementing <code>forward</code> and <code>backward</code> passes for a custom function, they be marked as <code>@staticmethod</code>. For example: <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/extending.html</a>.</p>
<p>What does the annotation <code>@staticmethod</code> do? I can’t find an explanation in the docs.</p>

tom: <p>Static methods are methods not attached to a particular instance - so they do take a <code>self</code> as first argument. They’re not PyTorch-specific but a general Python thing:<br>
<a href="https://docs.python.org/3.5/library/functions.html#staticmethod" class="onebox" target="_blank" rel="nofollow noopener">https://docs.python.org/3.5/library/functions.html#staticmethod</a></p>
<p>PyTorch’s autograd Functions store state in a special context object (passed as the first argument) rather than the Function object, that is why they’re static. (In ancient times before PyTorch 0.2 this used to be different.)</p>
<p>Best regards</p>
<p>Thomas</p>
Could for loop for a row in matrix be avoided where there's dependency between rows?

DLiang: <p>Hello, I have a simple question for you but confused me a lot.<br>
Suppose I got a tensor A in [batch_size, n, m], and I need to get a tenor B from A in the same shape, for B[:, i, :] = f(B[:, i-1, :], A[:, i, :]), the f could be any function that complex enough. The initial state of B, namely B[:, 0, :] could be any, say, all zero vector.</p>
<p>To get B, I need a for loop:<br>
for i in range(A.shape[1]):<br>
B[:, i, :] = f(B[:, i-1, :], A[:, i, :])</p>
<p>However this cost much time espetially when A.shape[1] is very large for the backpropogation of gradient.</p>
<p>For example, one f I meet is:    B[:, i, :] = (B[:, i - 1, :] + cat([zeros(batch_size, A.shape[1], 0), B[:, i - 1, : -1]], -1)) * A[:, i, :]</p>
<p>How to accelerate or avoid this for loop?</p>
<p>Thank you for your answer~~~~</p>
Winograd Convolution in Pytorch

MinahilRaza: <p>Has Winograd convolution been implemented in Pytorch? Is there a way that I can replace Conv2d with its Winograd implementation?<br>
I want to compare the run times of Winograd and simple convolution.</p>

tom: <p>PyTorch dispatches to multiple backends like CuDNN or MKLDNN which in turn use Winograd depending (mainly) on the kernel size.<br>
PyTorch’s design is to abstract the implementation details and does not support targeted invocation of the Winograd variants.<br>
The usual way to do such benchmarking would be to implement the functions you want to benchmark in a C++ extension.</p>
<p>Best regards</p>
<p>Thomas</p>
Product of two vectors return a matrix

abd: <p>Hi,<br>
I have v1 and v2 two vector:<br>
v1=torch.tensor([1 ,0 ,0 ,0])<br>
v2=torch.tensor([1, 0, 0, 0])<br>
The resultat will be v3=v1xv2=[[1 0 ,0 ,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]] ,v3 de size(4x4).<br>
hawo I implement this in PyTorch.</p>
<p>Thanks</p>

KFrank: <p>Hi Gues!</p>
<aside class="quote no-group" data-username="abd" data-post="1" data-topic="66445" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c6cbf5/48.png" class="avatar"> abd:</div>
<blockquote>
<p>v1=torch.tensor([1 ,0 ,0 ,0])<br>
v2=torch.tensor([1, 0, 0, 0])<br>
The resultat will be v3=v1xv2=[[1 0 ,0 ,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]] ,v3 de size(4x4).</p>
</blockquote>
</aside>
<p>You are asking for the <em>outer product</em> of two vectors.  pytorch has this<br>
function under the name <code>ger()</code>.  So try:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.ger (v1, v2)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Eta_C: <pre><code class="lang-python">result = v1[..., None] @ v2[None, ...]
</code></pre>

Hong_Cheng: <p>I think you should reshape your tensor first.<br>
**convert your tensor as shape of **<code> (4,1</code>) and <code>(1,4)</code></p>
<pre><code class="lang-auto">
In [16]: a = torch.rand((4,1))                                                  

In [17]: b = torch.rand((4,1))                                                  

In [18]: b.T.mm(a)                                                              
Out[18]: tensor([[0.7928]])

In [19]: a.mm(b.T)                                                              
Out[19]: 
tensor([[0.2572, 0.3554, 0.0888, 0.1154],
        [0.2528, 0.3493, 0.0873, 0.1134],
        [0.4203, 0.5808, 0.1451, 0.1886],
        [0.0919, 0.1270, 0.0317, 0.0412]])
</code></pre>

abd: <p>Thanks <a class="mention" href="/u/hong_cheng">@Hong_Cheng</a><br>
It works</p>
Multi attribute multi class classification

jinfagang: <p>Say I have 2 attributes which is color and direction.</p>
<p>In color we have blue, red, green,<br>
In direction we have left, right, front, back.</p>
<p>So How to design a classifier to make it work output 2 attributes and according class?</p>
<p>I know that we using 3x3=9 classes to classify is a solution, but that not very good seems. Does there any better way to do this?</p>

ptrblck: <p>You could use two classification “heads” and return two outputs.<br>
These outputs would separately compute the losses, which you could sum and <code>backward</code> together.</p>
ModuleNotFoundError: No module named 'utils.nms.cpu_nms'

111215: <p>Experts, why is that?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad6ce67b63c7a1ce77da58806dd2c4b96f961327.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad6ce67b63c7a1ce77da58806dd2c4b96f961327" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6ce67b63c7a1ce77da58806dd2c4b96f961327_2_642x500.png" alt="image" data-base62-sha1="oKbV3qPaDEOuAZxTuSmWphYIJgj" width="642" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6ce67b63c7a1ce77da58806dd2c4b96f961327_2_642x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6ce67b63c7a1ce77da58806dd2c4b96f961327_2_963x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad6ce67b63c7a1ce77da58806dd2c4b96f961327_2_1284x1000.png 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2627×2043 577 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c3d75368249922449947d7edeb3739d2340245b.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c3d75368249922449947d7edeb3739d2340245b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c3d75368249922449947d7edeb3739d2340245b_2_642x500.png" alt="image" data-base62-sha1="8AUjUh0roQ8szeOdxiI3egRpbnl" width="642" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c3d75368249922449947d7edeb3739d2340245b_2_642x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c3d75368249922449947d7edeb3739d2340245b_2_963x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c3d75368249922449947d7edeb3739d2340245b_2_1284x1000.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2627×2043 616 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Eta_C: <p>Complie this Cython code first.<br>
<strong>This problem is not related to PyTorch</strong>.</p>

111215: <p>Ok, Roger that, thank you</p>
Why is subclass of Model.__init__ never executed?

Bruce_Ho: <p>I am running the fastai TabularModel and placed some print statements in the <strong>init</strong>() method. The print statements in the forward method starting dumping out text, but the ones in <strong>init</strong> never showed up. I tried setting some self.xxx variables, and even put in a assert statement, none fired while forward kept on getting called.</p>
<p>There is something strange about how this works. Can someone throw some lights into this?</p>

ptrblck: <p>Could you post a code snippet to reproduce this issue?</p>

Bruce_Ho: <pre><code class="lang-auto">class TabularModel(Module):
    "Basic model for tabular data."
    def __init__(self, emb_szs:ListSizes, n_cont:int, out_sz:int, layers:Collection[int], ps:Collection[float]=None,
                 emb_drop:float=0., y_range:OptRange=None, use_bn:bool=True, bn_final:bool=False):
        super().__init__()
              
        self.emb_szs = emb_szs
        b = 0
        assert b != 0, "force an assertion to see who passed data_collate"  
        ps = ifnone(ps, [0]*len(layers))
        ps = listify(ps, layers)
        print('emb_szs={}'.format(emb_szs))
        for ni,nf in emb_szs:
            print('ni, nf={},{}'.format(ni, nf))
        self.embeds = nn.ModuleList([embedding(ni, nf) for ni,nf in emb_szs])
        print('self.embeds={}'.format(self.embeds))
        self.emb_drop = nn.Dropout(emb_drop)
        self.bn_cont = nn.BatchNorm1d(n_cont)
        n_emb = sum(e.embedding_dim for e in self.embeds)
        print('n_emb={}'.format(n_emb))
        self.n_emb,self.n_cont,self.y_range = n_emb,n_cont,y_range
        sizes = self.get_sizes(layers, out_sz)
        actns = [nn.ReLU(inplace=True) for _ in range(len(sizes)-2)] + [None]
        layers = []
        for i,(n_in,n_out,dp,act) in enumerate(zip(sizes[:-1],sizes[1:],[0.]+ps,actns)):
            layers += bn_drop_lin(n_in, n_out, bn=use_bn and i!=0, p=dp, actn=act)
        if bn_final: layers.append(nn.BatchNorm1d(sizes[-1]))
        self.layers = nn.Sequential(*layers)
</code></pre>
<p>All the assert and print statements have no effect, yet the <strong>init</strong> must have executed, otherwise the self. variables would not have been set, and the forward() call would have bombed out.</p>

ptrblck: <p>Could you post a link to the implementation?<br>
I’m currently not sure, if <code>Module</code> refers to <code>nn.Module</code> or a custom implementation in FastAI.</p>
<p>Of course removing all undefined variables, works fine:</p>
<pre><code class="lang-python">class TabularModel(nn.Module):
    "Basic model for tabular data."
    def __init__(self):
        super().__init__()
        b = 0
        assert b != 0, 'test'      
</code></pre>
<p>To debug this issue, we would need an executable code snippet.</p>
Info about binary cross entropy with logits

nivter: <p>The function <code>torch.nn.functional.binary_cross_entropy_with_logits</code> actually returns a call to the function <code>torch.binary_cross_entropy_with_logits</code>. But I can’t find any information about it. It doesn’t have any docstring either. What is the actual code that is called and how is it called?</p>

tenzo: <p>Hi, this is pretty old question but I think I found its definition:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L96-L112" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L96-L112" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L96-L112</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="96" style="counter-reset: li-counter 95 ;">
<li>Tensor binary_cross_entropy_with_logits(const Tensor&amp; input, const Tensor&amp; target, const Tensor&amp; weight, const Tensor&amp; pos_weight, int64_t reduction) {</li>
<li>    Tensor loss;</li>
<li>    auto max_val = (-input).clamp_min_(0);</li>
<li>    if (pos_weight.defined()) {</li>
<li>        // pos_weight need to be broadcasted, thus mul(target) is not inplace.</li>
<li>        auto log_weight = (pos_weight - 1).mul(target).add_(1);</li>
<li>        loss = (1 - target).mul_(input).add_(log_weight.mul_(((-max_val).exp_().add_((-input - max_val).exp_())).log_().add_(max_val)));</li>
<li>    } else {</li>
<li>        loss = (1 - target).mul_(input).add_(max_val).add_((-max_val).exp_().add_((-input -max_val).exp_()).log_());</li>
<li>    }</li>
<li>
</li>
<li>    if (weight.defined()) {</li>
<li>        loss.mul_(weight);</li>
<li>    }</li>
<li>
</li>
<li>    return apply_loss_reduction(loss, reduction);</li>
<li>}</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

solsol: <p>Hi,</p>
<p>My problem with using binary cross entropy is due to input and target shape.<br>
input size is [n,m] however target size is [n]. By upsampling target size to [n,m], I got “zero-Devision-Error-scaled” error in backward. Any help please?</p>
RuntimeError: derivative for _thnn_fused_gru_cell_backward is not implemented

halah: <p>Have no idea about this error.<br>
_thnn_fused_gru_cell_backward can be found in pytorch/tools/autograd/derivatives.yaml, so I assumed it had been implemented. Am I right?<br>
Are there any other possible reasons for this error to occur?</p>
<p>Thanks,<br>
Yann</p>

tom: <p>No, it only occurs <a href="https://github.com/pytorch/pytorch/blob/3803d1c901b6e2632682b02589b2d5aa888b0a88/tools/autograd/derivatives.yaml#L1484" rel="nofollow noopener">as the backward of <code>_thnn_fused_gru_cell</code></a>.<br>
If it had a line with <code>- name: _thnn_fused_gru_cell_backward(...)</code> then it would be differentiable itself.</p>
<p>Best regards</p>
<p>Thomas</p>

halah: <p>Hi Thomas,</p>
<p>Thanks for your reply.<br>
Does this mean I have to implement the backward function?</p>
<p>Thanks,<br>
Yann</p>

tom: <p>Well, so we don’t have RNN double backwards at the moment.<br>
If you wanted that, you would have to implement it yourself or implement an RNN cell yourself.</p>
<p>Best regards</p>
<p>Thomas</p>

Peter_Ham: <p>Do you mean I implement a cell as regular nn.module then it would automatically work?</p>

tom: <p>Yes, exactly.</p>
<p>Best regards</p>
<p>Thomas</p>
Torchtext may influence the function of torch

Hongzhi-Chen: <p>I have try more times that the torch.cuda.is_available() present None after using the torchtext package in Windows 10 system.</p>

zhangguanheng66: <p>That’s very wired. Could you open an issue on PyTorch Github and ping <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a>.</p>

Hongzhi-Chen: <p>Thank you,this question has been solved.Because pytorch edition was forced to update to cpu edition when I installed torchtext package in IDE environment.</p>
Preprocess images before feed into CNN

copyrightly: <p>I am doing some experiments on MNIST. I want to preprocess images before feed them into CNN. My preprocess function is like</p>
<pre><code class="lang-auto">def preprocess(image):
    ...
    return processed_image
</code></pre>
<p>the result is a processed image with the same size as the original image.<br>
I am trying to modify this code: <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" rel="nofollow noopener">pytorch examples mnist</a><br>
I am wondering where I should add this preprocess function into the code. Thank you!</p>

Eta_C: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/2796747f58516574f9fbdb2b62b709f034676060/mnist/main.py#L100-L112" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/2796747f58516574f9fbdb2b62b709f034676060/mnist/main.py#L100-L112" target="_blank" rel="nofollow noopener">pytorch/examples/blob/2796747f58516574f9fbdb2b62b709f034676060/mnist/main.py#L100-L112</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="100" style="counter-reset: li-counter 99 ;">
<li>train_loader = torch.utils.data.DataLoader(</li>
<li>    datasets.MNIST('../data', train=True, download=True,</li>
<li>                   transform=transforms.Compose([</li>
<li>                       transforms.ToTensor(),</li>
<li>                       transforms.Normalize((0.1307,), (0.3081,))</li>
<li>                   ])),</li>
<li>    batch_size=args.batch_size, shuffle=True, **kwargs)</li>
<li>test_loader = torch.utils.data.DataLoader(</li>
<li>    datasets.MNIST('../data', train=False, transform=transforms.Compose([</li>
<li>                       transforms.ToTensor(),</li>
<li>                       transforms.Normalize((0.1307,), (0.3081,))</li>
<li>                   ])),</li>
<li>    batch_size=args.test_batch_size, shuffle=True, **kwargs)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Write a <code>transforms</code>. <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Tutorial</a>.</p>
CUDA leakage when sum over the outputs?

Huimin_ZENG: <p>Hi! I have read this post: <a href="https://discuss.pytorch.org/t/cuda-memory-leakage/33970" class="inline-onebox">CUDA memory leakage</a>.</p>
<p>In the discussion, I find that this makes great sense to me:<br>
<em>Often memory leaks are created by trying to store some training information like the loss without detaching it from the computation graph, which will store the whole graph with it.</em></p>
<p>However, I don’t understand why the memory usage remains the same when I just add all the outputs into a variable (no detach operation). I think I am still storing the training information, right?</p>
<p>Thank you!</p>

ptrblck: <p>If you append a tensor with an attached computation graph (valid <code>.grad_fn</code>), the computation graph will be stored with it and should could see an increased memory usage in each iteration.<br>
Is this the case for you or are you seeing any other issue?</p>

harryhar: <p>Thank you for the reply! May post some code here, so that I can explain it more clearly.</p>
<p>net = net.cuda()<br>
sum = 0</p>
<p>while True:<br>
batch = 4<br>
h = 3<br>
w = 3<br>
num_outputs = 5<br>
x = torch.randn(batch,1, h, w).cuda()<br>
y = net(x)<br>
sum += y</p>
<p>So basically, I am just summing all the outputs, and the computation graph should be attched to the outputs, right? However, the GPU usage remains the same. I don’t understand why.</p>
<p>Thanks!</p>

Huimin_ZENG: <p>Thank you for the reply! May post some code here, so that I can explain it more clearly.</p>
<p>while True:<br>
x = torch.randn(batch,1, h, w).cuda()<br>
y = net(x)<br>
sum += y</p>
<p>So basically, I am just summing all the outputs, and the computation graph should be attched to the outputs, right? However, the GPU usage remains the same. I don’t understand why.</p>
<p>Thanks!</p>

ptrblck: <p>Yes, this should be the case, if you didn’t wrap the code block in a <code>with torch.no_grad()</code> block.</p>
<p>Which model are you using? Also, how are you checking the memory usage? Could you print it using <code>torch.cuda.memory_allocated()</code> in the loop?</p>

Huimin_ZENG: <p>Thank you for the instructions and sorry for the delayed reply.</p>
<p>I failed to reproduce the results I mentioned from the beginning. The usage of GPU is increasing now.</p>
<p>I am just using a really small toy model. And I keep typing command nvidia-smi to check the memory usage.</p>
<p>Thank you so much!</p>

ptrblck: <p>PyTorch uses a custom caching memory allocator, which will try to reuse the device memory.<br>
Thus <code>nvidia-smi</code> shows you the overall memory usage including the CUDA context, the allocated and the cached memory (also from other processes), which might show the increased memory usage after a couple of steps for small model.</p>
How to weight the loss?

oasjd7: <pre><code class="lang-auto">...
for step, input in dataloader:
    outptut = model(input)  
    # top_prob, _ = torch.topk(F.softmax(output, dim=1)[0], 1)
    loss = criterion(output, label)

</code></pre>
<p>I want to weight the highest probability of softmax to the loss.<br>
ex)<br>
If, prob of softmax : 0.75 -&gt;  Effect of this sample * 0.75<br>
If, prob of softmax : 0.12 -&gt;  Effect of this sample * 0.12</p>
<p>But the top_prob is the list (length == batch size)<br>
like <code>top_prob = [0.72, 0.75, 0.82, ... 0.94]</code><br>
and loss is one scalar value.<br>
(It means the loss is already computed with all elements of a batch.)</p>
<p>How to apply the probability of softmax to the loss ?</p>

LeviViana: <p>The CrossEntropy loss has a <code>weight</code> parameter for doing this, you can check it in the <a href="https://pytorch.org/docs/stable/nn.html#crossentropyloss" rel="nofollow noopener">documentation</a>.</p>

oasjd7: <p>Thanks for you answer.<br>
But as far as I know, the <code>weight</code> in nn.CrossEntropyLoss() uses for the class-wise weight.<br>
In my case, I need to weight sample-wise manner.</p>

LeviViana: <p>This is going to work:</p>
<pre><code class="lang-python">import torch

x = torch.rand(16, 20)
y = torch.randint(2, (16,))
# Try torch.ones(16) here and it will be equivalent to
# regular CrossEntropyLoss
weights = torch.rand(16) 

net = torch.nn.Linear(20, 2)

def element_weighted_loss(y_hat, y, weights):
    m = torch.nn.LogSoftmax(dim=1)
    criterion = torch.nn.NLLLoss(reduction='none')
    loss = criterion(m(y_hat), y)
    loss = loss * weights
    return loss.sum() / weights.sum()

weighted_loss = element_weighted_loss(net(x), y, weights)

not_weighted_loss = torch.nn.CrossEntropyLoss()(net(x), y)

print(weighted_loss, not_weighted_loss)
</code></pre>

oasjd7: <aside class="quote no-group" data-username="LeviViana" data-post="4" data-topic="66372">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>weights</p>
</blockquote>
</aside>
<p>Thanks!<br>
but weights.sum() is correct? not the size of batch?</p>

LeviViana: <p>Nope ! <code>weights.sum()</code> is there since the weights must sum up to 1.</p>
How can we measure the quality of features?

Sunny_Cloudy: <p>I’m recently working on time series data in my project, and I found that apart from traditional feature extract methods  like PCA, there are deep learning methods such as auto encoders. I want to share in my group about time series data using these and make comparisons between them.<br>
But you see they don’t predict results directly instead they make features.<br>
So how can I compare their performance in features? Or to put it more frankly, how can we measure the quality of features we extract? As far as i googled there seems to be few articles about that.</p>
Weibull Regression with LSTM

shashank_gupta1: <p>Hello,</p>
<p>I am trying to perform Regression with target distribution assumed to be Weibull.</p>
<p>The input is a time-ordered sequence with each timestamp having a regression target.</p>
<p>At each timestamp, the input is a mix of categorical and numerical values.</p>
<p>I transformed categorical variables using embedding and numerical values using an FFN.</p>
<p>When I train the network using LSTM, I am getting constant values as output, irrespective of the input.</p>
<p>Not sure how to debug/inspect this problem.</p>
<p>I am new to Pytorch, so I am confused.</p>
<p>Any suggestions wil be helpful.</p>
RuntimeError: “add_cpu/sub_cpu” not implemented for ‘Half’ when using Float16/Half

flynntax: <p>Hello, I am testing out different types in PyTorch and noticed when calling torch.jit.trace where the input is float16/half, I am getting a runtime error (RuntimeError: “add_cpu/sub_cpu” not implemented for ‘Half’). Does tracing not work for float16? I believe I also tested with int32 inputs which work.</p>

tom: <p>I think this might be more about operations that PyTorch supports on GPU than the types.<br>
Does the same code run in plain PyTorch?</p>
<p>Best regards</p>
<p>Thomas</p>

flynntax: <p>You are right in that this doesn’t seem to work for eager mode (guessing this is plain PyTorch) as just calling model(input) causes the same runtime error. But I am not running on a GPU right now (just a macbook). I guess I can probably change the category and rename the question. I guess Half is just not supported for CPU?</p>

tom: <p>Indeed. (That’d be a fun weekend project for you! <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> )</p>
Batch training in LSTM with variable lengths to get final (hn, cn)

sakura: <p>Hi, I have a sequence input: <code>input.size()=[17, 1, 64] #(seq_len, batch, channel)</code>, and use <code>nn.LSTM(64, 16, bidirectional=True)</code> to get <code>output, (hn, cn)</code>.</p>
<p>In such case, <code>output.size()=[17, 1, 32], hn.size()=[2, 1, 16], cn.size()=[2, 1, 16]</code>. I concat <code>(hn, cn)</code> as final feature for next application, which works well.</p>
<p>When there exists multiple inputs with variable lengths, I try to use <code>pad_sequence</code> and <code>pack_padded_sequence</code> to make a batch, i.e., a <code>PackedSequence</code> object. The model forwards successfully. However, when I using</p>
<pre><code class="lang-auto">h_n.view(num_layers, num_directions, batch, hidden_size).permute(2,0,1,3).contiguous().view(batch, -1)
</code></pre>
<p>to obtain final feature for each input sequence, the next applications get poor performance.</p>
<p>So, here the questions:<br>
(1) The <code>(hn, cn)</code> represent the final time step for the entire batch? or exactly represent the final time step for each variable seq?<br>
(2) If for the entire batch: how can I get the accurate final status (h, c) for each variable seq?<br>
(3) If already for each seq: why the performance drops? Is there other problems which I should check?</p>
Variable dropout rate throughout training

vainaijr: <p>How do I set a high dropout rate during the beginning of training, to make weight matrix more sparse, and after every certain epochs, keep reducing this dropout rate?<br>
for example, for the first 50 epochs, dropout rate could be 0.7, for next 50 epochs, 0.6, then 0.5, and for last 50 epochs, dropout rate could be 0.2.</p>
<p>And during evaluation use last dropout rate.</p>

ptrblck: <p>If you are using dropout as a module, you could manipulate the <code>.p</code> attribute after your specified number of epochs:</p>
<pre><code class="lang-python">drop = nn.Dropout(p=0.5)

x = torch.randn(10)
out = drop(x)
print((out==0.).sum())
&gt; tensor(5)

drop.p = 0.2
out = drop(x)
print((out==0.).sum())
&gt; tensor(1)
</code></pre>
<p>In your case, the manipulation would of course look like this:</p>
<pre><code class="lang-python">model.drop1.p = 0.2
</code></pre>
<p>Or alternatively you could also use the functional API from the beginning, and provide the drop probability in the <code>forward</code> method of your model:</p>
<pre><code class="lang-python">def forward(self, x, p):
    x = F.dropout(x, p, training=self.training)
    return x
</code></pre>
Question about training

chunchun: <p>I try to print loss.backward() then I got none,but the network training is still running.<br>
Why ?</p>

ptrblck: <p><code>tensor.backward()</code> won’t return anything. Instead the <code>.grad</code> attributes of all parameters used to compute the loss will be populated and accumulated.</p>

chunchun: <p>so should i print param_name.grad?</p>

kabilan: <p>it will compute backpropogation. if you want to compute manually use torch.no_grad()</p>

ptrblck: <p>Yes, if you want to print out all gradients, you could use:</p>
<pre><code class="lang-python">print(model.layer.param.grad)
# or to print all gradients
for name, param in model.named_parameters():
    print(name, param.grad)
</code></pre>

chunchun: <p>Thank you very much.</p>

kabilan: <p>thank you <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=9" title=":handshake:" class="emoji" alt=":handshake:"> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
What's num_chunks in rnncellbase class?

zjplab: <p>I am trying run old code written in PyTorch 0.3  in PyTorch 1.x. Parameter mismatch error occurred.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main_LM.py", line 113, in &lt;module&gt;
    args.tied, args.hard, args.res)
  File "/content/PRPN/model_PRPN.py", line 31, in __init__
    self.reader = nn.ModuleList([ReadingNetwork(ninp, nhid, nslots, dropout=dropout, idropout=idropout), ] +
  File "/content/PRPN/ReadingNetwork.py", line 19, in __init__
    self.memory_rnn = LSTMCell(ninp, nout)
  File "/content/PRPN/LSTMCell.py", line 23, in __init__
    super(LSTMCell, self).__init__()
TypeError: __init__() missing 4 required positional arguments: 'input_size', 'hidden_size', 'bias', and 'num_chunks'
</code></pre>
<p>I checked original PyTorch 0.3 source code. There was no num_chunks parameter in RNNCellBase class. So what does this newly add parameter do?</p>

ptrblck: <p><code>num_chunks</code> is used to initialize the parameters in different RNN variants.<br>
E.g. an LSTM uses <code>num_chunks</code>, as the weight and bias pararameters contain 4 parts as seen in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM">docs</a> (take a look at the “Variables” section).</p>
<p>It seems your code uses a custom <code>LSTMCell</code> implementation, so you could either use the PyTorch <code>LSTMCell</code>, would need to update the code for your custom layer or keep your code and add the necessary code from the base classes into your current implementation.</p>
How to get a model to run from CPU to GPU

JasonChenhx: <p>I want the model to be trained on the GPU. Besides the input and model to be cuda, what else needs to be cuda?</p>

ptrblck: <p>All data needed for an operation should be on the same device.<br>
So besides the model and input, you would have to push the target to the same device in order to calculate the loss.</p>
<p>This should be sufficient for a vanilla training loop.</p>

JasonChenhx: <p>Thanks！<br>
Your reply helped me a lot!<br>
Thanks again.</p>
Normalizing flow gone wrong

smu226: <p>Hello! I am using a normalizing flow (Neural Spline Flows) to approximate a probability and after some training, the average loss is around 0.5 (so logprob = -0.5). However, when I am trying it on some new test data, I am getting some values of logprob bigger than zero, which would mean that the probability for that element is bigger than one (which doesn’t make sense). Does anyone know what could cause this? Isn’t the flow supposed to keep all the probabilities below 1 automatically? Thank you!</p>

ptrblck: <p>I’m not familiar with your approach, but could you explain, how the log probabilities were calculated during training?<br>
I would assume you are using e.g. <code>F.log_softmax</code> to calculate them or are they generated by any other method?</p>
How to avoid memory out from python list

oasjd7: <p>Hi, all.<br>
I want to append features to a list.<br>
but out of memory occurs.</p>
<p>Here is my codes.</p>
<pre><code class="lang-auto">for epoch in range(total_epochs):
   list_a = []
   for step, data in range(data_loader):
        ...
         if some_condition:
              list_a.append(features.cpu()) # (both cuda() and cpu() occurs memory out)
   data_loader_a = torchdata.DataLoader(list_a, ...)
   for step, data in range(data_loader_a):
       ... train process ...
</code></pre>
<p>How can I solve this problem?</p>

ptrblck: <p>I guess <code>features</code> is still attached to the computation graph, thus all intermediate tensors (which would be needed for a <code>backward</code> call), are still on the device.<br>
If you don’t need to backpropagate using the features stored in <code>list_a</code>, you could just detach the tensors before appending to the list via:</p>
<pre><code class="lang-python">list_a.append(features.detach().cpu())
</code></pre>
Loss is nan after optimizer.step

helloWorld: <p>Hey,<br>
I’m trying to use the foolowing optimizer that I implemented (rmsprop) but after the first step of the optimizer the loss that is calculated in my main is nan.</p>
<pre><code class="lang-auto">def __init__(self, parameters,lr=0.001,beta=0.999,epsilon=sys.float_info.epsilon):
    self.layers_data_list=[]
    self.epsilon=epsilon
    self.beta=beta
    for layer_params in list(parameters):
      layer_dict = dict()
      layer_dict['params']=layer_params
      layer_dict['lr']=lr
      layer_dict['average_derrevative']=None
      self.layers_data_list.append(layer_dict)

  def step(self): 
    for layer_data in self.layers_data_list:
      p=layer_data['params']
      lr=layer_data['lr']
      d_p=p.grad.data
      if layer_data['average_derrevative'] is None:
        layer_data['average_derrevative'] = torch.clone(d_p).detach()
      else :
        layer_data['average_derrevative'].mul_(self.beta).addcmul_(1-self.beta,d_p,d_p)
      
      avg_root = torch.clone(layer_data['average_derrevative']).detach().sqrt().add_(self.epsilon) # tried to add .clone().detach() didnt help..
      layer_data['params'].data.addcdiv_(-lr,d_p, avg_root)
</code></pre>

helloWorld: <p>Not sure why, but after replacing the following line :<br>
<code>layer_data['average_derrevative'] = torch.clone(d_p).detach()</code></p>
<p>with the following :<br>
<code> layer_data['average_derrevative'] = torch.zeros_like(layer_data['params'].data)</code></p>
<p>solved my problem. To be honest, I’m not sure why the first command was a mistake. If someone can explain it will be gr8.</p>
Circular Correlation

Chrixtar: <p>Hello there,</p>
<p>I am trying to implement the ciruclar correlation defined as follows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cdcde56a6634e3e292c3c29b141853a8f608d2b2.jpeg" alt="PyTorch%20Question" data-base62-sha1="tmD0JOFU00lGUYiDagbaIgSRf7c" width="304" height="120"><br>
which can be efficiently calculated using the Fourier transformation (and its inverse), the elementwise product and the complex negation.</p>
<p>I am new to PyTorch and don’t know whether, this is possible and if so, whether it was already implemented.<br>
I tried it like this:</p>
<pre><code class="lang-auto">complex_s = complex_conjugation(torch.rfft(s, 1, onesided=False))
complex_o = torch.rfft(o, 1, onesided=False)
circular_correlation = torch.irfft(elementwise_complex_multiplication(complex_s, complex_o), 1, onesided=False)


def complex_conjugation(tensor):
    minor = tensor.clone()
    for i in range(minor.size(-1)):
        minor[i, 0] = 0
    res = tensor - 2 * minor
    return res


def real_part(tensor):
    dimensions = len(list(tensor.size()))
    return tensor.narrow(dimensions-1, 0, 1)


def imag_part(tensor):
    dimensions = len(list(tensor.size()))
    return tensor.narrow(dimensions-1, 1, 1)


def elementwise_complex_multiplication(t1, t2):
    dimensions1 = len(list(t1.size()))
    dimensions2 = len(list(t2.size()))
    if dimensions1 != dimensions2:
        print("Different number of tensor dimensions in elementwise_complex_multiplication")
        exit(1)
    real1 = real_part(t1)
    real2 = real_part(t2)
    imag1 = imag_part(t1)
    imag2 = imag_part(t2)
    return torch.cat((real1 * real2 - imag1 * imag2, real1 * imag2 + imag1 * real2), dim=dimensions1-1)
</code></pre>
<p>However, I don’t get the same results, as when I use the formula of the definition (without fft).<br>
I tested my self-defined functions and they should be correct.</p>
<p>I don’t know, whether I understood torch.rfft and torch.irfft correctly.<br>
Please don’t be too harsh with me. Thank you already for looking into this.</p>
<p>PS: I compute it as part of the forward method in a network. Is this autograd compatible and is there maybe another way to implement it more efficiently?</p>
<p>Best regards<br>
Chrixtar</p>

Amiro: <p>Hi Chris,</p>
<p>I am interested with the same thing in fact, have you succeeded to find the correct implementation?</p>
<p>Please let me know</p>
<p>Amir</p>
How can I CUDA code to support mixed precision?

justanhduc: <p>I have a CUDA kernel and I want to add FP16 support for it. Any idea where I should start?</p>

ptrblck: <p>What is your use case, i.e. would you like to perform the computation in FP16 or pseudo-FP16, i.e. FP32 math for FP16 inputs? Also, what kind of operations are you using inside your kernel?</p>
<p>Have a look at <a href="https://github.com/NVIDIA/apex/tree/master/csrc">nvidia/apex</a> for some use cases.</p>

justanhduc: <p>Thanks for your reply. I want to perform computation in FP16 directly. I have found the solution which is <code>AT_DISPATCH_FLOATING_TYPES_AND(at::ScalarType::Half, ...)</code>, which seems to be better considering that <code>DISPATCH_DOUBLE_FLOAT_AND_HALF</code> is being deprecated.</p>