Pytorch tensor abnormal, how to debug

walle_autoscale: <p>I have a program that produces a tensor, I can print the shape,dtype of this tensor, and it looks good. when I try to print the value of this tensor by <code>print(x)</code>, there goes the error message:</p>
<pre><code class="lang-auto">(Pdb)*** RuntimeError: numel: integer multiplication overflow
(Pdb)*** RuntimeError: numel: integer multiplication overflow
(Pdb)*** RuntimeError: CUDA error: misaligned addressCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect
</code></pre>
<p>But I do not know when this happed, it will only become a error when this tensor is accessed.</p>
<p>How to figure this out? Thanks a lot!</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the error as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

walle_autoscale: <p>Thanks for your reply! it requires several libs to reproduce this, if this really needed, I will provide one.<br>
I will provide more info about this error:</p>
<p>the program needs to compute <code>q = self.q_norm(q.transpose(1, 2).flatten(-2, -1))</code>, and I debugged the input of this code:</p>
<pre><code class="lang-auto">-&gt; q = self.q_norm(q.transpose(1, 2).flatten(-2, -1))[0].view(B_, N_, H_, D_).transpose(1, 2)                    
p q.shape                                                                                                        
(Pdb) torch.Size([2, 25, 257, 48])                                                                               
p q.dtype                                                                                                        
(Pdb) torch.bfloat16                                                                                             
p q                                                                                                              
(Pdb) tensor([[[[-1.2506e-12, -1.5010e-13,  3.3396e-13,  ...,  5.8265e-13,                                       
           -1.3589e-13, -5.2580e-13],                                                                            
          [-1.2434e-12, -1.5454e-13,  3.3218e-13,  ...,  5.7909e-13,                                             
           -1.4211e-13, -5.3646e-13],                                                                            
          [-1.2506e-12, -1.4833e-13,  3.3573e-13,  ...,  5.8620e-13,                                             
           -1.4033e-13, -5.3291e-13],                                                                            
          ...,          
</code></pre>
<p>the tensor looks good to me, but if i run <code> self.q_norm</code> I will get: <code>(Pdb) *** RuntimeError: CUDA error: misaligned address</code></p>

ptrblck: <p>Without a code snippet to be able to reproduce it, it might not be possible for me to debug the issue.<br>
Since you are apparently able to reproduce it, would it be possible to check your workload with <code>compute-sanitizer python script.py args</code> to check if it detects any issues?</p>

walle_autoscale: <p>Thanks. I runned with compute-sanitizer and got:</p>
<pre><code class="lang-auto">========= Invalid __global__ read of size 16 bytes
=========     at 0x1e0 in void layer_norm::ln_fwd_kernel&lt;layer_norm::Kernel_traits&lt;__nv_bfloat16, __nv_bfloat16, __nv_bfloat16, __nv_bfloat16, float, unsigned int, (unsigned int)1280, (unsigned int)1, (unsigned int)4, (unsigned int)1, (unsigned int)16, layer_norm::Kernel_traits_base&lt;(unsigned int)1280, __nv_bfloat16, __nv_bfloat16, __nv_bfloat16, __nv_bfloat16, float, unsigned int, (unsigned int)128&gt;&gt;, (bool)0, (bool)0, (bool)0, (bool)0&gt;(layer_norm::FwdParams)
=========     by thread (105,0,0) in block (26,0,0)
=========     Address 0x7f02facf1832 is misaligned
</code></pre>

ptrblck: <p>Thank you! Could you post the configs of all layernorm layers and the activation input shape so that I could try to create a code snippet to reproduce it?</p>

walle_autoscale: <p>Thanks. It requires some packages to reproduce this, might be complicated:</p>
<p>env:</p>
<pre><code class="lang-auto">pytorch.12+cu116 or Pytorch1.13+cu117+gcc10.2
</code></pre>
<p>dependency:</p>
<pre><code class="lang-auto"># modified deepspeed
git clone https://github.com/KimmiShi/DeepSpeed.git
cd DeepSpeed 
git checkout fmoe_v0.9.0
pip install .

git clone https://github.com/KimmiShi/fastmoe.git
cd fastmoe
# make sure you have nccl, if not, need to export NCCL path, both 2.10 and 2.14 are tested
export NCCL_PATH=/mnt/.../nccl_2.10.3-1+cuda11.4_x86_64/
export LIBRARY_PATH=$NCCL_PATH/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=$NCCL_PATH/lib:$LD_LIBRARY_PATH

# install with gpu machine
python setup.py install

# flash attn:
git clone https://github.com/HazyResearch/flash-attention
cd flash-attention
# install with gpu machine
python setup.py install
# install layer norm cuda ext
# these might get stuck, maybe related to g++ version
cd ../fused_dense_lib
python setup.py install

cd csrc/layer_norm
python setup.py install

</code></pre>
<p>and finally, the script:<br>
<a href="https://github.com/KimmiShi/TorchDistPackage/blob/2439664b41f69d9f228fef5bb2c0b40808009b87/explore/debug_ln_error.py" rel="noopener nofollow ugc">TorchDistPackage/explore/debug_ln_error.py at 2439664b41f69d9f228fef5bb2c0b40808009b87 · KimmiShi/TorchDistPackage · GitHub</a><br>
run this script with a gpu, if not using slurm, you need to modify the line +602  <code>setup_distributed_slurm()</code> to your version that does <code>torch.distributed.initialize(...)</code></p>
<p>The reason why this reproduce script requires a lot of deps is that the bug show up only when using <code>fmoe+deepspeed+flash attn layernorm</code> together. If only using <code>deepspeed+flash attn layernorm</code> there is no problem.</p>

walle_autoscale: <p>Hi, I have some new info:</p>
<p>I modified the code, to only create the problematic module(which may need a cuda ext) but not to use it to do any calculation, there is still the problem, when I remove the module completely, the program runs well.</p>
<p>I will invetigate into the following two possibility:</p>
<ol>
<li>the problematic module params caused the bug</li>
<li>the problematic module loaded cuda extention caused the bug</li>
</ol>

walle_autoscale: <p>update:<br>
It seems that a <code>nn.Linear(dim, 1)</code> in a module not related to layer_norm causes the problem. The shape of nn.Linear matters.<br>
I tried <code>nn.Linear(dim, 2)</code>, <code>nn.Linear(dim, 4)</code> these does not work;</p>
<p>Only when the dim is a multiple of 8 the program work.</p>
<p>I guess this is related to memory management, and maybe related to deepspeed.</p>

ptrblck: <p>No, I don’t think the linear layer is related and believe any CUDA operation could be the victim after the problematic <code>ln_fwd_kernel</code> kernel call as it could create a sticky error.<br>
Were you able to pull out the layernorm settings?</p>

walle_autoscale: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
So you mean that <code>ln_fwd_kernel</code> is the real problem?</p>
<p>about this op setting:</p>
<p>for python api:<br>
<a href="https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/ops/rms_norm.py#L70C11-L70C11" rel="noopener nofollow ugc">flash-attention/flash_attn/ops/rms_norm.py at main · HazyResearch/flash-attention (github.com)</a><br>
this python module is created with <code>flash_attn.ops.rms_norm.DropoutAddRMSNorm(1200, eps=1e-06, prenorm=True)</code></p>
<p>and calls to <a href="https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/ops/layer_norm.py#L10" rel="noopener nofollow ugc">flash-attention/flash_attn/ops/layer_norm.py at main · HazyResearch/flash-attention · GitHub</a></p>
<pre><code class="lang-auto">zmat, xmat, dmask, mu, rsigma = dropout_layer_norm.dropout_add_ln_fwd(
        x0mat, residualmat, gamma, beta, rowscale, colscale, None, None, dropout_p, epsilon,
        1.0, 0, None, residual_in_fp32, is_rms_norm
    )
</code></pre>
<p>with args:</p>
<pre><code class="lang-auto">x0mat.shape: torch.Size([514, 1200])
gamma.shape: torch.Size([1200])
residualmat: None
rowscale None
colscale None
dropout_p  0.0
is_rms_norm True
residual_in_fp32 False
</code></pre>

ptrblck: <p>Yes, <code>compute-sanitizer</code> shows the root cause in <code>ln_fwd_kernel</code> every other operation might only be the victim afterwards.</p>

walle_autoscale: <p>Does it make sense that <code>ln_fwd_kernel</code> requires 16B alignment, but the input data does not meet this requirement?. Is it reasonable for this kernel to have this requirement? Do I need to fix this kernel impl, any suggestions?</p>

ptrblck: <p>I doubt the input data is wrong and I guess some internal buffers might be misaligned, but I also didn’t take a deeper look into the code.</p>
<aside class="quote no-group" data-username="walle_autoscale" data-post="13" data-topic="183057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/walle_autoscale/48/52644_2.png" class="avatar"> walle_autoscale:</div>
<blockquote>
<p>Is it reasonable for this kernel to have this requirement?</p>
</blockquote>
</aside>
<p>It’s reasonable to expect an (internal) alignment, but it’s unreasonable to expect something from the user since users do not explicitly define any memory alignment guarantees in their code.</p>
<aside class="quote no-group" data-username="walle_autoscale" data-post="13" data-topic="183057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/walle_autoscale/48/52644_2.png" class="avatar"> walle_autoscale:</div>
<blockquote>
<p>Do I need to fix this kernel impl, any suggestions?</p>
</blockquote>
</aside>
<p>Assuming you are familiar with CUDA you could build the extension with debug symbols to narrow down which operation fails and why.<br>
I also see you’ve created an issue in their repository so I would also expect the code owners to take a look.</p>

walle_autoscale: <p>Thanks a lot! I will try to dive deeper into this bug and update this post if I had any new findings.</p>