Inconsistent Output for Identical Inputs When Using Linear Projection with Different squence length

seamoonlight-YBY: <p><code>query_states = self.q_proj(hidden_states)</code></p>
<p>Hello, I have encountered an issue regarding the implementation of an operator.<br>
Specifically, I input two <code>hidden_states</code> tensors: one with shape <code>(1, 10, 3584)</code> and the other with shape <code>(1, 41, 3584)</code>.<br>
It is important to note that the first 10 elements of <code>b</code> are exactly identical to those of <code>a</code>, and both tensors have <code>dtype=torch.bfloat16</code>.<br>
Given that <code>self.q_proj</code> is the same <code>Linear(in_features=3584, out_features=3584, bias=True)</code> module in both cases, I would expect that the first 10 elements of <code>query_states_b</code> would match exactly with those of <code>query_states_a</code>.<br>
However, I observed that there are slight differences between them. Could you please explain why this happens?<br>
Additionally, I noticed that when the shape of <code>a</code> is <code>(1, 35, 3584)</code>, this issue does not occur.</p>
<p>torch_version 2.1.2+cu121</p>

ptrblck: <p>You didn’t mention how large the differences are so I assume the error represents the expected noise caused by the limited floating point precision and a different order of operations for different kernels.</p>

seamoonlight-YBY: <p>Thank you very much for responding to my question!<br>
Let me try to describe the phenomenon more clearly.</p>
<p>I tested two input sequences — one of length 35 and the other of length 41. The first 35 tokens of both inputs are identical.<br>
<code>attn_output_10.shape torch.Size([1, 35, 3584])</code><br>
<code>attn_output.shape torch.Size([1, 41, 3584])</code><br>
Then, I ran a decoder layer from an LLM and extracted the <code>attention_output</code> from the first decoder layer (layer 0).<br>
Both outputs have the data type <code>torch.bfloat16</code>, meaning their numerical resolution is around 0.01.<br>
<code>attn_output.dtype torch.bfloat16</code><br>
After saving and comparing the results, I noticed that the outputs are exactly the same for the first 15 positions.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9866e8a4001a7a4fae63e2821c0553449bf0ca3d.png" alt="image" data-base62-sha1="lKd3v9ZloCCMcW76WFCe8sXnKgJ" width="517" height="35"><br>
However, starting from position 16, discrepancies begin to appear — and this is exactly what I don’t understand.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6234b16fbd8fc1a7701ebd5396d48f46ec00b7a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/6234b16fbd8fc1a7701ebd5396d48f46ec00b7a3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6234b16fbd8fc1a7701ebd5396d48f46ec00b7a3.png" alt="image" data-base62-sha1="e0LGzbRMRXrXEmWd6ukezdVZ2QX" width="593" height="100"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">593×100 22.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
To control for such random inconsistencies or noise, I tried to enforce deterministic behavior at the beginning of the inference script.</p>
<pre><code class="lang-auto">import os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
seed = 2025
import torch
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = False
torch.set_float32_matmul_precision("highest")
torch.backends.cudnn.allow_tf32 = False
import random
random.seed(seed)
import numpy as np
np.random.seed(seed)
</code></pre>
<p>Still, this issue persists, and I don’t know why.<br>
As far as I know, this occurs at the very first decoder layer, and such discrepancies accumulate with deeper layers, eventually resulting in significant <code>hiddenfeature</code> differences at each input position.</p>
<p><strong>Summary:</strong> I’d really appreciate it if you could let me know how to avoid this randomness.</p>
<p>PS: The reason I’m paying attention to those early hidden features — even the ones that don’t seem to contribute to the final generation — is because I’m using speculative decoding. But that’s a whole other story, haha. <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=14" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Using deterministic algorithms won’t help since you would enable deterministic results for the same sequence length not for all kernels.<br>
Since you are seeing differences starting from a specific sequence length another algorithm seems to be used and you could check the relative errors for <code>bfloat16</code> as well as other dtypes.</p>

jackshi: <p>Can reproduce with:</p>
<pre><code class="lang-auto">dim = 1280  # lower dimensions like 128 results in 0 diff.
qkv_proj = torch.nn.Linear(dim, 3 * dim).eval().to(_DEVICE, _DTYPE)
x = torch.rand(128, dim).to(_DEVICE, _DTYPE)

with torch.inference_mode():
    diff = qkv_proj(x[:54, :]) - qkv_proj(x)[:54, :]

print(diff.abs().max())  # Output: tensor(0.0020, dtype=torch.bfloat16)
print(diff.nonzero())  
</code></pre>
<p>Smees that smaller dimensions don’t have this problem. Higher precision also result in lower error. Even if <code>_DEVICE="cpu</code>, we still get this behaviour. Is there a way to force the computation to be deterministic?</p>