Speculative Decoding(do_sample=False) get different outputs
Author: nighty8
Pull request: no

> @transcend-0 hey!
> 
> 
> 
> The issue was solved in [#30068](https://github.com/huggingface/transformers/pull/30068). You can install transformers from `main` with the following line for the correct generation with assisted decoding:
> 
> 
> 
> `!pip install --upgrade git+https://github.com/huggingface/transformers.git` 

 _Originally posted by @zucchini-nlp in [#30608](https://github.com/huggingface/transformers/issues/30608#issuecomment-2089846816)_

### **System Info**

Python 3.10.11
transformers 4.49.0
torch 2.6.0+cu124

### **Same Reproduction**
Target_Model  = Qwen2.5-32B-Instruct
Draft_Model = Qwen2.5-7B-Instruct


`question = "Dienes are organic compounds with two adjacent double bonds in their structure, and they exhibit unique reactivity due to their conjugated pi-electron system. They play a significant role in organic chemistry and are involved in various chemical reactions and natural processes.\nAmong the given options which one is the possible reactant (A) for the given reaction also mention the correct sequence of the dienes according to their reactivity ( most reactive to least reactive) B.\nCyclohexene + A ---> 8,8-diiodobicyclo[4.2.0]octan-7-one\n(B) 1. 2,3-dimethylbuta-1,3-diene, 2. (2E,4E)-hexa-2,4-diene, 3. (2E,4E)-hexa-2,4-diene, 4. (2Z,4Z)-hexa-2,4-diene\n\n\nA. A = 2,2-diiodoethen-1-one, B = 3, 1, 2, 4\nB. A = 2,2-diiodoethen-1-one, B = 4, 2, 1, 3\nC. A = 4,4-diiodocyclobut-2-en-1-one, B = 3, 1, 2, 4\nD. A = 4,4-diiodocyclobut-2-en-1-one, B = 4, 2, 1, 3\n\n"`
`prompt = '
user' + question + 'Please reason step-by-step and put your choice letter without any other text with \\boxed{} in the end.'`

`['userDienes are organic compounds with two adjacent double bonds in their structure, and they exhibit unique reactivity due to their conjugated pi-electron system. They play a significant role in organic chemistry and are involved in various chemical reactions and natural processes.\nAmong the given options which one is the possible reactant (A) for the given reaction also mention the correct sequence of the dienes according to their reactivity ( most reactive to least reactive) B.\nCyclohexene + A ---> 8,8-diiodobicyclo[4.2.0]octan-7-one\n(B) 1. 2,3-dimethylbuta-1,3-diene, 2. (2E,4E)-hexa-2,4-diene, 3. (2E,4E)-hexa-2,4-diene, 4. (2Z,4Z)-hexa-2,4-diene\n\n\nA. A = 2,2-diiodoethen-1-one, B = 3, 1, 2, 4\nB. A = 2,2-diiodoethen-1-one, B = 4, 2, 1, 3\nC. A = 4,4-diiodocyclobut-2-en-1-one, B = 3, 1, 2, 4\nD. A = 4,4-diiodocyclobut-2-en-1-one, B = 4, 2, 1, 3\n\nPlease reason step-by-step and put your choice letter without any other text with \\boxed{} in the end. To solve this problem, we need to identify the reactant \\( A \\) that can react with cyclohexene to form 8,8-diiodobicyclo[4.2.0]octan-7-one. We also need to determine the correct sequence of the dienes according to their reactivity from most reactive to least reactive.\n\n### Step-by-Step Reasoning:\n\n1. **Identify the Product:**\n   - The product is 8,8-diiodobicyclo[4.2.0]octan-7-one. This suggests that the reactant \\( A \\) must be a compound that can undergo a Diels-Alder reaction with cyclohexene to form the bicyclic structure and then iodination at the appropriate positions.\n\n2. **Reactant Identification:**\n   - The reactant \\( A \\) should be a dienophile (a compound with a double bond that can participate in a Diels-Alder reaction). Among the given options, the possible candidates are:\n     - 2,2-diiodoethen-1-one\n     - 4,4-diiodocyclobut-2-en-1-one\n\n3. **Diels-Alder Reaction:**\n   - Cyclohexene is a diene, and it will react with a dienophile to form a bicyclic structure. The dienophile should have a double bond that can react with the diene to form the desired product.\n   - 2,2-diiodoethen-1-one has a double bond and iodine substituents, making it a suitable dienophile.\n   - 4,4-diiodocyclobut-2-en-1-one also has a double bond but is more complex and less likely to form the desired product directly.\n\n4. **Sequence of Dienes According to Reactivity:**\n   - The reactivity of dienes depends on the stability of the conjugated pi-electron system.\n   - Generally, the order of reactivity from most reactive to least reactive is:\n     1. (2E,4E)-hexa-2,4-diene (most stable and reactive)\n     2. (2E,4E)-hexa-2,4-diene (same as above)\n     3. 2,3-dimethylbuta-1,3-diene (less stable due to steric hindrance)\n     4. (2Z,4Z)-hexa-2,4-diene (least stable due to cis configuration)\n\n5. **Matching Options:**\n   - Option A: \\( A = 2,2 \\)-diiodoethen-1-one, B = 3, 1, 2, 4\n   - Option B: \\( A = 2,2 \\)-diiodoethen-1-one, B = 4, 2, 1, 3\n   - Option C: \\( A = 4,4 \\)-diiodocyclobut-2-en-1-one, B = 3, 1, 2, 4\n   - Option D: \\( A = 4,4 \\)-diiodocyclobut-2-en-1-one, B = 4, 2, 1, 3\n\nGiven the correct sequence of dienes and the suitable dienophile, the correct option is:\n\n\\boxed{A}']`
- targetDecoding - Running time: 41.82 s`

`['userDienes are organic compounds with two adjacent double bonds in their structure, and they exhibit unique reactivity due to their conjugated pi-electron system. They play a significant role in organic chemistry and are involved in various chemical reactions and natural processes.\nAmong the given options which one is the possible reactant (A) for the given reaction also mention the correct sequence of the dienes according to their reactivity ( most reactive to least reactive) B.\nCyclohexene + A ---> 8,8-diiodobicyclo[4.2.0]octan-7-one\n(B) 1. 2,3-dimethylbuta-1,3-diene, 2. (2E,4E)-hexa-2,4-diene, 3. (2E,4E)-hexa-2,4-diene, 4. (2Z,4Z)-hexa-2,4-diene\n\n\nA. A = 2,2-diiodoethen-1-one, B = 3, 1, 2, 4\nB. A = 2,2-diiodoethen-1-one, B = 4, 2, 1, 3\nC. A = 4,4-diiodocyclobut-2-en-1-one, B = 3, 1, 2, 4\nD. A = 4,4-diiodocyclobut-2-en-1-one, B = 4, 2, 1, 3\n\nPlease reason step-by-step and put your choice letter without any other text with \\boxed{} in the end. To solve this problem, we need to identify the reactant \\( A \\) that would lead to the formation of 8,8-diiodobicyclo[4.2.0]octan-7-one when reacted with cyclohexene. We also need to determine the correct sequence of the dienes according to their reactivity from most reactive to least reactive.\n\n### Step-by-Step Reasoning:\n\n1. **Identify the Product:**\n   - The product is 8,8-diiodobicyclo[4.2.0]octan-7-one. This suggests that the reactant \\( A \\) must be a compound that can form a bicyclic structure with cyclohexene and incorporate iodine atoms.\n\n2. **Reactant Identification:**\n   - Given the options, we need to choose between 2,2-diiodoethen-1-one and 4,4-diiodocyclobut-2-en-1-one.\n   - 2,2-diiodoethen-1-one is a simpler molecule compared to 4,4-diiodocyclobut-2-en-1-one.\n   - 4,4-diiodocyclobut-2-en-1-one has a cyclic structure which could potentially form a bicyclic structure with cyclohexene more readily.\n\n3. **Sequence of Dienes According to Reactivity:**\n   - The dienes provided are:\n     1. 2,3-dimethylbuta-1,3-diene\n     2. (2E,4E)-hexa-2,4-diene\n     3. (2E,4E)-hexa-2,4-diene (repeated)\n     4. (2Z,4Z)-hexa-2,4-diene\n   - Generally, trans (E) isomers are more reactive than cis (Z) isomers due to better overlap of the p-orbitals.\n   - Therefore, the sequence from most reactive to least reactive should be:\n     1. (2E,4E)-hexa-2,4-diene (twice)\n     2. 2,3-dimethylbuta-1,3-diene\n     3. (2Z,4Z)-hexa-2,4-diene\n\nGiven these points, the correct reactant \\( A \\) is 4,4-diiodocyclobut-2-en-1-one, and the sequence of dienes according to reactivity is 3, 1, 2, 4.\n\nThus, the correct answer is:\n\n\\boxed{C}']`
- speculativeDecoding - Running time: 29.14 s


Comments:
cc @gante @zucchini-nlp, but do we actually guarantee that speculative decoding won't slightly change the output?

Can you share a reproducer? Could be that it had `do_sample=True` and thus the results are different. Also, we should note that the logits for target model in assisted decoding will be slightly different from simple greedy decoding. I remember there was an issue somewhere on this. This is because in assisted decoding the target model gets a batch of ~5 new tokens every step, unlike greedy decoding where the models sees 1 token only, causing small numerical differences. Same thing as if we compared logits from cached or not-cached generation

@zucchini-nlp 
```
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
import torch
import os
import time


class Timer:
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        self.start = time.time()
        result = self.func(*args, **kwargs)
        print(f'- {self.func.__name__} - Running time: {time.time() - self.start :.2f} s\n')
        return result


set_seed(42)
os.environ["CUDA_VISIBLE_DEVICES"]='0,1'
device_map = 1
torch.cuda.empty_cache()


prompt = "Long long ago"
question = "Dienes are organic compounds with two adjacent double bonds in their structure, and they exhibit unique reactivity due to their conjugated pi-electron system. They play a significant role in organic chemistry and are involved in various chemical reactions and natural processes.\nAmong the given options which one is the possible reactant (A) for the given reaction also mention the correct sequence of the dienes according to their reactivity ( most reactive to least reactive) B.\nCyclohexene + A ---> 8,8-diiodobicyclo[4.2.0]octan-7-one\n(B) 1. 2,3-dimethylbuta-1,3-diene, 2. (2E,4E)-hexa-2,4-diene, 3. (2E,4E)-hexa-2,4-diene, 4. (2Z,4Z)-hexa-2,4-diene\n\n\nA. A = 2,2-diiodoethen-1-one, B = 3, 1, 2, 4\nB. A = 2,2-diiodoethen-1-one, B = 4, 2, 1, 3\nC. A = 4,4-diiodocyclobut-2-en-1-one, B = 3, 1, 2, 4\nD. A = 4,4-diiodocyclobut-2-en-1-one, B = 4, 2, 1, 3\n\n"
prompt = '
user' + question + 'Please reason step-by-step and put your choice letter without any other text with \\boxed{} in the end.'

target_checkpoint = ".../Qwen2.5-32B-Instruct/"
draft_checkpoint = ".../Qwen2.5-7B-Instruct/"

tokenizer = AutoTokenizer.from_pretrained(target_checkpoint)
inputs = tokenizer(prompt, return_tensors="pt").to(f"cuda:{device_map}")

target_model = AutoModelForCausalLM.from_pretrained(target_checkpoint,torch_dtype=torch.bfloat16).to(f"cuda:{device_map}")
draft_model = AutoModelForCausalLM.from_pretrained(draft_checkpoint,torch_dtype=torch.bfloat16).to(f"cuda:{device_map-1}")

target_model.generation_config.update(**{
    'max_new_tokens': 1024,
    'do_sample': False,
    'temperature': None,
    'top_k': None,
})

@Timer
@torch.no_grad()
def targetDecoding():
    outputs = target_model.generate(**inputs)
    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

@Timer
@torch.no_grad()
def draftDecoding():
    outputs = draft_model.generate(**inputs)
    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

@Timer
@torch.no_grad()
def speculativeDecoding():
    outputs = target_model.generate(**inputs, assistant_model=draft_model)
    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))


targetDecoding()
speculativeDecoding()
```
The reproduced code is almost identical to that in > #30608 and the setting of 'do_sample' is False;
In transformers 4.49.0, speculative decoding uses copy to make sure draft and target model have same generation config, so both of them should be  'do_sample'=False.
I only made an adjustment to the 'max_new_tokens' parameter.

Hmm, for me it shows the same gibberish output. What is the version you are using, can you try updating?

@zucchini-nlp 
Does “same” refer to token-level equality? I tested Transformers versions 4.51.3, 4.52.4, and 4.49.0, and found that target decoding and speculative decoding never produced exactly the same outputs even 'do_sample'=False.

Same after the tokens ate decoded, yes. Can you check that the logits are `allclose` since the difference is probably due to choosing one or another of two high probability tokens, and after that the generation diverges

@zucchini-nlp 
Hi, I recorded the first token where the two methods diverged across different versions of the transformer. But I'm not sure what's causing these differences.

**Transformer version:4.49.0**
```
Index 17 is different:
tuple_a[17] argmax: 646
Top-5 values: [24.125, 24.125, 23.75, 23.21428680419922, 23.125]
Top-5 indices: [646, 1035, 686, 11, 68845]
--------------------------------------------------
tuple_b[17] argmax: 1035
Top-5 values: [24.125, 24.0, 23.625, 23.125, 23.095239639282227]
Top-5 indices: [1035, 646, 686, 68845, 11]
```

**Transformer version:4.52.4**
```
Index 158 is different:
tuple_a[158] argmax: 17767
Top-5 values: [25.357145309448242, 25.25, 25.0, 23.21428680419922, 22.625]
Top-5 indices: [17767, 58676, 18320, 362, 24145]
--------------------------------------------------
tuple_b[158] argmax: 58676
Top-5 values: [25.25, 25.238096237182617, 24.875, 23.095239639282227, 22.625]
Top-5 indices: [58676, 17767, 18320, 362, 24145]
```

**Transformer version:4.51.3**
```
Index 17 is different:
tuple_a[17] argmax: 646
Top-5 values: [24.125, 24.125, 23.75, 23.095239639282227, 23.0]
Top-5 indices: [646, 1035, 686, 11, 68845]
--------------------------------------------------
tuple_b[17] argmax: 1035
Top-5 values: [24.125, 24.0, 23.625, 23.21428680419922, 23.125]
Top-5 indices: [1035, 646, 686, 11, 68845]
```



As you can see, the difference in logits for the two tokens is tiny and thus was caused by matrix multiplications. When doing target decoding the matmuls will be `(1, dim) x (weights_vector)` while in assisted decoding we are checking all candidates in a batch `(num_candidates, dim) x (weights_vector)`

These two modes of computation may trigger different optimized kernels internally in torch (e.g., single vs. batched GEMM), leading to slight numerical precision differences. These tiny deviations in logits sometimes are enough to change the sampling decision toward token A or B as in your example, especially when the logits are not much different

So yeah, it is expected

So may I conclude that the speculative decoding approach might be responsible for some minor discrepancies?

Yep :)

Thank you so much, that solves my issue.

@nighty8 there are a few seemingly input-invariant flags that actually cause tiny changes in the outputs, like speculative decoding or using KV caches! You can read more about it [here](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535) -- the root cause is as @zucchini-nlp explained 🤗 

@gante 
Thank you so much.  By the way, in this case, I found that in original Speculative Decoding, when the candidate input_ids are [17767, 362, 1124, 8, 429, 646, 1352], the token 646 will be rejected. However, when I adjust the candidate input_ids to [1124, 8, 429], the logits behavior becomes very similar to greedy decoding and accepts the extra token 646. In both cases, the whole input_ids before [646] are exactly the same. Could this also be caused by issues of this kind?
```
tensor([[[27.8750, 26.9048, 26.5476, 24.2500, 24.0000],
         [31.1905, 30.1190, 27.6190, 25.3571, 21.1250],
         [33.3333, 24.3750, 20.8750, 19.7619, 19.2857],
         [34.2857, 28.1250, 24.8750, 22.6250, 21.7500],
         [30.6250, 28.9286, 27.5000, 26.7857, 25.7500],
         [24.1250, 24.0000, 23.6250, 23.1250, 23.0952],
         [24.4048, 23.5000, 22.8750, 22.8750, 22.7500],
         [27.9762, 27.8571, 26.0714, 23.0000, 22.5000]]], device='cuda:0')
tensor([[[17767,   320,   362,   330,   429],
         [  362,    32,  1124,   320, 19788],
         [ 1124, 57758,    59,     8,   284],
         [    8,   701,  7731,   568,   982],
         [  429,   323,   369,   304,  2567],
         [ 1035,   646,   686, 68845,    11],
         [13767, 36671, 15759,  1352, 15963],
         [  279,   220, 17767,  3070,   330]]], device='cuda:0')
```
```
tensor([[[33.3333, 24.3750, 20.8750, 19.7619, 19.2857],
         [34.0476, 28.1250, 24.8750, 22.6250, 21.6250],
         [30.7500, 29.0476, 27.6190, 26.9048, 25.8750],
         [24.1250, 24.1250, 23.7500, 23.1250, 23.0952]]], device='cuda:0')
tensor([[[ 1124, 57758,    59,     8,   284],
         [    8,   701,  7731,   568,   982],
         [  429,   323,   369,   304,  2567],
         [  646,  1035,   686, 68845,    11]]], device='cuda:0')
```