wrong onnx output of OPUS en-ar
Author: logicvv
Pull request: no
Labels: name: Good Second Issue

name: bug

### System Info

transformers == 4.28 
python == 3.9
 


### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


Here is my decoder onnx transfer code:


```

class DecoderWithLMHead(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.decoder = model.model.decoder
        self.lm_head = model.lm_head
        self.final_logits_bias = model.final_logits_bias

    def forward(self, input_ids, encoder_hidden_states, encoder_attention_mask):
        dec_out = self.decoder(
            input_ids=input_ids,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask
        )
        hidden_states = dec_out.last_hidden_state
        logits = self.lm_head(hidden_states) + self.final_logits_bias
        return logits
def decoder_onnx_export():
    decoder = model.model.decoder
    decoder.eval()
    decoder_with_lm = DecoderWithLMHead(model)

    dummy_input_ids = torch.randint(0, tokenizer.vocab_size, (1, 64))  # (batch, tgt_seq_len)
    dummy_encoder_attention_mask = torch.ones((1, 64), dtype=torch.long)  # (batch, src_seq_len)
    dummy_encoder_outputs = torch.rand((1, 64, 1024))  # encoder_hidden_states
    torch.onnx.export(
        decoder_with_lm,
        args=(
            dummy_input_ids,
        ),
        kwargs={
            "encoder_hidden_states": dummy_encoder_outputs,
            "encoder_attention_mask": dummy_encoder_attention_mask,
        },
        f="decoder_with_lm.onnx",
        input_names=[
            "input_ids",
            "encoder_hidden_states",
            "encoder_attention_mask",
        ],
        output_names=["logits"],
        opset_version=17,
    `)```
```


The original model from hugging face is good, but the onnx output was wrong:

inference code:
```

input_sentence =  'Using handheld GPS devices and programs like Google Earth , members of the Trio Tribe , who live in the rainforests of southern Suriname , map out their ancestral lands to help strengthen their territorial claims .'

            input_ids, attention_mask = preprocess_input(input_sentence, tokenizer, max_length)


            encoder_outputs = encoder_session.run(['hidden_states'], {
                'input_ids': input_ids.astype(np.int64),
                'attention_mask': attention_mask.astype(np.int64)
            })


            decoder_input_ids = np.full((1, 64), 61246, dtype=np.int64)
            decoder_attention_mask = np.full((1, 64), 1, dtype=np.int64)

            decoder_input_ids[0, 0] = start_id

            decoder_input_ids = np.array(decoder_input_ids)

            # print(decoder_input_ids)
            # print(len(decoder_input_ids))
            # exit()


            # 用于存储每个时间步的 logits
            all_output_token_id = []

            count += 1
            decoder_count = 0
            # 4. 逐步解码
            for step in range(max_length-1):

  

                outputs = decoder_session.run(
                    ['logits'],  # logits
                    {

                        'input_ids': decoder_input_ids.astype(np.int64),  # Decoder
                        # 'attention_mask':decoder_attention_mask,
                        'encoder_hidden_states': encoder_hidden_states,  # Encoder
                        'encoder_attention_mask':   attention_mask.astype(np.int64)


                    }
                )

     
        
                current_logits = outputs[0][0, step + 1, :]

                next_token_id = np.argmax(current_logits).item()  #

                if step + 1 < 64:
                    decoder_input_ids[0, step + 1] = next_token_id


                if next_token_id == tokenizer.eos_token_id:
                    print(next_token_id)
                    print('end')
                    break
                all_output_token_id.append(next_token_id)
                # print(all_output_token_id)
            output_sentence = tokenizer.decode(all_output_token_id, skip_special_tokens=True)
            # output_sentence = sp_t.decode(all_output_token_id)
            print(output_sentence)
```


huggingface output：
باستخدام أجهزة GPS المحمولة وبرامج مثل Google Earth ، يقوم أعضاء Trio Tribe ، الذين يعيشون في الغابات المطيرة في جنوب سورينام ، برسم خرائط لأراضي أجدادهم للمساعدة في تعزيز مطالبهم الإقليمية.

onnx output:
باستخدام أجهزة GPS المحمولة وبرامج مثل جوجل أعضاء Tri، الذين يعيشون في الغابات جنوب سورينام رسم أراضي الفوركس للمساعدة تعزيز الإقليمية..


Could anyone help please?

### Expected behavior

huggingface output：
باستخدام أجهزة GPS المحمولة وبرامج مثل Google Earth ، يقوم أعضاء Trio Tribe ، الذين يعيشون في الغابات المطيرة في جنوب سورينام ، برسم خرائط لأراضي أجدادهم للمساعدة في تعزيز مطالبهم الإقليمية.

onnx output:
باستخدام أجهزة GPS المحمولة وبرامج مثل جوجل أعضاء Tri، الذين يعيشون في الغابات جنوب سورينام رسم أراضي الفوركس للمساعدة تعزيز الإقليمية..

Comments:
This is fixed.

PR #40182 is definitely not a usable fix, lol. However @logicvv, I notice your version of `transformers` is very out of date. Have you tried with the latest version?

Thanks for your comment, I tried to update my transformer before, but on another issue of this problem , they mentioned downgrade the version of transformer. that's why I tried it, but it doesn't work either.  @Rocketknight1 

Hmmn, okay. I'm not sure what the cause here is - if anyone wants to investigate, feel free! If you can figure out the cause we can maybe try to patch it

PR #41430 is also not a usable fix. Please only try to fix this one if you actually understand it well enough to review the outputs of your code agent - we're all AI engineers and we know how to run Copilot ourselves, so you are not helping by just blindly running it and sending us the output!

Hi @Rocketknight1,

You're absolutely right about my previous PR #41430. I apologize for the confusion I created, I realize now that I created external utilities rather than identifying and fixing the actual root cause in the core Transformers code.

Looking at the issue more carefully, I can see that @logicvv  is experiencing a discrepancy between PyTorch and ONNX outputs for Marian translation models, where the ONNX version produces incomplete/incorrect translations.

From analyzing the provided code, I suspect the issue might be related to:

Decoder attention mask handling during ONNX export - the custom DecoderWithLMHead wrapper doesn't handle decoder self-attention masks properly

Generation logic in the inference code - the way current_logits = outputs[0][0, step + 1, :] is indexed might not align with how the model expects autoregressive generation

Missing decoder attention mask in the ONNX export (commented out in the inference code)

Before I attempt another fix, could you point me toward the right direction? Should I be looking at:

The core Marian model files in src/transformers/models/marian/?

ONNX export utilities in the main codebase?

The way decoder attention masks are handled during export?

I want to make sure I understand the architecture properly and target the actual problem this time, rather than creating workarounds.

Thanks for your patience, and I appreciate the guidance on focusing on real fixes rather than external utilities.

@Rocketknight1 Please confirm if there is no one working on it then I'll try to finish this.

Hey @logicvv @InitialState9 

I looked into this a little bit for my own learning, but it works fine for me.

The script provided in the report is a bit incomplete, but I suspect your problem lies there. For inference, it's important that you either (1) only pass the already generated tokens to `decoder_input_ids` (Note: Use `dynamic_axes` in `torch.onnx.export` to make this work with onnx), or (2) Use `decoder_attention_mask` to mask the still unseen tokens. Masking is also done automatically if you use the correct padding token id. I see you're hard-coding `61246` instead of `pad_token_id`? That's probably your bug right there.

[Full working script.](https://gist.github.com/philiproeleveld/c011627cee1060b93c1a4ac129324dcf) Output:
```
باستخدام أجهزة تحديد المواقع المحمولة وبرامج مثل غوغل إيرث، أعضاء قبيلة تريو، الذين يعيشون في الغابات المطيرة جنوب سورينام، رسموا أراضي أسلافهم للمساعدة في تعزيز مطالبهم الإقليمية.
```
</details>

@Rocketknight1 I think this can be closed.

P.S. AI slop really getting out of hand 👀

Thank you for the investigation @philiproeleveld! Closing for now.