MarianMT architecture and onnx format
Author: goga334
Pull request: no

**MarianMT architecture**
I found an interesting detail about MarianMT implementation in huggingface. There is no "Softmax" layer after "Linear" at the end of the model, despite the default architecture of transformer.

```
MarianMTModel(
  (model): MarianModel(
    (shared): Embedding(58930, 512, padding_idx=58929)
    (encoder): MarianEncoder(
      (embed_tokens): Embedding(58930, 512, padding_idx=58929)
      (embed_positions): MarianSinusoidalPositionalEmbedding(512, 512)
      (layers): ModuleList(
        (0-5): 6 x MarianEncoderLayer(
          (self_attn): MarianAttention(
            (k_proj): Linear(in_features=512, out_features=512, bias=True)
            (v_proj): Linear(in_features=512, out_features=512, bias=True)
            (q_proj): Linear(in_features=512, out_features=512, bias=True)
            (out_proj): Linear(in_features=512, out_features=512, bias=True)
          )
          (self_attn_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (activation_fn): SiLUActivation()
          (fc1): Linear(in_features=512, out_features=2048, bias=True)
          (fc2): Linear(in_features=2048, out_features=512, bias=True)
          (final_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
    (decoder): MarianDecoder(
      (embed_tokens): Embedding(58930, 512, padding_idx=58929)
      (embed_positions): MarianSinusoidalPositionalEmbedding(512, 512)
      (layers): ModuleList(
        (0-5): 6 x MarianDecoderLayer(
          (self_attn): MarianAttention(
            (k_proj): Linear(in_features=512, out_features=512, bias=True)
            (v_proj): Linear(in_features=512, out_features=512, bias=True)
            (q_proj): Linear(in_features=512, out_features=512, bias=True)
            (out_proj): Linear(in_features=512, out_features=512, bias=True)
          )
          (activation_fn): SiLUActivation()
          (self_attn_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (encoder_attn): MarianAttention(
            (k_proj): Linear(in_features=512, out_features=512, bias=True)
            (v_proj): Linear(in_features=512, out_features=512, bias=True)
            (q_proj): Linear(in_features=512, out_features=512, bias=True)
            (out_proj): Linear(in_features=512, out_features=512, bias=True)
          )
          (encoder_attn_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (fc1): Linear(in_features=512, out_features=2048, bias=True)
          (fc2): Linear(in_features=2048, out_features=512, bias=True)
          (final_layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
  )
  (lm_head): Linear(in_features=512, out_features=58930, bias=False)
)
```

There is no problem when loading this model via "MarianMTModel.from_pretrained" and calling ".generate()" method, everything works fine, returning output shaped (batch_size, max_seq_len). 

**MarianMT onnx format**
However, when I tried to convert MarianMT huggingface model into onnx format via "torch.onnx.export" and use it with "onnxruntime.InferenceSession" calling "run()" method, I got raw embedding batches as outputs shaped (batch_size, max_seq_len, 58930), which I can't decode into text using MarianTokenizer. I suppose, it is caused by the absence of that Softmax layer at the end.

**Regarding this, I have two questions:**
- Is it normal that MarianMT in huggingface transformers has no Softmax layer at the end?
- Is there a way to decode output embeddings shaped (batch_size, max_seq_len, 58930) into text?

@gante @amyeroberts @sgugger, I'm not sure whether to consider this a bug or just a slight misunderstanding, so I'd be really grateful for some advice.

Comments:
No model in Transformers implements the softmax at the end, they return logits, or if labels are provided, the loss directly.

Assuming your output from the ONNX model really is the logits and is in numpy format you could probably use a code snippet like this to decode it into text:
```
import numpy as np
from transformers import MarianTokenizer
from scipy.special import softmax

tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-es', cache_dir='./estokenizer')

bsz = 1
seq_len = 42
vocab_size = 20000

logits = np.random.rand(bsz, seq_len, vocab_size) # Output from the model.

token_probs = softmax(logits, axis=-1)

token_ids = np.argsort(token_probs, axis=2)[:, :, -1] # Get top token ID.

tokenizer.batch_decode(token_ids)
```

But are you sure about manually converting it to ONNX? Huggingface's Optimum package should support converting Marian to ONNX off the bat, with beam search support and all the whistles.

@sgugger, thanks a lot, now I see how it works)

@SmartWashingMachine, yes output was logits indeed. Thanks for the code snipped, it works well, but it seems that the model should be converted in some other way. I used Huggingface's Optimum for MarianMT and it works perfect with conversion and inference. 

However, i couldn't use it for m2m100 and mbart, google collab crashes because of lack of memory during conversion. I'll look for some other way of speed up for these two.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> @SmartWashingMachine, yes output was logits indeed. Thanks for the code snipped, it works well, but it seems that the model should be converted in some other way. I used Huggingface's Optimum for MarianMT and it works perfect with conversion and inference.
> 
> However, i couldn't use it for m2m100 and mbart, google collab crashes because of lack of memory during conversion. I'll look for some other way of speed up for these two.

Hello, may I ask a question? After using optim cli to export MarianMT, the model is divided into two parts: encoder. onnx and decoder. onnx. I suspect that I am using the wrong part. How did you call the decoder?
```ptyhon
···
input_ids = [[40604, 24, 90, 34, 2588, 2, 187, 56, 21258, 9,
              11080, 11, 3,  5896, 526, 4947, 9, 11, 605, 3605, 11768, 6, 0]]
attention_mask = [[1, 1, 1, 1, 1, 1, 1, 1, 1,
                   1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]

encoder_out = encoder_session.run(None, {'input_ids': input_ids,
                                         'attention_mask': attention_mask})
print(encoder_out[0].shape)
# beam search
batch_size = 1
num_beams = 4
max_length = 512
pad_token_id = 65000
eos_token_id = 0

input_ids = np.ones((num_beams, 1), dtype=np.int64) * pad_token_id
beam_scorer = BeamSearchScorer(batch_size=batch_size, num_beams=num_beams, max_length=max_length)
if isinstance(eos_token_id, int):
    eos_token_id = [eos_token_id]

beam_scores = np.zeros((batch_size, num_beams), dtype=np.float32)
beam_scores[:, 1:] = -1e9
beam_scores = beam_scores.reshape((batch_size * num_beams,))

encoder_out = encoder_out[0].repeat(4, 0)
encoder_attention_mask = np.array(attention_mask, dtype=np.int64).repeat(4, 0)
while True:
    decoder_out = decoder_session.run(None, {'input_ids': input_ids, 'attention_mask': None,
                                      'encoder_hidden_states': encoder_out, 'encoder_attention_mask': encoder_attention_mask})
    print(decoder_out[0].shape)
···
```
decoder_output[0] shape is (4, 2, 65001)

Hey 🤗 We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!