Different generations with the NLP model MarianMT of HuggingFace
Author: yassine0abba
Pull request: no

I'm trying to use the NLP model " MarianMT" from HuggingFace, and I want to use my own implementation of decoding (ex : Greedy Decoding). I compared my implementatin to Hugging Face implementation. (model.generate())

I implemented it; I checked the code 100 times and I don't know why I have different text generation. Can you please help me on that ?

My implemntation :
```
# Hugging face
from transformers import MarianMTModel, MarianTokenizer
import torch
import torch.nn.functional as F
import numpy as np


device = torch.device("cuda" if torch.cuda.is_available() else  "cpu") 
tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de')
model = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-de').to(device)
vocab_size = tokenizer.vocab_size
max_t = 15
done = [False ] # To indicate if the generation is finished (we reached EOS token)

# My implementation of Greedy Search
source = [ 'While that might be a somewhat morbid thought, I think it has some really profound implications that are worth exploring.']
encoded = tokenizer.prepare_seq2seq_batch(source, return_tensors='pt').to(device)
generatd_tokens = torch.tensor([[58100] ]).to(device)

for t in range(1, max_t-1) :
    model_output = model(**{'input_ids':encoded["input_ids"], "attention_mask" :encoded['attention_mask'],"decoder_input_ids":generatd_tokens})['logits'].detach()[:,-1,:].reshape(1, vocab_size )
    distrib = F.softmax(model_output, dim =-1).reshape(1, vocab_size )
    distrib = torch.sort(distrib, dim =-1,descending=True )
    distrib_idx = distrib.indices
    distrib_values = distrib.values
    
    next_token = torch.tensor([[distrib_idx[0][0].item()] ]).to(device)
    next_token = torch.tensor(1 - np.array(done)*1).view(-1,1).to(device) * next_token
    generatd_tokens = torch.cat([generatd_tokens, next_token ], dim =-1)
    
    if (done[0] == False and next_token[0][0].item() == model.config.eos_token_id) or (t == max_t - 1) :
        done[0] = True
    # If all sentences are generated, we exit the loop
    if all(done) :
      break

gen_sentences = tokenizer.batch_decode(generatd_tokens, skip_special_tokens=True)
print(gen_sentences)
```
*Output : ['Obwohl das ein etwas morbider Gedanke sein könnte, denke']*

# Hugging Face Implementation
```
translated_greedy = model.generate(input_ids = encoded['input_ids'].to(device) ,max_length=max_t+2,do_sample = False)
translated_greedy_sen = tokenizer.batch_decode(translated_greedy, skip_special_tokens=True)
print(translated_greedy_sen)
```
*Output : ['Das mag zwar ein etwas morbider Gedanke sein, aber ich denke']*

Thanks in advance

Comments:
for the notice, 58100 is the pad_id of the model


Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.