Multi Lang Marian Translator not working (opus_mt_mul_en)
Author: rp13g10
Pull request: no

When attempting to use the [opus_mt_mul_en](https://huggingface.co/Helsinki-NLP/opus-mt-mul-en) model, no translations are generated. Based on [this issue](https://github.com/JohnSnowLabs/spark-nlp/issues/2472) on the SparkNLP repo, this has been happening for a while, but perhaps never raised here.

I'm currently accessing the model through SparkNLP on an Amazon EMR cluster (release 5.30.0). Spark version 2.4.5, SparkNLP version 3.1.0.. The same issue occurs when using SparkNLP 2.7.0.

Code to reproduce the issue:
```python
import os
! apt-get update -qq > /dev/null   
# Install java
! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]
! pip install pyspark==2.4.5 spark-nlp==2.7.0
from sparknlp.annotator import *
from sparknlp.common import *
from sparknlp.base import *
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline
from pyspark.sql.functions import array_contains
from pyspark.ml import Pipeline, PipelineModel
import sparknlp

from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
spark = sparknlp.start()
documentAssembler = DocumentAssembler()\
 .setInputCol("text")\
 .setOutputCol("document")

sentencerDL = SentenceDetectorDLModel.pretrained("sentence_detector_dl", "xx")\
 .setInputCols(["document"])\
  .setOutputCol("sentences")
marian = MarianTransformer.pretrained("opus_mt_mul_en", "xx")\
 .setInputCols(["sentences"])\
 .setOutputCol("translation")

marian_pipeline = Pipeline(stages=[documentAssembler, sentencerDL, marian])
sdf = spark.createDataFrame([[">>deu<< Hallo wie geht es dir Ich bin hubert aus Deutschland"], [">>fra<< Wikipédia est un projet d'encyclopédie collective en ligne, universelle, multilingue et fonctionnant sur le principe du wiki. Ce projet vise à offrir un contenu librement réutilisable, objectif et vérifiable, que chacun peut modifier et améliorer."]]).toDF("text") 
m_fit = marian_pipeline.fit(sdf ) 
res_Df  = m_fit.transform(sdf)
res_Df.select('translation').show(truncate=False)
```

I've tried using the `setLangId` method instead of putting the tags inline with the input text, with the same result.

Comments:
Hey @rp13g10,

We are not really familiar with the `sparkNLP` repo...from the issue I assume that the following is the error from our side:

```python
# opus-mt-mul-en
# opus-mt-en-mul

from transformers import MarianMTModel, MarianTokenizer

model_name = 'Helsinki-NLP/opus-mt-mul-en'
tokenizer = MarianTokenizer.from_pretrained(model_name)

tokenizer.supported_language_codes # this returns nothing
```

E.g. the tokenizers should return some supported language codes

@patil-suraj have you already taken a look at multi-lingual marian models? 

Also gently pinging the Marian OG @sshleifer - should we update https://huggingface.co/Helsinki-NLP/opus-mt-mul-en/blob/main/tokenizer_config.json analogs to https://huggingface.co/Helsinki-NLP/opus-mt-en-ROMANCE/blob/main/tokenizer_config.json ?   

This is still an issue as far as I can tell, and it would be cool if it was fixed. :)

Gently pinging @patil-suraj here - do you have an opinion here?

Not 100% sure, but language codes are required when there are multiple target languages as for such models we need to prepend the target language code to the source text. 

The `opus-mt-mul-en` models, translate multiple languages to English, so we do not need to insert any language codes as you can see from this example

```python3
model_name = 'Helsinki-NLP/opus-mt-mul-en'
tokenizer = MarianTokenizer.from_pretrained(model_name)

texts = [
    "c'est une phrase en anglais que nous voulons traduire en français", # french
    "Isto deve ir para o português.", # portuguese
    "Y esto al español", # Spanish
]
inputs = tokenizer(texts, return_tensors="pt", padding=True)

gen_ids = model.generate(**inputs)
tokenizer.batch_decode(gen_ids, skip_special_tokens=True)
# ['is a phrase in English that we want to translate into French',
 'This has to go to Portugal.',
 'And this is in Spanish']
```

So it seems there is no issue with the model.

And if you look at `opus-mt-en-mul` or `opus-mt-en-ROMANCE` where there are multiple target languages, it does return non-empty `supported_language_codes` list.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.