FSMT

**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign @stas00.

## [](#overview)Overview

FSMT (FairSeq MachineTranslation) models were introduced in [Facebook FAIR’s WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616) by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.

The abstract of the paper is the following:

_This paper describes Facebook FAIR’s submission to the WMT19 shared news translation task. We participate in two language pairs and four language directions, English <-> German and English <-> Russian. Following our submission from last year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling toolkit which rely on sampled back-translations. This year we experiment with different bitext data filtering schemes, as well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific data, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the human evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations. This system improves upon our WMT’18 submission by 4.5 BLEU points._

This model was contributed by [stas](https://huggingface.co/stas). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/wmt19).

## [](#implementation-notes)Implementation Notes

-   FSMT uses source and target vocabulary pairs that aren’t combined into one. It doesn’t share embeddings tokens either. Its tokenizer is very similar to [XLMTokenizer](/docs/transformers/v4.30.0/en/model_doc/xlm#transformers.XLMTokenizer) and the main model is derived from [BartModel](/docs/transformers/v4.30.0/en/model_doc/bart#transformers.BartModel).

## [](#transformers.FSMTConfig)FSMTConfig

### class transformers.FSMTConfig

[](#transformers.FSMTConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/configuration_fsmt.py#L41)

( langs = \['en', 'de'\]src\_vocab\_size = 42024tgt\_vocab\_size = 42024activation\_function = 'relu'd\_model = 1024max\_length = 200max\_position\_embeddings = 1024encoder\_ffn\_dim = 4096encoder\_layers = 12encoder\_attention\_heads = 16encoder\_layerdrop = 0.0decoder\_ffn\_dim = 4096decoder\_layers = 12decoder\_attention\_heads = 16decoder\_layerdrop = 0.0attention\_dropout = 0.0dropout = 0.1activation\_dropout = 0.0init\_std = 0.02decoder\_start\_token\_id = 2is\_encoder\_decoder = Truescale\_embedding = Truetie\_word\_embeddings = Falsenum\_beams = 5length\_penalty = 1.0early\_stopping = Falseuse\_cache = Truepad\_token\_id = 1bos\_token\_id = 0eos\_token\_id = 2forced\_eos\_token\_id = 2\*\*common\_kwargs )

This is the configuration class to store the configuration of a [FSMTModel](/docs/transformers/v4.30.0/en/model_doc/fsmt#transformers.FSMTModel). It is used to instantiate a FSMT model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FSMT [facebook/wmt19-en-ru](https://huggingface.co/facebook/wmt19-en-ru) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Examples:

```
>>> from transformers import FSMTConfig, FSMTModel

>>> 
>>> config = FSMTConfig()

>>> 
>>> model = FSMTModel(config)

>>> 
>>> configuration = model.config```

#### to\_dict

[](#transformers.FSMTConfig.to_dict)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/configuration_fsmt.py#L220)

( ) → `Dict[str, any]`

Dictionary of all the attributes that make up this configuration instance,

Serializes this instance to a Python dictionary. Override the default _to\_dict()_ from _PretrainedConfig_.

## [](#transformers.FSMTTokenizer)FSMTTokenizer

### class transformers.FSMTTokenizer

[](#transformers.FSMTTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/tokenization_fsmt.py#L135)

( langs = Nonesrc\_vocab\_file = Nonetgt\_vocab\_file = Nonemerges\_file = Nonedo\_lower\_case = Falseunk\_token = '<unk>'bos\_token = '
'sep\_token = '
'pad\_token = '<pad>'\*\*kwargs )

Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following:

-   Moses preprocessing and tokenization.
-   Normalizing all inputs text.
-   The arguments `special_tokens` and the function `set_special_tokens`, can be used to add additional symbols (like ”**classify**”) to a vocabulary.
-   The argument `langs` defines a pair of languages.

This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

#### build\_inputs\_with\_special\_tokens

[](#transformers.FSMTTokenizer.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/tokenization_fsmt.py#L404)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.FSMTTokenizer.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs to which the special tokens will be added.
-   [](#transformers.FSMTTokenizer.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of [input IDs](../glossary#input-ids) with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A FAIRSEQ Transformer sequence has the following format:

-   single sequence: `
 X 
`
-   pair of sequences: `
 A 
 B 
`

#### get\_special\_tokens\_mask

[](#transformers.FSMTTokenizer.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/tokenization_fsmt.py#L430)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → `List[int]`

Parameters

-   [](#transformers.FSMTTokenizer.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.FSMTTokenizer.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.
-   [](#transformers.FSMTTokenizer.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range \[0, 1\]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.FSMTTokenizer.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/tokenization_fsmt.py#L458)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.FSMTTokenizer.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.FSMTTokenizer.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).

Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ

Transformer sequence pair mask has the following format:

```
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |```

If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An FAIRSEQ\_TRANSFORMER sequence pair mask has the following format:

#### save\_vocabulary

[](#transformers.FSMTTokenizer.save_vocabulary)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/tokenization_fsmt.py#L491)

( save\_directory: strfilename\_prefix: typing.Optional\[str\] = None )

## [](#transformers.FSMTModel)FSMTModel

### class transformers.FSMTModel

[](#transformers.FSMTModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/modeling_fsmt.py#L1036)

( config: FSMTConfig )

Parameters

-   [](#transformers.FSMTModel.config)**config** ([FSMTConfig](/docs/transformers/v4.30.0/en/model_doc/fsmt#transformers.FSMTConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare FSMT Model outputting raw hidden-states without any specific head on top.

This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.FSMTModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/modeling_fsmt.py#L1058)

( input\_ids: LongTensorattention\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_input\_ids: typing.Optional\[torch.LongTensor\] = Nonedecoder\_attention\_mask: typing.Optional\[torch.BoolTensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_head\_mask: typing.Optional\[torch.Tensor\] = Nonecross\_attn\_head\_mask: typing.Optional\[torch.Tensor\] = Noneencoder\_outputs: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonepast\_key\_values: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Noneinputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonedecoder\_inputs\_embeds: typing.Optional\[torch.FloatTensor\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.Seq2SeqModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.Seq2SeqModelOutput) or `tuple(torch.FloatTensor)`

The [FSMTModel](/docs/transformers/v4.30.0/en/model_doc/fsmt#transformers.FSMTModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, FSMTModel
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/wmt19-ru-en")
>>> model = FSMTModel.from_pretrained("facebook/wmt19-ru-en")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state```

## [](#transformers.FSMTForConditionalGeneration)FSMTForConditionalGeneration

### class transformers.FSMTForConditionalGeneration

[](#transformers.FSMTForConditionalGeneration)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/modeling_fsmt.py#L1172)

( config: FSMTConfig )

Parameters

-   [](#transformers.FSMTForConditionalGeneration.config)**config** ([FSMTConfig](/docs/transformers/v4.30.0/en/model_doc/fsmt#transformers.FSMTConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The FSMT Model with a language modeling head. Can be used for summarization.

This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.FSMTForConditionalGeneration.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/fsmt/modeling_fsmt.py#L1192)

( input\_ids: LongTensorattention\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_input\_ids: typing.Optional\[torch.LongTensor\] = Nonedecoder\_attention\_mask: typing.Optional\[torch.BoolTensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_head\_mask: typing.Optional\[torch.Tensor\] = Nonecross\_attn\_head\_mask: typing.Optional\[torch.Tensor\] = Noneencoder\_outputs: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Nonepast\_key\_values: typing.Optional\[typing.Tuple\[torch.FloatTensor\]\] = Noneinputs\_embeds: typing.Optional\[torch.Tensor\] = Nonedecoder\_inputs\_embeds: typing.Optional\[torch.Tensor\] = Nonelabels: typing.Optional\[torch.LongTensor\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.Seq2SeqLMOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.Seq2SeqLMOutput) or `tuple(torch.FloatTensor)`

The [FSMTForConditionalGeneration](/docs/transformers/v4.30.0/en/model_doc/fsmt#transformers.FSMTForConditionalGeneration) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Translation example::

```
>>> from transformers import AutoTokenizer, FSMTForConditionalGeneration

>>> mname = "facebook/wmt19-ru-en"
>>> model = FSMTForConditionalGeneration.from_pretrained(mname)
>>> tokenizer = AutoTokenizer.from_pretrained(mname)

>>> src_text = "Машинное обучение - это здорово, не так ли?"
>>> input_ids = tokenizer(src_text, return_tensors="pt").input_ids
>>> outputs = model.generate(input_ids, num_beams=5, num_return_sequences=3)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
"Machine learning is great, isn't it?"```