BARTpho

Transformers documentation

Natural Language Processing

Performance and scalability

Reinforcement learning models

## [](#overview)Overview

The BARTpho model was proposed in [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://arxiv.org/abs/2109.09701) by Nguyen Luong Tran, Duong Minh Le and Dat Quoc Nguyen.

The abstract from the paper is the following:

_We present BARTpho with two versions — BARTpho\_word and BARTpho\_syllable — the first public large-scale monolingual sequence-to-sequence models pre-trained for Vietnamese. Our BARTpho uses the “large” architecture and pre-training scheme of the sequence-to-sequence denoising model BART, thus especially suitable for generative NLP tasks. Experiments on a downstream task of Vietnamese text summarization show that in both automatic and human evaluations, our BARTpho outperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future research and applications of generative Vietnamese NLP tasks._

Example of use:

```
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer

>>> bartpho = AutoModel.from_pretrained("vinai/bartpho-syllable")

>>> tokenizer = AutoTokenizer.from_pretrained("vinai/bartpho-syllable")

>>> line = "Chúng tôi là những nghiên cứu viên."

>>> input_ids = tokenizer(line, return_tensors="pt")

>>> with torch.no_grad():
...     features = bartpho(**input_ids)  

>>> 
>>> from transformers import TFAutoModel

>>> bartpho = TFAutoModel.from_pretrained("vinai/bartpho-syllable")
>>> input_ids = tokenizer(line, return_tensors="tf")
>>> features = bartpho(**input_ids)```

Tips:

-   Following mBART, BARTpho uses the “large” architecture of BART with an additional layer-normalization layer on top of both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts. For example:

```
>>> from transformers import MBartForConditionalGeneration

>>> bartpho = MBartForConditionalGeneration.from_pretrained("vinai/bartpho-syllable")
>>> TXT = "Chúng tôi là <mask> nghiên cứu viên."
>>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"]
>>> logits = bartpho(input_ids).logits
>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
>>> probs = logits[0, masked_index].softmax(dim=0)
>>> values, predictions = probs.topk(5)
>>> print(tokenizer.decode(predictions).split())```

-   This implementation is only for tokenization: “monolingual\_vocab\_file” consists of Vietnamese-specialized types extracted from the pre-trained SentencePiece model “vocab\_file” that is available from the multilingual XLM-RoBERTa. Other languages, if employing this pre-trained multilingual SentencePiece model “vocab\_file” for subword segmentation, can reuse BartphoTokenizer with their own language-specialized “monolingual\_vocab\_file”.

This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BARTpho).

## [](#transformers.BartphoTokenizer)BartphoTokenizer

### class transformers.BartphoTokenizer

[](#transformers.BartphoTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bartpho/tokenization_bartpho.py#L46)

( vocab\_filemonolingual\_vocab\_filebos\_token = '
'eos\_token = '
'sep\_token = '
'cls\_token = '
'unk\_token = '<unk>'pad\_token = '<pad>'mask\_token = '<mask>'sp\_model\_kwargs: typing.Union\[typing.Dict\[str, typing.Any\], NoneType\] = None\*\*kwargs )

Adapted from [XLMRobertaTokenizer](/docs/transformers/v4.30.0/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer). Based on [SentencePiece](https://github.com/google/sentencepiece).

This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

#### build\_inputs\_with\_special\_tokens

[](#transformers.BartphoTokenizer.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bartpho/tokenization_bartpho.py#L193)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.BartphoTokenizer.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs to which the special tokens will be added.
-   [](#transformers.BartphoTokenizer.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of [input IDs](../glossary#input-ids) with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An BARTPho sequence has the following format:

-   single sequence: `
 X 
`
-   pair of sequences: `
 A 

 B 
`

Converts a sequence of tokens (strings for sub-words) in a single string.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.BartphoTokenizer.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bartpho/tokenization_bartpho.py#L247)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.BartphoTokenizer.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.BartphoTokenizer.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of zeros.

Create a mask from the two sequences passed to be used in a sequence-pair classification task. BARTPho does not make use of token type ids, therefore a list of zeros is returned.

#### get\_special\_tokens\_mask

[](#transformers.BartphoTokenizer.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bartpho/tokenization_bartpho.py#L219)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → `List[int]`

Parameters

-   [](#transformers.BartphoTokenizer.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.BartphoTokenizer.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.
-   [](#transformers.BartphoTokenizer.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range \[0, 1\]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.