BERTweet

Transformers documentation

Natural Language Processing

Performance and scalability

Reinforcement learning models

## [](#overview)Overview

The BERTweet model was proposed in [BERTweet: A pre-trained language model for English Tweets](https://www.aclweb.org/anthology/2020.emnlp-demos.2.pdf) by Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen.

The abstract from the paper is the following:

_We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification._

Example of use:

```
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer

>>> bertweet = AutoModel.from_pretrained("vinai/bertweet-base")

>>> 
>>> tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", use_fast=False)

>>> 
>>> 

>>> 
>>> line = "SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:"

>>> input_ids = torch.tensor([tokenizer.encode(line)])

>>> with torch.no_grad():
...     features = bertweet(input_ids)  

>>> 
>>> 
>>> ```

This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BERTweet).

## [](#transformers.BertweetTokenizer)BertweetTokenizer

### class transformers.BertweetTokenizer

[](#transformers.BertweetTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bertweet/tokenization_bertweet.py#L68)

( vocab\_filemerges\_filenormalization = Falsebos\_token = '
'eos\_token = '
'sep\_token = '
'cls\_token = '
'unk\_token = '<unk>'pad\_token = '<pad>'mask\_token = '<mask>'\*\*kwargs )

Constructs a BERTweet tokenizer, using Byte-Pair-Encoding.

This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

Loads a pre-existing dictionary from a text file and adds its symbols to this instance.

#### build\_inputs\_with\_special\_tokens

[](#transformers.BertweetTokenizer.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bertweet/tokenization_bertweet.py#L184)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.BertweetTokenizer.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs to which the special tokens will be added.
-   [](#transformers.BertweetTokenizer.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of [input IDs](../glossary#input-ids) with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERTweet sequence has the following format:

-   single sequence: `
 X 
`
-   pair of sequences: `
 A 

 B 
`

Converts a sequence of tokens (string) in a single string.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.BertweetTokenizer.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bertweet/tokenization_bertweet.py#L238)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.BertweetTokenizer.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.BertweetTokenizer.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of zeros.

Create a mask from the two sequences passed to be used in a sequence-pair classification task. BERTweet does not make use of token type ids, therefore a list of zeros is returned.

#### get\_special\_tokens\_mask

[](#transformers.BertweetTokenizer.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/bertweet/tokenization_bertweet.py#L210)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → `List[int]`

Parameters

-   [](#transformers.BertweetTokenizer.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.BertweetTokenizer.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.
-   [](#transformers.BertweetTokenizer.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range \[0, 1\]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

Normalize tokens in a Tweet