CPMAnt

Transformers documentation

Natural Language Processing

Performance and scalability

Reinforcement learning models

## [](#overview)Overview

CPM-Ant is an open-source Chinese pre-trained language model (PLM) with 10B parameters. It is also the first milestone of the live training process of CPM-Live. The training process is cost-effective and environment-friendly. CPM-Ant also achieves promising results with delta tuning on the CUGE benchmark. Besides the full model, we also provide various compressed versions to meet the requirements of different hardware configurations. [See more](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live)

Tips:

This model was contributed by [OpenBMB](https://huggingface.co/openbmb). The original code can be found [here](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live).

⚙️ Training & Inference

-   A tutorial on [CPM-Live](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live).

## [](#transformers.CpmAntConfig)CpmAntConfig

### class transformers.CpmAntConfig

[](#transformers.CpmAntConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/configuration_cpmant.py#L29)

( vocab\_size: int = 30720hidden\_size: int = 4096num\_attention\_heads: int = 32dim\_head: int = 128dim\_ff: int = 10240num\_hidden\_layers: int = 48dropout\_p: int = 0.0position\_bias\_num\_buckets: int = 512position\_bias\_max\_distance: int = 2048eps: int = 1e-06init\_std: float = 1.0prompt\_types: int = 32prompt\_length: int = 32segment\_types: int = 32use\_cache: bool = True\*\*kwargs )

This is the configuration class to store the configuration of a [CpmAntModel](/docs/transformers/v4.30.0/en/model_doc/cpmant#transformers.CpmAntModel). It is used to instantiate an CPMAnt model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the CPMAnt [openbmb/cpm-ant-10b](https://huggingface.co/openbmb/cpm-ant-10b) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import CpmAntModel, CpmAntConfig

>>> 
>>> configuration = CpmAntConfig()

>>> 
>>> model = CpmAntModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.CpmAntTokenizer)CpmAntTokenizer

### class transformers.CpmAntTokenizer

[](#transformers.CpmAntTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/tokenization_cpmant.py#L88)

( vocab\_filebod\_token = '<d>'eod\_token = '</d>'bos\_token = '
'eos\_token = '
'pad\_token = '<pad>'unk\_token = '<unk>'line\_token = '</n>'space\_token = '</\_>'padding\_side = 'left'\*\*kwargs )

Parameters

-   [](#transformers.CpmAntTokenizer.vocab_file)**vocab\_file** (`str`) — Path to the vocabulary file.
-   [](#transformers.CpmAntTokenizer.bod_token)**bod\_token** (`str`, _optional_, defaults to `"<d>"`) — The beginning of document token.
-   [](#transformers.CpmAntTokenizer.eod_token)**eod\_token** (`str`, _optional_, defaults to `"</d>"`) — The end of document token.
-   [](#transformers.CpmAntTokenizer.bos_token)**bos\_token** (`str`, _optional_, defaults to `"
"`) — The beginning of sequence token.
-   [](#transformers.CpmAntTokenizer.eos_token)**eos\_token** (`str`, _optional_, defaults to `"
"`) — The end of sequence token.
-   [](#transformers.CpmAntTokenizer.pad_token)**pad\_token** (`str`, _optional_, defaults to `"<pad>"`) — The token used for padding.
-   [](#transformers.CpmAntTokenizer.unk_token)**unk\_token** (`str`, _optional_, defaults to `"<unk>"`) — The unknown token.
-   [](#transformers.CpmAntTokenizer.line_token)**line\_token** (`str`, _optional_, defaults to `"</n>"`) — The line token.
-   [](#transformers.CpmAntTokenizer.space_token)**space\_token** (`str`, _optional_, defaults to `"</_>"`) — The space token.

Construct a CPMAnt tokenizer. Based on byte-level Byte-Pair-Encoding.

#### build\_inputs\_with\_special\_tokens

[](#transformers.CpmAntTokenizer.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/tokenization_cpmant.py#L235)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.List\[int\] = None ) → `List[int]`

Parameters

-   [](#transformers.CpmAntTokenizer.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — The first tokenized sequence that special tokens will be added.
-   [](#transformers.CpmAntTokenizer.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`) — The optional second tokenized sequence that special tokens will be added.

The model input with special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CPMAnt sequence has the following format:

-   single sequence: `[BOS] Sequence`.

#### get\_special\_tokens\_mask

[](#transformers.CpmAntTokenizer.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/tokenization_cpmant.py#L253)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → `List[int]`

Parameters

-   [](#transformers.CpmAntTokenizer.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.CpmAntTokenizer.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.
-   [](#transformers.CpmAntTokenizer.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range \[0, 1\]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

## [](#transformers.CpmAntModel)CpmAntModel

### class transformers.CpmAntModel

[](#transformers.CpmAntModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/modeling_cpmant.py#L604)

( config: CpmAntConfig )

The bare CPMAnt Model outputting raw hidden-states without any specific head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters config ([~CpmAntConfig](/docs/transformers/v4.30.0/en/model_doc/cpmant#transformers.CpmAntConfig)): Model configuration class with all the parameters of the Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

#### forward

[](#transformers.CpmAntModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/modeling_cpmant.py#L642)

( input\_ids: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonepast\_key\_values: typing.Optional\[typing.Tuple\[typing.Tuple\[torch.Tensor\]\]\] = Noneuse\_cache: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None\*\*kwargs ) → [transformers.modeling\_outputs.BaseModelOutputWithPast](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or `tuple(torch.FloatTensor)`

The [CpmAntModel](/docs/transformers/v4.30.0/en/model_doc/cpmant#transformers.CpmAntModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoTokenizer, CpmAntModel
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("openbmb/cpm-ant-10b")
>>> model = CpmAntModel.from_pretrained("openbmb/cpm-ant-10b")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state```

## [](#transformers.CpmAntForCausalLM)CpmAntForCausalLM

### class transformers.CpmAntForCausalLM

[](#transformers.CpmAntForCausalLM)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/modeling_cpmant.py#L751)

( config: CpmAntConfig )

The CPMAnt Model with a language modeling head on top (linear layer with weights tied to the input embeddings).

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters config ([~CpmAntConfig](/docs/transformers/v4.30.0/en/model_doc/cpmant#transformers.CpmAntConfig)): Model configuration class with all the parameters of the Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

#### forward

[](#transformers.CpmAntForCausalLM.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/cpmant/modeling_cpmant.py#L764)

( input\_ids: typing.Optional\[torch.Tensor\] = Nonepast\_key\_values: typing.Union\[typing.List\[typing.Tuple\[torch.Tensor, torch.Tensor\]\], NoneType\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonelabels: typing.Optional\[torch.Tensor\] = Nonereturn\_dict: typing.Optional\[bool\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = None\*\*kwargs ) → [transformers.modeling\_outputs.CausalLMOutputWithPast](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.CausalLMOutputWithPast) or `tuple(torch.FloatTensor)`

The [CpmAntForCausalLM](/docs/transformers/v4.30.0/en/model_doc/cpmant#transformers.CpmAntForCausalLM) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> import torch
>>> from transformers import AutoTokenizer, CpmAntForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("openbmb/cpm-ant-10b")
>>> model = CpmAntForCausalLM.from_pretrained("openbmb/cpm-ant-10b")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs, labels=inputs["input_ids"])
>>> loss = outputs.loss
>>> logits = outputs.logits```