Yura Kuratov commited on
Commit
1e1a769
1 Parent(s): 5a7ea98

upload GENA-LM Athaliana

Browse files
README.md ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - dna
4
+ ---
5
+
6
+ # GENA-LM Athaliana 🌱 (gena-lm-bert-base-athaliana)
7
+
8
+ GENA-LM is a Family of Open-Source Foundational Models for Long DNA Sequences.
9
+
10
+ `gena-lm-bert-base-athaliana` is trained on Arabidopsis thaliana genome.
11
+
12
+ ## Model description
13
+ GENA-LM (`gena-lm-bert-base-athaliana`) model is trained with a masked language model (MLM) objective, following data preprocessing methods pipeline in the BigBird paper and by masking 15% of tokens. Model config for `gena-lm-bert-base-athaliana` is similar to the bert-base:
14
+
15
+ - 512 Maximum sequence length
16
+ - 12 Layers, 12 Attention heads
17
+ - 768 Hidden size
18
+ - 32k Vocabulary size
19
+
20
+ We pre-trained `gena-lm-bert-base-athaliana` on data obtained from [Kang et al.](https://doi.org/10.1038/s41467-023-42029-4), using this download [link](https://figshare.com/ndownloader/files/41661786) and contains chromosome-level genomes of 32 A. thaliana ecotypes.
21
+ Pre-training was performed for 1,700,000 iterations with batch size 256 and sequence length was equal to 512 tokens. We modified Transformer to use [Pre-Layer normalization](https://arxiv.org/abs/2002.04745). We upload the checkpoint with the best loss on validation set (iteration 425000) to `main` branch and the latest checkpoint to `step_1700000` branch.
22
+
23
+ Source code and data: https://github.com/AIRI-Institute/GENA_LM
24
+
25
+ Paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594
26
+
27
+ ## Examples
28
+
29
+ ### How to load pre-trained model for Masked Language Modeling
30
+ ```python
31
+ from transformers import AutoTokenizer, AutoModel
32
+
33
+ tokenizer = AutoTokenizer.from_pretrained('AIRI-Institute/gena-lm-bert-base-athaliana')
34
+ model = AutoModel.from_pretrained('AIRI-Institute/gena-lm-bert-base-athaliana', trust_remote_code=True)
35
+
36
+ ```
37
+
38
+ ### How to load pre-trained model to fine-tune it on classification task
39
+ Get model class from GENA-LM repository:
40
+ ```bash
41
+ git clone https://github.com/AIRI-Institute/GENA_LM.git
42
+ ```
43
+
44
+ ```python
45
+ from GENA_LM.src.gena_lm.modeling_bert import BertForSequenceClassification
46
+ from transformers import AutoTokenizer
47
+
48
+ tokenizer = AutoTokenizer.from_pretrained('AIRI-Institute/gena-lm-bert-base-athaliana')
49
+ model = BertForSequenceClassification.from_pretrained('AIRI-Institute/gena-lm-bert-base-athaliana')
50
+ ```
51
+ or you can just download [modeling_bert.py](https://github.com/AIRI-Institute/GENA_LM/tree/main/src/gena_lm) and put it close to your code.
52
+
53
+ OR you can get model class from HuggingFace AutoModel:
54
+ ```python
55
+ from transformers import AutoTokenizer, AutoModel
56
+ model = AutoModel.from_pretrained('AIRI-Institute/gena-lm-bert-base-athaliana', trust_remote_code=True)
57
+ gena_module_name = model.__class__.__module__
58
+ print(gena_module_name)
59
+ import importlib
60
+ # available class names:
61
+ # - BertModel, BertForPreTraining, BertForMaskedLM, BertForNextSentencePrediction,
62
+ # - BertForSequenceClassification, BertForMultipleChoice, BertForTokenClassification,
63
+ # - BertForQuestionAnswering
64
+ # check https://huggingface.co/docs/transformers/model_doc/bert
65
+ cls = getattr(importlib.import_module(gena_module_name), 'BertForSequenceClassification')
66
+ print(cls)
67
+ model = cls.from_pretrained('AIRI-Institute/gena-lm-bert-base-athaliana', num_labels=2)
68
+ ```
69
+
70
+ ## Evaluation
71
+ For evaluation results, see our paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594
72
+
73
+
74
+ ## Citation
75
+ ```bibtex
76
+ @article{GENA_LM,
77
+ author = {Veniamin Fishman and Yuri Kuratov and Maxim Petrov and Aleksei Shmelev and Denis Shepelin and Nikolay Chekanov and Olga Kardymon and Mikhail Burtsev},
78
+ title = {GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences},
79
+ elocation-id = {2023.06.12.544594},
80
+ year = {2023},
81
+ doi = {10.1101/2023.06.12.544594},
82
+ publisher = {Cold Spring Harbor Laboratory},
83
+ URL = {https://www.biorxiv.org/content/early/2023/11/01/2023.06.12.544594},
84
+ eprint = {https://www.biorxiv.org/content/early/2023/11/01/2023.06.12.544594.full.pdf},
85
+ journal = {bioRxiv}
86
+ }
87
+
88
+ ```
config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BertForMaskedLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoModel": "modeling_bert.BertForMaskedLM"
7
+ },
8
+ "attention_probs_dropout_prob": 0.1,
9
+ "gradient_checkpointing": false,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 768,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 3072,
15
+ "layer_norm_eps": 1e-12,
16
+ "max_position_embeddings": 512,
17
+ "model_type": "bert",
18
+ "num_attention_heads": 12,
19
+ "num_hidden_layers": 12,
20
+ "pad_token_id": 3,
21
+ "pre_layer_norm": true,
22
+ "position_embedding_type": "absolute",
23
+ "transformers_version": "4.6.0.dev0",
24
+ "type_vocab_size": 2,
25
+ "use_cache": true,
26
+ "vocab_size": 32000
27
+ }
events.out.tfevents ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a60902d19f54dbf550b321b08423c08dd5681077c1c9eda61dc64f8a90b355ea
3
+ size 2393729
modeling_bert.py ADDED
@@ -0,0 +1,2208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # code from huggingface transformers 4.17.0
2
+ # coding=utf-8
3
+ # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
4
+ # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
5
+ #
6
+ # Licensed under the Apache License, Version 2.0 (the "License");
7
+ # you may not use this file except in compliance with the License.
8
+ # You may obtain a copy of the License at
9
+ #
10
+ # http://www.apache.org/licenses/LICENSE-2.0
11
+ #
12
+ # Unless required by applicable law or agreed to in writing, software
13
+ # distributed under the License is distributed on an "AS IS" BASIS,
14
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
+ # See the License for the specific language governing permissions and
16
+ # limitations under the License.
17
+ """PyTorch BERT model."""
18
+
19
+ import importlib
20
+ import math
21
+ import os
22
+ import warnings
23
+ from dataclasses import dataclass
24
+ from typing import Optional, Tuple
25
+
26
+ import torch
27
+ import torch.utils.checkpoint
28
+ from packaging import version
29
+ from torch import nn
30
+ from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
31
+
32
+ from transformers.activations import ACT2FN
33
+ from transformers.file_utils import (
34
+ ModelOutput,
35
+ add_code_sample_docstrings,
36
+ add_start_docstrings,
37
+ add_start_docstrings_to_model_forward,
38
+ replace_return_docstrings,
39
+ )
40
+ from transformers.modeling_outputs import (
41
+ BaseModelOutputWithPastAndCrossAttentions,
42
+ BaseModelOutputWithPoolingAndCrossAttentions,
43
+ CausalLMOutputWithCrossAttentions,
44
+ MaskedLMOutput,
45
+ MultipleChoiceModelOutput,
46
+ NextSentencePredictorOutput,
47
+ QuestionAnsweringModelOutput,
48
+ SequenceClassifierOutput,
49
+ TokenClassifierOutput,
50
+ )
51
+ from transformers.modeling_utils import (
52
+ PreTrainedModel,
53
+ apply_chunking_to_forward,
54
+ find_pruneable_heads_and_indices,
55
+ prune_linear_layer,
56
+ )
57
+ from transformers.utils import logging
58
+ from transformers.models.bert.configuration_bert import BertConfig
59
+
60
+
61
+ logger = logging.get_logger(__name__)
62
+
63
+ _CHECKPOINT_FOR_DOC = "bert-base-uncased"
64
+ _CONFIG_FOR_DOC = "BertConfig"
65
+ _TOKENIZER_FOR_DOC = "BertTokenizer"
66
+
67
+ BERT_PRETRAINED_MODEL_ARCHIVE_LIST = [
68
+ "bert-base-uncased",
69
+ "bert-large-uncased",
70
+ "bert-base-cased",
71
+ "bert-large-cased",
72
+ "bert-base-multilingual-uncased",
73
+ "bert-base-multilingual-cased",
74
+ "bert-base-chinese",
75
+ "bert-base-german-cased",
76
+ "bert-large-uncased-whole-word-masking",
77
+ "bert-large-cased-whole-word-masking",
78
+ "bert-large-uncased-whole-word-masking-finetuned-squad",
79
+ "bert-large-cased-whole-word-masking-finetuned-squad",
80
+ "bert-base-cased-finetuned-mrpc",
81
+ "bert-base-german-dbmdz-cased",
82
+ "bert-base-german-dbmdz-uncased",
83
+ "cl-tohoku/bert-base-japanese",
84
+ "cl-tohoku/bert-base-japanese-whole-word-masking",
85
+ "cl-tohoku/bert-base-japanese-char",
86
+ "cl-tohoku/bert-base-japanese-char-whole-word-masking",
87
+ "TurkuNLP/bert-base-finnish-cased-v1",
88
+ "TurkuNLP/bert-base-finnish-uncased-v1",
89
+ "wietsedv/bert-base-dutch-cased",
90
+ # See all BERT models at https://huggingface.co/models?filter=bert
91
+ ]
92
+
93
+
94
+ def get_cls_by_name(name: str) -> type:
95
+ """Get class by its name and module path.
96
+
97
+ Args:
98
+ name (str): e.g., transfomers:T5ForConditionalGeneration, modeling_t5:my_class
99
+
100
+ Returns:
101
+ type: found class for `name`
102
+ """
103
+ module_name, cls_name = name.split(':')
104
+ return getattr(importlib.import_module(module_name), cls_name)
105
+
106
+
107
+ def load_tf_weights_in_bert(model, config, tf_checkpoint_path):
108
+ """Load tf checkpoints in a pytorch model."""
109
+ try:
110
+ import re
111
+
112
+ import numpy as np
113
+ import tensorflow as tf
114
+ except ImportError:
115
+ logger.error(
116
+ "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
117
+ "https://www.tensorflow.org/install/ for installation instructions."
118
+ )
119
+ raise
120
+ tf_path = os.path.abspath(tf_checkpoint_path)
121
+ logger.info(f"Converting TensorFlow checkpoint from {tf_path}")
122
+ # Load weights from TF model
123
+ init_vars = tf.train.list_variables(tf_path)
124
+ names = []
125
+ arrays = []
126
+ for name, shape in init_vars:
127
+ logger.info(f"Loading TF weight {name} with shape {shape}")
128
+ array = tf.train.load_variable(tf_path, name)
129
+ names.append(name)
130
+ arrays.append(array)
131
+
132
+ for name, array in zip(names, arrays):
133
+ name = name.split("/")
134
+ # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
135
+ # which are not required for using pretrained model
136
+ if any(
137
+ n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
138
+ for n in name
139
+ ):
140
+ logger.info(f"Skipping {'/'.join(name)}")
141
+ continue
142
+ pointer = model
143
+ for m_name in name:
144
+ if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
145
+ scope_names = re.split(r"_(\d+)", m_name)
146
+ else:
147
+ scope_names = [m_name]
148
+ if scope_names[0] == "kernel" or scope_names[0] == "gamma":
149
+ pointer = getattr(pointer, "weight")
150
+ elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
151
+ pointer = getattr(pointer, "bias")
152
+ elif scope_names[0] == "output_weights":
153
+ pointer = getattr(pointer, "weight")
154
+ elif scope_names[0] == "squad":
155
+ pointer = getattr(pointer, "classifier")
156
+ else:
157
+ try:
158
+ pointer = getattr(pointer, scope_names[0])
159
+ except AttributeError:
160
+ logger.info(f"Skipping {'/'.join(name)}")
161
+ continue
162
+ if len(scope_names) >= 2:
163
+ num = int(scope_names[1])
164
+ pointer = pointer[num]
165
+ if m_name[-11:] == "_embeddings":
166
+ pointer = getattr(pointer, "weight")
167
+ elif m_name == "kernel":
168
+ array = np.transpose(array)
169
+ try:
170
+ if pointer.shape != array.shape:
171
+ raise ValueError(f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched")
172
+ except AssertionError as e:
173
+ e.args += (pointer.shape, array.shape)
174
+ raise
175
+ logger.info(f"Initialize PyTorch weight {name}")
176
+ pointer.data = torch.from_numpy(array)
177
+ return model
178
+
179
+
180
+ class BertEmbeddings(nn.Module):
181
+ """Construct the embeddings from word, position and token_type embeddings."""
182
+
183
+ def __init__(self, config):
184
+ super().__init__()
185
+ self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
186
+ if config.position_embedding_type == 'absolute':
187
+ self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
188
+ self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
189
+
190
+ # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
191
+ # any TensorFlow checkpoint file
192
+ self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
193
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
194
+ # position_ids (1, len position emb) is contiguous in memory and exported when serialized
195
+ self.position_embedding_type = getattr(config, "position_embedding_type", "absolute")
196
+ self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))
197
+ if version.parse(torch.__version__) > version.parse("1.6.0"):
198
+ self.register_buffer(
199
+ "token_type_ids",
200
+ torch.zeros(self.position_ids.size(), dtype=torch.long),
201
+ persistent=False,
202
+ )
203
+
204
+ def forward(
205
+ self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None, past_key_values_length=0
206
+ ):
207
+ if input_ids is not None:
208
+ input_shape = input_ids.size()
209
+ else:
210
+ input_shape = inputs_embeds.size()[:-1]
211
+
212
+ seq_length = input_shape[1]
213
+
214
+ if position_ids is None:
215
+ position_ids = self.position_ids[:, past_key_values_length: seq_length + past_key_values_length]
216
+
217
+ # Setting the token_type_ids to the registered buffer in constructor where it is all zeros, which usually occurs
218
+ # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves
219
+ # issue #5664
220
+ if token_type_ids is None:
221
+ if hasattr(self, "token_type_ids"):
222
+ buffered_token_type_ids = self.token_type_ids[:, :seq_length]
223
+ buffered_token_type_ids_expanded = buffered_token_type_ids.expand(input_shape[0], seq_length)
224
+ token_type_ids = buffered_token_type_ids_expanded
225
+ else:
226
+ token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
227
+ if inputs_embeds is None:
228
+ inputs_embeds = self.word_embeddings(input_ids)
229
+ token_type_embeddings = self.token_type_embeddings(token_type_ids)
230
+ embeddings = inputs_embeds + token_type_embeddings
231
+ if self.position_embedding_type == "absolute":
232
+ position_embeddings = self.position_embeddings(position_ids)
233
+ embeddings += position_embeddings
234
+ embeddings = self.LayerNorm(embeddings)
235
+ embeddings = self.dropout(embeddings)
236
+ return embeddings
237
+
238
+
239
+ class BertSelfAttention(nn.Module):
240
+ def __init__(self, config, position_embedding_type=None, has_relative_attention_bias=False):
241
+ """Bert self-attention with abs/relative position encodings and sparsity.
242
+
243
+ Args:
244
+ config: HF model configuration loaded from json
245
+ position_embedding_type (str, optional): absolute, relative_key, relative_key_query or
246
+ relative_attention_bias . Defaults to None.
247
+ has_relative_attention_bias (bool, optional): Use it's own relative embeddings matrix. Defaults to False.
248
+ """
249
+ super().__init__()
250
+ if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
251
+ raise ValueError(
252
+ f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
253
+ f"heads ({config.num_attention_heads})"
254
+ )
255
+
256
+ self.config = config
257
+ self.is_decoder = config.is_decoder
258
+ # max_seq_len is used in absolute, relative_key & relative_key_query and to pre-define sparsity layout
259
+ self.max_seq_len = config.max_position_embeddings
260
+
261
+ self.num_attention_heads = config.num_attention_heads
262
+ self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
263
+ self.all_head_size = self.num_attention_heads * self.attention_head_size
264
+
265
+ # sparse attention configuration
266
+ self.is_sparse = False
267
+ sparse_config_cls_name = getattr(config, 'sparse_config_cls', None)
268
+ if sparse_config_cls_name:
269
+ self.is_sparse = True
270
+ sparse_config_cls = get_cls_by_name(sparse_config_cls_name)
271
+ self.sparse_config = sparse_config_cls(**self.config.sparse_attention)
272
+
273
+ if self.is_decoder and self.is_sparse:
274
+ raise RuntimeError('SparseAttention with BertModel decoder is not currently supported!')
275
+
276
+ self.query = nn.Linear(config.hidden_size, self.all_head_size)
277
+ self.key = nn.Linear(config.hidden_size, self.all_head_size)
278
+ self.value = nn.Linear(config.hidden_size, self.all_head_size)
279
+
280
+ self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
281
+ self.softmax = nn.Softmax(dim=-1)
282
+ self.position_embedding_type = position_embedding_type or getattr(config, "position_embedding_type", "absolute")
283
+ self.has_relative_attention_bias = has_relative_attention_bias
284
+
285
+ if self.is_sparse and self.position_embedding_type not in ['absolute', 'relative_attention_bias', 'rotary']:
286
+ raise RuntimeError(f'SparseAttention supports `absolute`, `relative_attention_bias` and `rotary` position '
287
+ f'embeddings, but: position_embeddings_type = {self.position_embedding_type}')
288
+
289
+ if self.is_decoder and self.position_embedding_type == 'relative_attention_bias':
290
+ raise RuntimeError(f'BertSelfAttention does not support `relative_attention_bias` with `is_decoder` '
291
+ f' = {self.is_decoder}')
292
+
293
+ if self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query":
294
+ self.max_position_embeddings = config.max_position_embeddings
295
+ self.max_seq_len = 2 * config.max_position_embeddings
296
+ self.distance_embedding = nn.Embedding(self.max_distance - 1, self.attention_head_size)
297
+ elif self.position_embedding_type == 'relative_attention_bias' and self.has_relative_attention_bias:
298
+ self.relative_attention_num_buckets = self.config.relative_attention_num_buckets
299
+ self.relative_last_bucket_distance = self.config.relative_last_bucket_distance
300
+ self.relative_attention_bias = nn.Embedding(self.relative_attention_num_buckets, self.num_attention_heads)
301
+ elif self.position_embedding_type == 'rotary':
302
+ self.rotary_base = getattr(config, 'rotary_base', None)
303
+ self.rotary_dim = getattr(config, 'rotary_dim', self.attention_head_size)
304
+ self.rotary_emb = RotaryEmbedding(self.rotary_dim, base=self.rotary_base)
305
+
306
+ if self.is_sparse:
307
+ try:
308
+ from deepspeed.ops.sparse_attention import SparseSelfAttention
309
+ except ImportError as e:
310
+ logger.error(f'DeepSpeed is required for Sparse Ops: {e}')
311
+ raise
312
+ self.sparse_self_attention = SparseSelfAttention(self.sparse_config, max_seq_length=self.max_seq_len)
313
+
314
+ def transpose_for_scores(self, x):
315
+ new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
316
+ x = x.view(new_x_shape)
317
+ return x.permute(0, 2, 1, 3)
318
+
319
+ def transpose_key_for_scores(self, x):
320
+ # to remove redundant transpose in attention_scores matmul operation
321
+ new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
322
+ x = x.view(*new_x_shape)
323
+ return x.permute(0, 2, 3, 1)
324
+
325
+ @staticmethod
326
+ def _relative_position_bucket(relative_position, bidirectional=True, num_buckets=32, max_distance=128):
327
+ """
328
+ Adapted from Mesh Tensorflow:
329
+ https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593
330
+
331
+ #todo: refactor, the same code is used in modeling_t5
332
+
333
+ Translate relative position to a bucket number for relative attention. The relative position is defined as
334
+ memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to
335
+ position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for
336
+ small absolute relative_position and larger buckets for larger absolute relative_positions. All relative
337
+ positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket.
338
+ This should allow for more graceful generalization to longer sequences than the model has been trained on
339
+
340
+ Args:
341
+ relative_position: an int32 Tensor
342
+ bidirectional: a boolean - whether the attention is bidirectional
343
+ num_buckets: an integer
344
+ max_distance: an integer
345
+
346
+ Returns:
347
+ a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets)
348
+ """
349
+ relative_buckets = 0
350
+ if bidirectional:
351
+ num_buckets //= 2
352
+ relative_buckets += (relative_position > 0).to(torch.long) * num_buckets
353
+ relative_position = torch.abs(relative_position)
354
+ else:
355
+ relative_position = -torch.min(relative_position, torch.zeros_like(relative_position))
356
+ # now relative_position is in the range [0, inf)
357
+
358
+ # half of the buckets are for exact increments in positions
359
+ max_exact = num_buckets // 2
360
+ is_small = relative_position < max_exact
361
+
362
+ # The other half of the buckets are for logarithmically bigger bins in positions up to max_distance
363
+ relative_postion_if_large = max_exact + (
364
+ torch.log(relative_position.float() / max_exact)
365
+ / math.log(max_distance / max_exact)
366
+ * (num_buckets - max_exact)
367
+ ).to(torch.long)
368
+ relative_postion_if_large = torch.min(
369
+ relative_postion_if_large, torch.full_like(relative_postion_if_large, num_buckets - 1)
370
+ )
371
+
372
+ relative_buckets += torch.where(is_small, relative_position, relative_postion_if_large)
373
+ return relative_buckets
374
+
375
+ def compute_bias(self, query_length, key_length):
376
+ """ Compute binned relative position bias """
377
+ context_position = torch.arange(query_length, dtype=torch.long)[:, None]
378
+ memory_position = torch.arange(key_length, dtype=torch.long)[None, :]
379
+ relative_position = memory_position - context_position # shape (query_length, key_length)
380
+ relative_position_bucket = self._relative_position_bucket(
381
+ relative_position, # shape (query_length, key_length)
382
+ bidirectional=(not self.is_decoder),
383
+ num_buckets=self.relative_attention_num_buckets,
384
+ max_distance=self.relative_last_bucket_distance,
385
+ )
386
+ relative_position_bucket = relative_position_bucket.to(self.relative_attention_bias.weight.device)
387
+ values = self.relative_attention_bias(relative_position_bucket) # shape (query_length, key_length, num_heads)
388
+ values = values.permute([2, 0, 1]).unsqueeze(0) # shape (1, num_heads, query_length, key_length)
389
+ return values
390
+
391
+ def get_relative_attention_bias(self, position_bias, batch_size, query_length, key_length):
392
+ if position_bias is None and self.has_relative_attention_bias:
393
+ position_bias = self.compute_bias(query_length, key_length)
394
+ position_bias = position_bias.repeat(batch_size, 1, 1, 1)
395
+ return position_bias
396
+
397
+ def forward(
398
+ self,
399
+ hidden_states,
400
+ attention_mask=None,
401
+ head_mask=None,
402
+ encoder_hidden_states=None,
403
+ encoder_attention_mask=None,
404
+ past_key_value=None,
405
+ position_bias=None,
406
+ output_attentions=False,
407
+ ):
408
+ mixed_query_layer = self.query(hidden_states)
409
+
410
+ # If this is instantiated as a cross-attention module, the keys
411
+ # and values come from an encoder; the attention mask needs to be
412
+ # such that the encoder's padding tokens are not attended to.
413
+ is_cross_attention = encoder_hidden_states is not None
414
+
415
+ if is_cross_attention and past_key_value is not None:
416
+ # reuse k,v, cross_attentions
417
+ key_layer = past_key_value[0]
418
+ value_layer = past_key_value[1]
419
+ attention_mask = encoder_attention_mask
420
+ elif is_cross_attention:
421
+ key_layer = self.transpose_key_for_scores(self.key(encoder_hidden_states))
422
+ value_layer = self.transpose_for_scores(self.value(encoder_hidden_states))
423
+ attention_mask = encoder_attention_mask
424
+ elif past_key_value is not None:
425
+ key_layer = self.transpose_key_for_scores(self.key(hidden_states))
426
+ value_layer = self.transpose_for_scores(self.value(hidden_states))
427
+ key_layer = torch.cat([past_key_value[0], key_layer], dim=2)
428
+ value_layer = torch.cat([past_key_value[1], value_layer], dim=2)
429
+ else:
430
+ key_layer = self.transpose_key_for_scores(self.key(hidden_states))
431
+ value_layer = self.transpose_for_scores(self.value(hidden_states))
432
+
433
+ query_layer = self.transpose_for_scores(mixed_query_layer)
434
+
435
+ if self.is_decoder:
436
+ # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.
437
+ # Further calls to cross_attention layer can then reuse all cross-attention
438
+ # key/value_states (first "if" case)
439
+ # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of
440
+ # all previous decoder key/value_states. Further calls to uni-directional self-attention
441
+ # can concat previous decoder key/value_states to current projected key/value_states (third "elif" case)
442
+ # if encoder bi-directional self-attention `past_key_value` is always `None`
443
+ past_key_value = (key_layer, value_layer)
444
+
445
+ bs, seq_len, _ = hidden_states.shape
446
+ # query shape: bs x n_heads x seq_len x head_dim
447
+ # key shape: bs x n_heads x head_dim x seq_len
448
+
449
+ if self.position_embedding_type == 'rotary':
450
+ # todo: in key_layer and value_layer past states already concatenated
451
+ # but rotary embeddings should not be applied to past states
452
+ if past_key_value is not None:
453
+ raise RuntimeError(f'past_key_values is not None are not supported in BertSelfAttention.forward with '
454
+ f'position_embedding_type = {self.position_embedding_type}.')
455
+ # traspose to bs x n_heads x seq_len x head_dim
456
+ key_layer = key_layer.transpose(-1, -2)
457
+ if self.rotary_dim < self.attention_head_size:
458
+ query_rot = query_layer[..., :self.rotary_dim]
459
+ query_pass = query_layer[..., self.rotary_dim:]
460
+
461
+ key_rot = key_layer[..., :self.rotary_dim]
462
+ key_pass = key_layer[..., self.rotary_dim:]
463
+ else: # full rotary
464
+ query_rot = query_layer
465
+ key_rot = key_layer
466
+
467
+ cos, sin = self.rotary_emb(key_rot, seq_len=seq_len)
468
+ query_layer, key_layer = apply_rotary_pos_emb(query_rot, key_rot, cos, sin, offset=0)
469
+ if self.rotary_dim < self.attention_head_size:
470
+ query_layer = torch.cat((query_layer, query_pass), dim=-1)
471
+ key_layer = torch.cat((key_layer, key_pass), dim=-1)
472
+ # transpose to bs x n_heads x head_dim x seq_len
473
+ key_layer = key_layer.transpose(-1, -2)
474
+
475
+ if not self.is_sparse:
476
+ # Take the dot product between "query" and "key" to get the raw attention scores.
477
+ attention_scores = torch.matmul(query_layer, key_layer)
478
+
479
+ if self.position_embedding_type in ["relative_key", "relative_key_query"]:
480
+ position_ids_l = torch.arange(seq_len, dtype=torch.long, device=hidden_states.device).view(-1, 1)
481
+ position_ids_r = torch.arange(seq_len, dtype=torch.long, device=hidden_states.device).view(1, -1)
482
+ distance = position_ids_l - position_ids_r
483
+ positional_embedding = self.distance_embedding(distance + self.max_position_embeddings - 1)
484
+ positional_embedding = positional_embedding.to(dtype=query_layer.dtype) # fp16 compatibility
485
+
486
+ # https://arxiv.org/abs/2009.13658
487
+ if self.position_embedding_type == "relative_key":
488
+ relative_position_scores = torch.einsum("bhld,lrd->bhlr", query_layer, positional_embedding)
489
+ attention_scores = attention_scores + relative_position_scores
490
+ elif self.position_embedding_type == "relative_key_query":
491
+ relative_position_scores_query = torch.einsum("bhld,lrd->bhlr", query_layer, positional_embedding)
492
+ relative_position_scores_key = torch.einsum("bhdr,lrd->bhlr", key_layer, positional_embedding)
493
+ attention_scores = attention_scores + relative_position_scores_query + relative_position_scores_key
494
+ elif self.position_embedding_type == 'relative_attention_bias':
495
+ position_bias = self.get_relative_attention_bias(position_bias, bs, seq_len, seq_len)
496
+ attention_scores = attention_scores + position_bias
497
+
498
+ attention_scores = attention_scores / math.sqrt(self.attention_head_size)
499
+ if attention_mask is not None:
500
+ # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
501
+ attention_scores = attention_scores + attention_mask
502
+
503
+ # Normalize the attention scores to probabilities.
504
+ attention_probs = self.softmax(attention_scores)
505
+
506
+ # This is actually dropping out entire tokens to attend to, which might
507
+ # seem a bit unusual, but is taken from the original Transformer paper.
508
+ attention_probs = self.dropout(attention_probs)
509
+
510
+ # Mask heads if we want to
511
+ if head_mask is not None:
512
+ attention_probs = attention_probs * head_mask
513
+
514
+ context_layer = torch.matmul(attention_probs, value_layer)
515
+ else:
516
+ # sparse attention
517
+ # todo: return attention_probs
518
+ # todo: support relative_key -> need to change einsum with sparse operators..
519
+ # sparse attention supports masks with following shapes:
520
+ # key_padding_mask: (bs x seq_len) or (bs x 1 x 1 x seq_len)
521
+ # attention_mask: seq_len x seq_len or (1 x 1 x seq_len x seq_len)
522
+ if self.position_embedding_type == 'relative_attention_bias':
523
+ position_bias = self.get_relative_attention_bias(position_bias, bs, seq_len, seq_len)
524
+
525
+ query_dtype = query_layer.dtype
526
+ if query_dtype != torch.half:
527
+ # deepspeed sparse_self_attention supports only fp16 inputs
528
+ # manually cast to half in case if running in fp32 or O1 modes
529
+ query_layer, key_layer, value_layer = query_layer.half(), key_layer.half(), value_layer.half()
530
+ # attention_mask = attention_mask.half()
531
+ if position_bias is not None:
532
+ position_bias = position_bias.half()
533
+ context_layer = self.sparse_self_attention(query_layer, key_layer, value_layer, rpe=position_bias,
534
+ key_padding_mask=attention_mask)
535
+ if query_dtype == torch.float:
536
+ context_layer = context_layer.float()
537
+
538
+ context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
539
+ new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
540
+ context_layer = context_layer.view(new_context_layer_shape)
541
+
542
+ if self.is_sparse and output_attentions:
543
+ # todo: return sparse attention_scores or None, to not break the run
544
+ raise RuntimeError(f'SparseAttention does not support output_attention = {output_attentions}')
545
+
546
+ outputs = (context_layer, attention_probs) if output_attentions else (context_layer,)
547
+
548
+ if self.position_embedding_type == 'relative_attention_bias':
549
+ outputs = outputs + (position_bias,)
550
+
551
+ if self.is_decoder:
552
+ outputs = outputs + (past_key_value,)
553
+ return outputs
554
+
555
+
556
+ class BertSelfOutput(nn.Module):
557
+ def __init__(self, config):
558
+ super().__init__()
559
+ self.pre_layer_norm = getattr(config, 'pre_layer_norm', False)
560
+ self.bert_output_layer = True
561
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
562
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
563
+ if not self.pre_layer_norm:
564
+ self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
565
+
566
+ def forward(self, hidden_states, input_tensor):
567
+ hidden_states = self.dense(hidden_states)
568
+ hidden_states = self.dropout(hidden_states)
569
+ if not self.pre_layer_norm:
570
+ hidden_states = self.LayerNorm(hidden_states + input_tensor)
571
+ return hidden_states
572
+
573
+
574
+ class BertAttention(nn.Module):
575
+ def __init__(self, config, position_embedding_type=None, has_relative_attention_bias=False):
576
+ super().__init__()
577
+ self.self = BertSelfAttention(config, position_embedding_type=position_embedding_type,
578
+ has_relative_attention_bias=has_relative_attention_bias)
579
+ self.output = BertSelfOutput(config)
580
+ self.pruned_heads = set()
581
+
582
+ def prune_heads(self, heads):
583
+ if len(heads) == 0:
584
+ return
585
+ heads, index = find_pruneable_heads_and_indices(
586
+ heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads
587
+ )
588
+
589
+ # Prune linear layers
590
+ self.self.query = prune_linear_layer(self.self.query, index)
591
+ self.self.key = prune_linear_layer(self.self.key, index)
592
+ self.self.value = prune_linear_layer(self.self.value, index)
593
+ self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)
594
+
595
+ # Update hyper params and store pruned heads
596
+ self.self.num_attention_heads = self.self.num_attention_heads - len(heads)
597
+ self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads
598
+ self.pruned_heads = self.pruned_heads.union(heads)
599
+
600
+ def forward(
601
+ self,
602
+ hidden_states,
603
+ attention_mask=None,
604
+ head_mask=None,
605
+ encoder_hidden_states=None,
606
+ encoder_attention_mask=None,
607
+ past_key_value=None,
608
+ position_bias=None,
609
+ output_attentions=False,
610
+ ):
611
+ self_outputs = self.self(
612
+ hidden_states,
613
+ attention_mask,
614
+ head_mask,
615
+ encoder_hidden_states,
616
+ encoder_attention_mask,
617
+ past_key_value,
618
+ position_bias,
619
+ output_attentions,
620
+ )
621
+ attention_output = self.output(self_outputs[0], hidden_states)
622
+ outputs = (attention_output,) + self_outputs[1:] # add attentions if we output them
623
+ return outputs
624
+
625
+
626
+ class BertIntermediate(nn.Module):
627
+ def __init__(self, config):
628
+ super().__init__()
629
+ self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
630
+ if isinstance(config.hidden_act, str):
631
+ self.intermediate_act_fn = ACT2FN[config.hidden_act]
632
+ else:
633
+ self.intermediate_act_fn = config.hidden_act
634
+
635
+ def forward(self, hidden_states):
636
+ hidden_states = self.dense(hidden_states)
637
+ hidden_states = self.intermediate_act_fn(hidden_states)
638
+ return hidden_states
639
+
640
+
641
+ class BertOutput(nn.Module):
642
+ def __init__(self, config):
643
+ super().__init__()
644
+ self.pre_layer_norm = getattr(config, 'pre_layer_norm', False)
645
+ self.bert_output_layer = True
646
+ self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
647
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
648
+ if not self.pre_layer_norm:
649
+ self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
650
+
651
+ def forward(self, hidden_states, input_tensor):
652
+ hidden_states = self.dense(hidden_states)
653
+ hidden_states = self.dropout(hidden_states)
654
+ if not self.pre_layer_norm:
655
+ hidden_states = self.LayerNorm(hidden_states + input_tensor)
656
+ return hidden_states
657
+
658
+
659
+ class BertLayer(nn.Module):
660
+ def __init__(self, config, has_relative_attention_bias=False):
661
+ super().__init__()
662
+ self.chunk_size_feed_forward = config.chunk_size_feed_forward
663
+ self.seq_len_dim = 1
664
+ self.pre_layer_norm = getattr(config, 'pre_layer_norm', False)
665
+ if self.pre_layer_norm:
666
+ self.pre_attention_ln = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
667
+ self.post_attention_ln = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
668
+ self.attention = BertAttention(config, has_relative_attention_bias=has_relative_attention_bias)
669
+ self.is_decoder = config.is_decoder
670
+ self.add_cross_attention = config.add_cross_attention
671
+ if self.add_cross_attention:
672
+ if not self.is_decoder:
673
+ raise ValueError(f"{self} should be used as a decoder model if cross attention is added")
674
+ self.crossattention = BertAttention(config, position_embedding_type="absolute")
675
+ self.intermediate = BertIntermediate(config)
676
+ self.output = BertOutput(config)
677
+
678
+ def forward(
679
+ self,
680
+ hidden_states,
681
+ attention_mask=None,
682
+ head_mask=None,
683
+ encoder_hidden_states=None,
684
+ encoder_attention_mask=None,
685
+ past_key_value=None,
686
+ position_bias=None,
687
+ output_attentions=False,
688
+ ):
689
+ # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
690
+ self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
691
+ self_attention_outputs = self.attention(
692
+ hidden_states if not self.pre_layer_norm else self.pre_attention_ln(hidden_states),
693
+ attention_mask,
694
+ head_mask,
695
+ position_bias=position_bias,
696
+ output_attentions=output_attentions,
697
+ past_key_value=self_attn_past_key_value,
698
+ )
699
+ attention_output = self_attention_outputs[0]
700
+
701
+ # if decoder, the last output is tuple of self-attn cache
702
+ if self.is_decoder:
703
+ outputs = self_attention_outputs[1:-1]
704
+ present_key_value = self_attention_outputs[-1]
705
+ else:
706
+ outputs = self_attention_outputs[1:] # add self attentions if we output attention weights
707
+
708
+ cross_attn_present_key_value = None
709
+ if self.is_decoder and encoder_hidden_states is not None:
710
+ if not hasattr(self, "crossattention"):
711
+ raise ValueError(
712
+ f"If `encoder_hidden_states` are passed, {self} has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`"
713
+ )
714
+
715
+ # cross_attn cached key/values tuple is at positions 3,4 of past_key_value tuple
716
+ cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
717
+ cross_attention_outputs = self.crossattention(
718
+ attention_output,
719
+ attention_mask,
720
+ head_mask,
721
+ encoder_hidden_states,
722
+ encoder_attention_mask,
723
+ cross_attn_past_key_value,
724
+ position_bias,
725
+ output_attentions,
726
+ )
727
+ attention_output = cross_attention_outputs[0]
728
+ outputs = outputs + cross_attention_outputs[1:-1] # add cross attentions if we output attention weights
729
+
730
+ # add cross-attn cache to positions 3,4 of present_key_value tuple
731
+ cross_attn_present_key_value = cross_attention_outputs[-1]
732
+ present_key_value = present_key_value + cross_attn_present_key_value
733
+
734
+ if self.pre_layer_norm:
735
+ attention_output = hidden_states + attention_output
736
+
737
+ layer_output = apply_chunking_to_forward(
738
+ self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
739
+ )
740
+
741
+ outputs = (layer_output,) + outputs
742
+
743
+ # if decoder, return the attn key/values as the last output
744
+ if self.is_decoder:
745
+ outputs = outputs + (present_key_value,)
746
+
747
+ return outputs
748
+
749
+ def feed_forward_chunk(self, attention_output):
750
+ intermediate_inp = attention_output if not self.pre_layer_norm else self.post_attention_ln(attention_output)
751
+ intermediate_output = self.intermediate(intermediate_inp)
752
+ layer_output = self.output(intermediate_output, attention_output)
753
+ if self.pre_layer_norm:
754
+ layer_output = layer_output + attention_output
755
+ return layer_output
756
+
757
+
758
+ class BertEncoder(nn.Module):
759
+ def __init__(self, config):
760
+ super().__init__()
761
+ self.config = config
762
+ self.pre_layer_norm = getattr(config, 'pre_layer_norm', False)
763
+ # last_layer_ln is used with pre_layer_norm:
764
+ # pre_layer_norm: https://arxiv.org/abs/2002.04745
765
+ # x = x + mha(ln(x))
766
+ # x = x + ffn(mha)
767
+ # if last_layer:
768
+ # x = ln(x)
769
+ # post_layer_norm (standart bert):
770
+ # x = ln(x + mha(x))
771
+ # x = ln(x + ffn(x))
772
+ self.last_layer_norm = getattr(config, 'last_layer_norm', self.pre_layer_norm)
773
+ if not self.pre_layer_norm and self.last_layer_norm:
774
+ raise RuntimeError('last_layer_norm could be used only with pre_layer_norm=True')
775
+ self.layer = nn.ModuleList(
776
+ [BertLayer(config, has_relative_attention_bias=bool(i == 0)) for i in range(config.num_hidden_layers)]
777
+ )
778
+ self.gradient_checkpointing = False
779
+ if self.pre_layer_norm and self.last_layer_norm:
780
+ self.last_layer_ln = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
781
+
782
+ def forward(
783
+ self,
784
+ hidden_states,
785
+ attention_mask=None,
786
+ head_mask=None,
787
+ encoder_hidden_states=None,
788
+ encoder_attention_mask=None,
789
+ past_key_values=None,
790
+ use_cache=None,
791
+ output_attentions=False,
792
+ output_hidden_states=False,
793
+ return_dict=True,
794
+ ):
795
+ all_hidden_states = () if output_hidden_states else None
796
+ all_self_attentions = () if output_attentions else None
797
+ all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None
798
+ position_bias = None
799
+
800
+ next_decoder_cache = () if use_cache else None
801
+ for i, layer_module in enumerate(self.layer):
802
+ if output_hidden_states:
803
+ all_hidden_states = all_hidden_states + (hidden_states,)
804
+
805
+ layer_head_mask = head_mask[i] if head_mask is not None else None
806
+ past_key_value = past_key_values[i] if past_key_values is not None else None
807
+
808
+ if self.gradient_checkpointing and self.training:
809
+
810
+ if use_cache:
811
+ logger.warning(
812
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
813
+ )
814
+ use_cache = False
815
+
816
+ def create_custom_forward(module):
817
+ def custom_forward(*inputs):
818
+ return module(*inputs, past_key_value, position_bias, output_attentions)
819
+
820
+ return custom_forward
821
+
822
+ layer_outputs = torch.utils.checkpoint.checkpoint(
823
+ create_custom_forward(layer_module),
824
+ hidden_states,
825
+ attention_mask,
826
+ layer_head_mask,
827
+ encoder_hidden_states,
828
+ encoder_attention_mask,
829
+ )
830
+ else:
831
+ layer_outputs = layer_module(
832
+ hidden_states,
833
+ attention_mask,
834
+ layer_head_mask,
835
+ encoder_hidden_states,
836
+ encoder_attention_mask,
837
+ past_key_value,
838
+ position_bias,
839
+ output_attentions,
840
+ )
841
+
842
+ hidden_states = layer_outputs[0]
843
+ if use_cache:
844
+ next_decoder_cache += (layer_outputs[-1],)
845
+ if output_attentions:
846
+ all_self_attentions = all_self_attentions + (layer_outputs[1],)
847
+ if self.config.add_cross_attention:
848
+ all_cross_attentions = all_cross_attentions + (layer_outputs[2],)
849
+
850
+ if self.config.position_embedding_type == 'relative_attention_bias':
851
+ if not output_attentions:
852
+ position_bias = layer_outputs[1]
853
+ else:
854
+ position_bias = layer_outputs[2]
855
+
856
+ if self.pre_layer_norm and self.last_layer_norm:
857
+ hidden_states = self.last_layer_ln(hidden_states)
858
+
859
+ if output_hidden_states:
860
+ all_hidden_states = all_hidden_states + (hidden_states,)
861
+
862
+ if not return_dict:
863
+ return tuple(
864
+ v
865
+ for v in [
866
+ hidden_states,
867
+ next_decoder_cache,
868
+ all_hidden_states,
869
+ all_self_attentions,
870
+ all_cross_attentions,
871
+ ]
872
+ if v is not None
873
+ )
874
+ return BaseModelOutputWithPastAndCrossAttentions(
875
+ last_hidden_state=hidden_states,
876
+ past_key_values=next_decoder_cache,
877
+ hidden_states=all_hidden_states,
878
+ attentions=all_self_attentions,
879
+ cross_attentions=all_cross_attentions,
880
+ )
881
+
882
+
883
+ class BertPooler(nn.Module):
884
+ def __init__(self, config):
885
+ super().__init__()
886
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
887
+ self.activation = nn.Tanh()
888
+
889
+ def forward(self, hidden_states):
890
+ # We "pool" the model by simply taking the hidden state corresponding
891
+ # to the first token.
892
+ first_token_tensor = hidden_states[:, 0]
893
+ pooled_output = self.dense(first_token_tensor)
894
+ pooled_output = self.activation(pooled_output)
895
+ return pooled_output
896
+
897
+
898
+ class BertPredictionHeadTransform(nn.Module):
899
+ def __init__(self, config):
900
+ super().__init__()
901
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
902
+ if isinstance(config.hidden_act, str):
903
+ self.transform_act_fn = ACT2FN[config.hidden_act]
904
+ else:
905
+ self.transform_act_fn = config.hidden_act
906
+ self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
907
+
908
+ def forward(self, hidden_states):
909
+ hidden_states = self.dense(hidden_states)
910
+ hidden_states = self.transform_act_fn(hidden_states)
911
+ hidden_states = self.LayerNorm(hidden_states)
912
+ return hidden_states
913
+
914
+
915
+ class BertLMPredictionHead(nn.Module):
916
+ def __init__(self, config):
917
+ super().__init__()
918
+ self.transform = BertPredictionHeadTransform(config)
919
+
920
+ # The output weights are the same as the input embeddings, but there is
921
+ # an output-only bias for each token.
922
+ self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
923
+
924
+ self.bias = nn.Parameter(torch.zeros(config.vocab_size))
925
+
926
+ # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
927
+ self.decoder.bias = self.bias
928
+
929
+ def forward(self, hidden_states):
930
+ hidden_states = self.transform(hidden_states)
931
+ hidden_states = self.decoder(hidden_states)
932
+ return hidden_states
933
+
934
+
935
+ class BertOnlyMLMHead(nn.Module):
936
+ def __init__(self, config):
937
+ super().__init__()
938
+ self.predictions = BertLMPredictionHead(config)
939
+
940
+ def forward(self, sequence_output):
941
+ prediction_scores = self.predictions(sequence_output)
942
+ return prediction_scores
943
+
944
+
945
+ class BertOnlyNSPHead(nn.Module):
946
+ def __init__(self, config):
947
+ super().__init__()
948
+ self.seq_relationship = nn.Linear(config.hidden_size, 2)
949
+
950
+ def forward(self, pooled_output):
951
+ seq_relationship_score = self.seq_relationship(pooled_output)
952
+ return seq_relationship_score
953
+
954
+
955
+ class BertPreTrainingHeads(nn.Module):
956
+ def __init__(self, config):
957
+ super().__init__()
958
+ self.predictions = BertLMPredictionHead(config)
959
+ self.seq_relationship = nn.Linear(config.hidden_size, 2)
960
+
961
+ def forward(self, sequence_output, pooled_output):
962
+ prediction_scores = self.predictions(sequence_output)
963
+ seq_relationship_score = self.seq_relationship(pooled_output)
964
+ return prediction_scores, seq_relationship_score
965
+
966
+
967
+ class BertPreTrainedModel(PreTrainedModel):
968
+ """
969
+ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
970
+ models.
971
+ """
972
+
973
+ config_class = BertConfig
974
+ load_tf_weights = load_tf_weights_in_bert
975
+ base_model_prefix = "bert"
976
+ supports_gradient_checkpointing = True
977
+ _keys_to_ignore_on_load_missing = [r"position_ids"]
978
+
979
+ def _init_weights(self, module):
980
+ """Initialize the weights"""
981
+ if isinstance(module, nn.Linear):
982
+ # Slightly different from the TF version which uses truncated_normal for initialization
983
+ # cf https://github.com/pytorch/pytorch/pull/5617
984
+ std = self.config.initializer_range
985
+ if hasattr(module, 'bert_output_layer') and self.config.pre_layer_norm:
986
+ std /= math.sqrt(2.0 * self.config.num_hidden_layers)
987
+ module.weight.data.normal_(mean=0.0, std=std)
988
+ if module.bias is not None:
989
+ module.bias.data.zero_()
990
+ elif isinstance(module, nn.Embedding):
991
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
992
+ if module.padding_idx is not None:
993
+ module.weight.data[module.padding_idx].zero_()
994
+ elif isinstance(module, nn.LayerNorm):
995
+ module.bias.data.zero_()
996
+ module.weight.data.fill_(1.0)
997
+
998
+ def _set_gradient_checkpointing(self, module, value=False):
999
+ if isinstance(module, BertEncoder):
1000
+ module.gradient_checkpointing = value
1001
+
1002
+
1003
+ @dataclass
1004
+ class BertForPreTrainingOutput(ModelOutput):
1005
+ """
1006
+ Output type of [`BertForPreTraining`].
1007
+
1008
+ Args:
1009
+ loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`):
1010
+ Total loss as the sum of the masked language modeling loss and the next sequence prediction
1011
+ (classification) loss.
1012
+ mlm_loss: masked language modeling loss
1013
+ nsp_loss: next sequence prediction loss
1014
+ prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
1015
+ Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
1016
+ seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`):
1017
+ Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation
1018
+ before SoftMax).
1019
+ hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
1020
+ Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of
1021
+ shape `(batch_size, sequence_length, hidden_size)`.
1022
+
1023
+ Hidden-states of the model at the output of each layer plus the initial embedding outputs.
1024
+ attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
1025
+ Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
1026
+ sequence_length)`.
1027
+
1028
+ Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
1029
+ heads.
1030
+ """
1031
+
1032
+ loss: Optional[torch.FloatTensor] = None
1033
+ mlm_loss: Optional[torch.FloatTensor] = None
1034
+ nsp_loss: Optional[torch.FloatTensor] = None
1035
+ prediction_logits: torch.FloatTensor = None
1036
+ seq_relationship_logits: torch.FloatTensor = None
1037
+ hidden_states: Optional[Tuple[torch.FloatTensor]] = None
1038
+ attentions: Optional[Tuple[torch.FloatTensor]] = None
1039
+
1040
+
1041
+ BERT_START_DOCSTRING = r"""
1042
+
1043
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
1044
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
1045
+ etc.)
1046
+
1047
+ This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
1048
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
1049
+ and behavior.
1050
+
1051
+ Parameters:
1052
+ config ([`BertConfig`]): Model configuration class with all the parameters of the model.
1053
+ Initializing with a config file does not load the weights associated with the model, only the
1054
+ configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
1055
+ """
1056
+
1057
+ BERT_INPUTS_DOCSTRING = r"""
1058
+ Args:
1059
+ input_ids (`torch.LongTensor` of shape `({0})`):
1060
+ Indices of input sequence tokens in the vocabulary.
1061
+
1062
+ Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and
1063
+ [`PreTrainedTokenizer.__call__`] for details.
1064
+
1065
+ [What are input IDs?](../glossary#input-ids)
1066
+ attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*):
1067
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
1068
+
1069
+ - 1 for tokens that are **not masked**,
1070
+ - 0 for tokens that are **masked**.
1071
+
1072
+ [What are attention masks?](../glossary#attention-mask)
1073
+ token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*):
1074
+ Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,
1075
+ 1]`:
1076
+
1077
+ - 0 corresponds to a *sentence A* token,
1078
+ - 1 corresponds to a *sentence B* token.
1079
+
1080
+ [What are token type IDs?](../glossary#token-type-ids)
1081
+ position_ids (`torch.LongTensor` of shape `({0})`, *optional*):
1082
+ Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
1083
+ config.max_position_embeddings - 1]`.
1084
+
1085
+ [What are position IDs?](../glossary#position-ids)
1086
+ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*):
1087
+ Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
1088
+
1089
+ - 1 indicates the head is **not masked**,
1090
+ - 0 indicates the head is **masked**.
1091
+
1092
+ inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*):
1093
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
1094
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
1095
+ model's internal embedding lookup matrix.
1096
+ output_attentions (`bool`, *optional*):
1097
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
1098
+ tensors for more detail.
1099
+ output_hidden_states (`bool`, *optional*):
1100
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
1101
+ more detail.
1102
+ return_dict (`bool`, *optional*):
1103
+ Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple.
1104
+ """
1105
+
1106
+
1107
+ @add_start_docstrings(
1108
+ "The bare Bert Model transformer outputting raw hidden-states without any specific head on top.",
1109
+ BERT_START_DOCSTRING,
1110
+ )
1111
+ class BertModel(BertPreTrainedModel):
1112
+ """
1113
+
1114
+ The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
1115
+ cross-attention is added between the self-attention layers, following the architecture described in [Attention is
1116
+ all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
1117
+ Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
1118
+
1119
+ To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set
1120
+ to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and
1121
+ `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass.
1122
+ """
1123
+
1124
+ def __init__(self, config, add_pooling_layer=True):
1125
+ super().__init__(config)
1126
+ self.config = config
1127
+
1128
+ if hasattr(config, 'sparse_attention'):
1129
+ self.is_sparse = True
1130
+ self.sparse_block_size = self.config.sparse_attention['block']
1131
+ else:
1132
+ self.is_sparse = False
1133
+
1134
+ if self.is_sparse and self.config.is_decoder:
1135
+ raise RuntimeError('SparseAttention with BertModel decoder is not currently supported!')
1136
+
1137
+ self.embeddings = BertEmbeddings(config)
1138
+ self.encoder = BertEncoder(config)
1139
+
1140
+ self.pooler = BertPooler(config) if add_pooling_layer else None
1141
+
1142
+ # Initialize weights and apply final processing
1143
+ self.post_init()
1144
+
1145
+ def get_input_embeddings(self):
1146
+ return self.embeddings.word_embeddings
1147
+
1148
+ def set_input_embeddings(self, value):
1149
+ self.embeddings.word_embeddings = value
1150
+
1151
+ def _prune_heads(self, heads_to_prune):
1152
+ """
1153
+ Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
1154
+ class PreTrainedModel
1155
+ """
1156
+ for layer, heads in heads_to_prune.items():
1157
+ self.encoder.layer[layer].attention.prune_heads(heads)
1158
+
1159
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
1160
+ @add_code_sample_docstrings(
1161
+ processor_class=_TOKENIZER_FOR_DOC,
1162
+ checkpoint=_CHECKPOINT_FOR_DOC,
1163
+ output_type=BaseModelOutputWithPoolingAndCrossAttentions,
1164
+ config_class=_CONFIG_FOR_DOC,
1165
+ )
1166
+ def forward(
1167
+ self,
1168
+ input_ids=None,
1169
+ attention_mask=None,
1170
+ token_type_ids=None,
1171
+ position_ids=None,
1172
+ head_mask=None,
1173
+ inputs_embeds=None,
1174
+ encoder_hidden_states=None,
1175
+ encoder_attention_mask=None,
1176
+ past_key_values=None,
1177
+ use_cache=None,
1178
+ output_attentions=None,
1179
+ output_hidden_states=None,
1180
+ return_dict=None,
1181
+ ):
1182
+ r"""
1183
+ encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
1184
+ Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
1185
+ the model is configured as a decoder.
1186
+ encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*):
1187
+ Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
1188
+ the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
1189
+
1190
+ - 1 for tokens that are **not masked**,
1191
+ - 0 for tokens that are **masked**.
1192
+ past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
1193
+ Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
1194
+
1195
+ If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that
1196
+ don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all
1197
+ `decoder_input_ids` of shape `(batch_size, sequence_length)`.
1198
+ use_cache (`bool`, *optional*):
1199
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
1200
+ `past_key_values`).
1201
+ """
1202
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
1203
+ output_hidden_states = (
1204
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
1205
+ )
1206
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1207
+
1208
+ if self.config.is_decoder:
1209
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
1210
+ else:
1211
+ use_cache = False
1212
+
1213
+ if input_ids is not None and inputs_embeds is not None:
1214
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
1215
+ elif input_ids is not None:
1216
+ input_shape = input_ids.size()
1217
+ elif inputs_embeds is not None:
1218
+ input_shape = inputs_embeds.size()[:-1]
1219
+ else:
1220
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
1221
+
1222
+ batch_size, seq_length = input_shape
1223
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
1224
+
1225
+ # if sparse attention is used, input sequence length should be divisible by block size
1226
+ if self.is_sparse and seq_length % self.sparse_block_size != 0:
1227
+ raise RuntimeError(f'BertModel with sparse attention is used, but seq_len = {seq_length} '
1228
+ f'is not divisible by block_size = {self.sparse_block_size}')
1229
+
1230
+ # past_key_values_length
1231
+ past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0
1232
+
1233
+ if attention_mask is None:
1234
+ attention_mask = torch.ones(((batch_size, seq_length + past_key_values_length)), device=device)
1235
+
1236
+ if token_type_ids is None:
1237
+ if hasattr(self.embeddings, "token_type_ids"):
1238
+ buffered_token_type_ids = self.embeddings.token_type_ids[:, :seq_length]
1239
+ buffered_token_type_ids_expanded = buffered_token_type_ids.expand(batch_size, seq_length)
1240
+ token_type_ids = buffered_token_type_ids_expanded
1241
+ else:
1242
+ token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device)
1243
+
1244
+ # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
1245
+ # ourselves in which case we just need to make it broadcastable to all heads.
1246
+ extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
1247
+
1248
+ # If a 2D or 3D attention mask is provided for the cross-attention
1249
+ # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
1250
+ if self.config.is_decoder and encoder_hidden_states is not None:
1251
+ encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.size()
1252
+ encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
1253
+ if encoder_attention_mask is None:
1254
+ encoder_attention_mask = torch.ones(encoder_hidden_shape, device=device)
1255
+ encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
1256
+ else:
1257
+ encoder_extended_attention_mask = None
1258
+
1259
+ # Prepare head mask if needed
1260
+ # 1.0 in head_mask indicate we keep the head
1261
+ # attention_probs has shape bsz x n_heads x N x N
1262
+ # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
1263
+ # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
1264
+ head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
1265
+
1266
+ embedding_output = self.embeddings(
1267
+ input_ids=input_ids,
1268
+ position_ids=position_ids,
1269
+ token_type_ids=token_type_ids,
1270
+ inputs_embeds=inputs_embeds,
1271
+ past_key_values_length=past_key_values_length,
1272
+ )
1273
+ encoder_outputs = self.encoder(
1274
+ embedding_output,
1275
+ attention_mask=extended_attention_mask,
1276
+ head_mask=head_mask,
1277
+ encoder_hidden_states=encoder_hidden_states,
1278
+ encoder_attention_mask=encoder_extended_attention_mask,
1279
+ past_key_values=past_key_values,
1280
+ use_cache=use_cache,
1281
+ output_attentions=output_attentions,
1282
+ output_hidden_states=output_hidden_states,
1283
+ return_dict=return_dict,
1284
+ )
1285
+ sequence_output = encoder_outputs[0]
1286
+ pooled_output = self.pooler(sequence_output) if self.pooler is not None else None
1287
+
1288
+ if not return_dict:
1289
+ return (sequence_output, pooled_output) + encoder_outputs[1:]
1290
+
1291
+ return BaseModelOutputWithPoolingAndCrossAttentions(
1292
+ last_hidden_state=sequence_output,
1293
+ pooler_output=pooled_output,
1294
+ past_key_values=encoder_outputs.past_key_values,
1295
+ hidden_states=encoder_outputs.hidden_states,
1296
+ attentions=encoder_outputs.attentions,
1297
+ cross_attentions=encoder_outputs.cross_attentions,
1298
+ )
1299
+
1300
+
1301
+ @add_start_docstrings(
1302
+ """
1303
+ Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next
1304
+ sentence prediction (classification)` head.
1305
+ """,
1306
+ BERT_START_DOCSTRING,
1307
+ )
1308
+ class BertForPreTraining(BertPreTrainedModel):
1309
+ def __init__(self, config):
1310
+ super().__init__(config)
1311
+
1312
+ self.bert = BertModel(config)
1313
+ self.cls = BertPreTrainingHeads(config)
1314
+
1315
+ # Initialize weights and apply final processing
1316
+ self.post_init()
1317
+
1318
+ def get_output_embeddings(self):
1319
+ return self.cls.predictions.decoder
1320
+
1321
+ def set_output_embeddings(self, new_embeddings):
1322
+ self.cls.predictions.decoder = new_embeddings
1323
+
1324
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
1325
+ @replace_return_docstrings(output_type=BertForPreTrainingOutput, config_class=_CONFIG_FOR_DOC)
1326
+ def forward(
1327
+ self,
1328
+ input_ids=None,
1329
+ attention_mask=None,
1330
+ token_type_ids=None,
1331
+ position_ids=None,
1332
+ head_mask=None,
1333
+ inputs_embeds=None,
1334
+ labels=None,
1335
+ next_sentence_label=None,
1336
+ output_attentions=None,
1337
+ output_hidden_states=None,
1338
+ return_dict=None,
1339
+ ):
1340
+ r"""
1341
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
1342
+ Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
1343
+ config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked),
1344
+ the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
1345
+ next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1346
+ Labels for computing the next sequence prediction (classification) loss. Input should be a sequence
1347
+ pair (see `input_ids` docstring) Indices should be in `[0, 1]`:
1348
+
1349
+ - 0 indicates sequence B is a continuation of sequence A,
1350
+ - 1 indicates sequence B is a random sequence.
1351
+ kwargs (`Dict[str, any]`, optional, defaults to *{}*):
1352
+ Used to hide legacy arguments that have been deprecated.
1353
+
1354
+ Returns:
1355
+
1356
+ Example:
1357
+
1358
+ ```python
1359
+ >>> from transformers import BertTokenizer, BertForPreTraining
1360
+ >>> import torch
1361
+
1362
+ >>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
1363
+ >>> model = BertForPreTraining.from_pretrained("bert-base-uncased")
1364
+
1365
+ >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
1366
+ >>> outputs = model(**inputs)
1367
+
1368
+ >>> prediction_logits = outputs.prediction_logits
1369
+ >>> seq_relationship_logits = outputs.seq_relationship_logits
1370
+ ```
1371
+ """
1372
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1373
+
1374
+ outputs = self.bert(
1375
+ input_ids,
1376
+ attention_mask=attention_mask,
1377
+ token_type_ids=token_type_ids,
1378
+ position_ids=position_ids,
1379
+ head_mask=head_mask,
1380
+ inputs_embeds=inputs_embeds,
1381
+ output_attentions=output_attentions,
1382
+ output_hidden_states=output_hidden_states,
1383
+ return_dict=return_dict,
1384
+ )
1385
+
1386
+ sequence_output, pooled_output = outputs[:2]
1387
+ prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)
1388
+
1389
+ total_loss = None
1390
+ masked_lm_loss = None
1391
+ next_sentence_loss = None
1392
+ if labels is not None and next_sentence_label is not None:
1393
+ loss_fct = CrossEntropyLoss()
1394
+ masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
1395
+ next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
1396
+ total_loss = masked_lm_loss + next_sentence_loss
1397
+
1398
+ if not return_dict:
1399
+ output = (prediction_scores, seq_relationship_score) + outputs[2:]
1400
+ return ((total_loss,) + output) if total_loss is not None else output
1401
+
1402
+ return BertForPreTrainingOutput(
1403
+ loss=total_loss,
1404
+ mlm_loss=masked_lm_loss,
1405
+ nsp_loss=next_sentence_loss,
1406
+ prediction_logits=prediction_scores,
1407
+ seq_relationship_logits=seq_relationship_score,
1408
+ hidden_states=outputs.hidden_states,
1409
+ attentions=outputs.attentions,
1410
+ )
1411
+
1412
+
1413
+ @add_start_docstrings(
1414
+ """Bert Model with a `language modeling` head on top for CLM fine-tuning.""", BERT_START_DOCSTRING
1415
+ )
1416
+ class BertLMHeadModel(BertPreTrainedModel):
1417
+
1418
+ _keys_to_ignore_on_load_unexpected = [r"pooler"]
1419
+ _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias"]
1420
+
1421
+ def __init__(self, config):
1422
+ super().__init__(config)
1423
+
1424
+ if not config.is_decoder:
1425
+ logger.warning("If you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`")
1426
+
1427
+ self.bert = BertModel(config, add_pooling_layer=False)
1428
+ self.cls = BertOnlyMLMHead(config)
1429
+
1430
+ # Initialize weights and apply final processing
1431
+ self.post_init()
1432
+
1433
+ def get_output_embeddings(self):
1434
+ return self.cls.predictions.decoder
1435
+
1436
+ def set_output_embeddings(self, new_embeddings):
1437
+ self.cls.predictions.decoder = new_embeddings
1438
+
1439
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
1440
+ @replace_return_docstrings(output_type=CausalLMOutputWithCrossAttentions, config_class=_CONFIG_FOR_DOC)
1441
+ def forward(
1442
+ self,
1443
+ input_ids=None,
1444
+ attention_mask=None,
1445
+ token_type_ids=None,
1446
+ position_ids=None,
1447
+ head_mask=None,
1448
+ inputs_embeds=None,
1449
+ encoder_hidden_states=None,
1450
+ encoder_attention_mask=None,
1451
+ labels=None,
1452
+ past_key_values=None,
1453
+ use_cache=None,
1454
+ output_attentions=None,
1455
+ output_hidden_states=None,
1456
+ return_dict=None,
1457
+ ):
1458
+ r"""
1459
+ encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
1460
+ Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention
1461
+ if the model is configured as a decoder.
1462
+ encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*):
1463
+ Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used
1464
+ in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
1465
+
1466
+ - 1 for tokens that are **not masked**,
1467
+ - 0 for tokens that are **masked**.
1468
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
1469
+ Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be
1470
+ in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100`
1471
+ are ignored (masked), the loss is only computed for the tokens with labels n `[0, ...,
1472
+ config.vocab_size]`
1473
+ past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
1474
+ Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up
1475
+ decoding.
1476
+
1477
+ If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those
1478
+ that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of
1479
+ all `decoder_input_ids` of shape `(batch_size, sequence_length)`.
1480
+ use_cache (`bool`, *optional*):
1481
+ If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
1482
+ (see `past_key_values`).
1483
+
1484
+ Returns:
1485
+
1486
+ Example:
1487
+
1488
+ ```python
1489
+ >>> from transformers import BertTokenizer, BertLMHeadModel, BertConfig
1490
+ >>> import torch
1491
+
1492
+ >>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
1493
+ >>> config = BertConfig.from_pretrained("bert-base-cased")
1494
+ >>> config.is_decoder = True
1495
+ >>> model = BertLMHeadModel.from_pretrained("bert-base-cased", config=config)
1496
+
1497
+ >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
1498
+ >>> outputs = model(**inputs)
1499
+
1500
+ >>> prediction_logits = outputs.logits
1501
+ ```
1502
+ """
1503
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1504
+ if labels is not None:
1505
+ use_cache = False
1506
+
1507
+ outputs = self.bert(
1508
+ input_ids,
1509
+ attention_mask=attention_mask,
1510
+ token_type_ids=token_type_ids,
1511
+ position_ids=position_ids,
1512
+ head_mask=head_mask,
1513
+ inputs_embeds=inputs_embeds,
1514
+ encoder_hidden_states=encoder_hidden_states,
1515
+ encoder_attention_mask=encoder_attention_mask,
1516
+ past_key_values=past_key_values,
1517
+ use_cache=use_cache,
1518
+ output_attentions=output_attentions,
1519
+ output_hidden_states=output_hidden_states,
1520
+ return_dict=return_dict,
1521
+ )
1522
+
1523
+ sequence_output = outputs[0]
1524
+ prediction_scores = self.cls(sequence_output)
1525
+
1526
+ lm_loss = None
1527
+ if labels is not None:
1528
+ # we are doing next-token prediction; shift prediction scores and input ids by one
1529
+ shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
1530
+ labels = labels[:, 1:].contiguous()
1531
+ loss_fct = CrossEntropyLoss()
1532
+ lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
1533
+
1534
+ if not return_dict:
1535
+ output = (prediction_scores,) + outputs[2:]
1536
+ return ((lm_loss,) + output) if lm_loss is not None else output
1537
+
1538
+ return CausalLMOutputWithCrossAttentions(
1539
+ loss=lm_loss,
1540
+ logits=prediction_scores,
1541
+ past_key_values=outputs.past_key_values,
1542
+ hidden_states=outputs.hidden_states,
1543
+ attentions=outputs.attentions,
1544
+ cross_attentions=outputs.cross_attentions,
1545
+ )
1546
+
1547
+ def prepare_inputs_for_generation(self, input_ids, past=None, attention_mask=None, **model_kwargs):
1548
+ input_shape = input_ids.shape
1549
+ # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
1550
+ if attention_mask is None:
1551
+ attention_mask = input_ids.new_ones(input_shape)
1552
+
1553
+ # cut decoder_input_ids if past is used
1554
+ if past is not None:
1555
+ input_ids = input_ids[:, -1:]
1556
+
1557
+ return {"input_ids": input_ids, "attention_mask": attention_mask, "past_key_values": past}
1558
+
1559
+ def _reorder_cache(self, past, beam_idx):
1560
+ reordered_past = ()
1561
+ for layer_past in past:
1562
+ reordered_past += (tuple(past_state.index_select(0, beam_idx) for past_state in layer_past),)
1563
+ return reordered_past
1564
+
1565
+
1566
+ @add_start_docstrings("""Bert Model with a `language modeling` head on top.""", BERT_START_DOCSTRING)
1567
+ class BertForMaskedLM(BertPreTrainedModel):
1568
+
1569
+ _keys_to_ignore_on_load_unexpected = [r"pooler"]
1570
+ _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias"]
1571
+
1572
+ def __init__(self, config):
1573
+ super().__init__(config)
1574
+
1575
+ if config.is_decoder:
1576
+ logger.warning(
1577
+ "If you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for "
1578
+ "bi-directional self-attention."
1579
+ )
1580
+
1581
+ self.bert = BertModel(config, add_pooling_layer=False)
1582
+ self.cls = BertOnlyMLMHead(config)
1583
+
1584
+ # Initialize weights and apply final processing
1585
+ self.post_init()
1586
+
1587
+ def get_output_embeddings(self):
1588
+ return self.cls.predictions.decoder
1589
+
1590
+ def set_output_embeddings(self, new_embeddings):
1591
+ self.cls.predictions.decoder = new_embeddings
1592
+
1593
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
1594
+ @add_code_sample_docstrings(
1595
+ processor_class=_TOKENIZER_FOR_DOC,
1596
+ checkpoint=_CHECKPOINT_FOR_DOC,
1597
+ output_type=MaskedLMOutput,
1598
+ config_class=_CONFIG_FOR_DOC,
1599
+ )
1600
+ def forward(
1601
+ self,
1602
+ input_ids=None,
1603
+ attention_mask=None,
1604
+ token_type_ids=None,
1605
+ position_ids=None,
1606
+ head_mask=None,
1607
+ inputs_embeds=None,
1608
+ encoder_hidden_states=None,
1609
+ encoder_attention_mask=None,
1610
+ labels=None,
1611
+ output_attentions=None,
1612
+ output_hidden_states=None,
1613
+ return_dict=None,
1614
+ ):
1615
+ r"""
1616
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
1617
+ Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
1618
+ config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
1619
+ loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
1620
+ """
1621
+
1622
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1623
+
1624
+ outputs = self.bert(
1625
+ input_ids,
1626
+ attention_mask=attention_mask,
1627
+ token_type_ids=token_type_ids,
1628
+ position_ids=position_ids,
1629
+ head_mask=head_mask,
1630
+ inputs_embeds=inputs_embeds,
1631
+ encoder_hidden_states=encoder_hidden_states,
1632
+ encoder_attention_mask=encoder_attention_mask,
1633
+ output_attentions=output_attentions,
1634
+ output_hidden_states=output_hidden_states,
1635
+ return_dict=return_dict,
1636
+ )
1637
+
1638
+ sequence_output = outputs[0]
1639
+ prediction_scores = self.cls(sequence_output)
1640
+
1641
+ masked_lm_loss = None
1642
+ if labels is not None:
1643
+ loss_fct = CrossEntropyLoss() # -100 index = padding token
1644
+ masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
1645
+
1646
+ if not return_dict:
1647
+ output = (prediction_scores,) + outputs[2:]
1648
+ return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output
1649
+
1650
+ return MaskedLMOutput(
1651
+ loss=masked_lm_loss,
1652
+ logits=prediction_scores,
1653
+ hidden_states=outputs.hidden_states,
1654
+ attentions=outputs.attentions,
1655
+ )
1656
+
1657
+ def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **model_kwargs):
1658
+ input_shape = input_ids.shape
1659
+ effective_batch_size = input_shape[0]
1660
+
1661
+ # add a dummy token
1662
+ if self.config.pad_token_id is None:
1663
+ raise ValueError("The PAD token should be defined for generation")
1664
+
1665
+ attention_mask = torch.cat([attention_mask, attention_mask.new_zeros((attention_mask.shape[0], 1))], dim=-1)
1666
+ dummy_token = torch.full(
1667
+ (effective_batch_size, 1), self.config.pad_token_id, dtype=torch.long, device=input_ids.device
1668
+ )
1669
+ input_ids = torch.cat([input_ids, dummy_token], dim=1)
1670
+
1671
+ return {"input_ids": input_ids, "attention_mask": attention_mask}
1672
+
1673
+
1674
+ @add_start_docstrings(
1675
+ """Bert Model with a `next sentence prediction (classification)` head on top.""",
1676
+ BERT_START_DOCSTRING,
1677
+ )
1678
+ class BertForNextSentencePrediction(BertPreTrainedModel):
1679
+ def __init__(self, config):
1680
+ super().__init__(config)
1681
+
1682
+ self.bert = BertModel(config)
1683
+ self.cls = BertOnlyNSPHead(config)
1684
+
1685
+ # Initialize weights and apply final processing
1686
+ self.post_init()
1687
+
1688
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
1689
+ @replace_return_docstrings(output_type=NextSentencePredictorOutput, config_class=_CONFIG_FOR_DOC)
1690
+ def forward(
1691
+ self,
1692
+ input_ids=None,
1693
+ attention_mask=None,
1694
+ token_type_ids=None,
1695
+ position_ids=None,
1696
+ head_mask=None,
1697
+ inputs_embeds=None,
1698
+ labels=None,
1699
+ output_attentions=None,
1700
+ output_hidden_states=None,
1701
+ return_dict=None,
1702
+ **kwargs,
1703
+ ):
1704
+ r"""
1705
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1706
+ Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair
1707
+ (see `input_ids` docstring). Indices should be in `[0, 1]`:
1708
+
1709
+ - 0 indicates sequence B is a continuation of sequence A,
1710
+ - 1 indicates sequence B is a random sequence.
1711
+
1712
+ Returns:
1713
+
1714
+ Example:
1715
+
1716
+ ```python
1717
+ >>> from transformers import BertTokenizer, BertForNextSentencePrediction
1718
+ >>> import torch
1719
+
1720
+ >>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
1721
+ >>> model = BertForNextSentencePrediction.from_pretrained("bert-base-uncased")
1722
+
1723
+ >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
1724
+ >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
1725
+ >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt")
1726
+
1727
+ >>> outputs = model(**encoding, labels=torch.LongTensor([1]))
1728
+ >>> logits = outputs.logits
1729
+ >>> assert logits[0, 0] < logits[0, 1] # next sentence was random
1730
+ ```
1731
+ """
1732
+
1733
+ if "next_sentence_label" in kwargs:
1734
+ warnings.warn(
1735
+ "The `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.",
1736
+ FutureWarning,
1737
+ )
1738
+ labels = kwargs.pop("next_sentence_label")
1739
+
1740
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1741
+
1742
+ outputs = self.bert(
1743
+ input_ids,
1744
+ attention_mask=attention_mask,
1745
+ token_type_ids=token_type_ids,
1746
+ position_ids=position_ids,
1747
+ head_mask=head_mask,
1748
+ inputs_embeds=inputs_embeds,
1749
+ output_attentions=output_attentions,
1750
+ output_hidden_states=output_hidden_states,
1751
+ return_dict=return_dict,
1752
+ )
1753
+
1754
+ pooled_output = outputs[1]
1755
+
1756
+ seq_relationship_scores = self.cls(pooled_output)
1757
+
1758
+ next_sentence_loss = None
1759
+ if labels is not None:
1760
+ loss_fct = CrossEntropyLoss()
1761
+ next_sentence_loss = loss_fct(seq_relationship_scores.view(-1, 2), labels.view(-1))
1762
+
1763
+ if not return_dict:
1764
+ output = (seq_relationship_scores,) + outputs[2:]
1765
+ return ((next_sentence_loss,) + output) if next_sentence_loss is not None else output
1766
+
1767
+ return NextSentencePredictorOutput(
1768
+ loss=next_sentence_loss,
1769
+ logits=seq_relationship_scores,
1770
+ hidden_states=outputs.hidden_states,
1771
+ attentions=outputs.attentions,
1772
+ )
1773
+
1774
+
1775
+ @add_start_docstrings(
1776
+ """
1777
+ Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled
1778
+ output) e.g. for GLUE tasks.
1779
+ """,
1780
+ BERT_START_DOCSTRING,
1781
+ )
1782
+ class BertForSequenceClassification(BertPreTrainedModel):
1783
+ def __init__(self, config):
1784
+ super().__init__(config)
1785
+ self.num_labels = config.num_labels
1786
+ self.config = config
1787
+
1788
+ self.bert = BertModel(config)
1789
+ classifier_dropout = (
1790
+ config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
1791
+ )
1792
+ self.dropout = nn.Dropout(classifier_dropout)
1793
+ self.classifier = nn.Linear(config.hidden_size, config.num_labels)
1794
+
1795
+ # Initialize weights and apply final processing
1796
+ self.post_init()
1797
+
1798
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
1799
+ @add_code_sample_docstrings(
1800
+ processor_class=_TOKENIZER_FOR_DOC,
1801
+ checkpoint=_CHECKPOINT_FOR_DOC,
1802
+ output_type=SequenceClassifierOutput,
1803
+ config_class=_CONFIG_FOR_DOC,
1804
+ )
1805
+ def forward(
1806
+ self,
1807
+ input_ids=None,
1808
+ attention_mask=None,
1809
+ token_type_ids=None,
1810
+ position_ids=None,
1811
+ head_mask=None,
1812
+ inputs_embeds=None,
1813
+ labels=None,
1814
+ pos_weight=None,
1815
+ output_attentions=None,
1816
+ output_hidden_states=None,
1817
+ return_dict=None,
1818
+ ):
1819
+ r"""
1820
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1821
+ Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
1822
+ config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
1823
+ `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
1824
+ """
1825
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1826
+
1827
+ outputs = self.bert(
1828
+ input_ids,
1829
+ attention_mask=attention_mask,
1830
+ token_type_ids=token_type_ids,
1831
+ position_ids=position_ids,
1832
+ head_mask=head_mask,
1833
+ inputs_embeds=inputs_embeds,
1834
+ output_attentions=output_attentions,
1835
+ output_hidden_states=output_hidden_states,
1836
+ return_dict=return_dict,
1837
+ )
1838
+
1839
+ pooled_output = outputs[1]
1840
+
1841
+ pooled_output = self.dropout(pooled_output)
1842
+ logits = self.classifier(pooled_output)
1843
+
1844
+ loss = None
1845
+ if labels is not None:
1846
+ if self.config.problem_type is None:
1847
+ if self.num_labels == 1:
1848
+ self.config.problem_type = "regression"
1849
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
1850
+ self.config.problem_type = "single_label_classification"
1851
+ else:
1852
+ self.config.problem_type = "multi_label_classification"
1853
+
1854
+ if self.config.problem_type == "regression":
1855
+ loss_fct = MSELoss()
1856
+ if self.num_labels == 1:
1857
+ loss = loss_fct(logits.squeeze(), labels.squeeze())
1858
+ else:
1859
+ loss = loss_fct(logits, labels)
1860
+ elif self.config.problem_type == "single_label_classification":
1861
+ loss_fct = CrossEntropyLoss()
1862
+ loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
1863
+ elif self.config.problem_type == "multi_label_classification":
1864
+ loss_fct = BCEWithLogitsLoss(pos_weight=pos_weight)
1865
+ loss = loss_fct(logits, labels)
1866
+ if not return_dict:
1867
+ output = (logits,) + outputs[2:]
1868
+ return ((loss,) + output) if loss is not None else output
1869
+
1870
+ return SequenceClassifierOutput(
1871
+ loss=loss,
1872
+ logits=logits,
1873
+ hidden_states=outputs.hidden_states,
1874
+ attentions=outputs.attentions,
1875
+ )
1876
+
1877
+
1878
+ @add_start_docstrings(
1879
+ """
1880
+ Bert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
1881
+ softmax) e.g. for RocStories/SWAG tasks.
1882
+ """,
1883
+ BERT_START_DOCSTRING,
1884
+ )
1885
+ class BertForMultipleChoice(BertPreTrainedModel):
1886
+ def __init__(self, config):
1887
+ super().__init__(config)
1888
+
1889
+ self.bert = BertModel(config)
1890
+ classifier_dropout = (
1891
+ config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
1892
+ )
1893
+ self.dropout = nn.Dropout(classifier_dropout)
1894
+ self.classifier = nn.Linear(config.hidden_size, 1)
1895
+
1896
+ # Initialize weights and apply final processing
1897
+ self.post_init()
1898
+
1899
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, num_choices, sequence_length"))
1900
+ @add_code_sample_docstrings(
1901
+ processor_class=_TOKENIZER_FOR_DOC,
1902
+ checkpoint=_CHECKPOINT_FOR_DOC,
1903
+ output_type=MultipleChoiceModelOutput,
1904
+ config_class=_CONFIG_FOR_DOC,
1905
+ )
1906
+ def forward(
1907
+ self,
1908
+ input_ids=None,
1909
+ attention_mask=None,
1910
+ token_type_ids=None,
1911
+ position_ids=None,
1912
+ head_mask=None,
1913
+ inputs_embeds=None,
1914
+ labels=None,
1915
+ output_attentions=None,
1916
+ output_hidden_states=None,
1917
+ return_dict=None,
1918
+ ):
1919
+ r"""
1920
+ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
1921
+ Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
1922
+ num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
1923
+ `input_ids` above)
1924
+ """
1925
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1926
+ num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]
1927
+
1928
+ input_ids = input_ids.view(-1, input_ids.size(-1)) if input_ids is not None else None
1929
+ attention_mask = attention_mask.view(-1, attention_mask.size(-1)) if attention_mask is not None else None
1930
+ token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) if token_type_ids is not None else None
1931
+ position_ids = position_ids.view(-1, position_ids.size(-1)) if position_ids is not None else None
1932
+ inputs_embeds = (
1933
+ inputs_embeds.view(-1, inputs_embeds.size(-2), inputs_embeds.size(-1))
1934
+ if inputs_embeds is not None
1935
+ else None
1936
+ )
1937
+
1938
+ outputs = self.bert(
1939
+ input_ids,
1940
+ attention_mask=attention_mask,
1941
+ token_type_ids=token_type_ids,
1942
+ position_ids=position_ids,
1943
+ head_mask=head_mask,
1944
+ inputs_embeds=inputs_embeds,
1945
+ output_attentions=output_attentions,
1946
+ output_hidden_states=output_hidden_states,
1947
+ return_dict=return_dict,
1948
+ )
1949
+
1950
+ pooled_output = outputs[1]
1951
+
1952
+ pooled_output = self.dropout(pooled_output)
1953
+ logits = self.classifier(pooled_output)
1954
+ reshaped_logits = logits.view(-1, num_choices)
1955
+
1956
+ loss = None
1957
+ if labels is not None:
1958
+ loss_fct = CrossEntropyLoss()
1959
+ loss = loss_fct(reshaped_logits, labels)
1960
+
1961
+ if not return_dict:
1962
+ output = (reshaped_logits,) + outputs[2:]
1963
+ return ((loss,) + output) if loss is not None else output
1964
+
1965
+ return MultipleChoiceModelOutput(
1966
+ loss=loss,
1967
+ logits=reshaped_logits,
1968
+ hidden_states=outputs.hidden_states,
1969
+ attentions=outputs.attentions,
1970
+ )
1971
+
1972
+
1973
+ @add_start_docstrings(
1974
+ """
1975
+ Bert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for
1976
+ Named-Entity-Recognition (NER) tasks.
1977
+ """,
1978
+ BERT_START_DOCSTRING,
1979
+ )
1980
+ class BertForTokenClassification(BertPreTrainedModel):
1981
+
1982
+ _keys_to_ignore_on_load_unexpected = [r"pooler"]
1983
+
1984
+ def __init__(self, config):
1985
+ super().__init__(config)
1986
+ self.num_labels = config.num_labels
1987
+ self.config = config
1988
+ if getattr(self.config, 'problem_type', None) is None:
1989
+ self.config.problem_type = 'single_label_classification'
1990
+
1991
+ self.bert = BertModel(config, add_pooling_layer=False)
1992
+ classifier_dropout = (
1993
+ config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
1994
+ )
1995
+ self.dropout = nn.Dropout(classifier_dropout)
1996
+ self.classifier = nn.Linear(config.hidden_size, config.num_labels)
1997
+
1998
+ # Initialize weights and apply final processing
1999
+ self.post_init()
2000
+
2001
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
2002
+ @add_code_sample_docstrings(
2003
+ processor_class=_TOKENIZER_FOR_DOC,
2004
+ checkpoint=_CHECKPOINT_FOR_DOC,
2005
+ output_type=TokenClassifierOutput,
2006
+ config_class=_CONFIG_FOR_DOC,
2007
+ )
2008
+ def forward(
2009
+ self,
2010
+ input_ids=None,
2011
+ attention_mask=None,
2012
+ token_type_ids=None,
2013
+ position_ids=None,
2014
+ head_mask=None,
2015
+ inputs_embeds=None,
2016
+ labels=None,
2017
+ labels_mask=None,
2018
+ pos_weight=None,
2019
+ output_attentions=None,
2020
+ output_hidden_states=None,
2021
+ return_dict=None,
2022
+ ):
2023
+ r"""
2024
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
2025
+ Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
2026
+ """
2027
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
2028
+
2029
+ outputs = self.bert(
2030
+ input_ids,
2031
+ attention_mask=attention_mask,
2032
+ token_type_ids=token_type_ids,
2033
+ position_ids=position_ids,
2034
+ head_mask=head_mask,
2035
+ inputs_embeds=inputs_embeds,
2036
+ output_attentions=output_attentions,
2037
+ output_hidden_states=output_hidden_states,
2038
+ return_dict=return_dict,
2039
+ )
2040
+
2041
+ sequence_output = outputs[0]
2042
+
2043
+ sequence_output = self.dropout(sequence_output)
2044
+ logits = self.classifier(sequence_output)
2045
+
2046
+ loss = None
2047
+ if labels is not None:
2048
+ if self.config.problem_type == 'single_label_classification':
2049
+ loss_fct = CrossEntropyLoss()
2050
+ loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
2051
+ elif self.config.problem_type == 'multi_label_classification':
2052
+ if labels_mask is None:
2053
+ loss_fct = BCEWithLogitsLoss(pos_weight=pos_weight)
2054
+ loss = loss_fct(logits, labels)
2055
+ else:
2056
+ loss_fct = BCEWithLogitsLoss(reduction='none', pos_weight=pos_weight)
2057
+ loss = loss_fct(logits, labels)
2058
+ loss = loss * labels_mask.unsqueeze(-1)
2059
+ loss = loss.sum() / labels_mask.sum() if labels_mask.sum() != 0.0 else torch.tensor(0.0, device=logits.device)
2060
+
2061
+ if not return_dict:
2062
+ output = (logits,) + outputs[2:]
2063
+ return ((loss,) + output) if loss is not None else output
2064
+
2065
+ return TokenClassifierOutput(
2066
+ loss=loss,
2067
+ logits=logits,
2068
+ hidden_states=outputs.hidden_states,
2069
+ attentions=outputs.attentions,
2070
+ )
2071
+
2072
+
2073
+ @add_start_docstrings(
2074
+ """
2075
+ Bert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
2076
+ layers on top of the hidden-states output to compute `span start logits` and `span end logits`).
2077
+ """,
2078
+ BERT_START_DOCSTRING,
2079
+ )
2080
+ class BertForQuestionAnswering(BertPreTrainedModel):
2081
+
2082
+ _keys_to_ignore_on_load_unexpected = [r"pooler"]
2083
+
2084
+ def __init__(self, config):
2085
+ super().__init__(config)
2086
+ self.num_labels = config.num_labels
2087
+
2088
+ self.bert = BertModel(config, add_pooling_layer=False)
2089
+ self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)
2090
+
2091
+ # Initialize weights and apply final processing
2092
+ self.post_init()
2093
+
2094
+ @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
2095
+ @add_code_sample_docstrings(
2096
+ processor_class=_TOKENIZER_FOR_DOC,
2097
+ checkpoint=_CHECKPOINT_FOR_DOC,
2098
+ output_type=QuestionAnsweringModelOutput,
2099
+ config_class=_CONFIG_FOR_DOC,
2100
+ )
2101
+ def forward(
2102
+ self,
2103
+ input_ids=None,
2104
+ attention_mask=None,
2105
+ token_type_ids=None,
2106
+ position_ids=None,
2107
+ head_mask=None,
2108
+ inputs_embeds=None,
2109
+ start_positions=None,
2110
+ end_positions=None,
2111
+ output_attentions=None,
2112
+ output_hidden_states=None,
2113
+ return_dict=None,
2114
+ ):
2115
+ r"""
2116
+ start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
2117
+ Labels for position (index) of the start of the labelled span for computing the token classification loss.
2118
+ Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
2119
+ are not taken into account for computing the loss.
2120
+ end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
2121
+ Labels for position (index) of the end of the labelled span for computing the token classification loss.
2122
+ Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
2123
+ are not taken into account for computing the loss.
2124
+ """
2125
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
2126
+
2127
+ outputs = self.bert(
2128
+ input_ids,
2129
+ attention_mask=attention_mask,
2130
+ token_type_ids=token_type_ids,
2131
+ position_ids=position_ids,
2132
+ head_mask=head_mask,
2133
+ inputs_embeds=inputs_embeds,
2134
+ output_attentions=output_attentions,
2135
+ output_hidden_states=output_hidden_states,
2136
+ return_dict=return_dict,
2137
+ )
2138
+
2139
+ sequence_output = outputs[0]
2140
+
2141
+ logits = self.qa_outputs(sequence_output)
2142
+ start_logits, end_logits = logits.split(1, dim=-1)
2143
+ start_logits = start_logits.squeeze(-1).contiguous()
2144
+ end_logits = end_logits.squeeze(-1).contiguous()
2145
+
2146
+ total_loss = None
2147
+ if start_positions is not None and end_positions is not None:
2148
+ # If we are on multi-GPU, split add a dimension
2149
+ if len(start_positions.size()) > 1:
2150
+ start_positions = start_positions.squeeze(-1)
2151
+ if len(end_positions.size()) > 1:
2152
+ end_positions = end_positions.squeeze(-1)
2153
+ # sometimes the start/end positions are outside our model inputs, we ignore these terms
2154
+ ignored_index = start_logits.size(1)
2155
+ start_positions = start_positions.clamp(0, ignored_index)
2156
+ end_positions = end_positions.clamp(0, ignored_index)
2157
+
2158
+ loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
2159
+ start_loss = loss_fct(start_logits, start_positions)
2160
+ end_loss = loss_fct(end_logits, end_positions)
2161
+ total_loss = (start_loss + end_loss) / 2
2162
+
2163
+ if not return_dict:
2164
+ output = (start_logits, end_logits) + outputs[2:]
2165
+ return ((total_loss,) + output) if total_loss is not None else output
2166
+
2167
+ return QuestionAnsweringModelOutput(
2168
+ loss=total_loss,
2169
+ start_logits=start_logits,
2170
+ end_logits=end_logits,
2171
+ hidden_states=outputs.hidden_states,
2172
+ attentions=outputs.attentions,
2173
+ )
2174
+
2175
+
2176
+ # todo: move to separate file with other position embeddings?
2177
+ # https://github.com/EleutherAI/gpt-neox/blob/8229d921d329266323706c01dd6778fa71649ac7/megatron/model/positional_embeddings.py#L24
2178
+ # https://blog.eleuther.ai/rotary-embeddings/
2179
+ class RotaryEmbedding(torch.nn.Module):
2180
+ def __init__(self, dim, base=10000):
2181
+ super().__init__()
2182
+ inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
2183
+ self.register_buffer("inv_freq", inv_freq)
2184
+ self.seq_len_cached = None
2185
+ self.cos_cached = None
2186
+ self.sin_cached = None
2187
+
2188
+ def forward(self, x, seq_dim=1, seq_len=None):
2189
+ if seq_len is None:
2190
+ seq_len = x.shape[seq_dim]
2191
+ if seq_len != self.seq_len_cached:
2192
+ self.seq_len_cached = seq_len
2193
+ t = torch.arange(seq_len, device=x.device).type_as(self.inv_freq)
2194
+ freqs = torch.einsum("i,j->ij", t, self.inv_freq)
2195
+ emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
2196
+ self.cos_cached = emb.cos()[None, None, :, :]
2197
+ self.sin_cached = emb.sin()[None, None, :, :]
2198
+ return self.cos_cached, self.sin_cached
2199
+
2200
+
2201
+ def rotate_half(x):
2202
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2:]
2203
+ return torch.cat((-x2, x1), dim=x1.ndim - 1) # dim=-1 triggers a bug in earlier torch versions
2204
+
2205
+
2206
+ def apply_rotary_pos_emb(q, k, cos, sin, offset: int = 0):
2207
+ cos, sin = cos[:, :, offset: q.shape[2] + offset, :], sin[:, :, offset: q.shape[2] + offset, :]
2208
+ return (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5effd8cb1c74689c397221cb81a5c2026dd3a9ac6d2037fcab0774812cc8af66
3
+ size 541130889
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"tokenizer_class": "PreTrainedTokenizerFast"}