Transformers
PyTorch
Graphcore
English
groupbert
Generated from Trainer
Inference Endpoints
File size: 7,757 Bytes
29e3760
 
 
feeaeac
 
 
29e3760
0485028
29e3760
feeaeac
 
 
29e3760
 
 
 
 
feeaeac
29e3760
feeaeac
 
 
29e3760
 
 
feeaeac
 
 
 
 
 
29e3760
 
feeaeac
29e3760
feeaeac
 
29e3760
 
 
feeaeac
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29e3760
 
 
feeaeac
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29e3760
 
feeaeac
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29e3760
 
 
 
 
 
 
 
 
 
bf20149
 
29e3760
 
 
 
 
 
 
 
 
 
bf20149
feeaeac
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
---
tags:
- generated_from_trainer
datasets:
- Graphcore/wikipedia-bert-128
- Graphcore/wikipedia-bert-512
model-index:
- name: groupbert-base-uncased
  results: []
license: apache-2.0
language:
- en
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Graphcore/groupbert-base-uncased 

Optimum Graphcore is a new open-source library and toolkit that enables developers to access IPU-optimized models certified by Hugging Face. It is an extension of Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on Graphcore’s IPUs - a completely new kind of massively parallel processor to accelerate machine intelligence. Learn more about how to take train Transformer models faster with IPUs at [hf.co/hardware/graphcore](https://huggingface.co/hardware/graphcore).

Through HuggingFace Optimum, Graphcore released ready-to-use IPU-trained model checkpoints and IPU configuration files to make it easy to train models with maximum efficiency in the IPU. Optimum shortens the development lifecycle of your AI models by letting you plug-and-play any public dataset and allows a seamless integration to our State-of-the-art hardware giving you a quicker time-to-value for your AI project.      

## Model description

GroupBERT (Bidirectional Encoder Representations from Transformers) is a transformers model which is designed by Graphcore to pretrain bidirectional representations from unlabelled texts. GroupBERT uses grouped convolutions and matmuls in the encoder, which allows to parallelize computation and achieve higher parameter efficiency. More details are described in the [GroupBERT paper](https://arxiv.org/pdf/2106.05822.pdf).    

It was trained with two objectives in pretraining : Masked language modelling (MLM) and Next sentence prediction(NSP). First, MLM is different from traditional LM which sees the words one after another while BERT allows the model to learn a bidirectional representation.  In addition to MLM, NSP is used for jointly pertaining text-pair representations. Similarly to BERT it enables easy and fast fine-tuning for different downstream tasks such as Sequence Classification, Named Entity Recognition, Question Answering, Multiple Choice and MaskedLM.

It reduces the need of many engineering efforts for building task specific architectures through pre-trained representation. And achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks.


## Intended uses & limitations
This model is a pre-trained GroupBERT-Base trained in two phases on the [Graphcore/wikipedia-bert-128](https://huggingface.co/datasets/Graphcore/wikipedia-bert-128) and [Graphcore/wikipedia-bert-512](https://huggingface.co/datasets/Graphcore/wikipedia-bert-512) datasets.

It was trained on a Graphcore IPU-POD16 using [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore).
Graphcore and Hugging Face are working together to make training of Transformer models on IPUs fast and easy. Learn more about how to take advantage of the power of Graphcore IPUs to train Transformers models at [hf.co/hardware/graphcore](https://huggingface.co/hardware/graphcore).

## Training and evaluation data

Trained on wikipedia datasets:
- [Graphcore/wikipedia-bert-128](https://huggingface.co/datasets/Graphcore/wikipedia-bert-128)
- [Graphcore/wikipedia-bert-512](https://huggingface.co/datasets/Graphcore/wikipedia-bert-512)

## Fine-tuning with these weights

These weights can be used in either `transformers` or [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore).

For example, to fine-tune the SQuAD v1 with `optimum-graphcore` you can do:

```
python examples/question-answering/run_qa.py \
--model_name_or_path Graphcore/groupbert-base-uncased \
--ipu_config_name Graphcore/groupbert-base-uncased \
--dataset_name squad \
--version_2_with_negative False \
--do_train \
--do_eval \
--pad_on_batch_axis \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 16 \
--gradient_accumulation_steps 10 \
--pod_type pod16 \
--learning_rate 4e-4 \
--max_seq_length 384 \
--doc_stride 128 \
--seed 42 \
--lr_scheduler_type linear \
--lamb \
--loss_scaling 64 \
--weight_decay 0.01 \
--warmup_ratio 0.1 \
--logging_steps 5 \
--save_steps -1 \
--dataloader_num_workers 64 \
--output_dir output/squad_groupbert_base
```

## Training procedure

Trained MLM and NSP pre-training scheme from [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://arxiv.org/abs/1904.00962).
Trained on a Graphcore IPU-POD16 using [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore).

It was trained with the IPUConfig [Graphcore/bert-base-ipu](https://huggingface.co/Graphcore/bert-base-ipu/).

Command lines:

Phase 1:
```
python examples/language-modeling/run_pretraining.py \
  --model_type groupbert \
  --tokenizer_name bert-base-uncased \
  --ipu_config_name Graphcore/bert-base-ipu \
  --dataset_name Graphcore/wikipedia-bert-128 \
  --do_train \
  --logging_steps 5 \
  --max_seq_length 128 \
  --max_steps 10500 \
  --is_already_preprocessed \
  --dataloader_num_workers 64 \
  --dataloader_mode async_rebatched \
  --lamb \
  --per_device_train_batch_size 8 \
  --gradient_accumulation_steps 2000 \
  --pod_type pod16 \
  --learning_rate 0.012 \
  --loss_scaling 16384 \
  --weight_decay 0.01 \
  --warmup_ratio 0.15 \
  --groupbert_schedule \
  --config_overrides "hidden_dropout_prob=0.0,attention_probs_dropout_prob=0.0" \
  --ipu_config_overrides device_iterations="1,matmul_proportion=0.22,layers_per_ipu=[1 3 4 4]" \
  --output_dir output-pretrain-groupbert-base-phase1
```

Phase 2:
```
python examples/language-modeling/run_pretraining.py \
  --model_type groupbert \
  --tokenizer_name bert-base-uncased \
  --ipu_config_name Graphcore/bert-base-ipu \
  --dataset_name Graphcore/wikipedia-bert-512 \
  --model_name_or_path ./output-pretrain-bert-base-phase1 \
  --do_train \
  --logging_steps 5 \
  --max_seq_length 512 \
  --max_steps 2038 \
  --is_already_preprocessed \
  --dataloader_num_workers 128 \
  --dataloader_mode async_rebatched \
  --lamb \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 2048 \
  --pod_type pod16 \
  --learning_rate 0.01 \
  --loss_scaling 128 \
  --weight_decay 0.01 \
  --warmup_ratio 0.15 \
  --groupbert_schedule \
  --config_overrides "hidden_dropout_prob=0.0,attention_probs_dropout_prob=0.0" \
  --ipu_config_overrides "device_iterations=1,embedding_serialization_factor=2,matmul_proportion=0.22,layers_per_ipu=[1 3 4 4]" \
  --output_dir output-pretrain-groupbert-base-phase2
```

### Training hyperparameters

The following hyperparameters were used during phase 1 training:
- learning_rate: 0.012
- train_batch_size: 8
- eval_batch_size: 1
- seed: 42
- distributed_type: IPU
- gradient_accumulation_steps: 200
- total_train_batch_size: 64000
- total_eval_batch_size: 20
- optimizer: LAMB
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.15
- training_steps: 10500
- training precision: Mixed Precision

The following hyperparameters were used during phase 2 training:
- learning_rate: 0.01
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- distributed_type: IPU
- gradient_accumulation_steps: 2048
- total_train_batch_size: 16384
- total_eval_batch_size: 20
- optimizer: LAMB
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.15
- training_steps: 2038
- training precision: Mixed Precision

### Training results



### Framework versions

- Transformers 4.20.1
- Pytorch 1.10.0+cpu
- Datasets 2.6.1
- Tokenizers 0.12.1