Update README.md

README.md

@@ -13,6 +13,9 @@ model-index:
 
 This model is a pre-trained BERT-Base trained in two phases on the [Graphcore/wikipedia-bert-128](https://huggingface.co/datasets/Graphcore/wikipedia-bert-128) and [Graphcore/wikipedia-bert-512](https://huggingface.co/datasets/Graphcore/wikipedia-bert-512) datasets.
 
+It was trained on a Graphcore IPU-POD16 using [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore).
+Graphcore and Hugging Face are working together to make training of Transformer models on IPUs fast and easy. Learn more about how to take advantage of the power of Graphcore IPUs to train Transformers models at [hf.co/hardware/graphcore](https://huggingface.co/hardware/graphcore).
+
 ## Model description
 
 Pre-trained BERT Base model trained on Wikipedia data.
@@ -28,7 +31,9 @@ Trained on wikipedia datasets:
 ## Training procedure
 
 Trained MLM and NSP pre-training scheme from [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://arxiv.org/abs/1904.00962).
-Trained on 16 Graphcore Mk2 IPUs using [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore)
+Trained on a Graphcore IPU-POD16 using [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore).
+
+It was trained with the IPUConfig [Graphcore/bert-base-ipu](https://huggingface.co/Graphcore/bert-base-ipu/).
 
 Command lines:
 
@@ -37,11 +42,11 @@ Phase 1:
 python examples/language-modeling/run_pretraining.py \
   --config_name bert-base-uncased \
   --tokenizer_name bert-base-uncased \
+  --ipu_config_name Graphcore/bert-base-ipu \
+  --dataset_name Graphcore/wikipedia-bert-128 \
   --do_train \
   --logging_steps 5 \
   --max_seq_length 128 \
-  --ipu_config_name Graphcore/bert-base-ipu \
-  --dataset_name Graphcore/wikipedia-bert-128 \
   --max_steps 10500 \
   --is_already_preprocessed \
   --dataloader_num_workers 64 \
@@ -66,12 +71,12 @@ Phase 2:
 python examples/language-modeling/run_pretraining.py \
   --config_name bert-base-uncased \
   --tokenizer_name bert-base-uncased \
+  --ipu_config_name Graphcore/bert-base-ipu \
+  --dataset_name Graphcore/wikipedia-bert-512 \
   --model_name_or_path ./output-pretrain-bert-base-phase1 \
   --do_train \
   --logging_steps 5 \
   --max_seq_length 512 \
-  --ipu_config_name Graphcore/bert-base-ipu \
-  --dataset_name Graphcore/wikipedia-bert-512 \
   --max_steps 2038 \
   --is_already_preprocessed \
   --dataloader_num_workers 128 \
@@ -129,3 +134,34 @@ The following hyperparameters were used during phase 2 training:
 - Pytorch 1.10.0+cpu
 - Datasets 1.18.3.dev0
 - Tokenizers 0.10.3
+
+## Fine-tuning with these weights
+
+These weights can be used in either `transformers` or [`optimum-graphcore`](https://github.com/huggingface/optimum-graphcore).
+
+For example, to fine-tune the GLUE task SST2 with `optimum-graphcore` you can do:
+
+```
+export TOKENIZERS_PARALLELISM=true
+python examples/text-classification/run_glue.py \
+  --model_name_or_path bert-base-uncased \
+  --ipu_config_name Graphcore/bert-base-ipu \
+  --task_name sst2 \
+  --do_train \
+  --do_eval \
+  --max_seq_length 128 \
+  --per_device_train_batch_size 1 \
+  --per_device_eval_batch_size 4 \
+  --gradient_accumulation_steps 32 \
+  --pod_type pod4 \
+  --learning_rate 2e-5 \
+  --lr_scheduler_type linear \
+  --warmup_ratio 0.25 \
+  --num_train_epochs 3 \
+  --seed 1984 \
+  --save_steps -1 \
+  --dataloader_num_workers 64 \
+  --dataloader_drop_last \
+  --overwrite_output_dir \
+  --output_dir /tmp/sst2
+```