Graphcore and Hugging Face are working together to make training of Transformer models on IPUs fast and easy. Learn more about how to take advantage of the power of Graphcore IPUs to train Transformers models at hf.co/hardware/graphcore.

RoBERTa Base model IPU config

This model contains just the IPUConfig files for running the roberta-base model on Graphcore IPUs.

This model contains no model weights, only an IPUConfig.

Model description

RoBERTa is based on BERT pretrain approach but it t evaluates carefully a number of design decisions of BERT pretraining approach so that it found it is undertrained.

It suggested a way to improve the performance by training the model longer, with bigger batches over more data, removing the next sentence prediction objectives, training on longer sequences and dynamically changing mask pattern applied to the training data.

As a result, it achieves state-of-the-art results on GLUE, RACE and SQuAD and so on on.

Paper link : RoBERTa: A Robustly Optimized BERT Pretraining Approach

Usage

from optimum.graphcore import IPUConfig
ipu_config = IPUConfig.from_pretrained("Graphcore/roberta-base-ipu")