Graphcore and Hugging Face are working together to make training of Transformer models on IPUs fast and easy. Learn more about how to take advantage of the power of Graphcore IPUs to train Transformers models at [hf.co/hardware/graphcore](https://huggingface.co/hardware/graphcore).

# RoBERTa Base model IPU config

This model contains just the `IPUConfig` files for running the [roberta-base](https://huggingface.co/roberta-base) model on Graphcore IPUs.

**This model contains no model weights, only an IPUConfig.** 

## Model description
RoBERTa is based on BERT pretrain approach but it  t evaluates carefully a number of design decisions of BERT pretraining approach so that it found it is undertrained.  

It suggested a way to improve the performance by training the model longer, with bigger batches over more data, removing the next sentence prediction objectives, training on longer sequences and dynamically changing mask pattern applied to the training data. 

As a result, it achieves state-of-the-art results on GLUE, RACE and SQuAD and so on on. 

Paper link : [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/pdf/1907.11692.pdf)

## Usage

```
from optimum.graphcore import IPUConfig
ipu_config = IPUConfig.from_pretrained("Graphcore/roberta-base-ipu")
```