model-index:
- name: Graphcore/gptj-mnli
results: []
datasets:
- multi_nli
tags:
- pytorch
- causal-lm
- text-generation
widget:
- text: >-
mnli hypothesis: Your contributions were of no help with our students'
education. premise: Your contribution helped make it possible for us to
provide our students with a quality education. target:
Graphcore/gptj-mnli
This model is the fine-tuned version of EleutherAI/gpt-j-6B on the MNLI dataset
MNLI dataset consists of pairs of sentences, a premise and a hypothesis. The task is to predict the relation between the premise and the hypothesis, which can be:
entailment: hypothesis follows from the premise,contradiction: hypothesis contradicts the premise,neutral: hypothesis and premise are unrelated.
The MNLI task is to take two sentences referred to as the hypothesis and the premise as input and decide if the sentences entail (support), are neutral (cover different subjects) or contradict each other.
We finetune the model as a Causal Language Model (CLM): given a sequence of tokens, the task is to predict the next token. To achieve this, we create a stylised prompt string, following the approach of T5 paper.
mnli hypothesis: {hypothesis} premise: {premise} target: {class_label} <|endoftext|>
For example:
mnli hypothesis: Your contributions were of no help with our students' education. premise: Your contribution helped make it possible for us to provide our students with a quality education. target: contradiction <|endoftext|>
Fine-tuning and validation data
Fine tuning is done using the train split of the GLUE MNLI dataset and the performance is measured using the validation_mismatched split.
validation_mismatched means validation examples are not derived from the same sources as those in the training set and therefore not closely resembling any of the examples seen at training time.
Fine-tuning procedure
Fine tuned on a Graphcore IPU-POD64 using popxl.
Prompt sentences are tokenized and packed together to form 1024 token sequences, following HF packing algorithm. No padding is used. Since the model is trained to predict the next token, labels are simply the input sequence shifted by one token. Given the training format, no extra care is needed to account for different sequences: the model does not need to know which sentence a token belongs to.
Fine-tuning hyperparameters
The following hyperparameters were used:
Framework versions
- Transformers
- Pytorch
- Datasets
- Tokenizers