BioLinkBERT-large-mnli-snli

This model is a fine-tuned version of BioLinkBERT-large on the GLUE MNLI and SNLI dataset.

The results are

Model	Dataset	Acc
Roberta-large-mnli	MNLI dev mm	90.12
	MNLI dev m	90.59
	SNLI test	88.25
BioLinkBERT-large	MNLI dev mm	33.56
	MNLI dev m	33.18
	SNLI test	32.66
BioLinkBERT-large-mnli-snli	MNLI dev mm	85.75
	MNLI dev m	85.30
	SNLI test	89.82

The labels are "0": "entailment", "1": "neutral", "2": "contradiction"

For finetuning BioLinkBERT-large on

• only MNLI (i.e. not SNLI), see cnut1648/biolinkbert-mnli
• MedNLI, see cnut1648/biolinkbert-mednli
• resampled MNLI such that the size is the same as MedNLI, see cnut1648/biolinkbert-large-mnli-resampled

Training procedure

This model checkpoint is created by mnli.py with the following command:

CUDA_VISIBLE_DEVICES=0,1,2; python -m torch.distributed.launch \
    --nproc_per_node 3 mnli.py \
    --model_name_or_path michiyasunaga/BioLinkBERT-large --task_name mnli --add_snli \
    --do_train --max_seq_length 512 --fp16 --per_device_train_batch_size 16 --gradient_accumulation_steps 2 \
    --learning_rate 3e-5 --warmup_ratio 0.5 --num_train_epochs 10 \
    --output_dir ./biolinkbert_mnli_snli

which will create a folder biolinkbert_mnli_snli that contains the checkpoints.

And all checkpoints are evaluated on MNLI dev (mismatched and matched) & SNLI test by

name=biolinkbert_mnli_snli
root=$PWD/$name
mkdir $PWD/eval/$name
for run in $root/checkpoint-*; do
    step=$( echo $run | rg "checkpoint-(?P<step>\d+)" -or '$step')
    out="./eval/${name}/eval_$step"
    echo "eval of $step ---- save to $out";

    CUDA_VISIBLE_DEVICES=0; python mnli.py \
        --model_name_or_path $run --task_name mnli --add_snli \
        --do_eval --max_seq_length 512 --fp16 --report_to none \
        --per_device_eval_batch_size 8 --output_dir $out
done;

which will create a folder eval/biolinkbert_mnli_snli that contains evaluation results for MNLI dev & SNLI test.

The best checkpoint is then selected according to mnli-m:

import json
import os
from pathlib import Path
import pandas as pd
from rich.console import Console
from rich.table import Table

pwd = Path(__file__).parent.resolve()
name = "eval/biolinkbert_mnli"
results = []
for evalckpt in os.listdir(pwd / name):
    step = evalckpt.split("_")[1]
    with open(pwd / name / evalckpt / "all_results.json") as f:
        data = json.load(f)
    results.append([int(step), data["mnli-mm_eval_accuracy"] * 100, data["mnli_eval_accuracy"] * 100, data["snli-test_eval_accuracy"] * 100])
results = pd.DataFrame(results, columns=["step", "mnli-mm", "mnli-m", "snli"]).sort_values(by=['step'])
console = Console()
table = Table(name)
table.add_row(results.to_string(float_format=lambda _: '{:.3f}'.format(_)))
console.print(table)
best = results["mnli-m"].idxmax()
print(results.loc[best])

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.5
• num_epochs: 10.0
• mixed_precision_training: Native AMP

Framework versions

• Transformers 4.22.2
• Pytorch 1.12.1+cu113
• Datasets 2.4.0
• Tokenizers 0.12.1