multiCorp_5e-05_0404

This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext on the None dataset. It achieves the following results on the evaluation set:

• eval_loss: 0.0657
• eval_precision: 0.6398
• eval_recall: 0.6267
• eval_f1: 0.6332
• eval_accuracy: 0.9847
• eval_runtime: 39.7302
• eval_samples_per_second: 32.544
• eval_steps_per_second: 2.039
• epoch: 3.41
• step: 1100

Multi Corp Training,

model = AutoModelForTokenClassification.from_pretrained( "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext", num_labels=41, id2label=id2label, label2id=label2id )

training_args = TrainingArguments( report_to = 'wandb', # enable logging to W&B output_dir = runname, # output directory/ name for huggingface hub learning_rate=5e-5, per_device_train_batch_size=16, per_device_eval_batch_size=16, weight_decay=0.01, evaluation_strategy = 'steps', # check evaluation metrics at each epoch max_steps = 2000, logging_steps = 25, # we will log every 25 steps eval_steps = 25, # we will perform evaluation every 25 steps save_steps = 25, load_best_model_at_end=True, metric_for_best_model = 'eval_loss', greater_is_better = False, push_to_hub=True, run_name = runname # name of the W&B run )

trainer = Trainer( model=model, args=training_args, train_dataset=tokenized_data["train"], eval_dataset=tokenized_data["validation"], tokenizer=tokenizer, data_collator=data_collator, compute_metrics=compute_metrics, callbacks = [EarlyStoppingCallback(early_stopping_patience=6)] )

[1101/2000 1:00:33 < 49:32, 0.30 it/s, Epoch 3.41/7]

25 0.836100 0.201612 0.000000 0.000000 0.000000 0.973546 50 0.149500 0.154239 0.233246 0.124420 0.162277 0.972420 75 0.136300 0.138105 0.145299 0.094708 0.114671 0.972385 100 0.129900 0.123477 0.425243 0.203343 0.275126 0.975886 125 0.103100 0.118570 0.297553 0.321727 0.309168 0.974136 150 0.117300 0.113230 0.393373 0.214949 0.277995 0.977039 175 0.117500 0.106183 0.320082 0.291551 0.305151 0.975930 200 0.093800 0.102443 0.353604 0.291551 0.319593 0.975297 225 0.091900 0.105976 0.446684 0.318942 0.372156 0.977127 250 0.088700 0.093393 0.439173 0.335190 0.380200 0.977734 275 0.113300 0.097715 0.522222 0.218199 0.307793 0.977637 300 0.092900 0.085730 0.473552 0.349118 0.401924 0.979405 325 0.085700 0.091731 0.380009 0.409471 0.394190 0.976960 350 0.081700 0.086656 0.554161 0.389508 0.457470 0.980162 375 0.062400 0.083441 0.538000 0.374652 0.441708 0.980769 400 0.077500 0.085072 0.486742 0.477252 0.481950 0.978869 425 0.073000 0.078521 0.516658 0.467967 0.491108 0.981103 450 0.081000 0.077073 0.552381 0.430826 0.484090 0.981288 475 0.075100 0.078478 0.483887 0.446147 0.464251 0.980408 500 0.062800 0.073298 0.550633 0.484680 0.515556 0.982247 525 0.060600 0.069571 0.542723 0.536676 0.539683 0.982608 550 0.063900 0.071559 0.539832 0.506500 0.522635 0.981983 575 0.060700 0.068333 0.564646 0.519034 0.540881 0.982546 600 0.062900 0.072810 0.602013 0.416435 0.492316 0.981886 625 0.051300 0.071469 0.550901 0.525070 0.537675 0.982335 650 0.059500 0.067657 0.553466 0.478180 0.513076 0.982528 675 0.047500 0.067443 0.594739 0.566852 0.580461 0.983663 700 0.052100 0.065269 0.564447 0.546890 0.555529 0.983039 725 0.041500 0.067790 0.593516 0.552461 0.572253 0.983672 750 0.046300 0.067922 0.609038 0.538069 0.571358 0.983461 775 0.054300 0.064636 0.646725 0.582173 0.612753 0.984499 800 0.049500 0.067722 0.650905 0.517642 0.576674 0.983830 825 0.043100 0.069327 0.630043 0.471216 0.539177 0.982880 850 0.048000 0.063814 0.631025 0.528784 0.575398 0.984068 875 0.042500 0.064527 0.644913 0.582637 0.612195 0.984543 900 0.043500 0.065475 0.608295 0.490251 0.542931 0.983522 925 0.039200 0.066043 0.635938 0.566852 0.599411 0.984323 950 0.046800 0.062491 0.646930 0.547818 0.593263 0.984719 975 0.043700 0.061204 0.634625 0.585422 0.609032 0.984543 1000 0.032000 0.066377 0.643390 0.560353 0.599007 0.984349 1025 0.038100 0.064764 0.666482 0.559424 0.608279 0.984745 1050 0.035300 0.065642 0.635359 0.587279 0.610374 0.984464 1075 0.032800 0.064835 0.657262 0.584030 0.618486 0.984587 1100 0.031700 0.065726 0.639810 0.626741 0.633208 0.984710

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 2000

Framework versions

• Transformers 4.27.4
• Pytorch 2.0.0+cu118
• Datasets 2.11.0
• Tokenizers 0.13.2