Edit model card

mt5-small-finetuned-Big-Patent-h

This model is a fine-tuned version of google/mt5-small on the big_patent dataset. It achieves the following results on the evaluation set:

  • Loss: 2.2622
  • Rouge1: 33.9091
  • Rouge2: 14.1731
  • Rougel: 30.105
  • Rougelsum: 30.3666

Model description

In this project, we fine-tuned mT5small, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. The model was fine-tuned on the electric patent corpus using a variety of techniques, including transfer learning, data augmentation, and hyperparameter tuning.

Intended uses & limitations

The fine-tuned model showed significant improvements in performance on the electric patent-specific tasks compared to the original pre-trained model.

Note: This project is suitable for researchers who are working on electric patent, as it's fine-tuned on electric patents and it can be used for related NLP problems for electric patent and electric patent research.

Training and evaluation data

A subset of electric patents were used to fine-tune the model.

The fine-tuned model was evaluated using the ROUGE metric on a variety of natural language processing tasks specific to the patent domain, including, named entity recognition, and summarization.

Training procedure

The model was fine-tuned on the electric patent corpus using a variety of techniques, including transfer learning, data augmentation, and hyperparameter tuning.

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5.6e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 8

Training results

Training Loss Epoch Step Validation Loss Rouge1 Rouge2 Rougel Rougelsum
2.5817 1.0 1071 2.3830 32.8521 13.2087 29.5594 29.7744
2.5657 2.0 2142 2.3345 33.9434 14.0573 30.0135 30.2533
2.4915 3.0 3213 2.2761 33.2033 13.2053 29.5126 29.8023
2.4365 4.0 4284 2.3041 33.8649 13.6629 30.0377 30.257
2.3952 5.0 5355 2.2722 33.9208 13.8018 30.1035 30.3432
2.3628 6.0 6426 2.2850 33.883 13.9537 30.0579 30.2417
2.3474 7.0 7497 2.2858 33.7201 14.0808 30.0762 30.255
2.331 8.0 8568 2.2622 33.9091 14.1731 30.105 30.3666

Framework versions

  • Transformers 4.24.0
  • Pytorch 1.12.1+cu113
  • Datasets 2.7.1
  • Tokenizers 0.13.2
Downloads last month
3

Dataset used to train AhiyaB/mt5-small-finetuned-Big-Patent-h

Evaluation results