---
tags:
- generated_from_trainer
model-index:
- name: training_bert
  results: []
license: mit
language:
- en
metrics:
- perplexity
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# training_bert

This model is a fine-tuned version of [Bert Base Uncased](https://huggingface.co/) on dataset composed of different jobs posted in several job platforms and thousands of resumes.
It achieves the following results on the evaluation set:
- Loss: 4.0495

## Model description

Pretraining done on bert base architecture. 

## Intended uses & limitations

This model can be used to generate contextual embeddings for textual data used in Applicant Tracking Systems such as resumes, jobs and cover letters. 
The embeddings can be further used to perform other NLP downstream tasks such as classification, Named Entity Recognition and so on.


## Training and evaluation data
THe training corpus is developed using about 40000 resumes and 2000 jobs posted scrapped from different job portals. This is a preliminary dataset 
for the experimentation. THe corpus size is about 2.35 GB of textual data. Similary evaluation data contains few resumes and jobs making about 12 mb of textual data.

## Training procedure
For the pretraining of masked language model, Trainer API from Huggingface is used. The pretraining took about 6 hrs 40 mins. 
### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 6.7862        | 0.11  | 500   | 6.9461          |
| 5.9428        | 0.22  | 1000  | 6.4640          |
| 5.5463        | 0.33  | 1500  | 6.2736          |
| 5.1871        | 0.44  | 2000  | 5.8517          |
| 4.896         | 0.55  | 2500  | 5.6070          |
| 4.6557        | 0.66  | 3000  | 5.4669          |
| 4.4832        | 0.77  | 3500  | 5.3318          |
| 4.3368        | 0.88  | 4000  | 5.2414          |
| 4.1887        | 0.99  | 4500  | 5.0666          |
| 4.053         | 1.1   | 5000  | 4.9532          |
| 3.9653        | 1.21  | 5500  | 4.8288          |
| 3.8865        | 1.33  | 6000  | 4.6741          |
| 3.8294        | 1.44  | 6500  | 4.7943          |
| 3.7565        | 1.55  | 7000  | 4.7336          |
| 3.673         | 1.66  | 7500  | 4.4760          |
| 3.6447        | 1.77  | 8000  | 4.5856          |
| 3.5808        | 1.88  | 8500  | 4.6133          |
| 3.5329        | 1.99  | 9000  | 4.4766          |
| 3.4916        | 2.1   | 9500  | 4.5085          |
| 3.4392        | 2.21  | 10000 | 4.5306          |
| 3.4333        | 2.32  | 10500 | 4.5433          |
| 3.3905        | 2.43  | 11000 | 4.1829          |
| 3.3701        | 2.54  | 11500 | 4.2976          |
| 3.3345        | 2.65  | 12000 | 4.2817          |
| 3.2815        | 2.76  | 12500 | 4.3146          |
| 3.2689        | 2.87  | 13000 | 4.2634          |
| 3.2401        | 2.98  | 13500 | 4.0907          |
| 3.2068        | 3.09  | 14000 | 4.1130          |
| 3.2097        | 3.2   | 14500 | 4.2001          |
| 3.1627        | 3.31  | 15000 | 4.0852          |
| 3.1647        | 3.42  | 15500 | 4.0383          |
| 3.1294        | 3.53  | 16000 | 3.9377          |
| 3.1166        | 3.64  | 16500 | 4.0733          |
| 3.1028        | 3.75  | 17000 | 3.8429          |
| 3.0903        | 3.86  | 17500 | 4.1127          |
| 3.0877        | 3.98  | 18000 | 3.8605          |
| 3.0407        | 4.09  | 18500 | 3.8482          |
| 3.0452        | 4.2   | 19000 | 4.0345          |
| 3.0496        | 4.31  | 19500 | 3.8602          |
| 3.0229        | 4.42  | 20000 | 4.2268          |
| 3.0157        | 4.53  | 20500 | 3.8028          |
| 3.0037        | 4.64  | 21000 | 3.8668          |
| 2.9992        | 4.75  | 21500 | 3.9542          |
| 3.016         | 4.86  | 22000 | 3.9090          |
| 2.9804        | 4.97  | 22500 | 4.0495          |


### Framework versions

- Transformers 4.25.1
- Pytorch 1.8.0+cu111
- Datasets 2.7.1
- Tokenizers 0.13.2