Model save

README.md

@@ -0,0 +1,122 @@
+---
+license: apache-2.0
+base_model: distilbert-base-uncased
+tags:
+- generated_from_trainer
+model-index:
+- name: distilbert_finetuned_ai4privacy_v2
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# distilbert_finetuned_ai4privacy_v2
+
+This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.0451
+- Overall Precision: 0.9438
+- Overall Recall: 0.9663
+- Overall F1: 0.9549
+- Overall Accuracy: 0.9838
+- Accountname F1: 0.9946
+- Accountnumber F1: 0.9940
+- Age F1: 0.9624
+- Amount F1: 0.9643
+- Bic F1: 0.9929
+- Bitcoinaddress F1: 0.9948
+- Buildingnumber F1: 0.9845
+- City F1: 0.9955
+- Companyname F1: 0.9962
+- County F1: 0.9877
+- Creditcardcvv F1: 0.9643
+- Creditcardissuer F1: 0.9953
+- Creditcardnumber F1: 0.9793
+- Currency F1: 0.7811
+- Currencycode F1: 0.8850
+- Currencyname F1: 0.2281
+- Currencysymbol F1: 0.9562
+- Date F1: 0.9061
+- Dob F1: 0.7914
+- Email F1: 1.0
+- Ethereumaddress F1: 1.0
+- Eyecolor F1: 0.9837
+- Firstname F1: 0.9846
+- Gender F1: 0.9971
+- Height F1: 0.9910
+- Iban F1: 0.9906
+- Ip F1: 0.4349
+- Ipv4 F1: 0.8126
+- Ipv6 F1: 0.7679
+- Jobarea F1: 0.9880
+- Jobtitle F1: 0.9991
+- Jobtype F1: 0.9777
+- Lastname F1: 0.9684
+- Litecoinaddress F1: 0.9721
+- Mac F1: 1.0
+- Maskednumber F1: 0.9635
+- Middlename F1: 0.9330
+- Nearbygpscoordinate F1: 1.0
+- Ordinaldirection F1: 0.9910
+- Password F1: 1.0
+- Phoneimei F1: 0.9918
+- Phonenumber F1: 0.9962
+- Pin F1: 0.9477
+- Prefix F1: 0.9546
+- Secondaryaddress F1: 0.9892
+- Sex F1: 0.9876
+- Ssn F1: 0.9976
+- State F1: 0.9893
+- Street F1: 0.9873
+- Time F1: 0.9889
+- Url F1: 1.0
+- Useragent F1: 0.9953
+- Username F1: 0.9975
+- Vehiclevin F1: 1.0
+- Vehiclevrm F1: 1.0
+- Zipcode F1: 0.9873
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine_with_restarts
+- lr_scheduler_warmup_ratio: 0.2
+- num_epochs: 5
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy | Accountname F1 | Accountnumber F1 | Age F1 | Amount F1 | Bic F1 | Bitcoinaddress F1 | Buildingnumber F1 | City F1 | Companyname F1 | County F1 | Creditcardcvv F1 | Creditcardissuer F1 | Creditcardnumber F1 | Currency F1 | Currencycode F1 | Currencyname F1 | Currencysymbol F1 | Date F1 | Dob F1 | Email F1 | Ethereumaddress F1 | Eyecolor F1 | Firstname F1 | Gender F1 | Height F1 | Iban F1 | Ip F1  | Ipv4 F1 | Ipv6 F1 | Jobarea F1 | Jobtitle F1 | Jobtype F1 | Lastname F1 | Litecoinaddress F1 | Mac F1 | Maskednumber F1 | Middlename F1 | Nearbygpscoordinate F1 | Ordinaldirection F1 | Password F1 | Phoneimei F1 | Phonenumber F1 | Pin F1 | Prefix F1 | Secondaryaddress F1 | Sex F1 | Ssn F1 | State F1 | Street F1 | Time F1 | Url F1 | Useragent F1 | Username F1 | Vehiclevin F1 | Vehiclevrm F1 | Zipcode F1 |
+|:-------------:|:-----:|:----:|:---------------:|:-----------------:|:--------------:|:----------:|:----------------:|:--------------:|:----------------:|:------:|:---------:|:------:|:-----------------:|:-----------------:|:-------:|:--------------:|:---------:|:----------------:|:-------------------:|:-------------------:|:-----------:|:---------------:|:---------------:|:-----------------:|:-------:|:------:|:--------:|:------------------:|:-----------:|:------------:|:---------:|:---------:|:-------:|:------:|:-------:|:-------:|:----------:|:-----------:|:----------:|:-----------:|:------------------:|:------:|:---------------:|:-------------:|:----------------------:|:-------------------:|:-----------:|:------------:|:--------------:|:------:|:---------:|:-------------------:|:------:|:------:|:--------:|:---------:|:-------:|:------:|:------------:|:-----------:|:-------------:|:-------------:|:----------:|
+| 0.6445        | 1.0   | 1088 | 0.3322          | 0.6449            | 0.7003         | 0.6714     | 0.8900           | 0.7607         | 0.8733           | 0.6576 | 0.1766    | 0.25   | 0.6783            | 0.3621            | 0.6005  | 0.6909         | 0.5586    | 0.0              | 0.2449              | 0.7095              | 0.2889      | 0.0             | 0.0             | 0.3902            | 0.7720  | 0.0    | 0.9862   | 0.8011             | 0.5088      | 0.7740       | 0.7118    | 0.5434    | 0.8088  | 0.0    | 0.8303  | 0.7562  | 0.5318     | 0.7294      | 0.4681     | 0.6779      | 0.0                | 0.8909 | 0.0             | 0.0107        | 0.9985                 | 0.4000              | 0.7307      | 0.9057       | 0.8618         | 0.0    | 0.9127    | 0.8235              | 0.9211 | 0.8026 | 0.4656   | 0.6390    | 0.9383  | 0.9775 | 0.8868       | 0.8201      | 0.4526        | 0.0550        | 0.5368     |
+| 0.222         | 2.0   | 2176 | 0.1259          | 0.8170            | 0.8747         | 0.8449     | 0.9478           | 0.9708         | 0.9813           | 0.7638 | 0.7427    | 0.7837 | 0.8908            | 0.8833            | 0.8747  | 0.9814         | 0.8749    | 0.7601           | 0.9777              | 0.8834              | 0.5372      | 0.4828          | 0.0056          | 0.7785            | 0.8149  | 0.3140 | 0.9956   | 0.9935             | 0.9101      | 0.9270       | 0.9450    | 0.9853    | 0.9253  | 0.0650 | 0.0084  | 0.7962  | 0.9013     | 0.9446      | 0.9203     | 0.8555      | 0.6885             | 1.0    | 0.7152          | 0.6442        | 1.0                    | 0.9623              | 0.9349      | 0.9905       | 0.9782         | 0.7656 | 0.9324    | 0.9903              | 0.9736 | 0.9274 | 0.8520   | 0.9138    | 0.9678  | 0.9922 | 0.9893       | 0.9804      | 0.9646        | 0.8556        | 0.8385     |
+| 0.1331        | 3.0   | 3264 | 0.0773          | 0.9133            | 0.9371         | 0.9250     | 0.9654           | 0.9822         | 0.9815           | 0.9196 | 0.8852    | 0.9718 | 0.9785            | 0.9215            | 0.9757  | 0.9935         | 0.9651    | 0.8742           | 0.9921              | 0.9438              | 0.7568      | 0.7710          | 0.0             | 0.8998            | 0.7895  | 0.6578 | 0.9994   | 1.0                | 0.9554      | 0.9525       | 0.9823    | 0.9910    | 0.9866  | 0.0435 | 0.8293  | 0.7824  | 0.9671     | 0.9794      | 0.9571     | 0.9447      | 0.9141             | 1.0    | 0.8825          | 0.7988        | 1.0                    | 0.9797              | 0.9921      | 0.9932       | 0.9943         | 0.8726 | 0.9401    | 0.9860              | 0.9792 | 0.9928 | 0.9740   | 0.9604    | 0.9730  | 0.9983 | 0.9964       | 0.9959      | 0.9890        | 0.9774        | 0.9247     |
+| 0.0847        | 4.0   | 4352 | 0.0503          | 0.9368            | 0.9614         | 0.9489     | 0.9789           | 0.9955         | 0.9949           | 0.9573 | 0.9480    | 0.9929 | 0.9846            | 0.9808            | 0.9927  | 0.9962         | 0.9811    | 0.9436           | 0.9953              | 0.9695              | 0.7826      | 0.8713          | 0.1653          | 0.9458            | 0.8782  | 0.7996 | 1.0      | 1.0                | 0.9809      | 0.9816       | 0.9941    | 0.9910    | 0.9906  | 0.3389 | 0.8364  | 0.7066  | 0.9862     | 1.0         | 0.9795     | 0.9637      | 0.9429             | 1.0    | 0.9438          | 0.9165        | 1.0                    | 0.9864              | 1.0         | 0.9932       | 0.9962         | 0.9352 | 0.9483    | 0.9860              | 0.9866 | 0.9976 | 0.9884   | 0.9827    | 0.9881  | 1.0    | 0.9953       | 0.9975      | 0.9945        | 0.9915        | 0.9841     |
+| 0.0557        | 5.0   | 5440 | 0.0451          | 0.9438            | 0.9663         | 0.9549     | 0.9838           | 0.9946         | 0.9940           | 0.9624 | 0.9643    | 0.9929 | 0.9948            | 0.9845            | 0.9955  | 0.9962         | 0.9877    | 0.9643           | 0.9953              | 0.9793              | 0.7811      | 0.8850          | 0.2281          | 0.9562            | 0.9061  | 0.7914 | 1.0      | 1.0                | 0.9837      | 0.9846       | 0.9971    | 0.9910    | 0.9906  | 0.4349 | 0.8126  | 0.7679  | 0.9880     | 0.9991      | 0.9777     | 0.9684      | 0.9721             | 1.0    | 0.9635          | 0.9330        | 1.0                    | 0.9910              | 1.0         | 0.9918       | 0.9962         | 0.9477 | 0.9546    | 0.9892              | 0.9876 | 0.9976 | 0.9893   | 0.9873    | 0.9889  | 1.0    | 0.9953       | 0.9975      | 1.0           | 1.0           | 0.9873     |
+
+
+### Framework versions
+
+- Transformers 4.35.0
+- Pytorch 2.0.0
+- Datasets 2.1.0
+- Tokenizers 0.14.1

all_results.json

@@ -0,0 +1,74 @@
+{
+    "epoch": 5.0,
+    "eval_ACCOUNTNAME_f1": 0.9946332737030411,
+    "eval_ACCOUNTNUMBER_f1": 0.9940323955669224,
+    "eval_AGE_f1": 0.9623601220752798,
+    "eval_AMOUNT_f1": 0.9643140364789851,
+    "eval_BIC_f1": 0.9929078014184397,
+    "eval_BITCOINADDRESS_f1": 0.9948293691830403,
+    "eval_BUILDINGNUMBER_f1": 0.9845313921747043,
+    "eval_CITY_f1": 0.9954669084315503,
+    "eval_COMPANYNAME_f1": 0.9962476547842402,
+    "eval_COUNTY_f1": 0.9877474081055608,
+    "eval_CREDITCARDCVV_f1": 0.9642857142857142,
+    "eval_CREDITCARDISSUER_f1": 0.995260663507109,
+    "eval_CREDITCARDNUMBER_f1": 0.9792979297929792,
+    "eval_CURRENCYCODE_f1": 0.8849557522123894,
+    "eval_CURRENCYNAME_f1": 0.22813688212927757,
+    "eval_CURRENCYSYMBOL_f1": 0.9561904761904763,
+    "eval_CURRENCY_f1": 0.7810526315789474,
+    "eval_DATE_f1": 0.9060716139076285,
+    "eval_DOB_f1": 0.7914438502673796,
+    "eval_EMAIL_f1": 1.0,
+    "eval_ETHEREUMADDRESS_f1": 1.0,
+    "eval_EYECOLOR_f1": 0.9836956521739131,
+    "eval_FIRSTNAME_f1": 0.9845632530120482,
+    "eval_GENDER_f1": 0.9970617042115573,
+    "eval_HEIGHT_f1": 0.991044776119403,
+    "eval_IBAN_f1": 0.9906040268456375,
+    "eval_IPV4_f1": 0.8125502815768303,
+    "eval_IPV6_f1": 0.7679245283018868,
+    "eval_IP_f1": 0.43488649940262847,
+    "eval_JOBAREA_f1": 0.9880404783808647,
+    "eval_JOBTITLE_f1": 0.9990917347865577,
+    "eval_JOBTYPE_f1": 0.9776586237712243,
+    "eval_LASTNAME_f1": 0.968421052631579,
+    "eval_LITECOINADDRESS_f1": 0.9721362229102166,
+    "eval_MAC_f1": 1.0,
+    "eval_MASKEDNUMBER_f1": 0.9635220125786164,
+    "eval_MIDDLENAME_f1": 0.9329852045256746,
+    "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+    "eval_ORDINALDIRECTION_f1": 0.990990990990991,
+    "eval_PASSWORD_f1": 1.0,
+    "eval_PHONEIMEI_f1": 0.9918478260869565,
+    "eval_PHONENUMBER_f1": 0.9962049335863378,
+    "eval_PIN_f1": 0.9477351916376306,
+    "eval_PREFIX_f1": 0.9545762711864407,
+    "eval_SECONDARYADDRESS_f1": 0.989247311827957,
+    "eval_SEX_f1": 0.9875835721107928,
+    "eval_SSN_f1": 0.9976076555023924,
+    "eval_STATE_f1": 0.9892857142857142,
+    "eval_STREET_f1": 0.9872727272727274,
+    "eval_TIME_f1": 0.9888983774551664,
+    "eval_URL_f1": 1.0,
+    "eval_USERAGENT_f1": 0.9952718676122931,
+    "eval_USERNAME_f1": 0.9975308641975308,
+    "eval_VEHICLEVIN_f1": 1.0,
+    "eval_VEHICLEVRM_f1": 1.0,
+    "eval_ZIPCODE_f1": 0.9872611464968153,
+    "eval_loss": 0.04511478543281555,
+    "eval_overall_accuracy": 0.9837712232834332,
+    "eval_overall_f1": 0.9549383718621983,
+    "eval_overall_precision": 0.9438162166018621,
+    "eval_overall_recall": 0.9663257852447041,
+    "eval_runtime": 118.6411,
+    "eval_samples": 8701,
+    "eval_samples_per_second": 73.339,
+    "eval_steps_per_second": 9.171,
+    "total_flos": 5695265572254720.0,
+    "train_loss": 0.37311450242996214,
+    "train_runtime": 1947.5078,
+    "train_samples": 34800,
+    "train_samples_per_second": 22.339,
+    "train_steps_per_second": 2.793
+}

eval_results.json

@@ -0,0 +1,68 @@
+{
+    "epoch": 5.0,
+    "eval_ACCOUNTNAME_f1": 0.9946332737030411,
+    "eval_ACCOUNTNUMBER_f1": 0.9940323955669224,
+    "eval_AGE_f1": 0.9623601220752798,
+    "eval_AMOUNT_f1": 0.9643140364789851,
+    "eval_BIC_f1": 0.9929078014184397,
+    "eval_BITCOINADDRESS_f1": 0.9948293691830403,
+    "eval_BUILDINGNUMBER_f1": 0.9845313921747043,
+    "eval_CITY_f1": 0.9954669084315503,
+    "eval_COMPANYNAME_f1": 0.9962476547842402,
+    "eval_COUNTY_f1": 0.9877474081055608,
+    "eval_CREDITCARDCVV_f1": 0.9642857142857142,
+    "eval_CREDITCARDISSUER_f1": 0.995260663507109,
+    "eval_CREDITCARDNUMBER_f1": 0.9792979297929792,
+    "eval_CURRENCYCODE_f1": 0.8849557522123894,
+    "eval_CURRENCYNAME_f1": 0.22813688212927757,
+    "eval_CURRENCYSYMBOL_f1": 0.9561904761904763,
+    "eval_CURRENCY_f1": 0.7810526315789474,
+    "eval_DATE_f1": 0.9060716139076285,
+    "eval_DOB_f1": 0.7914438502673796,
+    "eval_EMAIL_f1": 1.0,
+    "eval_ETHEREUMADDRESS_f1": 1.0,
+    "eval_EYECOLOR_f1": 0.9836956521739131,
+    "eval_FIRSTNAME_f1": 0.9845632530120482,
+    "eval_GENDER_f1": 0.9970617042115573,
+    "eval_HEIGHT_f1": 0.991044776119403,
+    "eval_IBAN_f1": 0.9906040268456375,
+    "eval_IPV4_f1": 0.8125502815768303,
+    "eval_IPV6_f1": 0.7679245283018868,
+    "eval_IP_f1": 0.43488649940262847,
+    "eval_JOBAREA_f1": 0.9880404783808647,
+    "eval_JOBTITLE_f1": 0.9990917347865577,
+    "eval_JOBTYPE_f1": 0.9776586237712243,
+    "eval_LASTNAME_f1": 0.968421052631579,
+    "eval_LITECOINADDRESS_f1": 0.9721362229102166,
+    "eval_MAC_f1": 1.0,
+    "eval_MASKEDNUMBER_f1": 0.9635220125786164,
+    "eval_MIDDLENAME_f1": 0.9329852045256746,
+    "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+    "eval_ORDINALDIRECTION_f1": 0.990990990990991,
+    "eval_PASSWORD_f1": 1.0,
+    "eval_PHONEIMEI_f1": 0.9918478260869565,
+    "eval_PHONENUMBER_f1": 0.9962049335863378,
+    "eval_PIN_f1": 0.9477351916376306,
+    "eval_PREFIX_f1": 0.9545762711864407,
+    "eval_SECONDARYADDRESS_f1": 0.989247311827957,
+    "eval_SEX_f1": 0.9875835721107928,
+    "eval_SSN_f1": 0.9976076555023924,
+    "eval_STATE_f1": 0.9892857142857142,
+    "eval_STREET_f1": 0.9872727272727274,
+    "eval_TIME_f1": 0.9888983774551664,
+    "eval_URL_f1": 1.0,
+    "eval_USERAGENT_f1": 0.9952718676122931,
+    "eval_USERNAME_f1": 0.9975308641975308,
+    "eval_VEHICLEVIN_f1": 1.0,
+    "eval_VEHICLEVRM_f1": 1.0,
+    "eval_ZIPCODE_f1": 0.9872611464968153,
+    "eval_loss": 0.04511478543281555,
+    "eval_overall_accuracy": 0.9837712232834332,
+    "eval_overall_f1": 0.9549383718621983,
+    "eval_overall_precision": 0.9438162166018621,
+    "eval_overall_recall": 0.9663257852447041,
+    "eval_runtime": 118.6411,
+    "eval_samples": 8701,
+    "eval_samples_per_second": 73.339,
+    "eval_steps_per_second": 9.171
+}

train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 5.0,
+    "total_flos": 5695265572254720.0,
+    "train_loss": 0.37311450242996214,
+    "train_runtime": 1947.5078,
+    "train_samples": 34800,
+    "train_samples_per_second": 22.339,
+    "train_steps_per_second": 2.793
+}

trainer_state.json

@@ -0,0 +1,496 @@
+{
+  "best_metric": 0.04511478543281555,
+  "best_model_checkpoint": "distilbert_finetuned_ai4privacy_v2/checkpoint-5440",
+  "epoch": 5.0,
+  "eval_steps": 500,
+  "global_step": 5440,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.46,
+      "learning_rate": 2.2977941176470588e-05,
+      "loss": 2.2046,
+      "step": 500
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 4.5955882352941176e-05,
+      "loss": 0.6445,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0,
+      "eval_ACCOUNTNAME_f1": 0.7607461476074614,
+      "eval_ACCOUNTNUMBER_f1": 0.8732624693376942,
+      "eval_AGE_f1": 0.6575875486381323,
+      "eval_AMOUNT_f1": 0.17664670658682632,
+      "eval_BIC_f1": 0.25,
+      "eval_BITCOINADDRESS_f1": 0.6782884310618067,
+      "eval_BUILDINGNUMBER_f1": 0.36213991769547327,
+      "eval_CITY_f1": 0.6005188067444877,
+      "eval_COMPANYNAME_f1": 0.6909350463954318,
+      "eval_COUNTY_f1": 0.558648111332008,
+      "eval_CREDITCARDCVV_f1": 0.0,
+      "eval_CREDITCARDISSUER_f1": 0.24489795918367346,
+      "eval_CREDITCARDNUMBER_f1": 0.7095490716180372,
+      "eval_CURRENCYCODE_f1": 0.0,
+      "eval_CURRENCYNAME_f1": 0.0,
+      "eval_CURRENCYSYMBOL_f1": 0.3902012248468941,
+      "eval_CURRENCY_f1": 0.2889137737961926,
+      "eval_DATE_f1": 0.772020725388601,
+      "eval_DOB_f1": 0.0,
+      "eval_EMAIL_f1": 0.986198243412798,
+      "eval_ETHEREUMADDRESS_f1": 0.801068090787717,
+      "eval_EYECOLOR_f1": 0.5087719298245614,
+      "eval_FIRSTNAME_f1": 0.7739769820971866,
+      "eval_GENDER_f1": 0.7118320610687023,
+      "eval_HEIGHT_f1": 0.5434173669467787,
+      "eval_IBAN_f1": 0.8087697929354446,
+      "eval_IPV4_f1": 0.8303078137332283,
+      "eval_IPV6_f1": 0.756244616709733,
+      "eval_IP_f1": 0.0,
+      "eval_JOBAREA_f1": 0.5317919075144508,
+      "eval_JOBTITLE_f1": 0.7294303797468354,
+      "eval_JOBTYPE_f1": 0.4680851063829787,
+      "eval_LASTNAME_f1": 0.6778523489932886,
+      "eval_LITECOINADDRESS_f1": 0.0,
+      "eval_MAC_f1": 0.890909090909091,
+      "eval_MASKEDNUMBER_f1": 0.0,
+      "eval_MIDDLENAME_f1": 0.010657193605683837,
+      "eval_NEARBYGPSCOORDINATE_f1": 0.9984779299847794,
+      "eval_ORDINALDIRECTION_f1": 0.39999999999999997,
+      "eval_PASSWORD_f1": 0.73071104387292,
+      "eval_PHONEIMEI_f1": 0.9056603773584907,
+      "eval_PHONENUMBER_f1": 0.8618181818181818,
+      "eval_PIN_f1": 0.0,
+      "eval_PREFIX_f1": 0.9127248500999333,
+      "eval_SECONDARYADDRESS_f1": 0.823529411764706,
+      "eval_SEX_f1": 0.9210526315789475,
+      "eval_SSN_f1": 0.8025613660618997,
+      "eval_STATE_f1": 0.465592972181552,
+      "eval_STREET_f1": 0.6390114739629303,
+      "eval_TIME_f1": 0.9382504288164666,
+      "eval_URL_f1": 0.9775086505190311,
+      "eval_USERAGENT_f1": 0.8867924528301887,
+      "eval_USERNAME_f1": 0.8201332346410067,
+      "eval_VEHICLEVIN_f1": 0.45255474452554745,
+      "eval_VEHICLEVRM_f1": 0.05504587155963303,
+      "eval_ZIPCODE_f1": 0.5368492224475998,
+      "eval_loss": 0.33221369981765747,
+      "eval_overall_accuracy": 0.8899911029030149,
+      "eval_overall_f1": 0.671429071681199,
+      "eval_overall_precision": 0.6448510123091411,
+      "eval_overall_recall": 0.7002921840759678,
+      "eval_runtime": 139.6015,
+      "eval_samples_per_second": 62.327,
+      "eval_steps_per_second": 7.794,
+      "step": 1088
+    },
+    {
+      "epoch": 1.38,
+      "learning_rate": 4.890245381382757e-05,
+      "loss": 0.3195,
+      "step": 1500
+    },
+    {
+      "epoch": 1.84,
+      "learning_rate": 4.477509350887423e-05,
+      "loss": 0.222,
+      "step": 2000
+    },
+    {
+      "epoch": 2.0,
+      "eval_ACCOUNTNAME_f1": 0.9708222811671087,
+      "eval_ACCOUNTNUMBER_f1": 0.9812606473594548,
+      "eval_AGE_f1": 0.7638190954773869,
+      "eval_AMOUNT_f1": 0.7427258805513018,
+      "eval_BIC_f1": 0.7836990595611286,
+      "eval_BITCOINADDRESS_f1": 0.8908045977011495,
+      "eval_BUILDINGNUMBER_f1": 0.8833034111310593,
+      "eval_CITY_f1": 0.8746569075937786,
+      "eval_COMPANYNAME_f1": 0.9814126394052045,
+      "eval_COUNTY_f1": 0.874880611270296,
+      "eval_CREDITCARDCVV_f1": 0.7601246105919003,
+      "eval_CREDITCARDISSUER_f1": 0.9777070063694268,
+      "eval_CREDITCARDNUMBER_f1": 0.8833619210977702,
+      "eval_CURRENCYCODE_f1": 0.4827586206896552,
+      "eval_CURRENCYNAME_f1": 0.005633802816901407,
+      "eval_CURRENCYSYMBOL_f1": 0.7785234899328859,
+      "eval_CURRENCY_f1": 0.5371980676328503,
+      "eval_DATE_f1": 0.8149158708503865,
+      "eval_DOB_f1": 0.3140243902439024,
+      "eval_EMAIL_f1": 0.9955780164245104,
+      "eval_ETHEREUMADDRESS_f1": 0.9934853420195441,
+      "eval_EYECOLOR_f1": 0.9100817438692098,
+      "eval_FIRSTNAME_f1": 0.9269979974513016,
+      "eval_GENDER_f1": 0.9449901768172889,
+      "eval_HEIGHT_f1": 0.9852507374631269,
+      "eval_IBAN_f1": 0.9252577319587629,
+      "eval_IPV4_f1": 0.008382229673093043,
+      "eval_IPV6_f1": 0.7961504811898513,
+      "eval_IP_f1": 0.06496927129060581,
+      "eval_JOBAREA_f1": 0.9013035381750465,
+      "eval_JOBTITLE_f1": 0.9445910290237467,
+      "eval_JOBTYPE_f1": 0.9202566452795601,
+      "eval_LASTNAME_f1": 0.8554913294797688,
+      "eval_LITECOINADDRESS_f1": 0.6885245901639345,
+      "eval_MAC_f1": 1.0,
+      "eval_MASKEDNUMBER_f1": 0.7151819322459222,
+      "eval_MIDDLENAME_f1": 0.6442199775533108,
+      "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+      "eval_ORDINALDIRECTION_f1": 0.9623059866962306,
+      "eval_PASSWORD_f1": 0.9348882410106901,
+      "eval_PHONEIMEI_f1": 0.9904761904761904,
+      "eval_PHONENUMBER_f1": 0.9781990521327014,
+      "eval_PIN_f1": 0.7655786350148368,
+      "eval_PREFIX_f1": 0.9323607427055702,
+      "eval_SECONDARYADDRESS_f1": 0.9903121636167922,
+      "eval_SEX_f1": 0.9736346516007532,
+      "eval_SSN_f1": 0.927400468384075,
+      "eval_STATE_f1": 0.851968503937008,
+      "eval_STREET_f1": 0.9137777777777779,
+      "eval_TIME_f1": 0.9677966101694915,
+      "eval_URL_f1": 0.9922212618841832,
+      "eval_USERAGENT_f1": 0.9893491124260355,
+      "eval_USERNAME_f1": 0.9804241435562805,
+      "eval_VEHICLEVIN_f1": 0.9645776566757492,
+      "eval_VEHICLEVRM_f1": 0.8555555555555556,
+      "eval_ZIPCODE_f1": 0.8385093167701864,
+      "eval_loss": 0.1259380429983139,
+      "eval_overall_accuracy": 0.947805382743676,
+      "eval_overall_f1": 0.8449014869560617,
+      "eval_overall_precision": 0.8170436325179954,
+      "eval_overall_recall": 0.8747260774287802,
+      "eval_runtime": 119.3348,
+      "eval_samples_per_second": 72.913,
+      "eval_steps_per_second": 9.117,
+      "step": 2176
+    },
+    {
+      "epoch": 2.3,
+      "learning_rate": 3.809937438642091e-05,
+      "loss": 0.1588,
+      "step": 2500
+    },
+    {
+      "epoch": 2.76,
+      "learning_rate": 2.973557696454571e-05,
+      "loss": 0.1331,
+      "step": 3000
+    },
+    {
+      "epoch": 3.0,
+      "eval_ACCOUNTNAME_f1": 0.9822064056939502,
+      "eval_ACCOUNTNUMBER_f1": 0.9814814814814814,
+      "eval_AGE_f1": 0.9196337741607323,
+      "eval_AMOUNT_f1": 0.8852459016393442,
+      "eval_BIC_f1": 0.971830985915493,
+      "eval_BITCOINADDRESS_f1": 0.9785056294779939,
+      "eval_BUILDINGNUMBER_f1": 0.9215328467153284,
+      "eval_CITY_f1": 0.975653742110009,
+      "eval_COMPANYNAME_f1": 0.9934640522875815,
+      "eval_COUNTY_f1": 0.9650613786591123,
+      "eval_CREDITCARDCVV_f1": 0.8742138364779874,
+      "eval_CREDITCARDISSUER_f1": 0.9921259842519685,
+      "eval_CREDITCARDNUMBER_f1": 0.943800178412132,
+      "eval_CURRENCYCODE_f1": 0.7710144927536231,
+      "eval_CURRENCYNAME_f1": 0.0,
+      "eval_CURRENCYSYMBOL_f1": 0.8998073217726397,
+      "eval_CURRENCY_f1": 0.7568113017154389,
+      "eval_DATE_f1": 0.7895287958115184,
+      "eval_DOB_f1": 0.6578199052132702,
+      "eval_EMAIL_f1": 0.9993658845909955,
+      "eval_ETHEREUMADDRESS_f1": 1.0,
+      "eval_EYECOLOR_f1": 0.9553805774278215,
+      "eval_FIRSTNAME_f1": 0.9525469688165795,
+      "eval_GENDER_f1": 0.9823182711198428,
+      "eval_HEIGHT_f1": 0.991044776119403,
+      "eval_IBAN_f1": 0.9866310160427807,
+      "eval_IPV4_f1": 0.8292682926829269,
+      "eval_IPV6_f1": 0.7823834196891193,
+      "eval_IP_f1": 0.04347826086956522,
+      "eval_JOBAREA_f1": 0.96709323583181,
+      "eval_JOBTITLE_f1": 0.9794091316025068,
+      "eval_JOBTYPE_f1": 0.9571428571428571,
+      "eval_LASTNAME_f1": 0.9447396386822529,
+      "eval_LITECOINADDRESS_f1": 0.9141104294478528,
+      "eval_MAC_f1": 1.0,
+      "eval_MASKEDNUMBER_f1": 0.8825,
+      "eval_MIDDLENAME_f1": 0.7987987987987989,
+      "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+      "eval_ORDINALDIRECTION_f1": 0.9796839729119639,
+      "eval_PASSWORD_f1": 0.9920948616600791,
+      "eval_PHONEIMEI_f1": 0.9931972789115645,
+      "eval_PHONENUMBER_f1": 0.9943074003795066,
+      "eval_PIN_f1": 0.8726114649681529,
+      "eval_PREFIX_f1": 0.9400798934753662,
+      "eval_SECONDARYADDRESS_f1": 0.9860365198711063,
+      "eval_SEX_f1": 0.9791666666666667,
+      "eval_SSN_f1": 0.9928400954653939,
+      "eval_STATE_f1": 0.9739910313901345,
+      "eval_STREET_f1": 0.9603603603603604,
+      "eval_TIME_f1": 0.9730185497470489,
+      "eval_URL_f1": 0.9982668977469671,
+      "eval_USERAGENT_f1": 0.9964497041420118,
+      "eval_USERNAME_f1": 0.9958915365653246,
+      "eval_VEHICLEVIN_f1": 0.989010989010989,
+      "eval_VEHICLEVRM_f1": 0.9774011299435028,
+      "eval_ZIPCODE_f1": 0.9246646026831785,
+      "eval_loss": 0.07725853472948074,
+      "eval_overall_accuracy": 0.9654142205266811,
+      "eval_overall_f1": 0.9250486696950033,
+      "eval_overall_precision": 0.9132616742596811,
+      "eval_overall_recall": 0.9371439006574142,
+      "eval_runtime": 119.3285,
+      "eval_samples_per_second": 72.916,
+      "eval_steps_per_second": 9.118,
+      "step": 3264
+    },
+    {
+      "epoch": 3.22,
+      "learning_rate": 2.076151950480346e-05,
+      "loss": 0.1071,
+      "step": 3500
+    },
+    {
+      "epoch": 3.68,
+      "learning_rate": 1.2333662705902018e-05,
+      "loss": 0.0847,
+      "step": 4000
+    },
+    {
+      "epoch": 4.0,
+      "eval_ACCOUNTNAME_f1": 0.9955237242614144,
+      "eval_ACCOUNTNUMBER_f1": 0.9948979591836735,
+      "eval_AGE_f1": 0.9573170731707318,
+      "eval_AMOUNT_f1": 0.9480314960629921,
+      "eval_BIC_f1": 0.9929078014184397,
+      "eval_BITCOINADDRESS_f1": 0.984646878198567,
+      "eval_BUILDINGNUMBER_f1": 0.9808219178082191,
+      "eval_CITY_f1": 0.9927404718693285,
+      "eval_COMPANYNAME_f1": 0.9962476547842402,
+      "eval_COUNTY_f1": 0.9810964083175803,
+      "eval_CREDITCARDCVV_f1": 0.943620178041543,
+      "eval_CREDITCARDISSUER_f1": 0.995260663507109,
+      "eval_CREDITCARDNUMBER_f1": 0.9695340501792116,
+      "eval_CURRENCYCODE_f1": 0.8713450292397662,
+      "eval_CURRENCYNAME_f1": 0.16528925619834708,
+      "eval_CURRENCYSYMBOL_f1": 0.9457659372026641,
+      "eval_CURRENCY_f1": 0.7826086956521738,
+      "eval_DATE_f1": 0.8781558726673986,
+      "eval_DOB_f1": 0.7996183206106869,
+      "eval_EMAIL_f1": 1.0,
+      "eval_ETHEREUMADDRESS_f1": 1.0,
+      "eval_EYECOLOR_f1": 0.9809264305177111,
+      "eval_FIRSTNAME_f1": 0.9816479400749063,
+      "eval_GENDER_f1": 0.9941291585127201,
+      "eval_HEIGHT_f1": 0.991044776119403,
+      "eval_IBAN_f1": 0.9906040268456375,
+      "eval_IPV4_f1": 0.8363636363636363,
+      "eval_IPV6_f1": 0.7066420664206642,
+      "eval_IP_f1": 0.33885350318471336,
+      "eval_JOBAREA_f1": 0.9861751152073732,
+      "eval_JOBTITLE_f1": 1.0,
+      "eval_JOBTYPE_f1": 0.9794826048171275,
+      "eval_LASTNAME_f1": 0.9636650868878357,
+      "eval_LITECOINADDRESS_f1": 0.942942942942943,
+      "eval_MAC_f1": 1.0,
+      "eval_MASKEDNUMBER_f1": 0.9438202247191011,
+      "eval_MIDDLENAME_f1": 0.9165186500888098,
+      "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+      "eval_ORDINALDIRECTION_f1": 0.9864253393665159,
+      "eval_PASSWORD_f1": 1.0,
+      "eval_PHONEIMEI_f1": 0.9931972789115645,
+      "eval_PHONENUMBER_f1": 0.9962049335863378,
+      "eval_PIN_f1": 0.9351535836177474,
+      "eval_PREFIX_f1": 0.9482874412357287,
+      "eval_SECONDARYADDRESS_f1": 0.986006458557589,
+      "eval_SEX_f1": 0.9866412213740459,
+      "eval_SSN_f1": 0.9976076555023924,
+      "eval_STATE_f1": 0.9884238646482637,
+      "eval_STREET_f1": 0.9827429609445959,
+      "eval_TIME_f1": 0.9880546075085324,
+      "eval_URL_f1": 1.0,
+      "eval_USERAGENT_f1": 0.9952718676122931,
+      "eval_USERNAME_f1": 0.9975308641975308,
+      "eval_VEHICLEVIN_f1": 0.994535519125683,
+      "eval_VEHICLEVRM_f1": 0.9914529914529915,
+      "eval_ZIPCODE_f1": 0.9840933191940615,
+      "eval_loss": 0.050253935158252716,
+      "eval_overall_accuracy": 0.9789317417415393,
+      "eval_overall_f1": 0.9489356333026912,
+      "eval_overall_precision": 0.9367949037332289,
+      "eval_overall_recall": 0.9613951789627465,
+      "eval_runtime": 119.1763,
+      "eval_samples_per_second": 73.009,
+      "eval_steps_per_second": 9.129,
+      "step": 4352
+    },
+    {
+      "epoch": 4.14,
+      "learning_rate": 5.53807997510338e-06,
+      "loss": 0.0752,
+      "step": 4500
+    },
+    {
+      "epoch": 4.6,
+      "learning_rate": 1.2504983052690406e-06,
+      "loss": 0.0557,
+      "step": 5000
+    },
+    {
+      "epoch": 5.0,
+      "eval_ACCOUNTNAME_f1": 0.9946332737030411,
+      "eval_ACCOUNTNUMBER_f1": 0.9940323955669224,
+      "eval_AGE_f1": 0.9623601220752798,
+      "eval_AMOUNT_f1": 0.9643140364789851,
+      "eval_BIC_f1": 0.9929078014184397,
+      "eval_BITCOINADDRESS_f1": 0.9948293691830403,
+      "eval_BUILDINGNUMBER_f1": 0.9845313921747043,
+      "eval_CITY_f1": 0.9954669084315503,
+      "eval_COMPANYNAME_f1": 0.9962476547842402,
+      "eval_COUNTY_f1": 0.9877474081055608,
+      "eval_CREDITCARDCVV_f1": 0.9642857142857142,
+      "eval_CREDITCARDISSUER_f1": 0.995260663507109,
+      "eval_CREDITCARDNUMBER_f1": 0.9792979297929792,
+      "eval_CURRENCYCODE_f1": 0.8849557522123894,
+      "eval_CURRENCYNAME_f1": 0.22813688212927757,
+      "eval_CURRENCYSYMBOL_f1": 0.9561904761904763,
+      "eval_CURRENCY_f1": 0.7810526315789474,
+      "eval_DATE_f1": 0.9060716139076285,
+      "eval_DOB_f1": 0.7914438502673796,
+      "eval_EMAIL_f1": 1.0,
+      "eval_ETHEREUMADDRESS_f1": 1.0,
+      "eval_EYECOLOR_f1": 0.9836956521739131,
+      "eval_FIRSTNAME_f1": 0.9845632530120482,
+      "eval_GENDER_f1": 0.9970617042115573,
+      "eval_HEIGHT_f1": 0.991044776119403,
+      "eval_IBAN_f1": 0.9906040268456375,
+      "eval_IPV4_f1": 0.8125502815768303,
+      "eval_IPV6_f1": 0.7679245283018868,
+      "eval_IP_f1": 0.43488649940262847,
+      "eval_JOBAREA_f1": 0.9880404783808647,
+      "eval_JOBTITLE_f1": 0.9990917347865577,
+      "eval_JOBTYPE_f1": 0.9776586237712243,
+      "eval_LASTNAME_f1": 0.968421052631579,
+      "eval_LITECOINADDRESS_f1": 0.9721362229102166,
+      "eval_MAC_f1": 1.0,
+      "eval_MASKEDNUMBER_f1": 0.9635220125786164,
+      "eval_MIDDLENAME_f1": 0.9329852045256746,
+      "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+      "eval_ORDINALDIRECTION_f1": 0.990990990990991,
+      "eval_PASSWORD_f1": 1.0,
+      "eval_PHONEIMEI_f1": 0.9918478260869565,
+      "eval_PHONENUMBER_f1": 0.9962049335863378,
+      "eval_PIN_f1": 0.9477351916376306,
+      "eval_PREFIX_f1": 0.9545762711864407,
+      "eval_SECONDARYADDRESS_f1": 0.989247311827957,
+      "eval_SEX_f1": 0.9875835721107928,
+      "eval_SSN_f1": 0.9976076555023924,
+      "eval_STATE_f1": 0.9892857142857142,
+      "eval_STREET_f1": 0.9872727272727274,
+      "eval_TIME_f1": 0.9888983774551664,
+      "eval_URL_f1": 1.0,
+      "eval_USERAGENT_f1": 0.9952718676122931,
+      "eval_USERNAME_f1": 0.9975308641975308,
+      "eval_VEHICLEVIN_f1": 1.0,
+      "eval_VEHICLEVRM_f1": 1.0,
+      "eval_ZIPCODE_f1": 0.9872611464968153,
+      "eval_loss": 0.04511478543281555,
+      "eval_overall_accuracy": 0.9837712232834332,
+      "eval_overall_f1": 0.9549383718621983,
+      "eval_overall_precision": 0.9438162166018621,
+      "eval_overall_recall": 0.9663257852447041,
+      "eval_runtime": 118.5637,
+      "eval_samples_per_second": 73.387,
+      "eval_steps_per_second": 9.176,
+      "step": 5440
+    },
+    {
+      "epoch": 5.0,
+      "step": 5440,
+      "total_flos": 5695265572254720.0,
+      "train_loss": 0.37311450242996214,
+      "train_runtime": 1947.5078,
+      "train_samples_per_second": 22.339,
+      "train_steps_per_second": 2.793
+    },
+    {
+      "epoch": 5.0,
+      "eval_ACCOUNTNAME_f1": 0.9946332737030411,
+      "eval_ACCOUNTNUMBER_f1": 0.9940323955669224,
+      "eval_AGE_f1": 0.9623601220752798,
+      "eval_AMOUNT_f1": 0.9643140364789851,
+      "eval_BIC_f1": 0.9929078014184397,
+      "eval_BITCOINADDRESS_f1": 0.9948293691830403,
+      "eval_BUILDINGNUMBER_f1": 0.9845313921747043,
+      "eval_CITY_f1": 0.9954669084315503,
+      "eval_COMPANYNAME_f1": 0.9962476547842402,
+      "eval_COUNTY_f1": 0.9877474081055608,
+      "eval_CREDITCARDCVV_f1": 0.9642857142857142,
+      "eval_CREDITCARDISSUER_f1": 0.995260663507109,
+      "eval_CREDITCARDNUMBER_f1": 0.9792979297929792,
+      "eval_CURRENCYCODE_f1": 0.8849557522123894,
+      "eval_CURRENCYNAME_f1": 0.22813688212927757,
+      "eval_CURRENCYSYMBOL_f1": 0.9561904761904763,
+      "eval_CURRENCY_f1": 0.7810526315789474,
+      "eval_DATE_f1": 0.9060716139076285,
+      "eval_DOB_f1": 0.7914438502673796,
+      "eval_EMAIL_f1": 1.0,
+      "eval_ETHEREUMADDRESS_f1": 1.0,
+      "eval_EYECOLOR_f1": 0.9836956521739131,
+      "eval_FIRSTNAME_f1": 0.9845632530120482,
+      "eval_GENDER_f1": 0.9970617042115573,
+      "eval_HEIGHT_f1": 0.991044776119403,
+      "eval_IBAN_f1": 0.9906040268456375,
+      "eval_IPV4_f1": 0.8125502815768303,
+      "eval_IPV6_f1": 0.7679245283018868,
+      "eval_IP_f1": 0.43488649940262847,
+      "eval_JOBAREA_f1": 0.9880404783808647,
+      "eval_JOBTITLE_f1": 0.9990917347865577,
+      "eval_JOBTYPE_f1": 0.9776586237712243,
+      "eval_LASTNAME_f1": 0.968421052631579,
+      "eval_LITECOINADDRESS_f1": 0.9721362229102166,
+      "eval_MAC_f1": 1.0,
+      "eval_MASKEDNUMBER_f1": 0.9635220125786164,
+      "eval_MIDDLENAME_f1": 0.9329852045256746,
+      "eval_NEARBYGPSCOORDINATE_f1": 1.0,
+      "eval_ORDINALDIRECTION_f1": 0.990990990990991,
+      "eval_PASSWORD_f1": 1.0,
+      "eval_PHONEIMEI_f1": 0.9918478260869565,
+      "eval_PHONENUMBER_f1": 0.9962049335863378,
+      "eval_PIN_f1": 0.9477351916376306,
+      "eval_PREFIX_f1": 0.9545762711864407,
+      "eval_SECONDARYADDRESS_f1": 0.989247311827957,
+      "eval_SEX_f1": 0.9875835721107928,
+      "eval_SSN_f1": 0.9976076555023924,
+      "eval_STATE_f1": 0.9892857142857142,
+      "eval_STREET_f1": 0.9872727272727274,
+      "eval_TIME_f1": 0.9888983774551664,
+      "eval_URL_f1": 1.0,
+      "eval_USERAGENT_f1": 0.9952718676122931,
+      "eval_USERNAME_f1": 0.9975308641975308,
+      "eval_VEHICLEVIN_f1": 1.0,
+      "eval_VEHICLEVRM_f1": 1.0,
+      "eval_ZIPCODE_f1": 0.9872611464968153,
+      "eval_loss": 0.04511478543281555,
+      "eval_overall_accuracy": 0.9837712232834332,
+      "eval_overall_f1": 0.9549383718621983,
+      "eval_overall_precision": 0.9438162166018621,
+      "eval_overall_recall": 0.9663257852447041,
+      "eval_runtime": 118.6411,
+      "eval_samples_per_second": 73.339,
+      "eval_steps_per_second": 9.171,
+      "step": 5440
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 5440,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "total_flos": 5695265572254720.0,
+  "trial_name": null,
+  "trial_params": null
+}