File size: 661 Bytes
160dfe2 b5ed404 160dfe2 b5ed404 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
---
license: apache-2.0
---
This is a distilbert-base-multilingual-cased-Model fine-tuned with a NER objective to tag tokens based on whether they belong to a code block or natural language text.
The dataset of 78210 examples was generated by randomly combining code and text blocks from other permissively-licensed datasets, with some examples containing only code and some only regular text.
The model achieves the following stats on the validation set:
| Metric | Value |
|--------------|-----------|
| Loss | 0.0788 |
| F1 Score | 0.8619 |
| Precision | 0.8362 |
| Recall | 0.8893 |
| Accuracy | 0.9792 | |