Update README.md

README.md

@@ -1,22 +1,42 @@
 ---
 license: mit
 ---
-# Scandinavian Linguistic Classifier NB-BERT
-Trained using code from [ComsmoPedia[https://github.com/huggingface/cosmopedia/tree/main/classification], but with the [nb-bert-base](https://huggingface.co/NbAiLab/nb-bert-base) as starting point. The [data](https://huggingface.co/datasets/north/scandinavian-llama3-annotations) used in classification is from [GlotCC](https://huggingface.co/datasets/cis-lmu/GlotCC-V1) and have been annotated using Gemini 1.5 Flash.
+# NB Linguistic Quality Regressor
 
+## Introduction
+
+This model is designed to rate the quality of Norwegian training corpora based on **linguistic quality**. It predicts a continuous score (float from 0 to 5), assessing the linguistic quality of Norwegian texts. The model is inspired by the classifiers used in the FineWeb project and is trained mainly on Norwegian content.
+
+## Model Architecture
+
+It is trained on top of the [nb-bert-base](https://huggingface.co/NbAiLab/nb-bert-base) model and utilizes code from [CosmoPedia](https://github.com/huggingface/cosmopedia/tree/main/classification).
+
+## Training Data
+
+The dataset used for training is derived from [GlotCC](https://huggingface.co/datasets/cis-lmu/GlotCC-V1) and has been annotated using Gemini 1.5 Flash.
+
+## Purpose
+
+The performance of large language models (LLMs) heavily depends on the quality and size of their pretraining datasets. This regressor aims to assess and enhance the linguistic quality of Norwegian textual data, contributing to better-performing Norwegian LLMs.
+
+This model is part of a pair; the other is the [NB Education Quality Regressor](https://huggingface.co/NbAiLab/nb-education-quality-regressor), which focuses on educational content.
+
+
+## Using the Model
+For convenience we also provide the `run_regressor_bert.py` script. This is also based on `run_edu_bert.py` from Cosmopedia. You can modify this script to annotate HuggingFace datasets directly. Cosmopedia also provides slurm-scripts here. We have not included these since we have had the opportunity to test them.
+
+
+## Training and Evaluation Procedure
 The following command where used for training. Please note that `train_regressor_bert.py` has a few minor changes to the original `train_edu_bert.py`:
 ```
- python train_regressor_bert.py --base_model_name="NbAiLab/nb-bert-base" --dataset_name="north/scandinavian-linguistic-annotations" --target_column="score" --checkpoint_dir="/home/pere/checkpoints/scandinavian_bert/"
+ python train_regressor_bert.py --base_model_name="NbAiLab/nb-bert-base" --dataset_name="user/linguistic-annotations" --target_column="score" --checkpoint_dir="/home/pere/checkpoints/scandinavian_bert/"
 ```
 
-We also provide an evaluation script. This is targetted toward the same dataset.
+The following script where used for evaluation.
 ```
- python eval_regressor_bert.py --checkpoint_dir="/home/pere/checkpoints/scandinavian_bert/final/" --dataset_name="north/scandinavian-linguistic-annotations"
+ python eval_regressor_bert.py --checkpoint_dir="/user/pere/checkpoints/scandinavian_bert/final/" --dataset_name="user/linguistic-annotations"
 ```
 
-For convenience we also provide the `run_regressor_bert.py` script. This is also based on `run_edu_bert.py` from Cosmopedia. You can modify this script to annotate HuggingFace datasets directly. Cosmopedia also provides slurm-scripts here. We have not included these since we have had the opportunity to test them.
-
-
 
 ## Classification Report