GIST-small-finefineweb

A fine-tuned version of the bert architecture (BertForSequenceClassification) optimized for the text-classification task.

• Model type: bert
• Problem Type: single_label_classification
• Number of Labels: 67
• Vocabulary Size: 30522
• License: MIT

Use

To get started with this model in Python using the Hugging Face Transformers library, run the following code:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "agentlans/GIST-small-finefineweb"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Replace this with your input text."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

predicted_class_id = logits.argmax().item()
predicted_class_name = model.config.id2label[predicted_class_id]

print(f"Predicted Class ID: {predicted_class_id}")
print(f"Predicted Class Name: {predicted_class_name}")

Intended Uses & Limitations

Intended Use

This model is designed for sequence classification tasks. Below are the specific class labels mapped to their corresponding IDs:

Label ID	Label Name
0	aerospace
1	agronomy
2	artistic
3	astronomy
4	atmospheric_science
5	automotive
6	beauty
7	biology
8	celebrity
9	chemistry
10	christianity
11	civil_engineering
12	communication_engineering
13	computer_science_and_technology
14	design
15	drama_and_film
16	economics
17	electronic_science
18	entertainment
19	environmental_science
20	fashion
21	finance
22	food
23	gamble
24	game
25	geography
26	health
27	history
28	hobby
29	hydraulic_engineering
30	instrument_science
31	journalism_and_media_communication
32	landscape_architecture
33	law
34	library
35	literature
36	materials_science
37	mathematics
38	mechanical_engineering
39	medical
40	mining_engineering
41	movie
42	music_and_dance
43	news
44	nuclear_science
45	ocean_science
46	optical_engineering
47	painting
48	pet
49	petroleum_and_natural_gas_engineering
50	philosophy
51	photo
52	physics
53	politics
54	psychology
55	public_administration
56	relationship
57	sociology
58	sports
59	statistics
60	systems_science
61	textile_science
62	topicality
63	transportation_engineering
64	travel
65	urban_planning
66	weapons_science

Training Details

Hyperparameters

The following hyperparameters were used during fine-tuning:

• Learning Rate: 5e-05
• Train Batch Size: 8
• Eval Batch Size: 8
• Optimizer: OptimizerNames.ADAMW_TORCH_FUSED
• Number of Epochs: 3.0
• Mixed Precision: BF16

Show Advanced Training Configuration

Optimization & Regularization

• Gradient Accumulation Steps: 1
• Learning Rate Scheduler: SchedulerType.LINEAR
• Warmup Steps: 0
• Warmup Ratio: None
• Weight Decay: 0.0
• Max Gradient Norm: 1.0

Hardware & Reproducibility

• Number of GPUs: 1
• Seed: 42

Training Results & Evaluation

During fine-tuning, the model achieved the following results on the evaluation set:

Metric	Value
Train Loss	1.3238
Validation Loss	1.4201
Validation F1 Score	N/A
Total FLOPs	5.6337e+15

For performance on the test set, click here.

Speed Performance

• Training Runtime: 720.633 seconds
• Train Samples per Second: 474.166
• Evaluation Runtime: 10.5434 seconds
• Eval Samples per Second: 1906.406

Show Detailed Training Logs

Training Logs History

Step	Epoch	Learning Rate	Training Loss	Validation Loss	Validation F1
500	0.035	4.9416e-05	3.8054	N/A	N/A
1000	0.07	4.8831e-05	3.0202	N/A	N/A
1500	0.105	4.8245e-05	2.5217	N/A	N/A
2000	0.14	4.7660e-05	2.2352	N/A	N/A
2500	0.176	4.7075e-05	2.0559	N/A	N/A
3000	0.211	4.6489e-05	1.9031	N/A	N/A
3500	0.246	4.5904e-05	1.8326	N/A	N/A
4000	0.281	4.5319e-05	1.7487	N/A	N/A
4500	0.316	4.4734e-05	1.6888	N/A	N/A
5000	0.351	4.4148e-05	1.6789	N/A	N/A
5500	0.386	4.3563e-05	1.6413	N/A	N/A
6000	0.421	4.2978e-05	1.5827	N/A	N/A
6500	0.457	4.2392e-05	1.5813	N/A	N/A
7000	0.492	4.1807e-05	1.588	N/A	N/A
7500	0.527	4.1222e-05	1.5676	N/A	N/A
8000	0.562	4.0637e-05	1.5305	N/A	N/A
8500	0.597	4.0051e-05	1.5383	N/A	N/A
9000	0.632	3.9466e-05	1.5032	N/A	N/A
9500	0.667	3.8881e-05	1.4895	N/A	N/A
10000	0.702	3.8295e-05	1.5036	N/A	N/A
10500	0.737	3.7710e-05	1.4761	N/A	N/A
11000	0.773	3.7125e-05	1.4902	N/A	N/A
11500	0.808	3.6540e-05	1.4635	N/A	N/A
12000	0.843	3.5954e-05	1.4686	N/A	N/A
12500	0.878	3.5369e-05	1.4882	N/A	N/A
13000	0.913	3.4784e-05	1.4641	N/A	N/A
13500	0.948	3.4198e-05	1.4443	N/A	N/A
14000	0.983	3.3613e-05	1.4215	N/A	N/A
14238	1.0	N/A	N/A	1.4201	N/A
14500	1.018	3.3028e-05	1.3587	N/A	N/A
15000	1.054	3.2443e-05	1.236	N/A	N/A
15500	1.089	3.1857e-05	1.2194	N/A	N/A
16000	1.124	3.1272e-05	1.2579	N/A	N/A
16500	1.159	3.0687e-05	1.2694	N/A	N/A
17000	1.194	3.0101e-05	1.1975	N/A	N/A
17500	1.229	2.9516e-05	1.2192	N/A	N/A
18000	1.264	2.8931e-05	1.2279	N/A	N/A
18500	1.299	2.8346e-05	1.2407	N/A	N/A
19000	1.334	2.7760e-05	1.2168	N/A	N/A
19500	1.37	2.7175e-05	1.242	N/A	N/A
20000	1.405	2.6590e-05	1.2581	N/A	N/A
20500	1.44	2.6004e-05	1.2209	N/A	N/A
21000	1.475	2.5419e-05	1.2384	N/A	N/A
21500	1.51	2.4834e-05	1.2286	N/A	N/A
22000	1.545	2.4248e-05	1.2079	N/A	N/A
22500	1.58	2.3663e-05	1.2561	N/A	N/A
23000	1.615	2.3078e-05	1.206	N/A	N/A
23500	1.651	2.2493e-05	1.2333	N/A	N/A
24000	1.686	2.1907e-05	1.2342	N/A	N/A
24500	1.721	2.1322e-05	1.1834	N/A	N/A
25000	1.756	2.0737e-05	1.2109	N/A	N/A
25500	1.791	2.0151e-05	1.2247	N/A	N/A
26000	1.826	1.9566e-05	1.2074	N/A	N/A
26500	1.861	1.8981e-05	1.2203	N/A	N/A
27000	1.896	1.8396e-05	1.1754	N/A	N/A
27500	1.931	1.7810e-05	1.2144	N/A	N/A
28000	1.967	1.7225e-05	1.188	N/A	N/A
28476	2.0	N/A	N/A	1.3501	N/A
28500	2.002	1.6640e-05	1.2042	N/A	N/A
29000	2.037	1.6054e-05	0.9849	N/A	N/A
29500	2.072	1.5469e-05	1.0076	N/A	N/A
30000	2.107	1.4884e-05	1.0033	N/A	N/A
30500	2.142	1.4299e-05	0.9742	N/A	N/A
31000	2.177	1.3713e-05	0.9856	N/A	N/A
31500	2.212	1.3128e-05	1.0047	N/A	N/A
32000	2.248	1.2543e-05	1.0172	N/A	N/A
32500	2.283	1.1957e-05	0.9914	N/A	N/A
33000	2.318	1.1372e-05	0.965	N/A	N/A
33500	2.353	1.0787e-05	0.991	N/A	N/A
34000	2.388	1.0202e-05	0.992	N/A	N/A
34500	2.423	9.6163e-06	1.0056	N/A	N/A
35000	2.458	9.0310e-06	0.9602	N/A	N/A
35500	2.493	8.4457e-06	0.9863	N/A	N/A
36000	2.528	7.8604e-06	0.9887	N/A	N/A
36500	2.564	7.2751e-06	0.9745	N/A	N/A
37000	2.599	6.6898e-06	0.9703	N/A	N/A
37500	2.634	6.1046e-06	0.9777	N/A	N/A
38000	2.669	5.5193e-06	0.9431	N/A	N/A
38500	2.704	4.9340e-06	0.9579	N/A	N/A
39000	2.739	4.3487e-06	0.9422	N/A	N/A
39500	2.774	3.7634e-06	0.9549	N/A	N/A
40000	2.809	3.1781e-06	0.9579	N/A	N/A
40500	2.845	2.5928e-06	0.9495	N/A	N/A
41000	2.88	2.0075e-06	0.9523	N/A	N/A
41500	2.915	1.4223e-06	0.9693	N/A	N/A
42000	2.95	8.3696e-07	0.9742	N/A	N/A
42500	2.985	2.5167e-07	0.988	N/A	N/A
42714	3.0	N/A	N/A	1.3795	N/A

Framework Versions

• Transformers: 5.14.0.dev0
• PyTorch: 2.9.1+cu128