aquiro1994/naics-github-classifier-multilingual

NAICS GitHub Repository Classifier, multilingual

A fine-tuned BGE-M3 that classifies GitHub repositories into 19 NAICS industry sectors from their name, description, topics and README.

It is the multilingual counterpart of aquiro1994/naics-github-classifier, which is RoBERTa-large and English-only. Same task, same 19 classes, same training data, same recipe; the difference is the encoder underneath.

Which one should you use?

	this model	naics-github-classifier
Encoder	BGE-M3 (XLM-RoBERTa, 568M)	RoBERTa-large (355M)
Vocabulary	250,002 tokens, 100+ languages	50,265 tokens, English
Test accuracy	86.95%	86.72%
Weighted F1	86.39%	86.33%
Macro F1	83.06%	82.95%
Same sector for a README and its own English translation	73%	19%
Calibrated as trained	yes, T = 1.04	yes, T = 1.01

On English text the two are indistinguishable: a 0.2-point difference on a 1,318-row test set is well inside run-to-run noise. Use this one when the corpus is not all English, which for public GitHub is about 12% of repositories. Use the RoBERTa model when the corpus is English and you want the smaller, faster network.

Important: the training data is English

This model reads other languages through cross-lingual transfer, not because it was trained on them. The 6,588 labelled repositories are English. What BGE-M3 brings is a shared multilingual representation space from its own pre-training, so a Spanish or Chinese README lands near its English equivalent and the classification head, trained on English, still applies.

That transfer is real but imperfect, and it is what the 73% above measures: 93 non-English repositories were classified twice, once as written and once from a human-quality English translation, and the two answers agreed 73% of the time. RoBERTa-large agrees with itself 19% of the time on the same test, which is what having no vocabulary for the text looks like.

	FR (31)	ES (28)	RU (14)	PT (11)	ZH (9)	all (93)
this model	84%	57%	79%	91%	56%	73%
RoBERTa-large	16%	11%	21%	18%	56%	19%

Two caveats worth stating plainly. Agreement is not accuracy: when the two readings differ, neither is known to be right. And the figure varies between training runs more than the English metrics do — a second run of the same recipe measured 81%, against an English F1 of 85.2. Treat 73% as one measurement, not a constant.

Training on multilingual labelled data would be the real upgrade, and nothing here measures what it would add.

Usage

import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else -1)

# truncation is required: without it any input over 512 tokens raises
# "index 514 is out of bounds"
classifier = pipeline(
    "text-classification",
    model="aquiro1994/naics-github-classifier-multilingual",
    device=device,
    truncation=True,
    max_length=512,
)

text = "Repository: fincas-app | Description: Gestión de alquileres y contratos de arrendamiento | README: Aplicación web para administradores de fincas..."
print(classifier(text))
# [{'label': '53', 'score': 0.97}]  # Real Estate and Rental

Build the input with the package, not by hand

The model was fine-tuned on text that goes through a cleaning step, so a hand-assembled string is raw markdown it never saw in training. Use the builder from naics-github-train, which is the same function the training pipeline calls:

from src.text_format import format_model_input

text = format_model_input(
    repo_name="fincas-app",
    description="Gestión de alquileres y contratos de arrendamiento",
    topics=["real-estate", "property-management"],
    readme="# Fincas\n\nAplicación web para **administradores de fincas**...",
)

The format is Repository: … | Description: … | Topics: a; b | README: …, with missing fields dropped along with their separator.

What the cleaning does, including two surprises

Badges removed, licence headers removed, URLs reduced to their domain, markdown stripped, whitespace collapsed. Two behaviours are not what the code comments suggest, and the training corpus was built with them, so they are load-bearing:

1. Everything after the first pip install, npm install or git clone is dropped, not just that line. On raw READMEs this costs 12.7% of repositories more than half their text.
2. Code block bodies survive. Fences are stripped; the code inside reaches the model.

Neither should be changed without retraining.

Thresholding

The model comes out of training calibrated (T = 1.04, expected calibration error 0.028), so its score means roughly what it says and no temperature scaling is needed. Filtering at score >= 0.8 keeps 81.8% of the test set and is right on 92.6% of what it keeps.

NAICS Classes

Label	Code	Sector	Label	Code	Sector
0	11	Agriculture, Forestry, Fishing and Hunting	10	53	Real Estate and Rental
1	21	Mining, Quarrying, Oil and Gas Extraction	11	54	Professional, Scientific, Technical Services
2	22	Utilities	12	56	Administrative and Support Services
3	23	Construction	13	61	Educational Services
4	31-33	Manufacturing	14	62	Health Care and Social Assistance
5	42	Wholesale Trade	15	71	Arts, Entertainment, and Recreation
6	44-45	Retail Trade	16	72	Accommodation and Food Services
7	48-49	Transportation and Warehousing	17	81	Other Services
8	51	Information	18	92	Public Administration
9	52	Finance and Insurance

NAICS code 55 (Management of Companies) is absent from the training data, so the model cannot predict it.

Training

Trained with naics-github-train, which regenerates this checkpoint:

python scripts/train.py --model bge-m3 \
    --data data/raw/train_data_gpt_ab8_score_with_code.parquet \
    --epochs 8 --batch-size 8 --gradient-accumulation-steps 8 \
    --min-samples 80 --eval-steps 100

Parameter	Value
Base model	BAAI/bge-m3 (568M parameters)
Data	6,588 labelled repositories, 19 classes
Split	4,611 train / 659 validation / 1,318 test, stratified, seed 42
Sequence length	512 tokens
Batch	8 with 8 gradient accumulation steps: effective 64
Epochs	8 (584 optimizer steps); best checkpoint by validation F1, at step 300
Learning rate	1.5e-5, polynomial decay, 15% warm-up
Weight decay	0.02
Optimizer	AdamW
Hardware	Apple M5 Max, fp32, 90 minutes

A 1,024-token run of the same recipe scored 0.5 points lower: the signal is in the opening of the README, not in its length.

The settings above are recorded in training_config.json in this repository, and scripts/evaluate.py reads them so evaluation cannot silently use a different split or sequence length.

Limitations

• The training data is English. See the section above.
• Agreement across languages is not accuracy, and it varies between runs (73% here, 81% in a second run of the same recipe).
• The labels are GPT-4.1 judgements, not human annotation.
• The model cannot abstain. The training data contains only positives, so every repository is assigned some sector. On a production corpus of public GitHub repositories, roughly half of what the model labels confidently arguably belongs to no sector at all: coursework, portfolios, exercises, generic tooling. Filtering by score helps but does not solve this.
• Sector 55 cannot be predicted, and rare sectors are weak: Wholesale Trade (42) has 24 test examples and its F1 is the lowest of the nineteen.
• One run. Seed-to-seed variation on a 1,318-row test set is about ±1 point.

Citation

@misc{naics-github-classifier-multilingual,
  author = {Quispe, Alexander and Xu, Kevin},
  title  = {NAICS GitHub Repository Classifier, multilingual},
  year   = {2026},
  url    = {https://huggingface.co/aquiro1994/naics-github-classifier-multilingual}
}

Repository

Training and inference code: github.com/alexanderquispe/naics-github-train