Rishabh157/spanmarker-unifieddatasets-multiconer-multilingual-ner-mdeberta

Unified Multilingual SpanMarker NER: mDeBERTa-v3 Fine-Grained (15+ Languages)

Developed by: Rishabh Kumar
Model Name: Rishabh157/krw-spanmarker-multilingual-ner
Base Architecture: microsoft/mdeberta-v3-base
Framework: SpanMarker

An enterprise-grade, state-of-the-art multilingual Named Entity Recognition (NER) model developed by Rishabh Kumar, combining microsoft/mdeberta-v3-base with the SpanMarker candidate span-classification framework.

This model is fine-tuned on the master KRW Unified Multilingual NER Dataset (Rishabh157/krw-unified-multilingual-ner), curated by Rishabh Kumar. The dataset harmonizes three foundational benchmarks: WikiANN, MultiCoNER 2023 (v2), and MultiNERD across 15+ languages and fine-grained entity types.

Unlike traditional sequence taggers that assign BIO labels token-by-token (which frequently suffer from boundary fragmentation, label inconsistency, and tokenizer-split artifacts), SpanMarker explicitly scores candidate phrase spans directly inside mDeBERTa's disentangled attention mechanism, yielding high boundary accuracy (87.32% Precision and 92.21% Sequence Accuracy).

---

📚 Training Dataset: KRW Unified Multilingual NER

This model was trained on the master unified corpus synthesizing three landmark benchmarks:

• 🤗 Master Unified Dataset: Rishabh157/krw-unified-multilingual-ner
• Dataset Curator: Rishabh Kumar
• Corpus Size: 2,252,976 sentences (~47.4 million tokens)
• Supported Languages (15+): English (en), German (de), Spanish (es), French (fr), Italian (it), Portuguese (pt), Swedish (sv), Dutch (nl), Polish (pl), Russian (ru), Bangla (bn), Hindi (hi), Farsi (fa), Ukrainian (uk), and Chinese (zh).

Foundational Datasets Integrated:

1. WikiANN (PAN-X): High-volume, cross-lingual Wikipedia sentences providing strong baseline supervision for core broad entity classes (PER, LOC, ORG).
2. MultiNERD: Multi-genre, multi-domain annotated corpus introducing specialized entity domains including biology (ANIM, PLANT), astronomy (CEL), temporal expressions (TIME), events (EVE), culinary (FOOD), and institutions (INST).
3. MultiCoNER 2023 (SemEval-2023 Task 2): Complex, short, low-context, and noisy web texts targeting fine-grained distinction among confusable subtypes (such as VisualWork, WrittenWork, MusicalWork, PublicCorp, PrivateCorp, Facility, and Station).

---

Model Details

• Developer / Creator: Rishabh Kumar (@Rishabh157)
• Base Backbone: microsoft/mdeberta-v3-base (278.8M parameters, 12 Transformer layers, 768 hidden dimension, 250k multilingual vocabulary)
• Framework: SpanMarker (Candidate span representation with start/end marker tokens $[S_i, E_j]$)
• Tuning Strategy: Full-Parameter Fine-Tuning (all encoder layers and span heads actively optimized)
• Sequence Parameters:
  • model_max_length: 384 tokens (100% empirical coverage across multi-sentence contexts)
  • entity_max_length: 24 words (covers 99.99% of complex multi-word entities)
  • marker_max_length: 128 candidate tokens
• Supported Languages (15+ Languages):
  • English (en), German (de), Spanish (es), French (fr), Italian (it), Portuguese (pt), Swedish (sv), Dutch (nl), Polish (pl), Russian (ru), Bangla (bn), Hindi (hi), Farsi (fa), Ukrainian (uk), Chinese (zh), and multilingual mixed texts.
• Entity Taxonomy: Fine-grained entities spanning Person, Location, Organization, Product, Creative Works, Biomedical, and Events

---

Empirical Benchmark Performance

Evaluated on the full unified validation suite of 186,295 sentences across all 15+ languages:

Metric	Score	Details
Overall Precision	87.32%	High fidelity with minimal false positives across languages
Sequence Accuracy	92.21%	Strict sequence-level correctness
Overall F1	71.18%	Macro/Micro balanced F1 across complex fine-grained classes
Overall Recall	60.08%	High-precision span boundaries
Evaluation Loss	0.0115	Cross-entropy span classification loss

Qualitative Benchmark & Real-World Evaluation

The model was qualitatively benchmarked against diverse multilingual test cases from the standard evaluation suite (samplefi/exper/sentence.json):

Test Sentence	Language	Detected Entities	Confidence	Analysis
"Barack Obama visited Paris last Tuesday and met representatives from Microsoft."	English	• [PER] "Barack Obama" • [LOC] "Paris"	0.999 0.996	Sharp boundary isolation; past-tense verbs and temporal tokens cleanly excluded.
"Taylor Swift performed at Wembley Stadium during The Eras Tour in London."	English	• [PER] "Taylor Swift" • [LOC] "Wembley Stadium" • [EVE] "The Eras Tour"	0.998 0.952 0.829	Successfully identifies multi-word event and venue spans without noun-verb bleed.
"Satya Nadella spoke at Microsoft's headquarters in Redmond on Monday morning."	English	• [PER] "Satya Nadella" • [ORG] "Microsoft's" • [LOC] "Redmond"	0.998 0.674 0.996	Clean boundary extraction of possessive corporate entities and city locations.
"विराट कोहली ने मुंबई के वानखेड़े स्टेडियम में ऐतिहासिक शतक बनाया।"	Hindi	• [Athlete] "विराट कोहली" • [Facility] "वानखेड़े स्टेडियम"	0.264 0.669	Cross-lingual non-Latin script transfer into fine-grained entity types (Athlete, Facility).
"Toyota unveiled a new electric vehicle at the Tokyo Motor Show."	English	• [ORG] "Toyota" • [EVE] "Tokyo Motor"	0.987 0.878	Extracts commercial organization and industry exhibition event.
"Angela Merkel besuchte gestern die Universität in München."	German	• [PER] "Angela Merkel"	0.998	High-confidence cross-lingual transfer on Germanic syntax.
"Gabriel García Márquez escribió Cien años de soledad en la Ciudad de México."	Spanish	• [PER] "Gabriel García Márquez"	0.996	Complete multi-token surname boundary preservation.

---

Model Labels & Taxonomy

The model unifies taxonomies across WikiANN, MultiCoNER 2023, and MultiNERD into cohesive fine-grained entity types:

Category Overview

• Person (PER): Artist, Athlete, Cleric, Politician, Scientist, SportsManager, OtherPER, PER
• Location (LOC): Facility, HumanSettlement, Station, OtherLOC, LOC
• Organization (ORG): AerospaceManufacturer, CarManufacturer, MusicalGRP, PublicCorp, PrivateCorp, SportsGRP, ORG
• Product (PROD): Clothing, Drink, Food, FOOD, Software, Vehicle, VEHI, OtherPROD
• Creative Works: ArtWork, MusicalWork, VisualWork, WrittenWork
• Biomedical & Science: AnatomicalStructure, Disease, DIS, MedicalProcedure, Medication/Vaccine, Symptom, ANIM, BIO, PLANT, CEL
• Events & Miscellaneous: EVE, INST, MEDIA, MYTH, TIME

Full Label Mapping & Examples

Label	Examples
ANIM	"Wandermuschel", "Pteranodon", "Acinetobacter baumannii"
AerospaceManufacturer	"স্পেস এক্স", "ফকার", "এয়ারবাস", "Boeing", "Airbus"
AnatomicalStructure	"পলক (চোখ)", "গলবিল", "গলা", "femur", "cerebral cortex"
ArtWork	"মোনা লিসা", "গুহাচিত্র", "রেনেসাঁ শিল্প", "Mona Lisa", "Guernica"
Artist	"চার্লস নেভিল", "এলভিন জোন্স", "Pablo Picasso", "Vincent van Gogh"
Athlete	"পল নিউম্যান", "Lionel Messi", "Cristiano Ronaldo", "Usain Bolt"
BIO	"Perlpilz", "benthischen", "Escherichia coli", "Bacillus subtilis"
CEL	"Venus", "Jupiter", "Andromeda Galaxy", "Hubble Deep Field"
CarManufacturer	"ভলভো গ্রুপ", "অ্যাস্টন মার্টিন", "জেনারেল মোটর্স", "Toyota", "Tesla"
Cleric	"পোপ পঞ্চদশ বেনেডিক্ট", "মুহাম্মাদ", "পোপ ফ্রান্সিস", "Dalai Lama"
Clothing	"ফেজ (টুপি)", "মেখলা চাদর", "শাড়ি", "kimono", "tuxedo"
DIS	"spastische Paraplegie", "AIDS", "meldepflichtigen Tierkrankheiten"
Disease	"মাংকিপক্স", "টাইফাস জ্বর", "गुटीবসন্ত", "COVID-19", "Alzheimer's"
Drink	"কফি", "ডিস্টিল্ড পানীয়", "বিয়ার", "espresso", "green tea"
EVE	"UEFA - Pokals 2008 / 09", "Ersten Weltkriegs", "Olympic Games"
FOOD	"Syrah", "Cinsault", "Grenache", "Parmigiano-Reggiano"
Facility	"স্ট্রিপ ক্লাব", "থিংভেৎলির", "ডাকঘর", "Wembley Stadium", "CERN"
Food	"চাল", "আলু", "কলা", "sushi", "baguette"
HumanSettlement	"জর্জ টাউন", "কোচি", "Paris", "Tokyo", "New York City", "Berlin"
INST	"OM 651", "The Clock of the Long Now", "Stradivarius violin"
LOC	"Atelocentra", "Durrães e Tregosa", "Alps", "Amazon River"
MEDIA	"Nash Bridges", "Nevermind", "Gangnam Style", "BBC News"
MYTH	"Laurentius", "Radha", "Thomas von Aquin", "Zeus", "Apollo"
MedicalProcedure	"যোগ (হিন্দুধর্ম)", "মনোবিজ্ঞানী", "গর্ভপাত", "angioplasty", "MRI"
Medication/Vaccine	"ইথানল", "কার্বন ডাই অক্সাইড", "দারুচিনি", "aspirin", "penicillin"
MusicalGRP	"ডেস্টিনি'স চাইল্ড", "বিশাল-শেখর", "ইউটু", "The Beatles", "Coldplay"
MusicalWork	"ও মিও বাব্বিনো কারো", "হ্যামিল্টন", "Bohemian Rhapsody", "Symphony No. 9"
ORG	"United Nations", "European Space Agency", "World Health Organization"
OtherLOC	"জিন্নাহ স্টেডিয়াম", "ইডেন পার্ক", "ফরমোসা প্রণালী", "Silicon Valley"
OtherPER	"এডিথ কির্‌চমাইয়ার", "শতবর্ষজীবি", "historical figures"
OtherPROD	"সৌর প্যানেল", "প্ল্যাটফর্ম স্ক্রিন ডোর", "চন্দ্র এক্স-রশ্মি মানমন্দির"
PER	"Barack Obama", "Marie Curie", "Antonín Rýgr", "Franz Liszt"
PLANT	"Lattiche", "Feigenkakteen", "Weißen Pestwurz", "Quercus robur"
Politician	"স্টিফেন জোসেফ হার্পার", "Emmanuel Macron", "Angela Merkel"
PrivateCorp	"সুরিন্দর ফিল্মস", "হুয়াওয়েই", "বেল ল্যাবস্", "ByteDance", "OpenAI"
PublicCorp	"ইন্টেল কর্পোরেশন", "অ্যাপল", "রেড হ্যাট", "Microsoft", "Google"
Scientist	"আর্কিমিডিস", "অ্যালেন নেওয়েল", "গ্যালিলিও গ্যালিলেই", "Albert Einstein"
Software	"আইওএস", "facebook", "লিনাক্স", "PyTorch", "Kubernetes", "Windows"
SportsGRP	"ইউটাহ জ্যাজ", "ফিনিক্স সান্‌স", "স্কুদেরিয়া ফেরারি", "Real Madrid"
SportsManager	"ডেভ সেক্সটন", "লিওনিদাস", "মিশেল প্লাতিনি", "Pep Guardiola", "Alex Ferguson"
Station	"রেল স্টেশন", "ব্রিস্টল বিমানবন্দর", "ইন্দিরা গান্ধী আন্তর্জাতিক বিমানবন্দর"
Symptom	"আমাশয়", "ব্যথা", "fever", "cough", "headache"
TIME	"Golden Goal", "Weltwirtschaftskrise", "Aufklärung", "21st century"
VEHI	"Hawker Hurricane", "Tupolew A - 3", "Airbus A 310", "Falcon 9"
Vehicle	"অতি-সাইবেরীয় রেলপথ", "জাহাজ", "বোয়িং ৭৪৭", "Model S", "Boeing 777"
VisualWork	"দ্য ফ্যালকন এন্ড দ্য উইন্টার সোলজার", "Inception", "Interstellar"
WrittenWork	"ওয়াশিংটন পোস্ট", "সাময়িক পত্র", "Principia Mathematica", "Hamlet"

---

Usage

Direct Use for Inference

Install the span_marker package:

pip install span_marker transformers torch

Run prediction on single sentences or batches:

```python
from span_marker import SpanMarkerModel

# Load the fine-tuned model from Hugging Face Hub
model = SpanMarkerModel.from_pretrained("Rishabh157/spanmarker-unifieddatasets-multiconer-multilingual-ner-mdeberta")

# Optional: Run on GPU for accelerated inference
# model.cuda()

# Single sentence inference
text = "Barack Obama visited Paris and met representatives from Microsoft."
entities = model.predict(text)

for entity in entities:
    print(f"[{entity['label']}] \"{entity['span']}\" (confidence: {entity['score']:.4f})")
# Output:
# [PER] "Barack Obama" (confidence: 0.9950)
# [LOC] "Paris" (confidence: 0.9837)
# [PublicCorp] "Microsoft" (confidence: 0.9521)

Multilingual Batch Inference

sentences = [
    "Satya Nadella announced Microsoft's new artificial intelligence campus in Redmond.",
    "Albert Einstein entwickelte die Relativitätstheorie in Bern und Zürich.",
    "SpaceX launched a Falcon 9 rocket carrying Starlink satellites from Florida.",
    "Marie Curie received the Nobel Prize in Physics and later the Nobel Prize in Chemistry.",
    "रविन्द्रनाथ ठाकुर नोबेल पुरस्कार प्राप्त करने वाले पहले गैर-यूरोपीय साहित्यकार थे।"
]

predictions = model.predict(sentences)

for text, preds in zip(sentences, predictions):
    print(f"\nText: \"{text}\"")
    for ent in preds:
        print(f"  • [{ent['label']}] \"{ent['span']}\" (score: {ent['score']:.3f})")

Downstream Fine-Tuning

You can also adapt or continue training this model on your domain-specific dataset:

Click to expand

from datasets import load_dataset
from span_marker import SpanMarkerModel, Trainer
from transformers import TrainingArguments

# 1. Load your pretrained SpanMarker model
model_id = "Rishabh157/spanmarker-unifieddatasets-multiconer-multilingual-ner-mdeberta"
model = SpanMarkerModel.from_pretrained(model_id)

# 2. Load dataset (requires "tokens" and "ner_tags" columns)
dataset = load_dataset("conll2003")

# 3. Define training arguments
args = TrainingArguments(
    output_dir="spanmarker_custom_finetuned",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    fp16=True,
    logging_steps=50,
    eval_strategy="epoch",
    save_strategy="epoch",
)

# 4. Initialize SpanMarker Trainer
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)

# 5. Train and save
trainer.train()
model.save_pretrained("spanmarker_custom_finetuned_final")

---

Citation

If you use this model or the KRW unified dataset in your work, please cite:

@software{kumar2026_mdeberta_spanmarker_unified,
  author       = {Kumar, Rishabh},
  title        = {Unified Multilingual SpanMarker NER: mDeBERTa-v3 Fine-Grained},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Rishabh157/spanmarker-unifieddatasets-multiconer-multilingual-ner-mdeberta}}
}

@misc{rishabhkumar2026_krw_unified_multilingual_ner,
  author       = {Kumar, Rishabh},
  title        = {KRW Unified Multilingual Named Entity Recognition Dataset},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/Rishabh157/krw-unified-multilingual-ner}}
}