Update README.md

README.md

@@ -1,3 +1,121 @@
----
-license: mit
----
+---
+language:
+- en
+- zh
+- es
+- ja
+- ru
+tags:
+- fill-mask
+- clinical-nlp
+- multilingual
+- bert
+license: mit
+---
+
+# MultiClinicalBERT: A Multilingual Transformer Pretrained on Real-World Clinical Notes
+
+MultiClinicalBERT is a multilingual transformer model pretrained on real-world clinical notes across multiple languages. It is designed to provide strong and consistent performance for clinical NLP tasks in both high-resource and low-resource settings.
+
+To the best of our knowledge, this is the first open-source BERT model pretrained specifically on multilingual real-world clinical notes.
+
+
+## Model Overview
+
+MultiClinicalBERT is initialized from `bert-base-multilingual-cased` and further pretrained using a two-stage domain-adaptive strategy on a large-scale multilingual clinical corpus (BRIDGE), combined with biomedical and general-domain data.
+
+The model captures:
+- Clinical terminology and documentation patterns
+- Cross-lingual representations for medical text
+- Robust performance across diverse healthcare datasets
+
+
+## Pretraining Data
+
+The model is trained on a mixture of three data sources:
+
+### 1. Clinical Data (BRIDGE Corpus)
+- 87 multilingual clinical datasets  
+- ~1.42M documents  
+- ~995M tokens  
+- Languages: English, Chinese, Spanish, Japanese, Russian  
+
+This dataset reflects real-world clinical practice and is the core contribution of this work.
+
+### 2. Biomedical Literature (PubMed)
+- ~1.25M documents  
+- ~194M tokens  
+
+Provides domain knowledge and medical terminology.
+
+### 3. General-Domain Text (Wikipedia)
+- ~5.8K documents  
+- ~43M tokens  
+- Languages: Spanish, Japanese, Russian  
+
+Improves general linguistic coverage.
+
+### Total
+- ~2.7M documents  >1.2B tokens  
+
+
+## Pretraining Strategy
+
+We adopt a **two-stage domain-adaptive pretraining approach**:
+
+### Stage 1: Mixed-domain pretraining
+- Data: BRIDGE + PubMed + Wikipedia  
+- Goal: Inject biomedical and multilingual knowledge  
+
+### Stage 2: Clinical-specific adaptation
+- Data: BRIDGE only  
+- Goal: Learn fine-grained clinical language patterns  
+
+### Objective
+- Masked Language Modeling (MLM)  
+- 15% token masking  
+
+
+## Evaluation
+
+We evaluate MultiClinicalBERT on **11 clinical NLP tasks across 5 languages**:
+
+- English: MIMIC-III Mortality, MedNLI, MIMIC-IV CDM  
+- Chinese: CEMR, IMCS-V2 NER  
+- Japanese: IFMIR NER, IFMIR Incident Type  
+- Russian: RuMedNLI, RuCCoNNER  
+- Spanish: De-identification, PPTS  
+
+### Key Results
+- Consistently outperforms multilingual BERT (mBERT)
+- Matches or exceeds strong language-specific models
+- Largest gains observed in low-resource settings
+- Statistically significant improvements (Welch’s t-test, p < 0.05)
+
+Example:
+- MedNLI: **83.90% accuracy**
+- CEMR: **86.38% accuracy**
+- IFMIR NER: **85.53 F1**
+- RuMedNLI: **78.31% accuracy**
+
+
+## Key Contributions
+
+- First BERT model pretrained on **multilingual real-world clinical notes**
+- Large-scale clinical corpus (BRIDGE) with diverse languages
+- Effective **two-stage domain adaptation strategy**
+- Strong performance across **multiple languages and tasks**
+- Suitable for:
+  - Clinical NLP
+  - Multilingual medical text understanding
+  - Retrieval-augmented generation (RAG)
+  - Clinical decision support systems
+
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("YLab-Open/MultiClinicalBERT")
+model = AutoModel.from_pretrained("YLab-Open/MultiClinicalBERT")