# Weakly-Supervised Multilingual Medical NER for Symptom Extraction for Low-Resource Languages
## 2 Materials and Methods
### Model Configuration  
For our symptom extraction pipeline, we fine-tuned the BERT-based model (cased), a widely-used transformer model pretrained on English text using a masked language modeling (MLM) objective [27]. This model, developed by Google, is case sensitive, differentiating between lowercase and uppercase words (e.g., \"english\" vs. \"English\"). BERT was pretrained on large English corpora, including BookCorpus and English Wikipedia, using two main objectives: MLM, where 15% of the words in each sentence are masked and predicted, and Next Sentence Prediction (NSP), which trains the model to predict sentence order [27]. The model configuration includes a vocabulary size of 30,000, a maximum token length of 512, and an Adam optimizer with learning rate warmup and linear decay [27]. BERT achieved high performance on a range of NLP tasks, including an average score of 79.6 on the GLUE benchmark, and is well-suited for token classification tasks, such as named entity recognition, when fine-tuned on specific labeled datasets [27]. This makes it an ideal base for developing a specialized model for medical-named entity recognition in our pipeline. The architecture of the BERT model specialized for NER tasks is displayed in Figure 2.  
The model was fine-tuned to classify entities into specific categories: PROBLEM, TEST, and TREATMENT, using the IOB tagging scheme (Inside-Outside-Beginning) [28], a widely-used Standard in NER, where additional tags (e.g., B-, I-, E-, S-) are used to capture entity boundaries. The key training parameters were set as follows: the architecture utilized was BERT-base-cased, a pretrained transformer model with 12 layers, 768 hidden units, and a vocabulary of 30,000 tokens.  
For the training parameters, the model was trained for 200 epochs with a batch size of 64; however, early stopping based on validation loss was applied, and in practice, training typically converged within 20-30 epochs. An AdamW optimizer was used, featuring a learning rate of \(3\times 10^{-5}\) and a weight decay of 0.01 to prevent overfitting. The sequence length was capped at 128 tokens to ensure efficient processing. To address the class imbalance, a focal loss function [29] was applied, emphasizing harder-to-classify samples.  
Table 4 displays more details on the parameters used for model training.  
\begin{tabular}{l c} \hline  & **BERT-BASE-CASED** \\ \hline max sequence length & 128 \\ \hline total parameters (million) & 109 \\ \hline batch size & 64 \\ \hline learning rate & \(3\times 10^{-5}\) \\ \hline warmup steps & 66,540 \\ \hline checkpoint every & 2000 \\ \hline weight decay & 0.01 \\ \hline max number of train epochs & 200 \\ \hline layers & 12 \\ \hline hidden states & 768 \\ \hline attention heads & 12 \\ \hline vocab size & 28,996 \\ \hline train time (hours) & 12–23 \\ \hline loss & focal loss \\ \hline number of GPUs & 1 \\ \hline GPU type & NVIDIA A100-PCIE-40 GB \\ \hline \end{tabular}  
These configurations were selected to maximize the model's ability to extract medical entities accurately from medical texts while maintaining efficiency and generalizability.