## 2 Key Concepts in Natural Language Processing
### Tokenization and Embeddings  
Tokenization is a crucial preprocessing step in NLP that converts text into tokens, which are smaller units of meaning. In the context of LLMs, tokenization is essential for transforming clinical case descriptions into a format that the model can understand and process. Effective tokenization ensures that the context and semantic meaning of the clinical data are preserved. Mathematically, tokenization can be described as:  
\[T=\text{Tokenizer}(X) \tag{1}\]  
where \(X\) is the input text, and \(T\) is the resulting tokenized representation.  
Tokenization is particularly important in symptom checkers, where the accuracy of the diagnostic predictions can directly impact patient outcomes. For clinical cases, where medical jargon and complex terminology are common, this method ensures that words like \"myocardial infarction\" or \"hypertension\" are adequately handled by breaking them down into subword units that the model can understand and process efficiently. The process involves breaking down the text into individual words or subwords, which are then converted into numerical representations called embeddings.  
Embeddings capture the semantic meaning of the text, allowing the LLM to understand the context and relationships between different parts of the clinical description. There are several types of embeddings, including word embeddings (Word2 Vec [18] and GloVe [19]), subword embeddings (Byte Pair Encoding and WordPiece), and contextual embeddings. In BERT [7] and RoBERTa [8], contextual embeddings are generated through a process that maps tokens to high-dimensional vectors \(\mathbf{e_{t}}\) using a pre-trained lookup table:  
\[e_{t}=\text{Embeddings}(t) \tag{2}\]  
where \(e_{t}\) is the vector representation of token \(t\). These vectors are used by the model to understand and generate text by considering the context in which a word appears, allowing the same word to have different embeddings based on its usage. For instance, the representation of a word like \"pain\" will change depending on the surrounding terms (e.g., \"chest pain\" vs. \"muscle pain\").  
Mixture of Experts (MoE) is an advanced technique used to enhance the capabilities of LLMs by leveraging a combination of specialized sub-models, or \"experts,\" each trained to handle specific aspects of a task. In an MoE architecture, the model dynamically selects the most appropriate experts to process different parts of the input, allowing for more efficient and effective handling of complex tasks. This approach has been shown to improve the performance and scalability of LLMs, especially in scenarios requiring specialized knowledge or fine-grained processing [20]. Subword tokenization is commonly used in MoE, where specialized embeddings are often computed based on the \"expert\" selected by the gating mechanism, enhancing the model's ability to specialize in specific domains. Thus, the embedding process can be formalized as:  
\[\mathbf{e}_{t}=\sum_{i=1}^{N}\alpha_{i}\cdot\mathbf{e}_{i,t} \tag{3}\]  
where \(a_{i}\) represents the attention weights from the gating mechanism selecting which experts \(i\) will contribute to the final embedding \(e_{t}\), and \(N\) is the number of experts.