## 2 Key Concepts in Natural Language Processing
### Large Language Models  
Large Language Models are a class of artificial intelligence models designed to understand and generate human language. These models are trained on vast amounts of text data and can perform a wide range of NLP tasks, such as text generation, translation, summarization, and question-answering [13; 14; 21; 22]. LLMs leverage deep learning techniques, particularly transformer architectures [10], to capture complex linguistic patterns and contextual information. Once the tokens are tokenized and embedded, they pass through layers of attention and neural activations, where the model focuses on the most relevant parts of the input. This activation is achieved using self-attention layers, which allow each token to attend to every other token in the input. The attention mechanism is defined mathematically as:  
\[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}} \right)V \tag{4}\]  
where \(Q\) is the query matrix (representing the current token), \(K\) is the key matrix (representing all tokens), \(V\) is the value matrix (containing the content of the tokens), and \(d_{k}\) is the dimension of the key matrix (scaling factor). This attention mechanism allows BERT to learn contextual relationships between tokens, which is crucial in complex medical diagnoses, where understanding the relationship between \"chest pain\" and \"shortness of breath\" is essential.  
In MoE, the gating mechanism selects the most appropriate subset of experts to process the tokens. Each token \(t\) is processed by a subset of experts \(E_{t}\):  
\[z_{t}=\sum_{i\in E_{t}}W_{i}\cdot\mathbf{e}_{t} \tag{5}\]  
where \(z_{t}\) is the activated output for token \(t\), and \(W_{i}\) represents the weights associated with each expert \(i\). The gating mechanism ensures that only the most relevant experts are activated, optimizing the network's efficiency and domain-specific performance.  
During propagation, information moves through multiple layers, refining the embeddings to generate a final diagnostic prediction. This process involves a series of transformations, with each layer applying a transformation \(f\):  
\[\mathbf{h}_{I}=f(\mathbf{h}_{I-1}) \tag{6}\]  
where \(h_{I}\) is the hidden state of layer \(l\), and \(f\) is the transformation function, which could be a feed-forward neural network or another attention layer. This iterative process allows the model to refine its understanding of the clinical case, producing a more accurate diagnosis.  
Fine-tuning is a process used to adapt pre-trained LLMs to specific domains or tasks by further training the model on domain-specific data. This approach allows the model to learn the unique linguistic patterns, terminology, and contextual nuances of the target domain,enhancing its performance and accuracy. Fine-tuning has been successfully applied in various domains, including healthcare, where models are fine-tuned using specialized clinical data to improve their understanding and generation of medical information [23; 24]. By adapting the model to the unique characteristics of the target domain, fine-tuning can significantly improve its accuracy and reliability. Fine-tuned models are better equipped to understand and generate contextually relevant responses, making them more effective in specialized applications.  
In the context of this study, LLMs are evaluated for their ability to predict medical diagnoses from Spanish textual descriptions of clinical cases. Techniques such as example-based prompting and Retrieval-Augmented Generation (RAG) are explored to enhance the quality of predictions. The impact of temperature settings and prompt strategies on diagnostic accuracy is assessed, contributing to the understanding of the strengths and limitations of LLMs in this domain.