###### Abstract  
The field of medical informatics has been significantly transformed in recent years with the emergence of Natural Language Understanding (NLU) and Large Language Models (LLM), providing new opportunities for innovative patient care solutions. This study aims to evaluate the effectiveness of publicly available LLMs as symptom checkers for cardiological diseases by comparing their diagnostic capabilities in real disease cases. We employed a set of 9 models, including ChatGPT-4, OpenSource models, Google PaLM 2, and Meta's LLMa, to assess their diagnostic accuracy, reliability, and safety across various clinical scenarios. Our methodology involved presenting these LLMs with symptom descriptions and test results in Spanish, requiring them to provide specialist diagnoses and recommendations in English. This approach allowed us to compare the performance of each model, highlighting their respective strengths and limitations in a healthcare context. The results revealed varying levels of accuracy, precision, and sensitivity among the models, demonstrating the potential of LLMs to enhance medical education and patient care. By analysing the capabilities of each model, our study contributes to a deeper understanding of artificial intelligence's role in medical diagnosis. We argue for the strategic implementation of LLMs in healthcare, emphasizing the importance of balancing sensitivity and realism to optimize patient outcomes. L +
Footnote †: journal: Medical Informatics  
design of these systems requires careful consideration to ensure accuracy and safety. This entails limiting user inputs to reduce the risk of misinterpretation and validating every possible scenario to prevent potentially harmful errors. The goal is to strike a balance between user-friendliness and dependable medical advice, thereby safeguarding users whilst maximizing the benefits of these digital tools. Large Language Models (LLMs) have been the cornerstone of machine learning research for many years, spanning a wide range of architectures and applications. However, the massive diffusion of advanced LLMs such as ChatGPT during 2023 has revolutionised several fields, starting also to impact medical practice [7; 8; 9; 10; 11; 12]. Our study focuses specifically on large, pre-trained, instruction-tuned generative language models, which represent a significant recent advance. These models, exemplified by systems such as ChatGPT, Google PaLM and Meta LLaMA (8B), differ from their predecessors in their ability to understand and follow complex instructions, generating human-like text in a variety of tasks. Unlike previous LLMs, used primarily for text classification or sentiment analysis, these instruction-adaptive models can engage in open-ended dialogues, answer questions and perform complex reasoning tasks. Their potential in medical applications has attracted particular attention because of their ability to rapidly analyse large volumes of data, which could help in diagnosis and treatment planning This subset of LLMs has gained prominence for its versatility and impressive performance in a variety of domains, such as healthcare, finance and creative writing. By focusing our analysis on these instruction-tuned models, we aim to provide insights into the cutting edge of AI language technology and its potential real-world applications, especially in critical areas such as medical practice. Large Language Models exhibit the capacity to swiftly synthesize extensive datasets, thereby empowering novice physicians to rapidly retrieve precise data, formulate diagnoses, and devise therapeutic strategies. Authors in [7] describe how LLMs are being used in healthcare and what problems need to be solved. These models are good at summarising health information, answering general health questions and eliciting information from patients. However, they are not so good at diagnosing diseases or giving advice based on symptoms. Most researchers are concerned about the accuracy and consistency of LLMs responses.