Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics

## Abstract

Abstract Large language models (LLMs) have demonstrated remarkable progress in healthcare. However, a significant gap remains regarding LLMs’ professionalism in domain-specific clinical practices, limiting their application in real-world diagnostics. In this work, we introduce Zodiac , an LLM-powered framework with cardiologist-level professionalism designed to engage LLMs in cardiological diagnostics. Zodiac assists cardiologists by extracting clinically relevant characteristics from patient data, detecting significant arrhythmias, and generating preliminary reports for the review and refinement by cardiologists. To achieve cardiologist-level professionalism, Zodiac is built on a multi-agent collaboration framework, enabling the processing of patient data across multiple modalities. Each LLM agent is fine-tuned using real-world patient data adjudicated by cardiologists, reinforcing the model’s professionalism. Zodiac undergoes rigorous clinical validation with independent cardiologists, evaluated across eight metrics that measure clinical effectiveness and address security concerns. Results show that Zodiac outperforms industry-leading models, including OpenAI’s GPT-4o, Meta’s Llama-3.1-405B, and Google’s Gemini-pro, as well as medical-specialist LLMs like Microsoft’s BioGPT. Zodiac demonstrates the transformative potential of specialized LLMs in healthcare by delivering domain-specific solutions that meet the stringent demands of medical practice. Notably, Zodiac has been successfully integrated into electrocardiography (ECG) devices, exemplifying the growing trend of embedding LLMs into Software-as-Medical-Device (SaMD).

## 1 Introduction

As technology continues to revolutionize healthcare, artificial intelligence (AI) has emerged as a crucial component of medical devices, driving the expansion of digital health in clinical practice (FDA, ). Among the most promising AI advancements, large language models (LLMs) are unlocking new possibilities in digital health. With their human-like conversational skills and vast pre-trained knowledge, LLMs are increasingly being adopted as clinical support tools by industry leaders, evolving into specialized clinical agents (Boonstra et al., ; Gala & Makaryus, ; Xu et al., ). This evolution has given rise to industrial products such as Microsoft’s BioGPT (Luo et al., ), Google’s Med-Gemini (Saab et al., ) and Med-PaLM (Tu et al., ), as well as a range of open-source medical specialist LLMs (Chen et al., ; ; ContactDoctor, ; Wang et al., ) built on Meta’s Llama (Touvron et al., ).

Figure: Figure 1: Zodiac attains cardiologist-level professionalism through a combination of advanced data integration with sophisticated technical designs.
Refer to caption: x1.png

Despite these advancements, integrating LLMs into real-world healthcare practice remains in its early stages, where a significant gap exists in their professionalism (Davenport & Kalakota, ; Asan et al., ; Weber et al., ; Quinn et al., ). Bridging these gaps is critical, especially when deploying LLMs in healthcare settings governed by the FDA’s Software-as-a-Medical-Device (SaMD) regulations (FDA, ). These regulations require software to demonstrate expert-level proficiency to function as a clinical assistant. However, current LLMs often fall short of this standard due to their general-purpose design, which lacks alignment with the specific standards of clinical practice (Khan et al., ; Wang et al., ; Kerasidou et al., ). In the context of SaMD, LLMs need not have universal capabilities but they must perform specialized tasks with the professionalism and accuracy expected in life-critical healthcare environments (Kelly et al., ; Yan et al., ). Achieving this alignment is vital to ensure LLMs meet the stringent requirements of real-world healthcare deployment.

Our Work. This study aims to address the challenge of aligning LLMs with the SaMD practice in the field of Cardiology, focusing on the clinical findings and interpretation of electrocardiogram (ECGs) (Yanowitz, ). We introduce Zodiac, an LLM-powered, multi-agent framework designed to achieve cardiologist-level professionalism. Zodiac assists cardiologists by identifying clinically relevant characteristics from patient data, detecting significant arrhythmias, and generating preliminary reports for expert review and refinement (details in 3). As illustrated in Figure 1, Zodiac combines advanced data integration with sophisticated technical designs to achieve professional performance. Specifically:

I) Data-Driven Professionalism: Zodiac is built on real-world data, including (1) patient data collected from clinics, (2) cardiologist-adjudicated texts, and (3) clinical guidelines. This ensures professionalism in two key aspects: First, Zodiac captures real-world cardiological characteristics, such as arrhythmias and their contributing factors, rather than relying on benchmarks or synthetic datasets that may not reflect clinical realities. Second, direct involvement from human experts (cardiologists) ensures Zodiac is fine-tuned to match expert-level performance, while adherence to clinical guidelines mitigates potential biases or errors, enhancing diagnostic accuracy and safety.

II) Technique-Driven Professionalism: The technical design of Zodiac aligns with cardiologist-level diagnostic practices. Our pipeline is multi-agent, utilizing multiple LLMs to analyze multimodal patient data, including clinical metrics in tabular format and ECG tracings in image format. This multi-agent framework represents a paradigm cardiologists use to identify key characteristics and interpret findings for clinically significant arrhythmias. Furthermore, during fine-tuning and inference, we employ cardiologist-adjudicated data, integrating multimodal diagnostic professionalism through instruction tuning and in-context learning. Instruction tuning embeds professionalism into the LLM’s parameters, while in-context learning provides professional demonstrations to further reinforce Zodiac’s diagnostics. Finally, we incorporate fact-checking against established cardiological guidelines to ensure the system generates accurate, expert-verified diagnostics.

Clinical Validation. Our experiments aim to bridge the gap between cardiologists and LLM-powered systems in terms of recognizing the professionalism of LLMs. To this end, we conduct a series of clinical validations to assess Zodiac’s clinical effectiveness and security. We utilize eight evaluation metrics and collaborate with cardiologists to evaluate Zodiac’s performance against leading LLMs, including OpenAI’s ChatGPT-4o, Google’s Gemini-Pro, Meta’s Llama-405B, and specialized medical LLMs like Microsoft’s BioGPT and Llama variants. With cardiologists endorsing Zodiac through its application in real patient diagnostics, we underscore its practical utility from the healthcare provider’s perspective.

Blueprint and Lifecycle Prospect. Zodiac has been successfully deployed on Amazon AWS and integrated with clinical settings to assist in cardiological diagnostics (details in 4.4). The design, development, and deployment of Zodiac provide a comprehensive blueprint for introducing professional-grade LLM agents into clinical-grade SaMD, encompassing data utilization, expert-level technical pipelines, and clinical validation. Furthermore, we incorporate human expertise (cardiologists) throughout data preparation and clinical validation, ensuring the professionalism of the proposed system while earning expert endorsement and trust through real-world validation. This approach promotes establishing a “humans-in-the-loop” lifecycle in responsible AI development (Food et al., ).

In summary, this work makes the following contributions:

- –
We introduce Zodiac, a cardiologist-level, multi-agent framework for patient-specific diagnostics, representing a significant step forward in aligning LLM capabilities with professional software-as-medical-device (SaMD) standards.
–
We provide a comprehensive blueprint for constructing Zodiac, offering a scalable framework that can guide the development of clinical-grade LLM agents across various clinical domains.
–
Through rigorous clinical validation, we demonstrate the effectiveness of Zodiac while establishing a model for integrating human oversight throughout the AI lifecycle, crucial for promoting responsible AI development under human regulation.

## 2 Related Work

LLMs in Clinical Diagnostics. LLMs have shown considerable progress in processing and interpreting vast amounts of unstructured medical data, such as patient records, medical literature, and diagnostic reports. For example, Han et al. () introduced a system that automatically summarizes clinical notes during interactions between patients and clinicians, while Ahsan et al. () explored the role of LLMs in retrieving key evidence from electronic health records (EHRs). Despite these successes, concerns persist regarding LLMs’ domain-specific expertise and professional performance in high-stakes, life-critical clinical settings (Nashwan & AbuJaber, ; Jahan et al., ; Wang et al., ; Li et al., ).
This work addresses these concerns by designing and validating Zodiac through our design and experiments specifically for cardiological diagnostics.

Multi-Agent Frameworks. Multi-agent frameworks have been extensively studied to enhance LLM capabilities in handling complex tasks and managing distributed processes (Wang et al., ; Hong et al., ; Du et al., ; Chan et al., ). In healthcare, where collaboration across different expertise is essential, multi-agent frameworks have shown their potential in optimizing patient management, coordinating care between various agents (e.g., doctors, nurses, administrative systems), and supporting decision-making processes (Furmankiewicz et al., ; Jemal et al., ; Shakshuki & Reid, ). Recent studies have also focused on leveraging multi-LLM agents to reduce manual tasks in healthcare workflows. For instance, Chen et al. () employed ChatGPT in distinct roles within a coordinated workflow, to automate tasks like database mining and drug repurposing, while ensuring quality control through role-based collaboration.

Cardiological Diagnostic Systems. Current cardiological diagnostic systems primarily depend on rule-based algorithms or single-agent approaches for identifying cardiovascular risk factors or predicting cardiac events (Goff Jr et al., ; Sud et al., ; Olesen et al., ). In recent years, deep learning models have been introduced into cardiology (Hannun et al., ; Acharya et al., ). However, there remains a significant gap in incorporating recent LLMs into cardiological diagnostics—a gap that this work addresses significantly.

## 3 Problem Formulation from Cardiologist-Level Diagnostics

This section outlines how Zodiac is aligned with cardiological diagnostics. Section 3.1 defines the task of cardiological diagnostics and its key components. Section 3.2 introduces the multimodal data used in real-world diagnostics. Finally, in Section 3.3, we formalize the task from the perspective of LLMs using a multi-agent framework.

Figure: Figure 2: Zodiac aligns with cardiological practice through a multi-agent framework that integrates patient data across various modalities: ➀ Patient data is collected in two modalities: tabular metrics and ECG tracings (images). ➁ A metrics-to-findings LLM agent processes the tabular metrics and generates text-based clinical findings. ➂ An tracings-to-findings LLM agent analyzes the ECG tracings to produce additional text-based clinical findings. ➃ The clinical findings from both agents are then combined. ➄ A findings-to-interpretation LLM agent synthesizes these findings with clinical guidelines into comprehensive diagnostic interpretation. ➅ Zodiac generates a patient-specific report by integrating the metrics, tracings, clinical findings, and diagnostic interpretation. ➆ A cardiologist validates the quality of the generated findings and interpretations (details in 5). For simplicity, we omit the biostatistics ($\mathcal{B}$) in this figure, which is considered in steps ➀➁➂ by default.
Refer to caption: x2.png

### 3.1 The Diagnostic Task and Its Key Components

The focus of this paper is the detection of clinically significant arrhythmias using patient data. We categorize the key components into two main categories: patient data and diagnostic outputs.

Patient Data is comprised of three sections: (1) Biostatistical information ($\mathcal{B}$) provides background details about the patient such as date of birth, gender, and age group. (2) Metrics ($\mathcal{M}$) summarize cardiological attributes and their corresponding values presented in a tabular format, providing an overview of 24-hour monitored statistics for the patient. For example, AF Burden: 12% indicates that the patient experienced atrial fibrillation for 12% of the whole monitoring period. (3) Tracings ($\mathcal{T}$) includes ECG images depicting clinically significant arrhythmias such as AFib/Flutter (Atrial Fibrillation / Atrial Flutter), Pause, VT (Ventricular Tachycardia), SVT (Supraventricular Tachycardia), and AV Block (Atrioventricular Block). $\mathcal{T}$ presents a concise but representative segment of the 24-hour monitoring, such as a 10-second strip highlighting the highest degree of AV block.

Diagnostic Outputs is comprised of two elements: clinical Findings ($\mathcal{F}$) and Interpretation ($\mathcal{I}$), both presented as expert-crafted natural language statements by cardiologists. $\mathcal{F}$ outlines key observations directly from clinically relevant characteristics, while $\mathcal{I}$ provides the final diagnostics, interpreting these findings. For example, the finding PR Interval is 210 milliseconds in the ECG tracings leads to the interpretation: The PR interval is slightly prolonged, suggesting a first-degree AV block.

Once $\mathcal{F}$ and $\mathcal{I}$ are completed by cardiologists (or by Zodiac), a clinical end-of-study report is generated for the patient, including $(\mathcal{B},\mathcal{M},\mathcal{T},\mathcal{F},\mathcal{I})$, as illustrated in the right part of Figure 2.

### 3.2 Cardiological Diagnostics with Multimodal Data

Cardiological diagnostics follows a process: $(\mathcal{B},\mathcal{M},\mathcal{T})\rightarrow\mathcal{F}\rightarrow\mathcal
{I}$. In addition, the final interpretation is guided by clinical guidelines, denoted as $\mathcal{G}$, which are consensus-based recommendations designed to support healthcare providers in making evidence-based decisions (detailed in D).

A cardiologist begins by reviewing the patient’s data $(\mathcal{B},\mathcal{M},\mathcal{T})$ to identify clinically relevant characteristics, such as the PR interval, which are key for diagnosing arrhythmias. These identified characteristics are then summarized into natural language statements, referred to as findings $\mathcal{F}$, which integrate insights from both tabular metrics $\mathcal{M}$ and image-based ECG tracings $\mathcal{T}$. For example, the PR interval is derived from $\mathcal{T}$, while the AF burden is obtained from $\mathcal{M}$. Finally, cardiologists synthesize the findings $\mathcal{F}$ with their clinical expertise and the established guidelines $\mathcal{G}$ to form the final interpretation $\mathcal{I}$.

### 3.3 Problem Formulation in Zodiac

As illustrated in Figure 2, Zodiac represents the diagnostic process through a multi-agent collaboration. Each LLM agent is tasked with a specific stage in the diagnostic workflow, enhancing the system’s ability to identify hybrid characteristics across multiple modalities. Zodiac is composed of three agents:

- 1.
Metrics-to-Findings Agent ($\theta_{\texttt{M2F}}$): A table-to-text LLM that extracts key characteristics from tabular metrics ($\mathcal{M}$), while incorporating patient biostatistics from $\mathcal{B}$ to generate clinical findings.
2.
Tracings-to-Findings Agent ($\theta_{\texttt{T2F}}$): An image-to-text LLM that identifies key factors from ECG tracings ($\mathcal{T}$), integrates relevant information from $\mathcal{B}$, and produces clinical findings.
3.
Findings-to-Interpretation Agent ($\theta_{\texttt{F2I}}$): A text-based LLM that synthesizes findings ($\mathcal{F}$) from both the $\theta_{\texttt{M2F}}$ and $\theta_{\texttt{T2F}}$, applies clinical guidelines from $\mathcal{G}$, and generates the interpretation ($\mathcal{I}$).

Formally, the process of Zodiac is formulated as:

$$ $\displaystyle\begin{split}\mathcal{I}\leftarrow\theta_{\texttt{F2I}}(\mathcal{ F},\mathcal{G})\,\,\,\,&s.t.\,\,\,\,\mathcal{F}\leftarrow\theta_{\texttt{M2F}} (\mathcal{M},\mathcal{B})\cup\theta_{\texttt{T2F}}(\mathcal{T},\mathcal{B}) \end{split}$ (1) $$

In this process, $\theta_{\texttt{M2F}}$ and $\theta_{\texttt{T2F}}$ independently generate clinical findings based on $\mathcal{M}$ and $\mathcal{T}$, respectively, which are then combined to form $\mathcal{F}$. This approach adheres to cardiological diagnostics as each finding in $\mathcal{F}$ corresponds to evidence derived from a specific modality – either metrics or ECG tracings.

## 4 Design, Development, and Deployment of Zodiac

This section details how Zodiac achieves cardiologist-level expertise, as well as its compliance with Software-as-a-Medical-Device (SaMD) regulations. We begin by discussing the process of real-world data collection and the professionalism-incorporated curation, outlined in Section 4.1. Next, we introduce the instruction fine-tuning in Section 4.2, where the curated data is used to imbue the LLM agents with domain-specific expertise. During inference, as described in Section 4.3, we leverage in-context learning and fact-checking to enhance diagnostic professionalism through collaborative multi-agent interactions. Finally, we present our approach to SaMD-compliant deployment in Section 4.4.

### 4.1 Data Collection and Professionalism-Incorporated Curation

Our data is characterized as real, representative, and professionalism-incorporated.

Real-World Patient Data. Instead of relying on public benchmarks from third-party or synthetic data—which often raise concerns about trustworthiness or misalignment with specific clinical applications (Chouffani El Fassi et al., ; Fehr et al., ; Youssef et al., )—we utilized ECG data sourced from our collaborating healthcare institutions(^1^11For anonymity, the names of our collaborating institutions are withheld but will be disclosed upon publication of this paper.) under an IRB-approved protocol, with removed patient identifiers to ensure privacy protection. The raw data collection consists of tabular metrics ($\mathcal{M}$) and ECG tracings ($\mathcal{T}$), as depicted in the left part of Figure 2. To ensure the clinical relevance, we engaged five independent cardiologists to review the data, resulting in a final dataset of 2,000+ patients. Of these, 5% were used for clinical validation (Section 5), while the remainder were used for fine-tuning (Section 4.2).

Representative Groups. Following FDA’s guideline (Food et al., ), it is crucial to include representative data, not simply to amass large volumes. Our dataset encompasses comprehensive arrhythmia types and ensures balanced representation across age and gender demographics, as detailed in Figure 3-(d).

Incorporating Cardiologist-Level Professionalism. When reviewing the raw data, cardiologists are asked to write professional findings ($\mathcal{F}$) and interpretation ($\mathcal{I}$) in accordance with established clinical guidelines ($\mathcal{G}$). This process facilitates fine-tuning the LLMs, embedding cardiologist-level reasoning, evidence-based statements, and a structured format into the models. To optimize the cardiologists’ time, medical research assistants first draft the $\mathcal{F}$ and $\mathcal{I}$, which are subsequently reviewed and independently adjudicated by the cardiologists. Additionally, each cardiologist randomly audits at least 50% of their peers’ drafts to address issues such as incompleteness, inconsistency, or diagnostic inaccuracies. This peer-review process not only improves the data quality but also ensures standardized findings and interpretation with professional accuracy.

Figure: Figure 3: (a)-(c) illustrate the prompts used for $\theta_{\texttt{M2F}}$ (prompts for $\theta_{\texttt{T2F}}$ and $\theta_{\texttt{F2I}}$ are in Figure 9): (a) represents the instructions (or “system prompt”) used for both fine-tuning and inference; (b) includes the demonstrations used for in-context learning during inference; and (c) shows the input and response structures. During fine-tuning, (c) is filled with cardiologist-adjudicated texts, whereas during inference, (c) retains the format presented above to specify the response format. (d) presents the statistics of our collected patient data, which is further subgrouped by gender, age, and arrhythmia classes – Class I: normal arrhythmias. Class II: clinically significant arrhythmias. Class III: life-threatening arrhythmias. Detailed clinical implications are provided in Appendix C.
Refer to caption: x3.png

### 4.2 Instruction Fine-Tuning

Utilizing cardiologist-adjudicated data, we apply instruction fine-tuning to instill cardiologist-level professionalism into agents $\theta_{\texttt{M2F}}$, $\theta_{\texttt{T2F}}$, and $\theta_{\texttt{F2I}}$. For $\theta_{\texttt{M2F}}$, we selected Llama-3.1-8B as the base model, LLaVA-v1.5-13B for $\theta_{\texttt{T2F}}$, and another Llama-3.1-8B for $\theta_{\texttt{F2I}}$. Each base model is fine-tuned individually, tailored to its specific task using relevant portions of cardiologist-adjudicated data. For instance, as shown in Figure 3-(a)(c), we fine-tune $\theta_{\texttt{M2F}}$ by utilizing system prompts from (a) and cardiologist-adjudicated texts in the format presented in (c), aligning with the metric-to-findings task handled by $\theta_{\texttt{M2F}}$.
Let $\theta_{\texttt{Agent}}$ denote the trainable parameters of any LLM agent, with $X$ and $Y$ representing the instructional input and the corresponding LLM responses for one patient, respectively. Given the cardiologist-adjudicated data $\mathcal{D}$, the tuning process can be formulated as follows:

$$ $\displaystyle\theta^{*}_{\texttt{Agent}}=\arg\min_{\theta_{\texttt{Agent}}} \mathbb{E}_{(X,Y)\in\mathcal{D}}\mathcal{L}(\theta_{\texttt{Agent}}(X),Y)$ (2) $$

The goal of instruction fine-tuning in Eq. 2 is to minimize the mean of the summed loss by $\mathbb{E}(\mathcal{L}(\cdot,\cdot))$ given each pair of $(X,Y)$ within $\mathcal{D}$. Specifically, when $\theta_{\texttt{Agent}}$ is $\theta_{\texttt{M2F}}$, we have $X=(\mathcal{M},\mathcal{B})$ and $Y=\mathcal{F}$. For $\theta_{\texttt{T2F}}$, $X=(\mathcal{T},\mathcal{B})$ and $Y=\mathcal{F}$. Lastly, for $\theta_{\texttt{F2I}}$, $X=(\mathcal{F},\mathcal{G})$ and $Y=\mathcal{I}$.

### 4.3 Inference with In-Context Learning and Fact-Checking

As outlined in Section 3.3, Zodiac’s inference process involves a multi-agent approach using the trained agents $\theta_{\texttt{M2F}}$, $\theta_{\texttt{T2F}}$, and $\theta_{\texttt{F2I}}$. First, $\theta_{\texttt{M2F}}$ processes patient metrics ($\mathcal{M}$) and $\theta_{\texttt{T2F}}$ handles ECG tracings ($\mathcal{T}$), together generating findings ($\mathcal{F}$). These findings are then interpreted by $\theta_{\texttt{F2I}}$ as the diagnostic interpretation ($\mathcal{I}$). Each agent leverages in-context learning to enhance diagnostic accuracy, with fact-checking applied at the final step for backward self-correction.

In-Context Learning. For each fine-tuned LLM agent, we implement in-context learning using a set of demonstrations (or “demos”) containing cardiologist-adjudicated findings and interpretation. The content of each demo is tailored to the specific LLM agent. For instance, demos for $\theta_{\texttt{M2F}}$ include cardiologist-adjudicated findings, as shown in Figure 3-(b). To ensure that each demo is relevant to the target patient’s case, we categorize the patient data by gender, age group, and arrhythmia class, following the subgrouping presented in Figure 3-(d). We then select three demos that match the patient’s gender, age group, and arrhythmia class. During inference, the input prompt is a combination of contents shown in Figure 3-(a)(b)(c).

Fact-Checking. Fact-checking occurs after $\theta_{\texttt{F2I}}$ generates the final interpretation ($\mathcal{I}$). Zodiac applies cardiological guidelines ($\mathcal{G}$) to verify whether the findings ($\mathcal{F}$) correctly lead to the interpretation ($\mathcal{I}$) that aligns with $\mathcal{G}$. Since $\mathcal{F}$ and $\mathcal{I}$ are itemized lists, it is convenient to match each item independently. If discrepancies are identified based on $\mathcal{G}$, Zodiac automatically prompts the corresponding agent to regenerate specific items in $\mathcal{F}$ or $\mathcal{I}$ using instructions derived from $\mathcal{G}$. Due to space constraints, the example of $\mathcal{G}$ and the fact-checking process are provided in Appendix D.

### 4.4 Towards In-Hospital Deployment

Figure: Figure 4: Workflow of Zodiac assisting cardiologists through AWS deployment.
Refer to caption: x4.png

In line with SaMD (FDA, ), Zodiac represents an important milestone in building professional LLMs in today’s standard clinical workflow. It has been deployed on Amazon AWS as the backend and connected to the in-hospital frontend to assist cardiologists by providing preliminary reports. As shown in Figure 4, cardiologists or their assistants can upload patient data, including monitored metrics and ECG tracings from wearable patches (Steinhubl et al., ) or Holters (Kim et al., ). This data is routed through AWS API Gateway, triggering AWS Lambda to invoke SageMaker, where Zodiac is hosted. On AWS SageMaker, Zodiac generates preliminary reports, including findings and interpretation based on the data, and performs fact-checking to ensure accuracy before finalizing the reports. These reports are then returned to the cardiologists, who can use them as a foundation to finalize their diagnoses, thus enhancing workflow efficiency and improving diagnostic accuracy.

## 5 Experimental Results

### 5.1 Clinical Validation Setting

Our experiments are designed to align with real-world clinical validation with the following settings:

Evaluation Metrics: As detailed in Table 1, we consider eight evaluation metrics commonly used among clinical validations (Tierney et al., ; Sallam et al., ). Metrics (a)-(e) evaluate the quality of generated outputs from a clinical perspective, while metrics (f)-(h) address potential security concerns. Each metric is rated on a scale from 1 to 5, reflecting varying degrees of alignment with ideal clinical standards.

Human Validation: Involving human experts in the validation is critical for enhancing the credibility and acceptance of advanced techniques (Tierney et al., ; Sallam et al., ). To this end, we engage cardiologists to evaluate Zodiac using the aforementioned eight metrics. To streamline their assessment process, we developed a structured questionnaire that begins with patient data, followed by generated findings and interpretation, and concludes with rating options (1-5). Notably, we anonymize the names of the LLMs to ensure fairness, preventing cardiologists from assigning biased scores based on their familiarity with or perceived reputation of specific models, particularly Zodiac.

Dataset: Instead of using public benchmarks, we adopt real patient data is to align with practical diagnostics. As described in Section 4.1, we use 5% samples from our data collection to validate Zodiac, reserving the remaining samples for instruction tuning. These samples encompass comprehensive subgroups that align with Figure 3-(d), which we discuss in Section 5.3.

Baselines: We compare three categories of LLMs: (1) Industry-Leading LLMs: GPT-4o, Gemini-Pro, Llama-3.1-405B, and Mixtral-8x22B.
(2) Clinical-Specialist LLMs: BioGPT-Large (Luo et al., ), Meditron-70B (Chen et al., ) derived from Llama-2, and Med42-70B (Christophe et al., ) derived from Llama-3.
(3) Ablations, including a single-agent Zodiac as detailed in Section 5.4.

Fair Comparison with Baselines: For text-based baselines (e.g., Llama-3.1-405B), we employ the same vision-text LLM, LLaVA-v1.5-13B, used in our image-to-text agent. Additionally, the inference-time prompts are identical to those in Figure 3, with one demonstration provided to establish a baseline for basic task understanding.

**Table 1: Evaluation metrics and their respective domains, abbreviations, and descriptions of ideal cases. Each metric is rated on a scale from 1 to 5, where: 1 — Not at all; 2 — Below acceptable; 3 — Acceptable; 4 — Above acceptable; 5 — Excellent.**
| Domain | Evaluation Metrics | Ideal Statements (Findings & Interpretation) are |
| --- | --- | --- |
| Clinics | a) Accuracy (ACC) | statistically correct, aligning with patient’s data. |
| b) Completeness (CPL) | containing complete items to use during diagnostics. |  |
| c) Organization (ORG) | well-structured and easier to locate the clinical evidence. |  |
| d) Comprehensibility (CPH) | are easier to understand without ambiguity. |  |
| e) Succinctness (SCI) | brief, to the point, and without redundancy. |  |
| Security | f) Consistency (CNS) | mutually supported, without contradicts any other part. |
| g) Free from Hallucination (FFH) | only containing information verifiable by the guideline. |  |
| h) Free from Bias (FFB) | not simply derived from characteristics of the patient. |  |

### 5.2 Diagnostic Performance Comparison with Other LLM Products

**Table 2: LLM diagnostic performance across various metrics. Each cell presents “mean ($\pm$std)” among ratings from all cardiologists across all patient data. We use boldface to indicate the best.**
| Model | Clinic-domain Metric | Security-domain Metric |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC | CPL | ORG | CPH | SCI | CNS | FFH | FFB |  |
| GPT-4o | 3.6 (1.0) | 4.2 (1.0) | 4.1 (0.7) | 4.2 (0.9) | 4.0 (1.1) | 3.8 (1.1) | 3.9 (1.0) | 4.3 (1.0) |
| Gemini-Pro | 3.7 (1.1) | 4.1 (1.1) | 3.9 (1.0) | 4.0 (1.1) | 4.0 (1.1) | 3.9 (1.1) | 4.3 (1.0) | 4.2 (1.2) |
| Llama-3.1-405B | 3.8 (1.2) | 4.0 (1.0) | 3.9 (1.0) | 4.2 (1.0) | 4.2 (1.2) | 3.8 (1.0) | 4.0 (1.0) | 4.3 (1.0) |
| Mixtral-8x22B | 3.7 (1.1) | 4.1 (1.1) | 4.0 (1.0) | 4.4 (0.9) | 4.2 (0.9) | 4.0 (1.0) | 4.1 (1.0) | 4.4 (0.8) |
| BioGPT-Large | 2.2 (0.4) | 2.8 (0.6) | 3.2 (0.8) | 3.3 (0.7) | 3.2 (0.6) | 3.0 (0.8) | 2.9 (0.7) | 3.8 (0.6) |
| Meditron-70B | 3.3 (1.1) | 3.3 (1.2) | 3.6 (1.1) | 3.6 (1.3) | 3.8 (1.1) | 3.4 (1.2) | 3.3 (1.3) | 3.4 (1.3) |
| Med42-70B | 3.6 (0.9) | 3.8 (0.9) | 3.6 (0.9) | 3.7 (1.1) | 3.7 (1.1) | 4.0 (0.8) | 3.7 (1.0) | 3.6 (1.1) |
| Zodiac | 4.4 (0.4) | 4.5 (0.7) | 4.7 (0.4) | 4.9 (0.2) | 4.4 (0.6) | 4.5 (0.2) | 4.8 (0.3) | 5.0 (0.0) |

Comprehensive evaluations in Table 2 highlight the remarkable capabilities of Zodiac. With fewer than 30B parameters (as noted in Section 4.2), Zodiac outperforms larger models like Llama-3.1-405B and advanced industrial products such as GPT-4o and Gemini-Pro, particularly in clinical professionalism (e.g., 4.9 CPH) and security assurance (e.g., 5.0 FFB). Additionally, Zodiac exhibits more stable performance, as evidenced by its lower standard deviation (e.g., $\pm$0.0 FFB). This underscores the importance of incorporating elaborated technical strategies (such as instruction tuning and in-context learning) to enhance diagnostic professionalism, rather than relying solely on prompting a generic model.

Interestingly, medical-specialist LLMs performed worse than generic LLMs. While the small scale of BioGPT-Large (1.5B parameters) understandably limits its diagnostic capabilities, a more critical issue is that the data used for fine-tuning models like Meditron-70B appear to be misaligned with real-world clinical practice. Even when aided by in-context learning demos, these specialist LLMs struggle to meet the specific requirements and security demands of clinical tasks.

Case Study. Figure 5 presents an example of a Zodiac-generated interpretation. Compared to other generated results (see Figure 11 to 13), Zodiac produces accurate and concise statements with clear, structured outputs that are easier for cardiologists to follow. In contrast, other LLMs often generate redundant statements (e.g., GPT-4o in Figure 11 and Gemini-Pro in Figure 12), inaccurate diagnoses (e.g., Llama-3.1-405B in Figure 13), and/or disorganized structures (e.g., GPT-4o and Gemini-Pro), making them more challenging for cardiologists to utilize effectively.

### 5.3 Evaluating Diagnostic Consistency via Subgroup Analysis

Figure: Figure 5: An example of interpretation generated by Zodiac.
Refer to caption: x5.png

Figure: Figure 6: (a)-(c) Subgroup analysis across arrhythmia classes, with the depth of cell color representing rating values (1-5). (d) Ablation baselines.
Refer to caption: x6.png

Subgroup analysis is important in clinical validation to evaluate whether a model demonstrates consistent effectiveness across diverse populations (Cook et al., ; Rothwell, ; Sun et al., ). Since our dataset includes various arrhythmia classes, age groups, and genders, we perform evaluations within these subgroups. Figure 6-(a)(b)(c) present results segmented by arrhythmia classes, with additional breakdowns by age and gender shown in Figure 14.

Observe that some industrial products exhibit obviously biased performance across arrhythmias. For example, GPT-4o presents less accuracy (ACC) and completeness (CPL) when diagnosing life-threatening arrhythmias, while Gemini-Pro shows an increase in hallucinations (less FFH) in normal cases, implying imbalances in their pre-trained knowledge. In contrast, Zodiac delivers consistent performance across all arrhythmia groups, underscoring the importance of data-driven professionalism by incorporating diverse and representative patient cases, as shown in Figure 3-(d).

### 5.4 Ablation Study and Diagnostic Properties of Zodiac

Ablation Study. We evaluate how removing key components from Zodiac affects its performance. Figure 6-(d) compares Zodiac with three baselines: without fine-tuning, without in-context learning, and without fact-checking. Notably, fine-tuning has the greatest impact on diagnostic performance across all metrics, followed by in-context learning, demonstrating the importance of using fine-tuning to permanently embed domain expertise into the LLMs’ parameters. Without fine-tuning, in-context learning can still guide the model toward proficiency, but with more limited improvements. We also notice that adding fact-checking improves security-related performance (e.g., FFH, FFB), highlighting the need to integrate clinical guidelines for safe and responsible diagnostics.

Single-Agent Performance. We evaluate a single-agent variant of Zodiac. To ensure an equivalent scale, we use the same vision-based LLM, LLaVA, as used in $\theta_{\texttt{T2F}}$ but with 34B parameters (compared to $\theta_{\texttt{T2F}}$’s 13B), applying the same tuning and inference techniques to perform tasks originally distributed among multiple agents. However, as shown in Figure 7-(b), we observe clear limitations in performance (e.g., ACC, ORG, SCI, and FFH) for the single-agent Zodiac. It is important to note that the multi-agent Zodiac, as described in Section 4.2, comprises fewer parameters in total (30B). This highlights the limited ability of a single LLM to manage various stages of the task, particularly when dealing with different modalities (e.g., table, image, text) and domain-specific expertise. These findings underscore the importance of using collaborative models, especially for complex tasks, as they enhance task-specific proficiency and prevent overloading a single model.

Diagnostic Lability. To evaluate the stability of diagnostic outputs, we check if LLMs produce consistent texts (findings and interpretation) across multiple runs. For each patient sample, we run LLMs (Zodiac or baselines) 10 times. We then convert the generated texts into embeddings using a sentence transformer (Reimers & Gurevych, ) and calculate variance of pairwise cosine similarities. The variance serves as the stability score for each patient. Figure 7-(a) shows the stability scores across all patients, and we observe that Zodiac demonstrates the most stable performance in generating consistent texts over multiple runs. We attribute this stability to the cardiologist-adjudicated texts with consistent structure and content. Through fine-tuning, Zodiac achieves highly stable outputs with minimal variability across different executions. This level of stability is crucial for ensuring reliability in patient care and building trust among healthcare providers.

Figure: Figure 7: Evaluations on (a) diagnostic lability among multiple executions; (b) single-agent Zodiac.
Refer to caption: x7.png

## 6 Future Work

Zodiac serves as our minimum viable product (MVP) toward functional completeness. Building on Zodiac, future work focused on security, trustworthiness, and transparency is crucial for ensuring long-term success in a competitive market.

As emphasized by FDA’s guiding principles (FDA, ), securing the development and deployment of LLMs is as important as achieving functional effectiveness. While our current evaluation addresses security-focused metrics, the next phase will prioritize further development of security measures to enhance trust. This will involve investigating third-party adversarial influences in data, identifying inherent weaknesses in LLMs that could lead to vulnerabilities (e.g., backdoors), proposing defensive strategies to safeguard Zodiac in life-critical diagnostic applications, and promoting transparency to foster human understanding and effective collaboration in human-machine intelligence.

## 7 Conclusion

We present Zodiac, an LLM-powered, multi-agent framework designed for cardiologist-level diagnostics. Zodiac aims to bridge the gap between clinicians and LLMs in the field of cardiology. Leveraging real-world, cardiologist-adjudicated data and techniques including instruction tuning, in-context learning, and fact-checking, Zodiac is enhanced to deliver diagnoses with the expertise of human specialists. Through clinical validation, we demonstrate that Zodiac produces leading performance across patients of different genders, age groups, and arrhythmia classes. In conclusion, Zodiac represent a significant step toward developing clinically viable LLM-based diagnostic tools.

## Appendix A A Real-world Cardiological Report

Figure 8 presents a real-world report on patient data and diagnostics (including findings and interpretation), with identifying information (such as patient name, date of birth, physician name, and company name) anonymized. The report layout is identical to that shown in Figure 2.

Figure: Figure 8: A real-world cardiological report, with identify-related information anonymized.
Refer to caption: x8.png

## Appendix B Additional Prompts

Corresponding to Figure 3-(a)(b)(c), we provide the prompts used for agents $\theta_{\texttt{T2F}}$ and $\theta_{\texttt{F2I}}$ in Figure 9.

Figure: Figure 9: Prompts used for $\theta_{\texttt{T2F}}$ and $\theta_{\texttt{F2I}}$: (a)(d) – instructions or “system prompt”; (b)(e) –demonstrations used during in-context learning; (c)(f) – LLM response template.
Refer to caption: x9.png

## Appendix C Details about Arrhythmia Classes

In this work, we categorize arrhythmias into three subgroups:

Class I — Normal Arrhythmias: Also known as benign or physiological arrhythmias, these irregular heart rhythms can occur in healthy individuals and typically do not lead to serious health issues. They are generally considered harmless and may not require treatment. In our patient data, Class I arrhythmias include Sinus Bradycardia, Sinus Tachycardia, and Sinus Arrhythmia.

Class II — Clinically Significant Arrhythmias: These arrhythmias involve abnormal heart rhythms that can cause symptoms, lead to complications, or require medical intervention. They may disrupt the heart’s ability to pump blood effectively, increasing the risk of serious events such as stroke, heart failure, or sudden cardiac death. In our patient data, Class II arrhythmias include Pause (¡3s), Ventricular Premature Beat (PVC), and Atrial Fibrillation (AF).

Class III — Life-Threatening Arrhythmias: These abnormal heart rhythms can result in severe consequences, such as cardiac arrest, stroke, or sudden cardiac death, requiring immediate medical attention and often emergency intervention. In our patient data, Class III arrhythmias include Ventricular Flutter (VF), Complete Heart Block (Third-Degree AV Block), Atrial Fibrillation (AFib) with Rapid Ventricular Response, Prolonged Pause, Atrial Flutter (AFL), Ventricular Tachycardia (VT), and Supraventricular Tachycardia (SVT).

In our experiments, we use these arrhythmia classes (I, II, III) for subgroup analysis rather than specific arrhythmias to avoid the limitations of small patient sample sizes for individual conditions. Subgroup analysis based on arrhythmia classes provides a comprehensive view of the LLMs’ diagnostic capabilities across different levels of urgency, offering valuable insights for data collection and performance improvement toward more balanced diagnostics.

## Appendix D Fact Checking Using Clinical Guideline

Clinical guidelines are systematically developed statements designed to assist healthcare providers and patients in making decisions about appropriate health care for specific clinical circumstances. These guidelines are based on the best available evidence and aim to standardize care, improve the quality of treatment, and ensure patient safety. For example, a section of clinical guidelines about PR Interval is provided in Figure 10.

Figure: Figure 10: Part of clinical guidelines.
Refer to caption: x10.png

Fact-Checking using Guidelines. We perform fact-checking by enumerating every itemized finding and corresponding interpretation to identify any misalignment with established guidelines. For example, if the PR interval exceeds 200 milliseconds, the interpretation should include a diagnosis of “a prolonged PR interval, which may indicate a first-degree AV block or the potential for a more advanced block”. Failure to include such a diagnosis would signal an inaccurate assessment by Zodiac. In response, we prompt the relevant LLM agents ($\theta_{\texttt{T2F}}$ and $\theta_{\texttt{F2I}}$ in this case) to re-examine the patient data, verify the accuracy of the findings, and update the interpretation accordingly.

## Appendix E Supplementary Experimental Results

Case Study. In line with Figure 5, we present diagnostic outputs from other LLMs using the same patient data, as shown in Figures 11 to 13. Unlike Zodiac (Figure 5), industrial LLMs tend to produce redundant statements regarding key clinical findings and interpretations, making it difficult for cardiologists to quickly identify key information and make revisions. For example, GPT-4o provides the statement “there were no occurrences of atrial fibrillation or atrial flutter,” which could be more succinctly expressed as “AF/AFL: not present”

Moreover, industrial LLMs may present statistical findings without supportive evidence. For instance, Llama-3.1-405B states “VT: not presen” without providing evidence-based interpretation to substantiate the claim, undermining the trustworthiness of the information for cardiologists.

Figure: Figure 11: An example of interpretation generated by GPT-4o. The corresponding patient case is same as Figure 5.
Refer to caption: x11.png

Figure: Figure 12: An example of interpretation generated by Gemini-Pro. The corresponding patient case is same as Figure 5.
Refer to caption: x12.png

Figure: Figure 13: An example of interpretation generated by Llama-3.1-405B. The corresponding patient case is same as Figure 5.
Refer to caption: x13.png

Subgroup Analysis. In alignment with the subgroup analysis presented in Section 5.3, Figure 14 provides additional analysis across different age groups and genders. We observe similar trends to those described in Section 5.3, where baselines such as GPT-4o, Gemini-Pro, and Llama-3.1-405B exhibit bias by offering better diagnostic completeness (CPL) and comprehensibility (CPH) for adults compared to the elderly population. A similar pattern is observed in the female group, indicating biased pre-training in these LLMs. In contrast, Zodiac demonstrates more balanced performance across different subgroups, indicating its equitable handling of responsibilities across diverse populations.

Figure: Figure 14: Additional subgroup analysis regarding (a)-(c) age groups and (d)-(e) genders.
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Throughout this work, ECG images have been used rather than the raw signal; Is there a specific reason why image instead of signal has been used? I am aware that, in some work, using 2D images has been proven to show higher performance than using 1D signal but curious if that was the case in this work as well.

Wu, Y., Yang, F., Liu, Y., Zha, X., & Yuan, S. (2018). A comparison of 1-D and 2-D deep convolutional neural networks in ECG classification. arXiv preprint arXiv:1810.07088.
soundness: 2
strengths: - The method and problem are well-defined, and the paper is well-written. 
- The proposed approach is built on cardiologist-adjudicated text inputs and is evaluated by clinicians, which enhances its clinical rigor. 
- The method incorporates ECG data along with relevant metadata and integrates current clinical guidelines to generate comprehensive reports.
summary: This paper presents ZODIAC, a human-in-the-loop multi-agent framework for electrocardiograms (ECGs) that leverages real-world patient data and cardiologist-adjudicated findings. ZODIAC includes multi-agent collaboration, where the LLM agent extracts findings from either metrics and tracings and extracts interpretation from the combined findings using clinical guidelines. Cardiologists evaluate the generated response with eight different metrics, where ZODIAC shows the best performance with stable outputs evaluated with diagnostic lability. The proposed framework brings commendable clinical rigor to the input data; however, the evaluation lacks sufficient rigor to thoroughly test ZODIAC and its proposed metrics, raising questions about the robustness of its findings. I am inclined to reject the paper in its current form but would be open to revisiting this decision if the concerns and questions are addressed comprehensively during the rebuttal process.
weaknesses: While the proposed method, ZODIAC, addresses a well-defined clinical problem of generating ECG reports and integrates valuable clinical insights through multi-agent collaboration, concerns remain regarding the rigor of its evaluation.

1. Evaluation Metric: The metrics defined in Table 1 are well-defined for qualitative evaluation, but appear subject to evaluator variability, raising concerns about the objectivity and rigor of its scale in Table 2. The following points are regarding the rigor of this evaluation method which would be expected to be improved in the following submission of the paper.

- Subjectivity of the Metric: The degree of subjectivity within the metrics may not be fully addressed. For example, what are the precise criteria for defining hallucination and bias within the FFH metric? Furthermore, the definition of bias is unclear regarding which 'characteristics' of the patient are considered. Does this refer to patient demographics, clinical features, or another criterion?

- Calibration of the Metric: There are also concerns about whether this metric can serve as a reliable, single quantitative standard for evaluating model outputs. A rigorous calibration process, such as Inter-Rater Reliability (IRR), would help substantiate the metric’s consistency and could address potential variability. This is particularly important because different evaluators may interpret the scale differently—for instance, what a score of 3 represents could vary among physicians.

- Details on Inter-Rater Reliability (IRR): The paper leaves IRR and confidence intervals unexplored, which could suggest inconsistencies in clinical outputs. Addressing IRR would help to ensure that ratings are stable and comparable across evaluators. There are standard deviations reported in Table 2, but it is better to separately report inter-rater confidence intervals as well.

2. Ablation Studies on $\theta_{M2F}$ only or $\theta_{T2F}$ only needed to see the effect of Triple agent vs. Double agents and the effect of leveraging metadata and ECG tracings.

3. There are more recent works on multi-agent collaboration or interaction, which could be included in the related works section of the paper. 

[1] Kim, Y., Park, C., Jeong, H., Chan, Y. S., Xu, X., McDuff, D., ... & Park, H. W. (2024). Adaptive Collaboration Strategy for LLMs in Medical Decision Making. arXiv preprint arXiv:2404.15155.

[2] Jin, Q., Wang, Z., Yang, Y., Zhu, Q., Wright, D., Huang, T., … & Lu, Z. (2024). AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learning.

[3] Li, J., Wang, S., Zhang, M., Li, W., Lai, Y., Kang, X., ... & Liu, Y. (2024). Agent hospital: A simulacrum of hospital with evolvable medical agents.

[4] Fan, Z., Tang, J., Chen, W., Wang, S., Wei, Z., Xi, J., ... & Zhou, J. (2024). Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis.

[5] Yan, W., Liu, H., Wu, T., Chen, Q., Wang, W., Chai, H., ... & Zhu, L. (2024). ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See detailed comments above. Briefly, 

- How does ZODIAC’s multi-agent framework compare with existing LLM-based multi-agent approaches?

- How robust and generalizable are the conclusions on the framework’s effectiveness across broader and unseen populations?

- How are the clinical guidelines sourced, selected, and integrated into the framework for fact-checking?

- Does the framework primarily rely on textual information, potentially minimizing the tracing2finding agent’s role? In practical scenarios with limited or low-quality cardiologist-adjudicated text, would the framework maintain comparable diagnostic performance relying on ECG tracings?
soundness: 1
strengths: - The studied problem is important and practical in healthcare applications. 
- The paper includes human validation and discussion on real-world deployment, which provide insights on how recent advances of LLMs can be utilized in clinical settings.
summary: The paper introduces ZODIAC, an LLM framework designed to perform diagnostics in cardiology, focusing on multi-agent collaboration for extracting, analyzing, and interpreting patient ECGs. ZODIAC contains three individual agents for transforming metrics to findings, tracing to findings, and findings to interpretation. Through clinical validation on real-world data, the authors claim ZODIAC outperforms leading LLMs. The studied problem is interesting and practical, but there are major issues as listed below.
weaknesses: - While ZODIAC effectively uses a multi-agent system, the framework is somewhat derivative of existing LLM-based multi-agent approaches. A more comprehensive comparison of this framework against existing multi-agent approaches in healthcare [1,2] is needed to demonstrate the novelty and effectiveness of the proposed framework. 

- The experiments are not convinced. 
  * The paper relies on a small validation set (only 5% of the dataset, ~100 patients). It raises concerns about the robustness of the framework. The size is far from sufficient to draw solid conclusions about its effectiveness across a broader population or diverse clinical scenarios.
  * All experiments are conducted on single-sourced private data of a limited size (2000+ patients). Though the authors aimed to align real clinical applications by using their own data, there are a bunch of publicly available, widely adopted real-world ECG datasets (e.g., PTB-XL [3], MIMIC-IV-ECG [4], CODE [5]) that can be utilized as additional evaluation sets. 
  * There is a lack of out-of-domain evaluation for demonstrating the generalizability of the framework. Both the training data and evaluation data are from the same data source. 

- Clinical guidelines are used for fact-checking to generate final interpretations.  However, there is insufficient information on how these guidelines are sourced or managed. There are no details on the selection criteria for clinical guidelines, their integration within the framework, or the mechanisms employed to handle potential conflicts arising from diverse guidelines.

- While the ZODIAC framework leverages a multi-agent setup to mirror cardiological diagnostic processes, the paper does not fully describe how they address potential redundancies or conflicts in findings across different modalities (e.g., ECG and tabular data). 
  
- While the tracing-to-finding agent is intended to interpret ECG images, it remains unclear how crucial this agent is in identifying visual patterns independently, given the availability of comprehensive cardiologist-adjudicated text and clinical guidelines within the framework. This raises the question of whether the model is truly leveraging visual information from ECGs or primarily relying on textual data. 

[1] Tang, Xiangru, et al. "Medagents: Large language models as collaborators for zero-shot medical reasoning." arXiv preprint arXiv:2311.10537 (2023).

[2] Bani-Harouni, David, Nassir Navab, and Matthias Keicher. "MAGDA: Multi-agent guideline-driven diagnostic assistance." International Workshop on Foundation Models for General Medical AI. Cham: Springer Nature Switzerland, 2024.

[3] Wagner, Patrick, et al. "PTB-XL, a large publicly available electrocardiography dataset." Scientific data 7.1 (2020): 1-15.

[4] Gow, B., et al. "MIMIC-IV-ECG: diagnostic electrocardiogram matched subset (version 1.0)." PhysioNet (2023).

[5] Ribeiro, Antônio H., et al. "Automatic diagnosis of the 12-lead ECG using a deep neural network." Nature communications 11.1 (2020): 1760.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Line 250: *we utilized ECG data sourced from our collaborating healthcare institutions1 under an IRB-approved protocol, with removed patient identifiers to ensure privacy protection.* The in-practice deployment with AWS makes me think about privacy concerns (even when removing patient indicators, patients can often be easily identified from the remaining information). 
- What is the output of the model? From figure 2 it seems that it generates a report. Is this the report from figure 8?
soundness: 2
strengths: - Well written and nicely presented
- Real-world validation of the approach
summary: A well-written work on the creation of an LLM application that can process data from patients, cardiologist reports, and clinical guidelines to perform several tasks.
weaknesses: Text:
- typo in figure 7b signle
- line 375: remove "is": Instead of using public benchmarks, we adopt real patient data is to align with practical diagnostics

General:
- *Representative Groups*: race is not indicated as statistic.
- Human validation: how many people have provided a report? Was the data that Zodiac was pre-trained on from the same institution as the physicians? This might explain why clinical-specialist LLM's performed poorly (overfitted to a certain dataset).
- Line 375: *Dataset: Instead of using public benchmarks, we adopt real patient data is to align with practical diagnostics.* Is there a reason you could not do both? Your own benchmark might have some biases as you could have developed your model to perform especially well on the metrics that you defined.
- I am missing a limitations section in your work.
- While I understand that evaluation of these frameworks is far from trivial and physician evaluation is an important part of the solution, more details need to be provided for these experiments. Moreover, public benchmarks should be added. If current benchmarks are imperfect, as mentioned in line 247, a small public benchmark can be provided for future work to be able to compare to your work.
- I urge you to provide code/configurations for as much of your work as possible. This is important, especially to convince the non-health ML field of your work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In line 190, it is mentioned that "$\mathcal{T}$ presents a concise but representative segment of the 24-hour monitoring". Is it possible that there are multiple such segments, e.g. multiple segments with AV block? If so, I think it would be helpful to discuss how the tracings-to-findings (T2F) agent handles multiple ECG images.
2. For equation (2), is the loss $\mathcal{L}$ next-token prediction? I think it would be helpful to clearly mention, or provide the formula, for the loss to help readers better understand.
3. For the evaluation, is it entirely human evaluation? Is there any AI-based evaluation or off-the-shelf metrics? If no AI-based evaluation is used, it may be challenging to scale up the human evaluation and it would be helpful to discuss how to make the human evaluation more scalable or efficient, perhaps in future work.
4. For baseline methods, only one demonstration is provided, while for ZODIAC, three demonstrations which match the patient's demographics and arrhythmia class are used for in-context learning. I am wondering whether this discrepancy in the number and relevance of few-shot examples could contribute to the performance gap between baseline methods and ZODIAC. If that is possible, it may be helpful to provide the baseline methods with the same demonstration examples to isolate this impact.
5. Will the findings and the interpretations data collected from the cardiologists be made public? I think that would be a great contribution to the research community.
soundness: 3
strengths: 1. This paper is clear and well-written.
2. Using LLMs for clinical decision support is a very important topic, which can help reduce clinician burnout and streamline healthcare administration.
3. The design of the ZODIAC framework is intuitive, extensible and flexible, with a correction mechanism with factchecking under clinical guidelines to ensure the safety of the generations.
4. The ablation study is quite comprehensive and justifies the design choice of each component.
summary: This paper proposes ZODIAC, an LLM-based, multi-agent framework for cardiology diagnostic support. ZODIAC leverages two agents to process clinical metrics (tabular data) and ECG tracings (image data) to generate findings, and a third agent to synthesize the findings into diagnostic interpretations under clinical guidelines. All three agents are instruction fine-tuned on high-quality data collected from expert cardiologists. On eight metrics for clinical validations, ZODIAC outperforms much larger LLMs such as GPT-4o, Gemini-Pro etc. and medical specific LLMs such as BioGPT-Large, Meditron-70B etc., under the judgements of expert cardiologists.
weaknesses: 1. In section 4.1, it is mentioned that data from 2000+ patients from collaborating healthcare institutions are used, and 5% of the data is used for evaluation. Given that the evaluation set is in the same distribution with the data used for fine-tuning the LLM agents, I am wondering whether the framework could generalize to a broader patient population, e.g. patients in other hospitals/medical centers, or patients from very different geographic locations or demographics (e.g. different race or ethnicity)?
2. In section 5.1 and table 2, it seems that the numeric results of the eight metrics including accuracy, completeness etc., are aggregated across the ratings from different cardiologists. Could there be any calibration bias across the cardiologists, e.g., some people are much stricter raters and some are more lenient? If that is the case, I think ranking-based metrics are more suitable than directly averaging the ratings across cardiologists.