RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Problem Statement
As mentioned above, the evaluation of medical necessity is conducted through a meticulous comparison between patient medical records and established coverage guidelines. These medical records are systematically structured , usually in HL7 or FHIR format, within Electronic Health Record (EHR) systems, categorized into discrete sections such as diagnoses, treatment histories, laboratory results, and patient demographics. We define a set of EHR documents as \mathcal{D}=\{d\}_{i=1}^{N_{D}} of size N_{D}

Further, coverage guidelines are formatted in a hierarchical, tree-like structure (referred as checklist in this paper), where each guideline statement ( parent node ) can encompass an arbitrary number of subordinate statements ( sub-checklist or leaf node ) nested within it as shown in Figure 2 . Thus, we define a coverage guideline or checklist as \mathcal{C}=\{c\}_{j=1}^{N_{C}} , where c is a checklist item.

Eventually, the task is to automatically determine the medical necessity y_{p}\in\{-1,0,1\} where -1 means the medical necessity is not justified, 1 means it is justified and 0 means there is a lack of sufficient evidence to justify the criteria.

Recognizing the importance of transparency in the task, we also aim to provide evidence \mathcal{E}_{c}=\{e_{c_{k}}\}_{k=1}^{N_{c}} . These evidences can be used downstream to cross-reference medical documents used to establish medical necessity for the procedure.

We aim to construct a machine learning model \mathcal{M} such that:

\mathcal{M}(\mathcal{D},\mathcal{C})=\{y_{p},\{\mathcal{E}_{c}\}\}\ \forall c\in\mathcal{C} (1)
Methodology
Recently Large Language Models have shown great performance improvements by breaking down complex tasks into simpler sub-problems Khot et al. ( 2022 ) . Motivated by this observation, we propose a two step solution for our problem statement. First we determine the judgement of each of the leaf node checklist items. Subsequently, we propagate the solution for parent nodes bottom-up using Depth First Search.
Leaf-Node Judgement Prediction
Given the vast set of documents with thousands of statements in each EHR document, we propose a Retrieval-Augmented Generation ( RAG-styled ) approach Gao et al. ( 2024 ) Ram et al. ( 2023 ) . This method initially narrows down the document pool to a set of plausible evidences ( top-k evidences ). Subsequently, a Retrieval Agent is employed to acquire the relevant set of evidences for the current checklist item, facilitating targeted and efficient data extraction.
Top-k Evidence Selection
Given the set of documents \mathcal{D} , we decompose each document into its constituent sentences and subsequently aggregate these into a comprehensive set of sentences derived from the EHR data. In order to filter-down sentences that are redundant towards the judgment, we first obtain top k candidate matches for the checklist item c from D . To achieve this, we propose to use a text encoder \mathcal{S} to derive semantic representations for each checklist item c and for each sentence in the EHR data. This method allows us to map both the checklist items and the sentences into a shared semantic space, facilitating more effective matching based on relevance. Subsequently, we employ a semantic similarity metric to calculate the similarity score between each sentence s_{i} and the checklist item c . Based on the similarity metric, we obtain top- k closest matched sentences with the checklist item c . Note that due to cost involved in using LLMs for this task, we keep 1 1 1 In our experiments section, we show how the performance of our approach varies with k k<100 .

\mathcal{S}(\{d_{j}\}|_{j=1}^{N_{D}},c)=\{s_{i}\}|_{i=1}^{k}\ \forall c\in\mathcal{C} (2)
Evidence Retrieval and Prediction
Our proposed Evidence Classification Agent \mathcal{M}_{e} , first looks at each sentence s_{i} in top- k sentences retrieved along with the checklist item c and gives a verdict v_{i} , whether the sentence s_{i} is a supporting evidence, a contradictory evidence or it does not affect the judgment y_{c} . Note that this agent is executed k times since there are k retrieved sentences.

\mathcal{M}_{e}(\{s_{i},c\}|_{i=1}^{k})=\{v_{i}\}|_{i=1}^{k}\ \forall c\in\mathcal{C} (3)

Then our Jury Agent \mathcal{M}_{j} picks up the complete set of sentences s_{i}|_{i=1}^{k} along with their verdicts v_{i}|_{i=1}^{k} and predicts the leaf-level checklist item judgment y_{c} along with evidences \mathcal{E}_{c}\subset s_{i}|_{i=1}^{k} that acted in favour of the judgement y_{c} . We run this jury agent multiple times ( n =10) and take vote of all predictions to determine the final judgement y_{c} and a confidence score f_{c} pertaining to the judgement.

\mathcal{M}_{j}(\{s_{i},v_{i}\}|_{i=1}^{k},\ c)=\{y_{c},f_{c},\{\mathcal{E}_{c}\}\}\ \forall c\in\mathcal{C} (4)
Parent-Node Judgement Prediction
The value of a parent node is contingent upon the values of its nested child nodes. Hence, we determine parent node’s value by aggregating children node’s value using bottom-up judgment propagation approach.
Bottom-Up Judgement Propagation
In order to obtain the decision over the complete checklist \mathcal{C} , we propose to use an iterative bottom-up approach. In another words, we start from the leaf nodes and keep obtaining the judgment of their parent nodes. The iterations are terminated when we obtain the judgment and scores of the root node in the checklist \mathcal{C} .

Mathematically, at every iteration i , we choose a set of leaf node checklist items c_{j}|_{j=1}^{N_{par}} having a common parent checklist item c_{par} , having judgements y_{j}|_{j=1}^{N_{par}} and confidence scores f_{j}|_{j=1}^{N_{par}} . We then calculate the judgement y_{par} and confidence score f_{par} of parent node as:

\mathcal{M}_{p}(c_{par},\{c_{j},y_{j},f_{j}\}|_{j=1}^{N_{par}})=\{y_{par},f_{par}\} (5)

where \mathcal{M}_{p} is our Propagator Agent
Experiments and Results
We have conducted preliminary tests for the Leaf-Node Judgment Prediction task on a limited set of examples, and a broader evaluation on a more extensive dataset is currently underway. Updates to the paper will follow upon completion of the study.

For the Bottom-Up Judgment Propagation task, we have prepared approximately 300 sub-checklists, where each sub-checklist includes a parent node and its associated child nodes. To evaluate our Propagator Agent , we intentionally apply the judgments from the Leaf-Node Judgment Prediction task to each leaf node in the checklist. This method of synthetic assignment is advantageous as it introduces a range of less likely or extraordinary judgment combinations, thereby challenging our Propagator Agent to maximize its robustness.

Our Propagator Agent is an LLM-powered Agent, which takes up a parent node and its corresponding leaf nodes (along with their judgments and confidence scores) to obtain the judgment and confidence score of the parent node. Due to proprietary constraints, the specific prompt utilized for this agent is not disclosed; however, we evaluate the performance of the Propagator Agent across various dimensions. The outcomes of this analysis are presented in Table 1 .
Effect of Prompting Strategy
We performed two sets of experiments. The first involved providing In-Context Learning (ICL) examples Min et al. ( 2022 ) and measuring accuracy, as illustrated in Figure 3 . Larger models such as GPT-4 and Opus yielded strong results, whereas smaller models like Mixtral-8X7B and GPT-3.5 exhibited suboptimal performance when relying solely on ICL prompts. However, in the second experiment, when supplemented with Chain of Thought (CoT) prompting Wei et al. ( 2022 ) , the performance of these smaller models markedly improved, demonstrating how the step-by-step reasoning process aids in decomposing the complex task of propagation into manageable segments.

However, the use of Chain of Thought (CoT) prompting substantially increases response times for larger models due to its generation of an increased number of tokens compared to ICL-only prompting. In contrast, the enhancements in performance observed with GPT-3.5 are achieved without a marked increase in latency, particularly when compared to larger models such as Opus and GPT-4 under similar conditions.
Effect of LLM Choice
We conducted an evaluation of the Propagator Agent utilizing various LLMs, with a particular emphasis on metrics such as accuracy and latency. Opus and GPT-4 emerged as the top performers, achieving approximately 94% accuracy when CoT prompting was combined with ICL examples, as illustrated in Figure 3 .

GPT-3.5 is ranked second in terms of accuracy but presents significant benefits in reduced latency compared to GPT-4 and Opus, as depicted in Figure 4 . Additionally, the operational costs associated with GPT-3.5 are substantially lower. Although selecting the optimal model involves a trade-off, GPT-3.5 stands out as the preferred option when considering a balance among cost, latency, and accuracy. Nonetheless, for scenarios where maximum accuracy is crucial, the larger models such as GPT-4 and Opus are more appropriate.
Conclusion
We have demonstrated that decomposing a large, complex problem into a series of smaller, specialized tasks handled by different agents can help us automate even sophisticated tasks that until now was extremely difficult. This approach also enables us to illustrate the transition from a monolithic AI solution ( \mathcal{M} ) to a more micro-service architecture driven solution ( \mathcal{M}_{r} , \mathcal{M}_{j} and \mathcal{M}_{p} ). At present, our methodology utilizes a constrained workflow; however, there is potential to develop a system incorporating loosely coupled agents. These agents would be more dynamic and agile, enhancing their capacity for problem-solving.

A fully developed implementation of this approach would essentially adopt an organizational-like structure, where the entire architecture is composed of several pods. Each pod contains worker agents, each specialized in different facets of the given problem, along with checker agents that reassess and validates worker agent’s output, prompting a rerun if the outputs appear to be incorrect. Overseeing these pods would be a super-orchestrator agent, analogous to a CEO at the organizational level, who coordinates and directs the activities across the architecture. This architectural design significantly addresses the issue of hallucinations, a common challenge in existing LLMs.

Moreover, given our work’s focus on the healthcare industry, there is a substantial emphasis on the explainability of the outputs generated by these LLM agents. The logs produced through Chain of Thought (CoT) prompting provide valuable insights into the decision-making processes of these agents, addressing, to some extent, the challenges of transparency associated with employing AI models in healthcare workflows.
Figure 1: Leaf-Level Judgement Prediction; Leaf-Level Judgement Prediction
Figure 2: Bottom-Up Judgement Propagation; Bottom-Up Judgement Propagation
Figure 3: Effectiveness of ICL vs. CoT Strategies Across Multiple LLMs; Effectiveness of ICL vs. CoT Strategies Across Multiple LLMs
Figure 4: Latency Analysis of LLMs Under ICL and CoT; Latency Analysis of LLMs Under ICL and CoT
Comparison of model performance