MedCalc: Bridging Medical Calculator and LLM Agent with Nested Tool Calling
Authors: Yakun Zhu, Shaohang Wei, Xu Wang, KUI XUE, Shaoting Zhang, Xiaofan Zhang


Reviewer paper summaries: This paper focuses on the downstream task of medical calculators and introduce MedCalc, which is an agent architecture for tool selected and nested tool calling. The paper also introduces CalcQA, a medical calculator assessment task with 100 samples, and a toolkit of 300 medical tools. Given the complexity of medical calculator tasks, the authors propose to break down the agent functions into specialized modules and nested tool calling.
Summary of strengths: 1. The introduction of MedCalc presents a novel approach to addressing the complexities of medical calculator tasks by using a combination of specialized and modular tools, allowing for dynamic and iterative tool usage.

2. The experimental results demonstrate significant improvements in accuracy and efficiency of tool using.

3. Collaborating with professional physicians to build the CalcQA benchmark and the tools can serve as a useful resource for the community for further research.
Summary of weaknesses: 1. While CalcQA is a valuable benchmark, its relatively small size may limit the generalizability of the findings. The benchmark is also designed for the method proposed in the paper, which might overfit the design decisions made in the framework. A larger dataset would provide more robust validation.

2. Many claims made in the paper are not backed up by experiments. For example, there are many places that emphasize the flexibility and generalizability of the framework, but the experiments are limited to the set of proposed task and tools.

3. How exactly is information transmitted throughout the framework and between each modules and components? The methods description in Section 3 feels a bit high level and vague. For example, Line 246 mentions the “Memory Repository,” what exactly is it? How is it operationalized? The framework technically needs to be reproducible from the paper description, but right now it is not the case.

4. The performance of MedCalc heavily depends on the underlying LLMs, which may vary in capabilities and limitations, potentially affecting the system’s overall effectiveness.
Comments, suggestions and typos: Line 062, 161, 163, 198, 201, 359, 527, etc.: space before parenthesis or next word

Line 111, Line 223-238, subsection titles in section 3, figure 2: please standardize the name/number of modules throughout the paper, currently it is sometimes 3 and sometimes 4 modules and would lead to confusions.

Line 531: Ours —> Our

Figure 2 and Figure 5 are not referred to in the paper. 

Overall, the writing and organization of the paper could benefit from another round(s) of editing. There are a lot of formatting mistakes, long and confusing sentences, and the description of the pipeline needs to be consolidated.
Overall assessment scores: 2.5