Domain: biology
Editorial outcome: major_revision

Abstract
Deriving personalized insights from popular wearable trackers requires complex numerical reasoning that challenges standard LLMs, necessitating tool-based approaches like code generation. Large language model (LLM) agents present a promising yet largely untapped solution for this analysis at scale. We introduce the Personal Health Insights Agent (PHIA), a system leveraging multistep reasoning with code generation and information retrieval to analyze and interpret behavioral health data. To test its capabilities, we create and share two benchmark datasets with over 4000 health insights questions. A 650-hour human expert evaluation shows that PHIA significantly outperforms a strong code generation baseline, achieving 84% accuracy on objective, numerical questions and, for open-ended ones, earning 83% favorable ratings while being twice as likely to achieve the highest quality rating. This work can advance behavioral health by empowering individuals to understand their data, enabling a new era of accessible, personalized, and data-driven wellness for the wider population.

Introduction
Personal health data, often derived from personal devices such as wearables, are distinguished by their multi-dimensional, continuous, and longitudinal measurements that capture granular observations of physiology and behavior in situ rather than in a clinical setting. Research studies have highlighted the significant health impacts of physical activity and sleep patterns, emphasizing the potential for wearable-derived data to reveal personalized health insights and promote positive behavior changes1–6. For example, individuals with a device-measured Physical Activity Energy Expenditure (PAEE) that is 5 kJ/kg/day higher had a 37% lower premature mortality risk2, and a large meta-analysis suggests that activity trackers improve physical activity and promote weight loss, with users taking 1800 extra steps per day7,8.

Prior work has focused on understanding the needs of wearable users and facilitating data exploration through conventional means. Studies deploying on-device apps found that users are interested in questions that analyze trends, compare values, and provide coaching, but that current systems do not adequately address this curiosity9–11. While some research has explored using visualization to help users interpret their data12–15, the advent of Large Language Models (LLMs) presents a new paradigm for interactive analysis. LLMs have demonstrated a growing capacity for complex reasoning and have been applied to tasks across the health domain, including medical question-answering16–19, medical education20,21, electronic health record analysis22–24, mental health interventions25–28, interpretation of medical images and assessments17,29, and generating diagnoses30,31.

However, despite their broad capabilities, applying LLMs to granular personal health data remains a significant challenge. Current LLMs frequently struggle with the numerical reasoning required for time-series analysis, and previous efforts have consequently relied on pre-aggregated, expert-defined statistical summaries rather than enabling direct, nuanced analysis of raw data. For instance, some models deliver fitness and sleep coaching by reasoning over 30-day summaries32 or perform classification tasks on daily metrics presented as text33, but are not designed for open-ended, numerically precise queries on high-resolution data. The development of LLM-based agents, which can be augmented with software tools like code interpreters and information retrieval systems34–38, offers a promising path forward. While agents have shown effectiveness for exploring objective tabular data in domains like electronic medical records23,39–44, they have not yet been adapted to handle the complex, open-ended, and personalized nature of wearable health queries, which often require substantial domain knowledge to translate data into actionable insights.

This gap is significant because even a seemingly straightforward user question, such as “Do I get better sleep after exercising?”, involves a series of complex analytical steps. Answering it properly requires checking data availability, selecting appropriate metrics, summarizing sleep quality on active days, contextualizing these findings within the individual’s broader health profile, and integrating knowledge of population norms to offer tailored recommendations. These steps require a combination of robust numerical analysis and an interpretive understanding of health, a capability that current systems lack. A system capable of fulfilling these steps is able to produce personal health insights, which refer to the outputs generated when an LLM system analyzes a user’s wearable time-series data in response to their health-related queries. This definition underscores that our focus—and the challenges we tackle—are rooted in multimodal wearable data streams.

In this work, we introduce the Personal Health Insights Agent (PHIA), the first open-ended question-answering system powered by an LLM-based agent designed specifically for nuanced reasoning over personal wearable data. PHIA leverages a state-of-the-art agent framework that combines multi-step iterative reasoning, code generation for direct data analysis, and web search integration to autonomously perform complex analyses and generate accurate, context-aware responses to thousands of diverse health queries. We demonstrate the superiority of this agentic approach through a 650-h human evaluation of more than 6000 model responses with 19 human annotators and an automatic evaluation of 16000 model responses, showing significant improvements in open-ended reasoning about wearable health data compared to non-agentic models. To facilitate future research, we also release a high-fidelity synthetic wearable dataset sampled from high-volume, anonymized production data and a personal health insights evaluation dataset, comprised of over 4000 closed and open-ended questions across multiple domains for both automated and human evaluation.

Results
Is a multi-step agentic framework really necessary to derive personal health insights from wearable data, or can simpler methods provide adequate results? To evaluate the necessity of the framework and tools (i.e., code generation, web search), we constructed four language model baselines to demonstrate PHIA’s performance as illustrated in Figs. 1–4. An example of responses from our baselines alongside PHIA can be found in Fig. 5.

PHIA achieves superior numerical correctness on objective queries
In Fig. 1a, we present the evaluation results for objective personal health queries. PHIA achieves an exact match accuracy of 84%, significantly outperforming the Code Generation baseline (74% accuracy), Numerical Reasoning (22% accuracy). We also evaluated a custom chain-of-thought prompting strategy designed for interpreting time-series wearable data with GPT-433 (53.6% accuracy). We observe that the PH-LLM model32 is not able to answer any of our objective queries due to its limitations in handling detailed, long-context tabular data inputs after being fine-tuned exclusively on aggregated coaching case study data. This demonstrates that the agent framework’s complexity and iterative reasoning substantially enhance performance on numerical queries, even those requiring limited abstract reasoning. The text results from our internal model reasoning approaches further emphasize that text-only reasoning is inadequate for precise numerical manipulations on personal health data, likely due to inherent limitations in current LLMs’ mathematical and tabular reasoning capabilities. Consequently, we excluded these methods from the costly human evaluation.

Human evaluation shows PHIA generates higher-quality reasoning and code
Overall, PHIA demonstrates a significant improvement in reasoning over the Code Generation baseline in all but two dimensions (Fig. 1b). Most notably, overall reasoning was substantially higher for PHIA than Code Generation (68 versus 52 in scaled Likert rating). Annotators rated 83% of PHIA’s responses as “Fair” ("3” on the Likert score, Supplementary Tables 1 and 2) or better. In Supplementary Fig. 1, we show that PHIA is also twice as likely to generate “Excellent” responses. Other significant improvements over the baseline include the domain knowledge category (63 vs 38) and logic. To better understand where PHIA’s increased performance comes from, in Fig. 2, we found that queries in general knowledge and compared to the cohort show the largest difference. This performance difference is likely attributable to the PHIA’s ability to query web search for external information and its ability to iteratively and interactively reason its internal parametric knowledge through Thought steps. For example, in Fig. 17, PHIA uses its web search function to supply information about a balanced workout routine. For “Personal Min/Max/Avg.” questions, which are characterized by aggregations well within the capabilities of the Code Generation baseline, the improvement was effectively zero. Examples of low-scoring and high-scoring PHIA outputs are available in the supplemental materials (Supplementary Figs. 2 and 3, respectively).

The two dimensions in which PHIA closely matched the Code Generation baseline are personalization and harm avoidance. For personalization, we believe this is because the Code Generation baseline tended to generate a similar amount of code and numerical insights as PHIA, making the responses comparable. The raters perceived the numerical insights generated through code as a form of personalization. Therefore, since both the Code Generation baseline and PHIA can generate code, their personalization appeared very similar to the raters. This hypothesis is also supported by our analysis of qualitative interviews. However, given the overall benefits in enhancing domain knowledge, we believe PHIA remains a superior model for reasoning about personal health queries. Additionally, we observe that the likelihood of harm avoidance is exceptionally high. The saturated ratings indicate that a combination of underlying model guardrails against harmful responses and the iterative thought process in PHIA effectively prevents harmful questions, with over 99% of responses rated as harmless. Taken as a whole, our evaluation indicates that PHIA’s agent-based method produces substantially higher-quality reasoning than the Code Generation baseline and is much more effective at addressing user-provided queries than its base language model alone. Inter-rater agreement was considerable, with results summarized in Supplementary Table 5. To understand the role of web search specifically, we ablate the feature and study it in Supplementary Fig. 4.

The results from expert evaluation indicate that PHIA improved over the Code Generation baseline in overall code quality, avoiding hallucinations, and personalization (Fig. 1c). Although the difference in performance on other perceived code quality metrics was insignificant, we demonstrate that PHIA is quantitatively less likely to generate code that raises an error. In Fig. 3, we found that the error rate of PHIA is half that of the Code Generation baseline (0.192 vs 0.395). The magnitude of this difference is perhaps particularly surprising considering that both methods use the same base language model. This implies that PHIA’s ability to strategically plan at the first Thought step and perform iterative reasoning about its outputs through the remaining Thought steps minimizes error-prone code generation.

Agentic framework reduces errors and enables recovery
Another notable advantage of using an agent framework in health data analysis is that PHIA can occasionally recover after it throws a fatal error by interpreting its mistake and correcting it in a subsequent step. PHIA recovered in 11.4% of cases (Fig. 4). In comparison, because Code Generation lacks the capacity to react to its own results, its recovery rate is zero. This means that agent-based approaches like PHIA are more stable with respect to fatal code errors. Our results in Fig. 4 show that PHIA is much less likely to make errors on complex tabular reasoning operations such as time-series indexing and joining multiple tables. PHIA is also substantially less likely to hallucinate responses or misinterpret the input data. This indicates that the additional complexity afforded by agents produces substantially more reliable results that can be better trusted by end users.

Qualitative analysis reveals raters prioritize personalization and domain knowledge
To better understand the rating process and provide insight into the nuances of evaluating model responses in health and fitness, we conducted qualitative interviews with two annotators and two experts. Several key themes emerged from these discussions:

All annotators agreed that the presence of numerical insights and metrics made them give higher ratings on personalization—“As long as there are numerical insights, that would be a ‘Yes’ on personalization” [Rater 2]. “I remember another example like how do I lose weight? And it gives a generic answer for getting active... For 150 minutes a week, but it does not reference what the user’s, like, current active minutes are. And I feel like that’s a missed opportunity. It could say if they’re only active for 10 minutes a week. That’s a clear personalization that could help. But it doesn’t really reference that. And so that was like a no.” [Rater 3]. These comments highlight the importance of referring to numerical insights to achieve better personalization.

Raters consistently emphasized the difficulty of accurately assessing model responses without full user context. While numerical data provides some insight, it lacks the rich tapestry of individual lifestyle, habits, and circumstances. As one annotator noted, “Understanding and reading can be challenging at times; you have to read it multiple times for the more subjective questions, but on the more closed ended ones, definitely easier” [Rater 1]. This highlights the inherent limitation of evaluating health advice based solely on quantified data, mirroring real-world scenarios where clinicians rely on a holistic understanding of their patients.

The inclusion of relevant and authorized domain knowledge consistently elevated the perceived quality of model responses. Raters looked for evidence that the model could integrate authoritative health information and go beyond generic advice. “If it did say you’re short on active minutes than the recommended exercise duration, then I would give a ‘Yes’ in domain knowledge” [Rater 4]. This reinforces the importance of grounding health and fitness recommendations in established medical and scientific consensus. Additionally, the annotators also commented that the model’s ability to connect insights to domain knowledge proved a key differentiator. For example, one annotator highlighted, “If the query is, ‘How many hours have I slept?’ and then it referenced some authorized domain knowledge on the recommended sleep duration and compared against to the personal sleep duration, that was a better overall response than just listing out the numerical insights” [Rater 2]. This suggests the importance of going beyond simply presenting data; models must demonstrate understanding of the user’s unique situation and interpret it in the context of relevant domain knowledge in order to tailor responses accordingly.

Raters expressed a heightened awareness of potential harm, particularly regarding medical advice. They favored cautious responses and emphasized the model’s responsibility to defer to healthcare professionals when appropriate. As one annotator explained, “I don’t believe [the model] should have the authority to tell the user diagnosis guidance and information" [Rater 1]. This underscores the ethical considerations inherent in developing AI for health applications, particularly when user safety is paramount. Quantitatively, annotators thought that model responses could cause harm in less than 0.1% of cases (Fig. 1b). Beyond navigating harms, annotators remarked that models would occasionally reference nonexistent data columns or metrics, impacting the overall quality and reliability of their responses.

Methods
Datasets for evaluating personal health insights
Wearable health trackers typically provide generic summaries of personal health behaviors, such as aggregated daily step counts or estimated sleep quality. However, these devices do not facilitate the generation of interactive, personal health insights tailored to individual user needs and interests. In this paper, we introduce three datasets aimed at evaluating how LLMs can reason about and understand personal health insights. The first dataset comprises objective personal health insights queries designed for automatic evaluation. The second dataset consists of open-ended personal health insights queries intended for human evaluation. Finally, we introduce a dataset of high-fidelity synthetic wearable users to reflect the diverse spectrum of real-world wearable device usage.

Objective personal health insights
Definition. Objective personal health queries are characterized by clearly defined responses. For example, the question, “On how many of the last seven days did I exceed 5000 steps?” constitutes a specific, tractable query. The answer to this question can be reliably determined using the individual’s data, and responses can be classified in a binary fashion as correct or incorrect.

Dataset curation. To generate objective personal health queries, we developed a framework aimed at the systematic creation and assessment of such queries and their respective answers. This framework is based on manually crafted templates by two domain experts, designed to incorporate a broad spectrum of variables, encompassing essential analytical functions, data metrics, and temporal definitions.

Consider the following example scenario: a template is established to calculate a daily average for a specified metric over a designated period, represented in code as daily_metrics[$METRIC].during($PERIOD).mean(). From this template, specific queries and their corresponding code implementations can be derived. For instance, if one wishes to determine the average number of daily steps taken in the last week, the query "What was my average daily steps during the last seven days?" and the code daily_metrics["steps"].during("last 7 days").mean() can be used to generate the corresponding response. It is worth noting that during() is a custom function to handle the date interpretation of the temporal span of a natural language query.

A total of 4000 personal health insights queries were generated using this approach. All of these queries were manually evaluated by a domain expert at the intersection of data science and health research to measure their precision and comprehensibility. Examples are available in Table 2.

Open-ended personal health insights
Definition. Open-ended personal health insights queries are inherently ambiguous and can yield multiple correct answers. Consider the question, “How can I improve my fitness?” The interpretation of “improve” and “fitness” could vary widely. One valid response might emphasize enhancing cardiovascular fitness, while another might propose a strength training regimen. Evaluating these complex and exploratory queries poses significant challenges, as it requires a deep knowledge of both data analysis tools and wearable health data.

Dataset curation. A survey was conducted with a sample of the authors’ colleagues, all of whom had relevant expertise in personal and consumer health research and development, to solicit hypothetical inquiries for an AI agent equipped with access to their personal wearable data. Participants were asked, “If you could pose queries to an AI agent that analyzes your smartwatch data, what would you inquire?” Participants were also solicited for “problematic” questions that could lead to harm if answered, such as “How do I starve myself?”

This survey generated approximately 3000 personal health insights queries, which were subsequently manually categorized into one of nine distinct query types (Table 1). For evaluation feasibility reasons, a smaller test dataset was created, comprising 200 randomly selected queries. From this subset, queries with high semantic similarities were excluded, resulting in a final tally of 172 distinct personal health queries. We manually ensured that the sampled subset of queries covered all the query types listed in Table 1. These were intentionally excluded from agent development to avoid potential over-fitting.

Synthetic wearable user data
Definition. To effectively evaluate both objective and open-ended personal health insights queries, high-fidelity wearable user data is essential. To maintain the privacy of wearable device users, we developed a synthetic data generator for wearable data. This generator is based on a large-scale anonymized dataset from 30,000 real wearable users who agreed to contribute their data for research purposes. Each of the synthetic wearable users has two tables – one of daily statistics (e.g., sleep duration, bedtime, and total step count for each day) and another describing discrete activity events (e.g., a 5km run on 2/4/24 at 1:00 PM). The schema of these tables are available in Supplementary Tables 6 and 7. Synthetic data not only ensures the privacy and confidentiality of real-world user data but also facilitates reproducibility and broader accessibility for the research community. Unlike many real-world datasets, our synthetic dataset incorporates detailed and event-based metrics (e.g., sleep score, active zone minutes), enabling more reliable evaluation of personal health insights.

Dataset curation. To build the training dataset for our synthetic data generation framework, we sampled wearable data from 30,000 users who were randomly selected from individuals with heart rate-enabled Google Fitbit and Google Pixel Watch devices. The study underwent review and approval by an independent Institutional Review Board (IRB), with all participants providing informed consent for their deidentified data to be used in research and development of new health and wellness products and services. Eligibility required users to have at least 10 days of data recorded during October 2023, with a profile age between 18 and 80 years old. This threshold was chosen to ensure the dataset captures day-to-day variability in user data while maintaining sufficient inclusion based on prior population distribution analyses. The dataset spans at most 31 days of October 2023, aggregated from daily metrics (e.g., steps, sleep minutes, heart rate variability, activity zone minutes) and exercise events listed in Supplementary Tables 6 and 7.

We used a Conditional Probabilistic Auto-Regressive (CPAR) neural network50,51, specifically designed to manage sequential multivariate and multi-sequence data, while integrating stable contextual attributes (age, weight, and gender). This approach distinguishes between unchanging context (i.e., typically static data such as demographic information) and time-dependent sequences. Initially, a Gaussian Copula model captures correlations within the stable, non-time-varying context. Subsequently, the CPAR framework models the sequential order within each data sequence, effectively incorporating the contextual information. For synthetic data generation, the context model synthesizes new contextual scenarios. CPAR then generates plausible data sequences based on these new contexts, producing synthetic datasets that include novel sequences and contexts. To further enhance the fidelity of the synthetic data, we incorporated patterns of missing data observed in the real-world dataset, ensuring that the synthetic data mirrors the sporadic and varied availability of data often encountered in the usage of wearable devices. A total of 56 synthetic wearable users were generated, from which 4 were randomly selected for evaluation.

The Personal Health Insights Agent (PHIA)
Language models in isolation demonstrate limited abilities to plan future actions and use tools52,53. To support advanced wearable data analysis, as Fig. 7 illustrates, we embed an LLM into a larger agent framework that interprets the LLM’s outputs and helps it to interact with the external world through a set of tools. To the best of our knowledge, PHIA is the first large language model-powered agent specifically designed to transform wearable data into actionable personal health insights by incorporating advanced reasoning capabilities through iterative code generation, web search, and the ReAct framework to address complex health-related queries.

Iterative & interactive reasoning. PHIA is based on the widely recognized ReAct agent framework34, where an “agent" refers to a system capable of performing actions autonomously and incorporating observations about these actions into decisions (Fig. 7d). In ReAct, a language model cycles through three sequential stages upon receiving a query. The initial stage, Thought, involves the model integrating its current context and prior outputs to formulate a plan to address the query. Next, in the Act stage, the language model implements its strategy by dispatching commands to one of its auxiliary tools. These tools, operating under the control of the LLM, provide feedback to the agent’s state by executing specific tasks. In PHIA, tools include a Python data analysis runtime and a Google Search API for expanding the agent’s health domain knowledge, both elaborated upon in subsequent sections. The final Observe stage incorporates the outputs from these tools back into the model’s context, enriching its response capability. For instance, PHIA integrates data analysis results or relevant web pages sourced through web search in this phase.

Wearable data analysis with code generation. During an Act stage, the agent engages with wearable tabular data through Python within a customized sandbox runtime environment. This interaction leverages the Pandas Python library, a popular tool for code-based data analysis. In contrast to using LLMs directly for numerical reasoning, the numerical results derived from code generation are factual and reliably maintain arithmetic precision. Moreover, this approach can help reduce the risk of leaking users’ raw data, as the language model only ever encounters the analysis outcome, which is generally aggregated information or trends.

Integration of additional health knowledge. PHIA enhances its reasoning processes by integrating a web search-based mechanism to retrieve the latest and relevant health information from reliable sources54,55. This custom search capability extracts and interprets content from top search results from reputable domains. This approach is doubly beneficial: it can directly attribute information to web sources, bolstering credibility, and it provides the most up-to-date data available, thereby addressing the inherent limitations of the language model’s training on historical data.

Mastering tool use. A popular technique for augmenting the performance of agents and language models is few-shot prompting56. This approach entails providing the model with a set of high-quality examples to guide it on the desired task without expensive fine-tuning. To determine representative examples, we computed a sentence-T5 embedding57 for all queries in our dataset. Next, we applied K-means clustering on these embeddings, targeting 20 clusters. We then selected queries closest to the centroid of each cluster as representatives. For each chosen query, we crafted a ReAct trajectory (Thought->Action->Observation) that demonstrates how to produce a high-quality response with iterative planning, code generation, and web search. Refer to the responses of PHIA Few-Shot in Supplementary Figs. 5–8 for more examples.

Choice of language model. For all of the following experiments, we fix Gemini 1.0 Ultra58 as the underlying language model. Our goal is not to study which language model is best at our task. Rather, we explore the effectiveness of agent frameworks and tool use to answer subjective, open-ended queries pertaining to wearable data.

Experimental design and evaluation
Baselines
Numerical reasoning. Since language models have modest mathematical ability52,59 it may be the case that PHIA’s code interpreter is not necessary to answer personal health queries. In the first baseline, the user’s data is structured in the popular Markdown table format and directly supplied to the language model as text, coupled with the corresponding query. Markdown has previously been shown to be one of the most effective formats for LLM-aided tabular data processing60.

Analogous to PHIA, we designed a set of few-shot examples to guide the model to execute rudimentary operations such as calculating the average of a data column in the last 30 days, as shown in Supplementary Fig. 9.

Code Generation. Is it necessary to use an agent to iteratively and interactively reason about personal health insights? To investigate this question, for our second baseline, we introduce a Code Generation model that can only generate answers in a single step. In contrast to PHIA, this approach lacks a reflective Thought step, which renders it unable to strategize and plan multiple steps ahead, as well as incapable of iterative analysis of wearable data. Moreover, this approach cannot augment its personal health domain knowledge as it does not have access to web search. This baseline builds on prior work in code and SQL generation for data science, where language models generate code in response to natural language queries61–64. To make a fair comparison, this baseline was fortified with a unique set of few-shot examples that employ identical queries to those used in PHIA, albeit with responses and code crafted by humans to mirror the restricted capabilities of the Code Generation model (i.e., no additional tool use and iterative reasoning). Examples are available in Supplementary Figs. 5–8.

Additional LLM-based wearable systems. We also compare PHIA against recent LLM-based methods, including the Personal Health Large Language Model (PH-LLM)32. It is a fine-tuned LLM based on Gemini Ultra 1.0, focused on delivering coaching for fitness and sleep based on aggregated 30-day wearable data (e.g., 95th percentile of sleep duration) instead of the high-resolution daily data. Rather than invoking tools, PH-LLM uses in-model reasoning to generate long-form insights and recommendations. Moreover, PH-LLM is fine-tuned specifically for providing coaching recommendations only, instead of providing numerical insights and recommendations for general wearable-based queries. Additionally, we compare our approach to a specialized chain-of-thought prompting strategy designed for interpreting time-series wearable data with the GPT-4 model33. This approach instructs the model to reason directly within its textual context window without external computational tools. Overall, unlike PHIA, these methods focus on internal model reasoning and do not incorporate an iterative agentic framework and external tools. In addition, this approach is based on GPT-4 instead of Gemini, enabling us to demonstrate that our proposed approach outperforms even strong baselines that leverage alternative language models.

Experiments. With the aforementioned baselines in mind, we conducted the following experiments to examine PHIA’s capabilities.

Automatically evaluating numerical correctness with objective queries. Some personal health queries have objective solutions that afford automatic evaluation (see “Methods” section for details). To study PHIA’s performance on these questions, we evaluated PHIA and the baselines on all 4000 queries in our objective personal health insights dataset. A query was considered correctly answered if the model’s final response was correct to within two digits of precision (e.g., given a ground truth answer of 2.54, a response of 2.541 would be considered correct, and the response 2.53 would be considered incorrect). We compared PHIA against numerical reasoning, code generation, and two LLM-based wearable systems (PH-LLM and custom-prompted GPT-4).

Evaluating open-ended insights reasoning with human raters. Open-ended personal health queries demand precise interpretation to integrate user-specific data with expert knowledge. To assess open-ended reasoning capability, we recruited a team of twelve independent annotators who had substantial familiarity with wearable data in the domains of sleep and fitness. They were tasked to evaluate the quality of reasoning of PHIA and our Code Generation baseline in the open-ended query dataset defined in Table 1. Due to annotators’ minimal experience with Python data analysis, two domain experts developed a translation pipeline with Gemini Ultra to translate Python code into explanatory English language text (examples available in Supplementary Figs. 10–14). Annotators were also provided with the final model responses.

Annotators were tasked with assessing whether each model response demonstrated the following attributes: relevance of data utilized, accuracy in interpreting the question, personalization, incorporation of domain knowledge, correctness of the logic, absence of harmful content, and clarity of communication. Additionally, they rated the overall reasoning of each response using a Likert scale ranging from one ("Poor”) to five ("Excellent”). All responses were distributed so that each was rated at least three unique annotators, who were blinded to the method used to generate the response. Rubrics and annotation instructions can be found in Supplementary Table 1. To standardize comparisons across different metrics, final scores were obtained by mapping the original ratings on a scale of 1–5 into a range of 0–100. Subsequent scores for “Yes or No” questions are the proportion of annotators who responded “Yes”. For example, an answer of “Yes” for domain knowledge would indicate that the annotator found the response to show an understanding of domain knowledge. In total, more than 5500 model responses and 600 h of annotator time were used in this evaluation. Additional reasoning evaluation with real-user data can be found in Supplementary Fig. 15.

Evaluating code quality with domain experts. To assess the quality of the code outputs of PHIA and our Code Generation baseline, we recruited a team of seven data scientists with graduate degrees, extensive professional experience in analyzing wearable data, and publications in this field. Collectively, these experts brought several decades of relevant experience (mean = 9 years) to the task. We distributed the model responses from PHIA and the Code Generation baseline such that each sample was independently evaluated by three different annotators. Experts were blinded to the experimental condition (i.e., whether the response was generated by PHIA or Code Generation baseline). Unlike in the reasoning evaluation, annotators were provided with the raw and complete model response from each method, including generated Python code, Thought steps, and any error messages. Experts were asked to determine whether each response exhibited the following favorable characteristics: avoiding hallucination, selecting the correct data columns, indexing the correct timeframe, correctly interpreting the user’s question, and personalization. Finally, annotators were instructed to rate the overall quality of each response using a Likert scale ranging from one to five (instruction details in Supplementary Table 2). To facilitate comparison, these ratings were again converted into 0–100 scores. In total, 595 model responses collected over 50 h were used in this evaluation.

Conducting comprehensive errors analysis. Additionally, we conducted a quantitative measurement of code quality by calculating how often a method fails to generate valid code while answering a personal health insights query. Toward this, we determined each method’s “Error Rate"-the number of responses that contain code that raises an error divided by the total number of responses that used code (e.g., indexing columns that don’t exist, importing inaccessible libraries, or syntax mistakes). To better understand the sources of errors, two experts independently performed an open coding evaluation on all the responses in the open-ended dataset. They were instructed to look for errors, including hallucinations, Python code errors, and misinterpretations of the user’s query. Results were aggregated into one of the following semantic categories: Hallucination, General Code Errors, Misinterpretation of Data, Pandas Operations, and Other.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Overall, this paper aims to use LLMs to interpret wearables data through the lens of human-generated queries to gain health insights. This paper is very innovative in its approach to interpreting wearables data and contributes a new way of thinking about wearables data analysis. From this perspective, the paper adds substantial value to the literature.\nHowever, many of the most interesting parts of the story are not included in the paper, including discussing the absolute performance of the PHIA model. The details on the methods require significant further elucidation which can be mostly addressed through the questions below. Unfortunately, one fundamental flaw in the paper appears to be the use of the synthetic wearables data, which in and of itself would require a paper to characterize and substantiate it as a suitable dataset for this task. Because there is a large amount of wearable device publicly available, it is unclear why the authors instead chose to go a route that makes their work less relevant and less reproducible.\nMajor Comments:\n• The overall framing of the paper uses general terms like “health insights” when the outputs are in reality very specific to health insights from wearables data. This should be clarified and corrected throughout the paper.\n• Claims made in the paper are often not substantiated by the data, and those claims need to be scaled back or changed. Specific examples include the first bullet point on page 2 and the first section header under 4.3 (where the text does not follow the header)\n• A disclaimer that the recommendations here should be for people who are generally healthy and that some health recommendations would not be appropriate for people with underlying conditions is key. For example, a recommendation to decrease resting heart rate (as in Fig 1 E) in a person with end-stage heart failure would be very problematic.\n• Absolute evaluations should be the more central part of the results instead of the comparisons to the relatively arbitrary baseline models (for example, describing the dark blue values in Fig 3B). The chosen baseline models (numerical reasoning and code generation) should also be further substantiated as to why these are the best comparators and more prior work on these should be covered. For example, wouldn’t it be obvious that an agent model that can update queries based on intermediate information would make its resultant queries better? Also, the details of those models (like their underlying makeup) should be explained, or if these exist elsewhere in the literature, should be referenced. It wasn’t until near the end of the paper that Gemini Ultra 1.0 is mentioned as the underlying model, and this is mentioned in passing. The limitations of the choice of these specific baseline models as opposed to other possible baseline models should be included in the limitations section.\n• Why was synthetic wearables data used as opposed to real wearable data? There is now substantial publicly available wearables data. How was the synthetic wearables data vetted? What were the 30k wearables and who were the wearers that the data was generated from? What was the duration of time, variables, and characteristics of that original dataset? Why was the model described in “dataset curation” the one chosen for this task? What were its performance characteristics and how was it vetted? Answers to these questions probably requires a full paper in and of itself.\n• How were the web-based recommendations authenticated? What was used to determine whether a source was reliable and reputable? When they were mentioned in the answers to queries, were they referenced and were those references checked? Otherwise how do we know these weren’t hallucinations?\n• Both query and wearable data needs to be available upon publication through a DOI-granting platform\nMinor Comments:\n• Details on the human annotators and data assessors are lacking– it should be made clear how the different subsets of annotators were recruited, what their motivations were for participating (compensation, authorship, etc?), and what their demographics were. For example, it is unclear who “the authors’ colleagues” are. Are they members of particular departments or professional societies? How as their expertise determined? For the question generation task, why wouldn’t every day people, rather than the authors’ colleagues, be selected to generate the questions? Also, were the contributions to the tasks split equally among the members or did some contribute more than others (possibly creating imbalanced representation in the data).\n• Section 2.1 – it would be more accurate to say “quantitative” rather than “objective”.\n• The classification of “could lead to harm” is very subjective and is dependent on what queries were actually generated by the contributors in the first place. This, as well as subjective tasks in the methods (like going from the 3k to 172 queries; from 30k real wearables datasets to 4 synthetic ones chosen for queries) and subjective choices of thresholds (like the wearables data choice of 31 days with 10 days of data availability; 20 clusters for sorting queries) should be included as limitations.\n• For the open-ended questions, it is important to note that the science may not be there in some cases which means that even human experts answering questions would not be able to answer some questions (for example, the strength vs aerobic question is still an open one).\n• Data visualization should be improved, for example, Figure 3A: were there multiple measures of accuracy per model type [parentheses for example, using bootstrapping, or cross validation techniques)? If yes, error, and or confidence intervals should be included, but if not, a bar chart is not the most effective way to convey this information. In Figure 4, the colors do not convey any meaning and should be removed. Figure 1 should include the “thought act and observe” stages of the ReAct framework described in the text.\n• Please clarify whether sections two and three are background information\n• A revised version with line numbers would make it easier for the reviewers to provide their feedback.\n• What are the advantages and disadvantages to doing this in Pandas vs SQL?\n• Figure 7 second column- the question does not make sense.\n(Remarks on code availability)\nCode has not been made available unfortunately, so it was not possible for it to be reviewed. Overall the work is not reproducible whatsoever in its current form.", "reviewer_2": "The paper titled \"Transforming Wearable Data into Health Insights Using Large Language Model Agents\" presents an LLM-based approach to analyzing personal health data derived from wearable devices through the use of a specialized agent called the Personal Health Insights Agent (PHIA). This agent leverages large language models (LLMs) integrated with advanced tools such as code generation and web search to provide personalized, actionable health insights. The study highlights the development and evaluation of PHIA, demonstrating its ability to accurately address both factual and open-ended health-related queries. The authors curated extensive datasets for testing and used rigorous evaluation methods, showing that PHIA can effectively interpret complex health data and offer valuable recommendations, marking a significant step towards more accessible and personalized health management.\nThe paper highlights several strengths of the proposed Personal Health Insights Agent (PHIA):\n1) Integration of Large Language Models (LLMs): PHIA utilizes advanced LLMs, which enhance its ability to effectively interpret complex health data and respond to diverse health-related queries accurately.\n2) Personalization: The agent provides tailored health insights based on individual wearable data, making the recommendations more relevant and actionable for users.\n3) Comprehensive Data Analysis: PHIA can handle both factual and open-ended queries, showcasing its versatility in analyzing various types of health information.\n4) User Accessibility: By translating wearable data into understandable and useful health insights, PHIA makes health management more accessible to users, potentially leading to better health outcomes.\nThe paper also acknowledges certain weaknesses and limitations of the proposed PHIA:\n1) Dependence on Data Quality: The accuracy and usefulness of PHIA's insights heavily depend on the quality and completeness of the wearable data it analyzes. Inconsistent or incomplete data could lead to less reliable recommendations.\n2) Limited Scope of Queries: Although PHIA can address a wide range of queries, there might be limitations in its ability to answer highly specialized or complex medical questions that require expert knowledge beyond the scope of wearable data.\n3) Potential Over-reliance on Technology: There is a risk that users might over-rely on the agent’s insights without consulting healthcare professionals, which could lead to misguided health decisions.\n4) Lack of Innovation in Approach: The integration of large language models (LLMs) with wearable health data, while effective, may not represent a groundbreaking innovation. Similar approaches are already being explored in various domains, which could limit the novelty and distinctiveness of PHIA's contribution.\n5) Insufficient Focus on Field-Specific Challenges: The paper does not provide in-depth consideration of the unique challenges and intricacies of the healthcare and wearable technology fields.", "reviewer_3": "This paper presents a study on using LLMs to answer open-ended analysis questions based on data collected from wearable devices, showing reasonable performances across quantitative and human verifications. Some strengths and weaknesses are summarized below.\nS1: The problem of open-ended question answering by analyzing wearable device data is important and challenging.\nS2: The use of LLM-based code generation, information retrieval and QA is reasonable and promising.\nS3: Comprehensive evaluations are down on several curated datasets with human verifications.\nW1: The datasets being used are pretty small and the way of curating data from scratch is not scalable.\nW2: The technical novelty is limited. For example, LLMs are not very good at reasoning about numbers and doing numerical calculations, but I don't find particular technical designs to address such limited capabilities, nor analysis about it.\nW3: Lack of baselines-- there is basically only the proposed PHIA method evaluated against itself, without real comparisons with any potential baselines at scale, and not even with different LLMs/tools.\n(Remarks on code availability)\nI didn't find the code or link to code."}, "author_response": "Point by Point Response of Nature Communications\nManuscript\nNCOMMS-24-41183-T\nGeneral Response ............................................................................................................... 1\nResponse to the Editor ....................................................................................................... 1\nResponse to Reviewer #1 ................................................................................................... 1\nResponse to Reviewer #2 ................................................................................................... 2\nResponse to Reviewer #3 ................................................................................................... 2\nGeneral Response\nDear Reviewers,\nWe deeply appreciate your thoughtful feedback and constructive suggestions. Their input has\nbeen invaluable in further strengthening the manuscript, and we are pleased to present a\nrevised version that addresses all concerns raised. Changes to the manuscript are marked\nwith blue text.\nWe acknowledge the specific concerns raised about synthetic data, baseline comparisons,\nframing, and the clarity of our analyses. In response, we have made the following key revisions:\n1. Synthetic Data Concerns (Reviewers 1 & 2):\n Conducted additional human reasoning experiments (totaling 480 hours of\nannotation) using real-world wearable data, validating our approach and\ndemonstrating that our results are robust across synthetic and real data\nsources (Supplement I).\n Clarified that multimodal wearable datasets that include discrete events such as\nactivities and demonstrate the capabilities of PHIA are not publicly available to the\nbest of our knowledge. This highlights the critical role of synthetic data, which\nenables reproducibility and future work that builds on our results (Section 2.3).\n Added detailed descriptions of the synthetic data generation process. (Section\n2.3)\n2. Improving the Interpretability of Absolute Performance Metrics (Reviewer 1):\n Included examples of model responses, showcasing both successes and errors\nto provide a balanced view of PHIA’s capabilities. (Supplements B, C, & D)\n Expanded on safety evaluations, defining harmful responses, and detailing the\nimplemented guardrails. We have added several new examples to Supplement C\nwhere PHIA refuses to answer harmful queries. (Supplement C)\n Added examples demonstrating meaningful contexts in which PHIA is\nused. (Supplement B)\n3. Clarifying Model Comparisons and Baselines (Reviewers 1 & 3):\n Substantiated the choice of baseline models (Numerical Reasoning and Code\nGeneration) and explained their relevance to our study with supporting citations.\n(Section 4.1)\n Included more details on the baseline models, such as Gemini Ultra 1.0, and their\nlimitations, referencing prior work to contextualize the comparisons. (Section 3 -\nChoice of Language Model)\n4. Framing and Language Adjustments (Reviewers 1, 2, & 3):\n Clarified that our insights are derived from wearable data and adjusted\ngeneral terms like \"health insights\" accordingly. (Section 1 Introduction)\n Added a disclaimer that the recommendations are intended for generally\nhealthy individuals and may not apply to those with specific conditions (e.g.,\nheart failure). (Section 7 - Focus on and Limitation of Wearable Data)\n Clarified the novelty of PHIA as the first large language model-powered agent\nspecifically designed to transform wearable data into actionable health insights in\nthe point-by-point response. (Section 3)\n5. Data Transparency and Availability (Reviewers 2 & 3):\n Confirmed that the wearable and query datasets will be made publicly available\nupon publication, facilitating reproducibility and follow-up studies.\n Added a link to showcase the data at the end of Introduction.\n6. Improving Data Visualization (Reviewer 1):\n Updated caption of Figure 3(A) to clarify the confidence intervals representation.\n Simplified Figure 4 by removing unnecessary colors.\n Revised Figure 1 to reflect the \"Thought, Act, and Observe\" stages of the ReAct\nframework.\n7. Detailed Methodology and Limitations (Reviewer 1):\n Expanded descriptions of human annotator recruitment, demographics, and roles\nin data evaluation. (Section F.5)\n Added more discussion on the subjective thresholds and decisions made in\ncurating queries and wearable datasets, incorporating this into the limitations\nsection. (Section 2.2, 2.3 and Section 7 Limitations - Subjective Thresholds on\nCuration)\n8. Code Availability (Reviewer 3):\n Added a link at the end of Introduction to include all the code and dataset for\nreview purposes; the code will be made publicly available at publication.\nWe have revised our manuscript's structure to improve clarity and accessibility, ensuring our\nresults and contributions are compelling and actionable. We hope these revisions address\nthe reviewers’ concerns and strengthen the paper’s impact.\nPlease find the detailed responses to each reviewer’s comments in the attached document.\nAdditionally, we have included two versions of the manuscript: one with tracked changes and a\nclean version for review.\nWe are enthusiastic about the opportunity to contribute to the field of wearable health AI and are\nhopeful that the revised manuscript meets the reviewers’ expectations. We look forward to your\nfeedback.\nSincerely,\nMike Merrill, Tim Althoff, Daniel McDuff, Xin Liu\nOn behalf of all authors\nResponse to Reviewer #1\nComment 1.1: The overall framing of the paper uses general terms like “health insights”\nwhen the outputs are in reality very specific to health insights from wearables data. This\nshould be clarified and corrected throughout the paper.\nResponse: We agree that we were not clear enough about this limitation in the first draft of our\npaper. To address your concern we added a new paragraph to the limitations section (Section\n7) which emphasizes the scope of the paper. In this paragraph we state that we only address a\nsubset of all possible health-related queries, and that conditions which are not measurable with\nwearable devices are not supported by PHIA. Nonetheless, we believe that future research\nbuilding on PHIA could integrate with medical records or other IoT devices like connected scales\nand blood sugar monitors. We have also ensured that the title and first usage of “PHIA” in the\nabstract clearly identifies the wearable data scope.\nComment 1.2: Claims made in the paper are often not substantiated by the data, and those\nclaims need to be scaled back or changed. Specific examples include the first bullet point on\npage 2 and the first section header under 4.3 (where the text does not follow the header)\nResponse: We appreciate the reviewer’s attention to the substantiation of claims and have\ncarefully reviewed the concerns raised. Regarding the first bullet point on page 2, which states\nthat we “introduce the first LLM-based health agent framework that utilizes multi-step iterative\nreasoning, code generation, and web search tools for in-depth analysis of thousands of health\ninsight questions,” we believe this claim is substantiated by the evidence presented in the\nmanuscript. Section 1 outlines the novelty of integrating these three components into a unified\nframework, and Sections 4.1 and 4.3 provide quantitative and qualitative evaluations\ndemonstrating PHIA’s superior performance compared to other methods. To further clarify the\ndistinctiveness of PHIA, Section 6.2 provides relevant work on agents for health and time-series\ndata, highlighting the clear differences between prior approaches and PHIA’s unique\ncapabilities. To enhance clarity, we have revised this statement to emphasize the novelty in\napplying these techniques to health insight generation and highlight PHIA’s unique combination\nof tools in the context of wearable health data.\nFor the first section header under Section 4.3,\nwe have updated the subtitle to “PHIA Demonstrates Precision in Answering Objective Health\nQueries” and have expanded the discussion to explicitly connect the results to PHIA’s\ndemonstrated precision on objective health queries.\nIf any potential concerns remained, we would be happy to work with the reviewer to address\nthem.\nComment 1.3: A disclaimer that the recommendations here should be for people who are\ngenerally healthy and that some health recommendations would not be appropriate for people\nwith underlying conditions is key. For example, a recommendation to decrease resting heart\nrate (as in Fig 1 E) in a person with end-stage heart failure would be very problematic.\nResponse: Thank you for raising this concern - we agree that it would be harmful for a reader\nof our paper to draw the conclusion that PHIA could provide medical advice for a patient with a\nserious condition. To address your comment we have added such a disclaimer to the limitations\nsection (Section 7).\nComment 1.4: Absolute evaluations should be the more central part of the results instead of\nthe comparisons to the relatively arbitrary baseline models (for example, describing the dark\nblue values in Fig 3B). The chosen baseline models (numerical reasoning and code generation)\nshould also be further substantiated as to why these are the best comparators and more prior\nwork on these should be covered.\nResponse: We appreciate the reviewer’s feedback regarding the importance of absolute\nevaluations in the results. To address this concern, we include detailed Likert score\ndistributions (Figure A.2) for the categories presented in Figures 3(B) and 3(C). Annotators find\nthatPHIA’s responses are of overall higher quality than the Code Generation baseline. In\nparticular,PHIAis twice as likely to generate “Excellent” responses. By doing so, we aim to\nprovide a clearer representation of the absolute quality of the model responses, complementing\nthe comparative results already shown.\nAdditionally, we provide examples of model outputs alongside these scores to demonstrate the\nscale of the model’s performance in a more tangible way (Figures C.1 and C.2). These\nexamples offer a clearer understanding of how the responses vary in quality and further support\nthe absolute evaluations of our model.\nLikert Score Distribution from Human Annotations\nComment 1.5: The details of those models (like their underlying makeup) should be explained,\nor if these exist elsewhere in the literature, should be referenced. It wasn’t until near the end of\nthe paper that Gemini Ultra 1.0 is mentioned as the underlying model, and this is mentioned in\npassing. The limitations of the choice of these specific baseline models as opposed to other\npossible baseline models should be included in the limitations section.\nResponse: Thanks for your comment. We added a paragraph (“Choice of Language Model”) in\nSection 3 to describe that PHIA is built on top of Gemini 1.0 Ultra. We have added a new\nparagraph, (“Generalization to Other Language Models”) in Section 7 to emphasize that our\nexperiments were conducted with specific models to isolate and study the benefits of agent\nframeworks and tool usage.\nComment 1.6: Why was synthetic wearables data used as opposed to real wearable data?\nThere is now substantial publicly available wearables data. How was the synthetic wearables\ndata vetted? What were the 30k wearables and who were the wearers that the data was\ngenerated from? What was the duration of time, variables, and characteristics of that original\ndataset? Why was the model described in “dataset curation” the one chosen for this task? What\nwere its performance characteristics and how was it vetted? Answers to these questions\nprobably requires a full paper in and of itself.\nResponse: Thanks for raising concerns about synthetic data usage.\nThe reason we used synthetic data is three fold:\n⚫ Reproducibility and Open Access: The use of synthetic data eliminates privacy\nconcerns inherent in real-world wearable datasets, allowing us to publicly share the\ndataset and ensure full reproducibility of our results. This commitment to open science\nenhances the broader impact and accessibility of our work.\n⚫ Data Quality and Feature Completeness: While publicly available wearable datasets\nexist, many lack detailed metrics critical for deriving nuanced health insights, such as\nactivity events, sleep scores, or active zone minutes. These limitations often stem\nfrom older-generation devices or inconsistencies in data resolution and sampling\nprotocols. Our synthetic dataset was specifically designed to address these gaps,\nensuring comprehensive coverage of health metrics relevant to our analyses.\n⚫ Minimal Differences Between Synthetic and Real Data for Deriving Health Insights: When\nusing LLM agents and tools to derive numerical health insights, the quality of data\n(synthetic vs. real) has minimal impact on the outcomes. This is because the analytical\nprocesses primarily rely on structured numerical data and statistical properties, which are\nfaithfully preserved in the synthetic datasets. Consequently, synthetic data effectively\nserves as a proxy for real-world data in achieving the objectives of our study.\nTo substantiate this claim, we conducted an supplementary 480-hour study involving\nreal-world users and observed similar conclusions in reasoning quality. Importantly,\nwhen employing multi-step iterative reasoning and planning, code generation, and web\nsearch tools, PHIA significantly outperformed baseline models in reasoning accuracy,\nrelevance, interpretation, and personalization. We include the results of this evaluation\nbelow and in Supplement I. Notably, PHIA had the same high performance on real,\nnon-synthetic data while the Code Generation baseline performed significantly worse.\nThis discrepancy is likely due to additional complexity (e.g., missingness, noise, or\nvariability) in real-world datasets, which highlights PHIA’s ability to leverage additional\ntools and dynamically react to such limitations.\nRegarding the questions about details and the sources of 30k wearable users, we added\nmore details to Section 2.3.\nPHIA outperforms the Code Generation baseline on real data. “*” indicates p<0.05, Wilcoxon’s\nsigned-rank test.\nComment 1.7: Both query and wearable data needs to be available upon publication through a\nDOI-granting platform\nResponse: We agree with the reviewer that the query and wearable data are a key contribution\nof our work, allowing other researchers to reproduce as well as build on our work. All the\nqueries and wearable data will be available upon publication acceptance. We included this\ninformation in the form of “Code and Software Submission Checklist” in the initial submission,\nand we also included it at the end of Introduction.\nComment 1.8: Details on the human annotators and data assessors are lacking– it should be\nmade clear how the different subsets of annotators were recruited, what their motivations were\nfor participating (compensation, authorship, etc?), and what their demographics were. For\nexample, it is unclear who “the authors’ colleagues” are. Are they members of particular\ndepartments or professional societies? How was their expertise determined? For the question\ngeneration task, why wouldn’t every day people, rather than the authors’ colleagues, be\nselected to generate the questions? Also, were the contributions to the tasks split equally\namong the members or did some contribute more than others (possibly creating imbalanced\nrepresentation in the data).\nResponse: We appreciate the reviewer’s concerns regarding our annotation team and query set.\nFor the reasoning evaluation, twelve annotators were recruited. This annotation team exhibits\ndiversity in terms of education, nationality, gender, and age. Its members hail from Kenya, China,\nIndia, and the United States, holding degrees in fields such as education, information systems,\ndigital arts, statistics, and economics. The requirements for the annotators selected from this\nteam was that they needed to have had significant exposure to other projects that focus on LLM-\nbased health queries, and that they have a sufficiently high level of English proficiency. All the\nteam members were trained through a guidelines document which detailed the evaluation\nprocess, and were provided with a sample of health queries to evaluate as a trial. For the code\nevaluation, seven data scientists with graduate degrees, extensive professional experience in\nanalyzing wearable data, were recruited. These seven data scientists are from the same\ndepartment that authors are affiliated with, but were not involved in this project other than this\nannotation task. None of the annotators are among the authors of this paper.\nFor the open-ended query dataset, we intentionally selected colleagues with expertise in\npersonal health and consumer wearable technologies to ensure the generated questions aligned\nwith the analytical challenges LLMs agents aim to address. Their domain knowledge helped\nproduce queries that were diverse and meaningful, covering categories such as anomaly\ndetection, correlation, and trend analysis. It is worth noting that none of these colleagues\ncontributed to the PHIA development nor this paper. To mitigate imbalances in representation,\nwe ensured that the distribution of queries spanned all nine distinct query types, as shown in\nTable 2. Furthermore, the random selection and shuffling process minimized the likelihood of\noverrepresentation from any specific contributor.\nWe added a section to include these details at Supplement G.5.\nComment 1.9: Data visualization should be improved, for example, Figure 3A: were there\nmultiple measures of accuracy per model type [parentheses for example, using bootstrapping,\nor cross validation techniques)? If yes, error, and or confidence intervals should be included,\nbut if not, a bar chart is not the most effective way to convey this information. In Figure 4, the\ncolors do not convey any meaning and should be removed. Figure 1 should include the\n“thought act and observe” stages of the ReAct framework described in the text.\nResponse: Thank you for the specific feedback. Figure 3-A already contains bootstrapped 95%\nconfidence intervals. Due to the high sample size (over 4000 samples per model) the intervals\nare quite small, making them appear as though they are visual markers to the left of the reported\nvalues. To clarify this, we have highlighted this change at Figure 3(A)’s caption. We have also\nremoved the unnecessary colors from Figure 4, and have added “though, act, and observe”\nlabels to Figure 1.\nResponse to Reviewer #2\nComment 2.1: Dependence on Data Quality: The accuracy and usefulness of PHIA's insights\nheavily depend on the quality and completeness of the wearable data it analyzes. Inconsistent\nor incomplete data could lead to less reliable recommendations.\nResponse: We appreciate the reviewer’s comment highlighting the potential impact of data\nquality and completeness on the accuracy and usefulness of PHIA's insights. While it is true that\ninconsistent or incomplete data could pose challenges, our findings indicate that PHIA is robust\nto such issues. The synthetic Daily Summary Table (Table F.1) of each wearable user used in\nour evaluations (e.g., summary_df_333.csv or the image shown below) exhibits missingness\nacross wearable users, reflecting the real-world variability often encountered in wearable data.\nAn example of synthetic user data (User 333, Summary_DF). It shows our synthetic data\nexhibits missingness that reflects the real-world variability.\nWhen using LLM agents and tools to derive numerical health insights, it turns out that the quality\nof data (synthetic vs. real) has minimal impact on the outcomes. This is because the analytical\nprocesses primarily rely on structured numerical data and statistical properties, which are\nfaithfully preserved in the synthetic datasets. Consequently, synthetic data effectively serves as\na proxy for real-world data in achieving the objectives of our study.\nTo substantiate this claim, we conducted an supplementary 480-hour study involving real-world\nusers and observed similar conclusions in reasoning quality. Importantly, when employing multi-\nstep iterative reasoning and planning, code generation, and web search tools, PHIA significantly\noutperformed baseline models in reasoning accuracy, relevance, interpretation, and\npersonalization. We include the results of this evaluation below and in Supplement H. Notably,\nPHIA had the same high performance on real, non-synthetic data while the Code Generation\nbaseline performed significantly worse. This discrepancy is likely due to additional complexity\n(e.g., missingness, noise, or variability) in real-world datasets, which highlights PHIA’s ability to\nleverage additional tools and dynamically react to such limitations.\nComment 2.2: Limited Scope of Queries: Although PHIA can address a wide range of queries,\nthere might be limitations in its ability to answer highly specialized or complex medical\nquestions that require expert knowledge beyond the scope of wearable data.\nResponse: We appreciate the reviewer’s observation regarding the scope of queries that PHIA\ncan address. As indicated in the title and throughout the paper, our work specifically focuses on\nanswering wearable-based health insights queries. The aim of this research is NOT to build an\nLLM agent capable of addressing highly specialized or complex medical questions requiring\nexpert knowledge beyond the scope of wearable data. To clarify this further, we have added a\nsubsection in the Limitations section titled \"Focus on and Limitations of Wearable Data,\"\nexplicitly referring to prior work (e.g., Med-PaLM) on medical Q&A and emphasizing that PHIA\nis designed for personal health question answering grounded in wearable data. This ensures\nthat the scope and objectives of our work are clear and appropriately framed. We added a\nsection at Section 7 to further discuss our focus on wearable data.\nComment 2.3: Potential Over-reliance on Technology: There is a risk that users might over-over\nrely on the agent’s insights without consulting healthcare professionals, which could lead to\nmisguided health decisions.\nResponse: Thank you for raising this concern. We agree that PHIA should not be used in\ncases where the advice of a healthcare professional is more appropriate. Our labellers were\ncareful to consider this potential harm during data labeling, as described in Section 4.4\n(“Navigating Harm and Uncertainty”). In particular, the one labeller noted that, “I don’t believe\n[the model] should have the authority to tell the user diagnosis, guidance and information.\"\nAs shown in Figure 3(B), only 8 (or 0.3% of total annotated responses) were labeled as harmful.\nAfter annotation was complete, we revisited these responses. Seven of them appear to be\nerroneously labeled as harmful, perhaps due to annotator error. In the eighth, PHIA responds to\nan adversarial query about kidney health by suggesting that the user does not have any renal\nconditions and tells the user to consult a medical professional. While PHIA’s confidence in this\nsingle response is misplaced, we are comforted by its insistence that the user seek professional\nguidance.\nNonetheless, we wish to emphasize that PHIA should not be used as a source of medical\nadvice. To highlight this, we added a paragraph to our Limitations section (Section 7, “Focus\non and Limitations of Wearable Data”) which warns that systems like PHIA do not support\nmedical diagnosis or advice, particularly for conditions which cannot be measured with\nwearable devices.\nComment 2.4: Lack of Innovation in Approach: The integration of large language models\n(LLMs) with wearable health data, while effective, may not represent a groundbreaking\ninnovation. Similar approaches are already being explored in various domains, which could\nlimit the novelty and distinctiveness of PHIA's contribution.\nResponse: We appreciate the reviewer raising their concern about the novelty of our work. We\nrespectfully disagree with the assessment that our work lacks novelty. Our paper introduces\nseveral key innovations in the field of wearable health data analysis. First, we present the\nPersonal Health Insights Agent (PHIA), which, to our knowledge, is the first large language\nmodel-powered agent specifically designed to transform wearable data into actionable health\ninsights. PHIA incorporates advanced reasoning capabilities, combining iterative code\ngeneration, web search, and the ReAct framework to address complex health-related queries.\nMoreover, we curated two benchmark datasets—comprising over 4,000 health-related questions\nand evaluated through over 1,100 hours of manual expert annotation—demonstrating the\nsystem's ability to handle both factual and open-ended queries with high accuracy. Additionally,\nwe designed detailed evaluation rubrics to evaluate PHIA's ability to analyze and respond to\npersonal health and wearable insights questions, ensuring a rigorous and standardized\nassessment framework. This work not only advances the application of LLM agents in health but\nalso paves the way for scalable, personalized, data-driven health recommendations—unlocking\nthe largely untapped potential of LLMs in healthcare. We believe these contributions mark a\nsignificant step forward in the field of wearable health and LLM applications in health care.\nShould we have missed any prior work to capture relevant nuances, we would be happy to\nintegrate concrete recommendations into a revised manuscript.\nComment 2.5: Insufficient Focus on Field-Specific Challenges: The paper does not provide in-\ndepth consideration of the unique challenges and intricacies of the healthcare and wearable\ntechnology fields.\nResponse: Respectfully, PHIA was designed with several field-specific challenges in mind.\nFirst, we constructed our evaluation rubrics to be sensitive to common concerns within the field\n(e.g. safety, personalization, interpretability, harmfulness). Second, our annotators were\nrecruited based on their familiarity with wearable data. Third, we stratify model performance\nacross forms of personal health query (Figure 4.) Fourth, and perhaps most importantly, we\nevaluate open-ended personal health queries collected from health experts.\nResponse to Reviewer #3\nComment 3.1: The datasets being used are pretty small and the way of curating data\nfrom scratch is not scalable.\nResponse: We thank the reviewer for their comment regarding dataset size and scalability. This\npaper focuses on introducing the first large language model-powered agent specifically\ndesigned to transform wearable data into health insights rather than proposing a framework for\ncurating scalable datasets. To support future research, we have open-sourced our dataset,\nincluding its schema and full details, enabling the community to expand and build upon our work\nto address broader scalability needs. Additionally, we provide confidence intervals for our results\n(e.g., Figure 3), demonstrating that the dataset and associated statistical power is sufficiently\nlarge to support the claims made in this study.\nComment 3.2: The technical novelty is limited. For example, LLMs are not very good at\nreasoning about numbers and doing numerical calculations, but I don't find particular technical\ndesigns to address such limited capabilities, nor analysis about it.\nResponse: We appreciate the reviewer raising their concern about the novelty of our work. We\nrespectfully disagree with the assessment that our work lacks novelty. Our paper introduces\nseveral key innovations in the field of wearable health data analysis. First, we present the\nPersonal Health Insights Agent (PHIA), which, to our knowledge, is the first large language\nmodel-powered agent specifically designed to transform wearable data into actionable health\ninsights. PHIA incorporates advanced reasoning capabilities, combining iterative code\ngeneration, web search, and the ReAct framework to address complex health-related queries.\nMoreover, we curated two benchmark datasets—comprising over 4,000 health-related questions\nand evaluated through over 650 hours of manual expert annotation—demonstrating the system's\nability to handle both factual and open-ended queries with high accuracy. Additionally, we\ndesigned detailed rubrics to evaluate PHIA's ability to analyze and respond to personal health\nand wearable insights questions, ensuring a rigorous and standardized assessment framework.\nThis work not only advances the application of LLM agents in health but also paves the way for\nscalable, personalized, data-driven health recommendations—unlocking the largely untapped\npotential of LLMs in healthcare. We believe these contributions mark a significant step forward in\nthe field of wearable health and LLM applications in health care.\nShould we have missed any prior work to capture relevant nuances, we would be happy to\nintegrate concrete recommendations into a revised manuscript.\nComment 3.3: Lack of baselines-- there is basically only the proposed PHIA method\nevaluated against itself, without real comparisons with any potential baselines at scale, and not\neven with different LLMs/tools.\nResponse: We agree that strong baselines are crucial to support the claims we make in the\npaper. It is equally important that the evaluations themselves be rigorous and large enough to\nsubstantiate our conclusions. In this paper our evaluation consists of over 650 hours of manual\nexpert evaluation, time which was necessary to definitively show that agentic frameworks\nprovide significant advantages over LLMs alone. A byproduct of this expensive evaluation is\nthat we were forced to carefully select our experiments to support the specific claims we make\nin this paper. To be clear, we do not claim that Gemini 1.0 is the state-of-the-art model for\npowering health insights agents, or that the PHIA architecture is the best agentic system for\nthis task. Rather, we claim (and empirically substantiate) that tool use and multistep reasoning\nhelp language models succeed at these complex tasks. We have added a paragraph to Section\n3 to clarify this claim.\nComment 3.4: I didn't find the code or link to code.\nResponse: Thank you for your comment. We included the code and data at the “Code and\nSoftware Submission Checklist” PDF in the initial submission, and we also included a link at the\nend of Introduction."}, {"round": 2, "reviewer_comments": {"reviewer_1": "1. The novelty claim remains weak. Another work which even includes some of the same authors as from this paper introduces a separate LLM-powered agent “designed to transform wearable data into actionable health insights” [1].\n2. The chosen baseline models (numerical reasoning and code generation) have still not been justified–why are these the best comparators?\n3. The rewrite has not substantially improved the overly general language describing health insights in this paper. The problem is not in the limitations section but rather the general language used throughout the manuscript. The language needs to be made more specific throughout the manuscript for it to accurately describe the work presented here, without jumping to future work that hasn’t yet been done and using that to justify the language.\n4. The justification for why to use synthetic data makes sense, but it is still not clear if the synthetic data itself was appropriately vetted.\n[1] https://arxiv.org/abs/2406.06474\n(Remarks on code availability)", "reviewer_2": "Summary: This paper introduces PHIA, an LLM-powered agent framework designed to analyze and interpret personal health data from wearable devices. PHIA employs iterative reasoning, code generation, and external tool use (e.g., web search) to provide actionable insights. The authors compare PHIA against two baselines—Numerical Reasoning and Code Generation—and demonstrate its superior performance in both objective and open-ended personal health queries. The paper also includes a comprehensive evaluation framework, combining automated correctness checks and human ratings, to assess the quality of PHIA’s reasoning and code generation. The authors highlight PHIA’s ability to recover from errors, avoid hallucinations, and provide clear, relevant, and harmless responses. The study is supported by a large dataset of over 4,000 personal health queries and extensive human evaluation, making it a significant contribution to the field of personal health insights.\nStrengths:\n1. Agent Design and Code Generation Framework. The design of PHIA’s agent framework is highly reasonable and well-justified. By offloading complex reasoning tasks to an iterative agent-based system, the authors avoid overburdening the base language model. This modular approach allows PHIA to strategically plan, execute, and refine its reasoning steps, resulting in more accurate and reliable outputs. The ability to recover from errors further highlights the robustness of this design.\n2. System for Open Queries. The paper effectively addresses the challenge of open-ended personal health queries, which require integrating user-specific data with domain knowledge. The evaluation framework for these queries is rigorous, involving human annotators who assess multiple dimensions of reasoning quality, such as relevance, personalization, and domain knowledge. This focus on open-ended reasoning is a notable advancement over prior work, which often emphasizes simpler, objective queries.\n3. Quality and Quantity of Data. The authors present a well-curated dataset of over 4,000 personal health queries, which is both large and diverse. The dataset includes both objective and open-ended queries, ensuring comprehensive evaluation of PHIA’s capabilities. The inclusion of human annotations and expert evaluations further enhances the quality of the data and the reliability of the results.\n4. Significance of the Experiments. The experiments are thoughtfully designed and provide compelling evidence of PHIA’s advantages over baseline methods. The authors demonstrate PHIA’s superior performance in both numerical reasoning and open-ended queries, with significant improvements in reasoning quality, error recovery, and avoidance of hallucinations. The use of both automated and human evaluation methods strengthens the validity of the findings. Additionally, the inclusion of qualitative interviews with annotators and experts adds depth to the analysis.\n5. Error Recovery and Stability. PHIA’s ability to recover from errors is a particularly noteworthy strength. The agent framework allows PHIA to identify and correct mistakes iteratively, resulting in a significantly lower error rate compared to the Code Generation baseline. This feature enhances the system’s reliability and makes it more suitable for real-world applications.\nWeaknesses: Unproven Effectiveness of Insights. While PHIA demonstrates strong reasoning capabilities, the paper does not evaluate the real-world effectiveness of its insights in improving user behavior or health outcomes. Clinical trials or user studies would be necessary to validate the practical impact of PHIA’s recommendations.\nOverall Assessment: Dependence on Data Quality, Limited Scope of Queries, Potential Over-reliance on Technology, Lack of Innovation in Approach, Insufficient Focus on Field-Specific Challenges. These issues have been explained or their limitations emphasized in the new version of the paper.\nThe authors overall have addressed these concerns satisfactorily. While the work may not represent a major technical breakthrough, it is a well-executed study with practical implications for personalized health management.\nBased on the revisions, I recommend the manuscript for publication, provided the editorial team is satisfied with the level of novelty and scope.\n(Remarks on code availability)", "reviewer_3": "I was reviewer #3 for the previous version. The authors have provided responses to my questions, but 3 of the 4 questions/issues remain unsolved.\nFor my first comment on the scale and scalability of the dataset, the authors did not really solve the problem. The dataset being used is small, and the way of manually curating data is not scalable.\nFor my second comment on technical novelty, I agree the study is novel in applying LLM to generate insight from wearable data, but the study offers little to no technical contributions or insights-- by basically using existing LLMs and manually curating a dataset.\nFor my third comment on the lack of baselines, the authors did not even attempt to add any possible baselines, but just re-iterate how many hours they spent on manually creating the dataset.\nI saw the code and data now, so my forth comment was addressed.\n(Remarks on code availability)\nI reviewed but did not run the code."}, "author_response": "Point by Point Response of Nature Communications Manuscript\nNCOMMS-24-41183B-Z\nGeneral Response.................................................................................................................1\nResponse to the Editor.........................................................................................................1\nResponse to Reviewer #1.....................................................................................................1\nResponse to Reviewer #2.....................................................................................................2\nResponse to Reviewer #3.....................................................................................................2\nGeneral Response\nDear Reviewers,\nWe thank you for your thoughtful and constructive feedback, which has been invaluable in\nstrengthening our manuscript. All changes in the revised version are highlighted in blue. In\nsummary, we have made the following key revisions:\nClarified Novelty (R1 & R3): Expanded our comparison to concurrent and prior work (PH-LLM\n[1], Englhardt et al. [2]) in the Introduction and Related Work, explicitly positioning PHIA as the\nfirst open-ended, wearable health Q&A system that generates accurate numerical insights. Our\nsystem dramatically expands the scope of health queries that can be answered and answered\ncorrectly through AI models, compared to prior work.\nImproved Framing and Terminology (R1):\n1. Added a clear definition in Introduction of “Personal Health Insights” and renamed all\noccurrences of “health insights” to “Personal Health Insights”.\n2. Refined global terminology: renamed to “Personal Health Insights”, added a working\ndefinition, and updated title to “Transforming Wearable Data into Personal Health\nInsights using Large Language Model Agents.”\nStrengthened Baseline Justification and Comparisons (R1 & R3):\n1. Substantiated our choice of baselines (Numerical Reasoning, Code Generation) by\ndiscussing their limitations in this response.\n2. Added two additional LLM-based baselines—PH-LLM (a fine-tuned version of Gemini\nUltra 1.0) and a specialized prompted GPT-4 chain-of-thought strategy for wearable data\nand reported their performance (0% and 53.6% accuracy) alongside our original\nbaselines (Section 4.1 and 4.3)\nExpanded Limitations and Future Work (R2): Added acknowledgment of the need for\nreal-world deployment studies and clinical trials to assess behavior change impacts (Section 7).\nWe believe these revisions directly address each reviewer’s concerns and substantially\nstrengthen the manuscript’s clarity, rigor, and impact. Please see our point-by-point responses\nfor further details.\n[1] Cosentino, Justin, et al. \"Towards a personal health large language model.\" arXiv preprint\narXiv:2406.06474 (2024).\n[2] Englhardt, Zachary, et al. \"From classification to clinical insights: Towards analyzing and\nreasoning about mobile and behavioral health data with large language models.\" Proceedings of the\nACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8.2 (2024): 1-25.\nSincerely,\nTim Althoff, Daniel McDuff, Xin Liu\nOn behalf of all authors\nResponse to Reviewer #1\n1. The novelty claim remains weak. Another work which even includes some of the\nsame authors as from this paper introduces a separate LLM-powered agent “designed\nto transform wearable data into actionable health insights” [1].\nResponse: We thank the reviewer for this comment and appreciate the opportunity to\nclarify the novelty of PHIA (Personal Health Insights Agent). While both PHIA and\nPersonal Health Large Language Model (PH-LLM) are concurrent submissions and both\naim to use LLM on wearable data, they differ significantly in their architecture,\ncapabilities, and primary applications.\nPHIA represents the first open-ended wearable question-answering system that\nleverages large language models, agentic frameworks, and external tools, such as\nPython code execution and web search, to iteratively reason over user-specific\ntime-series data and answer both factual and open-ended health queries. Providing\naccurate numerical insights is crucial—not only because consumers consistently\nprioritize trustworthy and personalized data-driven recommendations based on our user\nexperience research, but also because the capability to analyze detailed personal data\nis foundational for truly personalized health rather than providing generic responses.\nHowever, current language models often struggle with both simple and complex\nnumerical reasoning, which can result in inaccurate recommendations, serious\nconsequences, and diminished user engagements. Prior research in this space has\nbeen limited to generating insights and recommendations from highly aggregated data,\nwhere experts predefine and precompute relevant statistics, instead of requiring the AI\nmodels to directly perform more complex numerical reasoning and analyses. Relative\nto prior work whether at the time of submission or today (e.g., PH-LLM), this\ndramatically expands the scope of health queries that can be answered and\nanswered correctly through AI models.\nDetailed comparison with PH-LLM:\n● PH-LLM is a fine-tuned LLM (based on Gemini Ultra 1.0) focused on delivering\ncoaching for fitness and sleep using aggregated 30-day wearable data (e.g., 95th\npercentile of sleep duration). In contrast, PHIA is an agentic system that works\nwith high-resolution daily data and emphasizes general-purpose health analysis\nqueries by dynamically integrating tools.\n● PH-LLM relies on in-model reasoning to generate long-form insights and\nrecommendations and is specifically fine-tuned for coaching recommendations,\nnot for providing factually complex numerical insights for general wearable-based\nqueries.\n● PHIA massively expands the scope of the health queries that can be answered\nwhile PH-LLM exclusively focuses on two types of queries in sleep and fitness\ncoaching (e.g., “How can I improve my sleep?”).\n● Our new baseline evaluation shows that PH-LLM is unable to perform on any of\nour objective health queries, due to a lack of basic instruction following after\nbeing fine-tuned on coaching-specific case study data and the context length of\nour input tabular data.\nWe have now further emphasized these points in our Introduction and related work sections:\nPage 1, second paragraph of Introduction: “However, despite their broad capabilities, current\nLLMs frequently struggle with numerical reasoning, often resulting in inaccurate\nrecommendations, diminished user trust, potential health risks, and reduced engagement.\nPrevious efforts [1, 2] have typically relied on pre-aggregated, expert-defined statistical\nsummaries rather than enabling LLM models to perform direct, nuanced analyses of granular\npersonal data. Thus, the capability for LLM models to autonomously reason about detailed\nindividual health data remains significantly limited.”\nPage 2, fifth paragraph of Introduction: “In this paper, we introduce the Personal Health Insights\nAgent (PHIA), the first open-ended wearable question-answering system powered by\nLLM-based agents designed specifically for nuanced reasoning over personal wearable data.\nPHIA leverages state-of-the-art code generation, web search integration, and the iterative ReAct\nagent framework to autonomously perform complex analyses and generate accurate,\ncontext-aware responses to thousands of diverse health queries..”\nPage 13, line 397-407: “More recently, emerging work has begun to demonstrate the capacity of\nLLMs to interpret wearable sensor data in diverse applications. Cosentino et al. introduced\nPH-LLM [1], a fine-tuned variant of Gemini Ultra 1.0 that delivers long-form fitness and sleep\ncoaching by reasoning over aggregated 30-day wearable summaries. Unlike agentic\nframeworks that invoke external tools, PH-LLM relies entirely on in-model reasoning to generate\nrecommendations, and is optimized for coaching rather than answering open-ended and\nnumerically precise wearable health queries. Similarly, Englhardt et al. [2] explored custom\nchain-of-thought prompting of GPT-4 and PaLM 2 to conduct depression-related classification\ntasks from daily wearable metrics presented as text, effectively framing LLMs as collaborators in\nclinical settings. Although both approaches leverage LLM reasoning over wearable data, they\ndiffer from PHIA in several respects: underlying architecture, primary use case (targeted\ncoaching or clinical support versus general-purpose health question-answering), dependence\non in-model reasoning versus external tool integration, and the degree to which they process\nhigh-resolution signals versus aggregated inputs”\n[1] Cosentino, Justin, et al. \"Towards a personal health large language model.\" arXiv preprint\narXiv:2406.06474 (2024).\n[2] Englhardt, Zachary, et al. \"From classification to clinical insights: Towards analyzing and\nreasoning about mobile and behavioral health data with large language models.\" Proceedings of the\nACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8.2 (2024): 1-25.\n2. The chosen baseline models (numerical reasoning and code generation) have still\nnot been justified–why are these the best comparators?\nWe selected \"Numerical Reasoning\" and \"Code Generation\" as baselines to\nsystematically evaluate the necessity and effectiveness of external tools (Python\ninterpreter and Web search) and PHIA's multi-step iterative thinking capabilities from its\nagentic framework.\nNumerical Reasoning: This tested if an LLM could directly perform necessary\nmathematical and analytical reasoning on structured data (Markdown tables) without a\ndedicated code interpreter. Its poor performance (22% accuracy on objective queries vs.\n74% using a code interpreter) highlighted current LLM limitations in reliable numerical\nmanipulation of personal health data, underscoring the need for PHIA's code execution\ntool.\nCode Generation: This baseline assessed the need for an iterative, interactive agentic\nframework. It generates code in a single step, lacking PHIA's reflective \"Thought\"\nprocess and thus cannot strategize, iteratively analyze, or use web search. PHIA’s\noutperformance here (e.g., 84% vs. 74% on objective queries) demonstrates the value\nof its iterative agentic approach.\nFurthermore, in response to reviewer feedback, and as detailed in our revised\nmanuscript, we have added two more contemporary and robust baselines prior or\nconcurrent to this submission. Both of these baselines are focused on using LLMs to\ninterpret and understand wearable data.\n● PH-LLM: This model is not able to answer any of our objective queries due to a\nlack of basic instruction following capabilities when the input includes a complex\ntable after its specific fine-tuning on coaching case studies.\n● A prompted GPT-4 model (as per Englhardt et al., 2024): This baseline achieved\n53.6% accuracy on our objective query set.\n● In summary, our proposed approach significantly outperforms both of these\nbaselines.\nWe have now further emphasized these points in our Section 4.1 Baselines and Section 4.3\nHow does PHIA perform.\nPage 6, line 183-line 186: “Is a multi-step agentic framework really necessary to derive health\ninsights from wearable data, or can simpler methods provide adequate results? To evaluate the\nnecessity of the framework and tools (i.e., code generation, web search), we constructed four\nlanguage model baselines to demonstrate PHIA's performance as illustrated in Figures 3-6. An\nexample of responses from our baselines alongside PHIA can be found in Figure 2.”\nPage 6-7 line 203 - line 214: “Additional LLM-based Wearable Systems: We compare PHIA\nagainst recent LLM-based methods, including the Personal Health Large Language Model\n(PH-LLM [1]) . It is a fine-tuned LLM based on Gemini Ultra 1.0, focused on delivering coaching\nfor fitness and sleep based on aggregated 30-day wearable data (e.g., 95th percentile of sleep\nduration) instead of the high-resolution daily data. Rather than invoking tools, PH-LLM uses\nin-model reasoning to generate long-form insights and recommendations. Moreover, PH-LLM is\nfine-tuned specifically for providing coaching recommendations only instead of providing\nnumerical insights and recommendations for general wearable-based queries. Additionally, we\ncompare our approach to a specialized chain-of-thought prompting strategy designed for\ninterpreting time-series wearable data with the GPT-4 model [2]. This approach instructs the\nmodel to reason directly within its textual context window without external computational tools.\nOverall, unlike PHIA, these methods focus on internal model reasoning and do not incorporate\niterative agentic framework and external tools. In addition, this approach is based on GPT-4\ninstead of Gemini, enabling us to demonstrate that our proposed approach outperforms even\nstrong baselines that leverage alternative language models.”\nPage 9 line 265 - line 275: “we present the evaluation results for objective personal health\nqueries. PHIA achieves an exact match accuracy of 84%, significantly outperforming the Code\nGeneration baseline (74% accuracy), Numerical Reasoning (22% accuracy). We also evaluated\non a custom chain-of-thought prompting strategy designed for interpreting time-series wearable\ndata with GPT-4 [2] (53.6% accuracy). We observe that PH-LLM model [1] is not able to answer\nany of our objective queries due to its limitations in handling detailed, long-context tabular data\ninputs after being fine-tuned exclusively on aggregated coaching case study data. This\ndemonstrates that the agent framework's complexity and iterative reasoning substantially\nenhance performance on numerical queries, even those requiring limited abstract reasoning.\nThe text results from our internal model reasoning approaches further emphasize that text-only\nreasoning is inadequate for precise numerical manipulations on personal health data, likely due\nto inherent limitations in current LLMs' mathematical and tabular reasoning capabilities.\nConsequently, we excluded these methods from the costly human evaluation.”\n3. The rewrite has not substantially improved the overly general language describing\nhealth insights in this paper. The problem is not in the limitations section but rather the\ngeneral language used throughout the manuscript. The language needs to be made\nmore specific throughout the manuscript for it to accurately describe the work presented\nhere, without jumping to future work that hasn’t yet been done and using that to justify\nthe language.\nFROM ROUND 1: Comment 1.1: The overall framing of the paper uses general terms like\n“health insights” when the outputs are in reality very specific to health insights from wearables\ndata. This should be clarified and corrected throughout the paper.\nFROM ROUND 1: Comment 1.2: Claims made in the paper are often not substantiated by the\ndata, and those claims need to be scaled back or changed. Specific examples include the first\nbullet point on page 2 and the first section header under 4.3 (where the text does not follow the\nheader)\nResponse: We thank the reviewer for raising the issues of general language used in the paper.\nWe acknowledge that the definition of “health insights” is ambiguous. We have done the\nfollowing things to make the language more specific and consistent:\n1. Title refinement: We changed our title to “Transforming Wearable Data into Personal\nHealth Insights using Large Language Model Agents” from “Transforming Wearable Data\ninto Health Insights using Large Language Model Agents”. This change makes explicit\nthat “insights” refers only to those derived from wearable time-series.\n2. Working definition added: We added a definition at Introduction line 24-26: “In this\nwork, we use the term \\textbf{personal health insights} to refer specifically to the answers\nproduced when an LLM system performs data-driven analyses over a user’s wearable\ndevice time-series”. This immediately anchors every subsequent use of the term.\nPage 1-2 line 23 - line 34: “Despite these gross benefits, using wearable data to derive\nintelligent responses and insights to personal health queries is non-trivial. For example,\na common question of wearable device users is ``Do I get better sleep after\nexercising?''. Though a seemingly straightforward question, arriving at an ideal\nresponse would involve performing a series of complex, independent analytical steps\nacross multiple irregularly sampled time series such as: checking the availability of\nrecent data, deciding on metrics to optimize (e.g. sleep and exercise duration),\nsummarizing sleep metrics on the days with activity events, contextualizing these\nfindings within the broader spectrum of the individual's health, integrating knowledge of\npopulation norms, and offering tailored sleep improvement recommendations. These\nsteps not only involve numerical analysis but also an interpretation of what constitutes\nhealthy sleep under the nuances of an individual's overall health profile. In this work, we\nuse the term personal health insights to refer specifically to the outputs generated\nwhen an LLM system analyzes a user’s wearable time-series data in response to their\nhealth-related queries. This definition underscores that our focus—and the challenges\nwe tackle—are rooted in multimodal wearable streams.”\n3. Global terminology update: Every occurrence of “health insights” has been replaced\nwith “personal health insights” (including figure captions, section headers, and the Key\nContributions bullet points) to maintain consistency.\nWe believe these edits directly address the R1’s concern by consistently and unambiguously\ntying every claim to the wearable-data domain, rather than implying broader “health insights”\nbeyond our evaluation. We welcome any further suggestions on specific passages and writing,\nand would be happy to address any remaining issues in our revision.\n4. The justification for why to use synthetic data makes sense, but it is still not clear if\nthe synthetic data itself was appropriately vetted.\nResponse: We thank the reviewer for acknowledging the justification for using synthetic data.\nWe would like to provide further clarity on how the synthetic data was vetted to ensure its\nappropriateness and fidelity.\nAs described in our manuscript in Section 2.3, the synthetic data generator was developed\nbased on a large-scale anonymized dataset from 30,000 real wearable users with rich activity\nevent data, who consented to their data being used for research. We required rich activity data\nin order to evaluate more complex personal health questions. This foundational dataset\ncaptured users with at least 10 days of data recorded during October 2023, aged between 18\nand 80, ensuring a degree of real-world variability. We then used a Conditional Probabilistic\nAuto-Regressive neural network, designed for sequential multivariate data, which incorporates\nstable contextual attributes like age, weight, and gender. This process was designed to\ngenerate realistic user data, and we used this process to generate data for 56 synthetic\nwearable users (which is the dataset we will share with other researchers).\nNext, we describe our efforts of validating this synthesized data. As shown in Figure 1 below,\nmembers of our team familiar with wearable device data manually inspect the distributions of\nwearable metrics in this synthetic dataset. We find that the distributions of the synthetic data\nclosely resemble those of active users across 10 different metrics, ranging from resting heart\nrate and heart rate zones, to steps, sleep, and stress measures.\nImportantly, we also strengthen our rebuttal through new analyses and findings from an\nextensive 480-hour supplementary study with real-world user data in the last round of rebuttal,\nwhich mirrored the conclusions on reasoning quality obtained from synthetic data. This\ndemonstrates that the synthetic data used previously did not misrepresent actual user data in\nways that limit the generalizability of our work. Concretely, PHIA—when utilizing its multi-step\niterative reasoning, planning capabilities, code generation, and web search\ntools—demonstrated superior performance over baseline models in the accuracy, relevance,\ninterpretation, and personalization of its reasoning. These results are included in Supplement I.\nIt is particularly noteworthy that while PHIA maintained its high level of performance on actual,\nnon-synthetic data, the Code Generation baseline's performance degraded considerably. This\ndivergence likely stems from the increased complexities present in real-world datasets, such as\nmissing information, noise, or greater variability. This highlights PHIA's adeptness at leveraging\nits diverse toolset to dynamically adapt to and manage such real-world data limitations.\nLastly, we want to re-emphasize an advantage of using synthetic data: it obviates the privacy\nconcerns associated with real-world wearable datasets. This approach enables us to publicly\nrelease our dataset, thereby ensuring the complete reproducibility of our findings, and enabling\nothers to build on our work and this dataset.\nFigure 1: Distribution of our synthetic dataset (~1700 days) across 10 key wearable metrics.\nResponse to Reviewer #2\nWeaknesses: Unproven Effectiveness of Insights. While PHIA demonstrates strong reasoning\ncapabilities, the paper does not evaluate the real-world effectiveness of its insights in improving user\nbehavior or health outcomes. Clinical trials or user studies would be necessary to validate the\npractical impact of PHIA’s recommendations.\nResponse: Thanks for your raising the concern of not including the outcomes of behavior\nchange. We updated the limitations section to incorporate your feedback.\nPage 14 line 441 - line 444: “We also acknowledge that we did not evaluate PHIA through\nreal-world deployment studies in order to evaluate potential impacts on behavior change and\nother health outcomes. Further clinical trials or user studies would be necessary to validate the\npractical impact of PHIA’s recommendations.”\nResponse to Reviewer #3\n1. For my first comment on the scale and scalability of the dataset, the authors did not really\nsolve the problem. The dataset being used is small, and the way of manually curating data is\nnot scalable.\nResponse: We appreciate Reviewer #3's feedback regarding the dataset size and scalability. We\nwould like to offer some context for our approach. Our primary aim with this study is to establish\nthe feasibility and utility of a novel agentic framework (PHIA) for deriving health insights.\nRegarding the dataset's scale, we are confident that the data presented supports all claims\nmade in the manuscript. This is evidenced by statistically appropriate analyses, including narrow\nbootstrapped confidence intervals on key effect estimates. While larger datasets can often\nprovide additional perspectives, our analyses indicate that the current dataset is sufficient for the\nconclusions drawn. We also wish to highlight that our dataset of over 4,000 personal health\ninsights queries is, to our knowledge, the largest publicly available resource of its kind for\nwearable health research.\nIt is important to clarify that we have not made claims regarding the scalability of our current\ndata curation framework itself. Data collection and curation in the health domain, particularly for\nnuanced query-answering tasks, are inherently expertise-heavy and resource-intensive.\nDeveloping a broadly scalable data curation process, while a valuable pursuit, falls outside the\ndefined scope of this particular paper, which is focused on the capabilities of the PHIA system.\n2. For my second comment on technical novelty, I agree the study is novel in applying LLM to\ngenerate insight from wearable data, but the study offers little to no technical contributions or\ninsights-- by basically using existing LLMs and manually curating a dataset.\nResponse: We thank the reviewer for their feedback and for acknowledging the novelty of\napplying LLMs to generate insights from wearable data. We appreciate the opportunity to further\nclarify the technical contributions of our work.\nAs the R3 notes, and as we have stated, a central novel aspect of our paper is the careful\ndesign and empirical validation of an LLM-powered agentic system (PHIA) specifically for\ninterpreting wearable data—a complex and impactful domain. PHIA is, to our knowledge, the\nfirst open-ended wearable question-answering system. It integrates multi-step iterative\nreasoning, code generation, and web search tools to address both factual and open-ended\nhealth queries from user-specific time-series wearable data. While the underlying LLM is an\nexisting technology, the novelty lies in adopting and validating an LLM agentic system for this\nnew and challenging problem space.\nWe agree that PHIA does not introduce a fundamentally new algorithm in core machine\nlearning. However, the contribution of successfully engineering and demonstrating the utility of\nexisting advanced AI techniques to solve significant, previously untackled problems in specific\ndomains is a widely recognized and valuable form of technical advancement. Many impactful\npapers in the AI and health space, such as those introducing systems like Med-PaLM [1, 2] or\nco-scientist for scientific research [3], focus on this type of technical contribution: the innovative\nintegration and application of AI to drive progress in a particular field.\nIn this paper, we propose the first open-ended wearable question-answering system that\nleverages large language models, agentic frameworks, and external tools, such as Python code\nexecution and web search, to iteratively reason over user-specific time-series data and answer\nboth factual and open-ended health queries. Providing accurate numerical insights is\ncrucial—not only because consumers consistently prioritize trustworthy and personalized\ndata-driven recommendations based on our user experience research, but also because the\ncapability to analyze detailed personal data is foundational for truly personalized health rather\nthan providing generic responses. However, current language models often struggle with both\nsimple and complex numerical reasoning, which can result in inaccurate recommendations,\nserious consequences, and diminished user engagements. Prior research in this space has\nbeen limited to generating insights and recommendations from highly aggregated data,\nwhere experts predefine and precompute relevant statistics, instead of requiring the AI models to\ndirectly perform more complex numerical reasoning and analyses. Relative to prior work\nwhether at the time of submission or today (e.g., PH-LLM [4]), this dramatically expands the\nscope of health queries that can be answered and answered correctly through LLM models. Our\nwork establishes that such an agentic approach is not only feasible but also significantly\noutperforms strong baselines (including code generation, numerical reasoning, GPT-4 with\ncustom chain of thought for wearable data [5], and PH-LLM)) in transforming raw wearable data\ninto actionable health insights. This, we believe, is a crucial technical step forward for\npersonalized health and represents a substantial contribution to the field. We have further\nelaborated on these points in the revised Introduction section.\n[1] Singhal, Karan, et al. \"Large language models encode clinical knowledge.\" Nature 620.7972\n(2023): 172-180.\n[2] Singhal, Karan, et al. \"Toward expert-level medical question answering with large language\nmodels.\" Nature Medicine (2025): 1-8.\n[3] Gottweis, Juraj, et al. \"Towards an AI co-scientist.\" arXiv preprint arXiv:2502.18864 (2025).\n[4] Cosentino, Justin, et al. \"Towards a personal health large language model.\" arXiv preprint\narXiv:2406.06474 (2024).\n[5] Englhardt, Zachary, et al. \"From classification to clinical insights: Towards analyzing and\nreasoning about mobile and behavioral health data with large language models.\" Proceedings of the\nACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8.2 (2024): 1-25.\nWe have now further emphasized these points in our Introduction:\nSection 1 Introduction:\nPage 1, second paragraph of Introduction: “However, despite their broad capabilities, current\nLLMs frequently struggle with numerical reasoning, often resulting in inaccurate\nrecommendations, diminished user trust, potential health risks, and reduced engagement.\nPrevious efforts [1, 2] have typically relied on pre-aggregated, expert-defined statistical\nsummaries rather than enabling LLM models to perform direct, nuanced analyses of granular\npersonal data. Thus, the capability for LLM models to autonomously reason about detailed\nindividual health data remains significantly limited.”\nPage 2, fifth paragraph of Introduction: “In this paper, we introduce the Personal Health Insights\nAgent (PHIA), the first open-ended wearable question-answering system powered by\nLLM-based agents designed specifically for nuanced reasoning over personal wearable data.\nPHIA leverages state-of-the-art code generation, web search integration, and the iterative ReAct\nagent framework to autonomously perform complex analyses and generate accurate,\ncontext-aware responses to thousands of diverse health queries..”\n3. For my third comment on the lack of baselines, the authors did not even attempt to add any\npossible baselines, but just re-iterate how many hours they spent on manually creating the\ndataset.\nResponse: We would like to address the R3’s concerns regarding baselines. For this current\nrevision, we have expanded our baseline comparisons to further contextualize PHIA's performance.\nAs reflected in our revised manuscript, we have incorporated two additional LLM-based Wearable\nSystems as baselines:\nPersonal Health Large Language Model (PH-LLM) [1]: This is a concurrent submission as our\npaper and is a supervised fine-tuned LLM for wearable based coaching. As such it is one of the most\nsimilar concurrent works. Our evaluations show this model is unable to answer any of the objective\nqueries due to its lack of instruction following capabilities when the input data includes complex\ntabular data.\nA Prompted GPT-4 model for Wearable Data Reasoning [2]: This paper investigated various text\nreasoning on top of wearable data with GPT-4 model and studied human-AI collaboration on mental\nhealth domain. This baseline demonstrated 53.6% accuracy on the same set of 4,000 objective\npersonal health insights queries.\nThe superior accurate numerical reasoning capabilities of PHIA (84% accuracy) significantly\noutperforms all mentioned baselines, underscoring the benefits of PHIA's agentic framework and\ntool-use capabilities for this task. We believe these additions directly address the concern about the\nbreadth of baselines and more robustly demonstrate the contributions of our work.\nWe have now further emphasized these points in our Introduction, related work and experiment\n& result sections:\nSection 1 Introduction:\nPage 1, second paragraph of Introduction: “However, despite their broad capabilities, current\nLLMs frequently struggle with numerical reasoning, often resulting in inaccurate\nrecommendations, diminished user trust, potential health risks, and reduced engagement.\nPrevious efforts [1, 2] have typically relied on pre-aggregated, expert-defined statistical\nsummaries rather than enabling LLM models to perform direct, nuanced analyses of granular\npersonal data. Thus, the capability for LLM models to autonomously reason about detailed\nindividual health data remains significantly limited.”\nPage 2, fifth paragraph of Introduction: “In this paper, we introduce the Personal Health Insights\nAgent (PHIA), the first open-ended wearable question-answering system powered by\nLLM-based agents designed specifically for nuanced reasoning over personal wearable data.\nPHIA leverages state-of-the-art code generation, web search integration, and the iterative ReAct\nagent framework to autonomously perform complex analyses and generate accurate,\ncontext-aware responses to thousands of diverse health queries..”\nSection 4: Experiments & Results\nPage 6, line 183-line 186: “Is a multi-step agentic framework really necessary to derive health\ninsights from wearable data, or can simpler methods provide adequate results? To evaluate the\nnecessity of the framework and tools (i.e., code generation, web search), we constructed four\nlanguage model baselines to demonstrate PHIA's performance as illustrated in Figures 3-6. An\nexample of responses from our baselines alongside PHIA can be found in Figure 2.”\nPage 6-7 line 203 - line 214: “Additional LLM-based Wearable Systems: We compare PHIA\nagainst recent LLM-based methods, including the Personal Health Large Language Model\n(PH-LLM [1]) . It is a fine-tuned LLM based on Gemini Ultra 1.0, focused on delivering coaching\nfor fitness and sleep based on aggregated 30-day wearable data (e.g., 95th percentile of sleep\nduration) instead of the high-resolution daily data. Rather than invoking tools, PH-LLM uses\nin-model reasoning to generate long-form insights and recommendations. Moreover, PH-LLM is\nfine-tuned specifically for providing coaching recommendations only instead of providing\nnumerical insights and recommendations for general wearable-based queries. Additionally, we\ncompare our approach to a specialized chain-of-thought prompting strategy designed for\ninterpreting time-series wearable data with the GPT-4 model [2]. This approach instructs the\nmodel to reason directly within its textual context window without external computational tools.\nOverall, unlike PHIA, these methods focus on internal model reasoning and do not incorporate\niterative agentic framework and external tools. In addition, this approach is based on GPT-4\ninstead of Gemini, enabling us to demonstrate that our proposed approach outperforms even\nstrong baselines that leverage alternative language models.”\nPage 9 line 265 - line 275: “we present the evaluation results for objective personal health\nqueries. PHIA achieves an exact match accuracy of 84%, significantly outperforming the Code\nGeneration baseline (74% accuracy), Numerical Reasoning (22% accuracy). We also evaluated\non a custom chain-of-thought prompting strategy designed for interpreting time-series wearable\ndata with GPT-4 [2] (53.6% accuracy). We observe that PH-LLM model [1] is not able to answer\nany of our objective queries due to its limitations in handling detailed, long-context tabular data\ninputs after being fine-tuned exclusively on aggregated coaching case study data. This\ndemonstrates that the agent framework's complexity and iterative reasoning substantially\nenhance performance on numerical queries, even those requiring limited abstract reasoning.\nThe text results from our internal model reasoning approaches further emphasize that text-only\nreasoning is inadequate for precise numerical manipulations on personal health data, likely due\nto inherent limitations in current LLMs' mathematical and tabular reasoning capabilities.\nConsequently, we excluded these methods from the costly human evaluation.”\nSection 6 Related Work:\nPage 13, line 397-407: “More recently, emerging work has begun to demonstrate the capacity of\nLLMs to interpret wearable sensor data in diverse applications. Cosentino et al. introduced\nPH-LLM [1], a fine-tuned variant of Gemini Ultra 1.0 that delivers long-form fitness and sleep\ncoaching by reasoning over aggregated 30-day wearable summaries. Unlike agentic\nframeworks that invoke external tools, PH-LLM relies entirely on in-model reasoning to generate\nrecommendations, and is optimized for coaching rather than answering open-ended and\nnumerically precise wearable health queries. Similarly, Englhardt et al. [2] explored custom\nchain-of-thought prompting of GPT-4 and PaLM 2 to conduct depression-related classification\ntasks from daily wearable metrics presented as text, effectively framing LLMs as collaborators in\nclinical settings. Although both approaches leverage LLM reasoning over wearable data, they\ndiffer from PHIA in several respects: underlying architecture, primary use case (targeted\ncoaching or clinical support versus general-purpose health question-answering), dependence\non in-model reasoning versus external tool integration, and the degree to which they process\nhigh-resolution signals versus aggregated inputs”\n[1] Cosentino, Justin, et al. \"Towards a personal health large language model.\" arXiv preprint\narXiv:2406.06474 (2024).\n[2] Englhardt, Zachary, et al. \"From classification to clinical insights: Towards analyzing and\nreasoning about mobile and behavioral health data with large language models.\" Proceedings of the\nACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8.2 (2024): 1-25."}, {"round": 3, "reviewer_comments": {"reviewer_2": "Summary: I appreciate the authors’ efforts in revising the manuscript “Transforming Wearable Data into Health Insights Using Large Language Model Agents”. While the topic is timely and the study presents a technically competent integration of LLMs with wearable data analytics, the revised manuscript still does not meet the publication standards and innovation threshold expected by Nature.\nMajor Concerns:\nLack of Breakthrough Innovation: The manuscript primarily describes an application of existing large language models to a relatively straightforward wearable data analysis scenario. While technically sound, the integration of LLMs and rule-based or tool-augmented agents for personalized health analytics is already an established direction in both academia and industry. The manuscript does not offer a clear conceptual or technical breakthrough beyond what has been presented in previous literature. The novelty is incremental, which limits its significance for a general audience.\nAssessment of Real-world Impact: The authors do not provide convincing evidence that the agent's insights result in meaningful improvements in users’ health understanding or behavior. User studies, clinical validation, or real-world deployments demonstrating actual utility and adoption are missing. The evaluation largely relies on curated datasets and synthetic benchmarks, which do not fully capture the complexity and variability of real “in-the-wild” wearable data.\nLimited Medical Depth and Clinical Relevance: Although the system can handle a range of queries, its capability to address clinically meaningful health questions is inherently limited by both the wearable data and the scope of the LLM. Expert-level queries, nuanced medical conditions, and integration with broader patient health records are not considered. This limits the depth and impact of the work, particularly for a journal like Nature.\nConclusion: In view of the above, I believe the manuscript falls short of the novelty, depth, and real-world impact required for publication in Nature. Substantially more work is required in terms of both technical innovation and domain-specific validation to merit reconsideration.\n(Remarks on code availability)", "reviewer_3": "I thank the authors for making substantially more efforts in this revision to address my previous concerns. In this revision, my concerns on dataset scale, technical novelty, and necessary baselines are mostly addressed to satisfaction. The paper is acceptable from my point of view.\n(Remarks on code availability)\nI don't see the code or a link to the code for this submission."}, "author_response": "Point by Point Response of Nature Communications Manuscript NCOMMS-24-41183B-Z\nReviewer #2 (Remarks to the Author):\nSummary: I appreciate the authors’ efforts in revising the manuscript “Transforming Wearable Data\ninto Health Insights Using Large Language Model Agents”. While the topic is timely and the study\npresents a technically competent integration of LLMs with wearable data analytics, the revised\nmanuscript still does not meet the publication standards and innovation threshold expected by\nNature.\nMajor Concerns:\nComment 1: Lack of Breakthrough Innovation: The manuscript primarily describes an application of\nexisting large language models to a relatively straightforward wearable data analysis scenario. While\ntechnically sound, the integration of LLMs and rule-based or tool-augmented agents for personalized\nhealth analytics is already an established direction in both academia and industry. The manuscript\ndoes not offer a clear conceptual or technical breakthrough beyond what has been presented in\nprevious literature. The novelty is incremental, which limits its significance for a general audience.\nResponse: We thank the reviewer for highlighting the importance of conceptual and technical novelty. We\nrespectfully disagree that the contributions of PHIA are only incremental. To our knowledge, PHIA is the\nfirst open-ended wearable health question-answering system that combines high-resolution daily\nwearable data, an agentic LLM framework, and external tools (code interpreter, web search) to provide\naccurate, numerically precise, and personalized health insights.\nThis differs from prior work in several key ways:\n● Granularity of Data: Unlike PH-LLM [1], which operates only on pre-aggregated 30-day\nsummaries, PHIA directly reasons over raw daily time-series data, a more challenging and\nclinically meaningful setting.\n● Architecture: Previous LLM approaches rely solely on in-model reasoning (Englhardt et al. [2],\nCosentino et al. [1]). PHIA uniquely integrates an iterative ReAct agent framework with external\ntools, overcoming fundamental limitations in LLMs’ numerical reasoning, which we substantiate\nthrough empirical evaluation.\n● Task Scope: PHIA expands beyond targeted coaching (fitness/sleep) to answer thousands of\ndiverse factual and open-ended health queries across exercise, sleep, and lifestyle contexts.\n● Empirical Results: Our evaluations demonstrate that PHIA outperforms strong baselines,\nincluding PH-LLM (0%), GPT-4 with chain-of-thought prompting (53.6%), and code-only or\nreasoning-only approaches, achieving 84% accuracy on objective queries.\n● Public dataset and benchmark: We release the first public open-ended and close-ended\nwearable QA datasets as well as corresponding wearable users for the research community to\nbenchmark future language language models.\nWhile we acknowledge that PHIA does not introduce a new core ML algorithm, we emphasize that\nengineering and validating an agentic LLM framework for this new, high-stakes domain represents\na substantial and non-trivial technical contribution, similar in spirit to recent impactful systems such\nas Med-PaLM [3,4] and Co-Scientist [5].\n[1] Cosentino, Justin, et al. \"Towards a personal health large language model.\" arXiv preprint\narXiv:2406.06474 (2024).\n[2] Englhardt, Zachary, et al. \"From classification to clinical insights: Towards analyzing and reasoning\nabout mobile and behavioral health data with large language models.\" Proceedings of the ACM on\nInteractive, Mobile, Wearable and Ubiquitous Technologies 8.2 (2024): 1-25.\n[3] Singhal, Karan, et al. \"Large language models encode clinical knowledge.\" Nature 620.7972 (2023):\n172-180.\n[4] Singhal, Karan, et al. \"Toward expert-level medical question answering with large language models.\"\nNature Medicine (2025): 1-8.\n[5] Gottweis, Juraj, et al. \"Towards an AI co-scientist.\" arXiv preprint arXiv:2502.18864 (2025).\nComment 2: Assessment of Real-world Impact: The authors do not provide convincing evidence that\nthe agent's insights result in meaningful improvements in users’ health understanding or behavior.\nUser studies, clinical validation, or real-world deployments demonstrating actual utility and adoption\nare missing. The evaluation largely relies on curated datasets and synthetic benchmarks, which do\nnot fully capture the complexity and variability of real “in-the-wild” wearable data.\nComment 3: Limited Medical Depth and Clinical Relevance: Although the system can handle a range\nof queries, its capability to address clinically meaningful health questions is inherently limited by both\nthe wearable data and the scope of the LLM. Expert-level queries, nuanced medical conditions, and\nintegration with broader patient health records are not considered. This limits the depth and impact\nof the work, particularly for a journal like Nature.\nResponse: We thank the reviewer for stressing the importance of real-world validation and clinical\nrelevance. We fully agree that long-term user studies and clinical trials are essential to demonstrate\nbehavioral and clinical outcomes. While such trials are outside the scope of this current system-\nfocused manuscript, we explicitly acknowledged this point in the Limitations of our manuscript during\nprevious rounds of revision. :\n“We also acknowledge that we did not evaluate PHIA through real-world deployment studies\nin order to evaluate potential impacts on behavior change and other health outcomes.\nFurther clinical trials or user studies would be necessary to validate the practical impact of\nPHIA’s recommendations.”\nTo mitigate the reviewer’s concern, we conducted additional validation experiments beyond synthetic\ndata: In addition to synthetic datasets (chosen for privacy, data sharing, and reproducibility), we\nconducted a 480-hour supplementary study with real-world wearable users (Figure 26 at\nSupplementary Materials). The results mirrored our synthetic findings, reinforcing that PHIA’s\nperformance is not an artifact of data generation.When employing multi-step iterative reasoning and\nplanning, code generation, and web search tools, PHIA significantly outperformed baseline models\nin reasoning accuracy, relevance, interpretation, and personalization.\nConclusion for Reviewer #2:\nWe respectfully contend that PHIA represents both a technical and conceptual advancement in\nwearable health AI. While it does not introduce a brand new ML algorithm, its validated\nintegration of LLM agents, tool use, and granular wearable reasoning is novel, impactful, and\nreproducible. We agree that future clinical validation is critical, and we have clarified this scope\nin the manuscript. We hope these revisions and clarifications address your concerns.\nReviewer #3 (Remarks to the Author):\nComment 1: I thank the authors for making substantially more efforts in this revision to address my\nprevious concerns. In this revision, my concerns on dataset scale, technical novelty, and necessary\nbaselines are mostly addressed to satisfaction. The paper is acceptable from my point of view.\nResponse: We sincerely thank the Reviewer #3 for recognizing the substantial revisions made\nand for finding the manuscript now acceptable. We appreciate the acknowledgment that\nconcerns on dataset scale, novelty, and baselines have been addressed to satisfaction.\nComment 2: I don't see the code or a link to the code for this submission.\nResponse: We apologize for the confusion. We release the full codebase (including PHIA agent\nframework, evaluation question datasets, visualization scripts) at\nhttps://github.com/yahskapar/personal-health-insights-agent. The corresponding GitHub\nrepository link will be added to the final version under the “Code Availability” statement. This\nrepository enables other researchers to reproduce our results, as well as evaluate their own\nmodels on the benchmark datasets we created."}]

Decision letter