NutriBench: A Dataset for Evaluating Large Language Models in Nutrition Estimation from Meal Descriptions
Authors: Mehak Preet Dhaliwal
Andong Hua
Laya Pullela
Ryan Burke
Yao Qin
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Nutrition Estimation
Dataset and Benchmark
AI for healthcare
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Accurate nutrition estimation helps people make informed dietary choices and is essential in the prevention of serious health complications. We present NutriBench, the first publicly available natural language meal description nutrition benchmark. NutriBench consists of 11,857 meal descriptions generated from real-world global dietary intake data. The data is human-verified and annotated with macro-nutrient labels, including carbohydrates, proteins, fats, and calories. We conduct an extensive evaluation of Nutribench on the task of carbohydrate estimation, testing twelve leading Large Language Models (LLMs), including GPT-4o, Llama3.1, Qwen2, Gemma2, and OpenBioLLM models, using standard, Chain-of-Thought and Retrieval-Augmented Generation strategies. Additionally, we present a study involving professional nutritionists, finding that LLMs can provide comparable but significantly faster estimates. Finally, we perform a real-world risk assessment by simulating the effect of carbohydrate predictions on the blood glucose levels of individuals with type 1 diabetes. Our work highlights the opportunities and challenges of using LLMs for nutrition estimation, demonstrating their potential to aid professionals and laypersons and improve health outcomes. Our benchmark is publicly available at: https://mehak126.github.io/nutribench.html

Published as a conference paper at ICLR 2025
NUTRIBENCH: A DATASET FOR EVALUATING LARGE LANGUAGE MODELS ON NUTRITION ESTIMATION FROM MEAL DESCRIPTIONS
Andong Hual. Mehak Preet Dhaliwal
Laya Pullela'
Ryan Burke
Yao Qin²
University of California, Santa Barbara {dongx1997 mwhaliiiiiii linal, lpullela, yaoqin edu, ryan. e burke moaixe/@mmail....mmm com
ABSTRACT
Accurate nutrition estimation helps people make informed dietary choices and is essential in the prevention of serious health complications. We present Nu- TRIBENCH, the first publicly available natural language meal description nu- trition benchmark. NUTRIBENCH consists of 11,857 meal descriptions gen- erated from real-world global dietary intake data. The data is verineverified and annotated with macro-nutrient labels, including carbohydrates, proteins, fats, and calories. We conduct an extensive evaluation of NUTRIBENCH on the task of carbohydrate estimation, testing twelve leading Large Language Mod- els (LL.Ms), including GPT-4o, Llama3.1, Qwen2. Gemma2, and OpenBioL_LM models, using standard, Chain-of-Thouge and strategies. Additionally, we present a study involving professional nutritionists, finding that LLMs can provide comparable but significantly faster estimates. Fi- nally, we perform real-world risk assessment by simulating the effect of car- bohydrate predictions on the blood glucose levels of individuals with diabetes. Our work highlights the opportunities and challenges of using LLMs for nu- trition estimation, demonstrating their potential to aid professionals and layper- sons and improve health outcomes. Our benchmark is publicly available at: htttps://mehak/ 126, giitub.io/nutrbbnnch..nt
Generation
INTRODUCTION
Effective nutrition monitoring and dietary management are essential components of healthcare, closely linked to the prevention and control of chronic diseases, including obesity, heart disease, diabetes, and cancer (Amiri al., 2019, Min et al. 2019; Rollo et al. 2016). For instance, in- dividuals with diabetes must accurately estimate the carbohydrate content of meals to determine appropriate insulin doses (Contreras al. 2023 Buck et al. 2022). Inaccurate carbohydrate esti- mation can result in high blood sugar (hyperglycemial or low blood sugar (hypoglycemia) both of which can lead severe short- and long-term health complications (Cologne & in Health Care
IQWIG Sabarathinam 2023).
Despite technological advancements in dietary assessment methods, nutrition estimation from self- reported dietary intake has shown limited accuracy and high user burden (Rollo et al.) 2016 Taylo al. 2021: Tosi al., (022)) A major limitation is that most modern nutrition datasets typically include tabular data (USDA
2018 8 Li 2019 M019Stattttal Open Food Facts FoodDB, The Metaholomics Innovation Centre) or meal images paired with nutrition information (Contreras
et 2023] Ruede et al. 2021: Liang & Li. Taii et al.] 2023 Thames et al.) 2021 Nair et al. 2023 Ma et al., (023) but often lack natural language descriptions, restricting their usage and flexibility.
For example, tabular database searches generally require exact match for successful retrieval and multiple searches for meals with multiple food items, making the process time-consuming and
burdensome (Hezarjaribi systems are restricted to real-time predictions, pose privacy concerns (Hezarjaribi et al. 2017) and may encounter issues with food components being obseured in the image.
al.] 2017). In addition, image pooessing-basedd nutrition estimation
"Equal contribution, alphahetically ordered.

---

Published as a conference paper at ICLR 2025
At lunch, I treated myself to cup of bottled water, a piece of min-crust pepperoni pizza from cafateria, satisfying cup of chocolate milk that was low in suger.
Total carbohycrates: 36g
LLM
The meal consists of cup of bottle water. piece of thin-crust peoperoni cup
pizza, cup of chocolate piece of thin chust pepperoni pizza has approximately 30g carbs. cup of
has (g
lov-sugar chocolate milk has about 24g carbs. So the total carbs the meal (0 30 24) 54g
CoT
Retrieved Context
Food: Water, bottled, plain
Total carbohydrates: 26.39g
cup (240.0g) 0.0g
RAG
Food: Traditional thin crust
consists cup of bottled water, place of thin-orust
pepperoni pizza
pepperoni pizza, and cup of chocolate milk 1 cup of bottled water has carbs.
100g 28.39g carbs
(assuming standard serving of 100g) has 28 39g carbe. cup of chocolate milk (248. 0g) has 26 48g carbs. So the total carba in the meal = (0.0 28.30 26, 40) = 88g
thin-orust papparoni
Food: Chocolate milk drink 100g 10.68q carbs
RAG+GoT
cup (248. Og) 48g carba
Figure 1: GPT-4o-mini answers a query from NUTRIBENCH using different prompting strategies.
In contrast, using natural everyday language to describe meals provides greater flexibility, enabling users to detail various meal components and serving sizes without the constraints of real-time input. This also allows users to record meals in advance for planning retrospectively for tracking. We propose that Large Language Models (LLMs) are valuable tools for nutrition estimation from natural language meal descriptions due their advanced language understanding and reasoning capabílities, vast internal knowledge and ability to refer to external sources to provide precise nutrition estimates. However, there are currently no existing datasets available evaluate LLMs on this task.
To bridge this gap, we present NUTRIBENCH, a dataset comprising 11,857 natural language meal descriptions generated by GPT-4o-mini, derived from real-world global dietary intake data. NU- TRIBENCH is human-verified and annotated with macronutrient labels. including carbohydrates, proteins, fats, and calories. To our knowledge, NUTRIBENCH is the first publicly available bench- mark for evaluating the performance of L.LMs nutrition estimation from meal descriptions. Nu- TRIBENCH is constructed from dietary intake data from countries, sourced from the What We Eat in America (United States Department of Agriculture) and the FAO/WHO GIFT (Leclercq et al.) 2019) datasets. We generate meal descriptions that include single and multiple food items, specify- ing serving quantities both in precise metric measurements (such as grams) and in natural language terms (such as 'a tablespoon' or 'half a cup') to capture the variety of meal description styles used in the real world.
We evaluate 12 leading LLMs, including open-source models (Llama 3.1 (Dubey ct al. 2024). Llama
3 (Meta 2024), Gemma (Team al., 2024), and the Qwen 2 YYang al. 2024) family), closed- source models (GPT-40 and GP mini (Achiam et al. 2023 )) and a medical domain-specific model (OpenBioLLMM-701 (Ankit Pal 2024)), on the task of carbohydrate estimation from natural language meal descriptions. The evaluation spans four prompting paradigms, including Chain-of- Thought (CoT) (Wei et al.) 2022) and Retrieval- Setreval-Auumenteed Generation (RAG) (Lewis al. 2020). Figure I shows an example of GPT-4o-min's output across different prompting strategies on meal description from NUTRIBENCH.
Additionally, we conduct a study with three professional nutritionists to obtain carbohydrate esti- mates on sample of meal descriptions from NUTRIBENCH. Figure summarizes the performance of the evaluated LLMs and the nutritionists. GPT-4o with CoT prompting achieves the highest accu- racy of 66.82% with 16% the queries answered. Most LLMs provide more accurate and faster estimates than the nutritionists, highlighting their potential as valuable tools for obtaining precise and accessible nutrition estimates from natural language meal descriptions.
Finally, we performed real-world risk assessment by simulating the effeet of carbohydrate esti- mates on the blood glucose levels of Type 1 diabetes patients. An evaluation of blood glucose risk metrics across 44,800 simulations showed that nutrition estimates provided by LLMs can help Type I diabetes patients maintain their blood glucose within safe limits. Our experiments demonstrate the potential of LLMs offer accurate dietary guidance, benefiting both professionals and laypersons and ultimately improving health outcomes.
Our contributions can be summarized as follows:
2

---

Published as a conference paper at ICLR 2025
Basc
COT,
65
* Base, GPT-Go-min
Col, GPT-to-mini
Base, Ulmms11-768
Col, Liamw3.1-708
Uama3.1 4058
Liama3.1 4058
Basc, Liama3 88
CoT, Liama3 88
6asc, Ubma3
Liama3 708
Acc
50
Base, Qwer
Qwen2-78
Base, Qwern 2-728
CoT, Qwwn2-728
Base, Gamma2-98
CoT, Gemma2298
Base, Gemma2-276
CoT, Gemma2-278
OpenBleLUM-708
RAG, GPT-4ommm
RAG-CoT, GPT-4o-min
35
RAG, Lama 1amm3.1-455B-FP
RAG Libma3.1-4348
Humar- -Nttiiionist
85%
90%
95%
96%
97% 96% 93% 100%
Answer Rate
Figure 2: Accuracy and Answer Rate (defined in Section 4 for the evaluated LLMs across different prompting strategies on NUTRIBENCII.
NUTRIBENCH: We present NUTRIBENCH the first publicly available natural language meal description dataset labeled with macronutrient and calorie estimates. NUTRIBENCH consists of 11,857 human-verified meal descriptions derived from real-world dietary intake data from eleven countries spanning across the world.
Benchmarking LLMs:
We evaluate twelve LLMs varying size and expertise covering both open and closed source models with different prompting strategies including CoT and RAG on the task of carbohydrate estimation on NUTRIBENCH. This provides comprehensive insight into the current capabilities of L.LMs in nutrition estimation.
We conducted study with three professional nutritionists who provided carbohydrate estimates for a subset meal descriptions from NUTRIBENCH and measured the accuracy and speed of their predictions against those of the LLMs.
Expert Nutritionist Evaluation:
Real-World Risk Assessment: We conducted study simulating the effect of meal carbohy- drate estimation on blood glucose levels of individuals with type diabetes, demonstrating the real-world impact and potential health benefits of LLM-driven nutrition guidance.
RELATED WORK
Nutrition Estimation. Most mainstream nutrition datasets such as FoodData Central (FDC) (USDA 2019), MenuStat (MenuStat). FoodCom (Li, 2019), and Nutritionix (Nutrition 2018) fea- ture tabular food nutrition information curated from diverse sources. However, tabular datasets based retrieval methods require
precise terminology to ensure successful and relevant retrieval. In addition, these methods do not support retrieving multiple items that may be present in meal in single search, making the process time-consuming and burdensome (Hezarjaribi et al. 2017).
Another popular method for nutrition estimation involves predicting nutritional values from food images. Such approaches may involve identifying food items followed by retrieval from tabular nutrition database (Yunus et al. 2018)) or directly estimating the meal's 's nutritional breakdown from the image (Keller al. 2024). Many existing datasets contain images paired with nutrition informa- tion to facilitate research in this direction. These datasets may include images obtained from the web (Ruede et al. 202T), real world, (Liang & Tai al. 20033 Thames al.) (02T), ). or synthetically generated images Nair et al.) 2023). UMDFood90k (Ma et al. (202)) provide multimodel dataset with product images, text-based ingredient statements, and nutrient amounts. However, image- nutrition approaches
time-sene sensitivee (Hezarjaribi et al. 2017), requiring users to capture specific pictures of their meals at the time of consumption and may encounter issues with food components being obscured in the image.
Enabling users to input meal descriptions in natural everyday language can help mitigate these is- Initial exploration in this realm includes (Korpusik et al. 2017 Taylor al. 2021). who

---

Published as a conference paper at ICLR 2025
use Convolutional Neural Networks match food items from crowdsourced meal descriptions with an external tabular nutrition database. However, their data is not publicly released, preventing the evaluation of current state-of-the-e ahrtth-arttttttr language processing approaches, such as LLMs, for this task.
Motivated by the lack of a standardized benchmark for nutrition estimation based on natural lan- guage meal descriptions, introduce NUTRIBENCH, the first such publicly available benchmark, and evaluate offee- LLMs to gain insight into their current capabilities and limitations.
Large Language Models (LLMs). Large Language Models (LLMs) have made significant progress in recent years, with both closed-source models like GPT-4 (Achiam et al. (023) and open-source models like Llama 3 and 3.1 @Dubey al.] 2024), Gemma 2 [Team al.) 2024), and OpenBi- aLLMAnkit Pal 2024) enabling advancements in natural language processing as well as knowledge- intensive, reasoning, and cross-domain tasks including healtheare applications (Gramopadhye al.)
2024). Despite their extensive internal knowledge, LLMs still suffer from issues such as hallucina-
tions, incorrect or outdated information, and lack of interpretability (Lin al., 2021 Li et al.)
2023
Zhao et al.) 2024).
Chain-of-Thought (CoT) (Wei et al. 2022 prompting alleviates some these issues by enabling
models to reason about the answer step-by-sted Another promising solution is Retrieval-Augmented Generation (RAG) (1 (Lewis et al. 2020p, which provides the model with additional context relevant to the query by retrieving information from an external knowledge source. While previous works have identified shortcomings of both these approaches (Chen et al. 2024 Yao et al.) 2024), they have not been assessed the task of nutrition estimation, which may uncover unique challenges.
Our work comprehensively evaluates 12 state-of-ther of-the-art LLMs with standard, CoT, RAG, and RAG combined with CoT prompting for carbohydrate estimation from natural language meal descriptions. We present a detailed analysis of how different prompting strategies affect LLMs' performance in nutrition estimation in Section 5
NUTRIBENCH CONSTRUCTION
NUTRIBENCH derived from two source databases: What We Eat in America (United States De- partment of Agriculture) and FAO/WHO GIFT (Leclereg et al. 2019). which collectively provide real-world dictary intake data from 11 countries, including Argentina, Bulgaria, Ethiopia, India, Italy, Mexico, Nigeria, Peru, Philippines, Sri Lanka, and the United States. After processing the meal data with nutrition labels, we instruct GPT-4o-mini to generate natural language meal descrip- tions to construct NUTRIBENCHI Examples of the meal descriptions can be found in Appendix [
3.1 WHAT WR EAT IN AMERICA
The What We Eat in America (WWEIA) survey is national food survey jointly conducted by the U.S. Department of Health and Human Services (HHS) and the U.S. Department of Agriculture (USDA). Trained interviewers collect dietary intake data, including food names and portions through two 24-hour dietary recalls. As of September 2024, WWEIA has completed 10 surveys from 2001 to 2020, compiling total of 2,070,592 food items. To obtain nutrition labels, we utilize the Food and Nutrient Database for Dietary Studies (Montvilee et al. 2013) comprehensive database that provides nutrient values for foods and beverages within the W WWEIA data.
We cleaned the WWEIA data by removing entries with missing or invalid information (e.g.. "Quan- tity not specified" portion sizes) and integrated data across the years into one comprehensive dataset. Since both the recorded food names and nutrition labels from the FNDDS change across the years, we use the 2017-2018 dataset as our reference p Nutritional labels for each food item are calculated and normalized based on nutrition information derived from this reference data. Entries from the WWELA that contain food names not found in the reference dataset have been excluded from analysis.
3.2 FAO/WHO GIFT
The FAO/WHO GIFT (Leclereg et al. 2019) is an open-access online platform by the Food and Agri- culture Organization of the United Nations that provides individual food consumption data compiled from global national and sub-national food consumption surveys.
'The FNDDS 2017-2018 dataset is the most recent available data that aligns with the WWEIA survey prior the pandemic.
4

---

Published as a conference paper at ICLR 2025
We compiled meals from 10 countries based on the availability of indicators that allowed to separate food items into distinct meals (such as eating timestamps and meal markers), nutritional data, and geographical diversity. The countries displayed in Figure For each meal, obtained the food items, their quantities in grams, and corresponding nutritional information. Since we aim generate natural language meal descriptions with serving sizes expressed in everyday terms, converted the food quantities from grams to common serving units. This was done by mapping the food items in the FAO/WHO GIFT dataset The FoodData Central (FDC) (USDA) 2019) dataset and applying rule-based algorithm (Appendix F to translate the amounts in grams into natural serving sizes. FDC is the food composition information center of USDA (Fukagawa et al., (022) providing detailed information foods, including conversions from grams to natural servings. To avoid bias, we also considered meals where at least one item could not be mapped to an FDC item, keeping the original gram amounts.
Combining Food Items to Meals Food items in WWEIA FAO/WHO GIFT are recorded in- dependently. Therefore, we merge food items into single meal if they are consumed by the same person on the same day during the same eating occasion. For instance, meal may include "a slice of buttered toast, cup of black coffee, and half grapefruit" consumed together. For each meal, we maintain either natural or metric serving arres-ensuring there is no mix-up between the two. We compile 5,532 meal descriptions with natural servings, alongside the same 5,532 descriptions con- verted to metric servings from the WWEIA database. Additionally, Table summarizes the number of meals and unique food items sampled from each country from the FAO/WHO GIFT.
Country
Meals
Food items
(1) Argentina
70
67
(2) Bulgaria
26
39
(3) Ethiopia
22
20
(4) India
18
16
(5) Italy
141
124
(6) Mexico
181
254
(7) Nigeria
124
86
(8) Peru
93
92
(9) Philippines
84
104
Figure 3: The countries from which meals were extracted from the FAO/WHO GIFT. The indices on the map correspond to the countries in Table
(10) Sri Lanka
34
43
Table 1: Unique meals and food items sampled
from each country from FAO/WHO GIFT
3.3 GENERATING NATURAL LANGUAGE MEAL DESCRIPTIONS
GPT-4o-mini Based Generation. To obtain natural language meal descriptions from the collected meal data at scale, instruct GPT-4o-mini to generate these descriptions, which then used to create NUTRIBENCH To encourage diversity, we prompt the LLM to produce five varied meal descriptions for each food item single generation, from which we randomly select one as the final query. Details of the prompts used are in Appendix B.I
Human Verification. Human verification is conducted ensure that the meal descriptions gener- ated by the LLM contain no hallucinations or missing information. One of the authors acts as the verifier, confirming that the food names and serving sizes are accurately included in the final meal descriptions to match the nutrition labels. We identified two common mistakes made by GPT-4o- mini: missing food names and missing food servings. Additionally, we observed that GPT--4ommii has a higher probability of omitting food names when meal contains multiple food items and tends to make more mistakes generating descriptions with natural serving sizes compared to metric serv- ing sizes. For example, meal description prior to human verification, "During lunch, I had tasty medium crust pepperoni pizza and washed it down with a carton of 100% orange juice." was updated to, "During lunch, had piece of tasty medium crust pepperoni pizza and washed it down with carton of 100% orange juice.", after verification (note the change from a whole pizza to a single piece). More examples of this verification process displayed in Appendix B.2
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
LLM Models We conduct comprehensive evaluation of twelve state-of-the-art large language models (LLMs) using NUTRIBENCH on the task of carbohydrate estimation from meal descrip- tions. The evaluation spans a range of open-source models, including Llama 3.I-8B, Llama 3.1- 70B, Llama 3. 3.1-405B-FP8, Llama 3-8B, Llama 3-70B (Meta] 2024| Dubey et al. 2024), Gemma
2-9B, Gemma 2-27B (Team et al. 2024), and Qwen 2-7B, Qwen 2-72B (Yang et al. 2024]. We also
include closed-source models such as GPT-4o and GPT-4o mini (Achiam et al. 2023), well the medical domain-specific model OpenBioLLMM-0E (Ankit Pal 2024).
Prompting Methods We adapted four existing prompting methods with carefully designed prompts tailored for carbohydrate estimation.
Base: The first baseline prompt involves instructing LLMs estimate the carbohydrate tent based on the meal description provided in the query with basic instructions.
Chain-of-Thought (CoT) (Wei et al. 2022 y: Due to the complexity of meal descriptions with multiple items in varying quantities, we hypothesize that the step-by-step reasoning induced by chain-of -fthouhht prompting would reduce model errors by enabling the model to identify and reason about individual query components required to make the overall decision.
Retrieval-Augmennte Augmented Generation (RAG) (eewis et al. 2020): To further enhance the relia- bility of LLM predictions, we use RAG to ground their generations with nutrition knowledge retrieved from nutrition database. First, for a given meal query, we prompt the model to parse into individual food components. Next, we retrieve nutrition information about each food item in the query through a nearest neighbor semantic similarity search and concatenate the results to form a comprehensive set of facts about the food components in the query. Finally, we provide this retrieved context along with the original prompts for LLMs.
RAG+CoT: We combine the nutrition retrieval capability of RAG with step-by-step reasoning in CoT by concatenating the retrieved nutrition context with the CoT prompting for LLMs.
In all the cases above, we instruct the models to respond with '-1' they don't know the answer to reduce the risk of potentially harmful predictions.
Figure I shows the predictions generated by GPT-4o-mini using the four prompting paradigms. We evaluate RAG and RAG+CoT with GPT-4o-mini, Llama3.1-70B. and Llama3. 1-405B-FFF only due to computation constraints. The prompts used for each method and hyperparameters can be found in Appendix C
Evaluation Metrics We calculate the mean absolute error (MAE) to measure the deviation of the model responses from the true meal carbohydrates. addition, we report accuracy (Ace@7.5) by considering the model output "correct" if the predicted value is within +7.5g of the ground truth. This is based on the
corbbooyyrra ratio, which indicates the grams carbohydrates one unit of insulin can cover. While this ratio varies among individuals. it is generally considered 1:15 as a rule of thumb Since we aim to improve insulin management, we maintain a conservative threshold of 7.5g the absolute error to measure accuracy.
Finally, since we allow the models not to provide an estimate uncertain, we also report the answer rate (AR), indicating the percentage of answered questions. Overall, the models should have high answer rate and Acc@7.55 7.5, and low MAE.
5 RESULTS
Figure presents the accuracy and answer rate of the 12 evaluated large language models across four prompting paradigms. Across all models and methods, GPT-4o with Chain-of-Thought (CoT) prompting achieves the highest accuracy of 66.82%, with an answer rate of 99.169% Among open- source models, Llama 3. 3.1-405B-FFPP with RAG+CoT prompting performs the best, with accuracy of $9,899 99.89 and an answer rate of 96.0%.

---

Published as a conference paper at ICLR 2025
22
Gemma2-90
Gamma2-278
5000
Fannlline feer
20
Ower2-728
Qwen2-78
-18
Limma3.140558 40558-FF8
1-78
Count
3000
-14
Lama33 708
Lamm3 8B
2000
12
Ope
GPT-4o-mini
1000
6PT-4o
10
25
Diferencee Natural vs. Metric Servings
Food Number
Figure 4: Difference MAE between natural and metric serv- Figure 5: Frequency of queries and ing descriptions. Natural servings are preferred by most mod- model error with increasing number els (orange), whereas the GPT family prefers metric (blue).
of foods in descriptions.
5.1 IMPACT OF SERVING DESCRIPTIONS, PROMPTING STRATEGIES, AND RAG
Natural Serving Descriptions are Preferred by Most Models. We compare the performance of models on NUTRIB ENCH with natural versus metric serving descriptions. Figure&showe the differ- ence in MAE for each model between natural and metric serving descriptions. We observe that most models exhibit higher MAE when using natural serving measures (e.g.. "a cup of rice"), compared to metric measures (e.g.. "80g of rice"). Notably, both GPT-4o and GPT-4o mini outperformed in the metric serving category. We hypothesize that this may be attributed the inclusion of nutrition information associated with metric servings in the training data for the GPT family.
CoT Improves both Answer Rate and Accuracy, Especially on Complex Meal Descriptions. Tabeee[preeents the answer rate and accuracy across the twelve models NUTRIBENCH using Base and CoT prompting. We observe that CoT prompting significantly improves performance across both metrics. Furthermore, while larger models generally outperform their smaller counterparts, as shown in Figure 2 CoT prompting allows smaller models to exceed the performance of larger ones. For example, GPT-4o mini with CoT prompting achieves higher accuracy and answer rate than GPT-4o with Base prompting. This indicates that the step-by-step reasoning facilitated by CoT prompting improves the accuracy, demonstrating that even smaller models can achieve competitive superior performance when paired with effective prompting strategies.
Looking more closely, we discover that CoT prompting helps mitigate increasing errors in complex meal descrip- the meal descrip-
Table 2: Accuracy and answer rate aver- aged over 12 models for Base and CoT.
tions. Figure presents a histogram tions in NUTRIBENCH with increasing food items and the corresponding MAE for GPT-4o under Base prompting (red) and CoT prompting (green). There is sharp in- crease in the MAE with an increasing number food items in the meal for Base prompting. CoT prompting significantly mitigates the error. This anal- ysis demonstrates that enabling the model to parse meal descriptions into individual food items and perform explicit calculations through step-by-ster reasoning improves the accuracy of LLMs, particularly for more complex queries.
Base
CoT
Ace @ 7.5 Answer Rate
43.24%
47.46%
95.59%
97.14%
RAG Does Not Always Improve Performance. In our experiments with Retrieval-Augmented Generation (RAG), we built a retrieval database, RETRI-DB, using data from FDC (USDA 2019). First, we parsed meal descriptions into individual food items and retrieved relevant nutritional infor- mation from RETRI-DE DDB. This data was then used as context to prompt the LLMs for carbohydrate estimation. Figure Dstorssssswwss input query. Further details on RETRI-DB construction are Appendix D
shows an example of nutritional information retrieved for two food items in the
We evaluate the impact of RAG on GPT-4o-mini, Llama3. 1-405B-FP8, and Llama3.1-70B. Figure shows the accuracy and answer rates across four prompting methods for queries involving natu- ral and metric serving descriptions. For metric-serving queries, RAG and RAG+CoT significantly outperform the Base and CoT for Llama3.1, indicating that the additional information retrieved for RAG provides valuable context that improves the accuracy of their predictions. However, for GPT- 4o-mini, RAG shows only minor improvement over Base, while RAG+CoT performs worse than COT. Similar hypothesis regarding the impact of natural versus metric serving sizes, hy-
7

---

Published as a conference paper at ICLR 2025
pothesize that this may be due metric serving nutrition information being included in the training of GPT-4o models. Therefore, RAG may not provide additional information to improve outputs but could misguide the original prediction by potentially noisy retrieval results.
On the other hand, for nat- ural serving queries, while RAG shows some improve- ment over Base for the
Serving
Serving
70
2003
Basn, GPT-4o-min
Base, Liama 1amm3.1-106
05-
Liama3. 4058
GPT44ommin
Aco87.5
Acc@1.5
80
CoT, Lama2.1 708
Llama 3.1 models, CoT- only always outperforms RAG+CoT. An inspection of RETRI-DB reveals that for most food items, the nutritional data available in the retrieved context is for standardized metric 100g servings. We hypothesize that the LLMs are unable to successfully utilize this information when the food amounts in the query described using natural terms. This suggests that alignment between the RAG database and the queries crucial for improving results. A promising future direction is to augment RETRI-DB with more natural servings for RAG support
CoT, Limm33114058--P0
GPT-4c
50
Llamx3.1-708
RAG, Limma3:1-400B-4
40-
45
RAG4CaT cP GPT-4o-mini
Liama
90 Rate
RAG 3.1-4058-FF8
Arrwwer Rabe
Figure 6: Comparison of model performance with RAG on natural and metric serving queries
predictions with LLMs.
5.2 VARIATION IN LLM PERFORMANCE ACROSS DIETS AND CULTURES
We analyze whether there are differences in the performance of LLMs based on dietary pat- terns and cultural contexts.
1000
11
1000
Carda Por Wel
900
Correlation between meal carbohydrates and predic- tion error. Figure (Left) examines the relationship between meal carbohydrate content and prediction error. We plot the frequency and MAE by GPT-4o with CoT prompting for varying carbo- hydrate content of single-item meals in NUTRIBENCH. We focus on single-food item meals for this analysis avoid confounding the results with the effect of increasing food items in the meals. Our analysis reveals a positive correlation: the MAE increases as the carbohydrate content in the meal rises. This suggests that LLM predictions are likely to more accurate for individuals with generally now-carbohydrate diet compared to those following a cigh-carbohydrate diet.
200
10
25
53
75
135
100
Figure 7: Left: Food with higher carbohydrates tends to have larger MAE error. Right: Performance disparities across differ- ent countries in NUTRIBENCH.
Performance discrepancy across different countries. We also measure the performance of LLMs on meal descriptions generated from various countries. Figure 7 (Right) displays the MAE for GPT-4o with CoT prompting across different countries in NUTRIBENCH. We observe significant disparities in the performance across these countries, with the lowest error of 2.20 on meals from Nigeria and the highest error of 15.12 on meals from Sri Lanka. Further analysis indicates a cor- relation between the average carbohydrate content per meal for these countries (blue bars in Figure 7 (Right) and the MAE, with countries with a higher MAE generally tending to contain meals with higher carbohydrates. This pattern indicates that model performance may be influenced by the culinary variety and dietary habits of different cultures, such as the carbohydrate content in meals. Overall, findings suggest that L.MMs perform better for meals associated with certain cultural contexts and cuisines, highlighting the need for more equitable training practices that ensure diverse cultural and dietary representation.
5.3 FINE-TUNING WITH FDC DATA
To fine-tune our model, we use the FDC database to convert individual food items into natural lan- guage meal descriptions. This process yields dataset of 600K meal descriptions (details provided
8

---

Published as a conference paper at ICLR 2025
in the Appendix). We then apply the Base prompting method generate responses for all the meal descriptions, which serves as our training data. We select Gemma2-27B as the base model because offers similar performance to Llama3. 1lama3..-405BBFF... 1-405B-FP8 while being significantly smaller in size, making more suitable for fine-tuning. To fine-tune the model, we utilize qLoRA (Dettmers et al. 2024 with rank of 8, training for one epoch across A100 GPUs.
Table 3: Fine-tuning the LLM with nutrition data significantly improves MAE (lower is better), accuracy and answer rate (higher is better). Our fine-tuned model denoted as Gemma2-27B-FT.
Model
GPT-4o CoT
Gemma2-27B CoT
Gemma2-27B
Gemma2-27B-FT
Prompting Method
Base
Base
MAE
8.61
13.74
13.33
9.57
Acc @ 7.5
66.82%
49.35%
45.61 %
61.71% %
Answer Rate
99.16%
96.79%
88.65%
100%
As shown in Table 3 fine-tuning Gemma2-27B significantly improves performance over pretrained
Gemma2-27B by reducing MAF, while also increasing both accuracy and answer rate. However, Gemma2-27B-FT still underperforms GPT-4o with CoT prompting, highlighting GPT-4o's superior capability in carbohydrate estimation. This suggests that fine-tuning models from the GPT family could be a promising direction for future work achieve an LLM-based nutritionist.
LLMS ACHIEVE COMPARABLE ACCURACY TO NUTRITIONISTS BUT SIGNIFICANTLY OUTPERFORM IN SPEED
We invited three professional nutritionists in a voluntary study to provide carbohydrate estimates for subset of meal descriptions from NUTRIBENCH. We sampled 72 meal descriptions, including both natural and metric servings from the U.S. subset, since the nutritionists are based in the United States. For a fair comparison, we explicitly instructed the participants not to search nutrition apps or the web for carbohydrate estimates. We also provided them with three meal descriptions and their carbohydrates, identical to the few-shot examples given in the L.M prompts.
Notably, as shown in Figure 2 we find that several LLMs provide more accurate carbohydrate estimates than the nutritionists for the same 72 queries. Specifically, as detailed in Appendix LL] GPT models excel in estimating carbohydrates for complex, multi-comppneen meals and those with detailed measurements. In contrast, nutritionists perform better when estimating simpler, traditional meals that lack specific brand information. Additionally, the nutritionists report an average of 43 minutes spent estimating responses for all 72 queries. In contrast, LLMs can answer all 72 queries in just a few minutes, with GPT-4o-miní completing them in only 2 minutes.
These findings suggest that LLMs provide accurate carbohydrate estimates that can assist nutrition- ists in their work, while also significantly reducing the time required to process queries. However, when given access to nutritional lookup resources, nutritionists can achieve results comparable to GPT models, shown in Appendix L the speed and accuracy of nutrition assessments, making them valuable tools for both professionals and non-professional..
This highlights the significant potential of LLMs to enhance
RISK ASSESSMENT
We present real-world risk assessment demonstrating the impact of nutrition estimation by sim- ulating the effect of meal carbohydrate predictions on the blood glucose levels of individuals with Type diabetes (TID). To do this, we use the Tidepool Data Science Simulator [ The simulator includes a patient metabolism model that emulates how a virtual patient's blood glucose will change over a period of time in response external events (e.g.- carbohydrate insulin intake). It also includes a pump simulator running the FDA-cleared Loop algorithm (Tidepool) that delivers insulin the virtual patient in response to the blood glucose levels and reported carbohydrate intake or additional insulin administration. In real-world scenarios. individuals with Type diabetes either use such pumps or
insulin to manage their blood glucose.
The patient metabolism model is based on three key parameters: the Insulin Sensitivity Factor (ISF), Carb Insulin Ratio (CIR), and the basal rate. We ran simulations for 20 virtual patients using CIR,

---

Published as a conference paper at ICLR 2025
ISF, and basal rates derived from real, anonymized data from Tidepool. Each simulation modeled scenario where a virtual TID patient consumed a meal at the start. The patient's ' metabolic response was influenced by the actual carbohydrate content of the meal. We then simulated how the patient would manage glucose using estimated carbohydrates provided by nutritionists or L.LSs to calculate insulin doses input into their pump. Two scenarios patient, not using a pump, calculates their insulin dose manually based on estimated carbs 'x' and their CIR (using the equation x/CIR), and (2) the patient, using pump, enters estimated carbs into the pump, which adjusts the insulin dose automatically.
explored for each simulation: (1) the
We simulated scenarios where pa- tients obtained carbohydrate esti- mates for meals from nutritionists and GPT-4o. To capture wide range of real-world conditions, we also varied the patient's blood glu- cose at meal intake, setting it at 80, 120, 160, and 200 mg/dL. In total, across 20 virtual patients, 4 carbohy- drate estimation methods, 4 starting glucose levels, 2 simulation setups, and 70 meals, we conducted 44,800 simulations. Each simula-
Table 4: Simulation results: Carbohydrate estimates by GPT-4o with CoT prompting lead to the highest TIR, low- est %TBR, and lowest BGRI
Carb Estimator
%TRR
TTBR
%TAR
BGRI
Nutritionist 1
66.97
8.79
24.23
24.01
Nutritionist 2
66.88
6.76
26.36
22.48
Nutritionist 3
65.93
7.64
26.43
20.81
GPT-4o (CoT)
69.88
4.18
25.94
17.83
tion ran for hours, capturing the following blood glucose metrics:
% Time in Range (TR): Percentage of time blood glucose remained within the safe range of 70-180 mg/dL.
% Time Below Range (% TBR): Percentage of time below 70 mg/dL.
% Time Above Range ( % TAR): Percentage of time above 180 mg/dl..
Blood Glucose Risk Index (BGRI): A combined measure of high and low blood glucose risk,
calculated using the blood glucose risk curve defined by Clarke & Kovatchev (2009).
presenss the metrics across the different simulation scenarios based on meal carbohy- drate predictions from nutritionists and LLMs. We find that carbohydrate estimates made by GPT-4o with CoT prompting result in the high- est TIR and the lowest TBR. Although Nutri- tionist 1 estimates lead the lowest TAR, low blood glucose poses greater medical risk than higher levels (Clarke Kovatchev 2009). Ac- cordingly, the overall blood glucose risk is sig- nificantly lower with GPT-4o's s estimates com- pared both nutritionists. Figure shows sim- ulated glucose traces for a pump-using patient consuming a meal at 0. Predictions by Nutri- tionists 1-3 and GPT-4o (with CoT prompting) are used in the four scenarios. We observe that while glucose remains within the safe range for simulations with GPT-4o and Nutritionist 3, those involving Nutritionists and drop below range, particularly for Nutritionist 1.
Bloed Glucose Level Level (mgidd)
200
Nunthionist
Nutkisnist 1
175
Natrkionist
GPT44
150-
125
1000
75-
25-
0h
2h 3th
4h
dh
Time
Figure 8: Simulated glucose traces for a virtual patient using pump with carbohydrate estimates from GPT-4o and nutritionists. The safe glucose range (70-180 mg/dl) is in green, low (<70 mg/dl) in red, and high (> 180 mg/dl) in yellow.
This case study indicates that LLMs can be valuable tools in real-life healthcare settings, helping to prevent risky outcomes and promote better health.
CONCLUSION
We present NUTRIBENCII, the first publicly available benchmark for evaluating the performance of LLMs in nutrition estimation from natural language meal descriptions. NUTRIBENCH comprises 11,857 human-verified meal descriptions, generated from real-world dietary intake data across 11 countries. We conducted extensive experiments to evaluate twelve state-of-the-as LLMs carbo- hydrate estimation, with GPT-4o with CoT prompting achieving the highest accuracy, even surpass- ing professional nutritionists in both accuracy and speed.
10

---

Published as a conference paper at ICLR 2025
9 REPRODUCIBILITY STATEMENT
We describe our data creation process in Section 3 with the serving size conversion explained in Appendix The prompts used for meal generation are provided in Appendix [ and for LLM eval- uation are provided in Appendix C The details of constructing the RAG database and fine-tuning data are provided in Appendix and Appendix respectively.
10 ACKNOWLEDGMENT
We extend our sincere gratitude to Xuan Yang and Yifan Wei for the valuable initial discussions, data survey, and construction efforts for this project. We are also grateful to Xuezhi Wang for the insightful discussion and interpretation of our results. We further thank Naveen Mysore for participating in discussions and developing a demo. Finally, we thank Janice Macleod, Sandra Chmelnik, Susan R. McClendon for providing perspectives and nutrition estimates for the human study as a professional nutritionist. We especially appreciate Susan R. McClendon for providing results for both the look-up and non-look-up conditions and for engaging in discussions. Their collective contributions were essential to the success of our work.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.055 087774 2023.
Hadi Amiri, Andrew Beam, and Isaac S Kohane. Learning to estimate nutrition facts from food descriptions. AMIA, 2019.
Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large lan- guage models for healthcare and life sciences. lattps://hugeingfface cot.a/cee.cc/eoeeeeeeeeeeeeeeeee co/ htt.:/accc..//aaaaaa........
OpenBio -wLBB-ooLM--220 Ll 3 2024.
Sina Buck, Collin Krauss, Delia Waldenmaier, Christina Liebing, Nina Jendrike, Josef Högel, Boris M Pfeiffer, Cornelia Haug, and Guido Freckmann. Evaluation meal carbohydrate count- ing errors in patients with type diabetes. Experimental and Clinical Endocrinology & Diabetes, 130(07):475-48) 2022.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17754 17762, 2024.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint (0Ki::1909902166 2019.
William Clarke and Boris Kovatchev. Statistical tools analyze continuous glucose monitor data. Diabetes technology & therapeutics, 11(S1):S-45, 2009.
Germany: Institute for Quality Cologne and Efficiency in Health Care (IQWiG); 2006-. Hyper-
glycemia and hypoglycemia in type 2 diabetes. InformedHealthh.rg [Internet].
Ivan Contreras, Marti Guso, Aleix Beneyto, and Josep Vehi. Photo-based carbohydrates counting using pre-trained transformer models. PFAC-PapersOnLLiee, 5533: 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances Neural Information Processing Systems, 36, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 4rXiv:2407.2 1783, 2024.
11

---

Published as a conference paper at ICLR 2025
FoodDB, The Metabolomics Innovation Centre. Fooddb: Food database. https://Eooch: ca/ Accessed: 2024-09-30.
Naomi K Fukagawa, Kyle McKillop, Pamela R Pehrsson, Alanna Moshfegh, James Harnly, and John Finley. Usda's fooddata central: what is it and why is it needed today? The American journal of clinical nutrition, 115(3):619 624, 2022.
Ojas Gramopadhye, Saeel Sandeep Nachane, Prateek Chanda, Ganesh Ramakrishnan, Kshi- tij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, and Sachindra Joshi. Few shot chain-of- thought driven reasoning to prompt Ilms for open ended medical question answering. arXiv preprint arXiv:2403.0458 2024.
Niloofar Hezarjaribi, Sepideh Mazrouee, and Hassan Ghasemzadeh. Speech2health: a mobile framework for monitoring dietary composition from spoken data. IEEE journal of biomedical and health informatics, 22(1):252-264 2017.
Matthew Keller, Chi-en Amy Tai, Yuhao Chen, Pengcheng Xi, and Alexander Wong. Nutritionverse- direct: Exploring deep neural networks for multitask nutrition prediction from food images. arXiv preprint arXiv:2405. 07814, 2024.
Mandy Korpusik, Zachary Collins, and James Glass, Semantic mapping of natural language input to database entries via convolutional neural networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3685-5689. IEEE, 2017.
C Leclereq, P Allemand, Balcerzak, F Branca, RF Sousa, A Lartey, M Lipp, VP Quadros, and P Verger. Fao/who gift (global individual food consumption data tool): A global repository for harmonised individual quantitative food consumption studies. Proceedings of the Nutrition Soci- ety, 78(4))484-495, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-aie -aumentl-a genera- tion for Knowledge-intemt inndgge-nteesiveeeeeeeee nlp tasks. Advances Neural Information Processing Systems, 33: 1459-94744 2020.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian Yun Nie, and Ji-Rong Wen. Halueval: A large- scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6469,646, 2023.
Shuyang Li. Food.com recipes and interactions, 2019. URL https::/ www kaggle. com/ ESv/783630
Y Liang and J Li. Computer vision-based food calorie estimation: Dataset, method, and experiment. arxiv 2017. arXiv preprint
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
Peihua Ma, Yixin Wu, Ning Yu, Yang Zhang. Michael Backes, Qin Wang, and Cheng-I I Wei. Vision- language models boost food composition compilation. arXiv preprint arXiv:2306.01747, 2023.
MenuStat. Nutrition data from top restaurant chains. Accessed: 2024-09-30.
AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta Al, 2024.
Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. A survey on food computing. ACM Computing Surveys (CSUR), 52(5):1-36, 2019.
Janice B Montville, Jaspreet KC Ahuja, Carrie L Martin, Kaushalya Y Heendeniya, Grace Omolewa- Omolewa-Tomobi, Lois C Steinfeldt, Jaswinder Anand, Meghan E Adler, Randy P LaComb, and Alanna Moshfegh. Usda food and nutrient database for dietary studies (fndds), 5.0. Procedia Food Science, 212, 2013.
12

---

Published as a conference paper at ICLR 2025
Saeejith Nair, Chi-en Amy Tai, Yuhao Chen, and Alexander Wong. Nutritionversessytt:: An open access synthetically generated 2d food scene dataset for dietary intake estimation. arXiv preprint arXfv:2312.06192, 2023.
API Nutrition. by nutritionix. Retrieved March, 26, 2018.
Open Food Facts. Open food facts: Open data food products from around the world. https: Accessed: 2024-09-30.
Megan Rollo, Rebecca Williams, Tracy Burrows, Sharon Kirkpatrick, Tamara Bucher, and Clare E Collins. What are they really cating? review on new approaches to dietary intake assessment and validation. Current nutrition reports, 5:307-314, 2016.
Robin Ruede, Verena Heusser, Lukas Frank, Alina Roitberg. Monica Haurilet, and Rainer Stiefelha- gen. Multi-task learning for calorie prediction on novel large-scale recipe dataset enriched with nutritional information. In 2020 25th International Conference on Putlern Recognition (ICPR), pp. 40040004444 -008. IEFE, 2021.
Sarvesh Sabarathinam. A glycemic diet improves the understanding of glycemic control in diabetes patients during their follow-up. Future Science OA, 93):FSO843, 2023.
Chi-en Amy Tai, Saeejith Nair, Olivia Markham, Matthew Keller, Yifan Wu, Yuhao Chen, and Alexander Wong. Nutritionverse-rea. An open access manually collected 2d food scene dataset for dietary intake estimation. arXiv preprint arX1v:2401.08598, 2023.
Salima Taylor, Mandy Korpusik, Sai Das, Cheryl Gilhooly, Ryan Simpson, James Glass, and Susan Roberts. Use of natural spoken language with automated mapping of self-reported food intake to food composition data for low-burden real-time dietary assessment: method comparison study. Journal of Medical Internet Research, 23( (3(12)x5 26)2288, 2021.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Quin Thames, Arjun Karpur, Wade Norris, Fangting Xia, Liviu Panait, Tobias Weyand, and Jack Sim. NutritionSk: Towards automatic nutritional understanding of generic food. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8903 8903-8911, 2021. 2021.
Tidepool. Tidepool loop. https: htte:://www... www. http://wwww.eeool..ooo//eeeeeeee 2024-10-02.
-lop
Accessed:
Martina Tosi, Davide Radice, Giulia Carioni, Teresa Vecchiati, Federica Fiori, Maria Parpinel, and Patrizia Gnagnarella. Accuracy of applications to monitor food intake: Evaluation by comparison with 3-d food diary. Nutrition, 84:111018, 2021.
United nhanes.
States
Department
Agriculture.
Overview of wweia and
https: http://wwwarrrre www.:assssdaag..ooooeooo http://arr...uaaaoooororrtttttte |
URL
area/ centtr/eeeeeererreee
be
ooo.ep/cee....aarro.ooo/////aaaa 2024-10-01.
Accessed:
USDA. Fooddata central. https://Edc. nal www.:innnaa.uua...mmaeae www.. gov/ 2019. Accessed: 2024-09-30.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Raza Yunus, Omar Arif, Hammad Afzal, Muhammad Faisal Amjad, Haider Abbas, Hira Noor Bokhari, Syeda Tazeen Haider, Nauman Zafar, and Raheel Nawaz. framework to estimate the nutritional value of food in real time using deep learning techniques. IEEE Access, 7:2643-2652, 2018.
Yiran Zhao, Jinghan Zhang, Chern, Siyang Gao, Pengfei Liu, Junxian He, et al. Felm: Benchmark- ing factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A MORE EXAMPLES OF GENERATED MEAL DESCRIPTIONS
In this section, we give more examples of meal descriptions in NUTRIBENCII.
A.1 MEAL DESCRIPTION GENERATED FROM WWEIA
1. For my snack, poured myself a glass of refreshing white table wine. carbohydrates 4. 4.68g, energy 147. protein 11133,,Je fat- 0g
2. For dinner, feasted on a McDonald's Big Mac, enjoyed a large fast food order of French fries, and sipped on large drink of caffeine-free fruit-flavored soft drink.
carbohydrates 194.17g, energy 1394.25kcal protein 31.02g. fat 55.56g
3. For my evening meal, I savored firsh regular bagel with a slice of American cheese melted on top.
carbohydrates 86.86g, energy 341.67kcal, protein 14.47g, fat 6.23g
4. Tonight's dinner consisted of hearty 230g serving of macaroni noodles in a rich cheese sauce.
carbohydrates 50.46g, energy- 363.40kcz protein 11.04g, fat 12.72g
5. During my snack time, enjoyed 150g of ripe raw peach, complemented by 126g of banana and crunchy 28g serving of baked potato chips. carbohydrates 63.92g, energy 306.4666 46kca,, protein 4.13g, fat 5.92g
6. For a satisfying snack, savored a 152g turnover that was filled with scrumptious combi- nation of meat and cheese and topped with tomato-based sauce.
carbohydrates 46.63g, energy 383.04kcal protein 14. 17g, fat 15.55g
A.2 MEAL DESCRIPTION GENERATED FROM FAO/WHO GIFT
1. For lunch today, I had nutritious serving of 0.5 cup of raw lentils and 0.5 cup of crispy fried meat substitute cubes. (Argentina) carbohydrates /05.53g, energy 599..9999al protein 32.99g, far 6.30g
2. This morning, enjoyed raw quince, slicing half of the fruit savor its crisp texture and aromatic flavor. (Italy)
carbohydrates 7.04g. energy 26. 22kcl. protein 0.1, fat 0.05g
3. For breakfast, savored mixture half cup of mung beans and a quarter cup of fresh coconut, providing wholesome and energizing meal. (Sri Lanka) carbokydrates 68.05g, energy 40rrgy-434.377 434.37kcal, protein 25.40g, fat 8.31g
4. At dinner, I savored 100g of brown wheat bread complemented by 350g of rich quince compote. (Bulgaria) carbohydrates 104.40g, energy 466.4666000 00kal,, protein 8. 8.70g, fat 1.80g
5. At lunchtime, 1 indulged in 466g of leavened bread, made with common millet and similar grains. (Ethiopia)
carbohydrates 178.50g, energy 764. 764.00kcal, protein 19.10g, far 3.30g
6. For my snack, had 110g of raw common guavas and 123.75g fresh mango. (India) carbohydrates 34.29g, energy 149.05kcal, protein 3.82g, fat 1.52g
B DETAILS ON MEAL DESCRIPTION GENERATION
In this section, we provide details about the generation of meal descriptions, including the prompts used for creating these descriptions and human verification examples.
B. PROMPTS FOR MEAL DESCRIPTION GENERATION
We show the prompt used for generating meal descriptions in Figure and the prompt used for combining meal descriptions in fine-tuning section in Figure 10
15

---

Published as a conference paper at ICLR 2025
Prompts for GPT-4o-mini to generate meal descriptions, netural serving:
For the given food item slong with their eerving size and ealing occasion, create meal description, mimicking netural language.
Generate five diverse descriptions
item. Here are some examples:
Example input- Food list. ("description", ['emoothie, made with spinach, banans, almond milk, protein powder"). "unit": ["1.0 glase"). occaslon": "Breakfast");
Example output- My breakfast usually consists of a glass of smoothie made with spinech, benana, almond milk, and protein powder.
Exampie imput- serving (8 ozy, "1.0 tbep") "eeling, occasion": "Snack"))
d'description:: ['Beverages, NESTEA,
ready-to-drink,
"Frutt apple"). "unit": ["1.0
Example output- For my snack, enjoyed an 8 fl serving of NESTEA lemon black tea along with a tablespoon of apple butter.
("description:: ['Apples, dehydrated (low moisture), sulfured, uncosked": "Alcohalic beverage, bear, regular, BUDWEISER", "Frult butters, apple"]. "unit: [ cup". "1.0 serving (12 fi oz)" "1.0 toep"), "eeting_ooasee occasion" "Lunch"] Example output- For lunch. had
Exampic imput-
uncooked dehydrafed apples, tablespoon apple butter, and serving of Budweiser beer.
Example input- Food list: ("description:: ("Breskfast tart, lowfat", "Cereal (General Mills Lucky Charmsy'. "Milk, whole", 'Apple juice, 100%*). "unit": (*1 Pop Tari", "I preparckaed bowl", 'I cup", "I individual school container"), "cating_occasion" "Breakfast") Example output- For breekfast. had low-fet milk, and an individual school container of apple juice.
prepackaged
Lucky
cup whole
Example imput- Food list: ("description:: ["Frui flavored drink,. "Peanut butter sandwich, with regular butter, white bread", "Cor chips. plain (Frttox)". "Yogurt, low single serving bag". "I tube". "1 fun size bar"], "esting, oocaspoooon""".. "Lunch"]) Example output- For lunch. enjoyed Little Hug frui--lavoreed drink, peanut butter sandwich on white bread, small bag of Fritos, low-fat yogurt,
flavors other fruit. "Butterlinger"; "unk": ["1 Little Hug bottle", sandwich", small
Butterfinger.
Retum only the meal descriptions without additional information. Provide the tive descriptions separated by new line. Food list: (food_ lisf)
Prompts for GPT-4o-mini generate meel descriptions, metric serving:
For the givan food item along with thair serving size and nating occasion, create meal description, mimicking natural language.
Generate five diverse meal descriptions for each food item. Here are some examples:
Example input- Food liet ("ddecription" ['smoothie, made with spinach, banana, almond milk, protein powder"), "unit" 1200.0g']
"eating occasion": "Breakfast")
Example output- My brakkfatt usually consists of 200g smoothic made with spinach, banana, almond milk,
Example ingut- Food list ("description, ['Beverages, NESTEA, tea, black, ready-to-drink, lemon", "Fruit butters, apple"] "unit": "unit" [224,00". "17.0g"), "esting, occasion": : "Snack")) Example output- My snack today was 245 grams NESTEA black tea
a 17g portion apple
Example ingut- Food list: ("description:: ['Apples, dehydrated (low maisture), sulfured, uncooked", "Alcoholic beverage, beor, regular, apple"]. "unii" ('60 "357 0g" "17.0g"), Example outpul- For lunch, I had 60g uncooked dehydrated apples. 17g of apple butter, and 357 of Budweiser beer.
BUDWEISER", "Fruit
"Lunch"]:
Example input- Food list ("ddecription: ["Breakfast lowfa!" "Cereal (General Mills Lucky Chamm)", "Mil, whole", "Apple juice, 100%"", "urit": ("52.0g" 0g", "244.0g". "124.0g"), "eating occasion". "Breakfast")) Example outpul- For breaktest, had 52g low-tat breakfast tart. 27g General Mills Lucky Charms cereel, 244g whole milk, and 124g apple
Example input- Food list: ("description: : ['Fruit flavored drink", "Peanut butter sandwich, with regular peanut butter, on white bread". "Com chips, plain (Frtos/. "Yogurt, low fat milk, flavors other than fruit", "Buttorlnggr"". "unit": (248.0g", "92.0g". "28.0g": "64 Dg", 0g"). "eating_occasion"" occasion": "Lunch")) Example outpul- For lunch, I enjoyed 248g fruit- fruit-lavored drink, 92g peanut butter sandwich on white bread, 28g Fritos, 84g low-fat yogurt, and 21g Butteringer.
Return only the meal descriptions without additional information. Provide the five descriptinn separated by new Iine Food list: (food ist)
Figure 9: Prompts for meal description generation.
B.2 HUMAN VERIFICATION
After generating five meal descriptions with GPT-4o-mini for each food item, we randomly choose one meal description the final meal description. However, these descriptions still require refine- ment before practical application. Here are some examples of the raw generated descriptions and the descriptions after human verification:
Food descriptions: [' |Boiledd eggs. Boiling'. 'Vegetable fats and oils, edible. PROCESS Boiling', 'Strawberries', 'Cow milk, whole',
Hen eggs', 'Salt, PROCESS =
'White sugar', 'Tortilla", J. Serving size: 1'47.8'', '0.2g', '3.9g", '36.2g', '172.6g', '10.1g. '69.2g'1
16

---

Published as a conference paper at ICLR 2025
Prompts GPT-4a-mini to meal descriptions with serving:
Combine two meel descriptions into creeting nelural-eounding sentence spoken person. two queries describe different merge
single description.
Example queries: (For breakfast, am having large banana.; 'My lunch consists of a piece of refrigerated whole wheat nain bread.]
Examppe oulpul: For lunch, am having large benana and piece of refrigerated whole wheet neian bread.
Example queries: (1l had fresh salad with slice of garlic bread on the side.", " For lunch. had serving of griledd chicken steamed vegetables enjoyed Example output: For lunch, enjoyed a tresh green selad with slice of garlic bread, followed by a hearty bowl of tomato soup and one serving of grilled chicken with steamed vegetables.
tomato soup lunch.")
Example queries: (ll had dlassic burger with cheddar cheese and crispy bacon', 'Enjoyed one serving of seasoned fries', Forr breakfast, am having side of colesiaw, am having cup of chocolate milk for drink.] Example output: had
burger chedder
and crispy becon, with serving
coleslaw and a cup chocolate milk for breakfast.
Return only combined meal descriptions, excluding any additional information.
Queries: (query)
Prompts GPT-4o-mini combine descriptions
serving:
Combine two moal descripiinss into one, creating a natural-soundiing sentence spoken person. If two queries describe single meal description.
different meels, merge
Example queries: [FFor breakfast. I am having 136g banane.", ' 'My lunch consists of 106g refrigerated whole wheat nean bread.] 138g banana and 106g refrigerated whole wheal bread.
Example output: For lunch,
Example queries: (1l had 43g garlic bread", 'For lunch, had 100g grilled chicken with steemed vegetables.", enjoyed '100 tomato soup for lunch.]
Example output:
enjoyed 43g garlic bread,
100g tomato soup 100g grilled
steamed
vagelables.
Example queries: ('l 100g disseic burger em having 220g coleslew', having 100g chocolate milk for drfnk/'] Example output: had 100g classic burger with cheddar cheese and crispy bacon, served with 85g seasoned fries, 220g coleslaw and 100g chocolate milk for breakfast.
orispy bacon', 'Enjoyed 85g seasoned
Return only the combined meal descriptions, excluding any additional information.
(query)
Figure 10: Prompts for meal description combination in fine-tuning.
Before human verification: For a refreshing snack, I enjoyed 36.2g of strawberries alongside 47.8g of boiled eggs drizzled with 3.9g of vegetable oil. To complement the eggs, added touch of 10 of white sugar to sweeten my cup of whole cow milk. Additionally, I savored tortilla, making for delightful and satisfying snack experience.
After human verification: I enjoyed 47.8g boiled eggs, 3.9g vegetable, and a 69.2g tortilla with 0.2g salt and 10.1g sugar. addition, hvae 36.2g strawberries and 172.6g cow milk. Changes: Include missing Foods: salt and cow milk.
Food descriptions: l'Pancake syrup'. 'Pancakes, with fruit. from frozen") Serving size: ('I tablespoon', pouch') J
Before human verification: I kicked off my day with delightful breakfast of pancakes from frozen, complemented by a tablespoon of rich pancake syrup. After human verification: I kicked off my day with a delightful breakfast of one pouch of pancakes from frozen, complemented by a tablespoon of rich pancake syrup.
Changes: Include the missing serving size; one pouch.
Food descriptions: ['Chicken fillet biscuit, from fast food', 'Soft drink, cola'). Serving size: I'I sandwich, any size', 'I bottle (20 oz)'] Before human verification: savored a chicken fillet biscuit from a fast food joint for breakfast, accompanied by a 20 fl oz bottle of cola.
After human verification: savored a chicken fillet biscuit sandwich from a fast food joint for breakfast, accompanied by 20 fl oz bottle of cola.
Changes: Include the missing serving size: a sandwich.
17

---

Published as a conference paper at ICLR 2025
('Cream, half and half, flavored', "Coffee, brewed', 'Banana, raw"). Serving size: ['I individual container ft) 'I large', . 'I banana']
Food descriptions:
Before human verification; This morning, my breakfast consisted of rich brewed coffee, enhanced with a flavored half and half cream, and I couldn't resist having whole banana on the side.
After human verification:
This morning. my breakfast consisted of large brewed coffee, enhanced with individual container of flavored half and half cream, and I couldn't 't resist having a whole banana on the side.
Changes: Correct the wrong serving size: a large coffee and an individual container.
C DETAILS ABOUT EVALUATING LLMS ON CARBOHYDRATE ESTIMATION.
C.I HYPERPARAMETERS.
We use temperature = 0.6, and top-p 0.9 for open-sourced models they default gener- ating hyperparameters for LLama3.1. For GPT models, we use top p = 0.1 and temperature 0.1.
C.2 PROMPTS FOR CARBOHYDRATES ESTIMATION
For Base and COT methods, query the LLM model with the meal description and some instruc- tions as shown in Figure II For RAG, we will first parse the food description into food components. The parsing prompt is shown in Figure 12 Next, will retrieve the nutrition information about each food item and finally provide LLMS with the retrieved context as well as the meal description. The RAG prompt for GPT40/GPT-44-min as Llama and Gemma, follow their specific templates, though the content remains the same.
is shown in Figure 13] Prompts for other models, such
D CONSTRUCTION OF RAG DATABASE
We constructed RETRI-DBfrom FDC (USDA 2019) database, using the latest available version at the time of construction (April 2024). The full dataset included 2,046,761 food item entries, which 489,534 had unique food descriptions. We combined the entries with identical descriptions and filtered those without carbohydrate information. For entries with the same description but varying carbohydrate content, extreme outliers (defined as those with a z-score 1) were removed, and the median of the remaining values was taken as the final carbohydrate value for each food item. This process resulted final set of 450, food items with carbohydrate labels, including 13,048 items with natural serving sizes. Since models process unstructured information more effectively than tabular data (Chen al., 2019), we applied a rule-based transformation convert each entry's nutritional context into natural language. Specifically, for an entry containing a serving amount and nutrition amount for a particular netrieet, we convert to the string "serving amount has nutrition amount nutrient".
E CONSTRUCTION OF FINE-TUNING DATA
We also leverage the FDC data to generate our fine-tuning dataset. We apply rule-based transfor- mation to convert each food item into a meal description similar to the process in building RETRI- DB. addition, we sample 98K food items, incorporating both natural and metric servings, and use GPT-4o-mini to generate meal descriptions for these items. To include multiple foods in de- scriptions, we randomly select to 4 descriptions and use GPT-4o-miní to combine them into a single meal description. The prompt for combining meal descriptions is in Appendix [.I As sult, we obtain a dataset of 158,246 GPT-generated meal descriptions and 450,182 rule- rule-basee ba-dae meal descriptions.
18

---

Published as a conference paper at ICLR 2025
Base, GPT-4o-minVGPT-40:
For the given query inclucing meel desciiptin, calculate the amount of carbohydrates If the serving item the assume single standard serving based common
guidelines (e.g.. USDA).
Respond with a dictionary object containing the total carbohydrates in grams as follows: (Chotal carbohydratess" total grams of carbohydrates for the senving)) the carochydrates, respond
amount of carbohydrates
you
reepond with:
{["toial_c [[Tolal_carbohydratee carbohydratee"... -1))
Query: "This morning a
banana glass orange juice."
Answer: (["toal_carbboln wwt.bbbyyyyratsss 66.5))
Query: scrambled eggs
eggs breakfast"
Answer: (1 (ottal carbohydrates": 15))
Query: "Helf
jelly sendwich."
Answer: ("Yotal carbohydrates": 25.3))
Query: (query)
Answer:
GPT-4omin//GPT40:
For the given query including description, think step by step follows:
1. Parse the meal description into discrete food or beverage fiems along with their serving size. If the serving size of any item in the
it single standard serving based common nutriional guidelines (e.g.. USDA). Ignore additonal the 2. For each food or beverage item in the meal, calculate the amount of carbohydratee in grams for the specific serving size.
information specified,
serving size.
3. Respond with dictionary object containing the total carbohydrates grams as follows:
carbohydrates": lotal grams
serving))
For the total carbohydrates, respand with just the numeric amount carbohydrates without extra text. If you dan't know the answer, set the value of "total carbohydrates" to -1.
Follow the format of following examples when answering
"hii morning.
cup of catmeal with half sliced and glass orange juice."
Answer: Let's step stap.
The meel consiete of cup of ostmeel, 1/2 banana and glass of orange juice.
ostmesl has 27g carbe.
so glass of orange juice has 26g carbs.
banana (271(1/2)) 13.5g
So the total grams carbs in meel (27 13.5 26) 66.5
Output: ("total_cara carbooydraatss. 66.5))
Query: ale scrambled eggs made with 2 eggs and toast for breakfast"
Answer: Let's think stop stop.
The meal consists scrambled eggs made with 2 eggs and toast. Scrambled aggs made with 2 aggs has 2g carts.
tosst has 13g carbs.
grams
(2 13)
Output: ([ ("total_ carbooydrrates. 15))
Query: "Half
butter
Anewer: Let's think step by step.
The meal consists of 1/2 peanut butter and jelly sandwich.
peanut butter and sandwich has carbs half jolly sandwich (21/3)(1/)) 25.3g So the grams carbe in meel 25.3
Output: (( ["total_caat carbooydrrates" 25.33)
Query: (quary)
Answer: Let's think stop by stap.
Figure 11: Prompts for carbohydrate estimation using GPT-4o-mini/GPT-4 without RAG.
F RULE-BASED ALGORITHM FOR OBTAINING NATURAL SERVING UNITS IN THE FAO/WHO GIFT DATA
For converting the metric servings (grams) for food items the FAO/WHO GIFT data to natural servings, we first mapped the food items in the FAO/WHO GIFT dataset to those in the FDC dataset by identifying the nearest neighbors in the OpenAl eext-embedding---largg embedding space and choosing similarity score threshold of 0.5 to obtain valid mappings. A manual inspection of 100 sampled food items showed an accuracy of 85.5% with this threshold. For food items with valid converted to the nearest natural serving size. This
FDC mappings, the serving amount in grams was done by selecting the closest natural serving if it was within 10% the actual gram amount. If no suitable match found, the closest larger natural serving was chosen, along with multiple
19

---

Published as a conference paper at ICLR 2025
Parse, GPT-4o-mini: For the given query including meal description, parse the meal into a list of discrete food beverage item. Ensure that all the ttems are strings enclosed double quotes.
Fallow the format of the following examples when answering
Query: moming, haa cup oatmeal with sliced banana glass orange juice."
Anewer: ['oatmeal", "banans", "orange juice"]
Query: scrambled egge made egge
breekfast."
Answer: ('scrambled eggs". "toast"]
jelly Answer: ppeanul butter and jelly sandwich"]
Query: small
Query: large cappucsina made with whole milk"
Answer: ('cappuccino made with whole mil"]
Query: (query)
Figure 12: Prompts for parsing meal description into food items using GPT-4o-min//GPT-4.
Table 5: Mean Absolute Error (MAE) and Acc@7.5 of models fine-tuned using Llama and Gemma, comparing performance across different parameter sizes.
Mean Absolute Error
Acc@7: 7.5
13.79
46.84
Llama3.1-700 1-T0B-FT
12.60
49.61
Gemma2-27B-Fi
10.49
56.71
Ilama3. 1-8B-Base
19.97
36.20
Llama3. 1-70B-Base
14.73
42.05
Gemma2-27B-Bau
13.32
45.61
to reflect the correct quantity. To avoid unrealistic servings (c.g., 10 tbsp of rice), no multiple smaller than 3 was found, selected the closest smaller serving and expressed it using fractions (e.g., quarter tablespoon', 'half a cup'). For even smaller amounts, we applied volume conversions, such 1/8 of cup being equivalent to 2 tablespoons 1/16 of cup to teaspoon.
G ADDITIONAL RESULTS ON FINE-TUNING WITH FDC DATA.
To explore whether larger model different model will yield similar performance improvements with fine-tuning, we also conduct ablation studies by fine-tuning LLaMA 3.1 models with 8B and 70B parameters. Due computational constraints, fine-tuning limited GPT-generated data, comprising 39,745 metric samples and 19,745 natural samples. The results, presented in Table 5 indicate that larger model sizes generally lead to better performance. Furthermore, Gemma2 consistently outperforms LLaMA 3.1, aligning with the trends observed in the un-fine-tuned -uned models.
H EXAMPLES FROM SOURCE DATABASES TO NUTRIBENCH
H.1 EXAMPLE
The raw WWEIA and FNDDS data with selected rows and columns are Table 6 and 7 generated meal description is: "At dinner, I treated myself to delicious double cheeseburger from McDonald's paired with a delightful soft serve vanilla ice cream cone in waffle cone." " The associ- ated nutritional values are: carbohydrates 112.96g, energy 957.45kcal protein 39.45g, and fat 38.80 38.86. All nutritional data are sourced from WWEIA (refer to Ta
The
H.2 EXAMPLE 2
Table presents an example of a meal found in the FAO/WHO Gift raw data for India. The final query generated from this log 'For my snack, I had 30g of food industry prepared biscuits,
20

---

Published as a conference paper at ICLR 2025
RAG, GPT-4o-mini:
Context: (context)
description, meal is not speciled, assume it single standard serving based on common nutritional guidelines (e.g. USDA).
the given query including
amount carbohydrates grams. serving any
Respand with a diclionary object containing the total carbohydrates in grams follows:
(hooaal cartoohydaaes""
grams of carbohydreees the serving)) For the total carbohydrates, respond with just the numeric amount of carbohydrates without extra text. you don't know the answer.
respond with: (["taal carbohydrates" -1))
Query: "This morning. I had a cup of catmeal with half sliced banana and gless of orange juice."
Answer: ("Motal, carbohycrates" " 66.5))
Quary: ate scramibled eggs made with 2 eggs and a toast for breakfast"
Answer n'ooaal_ carbohyddatess" 15))
Query: "Half a peanut butter and jelly sandwich."
Answer (ototal carbohydrates" 25.3))
Query: (query)
Answer.
RAG+CoT, GPT-4o-mini: (context)
For the given query including meal description. think step by step follows:
Parse the meal description into discrete food beverage iiems along with their serving size. If the serving size of any item in the specified,
single standard serving based and serving size.
nutritonal guidelines (e.g.. USDA). Ignore additional
Information doeen't
the
2. For each food or beverege item in the meal, calculate the amount of carbohydratee in grame for the specific serving size. follows:
Respand with dictionary object containing the carbohydrates
(flotal, carbchydrates": For the total carbahydrates, respond with just the numeric amount carbohydrates without extra text. If you don't know the answer, set the value of "totel carbohydrates" to -1.
grame carbohydrates
serving))
Fallow the format of the following examples when answering
Query: "This moming,
cup of oatmeal with half sliced banana and glass orange juice."
Answer: Let's think step by step.
The meel consists of cup of cstmeal, 1/2 banana and glass of orange juice.
ostmeel has 27g carbe.
banana (272(1 13.5g carbs.
glass of orange juice has 26g carts.
So the total grams of carbs in the meal (27 13.5 26) 66.5
Output ("hotal carbohydrates", 68.5))
Query: ate scrambled eggs made with 2 eggs and loast for breakfast."
Answer: Let's think step step.
The meel consists scrambled egge made with 2 egge and toast. Scrambled eggs made with 2 eggs has 2g carbs.
foest has 13g carbe.
grams
(2
Output: (( [[Potal_ carbohydrates" 15))
jelly sandwich." Answer: Let's think step by step.
Query: peanut
The meal consists of 1/2 peanut butter and jelly sandwich.
jolly sandwich has cartss half peanut jolly sandwich (25.3"(1/21) 25.3g So the total grems of carbs in the meal 25.3
peanut butter
Output: ([ ("totaal carbohydrates" 25.3))
Query: (query)
Answer: Let's think step step.
Figure 13: Prompts for carbohydrate estimation using GPT-44-minn/GPT-4o with RAG.
SEQN
DR1-030Z
DRIIFDCD
DRIIGRMS
DRIICARB
(Interview ID) (Meal Occasion)
(Food Code)
(Food Weight)
(Carbohydrate)
100705
3.0 (dinner)
27510387
165
29.65
100705
3.0 (dinner)
13120786
255
83.31
Table 6: Selected raw data from WWEIA. This table focuses solely on carbohydrates, while other nutritional information, though omitted here, is available in the database.
paired with 26g of sweet bars made from palm trunk sap and sugar cane molasses, along with 23g of raw peanuts.', with labels 373kcal for energy, 7.5g for protein, 48.9g for carbohydrates, and 17.6g for fat.
21

---

Published as a conference paper at ICLR 2025
Food Code
Main food description
Portion weight (g)
Portion description
27510387
Double cheeseburger (McDonalds) Ice cream cone, soft serve, vanilla, waffle cone
165.0
double cheeseburger
13120786
255.0
cone
Table 7: Selected raw data from FNDDS
MEAL
FOODEX2INGH DESCR
FOOD WWW.AAMOUUTOOOOO
CARBOH
8
Biscuits, PREPARATION- PRODUCTION-PLACE
30
20.4
Food industry prepared
Sweet bars and other formed sweet masses, INGREDIENT = Palms (trunk sap), INGRE- DIENT = Sugar cane mo- lasses, PHYSICAL-STATE
26
24.8
Solid (soft or hard)
8
Peanuts, PROCESS Raw, heat treatment
23
3.7
Table 8: Example of meal in the raw data of FAO/WHO Gift from India. Several columns omitted for space considerations. The column MEAL_NAME notes the eating occasion (break- fast, lunch, dinner etc.), FOODEX2 DESCR refers the food items consumed in the meal, WWW.ONTTOOOSUUOOTAEEAO CONSUMED provides the portion size in grams of the consumed food item, and CARBOH g indicates the corresponding carbohydrates.
FOOD AMOUNT
ANALYSIS OF HIGH CARB VS. LOW CARB MEALS
We analyze the properties of single-food item high vs low-carb meals further to investigate the higher rates in the single-food high-carb meals observed in Section55.2) We defined high-carb meals as single food item meals with carbohydrate values above the third quartile (33.93g. N=869). Similarly, low-carb meals were the single food item meals with carbohydrate values below the first quartile (6.47g, N=871).
We found that high-carb meals exhibited significantly greater variability in carbohydrate content compared to low-carb meals (P < 0.05, Levene's test). Specifically, the standard deviation for high- carb meals was 26.88g, whereas for low-carb meals, was only 2.01g. This increased variance in high-carb meals suggests that the model may face greater difficulty when predicting carbohydrate content for foods with a broader range of values, which likely contributes the observed increase in eтror rates.
Additionally, we observed significant differences in the portion weights of low-carb and high-carb meals. High-carbohydrate meals had larger average portion sizes (95.27 + 139.89 g) compared to low-carbohydrate meals (226.80 + 161.73 g) (P < 0.05, Mann- Whitney U Test). The larger portion Mann-Whitney sizes of high-carb meals could also increase the complexity of accurate carbohydrate estimation, as larger meals may involve more varied ingredient types or require more precise portioning, contribut- ing to the higher error rates.
J ERROR ANALYSIS
We conducted detailed analysis of the queries in NutriBench leading to high errors with the best- performing model, GPT-4o with CoT prompting. First, we examined 100 queries sampled from NutriBench where the model exhibited high absolute error (>50g). We found that in all cases analyzed, the model successfully parsed meal components and serving sizes, accurately formulated equations, and correctly performed mathematical operations to calculate the total carbohydrate tent. However. it produced inaccurate carbohydrate estimates for one or more items within the meals, leading to overall high errors.
22

---

Published as a conference paper at ICLR 2025
To further identify patterns in the errors, compared high-error meals (absolute error >thhrd quartile, N=2,940) with low-error meals (absolute error <third quartile, N=2,939) using the Mann- Whitney U Test. The analysis revealed that high-error meals contained more food items average (2.13 + 0.93) compared to low-error meals (1.81 = 0.83, P <0.05). High-error meals also had signif- icantly higher carbohydrate content (64.58 + 38. 38.69g 24.13 25.27g, P <0.05) well larger portion weights 2268.37& = 250.08g) compared to low-error meals (437.28 = 301.25g, P <0.05).
These findings highlight that meal complexity, greater carbohydrate content, and larger portion sizes are key challenges for LLMs in carbohydrate estimation that should be focused on in future work to improve model performance.
K EVALUATING WITH PARAPHRASED PROMPTS
To evaluate the robustness of NutriBench to variations in prompt phrasing, we paraphrased the instructions while retaining specific output formats and demonstrations. This approach ensures con- sistency in expected outputs while testing the model's flexibility in understanding rephrased instruc- tions.
The three paraphrased instructions were as follows:
1. Given query describing meal, estimate the total carbohydrates in grams. If the serving size of any food item is not mentioned, assume standard single serving based on common nutritional guidelines (e.g., USDA). Provide the result as dictionary in the following format: ( (("total_carbll carotaalcabooyrrtee"" total grams of carbohydrates Include only the numeric carbohydrate value without additional text. If the information needed calculate the value is missing or uncertain, respond with: {("total carbohydrates": -1}}
2. Analyze the provided meal description and calculate the total carbohydrate content in grams. If the portion size of any ingredient is not specified, assume a default single serv- ing based standard nutritional guidelines (e.g., USDA). Return the result as dictionary in the format: {( ("ttota carbohydrates": total grams of carhohydrates) Ensure that only the numeric value included for the total carbohydrates. If the necessary information is unavailable or unclear, reply with: "aotaatcarboohydrrttt""" -1}}
3. Estimate the total carbohydrates in grams for given meal description. If the portion size for any food item not specified, assume standard single serving size based on com- mon nutritional guidelines (e.g.. USDA). Provide the result dictionary in this format: [("total caotallcarbohydratess.. total grams 3
Use only the numeric carbohydrate value your response without any additional text. If necessary information to compute the value is missing or ambiguous, respond with: {l (("total_care wawbooydrrrtsss
Experiment
Acc@7.5
Mean Absolute Error
Instruction (original)
51.43
11.46
Instruction
51.29
11.54
Instruction
52.70
11.78
Instruction
50.75
11.56
Mean
51.54
11.59
Standard Deviation
0.71
0.11
p-value
0.80
0.16
Table 9: Performance metries for different instructions.
The performance of GPT-4o-mini with Base prompt is summarized in Table 9 While the prompt does not need to adhere rigidly to specific formulation to produce results, it must maintain defined structure, including clear output formats and demonstrations, to guide the model effectively. Paraphrasing the prompt has minimal impact the results.
23

---

Published as a conference paper at ICLR 2025
L MORE ANALYSIS ABOUT GPT VS. NUTRITIONISTS
Among 72 meal descriptions, we identify 20 queries where GPT outperforms all nutritionists, and meal descriptions where all nutritionists outperform GPT. Our analysis reveals intriguing patterns:
GPT excels in complex, multi-component meals and those with detailed measure- ments. For instance, in the description "For breakfast, I had Burger King sandwich featuring egg, cheese, and sausage biscuit, paired with can of cola," GPT achieves a Mean Absolute Error (MAE) of 6.09, compared to the lowest MAE of 10.09 among nutritionists.
Nutritionists perform better with simpler, traditional meals lacking specific brand information. For example, in the description "Tonight's s dinner consisted of hearty 230g serving of macaroni noodles in rich cheese sauce,". GPT has an MAE of 20.84, while the highest MAE among nutritionists is 10.46.
Since we only have carbohydrate estimations from nutritionists, it difficult to directly evaluate their knowledge of specific meal descriptions. However, by analyzing the variance in their estimations, we uncover interesting patterns:
For meal descriptions with the highest variance among nutritionist estimations, GPT achieves a substantially lower Mean Absolute Error (MAE) of 18.9, compared to the lowest MAE of 34.4 among nutritionists (averaged over the top 10 high-variance descriptions).
For meal descriptions with the lowest variance, the MAEs are much closer: GPT achieves an MAE of 4.6, while nutritionists' MAEs range from 3.4 to 4.5.
These findings suggest that GPT performs better on meal descriptions where nutritionists show greater disagreement, possibly due to gaps in their knowledge or unfamiliarity with the meals.
The primary reason we did not initially allow nutritionists to search online was to prevent them from accessing our source database and obtaining the ground truth values directly. However, after discussions with nutritionists, we learned that they may rely on tools to look up information as part of their usual workflow. To ensure fair comparison, we conduct an additional human study where nutritionists are allowed to look up food items and use their standard methods to estimate carbohydrates.
In this study, we received only one result from a nutritionist, which achieved comparable perfor- mance the best AI model, CoT GPT-4o. Compared the scenario where nutritionists are not allowed to look up information, we observe two significant improvements: Meal descriptions with metric servings show better performance, demonstrated Table I0] Meal descriptions where nutritionists previously disagreed improved significantly, with the MAE of the top 10 high-variance descriptions dropping from 36.7 to 21.4, although GPT still performed better overall.
Method
Acc@7.55 all Acc@7.5. metric Ace@7.5, natural
Nutritionist
42.45
39.47
45.45
Nutritionist, look up
59.72
73.68
44.12
GPT-4o, CoT
60.56
63.16
57.58
Table 10: Performance metrics allowing nutritionists to look up.
24


Reviews:
[{"review_id": "CJ3lfsniGv", "reviewer": "Reviewer_JjPd", "summary": "This paper presents NUTRIBENCH, a new benchmark dataset designed to evaluate LLMs on nutrition estimation from natural language meal descriptions. The dataset includes 11,857 meal descriptions annotated with macronutrient labels (carbohydrates, proteins, fats, and calories), generated from global dietary intake data. The authors assess twelve LLMs.", "questions": "- What measures are taken to verify the nutritional information accuracy of each meal description, especially considering human verification only involves a single author?\n- Could you provide additional details on how you mapped food quantities to everyday serving sizes? For instance, how does the rule-based algorithm handle ambiguous measurements?\n- How do you explain the higher error rates in carbohydrate estimation for high-carbohydrate foods? Does the model struggle with certain food types or serving sizes?\n- What might be the underlying causes for cultural discrepancies in model performance, particularly for countries like Sri Lanka with higher MAE?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Nutribench is the first of its kind for the task of macronutrient estimation from the textual description of meals. \n- The paper benchmarks multiple LLMs and also explores/compares multiple prompting strategies. \n- The authors conduct an evaluation involving nutritionists as well. \n- The authors do a comprehensive analysis of performance across different dimensions.", "weaknesses": "- The authors have relied on GPT4o to generate meal descriptions. There is a concern with this. Were all of the outputs by GPT4o verified by humans? \n- The fine-tuning experiment with Gemma2-27B shows improvements but does not address whether larger or different models would perform similarly with fine-tuning.\n- The choice of specific prompting methods (e.g., why Chain-of-Thought outperforms other methods) should be discussed well.\n- Although the paper mentions about the performance disparities across cultural diets, it does not explicitly address if these are due to model biases or data representation gaps or any unique characteristics of the specific foods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YNIC6Jmil8", "reviewer": "Reviewer_rcnA", "summary": "This study presented NutriBench, a natural language meal description dataset labeled with macronutrient and calorie estimates. It evaluated 12 LLMs covering both open and closed source models with different prompting strategies on carbohydrate estimation. It also conducted a study with 3 nutritionists to obtain carbohydrate estimates on a sample of meal descriptions. Through a real-world risk assessment by simulating the effect of carbohydrate estimates on the blood glucose levels of Type 1 diabetes patiens, it found that LLM can help Type 1 diabetes patients maintain their blood glucose within safe limits.", "questions": "Questions:\n\n(1) NutriBench consists of 11,858 meal descriptions. Was the human verification applied to all of them?\n\n(2) It appears that human verification might induce subjective biases. Take the pizza case as an example, why should we update \"a tasty medium crust pepperoni pizza\" to \"a piece of tasty medium crust pepperoni pizza\"?\n\n(3) Is the benchmark robust and reproducible? Even though the temperature can be set 0 to obtain more deterministic responses, it is uncertain whether the responses will be reproducible. It would be better to include additional experiments for this concern. \n\n(4) Does the prompt have to be strictly formulated to produce the results? Would paraphrasing the prompt significantly change the results?\n\n(5) Would there be any biases when the data generator and carbohydrate estimator are identical? In this case, both are GPT-4o-mini.\n\n(6) What is the x-axis in Figure 6?\n\n(7) In the simulation experiment, it seems that the performance of different nutritionist varies significantly. Why is that?\n\n(8) According to Figure 2, the human nutritionist performed worse than a lot of LLMs. When human made mistakes, what were those mistakes? Did LLMs perform better just because they held more knowledge? If each food is associated with a pre-defined value, why can't nutritionists looked those up if they were not sure of the knowledge?\n\n(9) The authors discussed that previous benchmarks mainly focused on using images. I wonder whether it is feasible to caption the images from these datasets and expand the existing one. Additionally, why not caption the images instead of constructing a benchmark from the scratch?\n\nMinor:\n\n(1) It would be better to use verb tense consistently.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This study worked on an important issue which may influence diabetes management.\n\n(2) Most of the writings are clear.\n\n(3) In addition to a benchmark, this study also provided a preliminary experiments comparing LLMs with human experts. It also showed how it might be applied to real-world management via a case study.", "weaknesses": "(1) The study might be much improved if it could provide more insights into the success and failure cases of the LLMs on the carbohydrate estimation task. When LLMs made wrong estimations, what were those mistakes? Was it because LLMs did not have the information, or it was more like a math operation issue. To me, it is unclear what capabilities of LLMs did this study intend to find out. If it is just for evaluating LLMs for carbohydrate estimation, the research contribution might be limited. It would be better if the authors could further explain how the findings of this study may inspire future works. Why is it different from other LLM benchmark papers? \n\n(2) The study should disclose hyperpoparameter settings in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SiY9R4p4wH", "reviewer": "Reviewer_TX8E", "summary": "The paper introduces a benchmark called NutriBench, which includes 11,857 meal descriptions derived from real-world global dietary intake data. It employs Chain of Thought (CoT) and Retrieval-Augmented Generation (RAG) techniques to tackle the carbohydrate estimation task and evaluates twelve large language models (LLMs), such as GPT-4o and Llama 3.1. Ultimately, a real-world carbohydrate prediction task is conducted. The results demonstrate a significant improvement in both the accuracy and speed of carbohydrate estimation compared to traditional nutritionists.", "questions": "More downstream tasks and comprehensive analyses on nutrition can be performed to generate more informative insights, compared to just providing carbohydrate predictions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Data Collection: This study gathers 11,857 meal descriptions from various countries. This approach enhances text flexibility compared to traditional tabular data and addresses the challenges of acquiring image data.\n2. Model Evaluation: The research evaluates the carbohydrate prediction capabilities of different large language models (LLMs). The results show that these models outperform nutritionists in both prediction speed and accuracy.\n3. Social Influence: The paper conducts a real-world risk assessment by simulating how carbohydrate predictions affect blood glucose levels in individuals with diabetes. This demonstrates the potential positive impact of LLMs on public health.", "weaknesses": "1. Limited downstream tasks: the paper only focuses on the carbohydrate prediction task. More downstream tasks can be performed, e.g., protein prediction, etc.\n2. Over-reliance on description quality: the prediction performance generated by LLMs relies heavily on the quality (comprehensiveness, accuracy, etc) of description data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ySM3cn4nmT", "reviewer": "Reviewer_bDdB", "summary": "This paper describes a new dataset, NutriBench, that contains over 10,000 synthetically generated natural language meal descriptions and corresponding macronutrients from matching foods in the Food Data Central database. They run experiments with 12 open and proprietary large language models, including varients of GPT, Qwen, and Llama on the task of carbohydrate prediction. They found that GPT-4o with Chain-of-Thought prompting resulted in the highest accuracy and response rate.", "questions": "•\tNote that Answer Rate and RETRI-DB are used before they are defined—consider moving up the definitions to the first place these terms are used.\n•\tConsider explaining why you used generative AI to write the natural language meal descriptions instead of humans.\n•\tWere all the foods eaten by people in 11 countries in the USDA’s Food Data Central? I’m surprised you had full coverage of globally eaten meals just from the US food database.\n•\tPlease fix the caption for Figures 4 and 5.\n•\tIn section 5.2, you comment that LLM predictions are more accurate for individuals on a low-carb diet. However, aren’t people on a high-carb diet more likely to get diabetes?\n•\tIn section 5.3, why did you use the Base prompting method instead of CoT to generate responses when you said yourself that CoT is more accurate?\n•\tIn real life, when someone has diabetes, how do they dose their insulin typically? I’m guessing they’re not calculating their carbs every time they eat.\n•\tIn Appendix B.2, a couple of the human-verified responses actually seemed worse to me, with misspellings and unnecessary food words added (“sandwich” to “biscuit”).\n•\tWhy is your FDC database only 450K foods? Doesn’t it contain over 1.8M foods?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The greatest strength is the public release of the NutriBench dataset. Particularly impressive is that the dataset is based on real meals that people across 11 different countries eat. Another strength is the extensive experiments that were run, testing 12 LLMs, using three prompting strategies, comparing to nutritionists, and simulating the effect of carbohydrate predictions on blood glucose levels of individuals with diabetes. The takeaways for which LLMs perform best under which conditions (i.e., prompt, natural vs metric serving size, and diet/country) are important for researchers in this area. The real-world risk assessment with the Tidepool simulation of blood sugar levels is interesting, as are the LLM prompts in the appendix. Finally, this is an important problem with broad societal impacts—diet tracking is challenging, and finding ways to reduce user burden is critical for continued usage, which will help with obesity, diabetes, heart disease, and many other health conditions.", "weaknesses": "The biggest strength is the database, but the data that will be released and was included in the supplementary material contains only the natural language meal descriptions, no Food Data Central entities or their nutrition information. In addition, the only task was carbohydrate prediction, and the authors mention that the dataset contains only macronutrient information, but an LLM nutritionist should be able to predict protein and fat as well, and micronutrients as very important too. 11,857 meal descriptions is a small number, but is reasonable given they are human-verified. Another major weakness is the study in section 6 comparing LLMs to human nutritionists. It is understandable that human experts are slower, but instructing nutritionists not to search the web for carbohydrate estimates and then claiming that LLMs are more accurate seems quite problematic and results in a very misleading conclusion. Nutritionists do their job by using tools and resources. You can’t get an accurate measure of their ability to estimate carbohydrates if you take away the tools they use in the real world. The only actual takeaway here is that LLMs are faster, which is obvious. I love this work and really wanted to accept this paper, but can’t justify it due to these weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)