Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors introduced the Korean Menus–Ingredients–Nutrients–Diets (K–MIND) dataset for diet planning containing 1,140 diets, 3,238 menus, 3,036 ingredients. For creating the dataset, they used combinatorial optimization techniques in operations research (OR) to generate menus that satisfy the nutritional requirements. Then, dietitians edit the solutions to make them more desirable. Finally, an ML module is trained on the edited diet data. The ML module can be used to generate as many diets as necessary. They call this framework ORxML and have released its python package. 
strengths: 1. K–MIND dataset is the largest dataset that is ever introduced for the task of diet planning. 
2. ORxML framework that is suggested in this dataset can further encourage research in the area of diet planning.
weaknesses: 1. The first limitation of this work is that it only includes the Korean diet. The proposed ORxML framework is indeed more general-purpose, but the K–MIND dataset cannot be used for diet planning in any other part of the world. 
2. It is great that the authors take into account the dietitians' edits in the third step of the dataset creation process, but K–MIND is a synthetic dataset. The authors have evaluated the generated diets in terms of RDI score, but it doesn't consider the human element of the diet. It is essential to make sure that the generated diets are actually desirable for humans. Otherwise, it might be a very healthy and nutritionally perfect diet, but people might not want to take it. 
correctness: The procedure for creating the dataset seems legit to me.
clarity: The paper is well written.
relation to prior work: The authors have discussed the diet planning problem and previous approaches using machine learning to tackle the problem. However, a detailed comparison between the available datasets on the task of dier planning is missing. 
additional feedback: It would be great if the authors could run a clinical study to further evaluated the suggested diets. It is beneficial to see how participants would take the diest and hear their feedback. In addition, showing quantifiable health measures from the clinical study would make this work significantly stronger.
review_title: First large-scale dataset on diet planning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces the K-MIND dataset that has granularity at three different levels - Diet, Menu, and Nutrition.
It would be easier to navigate the paper if the authors can give a clear definition of all three key components.
Systematically breaks down food consumption under different abstractions through their proposed schema - diet, menu, ingredients and nutrition.
strengths: This dataset appears to be the first of its kind in this area. However, I am not an expert and might be missing out on related work. The authors have identified several important applications of various aspects of this work in the Appendix which I believe make this work useful beyond Korean food. The ML techniques applied here seem to improve performance on some of the metrics.
weaknesses: Introduction - 
Related work - Almost no discussion of related works.
Considering each diet as a collection of 19 menu items appears to be a restrictive formulation. As authors pointed out, they include dummy menus to fix that problem with diets with a fewer number of menu items. Nevertheless, I think adopting a data structure like a tree or graph might be a more generic and flexible approach here.
correctness: "Incidentally, a planner’s knowledge and belief rarely changes; therefore, the system of diet planning can be regarded as static and
explicit." This claim can be a controversial one. Since the whole point of integrating data driven techniques is to model the dynamic nature of diet planning.
clarity: The terms introduced in L155-163 need to be thoroughly revised so that their English definition is clear to the reader before we jump into mathematical symbols associated with the optimization.
relation to prior work: This paper lacks discussion with some of the prior work. Although the authors have cited many papers in the relevant areas. A proper taxonomy or a landscape of existing works and how they are different than this work should have been mentioned.
additional feedback: I am willing to bump up the rating if authors can address some of the writing and clarity-related concerns.
review_title: New and interesting contribution but the manuscript needs writing and presentation improvement

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors present a novel dataset for diet planning and healthcare research. The authors describe the process of dataset creation by combining approaches from operations research, machine learning and knowledge from domain experts, as well as evaluation of the generated resource. The authors claim that their resource is the first resource for diet planning of that type. The contributions of the paper include: the K-MIND dataset and a diet planning methodology that combines knowledge from operations research models, machine learning and domain experts
strengths: The significance of the work described in this paper is large since diet planning is a very important task in the domain of food and nutrition sciences. So having such a developed resource would lead to further developments of (semi)automatic methods for different aspects of software development in diet planning applications. Hence this work is important for the broader research community. The proposed resource is public and is easily accessible by using a Python library (dietkit). Given that this dataset is synthetic there are no open ethical issues. 

weaknesses: The limitations of the proposed dataset have not been explicitly discussed in the paper. In addition, one would expect a more elaborate discussion of the potential use cases and the advantages this new dataset would bring to the applications that would use the data. This is somewhat given in the Supplementary Material, but having a paragraph or two in the main text would significantly improve the paper.
correctness: In my opinion, the claims made in the paper are correct and the dataset was constructed in a sound way taking into account all relevant data sources and expert knowledge.
clarity: In my opinion, the paper is well written and contains enough information for researchers to start using the presented dataset as well as understand the process of dataset construction.
relation to prior work: The authors have discussed all relevant prior work in the context of diet planning theory and resources in general, as well as discussed the importance to combine existing operations research approaches 
additional feedback: - Incorporate a discussion about potential use cases in the main text
- Discuss the limitations of the use of the proposed dataset in the main text
review_title: Useful diet planning dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a multi-level synthetic dataset of information useful for diet-planning research.

In order to generate such a dataset, the authors propose a six-step methodology, incorporating (1) a component driven by operations research for the first two, i.e. in formulating and filtering for solutions in the combinatorial optimization problem; (2) a component driven by intervention by domain experts in the middle two, i.e. in accounting for implicit and human factors for "compositional quality"; and (3) a component driven by machine learning for the last two, i.e. in learning the patterns in the previous two components to generate synthetic diets, as well as explaining latent elements that are not explicit to experts.

The full dataset is presented in three databases, each containing synthetic examples illustrating the relationships between consecutive levels in the full hierarchy required for diet-planning---relating (a) nutrients to ingredients, (b) ingredients to menus, and (c) menus to diets. A discussion is provided on the evaluation of the proposed method of generation, as well as the final output dataset; these include (i) conformity to nutritional requirements, (ii) compositional quality of diets, and (iii) homogeneity of ingredient usage. It is shown that the machine learning module partially recovered some of the nutritional quality "lost" by the expert component, and it also outperforms even the expert 
strengths: 1. *Significance*. The authors are tackling an important problem. Both on ground-level advisory and more research-oriented settings in industry, the lack of high-quality data on diets and nutrition is indeed a major impediment to progress. To the best of my knowledge, this has not been tackled at this scale. The authors propose a hierarchical approach through multi-level relationships, and approach the problem through synthetic data generation, which solves the two primary hurdles---i.e. the unstructured nature of vast swaths of observational datasets out there, as well as the fact that no single consistent dataset is large enough for more in-depth research.

2. *Transparency*. On a high level, the 6-step structure proposed is clear. There appears to be no black magic and the authors make it known that what is being done is simply smartly combining known methods in operations research and machine learning with domain experts, to achieve the desired result. This is an advantage, as it is clear what goes into generating this dataset. Moreover, the process can be replicated (cf. "controlled generation") by other users that may desire a slightly different dataset.

3. *Human-in-the-Loop*. The use of domain experts in the loop is well-considered, especially at a time when the machine learning community as a whole is warming up to the notion that we can be smart about injecting human knowledge into models, instead of "learning from scratch".
weaknesses: 1. *Definitions*. There needs to be a clear definition of the various hierarchical terms being used. Understanding the terminology took several reads, especially going back to Figure 1 repeatedly. The impression given is as follows:

- "Nutrients": The lowest level, i.e. carbs, proteins, fat, etc.

- "Ingredients": Each ingredient contains a combination of nutrients. There needs to be an example of what an ingredient is.

- "Menu": Each menu contains a combination of ingredients. This is the most confusing term! It appears what is being referred to is a "food item". If this is an idiosyncrasy of the field, this still needs to be defined, with examples of what it can be.

- "Diet": Each diet contains a combination of menus. Again, this needs to be defined more clearly, as "diet" can mean any number of things in vernacular. Does this simply refer to what we take to be a "meal"?

- Sequence: Each sequence contains an ordered combination of diets. There does not appear to be a specific term that is used consistently throughout for this. However, it is present as the red box in Figure 1, so perhaps this can also be defined.

Here is a suggestion: Somewhere in the introduction, it would be much clearer if there were an explicit definition of all of these terms, so that readers unfamiliar with the hierarchy of diet-related concepts would be initiated appropriately. This can also be a text-box. It just needs to be up front. (At the moment some of these only make more sense after reading Appendix B).

2. *Vague Terms*. Some terms would benefit from more concrete illustration (either through concrete examples or concrete criteria) in the main manuscript:

- "Nutrient requirements": It appears this refers to the combinatorial optimization problem. What are the constraints, and are the constraints specified externally (i.e. this is simply RDI), or are some constraints introduced by the authors? This needs to be clearer.

- "Compositional quality": It appears that this refers to literally anything that humans may desire (i.e. "soft criteria") that are not captured by the combinatorial optimization problem (which could conceivably yield very odd meal recommendations). Is this correct? This needs to be clearer, with better examples.

3. *Clarity of Evaluation*. At the moment the evaluation section discusses evaluating RDI, percent mis-positioned, and Chi-square. Two questions:

- How is the RDI score (which is defined as "nutritional excellence") computed? How is the mis-positioning determined (i.e. relative to what correct ordering)? There are details in the Appendix, but at least one or two lines of explanation in the main manuscript would be good.

- Some of these measures are the input to the method. For example, RDI is the input to the OR component, which explains why that component does best on RDI score. Is correct positioning implicitly part of what the expert component encourages when they edit the plans? Same with Chi-square. This is important to briefly discuss, because that would give some context as to why the numbers are the way they are.
correctness: No issues other than those noted in "clarity of evaluation" above.
clarity: Apart from the specific clarity concerns above, in general the paper is well-organized on a high level. Minor comments:

- "RDI is a shortcut for recommended daily intake" --> shorthand?

- In Figure 1, it may be easier to understand if these were bolded and put first: "Food Item Level", "Menu Item Level", etc., and these are put second (or even removed): "diet problem", "diet planning", and "diet planning with ML", which are too vague.
relation to prior work: To the best of my knowledge, the relationship to prior work is covered satisfactorily.
additional feedback: Minor: The typesetting used appears to be incorrect. Is the correct latex template being used?
review_title: A well-thought solution to an important problem, with some clarity issues

Meta-review: Dear authors,

Thank you for submitting your paper and addressing the issues raised by the reviewers.

The paper introduces a novel dataset on diet planning and a tool that generated the data. The dataset was generated synthetically but edited and improved by human domain experts and Machine Learning.

The reviewers commend the importance of the data, use of human experts to improve the data quality, general clarity of the paper and easy to use python package.

The main weakness is the potentially limited use of the data and its limitation to Korean food. In the appendix A7 several additional use-cases are discussed.

The reviewer scores are 7,6,6 and 7 which is sufficient for accepting the paper.
Decision: Accept