DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization
Authors: Chengang Hu
Xiao Liu
Yansong Feng
Venue: EMNLP 2023 Main (2023)
Keywords: compositional generalization
dish name recognition
large realistic dataset
language model evaluation

Abstract: Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition.

DINER: a Large Realistic Dataset for Evaluating Compositional Generalization
Chengang Hu and Xiao Liu and Yansong Feng"
Wangxuan Institute of Computer Technology, Peking University [hcg, 1xlisa, fengyansong Sepku edu. cn
Abstract
Instructions:
Instructions:
Cut the pork belly and rinse
Clean the brisket and cut it into small pieces
Most of the existing compositional generaliza- tion datasets are syntheticall--geenaaee sulting in lack of natural language variation. While there have been recent attempts to intro- duce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate com- positional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DINER) task and create large realistic Chinese dataset. Given a recipe instruction, models are required recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 11 dishes and 228, recipes, and involves plenty of linguistic phe- nomena such as anaphora, omission and ambi- guity. We provide
re-
Add rice wine, light say sauce and dark soy sauce
Cook the meat pieces with the spices for about hour
Dish name: Bruised Pork (At 19)
Dish name: Marinated Beef Brisket (å 4-10)
Instructions:
Marinate the brisket for more than min
Add dark soy sauce and light soy sauce Stir fry well
Dish name: Bruised Beef Brisket (fr 4-10)
strong baselines based on T5 (Raffel et al., 2020) and large language models (LLMs).. This work contributes chal- lenging task, baseline methods tackle the task, and insights into compositional general- ization in the context dish name recognition.
Figure I: An example of compositional generalization in dish name prediction. Models are required to learn knowledge about braised and beef brisket and predict the composed dish name bruised beef brisket given the corresponding instructions.
1 Introduction
find the features corresponding to summarizing the staple ingredient beef brisket and abstract the ac- tion braise, thereby compose them into the name of this dish.
Compositional generalization, the capacity to com- prehend and create new combinations of observed components (Chomsky, 1956), has been shown to be an aspect of human intelligence both empiri- cally (Fodor and Pylyshyn, 1988) and experimen- tally (Lake al., 2019). Taking three recipes in Figure aa an example, by learning the instructions of cooking bruised pork (KI ( and marinated beef brisket ( / fi)) we can acquire knowledge about components in dish names, like the use of light søy sauce and dark soy sauce usually indi- cates the action braise, and beef brisket is more likely to be staple ingredient than spices. Given such knowledge, when reading the instructions of cooking braised beef brisket (*I ( we can
A bunch of synthetically been created for assessing compositional general- ization (Lake and Baroni, 2017: Bastings et al.. 2018; 8; Keysers et al., 2020; Kim and Linzen, 2020), and plain exhibit significant out-of-distribution (OOD) com- positional generalization performance loss compar- ing to in-distribution (ID) setting. While effective methods have been proposed overcome the diffi- culty in OOD compositional generalization (Lake, 2019; Nye et al., 2020; WeiBenhorn al., 2022), most of them mainly focus on semantic parsing, where some important abilities like summarization
datasets have
models
"Corresponding author.

---

Dataset
of samples
generalization forms
GEOQUERY (Shaw et al., 2021)
SPIDER-SS SSP (Shaw al., 2021)
4,376
SMCALFLOW-CS (Yin 2021)
28,054
COUNTERFACTUAL (Liu et al., 2022)
2,500
DINER(OurS)
223,581
Table Scale and diversity of non-synthetic compositional generalization datasets
and abstracting are not involved. Moreover, they are only evaluated on synthetic data, which is lack of natural language variation (Shaw et al., 2021) such as anaphora, omission and ambiguity (Keysers et al., 2020).
generalization. In this approach, training and test sets are partitioned in a manner that maximizes the divergence of compound distribution. The TMCD splits result in 4 categories of generalization forms, thereby raise more challenges in compositional generalization setting.
Recently, non-synthetic datasets have been pro- posed to evaluate compositional generalization on more realistic scenario (Shaw et al., 2021; Liu et al., 2022; Yin et al., 2021). However, as shown in Table 1, these datasets are either limited in data scale, which may lead to over-fitting and high vari- ance in evaluation; lack diversity of generaliza- tion forms (further explained in $2), which can result in biased models and limited generalization capabilities.
We develop baseline methods to solve the task, as existing compositional generalization models cannot fit in the dish name prediction task form. The first baseline, seq2seq trained T5, suffers from significant ID-OOD performance gap. Inspired by the chain of thought prompting method (Wei et al., 2022), we propose fine tune with composi- tional prompting (CP-FT). We first train a compo- nent model for predicting each kind of components, then we add the predictions of components to the in- put, and fine tune the model with the compositional prompt. To better fit the T5 model to the recipe domain, we also perform continual pretraining on large-scale recipe corpus equip models with do- main knowledge. The proposed CP-FT method and continual pretraining improve the OOD F1 score by large margin.
To address the aforementioned issues, we pro- pose the DIsh NamE Recognition (DINER) task, and build large-scale Chinese dataset with diverse collection of combinations. In this task, given the instructions of recipe, models are required pre- dict the dish name!. which is often composed of up three kind of components: food, actions and flavor. As visualized in Figure 1, models will be tested on unseen dish names composed by observed components.
With large scale data and distribution- split, the controllability of dataset is good. We demonstrate how evaluate the compositional gen- eralization ability of models on different training size and different level of distribution shift. We first find an inverse-scaling phenomenon between OOD performance and data scale. According to Kumar et al. (2022), on large distribution shift, fine tuning can distort pretrained features and under- perform OOD, We control the level of distribution shift, and empirically prove that the inverse-scaling phenomenon between OOD performance and data scale is caused by large distribution shift.
This task is not trivial without composi- tional generalization setting. Firstly, it takes sum- marization and abstracting to figure out which in- gredients, actions and flavors should be included in dish name. Secondly, the presence of anaphora, omission and ambiguity in recipes further increase the difficulty. We discuss such difficulty in $3.5 with more comprehensive analysis.
the XIACHUFANG Recipe Dataset (Liu al., 2022), we collect 3,803 dish names that can be parsed into ingredients. actions and flavors, with 223,581 corresponding recipes. As shown in Table 3, the large data scale brings a rich collection of combination forms. Then we generate TMCD (Shaw et al., 2021) splits of the dish names for assessing compositional
Based
Our contribution is three-fold: 1) We propose challenging compositional generalization task, Dish Name Recognition (DINER), and create large-scale realistic dataset. 2) We develop strong baseline: T5 with continual pretraining and fine tuning with compositional prompting and GPT-3.5 with seleetive demonstrations. 3) We demonstrate the controllability of our dataset and provide in-
'A dish refers to food prepared in particular way, and dish name is the conventional designation used by people to refer to that particular prepared food.

---

Category
# Dishes
Base Dishes
Target Dish
CombTwo
Beef, Bun
Beef Bun
AddOne
Noodles with Gravy, Steamed Perch
Steamed Noodles with Gravy
RecoTwo
619
Braised Pork, Marinated Beef Brisket
Braised Beef Brisket
Spicy Rice Noodle, Pork with Mushroom,
RecoThree
21
Braised Chicken
Rice Noodle with Mushrooms and Chicken
Table 2: Categories of generalization forms in TMCD splits
sights into compositional generalization in the con- text of dish name recognition.
datasets, our dataset has the advantage of data size and diversity compostional generalization forms.
2 Related Work
Dataset
Compositional generalization is the ability to under- stand and create combinations of elements and structures that have been seen before. To measure the compositional generalization ability of mod- cls, a series of studics have proposed synthetically- generated datasets based on semantic parsing tasks (Lake and Baroni, 2017; Bastings et al., 2018; Key- sers et al., 2020; Kim and Linzen, 2020). How- ever, the synthetic data lack natural language varia- tion (Shaw et al., 2021) such as anaphora, omission and ambiguity (Keysers al., 2020).
3.1 Data Preparation
We build DINER dataset large Chinese recipe corpus XIACHUFANG proposed by Liu et al. (2022), which consists of 1,550,151 real-world recipes from xiachufang. com. To adapt XIA- CHUFANG for dish name recognition task, we first collect (instruction, dish name) pairs from this cor- pus.
To collect (instruction, dish name) pairs, we face the challenge of identifying the actual dish name associated with each recipe. This is not straightfor- ward as XIACHUFANG provides a title that may not necessarily correspond to the dish name, for example, some titles include irrelevant descriptors and emojis. Based on the heuristic that frequently appearing titles stand good chance to be conven- tional dish name, we first clean the original titles by removing non-Chinese characters and content inside the parentheses, then keep the titles of 20 or more occurrences as dish names of their respec- tive recipes. Liu et al. (2022) proposed to map the title to a list of common dishes, we did not adopt this mapping method to avoid possible matching error. To enhance the quality of the recipes asso- ciated with each dish, we additionally eliminate any duplicates recipes containing fewer than 20 characters.
Recently, some works has proposed to evalu- ate the compositional generalization ability of the model on realistic data. Shaw et al. (2021) pro- posed two non-synthetic semantic parsing datasets, GEOQUERY and SPIDER-SSP and divided the data based on template, length and TMCD, but there are only hundreds thousands of sam- ples in these two datasets. Yin al. (2021) proposed larger scale semantic parsing dataset but this dataset only involves a compositional generalization from skills involving a single domain to skills involving two domains.
Liu et al. (2022) proposed counterfactual recipe generation task based on different combinations of ingredients, flavors and other components in recipe, and examined the model's ability to modify base recipe according the change of ingredi- ent. However, this task only involved 2500 pieces data and two forms of compositional generaliza- tion.
3.2 Dish Name Parsing
Chinese dish names are typically composed of food, actions and flavors (Yu and Huai, 2011; An, 2016), resulting in diverse range of combinations of these components. We build dish name parser to parse dish name to set of components, that can create a train-test split for evaluating compositional generalization based on various com- binations of components. Existing off-the-shelf Chinese word segmentation models are inadequate for parsing dish names due to the presence multi-
Inspired by Liu et al. (2022), we also study com- positional generalization based on the three com- ponents of food, action and flavor in recipes. Our dataset is based large-scale real-world recipe data generation. Compared with synthetic dataset, our dataset contains rich linguistic phenomena and combination forms. Our data splits result four types of compositional generalization forms. Com- paring to exsiting compositional generalization

---

component words. For example, braised pork $ ) is typically segmented single word, even though it consists of two distinct components: ac- tion ( and food pork(18) [.]). To this end, our parser uses maximal matching approach, with glossary of common food, actions and flavors con- structed as follows:
structions remains challenging, especially for Chi- nese ones (Liu et al., 2022). Therefore, in our approach, we utilize Target Maximum Compound Divergence (TMCD) (Shaw et al., 2021) 1) to create data splits, as it relies on the structure of outputs. TMCD split data by maximizing the divergence between compound distribution of training and test sets, while constrain the components in test set to appear least once in the training set.
Food Most of the food components in dish names are staple ingredients (e.g. pork, tofu) in the recipes. We collect such ingredients from the ingredient list of the recipe. There are also some food components indicating the form of the dish (e.g. pizza, bagel), which can be gathered by extracting the nouns in dish names.
We define compounds as subsets of the com- ponent set in dish names. For instance, in the dish name scrambled eggs with Chinese chives E My x
t with three components scramble ()))) eggs X f and Chinese chives ( the subset scrambled xg is compound. We em- ploy the greedy algorithm introduced by Shaw et al. (2021) generate splits that approximately maxi- mize compound divergence.
Actions The action components in dish names primarily refer to cooking techniques, such as fry- ing, stir-frying, and toasting. We extract the verbs from dish names to build the glossary of actions.
We study the construction of target dish names in the test set based on the observed components. As revealed in Table 2, there are four categories of compositional generalization forms in TMCD splits:
The flavor components are the taste ex- pression of a dish, e.g., sour and spicy. We extract the adjectives from dish names to build the glossary of actions.
Flavors
CombTwo: Combine two base single- component dish names into a target dish name.
Glossary Cleaning and Clustering The glossary collected in this manner contains many undesired components. To clean the glossary, we employ fil- tering process that removes components with fewer than 5 occurrences in dish names. Additionally, we manually check the glossary and eliminate multi- component words, resulting in 535 food items, 40 actions and 35 flavors. We also notice that some of the food items and actions are different mentions of the same food action, c.g. MVK and XB both refer to chicken feet. For fair evaluation and pre- venting data leakage in data splits, we cluster such food items or actions into on class, such that the components in one class are equivalent during eval- uation and generating data splits. The clustering results in 31 action classes and 413 food classes.
AddOne: Add a component from a base dish name to another hase dish name.
RecoTwo: Recombine components from two base dish names into target dish name.
RecoThree: Recombine components from three base dish names into target dish name.
Comparing to other approach to generate data splits, our approach provide more controllability that
evaluate compositional generalization different level of distributional shift (measured by compound divergence). To generate data splits of different compound divergence levels, we early stop the greedy algorithm at different times. We demonstrate how to evaluate compositional gener- alization on different levels of distributional shift in $6.
3.3 Distribution-based Data Splits
Most of the existing evaluations targeting com- positional generalization split data based on in- put/output patterns or length. Keysers et al. (2020) proposed to evaluate compositional generalization based on distribution, and generate data splits through Maximum Compound Divergence (MCD) approach. MCD requires that both input and output can be parsed into structured representations like trees or graphs. However, the parsing of recipe in-
3,4 Statistics
Basic Statistics We collect 3,803 dish names that can be parsed into these components. There are 223,590 recipes associated with these dish names, which averages to approximately 58.8 recipes per dish. Each recipe is of 180 characters and 8.3 steps on average. This extensive collection of data offers a wide range of combinations. There are 20 forms

---

Combination Forms
of Dishes
#o Recipes
Example
2 ingredients
1,080
64,421
Lettuce with Oyster Sauce (EE)))E))
3 ingredients
666
35,310
Congee with Minced Pork and Preserved Egg ( (1
action 2 ingredients
635
36,311
scrambled eggs with Chinese chives (115))))5
action ingredient
576
40,776
stir-fried pork (2)
flavor + ingredient
251
16,822
spicy chicken feet (B33))))))
Table 3: Top 5 combination forms in DINER Dataset.
Training
Test
of modeling the process of cooking and abstract the cooking skills and flavors.
# of Dishes
3,109
694
# Recipes
187,520
37,861
# of Components
479
383
Ambiguity Ambiguity occurs in components of dish names. For example, the word { means the action marinate in the recipe marinated beef brisket (i 4 (M))) (Figure 1), while it refers gravy, food component in another dish Noodles with Gravy (+T k H Among the dish names in our TMCD test set, we find 10 of them with ambiguous compo- nents. Such ambiguity makes it more challenging for models to summarize or abstract the compo- nents.
# of Compound
2,317
616
Table 4: Statistics of TMCD training and test set.
of combinations, and we illustrate the most popular combinations in Table 3.
TMCD Splits We divide the dish names into a training set and test set at approximate ratio of 4.5:1. Table 4 show the basic statistics of TMCD training and test set. The compound divergence of our TMCD splits is 0.98, and only 0.67% of the test set compounds are present in the training set. As for component occurrence, components in the test set appear 19.3 times on average in the training set. For comparison, we also generate random splits as in-distribution setting, where the (instruction, dish name) pairs are divided into training set and test set randomly.
3.6 Test Set Annotation
In-context learning (ICL) along with large language models (LLMs) has demonstrated impressive per- formance on both synthetic and non-synthetic compositional generalization datasets (Levy et al., 2022; An et al., 2023). However, because of the large data scale, assessing the entire test set on LLMs such as the GPT series incurs significant costs. Therefore, we propose to sample a high quality subset for LLM evaluation.
3.5 Pilot Study
We conduct pilot study focused on linguistic phe- nomena, aiming to analyze their impact on task complexity:
Although we have made efforts clean the recipe instructions, there are still instances where the information incomplete or contains too much irrelevant expressions, which means random sam- pling may result in an undesired test set for LLM evaluation. To mitigate this issue, we ask human experts to handpick a high quality recipe instruc- tion for each dish name in test set. The selected recipe instructions must furnish sufficient informa- tion to deduce the dish name and avoid excessive irrelevant content. This ensures that the samples in the test set provide sufficient information for dish name recognition and minimize the presence of irrelevant expressions.
Anaphora Recipe text contains rich anaphoric phenomena (Fang al., 2022). For example, in the recipe of marinated beef brisket in Figure 1, beef brisket is also mentioned as meat pieces. We ana- lyze 50 sampled recipes and find that each recipe has 1.9 anaphoric references of the staple ingredi- ent on average. This requires models to bridge the different mentions to decide the staple ingredient instead of simply counting the presences.
Omission Omission is prevalent phenomenon observed in Chinese (Huang, 1998), particularly in informal written contexts such as recipes. As shown in Figure the action stir fry well (b 5J recipe marinated beef brisket omits its object. each sampled recipe, there are 2.8 actions where the ingredient is omitted, which add to the difficulty
Baseline Methods
4.1 Fine Tuning T5
End-to-end Fine Tuning T5 (Raffel et al., 2020) is a strong baseline model for natural language un- derstanding (NLU) tasks like text summarization

---

and question answering. Some of these NLU capa- bilities are highly relevant to the dish name recog- nition task. We first build a baseline through fine tuning pretrained T5 in a seq2seq manner, where the input is <Instructions> What is the dish name of this recipe? and the output is the dish name.
(Demonstrations)
Input: "Slice the lotus root thinly..." What is the dish
name of this recipe?
Output: hot and sour lotus root slices (12 (6..2. ()
Input: "Slice the gizzards, heart and liver..." What the dish name this recipe?
Ouiput: chicken giblets with pickled pepper (:8
Fine Tuning with Compositional Prompting For compositional generalization tasks, it is good practice to enhance the primary task by address- ing the sub-tasks (Corona et al., 2021; Bursztyn et al., 2022). In the context of dish name predic- tion, there are three sub-tasks: recognizing the food /cttoon/ / flavor components. Based on this idea, we propose fine tuning with compositional prompting (CP-FT) to enhance compositional generalization in our task.
(Torget recipe)
Input: "Cut green pepper, red pepper and pickled pepper into small pieces..." " Whha is the dish name of this recipe?
Output:
Golden:
hot and sour chicken giblers A a2 (t)
GPT-3.5 3.5 Oulput: Hot and sour chicken giblets stir-fried with green poppor, red pepper and pickled pepper (Re
#4 A 1y I & éz. 8c.8
Figure 2: An example selective demonstrations. GPT- 3.5 learns the flavor hot and sour from demonstrations, but fails to figure out the staple ingredient.
We first train auxiliary model to address three sub-tasks in a seq2seq manner. For the action and flavor components, the input is <Instructions> What is the main action flavor of this recipe? and the output is the action / flavor, while for the food components, the input is <Instructions> What are the staple ingredients or food form of this recipe? and the output is the list of food components. If some of the components are missing, the output would be empty string. Prompting is good method to inject external knowledge or information language models (Li et al., 2022). To enhance our primary task, we add the auxiliary model's pre- dictions to original inputs as prompts during both the training and inference phases.
2022; An et al., 2023). Inspired by these princi- ples, for a test sample (target instruction, target dish name), we first retrieve diverse dish names with highest similarity to targer dish name from the training set. Then for each retrieved dish name, we find the instruction with highest similarity target instruction. We use the retrieved exemplars as demonstrations for LLMs, Figure shows an example of this procedure.
5
Evaluation
5.1 Evaluation Metrics
Previous works often use exact match accuracy evaluation metrics (Lake and Baroni, 2017; Bast- ings al., 2018; Keysers et al., 2020; Kim and Linzen, 2020). Nevertheless, such metric is too coarse for dish name recognition task, as it may overlook the distinctions between good and bad predictions. For example, given a golden dish name steamed noodles with gravy, a prediction steamed noodles is better than sleamed dumplings, but exact match accuracy cannot reflect such difference.
Continue Pretraining Continual pretraining is beneficial for domain adaptation (Wu et al., 2021; Kim et al., 2022). To better understand the recipe text and learn culinary knowledge about food, ac- tions and flavors, we continue pretrain the T5 model on the remaining XIACHUFANG corpus. We selectively retain recipes that have titles unrelated to any dish names in our dataset in order to avoid potential data leakage.
To measure the model performance more accu- rately, we first parse the prediction into compo- nents, then compute micro F1-score of predicted components given the golden components. To deal with those predictions that are unable to parse, suppose the golden component set is (c,,..., and the prediction is p. we define the truth pos- itive count be TP Ei 4[; E pl and ap- proximately count the number of components in p as the length of divided by the mean length of golden components ny = n len(p)/ / len(c;)
4.2 In-context Learning of LLMs
LLMs like GPT-3.5 with in-context learning has proven to be highly effective across a wide range of tasks (Logan IV et al., 2021; Bragg et al., 2021). However, in compositional generalization tasks, LLMs fail to generalize well under OOD settings (Hosseini al., 2022; Qiu et al., 2022). Some prin- ciples for demonstration selection like similarity and diversity have been proposed to improve com- positional generalization performance (Levy et al.,

---

Thus we can calculate precision np and recall TP/n.
TMCD
Random
EM
F1
EM
T5 +CP-FT
50.8
12.4
82.4
57.1
53.1
14.2
82.3
56.5
5.2 Experimental Settings and Hyper-Paramettrs
+Pretrain
55.8
18.4
84.3
60.3
+CP-FT & Pretrain
23.2
84.4
60.2
gpt-3.5-turbo
52.9
23.5
For T5-based methods, we use TS-base model pre- trained on large Chinese corpus (Zhang et al., 2021) base model. We use a batch size of 128 and learning rate of 5e We split 20% of the training samples validation set and stop training when validation loss stop decreasing. The Seq2seq train- ing takes about 4 hours on an A40 GPU. For con- tinual pretraining, we collect about 500k recipes and takes about 50 hours on an A40 GPU.
Table 5: Fi-score and exact match accuracy of T5-based methods under TMCD splits (OOD setting) and random splits (ID setting).
0.5
04
03
For LLM-base methods, we use the gpt-3.5- turbo model and evaluate it on the annotated test set. In examplar retrieval, we use Fl-score as the similarity score for dish names, and use BM25 to measure the similarity between recipe instructions. We only retrieve 2 examplars due to the lengthy text. To improve the diversity of demonstrations, after the retrieval of first dish name, we will use the uncovered components to retrieve the second examplar. It takes about dollars for API call.
03
BT5 RTSICN-Y
BTSHO (TBuretta
Figure 3: Fl-score on TMCD splits of each kind of components. When calculating the Fl score of one kind of components, the samples whose dish name doesn't have this kind of components will be ignored
generalization.
5.3 Results
LLM-based methods GPT-3.5 with selective demonstrations achieves a Fl-score of 52.9% and exact match accuracy of 23.5% on annotated test set. In comparison, T5 with continue pretrain- ing and CP-FT achieves Fi-score of 59.4% and an exact match accuracy of 22.9% on the same test set. GPT-3.5 fails to generalize on TMCD splits, and fall behind the fine tuned small language mod- els like T5 in Fl-score. GPT-3.5 get higher exact match accuracy than TS-based approach, which is possibly because it successfully recognizes small part of the recipes similar to observed recipes from its training data, but fails generalize larger part of unfamiliar recipes.
T5-based methods The evaluation results of T5- based methods are shown in Table 5. All the ap- proaches exhibit imperfect performance the ran- dom splits, proving that DINER task is challenging even under in-distribution setting. Comparing the results under TMCD splits and random splits, find that seq2seq fine tuned T5 model fails to gen- cralize to TMCD splits, and suffers from a severe ID-OOD performance drop: 31.6% in Fc-score and 44.7% in exact match accuracy.
Continue pretraining enhance the performance T5 both on TMCD splits and random splits, and the performace gain is larger on TMCD splits (5.0% in Fl) comparing random splits (1.9% in Fl). This may be due to the enhance on comprehen- sion of recipe text through continuous pretraining, enabling the model to overcome some of the chal- lenges associated with understanding unfamiliar recipes in the TMCD test set.
5.4 Analysis
Components We analyze the model's perfor- mance of recognizing each kind of components on TMCD splits. As shown in Figure 3, the food components are the easiest to recognize, and the flavor components are the hardest. Among the T5- based methods, continue pretraining enhance the performance all three kinds of components, indi- cating that pretraining on recipe text generally im- prove model's ability to understanding recipe text. The CP-FT approach enhances the performance on food components and action components, while
The proposed CP-FT method improves the F1- score of the original T5 and continue pretrained T5 by 2.3% and 3.4%. respectively, while achieve comparable performance on random splits. This means that the process of solving sub-tasks and enhancing the primary task with the help of sub- tasks is an effective approach for compositional

---

componnd shargence
compounn Ehergens
compoount shengenc +06
0.38
0.56
0.58
0.2
0.4
0.5
RaoTea
AwsTiwe
Figure 5: The Fl-score of continue pretrained T5 model under different data scales and different levels of distri- butional shift.
Figure 4: Fl-score TMCD splits of each category of generalization forms.
training (p E (0.2, 0.6, 0,8, 1.0)). We evalu- ate the continue pretrained T5 model three data splits (compound divergence 0.98, 0.8, 0.6) and five different ps.
underperforms on flavor components. The unde- sired performance on flavor components might be attributed to the unbalanced distribution of samples. Among the training data, only 12.8% of the sam- ples have flavor components, this may cause the auxiliary model to predict more empty strings, and mislead the primary task.
As revealed in Figure 5, under all the three data splits, there is inverse-scaling phenomenon be- tween OOD performance and data scale, which is opposite the usual case when the training data and test data are sampled from the same distribu- tion. According to Kumnr al. (2022), on large distribution shift, fine tuning can distort pretrained features and underperform OOD. We propose that the inverse-scaling phenomenon between OOD per- formance and data scale is caused by large distribu- tion shift. We empirically verify this comparing model's 's performance under different data splits. As the compound divergence decrease, the downward trend ease, indicating that the degree of inverse- scaling has a positive correlation with the level distributional shift.
Generalization Forms We calculate the F1- score of GPT-3.5 and T5 with continue pretraining and CP-FT on the annotated test set, and show the result in Figure 4. Model do surprisingly well on AddOne, however, we analyze model's predictions on AddOne and find that most of the predictions are the dish name from training set, which is short of one component than the golden dish name. In fact, the exact match accuracy of both models on AddOne is merely 9. 1.. This means model over- fits the training data and tend to generate observed dish names. Among the other three categories, both models performs best on CombTwo, and worst on RecoThree, which agrees with our intuition on the complexity of these two categories of generaliza- tion forms.
7 Conclusion
We propose DIsh NamE Recognition (DINER) task to evaluate natural language understanding ability under compositional generalization settings, and create large-scale realistic dataset with rich com- binations and generalization forms. We provide two strong baselines: 1) T5 with continual pretrain- ing and fine tuning with compositional prompting; 2) GPT-3.5 with selective demonstrations. Our T5-based approach outperforms the plain seq2seq trained version by large margin (8.4%). How- ever, the ID-OOD performance gap remains un- solved. We also demonstrate the controllability of dataset and provide insights into composi- tional generalization in the context of dish name recognition. By proposing this task, we provide a realistic and flexible benchmark for evaluating generalization.
6 Demonstrations of Data Controllability
With the large data scale and distribution bassed data splits, the DINER dataset has good control- lability, which enable to study the model's per- formance under different data scales and different levels distributional shift. In this section, we demonstrate such controllability by evaluating the continue pretrained model under multiple condi- tions.
To generate data splits of different levels of distri- butional shift, we early stop the splitting algorithm and generate another two splits with compound di- vergence 0.8 and 0.6. For each data splits, we sample different proportion p of training data for

---

Limitations
Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NIP, pages 47-55.
Limitation of dish name parser. Our dish name parser is based maximal matching. While this method has proven efficient in some instances, does come with inherent limitations such as am- biguity issues. For example, if a dish name can be segmented in multiple ways, the parser may struggle to determine the correct segmentation.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems, 34:15787-15800.
Victor Bursztyn, David Demeter, Doug Downey, and Larry Bimbaum. 2022. Learning to perform complex tasks through compositional fine-tuning of language models. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 1676-1686. Association for Computational Linguisties.
Imperfection of current solutions. While our proposed methods offer a strong baseline for com- positional generalization in the task of dish name recognition, they do not fully address the issue. Given the challenges outlined, we aim to develop more effective solutions in future.
Noam Chomsky. 1956. Syntactic structures. Mouton de Gruyter.
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. 2021. Modular networks for com- positional instruction following. In North American Chapter of the Association for Computational Lin- guistics.
Scalability to other languages. Although our ini- tial investigation centered around Chinese dishes, we acknowledge the importance of validating our conclusions across different languages. For in- stance, in English, dish names such as Spicy Fried Chicken and Mashed Sweet Potatoes are also com- posed of ingredients, flavors, and actions. Addi- tionally, there are extensive recipe databases like recipelM: MM+ available, which could be utilized to create a compositional generalization dataset for dish name recognition in English. Therefore, recognize the potential extend our research to other languages and will consider this as part of our future endeavors.
Biaoyan Fang, Timothy Baldwin, and Karin Verspoor. 2022. What does it take to bake a cake? the reciperef corpus and anaphora resolution in procedural text. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3481-3495
Jerry A Fodor and Zenon W Pylyshyn. 1988. Connec- tionism and cognitive architecture: A critical analysis. Cognition, 28(1-2)33-71
Arian Hosseini, Ankit Vani, Dzmitry Bahdanau, Alessandro Sordoni, and Aaron Courville. 2022. On the compositional generalization gap of in-context learning. arXiv preprint arXiv:2211.08477
Acknowledgments
C-T James Huang. 1998. Logical relations in Chinese and the theory of grammar. Taylor & Francis.
This work is supported by NSFC (62161160339) We would like thank the anonymous reviewers for the helpful suggestions, and our great annota- tors for their careful work. Also, we should thank Quzhe Huang and Chen Zhang for their detailed comments. For any correspondence, please contact Yansong Feng.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov. Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. 2020. Measur- ing compositional generalization: comprehensive method on realistic data. In 8th International Confer- Learning Representation.
References
Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. 2022. A broad study of pre-training for domain generalization and adaptation.
Shengnan An. Zeqi Lin. Qiang Fu, Bei Chen, Nanning Zheng. Jian- GuanG Lou, and Dongmei Zhang. 2023. How do in-context examples affect compositional generalization? arXiv preprint arXiv:2305.0433
Najoung Kim and Tal Linzen. 2020. COGS: compo- sitional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pages 9087-9105. Association for Computational Linguistics.
Xinkui An. 2016. Zhoncan caiming de fanyi ji qianzhici de shiyong. Zhongguo eyu jiaoxue, 3:40 44.
Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. 2018. Jump to better conclusions: Sean both left and right. In
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can

---

distort pretrained features and underperform out-of- distribution. In International Conference on Learn- ing Representations (ICIR).
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generaliza- tion and natural language variation: Can semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume 1: Long Papers), pages 922-938.
Brenden M Lake. 2019. Compositional generalization through meta scquence-to-sen -equeneecee learning. Ad- vances in neural information processing systems, 32.
Brenden M. Lake and Marco Baroni. 2017. General- ization without systematicity: On the compositional skills of sequence-to-sequuee recurrent networks. International Conference on Machine Learning.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1700..
Brenden M. Lake, Linzen, and Marco Baroni. 2019. Human few-shot learning of compositional instruc- tions. In Annual Meeting of the Cognitive Science Society.
Pia Weißenhorn, Lucia Donatelli, and Alexander Koller. 2022. Compositional generalization with broad- coverage semantic parser. Proceedings of the 11th Joint Conference on Lexical and Compulational Se- manties, pages 44-54.
Itay Levy, Ben Bogin, and Jonathan Berant, 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06000.
Han Wu, Kun Xu, Linfeng Song, Lifeng Jin, Haisong Zhang, and Linqi Song. 2021. Domain- pre- training methods for dialogue understanding. arXiv preprint arXiv:2105. 1366.
Haochen Li, Tong Mo, Hongcheng Fan, Jingkun Wang. Jiaxi Wang, Fuhao Zhang, and Weiping Li. 2022. KiPT: Knowledge-injected prompt tuning for event detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1943-1952, Gyeongju, Republic of Korea. Interna- tional Committee on Computational Linguistics.
Pengcheng Yin, Hao Fang. Graham Neubig. Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span- level supervised attention. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2810-2823, Associa- tion for Computational Linguistics.
Xiao Liu, Yansong Feng. Jizhi Tang, Chengang Hu, and
Dongyan Zhao. 2022. Counterfactual recipe gener-
ation: Exploring compositional generalization in realistic scenario. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7354-7370. Association for Com- putational Linguistics.
Zhentao Yu and Lihua Huai. 2011. Zhongguo caidian mingming fangfa yu guifanhua yanjiu. Nanning zhiye jishu xueyuan xuebao, 4:1-3.
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet inge- nious pre-trained models for chinese.
Robert L Logan IV, Ivana Balažević, Fric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint wrXi:2216613355
Maxwell Nye, Armando Solar-L Leaama, Josh Tenen- baum, and Brenden M Lake. 2020. Learning compo- sitional rules via neural program synthesis. Advances Neural Information Processing Systems, 33:10832- 10842.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. 2022. Evaluating the impact model scale for compositional generalization in semantic parsing, arXiv preprint arXiv:2205.122 12253.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text tranffrmer. Journal of Machine Learning Research, (140):1667.


Reviews:
[{"review_id": "XGWhsHd93f", "reviewer": "Reviewer_DC41", "summary": "", "questions": "- See reasons to reject.\n- At ln. 340 you mention asking experts to handpick a high quality recipe instruction, but it is a bit difficult to grasp what is a high quality instruction as it is not explained. \n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer DC41,\n\nWe appreciate your thorough reviews and insightful feedback. We would like to address your questions and concerns as follows:\n\n**A) Missing statistics for the dataset**\n\n>  Some useful statistics for the dataset are missing: compound frequency, atom frequency/occurence or co-occurence.\n\nWe acknowledge the significance of compound and atom frequencies, which were initially omitted from our paper due to space limitations. To address this, we provide aggregated statistics as follows:\n\n|                            | Training Set | Test Set |\n| -------------------------- | ------------ | -------- |\n| number of atoms            | 479          | 383      |\n| average atom frequency     | 0.0021       | 0.0026   |\n| number of compounds        | 2317         | 616      |\n| average compound frequency | 0.0004       | 0.0016   |\n\nAs for atom co-occurence, as stated at ln. 235 that the TMCD splits \"constrain the components in test set to appear at least once in the training set\". Specifically, test set atoms appear 19.3 times in average in the training set. \n\nMoreover, only 0.67% of the test set compounds are present in the training set, highlighting the TMCD's ability to minimize compound co-occurrence.\n\nIn our revision, we will provide a comprehensive statistical analysis that includes these aspects.\n\n**B) The language-transferablility of the findings/conclusion**\n\n> A point is unclear, is this a multi-lingual dataset? \n\nThe dataset is in Chinese. We apologize for any confusion and will make sure to clarify this by explicitly stating that the dataset is in Chinese within the introduction and abstract in our revision.\n\n> Are the findings/conclusion on compositional generalization from one language valid for another one?\n\nRegarding the applicability of our findings and conclusions on compositional generalization to other languages, although our initial investigation centered around a single language, we acknowledge the importance of validating our conclusions across different languages. \n\nFor instance, in English, dish names such as *Spicy Fried Chicken* and *Mashed Sweet Potatoes* are also composed of ingredients, flavors, and actions. Additionally, there are extensive recipe databases like recipe1M+ available, which could be utilized to create a compositional generalization dataset for dish name recognition in English. Therefore, we recognize the potential to extend our research to other languages and will consider this as part of our future endeavors.\n\n**C) \"high-quality recipe instruction\" in line 340**\n\n> At ln. 340 you mention asking experts to handpick a high quality recipe instruction, but it is a bit difficult to grasp what is a high quality instruction as it is not explained.\n\nThe recipe instructions for test set were handpicked to exclude the instances that have incomplete information or contain excessive irrelevant expressions (as mentioned at ln. 335). So the criteria for high-quality recipe instructions are as follows\n\n- The recipe instructions must furnish sufficient information to deduce the dish name.\n- The recipe instructions must avoid excessive irrelevant content, such as informal chat or descriptions of other dishes.\n\nWe will include a detailed description of test set annotation in or revision.\n\n**D) Typos Grammar Style And Presentation Improvements** \n\nIn our revision, we will add a brief explanation of TMCD when introduce it and correct the grammatical mistake.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem of predicting dish names from recipe text is interesting, especially framing the task under the composional generalization umbrella.\n- The propose dataset and the categorization of the task with increasing complexity appear to be valuable. \n- The baselines proposed are of interest, the worflow is clear and understanble.", "reasons_to_reject": "- Some useful statistics for the dataset are missing: compound frequency, atom frequency/occurence or co-occurence.\n- A point is unclear, is this a multi-lingual dataset? Are the findings/conclusion on compositional generalization from one language  valid for another one?", "questions_to_authors": "- See reasons to reject.\n- At ln. 340 you mention asking experts to handpick a high quality recipe instruction, but it is a bit difficult to grasp what is a high quality instruction as it is not explained. \n\n", "paper_topic_and_main_contributions": "This paper proposes a novel large-scale dataset and task called dish name recognition associated with the task of compositional generalization which is the ability to generalize to novel combinations of given familiar units.\nSpecifically, the authors build upon the XIACHUFANG recipe dataset collecting dish names and parse them to collect the units under the form of ingredients, flavours and actions. Loosely speaking, the novel task consists in predicting the name of the dish given the recipe instructions. The authors introduce two different baselines to assess the validity of the proposed method both based on LLM-based models a) T5 + fine-tuning and (b) GPT-3.5 with selective demonstrations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing reference.", "Typos_Grammar_Style_and_Presentation_Improvements": "- ln. 040: \"to summarizing\"\n- ln. 095: the acronym TMCD is introducted here but the meaning is explained at ln. 230"}, "extra_text": {}}, {"review_id": "76mOTywU0D", "reviewer": "Reviewer_m8Dw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer m8Dw,\n\nWe sincerely appreciate your thoughtful review of our paper. We've carefully considered your comments and would like to address the points you raised.\n\n> Compared to previous compositional generalization datasets for semantic parsing, the proposed dish name recognition have a narrower downstream application.\n\nWe acknowledge your point about the narrower downstream application of our proposed dish name recognition dataset compared to previous compositional generalization datasets for semantic parsing. While it's true that the application scope might seem more constrained, we believe this specificity offers unique advantages. This focused task provides a foundation for evaluating compositional generalization in a domain that mirrors real-world scenarios, such as recipe recommendation systems, culinary AI assistants, and menu generation. Therefore, even though the application may appear narrower, its relevance to practical contexts should not be underestimated.\n\nOnce again, we sincerely appreciate your time and attention to our paper. Your insightful review has enriched our perspective, and we're dedicated to addressing the concerns you've raised to improve the overall quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed large-scale, realistic dataset is a good resource contribution for compositional generalization. This is important as LMs are becoming more powerful. \n2. The proposed compositional prompting shows potential in a relatively small LM, helping it outperform the much larger GPT-3.5. ", "reasons_to_reject": "1. Compared to previous compositional generalization datasets for semantic parsing, the proposed dish name recognition have a narrower downstream application.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a large-scale, realistic dataset (DiNeR) consisting of 3k dishes and 228k recipes to evaluate compositional generalization. In this dataset, the input is a recipe instruction, and the output is the dish name, which requires an understanding of diverse combinations of ingredients, actions, and flavors. The authors proposed compositional prompting and leveraged continual pretraining to improve compositional generalization. The experimental results showed that the proposed method outperforms T5 and GPT-3.5 on the TMCD split, demonstrating stronger compositional generalization capability. The large scale of data also enables fine-grained control over the distributional shift, making it a testbed for diagnosing compositional generalization. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZgkavPup9s", "reviewer": "Reviewer_A6aj", "summary": "", "questions": "A) Why is the f1 metric preferred to exact match (also in view of what written above)?\n\nB) In the dataset preparation how much were human involved (cleaning the data, the labels ...)? It is not clear from the description whether it was mostly automatic work and it would be interesting to know/expand on it", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer A6aj,\n\nWe appreciate the thorough review of our paper and the insightful feedback provided by the reviewer. We would like to address the questions and concerns raised by the reviewer as follows:\n\n**A) Choice of Evaluation Metric** \n\n> In the experimental part, the f1 evaluation metric might not always be well suited. For example, in the AddOne category it should be easy for the model to \"Classify\" the parts already seen in training, which the metric rewards, while the difficult part is the generalization in the new component.\n\nWe understand the point raised about the AddOne category and how the F1 metric might favor parts seen in training. Actually, we have pointed out that F1 metric overestimates model's compositional generalization ability on the AddOne category at line 519.\n\nTo address this issue, we will include exact match metric when analyzing the generalization categories in our revision. For the whole test set and annotated test set, the exact match of models on AddOne category are as follows: (GPT-3.5 was only evaluated on the annotated test set)\n\n| test set              | T5   | T5+CP-FT | T5+Pretrain | T5+CP-FT&Pretrain | GPT-3.5 |\n| --------------------- | ---- | -------- | ----------- | ----------------- | ------- |\n| whole(756 samples)    | 11.1 | 12.3     | 15.2        | **29.9**          | -       |\n| annotated(11 samples) | 9.1  | 9.1      | 9.1         | 9.1               | 9.1     |\n\nThe exact match score for all models on the annotated test set is 9.1. However, this score might not accurately capture differences among the four methods due to the limited number of AddOne samples available for comparison. \n\nWhile on the whole test set, the compositional prompt method improve the exact match on AddOne category by a large margin, which is coherent to our findings.\n\n> Why is the f1 metric preferred to exact match\n\nAs mentioned at line 414, we favor the F1 metric over exact match (EM) because exact match can overlook nuances between predictions. For instance, given the reference \"steamed noodles with gravy\" the predictions \"steamed noodles\" and \"steamed dumplings\" are both incorrect by EM, but in reality, \"steamed noodles\" is a closer match. F1 considers precision and recall, capturing such distinctions better. For \"steamed noodles,\" F1 is 0.8 (precision=1, recall=2/3), and for \"steamed dumplings,\" it's 0.4 (precision=0.5, recall=1/3), offering finer evaluation.\n\n**B) Dataset Preparation and Human Involvement**\n\n> In the dataset preparation how much were human involved (cleaning the data, the labels ...)?\n\nThe steps of dataset preparation that involves human labor are dish name parsing and test set annotation:\n\n- In dish name parsing, human were involved when checking the glossary of food, actions and flavors (mentioned at line 206)\n- In test set annotation, human experts were asked to handpick a high quality recipe instruction for each dish name in test set, where the recipe instructions must furnish sufficient information to deduce the dish name and avoid excessive irrelevant content\n\nWe will provide a more detailed description of the dataset preparation process in our revision to address this concern.\n\n**C) Typos, Grammar, Style, and Presentation**: \n\nWe appreciate the suggestion to mention the language of the dataset (Chinese) in the abstract or introduction to provide immediate clarity to readers. We will make sure to include this information prominently.\n\nFurthermore, we acknowledge the recommendation to include a comprehensive related work section, especially focusing on compositional generalization. We will expand the related work section to provide a more thorough overview of existing research in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a new large dataset for compositional generalization in natural language and with different generalization categories.\n\n- The authors presents an experimental setup with some strong baseline methods. The results shows that the compositional splits are indeed hard w.r.t. to the iid one.\n\n- The paper is clear and well written.", "reasons_to_reject": "- In the experimental part, the f1 evaluation metric might not always be well suited. For example, in the AddOne category it should be easy for the model to \"Classify\" the parts already seen in training, which the metric rewards, while the difficult part is the generalization in the new component.", "questions_to_authors": "A) Why is the f1 metric preferred to exact match (also in view of what written above)?\n\nB) In the dataset preparation how much were human involved (cleaning the data, the labels ...)? It is not clear from the description whether it was mostly automatic work and it would be interesting to know/expand on it", "paper_topic_and_main_contributions": "The paper proposes a new dataset (DiNeR) for compositional generalization. The dataset consists of dish recipes with the target being the name of the dish, which is compositional of food, actions and flavors descriptors. DiNeR is in Chinese, features natural language and has a large size. The authors provides TMCD splits with different compositional generalization categories and experimental results with sequence to sequence models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Consider adding a statement that the dataset is in Chinese in the introduction or even in the abstract if possible, so that it is immediately clear to the reader.\n\n- While there are some references in the introduction, please consider adding a comprehensive related work section, especially for compositional generalization since there are many works on it currently."}}]

Author rebuttal:

Meta-review: This paper presents a large-scale realistic dataset, DIsh NamE Recognition (DINER), comprising  3k dishes and 228k recipes. This paper also presents the associated task (predicting the name of the dish given the recipe instructions) to evaluate compositional generalization. 
Experiments are well conducted and evaluated various models on this task (fine tuning T5, in-context learning LLM).
The article could benefit from the addition of corpus statistics. It would also be good to clarify the fact that it's in Chinese (which doesn't detract from the interest of the corpus) and that the conclusions may not apply.
Decision: Accept-Main