Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces a bilingual multi-domain dataset for end-to-end task-oriented dialogue modeling. Their dataset is large, diverse, and publicly available with over 7k multi-domain dialogues (144k utterances) together with a large and realistic bilingual knowledge base. The authors evaluate several state-of-the-art bilingual ToD baselines and show the importance of training bilingual dialog models over monolingual ones, which their dataset can help with. The authors also explore several cross-lingual transfer baselines in enabling ToD under low resource languages.
strengths: 1. The dataset will be useful towards better evaluating multilingual dialog systems for the development of robust end-to-end dialog systems for multilingual countries and regions.
2. The dataset and code is already publicly available.
3. The dataset is quite large and diverse (and multilingual) in comparison with existing dialog datasets.
4. The authors have done a good job implementing several classes of baselines and the results look interesting.
weaknesses: 1. There needs to be comparison with other multilingual dialog datasets and models beyond english and chinese, such as
Schuster et al., Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog. NAACL 2019 https://arxiv.org/abs/1810.13327
2. There could be more discussion regarding the annotation process since there may be possible biases resulting from the annotation process given that native english or chinese speakers follow different distributions, and may have different ways of responding and completing the dialog task. More discussion regarding checks for the accuracy of annotations would also help the paper.
3. There should be some references to machine translation literature and how using pretrained machine translation models for cross-lingual transfer can help.
correctness: 1. The evaluation methods and experimental design seem to follow existing work in dialog modeling (dialog state tracking and end-to-end task completion).
2. The authors have implemented and tested a good number of recent baselines on their dataset.
3. I am curious why the authors chose english and chinese as their 2 languages in the datasets given their very different vocabularies and language similarities. This might be a cause for poor cross-lingual transfer performance.
4. The paper does not seem to mention how annotators are selected, and whether english or chinese were the annotator's native language. There may be possible biases resulting from the annotation process given that native english or chinese speakers follow different distributions, and may have different ways of responding and completing the dialog task.
5. There should be some references to machine translation literature and how using pretrained machine translation models for cross-lingual transfer can help.
clarity: 1. Yes, the main paper is generally well written. The appendix could benefit from proofreading.
2. The figures and tables look good.
relation to prior work: 1. There could be more references and comparison to multilingual dialog datasets and models beyond english and chinese, such as these which I found on a cursory google search:
Schuster et al., Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog. NAACL 2019, https://arxiv.org/abs/1810.13327
Lacalle et al., Building a Task-oriented Dialog System for languages with no training data: the Case for Basque. LREC 2020, https://aclanthology.org/2020.lrec-1.340.pdf
2. Some references to machine translation literature would be relevant too, especially when applied in a dialog setting.
additional feedback: none.
review_title: review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a new dataset BiToD, a task-oriented dialogue benchmark in bilingual setting (English and Chinese) across multiple domains (such as: restaurant, attraction).  Experiments conducted on two state-of-the-art baselines demonstrate the need of such bilingual dataset.  
strengths: 1. The authors provides detailed explanation about the data collection process, intended data usage,  dataset statistics and maintenance plan.  
2. Two sate-of-the-art baseline models are evaluated across three evaluation settings (monolingual, bilingual, and cross-lingual). Analysis show leveraging bilingual data can train a better ToD system for both languages.
weaknesses: I feel some details/baselines are missing for better understanding about the dataset. I might be wrong about this, as I am not very familiar with dialogue systems.

(1) Is there one-to-one correspondence between the English dialogues and Chinese dialogues. For example, an English dialogue and a Chinese dialogue generated with the same user goal, hence share similar dialogue outline in meaning. 

If so, can we obtain a baseline performance based on machine translation. Is it possible to directly transfer a learned model with English training data to Chinese testing data via machine translation?

(2) How many paraphrased dialogues are collected for a dialogue outline? 

(3) Upper bound of model performance or human performance.
For all accuracy measure, I suppose the upper bound is 100%. It may be difficult to get human performance to mimic the whole dialogue pipeline. However, for generated response, what kind of BLEU score would suggest the generated response is good.  

(4) Cross-domain transferability. As BiToD includes dialogues from different domains, would it be possible to evaluate a dialogue system on cross-domain transferability? For example, train the model with data in one domain and evaluate on data in other domains.

(5) Visualization of model predictions. It would be nice to have a check on the qualities of model generated responses and identify the failure cases under different evaluation settings to highlight the difficulties of this bilingual dataset.

(6) Under the cross-lingual setting, if we keep increase the training data in the other language (e.g., the percentage of English data used for ZH->EN ) , when would the model get comparable performance with using the full English data under monolingual setting.
correctness: Yes. The dataset collection process sounds reasonable to me. 
clarity: Overall, I feel it is easy to follow the paper. 
relation to prior work: Yes. In section 6, the authors have summarized prior works on building datasets for end-to-end task-oriented dialogue modeling. The biggest difference between BiToD and prior works is the bilingual nature of BiToD.  

The authors have also provided detailed comparison of dataset statistics/features between BiToD and previous datasets in Table 2. In addition to the bilingual nature, BiToD provides several additional features: Deterministic API, Complex User Goal, Mixed-Language Context and Cross-API Entity Carry-Over. 
additional feedback: 
It is great to further extend BiToD to other languages as future work. What about diversifying the knowledge base? For example, current knowledge base are about entities in Hongkong, would it be possible to extend the dataset to other regions (cities or nations)? I am curious about whether a dialogue system trained with data designed for one city can be transferred to another city. I suppose the model should have leaned how to identify the APIs, understand the goal of a dialogue and etc. I would like to hear about the authors' thoughts.


**Post-rebuttal Updates** The authors have addressed my concerns. So I raised my rating to acceptance.
review_title: A new bilingual multi-domain dataset for end-to-end task-oriented dialogue modeling

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper describes a bilingual (chinese-English) multi-domain dataset for end-to-end task oriented dialogue. The paper involves the following contributions:

- First ToD bilingual multi-domain dataset
-  Three baselines for a range of settings (monolingual, bilingual, crosslingual)
- Evidence for using bilingual instead of monolingual ToD systems

The paper is clearly written and easy to follow in general, while the ethical section is also clear and straightforward with no problems whatsoever. This is definitely an important resource for dialogue NLP practicioners. A major weakness however, which is also quite commonly found in other papers in NLP, is the disregard of earlier work in dialogue modelling in linguistics and computational linguistics before the advent of DL in NLP. This is of course not a problem a priori, but because of that, misses key aspects of dialogue that have been extensively been discussed in the literature on dialogue in the last 40 years or so. The most clear example of the latter is that dataset involves cases of clean cut dialogue, namely cases where the abundant disfluencies and idiosyncracies found in natural language, i.e. interjections, split utterances, word repetitions, are not found. This is understandable for datasets that serve a specific practical application, but nevertheless casts doubt into claims on how "natural" such datasets are. In any case, datasets, of which the dataset described in the paper under cosideration is not an exception, involve a fraction of the wealth of patterns found in natural language dialogue. This does not mean that the datasets are not useful. To the contrary: they are extremely useful, but it's good time that these issues start to be discussed in the literature and properly acknowledged. 

In general the paper is an important contribution to the community and I could see it being accepted subject to the authors taking into consideration a number of more specific criticisms that are raised below:

- The authors claim that the dataset involves also cases of code-switching. Technically, this is true. However, code-switching is very restricted to the interchange between Chinese and English language names (i.e. of restaurants). This limitation has to be noted. 
- Give a clear definition of what constitutes a dialogue
- Give a definiton of what multi-domain is meant to be and what the limitations of those domains are (e.g. how they can be extended to include more domains) 
- Discuss the issue of wealth of dialogue patterns found in the dataset and why common features found in spontaneous dialogue (e.g. difluencies, split utterances) are not found in this dataset. 
strengths: - First dataset of its kind
- Sound collection procedure and useful baselines
weaknesses: - Limited variability compared to natural language dialogue patterns
- Not informed by any previous theoretical work on dialogue in theoretical and computational linguistics before the advent of deep learning in NLP. 
correctness: Dataset is constructed in a sound way and the evaluation methods are appropriate. 
clarity: The paper is clearly and well-written and easy to follow. 
relation to prior work: There is an adequate discussion on similar datasets. What is missing, which is something that most recent papers in the same category seem to miss, is a discussion or rather being informed from earlier literature in dialogue modelling spanning at least fifty years of research that could greatly inform the collection and variability of dialogue patterns found in natural language. 
additional feedback: Some references for the authors w.r.t dialogue work that can be useful in the discussion of dialogue disfluencies, range of dialogue patterns: 

Raquel Fernández. Non-Sentential Utterances in Dialogue: Classification, Resolution and Use. PhD thesis, Department of Computer Science, King's College London, University of London, London, UK, 2006.
Howes, C. (2012). Coordination in dialogue: Using compound contributions to join a party. Phd Thesis, Queen Mary University of London.
Gregoromichelaki, E., & Kempson, R. (2016). Joint utterances and the (split-) turn taking puzzle. In Interdisciplinary studies in pragmatics, culture and society (pp. 703-743). Springer, Cham.
Ginzburg J, Fernández R, Schlangen D. Disfluencies as intra-utterance dialogue moves. Semantics and Pragmatics. 2014 Jun 10;7:9-1.
Ginzburg, J., 2012. The interactive stance. Oxford University Press.
review_title: This paper describes a bilingual (chinese-English) multi-domain dataset for end-to-end task oriented dialogue to be used as a benchmark for evaluating ToD systems

Meta-review: This dataset introduces the first large-scale (144K utterances over 7K dialogues) bilingual multi-domain dataset for end-to-end task-oriented dialogue modeling. All reviewers agreed the experiments and baselines are sound and clearly presented. The experiments are done in monolingual, bilingual, and cross-lingual settings. The authors distinguish their dataset with existing datasets, which either only focuses on English or does utterance-level study instead of dialogue-level study for different languages. 

The authors have improved the paper based on the reviews, adding additional baseline (translate-test), results (using more target language in cross-lingual settings), and discussions (about limitations of the work). 

Lastly, as the authors bring up Hong Kong as the reason they investigate English and "Chinese", it would be better to be more specific which Chinese they are focusing on (Mandarin or Cantonese), as Cantonese is (still) the majority Chinese language in HK. If they are not making such distinctions during data collection, this should be clarified in the paper. Lastly, it's still unanswered who annotated this dataset -- were they native speakers of the annotated languages. Please add these details to revised paper. 
Decision: Accept