Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a formal grammar for describing complex queries on knowledge graphs, and extends the total number of possible queries.
strengths: * The framework is formal, extensible, and the particular choice of operators seems to have been thought through.
* Evaluation methods follow existing works
weaknesses: This paper may not be accessible to those without a background in formal logic, which may encompass most of those in ML (including myself). This may hinder users' understanding and also adoption of this benchmark. This paper may benefit from just a bit more explanation of tools used (Skolemization, prenex normal forms, etc), and why those choices make sense for this dataset

correctness: The paper seems correct.
clarity: Please see above note about explaining choices on tools from formal logic.
relation to prior work: The paper might benefit from a bit more explanation of the deficiencies of previous datasets, e.g. expanding or defending the statement that "existing datasets only contain queries from very few types, which might be insufficient for the investigation of combinatorial generalization"
additional feedback: n/a
review_title: A formal grammar for complex query answering on knowledge graphs

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper extends the scope from a few hand-crafted query types to the family of Existential First-Order queries with Single Free Variables, with an extendable framework. Based on this, a large-scale dataset of combinatorial queries is built. 
strengths: 1. An extendable framework is built by extending the scope from hand-crafted query types to the family of EFO-1.
2. A large-scale dataset is built for combinatorial queries.
3. New findings are observed for normal forms, training, and generalization. 
weaknesses: Related work should be introduced and compared.
correctness: The method is sound and correct.
clarity: This paper is well written. 
relation to prior work: Related work should be introduced and compared.
additional feedback: I am not quite familiar with this research topic, but I think this paper is well written.
review_title: A good benchmark for combinatorial generalizability of complex query answering on knowledge graphs

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper builds  a new dataset EFO-1-QA to benchmark the combinatorial generalizability of CQA models. This dataset consists about 301 query types which is larger than previous work. Meanwhile, the author explore the impact of the different operators and normal forms.  The proposed dataset is very important and will have a great impact to the community and promote future researches.
strengths: a. The larger query types and more data make the evaluation of the combinatorial generalization more comprehensive and convincing.

b. The deep analysis of normal forms and the investigation about how training query types affect the generalization are of great help to the future research. 

c. The proposed method is general enough to be compatible with different choices of 143 operators.

weaknesses: Some technical  details of baselines are missing, which may make some readers confusing.
correctness: The claims made in this paper is correct.
clarity: The author clearly demonstrates the construction of the dataset and the figures and tables in the paper are vivid.  
relation to prior work: The proposed dataset extended the query types in previous dataset. 
additional feedback: No
review_title: A great dataset about evaluating the combinatorial generalizability of CQA models.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents EFO-1-QA, a new synthetic dataset to benchmark the combinatorial generalizability of complex question answering (CQA) models. It provides a detailed description of the combinatorial generalizability problem and an overview of the proposed dataset but has a weak positioning. I recommend the authors change the paper's focus from the synthetic dataset to the comprehensive and systematic benchmark of useful CQA models.
strengths: - The provided dataset fills a gap in combinatorial generalizability of CQA models evaluation
- Both code and data are available (however, the license is not specified)
weaknesses: - The synthetic nature of the dataset is not justified
- Lack of clarity when describing the dataset and benchmark
correctness: I do not see any methodological issues in this paper.
clarity: The paper and the supplementary materials in great detail and clarity describe the problem for which the dataset has been created. I believe the paper would benefit from adding the evidence that real-life CGA on KGs benefits from tuning or evaluating combinatorial generalizability datasets.

Since the authors paid great attention to combinatorial generalizability, I expected this paper to focus on the benchmark, analysis, and discoveries rather than the dataset itself. I think this is the main weakness of the present form of the paper.
relation to prior work: Since the authors have produced EFO-1-QA using depth-first search, it will be useful to compare it to similar datasets, showing the similar and dissimilar parts.
additional feedback: After the clarifications and updates provided by the authors, I have updated my score.
review_title: A synthetic dataset for evaluating complex query answering methods

Meta-review: This paper presents a dataset for complex question answering. The dataset is significantly larger than previous datasets, and this problem space is underexplored but important. Reviewers' comments were addressed well in the rebuttals. All reviewers agree that this paper has merit and potential contribution for future research in this topic.
Decision: Accept