Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 1) The paper proposed BINDER to incorporate symbolic component into large language models, and the main benefit of the method is not requiring any fine-tuning.
2) BINDER outperforms state-of-the-art results on WIKITABLEQUESTIONS and TABFACT datasets 
3) useful analysis is presented to help readers understand the proposed approach.
strength_and_weaknesses: Strength:
1. Incorporating symbolic component into end2end learning systems is always an interesting idea
2. The proposed approach achieved state-of-the-art results

Weakness:
1. The writing is not clear throughout.  Most importantly, the paper directly jumps in talking about the proposed method without first clearly introducing the problem setup.
2. Again, presentation is not clear.  The main figure is confusing.  I do not understand how the prompting part is employed exactly in the system.  In particular, it is not clear how the prompting results are combined with the parsed results to feed in the program interpreter.
3. The results are a bit over sold.  Even though the fine-tuning methods require more task annotations, but GPT3 codex is pretrained on many more data.  The improvement is not that surprising.
clarity,_quality,_novelty_and_reproducibility: Clarity needs to be improved. Quality, Novelty And Reproducibility seem to be good.
summary_of_the_review: The proposed idea is exciting, the results are fairly strong, but the writing needs to be improved.
[Edited: The author has improved their writing to make the paper much clearer.  I have thus raised my score from 6 -> 8.]
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a training-free two-stage method based on Codex model for QA tasks. The first stage uses few-shot learning to prompt Codex and converts a natural language question to a SQL or Python program, where difficult-to-resolve parts are represented as API calls. In the second stage, Codex is prompted again to generate answers to each said API call, and then the whole program is executed to obtain the answer. Experimental results show new state-of-the-art on WikiTQ and TabFact datasets, with particular performance gains on questions that are unsolvable by natural-language-to-SQL translation.
strength_and_weaknesses: This paper is a creative usage of a code generation model. It also represents an interesting new approach to neural symbolic: code generation models can learn to divide a task into a high-level symbolic logical program and some low-level components for neural inference.

The experimental results are promising. With a pre-trained Codex model, without fine-tuning and with a small number of annotated examples, the proposed method is able to achieve state-of-the-art on two QA datasets. It achieves the desired behavior of a neural symbolic method, including better interpretability and answering some difficult questions.
clarity,_quality,_novelty_and_reproducibility: Clarity is good.

Quality is good.

Novelty is good.

Reproducibility is good as code is attached.
summary_of_the_review: The main idea is good and well executed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper propose BINDER, a training-free neural-symbolic framework that (1) maps the task input to a program using a pre-trained language model, where the model has to decide which part in the input can be converted to a target programming language and corresponding tasks API calls for further extension; (2) adopts an LM as both the program parser and the underlying model called by the API during the execution, where a language model serves to realize the API calls given the prompt and the return values feed back into the original programming language; (3) derive the final answer via a deterministic program interpreter executing the program without API calls. BINDER achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, where previous work is all fine-tuned on a large-scale training set.
strength_and_weaknesses: Strength:
- The system does not require extra training but can achieve better performance compared to fine-tuning settings.
- The in-depth analysis validates that BINDER can have significantly better performance on program-unsolvable questions.

Weakness:
No obvious weakness was observed.
clarity,_quality,_novelty_and_reproducibility: The BINDER system can benefit lots of semantic parsing tasks since it uses uniformed underlying API, i.e., SQL and Python, and requires on further training process.
summary_of_the_review: - This paper is clear and well-organized. Lots of complementary materials are also provided in the Appendix to help us understand the technical details better.
- The methods can be applied to a broad range of semantic parsing tasks since it use uniformed underlying APIs, i.e., SQL and Python, and require no further training process.
- The experimental results are impressive. It outperforms previous work which typically use finetuning settings based on large training set.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an interesting approach of solving question answering problems by generating a program first and then executing it to get answers.  The programs are generated by few-shot prompting a large language model (LLM).  The difficult to resolve part of the programs are delegated again to an LLM through the special LLM API calls, that are again resolved with few-shot prompting.  The inclusion of API calls used in the programs to query LLMs is a particularly interesting idea to combine programmatic solutions with neural models.  Results on a couple datasets show that this approach reaches state-of-the-art performance without additional training.

All reviewers liked the paper and recommended acceptance unanimously.  The clarity of the paper can still be improved, but overall this paper is well executed.
justification_for_why_not_higher_score: The main setting of the paper, question answering with a database, is still relatively narrow.  It would be more impactful if the targeted domain is more general.

The clarity of the paper can be improved, at the moment the paper is not the easiest to understand.
justification_for_why_not_lower_score: The paper is well executed, with interesting ideas and good experiment results and no major problems.  These factors distinguish this paper from other normal accepted papers.
Decision: Accept: notable-top-25%