Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1. It is said a end-to-end model for IQA, but in the training IQA, it is not just given question and answer. For each step, it is supervised with the feedback questions, which is similar to the Memory Network paper, not the End-to-end one.

2. In the attention model, at equ 6 and 9, it used the question representation U to calculate both word-level and sentence -level attention, why U could represent information in both word-level and sentence level, or word-leve and sentence-level information are in the same space?
review_title: Few questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1. the QA model is not novel, very similar to the existing model.
2. The IQA model is very confusing. If it needs human interactive in the training process, how could it be practical to ask human to join the training in each iteration? It sounds impractical. If the human interactive questions are predefined, then it is not interactive at all, since it is not based on the current state of model output.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an "interactive" version of the bAbI dataset by adding supporting questions/answers to the dataset in cases where there is not enough information to answer the question. Interactive QA is certainly an interesting problem and is well-motivated by the paper. However, I don't feel like the bAbI extension is adequately explained. For example, the baseline DMN and MemN2N models on the IQA task are "take both statements and question as input and then
estimate an answer." Their task is then fundamentally more difficult from the CAN's because they do not distinguish "feedback" from the original context; perhaps a more fair approach would be to treat **every** question (both supporting and original questions) as individual instances. Also, how were the supporting questions and the user feedback generated? How many templates / words were used to create them? The dataset creation details are missing, and if space is an issue, a lot of basic exposition on things like GRU / sentence encodings can be cut (or at least greatly shortened) and replaced with pointers to the original papers. 

Another issue I had is that the model attempts to generate these synthetic questions; if there are just one or two templates, why not just predict the values that fill these templates? So instead of generating "Which bedroom, master one or guest one?" with an RNN decoder, just predict "which" or "which bedroom"... isn't this sufficient? In the end, these just seem like more supporting facts, not actual interaction with users, and the fact that it is run on only three of the original twenty tasks make the conclusions hard to trust.

In conclusion, I think the paper has a strong idea and motivation, but the experiments are not convincing for the paper to be accepted at ICLR.
review_title: review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work describes 

1: a two stage encoding of stories in bAbI like setups, where a GRU is used to encode a sentence, word by word, conditioned on a sentence level GRU, and the sentence level GRU keeps track of a sentence level encoding.  Each is used

2: modifying the bAbI tasks so it is necessary to ask a question to correctly solve the problem

I am not convinced by the papers results:

1:   The new architecture does not do significantly better than DMN+, and in my view, is similar to DMN+.   What problem with DMN+ does your architecture solve?   

2:  There are now several papers doing the second thing, for example "Dialog-based Language Learning" by Weston and  "Learning End-to-End Goal-Oriented Dialog" by Bordes and Weston, and I think doing it more carefully and in more compelling ways.   In the current work, the correct answer to the question seems given independent of the what the agent asks, so any model that can output "unknown" and then input the extra response has an advantage.  Essentially all of the architectures that are used to solve bAbI can be modified to do this...  Indeed, the enc-dec* accuracies in appendix A show that this sort of module can be appended to any other model.  All of the standard models can be trained to output questions as a sequence of words.    Furthermore, I suspect you could generate the questions  in the authors' setting just by enumerating all the questions that occur in training, and taking a softmax over them, instead of generating word-by-word.

review_title: null

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. Unfortunately, all reviewers are leaning against accepting the paper. Authors are encouraged to incorporate reviewer feedback in future iterations of this work.
Decision: Reject