Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Interesting results and analysis but lack of novelty and details. 

This paper proposed a novel text explanation method which extracts keywords that are essential to infer an answer from question. The authors proposed VCBS based on CBS and use Natural language inference to identify the entailments when the answer is yes or no. Experiment results show better mean opinion score 100 random samples results compared with the previous method and better VQA binary classifications when flipping based on the NLI results. 

Different from [Park et. al. 2018], who collected the explanations, this paper directly use the coco captions dataset as the source for the explanations. One of my major concern about this paper is it actually generate relevant captions with respect to the question and answer instead of real explanations. It's true that correlated caption can sometimes serve as the explanation when they cover the same concept coincidently but, we can not use these captions to explain the reasoning process of VQA. 

The technique novelty of the proposed paper is also limited, the major novelty is the VCBS, which seems very similar to CBS. The only difference is VCBS adds relaxed parameters, which seems no technique novelty. Most annotations in Algorithms 1 is also not explained, making the readers hard to follow the actual content. The proposed model, although tied the vqa words with the explanation words, it suffers the same problems as PJ-X model, which didn't consider the VQA attention at all. 

The experiment is also weak, considering the results is conduct on 100 samples, there might be significant variance. It's interesting the compared approach is learned based on a different dataset, which makes the results harder to compare. The NLI model results are interesting but for a more fair comparison, I would expect the proposed method compare with a model trained with VQA and coco caption dataset, such as VQA-E. 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
 I thank the authors for their response.  I would keep my score unchanged (i.e., 6 Weak Accept). 

-----------------------------------------------

Strengths: 
- The paper enhances the beam search approach to generate explanations for answers to visual questions. The explanations are further used for verifying the yes/no answers.
- The paper constructs a VCBS algorithm with novelties in allowing soft constrains of the generated beams. Since I have not worked on the constrained beam search before, thus it is hard for me to measure the novelty of this method.
- The results of VQA 2.0 is pretty good. The accuracy of the Yes/No questions almost achieves the SotA systems. 


Weakness:
- I have a question about the NLI system. Since the NLI is a three-way classifier where the answers would be "Entail", "Contradictory", and "Neutral".  What would the system do when the relationship is "Neutral"? For now, I think that it just give an answer of "no" but I am not sure whether it is correct.  For example, in Fig. 3, it shows an explanation (premise) of "something (not the vegetable) on a plate" and the hypo is "there are vegetables on the plate". Since the hypo is not necessary to contradict the premise, the relationship should be neutral. It does not directly provide evidence of the answer.

Comments:
- Since more data are involved in training the explanation system, the proposed methods are not fairly compared with the BAN method. It would be better to mention this detail in the paper. 


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
The paper proposes a novel method for explaining VQA systems. Different from most previous work, the proposed approach generates textual explanations based on three steps. First, it extracts keywords from the question. Then an explanation sentence is decoded (based on an RNN image captioner) through the proposed Variable-Constrained Beam Search (VCBS) algorithm to satisfy the keyword constraints. Finally, 3) checking through linguistic inference whether the explanation sentence can be used as a premise to infer the question and answer.

I would recommend for acceptance. The paper proposes an alternative approach to VQA explanations, together with a few supporting algorithms such as VCBS. It is potentially helpful to future work on textual explanations and explainable AI in general.

At a high level, it is ambiguous to decide what is a reasonable explanation for many “no” answers. For example, one usually cannot provide stronger justification than “there is indeed no one” or “I don’t see anyone” to the question “Is there anyone in the room” with an answer “no.” The paper frames this explanation generation task as a linguistic inference task and checks entailment between the explanation and the question-answer pair. While it is debatable whether this is optimal, the proposed approach provides valuable insights on what constitutes a good explanation.

However, the proposed approach also has noticeable weaknesses. 

It relies on external models or tools for natural language inference, and such inference does not take into account the visual context of the image. Also, the explanations generated from the proposed model only justify the answer but are not introspective, and they do not reflect the decision process of the target VQA model.
review_title: Official Blind Review #2

Meta-review: This paper is good, with relatively positive support from the reviewers. However, there were also several legitimate issues raised, for example regarding the semantics of a negative answer and associated explanations. Though this paper cannot be accepted at this time, we hope the feedback here can help improve a future version, as all reviewers agree this is a valuable line of work.
Decision: Reject