Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper targets on the application problem of visual question answering, whose recent state of the art was set by attention (typically transformer) - based deep neural networks. This work propose to use program supervision to introduce extra loss terms in the training of these transformer based models, arguing that the guidance from these signals improve the model's robustness against noise and fluctuations in the visual input during transfer learning. Theoretical analysis and experimental evaluations are conducted on program supervision, leading to plausible findings and results.
main_review: This paper discusses a fundamental problem of how to assess and improve the "reasoning" capabilities of neural networks. In general, this problem is challenging to tackle as reasoning and pattern recognition are strongly entangled in most end-to-end networks with multi-modal inputs. On this note, using program supervision to guide representation learning and then perform transfer learning on more noisy inputs seems a plausible direction.

My questions challenges to this paper are listed as follows:

1. It seems to me that the CLEVR dataset is specifically designed for addressing a very similar if not the same issue of neural network being dominated by perceptual bias in VQA. Therefore it is the most natural dataset to study under the context of this paper. Is there a good reason why this dataset is not used?

2. This paper claims that program supervision improves the robustness against VISUAL inputs and surround all discussions and experiments around this claim. While the results seems plausible, it remains skeptical to me why this is the case. In most cases, programs are mainly depend on the question, not the visual context (in CLEVR and GQA, the program is one-to-one mapped to the question). Hence the program supervision will mostly lead to a better question parser rather than a more robust visual branch.

3. Following 1 and 2, I believe it would be more natural to study transfer learning against noise and biases in the language space rather than the visual space.

My overall score to this paper is "Marginally below threshold". No further experiments are requested in rebuttal.
limitations_and_societal_impact: Briefly discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new loss terms to encourage Transformer-based models perform "explicit reasoning", which is beneficial for transferring knowledge from oracle-trained VQA models to deployable ones. Specifically, the novel loss is similar with the one commonly used in modular network (e.g. converting questions to executable program/functions), but instead of relying on the predicted programs to assemble executable programs during inference, in this paper this loss is only used during training as an auxiliary regularization on top of regular Transformer-based VQA models. The authors provide theoretical proof about the benefits of using the additional loss, and empirically evaluated the advantage on GQA dataset (including out-of-distribution evaluation on GQA-OOD). The proposed system demonstrates not the best but still competitive performance especially considering the model size and training efficiency (without using extra data).

========================================================================================

Thanks for the response from the authors. I've updated my rating after reading the rebuttal as well as the comments from other reviewers.
main_review: 1. Originality
- This work is an extension following a very recent work (paper [21] in the reference) and the goal is to improve the knowledge transfer effectiveness from oracle-trained VQA systems to deployable (e.g. visual signals coming from upstream visual recognition systems), rather than directly training VQA models. This is a promising direction for building future VQA models.
- The proposed loss term is similar to existing work from modular networks (e.g. paper [6]), but the usage is different: in previous work the predicted executable program is used for producing execution graph especially during inference, but in this work the entire module is only used as regularization during training. 
- The authors provide theoretically analysis about their novel regularization term on a simplified framework (MLP).

2. Quality
- I don't see any obvious technical flaws in this paper, and the experiments are quite extensive with proper ablations. 
- One typo in L154: I assume the loss should be $L_{dep}$ rather than $L_{varg}$ (which is already defined in L149).

3. Clarity
- This paper is well-written and easy to follow. The main idea is straightforward, and the experimental sections are clear for readers to reproduce.
- I only quickly checked the math but not carefully verifying all the details in the proof (I'm not an expert in theory).

4. Significance
- This paper explores a novel approach (following a recent work) of building VQA systems, which aims at leveraging oracle-trained VQA systems (with perfect visual signals) and transferring the knowledge to accommodate noisy visual inputs. The high-level idea is different from the tradition (training from scratch with noisy visual signals) and the promising results may attract more future research efforts.
limitations_and_societal_impact: - I didn't find obvious mistakes from the theoretical analysis (I'm not an expert though), but it seems from the experimental validations there is no observation to directly or indirectly (with a measurable proxy) support the claim? More specifically, is there any metrics can be used to provide more straightforward evidence about the statement "... such program prediction can lead to decreased sample complexity ..." (from the abstract).

- One probably missing ablation about the proposed loss term is to directly train a VQA system with the loss from scratch but without oracle transfer. I feel this can be a good evidence to verify whether the proposed regularization is only helpful for knowledge transfer or generally can push the Transformer models to better handle the reasoning.

- Some minor issues: (1) why in Table 1 the accuracy numbers of binary, open and test-std for baseline models are missing? (2) L294 mentions an accuracy of 58.8 on testdev, when program supervision+lxmert is used but not oracle transfer. The oracle transfer+lxmert seems to be 58.4 according to the table, which is against the claim of L295 ("this is lower than oracle transfer's accuracy...")?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper describes a novel approach for doing more robust VQA that doesn’t just learn to exploit the biases in the data. They build on previous work which demonstrated that training a model on oracle visual data makes it less susceptible to learning biases in the data. While the previous work showed that the better reasoning skills learnt by the models trained on oracle visuals decline when transferred to the noisier visuals extracted by pre-trained vision models, here the authors demonstrate that adding the additional supervision step of making the model predict the reasoning program alongside the VQA answer makes it more robust to such distributional shifts.

main_review: The paper is exceptionally well written. It provides both theoretical and empirical results, and discusses ethical implications and risks well. 

The only problem is that it doesn’t achieve SOTA VQA results. Yes, the SOTA model (OSCAR [39]) requires 50 times more images, but the proposed model requires 2x more sentences and the additional program supervision, which is hard to obtain as the authors acknowledge. Saying this, it does better than the comparable baseline with program supervision trained on the same amount of data.

The authors show theoretical results suggesting that their additional program supervision results in better sample complexity by learning re-usable reasoning modules implicitly. This is probably the most interesting result of the paper, and I would have liked to see it explored more experimentally. For example, I am not sure what experimental results support the sample complexity claim. Also, could the authors maybe use the program module to analyse the failure cases? Does it bring extra interpretability to the model? 

The program prediction module is entirely independent from the VQA module. Why do the authors believe that it helps regularise VQA learning? Have they considered making VQA answer module dependent on the reasoning performed in the program module?

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the problem of training VQA models on noisy/imperfect visual input (e.g., representations from existing object detectors). It proposes using program supervision as an additional task to assist in transferring reasoning patterns learned through clean/oracle visual inputs to the imperfect inputs. The experiments are performed on the GQA/GQA-OOD benchmarks, and the empirical results show gains on both frequent (head) and infrequent (tail) classes, showing improvement in reasoning skills. Furthermore, the work theoretically shows that predicting programs can reduce sample complexity.
main_review: Overall, the paper is well written for most parts. The program supervision loss is simple and sensible, but of course requires ground truth programs to be available, which may not always be realistic. The theoretical result relating program supervision to reduction in sample complexity is an important contribution, but the empirical setup could be much better.
I will now present my concerns:

[C1] The phrase: ‘transfer of reasoning patterns’ seems a bit odd for the setup considered in this paper. With such a title/description, I was expecting more focus on the reasoning patterns themselves, instead of visual quality. For instance, the title would be more appropriate for the following scenarios: applying previously learned reasoning patterns to images from new domains or questions phrased differently or to novel concepts/compositions of reasoning steps, but these are not what is being studied here.

[C2] The theoretical result about program supervision reducing sample complexity in reasoning is an important contribution. However, the empirical setup leaves some open questions. Firstly, it would be nice to validate it empirically by fine-tuning on fewer samples. Secondly, does this apply to pre-training stages too? Would it apply to both visual oracle pre-training and BERT-style pre-training? It would be a plus to have those answers or at least have those discussions somewhere.

[C3] Obviously, there are some concerns about the practicality of the proposed method. Firstly, having access to oracle visual annotations is not realistic, yet most of the gains seem to come from them. Second, having access to the ground truth programs is also not realistic. Both of these concerns could be addressed if the sample complexity could be validated empirically i.e., would having program supervision help even if not all instances had oracle visual inputs? Furthermore, how much benefit would it show if only a subset of the instances had program supervision? The results on head/tail do not quite cut it since they were based answer type/answer frequencies if I recall correctly, and may not provide clear analyses based on training phase/type of annotation. I understand that these may require extra experiments, but even some discussions regarding these would help.

[C4] It is important to clarify early on that the program supervision refers to predicting the reasoning operations/arguments, but it is not necessary that the model actually follows that reasoning pattern (or does it?). It would be a big plus if the paper ran MMN [C2], which actually performs those reasoning operations with oracle pre-training too. For instance, if program supervision alone is sufficient and we do not actually need a separate modular architecture to run those reasoning operations, then that would be a very important finding for the community. But this is missing from the paper.

[C5] Some of the details were not completely clear:
I was confused about what ‘reasoning patterns’ meant in the context of this paper, and had to refer to [1] for clarification.
How much does the model trained after oracle pre-training alone achieve?

[C6] Missing citation: HAN [3] could be cited since it is also graph-based and outperforms the other graph-based method mentioned in the paper: NSM [4].

References:

[1] Kervadec, Corentin, et al. "How Transferable are Reasoning Patterns in VQA?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[2] Chen, Wenhu, et al. "Meta module network for compositional visual reasoning." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.

[3] Kim, Eun-Sol, et al. "Hypergraph attention networks for multimodal learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[4] Hudson, Drew A., and Christopher D. Manning. "Learning by abstraction: The neural state machine." arXiv preprint arXiv:1907.03950 (2019).

[5] Selvaraju, Ramprasaath R., et al. "Taking a hint: Leveraging explanations to make vision and language models more grounded." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
limitations_and_societal_impact: The paper acknowledges the limitations of the theoretical work i.e., analysis on MLPs instead of the models actually used in the paper. However, I think there are at least a few other limitations: a) assuming access to oracle visual inputs/programs and, b) lack of empirical studies on sample complexity for pre-training/fine-tuning tasks.
review_title: null

Meta-review: The authors addressed most of the reviewers concerns and all reviewers who engaged in post-rebuttal discussion recommend to accept the paper, one reviewer increasing their score.
The paper contributes an interesting approach to transfer reasoning patterns with program supervision in the context of visual question answering and a solid evaluation.

I recommend accepting the paper under the expectation that the authors will address the concerns as promised in the author response, including:
1) The empirical evaluation of the theoretical claim on sample complexity
2) improve clarity, typos
3) discuss related work Chen et al.

If possible, please also include a better failure analysis as suggested by reviewer uBf9.
Decision: Accept (Poster)