Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose an extended and unifying learning architecture – OmniNet- based on transformer, which tackles tasks with various modalities such as images, text and videos. This is attained with a spatio-temporal cache mechanism, that can both capture and store temporal information and spatial information. In addition, this proposed framework supports asynchronous multi-task learning with pre-trained neural networks on different modalities. OmiNet’s generalization capability is illustrated and demonstrated with experiments. The proposed model has multiple peripheral networks each majoring on one unique modality of data. These peripheral networks project input data into the same shared format/space that can be uniformly processed in a central neural processor working like a CPU. The central neural processor uses self-attention and RNN for temporal encoding and spatial encoding. The output of the encoding components are stored in temporal and spatial caches respectively. The two caches are then used as input to the spatial temporal decoder for different tasks. 


Overall, this paper is well-written, and technically sounds, with comprehensive experimental results. However, I still have two concerns below that prevent me from giving a direct acceptance.


1.	However, considering the proposed model attempts to solve the multi-task learning problem, there seems no multi-task learning methods compared as baselines, making it hard to justify the performance.

2.	Furthermore, in Table 1, the Omin-Net results are not as good as SOTA, without clear explanation. The parameter settings for SOTA are also missing. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a unified architecture in the context of multi-task learning where they demonstrate that training four tasks (with a variety of modalities like image, text, and videos) together results in about three times compressed model, while maintaining the performance similar to their respective individually trained models. The major components of this archtiecutre are (1) peripheral networks: used to encode the domain specific input into feature representations. (2) central neural processor: a fully attention based encoder-decoder model similar to the Transformer networks which encodes the spatio-temporal information. Further, this paper suggests that their unified architecture enables to perform decent on unseen tasks during its training. In this paper they test such scenarios on video captioning and video question answering. 

Overall, the paper is clear to read and thorough in its experiments, with the caveat that its missing many multi-task paper references and the ideas are not much novel. However, I would say the setup is well engineered. 

Arguments:

1) There are many works in multi-task learning after Luong et al., 2016, please refer them in the related work section (this section is very short!) and discuss on the differences of your model w.r.t. previous work (this is completely missing in the paper). I am pointing to some references below. 

2) Statistical significance tests are missing to show that MULT-3 or MULT-4 are able to “maintain” the performance w.r.t. IND. 


[1] Latent Multi-task Architecture Learning, Ruder et al., 2019
[2] A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks, Hashimoto et al., 2017
[3] Multi-Task Video Captioning with Video and Entailment Generation, Pasunuru & Bansal, 2017

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper describes the "OmniNet" architecture, which is essentially a transformer to convert any 2-dimensional (time and spatial) input into a sequence of output tokens. The idea is that a single model (the "Central Neural Processor") would learn to perform multiple tasks on multiple inputs at the same time. This seems like a reasonable approach, and would allow a single model to be able to process text, image, as well as video or potentially speech input just the same. Dedicated modality-specific "input peripherals" would "normalize" the data appropriately, and a "start token" seems to provide the model with information on what type of input data is to follow.

I am a bit confused by the problem formulation and implementation:
- Multi-task learning works best if there are clear dependencies and shared characteristics between the task - I am not sure if the tasks used here (POS tagging, image captioning, VQA, video activity recognition) share sufficient characteristics 
- Successful multi-task learning often leads to shared representations developing across the different tasks. Is there any indication of this happening in the proposed work? Would it help to make the modality specific encoders (BPE and ResNet) trainable, so that the transformer model has an easier time learning multi- and cross-modal representations? Or is this entirely the CNP's role?
- It is probably possible to use HogWild as a way to train a multi-modal and multi-task system in the described manner, but will this lead to shared representations developing? Is there any way to include regularization or other tricks to encourage the formation of shared cross-modal representations?

Here are a few ideas:
- Would it be possible to show how well the model works when trained and optimized (including hyper parameter optimization) on a single task only? it would then be possible to show that the multi-task model can outperform single-modality single-task models 
- What about tasks like speech input? There is other, published work on multi-task training for CTC models w.g. by Ramon Sanabria ("Hierarchical Multi Task Learning With CTC", SLT 2018), in addition to the reference provided
- Would it be possible to show how the model learns to develop cross-modal representations during training? this would be a convincing visualization and rationalization of the proposed approach

review_title: Official Blind Review #1

Meta-review: This paper presents OmniNet, an architecture based on the popular transformer for learning on data from multiple modalities and predicting on multiple tasks.  The reviewers found the paper well written, technically sound and empirically thorough.  However, overall the scores fell below the bar for acceptance and none of the reviewers felt strongly enough to 'champion' the paper for acceptance.  The primary concern cited by the reviewers was a lack of strong baselines, i.e. comparison to other methods for multi-task learning.  Unfortunately, as such the recommendation is to reject.  However, adding a thorough comparison to existing literature empirically and in the related work would make this a much stronger submission to a future conference.
Decision: Reject