OmniNet: A unified architecture for multi-modal multi-task learning
Authors: Subhojeet Pramanik
Priyanka Agrawal
Aman Hussain
Venue:  (2020)
Keywords: multimodal
multi-task
transformer
spatio-temporal
attention-networks
neural-network

Abstract: Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet. 

Under review as a conference paper at ICLR 2020
OMNINET: A UNIFIED ARCHITECTURE FOR MULTI- MODAL MULTI-TASK LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer is a popularly used neural network architecture, especially for lan- guage understanding. We introduce an extended and unified architecture that can be used for tasks involving variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimen- sion of the input in addition to the hidden states corresponding to the temporal in- put sequence. The proposed architecture further enables a single model to support tasks with multiple imput modalities well asynchronous multi-task learning, thus we refer to it as OmniNer. For example, single instance of OmniNer can con- currently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.
INTRODUCTION
Transformer Vaswani et al. 2017) is currently one of the best performing models for any sequence transduction tasks, especially those involving natural language. originally designed for single task at time. In fact, most of the generic deep learning architectures (Wu et al. 2016 Szegedy jet al. 2016 Battenberg et al. 2017) that have been designed and developed are able to learn, albeit very well, single task and handle one task specific input domain like image, text or audio. Fur- thermore with these models, we often rely on the generalization capability of the trained network to guarantee performance another popular paradigm used to adapt the model to learn a related task with similar input domain. The success of neural networks across these challenges is known to be due to their ability in learning effective representations of the data. For example, the self-attention mechanism in Transformers can capture the global temporal dependence in sequential data very well. Naturally, the question arises whether we can extend these architectures, like the Transformer, to be able to learn shared repre- sentations from multiple input domains and to be able attend on these representations perform multitude of tasks concurrently.
unseen examples. Transfer learning (Shu et al. 2018 Hu et al. 2018) is
The research into multi-task models that learn solve varied tasks across multitude of input domains is not new. Work done in Ngiam et al. (201T) demonstrates an architecture capable of learning shared representation across audio and video modalities. Similarly in Colloobert & Weston 2008) convolutional architecture has been designed to support variety of NLP tasks. However, most of these architectures are designed to learn specific set of tasks with known input domains. To the best of our knowledge, there does not exist single unified architecture that works out of the box for any combination of multi-modal inputs.
To address this gap, extend Transformer towards a unified architecture, namely OmniNet, which enables a single model to support tasks with multiple input modalities and asynchronous multi-task learning. We consider that most real-life data like image, text, speech, video, etc. is direct conjunc- tion of spatial and temporal components. Therefore, we employ a apatio-tempoor cache mechanism to learn a shared representation of the input data across the spatial (space) and temporal (time) di-

---

Under review as a conference paper at ICLR 2020
mension. Using generalized encode() function, OmniNet can process and store pratio-tempord representation for each of the input domains and then decode() predictions across a multitude of tasks. In experiments, train single instance of the OmniNet to solve number of tasks spanning multiple multi-domains such as part-of-speech tagging, image captioning, visual question answering and video activity recognition. To make work reproducible, open to scrutiny and fur- ther development, we will open source a demonstration of our system implemented using Pytorch (Paskke al. 2017).
RELATED WORK
Multi-task learning has been extensively studied the literature, with applications to a wide set of problems ranging from natural language processing (NLP) (Collobert & Weston 2008 Johnson
et al. 2017] Dong et al. 2015) to speech recognition (Seltzer & Droppo 2013 Krishna et al. 2018)
vision (Zhang al.] 2014 Chen et al.) 2018 Anderson al.) 2018). 8). It has also found
in a combination of diverse tasks like image captioning and text translation and parsing (Luong 2016 Zhao al., 2018). However, most of these architectures assume the of tasks to be known in advance. Similarly, multi-modal learning has been essential for solving broad range of interesting problems such as Visual Question Answering (Kim et al., 2016 2018) and Video Question Answering (Lei et al., (0188). Again, the state-of-the-ar models are highly specific to objective hand and not easily adaptable to different tasks or domains. (Kaiser al. (017) proposed MultiModel architecture for learning multiple tasks but lacks support for multi-modal tasks
al.
with more than one input domains such as visual question answering.
3 PROPOSED MODEL
We propose a unified architecture, namely OmniNet, to enable learning multi-modal tasks with mul- tiple input domains and support generic multi mustiitasking for any set of tasks. The OmniNer architecture consists of multiple sub-networks, called peripheral networks, connected to common central neu- ral network called the Central Neural Processor (CNP) (Figure [.. Each peripheral network used to encode the domain specific input into feature representations. In this work, we describe im- age, text and video peripherals (Section 3.T). One can add more, say speech peripheral, depending on the task. The output representation of peripheral network is always spatio-temporal tensor I E igtxsxdmodel, where & are the temporal and spatial dimensions of the input respectively, and dmadel is the model dimension input the CNP.
airpiane flying through cloudy sky
Output
Input Periphoral
Contral Nicural
Image Video
Other peripherats
9,
Temporal
Spatio Temporal
Output
yes
Spatial Cache
Peripheral
Text
Output
Is cloudy?
VBZ PRP RB
Figure 1: OmniNet performing image captioning, visual question answering and POS tagging at
The spatio-temporal representations generated by the peripheral networks corresponding to each in- put domain are then processed by the CNP. The CNP uses fully attention based encoder-decoder (Bahdanau al. 2015 Sutskever et al. 2014 Cho et al. 2014) model for sequence transduction similar to the Transformer architecture Vaswani et al. 2017), which is the ofate-of-the-art for mul- tiple language modeling tasks (Section 3.2). During the encoding stage, the CNP implements
2

---

Under review as a conference paper at ICLR 2020
generic encode(2, D) function first process and store the apatio-temporia representations of the imput, where x E 1Rtx*xdmmods is the spatio-temporal tensor produced by the peripheral networks and D E Z= 0 D < Dien is the domain id and Dien is the max number of domains supported by the CNP. The encode() function is called multiple times, once for each multi-modal input from respective peripheral. During the decoding stage, decode(yshi T) function is used to decode predictions as softmax probabilities, where Yshifted ZN-1 are the target outputs shifted one time- step to the right, N is the length of the output sequence; T E ž 0 < T < Tiem is task id and Ten is the total number of supported tasks. The decoding step is similar to Vaswani et al. (2017), modified to incorporate a two-step attention mechanism over spatial and temporal cache.
3.1 PERIPHERAL NETWORKS
First, we elaborate on how we support multiple input domains using peripheral networks. A periph- eral network can a pre-trained model from existing literature to ultimately encode given domain input to a standardized feature representation x E IR xRtx*xdmoote where t & s are the temporal and spatial dimensions of the input respectively, and dmmade is the model dimension input to the Central Neural Processor. Here we detail text and vision peripherals and one can add more peripherals or alter the peripheral design depending on the task.
Vision peripheral: This peripheral uses a convolutional neural network to encode image and video inputs in the tasks. For an image of dimension h w Ber this peripheral down-samples in to h x w' x n/g. where h, w, ne are the height, width and number of input channels respectively. For a video, each frame is input to the peripheral to produce F x x w' x xn',, where F is the total number of frames in the video. The encoding vectors are then projected to dimension dmmoee using fully connected layer. The output is then reshaped into a spatio-temporal tensor of x E IRxx w/ xdmodal where 1 = for an image and f = for a video. In our experiments, we use the pre-trained R5sNet-152 model, variant of ResNet (He et al. 2016) consisting of 152 convolutional layers. We remove the final fully connected and avg- avg-pooling layers to generate spatial feature representations for a given image/video.
Language peripheral: The Language peripheral uses byte-pair encoding (Sennrich et al. 2016 to generate subwords for given input sentence. The subwords are passed to embedding layer to generate subword embeddings of dimension demb and projected to dimension dymodel using a fully connected layer. The output is then reshaped into opatioltemporal tensor I equal to number of subwords in the input sentence. As we do not have any spatial dimension in textual data, the spatial dimension of from Language peripheral is always 1. In our experiments. We used praitedined subword embeddings with demo = 300 and vocab_size = 25000 from Heinz- erling & Strube (2018), which includes pre-trained subword embeddings of over 275 languages, to initialize the weights of the embedding matrix.
R Xdmmodel where
3.2 CENTRAL NEURAL PROCESSOR (CNP)
To process the spatio-temporal information in the input data, the CNP implements a spatial cache C.. temporal cache C, and link array L.. The spatial and temporal cache and the link array a list of elements, initialized as empty before the encoding process. During the encoding stage, an encode() routine takes as input, the tensor æ generated from the peripheral and corresponding domain/peripheral id D. This function processes the spatial and temporal information in the input x and stores them into the spatial cache C, and the temporal cache C. respectively and stores their dimensions & in the link array. For given task, this encode() routine is called K times, where K the number of inputs in the task. Note that these inputs can belong to same or different domains.
Encode (z, D): For given input E XXxssdm and domain identifier D. the encode() routine described in Algorithm T Since inputs can come from multiple peripherals, the algorithm first concatenates the input with the domain embedding to ensure a domain-aware encoding of the input (Steps 2 to 3). Steps 4 to 7 process the spatial information in x by unrolling the time dimension and adding these unrolled vectors into the spatial cache. Steps 8 to 10 process the temporal information in x by averaging the spatial dimension of x and then passing the averaged tensor to a self attentio based TenporalEncoder. This TemporalEncoder is similar to the encoder used in Vaswani et al. (2017) as shown in Figure is used to calculate temporal embeddings of the input sequence. The output from the TenporalEnnoddrr is appended to the temporal cache.
3

---

Under review as a conference paper at ICLR 2020
Culpul Pred ctoo
& Norm
Muiti Head Aftendion
Appended
Spotial Cacile
Tampara Scores
Tamporal
Add & Norn
Wuit-Head
Norm
Temporal Cache
Peed Forward
Mulll-taad Adtention
Haa--Mad
Mayaee
Ouggut Emeeabing
Outputs right))
(Spadislly iveeraged Impar)
Figure 2: left: TemporalEncoder architecture; right: OmniNet decode() architecture.
Algorithm encode((): Encodes spatial and temporal represtntio into spatial and temporal cache
Require: I E [Rtxsxdmaded D, C., L, C.
1: L (t
2: Demb +- Layerr)))
3: < FC((Concut x, Demb)) dimodet)
4: if > then
5: S Reshope(1, (ts, dmodde)) {where, output $ [S,,..., St.] s.t. S Rdmoddl is
spatial feature vector.)
6:
C. C. Sea] {Append spatial representations to spatial cache)
7: end if
8: T 1-Ei=11)) i)),
9: T h- Temporal Encoder(T) (where, output T [7,.... T:] s.t. T3 E Rodmdde is the encod-
ing of temporal dimension in J.}
10: C. + C. CT.......... ,T] (Append temporal representations to temporal cache)
The above encoding routine keeps appending spatio-temporal information to C₁ & C2 for each input Is R x&a Xdmadel Note the superscript k denote correspondence to k-tth input of the task, where 1e 11....,A K. After K calls, we have the temporal cache C. = [Ti,
Tnl, where = ΣX lr; the spatial cache C, = .S....., Sr), where P {Σp Lp 8p p E 1,..., K A 8p > 1} and the link array L [(t: + s1), ..... (tk - sx)]. Note that C, can also be empty in case the encode() is only called with inputs with S4 = 14k. Next, we use the decode() routine to generate predictions softmax probabilities.
Decode (Ushifteds 7): The architecture of the decode() function is shown in Figure 2 The decode() takes argument the output labels ysei shifted one time step the right, task id and gen-
4

---

Under review as a conference paper at ICLR 2020
erates predictions by attending from the spatial and temporal cache. The decode() function is struc- tured similar to the decoder used the Transformer architecture Vaswani et al. (2017) and jointly attends on the vectors stored in the temporal and spatial cache. Similar to Vaswani al. (2017), the decoding first starts by attending over the output embeddings using masked multi-head scaled dot product attention. The attention layer for the temporal cache uses scaled dot-product attention with multiple heads as specified in Vaswani et al. (017)). Attention layer for the spatial cache, uses gated multi-head attention attend over the elements of the spatial cache. For inputs with both time and space dimension (e.g. video), we want the spatial attention layer to attend more on frames which have relatively high attention scores in the temporal cache attention layer. Therefore, the attention score output from the temporal cache multi-head attention layer A E R X N R is used to calculate tensor E R xNxP used for gating the attention score output the spatial attention layer, where y, is the number of heads in multi-head attention as described in Vaswani et al. (2017). The tensor calculated using L as detailed in Algorithm2 2 Giver Q: the matrix queries, K; keys of dimension dæ and V: values of dimension de, the scaled dot-product attention for the spatial layer is modified as:
(Va. O oo0D
Attention K, V. Softmax
In order to use the same CNP for multiple tasks with varying output vocabularies, we use multiple output embedding layers OutputtEbedLagyer.. o to generate the output embeddings for each task. At the final layer, (FC + Softmar),,... (FC So Softmar)) Tu. classification layers for each task. We also cal-
use multiple
culate task embedding vector using and always start decoding using the task embedding vector.
Al Algorithm 2 Calculate G using output scores from temporal attention and link array
Require: A.A A
1: ida <-
2: for each in L do
3:
G G+D []
4: if then
5: A' + Al:,;, idx: idr +
6:
A' + Expand(A', (n4, tt, s)) {where Expand dimension) expands tensor
according to a given dimension)
7:
A' + Reshape A', NN ts))
8:
G GUA' (Append the respective temporal attention scores to G}
9:
end if
10: idæ <- idx +
11:
end for
12: G -Stack(G)) {Stack the list of tensors to construct tensor G of dimension (na, N, P)}
3.3 MULTI-TASK LEARNING
In order train single model simultaneously mutiple tasks used the Hog training approach as described in Recht et al. (011]). Similar to the approach described in Mnih et al. (2016), the main process holds global copy of the model. We create separate worker processes for each task, where each process maintains a local copy of model. At each training iteration, each process starts by synchronizing
local model with the global copy. This is done through forward and backward propagation on its local copy and then copying the locally computed gradients to the global model asynchronously. Each process then calls the global model optimizer asynchronously to update the weights of the global model. Instead of storing the model in CPU as in Mnih et al. 2016) we always store the local copies across multiple GPUs.
TASKS AND SETUP
To evaluate the effectiveness of our proposed framework for tasks spanning diverse modalities, we choose a set covering all possible spatio-temporal data archetypes: Image Captioning, Part-of-
For brevity, reuse the notations of Vaswani et al. 2017 in this description
5

---

Under review as a conference paper at ICLR 2020
Speech (POS) tagging. Visual Question Answering (VQA) and Video-activity Recognition. Each of these tasks explores a unique potential apatio-tempore configuration of the input containing differ- ent values of f and 8, where t and are the temporal and spatial dimensions of the input respectively. This enables us to perform a comprehensive study of the multi-modal and multi-input capabiliitess of the system. We further elaborate on the properties of these tasks below.
For training, we always use cross-entropy loss with Adam optimizer (Kingma & Bal 2015) and schedule the learning rate using Noam scheduler (Shazeer & Stern 2018) similar (Vaswani al.) 2 2 In the vision peripheral, we freeze the layers of pretrained ResNet model. Remaining parts of the architecture (peripherals and CNP) are kept trainable. In this section, we provide details on the datasets used and the model setup for each of these tasks.
Part-of-speech (POSyTagging: To illustrate the task with only temporal modality (t > & 1, where t and are the temporal and spatial dimensions of the input respectively), we consider POS tagging problem. Given an input sequence of words, the model should produce sequence of POS tags corresponding to each word. We use Penn Tree-bank (Marcus et al. 1994) which contains gold annotations on English WSJ articles by experts. During the encoding stage, each input sentence is processed by the language peripheral to generate a spatio-temporal tensor x: E R Rexxxdmadel, where I is the sequence length of subwords in the input. The CNP encode() function is then used to encode into the temporal cache. Note that spatial cache is empty for text inputs as decoding stage is same as that of Transformers to predict the sequence of POS tags.
1. Therefore, the
Image Captioning: This task represents the ones with inputs containing only spatial modality (t = && $ > 1). The captioning model is required to predict text caption for a given image. We use the MSCOCO 2014 dataset (Lin et al.) 2014) for training and present results on the COCO validation set. During the encoding stage, the input image is resized to 224 x 224 and processed by the vision peripheral containing pre-trained ResNett-12 to produce image embeddings x E 1 19xdmadde x is then imput to the encode() function which populates corresponding spatial and temporal cache. The decoding stage uses decode() function with output vocabulary size 25000 to generate the captions.
For the task with inputs from multiple domain, such that each contains either spatial or temporal modality (either t > 1 & = 1, or t = & s >> for each input), we choose the task of visual question answering. Given a question over an image as inputs, the model is supposed to predict the correct answer label. We use the recently introduced VQA v2.0 dataset (Goyal et al. (017)) for this purpose and perform evaluation on the VQA test-dev set. All the images are resized to dimension 224 x 224 before training. The encoding stage of this task utilizes two peripherals: the vision peripheral is used to generate tensor æ1 R' X49 for the input image. The language peripheral is used to encode the questions into 12 E IxxxIxdmoddd where t is equal to the length of the subwords in the question. The encode() function is the called two times, first with 2 and second with I2 as input. Finally, the decode() with output vocabulary size 3500 is to generate the answers as softmax probabilities in a single decoding step.
Visual Question Answering:
Video Activity Recognition: For tasks which contain both spatial and temporal modality in a single input (t > 1 & 8 > 1), we consider the action recognition task on videos. For this purpose, we use the HMDB dataset (Kuchne et al. 2011). The dataset consists of over 5000 short length clips of real life actions with 51 classes. We present our results on train-test split We use 16 frames per video and resize each of them to 224 x 224. During the encoding stage, each frame of the video is passed through the vision peripheral to cumulatively generate a video encoding æ E R T116449xémode which is then used as input to the encode() function. Finally, the decode() with output vocabulary size 51 is to predict the action as softmax probabilities in single decoding step.
5 RESULTS AND DISCUSSION
We present the evaluation on (a) Tasks of various modalities illustrated in Section Multi-tasking setup for these tasks (Table [ (c) Reuse of the multi-task model for an unseen task (Figure Bp. In addition, we also provide some ablation studies on the architecture (Table
2 Thee hyperparameter values used for nh, dmooets N Layerss de. de are same that specified in Trans- former hase model (Vaswani et al. 2017).
https:/caatloo.. Idc.upenn.edu/ D499T42; We use splits 0- 0-18 as training, 19-21 as development and 22-24 test sets

---

Under review as a conference paper at ICLR 2020
Performance of proposed architecture individual tasks: We choose set of four tasks with diverse imput modalities and combinations as described in previous Section 4 We train the OmniNer model independently across each of the above tasks. Each of the tasks demonstrates unique capa- bilities of this generic architecture. More specifically, in Table 1 we compare our results with the following state-of-the-art: POS tagging: Spoustová et al. (2009); image captioning & VQA: An- derson et al. (2018) and HMDB: (Simonyan & Zisserman (2014). It is important to note that we do not perform any parameee optimization. We believe that, with more computational power, fine tuning the hyperparameters towards these tasks should result in comparable or even improved performance to the state-of-the-are These results can indeed be used as a baseline for any future work which aims at using single architecture across various possible spatio-temporal archetypes. interesting to note that model extensible to a new domain without any modification to the CNP as long as one can add specific peripheral to convert domain inputs into spatio-temports tensors. This aspect of the architecture makes it applicable to several popular multi-modal tasks.
POS
Captioning
Visual Question Answering
HMDB
Acc.
BLEU-4
Meteor
Overall
Y/N
Num.
Other
Acc.
#PARAMS
SOTA
97.44
36.2
27.0
63.2
80.3
42.8
55.8
59.4
IND
95.61
28.9
25.2
55.31
74.09
35.17
46.35
55.29
450m
MULT-3
95.82
28.8
25.2
56.79
76.75
35.82
47.16
149.03m
MULT-4
95.44
27.4
24.5
55.76
75.49
35.64
46.08
54.44
149.07m
Performance of OmniNet on diverse set tasks. IND: Model trained individually for each of the given tasks; MULT-3: Multi-task model trained on POS, Captioning & VQA; MULT-4: Multi-task model trained across all the four tasks.
Table I:
Effeet of training diverse of tasks together: We trained two multi-task models: (1) MULT-3 PPOS+VOA-Captioning and (2) MULT-4 (POS=-QA--Captionnn-ttll
using hogwild ap- proach. While the MULT-3 model attains similar and sometimes better performance, the final MULT-4 model attains slightly reduced performance, when compared to the independent task scores. We believe this due to the skewness the size of the HMDB dataset containing only 5000 training samples. However strated below. Using a multi-task model also results in three times reduction in the total number of parameters. That is, when separate model is used for each task, we have a total of over 450 10 parameters. Whereas during multi-tasking since a single model is shared, we have a total of over 149 x 10t parameters, while achieving similar performance. Interestingly, the model is able to at- tend on spatio-temporal components of the inputs from different tasks and concurrently generate predictions across them, thus demonstrating the generalization capability of our architecture.
tradeoff, adding the HMDB task shows interesting zero-shot results demon-
Pradicted
What
Man
Action Labala:
Riding.Bli Bike
Swing, Basebell
Shoot, Bow
Riding near body
Predicted Caption:
man ridigg bike down the street
luseeball player swinging but a ball
A man is holdling tennis racket and a tennis ball
person
Figure 3: Results of zero-shot video captioning and video question-answering.
Towards zero-shot learning: reuse of pre-trained network for unseen task: Sharing represen- tations across multiple tasks provides the benefit to transfer of usefuul knowledge across multiple domains. Since, image and video are processed by the same vision peripheral, we conducted an experiment to see whether our model pre-trained on all the four tasks (MULT-4) can perform video
'Since most of these tasks are popular challenges, compare with state-of-the-ar which are generically applicable for the respective task instead of the challenge dataset.

---

Under review as a conference paper at ICLR 2020
captioning and video cresttonnansweriin without any explicit training on these tasks ie. zero-shot learning. The results of the evaluation on randomly picked instances from the HMDB test split shown in Figure E]
Interestingly, the model performs quite well related actions that were present in the COCO and VQA training set; such as captions related to horse riding and baseball; or questions related to concepts present in VQA. Without training on any video captioning & video QA instance, the model could use the trained information from image captioning, image QA (VQA) and video action recognition (HMDB) apply them on videos to generate meaningful predictions, hence demonstrating the capability of the model to transfer knowledge across related multi- tasks. However, on concepts that are not present in the trained datasets, the model either describes the environment in the video or replaces with alternate known concepts. This case study, although not comprehensive, shows the capability of the model to learn shared representations and ability to transfer knowledge across domains. We believe that adding more tasks and domains will lead to more interesting zero-shot learning results in future across a wide range of problems.
Impact of individual architectural components: In order to support different input domains, our architecture introduces spatial cache and link array components to the original Transformer architec- ture (which only consists of mechanisms to handle temporal data). We conducted an ablation study on each of these components to verify their importance across various tasks as shown in Table 2 The ablation was conducted the independent (IND) well the multi-tasking model (MULT-4). The second row ablates the link array from our architecture i.e. removing the multiplication of G in Equation[]] !1 The link array was designed assist in tasks with inputs such as video, containing both spatial as well as temporal modality in a single input. The total number of spatial components be- comes very large as number of frames in the video increases, thereby making it difficult to attend on various spatial regions throughout the video. Using link array the spatial attention layer can attend more on specific important frames in the video. Therefore, removal of link array leads a huge re- duction in performance in HMDB compared to other tasks as they do not have both spatio-temporal modalities for any single input. Removal of spatial cache, on the other hand, has significant effect on performance across all tasks containing spatial modality. Since, image captioning contains primarily spatial modality and hence the BLEU drops significantly after ablation. As other tasks utilize the temporal cache for prediction, in the multi-task setting the captioning task learns to utilize the spatial average of the image stored in the temporal cache for prediction and hence retains some performance after ablation of the spatial cache. On the other hand when trained independently on captioning, the network learns to utilize the information in the spatial cache only, and hence the score drops to zero after ablation. VQA leverages both, spatial information from image and temporal information from question, retains some performance from the use of temporal cache. Note that, POS tagging task is not affected by ablation of any of the components since only has temporal modality in the input.
POS (Acc.)
Captioning (BLEU-4)
VQA (Overall)
HMDB (Acc.)
IND
MULT4
IND
MULT-4
IND
MULT
IND
MULT
OmniNet
95.61
95.44
28.9
27.4
55.31
55.76
55.29
54.44
woo link array
95.61
95.44
28.9
27.4
54.05
55.30
45.94
46.79
w/o spatial cache
95.61
95.44
11.9
39.86
44.24
10.91
11.50
Table 2: Ablation study on the effect of proposed architectural components.
6 CONCLUSIONS AND FUTURE WORK
We present a unified neural network architecture OmniNet capable of learning tasks with multiple inputs of varying modalities. The architecture can be further adopted for multi-task learning across any set of tasks containing spatio-temporal data. Sharing one model across multiple tasks also results in significant reduction in the total number of parameters. We further demonstrate that this shared model can learn robust representations from various spatio-temporal inputs which are reusable for unseen tasks. We believe that this proposed architecture has wide applicability any task with apatio-temporte inputs. To extend its usability, we would like to introduce new peripherals supporting more domains such as speech. We also keen on exploring other aspects to the data beyond temporal and spatial dimensions such as graphs and relational data. Further, it would be interesting to investigate scheduling mechanisms for optimizing the multi-tasking framework.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, June 2018. doi: 10.1109/CVPR..2088.0064
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http://arxiv org/ abs/ 149.99944 0473
Eric Battenberg, Jitong Chen, Rewon Child, Adam Coates, Yashesh Gaur, Yi Li, Hairong Liu, Sanjeev Satheesh, David Seetapun, Anuroop Sriram, and Zhenyao Zhu, Exploring neural transducers for end-to-end speech recognition. CoRR, abs/1707.07413. 2017. URL http:
//axxiv.org/abe//70000 07413
Yaran Chen, Dongbin Zhao, Le Lv, and Qichao Zhang. Multi-task learning for dangerous object
detection in autonomous driving. Information Sciences, 432:559 - 571, 2018. ISSN 0020-0255.
doi: https://dolorr//1000166ii i01. 2017.00017..... URL htt.hhttp/// //www www.wccc...... encedi rect. com/
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, Oc- tober 2014. Association for Computational Linguisties. doi: 10.3115//1/D44-117, URL https: www ac hwtp:://wwww...////aaaaeeeeeee http.//or///...ooggg///eee/--/-
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 160-167, New York, NY, USA, 2008. ACM. ISBN 978- 1-60558- 205-4. doi: 105/131/39 13901777..11977
1390177
URL http::/ttooio acm. acm. ort//o..g/110000.
1390156. 1390177
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In ACL, 2015.
Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 770-778, June 2016. doi: 10. 10.1109/CVPR.20166
Benjamin Heinzerling and Michael Strube. BPEmb: Tokenization-free Pre-trained Subword Embed- dings in 275 Languages. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mar- iani. Hlne Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018 2018. European Language Resources Associa- tion (ELRA). ISBN 979- 109-5-9-556--00-9
Lanqing Hu, Meina Kan, Shiguang Shan, and Xilin Chen. Duplex generative adversarial network for unsupervised domain adaptation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339 2017. doi: 10.1 16.1/a/l 00065. httpst:s: /ttpss//wwwwt //www//www.//////////////.//.. wwww web. www.b.rrg//aathooooy///011110000

---

Under review as a conference paper at ICLR 2020
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. CoRR, abs/1706.05137, 2017. URL http:
//a wawxiv.org/ab///2.3...a. 1501
Jin-Hwa Kim, Sang-Wou Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Take Zhang. Multimodal residual learning for visual qa. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 361-369, Curran Associates, Inc., 2016. URL httptttt:p/pppss...t nips cc/papex/ 6446 www.mntiiomllllmmmmmm res
Jia-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 1564-1574. Curran Associates, Inc., 2018, URL http:/ /t:://pas pwppppappa..pe//p//eep nips cc/ httpp/pppp//33/ bi bilinear pdf
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. htt.lhtttt///ai...rr////////
6980
Kalpesh Krishna, Shubham Toshniwal, and Karen Livescu. Hierarchical multitask learning for ctc- based speech recognition. CoRR, abs/1807.00 00624,6,44 2018. URL http://erzxv..rr///aa
1807. 06234
H. Kuehne, H. Jhuang, Garrote, Poggio, and T. Serre HMDB: a large video database for human motion recognition. In Proceedings of the International Conference on Computer Vision (ICCV), 2011.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. TVQA: localized, compositional video question answering. CoRR, hts/18099001996 2018. URL http:://aaxxv.orr abs/////
11666
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-11
Thang Luong, Quoe V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. International Conference on Learning Representations, 2016.
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fer- guson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT '94, pp. 114- 119, Stroudsburg. PA, USA, 1994. Association for Computational Linguistics. ISBN 1-55860- 357-3. doi: 1.3115/1075812210078888 URL https://das org/ 31.15110077581111 1075335
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P. Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML 16, pp. 1928 1937. JMLR.org, 2016. URL http:// dl.acm. ort. www.orrgggiiiiiiiooooeeo wwt.iiion.cc.m.....mmmmm wttp::ccccmmcmcmmem0e 0003999..044599 3045594
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Mul- timodal deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Leurning, ICML' 11, pp. 689-696, USA, 2011. Omnipress. ISBN 978-1- 4503-0619-5. htLkttp::// //1,
acm. wtt.:/aawt.a.a......t...t..... /ttg/iiaaiii.oo..oo/.i... c
3104559
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
10

---

Under review as a conference paper at ICLR 2020
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock- free approach to parallelizing stochastic gradient descent. In J. Shawe-Taylos R. S. Zemel, P. L. Bartlett,
F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems
24, pp. 693-701. Curran Associates, Inc., 2011. URL /tap:r//ppppaaaaaa wiw.prrs.iip.cooooo..oo...... wwt.::cpppp.pcpppppppepppppppppe 4390- 1d- lock- rew-ree--ppaeee -otpae-appprccctt---papppaa httpa//ee....pp......ooaaaaaaaaa ott.ppappoccccccccooooooeeeeeeee centeeeeee...iiinneeeeee.e.. pdf
M. I.. Seltzer and J. Droppo. Multi-task learning in deep neural networks for improved phoneme recognition. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing,
pp. 6965-6969, May 2013, doi: 10.1 10.1199IICASPP:2133363990 661669111....
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: (0.18653/1/P11-1111 URL https: www aclweb. bxg//anthology//Pl61111
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1 0423500 2018. Ittp::////aaxx org/ hts//abs//// /104 04235
Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. CoRR, 1bs/188020,875, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp.
568- 176. Curran Associates, Inc., 2014. URL httt:::/pappsss.....h nip. cc papppprrr// www.nnnwwwss-ooooooo -ow.ttoorss-ooraatttoo--eeeooooo on-
5353 www---tttteaamcoooooo.t.....
vi videos.
pdf
Drahomíra "johanka" Spoustová, Jan Hajič, Jan Raab, and Miroslav Spousta. Semi-supervised training for the averaged perceptron POS tagger. Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pp. 763-771, Athens, Greece, March 2009, As- sociation for Computational Linguistics. URL https:://///. /ttp://wwww...... wwwwwb. wttp://www..o..a/htoooooooo/oo/
E09 E09-1087
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Processing Systems 27, pp. 3104 3112. Curran Associates, Inc., 2014. URL http:: nips. cttppperr/ppppr
Christian Szegedy, Sergey loffe, and Vincent Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections learning. CoRR, abs/1602.00261, 2016. URL http:
//arxiv.orrg/aa//11222.000
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL http: /ttp::://paprs....m nips wtapppcccc/eeee httc//aper//// 7718 attention: is -s-alllllllllll yeu..eedd.po
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens. George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick. Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.081 44, 2016. httt::/:///rrt
org/ abs 1609. 08144)
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In In ECCV. 94/08, 2014.

---

Under review as a conference paper at ICLR 2020
Wei Zhao, Benyou Wang, Jianbo Ye, Min Yang. Zhou Zhao, Ruotian Luo, and Yu Qiao. A multi-task learning approach for image captioning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. IJCAI-18, pp. 1205-1211. International Joint Conferences doi: 10.24963/jjcai.200 2018/166..
on Artificial Intelligence Organization, 2018.
URL https:
7/doi. htt.:/oo//1..2263/111e.. 2018a/122111111
12


Reviews:
[{"review_id": "SkxPBcahKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a unified architecture in the context of multi-task learning where they demonstrate that training four tasks (with a variety of modalities like image, text, and videos) together results in about three times compressed model, while maintaining the performance similar to their respective individually trained models. The major components of this archtiecutre are (1) peripheral networks: used to encode the domain specific input into feature representations. (2) central neural processor: a fully attention based encoder-decoder model similar to the Transformer networks which encodes the spatio-temporal information. Further, this paper suggests that their unified architecture enables to perform decent on unseen tasks during its training. In this paper they test such scenarios on video captioning and video question answering. \n\nOverall, the paper is clear to read and thorough in its experiments, with the caveat that its missing many multi-task paper references and the ideas are not much novel. However, I would say the setup is well engineered. \n\nArguments:\n\n1) There are many works in multi-task learning after Luong et al., 2016, please refer them in the related work section (this section is very short!) and discuss on the differences of your model w.r.t. previous work (this is completely missing in the paper). I am pointing to some references below. \n\n2) Statistical significance tests are missing to show that MULT-3 or MULT-4 are able to “maintain” the performance w.r.t. IND. \n\n\n[1] Latent Multi-task Architecture Learning, Ruder et al., 2019\n[2] A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks, Hashimoto et al., 2017\n[3] Multi-Task Video Captioning with Video and Entailment Generation, Pasunuru & Bansal, 2017\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlJ_jyW9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an extended and unifying learning architecture – OmniNet- based on transformer, which tackles tasks with various modalities such as images, text and videos. This is attained with a spatio-temporal cache mechanism, that can both capture and store temporal information and spatial information. In addition, this proposed framework supports asynchronous multi-task learning with pre-trained neural networks on different modalities. OmiNet’s generalization capability is illustrated and demonstrated with experiments. The proposed model has multiple peripheral networks each majoring on one unique modality of data. These peripheral networks project input data into the same shared format/space that can be uniformly processed in a central neural processor working like a CPU. The central neural processor uses self-attention and RNN for temporal encoding and spatial encoding. The output of the encoding components are stored in temporal and spatial caches respectively. The two caches are then used as input to the spatial temporal decoder for different tasks. \n\n\nOverall, this paper is well-written, and technically sounds, with comprehensive experimental results. However, I still have two concerns below that prevent me from giving a direct acceptance.\n\n\n1.\tHowever, considering the proposed model attempts to solve the multi-task learning problem, there seems no multi-task learning methods compared as baselines, making it hard to justify the performance.\n\n2.\tFurthermore, in Table 1, the Omin-Net results are not as good as SOTA, without clear explanation. The parameter settings for SOTA are also missing. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxUcO-2YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes the \"OmniNet\" architecture, which is essentially a transformer to convert any 2-dimensional (time and spatial) input into a sequence of output tokens. The idea is that a single model (the \"Central Neural Processor\") would learn to perform multiple tasks on multiple inputs at the same time. This seems like a reasonable approach, and would allow a single model to be able to process text, image, as well as video or potentially speech input just the same. Dedicated modality-specific \"input peripherals\" would \"normalize\" the data appropriately, and a \"start token\" seems to provide the model with information on what type of input data is to follow.\n\nI am a bit confused by the problem formulation and implementation:\n- Multi-task learning works best if there are clear dependencies and shared characteristics between the task - I am not sure if the tasks used here (POS tagging, image captioning, VQA, video activity recognition) share sufficient characteristics \n- Successful multi-task learning often leads to shared representations developing across the different tasks. Is there any indication of this happening in the proposed work? Would it help to make the modality specific encoders (BPE and ResNet) trainable, so that the transformer model has an easier time learning multi- and cross-modal representations? Or is this entirely the CNP's role?\n- It is probably possible to use HogWild as a way to train a multi-modal and multi-task system in the described manner, but will this lead to shared representations developing? Is there any way to include regularization or other tricks to encourage the formation of shared cross-modal representations?\n\nHere are a few ideas:\n- Would it be possible to show how well the model works when trained and optimized (including hyper parameter optimization) on a single task only? it would then be possible to show that the multi-task model can outperform single-modality single-task models \n- What about tasks like speech input? There is other, published work on multi-task training for CTC models w.g. by Ramon Sanabria (\"Hierarchical Multi Task Learning With CTC\", SLT 2018), in addition to the reference provided\n- Would it be possible to show how the model learns to develop cross-modal representations during training? this would be a convincing visualization and rationalization of the proposed approach\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents OmniNet, an architecture based on the popular transformer for learning on data from multiple modalities and predicting on multiple tasks.  The reviewers found the paper well written, technically sound and empirically thorough.  However, overall the scores fell below the bar for acceptance and none of the reviewers felt strongly enough to 'champion' the paper for acceptance.  The primary concern cited by the reviewers was a lack of strong baselines, i.e. comparison to other methods for multi-task learning.  Unfortunately, as such the recommendation is to reject.  However, adding a thorough comparison to existing literature empirically and in the related work would make this a much stronger submission to a future conference.
Decision: Reject