Title: OMNINET: A UNIFIED ARCHITECTURE FOR MULTI-MODAL MULTI-TASK LEARNING

Abstract: Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.

1. INTRODUCTION:

Transformer (Vaswani et al., 2017) is currently one of the best performing models for any sequence transduction tasks, especially those involving natural language. It is originally designed for a single task at a time. In fact, most of the generic deep learning architectures (Wu et al., 2016; Szegedy et al., 2016; Battenberg et al., 2017) that have been designed and developed are able to learn, albeit very well, a single task and handle one task specific input domain like image, text or audio. Furthermore with these models, we often rely on the generalization capability of the trained network to guarantee performance on unseen examples. Transfer learning (Shu et al., 2018; Hu et al., 2018) is another popular paradigm used to adapt the model to learn a related task with similar input domain. The success of neural networks across these challenges is known to be due to their ability in learning effective representations of the data. For example, the self-attention mechanism in Transformers can capture the global temporal dependence in sequential data very well. Naturally, the question arises whether we can extend these architectures, like the Transformer, to be able to learn shared representations from multiple input domains and to be able attend on these representations to perform a multitude of tasks concurrently.

The research into multi-task models that learn to solve varied tasks across a multitude of input domains is not new. Work done in Ngiam et al. (2011) demonstrates an architecture capable of learning a shared representation across audio and video modalities. Similarly in Collobert & Weston (2008) a convolutional architecture has been designed to support a variety of NLP tasks. However, most of these architectures are designed to learn specific set of tasks with known input domains. To the best of our knowledge, there does not exist a single unified architecture that works out of the box for any combination of multi-modal inputs.

To address this gap, we extend Transformer towards a unified architecture, namely OmniNet, which enables a single model to support tasks with multiple input modalities and asynchronous multi-task learning. We consider that most real-life data like image, text, speech, video, etc. is a direct conjunction of spatial and temporal components. Therefore, we employ a spatio-temporal cache mechanism to learn a shared representation of the input data across the spatial (space) and temporal (time) di-mension. Using a generalized encode() function, OmniNet can process and store spatio-temporal representation for each of the input domains and then decode() predictions across a multitude of tasks. In our experiments, we train a single instance of the OmniNet to solve a number of tasks spanning multiple multi-domains such as part-of-speech tagging, image captioning, visual question answering and video activity recognition. To make our work reproducible, open to scrutiny and further development, we will open source a demonstration of our system implemented using Pytorch (Paszke et al., 2017) .

2. RELATED WORK:

Multi-task learning has been extensively studied in the literature, with applications to a wide set of problems ranging from natural language processing (NLP) (Collobert & Weston, 2008; Johnson et al., 2017; Dong et al., 2015) to speech recognition (Seltzer & Droppo, 2013; Krishna et al., 2018) to vision (Zhang et al., 2014; Chen et al., 2018; Anderson et al., 2018) . It has also found its use in a combination of diverse tasks like image captioning and text translation and parsing (Luong et al., 2016; Zhao et al., 2018) . However, most of these architectures assume the set of tasks to be known in advance. Similarly, multi-modal learning has been essential for solving a broad range of interesting problems such as Visual Question Answering (Kim et al., 2016; 2018) and Video Question Answering (Lei et al., 2018) . Again, the state-of-the-art models are highly specific to the objective in hand and not easily adaptable to different tasks or domains. (Kaiser et al., 2017) proposed MultiModel architecture for learning multiple tasks but lacks support for multi-modal tasks with more than one input domains such as visual question answering.

3. PROPOSED MODEL:

We propose a unified architecture, namely OmniNet, to enable learning multi-modal tasks with multiple input domains and support generic multi-tasking for any set of tasks. The OmniNet architecture consists of multiple sub-networks, called peripheral networks, connected to a common central neural network called the Central Neural Processor (CNP) (Figure 1 ). Each peripheral network is used to encode the domain specific input into feature representations. In this work, we describe image, text and video peripherals (Section 3.1). One can add more, say speech peripheral, depending on the task. The output representation of a peripheral network is always a spatio-temporal tensor x ∈ R t×s×d model , where t & s are the temporal and spatial dimensions of the input respectively, and d model is the model dimension input to the CNP. The spatio-temporal representations generated by the peripheral networks corresponding to each input domain are then processed by the CNP. The CNP uses fully attention based encoder-decoder (Bahdanau et al., 2015; Sutskever et al., 2014; Cho et al., 2014) model for sequence transduction similar to the Transformer architecture (Vaswani et al., 2017) , which is the state-of-the-art for multiple language modeling tasks (Section 3.2). During the encoding stage, the CNP implements a generic encode(x, D) function to first process and store the spatio-temporal representations of the input, where x ∈ R t×s×d model is the spatio-temporal tensor produced by the peripheral networks and D ∈ Z : 0 ≤ D < D len is the domain id and D len is the max number of domains supported by the CNP. The encode() function is called multiple times, once for each multi-modal input from respective peripheral. During the decoding stage, a decode(y shif ted , τ ) function is used to decode predictions as softmax probabilities, where y shif ted ∈ Z N -1 are the target outputs shifted one timestep to the right, N is the length of the output sequence; τ ∈ Z : 0 ≤ τ < τ len is task id and τ len is the total number of supported tasks. The decoding step is similar to Vaswani et al. (2017) , modified to incorporate a two-step attention mechanism over spatial and temporal cache.

3.1. PERIPHERAL NETWORKS:

First, we elaborate on how we support multiple input domains using peripheral networks. A peripheral network can use a pre-trained model from existing literature to ultimately encode a given domain input to a standardized feature representation x ∈ R t×s×d model , where t & s are the temporal and spatial dimensions of the input respectively, and d model is the model dimension input to the Central Neural Processor. Here we detail text and vision peripherals and one can add more peripherals or alter the peripheral design depending on the task.

Vision peripheral: This peripheral uses a convolutional neural network to encode image and video inputs in the tasks. For an image of dimension h × w × n c , this peripheral down-samples in to h × w × n c , where h, w, n c are the height, width and number of input channels respectively. For a video, each frame is input to the peripheral to produce F × h × w × n c , where F is the total number of frames in the video. The encoding vectors are then projected to dimension d model using a fully connected layer. The output is then reshaped into a spatio-temporal tensor of x ∈ R t×h w ×d model , where t = 1 for an image and t = F for a video. In our experiments, we use the pre-trained ResNet-152 model, a variant of ResNet (He et al., 2016) consisting of 152 convolutional layers. We remove the final fully connected and avg-pooling layers to generate spatial feature representations for a given image/video.

Language peripheral::

The Language peripheral uses byte-pair encoding (Sennrich et al., 2016) to generate subwords for a given input sentence. The subwords are passed to an embedding layer to generate subword embeddings of dimension d emb and projected to dimension d model using a fully connected layer. The output is then reshaped into a spatio-temporal tensor x ∈ R t×1×d model , where t equal to number of subwords in the input sentence. As we do not have any spatial dimension in textual data, the spatial dimension of x from a Language peripheral is always 1. In our experiments, We used pre-trained subword embeddings with d emb = 300 and vocab size = 25000 from Heinzerling & Strube (2018), which includes pre-trained subword embeddings of over 275 languages, to initialize the weights of the embedding matrix.

3.2. CENTRAL NEURAL PROCESSOR (CNP):

To process the spatio-temporal information in the input data, the CNP implements a spatial cache C s , temporal cache C t and a link array L. The spatial and temporal cache and the link array are a list of elements, initialized as empty before the encoding process. During the encoding stage, an encode() routine takes as input, the tensor x generated from the peripheral and corresponding domain/peripheral id D. This function processes the spatial and temporal information in the input x and stores them into the spatial cache C s and the temporal cache C t , respectively and stores their dimensions t & s in the link array. For a given task, this encode() routine is called K times, where K is the number of inputs in the task. Note that these inputs can belong to same or different domains.

Encode (x, D): For a given input x ∈ R t×s×d model and domain identifier D, the encode() routine is described in Algorithm 1. Since inputs can come from multiple peripherals, the algorithm first concatenates the input with the domain embedding to ensure a domain-aware encoding of the input (Steps 2 to 3). Steps 4 to 7 process the spatial information in x by unrolling the time dimension and adding these unrolled vectors into the spatial cache. Steps 8 to 10 process the temporal information in x by averaging the spatial dimension of x and then passing the averaged tensor to a self-attention based TemporalEncoder. This TemporalEncoder is similar to the encoder used in Vaswani et al. (2017) as shown in Figure 2 is used to calculate temporal embeddings of the input sequence. The output from the TemporalEncoder is appended to the temporal cache. 

: x ∈ R t×s×d model , D, C s , L, C t 1: L ← L ∪ (t → s) 2: D emb ← EmbedLayer(D) 3: x ← F C(Concat(x, D emb ), d model ) 4: if s > 1 then 5: S ← Reshape(x, (ts, d model )) {where, output S = [S 1 , . . . , S ts ] s.t. S i ∈ R d model is a spatial feature vector.} 6: C s ← C s ∪ [S 1 , . . . , S ts ] {Append spatial representations to spatial cache} 7: end if 8: T ← ( s i=1 x[:, i, :])/s 9: T ← T emporalEncoder(T ) {where, output T = [T 1 , . . . , T t ] s.t. T j ∈ R d model is the encod- ing of temporal dimension in x.} 10: C t ← C t ∪ [T 1 , . . . , T t ] {Append temporal representations to temporal cache}

The above encoding routine keeps appending spatio-temporal information to C t & C s for each input x k ∈ R t k ×s k ×d model . Note the superscript k to denote correspondence to k-th input of the task, where k ∈ 1, . . . , K. After K calls, we have the temporal cache C t = [T 1 , . . . , T R ], where R = K r=1 t r ; the spatial cache C s = [S 1 , . . . , S P ], where P = { p t p * s p : p ∈ 1, . . . , K ∧ s p > 1} and the link array L = [(t 1 → s 1 ), . . . , (t K → s K )]. Note that C s can also be empty in case the encode() is only called with inputs with s k = 1∀k. Next, we use the decode() routine to generate predictions as softmax probabilities.

Decode (y shif ted , τ ): The architecture of the decode() function is shown in Figure 2 . The decode() takes as argument the output labels y shif ted shifted one time step to the right, a task id τ and gen-erates predictions by attending from the spatial and temporal cache. The decode() function is structured similar to the decoder used in the Transformer architecture Vaswani et al. (2017) and jointly attends on the vectors stored in the temporal and spatial cache. Similar to Vaswani et al. (2017) , the decoding first starts by attending over the output embeddings using masked multi-head scaled dot product attention. The attention layer for the temporal cache uses scaled dot-product attention with multiple heads as specified in Vaswani et al. (2017) . Attention layer for the spatial cache, uses gated multi-head attention to attend over the elements of the spatial cache. For inputs with both time and space dimension (e.g. video), we want the spatial attention layer to attend more on frames which have relatively high attention scores in the temporal cache attention layer. Therefore, the attention score output from the temporal cache multi-head attention layer A ∈ R n h ×N ×R , is used to calculate the tensor G ∈ R n h ×N ×P used for gating the attention score output in the spatial attention layer, where n h is the number of heads in multi-head attention as described in Vaswani et al. (2017) . The tensor G is calculated using A & L as detailed in Algorithm 2. Given1 Q: the matrix of queries, K: keys of dimension d k and V : values of dimension d v , the scaled dot-product attention for the spatial layer is modified as:

EQUATION

In order to use the same CNP for multiple tasks with varying output vocabularies, we use multiple output embedding layers OutputEmbedLayer 1 , . . . , OutputEmbedLayer τ len , to generate the output embeddings for each task.

At the final layer, we use multiple (F C + Sof tmax) 1 , . . . , (F C + Sof tmax) τ len classification layers for each task. We also calculate a task embedding vector using τ and always start decoding using the task embedding vector.

Algorithm 2 Calculate G using output scores from temporal attention and link array Require: L, A 1: idx ← 0 2: for each t, s in L do

3: G ← [] 4:

if s > 0 then

5::

A ← A[:, :, idx : idx + t] 6:

A ← Expand(A , (n h , N, t, s)) {where Expand(tensor, dimension) expands tensor according to a given dimension} 7:

A ← Reshape(A , (n h , N, ts))

8:

G ← G ∪ A {Append the respective temporal attention scores to G} 9:

end if 10:

idx ← idx + t 11: end for 12: G ← Stack(G) {Stack the list of tensors to construct tensor G of dimension (n h , N, P )}

3.3. MULTI-TASK LEARNING:

In order to train a single a model simultaneously on mutiple tasks we used the HogWild training approach as described in Recht et al. (2011) . Similar to the approach described in Mnih et al. (2016) , the main process holds a global copy of the model. We create separate worker processes for each task, where each process maintains a local copy of a model. At each training iteration, each process starts by synchronizing its local model with the global copy. This is done through forward and backward propagation on its local copy and then copying the locally computed gradients to the global model asynchronously. Each process then calls the global model optimizer asynchronously to update the weights of the global model. Instead of storing the model in CPU as in Mnih et al. (2016) we always store the local copies across multiple GPUs.

4. TASKS AND SETUP:

To evaluate the effectiveness of our proposed framework for tasks spanning diverse modalities, we choose a set covering all possible spatio-temporal data archetypes: Image Captioning, Part-of-Speech (POS) tagging, Visual Question Answering (VQA) and Video-activity Recognition. Each of these tasks explores a unique potential spatio-temporal configuration of the input containing different values of t and s, where t and s are the temporal and spatial dimensions of the input respectively. This enables us to perform a comprehensive study of the multi-modal and multi-input capabilities of the system. We further elaborate on the properties of these tasks below.

For training, we always use cross-entropy loss with Adam optimizer (Kingma & Ba, 2015) and schedule the learning rate using Noam scheduler (Shazeer & Stern, 2018) similar to (Vaswani et al., 2017) 2 . In the vision peripheral, we freeze the layers of pretrained ResNet model. Remaining parts of the architecture (peripherals and CNP) are kept trainable. In this section, we provide details on the datasets used and the model setup for each of these tasks.

Part-of-speech (POS)Tagging: To illustrate the task with only temporal modality (t > 1 & s = 1, where t and s are the temporal and spatial dimensions of the input respectively), we consider POS tagging problem. Given an input sequence of words, the model should produce a sequence of POS tags corresponding to each word. We use Penn Tree-bank3 (Marcus et al., 1994) which contains gold annotations on English WSJ articles by experts. During the encoding stage, each input sentence is processed by the language peripheral to generate a spatio-temporal tensor x ∈ R t×1×d model , where t is the sequence length of subwords in the input. The CNP encode() function is then used to encode x into the temporal cache. Note that spatial cache is empty for text inputs as s = 1. Therefore, the decoding stage is same as that of Transformers to predict the sequence of POS tags.

Image Captioning: This task represents the ones with inputs containing only spatial modality (t = 1 & s > 1). The captioning model is required to predict text caption for a given image. We use the MSCOCO 2014 dataset (Lin et al., 2014) for training and present results on the COCO validation set. During the encoding stage, the input image is resized to 224 × 224 and processed by the vision peripheral containing pre-trained ResNet-152 to produce image embeddings x ∈ R 1×49×d model . x is then input to the encode() function which populates corresponding spatial and temporal cache. The decoding stage uses decode() function with output vocabulary size 25000 to generate the captions.

Visual Question Answering: For the task with inputs from multiple domain, such that each contains either spatial or temporal modality (either t > 1 & s = 1, or t = 1 & s > 1 for each input), we choose the task of visual question answering. Given a question over an image as inputs, the model is supposed to predict the correct answer label. We use the recently introduced VQA v2.0 dataset (Goyal et al., 2017) for this purpose and perform evaluation on the VQA test-dev set. All the images are resized to dimension 224 × 224 before training. The encoding stage of this task utilizes two peripherals: the vision peripheral is used to generate a tensor x 1 ∈ R 1×49×d model for the input image. The language peripheral is used to encode the questions into x 2 ∈ R t×1×d model , where t is equal to the length of the subwords in the question. The encode() function is the called two times, first with x 1 and second with x 2 as input. Finally, the decode() with output vocabulary size 3500 is to generate the answers as softmax probabilities in a single decoding step.

Video Activity Recognition: For tasks which contain both spatial and temporal modality in a single input (t > 1 & s > 1), we consider the action recognition task on videos. For this purpose, we use the HMDB dataset (Kuehne et al., 2011) . The dataset consists of over 5000 short length clips of real life actions with 51 classes. We present our results on train-test split 1. We use 16 frames per video and resize each of them to 224 × 224. During the encoding stage, each frame of the video is passed through the vision peripheral to cumulatively generate a video encoding x ∈ R 16×49×d model which is then used as input to the encode() function. Finally, the decode() with output vocabulary size 51 is to predict the action as softmax probabilities in a single decoding step.

5. RESULTS AND DISCUSSION:

We present the evaluation on (a) Tasks of various modalities illustrated in Section 4 (b) Multi-tasking setup for these tasks (Table 1 ) (c) Reuse of the multi-task model for an unseen task (Figure 3 ). In addition, we also provide some ablation studies on the architecture (Table 2 ) Performance of proposed architecture on individual tasks: We choose a set of four tasks with diverse input modalities and combinations as described in previous Section 4. We train the OmniNet model independently across each of the above tasks. Each of the tasks demonstrates unique capabilities of this generic architecture. More specifically, in Table 1 we compare our results with the following state-of-the-art4 :-POS tagging: Spoustová et al. (2009) ; image captioning & VQA: Anderson et al. ( 2018) and HMDB: Simonyan & Zisserman (2014) . It is important to note that we do not perform any hyper-parameter optimization. We believe that, with more computational power, fine tuning the hyperparameters towards these tasks should result in comparable or even improved performance to the state-of-the-art. These results can indeed be used as a baseline for any future work which aims at using a single architecture across various possible spatio-temporal archetypes. It is interesting to note that the model is extensible to a new domain without any modification to the CNP as long as one can add a specific peripheral to convert domain inputs into spatio-temporal tensors. This aspect of the architecture makes it applicable to several popular multi-modal tasks. Effect of training a diverse set of tasks together: We trained two multi-task models: (1) MULT-3 (POS+VQA+Captioning) and ( 2) MULT-4 (POS+VQA+Captioning+HMDB), using hogwild approach. While the MULT-3 model attains similar and sometimes better performance, the final MULT-4 model attains slightly reduced performance, when compared to the independent task scores. We believe this is due to the skewness in the size of the HMDB dataset containing only 5000 training samples. However as a tradeoff, adding the HMDB task shows interesting zero-shot results demonstrated below. Using a multi-task model also results in three times reduction in the total number of parameters. That is, when a separate model is used for each task, we have a total of over 450 × 10 6 parameters. Whereas during multi-tasking since a single model is shared, we have a total of over 149 × 10 6 parameters, while achieving similar performance. Interestingly, the model is able to attend on spatio-temporal components of the inputs from different tasks and concurrently generate predictions across them, thus demonstrating the generalization capability of our architecture. Towards zero-shot learning: reuse of pre-trained network for unseen task: Sharing representations across multiple tasks provides the benefit to transfer of useful knowledge across multiple domains. Since, image and video are processed by the same vision peripheral, we conducted an experiment to see whether our model pre-trained on all the four tasks (MULT-4) can perform video captioning and video question-answering without any explicit training on these tasks i.e. zero-shot learning. The results of the evaluation on randomly picked instances from the HMDB test split 1 are shown in Figure 3 . Interestingly, the model performs quite well on related actions that were present in the COCO and VQA training set; such as captions related to horse riding and baseball; or questions related to concepts present in VQA. Without training on any video captioning & video QA instance, the model could use the trained information from image captioning, image QA (VQA) and video action recognition (HMDB) apply them on videos to generate meaningful predictions, hence demonstrating the capability of the model to transfer knowledge across related multi-modal tasks. However, on concepts that are not present in the trained datasets, the model either describes the environment in the video or replaces with alternate known concepts. This case study, although not comprehensive, shows the capability of the model to learn shared representations and ability to transfer knowledge across domains. We believe that adding more tasks and domains will lead to more interesting zero-shot learning results in future across a wide range of problems.

Impact of individual architectural components::

In order to support different input domains, our architecture introduces spatial cache and link array components to the original Transformer architecture (which only consists of mechanisms to handle temporal data). We conducted an ablation study on each of these components to verify their importance across various tasks as shown in Table 2 .

The ablation was conducted on the independent (IND) as well the multi-tasking model (MULT-4).

The second row ablates the link array from our architecture i.e. removing the multiplication of G in Equation 1. The link array was designed to assist in tasks with inputs such as video, containing both spatial as well as temporal modality in a single input. The total number of spatial components becomes very large as number of frames in the video increases, thereby making it difficult to attend on various spatial regions throughout the video. Using link array the spatial attention layer can attend more on specific important frames in the video. Therefore, removal of link array leads to a huge reduction in performance in HMDB compared to other tasks as they do not have both spatio-temporal modalities for any single input. Removal of spatial cache, on the other hand, has significant effect on performance across all tasks containing spatial modality. Since, image captioning contains primarily spatial modality and hence the BLEU drops significantly after ablation. As other tasks utilize the temporal cache for prediction, in the multi-task setting the captioning task learns to utilize the spatial average of the image stored in the temporal cache for prediction and hence retains some performance after ablation of the spatial cache. On the other hand when trained independently on captioning, the network learns to utilize the information in the spatial cache only, and hence the score drops to zero after ablation. VQA leverages both, spatial information from image and temporal information from question, retains some performance from the use of temporal cache. Note that, POS tagging task is not affected by ablation of any of the components since it only has temporal modality in the input. 

6. CONCLUSIONS AND FUTURE WORK:

We present a unified neural network architecture OmniNet capable of learning tasks with multiple inputs of varying modalities. The architecture can be further adopted for multi-task learning across any set of tasks containing spatio-temporal data. Sharing one model across multiple tasks also results in a significant reduction in the total number of parameters. We further demonstrate that this shared model can learn robust representations from various spatio-temporal inputs which are reusable for unseen tasks. We believe that this proposed architecture has wide applicability to any task with spatio-temporal inputs. To extend its usability, we would like to introduce new peripherals supporting more domains such as speech. We are also keen on exploring other aspects to the data beyond temporal and spatial dimensions such as graphs and relational data. Further, it would be interesting to investigate scheduling mechanisms for optimizing the multi-tasking framework.

:

For brevity, we reuse the notations ofVaswani et al. (2017) in this description

The hyperparameter values used for n h , d model , N Layers, d k , dv are same as that specified in Transformer base model(Vaswani et al., 2017).

3 https://catalog.ldc.upenn.edu/LDC99T42; We use splits 0-18 as training, 19-21 as development and 22-24 as test sets

Since most of these tasks are popular challenges, we compare with state-of-the-art which are generically applicable for the respective task instead of the challenge dataset.

Figures and Tables:

Figure 1: OmniNet performing image captioning, visual question answering and POS tagging at once

Figure 2: left: TemporalEncoder architecture; right: OmniNet decode() architecture.

Figure 3: Results of zero-shot video captioning and video question-answering.

Performance of OmniNet on diverse set of tasks. IND: Model trained individually for each of the given tasks; MULT-3: Multi-task model trained on POS, Captioning & VQA; MULT-4: Multi-task model trained across all the four tasks.

Table content: <table><tr><td/><td>POS</td><td colspan="2">Captioning</td><td colspan="4">Visual Question Answering</td><td>HMDB</td><td/></tr><tr><td/><td>Acc.</td><td colspan="3">BLEU-4 Meteor Overall</td><td>Y/N</td><td colspan="2">Num. Other</td><td>Acc.</td><td>#PARAMS</td></tr><tr><td>SOTA</td><td>97.44</td><td>36.2</td><td>27.0</td><td>63.2</td><td>80.3</td><td>42.8</td><td>55.8</td><td>59.4</td><td>-</td></tr><tr><td>IND</td><td>95.61</td><td>28.9</td><td>25.2</td><td>55.31</td><td colspan="3">74.09 35.17 46.35</td><td>55.29</td><td>450m</td></tr><tr><td colspan="2">MULT-3 95.82</td><td>28.8</td><td>25.2</td><td>56.79</td><td colspan="3">76.75 35.82 47.16</td><td>-</td><td>149.03m</td></tr><tr><td colspan="2">MULT-4 95.44</td><td>27.4</td><td>24.5</td><td>55.76</td><td colspan="3">75.49 35.64 46.08</td><td>54.44</td><td>149.07m</td></tr></table>

Ablation study on the effect of proposed architectural components.

Table content: <table><tr><td/><td colspan="2">POS (Acc.)</td><td colspan="2">Captioning (BLEU-4)</td><td colspan="2">VQA (Overall)</td><td colspan="2">HMDB (Acc.)</td></tr><tr><td/><td>IND</td><td colspan="2">MULT-4 IND</td><td>MULT-4</td><td>IND</td><td>MULT-4</td><td>IND</td><td>MULT-4</td></tr><tr><td>OmniNet</td><td>95.61</td><td>95.44</td><td>28.9</td><td>27.4</td><td>55.31</td><td>55.76</td><td>55.29</td><td>54.44</td></tr><tr><td>w/o link array</td><td>95.61</td><td>95.44</td><td>28.9</td><td>27.4</td><td>54.05</td><td>55.30</td><td>45.94</td><td>46.79</td></tr><tr><td colspan="2">w/o spatial cache 95.61</td><td>95.44</td><td>0</td><td>11.9</td><td>39.86</td><td>44.24</td><td>10.91</td><td>11.50</td></tr></table>

Peer review (iclr2020):
This paper describes the *OmniNet* architecture, which is essentially a transformer to convert any 2-dimensional (time and spatial) input into a sequence of output tokens. The idea is that a single model (the *Central Neural Processor*) would learn to perform multiple tasks on multiple inputs at the same time. This seems like a reasonable approach, and would allow a single model to be able to process text, image, as well as video or potentially speech input just the same. Dedicated modality-specific *input peripherals* would *normalize* the data appropriately, and a *start token* seems to provide the model with information on what type of input data is to follow. I am a bit confused by the problem formulation and implementation: - Multi-task learning works best if there are clear dependencies and shared characteristics between the task - I am not sure if the tasks used here (POS tagging, image captioning, VQA, video activity recognition) share sufficient characteristics - Successful multi-task learning often leads to shared representations developing across the different tasks. Is there any indication of this happening in the proposed work? Would it help to make the modality specific encoders (BPE and ResNet) trainable, so that the transformer model has an easier time learning multi- and cross-modal representations? Or is this entirely the CNP*s role? - It is probably possible to use HogWild as a way to train a multi-modal and multi-task system in the described manner, but will this lead to shared representations developing? Is there any way to include regularization or other tricks to encourage the formation of shared cross-modal representations? Here are a few ideas: - Would it be possible to show how well the model works when trained and optimized (including hyper parameter optimization) on a single task only? it would then be possible to show that the multi-task model can outperform single-modality single-task models - What about tasks like speech input? There is other, published work on multi-task training for CTC models w.g. by Ramon Sanabria (*Hierarchical Multi Task Learning With CTC*, SLT 2018), in addition to the reference provided - Would it be possible to show how the model learns to develop cross-modal representations during training? this would be a convincing visualization and rationalization of the proposed approach