The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings

1 INTRODUCTION . Natural organisms benefit from the fact that their sensory inputs and action outputs are all organized in the same space , that is , the physical universe . This consistency makes it easy to apply the same predictive functions across diverse settings . Deep multi-task learning ( Deep MTL ) has shown a similar ability to adapt knowledge across tasks whose observed variables are embedded in a shared space . Examples include vision , where the input for all tasks ( photograph , drawing , or otherwise ) is pixels arranged in a 2D plane ( Zhang et al. , 2014 ; Misra et al. , 2016 ; Rebuffi et al. , 2017 ) ; natural language ( Collobert & Weston , 2008 ; Luong et al. , 2016 ; Hashimoto et al. , 2017 ) , speech processing ( Seltzer & Droppo , 2013 ; Huang et al. , 2015 ) , and genomics ( Alipanahi et al. , 2015 ) , which exploit the 1D structure of text , waveforms , and nucleotide sequences ; and video game-playing ( Jaderberg et al. , 2017 ; Teh et al. , 2017 ) , where interactions are organized across space and time . Yet , many real-world prediction tasks have no such spatial organization ; their input and output variables are simply labeled values , e.g. , the height of a tree , the cost of a haircut , or the score on a standardized test . To make matters worse , these sets of variables are often disjoint across a set of tasks . These challenges have led the MTL community to avoid such tasks , despite the fact that general knowledge about how to make good predictions can arise from solving seemingly “ unrelated ” tasks ( Mahmud & Ray , 2008 ; Mahmud , 2009 ; Meyerson & Miikkulainen , 2019 ) . This paper proposes a solution : Learn all variable locations in a shared space , while simultaneously training the prediction model itself ( Figure 1 ) . To illustrate this idea , Figure 1a gives an example of four tasks whose variable values are measured at different locations in the same underlying 2D embedding space . The shape of each marker ( i.e. , ◦ , ,4 , ? ) denotes the task to which that variable belongs ; white markers denote input variable , black markers denote output variables , and the background coloring indicates the variable values in the entire embedding space when the current sample is drawn . As a concrete example , the color could indicate the air temperature at each point in a geographical region at a given moment in time , and each marker the location of a temperature sensor ( however , note that the embedding space is generally more abstract ) . Figure 1b-c shows a model that can be applied to any task in this universe , using the ◦ task as an example : ( b ) The function f encodes the value of each observed variable xi given its 2D location zi ∈ R2 , and these encodings are aggregated by elementwise addition ⊕ ; ( c ) The function g decodes the aggregated encoding to a prediction for yj at its location zj . Such a predictor can be viewed as a traveling observer model ( TOM ) : It traverses the space of variables , taking a measurement at the location of each input . Given these observations , the model can make a prediction for the value at the location of an output . In general , the embedded locations z are not known a priori ( i.e. , when input and output variables do not have obvious physical locations ) , but they can be learned alongside f and g by gradient descent . The input and output spaces of a prediction problem can be standardized so that the measured value of each input and output variable is a scalar . The prediction model can then be completely agnostic about the particular task for which it is making a prediction . By learning variable embeddings ( VEs ) , i.e. , the z ’ s , the model can capture variable relationships explicitly and supports joint training of a single architecture across seemingly unrelated tasks with disjoint input and output spaces . TOM thus establishes a new lower bound on the commonalities shared across real-world machine learning problems : They are all drawn from the same space of variables that humans can and do measure . This paper develops a first implementation of TOM , using an encoder-decoder architecture , with variable embeddings incorporated using FiLM ( Perez et al. , 2018 ) . In the experiments , the implementation is shown to ( 1 ) recover the intuitive locations of variables in space and time , ( 2 ) exploit regularities across related datasets with disjoint input and output spaces , and ( 3 ) exploit regularities across seemingly unrelated tasks to outperform single-tasks models tuned to each tasks , as well as current Deep MTL alternatives . The results confirm that TOM is a promising framework for representing and exploiting the underlying processes of seemingly unrelated tasks . 2 BACKGROUND : MULTI-TASK ENCODER-DECODER DECOMPOSITIONS . This section reviews Deep MTL methods from the perspective of decomposition into encoders and decoders ( Table 1 ) . In MTL , there are T tasks { ( xt , yt ) } Tt=1 that can , in general , be drawn from different domains and have varying input and output dimensionality . The tth task has nt input variables [ xt1 , . . . , xtnt ] = xt ∈ Rnt and mt output variables [ yt1 , . . . , ytmt ] = yt ∈ Rmt . Two tasks ( xt , yt ) and ( xt′ , yt′ ) are disjoint if their input and output variables are non-overlapping , i.e. , ( { xti } nti=1∪ { ytj } mt j=1 ) ∩ ( { xt′i } nt′i=1∪ { yt′j } mt′ j=1 ) = ∅ . The goal is to exploit regularities across task models xt 7→ ŷt by jointly training them with overlapping parameters . The standard intra-domain approach is for all task models to share their encoder f , and each to have its own task-specific decoder gt ( Table 1a ) . This setup was used in the original introduction of MTL ( Caruana , 1998 ) , has been broadly explored in the linear regime ( Argyriou et al. , 2008 ; Kang et al. , 2011 ; Kumar & Daumé , 2012 ) , and is the most common approach in Deep MTL ( Huang et al. , 2013 ; Zhang et al. , 2014 ; Dong et al. , 2015 ; Liu et al. , 2015 ; Ranjan et al. , 2016 ; Jaderberg et al. , 2017 ) . The main limitation of this approach is that it is limited to sets of tasks that are all drawn from the same domain . It also has the risk of the separate decoders doing so much of the learning that there is not much left to be shared , which is why the decoders are usually single affine layers . To address the issue of limited sharing , the task embeddings approach trains a single encoder f and single decoder g , with all task-specific parameters learned in embedding vectors zt that semantically characterize each task , and which are fed into the model as additional input ( Yang & Hospedales , 2014 ; Bilen & Vedaldi , 2017 ; Zintgraf et al. , 2019 ) ( Table 1b ) . Such methods require that all tasks have the same input and output space , but are flexible in how the embeddings can be used to adapt the model to each task . As a result , they can learn tighter connections between tasks than separate decoders , and these relationships can be analyzed by looking at the learned embeddings . To exploit regularities across tasks from diverse and disjoint domains , cross-domain methods have been introduced . Existing methods address the challenge of disjoint output and input spaces by using separate decoders and encoders for each domain ( Table 1c ) , and thus they require some other method of sharing model parameters across tasks , such as sharing some of their layers ( Kaiser et al. , 2017 ; Meyerson & Miikkulainen , 2018 ) or drawing their parameters from a shared pool ( Meyerson & Miikkulainen , 2019 ) . For many datasets , the separate encoder and decoder absorbs too much functionality to share optimally , and their complexity makes it difficult to analyze the relationships between tasks . Earlier work prior to deep learning showed that , from an algorithmic learning theory perspective , sharing knowledge across tasks should always be useful ( Mahmud & Ray , 2008 ; Mahmud , 2009 ) , but the accompanying experiments were limited to learning biases in a decision tree generation process , i.e. , the learned models themselves were not shared across tasks . TOM extends the notion of task embeddings to variable embeddings in order to apply the idea in the cross-domain setting ( Table 1d ) . The method is described in the next section . 3 THE TRAVELING OBSERVER MODEL . Consider the set of all scalar random variables that could possibly be measured { v1 , v2 , ... } = V . Each vi ∈ V could be an input or output variable for some prediction task . To characterize each vi semantically , associate with it a vector zi ∈ RC that encodes the meaning of vi , e.g. , “ height of left ear of human adult in inches ” , “ answer to survey question 9 on a scale of 1 to 5 ” , “ severity of heart disease ” , “ brightness of top-left pixel of photograph ” , etc . This vector zi is called the variable embedding ( VE ) of vi . Variable embeddings could be handcoded , e.g. , based on some featurization of the space of variables , but such a handcoding is usually unavailable , and would likely miss some of the underlying semantic regularities across variables . An alternative approach is to learn variable embeddings based on their utility in solving prediction problems of interest . A prediction task ( x , y ) = ( [ x1 , . . . , xn ] , [ y1 , . . . , ym ] ) is defined by its set of observed variables { xi } ni=1 ⊆ V and its set of target variables { yj } mj=1 ⊆ V whose values are unknown . The goal is to find a prediction function Ω that can be applied across any prediction task of interest , so that it can learn to exploit regularities across such problems . Let zi and zj be the variable embeddings corresponding to xi and yj , respectively . Then , this universal prediction model is of the form E [ yj | x ] = Ω ( x , { zi } ni=1 , zj ) . ( 1 ) Importantly , for any two tasks ( xt , yt ) , ( xt′ , yt′ ) , their prediction functions ( Eq . 1 ) differ only in their z ’ s , which enforces the constraint that functionality is otherwise completely shared across the models . One can view Ω as a traveling observer , who visits several locations in the C-dimensional variable space , takes measurements at those locations , and uses this information to make predictions of values at other locations . To make Ω concrete , it must be a function that can be applied to any number of variables , can fit any set of prediction problems , and is invariant to variable ordering , since we can not in general assume that a meaningful order exists . These requirements lead to the following decomposition : E [ yj | x ] = Ω ( x , { zi } ni=1 , zj ) = g ( n∑ i=1 f ( xi , zi ) , zj ) , ( 2 ) where f and g are functions called the encoder and decoder , with trainable parameters θf and θg , respectively . The variable embeddings z tell f and g which variables they are observing , and these z can be learned by gradient descent alongside θf and θg . A depiction of the model is shown in Figure 1 . For some integer M , f : RC+1 → RM and g : RM+C → R. In principle , f and g could be any sufficiently expressive functions of this form . A natural choice is to implement them as neural networks . They are called the encoder and decoder because they map variables to and from a latent space of sizeM . This model can then be trained end-to-end with gradient descent . A batch for gradient descent is constructed by sampling a prediction problem , e.g. , a task , from the distribution of problems of interest , and then sampling a batch of data from the data set for that problem . Notice that , in addition to supervised training , in this framework it is natural to autoencode , i.e. , predict input variables , and subsample inputs to simulate multiple tasks drawn from the same universe . The question remains : How can f and g be designed so that they can sufficiently capture a broad range of prediction behavior , and be effectively conditioned by variable embeddings ? The next section introduces an experimental architecture that satisfies these requirements .

Summary: This paper presents the traveling observer model (TOM), a general framework to learn multiple heterogenous supervized (input,output) tasks, which are indexed by a continuous "variable embedding" that is automatically learned by the system. The authors show on simple problems that the learned task embeddings can recover an intuitive organization of the problems' variables in space or time. They also show that the model simultaneously trained on 121 seemingly unrelated classification tasks can outperform state-of-the art supervized methods fine-tuned on single tasks.