The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings

1 INTRODUCTION . Natural organisms benefit from the fact that their sensory inputs and action outputs are all organized in the same space , that is , the physical universe . This consistency makes it easy to apply the same predictive functions across diverse settings . Deep multi-task learning ( Deep MTL ) has shown a similar ability to adapt knowledge across tasks whose observed variables are embedded in a shared space . Examples include vision , where the input for all tasks ( photograph , drawing , or otherwise ) is pixels arranged in a 2D plane ( Zhang et al. , 2014 ; Misra et al. , 2016 ; Rebuffi et al. , 2017 ) ; natural language ( Collobert & Weston , 2008 ; Luong et al. , 2016 ; Hashimoto et al. , 2017 ) , speech processing ( Seltzer & Droppo , 2013 ; Huang et al. , 2015 ) , and genomics ( Alipanahi et al. , 2015 ) , which exploit the 1D structure of text , waveforms , and nucleotide sequences ; and video game-playing ( Jaderberg et al. , 2017 ; Teh et al. , 2017 ) , where interactions are organized across space and time . Yet , many real-world prediction tasks have no such spatial organization ; their input and output variables are simply labeled values , e.g. , the height of a tree , the cost of a haircut , or the score on a standardized test . To make matters worse , these sets of variables are often disjoint across a set of tasks . These challenges have led the MTL community to avoid such tasks , despite the fact that general knowledge about how to make good predictions can arise from solving seemingly “ unrelated ” tasks ( Mahmud & Ray , 2008 ; Mahmud , 2009 ; Meyerson & Miikkulainen , 2019 ) . This paper proposes a solution : Learn all variable locations in a shared space , while simultaneously training the prediction model itself ( Figure 1 ) . To illustrate this idea , Figure 1a gives an example of four tasks whose variable values are measured at different locations in the same underlying 2D embedding space . The shape of each marker ( i.e. , ◦ , ,4 , ? ) denotes the task to which that variable belongs ; white markers denote input variable , black markers denote output variables , and the background coloring indicates the variable values in the entire embedding space when the current sample is drawn . As a concrete example , the color could indicate the air temperature at each point in a geographical region at a given moment in time , and each marker the location of a temperature sensor ( however , note that the embedding space is generally more abstract ) . Figure 1b-c shows a model that can be applied to any task in this universe , using the ◦ task as an example : ( b ) The function f encodes the value of each observed variable xi given its 2D location zi ∈ R2 , and these encodings are aggregated by elementwise addition ⊕ ; ( c ) The function g decodes the aggregated encoding to a prediction for yj at its location zj . Such a predictor can be viewed as a traveling observer model ( TOM ) : It traverses the space of variables , taking a measurement at the location of each input . Given these observations , the model can make a prediction for the value at the location of an output . In general , the embedded locations z are not known a priori ( i.e. , when input and output variables do not have obvious physical locations ) , but they can be learned alongside f and g by gradient descent . The input and output spaces of a prediction problem can be standardized so that the measured value of each input and output variable is a scalar . The prediction model can then be completely agnostic about the particular task for which it is making a prediction . By learning variable embeddings ( VEs ) , i.e. , the z ’ s , the model can capture variable relationships explicitly and supports joint training of a single architecture across seemingly unrelated tasks with disjoint input and output spaces . TOM thus establishes a new lower bound on the commonalities shared across real-world machine learning problems : They are all drawn from the same space of variables that humans can and do measure . This paper develops a first implementation of TOM , using an encoder-decoder architecture , with variable embeddings incorporated using FiLM ( Perez et al. , 2018 ) . In the experiments , the implementation is shown to ( 1 ) recover the intuitive locations of variables in space and time , ( 2 ) exploit regularities across related datasets with disjoint input and output spaces , and ( 3 ) exploit regularities across seemingly unrelated tasks to outperform single-tasks models tuned to each tasks , as well as current Deep MTL alternatives . The results confirm that TOM is a promising framework for representing and exploiting the underlying processes of seemingly unrelated tasks . 2 BACKGROUND : MULTI-TASK ENCODER-DECODER DECOMPOSITIONS . This section reviews Deep MTL methods from the perspective of decomposition into encoders and decoders ( Table 1 ) . In MTL , there are T tasks { ( xt , yt ) } Tt=1 that can , in general , be drawn from different domains and have varying input and output dimensionality . The tth task has nt input variables [ xt1 , . . . , xtnt ] = xt ∈ Rnt and mt output variables [ yt1 , . . . , ytmt ] = yt ∈ Rmt . Two tasks ( xt , yt ) and ( xt′ , yt′ ) are disjoint if their input and output variables are non-overlapping , i.e. , ( { xti } nti=1∪ { ytj } mt j=1 ) ∩ ( { xt′i } nt′i=1∪ { yt′j } mt′ j=1 ) = ∅ . The goal is to exploit regularities across task models xt 7→ ŷt by jointly training them with overlapping parameters . The standard intra-domain approach is for all task models to share their encoder f , and each to have its own task-specific decoder gt ( Table 1a ) . This setup was used in the original introduction of MTL ( Caruana , 1998 ) , has been broadly explored in the linear regime ( Argyriou et al. , 2008 ; Kang et al. , 2011 ; Kumar & Daumé , 2012 ) , and is the most common approach in Deep MTL ( Huang et al. , 2013 ; Zhang et al. , 2014 ; Dong et al. , 2015 ; Liu et al. , 2015 ; Ranjan et al. , 2016 ; Jaderberg et al. , 2017 ) . The main limitation of this approach is that it is limited to sets of tasks that are all drawn from the same domain . It also has the risk of the separate decoders doing so much of the learning that there is not much left to be shared , which is why the decoders are usually single affine layers . To address the issue of limited sharing , the task embeddings approach trains a single encoder f and single decoder g , with all task-specific parameters learned in embedding vectors zt that semantically characterize each task , and which are fed into the model as additional input ( Yang & Hospedales , 2014 ; Bilen & Vedaldi , 2017 ; Zintgraf et al. , 2019 ) ( Table 1b ) . Such methods require that all tasks have the same input and output space , but are flexible in how the embeddings can be used to adapt the model to each task . As a result , they can learn tighter connections between tasks than separate decoders , and these relationships can be analyzed by looking at the learned embeddings . To exploit regularities across tasks from diverse and disjoint domains , cross-domain methods have been introduced . Existing methods address the challenge of disjoint output and input spaces by using separate decoders and encoders for each domain ( Table 1c ) , and thus they require some other method of sharing model parameters across tasks , such as sharing some of their layers ( Kaiser et al. , 2017 ; Meyerson & Miikkulainen , 2018 ) or drawing their parameters from a shared pool ( Meyerson & Miikkulainen , 2019 ) . For many datasets , the separate encoder and decoder absorbs too much functionality to share optimally , and their complexity makes it difficult to analyze the relationships between tasks . Earlier work prior to deep learning showed that , from an algorithmic learning theory perspective , sharing knowledge across tasks should always be useful ( Mahmud & Ray , 2008 ; Mahmud , 2009 ) , but the accompanying experiments were limited to learning biases in a decision tree generation process , i.e. , the learned models themselves were not shared across tasks . TOM extends the notion of task embeddings to variable embeddings in order to apply the idea in the cross-domain setting ( Table 1d ) . The method is described in the next section . 3 THE TRAVELING OBSERVER MODEL . Consider the set of all scalar random variables that could possibly be measured { v1 , v2 , ... } = V . Each vi ∈ V could be an input or output variable for some prediction task . To characterize each vi semantically , associate with it a vector zi ∈ RC that encodes the meaning of vi , e.g. , “ height of left ear of human adult in inches ” , “ answer to survey question 9 on a scale of 1 to 5 ” , “ severity of heart disease ” , “ brightness of top-left pixel of photograph ” , etc . This vector zi is called the variable embedding ( VE ) of vi . Variable embeddings could be handcoded , e.g. , based on some featurization of the space of variables , but such a handcoding is usually unavailable , and would likely miss some of the underlying semantic regularities across variables . An alternative approach is to learn variable embeddings based on their utility in solving prediction problems of interest . A prediction task ( x , y ) = ( [ x1 , . . . , xn ] , [ y1 , . . . , ym ] ) is defined by its set of observed variables { xi } ni=1 ⊆ V and its set of target variables { yj } mj=1 ⊆ V whose values are unknown . The goal is to find a prediction function Ω that can be applied across any prediction task of interest , so that it can learn to exploit regularities across such problems . Let zi and zj be the variable embeddings corresponding to xi and yj , respectively . Then , this universal prediction model is of the form E [ yj | x ] = Ω ( x , { zi } ni=1 , zj ) . ( 1 ) Importantly , for any two tasks ( xt , yt ) , ( xt′ , yt′ ) , their prediction functions ( Eq . 1 ) differ only in their z ’ s , which enforces the constraint that functionality is otherwise completely shared across the models . One can view Ω as a traveling observer , who visits several locations in the C-dimensional variable space , takes measurements at those locations , and uses this information to make predictions of values at other locations . To make Ω concrete , it must be a function that can be applied to any number of variables , can fit any set of prediction problems , and is invariant to variable ordering , since we can not in general assume that a meaningful order exists . These requirements lead to the following decomposition : E [ yj | x ] = Ω ( x , { zi } ni=1 , zj ) = g ( n∑ i=1 f ( xi , zi ) , zj ) , ( 2 ) where f and g are functions called the encoder and decoder , with trainable parameters θf and θg , respectively . The variable embeddings z tell f and g which variables they are observing , and these z can be learned by gradient descent alongside θf and θg . A depiction of the model is shown in Figure 1 . For some integer M , f : RC+1 → RM and g : RM+C → R. In principle , f and g could be any sufficiently expressive functions of this form . A natural choice is to implement them as neural networks . They are called the encoder and decoder because they map variables to and from a latent space of sizeM . This model can then be trained end-to-end with gradient descent . A batch for gradient descent is constructed by sampling a prediction problem , e.g. , a task , from the distribution of problems of interest , and then sampling a batch of data from the data set for that problem . Notice that , in addition to supervised training , in this framework it is natural to autoencode , i.e. , predict input variables , and subsample inputs to simulate multiple tasks drawn from the same universe . The question remains : How can f and g be designed so that they can sufficiently capture a broad range of prediction behavior , and be effectively conditioned by variable embeddings ? The next section introduces an experimental architecture that satisfies these requirements .

Summary: .** Authors present a methodology for performing multi-task learning from data with disjoint and heterogeneous input domains. Particularly, they introduce an embedding of the inputs, in order to project each pair of input-output observations in a common continuous manifold where the exploration is significantly easier. Results show that the approach is valid with both synthetic and real-world data and they also demonstrate that the model is flexible when increasing/decreasing the dimensionality of the latent manifold.