Contextual HyperNetworks  for Novel Feature Adaptation

1 INTRODUCTION . In many deep learning application domains , it is common to see the set of predictions made by a model grow over time : a new item may be introduced into a recommender system , a new question may be added to a survey , or a new disease may require diagnosis . In such settings , it is valuable to be able to accurately predict the values that this feature takes within data points for which it is unobserved : for example , predicting whether a user will enjoy a new movie in a recommender system , or predicting how a user will answer a new question in a questionnaire . On the introduction of a new feature , there may be few or no labelled data points containing observed values for it ; a newly added movie may have received very few or even no ratings . The typically poor performance of machine learning models in this low-data regime is often referred to as the cold-start problem ( Schein et al. , 2002 ; Lika et al. , 2014 ; Lam et al. , 2008 ) , which is prevalent not only in recommender systems but also in applications where high quality data is sparse . This presents a key challenge : the adaptation of a deep learning model to accurately predict the new feature values in the low data regime . On one hand , it is often required to deploy the model in applications immediately upon the arrival of new features , so it is impractical for the adaptation to wait until much more data has been acquired . On the other hand , simply retraining the model every time a new feature is introduced is computationally costly , and may fall victim to severe over-fitting if there are only a small number of observations available for the new feature . Few-shot learning ( Snell et al. , 2017 ; Requeima et al. , 2019 ; Vinyals et al. , 2016 ; Gordon et al. , 2018 ) has seen great successes in recent years , particularly in image classification tasks ; however , these approaches typically treat all tasks as independent of one another . We wish to extend these ideas to the challenge of extending deep learning models to new output features , using a method which captures how a new feature relates to the existing features in the model . Furthermore , we seek a method that is computationally efficient , ideally requiring no fine-tuning of the model , and that is resistant to over-fitting in the few-shot regime . To address these needs simultaneously , our contributions are as follows : • We propose an auxiliary neural network , called Contextual HyperNet ( CHN ) , that can be used to initialize the model parameters associated with a new feature ( see Section 2 ) . CHNs are conditioned on both a context set made up of observations for the new feature , and any associated content information or metadata . CHNs amortize the process of performing gradient descent on the new parameters by mapping the newly observed data directly into high-performing new parameter values , with no additional fine-tuning of the model being required . This makes CHNs highly computationally efficient and scalable to large datasets . • We use a CHN to augment a partial variational autoencoder ( P-VAE ) and evaluate the system ’ s performance across a range of applications ( see Section 4 ) . While CHNs are applicable to a wide range of deep learning models , in this work we choose a P-VAE as the evaluation framework . The result is a flexible deep learning model able to rapidly adapt to new features , even when the data is sparsely-observed , e.g . in recommender systems . We show that this model outperforms a range of baselines in both predictive accuracy and speed across recommender system , e-learning and healthcare tasks . 2 MODEL . 2.1 PROBLEM SETTING . Our goal is to enable fast adaptation of a machine learning model when new output features are added to augment the originally observed data . Specifically , we consider the original observations as a set of vector-valued data points D = { x ( i ) } mi=1 , where each of the feature values in a given data point may be missing . We denote xj as the jth feature of a data point x ∈ D and group the observed and unobserved features within a data point as x = [ xO , xU ] . In many scenarios , such as recommender systems , a machine learning model p ( xU |xO ) aims then at predicting the unobserved features xU given observed ones xO . Now suppose a new output feature xn becomes available , so that each data vector x ∈ D is augmented to become x̃ = [ x ; xn ] . This happens when e.g . a new item is added to a recommender system , or a new type of diagnostic test is added in a medical application . We note that not every data point x ∈ D receives an observed value for the new feature : a newly added movie may have received very few ratings , or a new diagnostic test may have yet to be performed on all of the patients . We refer to the set of data points where the new feature is observed as the context set for the new feature n , i.e . Cn = { x̃ = [ x ; xn ] | x ∈ D , xn is observed } . context set is shown in yellow in Figure 1 Its complement , the target set Tn , is the set of those data points for which there is no associated observation for the feature : Tn = { x̃ = [ x ; xn ] | x ∈ D , xn is unobserved } . One can also split the augmented data into observed and unobserved parts , i.e . x̃ = [ x̃O , x̃U ] . Using this notation , it is clear that x̃O = [ xO ; xn ] , x̃U = xU for x̃ ∈ Cn , and x̃O = xO , x̃U = [ xU ; xn ] for x̃ ∈ Tn . In addition , we may also have access to some metadataMn describing the new feature . This could be categorical data such as the category of a product in a recommender system or the topic of a question in an e-learning system , or some richer data format such as images or text . We wish to adapt the machine learning model pθ0 ( xU |xO ) to pθ ( x̃U |xO ) so that it is able to accurately predict the value of the unobserved new features for data points x̃ ∈ Tn . A naive strategy would ignore the previous model pθ0 ( xU |xO ) and instead seek the maximum likelihood estimates ( MLE ) of the parameters for the new model pθ ( x̃U |xO ) . This is typically done by training the new model on the context set , by temporarily moving the observed new features xn to the prediction targets : θ̂ = argmax θ [ ∑ x̃∈Cn log pθ ( xn , xU |xO ) ] . However , in deep neural networks , the number of model parameters θ may be extremely large , so that maximising this log-likelihood is very expensive , particularly if new features are being introduced on a regular basis . Furthermore , optimising θ for one particular feature may lead to poor performance for another , as is the case in catastrophic forgetting ( Kirkpatrick et al. , 2017 ) in continual learning tasks . In order to address both of these concerns , we divide the model parameters into parameters θ0 inherent from the old model , and feature-specific parameters θn associated solely with the new feature . In other words , we use pθ0 ( xU |xO ) as a base model and pose a factorisation assumption on the augmented model as pθ ( x̃U |xO ) = pθ0 ( xU |xO ) pθ0 ( xn|xO ; θn ) , which together yield a predictive model for the new feature . We then hold θ0 fixed and only seek MLEs for θn . While this greatly reduces the dimensionality of the parameter space over which we optimize for a new feature , and decouples the optimization of parameters for one new feature from another , several issues still exist . This factorization still requires a gradient descent procedure , which can be computationally costly and risks severe overfitting when there is little data for the new feature . Furthermore , it is not immediately clear how to make the estimation of θn depend on the feature metadataMn . To address these problems , we introduce a Contextual HyperNetwork ( CHN ) Hψ ( Cn , Mn ) , an auxilliary neural network that amortizes the process of estimating θn . The goal is that when a new feature xn∗ is added at test time , the CHN will directly generate “ good ” parameters θ̃n∗ = Hψ ( Cn∗ , Mn∗ ) such that the new predictive model pθ0 ( xn∗ |xO ; θn∗ = θ̃n∗ ) can predict the values of the new feature accurately . 2.2 CONTEXTUAL HYPERNETWORKS . CHNs aim to map the context set Cn and metadataMn into an estimate of the new model parameters θ̃n . Since the size of Cn is variable for each feature , CHNs require an architecture that is able to adapt to a varying input dimension . This challenge is addressed through the use of a PointNet-style set encoder ( Qi et al. , 2017 ; Zaheer et al. , 2017 ) . For each context point x̃ ( i ) ∈ Cn , we concatenate the new feature x ( i ) n with a fixed-length encoding z ( i ) ( see below ) of the other observed features x ( i ) O within the data point . Each of these concatenated vectors [ z ( i ) , x ( i ) n ] is then input to a shared neural network f ( · ) , and the outputs f ( [ z ( i ) , x ( i ) n ] ) are aggregated by a permutation-invariant function such as summation in order to produce a single , fixed-length vector . This output is passed through a second neural network g ( · ) to produce a fixed-length encoding of cn we term a “ context vector '' . This architecture is displayed in Figure 2a . The fixed length encoding z ( i ) of the observed features x ( i ) O for each context data point x̃ ( i ) ∈ Cn is obtained by inputting the observed features to the base model pθ0 ( xU |xO ) and taking some intermediate representation from within the model : in an autoencoder model , this could be the encoded vector representing the data point at the information bottleneck , while in a feed-forward model it could be the output of an intermediate layer . By encoding the existing features in this way , we hope to enable the CHN to interpret the observed values for the new features in the context of the base model ’ s representation of remainder of the data point . Additionally , any feature metadataMn is passed through a neural network h ( · ) to produce a fixed length metadata embedding vector mn . In the case of image or text metadata , specialized architectures such as convolutional or sequence models can be used here . The concatenated vector [ cn ; mn ] is then input into a final feed-forward neural network which outputs the new feature-specific parameters θ̃n . CHNs can be applied to any predictive model with dynamically added output features : Figure 2b shows the application of CHN to an predictive model . As an example , we illustrate how CHN is used with an autoencoder-style model in Figure 3 . Since it is possible to parallelize the encoding of the context set Cn both across each data point x ( k ) and across the observed features within each data point xi , the computational costs for parameter prediction with CHNs are able to scale efficiently with both the context set size and the number of observed values within each data point in the context set . When combined with the lack of any costly iterative gradientdescent procedure , this makes CHNs an extremely efficient choice for parameter initialization .

Summary: This submission focuses on the cold start problem of new entities (new items in a recommender system, new treatments in a medical application, etc.). It combines the strengths of the *relations* between a new entity and the existing entities, and the *content* features of the new entity, by fusing the two kinds of information into a neural network that outputs the estimated representation of the new entity. The proposed method outperforms several intuitive naïve strategies as well as MAML.