Contextual HyperNetworks  for Novel Feature Adaptation
Authors: Angus Lamb
Evgeny Saveliev
Yingzhen Li
Sebastian Tschiatschek
Camilla Longden
Simon Woodhead
José Miguel Hernández-Lobato
Richard E Turner
Pashmina Cameron
Cheng Zhang
Venue:  (2021)
Keywords: Meta learning
few-shot learning
continual learning
recommender systems
deep learning

Abstract: While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new features remains a research challenge. This issue is particularly severe in online learning settings, where new features are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), which predicts the network weights associated with new features by incorporating information from both existing data as well as the few observations for the new feature and any associated feature metadata. At prediction time, the CHN requires only a single forward pass through a small neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches. In order to showcase the performance of CHNs, in this work we use a CHN to augment a partial variational autoencoder (P-VAE), a flexible deep generative model which can impute the values of missing features in sparsely-observed data. We show that this system obtains significantly improved performance for novel feature adaptation over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.

Under review as a conference paper at ICLR 2021
CONTEXTUAL HYPERNEtWORKS FOR NOVEL FEATURE ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
While deep learning has obtained sfate-of-the-an results in many applications, the adaptation of neural network architectures incorporate new output features remains challenge, as a neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe online learning settings, where new output features, such as items in a recommender system, are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), an auxiliary model which generates parameters for extending the base model to new feature, by utilizing both existing data as well as any observations and/or metadata associated with the new feature. At prediction time, the CHN requires only single forward pass through a neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches. To assess the performance of CHNs, we use a CHN to augment a partial variational autoencoder (P- VAE),,, deep generative model which can impute the values of missing features in sparsely-observe data. We show that this system obtains improved few-shot learning performance for novel features over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.
INTRODUCTION
In many deep learning application domains, it is common to see the set of predictions made by model grow over time: a new item may be introduced into a recommender system, new question may be added to a survey, or a new disease may require diagnosis. In such settings, it is valuable to be able to accurately predict the values that this feature takes within data points for which it is unobserved: for example, predicting whether user will enjoy a new movie in recommender system, or predicting how a user will answer a new question in a questionnaire.
On the introduction of a new feature, there may be few no labelled data points containing observed values for it; a newly added movie may have received very few or even no ratings. The typically poor performance of machine learning models in this low-data regime is often referred to the cold-start problem (Schein et al., 2002; Lika et al., 2014; Lam et al., 2008), which is prevalent not only in recommender systems but also in applications where high quality data is sparse. This presents key challenge: the adaptation of a deep learning model to accurately predict the new feature values in the low data regime. On one hand, is often required to deploy the model in applications immediately upon the arrival of new features, so it is impractical for the adaptation to wait until much more data has been acquired. On the other hand, simply retraining the model every time a new feature is introduced is computationally costly, and may fall victim to severe over-fitting if there are only small number of observations available for the new feature.
Few-shot learning (Snell et al., 2017; Requeima et al., 2019; Vinyals et al., 2016; Gordon et al., 2018) 1)) has seen great successes recent years, particularly in image classification tasks; however, these approaches typically treat all tasks as independent of one another. We wish to extend these ideas to the challenge of extending deep learning models to new output features, using a method which captures how new feature relates the existing features in the model. Furthermore, seek method that is computationally efficient, ideally requiring no fine-tuning of the model, and that is resistant to over-fitting in the few-shot regime.

---

Under review as a conference paper at ICLR 2021
To address these needs simultaneously, contributions are follows:
We propose an auxiliary neural network, called Contextual HyperNet (CHN), that can be used to initialize the model parameters associated with new feature (see Section 2).
CHNs are conditioned on both a context set made up of observations for the new feature, and any associated content information or meladata. CHNs amortize the process of performing gradient descent on the new parameters by mapping the newly observed data directly into high-performing new parameter values, with additional fine-tuning of the model being required. This makes CHNs highly computationally efficient and scalable to large datasets.
We use a CHN to augment a partial variational autoencoder (P-VAE) and evaluate the system's 's performance across range of applications (see Section 4).
While CHNs are applicable to a wide range of deep learning models, in this work we choose P- P-VAE the evaluation framework. The result is flexible deep learning model able to rapidly adapt to new features, even when the data is sparsely-observer systems. We show that this model outperforms range of baselines in both predictive accuracy and speed across recommender system, e-learning and healthcare tasks.
e.g. in recommender
2 MODEL
2.1 PROBLEM SETTING
Our goal is to enable fast adaptation of machine learning model when new output features are added to augment the originally observed data. Specifically, we consider the original observations as set of vector-valued data points D {x(") 1m J' where each of the feature values in a given data point may be missing. We denote z, the jth feature of a data point x D and group the observed and [xo. xy]. many scenarios, such as recommender systems, machine learning model p(xy (x)) aims then at predicting the unobserved features X given observed ones XO-
unobserved features within a data point x
Observed Features
Z+
Now suppose new output feature Iia becomes available, so that each data vector x D is augmented to become This happens when e.g. new item is added
x [x; ax] to a recommender system, or a new type of diagnostic test is added in medical application. We note that not every data point x E 'D receives an observed value for the new feature: newly added movie may have received very few ratings, or a new diagnostic test may have yet to be performed on all the patients. We refer the set of data points where the new feature is observed as the context set for the new feature n, i.e.
Dainsooins
Feelures
Observed value for datapoint in context set C,
Observed value datapoint in target set
Value unobserved in dataset
Cn = (x = [x; xn]]
X E: D, xn is observed].
Data used when adapting to new feature Fn. The rows which contain yellow blocks are C, and other rows are in Ta
Figure 1:
context set is shown in yellow in Figure Its complement, the target set T,, is the set of those data points for which there is no associated observation for the feature:
Tn { [x:
=n]
D, Nr is unobserved).
One can also split the augmented data into observed and unobserved parts, i.e. x [&o, Xt.. Using this notation, it is clear that Xo = [xo; Xy = Xy for x E C., and Xo = X0, Xy = [xt; xn) for Tn. In addition, we may also have access some metadata M, describing the new feature. This could be categorical data such as the category of product in a recommender system or the topic of a question in an e-Icarning system, or some richer data format such as images or text.
We wish to adapt the machine learning model pe. (xo) to po / xo) so that it is able to accurately predict the value of the unobserved new features for data points E T, A naive strategy would ignore the previous model Pa, (Xr xx) and instead seek the maximum likelihood estimates (MLE) of the parameters for the new model pa [xo). This is typically done by training the new model on the context set, by temporarily moving the observed new features H, to the prediction targets:
① arg max
Σ LECC..
log pe (In, xu xo)
2

---

Under review as a conference paper at ICLR 2021
C
xto
JC1
Xo
x!a)
Model &
xtx)
2
6.
CHN
MAIN MODEL
Mu
(a) Contextual HyperNet Architecture
(b) CHN applied predictive model
Figure 2: (a) Contextual HyperNetwork architecture. z(9) is a fixed-length internal representation of the previously observed values in the data point x(0) taken from the base model. (b) Complete architecture when a Contextual HyperNetwork is used to augment a predictive model peo (xu (xo) to be able to make predictions for new feature n. by initialising new parameters 0...
However, in deep neural networks, the number of model parameters may be extremely large, so that maximising this log-likelihood is very expensive, particularly if new features are being introduced on a regular basis. Furthermore, optimising for one particular feature may lead to poor performance for another, as is the case in catastrophic forgetting (Kirkpatrick et al., 2017) in continual learning tasks. In order address both of these concerns, we divide the model parameters into parameters 0o inherent from the old model, and feature-speciff parameters O, associated solely with the new feature. In other words, we use Peo (Xu (xo) as a base model and pose factorisation assumption on the augmented model predictive model for the new feature. We then hold O, fixed and only seek MLEs for On. While this feature,
pe (Xx xo) pe, (xx )e, (z,, xx: 0n), which together yield
greatly reduces the dimensionality of the parameter space over which optimize for and decouples the optimization of parameters for one new feature from another, several issues still exist. This factorization still requires gradient descent procedure, which can be computationally costly and risks severe overfitting when there is little data for the new feature. Furthermore, it is not immediately clear how make the estimation of O,, depend the feature metadata M,, To address these problems, we introduce a Contextual HyperNetwork (CHN) H (C... M,), an auxilliary neural network that amortizes the process of estimating 0,.. The goal is that when new feature T,," is added test time, the CHN will directly generate "good" parameters 0,. Ha (Cn-, Mn-) such that the new predictive model Po. (Z (xo; 0,, = 0a-) can predict the values of the new feature accurately.
2.2 CONTEXTUAL HYPERNETWORKS
CHNs aim to map the context set C., and metadata M,, into estimate of the new model parameters θ,. Since the size of Cn is variable for each feature, CHNs require architecture that is able to adapt to a varying input dimension. This challenge is addressed through the use of a PointNet-style set encoder (Qi et al., 2017; Zaheer et al., 2017). For each context point (+) Cn. we concatenate the new feature zn with fixed-length encoding (f) (see below) of the other observed features (6) within the data point. Each of these concatenated vectors (z) .xn is then input to shared neural network f()). and the outputs / [[[x(), .r((l)) are aggregated by permulation-invaaiiat function such as summation in order to produce a single. fixed-length vector. This output is passed through second neural network (-)) to produce fixed-length encoding of C, we term "context vector". This architecture is displayed in Figure 2a.
The fixed length encoding (+) of the observed features x for each context data point (9) E C. obtained by inputting the observed features to the base model pa, (xu|xo) and taking some intermediate representation from within the model: in autocncoder model, this could be the encoded vector representing the data point at the information bottleneck, while in a forw-forward model it could be the output of intermediate layer. By encoding the existing features in this way, we hope to enable the CHN to interpret the observed values for the new features in the context of the base model's representation of remainder of the data point.
(6)
Additionally, any feature metadata M, is passed through a neural network h(-) to produce a fixed length metadata embedding vector m.. In the case of image text metadata, specialized architectures
3

---

Since it is possible parallelize the encoding of the context set Cn both across each data point Under review as a conference paper at ICLR 2021
such convolutional sequence models can be used here. The concatenated vector [c,, m., is then input into a final feed-forward neural network which outputs the new feature-specific parameters
On. CHNs can be applied to any predictive model with dynamically added output features: Figure 2b shows the application of CHN to an predictive model. As example, illustrate how CHN is used with an autoencoder-style model Figure 3.
Thoonder Heads
x()) and across the observed features within each data point 24, the computational costs for parameter prediction with CHNs are able to scale efficiently with both the context size and the number of observed values within each data point in the context set. When combined with the lack of any costly iterative gradient- descent procedure, this makes CHNs an ex- tremely efficient choice for parameter initial- ization.
Fy
Set Enooder
Encoder
Sitarod Decoder
6..
Fa-i
Contextual HyperNet
M_
Figure 3: Contextual HyperNetwork applied to P-VAE. The CHN generates parameters 0, for new decoder head.
2.3 TRAINING CHNs WITH LEARNNNG
We adopt meta-learning approach to training the CHN, treating the prediction of the values of each new feature as an individual task, with the aim of producing model that can "learn how to learn" from Cn and M,,- First, base model Pa, (xul xo) is trained on the data observed before the adaptation stages; this model is then frozen during CHN training. To implement the training strategy, in the experiments we divide the dataset into three disjoint sets of features (see Figure 4): 'training" set for base model training the first stage, a 'meta-training' set for CHN meta-learning in the second stage, and meta-test set for CHN evaluation in the third stage.
Meta- Eeita-Trainig of the CHN
During meta-training, the parameters 0g of the base model are frozen, and learn the parameters vo of the CHN. We iterate the following training steps in mini-batches of features B sampled from the meta-training set for every step:
For each feature n B, sample ki, data points which this feature is observed form the context set Cn, and reveal the associated feature values to the model. In experiments we sample kn ~ Uniform (0, ..., 32) to ensure that a single CHN can perform well across a range of context sizes.
2. For each feature
B, compute feature-specift parameter predictions using the CHN,
0.. = Ha (Cn,MM))
3. For each feature B, estimate the log- ligeliiel of the CHN parameters &b given the ground truths for the hidden values of the feature n in the data points in its target set T.. using the augmented model Pa, (I,, (x. 0.):
((bb) = ΣNEB |T
Σ Σ
log pe, (x 1x00 0.).
REB (f)x()TT.)
4. Update the CHN parameters by taking a gradient ascent step in zb for ((xb).
Note that the log- ligeliikeioodd is only computed for the hidden values of the new feature in the target T,, and not for the observed values in C, this which generalize well, rather than overfitting to the context set. This approach is consistent with many meta-learning methods such as MAMI. (Finn et al., 2017), where the meta-learning model updated based on its performance on "test set" of previously unseen examples for each new task.
ensure that the CHN produces parameters
Evaluating the CHN At evaluation time, the parameters of both the base model and the CHN are now frozen. A fixed context set and metadata are provided for each feature in the meta-test set, and
4

---

Under review as a conference paper at ICLR 2021
these are used to initialize feature- seeturre-peeificc parameters for the meta-test features using the trained CHN. These parameters are then used to make predictions for all of the target set values for the new features, from which evaluation metries are computed.
3 RELATED WORK
Training Set
Meta-Train Set
CHNs aim to solve the problem of adapting to new fea- ture with very few available observations, and thus relate to few-shot learning and related fields such as meta-learning and continual learning. From a technical point of view,
Datspoints
auxiliary neural network to amortize the learn- ing of parameters associated with the new feature, which falls under the domain of hypernetworks. Furthermore, in the context of recommender systems, a number of re- lated methods have been proposed address the cold- start problem. We thus discuss related work in these three areas,
Features
Observed value; dalapoint in context set C..
Observed value; dalapoint larget set Ta
Value hidden from model; prediction target
Few-Shot Learning Few-shot learning is the problem of designing machine learning models that can adapt to new prediction tasks given small number of training examples. A popular approach to this problem is gradient- based meta-leaming, such as MAML (Finn al., 2017) and Reptile (Nichol & Schulman, 2018), which seek a parameter initialisation 0 that can rapidly adapt to tasks drawn from a task distribution p( methods do not directly condition the parameter initialisation for new task on any associated data or metadata, instead relying on computationally expensive and lead to severe overfitting when little data is available. Another line of methods seek to adapt classifier to task based on a context set of class examples. For instance, by embedding class examples to provide a nearest neighbours classifier (Snell et al., 2017). learning an attention mechanism between class examples and a new example (Vinyals et al., 2016), or modulating activation functions (Munkhdalai et al., 2017) within a feature extractor conditioned on the context set. Conditional Neural Adaptive Processes (Requeima al., 2019), which based on Conditional Neural Processes (Garnelo et al., 2018), adapt both classifier and feature extractor parameters based on the context for task. Similarly, Gidaris & Komodakis (2018) generate classifier weights for a new image class based on features extracted using the base model. However, in all cases, each task or image class is treated as independent from all others, whereas CHNs explicitly utilize all previously-observed features in the base model when adapting to new feature.
unobserved in dataset; ignore
Figure 4: Data splits when meta-training a CHN. Meta-testing proceeds analo- gously, using features from additional meta-test set of features, but using a con- stant set of observed values for each fea- ture on every meta-test set evaluation.
which can be both
closely related field is continual learning (Kirkpatrick et al., 2017; Nguyen et al., 2017), where model seeks to adapt to new tasks or shifting data distribution while avoiding catastrophic forgetting of old tasks. Continual learning does not necessarily address the few-shot scenario and is commonly applied in classification settings where the classifier/heads can be either shared or independent. CHNs can be seen as means of addressing continual learning in the few-shot learning regime, by generating parameters for new feature conditioned all of the features already learned by the model.
Hypernetworks Hypernetworks (Ha et al., 2016) are auxiliary neural networks which generate the parameters of a neural network. They were introduced in Ha et al. (2016) as a form of model compression, with the hypernetork taking as in input only structural information about the weight matrix they are currently predicting, such as its index in sequence of layers. By contrast, CHNs are explicitly conditioned on data relevant to the weights currently being predicted. Bertinetto et al. (2016) train hypernetwork to predict all of the parameters of a binary classifier for class of images, conditioned on a single exemplar image for the class. It is found that the output dimension of the hypernetwork grows extremely large for even small classifiers- in order to mitigate this, the authors propose a factorisation the parameters, whereas we instead choose to learn only a small number of Seature-specific parameters 0,.- task-conditioned hypernetorrks (von Oswald et al., 2019) provide an application of hypernetworks to multi-task continual learning, where weights for the entire neural network for different tasks predicted using hypernetwork, based learned task embedding.
5

---

Under review as a conference paper at ICLR 2021
This setting differs from our work the continual learning tasks are assumed to be independent, and the hypernetwork is not conditioned directly on data for the new task, instead requiring a gradient descent process to learn the associated task embedding.
Cold Starts in Recommender Systems Cold starts (Schein et al., 2002; Lika et al., 2014) occur when there is little no data associated with a novel item or user in recommender system. Collaborative filtering approaches to recommender systems have enjoyed great success for many years (Schafer et al., 2007; Sarwar et al., 2001; He et al., 2017), but can fail completely when there is very limited rating data associated with a new user or item (Lam et al., 2008). One potential solution cold starts is given by content-bass which use any available descriptive information about the new user or item. Hybrid approaches (Balabanović Shoham, 1997; Stern et al., 2009; Gomez-Uribe & Hunt, 2015) seek to marry these two approaches, making use of both collaborative and content-based methods. Meta-learning approaches also show promise for solving cold starts, including MAML-like approaches (Bharadhwaj. 2019) for initialising new items. Vartak et al. (2017) takes a similar approach to CHNs by adapting the weights of a classifier, but the adaption is based on user rather than item history, and relies on class-specific item embeddings and thus is not extensible to regression settings. In this work, when applied to recommender systems, CHNs combine the strengths of all of these approaches, using content information, ratings data and latent representations of the associated users generate accurate parameters for novel items. Recently, graph neural network (GNN)- based approaches (You al., 2020; Schlichtkrull et al., 2018; Wang et al., 2019; Hamilton al., 2017) have shown promise when applied to recommender systems. These methods formulate each item and each user as node, and incorporate new items by adding nodes to the graph, where the graph structure may be exploited for inductive reasoning about new features. In this work, we target more standard, vector-based neural networks which are currently deployed across a wide range of application domains where GNNs may not be suitable, but future work may explore the application of CHNs to GNNs and the relationships between the two.
methods (Pazzani & Billsus, 2007; Lops et al., 201 1),
EXPERIMENTS
We demonstrate the performance of the proposed CHN in three different real-world application scenarios, including recommender systems (Section 4.2), healthcare (Section 4.3) and e-learning (Section 4.4). Our method exhibits superior performance in terms of prediction accuracy across all these applications. We also demonstrate an advantage in terms of computational efficiency in a large scale real-life setting.
4.1 EXPERIMENT SETTINGS
In all our experiments, apply CHN to partial variational autoencoder (P- VAE) (Ma et al., 201 8b;a) as an exemplar model. This is a flexible autoencoder model that is able to accurately work with and impute missing values in data points, allowing to model sparsely-observed data such as that found in recommender systems. For each new feature n, we augment the P-VAE's decoder with new decoder head consisting of an additional column of decoder weights Wn, and an additional decoder bias term b, which extend the model's 's output to the new feature, so that 0, = {w,, ) Figure? illustrates how CHN used to extend P- VAE to make predictions for a new feature Fn.-
For all experiments, we train the CHN to output accurate feature parameters based on a range of context set sizes [0, .... 32) by randomly sampling on each occurrence of meta- vreta-training set feature. We then evaluate the performance of the CHN and baselines on the meta-test set features for a fixed range of context set sizes, ensuring that the same context sets are revealed to the CHN and each bascline. Further experimental results be found in Appendix C, and full details the model architectures and hyperparameters can be found in Appendix D. All results are averaged across random train/meta- teatttttrai/meaaaeee feature splits, and =1 error bars are plotted across these splits. Hyperparameters and model architectures were tuned on different data splits to those used in the final experiments.
We consider the following baselines for generating the new feature parameters 0,, = [w., b, All methods are applied to the same base trained P-VAE model to ensure fair comparison.

---

Under review as a conference paper at ICLR 2021
Rondom
Moan Head Params (Matching Motadatal
Mcan impubing
MANL
Muan Head Params
10-tuuree Neighbours
Frain from Hansoom 110 Epocha)
CnN
MovieLens-
Neuropathic Pain
E-Learning
0.70
0.75
1.5
Meto- Text RMSE
1.4
0.05
0.70
1.3
0.60
0.65
1.2
0.60
1.1
0.55
0.45
0.50
Context
Figure 5: Performances (+10) of CHN and benchmarks MovieL (left), Neuropathic Pain (middle) and E-learning (right) datasets for test features, with varying context size k. We report meta-test RMSE for MovieLens-IM (lower is better) and AUROC for the others (higher better). For the MAML baselines, the best-performing case is shown: fine-tuning epoch for MovieLens- 1M and 10 for others.
Random: Generate a random value for 0, for each new decoder head using Xavier initialisation for weights and 0 for biases.
Mean Imputing: Set weights and biases to always predict the mean of the observed values for the new feature in the context set, i.e. W, and o- (+. Eiec, _1(
Mean Head Parameters: Generate the new head parameters O,, the mean of all of the head parameters learned on the training set features.
Mean Head Parameters (Matching Metadata): As above, but averaging only parameters of heads whose associated feature has metadata categories matching those of the new feature.
k-Nearest Neighbour Head Parameters: Generate the new head parameters 0,, the mean of the head parameters of the k-nearest neighbour features in terms Euclidean distance, where column-wise mean imputing used to fill in unobserved values.
Train from Random: Initialize the new feature head parameters randomly, and then fine-tune these parameters on the data in the context set C, for a fixed number of epochs.
MAML: We meta-learn an initialisation of 0, using Model-Agnostic Meta Learning (Finn et al., 2017), where we treat the prediction each feature as separate task and fine-tune these parameters on the context set. In all experiments, we compare with the MAML baseline which has the best- performing number of fine-tuning epochs. For full details, see Appendix B.
4.2 RECOMMENDER SYSTEMS
In real-life recommender systems, new and items continuously added customers join and new products are launched. In deep learning based frameworks such Sedhain et al. (2015); Liang et al. (2018); Gong al. (2019): Ma al. (2018a), the deep neural networks are commonly used in a user-based manner. In this approach, each new user is treated as new data point, while each new item treated as new feature. To add new item, one must extend the network architecture to incorporate the new feature, and we propose CHNs an efficient way to predict the parameters associated with the new feature.
We evaluate the scenario above with MovieL-ens-1 M dataset (Harper & Konstan, 2015). The dataset consists of million ratings the range to given by 8094 users to 5660 movies, and is thus 2.2% observed. For each movie, have associated metadata M,, giving list of genres associated with the movie, such as Action or Comedy, which we encode in a binary format, alongside the year of release which normalize lie within [0, 1]. For each random data split, sampled 60% of movies as training data to train the base P. P.V model, used 30% as a meta-training set for CHN training and used the remaining 10% as meta-test set.
The plot in Figure (left) shows the performance our proposed CHN, comparing with all other baselines in terms of RMSE (lower is better). Our method shows advantage all considered
7

---

Under review as a conference paper at ICLR 2021
baselines other than MAML in the few-shot regime (k < 8), while achieving competitive performance with MAML without requiring costly fine-tuning.
4.3 HEALTHCARE
In healthcare applications, a new question is often added to an existing health-assessmmet question- naire, and in hospitals, new medical devices may be introduced to make physiological measurements. In this case it is desired for model to quickly adapt to the newly added feature for health assessment, even when relatively few tests have been administered and so data is scarce.
We assess the utility of CHNs in healtheare setting using synthetic data generated by the Neuropathic Pain Diagnosis Simulator (Tu et al., 2019). This simulator produces synthetic data using a generative model to simulate pathophysiologies, patterns and symptoms associated with different types of neuropathic pain. The data binary, where a represents the a diagnostic label that is not present in patient's 's record, and 1 indicates diagnostic label that is present. We simulated 1000 synthetic patients, and removed features with fewer than 50 positive diagnoses, resulting in 82 remaining features, with 17.3% of the values in the dataset being positive diagnoses. We used 50% of the features as training set; 30% of the features as the meta-test set and 20% of the features as the meta-test set.
The plot in Figure 5 (middle) shows the results in terms of AUROC (higher is better), as the dataset highly imbalanced. Our method consistently outperforms all baselines across all values of k while many methods including MAML suffer from severe overfitting when & is small. In contrast to the MovieLens- result, here the 10-ncarest neighbour approach does not seem to leverage more datapoints in the context set. This shows that our method is desirable in the cost-sensitive healthcare environment, even for highly imbalanced medical tests where results are largely negative.
4.4 E-LEARNING
We foresee CHNs being valuable in online education settings, potentially allowing teachers to quickly assess the diagnostic power of new question given a small number of answers, or to gauge whether new question's 's difficulty is appropriate for particular student.
We assess the performance of the CHN in an e-lcarning setting using a real-life dataset provided by the e-learning provider Eedi for the NeurIPS 2020 Education Challenge (Wang al., 2020). In particular, we use the dataset for the first tasks, filtered so that all students and questions have at least 250 associated responses. This results in dataset of for 6797 students across 4792 questions, detailing whether or not a student answered a particular question correctly. The dataset contains approximately 2.7 million responses, making it 8.2% observed. We treat each student and each question as a feature, and use binary encoding of each question's 's associated subjects as metadata. We used 60% of the questions as training set; 30% of the questions as the meta-test set and 10% of the questions as the meta-test set.
data point
The right panel in Figure illustrates the performance on prediction on the unseen meta-test set in terms of AUROC. Our method shows a significant improvement over all of the considered baselines over the entire range of suggesting promise for applying CHNs in educational settings.
4.5 TIMING EXPERIMENTS
We used the E-learning dataset to compare the time taken generate new feature parameters at meta-test time for a num- ber of methods considered in our exper- iments. The results are shown in Table
Table 1: Average time taken to initialize parameters for
feature in the e-learning dataset given & observations. All times are given in milliseconds, averaged across the whole meta-test set using a batch size of 128. T Random here indicates Train from Random.
We see that the CHN offers nearly a 4-fold speedup compared to the nearest- neighbours based approach. We see a sim- ilar difference in performance when com- pared to training the new heads on a single observation for just 10 epochs. Moreover,
Method/K
10-NN
400.8 0.5
402.3
405.1
T Random (1 Epoch)
47.4 4.5
4.7
81.+
Random (5 Epochs)
210.9 210.9:220.4
18.
331.1 10.1:2000 20.0
Random (10 Epochs)
48.
35.
651.7 4 38.8
Contextual HyperNet
1.3.7
116.5
119.9 +: 1.3
8

---

Under review as a conference paper at ICLR 2021
while this training time grows rapidly with
the number of observations in the context
set, the time taken for CHN remains nearly constant since it can efficiently parallelize along these observations, making CHNs an extremely efficient initialisation choice for larger context set sizes. The experimental settings are specified in Appendix D where the total computation time for training and evaluating a CHN was approximately minutes on the Neuropathic Pain dataset, 1.5 hours on the E-learning dataset, and hours MovieLens-IM.
CONCLUSION
We introduce Contextual HyperNetworks (CHNs), providing an efficient way to initialize parameters for new feature in model given context set of points containing the new feature and feature metadata. Our experiments demonstrate that CHNs outperform a range of baselines in terms of predictive performance across range of datasets, in both regression and classification settings, and are able to perform well across range of context set sizes, while remaining computationally efficient. In the future work, will evaluate CHNs in streaming setting with large-scale real-world applications.
REFERENCES
Marko Balabanović and Yoav Shoham. Fab; content-based, collaborative recommendation. Commu- nications of the ACM, 40(3):66-7,, 1997.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. Advances neural information processing systems, pp. 523-531, 2016.
Homanga Bharadhwaj. Meta-learning for user cold-start recommendation. In 2019 International Joint Conference on Neural Networks (ICNN), pp. 1-8. IEEE, 2019.
Chelsea Finn, Pieter Abbecl, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Voolum 70, pp. 1 1126-118 135. JMLR. org, 2017.
Marta Garnelo, Dan Rosenbaum, Chris Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J Rezende, and SM Eslami. Conditional neural processes. arXiv preprint arXiv:1807.016.1 2018
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367-4375, 2018.
Carlos Gomez-Uribe and Neil Hunt. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS), 6(4): 2015.
Wenbo Gong, Sebastian Tschiatschek, Sehastian Nowozin, Richard E Turner, José Miguel Hernández- Lobato, and Cheng Zhang. Icebreaker: Element wise efficient information acquisition with a bayesian deep latent gaussian model. In Advances in Neural Information Processing Systems, pp. 14791-14802, 2019.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Versa: Versatile and efficient few-shot learning. In Third workshop on Bayesian Deep Learning, 2018.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta-learning. arXiv preprint 1111:7111 2019.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint 1rXiv:160..09106. 2016.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.

---

Under review as a conference paper at ICLR 2021
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive inteiligent systems (tiis, 5(4):1-19, 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pp. 173-182, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.6980, 2014.
James Kirkpatriek, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526 2017.
Xuan Nhat Lam, Thuc Vu, Trong Due Le, and Anh Duc Duong. Addressing cold-start problem in recommendation systems. In Proceedings of the 2nd international conference on Ubiquitous information management and communication, pp. 208-211 2008.
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, pp. 689-698, 2018.
Blerina Lika, Kostas Kolomvatsos, and Stathes Hadjjefthymiades. Facing the cold start problem in recommender systems. Expert Systems with Applications, 41(4):2065-200 2014.
Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. Content- based recommender systems: State of the art and trends. In Recommender systems handbook, pp. 73-105. Springer, 2011.
Chao Ma. Wenbo Gong, José Miguel Hernández-Lobato, Noam Koenigstein, Sebastian Nowozin, and Cheng Zhang. Partial vae for hybrid recommender system. In NIPS Workshop on Bayesian Deep Learning, 201 8a.
Chao Ma, Sehastian Tschiatschek, Konstantina Palla, José Miguel Hernández-LLoatoo Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint arXiv: 1142, 2018b.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mchri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. arXiv preprint
2017.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv: 1rXii:1710.110 1010.1068, 2017.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2:2, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Michael J Pazzani and Daniel Billsus. Content-based recommendation systems. In The adaptive web, pp. 921-341. Springer, 2007.
Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, pp. 7957-7968. 2019.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pp. 285-295, 2001.
10

---

Under review as a conference paper at ICLR 2021
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. Collaborative filtering recommender systems. In The adaptive web, pp. 291-324, Springer, 2007.
Andrew I Schein, Alexandrin Popescul, Lyle H Ungar, and David M Pennock. Methods and metrics for cold-start recommendations. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information relrieval, pp. 253-260, 2002.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pp. 593-607. Springer, 2018.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th international conference on World Wide Web, pp. 111-112, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
David H Stern, Ralf Herbrich, and Thore Graepel. Matchbox: large scale online bayesian recommen- dations. In Proceedings of the 18th international conference on World wide web, pp. 111-120, 2009.
Ruibo Tu, Kun Zhang, Bo Bertilson, Hedvig Kjellstrom, and Cheng Zhang. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. In Advances in Neural Information Processing Systems, pp. 12773-12784 2019.
Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo Larochelle. A meta-learning perspective on cold-start recommendations for items. In Advances in neural information processing systems, pp. 6904-6914, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638 2016.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint
00695, 2019.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pp. 165-174, 2019.
Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, José Miguel Hernández-Lobar Richard Turner, Richard Baraniuk, Craig Barton, Simon Peyton Jones, et al. Diagnostic questions: The neurips 2020 education challenge. arXiv preprint rrXiv:2007.1206) 2020.
Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel J Kochenderfer, and Jure Leskovec. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems, 33, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.

---

Under review as a conference paper at ICLR 2021
A APPENDIX: PARTIAL VARIATIONAL AUTOENCODERS
A.1 PARTIAL VARIATIONAL AUTOENCODERS
For our experiments, we base our model on the Partial Variational Autoencoder (P-VAE) (Ma et al., 2018a) this model combines a traditional variational autoencoder (VAE) model with PointNet-style set encoder (Qi et al., 2017), allowing points. The P-VAE is based on the observation that typically the features in VAE are assumed to be conditionally independent when conditioned on the latent variable z. That is,
efficiently encode and reconstruct partially observed data
p(x z)
II p(zylz)
Then, given a data point x with observed features xo and unobserved features xv, we have that
p(Xy Xo. z) p(xy (z)
Hence, if we can infer posterior distribution over from the observed features, estimate p(xp/ (x)) The P-VAE infers a variational posterior distribution over z using an amortized inference network (or encoder network) (e(2)xx) and approximates the conditional data distribution given a value of using a decoder network (Xo, (z)
this
In our model, we extend the decoder decode the value of new feature 2n, by initialising an additional subnetwork in the decoder which we term a decoder head, with parameters Φn. to extend its output dimension by one. In principal this head could be of any architecture which takes as input the output of the shared layers of the decoder, but in practice we found that simply extending the final layer of weights and biases to accommodate new output dimension yielded good results while remaining parameter-efficient as the number of output features grows.
A.2 TRAINING P- P-ES
The P-VAE is trained to reconstruct observed features in the partially-observed data point, and in the process learn infer variational posterior qo (=(2(x)) over the latent variable z. The P-VAE is given batches of data points where features from both the meta-train and meta-test sets are hidden from the model. Additionally, each time particular data point input, some additional features are also randomly hidden from the model using a Bernoulli mask, in order to ensure the model is robust different sparsity patterns in the data. The P-VAE is then trained by maximising the Evidence Lower- Bound (ELBO) (Ma et al., 2018b):
log p(x)) 2 logr(xo) - DKL (xx.(11)))))]]]<<x>> Exng(s(xo) [log p(xo, xr|z)] DRr. [d(z)xo)l<((2) Lpartial(Xo) (Xo)
12

---

Under review as a conference paper at ICLR 2021
B APPENDIX: BASELINES
Here we provide additional details and results for the baselines used in our experiments.
B.I MAML
We adapt the Model-Agnostic Meta Learning (Finn et al., 2017) technique as a baseline. The decoder head parameters 0, are adapted using the MAMI. algorithm algoritt in the 'meta-training' stage. Each new feature X is viewed as a separate MAML task, with some observed and unobserved values. We sample the tasks in batches of size M and train the inner (a.k.a. fast) model over N steps. The inner model training loss is the ELBO of the PVAE on the observations Lxo- The meta-model (a.k.a. the slow outer model) is trained by being given the context set observations, and computing reconstruction loss on the target LT, (Jror X).. The gradient for the meta-model update is taken over the batch reconstruction losses mean. The full algorithm is detailed Algorithm 1.
Algorithm Feature-wise Model-Agnostic Meta-l with PVAE
Input:
p(At): distribution over features.
a. 3: learning rate hyperparameters.
M: meta-batch size, N: number inner iterations.
Initialize A
while not done do Sample M features X; p(2).
for all X; do
0..- <- 0
for j < O, do
Evaluate ELBO gradient Vo., Cxo (fo.. X2) w.r.t. observations in K examples Optimize inner model parameters: j1 +- 0i5 - Vo., (Jor
end for 0g - O
end for Evaluate gradient of mean reconstruction error V nt Expprry (ff
Optimize meta-model parameters: <- 0 V Expp(x) 2 (i)
end while Output:
Notably, since MAML aims to fit parameters that adapt quickly to new tasks, allows for fine-tuning evaluation time, that is, training the model for several iterations from the MAML parameter initialization. Here, we evaluate the model with and without fine-tuning.
In the MAML baseline experiments we M 4, N 10, ADAM (Kingma & Ba, 2014) with learning rate 3 = 10-2 for inner and outer model optimization. The model fine-tuned performance is evaluated over {1, 3,5, 5, 10) epochs and beset results are used. We make use of the higher order optimization facilitated by the higher library (Grefenstette et al., 2019) in the implementation of this baseline.
13

---

Under review as a conference paper at ICLR 2021
Fineturing
Epochs Finetuning
MAML: Epochs Finetuning
Epochs Finetuning
MMML: Epsch Finetuning
MovieLens-1M
Neuropathic Pain
E-Leaming
0.70
0.70
2.2
2.0
0.65
Meta-Tee Testeett Teaa-TettAll AUC
0.65
0.60
1.8
0.55
1.6
0.55
0.50
0.50
0.45
1.2
0.45
0.40
1.0
0.40
Context Set Size
Context Set Size
Context Set Size
Figure 6: MAML baseline performance comparison for 21,2,5, 5, 10) fine-tuning epochs and with fine-tuning. Left plot shows RMSE (lower is better), center and right plots show AUROC (higher is better).
1-Nearest Neighbour
S-Nearest Neighbours
10-Nearest Neighbours
MovieLens-1M
Neuropathic Pain
E-Learning
0.750
1.4
0.625
Meta-Tes Tata-Test RMSE
0.600
Meta-Test Test AUC
1.3
0.700
0.575
1.2
0.550
0.675
1.1
0.525
0.650
0.500
0.625
1.0
0.475
16 32
Context Set Size
Context Set Size
Context Set Size
Figure 7: k-Nearest Neighbour Head Parameters baseline performance for k E (1, 10}. Context set size here corresponds to the number of observed values used when determining the nearest neighbour heads.
C APPENDIX: ADDITIONAL RESULTS
Here we provide additional results not included in the main text.
C.1 MAML
Figure shows the performance of the MAML bascline for different numbers fine-tuning epochs and with fine-tuning. As expected, the bascline with fine-tuning is outperformed by those where fine-tuning employed. For the Neuropathic Pain and E-learning datasets, the increase in the number of fine-tuning epochs corresponds to improvement in performance (greater AUROC), whereas in case of MovieLens- IM, performance drops (RMSE increases) with longer fine-tuning, particularly for the smaller context set sizes.
C.2 K-NEAREST NEIGHBOUR HEAD PARAMETERS
We consider k-Nearest Neighbour Head Parameters baselines for the values & {1, 5, 10). Figure 7 shows the performance of this baseline for the different values of l across a range of context set sizes. We expect that as & is increased further, and the number of head parameters averaged over grows, the behaviour will approach that of the mean head parameter baselines. In the main text, 10-Nearest Neighbours is used throughout, it yields good performance in both the low and high-data regimes.
14

---

Under review as a conference paper at ICLR 2021
0.75
k-32
0.75
k-32
k-16
k-16
Noto TeSLAUC
k-8
0.65
Km4
kw4
k=1
k=1
K-0
k-0
0.35
5
16 Head Training Epochs
10
12
14
18
20
22
24
16 Head Training Epochs
10 12
14
18 20
22
(a) Train decoder heads from random initialisation
(b) Train decoder heads from CHIN initialisation
Figure 8: Comparing the predictive performance when training decoder new head parameters in P-VAE on range of context set sizes k on the E-learning dataset (higher is better).
Random
Mean Head Params (Matching Metadatal
Mean Imputing
MAML
Mean Head Panams
+
1C-Mearest Neighboars
Train from Random (10 Spochs)
CHN
Neuropathic Pain
E-Learning
0.55
0.65
Meta-Ter Test RASSE
0.60
0.45
0.40
0.35
0.45
0
2
8
16
32
2
16
32
Contoxt Set Size
Context
Size
Figure 9: Performances (#1o) of CHN and benchmarks on Neuropathic Pain (left) and E-learning (right) datasets for test features, with varying context size k. We report RMSE for both datasets, in constrast to the main text where AUROC reported.
C.3 TUN-TUNNNG
In the main text, we show the performance of training the new decoder heads their context sets from randomly initialized parameters for 10 epochs (see results for "Train from Random (10 epochs)"),, in order provide trade-off between predictive accuracy and computational cost. This section aims to elaborate further on this trade-off. In Figure 8a, we show the predictive performance of the V-V the meta-test after training randomly initialized head parameters for an increasing number of epochs, for range of context set sizes k. We see that the performance improves with training in all cases, with better performance achieved as the context set size increases, and thus the effect of over-fitting is lessened.
Furthermore, in Figure 8b, we perform the same experiment but instead initialising the heads with the CHN parameters. We see that in all cases except & = 0 and 1: = 32, training by gradient descent leads to decrease in performance due to over- over-fitting, suggesting that the CHN has an implicit regularising effect on the parameter initialisation. We note also that in all cases, the untrained CHN parameters substantially outperform those trained from the random initialisation for all values k even after 25 training epochs, with many of the training curves appearing to approach convergence.
C.4 RMSE VALUES FOR BINARY DATASETS
In Figure 9 we provide the RMSE values (lower is better) for the main paper experiments run on the Neuropathic Pain and E-learning datasets. We see that the relative performance ordering of methods in terms of RMSE is largely identical to the performance ordering obtained from the AUC results, suggesting that the proposed CHN approach is indeed better than other baselines.
15

---

Under review as a conference paper at ICLR 2021
CHN (No Metadata)
CHN (Metadata)
Mean Head Params (No Metadata)
Mean Head Params (Matching Metadata)
MovieLens 1M
1.12
Meta-Test RMSE
1.10
1.08
1.06
1.04
Context Set Size
Figure 10: Comparison RMSE (lower is better) with and without feature metadata for both the CHN and mean head parameter baselines, across 5 random data splits.
C.5 METADATA ABLATION STUDY
It is possible obtain ablation for removing the usage of the context set C, as input to the CHN by looking at the performance for k = 0 in any of the preceding performance plots. In order to further investigate the effect of metadata on the CHN's performance, we additionally perform an ablation on MovieLens-I1 1M by removing the use of metadata (movie genre and year) as input to the CHN. We also compare the performance of averaging all trained head parameters, vs. averaging all trained head parameters whose features are subset of those of the new feature under consideration. The results of these ablations can be found Figure 10. We see that both cases, in absolute terms the difference in performance due to the metadata is slight, with small improvement made to the mean head parameter performance, and very slight improvement to the CHN performance when This is likely due to the metadata carrying very little information for this dataset relative to the data points themselves, so that when & 2 1 the CHN is able to effectively disregard the metadata in favour of the observed data points. Future work could investigate the performance of CHNs datasets with richer and more information feature metadata.
0.
16

---

Under review as a conference paper at ICLR 2021
Table 2: Hyperparameters and architecture details for the P-VAE and CHN used each dataset.
Feed-forward neural networks are represented by list of the dimensions of their hidden layers.
MovieLens- 1M
Ncuropathic Pain E-learning
Training
Epochs
200
1000
50
Batch Size
1000
1000
1000
Learning Rate
Ie-3
le-2
1e-3
Weight Decay
0
0
0
Meta-Traie Trainnrin
Epochs
100
300
20
Batch Size
256
128
128
Learning Rate
le-4
le-3
lc-3
Weight Decay
le-3
le-3
1e-3
Set Encoder
Feature Embedding Dim.
50
30
50
Set Embedding Dim.
30
30
30
Encoder
Latent Dim.
150
20
150
Layers
[200]
[30]
[200]
Decoder
Shared Layers
[200]
[30]
[200]
Output Variance
0.1
CHN
Data point Embedding Dim.
50
25
50
Context Encoding Dim.
50
25
50
Context Encoder Layers
[128)
[50]
[50]
Metadata Encoding Dim.
5
20
Metadata Encoder Layers
[10]
[20]
Param. Pred. Net Layers [256.256,256]
[64,64]
[50, 100,
D APPENDIX: EXPERIMENT DETAILS
All models were implemented PyTorch (Paszke al., 2017). All experiments were performed on single Nvidia Tesla K80 GPU. For training both the P-VAE and the CHN's parameters, the ADAM (Kingma & Ba, 2014) optimizer was used with 31 = 0.9, B2 = 0.999 and æ 88
Details of hyperparameters and model architectures used for each dataset can be found in Table 2.
17


Reviews:
[{"review_id": "v5c8OMZj2PF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes CHN, a framework to extend an existing model to incorporate new features as they become available. The benefits of CHN are demonstrated by utilizing it to extend a P-VAE.\n\nThis work proposes a structured procedure to incorporate new features in an incoming data for use with a trained model. The proposed CHN uses a combination of a context vector and a meta vector (generated through a neural network from the new features's meta data) to produce parameters $\\hat{\\theta}_n$ (for the new feature $x_n$). This $\\hat{\\theta}_n$ can then be used with the base model's representations (encoded vector for the P-VAE setup) for downstream tasks.\n\nThe work does seem to have merit as existence of missing raw features / new information about data in the new samples is not uncommon for practical scenarios. Therefore, an approach that does not require extensive resource usage to produce a new model that can utilize the new information is very useful. \n\nThis work empirically demonstrates the benefit of CHN over several tasks along with demonstrating its efficiency. There are, however, some points that need to be addressed.\n\n\nIn Figure 5, the x-axis is the context size which I am assuming is $k$. However, in section 2.3 it is mentioned that $k_n$ is sampled from Uniform[0,32]. In such a setup, what does it mean to have a fixed point for each k on the x-axis ? Does it mean that $k_n$ in this case was not sampled from a uniform distribution ? Perhaps Figure 5 was misunderstood by me and a clarification would be useful.\n\n \nHow is the new feature incorporated in the model for future use i.e., in a continual learning type of setup. The new information might be useful for future feature revelations. Therefore, incorporating it in the base model or the model $\\psi$ will be useful as otherwise any new information received after the training of the base model (say the base P-VAE) will be lost to future instance of new feature introductions. This seems to be a very practical requirement. Is there a way to achieve this without having a big mitigating impact on speed and efficiency?\n\nIt does seem that the focus is on an unsupervised setting (as P-VAE) was used. How will the CHN be used for classification or other types of supervised learning tasks. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "sCpwKREHi_-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission focuses on the cold start problem of new entities (new items in a recommender system, new treatments in a medical application, etc.). It combines the strengths of the *relations* between a new entity and the existing entities, and the *content* features of the new entity, by fusing the two kinds of information into a neural network that outputs the estimated representation of the new entity. The proposed method outperforms several intuitive naïve strategies as well as MAML.\n\nPros:\n- The writing is clear.\n- Good reproducibility. Details, including hyperparameters, are listed in the appendix.\n\nCons:\n1. Similar models -- models that take the graph topology around a new node as well as the node’s feature as input and produces the node’s embedding without any extra training/finetuning step -- have long existed in the literature. For example, one can google “inductive learning + graph embedding”, “out-of-sample extension + graph embedding”, etc., and find plenty of related works. Not to mention that the nowadays (over-)popular graph neural networks naturally support this.\n2. It is unclear how this work improves upon Vartak 2017. Vartak 2017 views the cold-start problem in recommender systems as a meta-learning problem as well. It also combines the user rating data and the item’s content features. What’s new when compared to Vartak 2017, and how well the proposed method outperforms Vartak 2017 empirically?", "extra_scores": {}, "extra_text": {}}, {"review_id": "s7A5HdQjHmU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a few-shot meta-learning method for recommender system that uses a new feature's meta-information and observed samples for the features to predict the network weights for predicting the feature value from other features. The paper focuses on the cold-start problem where few samples with a new feature observed is available. The method outperforms a wide range of baselines on MovieLens-1M, a medical synthetic dataset, and a e-learning dataset.\n\nDisclaimer: I am not familiar with recommender systems.\n\nPros:\n- Methods straightforward and easy to understand, and does not have much ad-hoc design choices that are hard to validate.\n- The experiments are relatively thorough. A lot of baselines.\n\nCons\n- Novelty is limited -- a mixture of meta-learning and content-based method. There are also existing few-shot continual learning papers available (e.g. https://openaccess.thecvf.com/content_CVPR_2020/papers/Tao_Few-Shot_Class-Incremental_Learning_CVPR_2020_paper.pdf), so the motivation needs to consider differences from them.\n- Experiments does not do external comparisons with other recommender systems that deal with cold starting other than its own baselines.\n- Despite the large amount of baselines, ablation still lacks: (1) Train from random, but instead of training for a fixed number of epochs, just train an SVM or linear regression until convergence. This is what people would do as a baseline. (2) not important but it would be nice to ablate the CHN by taking out metadata and $C_n$ from input.\n- Not clear if the set of datasets is persuasive. One is a synthetic dataset. One recommendation system and one grading dataset.\n- There is no Appendix despite referencing it.\n\nMinor issues:\n- Not clear how the \"adapting to new features\" is different from meta-learning's adapting to new tasks, since they also use old tasks to inform new tasks. The abstract seems to suggest the method can take new features as input incrementally, which is a little confusing.\n- Overclaim at the end of Section 2.2: it is not O(1) if you have to do distributed computing. If you allow distributed computing, NP=P.\n- It's hard to tell why MAML is similar to CHN in one dataset but underperforms drastically in the e-learning dataset.\n\n\nPost-rebuttal:\nI appreciate the additional ablation study, but unfortunately the results did not strengthen the paper's distinction from related work. The explanation of the motives and related work comparisons only clarified differences between this paper and prior work that are either inherent to the task being addressed, or contribution unsupported by experiments. Unless the AC agrees with the authors that the paper is acceptable even without external comparisons (despite being a merge of two lines of work), I will not be changing my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eknH11FVgl3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes Contextual HyperNetworks (CHNs) as an auxiliary model to generate parameters from existing data, and observations and other metadata associated with new feature to address cold start problem of new feature. Besides, it doesn’t need either re-train or fine-tune at prediction time. The CHN is applied to P-VAE and some experimental results are provided to demonstrate its effectiveness in some application, i.e., recommender system, e-learning and healthcare tasks.\n\nThe motivation is clear and reasonable, and the proposed method with hyper-network is pretty interesting and attractive. However, experimental section has large improvement room.\n\n Pros:\n\n* The paper proposes an interesting direction to use hyper-network to solve cold-start problems in some critical tasks, such as recommender systems and etc.\n* The logic of paper is clear and easy to follow.\n\nCons:\n\n* The paper only illustrates how to apply CHNs to P-VAE, however, it’s better also to illustrate how to apply CHNs to other pretty common techniques in recommender system field (or e-learning or others), such as deep learning-based collaborative filtering methods. If there is no more one application, it is difficult to demonstrate the proposed method is generalizable enough to other models, including how easy it could be extended and how effectiveness it could have after extension.\n* The paper only shows the advantage of prediction time, but it doesn’t discuss a lot on the training latency. It’s better to also discuss this in the paper and compare with other methods.\n* The experiment setup is not strong enough to demonstrate the effectiveness of the proposed methods, for example,\n    * Lack of major important baselines and studies. In the paper, only several extension on how to handle new features on top of P-VAE is given in the comparison. However, it is unclear how it performs with other methods which target at solving cold-start problem. In other others, current experiments cannot provide evidence to show it outperforms other methods which could be used to address cold start problem.\n    * Fairness regarding to no meta information is incorporated in the baselines. CHNs is leverage additional meta information, however, other baselines don’t use this kind of information. \n    * Limited evaluation metics. Each task only either uses RMSE or AUC. So it lacks evidence to show whether the proposed method could outperform others in different validation metrics instead of bias.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a straightforward and easy to understand method for using hypernetworks to adapt existing models to be able to increase their output space. The method itself is also interesting and is detailed enough for reproducibility. However, the experiments and results should be improved by expanding the demonstration of CHNs beyond the narrow P-VAE application and comparing against relevant baselines in the recommendation system literature.

Pros
- Clear writing.
- Detailed hyperparameters to aid reproducibility.
- Straightforward model.

Cons
- Lack of sufficient comparison to related work, especially to existing recommendation systems that handle the cold-start issue and to Vartak, 2017.
- Limited results that only demonstrate application to P-VAE meaning it's still unknown if CHNs work well with other models. The result on the synthetic dataset is also less persuasive.
- Lack of sufficient ablations, i.e. training a SVM/linear regression model until convergence.
Decision: Reject