Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This submission focuses on the cold start problem of new entities (new items in a recommender system, new treatments in a medical application, etc.). It combines the strengths of the *relations* between a new entity and the existing entities, and the *content* features of the new entity, by fusing the two kinds of information into a neural network that outputs the estimated representation of the new entity. The proposed method outperforms several intuitive naïve strategies as well as MAML.

Pros:
- The writing is clear.
- Good reproducibility. Details, including hyperparameters, are listed in the appendix.

Cons:
1. Similar models -- models that take the graph topology around a new node as well as the node’s feature as input and produces the node’s embedding without any extra training/finetuning step -- have long existed in the literature. For example, one can google “inductive learning + graph embedding”, “out-of-sample extension + graph embedding”, etc., and find plenty of related works. Not to mention that the nowadays (over-)popular graph neural networks naturally support this.
2. It is unclear how this work improves upon Vartak 2017. Vartak 2017 views the cold-start problem in recommender systems as a meta-learning problem as well. It also combines the user rating data and the item’s content features. What’s new when compared to Vartak 2017, and how well the proposed method outperforms Vartak 2017 empirically?
review_title: An okay submission but with limited novelty.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes Contextual HyperNetworks (CHNs) as an auxiliary model to generate parameters from existing data, and observations and other metadata associated with new feature to address cold start problem of new feature. Besides, it doesn’t need either re-train or fine-tune at prediction time. The CHN is applied to P-VAE and some experimental results are provided to demonstrate its effectiveness in some application, i.e., recommender system, e-learning and healthcare tasks.

The motivation is clear and reasonable, and the proposed method with hyper-network is pretty interesting and attractive. However, experimental section has large improvement room.

 Pros:

* The paper proposes an interesting direction to use hyper-network to solve cold-start problems in some critical tasks, such as recommender systems and etc.
* The logic of paper is clear and easy to follow.

Cons:

* The paper only illustrates how to apply CHNs to P-VAE, however, it’s better also to illustrate how to apply CHNs to other pretty common techniques in recommender system field (or e-learning or others), such as deep learning-based collaborative filtering methods. If there is no more one application, it is difficult to demonstrate the proposed method is generalizable enough to other models, including how easy it could be extended and how effectiveness it could have after extension.
* The paper only shows the advantage of prediction time, but it doesn’t discuss a lot on the training latency. It’s better to also discuss this in the paper and compare with other methods.
* The experiment setup is not strong enough to demonstrate the effectiveness of the proposed methods, for example,
    * Lack of major important baselines and studies. In the paper, only several extension on how to handle new features on top of P-VAE is given in the comparison. However, it is unclear how it performs with other methods which target at solving cold-start problem. In other others, current experiments cannot provide evidence to show it outperforms other methods which could be used to address cold start problem.
    * Fairness regarding to no meta information is incorporated in the baselines. CHNs is leverage additional meta information, however, other baselines don’t use this kind of information. 
    * Limited evaluation metics. Each task only either uses RMSE or AUC. So it lacks evidence to show whether the proposed method could outperform others in different validation metrics instead of bias.


review_title: Experimental setup leaves something to be desired

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a few-shot meta-learning method for recommender system that uses a new feature's meta-information and observed samples for the features to predict the network weights for predicting the feature value from other features. The paper focuses on the cold-start problem where few samples with a new feature observed is available. The method outperforms a wide range of baselines on MovieLens-1M, a medical synthetic dataset, and a e-learning dataset.

Disclaimer: I am not familiar with recommender systems.

Pros:
- Methods straightforward and easy to understand, and does not have much ad-hoc design choices that are hard to validate.
- The experiments are relatively thorough. A lot of baselines.

Cons
- Novelty is limited -- a mixture of meta-learning and content-based method. There are also existing few-shot continual learning papers available (e.g. https://openaccess.thecvf.com/content_CVPR_2020/papers/Tao_Few-Shot_Class-Incremental_Learning_CVPR_2020_paper.pdf), so the motivation needs to consider differences from them.
- Experiments does not do external comparisons with other recommender systems that deal with cold starting other than its own baselines.
- Despite the large amount of baselines, ablation still lacks: (1) Train from random, but instead of training for a fixed number of epochs, just train an SVM or linear regression until convergence. This is what people would do as a baseline. (2) not important but it would be nice to ablate the CHN by taking out metadata and $C_n$ from input.
- Not clear if the set of datasets is persuasive. One is a synthetic dataset. One recommendation system and one grading dataset.
- There is no Appendix despite referencing it.

Minor issues:
- Not clear how the "adapting to new features" is different from meta-learning's adapting to new tasks, since they also use old tasks to inform new tasks. The abstract seems to suggest the method can take new features as input incrementally, which is a little confusing.
- Overclaim at the end of Section 2.2: it is not O(1) if you have to do distributed computing. If you allow distributed computing, NP=P.
- It's hard to tell why MAML is similar to CHN in one dataset but underperforms drastically in the e-learning dataset.


Post-rebuttal:
I appreciate the additional ablation study, but unfortunately the results did not strengthen the paper's distinction from related work. The explanation of the motives and related work comparisons only clarified differences between this paper and prior work that are either inherent to the task being addressed, or contribution unsupported by experiments. Unless the AC agrees with the authors that the paper is acceptable even without external comparisons (despite being a merge of two lines of work), I will not be changing my score.
review_title: Review (update)

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes CHN, a framework to extend an existing model to incorporate new features as they become available. The benefits of CHN are demonstrated by utilizing it to extend a P-VAE.

This work proposes a structured procedure to incorporate new features in an incoming data for use with a trained model. The proposed CHN uses a combination of a context vector and a meta vector (generated through a neural network from the new features's meta data) to produce parameters $\hat{\theta}_n$ (for the new feature $x_n$). This $\hat{\theta}_n$ can then be used with the base model's representations (encoded vector for the P-VAE setup) for downstream tasks.

The work does seem to have merit as existence of missing raw features / new information about data in the new samples is not uncommon for practical scenarios. Therefore, an approach that does not require extensive resource usage to produce a new model that can utilize the new information is very useful. 

This work empirically demonstrates the benefit of CHN over several tasks along with demonstrating its efficiency. There are, however, some points that need to be addressed.


In Figure 5, the x-axis is the context size which I am assuming is $k$. However, in section 2.3 it is mentioned that $k_n$ is sampled from Uniform[0,32]. In such a setup, what does it mean to have a fixed point for each k on the x-axis ? Does it mean that $k_n$ in this case was not sampled from a uniform distribution ? Perhaps Figure 5 was misunderstood by me and a clarification would be useful.

 
How is the new feature incorporated in the model for future use i.e., in a continual learning type of setup. The new information might be useful for future feature revelations. Therefore, incorporating it in the base model or the model $\psi$ will be useful as otherwise any new information received after the training of the base model (say the base P-VAE) will be lost to future instance of new feature introductions. This seems to be a very practical requirement. Is there a way to achieve this without having a big mitigating impact on speed and efficiency?

It does seem that the focus is on an unsupervised setting (as P-VAE) was used. How will the CHN be used for classification or other types of supervised learning tasks. 


review_title: Interesting work

Meta-review: This work presents a straightforward and easy to understand method for using hypernetworks to adapt existing models to be able to increase their output space. The method itself is also interesting and is detailed enough for reproducibility. However, the experiments and results should be improved by expanding the demonstration of CHNs beyond the narrow P-VAE application and comparing against relevant baselines in the recommendation system literature.

Pros
- Clear writing.
- Detailed hyperparameters to aid reproducibility.
- Straightforward model.

Cons
- Lack of sufficient comparison to related work, especially to existing recommendation systems that handle the cold-start issue and to Vartak, 2017.
- Limited results that only demonstrate application to P-VAE meaning it's still unknown if CHNs work well with other models. The result on the synthetic dataset is also less persuasive.
- Lack of sufficient ablations, i.e. training a SVM/linear regression model until convergence.
Decision: Reject