Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors of the paper address the problem of click-through rate (CTR) prediction customized to different instances. They propose an adaptive parameter generation (APG) method for generating weight matrices in instance-adapted CTR systems. Generating the entire weight matrix directly is very time and memory consuming. To reduce time and memory complexity, they propose to decompose the weight matrix into a product of three vectors, namely S, U, and V. Only S is dynamically generated to capture different instances of custom patterns, while U and V are shared instances that capture common patterns. Results from three public datasets and real search systems are shown.
strengths_and_weaknesses: Strengths:
1. By decomposing the matrix and then decomposing the feedforward, the time and memory complexity of the CTR model with APG is greatly reduced.
2. Improve scalability to large inputs through over-parameterization.
3. The paper is well written.

Weakness: 
1. Does not provide a comparison of between generating the S vector and using fixed S vector on the current v1-v5 versions. 
2. It does not provide a comparison with existing methods that automatically generate weight matrices (e.g., in the AutoML field). 
3. A small typo on line 210.
questions: 1. In a group-wise conditional design, is there any extra effort required to divide instances into different groups?
2. S is used to capture custom patterns because of its low rank. Have you tried using U and V for this purpose? Did they achieve similar results?
3. V5, over parameterization, appears to be a separate technology. How it connects to APG.
4. What is the APG version reported in Table 2?
5. The training time and memory complexity of v5 is not provided in Table 5. Is it similar to the basic version? What is the trade-off between the efficiency and effectiveness of v5?
limitations: 1. Can provide a comparison between using generated and fixed S in the current v1-v5 version.
2. Can provide comparisons with existing methods of automatically generating weight matrices (e.g., in the AutoML domain).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of enabling input-aware model parameters which are dynamically generated in order to boost representation power of deep CTR (click-through rate) prediction models. A novel and general method APG (Adaptive Parameter Generation) is proposed, being able to work together with a variety of existing CTR prediction models. By means of employing several techniques such as low-rank parameterization, decomposed feed-forwarding, parameter sharing and over parameterization, APG generates adaptive parameters in an efficient and effective way. Computational complexity is analyzed carefully; empirical results demonstrate APG's efficiency and effectiveness. Notably, APG is claimed to be deployed in an industrial sponsored search system, achieving performance gain in online A/B testing.
strengths_and_weaknesses: * **Originality**: Dynamic neural networks are widely studied in computer vision and natural language processing, in which model parameters and architectures could be input-aware and thus dynamically generated. It is claimed that this paper is the first to bring the idea about dynamic neural networks into deep CTR prediction models.
* **Clarity**: The paper is well organized and clearly presented. However, typos are widespread throughout the text.
* **Significance**: Computational complexity of the proposed APG is carefully analyzed, and the efficiency is also validated empirically. A wide variety of baselines are employed to demonstrate the generality of APG. It is noteworthy that APG is claimed to be deployed in an industrial sponsored search system, achieving performance gain in online A/B testing.
questions: In Line 283 “Since over parameterization does not introduce any cost, it is not considered here”: Why? It seems that over parameterization would introduce more computational cost during training.



limitations: The writing has to be polished carefully. Several typos are listed as below:

* Line 87: analysis → analyze
* Line 106: is lack → lacks
* Line 107: kinds → kind
* Line 118: General → Generally
* Line 215: detailed analyze → analyze in detail
* Line 242: Since APG → APG
* Line 282: analysis → analyze
* Line 349: detailed analyzed → analyzed in detail
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to learn input-aware parameters in deep CTR models to boost its representation power. A clear method iteration trace is presented with clear design and solid analysis. The final version archives both efficiency gain and accuracy improvement. Most impressively, the method is tested in industrial production system. 
strengths_and_weaknesses: Strengths
The paper is well written and quite readable. Methods' iteration records are plausible.
 
Weaknesses
Current baseline on parameter allocation is too weak. This is not necessarily a weakness since advanced baseline to learn fine-grained parameters for model parameters may not be existed. I am not an expert of related literatures.
questions: The novelty of such fine-grained input-aware parameter allocation methods. The related work section states that this is almost first work to apply such heuristic, right?

Do we have benchmark on the effectiveness of various condition designs?

The gain from v4 to v5 is huge, why an over parameterization trick so powerful? If parameter sharing is beneficial, base model should not be such poor?
limitations: The method iteration are surprisingly positive which may imply some deep insights. Let's say  
- base -> v1 parameter personalization improves performance 
- v1 -> v2 low rank parametrization doesn't change performance too much.
- v2 -> v4 appropriate parameter sharing improves performance 
- v4 -> v5 over-parameterization (more sharing parameters) largely improves performance

It seems model favors heavy parameter sharing plus certain parameter personalization. More effort is needed to study the root factor and a mode direct approach based on it.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper aims to improve an important application -- CTR prediction.

The authors claim that a static way of parameterization is suboptimal, where the weight of CTR network is shared by all the instances.
Instead, they prefer a dynamic way of parameterization, where a personalized weight is generated for each instance.

Specifically, they apply a hypernetwork to generate the weight for each instance. Moreover, they apply techniques like low-rank parameterization, Decomposed Feed-forwarding, Parameter sharing and Over Parameterization to improve the efficiency and effectiveness of their method.

They present the experimental results on both public datasets and online real-world search system.
strengths_and_weaknesses: Strengths:

(1) The paper is well-organized. The motivations for using low-rank parameterization, Decomposed Feed-forwarding, Parameter sharing and Over Parameterization are presented in a very clear way. The figures are very clear and beautiful.

(2) The experiments are comprehensive, including public datasets and online real-world search engine.

Weakness:

(1) The novelty is limited. The core idea of this paper is using a shared hypernetwork to generate weight for each instance or group, which was originally proposed in "Hypernetworks, David Ha, Andrew Dai, and Quoc V Le. 2016." and has been widely applied like in "Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021" The techniques like low-rank parameterization and Parameter sharing are also not new, which could be found in "HyperGrid Transformers: Towards A Single Model for Multiple Tasks, ICLR 2021."

(2) Related work is missing. Hypernetwork should be an important piece of the related work in the paper but authors seem to completely unaware of the existing of this piece.

(3) The improvement is limited on public datasets. In Table 2, it seems to me that the improvement of APG is not that large. For example, the average gain of APG is only 0.24 in terms of AUC on MovieLens dataset. In particular, the SOTA CTR architecture DIFM has only been improved 0.1, 0.06 and 0.66 on the three datasets, without significant test, it is really hard to be convinced that this method can substantially improve the CTR performance.

questions: May I ask if authors could read more related papers of hypernetworks and put this work into that context and re-evaluate the novelty?

Is this one the first paper that introduce hypernetworks into CTR prediction?

The current version seems to be a better fit to some application-toward conferences like SIGIR, KDD, considering this paper neither provide deep and theoretical analysis about why same and shared parameters for all instances is suboptimal nor present original idea of hypernetworks.

It is no doubt that this paper is a good application paper that apply hypernetworks for CTR prediction.
limitations: This paper has no potential negative societal impact.
review_title: null

Meta-review: metareview: The paper focuses on the application of click-through rate (CTR), and proposes input-aware model parameters which are dynamically generated in order to boost representation power of deep CTR prediction models. To reduce time and memory complexity, the method decomposes the parameters and dynamically generates only part of the decomposed parameters. Improved results are shown on three public datasets and an A/B testing on an industrial system as claimed. Overall, this is a nice application-focused work that applies the widely studied idea of parameter generation and decomposition onto the new problem of CTR.
confidence: Less certain
Decision: Accept