Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach for adapting a base model (in their case, a VQA model) given a task descriptor. The authors propose to perform such adaptation via a hypernetwork layer, and learn a latent representation through which one can control the modulation of the base model by training a VAE on said layer’s weights. This manipulation is achieved by either 1) using a task descriptor learned through joint embedding of task description-parameter pairs for classifier-like guidance in the hypernetwork latent space or 2), fitting a generative model on task descriptor-parameter pairs and using classifier-free guidance on the hypernetwork latent. They demonstrate that these techniques allow them to condition the base model on in-distribution test task descriptions so as to achieve a multi-task learning effect. While the methods considered are somewhat novel and interesting, their merit is not clearly demonstrated experimentally, as it does not clearly outperform the straight-forward yet strong baseline of conditional multi-task learning. Moreover, the setting considered in experiments may not exactly require meta-learning, since the task inference problem is not present at test-time (task description is given), so that the problem reduces to conventional multi-task learning.

strength_and_weaknesses: Strengths:

The authors propose a novel approach for modulating a base model in a task specific manner by guiding the prediction of dynamic weights with a learned representation of the task description.

The authors propose a novel approach for how said task representations can be learned by contrastive learning for joint embedding of task description and dynamic weights.

The technical background relevant to the method -- hyper-networks, generative hyper-networks, guidance with a contrastive classifier, as well as classifier-free guidance -- is clearly written.

Weaknesses:

The experimental setting used to validate the effectiveness of the approach does not necessarily require meta-learning (task inference followed by adaptation), since the model is conditioned on an explicitly provided task description. The work thus seems incorrectly positioned, insofar as it relates more to multi-task learning rather than meta-learning. This results in a lack of clarity, which could be resolved by positioning the approach as a solution for multi-task learning.

The proposed method is not shown to outperform the straight-forward yet strong baseline of conditional multi-task learning, despite being significantly more complex. More experiments are needed that clearly demonstrate the advantage of introducing such complexity.

Popular relevant approaches such as FiLM (Perez et al, 2017)  and AdaIN (Huang et al, 2017) are not discussed, but would serve as strong baselines for base model modulation with a conditioning signal.
clarity,_quality,_novelty_and_reproducibility: The technical background relevant to the method -- hyper-networks, generative hyper-networks, guidance with a contrastive classifier, as well as classifier-free guidance -- is clearly written. A key issue of clarity is the positioning of the work as pertaining to meta-learning -- which involves both task inference and subsequent adaptation -- since the task inference problem is not present in their train and test settings. This is because the model is given the corresponding task description as input, such that the setting reduces to conditional multi-task learning. Moreover, while the methods proposed -- namely, approaches for modulating a base model with hypernetwork layers guided with a task description -- are fairly original, such approaches contribute considerable added complexity without demonstrating clear advantages to the straight-forward baseline of conditional multi-task learning.
summary_of_the_review: While the methods proposed are fairly original, such approaches contribute considerable added complexity without demonstrating clear advantages to the straight-forward baseline of conditional multi-task learning. Perhaps more importantly, a key issue of clarity is the positioning of the work as pertaining to meta-learning -- which involves both task inference and subsequent adaptation -- since the task inference problem is not present in their train and test settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper proposes generative models that predicts (or generates) high-performing task-specific weights conditioned on text description of the task.
- Specifically, three "Hyper"-components are proposed: 
1) Hyper-"decoder": a decoder of VAE trained with (unconditional) auto-encoding of weight vector,
2) Hyper-"CLIP": task and text encoders trained to maximize cosine similarity of positive pair,
3) (optional) Hyper-"LDM": diffusion model for the latent space of 1) Hyper-"decoder".
- After training the above three components, we can sample task-specific weights through:
 $t \xrightarrow{\text{HyperCLIP}} e \xrightarrow{\text{HyperLDM}} z \xrightarrow{\text{HyperDecoder}} w$.
- The experimental results show that the proposed method performs better than the relevant baselines.



strength_and_weaknesses: Strength:
1. This multi-modal generative modelling of "neural network weight" is novel .
2. The task of Meta-VQA is firstly proposed in this paper, and it is very interesting for me.
3. This paper clearly benchmark the given task including convincing baselines and the proposed method, which largely contributes to (multi-modal) meta-learning community.

Weakness:
1. The proposed method consists of three components, and their usage is still confusing for me (they are used in two ways: w or w/o Hyper-LDM). What I understand is: a) $z_0$ is firstly sampled, e.g. $z_0\sim N(0,I)$, b) $z_0\rightarrow z_t$: the code is iteratively modified through either gradient descent on equation 4 (w/o Hyper-LDM) or LDM, and c) Hyper-Decoder decodes $z_t$ to obtain the weight. If this is not wrong, I would like to recommend either algorithm tables or figure explaining how they are used in the meta-test (inference time).
2. I can't find any information about how exactly the base model perform each task. Does the base model find a word from vocabulary by measuring the cosine distance? Or is it a classifier? Then, how the generated weights are used for the tasks? These questions can not be resolved from this paper. 


clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is overall well-written, except for that some details are confusing and missing: see weakness 1 and 2.

Quality: The evaluation of the proposed methods are fairly conducted: see strength 2.

Novelty: This paper provide a novel perspective of hypernetwork-based approach: see strength 1.

Reproducibility: This paper provides full code for reproducing the experimental results.
summary_of_the_review: This paper proposes multi-modal generative models learn to generate ``neural network weights'' conditioned on text description on new tasks, and provides a kind benchmark to this research direction. Some of details can be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents two new approaches to zero-shot learning via building generative models of classifier weights. Contrastive learning is used to build a CLIP-style model which matches task description with network weights. In one approach, this CLIP-style model is used at test time to maximize match between provided task description and the network weight. Here, the search space for network weights is constrained to the latent space of the generative model. 
In the other approach, a DM is trained on the latent space of the generative model, conditioned on the CLIP embedding of the task. Classifier-free guidance is used at sampling time to improve quality. A new dataset Meta-VQA is also proposed for evaluating zero-shot classification methods.
strength_and_weaknesses: While the use of hyper-networks in meta learning is not new, this particular choice of architecture and loss (VAE+ELBO) in this paper has not been investigated before. The proposed methods for zero-shot learning, in particular the LDM based variant, is novel to my best knowledge. 

I have some concerns regarding the design of the method and the experiments, listed below:

1. The VAE has the critical role of compressing and linearizing network weights into an euclidean-like latent space. Have you investigated the choice of architecture for both the encoder and decoder beyond shallow MLPs? Often we see adaptive-normalization layers being more appropriate than feedforward MLPs for outputting to non-euclidean vector spaces, have you experimented with any such models?

2. There exists a large existing literature on zero-shot learning, including both established benchmarks (e.g. OpenImages, Cub200-2011, Animals with Attributes AWA, imagenet 21k etc.) and well known methods (e.g. https://arxiv.org/abs/2101.11606v2, https://openaccess.thecvf.com/content/CVPR2022/html/Su_Distinguishing_Unseen_From_Seen_for_Generalized_Zero-Shot_Learning_CVPR_2022_paper.html, https://openaccess.thecvf.com/content/CVPR2022/html/Zhu_Dual_Cross-Attention_Learning_for_Fine-Grained_Visual_Categorization_and_Object_Re-Identification_CVPR_2022_paper.html, https://openaccess.thecvf.com/content/CVPR2021/html/Han_Contrastive_Embedding_for_Generalized_Zero-Shot_Learning_CVPR_2021_paper.html). Have you 1. experimented with running either of your proposed methods on standard ZSL benchmarks? And 2. Have you tried running any previous works (besides CLIP) on your proposed dataset? The related works section also do not cite any prior works in zero-shot learning.

3. Very few details are provided regarding the definition of the baselines that you compare to. For example, how is MAML used without any test-time adaptation? How do you solve the class assignment problem in that case? What is the “standard multitask learning method” referring to? How are the multitask methods adapted for the few-shot setting? 

4. Characteristics of the proposed meta-VQA datasets are missing: how many choices are there on average for each question? How do you split the training and test tasks (e.g. did you check if semantically identical questions exist between training and testing)? How big is the support set in the few-shot setting?

5. It is concerning that the performance of the proposed methods do not significantly out-perform baselines on the only zero-shot dataset evaluated, and is even out-performed on the few-shot task. I would suspect that if baselines designed for ZSL are included, the comparisons would appear even worse. Can you provide any hypothesis as to under what circumstances will the proposed method exhibit quantifiably better performance than existing methods?

6. Diffusion models, particularly when imbued with sufficient capacity, are known to overfit rapidly to small datasets, where latents are all mapped to known training examples. Have you validated the generative capabilities of your diffusion model? Is it really working as intended?

clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: Given the above concerns, I think this work is not suitable for publication in its current state, and hence I suggest rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a hypernetwork that generates the adapted weights for each task and runs the experiment on a adapted dataset called “meta VQA” for testing zero-shot and few-shot performance. The results suggest that the added benefit of the proposed module is tiny.
strength_and_weaknesses: Strengths:
- The authors made a huge effort designing many model components and have spent time writing a paper that describes their idea.
- The authors also contributed a new evaluation dataset based on existing data.

Weaknesses:
- Complexity vs. significance. Despite a plethora of new components proposed, e.g. VAE, HyperClip, Guidance, Hyper latent diffusion etc, the method does better than baselines only by 0.5% in zero-shot meta-VQA, which is surprisingly marginal. I suggest that the authors find a minimal set of modifications with the maximal difference of results.
- Although it is good to see a more realistic dataset on VQA proposed, the paper could also be evaluated on traditional zero-shot and few-shot tasks.
- The paper lacks a thorough description of how the meta-VQA dataset is created. And also there is no visualization of the examples, nor is there qualitative analyses. I also checked Appendix A.3 and it is still very unclear. A few more examples with model predicted outputs/groundtruth will be very helpful to understand the setup.
clarity,_quality,_novelty_and_reproducibility: - Reproducibility: The submission came with code in the supplementary, a plus for reproducibility.
- Novelty: The model is novel but empirical evidence suggests that it is not very useful.
- Clarity: The paper writing is ok, but Figure 1 is very intimidating. It also lacks a central hypothesis of what it is trying to show. Why a hyper clip guidance with latent diffusion in the first place?
summary_of_the_review: In summary, the experimental results show that it is probably unnecessary to introduce so many model complications. The newly introduced dataset is a highlight, but the paper could have been evaluated on existing few-shot / zero-shot datasets too. Therefore, I think it is definitely below the bar of acceptance.

---
I acknowledge that I have read the rebuttal. I thank the authors for providing additional results and raise my score from 1 to 3. Overall, the paper lacks a great amount of clarity in terms of model design and dataset evaluation, and the model design seems intimidating while the added benefits are small.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a meta-learning method that uses classifier(-free) guidance from the generative modeling literature to generate zero-shot adapted network weights. While the paper contains some interesting ideas, the majority of the reviewers generally agreed that the work is not strong enough for acceptance (especially the results of the initial manuscript are rather poor which may explain the low rating review scores). Reviewers raised some major weakness concerns about incremental empirical improvement as compared with the simple baselines, the weak experiments, the lack of convincing results to justify the added complexity of the method, and the confusing presentation issues. 

While we appreciate the authors have tried to improve the paper and add additional experimental results to address the review concerns during the rebuttal, the major concerns of the reviews remain and most of the reviewers were still not convinced that the revision is strong enough for acceptance (while some reviewer did acknowledge the efforts of adding additional results help improve the quality somewhat but was not convinced to change the reject decision). 

Overall, the quality of this work is below the acceptance bar and the paper would require significantly more revision efforts to improve the overall quality, which includes more extensive experiments, more convincing results to justify the significance of the proposed method with the added complexity, and better presentation and writing to improve the motivation of the proposed technique and the overall clarity of the work.
Decision: Reject