Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the weight pruning and weight sharing for image-text retrieval model. Contrary to previous works that applies the pruning and sharing on the layer level, the main idea of this paper is to apply weight sharing and weight pruning on the weight level. Therefore, the pruned model could achieve better compression rate and performance. The pruning and sharing is controlled by a hyper network, which is trained jointly with the image-text retrieval model. The proposed approach achieves better performance and better compression rate than the previous state-of-the=art.
strength_and_weaknesses: The paper is interesting to read and the proposed idea is novel.
I just have several quick questions and comments.
1. The proposed approach seems could be apply not only on the image-text retrieval model, but also on single modal models (e.g. image only model and text only model). I wonder why would the author choose the image-text retrieval model?
2. Just curious, as the ConvNet models also shares unified structure across the model, would the proposed approach work on the ConvNet models?
3. How to set the base layers? Is it set heuristically?
4. I might miss some details, but what is the 'z' in Eq.8 in implementation?
5. Just curious, how to enforce the Eq.6 and Eq.7 during training?
6. I wonder did the author observe any training instabilities for image-text retrieval model pre-training? 
clarity,_quality,_novelty_and_reproducibility: Although I am not the expert in model pruning, I think this paper is novel and should attract multiple readers in model pruning and vision+language communities.
As I didn't have experience in implementing the model pruning approach, I cannot judge the reproducibility of this paper.
summary_of_the_review: Given the improved performance and the novelty, I would recommend the rating of 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel compression algorithm for transformer-based vision and language models. It trains a hypernetwork to identify whether a network weight is able to share across multimodal backbones and across layers within one backbone or is able to be pruned off the original networks. Experiments show that the proposed model compression algorithm can reduce up to 40% of parameters without significant loss of accuracy in several downstream tasks.
strength_and_weaknesses: 1. Strengths:

+ The paper addresses an important topic and the proposed algorithm is well motivated. The paper is well written making it easy to follow.
+ Experiments are done carefully and the results are convincing. 

2. Weaknesses:

+ Some details are missing in the current version. In page 4, the authors divide a backbone into several blocks to identify whether network weights in those different blocks can be shared. By what criteria did you divide into these blocks? What is the optimal number of blocks? There is no justification by experiments or intuition for this in the paper. 

+ In page 4, after Eq. 5, the final W_q^vl is a combination of only two elements since s^l and s_{t_l} are binary. This makes the statement inaccurate.

+ In page 6, the authors mentioned that they only used a subset of training data to train HN. How does this affect the compression performance? What proportion of the subset is sufficient? Again, there are no justifications for why the authors do this.
+ The experiments conducted on the much smaller image size (224x224) compared to the original baselines, which raises doubts about the baselines' results.

+ Have the authors tried methods such as grid search to find the best hyperparameters of the backbone for that small image input?

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the proposed method is clearly explained.
summary_of_the_review: The paper is well written and the proposed method is clearly explained.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a framework to perform weight sharing and pruning across two transformer backbones and within the same transformer backbone. The framework is evaluated on vision and language tasks, including Referring Expression Comprehension (REC), Visual Question Answering (VQA), and Object Detection. The results show that the proposed framework achieves a prominent trade-off between number of parameters and accuracy.
strength_and_weaknesses: Strength
- The paper is well organized and convincing
- The paper proposes a novel framework for weight sharing and pruning of transformer backbones
- The method achieves a prominent trade-off between the number of parameters and the accuracy
- The method is tested on various vision and language grounding tasks, including Referring Expression Comprehension, Visual Question Answering, and Object Detection

Weakness
- The paper does not provide a detailed analysis of the proposed method
- Also, they only share part of hyper-parameters which makes the result hard to reproduce
- Lack of justification for experimental setup, hyper-parameter selection for model and baseline.
clarity,_quality,_novelty_and_reproducibility: This paper shows  a clear description of the proposed framework and the evaluation of the method on several vision and language tasks and the framework is evaluated on several vision and language tasks.
For reproducibility, it seems hard to reproduce the method without public code.

summary_of_the_review: The paper is well written and easy to follow and the proposed method is clear and well motivated. 
The experiments are well designed and the results are convincing, but the paper could provide a more detailed analysis of the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed a novel compression algorithm for transformer-based vision and language models and empirically show that it can reduce up to 40% of parameters without significant loss of accuracy in several downstream tasks. Reviewers generally agree the work is interesting, the proposed idea is novel and well motivated, the empirical results are convincing. There are some minor questions about the experiments and ablation studies and clarity of the presentation, authors made proper responses to address them during rebuttal. Overall, this is an interesting and practically useful paper that is recommended to accept. 


justification_for_why_not_higher_score: Although the proposed idea seems novel and the work is well executed, most techniques were borrowed from exiting literature and the technical depth is not particularly strong.
Decision: Accept: poster