Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Polyhistor and Polyhistor-Lite, consisting of Decomposed HyperNetworks and Layer-wise Scaling Kernels, to share information across different tasks with a few trainable parameters and address parameter-efficient multi-task adaptation for vision tasks. The authors construct a unified framework with the same implementation details and provide a comprehensive and fair comparison between existing parameter-efficient adaptation works in NLP on multi-tasking dense vision problems. Compared with the state-of-the-art multi-tasking parameter-efficient adaptation method, the method achieves competitive performance improvement with ∼ 90% reduction in the trainable parameters .
strengths_and_weaknesses: Strengths:
1. This paper conducts a thorough study on how the existing successful parameter efficient methods on NLP tasks perform on vision tasks, i.e., semantic segmentation, human part segmentation, saliency detection, and surface normals estimation.
2. The authors design a novel parameter-efficient method for adaptation to dense vision tasks. Specifically, the hyper-networks take input task embeddings and layer embeddings to produce low-rank matrices and further obtain the adapter weights.
3. Experimental results show that Polyhistor-Lite can achieve a competitive performance gain compared with the state-of-the-art method and only use a very limited amount of tunable parameters.

Weaknesses:
1. There are many hyper-parameters in this method, e.g., task embedding size, dimension of ranks, and the down-projection ratio of adapters. Searching for suitable hyper-parameters costs a lot of resources.
2. The method is implemented based on SwinTransformer backbone. It would be better to conduct experiments on more other backbones.
questions: 1. In Table 1 in Appendix, why the performance of Polyhistor-Lite (ρ = 32) in semantic segmentation is slightly better than Polyhistor-Lite (ρ = 1). It would be better to show more results of different down-projection ratio of adapters.
2. Can this method be applied to other backbones?
limitations: The method is novel and solve multiple tasks with limited tunable parameters. However, there are several hyper-parameters needed to be tuned in this method. Besides, The method only focuses on dense vision tasks. It would be better to include more common vision tasks like object detection.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The manuscript provides an extensive multi-task parameter-efficient benchmark and examines existing parameter-efficient fine-tuning NLP methods for vision tasks. The main contribution is that this work is the first to address parameter-efficient multi-task adaptation for vision tasks, and developed a unified framework to benchmark several parameter-efficient fine-tuning NLP methods on dense vision tasks. The paper is mostly written in a good manner, and the idea is straight yet effective.
strengths_and_weaknesses: The paper seems to combine several existing methods, and extend their applicable scene, making the contribution of this work less of a strength. However, they did several modifications to the existing methodologies, which should be detailed and illustrated.
questions: Major:
1.	The framework design of proposed Polyhistor is not explained clearly. Illustration on main steps of parameters choosing as well as the formulations are encouraged, and the usage of schematic diagram makes the idea easier to follow and understand.
2.	The difference between Visual Prompt Tuning and this work could be compared more concisely. 
3.	The novelty of this work mainly focused on the improving HyperNetwork and proposed scalable kernel, thus introduce more about these ideas would be of help.

Minor:
1.	A small grammar mistake: in line 82, it should be ‘a unified framework’, not ‘an’.

limitations: The true novelty of the work should be further justified.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a parameter-efficient multi-task adaptation method for dense vision tasks, called Polyhistor-lite. It is used to adapt a pre-trained hierarchical vision transformer for solving multiple dense vision tasks. The proposed method consists of two aspects, the Decomposed Hyper-networks and the Layer-Wise Scaling Kernels. Models are evaluated on PASCAL-Context datasets for semantic segmentation, human part segmentation, surface normals estimation, and saliency detection and are shown to be effective.
strengths_and_weaknesses: 
**Strengths**
1. It is interesting to explore the parameter-efficient adaptation techniques for dense vision tasks. It has not been investigated in this area. 
2. The presentation of this work is clear. This paper also provides a detailed discussion of the differences and relations with parameter-efficient multi-task adaptation methods in NLP tasks. 
3. The proposed parameter-sharing method is reasonable and is shown to be helpful in reducing the learning parameters but also keeping relatively high performance. 
4. Comparisons with existing approaches are thorough and significant.

**Weaknesses** 
1. In Figure 2(b), the Transformer in the lower part has no direct connection with the upper part and is meaningless. 
2. The proposed method is only evaluated with Swin-Transformer. As claimed by this paper, it is designed for hierarchical vision transformers. Would it also work well with other hierarchical vision transformers?

questions: 
See Weaknesses point 2 for the question. Overall, this proposed method is novel and effective with well-presentation. 

limitations: Limitations are discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes Polyhistor, a parameter efficient tuning method for jointly tuned dense vision tasks. Previous adapter-like methods in NLP are benchmarked in detail for dense vision tasks. The proposed method is proved to give better parameter-performance trade-off on the studied tasks.
strengths_and_weaknesses: Strengths:

1. The paper is neatly written. It is easy to follow and understand.
2. I haven't seen a lot of paper to study parameter-efficient tuning specifically for dense vision tasks, so the benchmark for adapter-like methods in this paper should be helpful for the community. The authors also promised to release the code. 
3. The idea makes sense to me, and the performance is well-supported by the experiments. Low-rank decomposition and dynamic weight are nothing new, but they are properly applied. The Layer-wise Scaling Kernel is novel and effective.

Weakness:

The comparison w.r.t single task baselines doesn't seem fair to me. A model jointly trained on similar tasks is expected to outperform models trained on single tasks respectively. I feel results on single-task should be reported. It's ok to perform worse on a few tasks, since this paper is mainly comparing with multitask methods, but the results should be presented for completeness.

questions: I am just curious about why the four tasks are jointly tuned. The reason that we need parameter-efficient tuning is that we can not afford replicating a model with 500B parameters on 1000 down stream tasks, especially when the tasks come one by one, unexpectedly. If we jointly tune the model for a few tasks, how do we do with the upcoming tasks?

I do not mean to criticize the paper on this point. It's a good paper, and the world needs diversity. I just have this confusion, and some other readers may also have this confusion. So, maybe this is a problem worth addressing in the paper.
limitations: 1. Only pretrained Swin Transformer is studied in this paper, but I feel this method can be easily extended to ConvNets.
2. More experiments on models pretrained on SSL tasks should also make this paper stronger, since SSL models are competitive against IN-pretrained models, and the features are very different. The behavior of a method can be very different depending on the pretrain task.
review_title: null

Meta-review: metareview: The proposed Polyhistor and Polyhistor-Lite for parameter-efficient multi-task adaptation achieves competitive performance gains on dense vision datasets. All reviewers give consistent positive scores. The requested experiments for more backbones, self-supervised backbones and analyses have been accordingly added during the discussion phase. Reviewer Gyt3 is concerned about the unclear explanation of the framework, and why HyperNetwork and scalable kernels could help. The authors addressed the issues and modified the paper. The meta-reviewers thus recommend to accept this paper, and encourage the authors to add all new experiments and make the presentation more clear in the camera ready.
confidence: Certain
Decision: Accept