Efficient In-Context Visual Learning with Trident Block and Cross Blocks

Authors: Hao Wen, Hu Xiao, Antonio Bianchi
Primary area: optimization
Keywords: Visual Prompting Large Vision Model, Efficient In-context Learning, Vision Transformers

Abstract:
Visual prompt-based large vision models exhibit remarkable performance in a range of vision tasks. However, visual prompting large vision models are computationally intensive and resource-demanding due to their large parameter sizes and the complexity of processing visual prompts, resulting in inefficiencies in speed and memory usage. To tackle these challenges, we propose the Efficient Painter model, which leverages a novel context-aggregated attention based trident block to alleviate cross-task gaps and reduce memory and computation overhead. Furthermore, we introduce a cross-blocks feature union module to capture global contextual information at different levels and speed up training. This architecture mitigates training costs and memory requirements during inference. Our model strikes a balance between speed and memory efficiency, achieving a 19$\times$ reduction in FLOPs. Moreover, our model is 9$\times$ smaller in model size and runs 4.1$\times$ and 27$\times$ faster during training and inference, respectively. Comprehensive experiments demonstrate that our design effectively processes additional visual prompts and outperforms baseline methods on standard benchmarks like \textit{SIDD} and \textit{LoL} in zero-shot settings, improving performance by 0.4\% and 1.2\% respectively.

Decision: reject
Reviewer scores: 3, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729836882988

thread_type:
official_review

<thread_object>
creator: reviewer_jqqw

<content>
summary:
This paper mainly focuses on visual in-context learning, specifically the previous work Painter. The authors point out that the previous in-context visual learning methods require unaffordable computational resources. To this end, the authors propose a more efficient network structure specifically designed for visual in-context learning, which can alleviate cross-task gaps and reduce memory and computation overhead.

strengths:
1. The method is clearly explained.
2. The idea of reducing computational complexity for visual in-context learning is interesting and practical.

weaknesses:
1. I am not sure how the proposed CAA is implemented. It seems the information is sequentially propagated between different attention heads.
2. It seems the proposed method may be overly complex. To achieve the goal of better efficiency, the authors rebuild almost all basic modules in the original Painter. While there are several ablation studies showing the efficacy of these designs, these experiments only focus on the high-level design. It is still not clear if it is necessary to build the modules the same as in L266 and L306.
3. I am not sure why the authors have to include the Painter+EViT with all NaN restuls in Tab.5?
4. I wonder if the author can compare the proposed method with methods such as distillation and pruning?
5. It would be better if the authors can test the proposed methods on different in-context learning tasks. For example, use MAE-VQGAN [1] for segmentation, detection and colorization.



[1] Bar A, Gandelsman Y, Darrell T, et al. Visual prompting via image inpainting[J]. Advances in Neural Information Processing Systems, 2022, 35: 25005-25017.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730442165432

thread_type:
official_review

<thread_object>
creator: reviewer_fa3m

<content>
summary:
While Visual prompt-based large vision models exhibit remarkable performance in a range of vision tasks, visual prompting large vision models are computationally intensive and resource-demanding due to their large parameter sizes and the complexity of processing visual prompts. In light of this view, the authors propose the Efficient Painter model, leveraging a novel context-aggregated attention-based trident
block to alleviate cross-task gaps and reduce memory and computation overhead.

strengths:
1 The authors propose the Efficient Painter model which is faster and more memory-inefficient than Painter [Ref1].
2 The authors present ideas clearly, making the paper easy to follow.

weaknesses:
1 The CAA passes the feature from the previous head to the next head. This will make heads can't run in parallel leading to increasing running time. Could you prove the effectiveness of this design compared to running them in parallel? More discussion should be added.

2 In CAA, the authors use a smaller dimension for Q and K. Could authors apply the same idea to ViT of Painter and check whether it can get similar results with fewer parameters? This actually challenges the novelty/efficiency of this proposed method, if setting the dimensions of Q, K to a lower dimension to ViT of Painter can reach similar results, then the proposed CAA might be a insufficient design.

3 The specialized models in Table 5 are outdated. The authors should compare the proposed model with the newest best-specialized methods.

4 Could authors evaluate the proposed model with other datasets (for generality), not only the datasets in Painter [Ref1]?

5 Why are the results of Painter (Efficient-ViT-M5) all NA in Table 5? Table 5 is extremely incomplete in this case, it becomes really hard to evaluate the performance across different methods systematically. I understand that  the benchmark might be hard to build, however, when reporting related performance, the relative results should be reported for completeness.

[Ref1]: Images Speak in Images: A Generalist Painter for In-Context Visual Learning.

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598604944

thread_type:
official_review

<thread_object>
creator: reviewer_f6ai

<content>
summary:
This paper aims to address the computational inefficiencies and slow speeds resulting from the large number of parameters and complex prompt processing in visual promoting large vision models. The paper proposes a context-aggregate attention based on the trident block and a cross-blocks feature union. Compared to previous models, it is 9× smaller in model size and runs 4.1× and 27× faster during training and inference, respectively.

strengths:
+ From the experimental results, the architecture of this paper shows certain improvements in computational efficiency and speed.
+ The designed architecture can be applied to multiple tasks, including denoising and low-light enhancement tasks.

weaknesses:
- There are grammar issues in the writing, such as in the related work on the second page, ‘Traditional approaches such as DeiT, and MobileViT  leverage knowledge distillation or architectural downsize fail to enhance 
actual inference speed or throughput.’
- Lack of innovation. The core contribution of this article is a lightweight architecture, but many of the operations that affect parameter changes are existing methods, such as integrating existing DWConv or reducing the dimension of queries.
- Non-standard use of abbreviations in the writing.

questions:
1. What is the innovation in this article? In addition to existing dimension reduction methods, what is the innovative dimension reduction strategy proposed in this article?
2. From the description in Section 4.2 on the fourth page and Figure 3, The CAA module primarily computes attention weights for different channel features and enhances the response of important features. It then accumulates and stacks important features from different channels. How are deep connections established between different channels? More prominently observed in the current design is feature aggregation rather than the establishment of correlations.
3. The mention of aligning channel attention mechanisms on lines 204-205 can reduce computational costs. The related reasons need to be further elaborated with specificity.
4. The article mentions that in the design of sub-modules, smaller dimensions are used to reduce the parameter count. This is indeed a simple and effective strategy, but further experiments and analysis are needed for the core innovative operations that can reduce the parameters.
5. Please avoid using abbreviations for the first occurrence whenever possible.
6. In lines 310-316 on page six, it is explained that aligning the channels of the sub-decoder can alleviate gradient explosions. Please provide specific reasons why this operation is believed to mitigate gradient explosions.
7. The alpha and beta variables used in section 4.4 are not clearly elucidated regarding the numerical selection strategy. Experimental analysis of the parameters chosen is lacking.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627271958

thread_type:
official_review

<thread_object>
creator: reviewer_xvqm

<content>
summary:
The authors propose an efficient large vision model (LVM) called Efficient Painter, designed to reduce memory and computational demands for visual tasks. Key contributions include a context-aggregated attention (CAA) mechanism that allows dynamic context processing with lower resource usage and a cross-block feature union (CBFU) module that improves multi-level global context processing, effectively speeding up training. The authors report significant reductions in memory consumption and substantial speed improvements during inference.

strengths:
If understood correctly, the Efficient Painter model demonstrates substantial reductions in computational load and memory usage compared to its baselines. There appears to be a meaningful architectural novelty, especially in the design of the CAA and CBFU components, which seem well thought out and targeted toward efficiency.

weaknesses:
The paper’s writing quality is quite limited, making it difficult to follow and understand key ideas, especially in the introduction, motivation, and experimental sections. The paper is filled with confusing sentences, which need substantial improvement for clarity and readability.


Moreover, the paper appears unfinished, with placeholders such as "NaN" in the result tables, which is especially noticeable in Table 5. The experimental results are presented in a messy and confusing manner, further detracting from the paper's overall quality.

questions:
In its current state, I don’t believe this paper meets the standards for publication and would require significant revisions to be considered.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732695022091

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response about paper revision details.

comment:
In our revised manuscript, we have made several improvements to enhance clarity and better articulate the motivations and challenges of our work:

1. **Introduction – Second Paragraph:**  
We have corrected grammatical issues to improve readability and have provided a more precise exposition of previous works and their limitations. Specifically, we have clarified that prior approaches not only fail to achieve overall efficiency but also overlook their applicability to visual prompt-based Large Vision Models (LVMs). This revision ensures a clearer understanding of the existing gaps that our research aims to address.

2. **Introduction – Third Paragraph:**  
To highlight the primary challenge of our study more prominently, we underscore the delicate balance we aim to achieve between overall model efficiency (computation and memory efficiency) and the requisite in-context visual learning capacity for visual prompt-based models at the beginning of the paragraph. By tackling this challenge, we make our research objectives more immediately apparent to the reader.

3. **Section 3 – Motivation Part:**  
We have added a new paragraph to the motivation section to delineate the process through which we identified the core problem. This addition clarifies the problem's significance and relevance and introduces our design stemming from this challenge.

4. **Section 5.1 - Implementation Details:**  
We have revised this part to give more details about our selected parameter values in 

5. **Abbreviation Clarifications:**  
We have revised Section 5.1 Implementation Details to explain the selected parameter values of our context-aware reconstruction loss function.

We believe these revisions significantly enhance the manuscript by providing a clearer narrative and stronger justification for our research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733201406521

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Awaiting Feedback on Rebuttal Submission

comment:
Dear Reviewers,

I hope this message finds you well. We eagerly await your feedback on our rebuttal and would greatly appreciate any additional comments or follow-up questions you might have regarding our work. Your insights will be invaluable for improving our research.

Best,
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Efficient Painter, a large vision model designed to address computational inefficiencies in visual tasks. The key innovations include: (i) Context-Aggregated Attention: A mechanism for dynamic context processing that reduces memory and computational demands. (ii) Cross-Block Feature Union: A module for enhanced multi-level global context processing, improving efficiency during training and inference. The Efficient Painter model achieves significant reductions in computational load and memory usage compared to its baselines. However, several concerns remain. The paper’s writing quality is limited, making it challenging to follow, and the description of its main contribution, the CAA module, remains unclear. Considering the reviewers' feedback, the AC has decided to reject this paper.