Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures

Authors: Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, Wenhai Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: RWKV, Visual Perception, Linear Attention

Abstract:
Transformers have revolutionized computer vision and natural language processing, but their high computational complexity limits their application in high-resolution image processing and long-context analysis. This paper introduces Vision-RWKV (VRWKV), a model that builds upon the RWKV architecture from the NLP field with key modifications tailored specifically for vision tasks. Similar to the Vision Transformer (ViT), our model demonstrates robust global processing capabilities, efficiently handles sparse inputs like masked images, and can scale up to accommodate both large-scale parameters and extensive datasets. Its distinctive advantage is its reduced spatial aggregation complexity, enabling seamless processing of high-resolution images without the need for window operations. Our evaluations demonstrate that VRWKV surpasses ViT's performance in image classification and has significantly faster speeds and lower memory usage processing high-resolution inputs. In dense prediction tasks, it outperforms window-based models, maintaining comparable speeds. These results highlight VRWKV's potential as a more efficient alternative for visual perception tasks. Code and models are available at~\url{https://github.com/OpenGVLab/Vision-RWKV}.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730246056295

thread_type:
official_review

<thread_object>
creator: reviewer_weiy

<content>
summary:
The authors describe a Vision-RWKV architecture that employs novel techniques like bi-directional RWKV, quad token shifting method Q-shift etc. These techniques helps Vision-RWKV  architecture surpass window-based ViTs and comparable to global attention ViTs along with lower FLOPs, lower GPU memory cost and faster processing as shown in Figure 1. Although the MAE finetuning is not as straightforward as typical finetuning on downstream task, overall this paper has a good contribution for the vision research community.

strengths:
- Novel contribution for quad-directional token shifting called Q-shift. This essentially increases the models range of semantic understanding
- Authors expanded causal RWKV to a bidirectional global RWKV. They modified the exponent in the RWKV attention that leads to transforming the absolution positional bias to relative bias.
- Well written paper with through experimentation including MAE pretraining

weaknesses:
- The authors implemented a bidirectional shift operation that removed the vertical shift in Q-shift, thereby enabling for MAE pretraining. IMO this is a source of complexity. As a result, MAE finetuning needs to be done in Q-shift manner

questions:
- Can you please clarify what you meant in line 527-528 by task fine-tuning using Q-shift manner?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730588296922

thread_type:
official_review

<thread_object>
creator: reviewer_cuxv

<content>
summary:
Authors propose an approach that adapts NLP models for RWKV to vision tasks, termed as V-RWKV. Their proposed method could be looked upon as a cost effective solution than ViTs. They propose Quad-directional Shift and change the causal attention to bidirectional one towards learning locl concepts that are more relevant in vision rather than simple text. They evaluate their approach on the image classification, object detection and semantic segmentation tasks.

strengths:
- Comprehensive results across three tasks.
- Interesting direction to replace ViTs with other efficient techniques that provide on-par or better performance.

weaknesses:
- The gains in efficiency seem to be relatively minor, i.e., it is not an order of magnitude which still brings the question whether it is worth exploring these type of models to replace ViTs to begin with or not.

 For example, in Table 2 ViT-L vs. VRWKV-L 191.1 vs 189.5G FLOPS and parameters at 309.5M vs. 334.9M respectively. I think the reduction in FLOPS when scaling to Large variant seems to be around 3G. Similarly in Table 4, FLOPS 446.8 vs. 421.9G with the expense of an increase in the number of parameters. I am not expert in such type of methods focused on improving efficiency but I do not see the results are impressive enough to show the benefit from the V-RWKV design, especially that it is increasing the parameters.

questions:
Clarifying the practical benefit from their proposed approach and why not simply use vanilla ViTs considering the current gains are not that considerable when looking at Tables 2 and 4. Hence, the reason I am leaning towards a marginal reject but since the method seems to provide interesting direction and their results for detection seems sufficiently good I am not going lower.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730861568480

thread_type:
official_review

<thread_object>
creator: reviewer_qwnu

<content>
summary:
The paper introduces a new network architecture, VISION-RWKV, a vision-adapted version of the RWKV network from the NLP community, which employs an RNN-based linear attention mechanism. The authors have made necessary adaptations to the RWKV to better suit visual tasks, recognizing the differences between vision and language processing tasks.

strengths:
1.The paper is well-written and easy to follow;

2.Extensive empirical evidence supports the effectiveness of the model, indicating its practical application potential;

3.The adaptation of RWKV for visual tasks goes beyond mere transfer, incorporating modifications that enhance its suitability for image processing.

weaknesses:
Major：Although the paper discusses RWKV and mamba as prominent RNN-based linear attention models transitioning from NLP to computer vision, it lacks a direct comparison with the mamba (vision) model. Such a comparison would be valuable for assessing the respective strengths and weaknesses of each model in the field of computer vision.

Minor：From an innovation perspective, the approach of adapting other domains' mature architectures to computer vision, similar to what was done post-transformer, appears somewhat incremental.

questions:
see weaknesses
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings

The paper presents VISION-RWKV, an adaptation of the RWKV network from NLP to vision tasks, incorporating an RNN-based linear attention mechanism. Reviewer QwNu notes that the authors have made necessary modifications to suit visual tasks, while cUXV highlights the introduction of Quad-directional Shift and bidirectional attention to better capture local concepts in vision. weiY emphasizes the novel techniques like bi-directional RWKV and quad token shifting, which help the architecture surpass window-based ViTs in efficiency and performance.

(b) Strengths

The paper is well-written and provides extensive empirical evidence supporting the model's effectiveness, as noted by QwNu. cUXV appreciates the comprehensive results across multiple tasks and the exploration of efficient alternatives to ViTs. weiY highlights the novel contributions, such as the quad-directional token shifting and the expansion of causal RWKV to a bidirectional global RWKV, which enhance the model's semantic understanding and efficiency.

(c) Weaknesses

A major weakness identified by QwNu is the lack of direct comparison with the mamba (vision) model, which would help assess the model's strengths and weaknesses in computer vision. cUXV points out that the efficiency gains are relatively minor, questioning whether the model is a worthwhile replacement for ViTs given the modest reduction in FLOPs and increase in parameters. weiY mentions the complexity introduced by the bidirectional shift operation, which complicates MAE finetuning.

(d) Decision Reasons

The decision to accept the paper is based on its novel contributions to adapting NLP models for vision tasks and the comprehensive empirical validation provided. Despite some concerns about the incremental nature of the approach and the modest efficiency gains, the paper offers a promising direction for efficient vision models, as highlighted by all reviewers. The strengths in presentation, empirical results, and novel techniques outweigh the weaknesses, leading to an overall positive assessment.