Title: Learning 1D Causal Visual Representation with De-focus Attention Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: visual causal modeling; vision model; representation learning;
Abstract: Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using 1D causal modeling. We identify an "over-focus" issue in existing 1D causal vision models, where attention overly concentrates on a small proportion of visual tokens. The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns. During training, large and scheduled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks are introduced. These two strategies encourage the model to attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code shall be released.

Learning 1D Causal Visual Representation with De-focus Attention Networks

Abstract

Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using 1D causal modeling. We identify an "over-focus" issue in existing 1D causal vision models, where attention overly concentrates on a small proportion of visual tokens. The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns. During training, large and scheduled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks are introduced. These two strategies encourage the model to attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code shall be released.

1 Introduction

Due to inherent modality differences, vision and language models have evolved into distinct heterogeneous architectures. A key difference is that images usually require 2D non-causal modeling, while texts often utilize 1D causal modeling. This distinction presents a significant challenge in constructing unified multi-modal models. Many existing multi-modal models [37, 3, 11, 5, 29] have to train vision and language encoders separately before combining them. A crucial question in advancing unified vision-language modeling is how to represent images using 1D causal modeling.

Following the success of causal language modeling (e.g., GPT-series [52, 53, 8]), some studies [10, 17] have explored causal modeling in the vision domain. These efforts primarily focus on auto-regressive visual pre-training by adding a causal attention mask to standard Transformers [15]. Despite numerous attempts, the gap between 1D causal and 2D non-causal vision models remains unbridged. As shown in Sec. 5, many 1D causal vision models, such as State Space Models [61, 21] and causal ViTs [15], perform inferiorly compared to their modified 2D non-causal counterparts.

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

(a) Attention Map

(b) Gradient Map

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

In this paper, we identify an "over-focus" issue in existing 1D causal vision models. Fig. 1 visualizes the attention and gradient maps of several ImageNet-trained networks, including 2D non-causal ViT, 1D causal ViT, and 1D causal Mamba. The results show that in 1D causal vision models, the attention patterns are overly concentrated on a small proportion of visual tokens, especially in the deeper network layers close to the output. This phenomenon hinders the model from extracting diverse visual features during the forward calculation and obtaining effective gradients during the backward propagation. We refer to this phenomenon as the "over-focus" issue in 1D causal vision models.

To address the issue, a "de-focus attention" strategy is introduced. The core idea is to guide the network to attend to a broader range of tokens. First, learnable bandpass filters are introduced to filter different sets of token information, and then combine their attention patterns. This ensures that even if over-focusing occurs, the attention pattern remains diverse due to the varying constraints of each set. Second, optimization strategies are improved. A large drop path rate is employed to encourage the network to attend to more tokens within one layer, rather than relying on depth to get large receptive fields. For tasks requiring global understanding (e.g., image classification), an auxiliary loss is applied to the globally pooled features to enhance the effective gradients for all tokens in the sequence.

Extensive experiments demonstrate the effectiveness of our De-focus Attention Networks for 1D causal visual representation learning. It achieves comparable or even superior performance to 2D non-causal ViTs across various tasks, including image classification, object detection, and image-text retrieval. Our method has been validated on both ViTs and Mambas. Our contributions can be summarized as follows:

We identify the over-focus issue in 1D causal visual modeling, where the model overly focuses on a small proportion of visual tokens in the deeper layers of the network.

To address this issue, we propose a "de-focus attention" strategy. This involves integrating learnable bandpass filters into the existing attention operators to achieve diverse attention patterns. Additionally, we introduce a large drop path probability and an auxiliary loss on average pooled features during training to enhance network optimization.

Our De-focus Attention Networks have demonstrated that 1D causal visual representation can achieve performance equivalent to 2D non-causal representation in tasks requiring global perception, dense prediction and multi-modal understanding tasks.

2 Related Work

State Space Models (SSMs) are intrinsically causal models, originated from the classic Kalman filter[31]. SSMs describe the behavior of continuous-dynamic systems, enabling parallel training and linear complexity inference. [24] proposed a Linear State Space Layer, merging the strengths of continuous-time models, RNNs and CNNs. HIPPO [22] introduced methods to facilitate continuoustime online memorization. Building on these foundations, Structured SSMs (e.g., S4 [23], Diagonal State Spaces (DSS) [25], S5 [58]), Recurrent SSMs (e.g., RWKV [49], LRU [45]) and Gated SSMs (e.g., GSS[42], Mega[41]) further expand the SSMs landscape. Notably, Mamba [21] excels in long-sequence modeling with its selective scan operator for information filtering and hardware-aware algorithms for efficient storage of intermediate results. As SSMs have drawn more and more attention recently, they also have extensive applications in domains that need long sequences processing such as medical [44, 6], video [34], tabular domain [2] and audio/speech [20, 30]. These successes achieved by SSMs prompt us to explore their application in visual modeling within this causal framework.

2D Non-Causal Visual Modeling are dominant in vision domains. Convolutional Neural Networks (CNNs), operating in a 2D sliding-window manner [33] with inductive biases such as translation equivariance and locality, have demonstrated remarkable adaptability [32, 57, 62, 71, 28, 27, 63]. Vision Transformers (ViTs) [15] utilize a non-causal self-attention mechanism, enabling global receptive fields. Subsequent improvements focus on enhancing locality[39], refining self-attention mechanisms[74, 4], and introducing novel architectural designs [69, 70, 46, 26], while maintaining non-causality. Recent advances in State Space Models (SSMs) have inspired new vision backbone networks, such as VMamba [38], Vision Mamba [75], and Vision-RWKV [16]. Although SSMs are inherently causal, these works incorporate non-causal adjustments to enhance vision performance. VMamba introduced a four-way scanning strategy, Vision Mamba incorporated bidirectional SSMs, and Vision-RWKV adopted bidirectional global attention and a special token shift method. These designs of arrangement hinder the unification of vision and language modeling.

1D Causal Visual Modeling. While 1D causal modeling has primarily been used in language[7] and speech[66], it has also been explored for visual representation. In recent years, the causal visual modeling has been adopted in Transformer-based visual generation methods such as Image Transformer [47] and VQGAN [18]. These models first discretize images into grids of 2D tokens, which are then flattened for auto-regressive learning. However, their performance significantly lags behind [48, 1]. Of particular interest, iGPT [10] also employed auto-regressive causal modeling for pre-training, followed by linear probing or fine-tuning to achieve commendable results in various downstream tasks, though still worse than non-causal models [14, 9]. Similarly, AIM [17] applied causal masks to the self-attention layers, and pre-trained with an auto-regressive objective, showing good scaling potential. Despite many attempts, the performance gap between 1D causal and 2D non-causal vision models remains.

3 Preliminary

Transformers [67] with causal attention consist of multiple attention layers. Each attention layer computes a weighted average feature from the preceding context for every input token, with aggregated features weighted by the similarities between tokens. The attention layer is written as:

where s and t are indexes of different locations of the input sequence, Q i , K i , V i are projections of input x i , and y t is the output of the attention layer.

State Space Models (SSMs) are classical latent state models widely used in various scientific fields [44, 6, 34, 68, 50, 73]. Originally, SSMs are defined for continuous signals, mapping a 1D input signal x ( t ) ∈ R to a latent state h ( t ) ∈ R N and computing the output y ( t ) ∈ R from the latent state. To apply SSMs to discrete sequences, their discrete form is defined as

where A t ∈ R N × N , K t ∈ R N × 1 , Q t ∈ R N × 1 are parameters of the system. Note that we use notations different from the original SSMs ( K t , Q t instead of B t , C t ) for a better comparison with Transformers above.

SSMs can also be transformed into another formulation by expanding the recurrent process:

This formulation resembles the conventional attention module and explicitly reveals the relationship between different inputs in the sequence. We use this form for further discussion.

There are multiple variants of SSMs, mainly differing in the parameterization of ( A t , K t , Q t ) . We introduce some well-known SSMs and discuss their differences below.

RetNet [61] and Transnormer [51] employ a fixed A and convert it into an exponential decay (defined by λ ∈ R ) with a relative positional embedding (defined by θ ∈ R N ):

Mamba [21] and S4 [23] use zero-order hold (ZOH) rule for discretization, introducing a time-scale parameter ∆ t . The discretization rule is A t = exp(∆ t ˆ A ) and K t = (∆ t ˆ A ) -1 (exp(∆ t ˆ A ) -I ) · ∆ t ˆ K t , where ˆ A and ˆ K t are learnable parameters. S4 uses data-independent parameters, while Mamba computes these parameters based on inputs. The formulation can be written as:

4 Method

This section introduces our De-focus Attention Networks for 1D causal visual representation learning. Sec. 4.1 elucidates the main components of De-focus Attention as Learnable Bandpass Filter, while Sec. 4.2 further discusses two training strategies adopted in De-focus Network. The overall architecture of our model is presented in Fig. 2.

4.1 De-focus Attention with Learnable Bandpass Filter

To de-focus on a few salient tokens and enhance the extraction of diverse features from images, learnable bandpass filters are incorporated to first adaptively filter diverse information from the input and their attention patterns are then combined together. Due to the varying contents from different filters, the attention can still be diverse even if the over-focus issue happens.

These bandpass filters can be implemented through exponential spatial decay and relative position embedding similar to those in RoPE [59] and xPos [60], both of which are further made learnable. Our results demonstrate that these factors are crucial for the model to learn diverse attention patterns.

To show how spatial decay and relative position embeddings work as a bandpass filter, consider a simplified version of 1D causal attention equipped with them:

where x ( s ) is the input signal at time s . e λ ( t -s ) ( λ < 0 ) represents the simplest version of exponential spatial decay, which is also used by RetNet [61] and Transnormer [51]. e iθ ( t -s ) is the relative position embedding proposed by RoPE [59] and xPos [60]. Here, the continuous time domain is used to facilitate derivation without losing generality.

The above equation implies a time domain convolution between e λ ( t -s ) e iθ ( t -s ) and x ( s ) . By transforming Eq. (6) into the frequency domain and using ˆ x ( ω ) , ˆ y ( ω ) to represent Fourier transform of corresponding x ( s ) , y ( t ) , the frequency domain expression becomes:

This equation indicates that Eq. (6) is actually a bandpass filter, where θ is its center frequency and λ controls its passband width. Eq. (7) presents some interesting properties of 1D causal modeling:

If there is no spatial decay or relative position embedding (e.g., Transformers without Softmax ), Eq. (6) will degenerate to a summation of the inputs, losing the ability to filter spatial information;

If there is no relative position embedding (e.g., Mamba), 1D causal attention will perform low-pass frequency filtering, causing the query to miss the full information of features and resulting in information loss;

If only relative position embedding is used, it will degenerate to specific frequency selecting, which may also result in information loss;

If both spatial decay and relative position embedding are used (suggested), 1D causal attention will act as a bandpass filter. For a given query, when different components of the feature vector use different center frequencies (i.e., different θ ) and passbands width (i.e., different λ ), a more diverse range of information will be gathered. Due to the diverse frequency passbands, even if the over-focus issue occurs, the attention remains diverse across different components of the feature vector.

To fully leverage the bandpass filtering mechanism, a learnable one is preferable. Experiments demonstrate that performance worsens when values are fixed or not well set.

Our De-focus Attention can be incorporated into different architectures. Below, examples of its implementation in causal ViT and Mamba are presented.

De-focus Causal ViT. ViT has additional attention activation (i.e., Softmax ) compared with SSMs. Learnable exponential spatial decay and learnable relative position embeddings are appended before applying the attention activation, following the common implementation of RoPE, as shown below:

where the terms of e λ and e iθ function as the learnable bandpass filter.

De-focus Mamba. Since Mamba already has learnable and data-dependent exponential spatial decay, only attachment of learnable relative position embeddings to it is necessary:

where the terms of ˆ A ∆ and e iθ function as the learnable bandpass filter.

4.2 De-focus Attention in Network Optimization

During network training, performance of 1D causal models can be further enhanced with improved optimization strategies. Specifically, using a large drop path rate with a linear schedule helps the model attend to more tokens in each layer. Additionally, applying an auxiliary loss to the global average feature mitigates the under-learning of features in deeper layers. The effects of these training strategies are illustrated in Fig. 3.

Large Drop Path Rate with Linear Schedule. Two ways for the final prediction to access information from previous inputs are observed: 1) Network Depth : Progressively looking forward a few tokens in each layer until reaching the earliest tokens; 2) Intra-Layer Attention : Using the attention mechanism within the same layer to directly capture information from more distant tokens.

Our goal is for each layer to fully utilize the existing attention mechanism to capture more and further information in one layer. Therefore, a large drop path rate (up to 0.7) is employed to encourage the network to rely less on depth and rely more on training the attention mechanism in each layer. Since a large drop path rate may hinder the model when only a few features are learned, i.e., at the beginning of training, a linear schedule that gradually increase the drop path rate is followed.

Fig. 3 demonstrates the effectiveness of this strategy, indicating that without large drop path strategies, the network tends to prefer to see less tokens in one layer and rely on network depth to increase the receptive field.

Auxiliary Loss for Image Classification. To address over-focus issue in backward gradients, an auxiliary loss is proposed to enrich the gradients variety and aid in the representation learning of

N

×

…

…

…

…

<CLS>

Loss

De-focus Causal Attention Block

Auxiliary Loss

FC

Average Pooling

FC

Causal Attention Layer

Learnable

Decay

exp(1 ⋅)

"

#∆

Query Tokens Output

∆

(

)

& '

!

(

Learnable

RoPE

exp 567 ×

Learnable

Bandpass

Filter

unnoticed tokens. The final representations of all image tokens (excluding the final <CLS> token) are averaged and fed into an additional linear layer. The auxiliary loss function is defined as the cross-entropy loss between the output of the additional linear layer and the ground truth label. This approach helps enrich the backpropagated gradients, thereby addressing the over-focus issue.

As shown in Fig. 3, after applying the auxiliary loss strategy, the backward gradients are significantly improved in its density, globality, and diversity in deeper layers.

4.3 Overall Architecture.

The overall architecture of our De-focus Attention Networks is illustrated in Fig. 2. The following explains how learnable bandpass filters and optimization strategies are integrated into existing models.

De-focus Attention Blocks. Each block consists of three main parts, which are a projection layer, a learnable bandpass filter, and a causal attention layer. The input tokens are first projected into a query Q , a key K . ∆ is data-dependent in De-focus Mamba, while is set to 1 in De-focus ViT. Other projections may be required by the causal attention layer. The block has learnable decay parameters A (corresponds to λ in De-focus ViT) and learnable relative position embedding parameters θ . Given these learnable parameters, exponential spatial decay and relative position embedding are computed as illustrated in Eq. (8) and Eq. (9). Q,K and the exponential spatial decay term are integrated into the causal attention layer. Thus, the outputs of a De-focus Attention block aggregate the input information filtered by a series of learnable bandpass filters.

De-focus Attention Networks. Given an image, our De-focus Attention Networks first transform it into a sequence of image tokens and append an extra <CLS> token to the sequence end. The whole network then stacks N De-focus Attention blocks to process the input sequence. Each block is equipped a drop path rate, which increases linearly during training. In the final layer, the <CLS> token is fed through a linear layer and used to compute a cross-entropy loss with class labels. All image tokens, excluding the final <CLS> token, are averaged and passed through a separate linear layer. An auxiliary cross-entropy loss is applied to this projected averaged feature. The two losses are then added with equal weights to form the final loss function.

5 Experiments

5.1 Experiment Setup

Implementation Details . The De-focus Attention mechanisms are integrated into Mamba, RetNet, and ViT, referred to as De-focus Mamba, De-focus RetNet, and De-focus ViT, respectively. To improve optimization stability, λ = -exp( ˆ λ ) is used and ˆ λ is the parameter to be optimized. In De-focus ViT and De-focus RetNet, different λ s are assigned to different heads. Mamba inherently implements data-dependent decay ˆ A ∆ , where ˆ A is a learnable parameter and ∆ is a projection from the input. The drop path rate increases following a linear schedule from 0.1 to 0.7.

Image Classification. ImageNet-1k [13] is used, which contains 1.28M images for training and 50K images for validation. The training recipe of DeiT [64] is followed. The small- and base-size models are trained on ImageNet for 300 epochs. The large-size model is firstly pre-trained on ImageNet-21k [55] for 90 epochs, and then fine-tuned on ImageNet-1k for 20 epochs. The AdamW optimizer [40] with a peak learning rate of 5e-4, a total batch size of 1024, a momentum of 0.9, and a weight decay of 0.05 are used. These models are trained on 32 Nvidia 80G A100 GPUs for 30 hours.

Object Detection. The MS-COCO dataset [36] and the DINO detection framework [72] are used, with different networks serving as the backbones. The De-focus Attention Networks implemented here are pre-trained on ImageNet-1K dataset for 300 epochs. These models are trained on 16 Nvidia 80G A100 GPUs for 20 hours.

The entire network is fine-tuned using both a 1 × schedule (12 epochs) and a 3 × schedule (36 epochs). The base learning rate is set to 2e-4, with a multi-step learning rate strategy employed to decrease it by a factor of ten after 11 epochs (1 × schedule) or after 27 and 33 epochs (3 × schedule). The weight decay and the total batch size is set to 1e-4 and 16, respectively.

Contrastive Language-Image Pre-training (CLIP). The Laion-400M dataset [56] is used for pre-training. Strategy introduced in OpenCLIP [12] is followed to train the model for 32 epochs. The zero-shot classification performance is evaluated on ImageNet-1K. The AdamW optimizer [40] is employed with a peak learning rate of 5e-4, a total batch size of 32768, a momentum of 0.9, and a weight decay of 0.1. These models are trained on 128 Nvidia 80G A100 GPUs for 128 hours.

5.2 Main Results

Image Classification . The classification results are presented in Table 1. Evaluation of different types of De-focus Networks at various scales is conducted, with comparisons to both causal and non-causal models. The results show that previous causal models have inferior performance. In contrast, our model defies this trend, significantly outperforming other 1D causal models and achieving comparable performance to 2D non-causal models.

Notably, the De-focus Attention mechanism works well across various networks, e.g., Causal ViT, Mamba, and RetNet. And as the model size increases from small to large, it remains on par with the 2D non-causal ViTs.

Object Detection . As shown in Table 2, De-focus Mamba remarkably outperforms non-causal models such as DeiT and ResNet-50. This trend of superior performance persists even with an increasing number of training epochs. Additionally, excellent performance on the AP box 75 metric may suggest that De-focus Attention Networks are more effective at fine-grained localization.

Image-text CLIP Pre-training . The model is pre-trained using OpenCLIP to demonstrate its outstanding performance on large-scale image-text training. As shown in Table 3, the model performs comparably to 2D non-causal models. We also report cross-modal retrieval results in Table 4, which further validate that the learned causal feature can achieve similar results with non-causal features. These results indicate that the model has a similar scaling law to non-causal ViTs on larger dataset, demonstrating its robustness and scalability across various of tasks and datasets. Additionally, this experiment demonstrates the potential of 1D causal modeling for unified vision-language modeling.

5.3 Ablation Study

Learnable Bandpass Filter. As discussed in Sec. 4.1, exponential spatial decay and relative position embedding (RoPE) together act as a bandpass filter. Tab. 5(a) shows the effects of different configurations. When decay is not used, the performance significantly deteriorates. Employing learnable decay leads to an improvement of approximately 0.5% compared to fixed decay, while learnable RoPE can further enhance performance by 0.8%. In contrast, the data-dependent decay used in Mamba only results in a marginal improvement of 0.1%. These results indicate the integration of learnable decay and RoPE are necessary for good performance.

(b) Reception field w/ Scheduled DropPath

(linear(0.1, 0.7))

(constant 0.1)

(c) Gradient map w/o Auxiliary Loss

(d) Gradient map w/ Auxiliary Loss

Drop Path. Tab. 5(b) shows the performance of different drop path strategies, with rates ranging from 0.1 to 0.7. The best performance is achieved with a scheduled drop path rate linear(0 . 1 , 0 . 7) . Fig. 3(a)-(b) visualize the receptive field of the 22 nd layer of the network. The results demonstrate that using a large and scheduled drop path rate strategy allows for larger receptive field and helps capture more dense semantic details.

Auxiliary Loss. Tab. 5(c) compares various implementations of the loss function, which are generated from <CLS> token only, average token only, concatenation of <CLS> token and average token, and <CLS> token with auxiliary average token. The results reveal that the average pooled feature alone performs poorly in training the network. It may result from the fact that previous tokens often have incomplete information. However, it serves as an effective auxiliary component, thereby enhancing the network training. The visualization of gradient maps at the 22 nd layer of the network are shown in Fig. 3(c)-(d). When training with auxiliary loss, the density, globality, and diversity of backward gradients are significantly improved.

6 Conclusion

We propose De-focus Attention Networks to enhance the performance of causal vision models by addressing the issue of over-focus in them. The over-focus phenomenon, i.e. attention pattern is overly focused on a small proportion of visual tokens, is observed both during the forward calculation and backpropagation. These De-focus models incorporate a decay mechanism and relative position embeddings, functioning together as diverse and learnable bandpass filters to introduce various attention patterns. The models are trained with a large scheduled drop path rate and auxiliary loss to enhance the density, globality, and diversity of backward gradients. A series of De-focus models based on Mamba, RetNet, and ViT significantly outperform other causal models and achieve comparable or even superior performance to state-of-the-art non-causal models. By implementing the de-focus

strategy, our work bridges the performance gap between causal and non-causal vision models, paving the way for the development of state-of-the-art unified vision-language models.

Limitations. While the De-focus Attention has achieved very promising results, some differences between causal and non-causal vision models still need further exploration. For example, compared with non-causal models, De-focus Mamba-Base excels at AP box 75 metric but is inferior at AP box 50 metric on object detection. It is worth studying how causal models perform dense prediction tasks. Furthermore, currently we only explore 1D causal modeling in purely visual settings, while exploring unified 1D causal modeling of vision and language remains to be verified.

Broader Impacts. De-focus Attention Networks demonstrate the potential of building unified multi-modal models, so they may also cause similar problems as previous works. For instance, it may also require huge computational resources, contain dataset bias, and raise ethical concerns when being adopted for training large multi-modal foundation models.

Acknowledgements. The work is partially supported by the National Natural Science Foundation of China under Grants 62321005.

References

Alpha-vllm.large-dit-imagenet. https://github.com/Alpha-VLLM/LLaMA2-Accessory/ tree/f7fe19834b23e38f333403b91bb0330afe19f79e/Large-DiT-ImageNet , 2024.

M. A. Ahamed and Q. Cheng. Mambatab: A simple yet effective approach for handling tabular data. arXiv preprint arXiv:2401.08867 , 2024.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS , 2022.

A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, et al. Xcit: Cross-covariance image transformers. NeurIPS , 2021.

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.

M. Ballarotto, S. Willems, T. Stiller, F. Nawa, J. A. Marschner, F. Grisoni, and D. Merk. De novo design of nurr1 agonists via fragment-augmented generative deep learning in low-data regime. Journal of Medicinal Chemistry , 2023.

Y. Bengio, R. Ducharme, and P. Vincent. A neural probabilistic language model. NeurIPS , 2000.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurIPS , 2020.

H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML . PMLR, 2020.

Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 , 2023.

M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR , 2023.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR . Ieee, 2009.

J. Donahue and K. Simonyan. Large scale adversarial representation learning. NeurIPS , 2019.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Y. Duan, W. Wang, Z. Chen, X. Zhu, L. Lu, T. Lu, Y. Qiao, H. Li, J. Dai, and W. Wang. Visionrwkv: Efficient and scalable visual perception with rwkv-like architectures. arXiv preprint arXiv:2403.02308 , 2024.

A. El-Nouby, M. Klein, S. Zhai, M. A. Bautista, A. Toshev, V. Shankar, J. M. Susskind, and A. Joulin. Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.08541 , 2024.

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR , 2021.

I. Flickr. Flickr terms & conditions of use. https://www.flickr.com/help/terms .

K. Goel, A. Gu, C. Donahue, and C. Ré. It's raw! audio generation with state-space models. In ICML . PMLR, 2022.

A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. NeurIPS , 33, 2020.

A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. NeurIPS , 2021.

A. Gupta, A. Gu, and J. Berant. Diagonal state spaces are as effective as structured state spaces. NeurIPS , 35, 2022.

B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh. Rethinking spatial dimensions of vision transformers. In ICCV , 2021.

J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In CVPR , 2018.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR , 2017.

S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al. Language is not all you need: Aligning perception with language models. NeurIPS , 2024.

X. Jiang, C. Han, and N. Mesgarani. Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation. arXiv preprint arXiv:2403.18257 , 2024.

R. E. Kalman. A new approach to linear filtering and prediction problems. 1960.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS , 2012.

Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium on circuits and systems , pages 253-256. IEEE, 2010.

K. Li, X. Li, Y. Wang, Y. He, Y. Wang, L. Wang, and Y. Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977 , 2024.

S. Li, H. Singh, and A. Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892 , 2024.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. NeurIPS , 2024.

Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166 , 2024.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

X. Ma, C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and L. Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.10655 , 2022.

H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947 , 2022.

MIT. Laion-400m terms & conditions of use. https://github.com/LAION-AI/ laion-datasets/blob/main/LICENSE .

E. Nguyen, K. Goel, A. Gu, G. W. Downs, P. Shah, T. Dao, S. A. Baccus, and C. Ré. S4nd: Modeling images and videos as multidimensional signals using state spaces. arXiv preprint arXiv:2210.06583 , 2022.

A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In ICML . PMLR, 2023.

Z. Pan, B. Zhuang, J. Liu, H. He, and J. Cai. Scalable vision transformers with hierarchical pooling. In ICCV , 2021.

N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In ICML . PMLR, 2018.

W. Peebles and S. Xie. Scalable diffusion models with transformers. In ICCV , 2023.

B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048 , 2023.

Y. Qiao, Z. Yu, L. Guo, S. Chen, Z. Zhao, M. Sun, Q. Wu, and J. Liu. Vl-mamba: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600 , 2024.

Z. Qin, D. Li, W. Sun, W. Sun, X. Shen, X. Han, Y. Wei, B. Lv, F. Yuan, X. Luo, et al. Scaling transnormer to 175 billion parameters. arXiv preprint arXiv:2307.14995 , 2023.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML . PMLR, 2021.

T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 , 2021.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

J. T. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933 , 2022.

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 2024.

Y. Sun, L. Dong, B. Patra, S. Ma, S. Huang, A. Benhaim, V. Chaudhary, X. Song, and F. Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554 , 2022.

Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621 , 2023.

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR , 2015.

M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML . PMLR, 2019.

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML . PMLR, 2021.

P. University and S. University. Imagenet terms & conditions of use. https://image-net. org/download .

A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, K. Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 2016.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. NeurIPS , 2017.

C. Wang, O. Tsepa, J. Ma, and B. Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789 , 2024.

W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV , 2021.

W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media , 8(3): 415-424, 2022.

S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR , 2017.

H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022.

H. Zhao, M. Zhang, W. Zhao, P. Ding, S. Huang, and D. Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403.14520 , 2024.

D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 , 2021.

L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 , 2024.

A More Experiment Results

A.1 Visualization

This section provides visualization results of attention maps and gradient maps from more different layers of different models, as shown in Fig. 4, Fig. 5 and Fig. 6. Compared to other causal models, our de-focus attention network has denser attention maps and diverse gradient maps across all layers.

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

(a) Attention Map

(b) Gradient Map

A.2 Resolution Transfer

Our model is trained on images with 224 resolution. We test the transfer performance under resolution 336 and report the results in Table 6. The results demonstrate that our De-focus Networks can also transfer to different resolutions effectively.

B More Implementation Details

B.1 Visualization

This subsection discusses the detailed implementation of different visualization methods adopted in our paper, including receptive fields (Fig. 3(a)-(b)), attention maps (Fig. 1(a), Fig. 4(a), Fig. 5(a), Fig. 6(a)), and gradient maps (Fig. 1(b), Fig. 3(c)-(d), Fig. 4(b), Fig. 5(b), Fig. 6(b)).

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

(a) Attention Map

(b) Gradient Map

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

Non-causal

ViT

Causal

ViT

Causal

Mamba

De-focus

Attention Network

(a) Attention Map

(b) Gradient Map

Receptive fields of a certain layer are defined as the gradient norms of all image tokens on the input side. The gradients here are obtained by back-propagating from the L2-norm of the <CLS> token feature on output side of the same layer. Redder colors indicate larger receptive scores.

Attention maps . Similar to receptive fields, the approximated attention maps in our paper are also the gradient norms of all input image tokens (as 'key'). However, different from receptive fields, these gradients come from back-propagation of the feature norm across all image tokens (as 'query') on the same layer's output side. Brighter colors indicate larger attention weights.

Gradient maps . Different from receptive fields, the gradient maps of a certain layer are calculated by directly back-propagating from the final training loss to this layer's input image tokens. Then the L2-norm of each image token's gradient is used for plotting the gradient maps. Redder colors indicate larger gradient norms.

By default, the values of receptive fields, attention maps, and gradient maps are divided by the maximum value among all input image tokens for normalization. For attention maps, the diagonal values are set as 0 manually to eliminate the influence induced by residual connection. All image samples are randomly selected.

B.2 Image Classification

The hyper-parameters for training on ImageNet-1k [13] from scratch are provided in Tab. 7.

The hyper-parameters for pre-training on ImageNet-21k [55] are provided in Tab. 9. The hyperparameters for finetuning on ImageNet-1k [13] after pre-training are provided in Tab. 10.

B.3 Object Detection

The hyper-parameters for training on COCO object detection [36] are provided in Tab. 8.

Since an ImageNet-1K pre-trained model is used, to reduce the discrepancy between the resolutions of images in the COCO dataset and those in the ImageNet-1K dataset, several moditifications are made. Spatial decay parameters (e.g., ∆ ) and position embedding indices are initially scaled down by factors of approximately 4 and 20 (which are COCO resolution to ImageNet-1K resolution ratios), respectively. The order of image tokens is rearranged as shown in Fig.7, with each 224 × 224 section of the image first being spanned, followed by concatenation of these spanned sequences in a z-scan order.

B.4 Contrastive Language-Image Pre-training (CLIP)

The hyper-parameters for Contrastive Language-Image Pre-training on Laion-400m [56] are provided in Tab. 11.

C Licenses of Datasets

ImageNet-1k [13] is subject to the ImageNet terms of use [65].

COCO [36] is subject to the Flickr terms of use [19].

Laion-400m [56] is subject to the Laion-400m LICENSE of use [43].

Image Patches divided into Sections

Sections Spanned Independently

Sections Concatenated

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are clearly stated in the abstract and introduction, including contributions, scope and corresponding results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All information for reproduction are provided, which can be found in Sec. 5 and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We promise to release our code upon acceptance. We also provide sufficient information for reproduction in Sec. 5 and appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are included. Please refer to Sec. 5 and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Experiments on ImageNet and COCO are known to be stable, with an error of less than 0.2. Calculating error bars on LAION400M is too computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information about compute resources in Sec. 5 and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked the NeurIPS Code of Ethics carefully and made sure that we follow all of them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts are discussed in Sec. 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: To the best of our knowledge, all license of used assets are cited and respected in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Documentation will be provided when releasing the code and models.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper mainly improves the existing 1D causal models when handling visual inputs, generally 2D non-causal. The key insight of the authors mainly comes from Figure 1: the 1D causal models will over-focus to a few tokens instead of capturing the rich information from the whole image. To fix this issue, the authors propose the de-focus network. They first view the spatial decay and positional embedding as bandpass filters and learn that such filters lead to the capture of diverse information. In addition, the improvement in optimization, especially a significantly larger dropout, also encourages the model to make use of diverse sources of information. Finally, the authors experiment on image classification, object detection, and image-text training to prove the effectiveness of their approach.

STRENGTHS:
1. In Figure 1 and Sec. 4.1, the authors make a very good demonstration and reasoning of their intuitions.
2. The De-focus network is simple and reasonable, including both the bandpass filter part and the optimization improvement.
3. The experiments across a wide range of fundamental tasks prove the effectiveness of the De-focus network.

WEAKNESSES:
I think the paper is good overall. I only have one question regarding the generalization of the method, which is not discussed in the paper.

Transformer-based architectures like ViT in image classification and CLIP show certain generalizations to different resolutions. For example, people can train a model on 224x224 and inference on 336x336 by interpolating the positional embeddings. I am curious how well the de-focus network will perform under such generalization scenarios, compared with the 2D non-causal transformer-based architectures.

QUESTIONS:
See weakness.

LIMITATIONS:
Yes, the authors have addressed such limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of over-focus in vision models by proposing strategies of using large and scheduled drop path rates and an auxiliary loss on globally pooled features. These strategies aim to encourage the model to focus on a wider range of tokens and improve network optimization. The paper is logically structured, the methods are novel, and the experimental results validate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. The paper is well-organized and easy to follow.\n2. The authors have an insightful perspective on existing vision models.\n3. The research perspective of this paper is interesting and will be inspiring for future studies.\n\nWEAKNESSES:\nI am confused about some of the author's viewpoints. For example, in the introduction, the author mentioned that the causal modeling methods explored in existing vision models give them causal reasoning capabilities. \n\n1. Why do vision models based on standard Transformers without adding a causal attention mask still possess some level of causal reasoning abilities? \n\n2. Can we consider that vision Transformers inherently have causal abilities, and that existing improved methods simply amplify these models' causal capabilities?\n\nQUESTIONS:\n1. Can using large spatial dropout/dropblock or randomly dropping masked image tokens achieve similar effects compared to using a Large Drop Path Rate? If these methods are feasible, could additional experimental results comparisons be included?\n\n2. The authors need to provide further explanation on the rationale behind the positioning of learnable decay and learnable RoPE in the attention mechanism. Besides, can we directly deploy them on the feature maps before the attention mechanism?\n\nLIMITATIONS:\n1. The authors compared the parameter counts between models in terms of efficiency, but parameter count alone may not fully reflect model efficiency. I suggest the authors consider adding metrics related to model FLOPs and latency in Table 1 and Table 2.\n\n2.  Adding a classification loss to the model is not a novel method. \n\n3. The ablation study lacks experimental results comparing with other causal methods.\n\n4. There is a small grammatical error. In the paragraph on line 38, the phrase \"On one hand xxx, On the other hand xxx\" typically indicates a contrasting situation, but in this context, the authors may not intend to convey such a situation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenges of using 1D causal modeling for images, which traditionally require 2D non-causal modeling due to inherent modality differences between vision and language models. It identifies a significant issue in existing 1D causal vision models termed \"over-focus,\" where the model's attention is overly concentrated on a few visual tokens, hindering diverse feature extraction and optimization. To combat this, the paper introduces De-focus Attention Networks that use learnable bandpass filters to diversify attention patterns and incorporate strategies like large drop path rates and an auxiliary loss on globally pooled features to broaden token attention and enhance model optimization. Experiments on several image underst show that these innovations enable 1D causal visual representation to achieve performance comparable to 2D non-causal models in various tasks, including global perception and multi-modal understanding.\n\nSTRENGTHS:\n+ Identifying over-focus issue in 1D causal visual modeling is an interesting and provide a starting point for the work in thissubmission. \n\n+ A \"de-focus attention\" strategy is introducted to address this issue. It incorporates learnable bandpass filters into the existing attention mechanisms to generate diverse attention patterns. The proposed idea is resonable.  A high drop path probability and an auxiliary loss is also proposed for better optimization.\n\n+ Experiments on seval visual understanding benchmarks show that via the proposed De-focus Attention Networks, 1D causal visual representation can match the performance of 2D non-causal representation in tasks involving global perception, dense prediction, and multi-modal understanding.\n\nWEAKNESSES:\n- For CLIP experiments, it is helpful to also report comparisons for cross-modal retrival on MSCOCO to follow existing routines of CLIP model comparisons.  \n\n- The major motivation is about addressing challenges in constructing unified multi-modal model, while the main experments are about image understanding tasks. To this point, seems discussions/explorations about impact of proposed method on MLLM is missing. I'm also curious about its potential applications for image generation models.\n\nQUESTIONS:\nPlease refer to details in Weakness part above.\n\nLIMITATIONS:\nSome limitations of the work have been discussed in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper explores the feasibility of representing images with 1D causal modeling in unified multi-modal vision and language models, addressing the "over-focus" issue in existing models by proposing De-focus Attention Networks (DANs) with learnable bandpass filters and enhanced training strategies. Extensive experiments show that 1D causal visual representation can perform comparably to 2D non-causal representation in global perception, dense prediction, and multi-modal understanding tasks, with code to be released.

STRENGTHS:
1. The proposal to use 1D causal modeling for images is a novel approach that challenges the traditional 2D non-causal representation, opening new avenues for unified multi-modal models.

2. The use of large and scheduled drop path rates and an auxiliary loss on globally pooled features are effective training strategies that promote broader attention and better network optimization.

WEAKNESSES:
1. The evaluation of the proposed method is inconsistent, e.g., in table 1, the performance of Base ViT model is reported, but Small and Large model are missing. In table 2 and 3, only Mamba model results are compared.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
Limitations have been discussed in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15