Learning 1D Causal Visual Representation with De-focus Attention Networks
Authors: Chenxin Tao
Xizhou Zhu
Shiqian Su
Lewei Lu
Changyao Tian
Xuan Luo
Gao Huang
Hongsheng Li
Yu Qiao
Jie Zhou
Jifeng Dai
Venue: NeurIPS 2024 poster (2024)
Keywords: visual causal modeling; vision model; representation learning;
Primary area: machine_vision

Abstract: Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using 1D causal modeling. We identify an "over-focus" issue in existing 1D causal vision models, where attention overly concentrates on a small proportion of visual tokens. The issue of "over-focus" hinders the model's ability to extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns. During training, large and scheduled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks are introduced. These two strategies encourage the model to attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code shall be released.

Learning 1D Causal Visual Representation with De-focus Attention Networks
Chenxin T Xizhou L2" Shiqian Sul.3*, Lewei Lu Changyao Tian', Xuan Luo', Gao Huang', Hongsheng Lit, Yu Qiao?, Jie Zhou', Jifeng Dai 1.20
Shanghai Artificial Intelligence Laboratory "The Chinese University of Hong Kong (tcx20, ssqq0, (uoruan21]$maill {zhuxizhou, (chuxizhou,gaaohuan,, jahou, daij htw.ammmmaaa.sllm....aaaaa tsinghua. ediu. edu. edu. en, lutttosseenee tcyhost&link.. cuhk, edu.hk, hsli@ee. cuhk, edu.hk, qiaoyu0pj jaappllabbbb lab lab.org....................... en
'Tsinghua University
"SenseTime Research
Abstract
Modality differences have led to the development heterogeneous architectures for vision and language models. While images typically require 2D non-causal model- ing, texts utilize 1D causal modeling. This distinction poses significant challenges in constructing unified multi-modal models. This paper explores the feasibility of representing images using ID causal modeling. We identify an "over-focus" issuc small in existing ID causal vision models, where attention overly concentrates proportion of visual tokens. The issue of "over-focus" hinders the model's 's ability extract diverse visual features and to receive effective gradients for optimization. To address this, we propose De-focus Attention Networks, which employ leamable bandpass filters create varied attention patterns. During training, large and sched- uled drop path rates, and an auxiliary loss on globally pooled features for global understanding tasks introduced. These two strategies encourage the model attend to a broader range of tokens and enhance network optimization. Extensive experiments validate the efficacy of our approach, demonstrating that 1D causal visual representation can perform comparably to 2D non-causal representation in tasks such as global perception, dense prediction, and multi-modal understanding. Code shall be released.
Introduction
Due inherent modality differences, vision and language models have evolved into distinct heteroge- neous architectures. A key difference is that images usually require 2D non causal modeling, while texts often utilize 1D causal modeling. This distinction presents significant challenge in constructing unified multi-modal models. Many existing multi-modal models [37 and language encoders separately before combining them. crucial question in advancing unified vasion-language modeling is how to represent images using 1D causal modeling.
5 29 have to train vision
Following the success of causal language modeling (e.g., GPT-series [52 53 81), some studies [10 17
have explored causal modeling in the vision domain. These efforts primarily focus on auto-regressive visual pre-training by adding causal attention mask to standard Transformers [15]. Despite numerous attempts, the gap between ID causal and 2D non-causal vision models remains unbridged. As shown in Sec. 5 many ID causal vision models, such as State Space Models [6] 21 and causal ViTs [5]. perform inferiorly compared to their modified 2D non-causal counterparts.
"Equal contribution. This work done when Chenxin Tao and Shiqian Su are interns at SenseTima Research. Corresponding to Jifeng Dai <daijifeng @ eei..c...edu.cn>.
x
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Moon-causal
Causal ViT
De-focus Allention Network
Non-causal VII
Causal Vir
De-focus
VIT
Mamba
Mamba
Atication Network
Attention Map
(b) Gradient Map
Figure 1: Visualizations of (a) Attention Map and (b) Gradient Map of different models, including Non-causal ViT, Causal ViT. Causal Mamba and our De-focus Attention Network Haaba-based The results are from the 1h layer of ViT (12 in total) and 22nd layer of Mamba (24 in total). (a) The approximated attention maps of all image tokens: The row and column axes represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back preck--oppggtion: Redder colors indicate larger gradient norms. See Appendix A for more visualizations on different layers.
In this paper, identify "over-focus" issue in existing ID causal vision models. Fig. visualizes the attention and gradient maps of several ImageNet-trained networks, including 2D non-causal ViT, ID causal ViT, and 1D causal Mamba. The results show that in ID causal vision models, the attention patterns are overly concentrated on a small proportion of visual tokens, especially in the deeper network layers close to the output. This phenomenon hinders the model from extracting diverse visual features during the forward calculation and obtaining effective gradients during the backward propagation. We refer this phenomenon as the "over-focus" issue in 1D causal vision models.
To address the issue, a foc-f attention" strategy is introduced. The core idea is to guide the network to attend to broader range of tokens. First, learnable bandpass filters are introduced to filter different sets of token information, and then combine their attention patterns. This ensures that if over-focuue wver-focusing occurs, the attention pattern remains diverse due to the varying constraints of each set. Second, optimization strategies are improved. A large drop path rate is employed to encourage the network to attend to more tokens within one layer, rather than relying on depth to get large receptive fields. For tasks requiring global understanding (e.g., image classification), an auxiliary loss is applied the globally pooled features to enhance the effective gradients for all tokens in the sequence.
Extensive experiments demonstrate the effectiveness of our De-focus Attention Networks for ID causal visual representation learning. It achieves comparable or even superior performance to 2D non-causal ViTs across various tasks, including image classification, object detection, and image-text retrieval. Our method has been validated on both ViT's and Mambas. Our contributions can be summarized follows:
We identify the over-focus issue in ID causal visual modeling, where the model overly focuses on small proportion visual tokens the deeper layers the network.
To address this issue, we propose "de-focus attention" strategy. This involves integrating learnable bandpass filters into the existing attention operators to achieve diverse attention patterns. Additionally, we introduce a large drop path probability and an auxiliary loss on average pooled features during training to enhance network optimization.
Our fe-focus Attention Networks have demonstrated that 1D causal visual representation can achieve performance equivalent to non-causal representation tasks requiring global percep- tion, dense prediction and multi-modal understanding tasks.

---

2
Related Work
State Space Models (SSMs) are intrinsically causal models, originated from the classic Kalman filter[31]. SSMs describe the behavior of continuous-yynaiic systems, enabling parallel training and linear complexity inference. (24) proposed a Linear State Space Layer, merging the strengths of
continuous-time models, RNNs and CNNs. HIPPO [22] introduced methods to facilitate continuous-
time online memorization. Building on these foundations, Structured SSMs (e.g.. S4 [23]. Diagonal
State Spaces (DSS) (25. S5 [58), Recurrent SSMs (e.g., RWKV [4)] LRU [45] and Gated SSMs
(e.g., 42 Mega 41) further expand the SSMs landscape. Notably, Mamba [2] excels in
long-sequence modeling with its selective scan operator for information filtering and hardware-awan algorithms for efficient storage of intermediate results. As SSMs have drawn more and more attention recently, they also have extensive applications in domains that need long sequences processing such as medical 44 video (4), tabular domain [2] and audio/speech [20 [3]. These successes achieved by SSMs prompt to explore their application in visual modeling within this causal framework.
2D Nau-Ca Visual Modeling are dominant in vision domains. Convolutional Neural Networks (CNNs), operating in a 2D sliding-window manner (33 with inductive biases such as translation equivariance and locality, have demonstrated remarkable adaptability [32
57
28 27 63
Vision Transformers (ViTs) [[5] utilize a non- causal self-attention mechanism, enabling global receptive fields. Subsequent improvements focus on enhancing locality 33 refining self-attention
mechanisms 74| 4 and introducing novel architectural designs (69 77 46 26], while maintaining
non-causality. Recent advances in State Space Models (SSMs) have inspired new vision backbone networks, such as VMamba (38), Vision Mamba [75], and Vision-RWKV [16]. Although SSMs are inherently causal, these works incorporate non-causal adjustments to enhance vision performance. VMamba introduced four-way scanning strategy, Vision Mamba incorporated bidirectional SSMs, and -ision-RWVV adopted bidirectional global attention and a special token shift method. These designs of arrangement hinder the unification of vision and language modeling.
ID Causal Visual Modeling. While ID causal modeling has primarily been used in language and specch[[66. it has also been explored for visual representation. In recent years, the causal visual modeling has been adopted in Transformer-based visual generation methods such as Image Transformer [47 and VQGAN [8] These models first discretize images into grids of 2D tokens, which are then flattened for auto-regressive learning. However, their performance significantly lags behind [48 [. Of particular interest, iGPT [00] also employed auto-regressive causal modeling for pre-training, followed by linear probing or fine-tuning to achieve commendable results in various downstream tasks, though still worse than non-causal models 4 9 Similarly, AIM [77 applied causal masks to the self oeff-attention layers, and pre-trained with an auto-regressive objective, showing good scaling potential. Despite many attempts, the performance gap between ID causal and 2D non-causal vision models remains.
Preliminary
Transformers 67 with causal attention consist of multiple attention layers. Each attention layer com- putes a weighted average feature from the preceding context for every input token, with aggregated features weighted by the similarities between tokens. The attention layer is written as:
V Softmax(Q, K.)V..
(1)
where s and are indexes of different locations of the input sequence, Qi, K2, Vi2 are projections of input xi, and y/ the output of the attention layer.
State Space Models (SSMs) are classical latent state models widely used in various scientific fields 44 6.
34 68 50 73. Originally, SSMs are defined for continuous signals, mapping a 1D input
signal (t) E to a latent state h(t) E V and computing the output y(t) E from the latent state. To apply SSMs to discrete sequences, their discrete form is defined as
h = A,h4-1 + KYTG,
3e = QT hes
(2)
where A, E NNN, K. E Nx1, Q, E Nx1 are parameters of the system. Note that we use notations different from the original SSMs (K,, Q, instead of B,, C₂) for a better comparison with Transformers above.
3

---

SSMs can also be transformed into another formulation by expanding the recurrent process:
y/ Σ0 [OT(AA.. A.. K,T..
(3)
#5t
This formulation resembles the conventional attention module and explicitly reveals the relationship between different inputs in the sequence. We use this form for further discussion.
There multiple variants of SSMs, mainly differing in the parameterization of (AA, KG, Qt). We introduce some well-known SSMs and discuss their differences below.
RetNer [6]] and Transnormer [5]] employ fixed A and convert it into an exponential decay (defined by E IR) with a relative positional embedding (defined by E R.
y/ ΣΦ! ((t-a)
(0(1-a)
Kg
eap decay relative pos embed
Mamba [21] and S4 [23 use zero-order hold (ZOH) rule for discretization, introducing time-scale parameter Å, The discretization rule is A, exp(Ar and K, where Å and K, learnable parameters. S4 uses data-indepp data-independent parameters, while Mamba computes these parameters based on inputs. The formulation can be written as:
1/t Σ OT exp(A(A,,)) + A.)) KgTs. $51
(5)
learnalle expenential decay
4 Method
This section introduces our De-focus Attention Networks for ID causal visual representation learn- ing. Sec. [ elucidates the main components of De-focus Attention Learnable Bandpass Filter, while Sec. |.2 further discusses two training strategies adopted in De-focus Network. The overall architecture of our model is presented in Fig. 2
4.1 De-focus Attention with Learnable Bandpass Filter
To de-focus on a few salient tokens and enhance the extraction of diverse features from images, learnable bandpass filters are incorporated to first adaptively filter diverse information from the input and their attention patterns are then combined together. Due to the varying contents from different filters, the attention can still be diverse even if the over-focus issue happens.
These bandpass filters can be implemented through exponential spatial decay and relative position embedding similar to those in RoPE (59) and xPos [60], both of which are further made learnable. Our results demonstrate that these factors crucial for the model to learn diverse attention patterns.
To show how spatial decay and relative position embeddings work as a bandpass filter, consider a simplified version of ID causal attention equipped with them:
y(t)
e A(t-o) (0((=>) (00tt=))((s)ss
(6)
where zss) is the input signal at time s. c(t-ss) (A < 0) represents the simplest version of exponential spatial decay, which also used by RetNet [6] and Transnormer [31]. (i(tts) is the relative position embedding proposed by RoPE [59] and xPos (60). Here, the continuous time domain is used to facilitate derivation without losing generality.
The above equation implies a time domain convolution between e((t=s)ci((-a)) and a(s). By transforming Eq. (6) into the frequency domain and using ((w), ((w) to represent Fourier transform of corresponding æ z(s). (f)) the frequency domain expression becomes:
1
1
((w)
i(w 0) k(w),
[((W]]
[X
(
This equation indicates that Eq. controls its passband width. Eq. presents some interesting properties of ID causal modeling:
is actually a bandpass filter, where 0 its center frequency and

---

1. If there is no spatial decay or relative position embedding (e.g. Transformers without Softmax), Eq. (6)) will degenerate to summation of the inputs, losing the ability to filter spatial information;
2. If there is no relative position embedding (e-g., Mamba), ID causal attention will perform low-pass frequency filtering, causing the query to miss the full information of features and resulting in information loss;
3. If only relative position embedding is used, it will degenerate to specific frequency selecting, which may also result in information loss;
If both spatial decay and relative position embedding are used (suggested), ID causal attention will act as bandpass filter. For a given query, when different components of the feature vector use different center frequencies (i.e., different 0) and passbands width (i.e., different X), a more diverse range of information will be gathered. Due to the diverse frequency passbands, even if the over-focus issue occurs, the attention remains diverse across different components of the feature vector.
To fully leverage the bandpass filtering mechanism, learnable one preferable. Experiments demonstrate that performance worsens when values are fixed or not well set.
Our De-focus Attention can be incorporated into different architectures. Below, examples of its implementation in causal VIT and Mamba are presented.
De-focus Causal VIT. ViT has additional attention activation (i.e., Softmax) compared with SSMs. Learnable exponential spatial decay and learnable relative position embeddings are appended before applying the attention activation, following the common implementation of RoPE, shown below:
Vt Softmax (Q.
A(t-a)
Kg) Fes
Icaraall decay learnable RoPE
where the terms of e and e function as the learnable bandpass filter.
De-focus Mamba. Since Mamba already has learnable and data-dependent exponential spatial decay, only attachment of learnable relative position embeddings to it is necessary:
yt Σ Qi exp(A(AA+))
A.))
(i(t_
Kg
learnable RoPE
learnable expomential decay
where the terms of ÁA and e0 function as the learnable bandpass filter.
4.2 De-focus Attention in Network Optimization
During network training, performance of ID causal models can further enhanced with improved optimization strategies. Specifically, using large drop path rate with a linear schedule helps the model attend to more tokens each layer. Additionally, applying an auxiliary loss to the global average feature mitigates the under-learning of features in deeper layers. The effects of these training strategies are illustrated in Fig.
Large Drop Path Rate with Linear Schedule. Two ways for the final prediction to access infor- mation from previous inputs are observed: 1) Network Depth: Progressively looking forward few tokens in each layer until reaching the earliest tokens; 2) Intra-Layer Attention: Using the attention mechanism within the same layer to directly capture information from more distant tokens.
Our goal is for each layer to fully utilize the existing attention mechanism to capture more and further information in one layer. Therefore, large drop path rate (up 0.7) employed encourage the network to rely less on depth and rely more on training the attention mechanism in each layer. Since large drop path rate may hinder the model when only a few features are learned, i.c., at the beginning of training, linear schedule that gradually increase the drop path rate is followed.
Fig. 3 demonstrates the effectiveness of this strategy, indicating that without large drop path strategies, the network tends to prefer to see less tokens in one layer and rely on network depth to increase the receptive field.
Auxiliary Loss for Image Classification. To address over-focus issue in backward gradients, an auxiliary loss is proposed to enrich the gradients variety and aid in the representation learning of
5

---

Query Tokens Output
Auxiliary Loss
Loss
Causal Attention Layer
(e.g. Causal Transformer, Mamba)
FC
Average Pooling
Learmable
Bandpass
AA
R
Nx
Drop Path
Learnable Decay exp(A) )
Learnable RoPE exp(i0t) x
De-focus Causal Attention Block
A
K
CLS
Projection Layer
Query Tokens Input
Input Image
Figure 2: Architecture of our De-focus Attention Network. Left: Detailed architecture of De-focus Attention Block: The input tokens are projected to Q, K, and other parameters required by certain causal attention layer (e.g. Transformer or Mamba). A is data-dependent De-focus Mamba, while is set to 1 in fe-fu VIT. Learnable decay and learnable relative position embeddings form learnable bandpass filter and are calculated before being fed into the causal attention layer. Parameter À in De-focus VIT corresponds to A in this figure. Right: Overall architecture of De-focus Attention Network: Drop paths are incorporated after each De-focus Attention Block. All output image tokens are passed through Average Pooling and fully connected layer to produce the auxiliary loss.
unnoticed tokens. The final representations of all image tokens (excluding the final <CLS> token) averaged and fed into additional linear layer. The auxiliary loss function is defined as the cross-entropy loss between the output of the additional linear layer and the ground truth label. This approach helps enrich the backpropagated gradients, thereby addressing the over-focus issue.
As shown in Fig. 3 after applying the auxiliary loss strategy, the backward gradients are significantly improved in its density, globality, and diversity in deeper layers.
4.3 Overall Architecture.
The overall architecture of our De-focus Attention Networks is illustrated in Fig. 2 The following explains how learnable bandpass filters and optimization strategies are integrated into existing models.
De-focus Attention Blocks. Each block consists of three main parts, which are a projection layer, learnable bandpass filter, and a causal attention layer. The input tokens are first projected into query Q, key K. A is date-dependent in De-focus Mamba, while is set to in De-focus ViT. Other projections may be required by the causal attention layer. The block has learnable decay parameters A (corresponds to À in De-focus VIT) and learnable relative position embedding parameters 0. Given these learnable parameters, exponential spatial decay and relative position embedding are computed as illustrated in Eq. 8 and Eq. (9) Q, K and the exponential spatial decay term are integrated into the causal attention layer. Thus, the outputs of a De-focus Attention block aggregate the input information filtered by series of Icarnable bandpass filters.
De-focus Attention Networks. Given an image, our De-focus Attention Networks first transform into a sequence of image tokens and append an extra <CLS> token to the sequence end. The whole network then stacks N De-focus Attention blocks process the input sequence. Each block is equipped drop path rate, which increases linearly during training. In the final layer, the <CLS> token fed through a linear layer and used to compute a cross-entropy loss with class labels. All image tokens, excluding the final <CLS> token, are averaged and passed through separate linear layer. An auxiliary cross-entropy loss is applied to this projected averaged feature. The two losses are then added with equal weights to form the final loss function.

---

Table 1: Comparison of causal and non-causal attentions for image classification on ImageNet- 1K.
ImageNet Top-1 Acc
Method
Causal
Size
#Param
DeiT-Small
04
224
22.1M
79.9
Mamba-ND LaabbaNDD-Salll
35
224
24M
79.4
Vision Mamba-S -mall
224
26M
80.5
Vision RWKV-Small
16
224²
23.8M
80.1
DeiT-Small
2242
22.1M
78.6
Mamba-Smal Small 21
224
24.7M
78.7
Mamba-ND-S Daaaa-D---aalllll
35
224a
24M
76.4
De-focus VIT-Small
224
22.4M
79.6
De-focus Mamba-Small
2242
25.8M
80.3
DeiT-Base Base 64
224
86.6M
81.8
$4ND- VIT-B
2212
88.8M
80.4
Vision RWKV-Basc
16
2242
93.7M
82.0
De-focus VIT-Base
224
87.4M
81.8
DeiT-Base
224
86.6M
80.1
RetNet-Base 61
V
22422
93.6M
79.0
Mamba-Base 21
V
2242
91.9M
80.5
De-focus VIT-Base
224
87.4M
81.5
De-focus RetNet-Base
224
94. IM
81.7
De-fo Mamba- Base
224
94. IM
82.0
ViT-Large [TS]
3842
309 5M
85.2
Vision RWKV-Large
16
384
334.9M
86.0
De-focus Mamba-Large
3842
327.4M
85.4
Experiments
5.1 Experiment Setup
Implementation Details. The De-focus Attention mechanisms are integrated into Mamba, RetNet, and ViT, referred to as De-focus Mamba, De-focus RetNet, and De-focus ViT, respectively. To improve optmizatiion stability, A -exp(A) is used and is the parameter to be optimized. In De-focus ViT and De-focus RetNet, different As are assigned different heads. Mamba inherently implements data-dependent decay ÅA, where Å is a learnable parameter and A a projection from the input. The drop path rate increases following linear schedule from 0.1 to 0.7.
Image Classification. ImageNet- 1k [ is used, which contains 1. 1.28M images for training and 50K images for validation. The training recipe of DeiT [64] is followed. The small- and base-size models are trained on ImageNet for 300 epochs. The large-size model firstly pre-trained on ImageNet-2 1k (55) for 90 epochs, and then fine-tuned on ImageNet-Ik lk for 20 epochs. The AdamW 40 with peak learning rate of 5e-4, total batch size of 1024, a momentum of 0.9, and weight decay of 0.05 are used. These models are trained on 32 Nvidia 80G A100 GPUs for 30 hours.
optimizer
Object Detection. The MS-COCO dataset 36 and the DINO detection framework 722 are used, with different networks serving as the backbones. The De-focus Attention Networks implemented here pre-trained ImageNet-1 K dataset for 300 epochs. These models trained 16 Nvidia 80G A100 GPUs for 20 hours.
The entire network is fine-tuned using both 1x schedule (12 epochs) and 3x schedule (36 epochs). The base learning rate is set to 2c-4, with a multi-step learning rate strategy employed to decrease by a factor of ten after 11 epochs (1 x schedule) or after 27 and 33 epochs (3x schedule). The weight decay and the total batch size is set to le-4 and 16, respectively.
Contrastive Language-Image Pre-training (CLIP). The Laion-400M dataset [56] is used for pre-training. Strategy introduced in OpenCLIP 212 followed train the model for epochs. The zero-shot classification performance is evaluated on ImageNet-I 1 K. The AdamW optimizer 440 is employed with peak learning rate of 5e-4, a total batch size of 32768, a momentum 0.9, and weight decay of 0.I. These models are trained on 128 Nvidia 80G A100 GPUs for 128 hours.

---

Table 2: Results of object detection on the COCO
36] dataset with DINO 72 detector.
Method
Causal
#Param
Epochs
Ap
AP
AP
ResNet-50
49.0
66.6
53.5
DeiT-Base
49.1
69.9
52.7
De-focus ViT-Basc
113M
48.9
67.1
53.3
De-focus Mamba- Hamba-Base
115M
50.8
68.9
55.2
ResNet-50 722
47M
36
50.9
69.0
55.3
DeiT-Base
110M
36
52.3
72.5
56.7
De-focus Mamba-Bane
115M
36
53.5
71.9
58.3
5.2
Main Results
Image Classification. The classification results are presented in Table Evaluation of different types of De-focus Networks various scales conducted, with comparisons both causal and non-causal models. The results show that previous causal models have inferior performance. In contrast, model defies this trend, significantly outperforming other ID causal models and achieving comparable performance to 2D non-causal models.
Notably, the De-focus Attention mechanism works well across various networks, e.g., Causal ViT, Mamba, and RetNet. And as the model size increases from small to large, it remains on par with the 2D non-causal ViTs.
Object Detection. As shown Table 2 De-focus Mamba remarkably outperforms non-causal models such DeiT and ResNet-50. This trend of superior performance persists even with increasing number of training epochs. Additionally, excellent performance the AP metric may suggest that De-focus Attention Networks are more effective at fine-grained localization.
Table 3: Results on zero-shot image classification of CLIP pre-trained models.
ImageNet Zero-shot
Method
Causal
#Param
Top-1 Acc
OpenAI CLIP-Base/33
151.3M
OpenCLIP OpeseLLP-Base/32
151.3M
62,9
De De-fo Mamba- Base/32
161.9M
Table Results on image-text retrieval on the COCO 36
dataset of CLIP pre-trained models.
Method
Causal
#Param
1
Image Retrieval
Recall
Recall@ 10
Recall@ 1
Text Retrieval Recalle
Recall 10
OpenAl CLIP-Base/2 54
151.3M
30.4
65.7
49.2
73.4
82.4
OpenCT IP. Base/32 12
151.3M
35.3
61.0
71.8
52.5
77.0
84.9
Decuu
Image-text CLIP Pre-training. The model is pre-trained using OpenCLIP demonstrate its outstanding performance on large-scale image-text training. As shown in Table 3 the model performs comparably 2D non-causal models. We also report cross-modal retrieval results in Table 4 which further validate that the learned causal feature can achieve similar results with non-causal features. These results indicate that the model has similar scaling law non non-causal VITs on larger dataset, demonstrating its robustness and scalability across various of tasks and datasets. Additionally, this experiment demonstrates the potential of 1D causal modeling for unified vision-language modeling.
5.3 Ablation Study
Learnable Bandpass Filter. As discussed in Sec. 4.1 exponential spatial decay and relative position embedding (RoPE) together act as a bandpass filter. Tab. (Ja) shows the effects of different configurations. When decay is not used, the performance significantly deteriorates. Employing learnable decay leads to an improvement of approximately 0.5% compared to fixed decay, while learnable RoPE can further enhance performance by 0.8%. In contrast, the data-dependent decay used in Mamba only results in a marginal improvement of 0. 0.1% These results indicate the integration of learnable decay and RoPE necessary for good performance.

---

Table 5: Ablation studies of various design choices of De-focus Mamba-Base model ImageNet- 1k [13]. . The default settings are set as (a) dpr 0.4, with auxiliary loss, (b) with auxiliary loss, data dependent decay and learnable RoPE, (c) dpr 0.4, with data dependent decay and learnable RoPE. "dpr" is drop path rate. The text in (c) denotes the input feature for the loss function.
(a) Ablation on Bandpass Filter
(b) Ablation on Drop Path.
(c) Ablation on Loss Function
Decay
RoPF
Acc
Drop Path
Acc
Loss
Aux Loss
Acc
w/o
w/o
75.2
w/o
fixed
75.3
0.1
79.6
<CLS>
81.6
fixed
w/o
79.9
0.4
81.6
77.2
fixed
fixed
80.0
0.7
80.9
<CLS> avg
79.7
fixed
Icarnable
80.6
linear(U.1. 0.7)
82.0
<CLS>
avg
82.0
learnable
w/o
80.4
learnable
learnable
81.2
data dependent
learnable
81.3
(a) Reception Goold wio Schoduled DropPaith (constant 0.1)
(c) Gradicnt mapp wlo Auxiliary Loss
(b) Reception fild wi Scheduled DropPath (0..ear(0.1, 0.7))
(d) Gradient map w/ Auxiliary Loss
Figure 3: Qualitative ablation results of using scheduled drop path and auxiliary loss. (a)-(b): The receptive fields of model trained with and without scheduled drop path. The scheduled drop path strategy enables a larger receptive field, facilitating the capture of denser semantic details. (c)-(d): The backward gradient maps of our model trained with and without auxiliary loss. When trained with the auxiliary loss, the model can attend to denser and more diverse image tokens, particularly those at the front of the sequence.
Drop Path. Tab. (bbb shows the performance of different drop path strategies, with rates ranging from 0.1 to 0.7. The best performance is achieved with a scheduled drop path rate linear (0.1ar(..1,. 0.7). Fig. 3/a [b) visualize the receptive field of the 22nd layer of the network. The results demonstrate that using a large and scheduled drop path rate strategy allows for larger receptive field and helps capture more dense semantic details.
Auxiliary Loss. Tab. cjc) compares various implementations of the loss function, which are generated from <CLS> token only, average token only, concatenation of <CLS> token and average token, and <CLS> token with auxiliary average token. The results reveal that the average pooled feature alone performs poorly in training the network. It may result from the fact that previous tokens often have incomplete information. However, serves as an effective auxiliary component, thereby enhancing the network training. The visualization of gradient maps the 22nd layer of the network shown in Fig. (d)) When training with auxiliary loss, the density, globality, and diversity of backward gradients are significantly improved.
Conclusion
We propose De-focus Attention Networks to enhance the performance of causal vision models by addressing the issue of over-focus them. The over-focus phenomenon, i.e. attention pattern is overly focused on small proportion of visual tokens, is observed both during the forward calculation and backpropagation. These De-focus models incorporate decay mechanism and relative position embeddings, functioning together as diverse and learnable bandpass filters to introduce various attention patterns. The models are trained with large scheduled drop path rate and auxiliary loss to enhance the density, globality, and diversity of backward gradients. A series of De-focus models based on Mamba, RetNet, and ViT significantly outperform other causal models and achieve comparable or even superior performance to statc-of-the-ar non-causal models. By implementing the de-focus

---

strategy, our work bridges the performance gap between causal and non-causal vision models, paving the way for the development of state-of-the-art unified vision-language models.
Limitations. While the De-focus Attention has achieved very promising results, some differences between causal and non-causal vision models still need further exploration. For example, compared with non-causal models, De-focus Mamba-Base excels AP metric but is inferior A metric on object detection. It is worth studying how causal models perform dense prediction tasks. Furthermore, currently we only explore 1D causal modeling in purely visual settings, while exploring unified ID causal modeling of vision and language remains to be verified.
Broader Impacts. De-focus Attention Networks demonstrate the potential of building unified multi-modal models, so they may also cause similar problems as previous works. For instance, may also require huge computational resources, contain dataset bias, and raise ethical concerns when being adopted for training large multi-modal foundation models.
Acknowledgements. The work is partially supported by the National Natural Science Foundation of China under Grants 62321005.
10

---

References
[1] Alphavvlnnllrrg--di--maaaaa http://ww............./.o.oo.oon
2024.
[2] M. A. Ahamed and Q. Cheng. Mambatab: A simple yet effective approach for handling tabular
data. arXiv preprint arXiv:2401.000 008667, 2024.
[3] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
[4] A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova,
G. Synnaeve, J. Verbeek, et al. Xcit: Cross-covariance image transformers. NeurlPS, 2021.
[5] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile rision-language model for understanding, localization, text reading, and beyond. 2023.
[6] M. Ballarotto, S. Willems, T. Stiller, F. Nawa, A. Marschner, F. Grisoni, and D. Merk. De novo design of nurrl agonists via boggennt-augment generative deep learning in low-data regime. Journal of Medicinal Chemistry, 2023.
[7] Y. Bengio, R. Ducharme, and Vincent. neural probabilistic language model. NeurIPS, 2000.
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
[9] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang. K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301 100004, 2023.
[10] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and Sutskever. Generative pretraining from pixels. In ICML. PMLR, 2020.
[11] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- -inguuiin tasks. arXiv preprint arXiv:2312. 142338 2023.
[12] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann,
L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning.
In CVPR, 2023.
[13] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and I.. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR. leee, 2009.
[14] J. Donahue and K. Simonyan. Large scale adversarial representation learning. NeurIPS, 2019.
[15] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020.
[16] Y. Duan, W. Wang, Z. Chen, X. Zhu, L. Lu, T. Lu, Y. Qiao, H. Li, J. Dai, and W. Wang. Vision- rwkv: Efficient and scalable visual perception with rwkv-like architectures. arXiv preprint arXiv:2403.022 02308 2024.
[17] A. El-Nouby, M. Klein, S. Zhai, M. A. Bautista, A. Toshev, V. Shankar, J. M. Susskind, and A. Joulin. Scalable pre-training of large autoregressive image models. arXiv preprint ar arXiv:2401.08541 2024.
[18] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR. 2021.
[19] I. Flickr. Flickr terms conditions of use. lattps://ww.flictt wtwp:/ffiicc..ccomee//eeeeeeee com.help/terms
[20] K. Goel, A. Gu, C. Donahue, and C. Ré. It's raw! audio generation with state-space models. In ICML. PMLR, 2022.
[21] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.000552 2023.
[22] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. NeurIPS, 33, 2020.

---

[23] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
[24] A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. NeurlPS, 2021.
[25] A. Gupta, A. Gu, and J. Berant. Diagonal state spaces are as effective as structured state spaces. NeurlPS, 35, 2022.
[26] B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021.
[27] J. Hu, L. Shen, and G. Sun. Squeeze-and-excittioon networks. In CVPR, 2018.
[28] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional
networks. CVPR, 2017.
[29] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al. Language is not all you need: Aligning perception with language models. NeurlPS, 2024.
[30] X. Jiang, C. Han, and N. Mesgarani. Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation. arXiv preprint arXiv:2403. 18257, 2024.
[31] R. E. Kalman. A new approach to linear filtering and prediction problems. 1960.
[32] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 2012.
[33] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium on circuits and systems, pages 253 256. IEEE, 2010.
[34] K. Li, X. Li, Y. Wang. Y. He, Y. Wang, L. Wang. and Y. Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977, 2024.
[35] S. Li, H. Singh, and A. Grover. Mamba-nd: Selective state space modeling for multi-dimeensonal data. arXiv preprint arXiv:2402.0588) 2024.
[36] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[37] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. NeurIPS, 2024.
[38] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu. Vmamba: Visual state space model. arXiv preprint 24X1::2401.10166 2024.
[39] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[40] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17X 2017.
[41] X. Ma, C. Zhou, X. Kong, J. He, I. Gui, G. Neubig. J. May, and I.. Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209. 10655, 2022.
[42] H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.1388 2022.
[43] MIT.
Laion-400m terms & conditions of use. https:// htt:://gitttb..... Low.La....ooLLIONNNN AI/ daiionddatasets/bloo wttpo///miiinnnnnEESS.
[44] E. Nguyen, K. Goel, A. Gu, G. W. Downs, P. Shah, T. Dao, S. A. Baccus, and C. Ré. S4nd: Modeling images and videos as multidimensional signals using state spaces. arXiv preprint arXiv:2200006583. 2022.
[45] A. Orvieto, S. I.. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In ICML. PMLR, 2023.
[46] Z. Pan, B. Zhuang, J. Liu, H. He, and J. Cai. Scalable vision transformers with hierarchical pooling. In ICCV, 2021.
[47] N. Parmar, A. Vaswani, Uszkoreit, I.. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In ICML. PMLR, 2018.
12

---

[48] W. Peebles and S. Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[49] B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, al. Rwkv: Reinventing rnns for the transformer era. arXiv preprini
MrXiv:2305.13045 2023.
[50] Y. Qiao, Z. Yu, L. Guo, S. Chen, Z. Zhao, M. Sun, Q. Wu, and J. Liu, Vl-mamba: Exploring state space models for multimodal learning. arXiv preprini arXiv:2403. 13600, 2024.
[51] Z. Qin, D. Li, W. Sun, W. Sun, X. Shen, X. Han, Y. Wei, B. Lv, F. Yuan, X. Luo, al. Scaling transnormer to 175 billion parameters. arXiv preprint arXiv:2307.1499) 2023.
[52] A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language understanding
by generative pre-training. 2018.
[53] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[54] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
[55] Ridnik, E. con-Baruch, A. Noy, and L. Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10977 2021.
[56] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes,
J. Jitsev, and A. Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0222 2021.
[57] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
[58] J. T. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208. 04933, 2022.
[59] Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
[60] Y. Sun, L. Dong, B. Patra, S. Ma, S. Huang, A. Benhaim, V. Chaudhary, X. Song, and F, Wei. A cength-extrapolattbe transformer. arXiv preprint arXiv:22 10554, 2022.
[61] Y. Sun, L. Dong, S. Huang, S. Ma. Xia, J. Xue, J. Wang, and Wei. Retentive network: A successor to transformer for large language models. arXiv preprint urXiv:2307.08622 2023.
[62] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and
A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[63] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In
ICML. PMLR, 2019.
[64] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, Training data-efficient image transformers & distillation through attention. In ICML. PMLR, 2021.
[65] P. University and S. University. Imagenet terms & conditions of use.
https: petps://image-net
org/ org/dowmload
[66] A. Van Den Oord, Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, K. Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint
"XXv:1660.00399 2016.
[67] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. NewrlPS, 2017.
[68] C. Wang, O. Tsepa, J. Ma, and B. Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00788 2024.
[69] W. Wang. E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. ICCV, 2021.
[70] W. Wang, E. Xie, X. Li, D.PP. Fan, K. Song, D. Liang, Lu, P. Luo, and L. Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3): 415-424, 2022.
13

---

[71] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
[72] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, Zhu, L. M. Ni, and H.-Y. Shum. Dino:
Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprini arXiv:2203.03605. 2022.
[73] H. Zhao, M. Zhang, W. Zhao. P. Ding, S. Huang, and D. Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint 2455: 2024.
[74] D. Zhou, B. Kang, X. Jin, L. Yang. X. Lian, Z. Jiang, Q. Hou, and Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. / 1886, 2021.
[75] I.. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09117, 2024.
14

---

A More Experiment Results
A.1
Visualization
This section provides visualization results of attention maps and gradient maps from more different layers of different models, as shown in Fig- Fig. 5 and Fig. 6 Compared to other causal models, our de-focus attention network has denser attention maps and diverse gradient maps across all layers.
Non-causal
Causal
Causal
De-focus
Non-camsal
Causal
Cousal
De-focus
VIT
Attention
Allention Network
Map
Map
Figure Visualizations Attention Map (a) and Gradient Map (b) different models, including non-causal ViT, causal ViT, Causal Mamba and our De-focus Attention Network (Mamba-based), The results are from the 3rd layer of ViT (12 in total) and 6th layer Mamba (24 in total). (a) The approximated attention maps of all image tokens: The row and column axis represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back- :ropaaatio:::::
Redder colors indicate larger gradient norms.
A.2 Resolution Transfer
Table 6: Resolution transfer results on ImageNet dataset.
Resolution 224
Resolution 336
Deit--Base
81.8
81.6
De-focus Namba-Base
82.0
81.6
Our model is trained on images with 224 resolution. We test the transfer performance under resolution 336 and report the results in Table 6 The results demonstrate that our De-focus Networks can also transfer to different resolutions effectively.
B
More Implementation Details
B.1
Visualization
This subsection discusses the detailed implementation of different visualization methods adopted
in our paper, including receptive fields (Fig. 3 (b)),)),, attention maps (Fig. [a), Fig. 4()) Fig. 5[a),
Fig. (()),, and gradient maps (Fig, I (), Fig. (d). Fig. 4 bb)) Fig. 5 Fig. bb).
15

---

Non-causal VIT
Causal Vir
Causal Mamba
De-focus Attention Netwerk
Non-causal VIT
Causal Vir
Causal Mamba
De-focus
Atication Network
(a) Altention Map
(b) Gradicut Map
Figure 5: Visualizations of Attention Map (a) and Gradient Map (b) of different models, including non-causal VIT, causal ViT, Causal Mamba and our De-focus Attention Network (Mamba-based). The results are from the 6th layer of VIT (12 in total) and 12 layer of Mamba (24 in total). (a) The approximated attention maps of all image tokens: The row and column axis represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back- nack-propagation: Redder colors indicate larger gradient norms.
Non-cmusal VIT
Causal VIT
Cansal Mamba
De-focus Attention Network
Non-emusal VIT
Causal
Causal Mamba
De-focus Astention Network
WT
(n) Artention Map
Gradient Map
Figure 6: Visualizations of Attention Map (a) and Gradient Map (b) of different models, including non-causal VIT, causal ViT, Causal Mamba and our De-focus Attention Network (Mamba-based) The results are from the 9th layer of ViT (12 in total) and 18th layer of Mamba (24 in total). (a) The approximated attention maps of image tokens: The row and column axis represent the query and key token index respectively. Brighter color indicates larger attention values. (b) The gradient maps of each image token input after back- propek-ropagaiion: Redder colors indicate larger gradient norms.
16

---

Receptive fields of certain layer are defined as the gradient norms of all image tokens on the imput side. The gradients here are obtained by propagating from the L2-norm of the <CLS> token feature on output side of the same layer. Redder colors indicate larger receptive scores.
Attention maps. Similar receptive fields, the approximated attention maps in paper also the gradient norms of all input image tokens (as 'key".. However, different from receptive fields, these gradients come from back-propagation of the feature norm across all image tokens (as 'query") on the same layer's output side. Brighter colors indicate larger attention weights.
Gradient maps. Different from receptive fields, the gradient maps of certain layer calculated by directly back-propase propagaatng from the final training loss to this layer's input image tokens. Then the L2-norm of each image token's 's gradient is used for plotting the gradient maps. Redder colors indicate larger gradient norms.
By default, the values of receptive fields, attention maps, and gradient maps are divided by the maximum value among all input image tokens for normalization. For attention maps, the diagonal values are set 0 manually to eliminate the influence induced by residual connection. All image samples are randomly selected.
B.2 Image Classification
The hyper-parameters for training on ImageNet-1k K from scratch are provided in Tab.
The hyper-parametert for pre-training ImageNet-21k (55) provided in Tab. 9 The hyper-
parameters for finetuning on ImageNet-1k [3] after pre-training are provided Tab. 10
B.3
Object Detection
The
for training COCO object detection (36 are provided in Tab.
Since an ImageNet- pre-trained model is used, to reduce the discrepancy between the resolutions of images in the COCO dataset and those in the ImageNet-IK I dataset, several moditifications are made. Spatial decay parameters (e.g.. A) and position embedding indices are initially scaled down by factors of approximately 4 and 20 (which are COCO resolution to ImageNet- 1K resolution ratios), respectively. The order of image tokens is rearranged as shown in Fig with each 224 x 224 section of the image first being spanned, followed by concatenation of these spanned sequences in z-scan order.
B.4 Contrastive Language-Image Pre-training (CLIP)
The in Tab.
for Contrastive Language-Image Pre-training on Laion-400m 56 are provided
11
C Licenses of Datasets
ImageNet-1k 1k 13 is subject to the ImageNet terms of use [65]
COCO 36 is subject the Flickr terms of use [99
Laion-400m [56] is subject to the Laion-400m LICENSE of use 43
17

---

Image Patches divided into Sections
Scctions Spanned Independently
Sections Concatenated
Figure 7: Rearranging Procedure in Object Detection. This illustration presents image divided into several 2x2 sections.
Table Hyper-parameters for training from seratch on ImageNet-Ik
Hyper- parameters
Value
Input resolution
221 x 224
Training epochs
300
Warmup epochs
20
Batch size
1024
Optimizer
AdamW 1.0 x 10-3
Peak learning rate
Learning schedule
cosine
Weight decay
0.05
AdamW /
(0.9, 0.999)
Augmentation
Color jitter
0.4
Rand augment
9/0.5
Erasing prob.
0.25
Mixup prob.
Cutmix prob.
1.0
Label smoothing
0.1
repeated augmentation
True
Drop path rate
I0.. 0.7)
18

---

Table 8: Hyper-parameter for COCO object detection.
Hyper-paramettrss
Value
Input resolution
1024 1024
Finetuning epochs
12 /6
Batch size
Optimizer
AdamW
Peak learning rate
x 10
Learning rate schedule
Step(11)/ Step(27.33)
Weight decay
x 10 (0.9, 0.999)
Adam /
Augmentation
Random flip
0.5
Drop path rate
0.5
Table 9: Hyper-parametert for pre-training on ImageNet-21k,
Hyper-parameters
Value
Input resolution
192 192
Training epochs
90
Warmup epochs
Batch size
4096
Optimizer
Adam W
Peak learning rate
1.0 10
Learning rate schedule
cosine
Weight decay
0.05
Adam W /
(0.9, 0.999)
Augmentation
Mixup proh.
0.8
Cutmix proh.
1.0
Label smoothing
0.1
Drop path rate
linear(O.11, 0.5)
Table 10: Hyper-parameters for finetuning on ImageNet- Net-1k 1k.
Hyper-parametters
Value
Input resolution
384 384
Finetuning epochs
Warmup epochs
Batch size
1024
Optimizer
Adam W
Peak learning rate
x 10 5
Learning rate schedule
cosine 0.05 (0.9, 0.999)
Weight decay
Adam
Augmentation
Mixup prob.
Cutmix prob.
1.0
Label smoothing
0.1
Drop path rate
1inear(O..1, 0.5)
Table 11: Hyper-paramettrss for contrastive vision-language pre-training on Laion-400m.
Hyper-parameters
Value
Input resolution
224 224
Training epochs
Warmup epochs
20000
Batch size
32768
Optimizer
Adam W
Peak learning rate
10
Learning rate schedule
cosine
Weight decay
0.1
AdamW W 3
(0.9, 0.98)
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims are clearly stated in the abstract and introduction, including contributions, scope and corresponding results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information for reproduction are provided, which can be found in Sec. and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We promise to release our code upon acceptance. We also provide sufficient information for reproduction in Sec. | and appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wtt.:g/ddddeeeeSSuuuuutr ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are included. Please refer Sec. 5la appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments on ImageNet and COCO are known to be stable, with an error of less than 0.2. Calculating error bars on LAION400M too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information about compute resources in Sec. 5 and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: httpp///w.//.crrc..cc//oo//////a
Answer: [Yes]
Justification: We have checked the NeurlPS Code of Ethics carefully and made sure that we follow all them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: To the best of our knowledge, all license used assets are cited and respected in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation will be provided when releasing the code and models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "E6E6gJJeCN", "reviewer": "Reviewer_WFuU", "summary": "The paper explores the feasibility of representing images with 1D causal modeling in unified multi-modal vision and language models, addressing the \"over-focus\" issue in existing models by proposing De-focus Attention Networks (DANs) with learnable bandpass filters and enhanced training strategies. Extensive experiments show that 1D causal visual representation can perform comparably to 2D non-causal representation in global perception, dense prediction, and multi-modal understanding tasks, with code to be released.", "questions": "Please refer to the weakness section.", "limitations": "Limitations have been discussed in this paper.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposal to use 1D causal modeling for images is a novel approach that challenges the traditional 2D non-causal representation, opening new avenues for unified multi-modal models.\n\n2. The use of large and scheduled drop path rates and an auxiliary loss on globally pooled features are effective training strategies that promote broader attention and better network optimization.", "weaknesses": "1. The evaluation of the proposed method is inconsistent, e.g., in table 1, the performance of Base ViT model is reported, but Small and Large model are missing. In table 2 and 3, only Mamba model results are compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer WFuU for the comments, yet we must emphasize that the weakness Reviewer WFuU has identified is not a sufficient reason for rejection. Our experiments based on Mamba have demonstrated that 1D visual causal modeling can achieve comparable performance to non-causal models. Other results based on ViT are included primarily to supplement this evidence. Their performance, whether superior or not, does not compromise the soundness of the arguments or the validity of the conclusions drawn.\n\n---\n\n> **Q1: The evaluation of the proposed method is inconsistent, e.g., in table 1, the performance of Base ViT model is reported, but Small and Large model are missing. In table 2 and 3, only Mamba model results are compared.**\n\nA1: The aim of our research is to assess the feasibility and performance potential of causal modeling, a goal we have demonstrated by using the Mamba-based model. The structure of specific models, such as the ViT-based model mentioned in the paper, serves merely to provide additional evidence, further proving that our method works well and can be applied generally.\n\nDue to computational constraints, we conducted only the most essential experiments and presented these in the paper. Addressing your concerns, we have also conducted supplementary experiments employing the ViT model:\n\n- We began by training the De-focus ViT-Small model on the ImageNet dataset. The results of this experiment are depicted in the following table.\n\n|  | Causal | ImageNet Top1 Acc |\n|-----|----|----|\n| (DeiT) ViT-Small | No | 79.9 |\n| De-focus ViT-Small | Yes | 79.6 |\n| De-focus Mamba-Small | Yes | 80.3 |\n\n\n- Additionally, we conducted an experiment in a detection task setting. We did not change the existing detection architecture but instead utilized the ViT-base model listed in Table 1 as the pretrained model. \n\n|  | Causal | Epoch | AP | AP50 | AP75 |\n|-----|----|----|----|----|----|\n| (DeiT) ViT-Base | No | 12 | 49.1 | 69.9 | 52.7 |\n| De-focus ViT-Base | Yes | 12 |  48.9 | 67.1 | 53.3 |\n| De-focus Mamba-Base | Yes | 12 |  50.8 | 68.9 | 55.2 |\n\n\nThese additional experiments highlight the adaptability of our approach. Nonetheless, the absence of these experiments from the initial submission does not weaken our main points or lessen the reliability of the conclusions we made.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHmHgwKX6D", "reviewer": "Reviewer_4dxw", "summary": "The paper addresses the challenges of using 1D causal modeling for images, which traditionally require 2D non-causal modeling due to inherent modality differences between vision and language models. It identifies a significant issue in existing 1D causal vision models termed \"over-focus,\" where the model's attention is overly concentrated on a few visual tokens, hindering diverse feature extraction and optimization. To combat this, the paper introduces De-focus Attention Networks that use learnable bandpass filters to diversify attention patterns and incorporate strategies like large drop path rates and an auxiliary loss on globally pooled features to broaden token attention and enhance model optimization. Experiments on several image underst show that these innovations enable 1D causal visual representation to achieve performance comparable to 2D non-causal models in various tasks, including global perception and multi-modal understanding.", "questions": "Please refer to details in Weakness part above.", "limitations": "Some limitations of the work have been discussed in the submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Identifying over-focus issue in 1D causal visual modeling is an interesting and provide a starting point for the work in thissubmission. \n\n+ A \"de-focus attention\" strategy is introducted to address this issue. It incorporates learnable bandpass filters into the existing attention mechanisms to generate diverse attention patterns. The proposed idea is resonable.  A high drop path probability and an auxiliary loss is also proposed for better optimization.\n\n+ Experiments on seval visual understanding benchmarks show that via the proposed De-focus Attention Networks, 1D causal visual representation can match the performance of 2D non-causal representation in tasks involving global perception, dense prediction, and multi-modal understanding.", "weaknesses": "- For CLIP experiments, it is helpful to also report comparisons for cross-modal retrival on MSCOCO to follow existing routines of CLIP model comparisons.  \n\n- The major motivation is about addressing challenges in constructing unified multi-modal model, while the main experments are about image understanding tasks. To this point, seems discussions/explorations about impact of proposed method on MLLM is missing. I'm also curious about its potential applications for image generation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 4dxw for the thoughtful review and the insights provided. We appreciate the opportunity to discuss the enhancements and implications of our research further.\n\n---\n\n> **Q1: For CLIP experiments, it is helpful to also report comparisons for cross-modal retrival on MSCOCO to follow existing routines of CLIP model comparisons.**\n\nA1:  In response to your comments, we have conducted additional experiments using the CLIP model on the MSCOCO dataset. These new results further reinforce the findings presented in our initial submission, emphasizing the robustness and applicability of our approach across different data sets.\n\n| Model | Causal | Image Retrieval Recall@1 | Image Retrieval Recall@5 | Image Retrieval Recall@10 |  Text Retrieval Recall@1 |  Text Retrieval Recall@5 |  Text Retrieval Recall@10 |\n|-------|-------|-------|-------|-------|-------|-------|-------|\n| OpenAI CLIP-Base/32 | No | 30.4 | 55.0 | 65.7 | 49.2 | 73.4 | 82.4 |\n| OpenCLIP-Base/32 | No | 35.3 | 61.0 | 71.8 | 52.5 | 77.0 | 84.9 |\n| De-focus Mamba-Base/32 | Yes | 34.6 | 60.3 | 71.2 | 51.7 | 76.3 | 84.8 |\n\n---\n\n> **Q2: The major motivation is about addressing challenges in constructing unified multi-modal model, while the main experiments are about image understanding tasks. To this point, seems discussions/explorations about impact of proposed method on MLLM is missing.**\n\nA2: Thank you for your suggestions. In our research, we mainly study the application of causal modeling to images and achieve competitive performance. These results show a promising path towards a unified framework for joint causal modeling of images and text.\n\nHowever, the integration of these modalities remains a considerable challenge. Models such as Fuyu-8B[1] and Chameleon[2] that attempt to fuse image and text modeling are examples in this regard. Unfortunately, the details of these models and their training process are not publicly available, and they have not yet achieved the performance level of state-of-the-art MLLMs. This limitation underscores the crucial need for ongoing research in this domain to bridge the existing gap.\n\n---\n\n> **Q3: I'm also curious about its potential applications for image generation models.**\n\nA3:  Recent preliminary studies, such as \"Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation\"[3], have begun to uncover the potential of causal modeling in image generation. While our work primarily focuses on perception and understanding, these two works together indicate that causal modeling is a viable alternative to traditional methods across various areas of AI research.\n\n---\n\n[1] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023.\n\n[2] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.\n\n[3] Sun, Peize, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv preprint arXiv:2406.06525, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TiobKOyFMd", "reviewer": "Reviewer_J7C5", "summary": "This paper addresses the issue of over-focus in vision models by proposing strategies of using large and scheduled drop path rates and an auxiliary loss on globally pooled features. These strategies aim to encourage the model to focus on a wider range of tokens and improve network optimization. The paper is logically structured, the methods are novel, and the experimental results validate the effectiveness of the proposed approach.", "questions": "1. Can using large spatial dropout/dropblock or randomly dropping masked image tokens achieve similar effects compared to using a Large Drop Path Rate? If these methods are feasible, could additional experimental results comparisons be included?\n\n2. The authors need to provide further explanation on the rationale behind the positioning of learnable decay and learnable RoPE in the attention mechanism. Besides, can we directly deploy them on the feature maps before the attention mechanism?", "limitations": "1. The authors compared the parameter counts between models in terms of efficiency, but parameter count alone may not fully reflect model efficiency. I suggest the authors consider adding metrics related to model FLOPs and latency in Table 1 and Table 2.\n\n2.  Adding a classification loss to the model is not a novel method. \n\n3. The ablation study lacks experimental results comparing with other causal methods.\n\n4. There is a small grammatical error. In the paragraph on line 38, the phrase \"On one hand xxx, On the other hand xxx\" typically indicates a contrasting situation, but in this context, the authors may not intend to convey such a situation.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow.\n2. The authors have an insightful perspective on existing vision models.\n3. The research perspective of this paper is interesting and will be inspiring for future studies.", "weaknesses": "I am confused about some of the author's viewpoints. For example, in the introduction, the author mentioned that the causal modeling methods explored in existing vision models give them causal reasoning capabilities. \n\n1. Why do vision models based on standard Transformers without adding a causal attention mask still possess some level of causal reasoning abilities? \n\n2. Can we consider that vision Transformers inherently have causal abilities, and that existing improved methods simply amplify these models' causal capabilities?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer J7C5's feedback. It should first be clarified that the reviewer seems to have confused the concept of causal reasoning with 1D causal modeling in our paper. This may have led to some misunderstandings, such as the mistaken belief that we did not compare with other causal modeling methods in the experiments. And other concerns, like additional experiments with dropout, do not constitute valid reasons for rejection. Detailed responses are provided below.\n\n---\n\n> **Q1: Why do vision models based on standard Transformers without adding a causal attention mask still possess some level of causal reasoning abilities? Can we consider that vision Transformers inherently have causal abilities, and that existing improved methods simply amplify these models' causal capabilities?**\n\nA1: Our paper does not involve the concept of causal reasoning. There is a misunderstanding: causal modeling [1,2] does not refer to causal reasoning, it means that image patches are treated as time series like language, and the patches at the front of the sequence are used to predict subsequent patches. Whether current vision models possess causal reasoning abilities is orthogonal to our paper. We will add the definition of causal modeling in the introduction.\n\n---\n\n> **Q2: Can using large spatial dropout/dropblock or randomly dropping masked image tokens achieve similar effects compared to using a Large Drop Path Rate? If these methods are feasible, could additional experimental results comparisons be included?**\n\nA2: Thanks for your suggestion. It should be first noted that while these attempts are interesting, they do not diminish the contributions of our paper. We have conducted additional experiments and the results are provided in the table below. It's shown that using large spatial dropout or dropping masked image tokens is inferior to using a large drop path rate. \n\n|   | ImageNet Top1 Acc |\n|-------|-------|\n| Large Drop Path Rate | 82.0 |\n| Large Spatial Dropout | 80.9 |\n| Drop Masked Image Tokens | 80.2 |\n\n---\n\n> **Q3: (a) The authors need to provide further explanation on the rationale behind the positioning of learnable decay and learnable RoPE in the attention mechanism. (b) Besides, can we directly deploy them on the feature maps before the attention mechanism?**\n\nA3: (a) Please refer to L38-L42 in the introduction and L127-L130 in Section 4.1 for the reason of using learnable decay and RoPE in the attention mechanism. Specifically, the motivation stems from our observation of the \"over-focus\" issue. Learnable decay helps increase or decrease the network's emphasis on distant data. Learnable RoPE allows the network to focus on different aspects of data within various spatial frequencies. By employing multiple settings for both decay and RoPE, the network can create a more balanced and varied pattern of attention.\n\n(b) Learnable decay and learnable RoPE can not be applied to the feature maps, because the relative positions of different tokens should be taken into account.\n\n---\n\n> **Q4: The authors compared the parameter counts between models in terms of efficiency, but parameter count alone may not fully reflect model efficiency. I suggest the authors consider adding metrics related to model FLOPs and latency in Table 1 and Table 2.**\n\nA4: It should be noted that efficiency is not the focus of our paper. As in many other papers, parameter counts are reported to ensure a fair comparison between models of similar size, which is not directly related to efficiency. To address the concern, we test the FLOPs and latency of our model, and our \"de-focus attention\" strategy adds less than 1% to both the model's FLOPs and latency.\n\n---\n\n> **Q5: Adding a classification loss to the model is not a novel method.**\n\nA5: The novelty of our paper includes the observation of the \"over-focus\" issue in training causal models, the proposal of the \"de-focus attention\" strategy, as well as the demonstration that 1D causal models can deliver comparable performances to non-causal models (as noted by the summary of Reviewer WFuU, Reviewer 4dxw and Reviewer WAfJ). It should be emphasized that the use of auxiliary loss is only a part of our overall \"de-focus attention\" strategy.\n\n---\n\n> **Q6: The ablation study lacks experimental results comparing with other causal methods.**\n\nA6: We have compared with other causal methods in the main results, including RetNet[3], Mamba[4], Mamba-ND[5]. Please refer to Table 1 in the paper.\n\n---\n\n> **Q7: There is a small grammatical error. In the paragraph on line 38, the phrase \"On one hand xxx, On the other hand xxx\" typically indicates a contrasting situation, but in this context, the authors may not intend to convey such a situation.**\n\nA7: Thanks for your advice, we shall fix it in the final version. \n\n---\n\n[1] Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung et al. UL2: Unifying Language Learning Paradigms. In The Eleventh International Conference on Learning Representations, 2023.\n\n[2] Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309, 2023.\n\n[3] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.\n\n[4] Albert Gu, and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.\n\n[5] Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xivnba6H0C", "reviewer": "Reviewer_WAfJ", "summary": "This paper mainly improves the existing 1D causal models when handling visual inputs, generally 2D non-causal. The key insight of the authors mainly comes from Figure 1: the 1D causal models will over-focus to a few tokens instead of capturing the rich information from the whole image. To fix this issue, the authors propose the de-focus network. They first view the spatial decay and positional embedding as bandpass filters and learn that such filters lead to the capture of diverse information. In addition, the improvement in optimization, especially a significantly larger dropout, also encourages the model to make use of diverse sources of information. Finally, the authors experiment on image classification, object detection, and image-text training to prove the effectiveness of their approach.", "questions": "See weakness.", "limitations": "Yes, the authors have addressed such limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. In Figure 1 and Sec. 4.1, the authors make a very good demonstration and reasoning of their intuitions.\n2. The De-focus network is simple and reasonable, including both the bandpass filter part and the optimization improvement.\n3. The experiments across a wide range of fundamental tasks prove the effectiveness of the De-focus network.", "weaknesses": "I think the paper is good overall. I only have one question regarding the generalization of the method, which is not discussed in the paper.\n\nTransformer-based architectures like ViT in image classification and CLIP show certain generalizations to different resolutions. For example, people can train a model on 224x224 and inference on 336x336 by interpolating the positional embeddings. I am curious how well the de-focus network will perform under such generalization scenarios, compared with the 2D non-causal transformer-based architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer WAfj for the constructive suggestions. Please see our detailed response below.\n\n---\n\n> **Q1: Transformer-based architectures like ViT in image classification and CLIP show certain generalizations to different resolutions. For example, people can train a model on 224x224 and inference on 336x336 by interpolating the positional embeddings. I am curious how well the de-focus network will perform under such generalization scenarios, compared with the 2D non-causal transformer-based architectures.**\n\nA1: Thanks for your suggestion. We test the transfer performance under resolution 336 and report the results in the table below.\n\n|   | Resolution 224 (same as training) | Resolution 336 |\n|-------|-------|-------|\n| DeiT-Base | 81.8 | 81.6 |\n| De-focus Mamba-Base | 82.0 | 81.6 |\n\nThe results demonstrate that our De-focus Networks can also transfer to different resolutions effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper was reviewed by four expert reviewers and post-rebuttal received somewhat divergent ratings of: 2 x Borderline reject, 1 x Weak Accept, 1 x Accept. Overall, most reviewers agree that the proposed 1D causal attention is interesting and is a viable alternative to more traditional attention mechanisms. However, some concerns have also been raised by the reviewers, including with (1) inconsistent evaluation [WFuU], (2) effectiveness of a large drop path rate compared to large spatial dropout/dropblock [J7C5], (3) ability to generalize to different image resolutions [WAfJ], as well as a couple of others. In general, authors have addressed most of these comments in the rebuttal, including with additional experimental evidence. 

Unfortunately, [WFuU] has not engaged in the discussion; [J7C5] has engaged in the discussion and acknowledged that some of the concerns have been alleviated in author responses, but remained concerned with novelty of auxiliary loss. The remaining two reviewers appear happy with author responses and argue for acceptance. Overall, all reviews agree that "Contributions" of the work are "good" and most feel that "Presentation" is "excellent". Only [WFuU] raised issue with "Soundness", rating it "fair", while other reviewers rated it "good" or "excellent". 

AC had considered the reviews, rebuttal, discussion that followed and the paper itself. AC agrees that the work is interesting and may be broadly valuable (e.g., the learnable band pass filter can be broadly useful in attention mechanisms). While AC acknowledges that more experiments in additional tasks (e.g., VLMs) would make the work even stronger, this should not take away from contributions already made by the paper. Overall, AC agrees with more favorable reviewers that the work is solid, leading to an Acceptance recomendation.
Decision: Accept (poster)