Title: CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Attention Models

Efficient Inference Methods

Computer Vision

Natural Language Processing

Representation Learning
Abstract: Transformers have driven remarkable breakthroughs in natural language processing and computer vision, yet their standard attention mechanism still imposes $O(N^2)$ complexity, hindering scalability to longer sequences. We introduce Circular-convolutional ATtention (CAT), a Fourier-based approach that efficiently applies circular convolutions to reduce complexity without sacrificing representational power. CAT achieves $O(N \log N)$ computations, requires fewer learnable parameters by streamlining fully connected layers, and introduces no heavier operations, resulting in consistent accuracy improvements and about a 10\% speedup in naive PyTorch implementations. Based on the engineering-isomorphic transformer framework, CAT's design not only offers practical efficiency and ease of implementation, but also provides insights to guide the development of future high-performance Transformer architectures. Finally, our ablation studies highlight the key conditions underlying CAT’s success, shedding light on broader principles for scalable attention mechanisms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Circular Convolutional Attention (CAT), a Fourier-based alternative to standard attention that replaces the O(N^2) complexity with a more scalable O(N log N) formulation. By leveraging circular convolutions in the frequency domain, CAT preserves representational capacity while reducing computational cost and parameter count by simplifying the fully connected layers. The method reports ~10% speedups in naive PyTorch implementations without relying on heavier operations. CAT demonstrates improvements in accuracy and perplexity across both vision and language tasks, under both masked and causal transformer settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed approach achieves O(N log N) complexity by leveraging the fourier transform and viewing attention as a circular shift matrix, which is a creative and well-motivated direction. I believe there is novelty in this approach, which makes it an interesting solution.\n\n2. The authors target a known bottleneck—attention's quadratic complexity—so the potential savings are practically meaningful, especially in long-sequence settings. Reducing parameter count and maintaining accuracy while improving speed is a valuable contribution.\n\nWeaknesses:\nDespite the well-motivated and creative interpretation, I have some concerns:\n\n1. Limited Baselines: The paper primarily compares CAT to a standard baseline model, without including other relevant efficiency-focused attention methods. Notably, BlockSparse Attention variants and Grouped Query Attention (GQA) are not included. These are commonly used in practice to reduce computation and memory. Without such comparisons, it's unclear how CAT performs relative to the current state-of-the-art. The discussion on MHA in the main paper is sufficient; however, MHA won't have savings natively (i.e, parameters are still full rank, only head-independence aids in performance). In a case like GQA, LLama3 may have 1/4 output channels on KV, meanwhile Qwen2 is far less (~1/8), so to validate the effectiveness of CAT it should be compared with current methods of compute reduction.\n\n2. Training from Scratch: It is not clearly stated whether CAT can be applied to existing pretrained models or if it requires training from scratch with new parameters. If the latter, this significantly limits its adoption potential, especially in settings where full retraining is infeasible.\n\n3. Ambiguity in Efficiency Target: The paper does not clearly distinguish whether the proposed method targets training efficiency, inference speed, or both. These have different runtime bottlenecks and trade-offs. Furthermore, the absence of a detailed runtime profile or FLOPs breakdown for the baseline makes it difficult to assess the real-world impact of the claimed 10% speedup.\n\n4. Memory Reduction Claims: The claim of reduced parameter count is made, but no concrete statistics are given for memory usage or parameter reduction in practice -- the same applies to the latency/runtime; very little information is provided about it.\n\n5. Representation: Since CAT replaces traditional attention weights with a form of circular convolution (circular shift matrix), it's important to validate that representational capacity is preserved. However, the paper lacks any qualitative or visual support (e.g., attention maps) to demonstrate this.\n\n6. Unclear Scope: In Figure 2, the improvements seem to target the V projection paths. However, with the original goal being to optimize the N^2 component (i.e, QK) it begs the question that if the V is considered, why not the linear layers in the FFN, which would have similar scaling to the linear V projection.\n\n7. Presentation: The paper’s explanation of the method, particularly in Figure 1, lacks clarity. The overall changes to the attention architecture are not well localized or visualized. This makes it difficult to understand exactly what components are being modified or replaced. Overall, the writing clarity/quality (and inclusion of related works, which is missing many sparse attention works) needs significant improvement.\n\nQUESTIONS:\n1. Compatibility: Can CAT be combined with other efficient attention mechanisms like GQA or FlashAttention, or is it intended as a standalone replacement? Please provide numerical comparisons for this.\n\n2. Deployment: Does CAT support transfer to pretrained models (e.g., via fine-tuning), or does it require full training from scratch? If from scratch, then this should be discussed in the paper regarding its practicality.\n\n3. Scope of Modifications: Are the convolutional replacements limited to QK projections, or can the same principle be extended to FFN layers or other modules?\n\n4. Target Efficiency: Is the focus of CAT primarily on improving inference speed or training efficiency? Can the authors provide wall-clock runtimes and detailed bottleneck breakdowns for the baseline and method in the target setting? Also, consider both the FLOPs scaling and memory costs.\n\n5. Visual Evidence: Can the authors include attention map visualizations or other forms of qualitative analysis to demonstrate that CAT preserves the structure and utility of standard attention?\n\nCurrently, my biggest concern with the work is that I am uncertain whether it really performs better than existing, well used and performing sparse or optimized attention algorithms such as GQA.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Circulant Attention Transformer (CAT), a variant that replaces dense soft-max attention with a circulant structure amenable to FFT, yielding sub-quadratic theoretical complexity. It formalizes four “Engineering-Isomorphic Transformer (EIT)” principles showing how CAT integrates seamlessly into standard Transformer blocks, then supplies proofs-of-concept on vision (ImageNet/ViT, CLIP) and language (Transformer-XL, GPT-2) tasks. Experiments indicate comparable or slightly better accuracy in several masked- or average-pooling settings while reducing memory and wall-clock time, and the approach generalizes across modalities without architectural changes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* Researching and optimizing foundational model architectures is a valuable and important direction.\n\n* This work offers clear complexity analysis and provides an explanation for why circulant softmax preserves key properties of full attention.\n\n* In terms of experiments, the authors demonstrate the method’s viability on both image (ViT, CLIP) and text (language modeling) benchmarks, rather than focusing on a single domain.\n\n* Even with a naïve PyTorch gather implementation, the paper reports about a 10% wall-clock speed-up and a 25% peak memory reduction on 256-token sequences. The authors suggest that larger gains are possible with custom FFT kernels.\n\n**Weaknesses:**\n\n* The scale of the experiments is somewhat small and does not reach the long-sequence capability claimed in the paper. All experiments are limited to 256-token or 224×224 image scales, while the paper’s motivation emphasizes “substantially longer sequences” (see lines 49-50: “opens new directions for longer-sequence modeling across language and vision tasks”). The experiments do not include longer context benchmarks like Long Range Arena, PG-19, or Books3, which raises concerns about the feasibility and stability of the method in real long-sequence scenarios. Although the authors mention in the conclusion that experimenting with sequences of tens of thousands of tokens is still an open challenge, evidence from only 256-token experiments is not very convincing for supporting the method’s effectiveness in long-sequence modeling.\n\n* Missing baselines: The experiments only compare standard Attention with the proposed CAT method, without comparing to other efficient attention methods such as BigBird or Longformer as mentioned in the paper. While the authors state that they tried to reproduce linear attention but gave up due to NaN issues, they did not provide details about tuning or attempts to stabilize the method.\n\n* No end-to-end latency and throughput tests: Although the paper provides theoretical analysis showing CAT’s advantages in algorithmic complexity, the experiments mainly focus on comparing CAT to standard attention in terms of performance, lacking more direct end-to-end latency and throughput tests. For example, in text generation tasks, it would be helpful to record the end-to-end generation time (in seconds) and throughput (in tokens/s) under different input/output settings. Reference can be made to works like “H$\\_2\\$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models” and “Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs”.\n\n* Lack of explanation for experimental phenomena: In at least half of the experimental results, the CAT method performs worse than the baseline, but the CAT-alter variant performs better than the baseline. This is confusing, because CAT-alter is essentially a mixture of CAT attention and standard attention. Intuitively, CAT-alter’s performance should fall between CAT and standard attention, not exceed the baseline when CAT itself underperforms. For example, Table 1 shows that under the pool type “token,” CAT performs worse than Attention, but CAT-alter is significantly better than Attention. Similar results can be seen in Table 2 for the LLM type “causal” experiments.\n\n* The statement “avoiding many of the pitfalls (e.g., numerical instability, partial token coverage) seen in previous approximations” (lines 29-30) is not easy for readers unfamiliar with efficient attention research to understand, and the high level of summarization does not resonate well. Also, I assume “previous approximations” refers to previous work on attention approximations? If so, I suggest the authors add proper citations.\n\nQUESTIONS:\n* The abbreviation “FFT” (line 32) is not written out in full the first time it appears, nor is it explained.\n* “RFFT” (lines 136-137) is not explained.\n* Some of the font sizes in Figure 1 are too small and hard to read without zooming in. Also, the icons for different operations in Figure 1 are a bit small. Since there is still a lot of space in Figure 1, it is suggested to increase the font size and the size of the icons.\n* Typo: In Table 1, the “A” in “attention” in the fifth row and the third row from the bottom should be capitalized.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is aims to improve the quadratic complexity of attention, while keeping the softmax component of it. The main approach is to define an attention operator that uses circulant operator, which can be efficiently computed via FFT. Experiments are conducted to compare the accuracy and efficiency of the proposed CAT with respect to standard self attention.\n\nSTRENGTHS AND WEAKNESSES:\n## Significance\n\n1. The first contribution of “EITs: a new framework” is questionable.   \n- Firstly, for a framework/class to be non-trivial, one needs multiple instances under the framework. However, it appears that the only instance the authors mention is CAT. Thus, EIT seems more like the properties of the proposed method, rather than a framework.  \n- Moreover, I do not understand why “Softmax preservation” is a property that is a priori ideal. At the end of the day, what matters seem to be efficiency (space / time), accuracy, theoretical grounds, among others. It is unclear whether “Softmax preservation” is sufficient or necessary to any of these metrics.  \n2. The proposed CAT is not well motivated and justified.   \n- The definition of CAT comes out of the blue, without any motivation on why one should use this design.   \n- The proposed CAT loses a property that self-attention has, which is permutation equivariance.   \n  1. In self-attention, if one exchanges the first two tokens in the input x, the output tokens are the same as the original ones, except that the tokens at the first two tokens are swapped.   \n     2. In CAT, one can see that $$o\\_1=z\\_1^\\*v\\_1 \\+ z\\_2^\\*v\\_2 \\+ \\\\dots \\+ z\\_Nv\\_N $$ $$o\\_2 \\= z\\_N^\\*v\\_1+z\\_1^\\*v\\_2 \\+ \\\\dots \\+ z\\_{N-1}^\\*v\\_N $$, which does not satisfy permutation equivariance. \n\n  Why would this be a good choice? \n\n3. The efficiency claim of the third contribution lacks detailed evidence.  \n- The paper mentioned multiple times that they gained 10% speedup. For the results in tables 1-3, what are the running times? Which libraries are self-attention and FFT implemented in?   \n- The authors claimed that calling FFT often “offsets theoretical advantage” \\- did the authors observe time or space usage? Again, how was FFT implemented? There seems to be a few efficient FFT implemetnations \\- have the authors tried?  \n  - [https://developer.nvidia.com/cufft](https://developer.nvidia.com/cufft)  \n  - [https://github.com/fkodom/fft-conv-pytorch](https://github.com/fkodom/fft-conv-pytorch)  \n- The ideal comparison would be to compare efficient self-attention (e.g., flashattention) with efficient CAT implementation, since this is the regime that would make an impact. \n\n## Clarity\n\n1. Notations: the notation convention is a bit confusing: Q, W\\_Q and x are all matrices, but x is the only one lower cased.   \n2. I was trying to understand the variants in the ablation study. Can the authors write down the expressions of each of the variants just as how they define CAT in section 4? Otherwise, it is hard to understand what these variants are.\n\nQUESTIONS:\nThe reviewer would love to see the concerns in the above addressed and revise the rating.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}