Title: Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representation Learning; Causal Inference; Positional Encoding
Abstract: Positional encoding is essential for supplementing transformer with positional information of tokens. Existing positional encoding methods demand predefined token/feature order, rendering them unsuitable for real-world data with non-sequential yet causally-related features. To address this limitation, we propose CAPE, a novel method that identifies underlying causal structure over non-sequential features as a weighted directed acyclic graph (DAG) using generalized structural equation modeling. The DAG is then embedded in hyperbolic space where its geometric structure is well-preserved using a hyperboloid model-based approach that effectively captures two important causal graph properties (causal strength & causal specificity). This step yields causality-aware positional encodings for the features, which are converted into their rotary form for integrating with transformer's self-attention mechanism. Theoretical analysis reveals that CAPE-generated rotary positional encodings possess three valuable properties for enhanced self-attention, including causal distance-induced attenuation, causal generality-induced attenuation, and robustness to positional disturbances.  We evaluate CAPE over both synthetic and real-word datasets, empirically demonstrating its theoretical properties and effectiveness in enhancing transformer for data with non-sequential features. Our code is available at https://github.com/Catchxu/CAPE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method to embed the structure of a causal graph into a Transformer model. It firstly encodes causal graphs into hyperbolic space, translating causal strength and hierarchy into geometric distance and position, then converts them into a rotary form. Experimental results verify the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n- The motivation of applying causality-induced positional encoding is clear and significant.\n- The method is clearly described and the paper is well-written. \n## Weakness:\n- This paper could benefit from more discussions on the correctness of the causal structure learning step. See questions for details.\n\nQUESTIONS:\n1. To my understanding, causal structure learning usually requires some fundamental assumptions, e.g., causal faithfulness and causal sufficiency. It is unknown whether these assumptions remain true in the context of real-world datasets. Could the proposed approach remain effective when these assumptions are not satisfied, e.g., when latent confounders exist?\n2. A generalized nonlinear SEM model is used to infer the causal DAG. However, in many real-world scenarios, the DAG is non-identifiable from the data, so I think the inferred causal DAG should contain unavoidable errors under these scenarios. If this is correct, how does the proposed approach solve this problem? Moreover, if the proposed method can be effective under inaccurate causal DAG, could it be used in traditional sequential features?\n3. RoPE is not used as benchmark as they rely on predefined feature order. Is it possible to use RoPE or other order-based methods with the topological order of the causal DAG? I think the comparison could make the experimental results more comprehensive.\n4. As also mentioned in Sec. C.2, there are many casual structure learning methods. Why nonlinear SEM method is used in the causal structure learning step? Are there any other alternatives?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CAPE, a positional encoding method that learns causal structures among non-sequential features to provide Transformer with positional information related to causal structure rather than sequential structure. CAPE first uses generalized structural equation model (SEM) to identify causal directed acyclic graphs (DAGs) among features. It then embeds the identified DAGs into hyperbolic space and uses the hyperbolic surface model to preserve two key properties: causal strength and causal specificity. Finally, the hyperbolic embedding is converted into a rotated positional encoding form, which is compatible with the self-attention mechanism of Transformer. The authors provide theoretical analysis demonstrating that CAPE possesses desirable properties including causal distance-induced attenuation, causal generality-induced attenuation, and robustness to positional disturbances, and validate the method's effectiveness on synthetic data and multi-omics datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Well-motivated - Addresses the important practical problem that traditional positional encoding cannot handle non-sequential causally-related features, with significant application value\n2. Novel approach - Cleverly combines causal discovery, hyperbolic geometry, and rotary attention mechanisms\n3. Solid theoretical foundation and well-designed experiments - Provides rigorous theoretical analysis with comprehensive experimental validation\n\nWeaknesses:\n1. Limited baseline comparisons - Lacks comparison with more diverse baseline methods, with experiments confined primarily to biomedical domains\n2. Potential computational overhead - May increase computational costs, uncertain practical utility\n\nQUESTIONS:\n1. Computational efficiency - How does the computational efficiency compare? Can you provide complexity analysis?\n2. Generalization - Is the method still robust when dealing with larger causal graphs or noisy conditions? Does this approach still work in a causal diagram that only shows a partial set of causal relationships? Is it possible to modeling different causal strengths?\n\nLIMITATIONS:\nThe effectiveness in other domains remains to be confirmed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an end-to-end deep neural network for 1) causal discovery/causal strength estimation 2) causal graph encoding 3) self attention of non-sequential feature with causal positional encoding. It provides theoretical properties of the attention with respect to the positional encoding, verified by ablation experiments, and shows experimentally how the introduced encoding improves a transformer network compared to other encoding methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strong points**\n - Clarity: the paper is precise, well detailled, organized, and the extensive proof rely on unified and clear notations.\n - Quality: The theoretical results underline desired properties that are validated in ablation experiments, despite some theoretical guarantees remaining somewhat weak. Real data experiments were reported with standard deviation that seems to indicate that the improvements are statistically significant.\n - Originality: While the encoding of graph in Pointcarré space is not novel, its use to create positional encoding from *causal* information is, to my knowledge, novel.\n - Significance: Combining causal analysis and the more popular architectures of Deep Neural Network is an unsolved yet high impact problem.\n\n**Weak points**\n\n**Clarity**\n - Lines 212+: pointcarre encodings denote a point on a disk, so why is it that phi = c.e is an angle vector? The multiplication does not change the angle, just the magnitude of e, since e was defined using cartesian coordinates. Therefore, phi does not correspond to a coordinate system that involves angles.\n - Typo line 929, should be an inequality. The proof remains sound.\n\n**Quality**\n - Line 831+: There is nothing guaranteeing that A+ and A- tend to 0. In fact, the term Beta_i only depends on W_q, W_k, v_q and v_k, none of which involve the positonal causal aware encoding.\n - In fact, if a first point has pointcarre embedding 0 and another is (a,0,0,0...) with a varying from 0 to 1, the changes to matrix R will only affect the first 2D block. This countradicts A tending to 0 when dp tends to infinity.\n - The same example shows that for those two points, the attention (line 246) may have large upper bound (lower bound by symmetry) despite causal generality.\n - Going further, since feature encoding v and positional encoding phi are computed separately, and since rotation matrices have determinant 1, for every feature embedding v1 and rotation matrix, there exists and embedding v2 (perhaps unused), such that the attention score is ||v1||. To do so, it is enough to invert R and multiply by v1. Relying only on a rotation matrix is not sufficient to guarantee that the causal proximity affects each attention scores. Attention score bounds may only be **probabilistic**.\n - Proposition A.8 also falls short of the stated goal. It relies on that vm and vn are of same mean, are independent, vm and delta are also independent, so their attention cancel out on average. The only term left is the attention of vm with itself, reasonably positive. The problem is that we would like that for any vm and vn, reasonably far from each other, vm+delta has higher attention score.\n\nQUESTIONS:\nOther than making sure that the theoretical properties are correct, and that the plain text explanation properly reflects the strength of the guarantee (which would be my main concerns), I have one other question.\n\nThe greatest limitation of the encoding method used is that rotation matrices do not guarantee that distance on the pointcarre space result into null attention. This is due to the necessity of gamma being function of the difference in positional encoding, which results in the radius function being independent of the positional encoding. **What if gamma was function of the encoding difference AND causal generality?** While this differs from the usual setting, this could be fundamental in bridging Pointcarré space with positional encoding. Note that I did not carefully check if this suggestion is viable.\n\nLIMITATIONS:\nOn top of the limitations identified in appendix (which should be put in the main paper), the authors should mention that the rotation-based encoding does not entirely reflect the information encoded in pointcarré space. It might be possible to obtain better guarantees with a different positional encoding transformation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}