Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features
Authors: Kaichen Xu
Yihang Du
Mianpeng Liu
Zimu Yu
Xiaobo Sun
Venue: NeurIPS 2025 poster ()
Keywords: Representation Learning; Causal Inference; Positional Encoding
Primary area: machine_learning_for_sciences

Abstract: Positional encoding is essential for supplementing transformer with positional information of tokens. Existing positional encoding methods demand predefined token/feature order, rendering them unsuitable for real-world data with non-sequential yet causally-related features. To address this limitation, we propose CAPE, a novel method that identifies underlying causal structure over non-sequential features as a weighted directed acyclic graph (DAG) using generalized structural equation modeling. The DAG is then embedded in hyperbolic space where its geometric structure is well-preserved using a hyperboloid model-based approach that effectively captures two important causal graph properties (causal strength & causal specificity). This step yields causality-aware positional encodings for the features, which are converted into their rotary form for integrating with transformer's self-attention mechanism. Theoretical analysis reveals that CAPE-generated rotary positional encodings possess three valuable properties for enhanced self-attention, including causal distance-induced attenuation, causal generality-induced attenuation, and robustness to positional disturbances.  We evaluate CAPE over both synthetic and real-word datasets, empirically demonstrating its theoretical properties and effectiveness in enhancing transformer for data with non-sequential features. Our code is available at https://github.com/Catchxu/CAPE.

Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features
Kaichen X Yihang Du?, Mianpeng Liu?, Zimu Yu², Xiaobo Sun Deppartment of Computer Science, Emory University School of Statistics and Mathematics, Zhongnan University of Economics and Law School of Medicine, Department of Human Genetics, Emory University
Abstract
Positional encoding is essential for supplementing transformer with positional information of tokens, Existing positional encoding methods demand predefined ken/feature order, rendering them unsuitable for real-world data with sequenttal yet causally-related features. To address this limitation, we propose CAPE, novel method that identifies underlying causal structure over non-sequential features as a weighted directed acyclic graph (DAG) using generalized structural equation mod- eling. The DAG is then embedded in hyperbolic space where its geometric structure is well-preserved using hyperboloid model-based approach that effectively cap- tures two important causal graph properties (causal strength & causal specificity). This step yields causality-aware positional encodings for the features, which are converted into their rotary form for integrating with transformer's 's self-attention mechanism. Theoretical analysis reveals that AAPE-generated rotary positional encodings possess three valuable properties for enhanced self-attention including causal distance-indee induced attenuation, causal canerality-inducee attenuation, and robustness positional disturbances. We evaluate CAPE over both synthetic and real-word datasets, empirically demonstrating its theoretical properties and effectiveness in enhancing transformer for data with non-sequential features. Our code is available at wttttt://// //tp//ggttttttt//// com www.cooo//cccceeeco CCatchxu/CAPPE
Introduction
Transformer has become the cornerstone modern deep learning models, powering advances in natural language processing (NLP) 225 computer vision 6 speech and audio processing [ 10], and multimodal learning UHE At the core of transformer the self-attention mechanism, which effectively captures dependencies among sequential elements. However, this mechanism inherently
and cormutation-invariann [14.. Positional information is crucial in learning semantics because it encodes sequential dependencies analogous to directional causal structure
[ as opposed the undirected associations captured by self-attention. To inject positional information into the transformer architecture, plethora of strategies have been proposed to generate positional encodings that are integrated with contextual embeddings. These approaches include fixed sinusoidal functions [1, trainable absolute or relative positional encodings 2 5 17 and more
recent rotary positional encodings (RoPE) [18 [9]. Notably, these methods generally assume natural, inherent ordering in the data, such the sequence of words in sentence the spatial arrangement of image patches.
'These authors contributed equally. 'Coreepondeene: xaun28demory .dd
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(Sequentia data
Sequence
Static toxen embedding
CAPE
encodes
causal
positions
Conrantional poostion encoder
Contertull zed. ward embedding
3
Postion encoding
Explicit position
Non-sequential teatures
X Not applicable
Non connseqquettal data
Sample
U1
0g
5
position encoding
Causality-aware
8 TLL Conontama
Cornenkullzed caunality sware feature entbediing
Pusittion Gicocer
Static feature embeddiing
Cousal positionll encading
Cocua suaciicy
grash
Figure 1: Causal position information can be utilized in place of explicit position information for wranffrrmr-bbsse representation learning of data with non-sequential yet cousally-related features.
However, this assumption breaks down in the context of many real-world datasets, where rows represent independent observations, columns represent non-sequential features, and entries capture the quantity or state of each feature for given observation. For example, in biomedical sciences, multi-omics data, such as transcriptomics (00 and proteomies [21], measure gene and protein expression levels within samples. The expressions of genes and proteins lack predefined sequence, despite their intricate causal order. Similarly, economic studies often involve -on-sequettial but causally linked economic indicators collected across different regions countries. Therefore, existing positional encoding methods struggle to capture these underlying causal structures, limiting the applicability of transformers to such data. In response, some preliminary efforts have emerged within specialized domains. For example, in single-cell transcriptomics. transformer -aserr---dddddeee foundational models seek to generate distributed gene representations from large corpus of scRNA-seq datasets. To impose a pseudo-order on genes, some of these models [22 23 organize trainable gene embeddings into sequences based on their expression levels, while others [24 25 use pretrained, static gene embeddings as pseudo-pont sosuddopssiiionll encodings. However, a critical limitation of these methods is that they neglect the underlying causal structure among genes.
In this study, we propose Causality-A Aware Position Encoder (CAPE), a novel method for generating causality-aware positional encodings that extend the transformer architecture to data with non- sequential yet causally-related features. Initially, CAPE leverages a generalized structural equation model (SEM) to model the hidden causal structure among features weighted directed acyclic graph (DAG), which is efficiently identified through neural variational inference with a constraint- based, continuous optimization technique (8)228. Next, inspired by the theory of special relativity, which links causal connections between events to their relative positions in hyperbolic spacetime we utilize the hyperboloid modef"tto embed the DAG into the hyperbolic space, which is known for its ability to model tree-like like networks commonly seen in DAGS [31] Specifically, nodes in the DAG represented as points
29 30
Riemannian manifold, with their positions learned through regularized graph contrastive learning, optimized via Riemannian stochastic gradient descent (RSGD)
33). This approach ensures that the learned embeddings capture two critical causal graph properties, including causal strength and causal specificity (see Section 3.4 (33], thus preserving the original causal structure of the DAG. Finally, CAPE converts the hyperbolic positional encodings into rotary form, a causality cansaeityinduued induced version of RoPE [18]. This form offers several key benefits, including compatibility with linear self-attention [18 and enhanced understanding of contextual knowledge [19] We further theoretically demonstrate that the causality-aware rotary positional encodings offers three valuable properties in computing elf-attention: Attention strength attenuates with increasing causal distance (causal distance-induced attention attenuation, Section 4.1 or decreasing causal specificity (causal generality induced astention attenuation, Section 4.2). and attention scores exhibit robusiness to positional disturbances (Section In summary, our main contributions include:
We propose CAPE, a novel method for generating causality-aware positional encodings for data with non-sequential yet causally-related features. It eliminates the need for predefined feature ordering required by conventional positional encoding methods, while incorporating causal structure information into transformer-based representation learning.
htttp::/een.wikiieddaa..o httppp/dw..oog//..t////otoooooot

---

CAPE adopts a hyperboloid model-based approach to embed causal graphs, effectively capturing two fundamental causal graph properties: causal strength and causal specificity.
We theoretically demonstrate that CAPE-generated rotary positional encodings possess several valuable properties that enhance the effectiveness of self-attention.
We empirically validate CAPE's theoretical properties using synthetic data, and evaluate its effectiveness in enhancing representation learning of non-sequential data using various real-world multi-omics datasets.
Step!
Step Map Hyporbolic Space
Step IN Transform into Rotery Posttonal Encoding
Causal Structure Leaming
Hyperbeloid
Pk
SE54-aased
undi stactivery
Pope boildd mode: ssped
222
311
& Adds
Norm 8 Apw
Causelky-Aware, Rotary Postional Ensoding
NaB
0.5
Puarsa bul
Weighted DAG
CAPE
Backbone Network
Causally-agnosstic feature ambedding
Figure Overview of CAPE.
2
Related Work
postpone this section to Supp. Cue to limited space.
Methodology
3.1 Preliminary
Let V = {
L. be a sequence of input tokens (e.g., words and image patches). The conventional procedure of applying standard transformer [I] to V can be described as:
G TM = T(A(F(u) sp1,, ... FFu,, (PM))
(I)
Here, Vj e [1. [1... MJ. ūy and by represent the position-aware, contextualized embedding and static, pretrained embedding of token uy respectively. 4i is the positional encoding of the j-th position. F denotes the function for fusing and yp, A represents the self-attention function, and T is the transformer function.
When V consists of non-sequential features, cannot be directly derived from predefined sequential order. such cases, we assume that { are causally related and organized into tabular measurement dataset X E RNN. where row represents the i-th observation, the j-th column corresponds to ej. and X,, denotes the quantity of " measured in x; For example, ey might represent gene and Xúj the read counts of gene in cell i. We aim to derive causality-aware positional encodings from as:
( M P(X),
(2)
where P denotes the causality-aware positional encoding function. We then inject {4,, }MM into the transformer architecture as:
6 S ((A(F(G(())) (. Pow))) Vi = 1. N (3)
where T represents the contextualized, causality-aware embedding vy within the i-th observation. G is a function to generate contextualized, causality intermediate feature embeddings (see

---

Supp. F22 for examples of 9). We can further obtain observation-level embeddings as:
h' Agg 0j. (M,)
where Agg denotes an aggregate function (e.g., mean or max pooling), h' the embedding of th observation. These contextualized, causality-aware feature embeddings and observation- embeddings can be used in downstream tasks for improved performance, shown in Section 5.2
3.2 Methodology Overview
As stated in the previous section, given set of non-sequential, causally- related features V and their associ- ated measurement data X, our goal is to generate contextualized, causality- aware positional encoding Pu,, Rd for each Uy E V. To this end, CAPE introduces an integrated three-step framework. In Step / (Section (.3). CAPE identifies the causal structure
Table I: Summary of main notations.
Notations
Descriptions
N
Number of observations.
M
Number of variables.
R MMM
Adjacency matrix of causal graph.
d
Dimensionality of positional encoding.
Dimensionality of feature embedding.
Pe, IRd+1 over V as weighted DAG G(V,E) where the node representing features, & is the edge set represent- ing causal relationships, and the edge weights quantify causal strengths. The presence and weights edges are inferred using a non non-lineat SEM that captures complex, potentially non-linear causal dependencies. In Step II (Section 3.4). the identified G is embedded in hyperbolic space using the hyperboloid model, assigning each Vj E V a positional embedding Pv, gd+1 where d is the dimensionality of the hyperbolic space. These embeddings are optimized using regularized graph contrastive loss that accounts for both causal strength and causal specificity, effec- tively preserving the original causal structure of G. In Step III (Section (35).. hyperboloid positional embeddings are mapped into a unit Poincaré ball via diffeomorphism before being transformed into their rotary form for modulating feature-wist attention scores in the transformer. We will elaborate each of these steps in the following sections.
Hyperbolic embedding.
Cuy E Rd
Poincaré ball embedding.
PPy E Rd
Rotary positional encoding.
E R
Static feature embedding.
3.3 Causal Structure Learning (Step I)
CAPE infers the causal structure over V using generalized nonlinear SEM defined as:
+ g( (Z).
where f N M N x M is nonlinear function that models the functional relationships among observed features (endogenous variables), AA E M is a directed, weighted adjacency matrix representing the hidden causal graph G(),E), and g N M N M models the distribution of unseen noises Z E N (exogenous variables). Given the universal approximation theorem [34, we let f be a multi-layer perceptron (MLP), and g( 7 = E NxM ~ (N, I) for simplicity, where A denotes the Gaussian distribution. Then Eq. (5) can be reformed as:
Z = Enc X A, We) (X - A),
X = Dec( Z A, Wa) = r (2(I -A)-1,
where Enc acts as an encoder mapping X to Gaussian noises Z, while Dec serves as a decoder that recovers X from Z, similar to the encoder and decoder of variational autoencoder (VAE) (35 Here, A, We. Wa can be estimated using a variational inference approach, with an evidence lower bound (ELBO) training objective (27:
LELBO (2)x) [log pl x KL (o(Z)))lp(Z))
(8)
Reconstruction Loss
where p( 2) the reconstruction distribution, q( (2 prior distribution. We also impose regularization term ||A]]] to encourage sparsity, and smooth
()X+ the variational posterior, and p( Z) the

---

constraint (AA): tr (e
M
0 to ensure the acyclicity of A (26). where @ denotes the The overall loss function then reads:
element-wise product (see Supp. A.I
min We. A
LELBO + >,|1A|11 S.t. ((A)= = 0,
where A 2 is the regularization coefficient.
To solve Eq. (9) efficiently. we employ the augmented Lagrangian method [3), yielding the following unconstrained subproblem:
min W., Wa.A
fDAG = LELBO + XxIlA11 + (I(( + ah(A),
(10)
where a is the Lagrange multiplier and p > 0 is the penalty parameter. The optimization proceeds by alternating updates [27]. Finally, a threshold т >> 0 is applied to A to prune noisy, false-positive
causal edges, A + A O I[[A| r). See Supp. G.3 for sensitivity analysis of r.
3.4 Mapping Causal Structure to Hyperbolic Space (Step II)
To translate the identified causal graph into spatial positions while preserving its geometric structure,
we project G(V, E) into hyperbolic space using the hyperboloid model. Specifically, each node
My E V is assigned an embedding Pv, E Rd+1 in a Riemannian manifold L = (H 9t), where
g: 1,, 1,,1,.........,. 1) E [[(+1))(d++)) denotes the metric tensor and where
Hd = {p = (p p) E Rd++1
(p.P): 11, 0}.
(11)
Peom. Pu.)i P... 9: Pv., -P -Pe.. Pv.. P Po..
(0) (0)
(12)
denotes the upper sheet of two-sheeted hyperboloid with origin Po = (1, in (d 1) dimmennioa Minkowski space (37]. The distance between two points P : Pv, E Hd reads
(P Pe..) = arcosh(- (Pe.. ) 38). based on which two critical causal graph properties are
defined as:
Causal strength : ofmms Un) x di Pv,)
(13)
Causal specificity: ((Um) (p.... Prm 1 Pr...
(14)
where ||-|| denotes the Euclidean norm. Intuitively, as shown in Fig. 2 the strength of the causal relationship between Um and Un attenuates as their hyperbolic distance increases, reflecting their weaker connection in the causal graph. Meanwhile, since hyperbolic space can be thought of as a continuous analogue to discrete trees with roots near the origin [39]. a causally general feature (e.g., root feature that is causally related to many its causal descendants in the causal graph) should poise close to the origin. To implement these properties into the positional encodings, we adopt a regularized graph contrastive learning framework, with the objective:
M
min dog Pop Edd
LH =
Σ
Loon (P,,) + XpP(Pe)),
(15)
M
c dd (pom .on)
Leon (Prm))
A log
(16)
E dd( Pua + En'eN (m²)
di (pem Peae)
NEN (Wm)
2(p ) = Te di Po),
(17)
where Leen is the contrastive term, Ω is the regularization term, and Ag is the regularization weight. In Lcons Nt (Em) C V denotes the set of positive samples of Um. consisting of those connected to Um via k-hop (k defaults to 2, see sensitivity analysis in Supp. G.3 causal paths in G, while No (Um) V (m) denotes its set of negative samples. |Amm] reflects the estimated causal strength between Um and va- By pulling closer features with strong causal relationships (positive pairs), while distancing those with weak causal relationships (negative pairs). Lcon ensures the causal strength property. In 82 Tem E I is the m-th value in the PageRank vector π E (0, 1) as:
P = 1
P (1 W) +
M
(18)
PT stt. π I =
5

---

where Dm denotes the diagonal in in-degree matrix of |A], P is the in-degree normalized transition matrix, and 1r M M is the transition restart matrix ensure P is strongly connected and an ergodic Markov chain w E (0, 1) is the relative weight for the restart matrix. is essentially steady-state probability vector with larger values for more causally general nodes (e.g., those with more outgoing edge: Consequently, features with larger causal generality are more penalized by 12 forcing their positions to be close to the origin Po. thus ensuring the causal specificity property.
To minimize Lu in Eq. (1)).
RSGD update Puy in iterations. Specifically, we first compute the Euclidean gradient V Pu, Lu at Pu,. which then converted into Riemannian gradient as:
VR Lu = 9 -V. LH-
(19)
Pry
Next, the Riemannian gradient direction projected onto the tangent space at Pa; via an orthogonal projection (see Prop. A.4) as:
VT
Cu = =propp;; Pay
(VR Cu)
(20)
Po; is updated along the direction of VI Lx in the tangent space with a learning rate of > 0, and Ps;
then retracted onto the hyperboloid via an exponential map function (see Prop. A.3) as:
Poj cxPp., (-7- VT Pr; Cu)
(21)
Due the closed-form computation of the geodesics on the hyperboloid, this optimization is computationally efficient (see Supp. A.2 for details).
3.5 Transforming Hyperbolic Positional Encoding to Rotary Form (Step III)
As demonstrated in [18 and [19, rotary positional encodings exhibit the advantages of compatibility with linear self-attention and enhanced understanding of contextual knowledge. Adherent to this notion, we first map the optimized positional encodings {P0, J j=1 from the hyperboloid into Poincaré ball as {e,, J.M via the diffeomorphism fa : H.d Bdd (see Supp. A.2.3 for details).
The Poincaré ball is Riemannian manifold pd = (Bc, 9p). where Bd = {e E R- ||el| < 1}
represents the open d-dimensional unit ball and the metric tensor 9p = (1-
I is conformal
transformation of the Euclidean metric I. This mapping is motivated by the fact that the Poincaré ball, with spherical geometry centered at the origin Oa, is more naturally suited for rotary encodings. Importantly, since it also represents hyperbolic space, the two causal graph properties (causal strength and causal specificity) are preserved (see Eqs. (3 and (44).
To transform the Poincaré ball embeddings {e,, into their rotary form for injecting into the transformer architecture, we refine the standard query-key mapping and inner product used in the self-attention mechanism, in way similar to RoPE [8]:
dum = Ig( evm)
ki, = In (vn ev.):
(22)
(d ) A (v ofy (erm ev,.))
(23)
where q:., and ko,, are the query and key derived from Um and U, in the context of observation x;, respectively. vi := G(v, ax)) ) E RD denotes the contextualized, causality-agonistic embeddings (see Section B.I). I. and It are functions that inject positional encodings into queries and keys, respectively. A is an attention scoring function that accounts for the relative causal position between Um and Un. which is represented as 7(e..: Cv Supp. A.3 gives the explicit solutions to Igp Ig. and A as:
Ig (uis ev... == R(Y W vi = R((pv..) Aum. Ie Ie ev,,) R(Po,, W R(yo,, (
(24)
'By the unique.
theorem, the left eigenvector π of the largest eigenvalue (Amax 1) of P is
$ Seee Supp. B for discussion of why we use π rather than out-degree
FFollowing RoPE, positional encodings are only injected into keys and queries, not values.

---

A (v 2n (evm: Cv.))
(g
R(Gs. - -uum >um)),
(25)
where fo == cep denotes vector whose components used the rotation angles in the sub- sequent rotary embedding, c = T/4 constant scale factor to control the range of angles and (evm Cv.. Btm ev. R((Px)) is a rotation matrix induced by Pe:
T((p!))
0
r(pb)) (2)
0
(p!!) sin( (0)
sin(sp!)) cos(
R(G,)
0 r(p!")==
(26)
0
0
r(pp!))
(4)
where d
2
4 Theoretical Properties of Causality-Aware, Rotary Positional Encoding
4.1 Causal Distance-Induced Attention Attenuation
As demonstrated in Prop. 4.1 and Remark 4.1 injecting CAPE-generated positional encodings into the self-attention calculation allows two features be assigned reduced attention scores when they are causally distant.
Proposition 4.1. Given v'n and vi the attention scoring function A in Eq. 25 is bounded by A > and A satisfying:
OA+ (win eum - ev.) I 20,
Cum
0A~ (vnn 1/ Cum Enn,)
0,
(27)
ddp(erm. Ev..)
ddp(er., Cv..)
where dem Bv,) is the distance between eum and Eu,, on the Poincaré ball manifold, computed
||eum Cv., 112
dp(enm Cv,)) arcosh 1 21+2
(28)
Cv. 12)
Ceu 122)
The functions A+ and A are given in Supp. A.4
Proof. See Supp.
A.4
Remark 4.1. As the causal distance dplet evn) + +oo, both A+ and A attenuate and converge towards smaller magnitudes (though not necessarily to 0). Since bounded between A+ and A=, its range of possible variation also shrinks significantly.
4.2 Causal Generality-Indueee Attention Attenuation
As discussed in Section 3.5 the causal specificity of um increases with |/e,,,, in hyperbolic space. Here, we further define the causal generality unit Poincaré ball manifold below.
Definition 4.1 (Causal generality unit Poincaré ball manifold). Given a causal graph G(V, E) embedded in the Poincaré ball manifold pd = (Bdd 9p), where Bdd {e Rd |ell 1} the causal generality of node Um E V is defined as the.. := 1- || ev 1l-
Causally general features (c.g., those that influence many other features) distribute their attention more broadly, resulting in lower attention scores for each of their individual causal descendants. Consequently, their attention scores tend to span a narrower range compared to more causally specific features. For example, both the "Big Bang" and "amino acids" are causes of the "emergence of life". but the former is a more causally general event, as represents the origin of all things. Therefore, the "Big Bang" should receive less attention than "amino acids" when reasoning about the origins of life. This property emerges from the attention scores computed with CAPE-generated rotary positional encodings, formally demonstrated in Prop. 4.2 and Remark 4.2
7 is set be m make sPR small but not negligible angle
7

---

True
Inferred
Poincaré Ball Embeddings
Positional Encoding
XX
Coordinates
1.0
1D
Norus
0.16
0.25
-0.25
0.38
0.5
2
0.11
-0.05
-0.18
0.25
b
0.0
0.01
True Adj Matrix
Inferred Adj Matrix
0.36
0.87
0.16
0.78
-0.5
40.23
0.45
0.47
0.70
0.39
4.19
.011
0.29
-1.0
-0.5
-0.5
-1.0
0.15
0.41
0.28
0.52
0.0
0.0
0.5
0.5
x
1.0
1.0
Figure 3: (a) True and inferred DAGs on synthetic data. (b) True and inferred adjacency matrices. (c) 3D visualization of Poincaré ball embeddings of nodes in pa
Proposition 4.2. Given iin vi and fixed causal distance d, (evm Ev,)) in the Poincaré ball manifold defined in Def. 4.1 A and A satisfy:
0A+ (w 2/ Cum Cv.)
0A (w ot, Cum Cv.). 2
S 50,
(29)
The same holds for oow
Proof. See Supp. A.S
Remark 4.2. As the causal generality ben. 1, A upper boundary A+ and lower boundary A asymptotically attenuate towards fixed constants >> and -a < 0, respectively (See Supp. AA.5).
4.3 Robustness to Positional Disturbances
In practice, the measurement data X (see Section is often subject to measurement errors, leading to biased estimation of the causal structure and perturbed Poincaré ball positional encodings. The following proposition demonstrates the robustness the resulting attention scores to such disturbances.
Proposition 4.3. Assume that the noise-perturte peite-perturbed Poincaré ball positional encoding of can be represented as e',, -e, + Ej, where E N I5) is small random Gaussian disturbance with E IRd and I d "220".2..... ".22...,,,,, Then, the noise-disturbed attention score A' remains robust to such disturbances in three aspects, including Distinguishability (Prop. A.8, Unbiasedness (Prop. A.9. and and Asymptotic Convergence (Prop. A.
Proof. See Supp.
A.6
5 Experiments
Due to space constraints, we defer the dataset descriptions to Supp. D and the specifics of the evaluation tasks and metrics to Supp. E Implementation details, including data preprocessing. model architecture. and training procedures, are provided in Supp. F Finally, Supp. G presents additional results, including a full empirical analysis CAPE properties, comprehensive multi-omics benchmarks, ablation studies, sensitivity analysis, and complexity analysis.
5.1 Empirical Evaluation of CAPE's Properties
CAPE effectively identifies the causal structure and preserves it in the hyperbolic manifold. To facilitate this evaluation, we simulate tabular dataset X syn 5000x100 consisting of 5, 000 observations over set of ten non-sequential features. shown in Fig. 3a, the underlying causal graph G(V,E) is generated as a directed adjacency matrix A & 10x using the Albert

---

Table 2: Performance comparison of gene perturbation prediction on scRNA-seq datasets. Mean squared eтror (MSE) of perturbation predictions for top 20 differentially expressed genes are reported. i indicates the original positional encoding used in this method.
Methods
Positional Encoding
Single-gene Perturbation
Double-gene Perturbations
Statie absolute
0.224
0.230
scBERT 24
Trainable relative
0.219 (- 0.005)
0.215 0.005)
CAPE
0.193 0.031)
0.189 (-0.041)
Trainable absolute
0.202
0.201
scGPT 25
Trainable relative
0.195 (- 0.007)
0.204 (++0.003)
CAPE
0.182 (-0.020)
0.176 (-0.025)
which is characterized by a preferential attachment mechanism. This mechanism assigns higher probability of gaining new connections to nodes with larger number of existing connections, thus varying the causal specificity across nodes [41] We then utilize the I/DSimulation package from gCastle 442 to generate X Xay based on A, employing MLP-based nonlinear assignments with added
model 40).
Gaussian noises.
CAPE is trained to estimate A from X sya Fig. and b show that the estimated adjacency matrix À closely approximates the ground truth. Next, CAPE generates d-dimensional Poincaré ball positional encoding ev for each feature e E V based on A, with = 3 for visualization (Fig. 30). These embeddings effectively encode causal strengths pairwise distances and causal specificity as the distance the origin, thereby accurately preserving the causal structure of A. For example, nodes with strong influence in the true DAG, such as node pairs (7, 0) and (3, 5), are embedded in close proximity on the Poincaré ball, demonstrating the model's s ability to encode causal strength. Meanwhile, embeddings near the boundary (e.g,, nodes 5 and 4) correspond to leaf nodes with high specificity, while those near the origin (e.g., nodes 6) correspond root nodes with general causal influence, demonstrating the model's 's ability to model causal specificity.
CAPE enhances the causality-awarenn and robustness of the self-attention mechanism. See Supp. G.1
5.2 Empirically Evaluating Representation Learning with CAPE over Real Multi-Omics Data
To assess the effectiveness of CAPE in enhancing performance of transformer models over data with non-sequential yet causally-related data, we conduct evaluations using data from multiple omics domains [43. including transcriptomics, epigenomies, and proteomies, (see Supp. D for the data description). Feature and observation representations generated by the CAPE-transformer model are evaluated in various feature-level and observation-level downstream tasks. Here, we focus on the feature-level task, gene perturbation prediction (GPP) with scRNA-seq data 44], and leave the results of other tasks, e.g., cell clustering with proteomics data 45 and age prediction with epigenomics data [46). to Supp. G.2
GPP aims to leverage the learned gene representations predict perturbation (e.g., gene knockout activation) activation)--nduuced changes in gene expression profiles, facilitating the exploration of gene functions and regulatory networks. Here, use human leukemia cell dataset (23 25 48 which includes unperturbed and perturbed cells under both single- and double-gene perturbations. Gene representations are learned using two prevalent transformer-based single-cell foundational models, including scBERT (24) and scGPT (25). Different position encoding approaches, which do not rely on predefined feature order are evaluated with the two models, including CAPE, their default methods, and a trainable, causality- using standard scRNA-seq workflows [50]. including quality control, normalization, and highly variable gene selection, as detailed in Supp. [.I Following (24)25)) the two methods are trained on unperturbed cells to learn contextualized gene representations, which are fed into GEARS (4), perturbation prediction model, to predict the verturbation-induns induced gene expression changes. Detailed implementations of the transformer models and positional encoding mechanisms can be found in respectively. The prediction accuracy is measured as the mean squared error
relative position encoder [49]. The dataset is first preprocessed
Supp. F2 and Supp. F3
'This explains why RoPE is not used benchmark

---

Table 3: Ablation studies on gene perturbation prediction on scRNA-seq datasets. The standard deviation of each experiment indicated in parentheses.
Models
Single-gene Perturbation
Double-gene Perturbations
CAPE-null
0.234 (+0.014)
0.238 (+0.017)
CAPE-WO-CSL
0.209 (#0.010)
0.213 (#0.011)
CAPE-w/o-byperbolic
0.192 (#0.008)
0.196 (#0.008)
CAPE-w/orottry
0.201 (#0.009)
0.208 (=0.010)
CAPE
0.182 (#0.005)
0.176 (=00.088)
(MSE) of the predicted and true expressions of the top 20 differentially expressed genes Tab. 2 We find that both models equipped with CAPE consistently yield substantial performance gains (11.1% % average reduction in MSE) compared to their respective default approaches. This contrasts with using the aggssliy relative position encoder, which achieves only a 2.7% reduction.
5.3 Model Analyses
We provide a comprehensive analysis of CAPE, including ablation studies, sensitivity analysis, and complexity analysis. In this section, we primarily present the ablation studies, while the results of sensitivity and complexity analyses are deferred Supp. G.3 and Supp. G.4 respectively.
We conduct a series of ablation studies to assess the contributions of CAPE's key components, using the same GPP dataset as described in Section 5.2 In these experiments, we adopt scGPT as the transformer backbone, replacing its default position encoding mechanism with four CAPE ablation variants. Specifically, the first variant completely omits CAPE (CAPE-null). The second variant (CAPE-W/OCCLL excludes the Causal Structure Learning step (Step I) by replacing the learned causal graph with a similarity graph constructed from pairwise feature correlations. The third variant CCAPF--w/ohyperbooll
removes the hyperbolic modeling component, replacing the hyperbolic distance (d, in Eq. (T6) and Eq. (17) with Euclidean distance and using standard SGD instead of RSGD for optimization. Lastly, the fourth variant (CAPE-w/o-rotar) bypasses the rotary form conversion (Step III), directly adding hyperbolic positional encodings to feature embeddings.
As shown in Tab. removing any of these components leads to performance degradation, particularly in the double-gene perturbation prediction task. This task is inherently more challenging than single- gene perturbation prediction, as it requires more semantical informative gene representations. As expected, the CAPE-null variant yields the most significant increase in MSE for both prediction tasks, demonstrating the importance of CAPE's overall design. The second- largest performance drop arises with CAPE-w/O-SSL, where replacing the learned causal DAG with an undirected similar matrix impairs the model's s ability to encode causal relationships in the positional encodings. Similarly, APE-w/o-rotary leads to notable decrease in performance, as omitting the rotary form forfeits the attention mechanism's sensitivity to causal distance and causal generality, two key causal properties for capturing hierarchical causal semantics. Moreover, this omission also undermines CAPE's sensitivity to nuanced changes in underlying causal semantics, consistent with prior findings that highlight the importance of concentrated attention scores in context modeling [19]. Finally, although HAPEEw/o-Bee /oohypeerbolic leads to more modest decline in performance, we emphasize the essential role of space curvature- optimization, which allows the positional encodings to be placed in optimal locations that better reflect the underlying causal graph structure.
Conclusion
In this study, we present CAPE, causality-aware positional encoding that enables transformers to handle non- sequential yet causally-relater features by modeling their latent structure as DAG and embedding it in hyperbolic space. By unifying causal discovery, hyperbolic geometry, and rotary attention. CAPE effectively captures core properties of causal graphs, causal strength and causal specificity, and translates them into position-aware attention dynamics, Our theoretical analysis reveals desirable behaviors of the resulting self-attention, and extensive empirical results across synthetic and multi-omics datasets validate the benefits. CAPE opens a pioneering path toward causal representation learning in domains where traditional positional encodings fail.
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the International Conference on Neural Information Processing Systems, volume 30, 2017.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the International Conference Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[4] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 2019.
[5] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representattonns 2021.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229, Springer, 2020.
[8] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821 1-831. Pmlr, 2021.
[9] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 6706 6706-6713, 67,1,, 2019.
[10] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492-28518, PMLR, 2023.
[11] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763. PMLR, 2021.
[12] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified lision-language understanding and generation. In International Conference on Machine Learning, pages 12800. PMLR, 2022.
[13] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the International Conference on Neural Information Processing Systems, volume 36, pages 34892-34916. 2023.
[14] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators sequence-to-seqquenc functions? In International Conference on Learning Representations, 2020.

---

[15] Zhiheng Huang. Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
[16] He Pengcheng, Liu Xiaodong, Gao Jianfeng, and Chen Weizhu. DeBERTa: Decoding- enhanced bert with disentangled attention. In International Conference on Learning Represen- tations, 2021.
[17] Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2021.
[18] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 368:127066 127063, 2024.
[19] Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du Zirui Liu, and Yongfeng Zhang. Massive values in self-attentio modules are the key to contextual knowledge understanding. In International Conference on Machine Learning. 2025.
[20] Christoph Ziegenhain, Beate Vieth, Swati Parekh, Bjöm Reinius, Amy Guillaumet-Adking Martha Smets, Heinrich Leonhardt, Holger Heyn, Ines Hellmann, and Wolfgang Enard. Comparative analysis of single-cell ma sequencing methods. Molecular Cell, 65(4))631 2017.
[21] Ryan T Kelly. Single-cell proteomics: progress and prospects. Molecular and Cellular Proteomics, 19(11):1739-1745, 2020.
[22] Christina Theodoris, Ling Xiao, Anant Chopra, Mark Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 618(7965))116-624, 2023.
[23] Xiaodong Yang. Guole Liu, Guihai Feng, Dechao Bu, Pengfei Wang, Jie Jiang, Shubai Chen, Qinmeng Yang. Hefan Miao, Yiyang Zhang, et al. GeneCompass: deciphering universal gene regulatory mechanisms with a knowledge-inforned cross-species foundation model. Cell Research, pages 1-16, 2024.
[24] Fan Yang. Wenchuan Wang, Fang Wang. Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. ScBERT as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence, 4 4(10):852-866, 2022.
[25] Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scGPT: toward building a foundation model for single-cell multi-omics using generative ai. Nature Methods, pages 1-11, 2024.
[26] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS: Continuous optimization for structure learning. Proceedings of the International Conference Neural Information Processing Systems, volume 31, 2018
[27] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pages 7154-7163. PMLR, 2019.
[28] Hantao Shu, Jingtian Zhou, Qiuyu Lian, Han Li, Dan Zhao, Jianyang Zeng, and Jianzhu Ma. Modeling gene regulatory networks using neural network architectures. Nature Computational Science, 1(7):49-551, 2021.
[29] Hermann Minkowski. Raum und zeit. Springer, 1988.
[30] Athanasios Vlontzos, Henrique Bergallo Rocha, Daniel Rueckert, and Bernhard Kainz. Causal future prediction in a minkowski space-time. arXiv preprint 0xIv:22008.09154, 2020.
[31] Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vabdat, and Marián Boguná. Hyperbolic geometry of complex networks. Physical Review E E-Sattistical, Nonlinear, and Soft Matter Physics, 82(3):036106, 2010.
[32] Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IFEE Transactions Automatic Control, 58(9):2217-222)) 2013.
12

---

[33] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International conference on machine learning, pages 3779-3788. PMLR, 2018.
[34] Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359- -66,, 1989.
[35] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[36] Dimitri Bertsekas, Nonlinear programming. Journal of the Operational Research Society, 48(3):334-334, 1997.
[37] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pages 3779-3788. PMLR, 2018.
[38] James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of Geomerry, 31(59-115):2, 1997.
[39] Menglin Yang. Min Zhou, Rex Ying, Yankai Chen, and Irwin King. Hyperbolic representation learning: Revisiting and advancing. In International Conference on Machine Learning, pages 39639-39659. PMLR, 2023.
[40] Albert- Barabási and Réka Albert. Emergence of scaling in random networks. Science, 286(5439):509-511 1999.
[41] Aleksander Molak. Causal Inference and Discovery in Python: Unlock the secrets of modern causal machine learning with DoWhy, EconML, PyTorch and more. Packt Publishing Ltd, 2023.
[42] Keli Zhang, Shengyu Zhu, Marcus Kalander, Ignavier Ng. Junjian Ye, Zhitang Chen, and Lujia Pan. gCastle: python toolbox for causal discovery, 2021.
[43] Alev Baysoy, Zhiliang Bai, Rahul Satija, and Rong Fan. The technological landscape and applications of single-cell multi-omics. Nature Reviews Molecular Cell Biology, 24(10):695
713, 2023.
[44] Yusuf Roohani, Kexin Huang, and Jure Leskovec. Predicting transcriptional outcomes of novel multigene perturbations with gears. Nature Biotechnology, 42(6))927-935. 2024.
[45] Harrison Specht, Edward Emmott, Aleksandra A Petelski, R Gray Huffman, David H Perlman, Marco Serra, Peter Kharchenko, Antonius Koller, and Nikolai Slavov. Single-cell proteomic and transcriptomic analysis of macrophage heterogeneity using scope2. Genome biology, 22:1-27, 2021.
[46] Lucas Paulo Lima Camillo, Louis Lapierre, and Ritambhara Singh. A pan-tissue dna- methylation epigenetic clock based on deep learning. Npj Aging. 8(1):4, 2022.
[47] Thomas M Norman, Max A Horlbeck, Joseph M Replogle, Alex Y Ge, Albert Xu, Marco Jost, Luke A Gilbert, and Jonathan Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science, 365(6455) 2019.
[48] Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Xuegong Zhang, and Song. Large-scale foundation model on single-cell transcriptomics. Nature Methods, pages 1-11, 2024.
[49] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive language models beyond a fixed-length context. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2978-29988 2019.
[50] F Alexander Wolf, Philipp Angerer, and Fabian Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome Biology, 19:1-5, 2018.
[51] Joel W Robbin and Dietmar A Salamon. Introduction to differential geometry. Springer Nature, 2022.
13

---

[52] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical repre- sentations. In Proceedings of the International Conference on Neural Information Processing Systems, volume 30, 2017.
[53] W. Grafarend. Linear and Nonlinear Models: Fixed Effects, Random Effects, and Mixed Models. Walter de Gruyter, 2006.
[54] Konstantin Avrachenkov, Alexey Piunovskiy, and Yi Zhang. Markov processes with restart. Journal of Applied Probability, 50(4))960-96) 2013.
[55] Alec Radford, Karthik Narasimhan, Tim Salimans, lly Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[56] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[57] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher Manning. Elec- tra: Pre-training text encoders as discriminators rather than generators. arXiv preprini arXiv:2003. 10555, 2020.
[58] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 464-468. 2018.
[59] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
[60] Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding, Youbang Sun, Biqing Qi, Yuchen Fan, Xuekai Zhu, and Bowen Zhou. Fourier position embedding: Enhancing attention's periodic extension for length generalization. arXiv preprint
2024.
[61] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extraaollatbbe transformer. arXiv preprint 1.Xiv:2212.105544 2022.
[62] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context win- dow of large language models via positional interpolation. arXiv preprint arXiv:2306.15. 15595,
2023.
[63] Hongru Shen, Jilei Liu, Jiani Hu, Xilin Shen, Chao Zhang, Dan Wu, Mengyao Feng, Meng Yang, Yang Li, Yichen Yang. et al. Generative pretraining from large-scale scage-scale transcriptomes for single-cell deciphering. Iscience, 26(5), 2023.
[64] Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
[65] Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2000.
[66] Diego Colombo, Marloes H Maathuis, al. Order-independent constraint-based causal structure learning. J. Mach. Learn. Res., 15(1):374137822 2014.
[67] Thuc Duy Le, Tao Hoang, Jiuyong Li, Lin Liu, Huawen Liu, and Shu Hu. A fast pe algorithm for high dimensional causal discovery with multi-core pcs. JEEE/ACM transactions on computational biology and bioinformatics. 16(5):1483-1495, 2016.
[68] Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Schölkopf, and Clark Glymour. Generalized score functions for causal discovery. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1551-1560, 2018.
[69] Max Chickering. Statistically efficient greedy equivalence search. In Conference on Uncer- tainty in Artificial Intelligence, pages 241-249. Pmlr, 2020.
14

---

[70] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlin- ear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.
[71] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non- gon-saassian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
[72] Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer. Directlingam: A direct method for learning a linear non- gaussian structural equation model. Journal of Machine Learning Research-JMLR, 12(Apr):1225 (25-):::222-222:: 2011.
[73] Ignavier Ng. AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing 3 2020.
[74] Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient- based neural dag learning. arXiv preprint
2019.
[75] Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.0211 2022.
[76] Colin Megill, Bruce Martin, Charlotte Weaver, Sidney Bell, Lia Prins, Seve Badajoz, Brian McCandless, Angela Oliveira Pisco, Mareus Kinsella, Fiona Griffin, et al. Cellxgene: a performant, scalable exploration platform for high dimensional sparse matrices. bioRxiv, pages 2021-04, 2021.
[77] Kejun Ying, Jinyeop Song, Haotian Cui, Yikun Zhang, Siyuan Li, Xingyu Chen, Hanna Liu, Alec Eames, Daniel L. McCartney, Riccardo E Marioni, et al. Methylgpt: a foundation model for the dna methylome. bioRxiv, pages 2024-10, 10, 2024.
[78] Zhuang Xiong, Mengwei Li, Fei Yang, Yingke Ma, Jian Sang, Rujiao Li, Zhaohua Li, Zhang Zhang, and Yiming Bao. Ewas data hub: a resource of dna methylation array data and metadata. Nucleic acids research,
2020.
[79] Kejun Ying. Alexander Tyshkovskiy, Alexandre Trapp. Hanna Liu, Mahdi Moqri, Csaba Kerepesi, and Vadim N Gladyshev. Clockbase: a comprehensive platform for biological age profiling in human and mouse. bioRxiv, pages 2023 02, 2023.
[80] Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo. Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature Communications, 8(1):14049, 2017.
[81] Asa Segerstolpe, Athanasia Palasantza, Pernilla Eliasson, Eva-Marie Andersson, Anne- Christine Andréasson, Xiaoyan Sun, Simone Picelli, Alan Sabirsh, Maryam Clausen, Magnus K Bjursell, et al. Single-cell transcriptome profiling of human pancreatic islets in health and type 2 diabetes. Cell Metabolism, 24(4):593-607, 2016.
[82] Jeffrey M Granja, Sandy Klemm, Lisa M McGinnis, Arwa S Kathiria, Anja Mezger, M Ryan Corces, Benjamin Parks, Eric Gars, Michaela Liedtke, Grace XY Zheng, et al. Single-cell multiomic analysis identifies regulatory programs in mixed-phenotyy acute leukemia. Nature Biotechnology. 37(12):458-1165 2019.
[83] Meng Zhang, Xingjie Pan, Won Jung, Aaron R Halpern, Stephen W Fichhom, Zhiyun Lei, Limor Cohen, Kimberly A Smith, Bosiljka Tasic, Zizhen Yao, et al. Molecularly defined and spatially resolved cell atlas of the whole mouse brain. Nature, 624(7991):343-354, 2023.
[84] Ana P Montalvo, Zoe I. Gruskin, Andrew Ledue, Mai Liu, Zihan Gao, June H Ahn, Juerg R Straubhaar, Nikolai Slavov, and Juan R Alvarez-Dominguez, An adult clock component links circadian thythms to pancreatic 3-cell maturation. BiorXiv, 2023.
15

---

[85] Andrew Ledue, R Gray Huffman, Joshua Cantlon, Saad Khan, and Nikolai Slavov. Exploring functional protein covariation across single cells using npop. Genome Biology, 23(1):261, 2022.
[86] Lucas Paulo de Lima Camillo, Louis R Lapierre, and Ritambhara Singh. A pan-tissue dna- methylation epigenetic clock based on deep learning. npj Aging, 8(1):4, 2022.
[87] Suyuan Zhao, Jiahuan Zhang, Yushuai Wu, Yizhen Luo, and Zaiqing Nie. LangCell: Language- cell pre-training for cell identity understanding. In International Conference Machine Learning, 2024.
[88] Wei Li, Fan Yang, Fang Wang, Yu Rong, Linjing Liu, Bingzhe Wu, Han Zhang, and Jianhua Yao. seprotein: versatile deep graph contrastive learning framework for single-cell proteomies embedding. Nature Methods, 21(4 634,,63322200 2024.
[89] Laurent Gatto, Ruedi Aebersold, Juergen Cox, Vadim Demichev, Jason Derks, Edward Emmott, Alexander M Franks, Alexander R Ivanov, Ryan T Kelly, Luke Khoury, al. Initial recom- mendations for performing, benchmarking and reporting single-cell proteomics experiments. Nature methods, 20(3):375-386 2023.
[90] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193- 218, 1985.
[91] Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. Normalized mutual information feature selection. IEEE Transactions on neural networks, 20(2):189-2011 2009.
[92] Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65 66, 1987.
[93] Kaichen Xu, Yueyang Ding, Suyang Hou, Weiqiang Zhan, Nisang Chen, Jun Wang, and Xiaobo Sun. Domain adaptive and fine-grained anomaly detection for single-cell sequencing data and beyond. In International Joint Conference Artificial Intelligence, pages 6125-6133, 8 2024.
[94] The UniProt Consortium. Uniprot: the universal protein knowledgebase. Nucleic acids research, 46(5):2699-2699, 2018.
[95] Yuhan Hao. Stephanie Hao, Erica Andersen- Nisseen, William M Mauck, Shiwei Zheng, Andrew Butler. Maddie Lee, Aaron Wilk, Charlotte Darby, Michael Zager, et al. Integrated analysis of multimodal single-cell data. Cell, 184(13):3533-3587, 2021.
[96] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, An- dreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et Rethinking attention with performers. In International Conference on Learning Representations, 2020.
[97] Jingcheng Du, Peilin Jia, Yulin Dai, Cui Tao, Zhongming Zhao, and Degui Zhi. Gene2vec: distributed representation of genes based on co-expression. BMC Genomics, 20:7-15, 2019.
[98] Christophe Vanderaa and Laurent Gatto. Replication of single-cell proteomics data reveals important computational challenges. Expert Review of Proteomics, 18(10):835884) 843, 2021.
[99] David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell, 174(3):716-729 2018.
[100] Hui Li, Cory R Brouwer, and Weijun Luo. A universal deep neural network for in- cleaning of single-cell maasee data. Nature Communications, (11:11991, 2022.
[101] Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature methods, 16(12):129--1226 2019.
16

---

[102] Brian Hie, Bryan Bryson, and Bonnie Berger. Efficient integration of heterogeneous single-cell transcriptomes using scanorama. Nature biotechnology. 37(6):685-691, 2019.
[103] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology. 67(2)3001-320 2005.
[104] Steve Horvath, Junko Oshima, George M Martin, Ake T Lu, Austin Quach, Howard Cohen, Sarah Felton, Mieko Matsuyama, Donna Lowe, Sylwia Kabacik, et al. Epigenetic clock for skin and blood cells applied to hutchinson gilford progeria syndrome and vivo studies. Aging (Albany NY), 10(7):1758, 2018.
[105] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learming systems, 32(1):4-22 24, 2020.
[106] Zhuangyan Fang, Shengyu Zhu, Jiji Zhang, Yue Liu, Zhitang Chen, and Yangho He. On low-rank directed acyclic graphs and causal structure learning. IEEE Transactions on Neural Networks and Learning Systems, 35(4):4924-4937, 2023.
[107] Shuyu Dong and Michèle Sebag. From graphs to dags: a low-complexity model and scalable algorithm. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 107-122. Springer, 2022.
[108] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-ima -mage learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 281 18-92,9,, 2023.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We emphasize the scope of this paper in both the abstract and the introduction, namely data with non-sequential features. At the same time, we also elaborate on the main contributions in points at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have clarified the limitations of our work in Supp.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[Yes]
Justification:
We have provided the complete proof in Supp. A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We provide the detailed dataset descriptions to Supp. D and the specifica- tion of evaluation tasks and metrics to Supp. B Implementation details. including data preprocessing, model architecture, and training procedures, are provided in Supp.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All data in this paper are publicly accessible, and we provide as detailed description of the experimental design as possible ensure reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Implementation details, including data preprocessing, model architecture, and training procedures, provided in Supp. F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Although error bars for each set of experiments are not reported for the sake of simplicity, ensure that each experiments repeated times independently and then averaged to ensure the stability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the required computational resources in detail in the implementation details Supp.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We ensure full compliance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided the Broader impacts in Supp. !I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The resources used in this article all open source content obtained legally from public websites.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:////oorrppsscc///oofrrrre// for what should or should not be described.
24

---

Supplementary Material
In this supplementary material, we first provide theoretical analysis in Supp. and justify the use of gregariousness (TT in regularization in Supp. B Supp. C related work, while Supp. [ and Supp. E detail the datasets and evaluation protocols. Supp. Fotllines implementation details, including preprocessing, architecture, and training. Finally, Supp. presents additional results, covering CAPE's empirical properties, multi-omics benchmarks, sensitivity, and complexity analyses. The code for our model and experiments available at https: pattps://githb. com CCatchxu/CAPE
G
A Theoretical Analysis
A.1 Equivalence between Constraint Condition and Acyclicity
Proposition A.I 26 ). A matrix A E MM can induce DAG, if and only if:
h(A) = - -
(A.1)
where @ is the element-wise product. Moreover, the gradient of h(A) follows a simple form of Vh( () = (0((A))((AAA0))) (T ④ 2A.
Proof. Given non-negative matrix B E TR- the adjacency matrix of DAG G, B reflects its k-hop connectivity. Specifically, B * indicates that a path of length & exists from u v. Consequently, tr( tr(B²) equals to the number of length-k self-cycles in G. That is, G is acyclic if and only if:
tr(B*) = 0, Vk E 2
(A.2)
This infinite family of constraints can be reformulated using matrix exponential:
tr(c")= tr())
k!
k=1
+00
(B*)
uu
(A.3)
M
+
k!
k=l
= - tr(e
B
M =
Replacing B with the element-wise product matrix A C A, which ensures the non-negativity, we reach the constraint A0A) M = 0. In particular, this constraint exhibits a simple form of gradient for easy optimization:
0tr(es)
atr(es)
as
Vh(A)
AAATT o 2A
(A.4)
0A
aS
0A
where S A O A. This completes the proof.
A.2 Optimization and Diffeomorphism of Hyperbolic Models
CAPE involves two hyperbolic models, including the hyperboloid model and Poincaréball model, to embed the identified DAG in a hyperbolic space. On one hand, the hyperboloid model is employed for an efficient RSGD optimization of the node embeddings, due its simple closed-form computation of geodesics on the hyperboloid. On the other hand, we utilize a diffeomorphism to map hyperboloid embeddings Poincaré ball embeddings, which are more natural for both conversion into their rotary form and visualization. In the following subsections, we will discuss the geometric properties, the optimization, and the diffeomorphism of the two models.
A.2.1 Hyperboloid Model
Recall the definition of the hyperboloid model:

---

Definition A.1 ((II). d-dimensional hyperboloid model is a Riemannian manifold L = (H.,..),
where == diag(- 1-,, 1, ... ,1] IR (d+1)x((d+1) denotes the metric tensor and where
Ht = {p: (p (0) E Rd++ (p.p): = -11pp > 0}:
(A.5)
(pem Pualt = Pum 91 Pu. = -Pem Pi.. + Pum. Pe..
(A.6)
denotes the upper sheet of two-sheeted hyperboloid with an origin po (1, in (d+1) dimensional Minkowski space 137V.
The distance of Pr Pri E Hd on L given as 38
di (P Pu..) arcosh( (Pr.. Pe.)t)-
(A.7)
Geometrically, the tangent space TpH at point E Hdd is defined as the set of vectors orthogonal to p 38 51
Tp = (o E RRd (v,p), = 0}.
(A.8)
The geodesics of H. can then be computed based on the following proposition. Proposition A.2. Given p E Hd and an unit tangent vector E TTp?d where (w, = 1, the unique unit-speed geodesic &p.u : [0, 1] - Hdd can be parameterized as:
Qp. cosh(t)p sinh(t)v s.t. (0. p, pp. v,
(A.9)
where E [0, 1]. (0) denotes the derivative of (t. to t, or the velocity of the geodesic (p.((t)
time t.
Proof. According to the definition of geodesic 38 5 Φ [0, + 7d a geodesic and only it satisfies the equivalent conditions:
(Ø) Φ) Φ.
(A.10)
This means that the acceleration vector V is zero. In other words, along the geodesic o, the direction of the velocity vector does not "tur".. When (ò, Φ) 1, Eq. (A.10) be formulated as ordinary differential equation (ODE):
((t) = (tt).
(A.11)
with general solution:
<(t) = ae' + Be
(A.12)
where az, B E IR++1 are constant vectors. a, β be determined by the initial conditions
(0) p
+ 3 p.
(A.13)
(1) = v = ae - Be- = =a --3=0
Then, we have:
1
1
= (p + v),
B= = (p- v).
(A.14)
2
2
(t) = cosh(t)p + sinh(t)v.
(A.15)
This completes the proof.
Eq. (A.10) can be extended to tangent vectors of any length using the exponential map, defined as: Proposition A.3 (Exponential map of hyperboloid model). Given E Hd and v E Tp with
length |laa| viv, u)p there exists a unique geodesic ő [0, Hd with ø(0) p. ø' (0)
The exponential map exp, : TTH + Hd is defined as exp,(v) == (1), which is the end point of the geodesic on the munifold. Based Prop. A.2 expp can be represented as:
expp(v) = cosh( (I (1ll))) + sinh([|u|la
(A.16)
|0

---

In other words, the exponential map yields the unique point on 7! reached by traveling along the geodesic originating p along the direction of for hyperbolic distance of ||v||--
Moreover, given any Riemannian gradient direction Rd+1 at p. it can be projected to the tangent space using the following proposition.
Proposition A.4 (Projecting Riemannian gradient to tangent space). Given p E H., Riemannian gradient u E Rd++ at can be projected to the tangent space Tp TH via:
projp (u) = u + (p, u))
(A.17)
Proof. Since p orthogonal the tangent space, we can derived the projected using the generalized Gram-Schmidt Orthogonalization:
projp (rojp(u)= = u (u, (u,p),
(A.18)
(p.
Given (p.P): = -1, we have:
proip(u) a + (p. u)
(A.19)
This completes the proof.
Prop. A.4 and Prop. A.3 together allow the RSGD steps in Eq. 199 Eq. 20
and Eq.
21 in
Section
A.2.2 Poincaré Ball Model
Definition A.2 ((22). A d-dimensional Poincaré ball manifold is Riemannian manifold defined as pd = (Bá, 9p), where Bd = {e IRd ||e|| < 1} represents an open d-dimensional unit ball with
the metric tensor:
||l|2 )
gp = 1-
I,
(A.20)
which is a conformal transformation of the Euclidean metric
The distance between points Cun Cv., E Edd reads 38
|eee.. Cr.
(eem Cen.) arcosh
+ 2 (1
(A.21)
lmm. 12) 11
Cv, 12)
straightforward to see that as |/e,,. ll 1 or Cv.. - dypeemm Cv.)) +00. Hence, the regions closer to the boundary of the Poincaré ball manifold have larger space capacity, thus allowing it model data with hierarchical structures and power-law distributions, e.g., causal graph.
On the other hand, as a point e approaches the origin, ||el| - 0. And Eq. (A.21) suggests that, on average, it closer other points, thus represent casually more general node that connects many its causal descendants. Meanwhile, |ee|| can be used to represent the causal specificity of the corresponding node.
A.2.3 Transforming Hyperboloid manifold to Poincaré ball manifold via diffeomorphism
We first define diffeomorphism fa as follows.
Definition A.3 (Difteumorpphssm) Given two differentiable manifolds M1 and M2, a continuously differentiable map fa Mi M2 is diffeomorphism if is is bijection and its inverse f
M2 > Mi is differentiable as well. And we claim that Mi and M2 are diffeomorphic.
The following proposition allows the bidirectional transformation between the hyperboloid manifold and the Poincaré ball manifold via diffeomorphism.
Proposition A.S. The manifold Ld in Def. A.l and manifold Pd in Def. A.2 are diffeomorphic, where the diffeomorphism fa Hd - 1 and its inverse fa Bd Hd are given by:
fa(p) = (22)
(A.22)
p(0)

---

(1 + |||| 2e(1), ... 2c!a))
Ja (e)
(A.23)
ee
where p E Hd and E B".
Proof. The proof consists of three parts:
1. fa is mapping from L to pd;
2. [ is mapping from pd L
3. Ja and S are differentiable and mutually inverse,
which are proved sequentially in the following proofs.
Proof 1. fa is a mapping from L to pd. For p E Hd, it satisfies:
(p.p): = -p())
+
(0) = =1
(A.24)
t=1
Then:
21 p0102
p(0)2
-2((0 +
||a(p)]]
(A.25)
(p(0)) + 1)2 (p(0) + +1)
(p(() + +1)
Since p!)) > 0, [If(())] <1 fa(p) 1d, which indicates that fa is a valid mapping function from L to pd.
Proof 2. fa is a mapping from pd to Ld.
Suppose r = ||e||, then:
1 2
2
d
(t
p2
2
4r
fT'))
22
1-
=
x2
+
(1 r2)3
t=1
(A.26)
- +2)+ + 4r²
-(1 - 22)2
1.
22
(1 22)
Therefore, f E Hd, which completes this proof.
Proof 3. fa and Sa are differentiable and mutually inverse. Since fa and fūl are both rational
functions and their denominators remain nonzero in their domains (e.g., (()) 1 > 0, 1 they are infinitely differentiable. Moreover, we have:
||e|| > 0),
(f((() 1+ [Ifa(p))1 (p(0)² + 2p
1- -If((p))l 2p(0) + (0)
(A.27)
fa (fa(()))) (p(() +1) (p(()
(p))) +1) 2
2p(0) +2 (1)) Vt = 1,2... d. (A.28)
1 111aapp)112
p(u) +
Thus, ff (fa(p) P. and f-1-1) is indeed the inverse function of fa- Similarly, it is trivial to prove Sa(ST (e)) e, indicating that fa and f are mutually inverse. This completes the proof.
A.3 Solving the Query and Key Mapping Functions
We solve Eq. 22 and Eq. 23 in Section 3.5 in complex space. this end, we begin with some preliminaries of the complex space.

---

Proposition A.6. When D is even, real space RD and complex space CD/2 are diffeomorphic (Def. A.3), where the diffeomorphism T R - C/2 and its inverse T DD/2 RD are given
by:
Re(2(1)
x(1)
(1) + ix [33) ix(
2(1)
Im(
x12)
x(2)
T(x) T
T -2)
T
Re(2(D/2))
(D-1) +ir(
22122)
Tm(2(D/F)
(A.29)
where æ E RRD and = E CDD/2
Proof. Given T(x) E C/2 and (z) E R", it is obvious to have T (T(x)) = and T (T (z)) 2. Moreover, each component of I' is linear mapping as (xx(1) , (t+1)) + (+)
(x(++)) t 1,3, D 1, and the same holds for T Therefore, T is a bijection, and T and T-1 are differentiable and mutually inverse. This completes the proof.
Furthermore, the canonical inner product for two vectors 21, #1, E CP/2 is defined as:
D/2
(21, x2) = 2 =
Σ
(1) (1) 22
(A.30)
where z denotes the complex conjugate of z2.
Next, as defined in Section 3.5 Um denotes a non-sequential feature, with v E RR representing its position-agnostic feature embedding in the context of the i-th observation, and ev., Bd representing its Poincaré ball positional encoding. We define the dimensionality of positional encodings as D/2, and give Eq. (23) in Cd as:
(T [I evm. }] T [I en,, }}> A (u u/ (eom Cv,)),
(A.31)
where T: RRP Cd is the mapping function in Prop. A.6 and A is scoring function. Both left- and right-band sides of the above equation can be represented as exponential forms:
T (1 Gem, )] = Pa( Cvm, exp {10, (..)),
ev..),
(A.32)
T [v ev..]] pu(p), Go,. exp (i0 exp
(v ev.))
(A.33)
A (wi wn ((em )) = PA (Dino 1/ (e ev..)) 1 exp 1 (pmm 1i (er... Cv.,
(A.34)
where Pg. Ph: : RRD x Rd + PA: RD x IRD x Rd + R denote the radius functions to be solved;
and 0,, O2: RDD x RR + Rdd OA R x RDD x Rd + Rd denote the angle functions to be solved.
Meanwhile, have:
(e ( Rmm)) 10 (m (ua))
Σ
exp (10%)) (Din evm) exp (10% (wi
(ei evn)
(=1
(A.35)
)mm) exp (1) Com) - -1(()) (00 (oh, cvn)
[=1
exp 10 (vi Gom) 0x(2 evn)
Substituting Eqs. A.32 to AA35 into Eq. AA.l yields:
Po Cun )px (u, ev )1" exp (i [i, evm) Ox(2,, )..)]}
PA (w',, vi 7(erm. en.)) I exp { (wim vy 7e (v.))). (A.36)
straightforward solution the above equation reads:
( Pq PylUim Cum )pn(1, ev,) PA (i), Un 7(eum een)); (0(00 eom) (v G..)) ) = O (vi 2y ((emm en,))
(A.37)
5

---

where r(ee,, Cv,)) is function that reflects the relative information between Cun and ev. We further assume the relative ceffinnformation is constant:
(eer Co C, Vk = 1,2,.... ,M,
(A.38)
where E Id is a constant vector. The explicit solutions of the radius and angle functions in Eq. (A.37) are given as follows.
(1) Radius Functions. For any ev.. - Ev,, we have:
Po(u'n ev,,. (r,.))((('s ev:,.) (m (v 0) PA (u/ o/ c)
(A.39)
which indicates that the radius functions independent of ev. This causal position analogue of the radius function used in RoPE [18), with the solutions:
Pa Pa evm) = lletulls ev, |
(A.40)
PA (m's vn, 7(Eum, ev,)) = ll |
(A.41)
(2) Angle Functions.
For any Cvm. = Cun, it always exists:
( evm) (v(v; evm) = 00(('n 0) Ou(v', 0) 0^ (v c).
(A.42)
follows:
(1 0). (v Cvm) - 0g ( 0) Os( eum) ) - Ou((
(A.43)
We simplify the solution by using the same function Ç for both 02 and Oz. yielding:
C(ui evm) ((v'n 0) C(e eum <( 0).
(A.44)
which indicates ((u evm) ) ((vin: 0) is independent of by (m.n) and can be expressed as:
S(win ev...) C(v'n. 0) (= (peen.)
(A.45)
where Φ Rd + Fd is a function that captures the effects of eem, which can also be viewed a causal position analogue of the angle function used in RoPE.
The following analysis will focus Eq. (A.32) since it also applies to Eq. AA.3)..
Substituting
Eqs. (0.40) and (A.45) into Eq. (A.32) we have:
T (T, Ce.. ]]
= | ll exp (i0,(v'.. 0)} @ cxp fie()....)
(A.46)
where C denotes the element-wise product. Since the term before O on the RHS of the above equation depends only on u we denote it
| exp (v 0)} = T (W = (g Cd
(A.47)
where W, E DDD are trainable weights for query or key mapping. Note that any complex exponential can be transformed into rotation matrix in real space as shown in the lemma below.
Lemma A.1. Given z = 21 + izz E C, 0 E R. and the mapping function T IR? C defined as T [21, 22)" = the following equivalence exists:
T (z exp(i0)) = T [[21 + iz2) (cos(f) + issi (0))]
- (oos()==) sin(0) (sin + (0))))22))
(A.48)
(cos(0)2; sin(
[cos(O) (in cos 22
sin 21
sin(0)21 + cos (0)
Lemma A.I allows us to rewrite the solution of Eq. (A.46) as:
Ig(D", Cem)
T [T (W Wr m
exp (ie(c...)) )}]
= )) Wo
(A.49)
= -o(((e dim.
where R is defined in Eq. 26 in Section 3.5 in which (fe,., is replaced with Pu... == Ce Put together, we reach the solutions for T, and I& in Eq. (24) Section 3.5 whose validity in satisfying Eq. 23 are demonstrated in the following proposition.

---

Proposition A.7. Given an Vn Cemi Con, the Ig and Ia given in Eq. 24 satisfy:
(Iy( eum ( evn)) = A (m un, (eeu.. Ev..) = (q Ri @u Ken (A.50)
where (( Cra) = Coa. Com
Proof.
(Ig(''n Com) In (w ev.)) [R(ye.. pm Win R(gr.) WEo
(A.51)
v W R(p ) R( Wa
where, based Lemma A.2 Rl ) R((Pv.) can be written as:
R(4 um
R((Pva))
r((p!.))
0
r (ipum))
0
0
r
(2)
0
0
r(pe)))
(d)
0
0
r(p (d)
r (1)
(ppe
0
r ((p
(2))
r (((p))))
0
(A.52)
0
r( ((p
(d)
r(Pon
0
r
(2)
(2)
Un
(d)
(d)
r rspo
R R(
pp.) R |c (ee.
Com }]
where r r(pb!)) is defined in Eq. 26) Section 3.5 Then, Eq. A.51
reads:
(Ig evm. ). In("", ev,)) )) = v'n W R (cY(e Cu,,
WAR = A (v vi,, 7(vmm ev, ))
(A.53)
This completes the proof.
Lemma A.2. The following equation always exists:
(os((a) - -sin(a)] sin(a) cos(a)
(30((3) - -sin
coo((3 (3 a) sin( a) cos(B - a)
sin(3
Vaa TR. (A.54)
sin(B)
cos(B) (3)
Remark A.1. Since PPum. ce.m and Cum Bdd lies within the unit Poincaré ball. Pu... is bounded for any Um- is a scale factor to control the range of angles. Here, we set c = T 4 to ensure that PPum is small but not negligible angle constrained to [-n /4, π /4). This also brings several valuable properties as demonstrated in Supps. A.4 to A.6
Remark A.2. Since Eq. (A.55) is sparse, it can be efficiently computed as [8]]
cos(pptt) (1)
(11)
sin(
(1)
(2)
cos(ppt (1)
sin (pp)))
(1)
cos(sa
sin
(4)
(os(p
(2)
sin ((n())(())
(3)
(A.55)
cos( cos(yplp)) (D/2))
((D-1)
sin
(p (D/2))
(D)
(D/2)
w(D)
(D/2))
(D-1)
cos( pm
sin p0om
where 3 is the element-wise product.

---

A.4 Causal Distance-Induced Attention Attenuation
Proposition 4.1. Given vnn and vi the attention scoring function A in Eq. A > O and A < O, satisfying:
25 is bounded by
OA+ (v'm Cum eu,)
Cum
0A (u'm Vn Cum en..)
S S0,
0,
(27)
@dp(erm= Ev..)
Ōdp Cv..)
where dp (em evn) is the distance between Eum and e.n on the Poincaré ball manifold, computed as:
||eun. Cv, 1|2
de(e Cvn)) arcosh 1 2
(28)
ev 12)
Cun 12)
The functions At and A are given in Supp. A.4
Proof. We first prove that for any on vn and 7(e Cv,, ). A is bounded by two functions A+,A=
(I) Boundedness. Recall that Iim
Wapf,
(go (1) o (2)
Iom (D)) and k
( (1) kin (2) 1... hen (D))
WEO
Then, we have:
A (o 2/ Cum -vv,) qum
R(ypu, Pum yky
(2r-1)
(2t) ) -(( (ki
(2r-1)
kg (26))T
cos(ppb) Fim Tom
(t) (t) (2t-1) + sin(p!? pp Tom
(t)
(2t)
ki (2f-1)
(A.56)
sin( -ppb nim
(t) (2c-1) + cos(up!. (t)
(1)
(21)
(21) ki
(1)
cos(sp!1) - 2!!.)) + 3((0 sin(s!!!) - -(1))
(1)
df (2r-1) 1) kin (21-1) + dom
(20) ki (21) B(1) di (20) kin
(2t-1)
(20-1) 1) ki (2r)
where 0;
Eq. (A.56) is bounded
A (U uh
Cum Cum
eva)
Σ [=1
a
(t) cos(pp!!) - -p!!!) +,05001
(A.57)
$ |a
cos(1
cos
-p!!0 + 13(0),
where |a.l maxella (1) As Pu Pon 11-//4 T /4. we have Ippt
(
(1)
(0) - -00 E [0,T/2] Over this
interval, cos(1sp(1) pbx is concave. Thus, Jensen't 's inequality indicates the inequality:
cos(1:p!) x(2) S cos
d
Pem
2=
(A.58)
< cos
7 :1lspum
which leads to the upper bound function of Eq. A.56) as:
d
A (v'm Ui eu... - ev..) = (la, d)) cos
7 illp.... -vu 11 + Σ 13
(A.59)
Since Pp.m $ Eum Pin Tew, and Cum Cu, E Bd (e E Rdd
||el| < 1}. we have
Ilypnm (P.,,
Z Ellev. ev, T/2]. which further leads to A+ 0. Similarly, the lower
bound function A < can be defined as:
d
A (wi 0/ ev... er.,) == - -(aaidd) cos () 140.... - spmlll)
Σ
(A.60)
1=1
8

---

Subsequently, we prove that both A and A attenuate as dp (ev Cv, increases.
Attenuation.
The partial derivative of A with respect llevm. ev,, reads:
aA+ (u Un Ev... Ev... (d(a?))
cos (1 pt... - 4in II) I a1yyp... - Gun
emm ev.
0 vm -uuu, evm evall
(A.61)
-- nd a sin Elliem - :Poull)
Given the Poincaré distance in Eq. 28
have:
ddg(Cvm Crn)
i arcosh
(1 + 20 ev Eun 112)
0 eu Cv2
@m - -vv,
40 40-11|es eleum Gvn
2 v Ilee Cun 12+C
(A.62)
V(1 20 11eem Cea, 112)
where C (1 ||e 13) - -1e,,, 113 This leads to:
0A+ (win: o/ Com Con)
0A+ (v', wh Com Coo)) @llev. Cra
ddlp ev Cvr.)
el ev,11 11 (erler... Bun
(A.63)
Td
aa,1 |a,|sin sin ( glllp... Pv..ll 11)
evm
Bvn,
12 + C
<0
4
2
where llyp... &u.,l E [0, /2]. Similarly, we can prove the attenuation for A= This completes the proof.
Remark A.3. As the causal distance dp(eum, ev.) +00, both At and A altenuate and converge towards smaller magnitudes (though not necessarily to 0). Since A is bounded between At and A its range of possible variation also shrinks significantly. In particular, when q and le collinear, A exhibits a stronger attemuation property shown below.
Corollary A.1. Following the definition Prop.
4.1 when oq, C. E R. and + 0, exists:
aA (v Ui e.... Gv, ks cq)
sgn(() S 80
(A.64)
(do eoa)
where sgn(c) = c/|cc is a sign function.
Proof. According to Eq. A.5.), given KS = cq. we have:
(1hm dhm (2(-111) 2 + db (200) 2 cos(>p!t) - =(( (A.65)
A (vis w/y
Com Com
-v, 1k kk= =
Σe
1=1
When 0, we have az =c c((1thn (20-1)) 2
(1t (201)² 27
> 0, and:
0.A (v vh eum - Cun 1k cq)
aA (vi vi Gum
eun
kk cq) allyr.m - -Pimll
@llemm - em
0lly.mm
I Ileem. -cv,
π
a
(1) cos(p - - 0(p - -p0))
π Σ
aa sin ((t) -p!!!)
|14p...
4
0(p -!p(!)) 01(.m... v.
(0) (t)
4
(1)
1=1
t=
60em
4 in
(A.66)
Combining Eq. (.66)) and Eq. A.62) we obtain:
aA (vin Vi, Gum er, kk = cg. > 0)
Üd,, ).n))
(A.67)
W ev. C
d
(0) sin (2)
(pm -Punn
(t)
||ypm.. - sr..
0.
8
(t)
(t)

---

Similarly, when < O, we have at < 0, and:
0A (m 2/ Cum Bv, === 10== 0)
20
(A.68)
ddp Cum Cv.)
This completes the proof.
A.5 Causal Generality-Inducee Attention Attenuation
Proposition 4.2, Given vim vi and fixed causal distance dy (e Cv,) in the Poincaré ball manifold defined in Def. 4.1 A and A satisfy:
0A+ (v'm viny Cum
Com - ev.) 50
0AA (u D Cum Cval 2 0.
(29)
@npun..
The same holds for Ven.
Proof. As proved in Supp. A.4, bounded by the functions A+ and A in Eqs. (A.59) and (A.60).
We have also shown in Eq. A that:
0A+ (o U Cum Cva)
sin 10lm-m-pun))
(A.69)
e e...l
4
The Euclidean norm of eum and Ce, can be expressed in terms of their Poincaré distance based on Eq. (28 as:
Cv, V2
leew
- 1] (1 - lee,,. [1) - ||er,, 11
(A.70)
Given fixed dp (em eu. ) c= 1 [cosh(dppeen Ev,)) 11 > 0 represents non-negative constant. Given tie.. e
we have:
El evall
@lleu.. eval
erm - Ev.
0(1 ev |]
0| aleem
(A.71)
0VC(1-
llem
12)
lev, 12)
|lerm 11
V C(1 - lev, 113),
0| evmll ||
e
Combining Eq. (A.69) and Eq. AA.7I we obtain:
0AA+ (win: wr: ev... Bew.
Orim..
( - |a,
.d.
sin (l...... - -s.u.l)
| le
VC(1
Iiee.,,11. (12) -le,,,
(A.72)
where ||ypm. - uu || [0, T/2). Similarly, we can prove the attenuation for A The same proof also applies to Ve... This completes the proof.
Corollary A.2. When the =1-1|eam upper bound function At (w'm, vi,, Co... ev.) and lower bound function A (um, 2/ Bvm. ev..) attenuate towards constants a and -a, respectively, where
and the causal distance de Bu,,) ) is fixed, the
d
a = (1a d) cos
π 4d
(A.73)
4dV
where C = 2 [cosh(d,.er.... e,,.) - 2 represents a non-negative constant. Moreover, decreases monotonically with increasing causal distance.
10

---

Proof. Since com
+ 0 as Vem + 1, we have the following limits of Eq. AA700
lim llen - Cv,, =1
lim VC(1 - -eee [12) l,,,, []
Pum
+1
Wum +1
(A.74)
lim lleum
Eun ||
lim | 11
lim
VC(L - lleu, 112).
Vom
Dum
Pem +1
Let
ev.
= limet..
-1 11e
Eq. A.74
yields:
1
2
Cv. 1
(A.75)
C
C+1
Hence,
lim 11pm -vi. 4
π
C
Vum -1
C+1
lim cos ( (2lipim - -Puu.ll
C
Pum +1
cos
4d
C+1
(A.76)
lim Pum
A (iys u/s Bum Cvn ev.)= (la)ld) = cos
C
+
3 = a.
+1
C+1
According to Prop. 4.2 A+ (v) Vn Sum - ew.)) asymptotically attenuates towards a as t/ - 1. Similarly, the asymptotical attenuation of A (v vi er... ev..) towards as then.. 1 can be proved. Finally, the monotomic decrease of w.rt dp (e ev,,) can be told from x f(g()))),
where == cos(.) and = La / C+I E [0, 7) It is straightforward to verify that f is monotonically decreasing in g. and is monotonically increasing in C, which itself increases with dp(er.. ev, Therefore, decreases monotonically de Ce,) increases. This behavior aligns with the expectation that the range of the attention score between Um and U2 shrinks around as their causal distance grows. This completes the proof.
Corollary A.3. Following the definition Prop.
4.2 when oq. C. E R and + exists:
a.A (m Ui Co...
ev, k eq)
(c) S 0,
(A.77)
Dwham.
where sgn(c) = c/(cl is sign function. And the same holds true for De..
Proof. The proof follows that of Corollary A.1
A.6 Robustness to Positional Disturbances
Proposition 4.3, Assume that the noise prise-perturbed Poincaré ball positional encoding of can be represented as e',, = Ce, + Ej, where E ((,, I5) is small random Gaussian disturbance with I E &Rd and I, = d ".......... ..))). Then, the noise-disturbed attention score A' remains robust such disturbances in three aspects, including Distinguishability (Prop. A.8, Unbiasedness (Prop. A.9). and Asymptotic Convergence (Prop. A.10
A.6.1 Distinguishability
Here, distinguishability refers to the property that the attention score between two feature embeddings. differing only due to random noise, should remain larger than the score between embeddings of two truly distinct features. Importantly, this property should preserved even when the positional embeddings are perturbed. Formally, this is captured in the following proposition.

---

Proposition A.8. Given embeddings of two distinct features (v vi,), the noisy embedding T. v'n + 8. where ó EDD is a random noise with zero mean and finite second moment, and the noise-perturbed positional encodings (e'm, G,,, exists:
Ben 8 [A (w Ums e'm en,,] > Bui [A (u Un e.m - e,,,)]
(A.78)
Proof. We first define:
Dl un 8, e'm - -b'n)) = A (v/m unn + + e'm - -v-n) A (mms un e'm E
(A.79)
= W R(4's, -4''m WW + + -u' T W R(v'', Φ W
where Pe... = ce'... P/c.. = cely. Let μ E(v')) = E(v ) Σ Cov(Um)) Since both Um and are randomly sampled from V, for any e'um e the expectation of D satisfies:
Evi, og (D(v'n. vn 8, 8, -
B U W R(vl'.
WWe( + -3)-0 v'n W. R(yl', -4'm W
Byg U W R(yp'.. -4' W Beinnt vi (oin W R(yp'. (
[W,] RiPe', -y' WW = { - -40'
EWW
tr RR(v'e - pl.,, Cov()o W
Cov ge.m (21-1) ke ( + Cov frm
(20)
(20) cos(p' (0) 4',, 19),
(A.80)
where the third equal sign is achieved based on Lemma the same embedding Um. is safe to assume:
A.3 Since qi.. and kt.,, generated from
Cov ge...
ki 10)
A 1,2,....d
(A.81)
By substituting Eq. (A.8T into Eq. (A.80), we obtain:
Evg, .8.8 [D(vi oi 6, e'm -',,)]
Cov do
(20-1) 1) ke (2-11) + Cov
(2t)
kim
(20))
cos
(v0 (+)
cm 4 (4)
(A.82)
20
20
c|-x/2,5/21
ΣCon
Cov dom
(0)
"0, kom (1) #020. =))000
This completes the proof.
Lemma A.3 (Expectation of quadratic form). Given a random vector x E TRD and constant
matrix A R DDD, where B(x)) ar and Cov(x) E, it always exists:
Ez(x T Az) E, [tr(xx] Am)] En [tr((Aaa }] tr (Axxx 7)] = tr [A(E u4 J]
=tr(AE) + Att(AAuuu)) tr(AE) + 4 Aμ..
(A.83)
A.6.2 Approximate Unbiasedness
Proposition A.9. Given the original and noise-perturbed positional encodings defined Prop. the disturbed attention score approximates the original score in expectation:
4.3
Eg..e, e.. (A 2/ ebmm e',,)] = A A(v Un Cu... Ev..)
(A.84)
12

---

Proof.
Let Omn = Pm Pvn + Em Ens then it has:
8mn N (4Pum Pon Im In).
(A.85)
Each component onn,
1,2, in 8mn satisfies:
6(t) mn ~ N/ Ppom
p on + dne)
(A.86)
Utilizing the characteristic function of Gaussian distribution, we have:
9 (C))
E
oian
(pim -pin
x2 (o + me
exp { (ir( - -p!!))
exp
2
2² (o + me
loss (x(p!2 (p(h)) + issn (s(0lh) p!!))
exp
2
ind
Re
E ( (W
2 tofe)
+ Im s
( wit the mne)) E
= =E,0) 6mn -N -000 ometta))
cos(zzd )
+ ilE
bin (v -pol om
sin (in(xo((tn) (
(A.87)
Then, we can obtain:
x (o + oni)
E (t)) (vbii -pon of to
(oo((xo(11)) min
eex
exp
ccos (a((p(0)) - (200)
2
(A.88)
x² (0 + ont
sim (x((e!()
E (e)) N (W(tth -om--ppll -pll mne
exp
2
(A.89)
For Eq. A.56). the attention score calculated with noise-perturbed positional encodings can be expressed as:
A (u o e'... e,,,)
cos cos()(t)) (.0"))(snn sin(
(A.90)
Substituting Eqs.
A.88 and A.89 into Eq. (A.90). we have:
Eeeio Cop [A (win "n e'mm elm)) EEs.... Σ O cos( + 11000) sin sin()
mn
d
(co((10((1)) ) c0 1
+ +.5000 E sin (sin
(A.91)
t=1
Σ exp
d
+ ont
[a.00 (1) cos(pp!t)
- -(( (000) sin(y!!!) - 1
1=1
Since -T/4 S (1)
S 1/4 we can assume that (ame J are a series of small quantities, leading to:
0 + 02
exp
1.
(A.92)
2
By substituting Eq. (..2 into Eq. (A.9I). we reach the Prop. A.9 To be more rigorous, we have: lim Im.Ia -
Ee eboy [A (wm uy e'u.. e',,)] = A (v) vn evn. - ev..)
(A.93)
This completes the proof.
Remark A.4. We perform numerical experiments to verify Eq. (A.92). mation is defined as:
The accuracy of this approxi-
(-
0
+
Acc = exp
E =(0,1)
(A.94)
2
13

---

According to the three-sigma rule (53), 99.73% of samples will be within three standard deviations
the mean, indicating that the vast majority of ppe. + are within (:p (t) -30mms Po m C
(t)
[-x/4, \/4] Therefore, the accuracy is computed with mmt, Ont E [- / T/ //.. The surface plot of Ace against me and na shown in Fig. SS] demonstrating that Eq. (A.92) guarantees at least 93.8% accuracy.
Accuracy
0.99
0.99
0.98
0.98
0.97
0.96
0.97
0.95
20.2.0.28
0.94
0.96
ACC 0:95
0.95
0.2
0.1
-0.2
-01
0.28 0.28.00200
-0.1
0.0
0.94
0.0
Omt
0.1
0.2
-0.2
Acc
0.938
Figure S1: Surface plot of accuracy of approximation as mt, Tne change. The plane representing Acc=0.95 is marked transparent red.
A.6.3 Approximate Asymptotic Convergence
Proposition A.10. We define the average effect of noise disturbance on attention score as:
N A (mm Uye 6'mm -' - A (w Uyy Cum.. ev..)
En
(A.95)
N
where N denotes the number of observations.
Given an error term it approximately holds:
(_eN 2N
IP (lENl 2 E) 2 2exp
(-85]
(A.96)
88
where = 1
N Σ (l1g)., Il], II)² is a constant independent of ev., Cv, Em and En. Further-
more,
(- 22N (_eN
P(EN 2 N
lim
lim 2 cxp No+000
0 Vc
0,
(A.97)
85
which indicates the positional disturbance-induced bias on attention score asymptotically converges 0 in probability, e.g., lim +00 Łn - 0.
14

---

The difference between noise-disturbed and original attention scores of the i-th observation is bounded as:
Proof.
A (v un e'm.. - -v' - A (w'm un eum - -vu))
cos(onn)) + +,300 sin() - az cos(p!n)
- -
-8 sin - v
Σ
V Vat (1)2 + =0(0? (cos(of") - 8 cos( p!th + +
S 2
V a (1)
+ B/01)
1=1
d
(sti.. 12 + (tt... (20)) 27
[
[ ki (21-11) 2 + Kitt (2t) 27
2
>
[=1
C.S.
Iem (21-11) (gt gb. (2t) )
kep (30-11)
(kt (261)
2
= =2 B1 1?21 112 = 2|14 |b 11:
(A.98)
where C.S. denotes the Gauchy-Schwand inequality, and 0!") [[-T, a directed angle defined as: a
3100
cos(O Valor + 00002
sin(o!")
(A.99)
V /a(02 + 0010
Applying Lemma A.4 we get:
e2N
PP PIEN - E(EN)) 2 E) < 2exp
(A.100)
85
where S = 1 N i= (1g ||| (1,,11) 11)
As shown by Eq. AA.I.. the expectation of En satisfies:
N
B(EN) N
Bein,, ei
(u(in un e'r. - -e'..) - A (win wn Cum - ev..]
1=1
02 ml + on
exp
1
(2) cos(sp!!) - -p!!)) + 0(0) sin(!!!)) -
-p!!!)
10.
As analyzed in Prop.
A.9 and Remark
A.4,
we have:
2 + o
exp
1 R0 0,
(A.102)
2
which leads to:
22N
IP (IEN - E((N)] P(ENN 2 e) 2exp
(A.)03)
88
To be more rigorous, we have:
e2N
lim IP (IÊN -[(N) 2 -P (PEN) 2 t) < 2 exp
(A.104)
Im In +
85
Since the second moment of || | ||. which can be assumed to be finite empirically, when
N + 0o, we have exp
=N (-2+)
85 + Thus P (lExl 2 e) asymptotically converges to zero. This
completes the proof.
Lemma A.4 (Hoeffding's inequality). Given series of i.i.d. random variables {X, with
P(X; [a,, b.l) 2 Vi [1 [1...n],
n). the sum Sn = En= X, satisfies:
262
P ([S, E((.) 2 e) < 2exp
(A.105)
Σm (b: (b:-a;)2
where 0 denotes the error term.
15

---

B Measuring Causality-Generaaity of Nodes within Directed Causal Graph
We define the gregariousness (TT)) of a node outgoing connections to other nodes, which is equivalent to the degree of causal generality when G represents a causal graph. While a node's out-degree can serve as simple proxy for gregariousness, it only captures 1-hop local connectivity. For example, node might connect to many immediate neighbors that themselves have no further outgoing connections. To capture the global connectivity, adopt PageRank-liks approach. As shown in Eq. (I8), we first compute a probability transition matrix P by transposing the absolute adjacency matrix A of G normalized by its in-degrees, with P... representing the probability that incoming connection of o; comes from vy- Note that P differs from a conventional transition matrix normalized by out-degrees, which instead represents outgoing connection probabilities. Next, restart probability matrix is added to P to ensure that the resulting Markov chain is strongly connected and ergodic [54]. In this random walk, nodes with higher global influence (i.e., more gregarious nodes) will accumulate larger steady-state probabilities, reflecting their broader reach across the graph and higher gregariousness. This steady-state distribution, given by the left eigenvector of P corresponding to the largest eigenvalue 1, defines the PageRank vector as shown in Eq. (18).
DAG C(V,E) as its propensity to establish
C Related Work
C.1 Position Encoding
For sequential data, position encoding methods broadly fall into two paradigms: Absolute Positional Encoding (APE) and Relative Positional Encoding (RPE). The canonical APE approach [[] employs fixed sinusoidal functions of varying frequencies to encode each token's s absolute position. Beyond this fixed design, various trainable absolute positional encoding schemes have been proposed to enhance performance (75) However, these approaches often fails to generalize to sequences longer than those seen during training. To address this limitation, RPE methods 22 S 15 [8
60 modulate attention scores based on the relative distance between tokens. Among these, rotary positional encoding (RoPE) [I8] applies position-dependen rotations to Query and Key vectors, using angles proportional to their absolute positions. These rotated vectors are directly involved in the computation of Query-Ke Key attention scores. RoPE offers several key benefits, including that long-term decaying attention scores, compatibility with linear self-attention [18), and enhanced understanding of contextual knowledge [19]. Most recently, several RoPE-based methods have been proposed to improve the extrapolation ability of Transformers to longer contexts by modifying the frequency-domanl representation of RoPE [60). incorporating decay-aware embeddings [6]] or employing interpolation- based techniques 62 However, these methods all assume a predefined sequential order among tokens, making them unsuitable for data without inherent ordering even when such data exhibit an implicit causal structure.
Positional encoding methods are limited and primarily developed for specialized domains. In the context of single-cell RNA sequencing (scRNA-seq), two main strategies have emerged for generating positional encodings for genes, which inherently lack natural ordering. The first strategy 24 25 48
utilizes static pscudo-positional encodings derived from large scale gene co-expression data, capturing association patterns between genes. This approach is analogous to static word embeddings generated using models like CBOW, where positional encodings are assigned based on gene "proximity" in expression space. However, these encodings not contextualized and fail to represent causal relationships between genes. The second strategy 22
generates contextualized pseudo- positional encodings by assigning gene orderings based ranked expression levels within the dataset. Positional encodings, either static or trainable, are then constructed according to these induced pseudo- orders. While more adaptive, this approach still primarily reflects superficial relationships based on relative expression and cannot capture more complex dependencies, such as causal interactions among genes.
23 63
C.2 Causal Structure Learning
Generally, there are four families of casual structure learning methods, including constraint- -ased methods, score-based methods, functional causal discovery, and gradient-base causal learning
41 64]. Constraint-basee methods (65 typically start with a fully-comnected graph from which
16

---

they learn causal graph structure by leveraging the independence between graphical structures (e.g.. chains, forks, and colliders). Score-based methods (8, 69] on the other hand, start with an empty graph and iteratively add or prune edges to maximize a scoring function (e.g., BIC) that measures how well the graph explains the observed distribution. A common drawback of both approaches their computational inefficiency and limited ability to estimate causal effect strength. Functional causal discovery methods [70 assume explicit functional forms (e.g., spline regression) and distributional properties (e.g. -on-Gaussianity) to recover both the causal structure and the strength of causal relationships. More recently, gradient-basee causal discovery methods have advanced causal structure learning by formulating the task as continuous optimization problem. For example, NOTEARS [26] enforces acyclicity through a smooth constraint embedded a data reconstruction loss, allowing efficient gradient- based optimization without combinatorial search independence testing. GOLEM [73] extends NOTEARS by incorporating a fikelihood-base objective with sparsity regularization, while retaining acyelicity constraints. However, both methods limited modeling linear causal dependencies. In contrast, neural network- approaches [27] 74, 75 introduce deep learning architectures to model complex nonlinear causal mechanisms, enabling scalable and flexible estimation of both structure and effect strengths in high-dimensional settings.
D Dataset Description
D.1 Pre-Training Datasets
D.1.1 Single-Cell Sequencing Data
We collect wide variety of single-cell multi-omics datasets from Homo sapiens and Mus musculus, cziscience com/ This collection includes 1,465 datasets, encompassing around 91.5 million cells and covering approximately 900 different cell types, with data spanning several sequencing methods and omics modalities.
which sourced from the CELLxGENE database (76)
The datasets are primarily divided into two broad categories: single-cell transcriptomics and single- cell epigenomics, depending on the type of molecular feature being analyzed, such as RNA expression or chromatin modifications. All datasets are organized into a standardized high-dimensional matrix X E R.N Nx xn, where each element Ti represents the gene expression values of gene g in cell j. Here, N denotes the total number of cells, and n refers to the number of genes. It is noteworthy that the format spatial transcriptomics data (e.g., Slide-seq) is processed consistently and does not take into account spatial coordinates and H&E images.
D.1.2 DNA Methylation Data
We adopt the pretraining dataset released by MethylGPT (77). which consists of DNA methylation data collected from 154,063 human samples through the EWAS Data Hub (78) and Clockbase [79]. The dataset includes approximately 300,000 patients, with low-quality entries filtered. The cleaned data was deduplicated, ensuring no repetitions in the training set, and randomly sampled to cover 20 distinct tissue types. We specifically focus on 49, 156 CpG sites selected for their biological relevance and array format compatibility, as detailed by the EWAS catalog. The data is structured into matrix X N xM where each element X,,, denotes the methylation level of CpG site j in sample i. Here, N is the number of samples and M corresponds to the number of CpG sites.
D.2
Held-Out Datasets
D.2.1 Gene Perturbation Prediction
Human Leukemia Cell Dataset The human leukemia cell dataset consists of three distinct datasets.
We use the Norman dataset (47 for GPP expriment. The Norman perturbation dataset provides gene expression profiles from the K562 leukemia cell line treated with Perturb-seq. This dataset includes 131 dual-gene perturbations and 105 single-gene perturbations, with each perturbation represented by approximately 300 to 700 cells.
17

---

D.2.2 Cell Type Annotation
hPBMC The hPBMC [80] dataset, sourced from a healthy donor, contains gene expression profiles for 68,450 peripheral blood mononuclear cells (PBMCs). It includes eleven distinct cell types: CD4+ T cells, CDS+ cells, B cells, natural killer (NK) cells, CD14+ monocytes, FCGR3A+ monocytes, dendritic cells, memory cells, helper2 cells, and megakaryocytes. These cells were processed using the 10x platform with scRNA-seq technology.
hPancreas The hPancreas 81 dataset comprises 2.209 single cells from human pancreatic islets, collected from six healthy donors and four type 2 diabetes (T2D) donors. It includes both endocrine and exocrine cells, representing eight cell types: alpha, beta, gamma, delta, and epsilon endocrine cells, well as acinar, ductal, and pancreatic stellate cells (PSCs). The cells were dissociated into single-cell suspensions, sorted via fluorescence-activate cell sorting (FACS), and subjected to RNA sequencing using the Smart-seq2 protocol.
hBMMC The hBMMC dataset [82 includes 35,882 bone marrow mononuclear cells (BMMCs) from healthy donors, containing six distinct cell types: progenitor cells, cells, T cells, NK cells, monocytes, and dendritic cells. These cells were profiled using single-cell assay for transposase- accessible chromatin sequencing (scATAC-seq) technology on the 10x platform.
mOP The mOP 83 dataset provides spatially resolved cell atlas of the mouse brain, containing molecular profiles for 338 major cell types from more than ten million cells, spanning eleven brain regions. was generated using Multiplexed Error-Robust Fluorescence In Situ Hybridization (MER- FISH), a spatial transcriptomic technique that enables gene expression profiling while preserving the spatial organization of cells within tissue sections.
D.2.3 Cell Clustering
(CoPE2_Specht SCoPE2 SSeec_ [5 is a representative single-cell proteomic dataset that quanti- fies 3,042 proteins 1,490 cells using the SCoPE2 method. It includes cell types: monocytes and macrophages. Notably, without polarizing cytokines, monocytes may adopt macrophage-lin traits, increasing cell clustering difficulty due their similarity.
SCoPE2 Montalvo SCoPE2_Monen Montalvo g4 quantifies 843 proteins in 508 cells using the SCoPE2 method. It contains five cell types: Vasculature, Beta cells, Beta cells, Delta cells, Alpha cells.
was generated by the pSCoPE technique. quantifies 2,844 different proteins in 1,543 cells, comprising two cell types: melanoma cells and U-93 cells.
PSCoPE Leduc pSCoPE Leduc 85
D.2.4 Age Prediction
We use a widely used DNA methylation dataset for age prediction, collected by [86), which includes 13,505 samples (21, (21,368 CpG sites) from multiple tissues. The dataset covers ages from 0 to 100 years, with the majority of samples derived from whole blood (47.2%) and brain tissue (34.5%).
E Evaluation Protocols
E.1
Gene Perturbation Prediction
The Gene Perturbation Prediction (GPP) experiment leverages learned gene representations predict the effects of targeted perturbations. These embeddings, generated by foundational models, serve as input to downstream classification heads that predict perturbation status, thereby elucidating gene function and regulatory network dependencies. For each foundational model incorporating distinct positional encodings, we computed the mean squared error (MSE) across the top 20 differentially expressed genes between pre- and portuueart expression profiles the evaluation metric.
18

---

E.2 Cell Type Annotation
As standard classification task, we adopt the evaluation framework established prior studies [22 87]. Under the fine-tuning setting, we append an additional classifier to the cell embeddings generated by each model and perform supervised fine-tuning on the model parameters to optimize task-specific performance. Then, we employ accuracy and macro F1 score as the evaluation metrics.
E.3
Cell Clustering
To assess the quality of the cell embeddings generated by our proposed method, we conducted cell clustering experiment, which is a standard practice in single-cell proteomics [88] 89]. We employed the k-means algorithm obtain the cell clusters and subsequently evaluated the clustering performance using three commonly used metrics, including ARI (90), NMI [ and ASW [92].
E.4 Age Prediction
Following established DNA methylation foundational models 777,, fine-tuned both our model and MethyIGPT using a ResNetID prediction head. During joint optimization, both the pre-trained MethyIGPT and the downstream ResNet1 were trained end-to-end, with mean squared error (MSE) as the objective function. Other con-pre-trainee models were also trained and evaluated on the same data splits. To robustly assess model performance, we employed the median absolute error (MedAE) as the evaluation metric.
F
Implementation Details
F1 Data Preprocessing
To ensure methodological consistency, we adopt unified data processing workflow for each omics included this study.
F111 Single-Cell RNA Sequencing
Gene List Mapping. After collecting the single-cell datasets, standardize their gene symbols to the HUGO Gene Nomenclature. Technological discrepancies between sequencing platforms occasionally result in absent gene annotations within specific datasets. To address this, unmapped genes are assigned zero expression values, thereby enforcing uniform gene symbol compatibility across all processed matrices.
Quality Control and Normalization. Quality control performed using Scanpy (50 to eliminate low-information cells, defined as those with fewer than 200 detected genes. To mitigate technical variability, raw expression counts are normalized by scaling each cell's total transcript count to 10,000 (library size normalization). Subsequently, non-zero expression values undergo loglp transformation to stabilize variance and reduce skewness in the data distribution.
Dataset Splitting. We split the dataset in similar way previous studies 87 Specifically, collect the large-scale datasets for pre-training and several held-out datasets for evaluation. The former does not need be split, while the latter needs be split into fine-tuning dataset and test dataset in a 3:7 ratio according to different uses. Furthermore, these held-out datasets usually come from different experimental conditions, donors, and due the common batch effects in single-cell data 193] they can be regarded as new data that differ from pre-training datasets.
F.1.2 Single-Cell Proteomics
The single-cell proteomic data were processed according the SCoPE2 pipeline 45). Raw MS files were analyzed in MaxQuant using the UniProt human proteome database [94] with TMT labeling modifications and 1 1% FDR filtering. Cells with <500 peptides or >20% mitochondrial proteins were excluded. Proteins detected in <10% of cells were removed. Missing values were imputed via k-nearest neighbors, and batch effects were corrected using LOFSS normalization and ComBat. Analyses used R with SCoPE2
45 and Seurat packages (95).
19

---

F.1.3 DNA Methylomics
The DNA methylation data preprocessing followed the MethyIGPT pipeline [77]. Initially, stringent quality control was applied to exclude samples with missing values exceeding 40% of CpG sites and remove duplicate entries. Subsequently, CpG sites were selected based on their biological relevance (associated with 25 EWAS traits) and cross-platform compatibility (detected 295% of samples). The methylation 8-values were standardized, with missing values intentionally preserved for downstream masked modeling tasks. The processed data were structured into a matrix X e R RN M, where N and M denote the number of samples and CpG sites, respectively, enabling systematic analysis of methylation patterns.
F.2
Transformer-Basel Backhone Models
We use the following transformer backbones with CAPE-generated positional encodings to learn both feature and observation-level representations for tasks in 3.ction[5.2aan Supp. G.2 For both backbones, CAPE-generated positional encodings used in place of the original positional encodings as described below.
discretizes continuous gene expression values via binning, mapping each to a learnable token embedding. To encode positional information, it assigns each gene fixed embedding.
ScBERT scBERT 24]
which remains static during training. Finally, the expression embeddings and positional encodings
are directly added and input into the transformer backbone (Performer (96) and the expression embeddings are updated with masked reconstruction learning. Specifically, given the sample matrix X E R XM where N is the number of observations (cells), and M is the number of features (genes). For each non-zero expression count Zij in each cell, it calculates the raw absolute values and divide them into B consecutive intervals [bx, bu+1), where k = 1, ... B, and each interval is assigned feature embedding in the code book C with items. Then, 9 in Section 3.1] which is a function to generate contextualized, causality aagossit intermediate feature embeddings, is defined as:
0 = G(v,; a.) C(bin(x:;)), bin(x.y)== ) {(:
Fiz > and Fij E [byy bs+
(F.166
otherwise.
In the original study, the fusion function F. to integrate feature embeddings and positional encodings simply defined as:
Flu), po,)) 2 Puy,
(F.107)
where Pey E R denotes the gene embedding of gene j generated by pretrained gene2vec 197]. In our study, we instead use the CAPF-generated positional encodings Pu, Rd and modify the F function as:
F(o), Pv,)) R((P,)))))) )ui,
(F.108)
where R the rotary matrix define as Eq. (26). Additionally, at the beginning of the input sequence vi. ...... vis of cell i, scBERT sets a special <cls> token, which uses the attention module to extract the cell-level embedding from {ut )
uses similar architecture to scBERT, with the main differences being: (1) it uses a different positional encoding; (2) it is pre-trained on a wider range of datasets, making it suitable for multi-omics: and (3) it adopts multi-task pre- mreitrainig paradigm. In particular, in terms of positional encoding, scGPT sets learnable gene embeddings for each gene and updates it during the training process. Therefore, scGPT maintains two different codebooks, Cbin with B items and Cgene with M items, one for assigning v) and one for Pny, as:
scGPT ScGPT 25
= g ..:) = Cbin (bin((())))))) Puz = Cyene(j).
(F.109)
Note that we use the CAPE- -ePE-geneatted positional encodings as sp,, for scGPT in our study, as described in Equation (F.108).
In summary, ScBERT is equivalent to using static absolute position encodings, while scGPT uses trainable absolute position encodings. When we practice CAPE on these models, we replace the positional encodings B.y and fusion function F set by CAPE with the native ones, while keeping the other model architectures unchanged.
20

---

General case When measurement values of features are not on a comparable scale, the binning- based G functions in Eq. (F106) and Equation (F.109) are no longer applicable for generating contextualized, causality-agoniste intermediate feature embeddings (e.g., fi ). In such cases, the canonical transformer without positional encodings is used as the & function in Eq. (3) to generate these intermediate feature embeddings via self-supervised reconstruction-bet based training objective.
F.3
Benchmark Methods
F33.1 Positional Encoding
Trainable Relative Position Encoding We the trainable relative position encoding proposed by 49] as our benchmark. Instead of relying on absolute position embeddings, this method represents the distance between query position m and key position using sinusoidal-based vector Pm-n-
Content vectors E, and these relative encodings are projected separately (via W and Ws) and combined with two global bias vectors (content bias) and (position bias). The resulting attention score
qm kn = T T W WE + I. W Wx Pm-n + u W W 2n + v W W&Pm-n (F.110)
ensures that attention depends only on relative distances, giving the model translation invariance and better generalization longer sequences.
F.3.2 Multi-Omics Analysis Benchmark Models
KNN-ComBat KNN-ComBat is a standard method in the existing single-cell proteomics data analysis pipeline (98), which combines KNN-based imputation with ComBat-based batch correction for routine data preprocessing.
MAGIC MAGIC is diffusion-basee method for data cleaning single-cell RNA sequencing, effectively imputing missing data and recovering gene interactions by sharing information across similar cells.
a deep neural network cleaning single-cell RNA-seq data, using an autoencoder and classifier to remove noise and recover missing data, improving downstream analysis.
AutoClass AutoClass 100)
effectively corrects batch effects by iteratively clustering cells and adjusting their positions in PCA space, ensuring the integration reflects biological rather than technical variation.
Harmony Harmony [01
Scanorama Scanorama 102 is tool for integrating single-cell RNA-seq data across multiple datasets while correcting for batch effects. It uses fast, alignment-based method that projects data into shared low-dimensional space, ensuring that the biological variation preserved while mitigating technical variability.
sePROTEIN scPROTEIN 88 framework addresses peptide uncertainty, missing data, batch effects, and noise in single-cell proteomics. uses multitask heteroscedastic regression for peptide uncertainty and graph contrastive leamingg for cell embedding, enhancing clustering, batch correction, and annotation.
MethylGPT MethyIGPT 77 is a transformer-based foundation model DNA methylation analysis, demonstrates superior performance across key tasks including age prediction, disease risk prediction and missing data imputation.
AltumAge AltumAge 86 is a deep learning- epigenetic clock designed to predict human age using DNA methylation data from multiple tissues. It outperforms traditional linear models by leveraging a neural network architecture capable of capturing complex interactions between CpGi sites.
21

---

ElasticNet ElasticNet [103] is linear regression model widely used in the construction of epige- netic clocks. By applying regularization to DNA methylation data, effectively selects CpG sites related to age prediction in high-dimensional data.
Horvath's clock
Horvath's clock [104] a DNA methylation- biomarker developed by Steve Horvath to estimate the biological age of skin and blood cells.
E.4 Training Details
Causal Structure Learning (Step I) Given preprocessed matrix X E RNXM, we parameterize the causal graph as a learnable matrix A E I MMM Both encoder and decoder are 1-64-1 MLPs. We train A via Eq. (9) with regularization coefficient X₂ where size of 128, a learning rate 3e-3, and 100 epochs for optimization. After training, we apply a pruning threshold of 0.2 to obtain the final adjacency matrix.
AdamW with batch
Mapping Causal Structure to Hyperbolic Space (Step II) Given the trained A RMMx from Step I, map each variable into a d-dimensional hyperbolic space, where d D/2, and the dimensionality variable embeddings D determined by the selected transformer backbones (c.g.. D 200 for scBERT and D 512 for scGPT), Then, k hop in the graph contrastive learning Eq. (66] is set as 2, while the regularization weight Ag is set as 0.1, and the relative weight for the restart matrix is set as 0.15. Finally, we also choose the AdamW optimizer with batch size of 32, a learning rate of 1e-3, and 1000 epochs for optimization.
Transforming Hyperbolic Positional Encoding to Rotary Form (Step III) For scBERT, set the dimension of feature embeddings to 200 and the backbone network adopts the performer architecture. The pre-training process is consistent with the values in the original scBERT study, that is, epochs is set to 100, batch size 3, learning rate is le-4, and Adam is used for optimization. For scGPT, we set the dimension of feature embeddings to 512. The backbone network has 4 transformer blocks, each with attention heads. The pre-training process is consistent with the values in the original scBERT study, that is, epochs is set to 60, batch size is 5, learning rate is le-4, and Adam is used for optimization.
G Additional Experiments
G.1 Empirical Evaluation of CAPE's Properties
In this empirical analysis, we evaluate the effectiveness of CAPE in enhancing both the causal awareness and robustness of the self-attention mechanism. Across all experiments, the query and key vectors are fixed and generated as 128-dimensional random vectors: qum. ky,, E R ~ (0, where D 128. The dimensionality of the Poincaré ball positional encodings Cum. Cv, is set to = D/2 2 64.
G.1.1 Attention Attenuation Induced by Causal Distance and Causal Generality
In this analysis, pairs of {eu.. Cu,. are sampled from isotropic Gaussian in Rd and subsequently normalized to lie within the unit Poincaré ball. As a result, the norm r = ||e,., I| = ||e,,. varies within the open interval (0, 1), and the Poincaré distance de(e : ev..) spans the range 5]. The upper bound of the attention score, A+,, is computed for various combinations of dp e and 3D surface in Fig. S2 On one hand, for fixed values of r, A monotonically
and visualized decreases as the Poincaré distance (causal distance) increases, consistent with the causal distance- induced attention attenuation stated in Prop. 4.1 On the other hand, for fixed values of dly (er Cv., ) A also monotonically decreases as the causal generality (1 - r) increases, aligning with the causal generality-nnduced attention attenuation stated in Prop. 4.2
G.1.2 Robustness to Positional Disturbances
Here, we sample single pair of {ee... Cv, } as described in Supp. G.I.I To simulate perturbations, we generate varying number (N [1, 100 2 of Gaussian noise pairs (Ee...: Ev,. AO Diag(o) which are added to ev... and ev,, to obtain perturbed positional encodings {e'm.. e,,, J. from which we
22

---

Upper bound of attention score
A
30
28
28
27
26
0.10
0.30
Causal generality
0.50
Poincare distance d, e...
0.70
e._
Figure S2: 3D surface showing the effect Poincaré distance (causal distance) and causal generality As Poincaré distance and causal generality increase,
on the upper bound of attention score A+. attention attenuation decreases.
Noise std 0.1
Noise
Noise std 0.3
2
0
-2
-2
-4
-4
-6
-6
100
N (number of noise pairs)
N (number of noise pairs)
N (number of noise pairs)
Zero
Average attention bias
Figure S3: Robustness CAPE-derived attention scores to positional noise. Each subplot shows the average attention bias against the number of the noise pairs N under three Gaussian noise levels(a = 0.1, 0.2, 0.3). The red horizontal line marks the zero bias.
compute the average attention bias εN as defined in Eq. (A.55. For each value of N. the experiment is repeated for 100 times, and the results are visualized as scattered plots in Fig. S3 Each panel in Fig. S3 corresponds to different noise level, with standard deviations that as N grows from to 100, distribution of E.v becomes increasingly concentrated around zero
0.1, 0.2, 0.3. We observe
all noise levels. This empirical trend aligns with the theoretical result oom EN 4 stated in Prop. A0 confirming the asymptotic robustness of CAPE-derived attention scores to random perturbations.
G.2 Multi-Omics Analysis
In non-sequential data (e.g., single-cell multi-omics), learning high quality feature embeddings is critical for improving observation-level representations. For instance, in single-cell analysis, robust embeddings of genes proteins inherently capture latent biological states (e.g., cell types or developmental trajectories), which directly enhance downstream tasks like clustering or classification. This aligns with practices in natural language processing (NLP): models such as BERT leverage [CLS] token aggregate sequence-level semantics for sentence classification. To validate CAPE's capability in bridging feature embeddings and observation- semantics, we further applied CAPE to three representative observation-leva level tasks spanning multiple omics: (1) cell type annotation
23

---

Table S1: Performance comparison of cell type annotation scRNA-seq datasets. Acc and MF1 denote accuracy and macro Fl Sc-re (%), respectively.
hPBMC
haaancraas
hBMMC
mOP
Methods
Pos Encoding
Acct
MF11
Acc1
MFL
Acct
MF1
Ace
MF11
Static absolute' i
75.74
67.34
69.21
67.03
67.09
59.25
74.37
70.22
scBERT
Truinable relative
77.51
70.66
73.48
71.14
69.79
66.90
77. 77.64
71.79
CAPE
80.71
72.32
78.07
74,31
74.49
71.55
85.27
80.41
Trainable absolute!
84.48
75.39
70.76
68.03
67.18
60.93
80.14
77.03
scGPT
Trainable relative
84.47
77.01
74.87
72.42
75.65
73,91
85.37
79.40
CAPE
85.14
77.09
82.27
75.10
78.14
70.76
87.62
82.07
in scRNA-seq data, (2) cell clustering in single cell proteomics, and (3) age prediction in DNA methylomics.
We begin with cell type annotation, the most common task in single-cell foundational model to evaluate the cell embeddings generated by models. Following a similar experimental setup as described in Section cell embeddings are learned using scGPT and scBERT with three types of positional encoding across three human datasets (hPBMC, hPancreas, and hBMMC) and one mouse dataset (mOP) (See Supp. D.2 for details). We find that both models, scGPT and scBERT, when combined with CAPE, achieve the best performance.
For single cell proteomics, we evaluate the cell embeddings in the cell clustering task, applying it to two datasets: SCoPE2 Specht and SCoPE2 Montalvo (see Supp. D.2 details). Given the absence of transformee-based models designed for single cell proteomics, we leveraged scGPT to generate cell embeddings. Although scGPT'ss is originally designed for scRÑA-seq data, fine-tuned it the PSCoPE Leduc dataset to adapt it for proteomics data, After fine-tuning, we used scGPT to obtain the cell embeddings for clustering, respectively using its default position encoding and CAPE for comparison.
Due to the lack of established foundational models and the scarcity of single cell proteomic com- putational methods, current work in single cell proteomics often uses methods originally developed for scRNA-seq as baseline. To ensure comprchensive benchmarking,
evaluated our method against: (1) scPROTEIN, state-of-the-arr representation learning framework specifically designed : (2) the common proteomics analysis pipeline
com- bining KNN-based imputation with ComBat batch correction [98] and (3) established scRNA-seq computational methods adapted for proteomic data, including 192 (See Supp. [F3] for details). As shown in Tab, S2] scGPT with CAPE significantly outperforms both the original scGPT and other baseline methods across all evaluated metrics in clustering. This demonstrates that CAPE-derived cell embeddings preserve substantially richer biological information.
for single-cell proteomies [
We further assess CAPE's performance in predicting age from DNA methylation patterns. Similar to the experimental setup in GPP (Section (5.2), we utilize a transformer-basee DNA methylation foundational model, MethylGPT [77). to generate cell embeddings, using its default position encoding strategy along with CAPÉ. Similar to the position encoding used in single-cell foundational models like ScGPT 25 and ScBERT 24 MethylGPT assigns embeddings each CpG site, similar to how genes are represented in ScGPT. Additionally, methylation values, much like gene expression counts in scRNA-seq, are also assigned embeddings, which are then summed and subsequently used as input to the transformer blocks. For a comprehensive assessment, we benchmark CAPE's age prediction performance against three widely use methods including AltumAge [6 ElasticNet [0X) and Horvath's 's clock [144.
After fine-tuning for age prediction, MethylGPT, enhanced with CAPE, achieved superior accuracy and exhibited the lowest median absolute error among all methods [S3). This demonstrates CAPE's capacity capture the hidden causal structure between CpG sites, effectively learning biologically meaningful age-related patterns.
G.3 Sensitivity Analysis
In this section, we conduct sensitivity analysis for three key hyperparameters of CAPE's training objective, including the DAG pruning threshold 7 (see Section 3.3), the k-hop neighborhood size for graph contrastive learning (Eq. (16). and the weight X, of the causal generality penalty term
24

---

Table S2: Performance comparison of cell clustering single-cell proteomics.
SCoPE2 Seeecht
SCoPE2 Montallvo
Methods
ARI
NMI
ASW
ARI
NMI
ASW
KNN-ComBat
0.317
0.066
0.375
0.274
0.053
0.536
MAGIC
0.245
0.375
0.339
0.452
0.389
0.693
AutoClass
0.02
0.313
0.211
0.316
0.253
0.421
Harmony
0,406
0.230
0.422
0.443
0.132
0.682
Scanorama
0.013
0.215
0.003
0.001
0.138
0.006
scPROTEIN
0.435
0.428
0.469
0,502
0.465
0.689
scGPT
0.396
0.405
0.156
0,482
0.377
0.383
scGPT CAPE
0.513
0.497
0.475
0.516
0.572
0.631
Table S3: Performance comparison of age prediction DNA methylomics datasets. MedAE denotes Median Absolute Error.
Methods MethyIGPT
AltumAge
ElasticNet
Horvath's s clock
MethylGPT w/ CAPE
MedAE
4.59
6.53
5.16
6.88
4.07
0.254
0.2
8.0.116
Prediction MSE
0.24
0,237
0.22
0.210
021
0.208
0.203
0.22
0.212
0.189
0.20
0.20
0.181
0.184
0.19
0.187
0.188
0.19
0.182
178
0.186
0.182
0.188
0.18
0.184
0.175
0.174
0.178
0.175
0.1
0.2
0.3
0.4
0.5
0.6
2
0.00
0.05
0.10
0:15
0.20
DAG pruning threshald
k-hop neighborhood
Regularization weight
Single-gene perturbetion
Double-gene perturbelions
Figure $4: Sensitivity analysis about DAG pruning threshold, k-hop neighborhood, and regularization weight.
Ω in Eq. (6))- Fig. illustrates the Prediction Mean Squared Error (MSE) single-gene and double-gene perturbation tasks as these hyperparameters are varied.
In the lefimost panel, observe that increasing from 0.1 to 0.2 significantly improves prediction
accuracy. However, as т continues to increase beyond 0.2, performance gradually declines. This pattern reflects trade-off: low threshold (r 0.1) fails sufficiently eliminate noisy, false- positive causal edges, whereas high threshold (r > 0.3) may excessively prune true causal edges, thus degrading the quality of the learned causal structure.
The middle panel shows that performance peaks designate features with strong 2-hop causal relationship as negative causal pairs Conversely, large (c.g., 2 4) risks misclassifying weakly or non-causally related features as positive causal pairs. Both extremes undermine the effectiveness of the graph contrastive loss in preserving salient causal relationships, leading to suboptimal positional encodings.
2. small l: (e.g., k = 1) may incorrectly
The rightmost panel displays V-shaped trend with respect to Ag. with optimal accuracy achieved A = 0.1. diminutive Àg may not sufficiently regularize causally general features towards the origin, thereby weakening the encoding of causal specificity. Conversely, an excessively large Xg could force all features towards the origin, collapsing causal distances and diminishing the model's capacity to discern varying causal strengths. Empirically, setting >g in the range of 0.1 to 0.15 appear to offer a favorable balance, preserving both causal specificity and relative causal distances in the positional encodings.
25

---

Causal structure learning (Step
Mapping causal structure (Step
Average time (saconds)
0.4
0.388
1.5
1:545
0.173
0.3
0.311
1.0
0.953
0.10
0.107
0.2
0.208
0 1.542
0.05
0.081
0.1
0.131
0.257
0.028
0.043
0.043
0
0.005
20000
40000
60000
80000
200
400
600
800
200
400
600
800
(number observations)
M (number of variables)
(number variables)
Figure S5: Runtime per epoch during training Step and Step Il sampled sub-datasets.
G.4 Complexity Analysis
We begin by analyzing the computational complexity of CAPE. Step I, given a data matrix X E N with N observations and M features, CAPE implements the encoder and decoder functions (Eqs. (6) and (7)) using MLPs, and solves the nonlinear SEM through trainable adjacency matrix E RR N complexity of (N 2² [105]. To enforce acyclicity, CAPE introduces smooth constraint term h(A), which involves computing matrix exponential and incurs time complexity of (M3) [06 To mitigate this computational bottleneck, we adopt the low-rank approximation strategy proposed by Dong, et al. [177] our implementation. Specifically, A is approximated as UV with U, V gMMr and rank r = 40. This approximation reduces the computation complexity of acyclicity constraint to O(M2r), yielding an overall complexity of O(N + r)M for Step I. Thereby, CAPE achieves significantly improved scalability while maintaining its expressiveness. In Step II, CAPE maps each feature the DAG to a d-dimensional hyperbolic embedding. The dominant cost in this step arises from graph constrastive learning, which has a time complexity of ( [08].
This step analogous to training graph neural network (27), with a time
We empirically assess CAPE's scalability with respect to the number of samples (N) and features (M).
To evaluate the impact of N, we subsample N 1, 3, 5, 7, 10 instances from the GPP dataset
(Tab. 2 with M = 100 fixed. To assess the effect of M, we vary M 100, 300, 500, 700, 900 while fixing N 10 For each configuration, and reuse it in Step II. We repeat each experiment 10 times and report the average runtime in Fig. [S5 As expected, runtime scales linearly with N, and approximately quadratically with M, reflecting the theoretical complexities of M2 and (dd22) in Steps and II, respectively.
learn a separate adjacency matrix A E R M in Step I
H Limitations
While CAPE offers a general and theoretically grounded solution for encoding causal structure in non- sequential data, its effectiveness currently relies on the quality of the inferred causal graph. Although we adopt robust variational formulation for causal discovery. inaccuracies may arise extremely noisy or undersampled settings. Additionally, our current implementation assumes feature-wise causal structure to static across samples, which may not fully capture sample-specific heterogeneity in highly dynamic systems. These limitations point to promising directions for future work, such as incorporating uncertainty-aware causal discovery or adapting CAPE to sample-dependent causal structures.
Broader Impacts
By enabling transformers to model non-sequential yet causally related features, CAPE has the poten- tial to advance representation learning in wide range of scientific domains where causal structure is key-such as biomedicine, economics, and environmental science. In particular, our method may as- sist researchers in uncovering interpretable, causally grounded representations from high-dimensional
26

---

biological data, potentially informing therapeutic target discovery or precision medicine. As with any causal inference technique, misuse or overinterpretation of inferred relationships remains risk. especially in domains where observational biases are strong. We encourage responsible use of CAPE in conjunction with domain expertise, and highlight the importance of open datasets, reproducible code, and transparent evaluation to mitigate unintended consequences.
27


Reviews:
[{"review_id": "PNTnqPsfoI", "reviewer": "Reviewer_TuE7", "summary": "The paper introduces an end-to-end deep neural network for 1) causal discovery/causal strength estimation 2) causal graph encoding 3) self attention of non-sequential feature with causal positional encoding. It provides theoretical properties of the attention with respect to the positional encoding, verified by ablation experiments, and shows experimentally how the introduced encoding improves a transformer network compared to other encoding methods.", "questions": "Other than making sure that the theoretical properties are correct, and that the plain text explanation properly reflects the strength of the guarantee (which would be my main concerns), I have one other question.\n\nThe greatest limitation of the encoding method used is that rotation matrices do not guarantee that distance on the pointcarre space result into null attention. This is due to the necessity of gamma being function of the difference in positional encoding, which results in the radius function being independent of the positional encoding. **What if gamma was function of the encoding difference AND causal generality?** While this differs from the usual setting, this could be fundamental in bridging Pointcarré space with positional encoding. Note that I did not carefully check if this suggestion is viable.", "limitations": "On top of the limitations identified in appendix (which should be put in the main paper), the authors should mention that the rotation-based encoding does not entirely reflect the information encoded in pointcarré space. It might be possible to obtain better guarantees with a different positional encoding transformation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strong points**\n - Clarity: the paper is precise, well detailled, organized, and the extensive proof rely on unified and clear notations.\n - Quality: The theoretical results underline desired properties that are validated in ablation experiments, despite some theoretical guarantees remaining somewhat weak. Real data experiments were reported with standard deviation that seems to indicate that the improvements are statistically significant.\n - Originality: While the encoding of graph in Pointcarré space is not novel, its use to create positional encoding from *causal* information is, to my knowledge, novel.\n - Significance: Combining causal analysis and the more popular architectures of Deep Neural Network is an unsolved yet high impact problem.\n\n**Weak points**\n\n**Clarity**\n - Lines 212+: pointcarre encodings denote a point on a disk, so why is it that phi = c.e is an angle vector? The multiplication does not change the angle, just the magnitude of e, since e was defined using cartesian coordinates. Therefore, phi does not correspond to a coordinate system that involves angles.\n - Typo line 929, should be an inequality. The proof remains sound.\n\n**Quality**\n - Line 831+: There is nothing guaranteeing that A+ and A- tend to 0. In fact, the term Beta_i only depends on W_q, W_k, v_q and v_k, none of which involve the positonal causal aware encoding.\n - In fact, if a first point has pointcarre embedding 0 and another is (a,0,0,0...) with a varying from 0 to 1, the changes to matrix R will only affect the first 2D block. This countradicts A tending to 0 when dp tends to infinity.\n - The same example shows that for those two points, the attention (line 246) may have large upper bound (lower bound by symmetry) despite causal generality.\n - Going further, since feature encoding v and positional encoding phi are computed separately, and since rotation matrices have determinant 1, for every feature embedding v1 and rotation matrix, there exists and embedding v2 (perhaps unused), such that the attention score is ||v1||. To do so, it is enough to invert R and multiply by v1. Relying only on a rotation matrix is not sufficient to guarantee that the causal proximity affects each attention scores. Attention score bounds may only be **probabilistic**.\n - Proposition A.8 also falls short of the stated goal. It relies on that vm and vn are of same mean, are independent, vm and delta are also independent, so their attention cancel out on average. The only term left is the attention of vm with itself, reasonably positive. The problem is that we would like that for any vm and vn, reasonably far from each other, vm+delta has higher attention score.", "was_revised": true, "final_justification": "Authors responded and amended their statements as asked. Other reviews are positive, I believe that the paper should be accepted.", "rebuttal": "We sincerely thank you for your insightful comments and careful check over our theoretical parts, which greatly improve the quality and clarity of our work.\n***\n**Weak points Clarity 1**\n\nWe appreciate the reviewer for pointing out that the term \"angle vector\" might be misleading. In our formulation, the word “angle” refers not to the polar coordinate of a point on the Poincaré disk, but to the **rotary phase** employed by RoPE. For every 2-D sub-space $(2t-1,2t)$ , the $t$-th element of $\\varphi_{v}$ serves as the rotation phase in the rotation matrix $R(\\varphi_{v})$. Thus, when computing the attention score (Eq. 25), the factor $R(\\varphi_{v_m}-\\varphi_{v_n})$ lets tokens $v_m$  and $v_n$ to be aware of the difference for *each coordinate* of their Poincaré embeddings and utilize this positional information during their training. To avoid confusion, we revise the description of $\\varphi_{v}$ on Line 212 to read: \"$\\ldots$where $\\varphi_{v}\\coloneqq ce_v$ denotes a vector whose components are used as the rotation angles in the subsequent rotary embedding$\\ldots$\". \n***\n**Weak points Clarity 2**\n\nThe typo on line 929 has been corrected.\n***\n**Weak points Quality 1&2**\n\nYou are entirely correct in catching this mistake due to our negligence. As $d_p(e_{v_m}, e_{v_n})\\rightarrow+\\infty$, the correct attenuation bounds for $\\mathcal{A}^+$ and $\\mathcal{A}^-$ are $| \\alpha_i^* | d \\cos\\left(\\frac{2\\pi}{d}\\right) + \\sum_{t=1}^d | \\beta_i^{(t)} | $ and $-| \\alpha_i^* | d \\cos\\left(\\frac{2\\pi}{d}\\right) - \\sum_{t=1}^d\\lvert\\beta_i^{(t)}\\rvert$ , respectively. In fact, we realized these bounds intuitively are more sensible for several reasons: \n\ni) **Semantic component persists.** $ \\alpha_i^* $ and $\\beta_i^{(t)}$  encode the *semantic* similarity between tokens through $W_qv_m$ and $W_kv_n$. They are independent of positional information and therefore do not vanish with causal distance. \n\nii) **Causal attenuation is still present.** The rotary phase affects only the $\\alpha_i^*$ term via the cosine factor shown above, ensuring that, for a fixed semantic similarity, the attention score is attenuated when the Poincaré distance grows. \n\niii) **Desirable behavior.** The attention score should not vanish for two features that are semantically related, even if they are causally distant. For instance, two genes may belong to the same protein family (high semantic similarity) but be regulated by entirely different pathways (large causal distance). Their attention score should remain non-zero to reflect their shared function, even as it is attenuated by their causal separation. We have corrected this mistake in the Remark A.3. section in our manuscript.\n***\n**Weak points Quality 3**\n\nYou are correct that the absolute magnitude of the attention score's bounds is not determined by causal generality alone, and can indeed be large. Our intention was not to claim that high generality guarantees a low attention score in all cases, but rather to describe its specific attenuating effect. We will clarify this in the manuscript and below:\n\nAs shown in Eq. A.73, the two bounds depend on $ \\alpha_i^* $, $\\beta_i^{(t)}$, which encode the *semantic* similarity between tokens,  and causal distance $d_p(e_{v_m},e_{v_n})$ (in $C$) .  Thus, the attention score between $v_m$ and $v_n$ can remain significant even under extreme causal generality since causal generality is not the sole factor that determines the attention. The main point is, given fixed semantic similarity and causal generality, it is desirable that the attention score is attenuated when causal generality grows. Crucially, the raw attention scores are not the final weights used by the model. These scores are inputs to a **softmax function**, which normalizes them across all features in the sequence to produce a probability distribution. This means the **relative values** of the attention scores are more important than their absolute magnitudes. While a highly general feature might still produce a large raw score with a semantically similar neighbor, the causality-induced attenuation ensures this score is *relatively lower* than it would be for a more specific feature at the same causal distance. This relative difference is preserved and utilized by the softmax function to weigh the final feature representations.\n***\n**Weak points Quality 4**\n\nThis is a sharp theoretical observation. You are correct that for any feature encoding $v_1$ and rotation matrix $R$, it is mathematically possible to construct an encoding $v_2=R^Tv_1 $that perfectly cancels the rotational effect in their dot product. However, our framework's effectiveness relies on what the model is incentivized to learn during end-to-end training, not on this theoretical edge case. There are two main reasons why this is not a concern in practice: \n\ni) **Inductive Bias and the Learning Objective:** The core premise of CAPE is that injecting causal geometry provides a powerful **inductive bias**. That is, the feature encodings are not fixed but trained to minimize a loss on a downstream task, being rewarded for leveraging the information provided by the causal positional encodings to make better predictions. It is highly improbable that the optimal solution for minimizing the global loss would involve learning a complex set of feature embeddings that systematically \"conspire\" to cancel out this useful causal structural information. The model is far more likely to learn representations that benefit from the provided geometry. \n\nii) **Empirical Validation from Ablation Studies:** Our experiments provide strong evidence that the model does, in fact, learn to use the causal information effectively. As shown in our ablation study (Table S4), the performance of the full CAPE model is substantially better than variants where the causal or rotary components are removed. This empirically demonstrates that the causal-rotary mechanism is not being ignored or counteracted by the learned feature embeddings but is critical to the model's success.\n***\n**Weak points Quality 5**\n\nWe are very grateful for this insightful critique. Our original intent with this proposition was to show that the rotary attention mechanism could distinguish genuine semantic differences from random noise. However, you have correctly identified the proposition's limitations. We agree that its proof relies on strong statistical assumptions that cause the attention terms to cancel out on average, and it does not guarantee that it is true for any pair of features. Upon reflection, we concur that this \"in-expectation\" result falls short of its goal. To improve the focus and rigor of our paper, we have removed Proposition A.8. This allows our theoretical analysis to center on the more impactful, instance-level guarantees of causal attenuation. We sincerely thank you for this comment, which has helped us to improve the quality of our paper.\n***\n**Questions**\n\nWe thank the reviewer for this thought-provoking suggestion. Indeed, in our current formulation, the function $\\gamma$ is dependent only on the difference in positional encodings. While our method does not explicitly factor causal generality into the $\\gamma$ function, we argue that this information is **implicitly encoded and available to the model**. A feature's causal generality is defined by its proximity to the origin of the Poincaré disk. This centrality is inherently reflected in its set of pairwise causal distances to all other features; a more general feature will, on average, be closer to all others. Since the model has access to all pairwise causal positional differences during training, it can therefore learn to recognize a feature's generality from this context. This is conceptually similar to how kernel methods can recover geometric properties from a matrix of pairwise similarities. \n***\n**Limitations**\n\nWe understand this limitation also refers to the same point you raised in the *Questions* section, which we have responded above. Please let us know if it refers to something else."}, {"review_id": "FL6vtKZ0wf", "reviewer": "Reviewer_wHJ8", "summary": "This paper introduces CAPE, a positional encoding method that learns causal structures among non-sequential features to provide Transformer with positional information related to causal structure rather than sequential structure. CAPE first uses generalized structural equation model (SEM) to identify causal directed acyclic graphs (DAGs) among features. It then embeds the identified DAGs into hyperbolic space and uses the hyperbolic surface model to preserve two key properties: causal strength and causal specificity. Finally, the hyperbolic embedding is converted into a rotated positional encoding form, which is compatible with the self-attention mechanism of Transformer. The authors provide theoretical analysis demonstrating that CAPE possesses desirable properties including causal distance-induced attenuation, causal generality-induced attenuation, and robustness to positional disturbances, and validate the method's effectiveness on synthetic data and multi-omics datasets.", "questions": "1. Computational efficiency - How does the computational efficiency compare? Can you provide complexity analysis?\n2. Generalization - Is the method still robust when dealing with larger causal graphs or noisy conditions? Does this approach still work in a causal diagram that only shows a partial set of causal relationships? Is it possible to modeling different causal strengths?", "limitations": "The effectiveness in other domains remains to be confirmed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Well-motivated - Addresses the important practical problem that traditional positional encoding cannot handle non-sequential causally-related features, with significant application value\n2. Novel approach - Cleverly combines causal discovery, hyperbolic geometry, and rotary attention mechanisms\n3. Solid theoretical foundation and well-designed experiments - Provides rigorous theoretical analysis with comprehensive experimental validation\n\nWeaknesses:\n1. Limited baseline comparisons - Lacks comparison with more diverse baseline methods, with experiments confined primarily to biomedical domains\n2. Potential computational overhead - May increase computational costs, uncertain practical utility", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal and the comments provided by other reviewers. The authors have addressed the majority of my concerns effectively. In particular, I appreciate the clarification regarding the algorithm's complexity and its behavior on sparse graphs. Overall, I believe this is a promising piece of work, and I have decided to maintain my final score of 5.", "rebuttal": "We greatly appreciate your careful evaluation and constructive comments. To better respond to your concerns, we have summarized and responded to the following points.\n***\n**Limited baseline comparisons and effectiveness in other domains**\n\nThank you for this constructive comment about applying CAPE to other domains. We focus on the biomedical domain as the rapid development in omics technologies in recent years allows the accumulation of enormous data that CAPE can be applied to. So we consider the biomedical domain the main battleground for CAPE. However, we agree that the application of CAPE to other domains is also important. Due to the limited rebuttal time, we conducted a simple experiment using the MovieLens-32M dataset (with 87,585 movies and 200,948 users), which is commonly used in test recommendation algorithms. This data has a table format with rows representing users, columns movies, and entries ratings. Our goal is to learn causality-aware embeddings of movies for their multi-label genre classification (20 genres). The benchmark methods include Transformer w/o positional encodings (Transformer-w/o) and an MLP-based classifier that uses the raw data. For CAPE-Transformer and Transformer-w/o, linear probing is adopted to train a projection head for multi-label classification using the derived movie embeddings of the training set. The benchmark MLP classifier is trained in a supervised manner using columns of the raw data as inputs. All movies are partitioned into training (80\\%), validation (10\\%), and test (10\\%) sets. \n\n| Inputs | Classifier | Jaccard Index | Macro-F1 |\n| -- | -- | -- | -- |\n| Raw Column Data | MLP | 0.457 | 0.583 |\n| Transformer-w/o-generated movie embeddings | Projection head | 0.633 | 0.659 |\n| **CAPE-Transformer**-generated movie embeddings | Projection head | **0.678** | **0.690** |\n\n***\n**Computational efficiency**\n\nWe provide a detailed complexity analysis in Supplementary Section G.5. The computational cost of CAPE is primarily concentrated in two preprocessing steps: Step I (Causal Discovery) and Step II (Hyperbolic Embedding). Specifically, in Step I, the main computational bottleneck arises from enforcing the acyclicity constraint during causal structure learning. To mitigate this, we adopt a low-rank approximation of the adjacency matrix, reducing the constraint’s complexity to $\\mathcal{O}(M^2 r)$, where $r \\ll M$ is the rank parameter. As detailed in our analysis, this results in an overall complexity of $\\mathcal{O}((N + r) M^2)$ for this step, where $N$ is the number of samples. Step II, which performs hyperbolic embedding of the discovered graph, has a complexity of $\\mathcal{O}(d M^2)$, where $d$ is the embedding dimension. The causal encodings generated in these steps are then fed into a standard transformer architecture, where attention is computed across $M$ feature embeddings for each of the $N$ samples, with a complexity of $\\mathcal{O}(N M^2)$. Thus, although CAPE introduces a modest additional cost during the construction of positional encodings, it preserves the overall complexity of standard transformers. The total computational cost remains $\\mathcal{O}(N M^2)$, with only a constant-factor increase, rather than a higher-order growth such as $\\mathcal{O}(M^3)$. Finally, our \\textbf{empirical results} support the theoretical complexity analysis, showing that the runtime of CAPE scales linearly with $N$ and quadratically with $M$.\n***\n**Robustness and Generalization.**\n\nThis is a great and constructive question. \n\n**i) Graph size**. We agree that accuracy can decrease as the graph size increases due to the exponentially increased search space and the elevated risk of converging to local optima. However, in practice, the local solutions can often be high-quality as demonstrated by other gradient-based causal discovery methods like NOTEARS. Furthermore, performance is often better on sparse graphs due to constrained search space, which are common in real-world systems like gene regulatory networks. CAPE leverages regularization and thresholding on the adjacency matrix to promote sparsity. \n\n**ii) Noisy condition**. CAPE is designed to be robust to additive Gaussian noise, as this is an explicit component of the SEM framework. However, we acknowledge two limitations: First, performance can degrade as the noise variance increases, reducing the signal-to-noise ratio in the data. Second, performance may decline if the true noise follows a non-Gaussian distribution, as this constitutes a model misspecification. \n\n**iii) Effectiveness with partial causal graphs.** CAPE's sparsity promoting of inferred DAG can result in partial causal graphs, which may affect accuracy of Poincaré positional encodings during the hyperbolic mapping.\n\nTo empirically validate these points, we conducted three experiments: for **i)**, we simulated four pairs of DAGs with the number of nodes increasing from 10 to 100 using the Erdos-Renyi algorithm. DAGs in the same pair differ in their sparsity. For **ii)**, we increased the standard deviation of Gaussian noise of simulated data from 1 to 3. We aslo simulated data with non-Gaussian noises (Gumble noise). For **iii)**, we randomly drop edges from the simulated DAG with a increasing rate from 10\\% to 40\\% edges. The first two experiments are evaluated using structural Hamming distance (SHD) and false discovery rate (FDR), while the third one is evaluated by checking the average changes in Euclidean distances between the two nodes with removed edges. The results are shown below.\n\n| Number of nodes | Low-sparsity SHD | Low-sparsity FDR | High-sparsity SHD | High-sparsity FDR |\n| -- | -- | -- | -- | -- |\n| 10 | 5.8 ($\\pm$1.3) | 0.22 ($\\pm$0.10) | 2.9 ($\\pm$1.1) | 0.09 ($\\pm$0.08) |\n| 30 | 17.9 ($\\pm$5.2) | 0.24 ($\\pm$0.05) | 14.5 ($\\pm$3.2) | 0.15 ($\\pm$0.10) |\n| 50 | 29.0 ($\\pm$6.5) | 0.21 ($\\pm$0.05) | 22.5 ($\\pm$4.2) | 0.17 ($\\pm$0.08) |\n| 100 | 60.0 ($\\pm$10.0) | 0.26 ($\\pm$0.04) | 45.0 ($\\pm$8.0) | 0.11 ($\\pm$0.06) |\n\n| Noise pattern | SHD | FDR |\n| -- | -- | -- |\n| Gaussian(0,1) | 5.8 ($\\pm$1.3) | 0.22 ($\\pm$0.10) |\n| Gaussian(0,2) | 5.6 ($\\pm$1.7) | 0.25 ($\\pm$0.12) |\n| Gaussian(0,3) | 6.0 ($\\pm$2.0) | 0.27 ($\\pm$0.14) |\n| Gumbel(0,1) | 6.3 ($\\pm$1.5) | 0.25 ($\\pm$0.10) |\n\n| Removal rate | Average changes |\n| -- | -- |\n| 10% | 0.1589 ($\\pm$0.0269) |\n| 20% | 0.1858 ($\\pm$0.0281) |\n| 40% | 0.3530 ($\\pm$0.0362) |\nPS: The Euclidean distance on the Poincaré ball ranges from 0 to 2.\n\nGenerally, the results align with our expectations explained above. Interestingly, in the third experiment, the difference in average Poincaré distances is relatively smaller when the edge dropout rate is below 20\\% and rapidly increases when the rate is 40\\%. This is probably due to that when the dropout rate is low, the causal relationships between two nodes of a removed edge can be maintained via other paths until those paths are eliminated by the increasing dropout rate. We also emphasize that, since CAPE's primary goal is proposing a methodology of creating causality-aware positional encodings from a learned causal structure, rather than proposing a novel causal discovery algorithm, we adopt a modular design for CAPE's causal discovery component, thus allowing for the future integration of more robust causal discovery algorithms.\n\nFinally, CAPE accounts for causal strength, which are explicitly modeled by its SEM and translated into Poincaré distances during the hyperbolic mapping."}, {"review_id": "umP635XrRf", "reviewer": "Reviewer_Jacd", "summary": "This paper proposes a novel method to embed the structure of a causal graph into a Transformer model. It firstly encodes causal graphs into hyperbolic space, translating causal strength and hierarchy into geometric distance and position, then converts them into a rotary form. Experimental results verify the effectiveness of the proposed method.", "questions": "1. To my understanding, causal structure learning usually requires some fundamental assumptions, e.g., causal faithfulness and causal sufficiency. It is unknown whether these assumptions remain true in the context of real-world datasets. Could the proposed approach remain effective when these assumptions are not satisfied, e.g., when latent confounders exist?\n2. A generalized nonlinear SEM model is used to infer the causal DAG. However, in many real-world scenarios, the DAG is non-identifiable from the data, so I think the inferred causal DAG should contain unavoidable errors under these scenarios. If this is correct, how does the proposed approach solve this problem? Moreover, if the proposed method can be effective under inaccurate causal DAG, could it be used in traditional sequential features?\n3. RoPE is not used as benchmark as they rely on predefined feature order. Is it possible to use RoPE or other order-based methods with the topological order of the causal DAG? I think the comparison could make the experimental results more comprehensive.\n4. As also mentioned in Sec. C.2, there are many casual structure learning methods. Why nonlinear SEM method is used in the causal structure learning step? Are there any other alternatives?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths:\n- The motivation of applying causality-induced positional encoding is clear and significant.\n- The method is clearly described and the paper is well-written. \n## Weakness:\n- This paper could benefit from more discussions on the correctness of the causal structure learning step. See questions for details.", "was_revised": true, "final_justification": "Thanks for the authors' detailed response. I have carefully read the authors' rebuttal as well as the comments from other reviewers. I believe the authors' response has clarified most of my questions. Overall, this is an excellent piece of work, and I have decided to raise the final score to 5 points (accept).", "rebuttal": "We sincerely thank you for the thoughtful and constructive feedback. \n***\n**Question 1**\n\nThank you for raising this fundamental and important point. We agree that our causal discovery step, like most SEM-based methods, assumes causal sufficiency and faithfulness. Therefore, in the presence of strong latent confounders or faithfulness violations, the learned DAG may not be the \"true\" underlying causal graph (please see our response to the next question for some tentative solutions). Since the primary contribution of our work is to introduce a method for creating causality-aware positional encodings from a learned causal structure, rather than proposing a novel, assumption-free causal discovery algorithm, we adopt the current methodology,  which extends NOTEARS that is known for easy optimization and balanced computational efficiency and performance. Our Ablation Studies (Supplementary G.3) provide strong empirical support for this approach. The results show that using this learned DAG, even if it is an approximation of the true causal structure, provides a far more effective basis for representation learning than assuming no structure or using a simple similarity graph. We agree that explicitly handling violations of these assumptions is a critical direction for future research. The modular design of CAPE is well-suited for this, as it would allow for the future integration of more advanced causal discovery algorithms that are robust to such violations. We will add this to our discussion.\n***\n**Question 2**\n\nThis is an important consideration and we agree that in case of unidentifiable causal effects, CAPE-inferred DAG will inevitably contain errors. Actually, how to accurately infer causal DAG while maintaining the computational efficiency remains a significant challenge currently, particularly when using real-word datasets. Here, we tentatively propose several possible solutions: \n\ni) **Improve CAPE's causal structure learning**. The nonlinear SEM can be modified to incorporate hidden confounding variables $h_i$ for each variable $x_i$ in addition to its parents $PA(x_i)$, leading to $x_i=f_i(PA(x_i),h_i)+z_i$. During training, the VAE encoder learns to approximate the posterior distribution of all the latent confounders ($q_\\theta(H|X)$) given the observed data. It takes the observed data $x_i$ as input and outputs the parameters (e.g., mean and variance) for the distributions of the latent variables $h_i$. The decoder reconstructs the original data $X$ with $\\tilde{H}$ sampled from the variational distribution. If the model can achieve a better objective score (a higher ELBO) by using a latent variable to explain the correlation, it will infer the presence of a confounding arc. Meanwhile, the SEM can be improved by assuming non-Gaussian noises with asymmetrical distribution that will help to correctly identify the direction of node links, given our current method might just return a sample DAG from a Markov equivalence class. \n\n ii) **Improve the ex post CAPE-derived DAG**. We can apply conditional independence tests (e.g., Fisher's Z-test) against the chain, fork, and collider structures within *ex post* CAPE-inferred DAG, thus reducing the incorrect node links. Additionally, suspicious graph structures in the DAG can be validated using various refutation tests. \n\niii) **Robustify the mapping of DAG to hyperbolic space**. Confidence scores can be assigned to node links within the CAPE-derived DAG based on the possibility of their association with hidden confoundings and the statistical significance of their independence and refutation tests. During the subsequent mapping of DAG to hyperbolic embeddings, variable pairs are assigned weights based on their confidence scores so that accurate causal structures in the DAG dominate over those less accurate. However, these solutions can come with substantial computational overheads, particularly when dealing with large DAG. Thus, they should be adopted only when we are certain about the presence of impactful hidden confoundings, and when the accuracy of inferred DAG and causal positional encodings are critical. Due to the limited rebuttal time, we will implement and test these solutions in the future and add the results to our manuscript. \n\nYour suggestion on applying CAPE to traditional sequential features is very interesting. Yet, we believe it may not be the most practical application of CAPE in its current form. This is because for traditional sequential features, their predefined order already provides a strong and computationally efficient causal prior—that an element is primarily caused by the elements that precede it. This is the principle behind successful causal autoregressive models like GPT. While applying CAPE may capture more complex, non-local causal relationships, the significant computational overhead of the causal discovery step would likely outweigh the marginal performance benefits. We believe the main strength of CAPE lies in domains where no such natural order exists. In our future work, we will try to migrate CAPE to sequential features, improving its computational efficiency and effectiveness by utilizing sequential order information.\n***\n**Question 3**\n\nThis is an excellent suggestion. We can use algorithms such as the Kahn's algorithm to perform a topological sorting on the inferred DAG to derive feature nodes' topological order, which can be used by RoPE. However, this method has several limitations. First, the topological order also predicates on the correctness of the inferred DAG, thus can be erroneous. Second, a topological sort flattens the DAG into a single, discrete sequence. This process cannot capture causal strength; two causally related nodes might be placed far apart in the sequence if there are many other nodes with stronger causal relation with them. In contrast, CAPE's continuous geometric embedding explicitly models these strengths via hyperbolic distances. Third, topological sorting algorithms can have major limitations. For example, in Kahn's algorithm, if multiple nodes have no incoming edges, their relative order is arbitrary, leading to ambiguous topological order. As per your suggestion, we ran a new ablation experiment using a topological sort of the inferred DAG to generate positional encodings. The results are shown below:\n\n| Models | Single-gene perturbation | Double-gene perturbations |\n| -- | -- | -- |\n| Original CAPE | 0.182 ($\\pm$0.005) | 0.176 ($\\pm$0.008) |\n| Topological order | 0.201 ($\\pm$0.012) | 0.210 ($\\pm$0.015) |\n| Similarity | 0.209 ($\\pm$0.010) | 0.213 ($\\pm$0.011) |\n| Non-positional | 0.234 ($\\pm$0.014) | 0.238 ($\\pm$0.017) |\n\nAs expected, although topological order performs better than non-positional encodings and similarity graph, CAPE significantly outperforms it. \n***\n**Question 4**\n\nWe thank the reviewer for this important question. The choice of a nonlinear SEM was a deliberate decision based on the specific goals of our work and the practical trade-offs among different classes of causal discovery algorithms. While alternatives exist that can relax certain assumptions, they often come with significant limitations. For example, many constraint-based methods are more robust to hidden confounders but are typically computationally intractable and rely on discrete statistical tests, making them difficult to integrate into a gradient-based deep learning framework. Other approaches like functional causal models may be limited by strong assumptions about the data-generating process, such as linearity. Our primary goal is not to solve the general problem of causal discovery but to learn a plausible DAG to serve as an effective inductive bias for the Transformer. For this purpose, our approach was chosen for three key reasons: First, it is based on gradient-based continuous optimization, which not only allows for easy implementation but also natural integration with neural network to model complex nonlinear causal relationships. Second, our method is an end-to-end causal framework that simultaneously retrieves causal structure and causal strength. Finally, our method extends NOTEARS, a well-received causal discovery method known for well-balanced computational efficiency and performance. We will add these discussions to the Related Work Section."}]

Author rebuttal:

Meta-review: Capturing causation in transformers is an especially important direction, and using causation in positional encoding is an innovative approach.  The paper is technically sound, the reviewers and authors had a great discussion, and the reviewers are unanimous in recommending acceptance.
Decision: Accept (poster)