A Pre-training Framework for Relational Data with Information-theoretic Principles
Authors: Quang Truong
Zhikai Chen
Mingxuan Ju
Tong Zhao
Neil Shah
Jiliang Tang
Venue: NeurIPS 2025 poster ()
Keywords: graph learning
tabular learning
relational database
Primary area: deep_learning
TL;DR: A predictive task-aware pre-training framework for Relational Deep Learning.

Abstract: Relational databases underpin critical infrastructure across a wide range of domains, yet the design of generalizable pre-training strategies for learning from relational databases remains an open challenge due to task heterogeneity. Specifically, there exist many possible downstream tasks, as tasks are defined based on relational schema graphs, temporal dependencies, and SQL-defined label logics. An effective pre-training framework is desired to take these factors into account in order to obtain task-aware representations. By incorporating knowledge of the underlying distribution that drives label generation, downstream tasks can benefit from relevant side-channel information. To bridge this gap, we introduce Task Vector Estimation (TVE), a novel pre-training framework that constructs predictive supervisory signals via set-based aggregation over schema traversal graphs, explicitly modeling next-window relational dynamics. We formalize our approach through an information-theoretic lens, demonstrating that task-informed representations retain more relevant signals than those obtained without task priors. Extensive experiments on the RelBench benchmark show that TVE consistently outperforms traditional pre-training baselines. Our findings advocate for pre-training objectives that encode task heterogeneity and temporal structure as design principles for predictive modeling on relational databases. Our code is publicly available at https://github.com/quang-truong/task-vector-estimation.

A Pre-training Framework for Relational Data with Information-theoreti Principles
Quang Truong', Zhikai Chen' Mingxuan Ju² Tong Zhao? Neil Shah Jiliang Tang'
'Michigan State University, SSnap Inc. (truongc4, chenzh85, tangjili)@mse {mju, tong, lshahp-@nap. com
Abstract
Relational datahases underpin critical infrastructure across a wide range of domains, yet the design of generalizable pre-training strategies for learning from relational databases remains an open challenge due to task heterogeneity. Specifically, there exist many possible downstream tasks, as tasks are defined based on relational schema graphs, temporal dependencies, and SQL-defined label logics. An effective pre-training framework is desired to take these factors into account in order obtain task-aware representations. By incorporating knowledge of the underly- ing distribution that drives label generation, downstream tasks can benefit from relevant ade-chaanell information. To bridge this gap, we introduce Task Vector Estimation (TVE), a novel pre-training framework that constructs predictive super- visory signals via set- aggregation over schema traversal graphs, explicitly modeling next-window relational dynamics. We formalize our approach through inforrattoonthhoretiic lens, demonstrating that task-informed representations retain more relevant signals than those obtained without task priors. Extensive experiments on the RelBench benchmark show that TVE consistently outperforms traditional pre- pre-traimen vee-traiing baselines. Our findings advocate for pre- training objec- tives that encode task heterogeneity and temporal structure as design principles for predictive modeling on relational databases. Our code is publicly available at https: ftttps://githun ght..b.t........ com/quang-
ht////uuunuuungggeeeeee truutguuunnneo taauong/tsk--uee www.taaaaatteettttiieiiieeeeeeee estimation
Introduction
Relational databases (RDBs) have long served as the de facto standard for managing data storage across diverse industries 4 28 and learning tasks over RDBs depend on domain expertise for manually engincering meaningful features [24] 34]. While manual feature engineering can yield effective models, it demands significant human time and expertise, and it inherently introduces bias. susceptibility to human errors, and the risk of overlooking valuable predictive signals. Relational Deep Learning (RDL) 21 24 has emerged as powerful paradigm that exploits relational structures- defined by primary-foreige key (PK-FK) links across tables. By treating RDBs as graphs of tables, we can leverage existing state-of-the-art tabular feature encoders I17 27 and Graph Neural Networks extract meaningful embeddings in an end-to-end manner. This integrated approach has already demonstrated significant improvements in efficiency over conventional manual feature engineering techniques due to the end-to-end learning pipeline 24 46).
(GNNs) 126] 29 35
Designing pre-training objectives for RDL is particularly challenging. single RDB typically supports multiple tasks, each definable by distinct SQL queries. Mcanwhile, due to potential differences in label semantics, the pre-training stage is relevant to downstream tasks, which leads to unreliable pre-trained models. Moreover, downstream tasks depend on temporal dynamics (e.g. nendow properties), which cannot be simply modeled by just pre- training on unlabeled input data alone. A pre-training framework desired to model the dynamics in the latent representation intrinsically without having to rely 39th Conference on Neural Information Processing Systems (NcurIPS 2025).
hard to guarantee that the latent representation obtained during

---

Set Next- Next-window
Topuc up
Inpul up lo cutoff Ang Aug. Ioput
RDL Buckbase
+Decndar+
Lutent Repe.
Vector Coostruction
Taok Vector
Aug.
to outoff
up lu
Values
II
Aug. Input
to
RDL Bockbone
Doodler
Rec. Input
to cutoff
to cutoll
Buckhhme
+Decnder
Aug. Lateut
Repr. up outoll
Iuput up to cutoff
IDD Bacckoons
DDoodar
Task Vector
Estimated
cutoff
(a) Masked Autoencoder
(b) Contrastive Learning
(c) Task Vector Estimation
Figure I: Overview of pre-training frameworks. Input graph is omitted for brevity.
on downstream fine-tuning for predictive modeling. In addition, what makes RDL distinct is the dependency of tasks relational schema graphs, which leads to growing complexity complicated RDBs. Recently, Liu et al. [40] showed that adapting traditional SSL methods [8] 30 to RDBs can inadvertently capture spurious yet seemingly informative cesterns-so-callle
pnreerestingloooiiin noises"- that undermine downstream performance. This contradicts the core objective of traditional SSL, which aims retain task-relevany information while discarding irrelevant noise [2 [2 37), based on the assumption that different augmented views still share the same labels. Accordingly, our goal is to design a pre-training objective for RDL that avoids these pitfalls.
Main Contributions Our main contributions are summarized as follows:
1. Task Vector Estimation (TVE): pre-training framework for RDL that leverages schema graphs, next-window dynamics, and task heterogeneity to produce temporal, task-aware representations. Theoretical insights: we prove that pre-training on full relational databases with our objectives preserves more downstream-relevvnn information than standard self-supervised methods.
3. Empirical validation: across diverse downstream lasks-especially, in low-data regimes- TVE outperforms traditional SSL methods; ablation studies further uncover "flaky" task-dependent behaviors, underscoring the need to model both temporal dynamics and task heterogeneity.
2 Preliminaries
Relational database [24] RDB (T, formally defined as a set of tables (or entity types) T ...) that are connected by a set of PK-FK relations (or edge types) L C T x T 24]. Each table T; can contain different data types for each column: numerical, categorical, text embeddings, etc. Additionally, each table contains key column, which is either for primary keys or foreign keys, serving as links between tables. Tables are either fact or dimension tables [4]. Fact tables contain interaction between entities (e.g. Treview) with corresponding timestamps, while dimension tables contain static, immutable properties of entities (e.g. Teustomer or Tpwdduct. Additionally, refer root table to the table would like to perform predictive modeling.
Graph perspectives of RDBs. RDB can be understood from two different perspectives, namely schema graph and relational entity graph. Schema graph (T, R) is blueprint of the RDB, in which the relation bi-directional R -CUL- 24 Relational entity graph is the input graph for learning, and is fine-grate tne-grrined view of the schema graph: we treat rows in table nodes, and they are connected to rows in other tables via the PK-FK links [24]. We use "nodes" and "rows" interchangeably in this work. We are interested in the relational entity graph snapshot up cutoff timestamp t:
g(->o.1l (V(-=o.f), g( pv, Pe, fv.
where (o,0) and (foo are the sets of nodes and edges observed up to timestamp f, sy T maps each node to an entity type, ig E R maps each edge to PK-FK link type, and fv V Rd and fe + I map nodes and edges to their feature vectors, respectively.
Different tables have varying attribute sets, so node feature representations are inherently heteroge- neous. Although edges could carry features, we use exclusively node features simplify discourse.
Predictive tasks over RDBs. RDL spans range tasks, from table cell prediction [5I to next- window property forecasting (46). In this study, we concentrate predictive modeling over relational

---

data: given a training table Tuain, the objective is to forecast future node-level properties as formalized by RelBench [46). Each training node is associated with a labeled entity (Ku, tus yo b A where K2 is a foreign key linking the entity to its originating table row, by is the cutoff timestamp, (A. Att)) is the wixdww label. The learning objective is to estimate:
and yo
g - = (( ,t,),
where fe is a differentiable function parameterized by that uses the node's tabular features, the graph structure, and temporal context up to by to predict the future label.
Next, would like to Fig. 2 schema graph to demonstrate the process of forecasting customers' review counts in the seven days following time as example. First, we extract G(->c.0] for learning. Second, we label each node by issuing an SQL query that joins the root table (e.g. T Teustomer) with with fact table (c.g. Teeview) and aggregates event counts over the interval (t, t + 7 days]. The node ID, the cutoff timestamp, and the corresponding label are recorded in the training table Turin This step is necessary because the root table is dimension table, thus relying neighboring fact tables to provide temporal contexts. Third, follow [24 46). we train a model to predict these labels. Tuain is used only for lookup and loss computation, not for message passing.
RDL is inherently distinct from both static heterogeneous and temporal graph problems. Static heterogeneous graph methods [52 60) omit temporal dynamics, while temporal graph approaches represent data as discrete-time snapshots or continuous-time event streams to capture graph evolution. By contrast, RDI. treats timestamps as tabular features and uses them to impose causal constraints when extracting subgraphs. Lastly, RDL generates labels via SQL-driven schema- graph joins, leveraging fact tables, aggregates, and multi-hop joins absent from standard graph problems.
16 388
3 An Overview of the Proposed Pre-training Framework
We would like introduce an overview of our pre-training framework, namely Task Vector Estimation (TVE), as shown in Fig. Ic Besides the RDL backbone to extract features and the decoder to map the extracted latent representation to the final task vector, our framework contains an external module called Task Vector Construction to construct an objective vector extracted from the set of next-window values. By depending on temporal dynamics, schema graph, and simple SQL logics [19, the vector represents a proxy representation for the set of next-window -wichh plays an important roles in label generation process (see Section 4.1). Details are discussed in Section 4 where we outline the motivation, implementation, and theoretical justifications.
Most popular SSL frameworks, namely Masked Autoencoder (MAE) 30 and Contrastive Learning (CL) [ are shown in Fig. Ia and Fig. [bb] respectively. MAE's objective is to reconstruct original feature values up to timestamp cutoff given augmentations, while CL attempts to minimize the distance between latent representations of two views of the same features (e.g. original features vs. corrupted features). Notably, the roles of decoders are different for MAE and CL. The decoder in MAE maps the extracted latent representation back to the original input space for reconstruction, whereas the decoder (often called the projection head) in CL. maps the latent representation to a different, usually smaller space for contrastive learning. Both these methods depend on separation of data from inputs without modeling the next-window dynamics, which are critical for predictive modeling. Meanwhile, what sets TVE apart the incorporation of an objective conditioned on next-window dynamics. TVE estimates task-relevant information during pre- pre-training. and can be complementary to both MAE and CL (see Eq. for the combined objective and Table 2 for experimental results). Our objective correlates input features and next-window dynamics, thus yielding task-aware latent representation.
Predictive Task-aware Self-supervised Learning
In this section, we would like to shed light on the downstream label generation process, then propose our SSL. method, namely Task Vector Estimation (TVE), based on the label generation insights. Finally, we establish the benefits of this SSL objective over other methods from an information- thcoretic perspective.
3

---

8 Sversed xcfuuttdd)
product
cudtsDez
custorar_.10
(rerion I (11
TWVING
raviez
(rerien 2xx fletet d
custonor
product 1d product,
(predees 2x! Mude 150)
CUSTADAT
CUSTOROE
zoviow
zevieи
product
Gouple
Grupla
Figure 2: Construction of task vectors from relational schema. From schema graph, we derive schema traversal graph, where edges represent valid join keys. For each reachable table k-hop
from the source node we apply set function S to the set of next-window values 2 x (f reachable via the traversal path, where only paths contributing to the prediction task are retained. The task vector concatenates outputs from these set functions. A null indicator (purple cell) is activated when all other entries are absent (i.e., zero-valued).
4.1 Label Generation Process Set Function
Given the nature labels defined the training table introduced in Section 2 observations regarding label generation process, First, the training process is seiity-centrc-whhee we aim to perform predictive modeling for entities in the root table. Second, downstream tasks span across multiple tables in the RDB, and are conditioned Third, because the root table is often a dimension table, assigning a cutoff timestamp t, requires joining it with fact tables. For example, tasks related to the customer table are timestamped via interactions with the review table, which is one hop away in the schema graph as shown in Fig. 2
we make the following
set of nodes connected to the root table.
The set of next-window values depends not only on temporality, but also the schema graph structure and SQL logics. For example, predicting the number of reviews customer will submit in the next window requires one-hop join between the customer and review tables, with the COUNT aggregation. Likewise, computing the average price products reviewed by a customer in that same window involves two-hop join spanning customer, review, and product, coupled with the AVERAGE aggregation. For the sake simplicity, assume we only consider the k-hop neighbors with the same entity-type, given entity and the timestamp t, its label y is the output of a set function I defined over the features of its set of k-hopp neighbors (+++1 in the next timestamp t
v = I(x(++)) x((+1) = (fv(4) A(t+))
The formal definition of Eq. where we additionally differentiate sets of neighbors conditioned based on their entity type R, is included in Appendix B Importantly, the traversal graph permits self- self-joins, e.g., joining entries in revies vew by both customer_it and product_ia idd When the labeling function I is an identity function and sets contain single element, Eq. reduces to standard cell prediction, as often studied in Tabular Deep Learning
55
4.2 Task Vector Estimation
As demonstrated Section 4.1 there are three key factors governing the label generation pro- cess: schema graphs, temporal dependencies, and SQL logics. While these three factors can complicate possible tasks, all tasks depend on
+ T(Y) - ((Y)
pre-training downstream task
Figure 3: Markov Chain of Labels.
(to.ff.u which are represented by all pos- sible rows, possibly with different entity types R, that are k-hop from the root tables and associated with the wendww period (t,,, t. + At).
TVE's goal is to model task heterogeneity via estimating statistics representing * x (x(oo.tt Gen- Abu erally, let Y represent next-window set of values (c.g. x ( Our goal is to capture the

---

distributional properties of Y. parameterized by @ as illustrated in Fig. 3 Here, T(Y) denotes sufficient statistic of Y for θ. If the latent representation fully captures YY) during pre pre-training, then the latent representation has all information regarding Y for + [20] Finally, the downstream la- bels !(Y) depend on Y as illustrated in Fig- 3 and formally defined Section Thus, pre-training model to estimate T(Y) yields task-aware knowledge for downstream tasks.
TVE consists of steps:
Generate pre-training table by performing a CROSS JOIN between entity rows and candidate timestamps, then filtering out timestamps prior the entity's first fact entry (e.g.. first review).
2. Attach pretext labels by computing normalized set-level statistics 5 (mean, count, etc.) per column across all reachable (f ( using aggreggagn
Regarding the first step, the intuition is that non-events entries are also important for downstream tasks such as churn prediction. The second step is illustrated in Fig. 2 Specifically, given relational schema graph, we construct a schema traversal graph based on joinability between two tables via PK or FK, in which edges denote which key is used for joining. Paths in this graph possess semantic meanings. Given customer, we can find the set of next-window reviews made by the customer (reviee X ((
), the set of next-window products S S(produce
(11 ), the set of 2x!"
reviews for all products made by the customer ((reviev_ 2 2x. ) and the set of reviews made by other customers for next-window products that the customer reviewed ((zeview_2 2 (f Importantly, the extracted set must be causal, which means that the joined entries in the next hop must have the timestamp less than or equal to that of the entries in the previous hop. Then, we aggregate values in sets by simple SQL aggregators to get predictive pretext tasks. Note that adopt the simple statistic S(Y) for computational efficiency, while approximating T(Y) is challenging. For further discussion regarding the sufficiency of simple statisties and future works designing better approximations of sufficient statistics, please kindly refer to Appendix D Our proposed task vector construction operates similarly to Breadth First Search, and tasks extracted from a schema path is concatenated to the final task vector, leading to a variable-length task vector depending on the traversal paths. The task vector summarizes statistics for every column of tables reachable within k-hops of the root table in the schema traversal graph, estimating this vector ensures that column is prematurely excluded during pre-training. It is important design choice because downstream tasks may depend on any columns of any table. Optionally, the vector can be compressed via dimensionality reduction techniques such PCA [8] or MCA (1. The pre-training loss the Scaled Cosine Error 31:
11
U yo l@oll 11
CSCE
a a21,
(4)
[Trainl
of Tuai
where Ye is the ground-truth task vector, f. is the estimation of yv. and is scaling factor. Additionally, to account for the potential imbalance due null task vectors outnumbering others we reweight each group's 's contribution to the loss inversely proportional to its relative frequency.
Optionally, mitigate the potential loss of input-specific information during pre-training, we extend our objective beyond the TVE loss (Eq. by combining a traditional self-supervised loss:
C LTVE + 3 Lssi.-
(5)
where is a balancing coefficient, and LSSL denotes standard self- supervseee loss either reconstruction objective contrastive loss. This formulation encourages the model to retain both predictive information (captured via task vectors) and general input features.
4.3 Theoretical Evidence of Task Vector Estimation
In this section, we would like to characterize relationship between the latent representation of pre-trained model and downstream tasks, and then we justify our design choice which can be proved to contain more task-relevant information than vanilla SSL methods.
5

---

Task heterogeneity introduces unique challenge: how to design predictive aware SSL method for pre-training so that the model can generalize across a combinatorially large set of potential downstream tasks? Each task is conditioned on different column subsets of k these tasks may may not be correlated. This raises a natural question: how much task-relevant information must a latent representation encode to fully capture the entire set of tasks?
Rt x (vv.fttA and
Remark 4.1. Suppose we have set of tasks whose labels are denoted by Y1.Y2," .n. The lower bound of latent coding length L is:
15 Σ
L L2 H(y1. .. .nn)
H(vi)
TC(y1.y2.". 1...2....30. .Yn).
(6)
where Hy:: is the entropy y; y: and TC(y1. .nn is total correlation of all tasks. Addition- ally, 1,,,1,y2 .n) 2 effectively becomes zero once all tasks are independent.
The lower-bound of code length is covered 220,, and the relationship between joint entropy and total correlation is studied in [54]. Remark 4. underscores important property of any generalizable latent representation: the more uncorrelated tasks to support, the longer the required latent code.
Some tasks may degrade model performance. For example, if spending behavior is independent of gender, then training on gender-conditionel spending prediction tasks introduces noise. By the Data Processing Inequality, information can only decrease or remain the same through processing:
Remark 4.2. Given the Markov chain - >x= - x' where y is label, x is original data, and x' is the filtered data, then xy; T(y:'') by Data Processing Inequality.
Next, we provide an information-theonm characterize how guiding models toward predictive task-aware objective can benefit downstream tasks compared with vanilla contrastive separation or mask reconstruction from input data only.
justification for TVE. Specifically, we would like to
Let denote input features, t be random variable representing iide-channel information (which includes task vector proposed Section 4.2). and y denote downstream labels. Let z₁ be latent representation learned from both x and t, and be learned from x alone.
Prior literature 22 defines sufficiency of representation Z2 of x for y. in which 22 must be as predictive as x to predict y. Similarly, we can define a condition for a sufficient representation of inputs and side channels for a downstream task.
Definition 4.1 (Sufficient representation given inputs and side-channel information). A representation 21 sufficient for predicting y given inpul features x with side-channel information if and only if T 21 0.
We can prove that representation of with additional knowledge of shares as least as much information with the downstream task than that of x without t does (see proof in Appendix [A.I).
Theorem 4.1. Let x, L, be random variables with joint distribution p(x, t, y, Assume that zı is sufficient representation of x with side-channel information ffr y, and 22 is sufficient representation of x for y. Then mutual information between 21 and y is at least as much as that between Z2 and y:
I [Zzıy) 2 I I(zzyy).
The theorem justifies the construction of the task vector, introduced in Section 4.2 In other words, whenever side-channel signals actually carry extra task- relk-nelevant cues, using them in pre-training can equip the model with task-aware representations. The task vector is task-relevant, as it serves as the proxy representation of set of values that are used during label generation process (as demonstrated in Eq. B.. thus improving fine-tuning. In contrast, standard SSL learns moise-invarian embeddings and only later fits a direct input-label mapping the fine-tuning stage, without ever modeling the underlying set distribution in the pre-training stage.
5 Experiments
In this section, we conduct comprehensive experiments to answer the following research questions:
RQI: How does proposed task vector improve model performance downstream tasks?

---

Table Performance across datasets and model variants in the low-data regime. Reported metrics in are ROC-AUC for classification tasks, and Mean Absolute Error for regression tasks. Split, Validation, and Test are abbreviated as S. V, and T for readability. Bold is the best performance.
Model
Task
S
Baseline
MAE-0.25
MAE-0.5
CTR-0.25
CTR-0.5
TVE-1
TVE-2
Classification
least V 76.04 3.84
76.07 = 4.38
76.06 = 3.19
84.82 2.12
84.32 = 2.41
86,56 = 1,31
87.11 = 2.06
rel-amz/
50 T 73.39 + 6.63
71,33 + 5.96
67.02 # 6.89
83,97 + 2.72
79.89 + 4.41
81.97 + 0.95
83.25 + 2.40
item-chum
least V 76.64 # 1.11
75,31 + 2.05
74.03 + 2.47
80,99 * 1.27
80.09 # 1.68
80.94 # 1.35
82.79 # 1.20
100
T 68.65 + 1,91
68.48 + 2.58
70.12 1.58
74.78 + 1,56
73.86 + 1.68
75.29 +: 2.27
76.74 + 1.46
top V 91.35 # 1.27
91.53 + 1.47
91.66 + 1.13
91.01 * 1.82
91.09 # 1.10
93.31 # 1.19
91.82 + 1.25
rel-amnz/
50
89.07 2.14
87.86 + 2.00
85.71 + 3.37
89.66 + 2.26
89.71 + 1.90
93.21 + 1.58
91.53 # 1.84
uscr-chumm
top
V 88.99 + 0.61
90.00 # 0.60
89.24 + 0.68
91.32 + 0.58
90.52 # 0.86
91.35 + 0.48
90.92 + 0.81
100
86.23 + 0.64
86.69 # 1.00
85.59 + 1.66
84.86 1.39
86.32 + 1.41
87.74 +: 1.04
88.13 + 1.14
top
V 74.41 + 0.85
76.30 1.48
76.47 + 1.20
84.07 + 2.07
84.14 + 1.74
76.47 + 2.43
74,97 # 1.42 64.12 + 3.13
rel-hm/
50
60.23 a 1.84
56.94 + 3.59
58.11 +: 2.38
56.82
3.05
57.48 + 3.55
65.01 3.82
user-churm
top
V 72.63 # 1.30
74.61 # 1.48
76.22 + 1.24
79.74
79.85 + 1.41
75.42 + 1.53
75.60 + 1.40
100
63.43 # 1.14
59.26
60.25 = 2.20
61.58 # 1.15
60.69 = 1.50
62.85 = 1.87
64.06 + 2.29
Regression
rel-amz/
0.172 + 0.003 0.175 # 0.001 0.174 a 0.002 0.1 # 0.005 0.165 = 0.006 least V 0.175 = 0.000 0
item-ltv
100 T 0.186 + 0.000 0.184 + 0.003 0.185 0.002 0.187 = 0.000 0.188 0.003 0.181 + 0.005 0.188 + 0.006
rel-amz/
bad-
V 8.172 = 0.000 8.173 # 0.000
8.172 + 0.000
8.172 = 0.000 8.172 0.000
8.117 = 0.008 8,115 = 0.010
user-Itv
rev.
9.520 = 0.000 9.520 + 0.000 9.520 0.000 9.520 + 0.000 9.520 #: 0.000 9.372 + 0.026 9.373 + 0.010
rel-hm/
top V 3,203 = 0.043 3.139 # 0.083 3.111 = 0.062 3.170 = 0.090 3.113 # 0.084 2.974 # 0.098 2.945 = 0.028
item-sales
200
3.240 = 0.049 3.271 # 0.1 3.249 0.039 3.277 = 0.135 3.284 + 0.1 3.176 + 0.078 3.205 = 0.081
Table 2: Performance across datasets and model variants in the sufficient-data regime 46 Reported metries are ROC-AUC for classification tasks, and Mean Absolute Error for regression tasks. Split, Validation, and Test are abbreviated as S, and T for readability. Bold is the best performance.
Model
Task
Baseline
MAE-0.25
MAB+TVE-1 MAE+TVE-2
CTR-0.25
CTR+TVE-1
CTR+TVE-2
Classification
rel-amz//
V 81.54 0.06
81.61 + 0.14
81.13
0.03
81.27 = 0.08
81.29 0.09
81.28
item-chur
T
81.85 1 0.10
81.92 0.16
81.47 = 0.04
81.49
81.53 = 0.09
81.57 : 0.13
81.56 + 0.12
rel-amz/ V 69.95 0.09
69.26 0.22
69.92 0.04
69,93 0.05
69.85 = 0.05
70,04 : 0.04
70.03 + 0.06
user-chumn
T
69.82 2 0.05
68.89 0.29
69.78 = 0.07
69.80
69.64 0.09
69.95 + 0.02
69.85 0.07
rel-hm/ V 70.34 1 0.09
70.39 = 0.06
70.29 = 0.09
70.35 = 0,10
70.49 a 0.13
70.53 =: 0.04
70.54 2.16
user-chum
T 69.77 a 0.33
69.81 + 0.20
69.89 = 0.09
69.80 0.11
69.97 + 0.09
70.13 # 0.22
69.72 a: 0.51
Regression
rel-amz/
45.818 + 0.301
45.573 + 0.091 45.590 = 0.121 45.573 0.045
45.630 + 0.161 45.030 + 0.100
44.956 0.078
item-lt
50.603 0.351
50.471 + 0.308 50.437
0.395 50.659 0.417 50.472 + 0.432 50.012 + 0.287 49.753 0.192
rel-amz/
12.416 0.016
15.989 # 1.154 12.429 = 0.016
12.437=0.00
12.434
0.014
12,397 = 0.010
12.371 0.018
user-Ity
T 14.665 0.039
17.397 : 0.907 14.658 0.028
14.626
0.019
14.687 = 0.057
14.622
0.046
14.577 0.020
rel-hmm
0,0643 = 2c-4
0.0630 lc44 0.0627 = lc-4
0.0624 3c-4
0.0635 2c-4
0.0628
4e-4
0.0627 = 3c-4
item-sales
T 0.0556
0.0547 le-4 0.0544 = 2c-4
0.0543 + 3e-4 0.0554 + 3e-4 0.0547 # 3e-4
0.0543 = 3e-4
RQ2: Can TVE provide better latent representations compared to traditional SSL methods for downstream tasks?
RQ3: How sensitive is our model to different hyperparameter settings and last linear head's initialization?
5.1
Experimental Settings
We compare TVE with other SSL methods that do not take into account next-window dynamics Graph Masked Autoencoder and Graph Contrastive Learning. Specifically, these architectures resemble GraphMAF [31] and GraphCL. (56), with minor modifications for RDL pre-training. All evaluated models used the same backbone proposed by [46]. We compare several models: a non- pretrained Baseline; Graph Masked Autoencoder with mask rates of 0.25 and 0.5 (MAE-0.25.
7

---

Baseline
MAE-0.25
MVE+
CTR-0.25
CTR+TVE-11
TVE-1
amazooviteee
amazon/ttem
ht-hmuussechhur
5.7 x 10
5.6 10
roc auc
0.81
5.4 10
5.5 10
0.69
0.69
0.80
5.3 10
0.68
5.2% 10
0.79
0.68
0.67
5x
15
16
16
16
Epoch
Epoch
Epoch
Epoch
Figure 4: Test performance curves averaged over runs on data sufficient tasks.
Baseline
MAE-0.25
CTR-0.25
TVE
TVE 2
rel-amazon/teem-chum -least00-spendiing
ti-amazzon/tem-tvv
rel-hm/item-sales =ogg20-spenddng
rel-hm/user-chum
-east100-spending
top100-spending
2.15 x 10
6 10
0.65
2.1
roc. jauu
0.7
2.05 10-
5 10
roc aau
2 1
mae
0.6
1.9 10
10
x 10"
0.55
0.50
Epoch
Epoch
Epoch
Epoch
Figure 5: Linear probing results averaged over runs on test set.
Graph Contrastive Learning with the same mask rates (CTR-0.25, CTR-0.5); and our proposed method, TVE, using I-hop and 2-hop schema traversals (TVE-I, TVE-2). All pre-training is conducted using the same training table constructed via CROSS PRODUCT described in Section 4.2 While there are many pre-training techmiques designed for vertical domains, their translations to RDL are non- non-trivial due to the fundamental differences outlined Section 2 therefore, we only consider pre-training techniques based on tabular feature perturbations (see Appendix [ details).
MAE-0.5);
We evaluate proposed method rel aol-amazon and rel-hm. Additionally, we introduce a suite new tasks designed to reflect realistic scenarios. For instance, business might wish to build predictive models for specific cohort (e.g. low-spending customers) where label data is scarce. These scenarios often lack rich temporal context, so dynamics presented by labels alone may support high-quality forecasting. Yet these tasks are crucial, since companies increasingly aim to personalize offerings for these groups to boost retention. To generate these tasks, we apply attribute-based filters the prior period: for instance, an SQL query retrieves the top-k spenders over the last three months, and other customers are excluded from the training data for the current period.
variety of node-level tasks defined
datasets:
To address this, we explicitly evaluate model performance under two regimes: (i) data limited tasks and (ii) data sufficient tasks (the original RelBench tasks). More details are outlined in Appendix C
5.2 Data Limited Tasks
Table I presents results for tasks with limited labeled supervision. We find that TVE consistently outperforms other SSL baselines, achieving the best performance in out of 6 classification tasks. The only exception is rel rel-amazon/ mamaazoniitee teiiemmchiii wwwe-ccuunnnat---pppeeeeee
where CTR-0.25 achieves the highest test ROC-AUC. However, this task illustrates a form of flaky behavior: while CTR-0.25 and CTR-0.5 exhibit nearly identical validation scores (84.82 2.12 84.32 + 2.41), their performance diverges significantly on test set (83.97 2.72 vs. 79.89 # 4.41). This suggests sensitivity to hyperparameter tuning (e.g., augmentation strength), which are later studied in Section 5.5
More broadly, MAE and CTR often excel on validation but falter on the test set sometimms below the Baseline- Baeeline-likely due to distribution shift and recency bias from temporal splits. Our method explicitly captures next-window dynamics during pre-training, allowing it to generalize across time. Similar patterns hold in regression tasks: for instance, in rel sel--mazon//user hase--lv-baddreeee CTR match Baseline performance, while TVE allows the model to achieve better performance. These results answer RQL, which demonstrates the effectiveness of TVE.
MAE and

---

1r-0.002
1=-00001
Tr-0.005
1r-0.00075
Beseline
MAE-0.25
CTR:0.25
0.53
0.06
11.35
0.55
Centroid:
Centroid:
Centroks:
Centrold
10050 spending
0:50
0.50
K:
900
0.90
K=0.099
0.90
0.85
0.81
0.85
0.75
0.80
0.85
0.90
0.55
0.75
0.60
0.55
0.50
0.55
0.75
0.80
0.25
0.50
0.55
0.75
0.60
0.85
0.00
0.95
Validation
Validatton
Validaatio
Validation
Figure 6: Validation vs. Test scatter plots for data limited node classification tasks across varying
learning rates. Average distance centroid (denoted (k) used to measure performance variance.
5.3 Data Sufficient Tasks
In Table 2 we benchmark each model on the original RelBench tasks, which involve substantially larger downstream datasets. In data-rich settings, all SSL methods-including vanilla TVE-produce only marginal gains or even underperform, since plentiful fine-tuning data alone can recover temporal patterns. This matches Theorem 4.1 TVE's extra pre-training benefit arises only if the representation remains jointly sufficient for both inputs and task vectors, so any drop signals lost input information. As mentioned in Section TVE is complementary to existing SSL methods. Therefore, the combined- loss model from Eq. []] which penalizes the omission original information. As Fig. demonstrates, this hybrid objective produces a stronger initialization, and Table 2 confirms that outperforms single-objective models on five of the six tasks.
employ
4
5.4 Linear Probing Performance
To answer RQ2, we evaluate the quality the learned representations independently of downstream fine-tuning by linear probing experiments [62]. Specifically, we freeze the pre-trained backbone and train only linear head that maps latent representations to task outputs. All model shared the same backbone with the channel size of 128.
As shown in Fig. 5 representations learned via TVE outperform all other SSL methods, as well as the Baseline. Furthermore, CTR-0.25 achieves strong performance on three of four tasks, yet falls below the Baseline on rel-amazon/ teiittemccee churn- teast100- -penddddn illustrating the presence of "interesting-lookee noise" [40]. In contrast, TVE does not rely augmentations, thus yielding stable and transferable embeddings.
5.5 Performance Sensitivity Across Splits and Tasks
In order answer RQ3, further investigate the robustness of each method hyperparameter variation by analyzing the stability of performance across learning rates and last linear head's 's random initialization. We vary the learning in Fig. All models share the same backbone with the channel size of 128.
across 10 runs per task, yielding 40 data points per scatterplot
The results show that MAE-0.25 and CTR-0.25 are highly sensitive to hyperparameters and last linear head's s random initialization: the spread of points is substantially larger compared to TVE. These results suggest that the task-aware, temporal pre-training of TVE leads to more stable and transferable representations in the low-data setting. Experiments for other tasks are provided in Appendix C
Related Works, Discussion, and Conclusion
Relational Deep Learning. RDL has emerged as paradigm for learning from RDBs 10
21
24
25
46
5T. Although SSL techniques are advanced for individual Domains-temppore temporal graphs
15
16
38
39
48 static graphs [1 [50 [6], and tabular data ( 49 55 SSL investigations on
RDBs are still limited 44). Drawing on 43 k-means column-subset pretext tasks, TVE instead generates objectives via schema-graph traversals and temporal SQL aggregations. Unlike SQL-join
feature synthesis 34 or recent RDL advances on architectures [57] 61 and efficiency (36]. our
approach centers task-space representations for SSL pre-training.

---

Task Modeling, Generalizing to new tasks remains a core challenge. Prior approaches include task taxonomies to reveal inter-task structure [59] task embeddings for selecting feature extractors [4] and modeling Sata-auxiliary task node interactions [3] In contrast, we analyze label-generation in relational deep learning and propose a pretext objective that explicitly preserves the next-window set. representations from which any downstream labels are derived.
Representation Learning. Recent SSL-innormation-theoreete work frames representations as noise- invariant 2 and studies how multi-view SSL discards redundant features for downstream tasks [22]. Augmentations can erase view-specific cues, underscoring the need to preserve task-relevant signals 37. Another important SSL technique is to train models to predict future observations (e.g., frames or tokens) [4T] 44). TVE follows these principles by forecasting next- window dynamics to capture the core information for downstream labels.
Limitations. Although our theory leverages sufficient statisties, task vectors- via simple SQL aggregations-naa -ayi not fully capture the original next-window sets, thus leading to loss of information. Their dimensionality grows with the number of columns and joinable paths, which necessitates dimensionality reduction techniques [ g Finally, learnable representation for next-window sets may be more expressive; leave this exploration future work.
Conclusion. We introduce Task Vector Estimation, a pre- tre-training framework that integrates schema graphs, temporal dynamics, and task heterogeneity. Our theoretical results show that lask-aware objectives retain more downstream--elevamn information than conventional SSL methods, producing representations that predictive for downstream tasks. Our findings underscore the importance of treating task heterogeneity and temporal context as first-class citizens in RDL pre-training strategies.
7 Acknowledgements
Quang Truong, Zhikai Chen, and Jiliang Tang are supported by the National Science Foundation (NSF) under grant numbers CNS2321416, IIS2212032, IIS2212144, IIS 2504089, DUE2234015, CNS2246050, DRL24054483 and IOS2035472, the Michigan Department of Agriculture and Rural Development, US Dept of Commerce, Gates Foundation, Amazon Faculty Award, Meta, NVIDIA, Microsoft and SNAP.
References
[1] Hervé Abdi and Dominique Valentin. Multiple correspondence analysis. Encyclopedia of measurement and statistics, 2(4):651-63 667,, 2007.
[2] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50)11-34, 2018
[3] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta- learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
[4] Aditya, Gaurav Bhalotia, Soumen Chakrabarti, Arvind Hulgeri, Charuta Nakhe, S Sudar- shanxe, al. Banks: Browsing and keyword searching in relational databases. In VLDB '22: Proceedings of the 28th International Conference on Very Large Databases, pages 1083-1086. Elsevier, 2002.
[5] Rakesh Agrawal, Amit Somani, and Yirong Xu. Storage and querying of e-commerce data. In VLDB, volume pages 149-158, 2001.
[6] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017. URL
[7] Mohammad Alomrani, Mahdi Biparva, Yingxue Zhang, and Mark Coates. Dyg2vec: Efficient representation learning for dynamic graphs. Tronsactions on Machine Learning Research, 2024.
ISSN 2835-8856. URL httppp///ppppeeeeeeeeeee/rarrtee
10

---

[8] Karl Pearson and. Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11): 559-577 1901. doi: 10.10800114864440000908800000000 14786440109462720
URL https: /doi/ org/ 10. 1080/
[9] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learn- ing using random feature corruption. In International Conference on Learning Representations, 2022. URL/aatps://pperrriir
ceet/forum?id-eee -Yk=uVggggkKkb3
[10] Gleb Bazhenov, Oleg Platonov, and Liudmila Prokhorenkova. Tabgraphs: A benchmark and strong baselines for learning on graphs with tabular node features, 2024. URL https
//arxiv. org/ abs/2499. 14500
[11] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531-540. PMLR, 10-15 Jul 2018. www.hhttss:/prrceedinne
mlr. www.srresseee..............e wwwpppewssssseeeeee http//80//eeee wa/8///eegaaiiaa... 18a.
[12] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: www wandb.com/
Software available from wandb.com.
[13] Kaidi Cao, Jiaxuan You, and Jure Leskovec. Relational multi-task learning: Modeling relations between data and tasks. arXiv preprint arXhh:2303.07666 2023.
[14] Kaidi Cao, Jiaxuan You, Jiaju Liu, and Jure Leskovec. Autotransfer: Automl with knowledge transfer-an an application to graph neural networks. arXiv preprint arXiv:2303.06666 2023.
[15] Ke-Jia Chen, Jiajun Zhang, Linpu Jiang, Yunyun Wang, and Yuxuan Dai. Pre-training dynamic graph neural networks. Neurocomputing, 500:679 2022.
[16] Ke-Jia Chen, Linsong Liu, Linpu Jiang, and Jingqiang Chen. Self-supervised dynamic graph representation learning via temporal subgraph contrast. ACM Trans. Knowl. Discov, Data, 18 (1), September 2023. ISSN 1556-4681. doi: 10.1145/3612931. 1145/3612931
[17] Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting- Wei Chen, and Darby Tien-Hao Chang. Trompt: towards better deep neural network for tabular data. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023.
[18] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[19] Edgar F Codd. relational model of data for large shared data banks. Communications of the ACM, 13(6))377-387, 1970.
[20] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241955
[21] Milan Cvitkovic. Supervised learning relational databases with graph neural networks, 2020.
URL htttt:://taxiii>> org/ wwwg/bbb///0/// 202000000000066
[22] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2020.
[23] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[24] Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Relational deep learning: Graph representation learning relational databases, 2023, URL ARLhttps:: :/abxiiv//rxiv org/ abs 2tlpe/2311.ccm 04615

---

[25] Quan Gan, Minjie Wang, David Wipf, and Christos Faloutsos. Graph machine learning meets multi-table relational data. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 6512-6512, New York, NY, USA, 2024, Association for Computing Machinery. ISBN 9798400700401. doi: 10.1 1414446677288677144777777 URI.htts::///oo org/ 10. 0011111533335222 S51
[26] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. Proceedings of the 34th International Conference Machine Learning- Volume 70, ICMI'117, page 1263-1272. JMLR.org. 2017.
[27] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Proceedings of the 35th International Conference on Neural Information P Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845993.
[28] Terry Halpin and Tony Morgan. Information modeling and relational databases. Morgan Kaufmann, 2010.
[29] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[31] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang. Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 594-604, 2022.
[32] Weihua Hu, Yiwen Yuan, Zecheng Zhang, Akihiro Nitta, Kaidi Cao, Vid Kocijan, Jure Leskovec, and Matthias Fey. Pytorch frame: A modular framework for multi-modal tabular learning. arXiv preprint arXiv:2404.00776, 2024.
[33] Tao Huang, Yanxiang Ma, Shan You, and Chang Xu. Learning mask invariant mutual infor- mation for masked image modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL httttss://ppereviiiiiis
[34] James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 JEEE international conference on data science and advanced analytics (DSAA), pages 1-10. IEEE, 2015.
[35] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https //openreview
[36] Veronica Lachi, Antonio Longa, Beatrice Bevilacqua, Bruno Lepri, Andrea Passerini, and Bruno Ribeiro. Boosting relational deep learning with pretrained tabular models, 2025. URL
https://arxx: wwweexxvv.orggaaa / 04934
[37] Paul Pu Liang, Zihao Deng, Martin Q. Ma, James Zou, Louis-Philippe Morency, and Russ Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https wwt.:eennneettoooomtdiidllal
/
[38] Lingwen Liu, Guangqi Wen, Peng Cao, Jinzhu Yang, Weiping Li, and Osmar R. Zaiane. Capturing temporal node evolution via self-supervised learning: A new perspective on dynamic graph learning. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24, page 443-451, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703713, doi: 10.1145526168555363377777 URL https 00..11455 /111/1111111111 /555//66668555333765555
[39] Meng Liu, Ke Liang. Yawei Zhao, Wenxuan Tu, Sihang Zhou, Xinbiao Gan, Xinwang Liu, and Kunlun He. Self-supe temporal graph learning with temporal and structural intensity alignment. IEEE Transactions on Neural Networks and Learning Systems, 2024.
12

---

[40] Shengchao Liu, David Vazquez, Jian Tang. and Biere-AAndr Noël. Flaky performances when pretraining on relational databases (student abstract). In Proceedings of the Thirty-Sevente AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 10. 11..///aai..333133.33 266993. URL htt.:htps:/// posttps://dois http:///ooooooog/////// 1wwp://..100//////1 aaa00//aai.... v37i13. 26993
AAAI Press, 2023. ISBN 978-1-57735-88044 doi:
[41] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv: 160::66550000 08100, 2016.
[42] Jan Modl and Oliver Schulte. The ctu prague relational learning repository. arXiv preprint 1XX1:0151 2015.
[43] Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin. STUNT: Few-shot tabular learning with self-generated tasks from unlabeled tables. In The Eleventh International Conference on Learning Representations, 2023. URL
www.ereeiieweeeeeeeeeeeeeeeeeeee net/forum?
id=_xlsj [ls_xxljeeDvly
[44] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0878003778, 2018.
[45] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. URL http:///:ppppperrr http::/ cttpuriis.ccpppppppeeeep cc/paper/ 9015 9015-pytorch wentorch-a--nnggee an- un-imperative htwpeiiiieeetlll--i---e---eeeeee wwwp-poeooaaaaeeeeeeeeeeeeee--ee -ig.rrinnnnnnnbrrrryy library. pdf
[46] Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan Eric Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec. Relbench: A benchmark for deep learning on relational databases. In The Thirty-eight Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL horppprrrriiee.nne/fouummmm wwt.::tttteoummmmmmmm 2tmee//foormm22dddd88000mm333
https://openrevie::
[47] Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning, 2022. URL http:ppp.//aarr...orr/b///////11 03208
[48] Sheng Tian, Ruofan Wu, Leilei Shi, Liang Zhu, and Tao Xiong. Self-supervised representation learning on dynamic graphs In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, page 1814-1823 New York, NY, USA, 2021. Association for Computing Machinery. ISBN 1781450384469. 3482389, URL https:://tt....oo org/ 10. 1145/3459637 348259997733888899
doi: 10.1145/3459637.
[49] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for supervisd representation learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL
https://openrevie::
[50] Petar Veličković, William Fedus, William L. Hamilton, Pietro Lio, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations, 2019.
URL URL.https///opeervviie
[51] Minjie Wang, Quan Gan, David Wipf, Zheng Zhang, Christos Faloutsos, Weinan Zhang, Muhan Zhang, Zhenkun Cai, Jiahang Li, Zunyao Mao, et al. 4dbinfer: A 4d benchmarking toolbox for graph-centric predictive modeling on rdbs. Advances in Neural Information Processing Systems, 37:27236-27273, 2024.
[52] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Hetero- geneous graph attention network. In The World Wide Web Conference, WWW '19, page 2032-2032, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366748. doi: 10. 00.14457308558833135566 URL https://doi. org/10.11455
3308558.3313562
13

---

[53] Yanbo Wang, Xiyuan Wang, Quan Gan, Minjie Wang. Qibin Yang, David Wipf, and Muhan Zhang. Griffin: Towards a graph-centric relational database foundation model. In secood International Conference Machine Learning, 2025. URL. https:://pperret Horum?ideTxeC<Nh3cc
net/
[54] Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of Research and Development, 4(1):66 82, 1960. doi: 10.1 10.1147/rd.41.0066
[55] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela Van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in neural information processing systems, 33: 1033-11043, 2020.
[56] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823 2020.
[57] Yiwen Yuan, Zecheng Zhang, Xinwei He, Akihiro Nitta, Weihua Hu, Manan Shah, Blaž Stojanovič, Shenyang Huang, Jan Eric Lenssen, Jure Leskovec, and Matthias Fey. ConteXIGNN: Beyond two-tower recommendation systems. In The Thirteenth International Conference on Learning Representations, 2025. URL httts::/
[58] Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J Smola. Deep sets. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 1339-3400 Red Hook, NY, USA, 2017.
Curran Associates Inc. ISBN 9781510860964.
[59] Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[60] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla. Heteroge- neous graph neural network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 199, page 793-803, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 7781450362016 doi: 10.1145/22925003330961
URL https:///oi....... org/ 10. 1141 3330961
[61] Han Zhang, Quan Gan, David Wipf, and Weinan Zhang. Gfs: Graph-based feature synthesis for prediction over relational databases, 2023. URL http:::///:aarr /tt::://axxvvve/ http:///aaarl..oo/aa/////// 02033
[62] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV,
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduce accurately describe the proposed framework, namely Task Vector Estimation and theoretical contributions in Section Experimental results with performance gains included in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: The theoretical assumptions are clearly stated the statements in main texts and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper discusses the proposed method depth in Section 3 and Section 4
with experiment details in Section 5 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Codes, with sufficient instructions, are included in public URL.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/ppll wttpcccppppppppeeeeeeeeeeeeeeeee httae//ggiieee/eeeeebbissssssss /Cc////oddiiisssseeeeeee ssiiiisisiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Detailed experimental settings are stated in both main texts and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Result tables include mean performance and standard deviation for every experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational details are included in Appendix, where we state the CPU and GPU of machines use.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::/neuripp: http:///wrrrp...cc/o/o//ccccaaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not observe any negative societal impacts for our research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not observe any risks associated with our research.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have given proper attributions to the original authors cited in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide details of our proposed tasks in the appendix and codes in the public URL.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for editing writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanntaddard components.
Please refer to our LLM policy (https: /att:e///earrisccc//offeeeeeeeee for what should or should not be described.
21

---

A Proofs
In this section, we assume that all random variables x, y, 21, Z2 are discrete. We the following
identities to prove the theorem reported in Section 4.3 the following properties:
For any random variable x, y, and Z, we have
Non-negativity
I(x:y) 2 0, I(x, 2) 2 0.
(7)
Chain rule:
I(xy: = z(y; + [xx; 1 y).
(8)
Chain rule (Multivariate Mutual Information):
T(x:y;2) I(y;2) I(y; xx).
(9)
Entropy and Mutual Information:
H(x) Hx y) + I(x:)).
(10)
Conditional Mutual Information and Conditional Entropy:
I y 12) Hx z) Hx yz).
(11)
Federici et assumes the following property when Z2 representation x, which implies that the only source of stochastic in is from x:
T x) =. 0.
(12)
Federici et al. 22 also proves the following proposition, which implies that Z2 must be as predictive as x to predict y when Z2 a sufficient representation of for y.
Proposition A.1 ((22). Let x and y be random variables with joint distribution p(x, y). Let 22 is representation of x. Then z2 is sufficient for y if and only if:
T(x; y) T(y: 22))
(13)
Similarly, we also assume stochasticity for 11 is conditionally independent from any other variables in the system once x and t are observed:
[(z1yy xt) = 0.
(14)
A.1 Proof for Theorem
First, we need to prove sufficient representation of inputs and side-channel information for task implies that the representation does not discard any task-relevant information contained in the inputs and side channels.
Proposition A.2. Let x, t, be random variables with joint distribution p(x, t, y). Let Zı be a latent representation of x with additional side-channel information Then Zı is sufficient of and for y if and only if (xt; = 21);
Proof.
T(xt:y y 21)) I(xt: y) I(xt; 21)
T(xt:y) - T(y:41) - T(y; xt)
T(xt: T(y;21)
Thus, is sufficient of x and for (I(xt:y 22) 0) if and only if 7(xt;y) = I(y;21)
Consequently, the mutual information between the sufficient representation 2₁ and the label y is at least large as that of the input and the label y.
22

---

Corollary A.1. Xetx, y be random variables join distribution p(x, y. Let 21 be sufficient latent representation of x with additional side-channe information t for y. Then, ((zi:) [(x:
Proof.
[y: z1) [xt:: ((x,y) + I(::y x)
y(
The equality occurs when the side-channel information irrelevant to the downstream task given the observation of the input. Lastly, we prove our main theorem.
Let t, be random variables with joint distribution p(x, t,y). Assume that is sufficient representation of x with side-channel information t for y, and 22 sufficient representation of for y. Then mutual information between 21 and y is least much as that between 22 and y:
Theorem 4.1
I ((iiy) 2 ((z2i)..
Proof. From Corollary A.1
we have:
I(21;y) 2 ((x:y)
(15)
10
H(y 21) + H(y) 2 -H(y x) + H(y)
(16)
H(y 21) S H(y x)
(17)
Then we expand [(21;y):
T(21;) T(x21:y) - T(x:y 21)
T(x;y) + T(zi:y x) -I(x;y (21)
[(x:y) + +H((y xx H(y (xz1)) - (H(y 21) - H(y (zz)
I(x:y) H x) H(y | 21)
Given Eq. 17 and Proposition A. we have:
[(21;y) 2 T(22:y)
B Formal Definition of Label Generation
In Section 4.1 we provided a compact notation for RDL labels, assuming that k-hop neighbors share the same entity type and that timestamps are discretized with natural integer indices. In this section, we provide the full notations for the label generation process. Formally, downstream labels can be represented as:
ye
(tott Adl 1(x X (( (fotte
(18)
Ex(lodttall (fr(u) u E &N Aeleadotall (v) A (u) R].
(19)
where / is labeling function and & xx! (fu denotes the set of k-hop neighbors' features of v. of entity type R, which appear within the prediction window, and A (fost + At] denotes the set of
temporal k-hop neighbors based on schema traversal. ye (ta.t++At) corresponds to ye in Eq. 3 which represents the label associated with the entity at the cutoff timestamp
23

---

C Experiments
In this section, we would like provide further details regarding our experiments. All of the experiments are implemented in Pytorch 45). Pytorch Geometric [23] and Pytorch Frame [32]. All of the experiments are executed on one of the following 4 machines: 1) AMD EPYC 7763 64-Core Processor and NVIDIA RTX A6000, 2) AMD EPYC 7543 32-Core Processor and NVIDIA RTX A5000, 3) AMD EPYC 7543 32-Core Processor and NVIDIA RTX A5000, and 4) AMD EPYC 7513 32- 32-Core Processor and NVIDIA RTX A6000, Experiments logged and kept tracked by Weights & Biases 12].
C.1 Dataset and Task Descriptions
Here, we provide details regarding datasets and tasks for experiments. Datasets proposed by RelBench [46.. In addition to the original tasks proposed by RelBench [46). we also provide additional data limited tasks.
The rel-amazan dataset is sourced from the Amazon e-commerce platform. It comprises four original downstream tasks [46]:
rel-amazon
user-churn: predict whether user remains inactive (i.e., has interactions) over the next three months.
item-churn: predict whether an item remains without interactions over the next three months.
user-1tv: estimate the total monetary value of purchases made by over the next three months.
item-ltv: estimate the total monetary value of purchases received by an item over the next three months.
Additionally, we propose data limited tasks, which are conditioned on a certain attribute of entities up the cutoff timestamp and period At, where At 90 days:
among k least bought products in the previous period (t - At, t], predict products that remains without interactions over the next three months.
i -awpauuun-easst----sseeeee-ee-ee
among è top spending customers in the previous period (t - At, t], predict those who will be inactive over the next three months.
among l least spending products in the previous period (t - At, t). estimate the total monetary value of purchases received by an item over the next
three months.
in Table among those customers who only left bad reviews (review score 1, 2, or 3) in the previous period
(abbreviated
(t - At, f). estimate the total monetary value of purchases made by those users over the next three months.
rel-hn The rel-he dataset is sourced from the H&M relational database. Similar to rel-amazon, rel-hm comprises similar tasks as originally proposed by [46]:
user-churm predict if a customer does not make any transactions in the next 7 days.
item-sales: predict total sales for an article in the next days.
Additionally, we also propose the following tasks, which are conditioned on certain attribute of entities up to the cutoff timestamp and period At, where At 7 days:
usr-cchur-top----peedde
among & top spending customers in the previous period
(L - At, f]. predict those who do not make any transactions in the next 7 days.
taolee--to---sppnddde
among k top bought articles in the previous period (1
tt, estimate estimaat the total sales for those article in the next 7 days.
24

---

Table Performance across datasets and TVE variants in the sufficient-data regime 46) Reported metrics are ROC-AUC for classification tasks, and Mean Absolute Error for regression tasks. Split, Validation, and Test are abbreviated S, V, and T for readability. Bold is the best performance.
Model
Task
s
TVE-1
TVE-2
MAE+TVE-I MAE+TVE-2
CTR+TVE-1 CTR+TVE-2
Classification
rel-amz/ V
80.67 = 0.06
80.76 = 0.08
81.13 + 0.04
81.19 0.03
81.29 = 0.09
81.28 0.10
item-churm
T
80.91 # 0.13
81.03 # 0.10
81.47 + 0.04
81.49 # 0.04
81.57 0.13
81.56 + 0.12
rel-amz/ 69.85 = 0.03
0.04
69.92 + 0.04
69.93 0.05
0.00
70.03 0.06
user-churn
T
69.67 = 0.08
69.78 0.05
69.78 4: 0.07
69.80 : 0.03
6.02
69.85 + 0.07
rel-hm/
70.35 # 0.11
70.38 0.07
70.29 0.09
70.35 # 0.10
7.05
70.54 0.166 69.72 # 0.51
user-churn
69.86 = 0.17
69.81 = 0.17
69.89 + 0.09
69.80 = 0.11
70.13 = 0.22
Regression
rel-amz/
V 48. = 18.183 0.804 47.667 = 00.40
45.590 + 0.121 45,573 # 0.045 5
44.956 + 0.078
item-ltv
T 51.244 = 0.229 51.167 # 0.341 50.437 # 0.395 50.659 # 0.417 50.012 # 0.287 49.753 0.199
rel-amz/
V 12,442 = 0.021 12.455 +: 0.018
12.429 + 0.016
12.437 # 0.031 12.397 = 0.010
12.371 0.018
user-Ity
T 14.690 # 0.058
14.676 # 0.034
14.658 + 0.028
14.626 + 0.019
14.622 # 0.046
14.577 + 0.020
rel-hm//
0.0637 + 3e-4
0.0636 2e-4
0.0627 + le-4
0.0624 3e-4
0.0628 4e-4
0.0627 + 3e-4 0.0543 : 3e-4
item-sales
T 0.0554 = 4c-4
0.0553 + 2c-4
0.0544 # 2c-4
0.0543 = 3c-4 0.0547 + 3e-4
C.2
Models
In this section, we would like provide further details regarding models included in this study.
Baseline
This is the base model adopted from RelBench 46 for node-level tasks.
Masked Autoencoder (MAE) Building on GraphMAE 31. we introduce the following modifica-
tions to adapt it to relational data. First, instead of masking entire node feature vectors, apply cell-level masking- technique well established in tabular learning 99] 27 55) replace masked entries with values sampled from each column's 's marginal distribution. This obviates the need for a dedicated mask token and a re-masking stage. Our decoder still retains a projection layer after the GNN encoder, a single-layer GNN decoder, and last linear heads. Second, because cell-Ievel masking can alter a large fraction of each row, we reconstruct the full feature vector, as in 40 55 We limit reconstruction numerical, categorical, and textual features: normalized numerical values and text embeddings are recovered using mean-squared eтror loss, while categorical features are recovered via cross-entropy loss. Any entries missing in the original database are excluded from the loss computation. The final training objective is the sum of these individual reconstruction losses for all tables.
Contrastive Learning (CTR) Building on GraphCL [56], we make several adjustments similar to MAE. Specifically, we adopt the same masking augmentation as outlined in the previous paragraph. Also, follow SCARF D. we consider two views of the inputs be original features and augmented features, and the final objective is to maximize agreements between latent representations of two views via a symmetric InfoNCE loss (4) for all tables.
Task Vector Estimation (TVE) This is proposed method, where provide details in Section We also apply mask augmentation with probability (p- = 0.15) similar to MAE and CTR for regularization. A noteworthy observation is that our method is robt-table- which means that the objective (Eq. 4 is computed for a single table only. The hybrid models, namely MAE+TV TVV and CTR+TVE, also follow the same pipeline, with only changes in the decoder cater for different loss objectives. For both hybrid models, we set the balancing coefficient 3 to 0.1 (see Eq. 5).
4
C.3 RelBench Experiments on Data Limited and Data Sufficient Tasks
In this section, we would like to provide further details regarding experiments reported in Section 5.2 and Section 5.3
25

---

Baseline
MAE-0.25
CTR-0.25
TVE-1
TVF-2
relaamazonitem-chum
rel-amazon/tee-ttv
rel-hmilser-chum
-least100-spending
leas110-speending
-topp00 00--ednnngn
top100-spending
0.65
roc. juu
0.7
2.1 10
roc. s_auuc
roc aaaa
0.60
2x 10
0.6
10-
0.80
0.55
18x10
0.50
Epoch
Epoch
Epoch
Epoch
Figure 7: Test performance curves averaged over runs data limited tasks.
Table 4: Hyperparameter settings for each model in Table
Each configuration is in the format
[learning rate, channel size].
Model
Task
Baseline
MAE-0.25
MAE-0.5
CTR-0.25
CTR-0.5
TVE-1
TVE-2
rel-amz/
least50
[0.001, 128]
[0.005, 256]
[0.001, 256]
[0.002, 256]
[0.005, 256]
[0.001, 256]
10.001,
item-churn
1east100
[0.00075, 128]
[0.005, 128)
[0.005, 128]
[0.001, 256]
[0.002, 256]
[0.00075, 256]
[0.00075, 256]
amzz
top50
[0.005,
128]
[0.005, 256]
[0.005, 256]
[0.005, 256]
[0.005, 128]
[0.00075, 128)
[0,002, 256]
user-churn
lop100
[0.002,
256]
[0.002, 256]
[0.005, 256]
[0.002, 128]
[0.001, 128)
[0.005, 256]
[0.005, 128]
rel-hm/
top50
[0.001,
128]
[0.002, 128]
[0.002, 256]
[0.001, 256]
[0.00075, 256]
[0.005, 128]
[0.005, 128]
churm
top100
[0.005, 128)
[0.002, 128)
[0.001, 128]
[0.00075, 256)
[0.001, 256]
[0.005, 256]
[0.002, 256)
amzzitem-lt Itv
1e0
[0.00075, 128]
[0.002, 128)
[0.002, 256]
[0.005, 256]
[0.005, 256]
[0.001, 256]
[0.002, 256]
rel amzzuser ltv
bad
[0.00075, 128]
[0.00075, 128]
[0.00075, 128)
[0.00075, 128]
[0.00075, 128]
(0.00075, 256]
[0.00075, 128]
rel sol-hhu/item sales lop200
[0.001, 256]
[0.005, 256]
(0.005, 128)
[0.005, 256]
[0.005, 256)
[0.005, 256]
[0.005, 256]
For pre-training, we train these models for 81 epochs with the max steps per epoch to be 1000 per GPU, which effectively means 4000 steps per epoch due to distributed training. Learning rate is 0.001 for the first 50 epochs and reduces to 0.0001 for the remaining epochs. During fine-tuning, we fine-tune the pre- pre-trained models for 20 epochs (except to where we fine-tune the models for 50 epochs). We also perform light hyperparameter tuning where we vary the following hyperparameters: learning rate E {0.0075, 0.001, 0.002, 0.005) and number of channels E (128, 256]. For the original tasks proposed by RelBench, we adopt the learning rate of 0.005 and 0.002 for classification and regression tasks respectively. Similarly to the pre-training stage, we also perform distributed training on tasks with large amount of data. Data limited tasks repeated 10 times (except to where we only repeat 5 times), while data sufficient batch size of 512 (reduced to 256 batches exceed
and
rel-hm/Se
tasks are repeated times. All experiments 24 GB VRAM), mean aggregation, two GNN layers, neighbor sampling of 128 then 64 hops, and uniform temporal sampling. Detailed hyperparameter differences are reported Table and Table for data limited and data sufficient tasks, respectively.
Fig. 7 shows learning curves on low-data tasks, supplementing Table We can observe that TVE pre- training both accelerates convergence and outperforms MAE and CTR. Table which complements Table 2 compares all TVE variants in the data-rich regime and shows that combining vanilla TVE with any standard SSL objective yields additional gains, underscoring their complementary strengths.
C.4 Linear Probing Experiments
We would like to give more results supplementing the one reported in Section 5.4 Table 0 reports numerical results that are visualized in Fig. [ Specifically, in the linear probing setting, TVE consistently outperforms both MAE and CTR on ev- ery evaluated task. Notably, for rel-amazon/ http:/ittemccch-eaaeeeeee weteeestttssppeeee 100-sddnning and rel wtw.://tttee-t--eeettf000o provement, highlighting their unreliability as pre-training objectives for downstream tasks.
neither MAE-0.25 nor CTR-0.25 yields any
26

---

Table 5: Hyperparameter settings for each model in Table 2 [learning rate, channel size].
Each configuration is in the format
Model
Task
Baseline
MAE-0.25
MAE TVE-1 MAE TVE-2
CTR-0.25
CTR TVE-1 CTR TVE-2
ael-amziiem-churn [0.005, 128] [0.005, 256]
[0.005, 128]
[0.005, 128]
[0.005, 256]
[0.005, 256]
[0.005, 256]
rel-amuzuusr--cuur [0.005, 128]
[0.005, 128]
[0.005, 128)
[0.005, 128]
[0.005, 128]
[0.005, 128]
[0.005, 256]
31-hm/user-chum [0.005, 256] [0.005, 128]
[0.005, 256]
[0.005, 128]
[0.005, 256]
[0.005, 256]
[0.005, 256]
am-aaittttmtit [0.002. 256] [0.002, 256]
[0.002, 128]
[0.002. 128]
[0.002, 256]
[0.002, 256]
[0.002, 256]
rel-amzzuserr-tv [0.002, 128]
[0.002, 128]
[0.002, 256]
[0.002, 256]
[0.002, 256]
[0.002, 256]
[0.002, 256]
rel ti-hm/item-sslee [0.002. 256] [0.002, 256]
[0.002, 256] [0.002, 256]
[0.002, 256]
[0.002 256]
[0.002, 256]
Table 6: Linear probing results. Reported metric is ROC-AUC (higher is better) for classification
tasks, and Mean Absolute Error (lower better) for regression tasks. Split, Validation, and Test are abbreviated as S, V, and T for readability. Bold is the best performance.
Model
Task
Baseline
MAE-0.25
CTR-0.25
TVE-1
TVE-2
Classification
rel-ann/htem-chum-lees (0--penning
V
69.72 = 4,98
61.18 1.07
68.13 + 1.78
81.57 = 0.37
82.76 0.43
63.01 # 4,98
52.92 # 0.79
56.96 + 2.07
78.06 + 0.65
77.83 * 0.95
V
84.53 = 1.97
88.44 : 0.29 83.86 1.04
88.62 = 0.38 81.63 +
91.42 = 0.07 86.56 0.36
90.94 # 0.14 85.72 + 0.38
78.36 + 1.72
rel hatthm/uurncccumm-oop000--ppppp
69.43 = 1.56
64.27 = 0.60 56.65 0.53
74.57 = 0.73
71.34 0.24
70.09 = 0.61
53.93 1.46
64.22 + 1.47
65.33 0.35
64.15 0.61
Regression
0.1753 0.0005
0.1751 0.0001 0.1755 0.0005 0.1749 + 0.0011
0.1746 0.0013
ael-amz/item- Iev-lest1100 least100-spediing
T 0.1867 0.0004 0.1865 = 0.0001 0.1866 0.0007 0.1841 = 0.0014 0.1836 0.0013
V 3.6312 0,0118 3.6948 0.0126 3.6524
0.0123 3,7160 + 0.0140
3,5519 0.0056
od--hn/item--sale--ooon wttp:/saaaaaaaoppppppppooaeieiia
T 3.5325 # 0.0386 3.4925 = 0.0059 3.5087 = 0.1426 3.4815 # 0.0210
3.4583 = 0.0320
C.5 Experiments on Weakly Schema-dependent Tasks
So far, we have only focused on scenarios where tasks are strictly defined by SQL queries. However, some tasks may weakly depend on the schema graph or relational entity graph-such those based on manual labels or involving database errors. To quantify TVE's sensitivity to the underlying relational entity graph's quality, we simulated these weak dependencies by running controlled "noise" experiments on the RelBench's 's rel-amazon database, using the following steps:
1. We randomly dropped 20% and 40% of the rows in the review table to corrupt the relational entity graph. The corrupted graph here is considered clean signals, and tasks depend on this newly created graph.
2. We treat the unaltered database noisy database; thus, the downstream tasks defined in Step are no longer fully dependent on this noisy database or its original graph structure. Therefore, robust pre-training paradigm overfitting to the noisy relational entity graph cannot generalize well to the tasks in Step 1.
3. We compared three pre-training paradigns-MAN CTR, and our TVE-on on two tasks wttpr-ccuuttttto--spppee
and churn. We then (a) pre-trained our models on the noisy graph (including task vector constructed from this noisy graph), and (b) fine-tuned (and evaluated) on the same noisy graph using labels generated from the clean database (constructed in Step 1). In other words, the only factor that deviates from prior experiments is the downstream labels, which are generated based on a simulated clean database.
Table 7 compares robustness to noises of different SSL paradigms. We can observe that even when the downstream tasks loosely depend on schema graph understanding, and when TVE pre-trained on a noisy relational database, TVE's task-aware pre-training consistently outperforms standard SSL objectives. This demonstrates that our framework robust to errors in the schema graph/relational entity graph and retains its effectiveness across range of noise levels.
27

---

Table 7: Performance across different model variants on the scenarios where tasks weakly depend on the relational entity graphs of rel- -mazon. Reported metric is ROC-AUC. Task
Baseline
MAF-0.25
CTR-0.25
TVE-1
Drop 20%
7.14
82.26 0.27
78.98 1.17
86.79 0.26
Lincar Probing
user-churnttop100-0 -pppeendnnnnggggnnnnngn...
Drop 40%
73.95
1.76
78.16 0.46
76.38 = 1.80
81.44 0.17
Drop 20%
80.72 1.14
82.83 # 0.41
81.46 + 2.31
85.04 0.92
Fine-tuning
Drop 40%
76.39 1.16
76.97 # 1.43
75.72 = 1.26
78.29 + 1.79
Drop 20%
60.92 = 0.61
65.62 0.04
64.59 + 0.01
70.17 +
Linear Probing
uscr-churn
Drop 40%
61.95 = 0.67
66,96 0.03
66.14 0.04
7
Drop 20%
70.09 0.15
69.74 0.21
69.98 + 0.13
70.26 0.10
Fine-tuning
Drop 40%
71.35 = 0.06 70.61 0.24
71.20 + 0.17
71.37 + 0.20
Table 8: Comparison of linear probing performance of TVE-1 pretrained on different entity types and other models. Reported metric ROC-AUC.
Model
iten-chumm-leass100-pppniii tuerechurm--oppll0pppeeennnn
Baseline
63.01 + 4.98
78.36 + 1.72
MAE-0.25
52.92 # 0.79
83.86 = 1.04
CTR-0.25
56.96 + 2.07
81.63 = 0.46
TVE-1 (item)
78.06 = 0.65
83,28 + 0.63
TVE-1 (user)
65.88 + 2.25
86.56 + 0.36
C.6 In-database Cross-entity Transfers
Since our proposed pre-training approach is teeeey-tpp--enticc, where the constructed task vector is limited to a certain entity type in the database, it is natural to question the transferability between pre-trained models on two different entity types in the same database. To better understand the separability of the pre-trained latent representation, we further conduct a linear probing experiment, where we investigate if item-centie pre-trained model can be fine-tuned for user-task, and vice versa. Table [ reports the results. TVE-1 (item) refers to the pre-trained model on the item's 's task vector, while TVE-1 (user) refers the pre-tre trairrried model on the user's task vector. Surprisingly, even when the task vector is not matched with downstream tasks' entity type, it can still yield better performance than other generic SSL methods. For example, TVE-1 (user) is only worse than TVE- and TVE-1 (item) is as strong as MAE-0.25 on Consequently, this experiment highlights the necessity create
i (item) on user- asee-chhurn-top1100--eeedddd an entity type-specific pre-training objective to maximize downstream performance gains.
C.7 Experiments Cell-level Prediction
While previous experiments focus predictive modeling over the RelBench benchmark [46], there exists other benchmarks such as 4DBInfer [51) where tasks are defined differently. Specifically, RelBench focuses on inference SQL-generated label attached to rows in the dimension table (the table that is static and does not contain timestamps), while 4DBInfer focuses on cell-level prediction.
We would like to show that our proposed framework is performant regardless of evaluation settings. We pre-trained TVE-1 on seznam dataset [42), and evaluated downstream performance on an entity attribute classification task called charge, where we would like to predict cells under column sluzba of entity table dobito. We reported metrics ActticccAcccracy... Macro FL, Micro F1, and Mean Reciprocal Rank (MRR)-oV on both validation and test splits. We leverage the AutoCompleteTaa implementation provided by RelBench [46 to create the corresponding training table for cell-level prediction, and we split the dataset such that the cut-off timestamps for the validation set and test set are at 2015-03-31 and 2015-07-31, respectively. Table 9 reports results for both linear probing and fine-tuning settings.
We can observe that the performance of MAE and CTR is not consistent across datasets, while TVE consistantly performs well and exhibits competitive performance (matching MAE for linear probing and matching CTR for full-finttuning). This is because both MAE and CTR have limitations. For example, CTR does not offer separable representation after pre-training (low performance
28

---

Table Results on seznam/charge. Split, Validation, and Test are abbreviated as V, and T for readability.
Setting
Model
Accuracy
Macro F1
Micro F1
MRR
V 60.27 + 1.22
28.89 + 2.09
60.27 = 1.22
76.96 + 0.70
Baseline
T
59.36 1.00
27.99 = 1.86
59.36 + 1.00
76.48 = 0.50
71.03 0.04
44.50 1.24
71.03 = 0.04
84.00 = 0.03
Linear Probing
MAE-0.25
70.52 0.05
43.32 1.74
70.52 # 0.05
83.76 = 0.04
42.88 0.09
12.74 0.81
42.88 + 0.09
64.56 + 0.10
CTR-0.25
41.11 0.32
10.01 = 0.48
41.11 + 0.32
62.79 = 0.21
69.98 0.21
44.20 + 1.00
69.98 + 0.21
83.44 = 0.09
TVE-1
T
69.90 + 0.16
43.39 + 0.64
69.90 = 0.16
83.55 = 0.08
V
81.13 = 0.21
62.02 0.76
81.13 = 0.21
89.96 = 0.12
Baseline
80.97 0.50
63.43 0.83
80.97 + 0.50
89.91 = 0.28
81.18 0.15
62.00 2.37
81.18 = 0.15
89.99 = 0.09
Fine-tuning
MAE-0.25
81.08 0.32
63.31 1.00
81.08 + 0.32
89.97 = 0.18
81.21 0.17
63.28 = 1.05
81.21 + 0.17
90.01 = 0.10
CTR-0.25
81.31 # 0.41
63.69 # 0.92
81.31 # 0.41
90.10 = 0.22
V
81.39 + 0.22
63.92 + 1.17
81.39 + 0.22
90.11 = 0.12
TVE-1
T
81.26 # 0.29
63.82 # 0.75 81.26 + 0.29
90.07 = 0.16
during linear probing), while MAE is not as performant as CTR during full fine-tuning, Moreover, these gains come despite a conceptual mismatch-TTVE was optimized for next-state prediction on dimension tables, whereas the downstream tasks operate on fact-table rows. Regardless of evaluation setting, TVE consistently performs well across different evaluation settings.
C.8 Performance Sensitivity Across Splits and Tasks
To further assess model robustness, Fig. g presents additional scatterplots
for three more tasks, namely rel- rel-amas maazoon/ten-ccuua--aasttto- and rel-hm hm wwt.muuurrccce cchurn-top55 -pepdingenningg
aalaamazon/ttem- wwwpo//itemcccuunn-eeee with the purpose of complementing the results reported in Fig. 6 Across all three tasks, TVE achieves higher validation and test scores with consistently smaller average distances to the centroid, whereas traditional SSL methods suffer from recency bias, where they exhibit inflated validation performance but degraded test results (e.g. hal--mmusserccne findings underscore TVE's stability to hyperparameter choices and the last linear head's weight initialization.
These
C.9
Complexity Analysis
In this section, we present complexity analysis on task vector construction and provide comparison of pre-training runtimes between our proposed approach and the baseline SSL methods.
C.9.1 Task Vector Construction
First, we provide formal worst-case time complexity task vector construction, and then provide empirical analysis of task vector construction on rel rel-amazon and rel-hm via elapsed time and peak memory usages.
Remark C.1. Let the number of rows per table O(N), the number of timestamps brunching factor of relational entity graph (the input graph for learning) be b. the number of aggregation functions v, the branching factor of schema graph (the blueprint of the Relational Database) be p. and the number of hops for generating the task vector be k. Then, the time complexity
the
29

---

of task vector construction is
TIVE - O
Proof. We would like to compute the computational cost each step for constructing the task vector. First, we compute the cost taken for single path from the schema graph perspective (Step 1 3), and then sum up to get the final cost (Step 4). Remember that for every schema graph path, there are many paths fall under the same schema graph path since schema graph is just a blueprint of the input graph.
Step 1. Joining with the time table, which is a necessary step regardless of pre-training paradigm, costs (NTT).
Step 2. Next, every join causes each row to fan out by a factor of Obb), so the cost of joining up to u successive times is:
O( + NTb² + NTb") = (NNT))
Σ
b¹).
i=1
For > 1, the geometric sum is scaled on the order of b":
gut1
b
Σ 1=1
b 6" O(6").
b
Therefore, the total cost of for joining up to length u is O(NTb").
Step 3. One aggregation of r functions over the joined rows costs OrrNTb") every path (from the schema graph perspective) cost O(NTI" rNTbb) = O(TNbb).
Step 4. There are O(p") paths of length u from the schema graph perspective, summing the cost:
TivE Σ 1r" O(PNT6")] O [ Σ (ob) -O(NTTp6
Therefore, the cost blows up exponentially in k through both pd and b^, with linear dependence on N, and T. In practice, branching factor of the schema graph p typically small, and the schema graph is small, leading to small k. Furthermore, path is only meaningful if it contains at least one fact table (the table containing timestamps), so that we can construct the predictive pre-training objective. Finally, this operation is only done once, and is not involved during pre-training optimization.
To demonstrate our point, we would like report the time and peak memory usage for constructing task vector across datasets in Table 10 As we can see from the table, this operation is feasible even with large-scale databases, where rows, respectively. For our experiments, the maximum hop is 2, while the possible aggregation functions include MEAN, MIN, MAX, SUM, COUNT, and STDDEV for numerical values, and MODE, COUNT, and COUNT DISTINCT for categorical values. In industry scenario, such cost is expendable.
and rel-hm contain approximately 24M and 33M
C.9.2 Pre-training Runtimes
We present an empirical comparison of per-batch pre- pre-training runtimes (batch size 512) for all SSL methods. Using shared RDL backbone, we measure the wall-clock time for single training step, averaged over 100 runs per model. As shown Table [] TVE achieves the lowest overhead, since computes losses to approximate task vectors only for root-table entities. In contrast, MAE and CTR must compute losses for all nodes the sampled subgraphs, and CTR incurs additional cost from its double forward passes and pairwise distance calculations.
30

---

Table 10: Elapsed time and peak memory usage for different datasets given root tables.
Model
rel-amazon/item
rel-amazon/user
rel- hi-/mmmtem
rel-hm/user
TVE-1
64.59 seconds 39.98 GB
90.17 seconds / 40.07 GB
19.38 scconds 4.62 GB
11.74 scconds 19.85 GB
TVE-2
382.75 seconds 40.03 GB
652.85 seconds 52.67 GB
98.03 seconds 16.15 GB
871.26 seconds 148.19 GB
Table 11: Time complexity of different SSL methods for every step in seconds.
Model
Dataset
Root Table
MAE
CTR
TVE-1
TVE-2
0.0554 0.0071
0.0587 = 0.0088
0.0445 = 0.0102
0.0432 + 0.0165
rel-amazon
item
0.0583 # 0.0065
0.0663 = 0.0059
0.0440 = 0.0089
0.0478 + 0.0101
rel-hm
user
0.0610 = 0.0115 0.0814 0.0075
0.0615 = 0.0062
0.0504 = 0.0288
0.0466 + 0.0053
item
0.3110 = 0.0552
0.0609 = 0.0166
0.0571 = 0.0038
D Further Discussions
D.I
On Designing Better Approximation of Sufficient Statistics
Designing richer approximations of sufficient statistics is indeed vital and challenging direction. For example, beyond fixed aggregators like task vectors, we can employ a small neural "set encoder" (e.g.. DeepSets [58] or transformer based pooling) that learns compress each next- wextow set into vector. This approach can, in principle, capture all information in the set without hand-crafting statistics. However, training two networks one to encode the and one to predict the label- introduces instability. For instance, a GAN-style formulation (where the encoder acts as a generator and the predictor as discriminator) risks mode collapse, mapping diverse inputs to the same latent code. Therefore, designing a sophisticated loss function to avoid mode collapse is challenge for this strategy.
Our current task-vector objective optimizes only one encoder for input graph, which makes training stable and efficient. Despite being simpler, these vectors capture key moments of the distribution and are not susceptible to GAN-style collapse. They provide a reliable surrogate that practitioners can use immediately, while leaving more complex, learnable summaries to future work.
D.2 Sufficiency of Simple Statistics
Computing minimal sufficient representations has long been a grand challenge. Because direct evaluation of mutual information is often intractable, Alemi et al. (6] instead optimize tractable lower bound, while Belghazi et al. 1 turn to neural estimators to approximate mutual information directly.
Given the above challenges of sufficient statistics approximation, we would like to provide comple- mentary arguments showing why-and under what conditions conditions-sinple statistics can suffice, and how we mitigate the information loss when compared with sufficient statistics in practice.
Statistical sufficiency in common distributions. In many distributions, small collection of aggregators is provably sufficient. Notable examples include:
The sum of samples is sufficient for exponential, Poisson, and Bernoulli distributions.
The max is sufficient for a uniform distribution over bounded interval.
The mean is sufficient for normal distribution with known variance.
Whenever the underlying dynamics match one of these canonical families, no information is lost by reducing the full set of observations to the corresponding statistic.
Enriching the task vector minimize loss. Real-world derivviino queries often depend on different columns, thus leading to potential different distributions. To guard against this, we augment our task vectors with as many simple statistics-includee MEAN, MIN, MAX, SUM, COUNT, and STDDEV for numerical values, and MODE, COUNT, and COUNT DISTINCT for categorical values. This "overcomplete" summary ensures that, even if some statistics are redundant, we capture a broader range of moment and order information, thereby shrinking the worst-case information gap.
31

---

1=0.002
F=0.001
Ir=D 005
Ir=0.000)
CIR-9.25
Dist: 0.052
Dist: 0.065
Dist: 0.034
Dist: 0,020
Centald:
0.85
Centroid:
Centroid:
0.85
Cantrold:
amszervitio
1esst50-peeni
ym0.730
y=b
0.75
0.75
0.75
0.13
0.70
0.60
0.60
0.60
0.60
0.7
0.8
0.5
0.7
0.3
0.6
0.7
0.8
0.6
0.7
amazonittmm churn
0002200--ppeeee spending
0.80
X-0, 7.9
Centroid:
0.80
x-0,736
Centroid:
0.00
Centroid:
7-0
0.80
x-0.800
Centroid:
0.015
0.75
y=0.683
0.75
y=0.699
0.75
y=u 722
0.75
y=0.761
Test
1e320000
0.70
0.70
0.70
0.70
0.05
0.60
0.60
0.70
0.75
0.80
Dist: 0.021 Contrald:
Dist: 0.031
Dist: 0.039
Dist: 0.033
Cantroid:
Contraid:
Centroid:
rel relermtuerr chum
tep00- apending
0.8
x=O
0.8
k=0,
0.8
x=0 80B
0.8
x=
y=0.595
y=O 584
y-
0.7
0.7
0.7
0.7
tep50
0.6
0.6
0.7
0.8
0.6
0.7
0.8
0.6
0.7
0.8
0.6
0.7
0.8
Velidation
Velidation
Velidation
Velidation
Figure 8: Additional Results for Validation vs. Test scatter plots for low-data node classification tasks across varying learning rates. Distances centroid (denoted xir) are used to measure performance variance.
Set-theoretic recovery and downstream expressivity. From a set- set-theoretic standpoint, certain combinations of simple statistics can perfectly reconstruct small multisets. For instance, if exactly three purchases occur in the prediction window, then the triplet (MIN, MAX, SUM} uniquely determines the multiset. More generally, accurate pre-training on these statistics helps the model learn an implicit approximation of the original value-set, which turn improves its ability to approximate any downstream labeling function-sing since most such functions operate on those same sets (cf. Eq. 3).
Empirical validation in tabular domains. Prior work Deep Feature Synthesis (DFS)
demonstrates that stacking joins with simple aggregations often matches more expressive graph-based architectures on many benchmarks [5I [53. While DFS features incur theoretical information loss relative to a full relational graph encoding. their competitive accuracy shows that richly engineered statisties can suffice in practice. Our investigation orthogonal: we systematically characterize all node-level SQL-generated labels as set functions and show how task-vector approach can act as surrogate representation for next-window sets. Additionally, our task vector includes similar aggregators to that of DFS.
Together, these points explain why simple statistics are not only theoretically grounded but also practically powerful-and how we effectively minimize any information loss.
32


Reviews:
[{"review_id": "kpCHtC7Oli", "reviewer": "Reviewer_5cD2", "summary": "The paper introduced Task Vector Estimation (TVE), a novel pre-training framework for relational databases (RDBs) that explicitly models task heterogeneity and temporal dynamics. Different from the traditional self-supervised learning methods that focus on input features, TVE incorporates knowledge about the underlying distribution driving label generation by constructing predictive supervisory signals through set-based aggregation over schema traversal graphs. The paper formalized their approach through information theory, demonstrating that task-informed representations retain more relevant signals for downstream tasks. Extensive experiments on the RelBench benchmark demonstrated that  TVE consistently outperforms traditional pre-training baselines, especially in low-data regimes.", "questions": "1. Please respond to W2.\n2. Please respond to W3.\n3. How sensitive is TVE to the quality of the schema graph representation? Would errors in schema understanding significantly impact performance?", "limitations": "No", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nS1. The paper identified a key limitation in current pre-training strategies for relational data which overlook task heterogeneity and temporal dynamics. The paper argues that different tasks defined on RDBs depend on schema graphs, temporal dependencies, and SQL-defined label logics.\n\nS2. The information-theoretic analysis provides mathematical justification for why task-aware representations should outperform traditional self-supervised methods. Theorem 4.1 proves that representations incorporating side-channel information share at least as much information with downstream tasks as those without.\n\nS3. The paper included detailed ablation studies, sensitivity analyses, and linear probing experiments that provided insights into the stability and transferability of the learned representations.\n\nWeakness\n\nW1. While the paper claimed that TVE addresses task heterogeneity, the experiments mainly focus on performance on individual tasks rather than demonstrating transfer across multiple related tasks within the same database.\n\nW2. It will be nice if the paper can provide more discussion about how to design better approximations of sufficient statistics.\n\nW3. The paper does not analyze how well the chosen SQL aggregations approximate the ideal sufficient statistics of the underlying distribution.", "was_revised": true, "final_justification": "The rebuttal has strengthened my confidence in both the theoretical foundations and practical robustness of the proposed approach.", "rebuttal": "We would like to express our appreciation to the reviewers for the thorough evaluation and helpful recommendations. We would like to address the reviewer’s key concerns below.\n\n> **[W1]** **While the paper claimed that TVE addresses task heterogeneity, the experiments mainly focus on performance on individual tasks rather than demonstrating transfer across multiple related tasks within the same database.**\n> \n\nWe would like to clarify that the goal of our work is to create a pre-training objective that can transfer across multiple downstream tasks like traditional pretraining in other fields such as language and image. Contrary to other fields, Relational Deep Learning offers a flexible framework where one can create a novel task that is based on temporality, structure, and SQL logics. Our goal is to systematically formulate that SQL-generated node-level labels are depending on sets of next-window values, and how to retain information of these sets during pre-training. In other words, we establish a universal mechanism for capturing supervisory signal of all downstream tasks in a single framework. As such, explicitly evaluating “bidirectional transfer” between two specific tasks lies beyond our core contribution, which is to prove and realize a unified pre-training paradigm that inherently supports transfer across the entire, potentially unbounded family of relational tasks.\n\n> **[W2] It will be nice if the paper can provide more discussion about how to design better approximations of sufficient statistics.**\n> \n\nThank you for this perceptive suggestion. Designing richer approximations of sufficient statistics is indeed a vital and challenging direction, and we would like to engage in the discussion regarding possibilities of better representation for next-window sets. \n\nBeyond fixed aggregators like task vectors, we can employ a small neural “set encoder” (e.g., DeepSets or transformer-based pooling) that learns to compress each next-window multiset into a vector. This approach can, in principle, capture all information in the set without hand-crafting statistics. However, training two networks—one to encode the set and one to predict the label—introduces instability. For instance, a GAN-style formulation (where the encoder acts as a generator and the predictor as a discriminator) risks mode collapse, mapping diverse inputs to the same latent code. Therefore, designing a sophisticated loss function to avoid mode collapse is a challenge for this strategy.\n\nOur current task-vector objective optimizes only one encoder for input graph, which makes training stable and efficient. Despite being simpler, these vectors capture key moments of the distribution and are not susceptible to GAN-style collapse. They provide a reliable surrogate that practitioners can use immediately, while leaving more complex, learnable summaries to future work.\n\n> **[W3] The paper does not analyze how well the chosen SQL aggregations approximate the ideal sufficient statistics of the underlying distribution.**\n> \n\nThank you very much for your thoughtful question. First, we would like to highlight that computing minimal sufficient representations has long been a grand challenge in both the Machine Learning and Information Theory communities. Because direct evaluation of mutual information is often intractable, Alemi et al. [1] instead optimize a tractable lower bound, while Belghazi et al. [2] turn to neural estimators to approximate mutual information directly.\n\nNext, we would like to clarify complementary arguments showing why—and under what conditions—simple statistics can suffice, and how we mitigate the information loss when compared with sufficient statistics in practice.\n\n**Statistical sufficiency in common distributions.**\n\nIn many distributions, a small collection of aggregators is provably sufficient.  For example:\n\n- The **sum** of samples is sufficient for exponential, Poisson, and Bernoulli distributions.\n- The **max** is sufficient for a uniform distribution over a bounded interval.\n- The **mean** is sufficient for a normal distribution with known variance.\n\nWhenever the underlying dynamics match one of these canonical families, no information is lost by reducing the full set of observations to the corresponding statistic.\n\n**Enriching the task vector to minimize loss.**\n\nReal-world label‐derivation queries often deviate from textbook distributions.  To guard against this, we augment our task vectors with as many simple statistics—include `MEAN`, `MIN`, `MAX`, `SUM`, `COUNT`, and `STDDEV`  for numerical values, and `MODE`, `COUNT`, and `COUNT DISTINCT` for categorical values.  This “overcomplete” summary ensures that, even if some statistics are redundant, we capture a broader range of moment and order information, thereby shrinking the worst‐case information gap.\n\n**Set‐theoretic recovery and downstream expressivity.**\n\nFrom a set‐theoretic standpoint, certain combinations of simple statistics can perfectly reconstruct small multisets.  For instance, if exactly three purchases occur in the prediction window, then the triplet { `MIN`, `MAX`, `SUM` } uniquely determines the multiset.  More generally, accurate pre‐training on these statistics helps the model learn an implicit approximation of the original value‐set, which in turn improves its ability to approximate any downstream labeling function—since most such functions operate on those same sets (cf. Eq. (3)).\n\n**Empirical validation in tabular domains.**\n\nPrior work on **Deep Feature Synthesis** (DFS) [3] demonstrates that stacking joins with simple aggregations often matches more expressive graph‐based architectures on many benchmarks [4, 5].  While DFS features incur theoretical information loss relative to a full relational graph encoding, their competitive accuracy shows that richly engineered statistics can suffice in practice.  Our investigation is orthogonal: we systematically characterize **all** node‐level SQL‐generated labels as set functions and show how a task‐vector approach can act as surrogate representation for next-window sets. Additionally, our task vector includes similar aggregators to that of DFS.\n\nTogether, these points explain why simple statistics are not only theoretically grounded but also practically powerful—and how we effectively minimize any information loss.\n\n> **[Q3] How sensitive is TVE to the quality of the schema graph representation? Would errors in schema understanding significantly impact performance?**\n> \n\nThank you for this excellent question. To quantify TVE’s sensitivity to schema‐graph quality, we ran controlled “noise” experiments on the RelBench rel-amazon database:\n\n1. We randomly dropped 20% and 40% of the rows in the review table to corrupt the relational entity graph. The corrupted graph here is considered clean signals, and tasks are depending on this newly created graph.\n2. We treat the unaltered database as noisy database; thus, the downstream tasks defined in 1. no longer fully dependent on the original schema graph. Therefore, a robust pre-training paradigm overfitting to the noisy relational entity graph won’t generalize well to the tasks in 1. It simulates the case when tasks are not based on the schema graph due to errors in schema understanding.\n3. We compared three pre-training paradigms—MAE, CTR, and our TVE—on two tasks user-churn-top100-spending and user-churn. We then (a) pre-trained our models on the noisy graph (including task vector constructed from this noisy graph), and (b) fine-tuned (and evaluated) on the same noisy graph using labels generated from the clean database.\n\nrel-amazon/user-churn-top100-spending (linear probing)\n\n| Model | Baseline | MAE-0.25 | CTR-0.25 | TVE-1 |\n| --- | --- | --- | --- | --- |\n| Drop 20% | 75.42 ± 3.10 | 82.26 ± 0.27 | 78.98 ± 1.17 | 86.79 ± 0.26 | \n| Drop 40% | 73.95 ± 1.76 | 78.16 ± 0.46 | 76.38 ± 1.80 | 81.44 ± 0.17 |\n\nrel-amazon/user-churn-top100-spending\n\n| Model | Baseline | MAE-0.25 | CTR-0.25 | TVE-1 |\n| --- | --- | --- | --- | --- |\n| Drop 20% | 80.72 ± 1.14 | 82.83 ± 0.41 | 81.46 ± 2.31 | 85.04 ± 0.92 |\n| Drop 40% | 76.39 ± 1.16 | 76.97 ± 1.43 | 75.72 ± 1.26 | 78.29 ± 1.79 |\n\nrel-amazon/user-churn (linear probing)\n\n| Model | Baseline | MAE-0.25 | CTR-0.25 | TVE-1 |\n| --- | --- | --- | --- | --- |\n| Drop 20% | 60.92 ± 0.61 | 65.62 ± 0.04 | 64.59 ± 0.01 | 70.17 ± 0.03 | \n| Drop 40% | 61.95 ± 0.67 | 66.96 ± 0.03 | 66.14 ± 0.04 | 71.31 ± 0.01 |\n\nrel-amazon/user-churn\n\n| Model | Baseline | MAE-0.25 | CTR-0.25 | TVE-1 |\n| --- | --- | --- | --- | --- |\n| Drop 20% | 70.09 ± 0.15 | 69.74 ± 0.21 | 69.98 ± 0.13 | 70.26 ± 0.10 | \n| Drop 40% | 71.35 ± 0.06 | 70.61 ± 0.24 | 71.20 ± 0.17 | 71.37 ± 0.20 |\n\nEven when the downstream task loosely depends on schema graph understanding, and when TVE is pre-trained on a noisy relational database, TVE’s task‐aware pre-training consistently outperforms standard SSL objectives. This demonstrates that our framework is robust to errors in schema understanding and retains its effectiveness across a range of noise levels.\n\n[1] Deep Variational Information Bottleneck.\n\n[2] Mutual Information Neural Estimation.\n\n[3] Deep feature synthesis: Towards automating data science endeavors.\n\n[4] 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs.\n\n[5] Griffin: Towards a Graph-Centric Relational Database Foundation Model."}, {"review_id": "Lpd2sfKYHK", "reviewer": "Reviewer_yYsy", "summary": "This paper introduces Task Vector Estimation (TVE), a pre-training framework for relational databases that constructs predictive signals by summarizing next-window temporal dynamics through schema graph traversals and SQL aggregations. Unlike traditional self-supervised methods that can capture spurious patterns, TVE explicitly models the underlying data generation process by creating task vectors that represent statistical summaries of future values relevant to downstream prediction tasks. The authors provide theoretical justification showing that representations incorporating both input features and temporal side-channel information retain more task-relevant signals than input-only approaches. Experimental results on RelBench demonstrate that TVE consistently outperforms traditional baselines, particularly in data-limited scenarios, while showing greater stability across hyperparameters and producing more transferable representations.", "questions": "- The authors don't talk about the computational complexity. Please discuss the complexity of task vector estimation\n- The authors should show results on some other benchmark other than Relbench. Please see the 4DBInfer benchmark.\n- The authors must include more recent baselines. One baseline is for 2023, and the other is for 2020. Please include recent models.\n- The authors should talk about the memory requirements while calculating task vector estimation. \n- Authors should show the results on cross-domain transfer. As SSL has shown its strength in cross-domain transfer.\n- Expressions like \"infinitely many possible downstream tasks\"  are imprecise. Please improve\n- The paper is very difficult to read. The paper introduces mathematical notation without sufficient explanation.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- The paper provides formal information-theoretic justification showing that incorporating side-channel information (task vectors) preserves more task-relevant signals than input-only methods\n- Experiments on the RelBench benchmark show that TVE consistently outperforms traditional pre-training baselines.\n\n# Weakness\n- The authors don't talk about the computational complexity. Please discuss the complexity of task vector estimation\n- The authors should show results on some other benchmark other than Relbench. Please see the 4DBInfer benchmark.\n- The authors must include more recent baselines. One baseline is for 2023, and the other is for 2020. Please include recent models.\n- The authors should talk about the memory requirements while calculating task vector estimation. \n- Authors should show the results on cross-domain transfer. As SSL has shown its strength in cross-domain transfer.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their feedbacks, and we have addressed each point below.\n\n> **[Q1] The authors don't talk about the computational complexity...memory requirements…**\n> \n\nTable 7 (Appendix B.6) shows each method’s pre-training time in seconds, demonstrating that TVE is relatively fast compared to others during pre-training. Next, we address computational complexity and memory requirements.\n\nBefore deriving the computational complexity, assume the following:\n\n- Number of rows per table is $O(N)$.\n- Number of timestamps is $T$.\n- Branching factor of relational entity graph (the input graph for learning) is $b$.\n- Number of aggregation functions is $r$.\n- Branching factor of the schema graph (the blueprint of the Relational Database) is $p$.\n- Number of hops for generating the task vector is $k$.\n\nWe derive the per-path cost (Steps 1–3) and then sum over all paths (Step 4). Remember that for every schema graph path, there are many paths that fall under the same schema graph path since a schema graph is just a blueprint of the input graph.\n\n1. Joining with the time table, which is a necessary step regardless of pre-training paradigm, costs $O(NT)$.\n2. Next, every join causes each row to fan out by a factor of $O(b)$, so the cost of joining up to $u$ successive times is:\n    \n    $$\n    O(NTb + NTb^2 + ... + NTb^u) = O(NT\\sum_{i=1}^u b^i) = O(NTb^u)\n    $$\n    \n3. Aggregating $r$ functions adds $O(rNTb^u)$ per path.\n4. There are $O(p^u)$ paths of length $u$ from the schema graph perspective, summing the cost:\n\n$$\n\\sum_{u=1}^k \\left[p^u \\times O(rNTb^u)\\right] = O\\left(rNT\\sum_{u=1}^k (pb)^u\\right) = O(rNTp^kb^k)\n$$\n\nTherefore, the cost blows up exponentially in $k$ through both $p^k$ and $b^k$, with linear dependence on $r$, $N$, and $T$. Since $p$ and $k$ are small in real schemas, the exponential blow-up is limited. Furthermore, a path is only meaningful if it contains at least one fact table (the table containing timestamps), so that we can construct the predictive pre-training objective. Finally, this operation is only done once, and is not involved during pre-training optimization.\n\nTo demonstrate our point, we report the time and peak memory usage for constructing task vectors. As we can see from the below table, this operation is feasible even with large-scale databases, where rel-amazon and rel-hm contain approximately 24M and 33M rows, respectively.\n\n| Model | rel-amazon/item | rel-amazon/user | rel-hm/item | rel-hm/user |\n| --- | --- | --- | --- | --- |\n| TVE-1 | 64.59 seconds / 39.98 GB | 90.17 seconds / 40.07 GB | 19.38 seconds / 4.62 GB | 11.74 seconds / 19.85 GB |\n| TVE-2 | 382.75 seconds / 40.03 GB | 652.85 seconds / 52.67 GB | 98.03 seconds / 16.15 GB | 871.26 seconds / 148.19 GB |\n\n> **[Q2] The authors should show results on some other benchmark…**\n> \n\nFirst, we would like to clarify the core difference between RelBench [1] and 4DBInfer[2]. RelBench focuses on inference an SQL-generated label attached to rows in the dimension table (the table that is static and doesn’t contain timestamps), while 4DBInfer focuses on cell-level prediction.\n\nWe would like to show that our proposed framework is performant regardless of the evaluation method. We pre-trained TVE on seznam, and evaluated downstream performance on an entity attribute classification task. We reported 4 metrics— Accuracy, Macro F1, Micro F1, and Mean Reciprocal Rank (MRR)—on both validation and test splits. We include two separate tables, one for linear probing and one for full-finetuning.\n\nseznam/charge (linear probing)\n\n| Model | V/Accuracy | V/Macro F1 | V/Micro F1 | V/MRR | T/Accuracy | T/Macro F1 | T/Micro F1 | T/MRR |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Baseline | 60.27 ± 1.22 | 28.89 ± 2.09 | 60.27 ± 1.22 | 76.96 ± 0.70 | 59.36 ± 1.00 | 27.99 ± 1.86 | 59.36 ± 1.00 | 76.48 ± 0.50 |\n| MAE-0.25 | 71.03 ± 0.04 | 44.50 ± 1.24 | 71.03 ± 0.04 | 84.00 ± 0.03 | 70.52 ± 0.05 | 43.32 ± 1.74 | 70.52 ± 0.05 | 83.76 ± 0.04 |\n| CTR-0.25 | 42.88 ± 0.09 | 12.74 ± 0.81 | 42.88 ± 0.09 | 64.56 ± 0.10 | 41.11 ± 0.32 | 10.01 ± 0.48 | 41.11 ± 0.32 | 62.79 ± 0.21 |\n| TVE-1 | 69.98 ± 0.21 | 44.20 ± 1.00 | 69.98 ± 0.21 | 83.44 ± 0.09 | 69.90 ± 0.16 | 43.39 ± 0.64 | 69.90 ± 0.16 | 83.55 ± 0.08 |\n\nseznam/charge\n\n| Model | V/Accuracy | V/Macro F1 | V/Micro F1 | V/MRR | T/Accuracy | T/Macro F1 | T/Micro F1 | T/MRR |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Baseline | 81.13 ± 0.21 | 62.02 ± 0.76 | 81.13 ± 0.21 | 89.96 ± 0.12 | 80.97 ± 0.50 | 63.43 ± 0.83 | 80.97 ± 0.50 | 89.91 ± 0.28 |\n| MAE-0.25 | 81.18 ± 0.15 | 62.00 ± 2.37 | 81.18 ± 0.15 | 89.99 ± 0.09 | 81.08 ± 0.32 | 63.31 ± 1.00 | 81.08 ± 0.32 | 89.97 ± 0.18 |\n| CTR-0.25 | 81.21 ± 0.17 | 63.28 ± 1.05 | 81.21 ± 0.17 | 90.01 ± 0.10 | 81.31 ± 0.41 | 63.69 ± 0.92 | 81.31 ± 0.41 | 90.10 ± 0.22 |\n| TVE-1 | 81.39 ± 0.22 | 63.92 ± 1.17 | 81.39 ± 0.22 | 90.11 ± 0.12 | 81.26 ± 0.29 | 63.82 ± 0.75 | 81.26 ± 0.29 | 90.07 ± 0.16 |\n\nWe can observe that the performance of MAE and CTR is not consistent across datasets, while TVE consistantly performs well and exhibits competitive performance (matching MAE for linear probing and matching CTR for full-finetuning). This is because both MAE and CTR have limitations. For example, CTR doesn’t offer a separable representation after pre-training (low performance during linear probing), while MAE isn’t as performant as CTR during full fine-tuning. Moreover, these gains come despite a conceptual mismatch—TVE was optimized for next-state prediction on dimension tables, whereas the downstream tasks operate on fact-table rows. Regardless of evaluation setting, TVE consistently performs well across two different benchmarks RelBench and 4DBInfer.\n\n> **[Q3] The authors must include more recent baselines…**\n> \n\nRelational‐database–specific pre‐training remains an emerging area, and to our knowledge no paradigms beyond generic adaptations have been proposed. The only relevant study is [3]. Because of the substantial engineering required to adapt SSL methods for relational data contexts, we restrict our comparison to two representative branches of SSL, including generative and contrastive pre-training (cf. Appendix B.2 for detailed changes for adaptation). Through our study, we demonstrate that these SSL methods aren’t sufficient because they do not explicitly model next-window dynamics. Meanwhile, Task-Vector Estimation (TVE) explicitly models the schema-driven factors that govern downstream label generation. As the field matures, we look forward to new pre-training methods and will include them in our future evaluations.\n\n> **[Q4] Authors should show the results on cross-domain transfer...**\n> \n\nUnfortunately, cross-domain transfer is beyond the scope of our study; we focus on theoretical understanding of underlying factors driving downstream label generation process, and how to derive a task-aware pre-training objective. We adopted the baseline proposed in [1] for our study, which leads to different encoders for different relational databases.\n\nMeanwhile, we believe that cross-domain transfer in the same database poses another interesting question. To better understand the separability of the pre-trained latent representation, we further conduct a linear probing experiment. TVE-1 (item) refers to the pre-trained model on the item's task vector, while TVE-1 (user) refers to the pre-trained model on the user's task vector.\n\nLinear probing experiment:\n\n| Model | item-churn-least100-spending | user-churn-top100-spending |\n| --- | --- | --- |\n| Baseline | 63.01 ± 4.98 | 78.36 ± 1.72 |\n| MAE-0.25 | 52.92 ± 0.79 | 83.86 ± 1.04 |\n| CTR-0.25 | 56.96 ± 2.07 | 81.63 ± 0.46 |\n| TVE-1 (item) | 78.06 ± 0.65 | 83.28 ± 0.63 |\n| TVE-1 (user) | 65.88 ± 2.25 | 86.56 ± 0.36 |\n\nSurprisingly, even when the task vector isn’t matched with downstream tasks’ entity type, it can still yield better performance than other generic SSL methods. For example, TVE-1 (user)  is only worse than TVE-1 (item) on item-churn-least100-spending, and TVE-1 (item) is as strong as MAE-0.25 on user-churn-top100-spending.\n\nConsequently, this experiment highlights the necessity to create an entity type-specific pre-training objective to maximize downstream performance gains.\n\n> **[Q5] Expressions like \"infinitely many possible downstream tasks\" are imprecise…**\n> \n\nWe agree that “infinitely many possible downstream tasks” are imprecise. We will rephrase the term to be “diverse downstream tasks.”\n\n> **[Q6] The paper is very difficult to read...**\n> \n\nBelow is how we will revise the manuscript to further enhance clarityr:\n\n- We will refactor Eq. 3 by introducing a compact notation for the set‐extraction step as follow:\n    \n    $$\n    \\mathbf{X}_v^{(t + 1)}  =  \\\\{ f\\_{\\mathcal{V}}(u) \\mid u \\in \\mathcal{N}_k^{(t+1)} (v) \\\\}\n    $$\n    \n    $$\n    y_v^{(t)}  = l(\\mathbf{X}_v^{(t + 1)})\n    $$\n    \n    where we already make assumptions that these values are from a specific relation $R$. We’ll move the full expansion to the appendix remark for readers who want the full notations.\n    \n- We agree that Figure 3 should be elaborated more, and we will add more details regarding Figure 3 in the camera-ready version.\n\nWe welcome any further clarity suggestions and thank the reviewers for their time.\n\n[1] RelBench: A Benchmark for Deep Learning on Relational Databases.\n\n[2] 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs.\n\n[3] Flaky Performances when Pretraining on Relational Databases."}, {"review_id": "zkcrKbeyXF", "reviewer": "Reviewer_r84r", "summary": "The authors introduce Task Vector Estimation (TVE), a novel pre-training framework that explicitly models task heterogeneity and temporal patterns by aggregating over schema traversal graphs and predicting next-window relational behavior. Information-theoretic analysis shows that, by incorporating side-channel task vectors, TVE retains more predictive, task-specific information than standard self-supervised methods, which rely only on input data and risk losing relevant signals due to task diversity and noise. The experimental results also demonstrates the capability using TVE has better downstream task performance on a variety of tasks and models.", "questions": "1. How do you ensure that the task vector maintains a consistent size across different traversals?\n2. The theoretical section discusses on SQL aggregation.  In practice, the SQL queries might cover much more operators. How does this information loss affect learning effectiveness?\n3. How complex are the expected SQL queries in practice? How many join hops and aggregations does the dataset typically involve?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality (4/5):\nThe paper demonstrates good research quality with well-designed experiments that directly address the stated research questions. The empirical results are comprehensive and robust, covering both data-limited and data-rich scenarios to show the method’s effectiveness under different conditions. Additionally, the inclusion of theoretical proofs adds rigor and strengthens the work’s technical foundation.\n\nClarity (2.5/5):\nThe overall structure of the paper is clear and logically organized. The mathematical notations are understandable, but some parts, especially Equation 3, could be simplified to improve readability for a broader audience. Figure 1 could be clearer if it focused more narrowly on the core pipeline and provided additional details specific to the proposed framework. Figure 2 explains how the task vector is constructed but does not clarify whether the traversal produces variable-length vectors; this point would benefit from further elaboration. Also, Figure 3 feels disconnected from the main text, it would help to explain its context and relevance more explicitly.\n\nSignificance (3.5/5):\nRelational databases remain critical data sources, especially for managing sensitive internal data within organizations. While large language model pre-training has shown great success, directly applying it to private relational data may not be the best approach. This work addresses that gap by exploring an alternative strategy tailored to relational databases, which could have practical impact.\n\nOriginality (4/5):\nThe paper offers a novel contribution by effectively bridging ideas from graph neural networks for encoding relational structures with model pre-training for downstream tasks. This integration of relational reasoning into pre-training design is both original and promising.", "was_revised": true, "final_justification": "The authors has well addressed my concerns.", "rebuttal": "We appreciate the reviewers’ thoughtful feedback and constructive suggestions. We addressed each point below.\n\n> **[W1] The mathematical notations are understandable, but some parts, especially Equation 3, could be simplified to improve readability for a broader audience. Figure 1 could be clearer if it focused more narrowly on the core pipeline and provided additional details specific to the proposed framework. Figure 2 explains how the task vector is constructed but does not clarify whether the traversal produces variable-length vectors; this point would benefit from further elaboration. Also, Figure 3 feels disconnected from the main text, it would help to explain its context and relevance more explicitly.**\n> \n\nWe agree that improving the clarity of our notation and figures will make the core ideas of TVE more accessible. Below is how we will revise the manuscript to address each point:\n\n- We will refactor Eq. 3 by introducing a compact notation for the set‐extraction step as follow:\n    \n    $$\n     \\mathbf{X}_v^{(t + 1)}  =  \\\\{ f\\_{\\mathcal{V}}(u) \\mid u \\in \\mathcal{N}_k^{(t+1)} (v) \\\\}\n    $$\n    \n    $$\n    y_v^{(t)}  = l(\\mathbf{X}_v^{(t + 1)})\n    $$\n    \n    so that the reader sees at a glance “apply labeling function $l$ to the set of next-window k-hop neighbor set,” where we already make assumptions that these values are from a specific relation $R$. We also simplify the $\\Delta t$ and represent the next-window variables via the upper-script $(t+1)$.  We’ll move the full expansion to appendix remark for readers who want the full notations.\n    \n- Regarding Figure 1, we would like to provide a brief overview of different pre-training paradigms. We agree that Figure 1 could be clearer with more details describing the figure, so that readers can understand the general intuitions of our work. We will add more descriptions regarding inputs and outputs for each pre-training paradigm in Section 3.\n- Regarding Figure 2, we will add a clarification on the variable-length vectors, as we answer below (cf. [Q1]).\n- Figure 3 illustrates the Markov Chain that driving the label generation process. Specifically, for any set $\\mathbf{Y}$ that is a generated by the parameter $\\mathbf{\\Theta}$, the downstream task is typically a labeling set function $l(\\mathbf{Y})$. Our pre-training framework attempts to capture sufficient statistics $T(\\mathbf{Y})$ of $\\mathbf{Y}$ for $\\mathbf{\\Theta}$, which ultimately leads to task-aware latent representation. We agree that Figure 3 should be elaborated more, and we will add more details regarding Figure 3 in the camera-ready version.\n\n> **[Q1] How do you ensure that the task vector maintains a consistent size across different traversals?**\n> \n\nOur proposed task vector construction operates similarly to Breadth First Search, and tasks extracted from a schema path is concatenated to the final task vector (cf. Figure 2). While it true that different path leads to different length of task vector, the final objective is an estimation of the final task vector. We can enforce different traversal paths to have consistent length by leveraging data compression techniques like PCA or MCA, as we outlined in Section 4.2.\n\n> **[Q2] The theoretical section discusses on SQL aggregation. In practice, the SQL queries might cover much more operators. How does this information loss affect learning effectiveness?**\n> \n\nThank you for raising this important point. We’d like to clarify complementary arguments showing why—and under what conditions—simple statistics can suffice, and how we mitigate the information loss when compared with sufficient statistics in practice.\n\n**Statistical sufficiency in common distributions.**\n\nIn many distributions, a small collection of aggregators is provably sufficient.  For example:\n\n- The **sum** of samples is sufficient for exponential, Poisson, and Bernoulli distributions.\n- The **max** is sufficient for a uniform distribution over a bounded interval.\n- The **mean** is sufficient for a normal distribution with known variance.\n\nWhenever the underlying dynamics match one of these canonical families, no information is lost by reducing the full set of observations to the corresponding statistic.\n\n**Enriching the task vector to minimize loss.**\n\nReal-world label‐derivation queries often deviate from textbook distributions.  To guard against this, we augment our task vectors with as many simple statistics—include `MEAN`, `MIN`, `MAX`, `SUM`, `COUNT`, and `STDDEV`  for numerical values, and `MODE`, `COUNT`, and `COUNT DISTINCT` for categorical values.  This “overcomplete” summary ensures that, even if some statistics are redundant, we capture a broader range of moment and order information, thereby shrinking the worst‐case information gap.\n\n**Set‐theoretic recovery and downstream expressivity.**\n\nFrom a set‐theoretic standpoint, certain combinations of simple statistics can perfectly reconstruct small multisets.  For instance, if exactly three purchases occur in the prediction window, then the triplet { `MIN`, `MAX`, `SUM` } uniquely determines the multiset.  More generally, accurate pre‐training on these statistics helps the model learn an implicit approximation of the original value‐set, which in turn improves its ability to approximate any downstream labeling function—since most such functions operate on those same sets (cf. Eq. (3)).\n\n**Empirical validation in tabular domains.**\n\nPrior work on **Deep Feature Synthesis** (DFS) [1] demonstrates that stacking joins with simple aggregations often matches more expressive graph‐based architectures on many benchmarks [2, 3].  While DFS features incur theoretical information loss relative to a full relational graph encoding, their competitive accuracy shows that richly engineered statistics can suffice in practice.  Our investigation is orthogonal: we systematically characterize **all** node‐level SQL‐generated labels as set functions and show how a task‐vector approach can act as surrogate representation for next-window sets. Additionally, our task vector includes similar aggregators to that of DFS.\n\nTogether, these points explain why simple statistics are not only theoretically grounded but also practically powerful—and how we effectively minimize any information loss.\n\n> **[Q3] How complex are the expected SQL queries in practice? How many join hops and aggregations does the dataset typically involve?**\n> \n\nBefore deriving the computational complexity, assume the following:\n\n- Number of rows per table is is $O(N)$.\n- Number of timestamps is $T$.\n- Branching factor of relational entity graph (the input graph for learning) is $b$.\n- Number of aggregation functions is $r$.\n- Branching factor of schema graph (the blueprint of the Relational Database) is $p$.\n- Number of hops for generating the task vector is $k$.\n\nWe would like to compute the computational cost at each step for constructing the task vector. First, we compute the cost taken for a single path from the schema graph perspective (Step 1 - 3), and then sum up to get the final cost (Step 4). Remember that for every schema graph path, there are many paths fall under the same schema graph path since schema graph is just a blueprint of the input graph.\n\n1. Joining with the time table, which is a necessary step regardless of pre-training paradigm, costs $O(NT)$.\n2. Next, every join causes each row to fan out by a factor of $O(b)$, so the cost of joining up to $u$ successive times is:\n    \n    $$\n    O(NTb + NTb^2 + ... + NTb^u) = O(NT\\sum_{i=1}^u b^i).\n    $$\n    \n    For $b > 1$, the geometric sum is scaled on the order of $b^u$:\n    \n    $$\n    \\sum_{i=1}^u b^i = \\frac{b^{u+1} - b}{b-1} \\approx \\frac{b^{u+1}}{b-1} = \\frac{b}{b-1}b^{u} = \\Theta(b^u).\n    $$\n    \n    Therefore, the total cost of for joining up to length $u$ is $O(NTb^u)$.\n    \n3. One aggregation of $r$ functions over the joined rows costs $O(rNTb^u)$, so every path (from the schema graph perspective) cost $O(NTB^u + rNTb^u) = O(rNTb^u)$ .\n4. There are $O(p^u)$ paths of length $u$ from the schema graph perspective, summing the cost:\n\n$$\n\\sum_{u=1}^k \\left[p^u \\times O(rNTb^u)\\right] = O\\left(rNT\\sum_{u=1}^k (pb)^u\\right) = O(rNTp^kb^k)\n$$\n\nTherefore, the cost blows up exponentially in $k$ through both $p^k$ and $b^k$, with linear dependence on $r$, $N$, and $T$. In practice, branching factor of the schema graph $p$ is typically small, and the schema graph is small, leading to small $k$. Furthermore, a path is only meaningful if it contains at least one fact table (the table containing timestamps), so that we can construct the predictive pre-training objective. Finally, this operation is only done once, and is not involved during pre-training optimization.\n\nTo demonstrate our point, we would like to report the time and peak memory usage for constructing task vector across datasets. As we can see from the below table, this operation is feasible even with large-scale databases, where rel-amazon and rel-hm contain approximately 24M and 33M rows, respectively. For our experiments, the maximum hop is 2, while the possible aggregation functions include `MEAN`, `MIN`, `MAX`, `SUM`, `COUNT`, and `STDDEV`  for numerical values, and `MODE`, `COUNT`, and `COUNT DISTINCT` for categorical values. In industry scenario, such cost is expendable.\n\n| Model | rel-amazon/item | rel-amazon/user | rel-hm/item | rel-hm/user |\n| --- | --- | --- | --- | --- |\n| TVE-1 | 64.59 seconds / 39.98 GB | 90.17 seconds / 40.07 GB | 19.38 seconds / 4.62 GB | 11.74 seconds / 19.85 GB |\n| TVE-2 | 382.75 seconds / 40.03 GB | 652.85 seconds / 52.67 GB | 98.03 seconds / 16.15 GB | 871.26 seconds / 148.19 GB |\n\n[1] Deep feature synthesis: Towards automating data science endeavors.\n\n[2] 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs.\n\n[3] Griffin: Towards a Graph-Centric Relational Database Foundation Model."}]

Author rebuttal:

Meta-review: This work introduces Task Vector Estimation (TVE), a novel pre-training framework for relational databases (RDBs) that explicitly models task heterogeneity and temporal dynamics. The approach is formalized via information theory, and the authors prove that task-informed representations retain at least as much relevant signal for downstream tasks. Experimental evaluation on RelBench benchmark demonstrate consistently performance improvements over traditional pre-training baselines.

Strengths:
1) The paper identifies a key limitation of current pre-training strategies for relational data which overlook task heterogeneity and temporal dynamics. 
2) The paper provides formal information-theoretic justification showing that representations incorporating side-channel information share at least as much information with downstream tasks as those without

Weaknesses:
1)  Clarity could be improved in some parts (see r84r).
2)  Consider including additional baselines (authors explain their decisions well in the rebuttal).

Overall:  The reviewers were overall positive on this work both before and after the rebuttal with only minor suggestions for improvement.
Decision: Accept (poster)