Title: A Pre-training Framework for Relational Data with Information-theoretic Principles
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph learning

tabular learning

relational database
Abstract: Relational databases underpin critical infrastructure across a wide range of domains, yet the design of generalizable pre-training strategies for learning from relational databases remains an open challenge due to task heterogeneity. Specifically, there exist many possible downstream tasks, as tasks are defined based on relational schema graphs, temporal dependencies, and SQL-defined label logics. An effective pre-training framework is desired to take these factors into account in order to obtain task-aware representations. By incorporating knowledge of the underlying distribution that drives label generation, downstream tasks can benefit from relevant side-channel information. To bridge this gap, we introduce Task Vector Estimation (TVE), a novel pre-training framework that constructs predictive supervisory signals via set-based aggregation over schema traversal graphs, explicitly modeling next-window relational dynamics. We formalize our approach through an information-theoretic lens, demonstrating that task-informed representations retain more relevant signals than those obtained without task priors. Extensive experiments on the RelBench benchmark show that TVE consistently outperforms traditional pre-training baselines. Our findings advocate for pre-training objectives that encode task heterogeneity and temporal structure as design principles for predictive modeling on relational databases. Our code is publicly available at https://github.com/quang-truong/task-vector-estimation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Task Vector Estimation (TVE), a novel pre-training framework that explicitly models task heterogeneity and temporal patterns by aggregating over schema traversal graphs and predicting next-window relational behavior. Information-theoretic analysis shows that, by incorporating side-channel task vectors, TVE retains more predictive, task-specific information than standard self-supervised methods, which rely only on input data and risk losing relevant signals due to task diversity and noise. The experimental results also demonstrates the capability using TVE has better downstream task performance on a variety of tasks and models.\n\nSTRENGTHS AND WEAKNESSES:\nQuality (4/5):\nThe paper demonstrates good research quality with well-designed experiments that directly address the stated research questions. The empirical results are comprehensive and robust, covering both data-limited and data-rich scenarios to show the method’s effectiveness under different conditions. Additionally, the inclusion of theoretical proofs adds rigor and strengthens the work’s technical foundation.\n\nClarity (2.5/5):\nThe overall structure of the paper is clear and logically organized. The mathematical notations are understandable, but some parts, especially Equation 3, could be simplified to improve readability for a broader audience. Figure 1 could be clearer if it focused more narrowly on the core pipeline and provided additional details specific to the proposed framework. Figure 2 explains how the task vector is constructed but does not clarify whether the traversal produces variable-length vectors; this point would benefit from further elaboration. Also, Figure 3 feels disconnected from the main text, it would help to explain its context and relevance more explicitly.\n\nSignificance (3.5/5):\nRelational databases remain critical data sources, especially for managing sensitive internal data within organizations. While large language model pre-training has shown great success, directly applying it to private relational data may not be the best approach. This work addresses that gap by exploring an alternative strategy tailored to relational databases, which could have practical impact.\n\nOriginality (4/5):\nThe paper offers a novel contribution by effectively bridging ideas from graph neural networks for encoding relational structures with model pre-training for downstream tasks. This integration of relational reasoning into pre-training design is both original and promising.\n\nQUESTIONS:\n1. How do you ensure that the task vector maintains a consistent size across different traversals?\n2. The theoretical section discusses on SQL aggregation.  In practice, the SQL queries might cover much more operators. How does this information loss affect learning effectiveness?\n3. How complex are the expected SQL queries in practice? How many join hops and aggregations does the dataset typically involve?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Task Vector Estimation (TVE), a pre-training framework for relational databases that constructs predictive signals by summarizing next-window temporal dynamics through schema graph traversals and SQL aggregations. Unlike traditional self-supervised methods that can capture spurious patterns, TVE explicitly models the underlying data generation process by creating task vectors that represent statistical summaries of future values relevant to downstream prediction tasks. The authors provide theoretical justification showing that representations incorporating both input features and temporal side-channel information retain more task-relevant signals than input-only approaches. Experimental results on RelBench demonstrate that TVE consistently outperforms traditional baselines, particularly in data-limited scenarios, while showing greater stability across hyperparameters and producing more transferable representations.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- The paper provides formal information-theoretic justification showing that incorporating side-channel information (task vectors) preserves more task-relevant signals than input-only methods\n- Experiments on the RelBench benchmark show that TVE consistently outperforms traditional pre-training baselines.\n\n# Weakness\n- The authors don't talk about the computational complexity. Please discuss the complexity of task vector estimation\n- The authors should show results on some other benchmark other than Relbench. Please see the 4DBInfer benchmark.\n- The authors must include more recent baselines. One baseline is for 2023, and the other is for 2020. Please include recent models.\n- The authors should talk about the memory requirements while calculating task vector estimation. \n- Authors should show the results on cross-domain transfer. As SSL has shown its strength in cross-domain transfer.\n\nQUESTIONS:\n- The authors don't talk about the computational complexity. Please discuss the complexity of task vector estimation\n- The authors should show results on some other benchmark other than Relbench. Please see the 4DBInfer benchmark.\n- The authors must include more recent baselines. One baseline is for 2023, and the other is for 2020. Please include recent models.\n- The authors should talk about the memory requirements while calculating task vector estimation. \n- Authors should show the results on cross-domain transfer. As SSL has shown its strength in cross-domain transfer.\n- Expressions like \"infinitely many possible downstream tasks\"  are imprecise. Please improve\n- The paper is very difficult to read. The paper introduces mathematical notation without sufficient explanation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduced Task Vector Estimation (TVE), a novel pre-training framework for relational databases (RDBs) that explicitly models task heterogeneity and temporal dynamics. Different from the traditional self-supervised learning methods that focus on input features, TVE incorporates knowledge about the underlying distribution driving label generation by constructing predictive supervisory signals through set-based aggregation over schema traversal graphs. The paper formalized their approach through information theory, demonstrating that task-informed representations retain more relevant signals for downstream tasks. Extensive experiments on the RelBench benchmark demonstrated that  TVE consistently outperforms traditional pre-training baselines, especially in low-data regimes.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nS1. The paper identified a key limitation in current pre-training strategies for relational data which overlook task heterogeneity and temporal dynamics. The paper argues that different tasks defined on RDBs depend on schema graphs, temporal dependencies, and SQL-defined label logics.\n\nS2. The information-theoretic analysis provides mathematical justification for why task-aware representations should outperform traditional self-supervised methods. Theorem 4.1 proves that representations incorporating side-channel information share at least as much information with downstream tasks as those without.\n\nS3. The paper included detailed ablation studies, sensitivity analyses, and linear probing experiments that provided insights into the stability and transferability of the learned representations.\n\nWeakness\n\nW1. While the paper claimed that TVE addresses task heterogeneity, the experiments mainly focus on performance on individual tasks rather than demonstrating transfer across multiple related tasks within the same database.\n\nW2. It will be nice if the paper can provide more discussion about how to design better approximations of sufficient statistics.\n\nW3. The paper does not analyze how well the chosen SQL aggregations approximate the ideal sufficient statistics of the underlying distribution.\n\nQUESTIONS:\n1. Please respond to W2.\n2. Please respond to W3.\n3. How sensitive is TVE to the quality of the schema graph representation? Would errors in schema understanding significantly impact performance?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}