Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data
Authors: Rishabh Ranjan
Valter Hudovernik
Mark Znidar
Charilaos I. Kanatsoulis
Roshan Reddy Upendra
Mahmoud Mohammadi
Joe Meyer
Tom Palczewski
Carlos Guestrin
Jure Leskovec
Venue: ICLR 2026 Poster (2026)
Keywords: foundation models
relational deep learning
relational data
transformer
Primary area: foundation or frontier models, including LLMs
TL;DR: A novel architecture for relational data that shows strong zero-shot abilities on unseen datasets after pre-training.

Abstract: Pretrained transformers readily adapt to new sequence modeling tasks via zero-shot prompting, but relational domains still lack architectures that transfer across datasets and tasks.
The core challenge is the diversity of relational data, with varying heterogeneous schemas, graph structures and functional dependencies.
In this paper, we present the Relational Transformer (RT) architecture,
which can be pretrained on diverse relational databases and directly applied to unseen datasets and tasks without task- or dataset-specific fine-tuning, or retrieval of in-context examples. RT (i) incorporates task specification via task table prompting, (ii) tokenizes cells with table/column metadata, (iii) is pretrained via masked token prediction, and (iv) utilizes a novel Relational Attention mechanism over columns, rows, and primary-foreign key links.
Pretrained on RelBench datasets spanning tasks such as churn and sales forecasting, RT attains strong zero-shot performance,
averaging 93% of fully supervised AUROC
on binary classification tasks
with a single forward pass of a 22M parameter model,
as opposed to 84% for a 27B LLM.
Fine-tuning yields state-of-the-art results with high sample efficiency. Our experimental analyses show that RT's zero-shot transfer leverages task context,
relational attention patterns and schema semantics. Overall, RT provides a practical path toward foundation models for relational data.
Code, models, data: https://github.com/snap-stanford/relational-transformer.

Published as a conference paper at ICLR 2026
RELATIONAL TRANSFORMER: TOWARD ZERO-SHOT FOUNDATION MODELS FOR RELATIONAL DATA
Rishabh Ranjan"; xx Valter Hudovernik"! Mark Znidar Charilaos Kanatsoulis", Roshan Upendra', Mahmoud Mohammadi Joe Meyer Tom Palczewski", Carlos Guestrin,, Jure Leskovec"
"tannford University. 'Kumo AI, University of Oxford, "SAP Labs LLC franjanr, guestrin, jure] esteefrraooord.. edu
ABSTRACT
Pretrained transformers readily adapt new sequence modeling tasks via zero- shot prompting, but relational domains still lack architectures that transfer across datasets and tasks. The core challenge is the diversity of relational data, with varying heterogeneous schemas, graph structures and functional dependencies. this paper, we present the Relational Transformer (RT) architecture, which can be pretrained on diverse relational databases and directly applied to unseen datasets and tasks without task- or dataset-specific fine-tuning, or retrieval of in-context examples. RT (i) incorporates task specification via task table prompting, (ii) tokenizes cells with table/column metadata, (iii) pretrained via masked token prediction, and (iv) utilizes a novel Relational Attention mechanism over columns, rows, and primary-foreign key links. Pretrained on RelBench datasets spanning tasks such as churn and sales forecasting, RT attains strong zero-shot performance, averaging 93% of fully supervised AUROC on binary classification tasks with single forward pass of 22M parameter model, as opposed to 84% for 27B LLM. Fine-tuning yields state-of- experimental analyses show that RT's zero-shot transfer leverages task context, relational attention patterns and schema semantics. Overall, RT provides practical path toward foundation models for relational data
results with high sample efficiency. Our
INTRODUCTION
Foundation models 3 49 have redefined natural language processing (NLP) ? and computer vision (CV) by demonstrating the effectiveness of general-purpose architectures across diverse domains and tasks. This success is driven by the transformer architecture [39], whose design makes large-scale pretraining possible and yields models that are powerful and broadly transferable. analogous breakthrough has not yet been achieved for relational data, despite relational databases being the dominant repository of structured enterprise information. Unlike sequences or images, relational databases comprise multiple interconnected tables with heterogeneous columns linked through primary-forcige key relationships. As a result, predictive signal is often scattered across rows, columns, linked tables, and time, making model design substantially more challenging.
Despite its difficulty, designing foundation model for relational databases is of utmost impor- tance [40]. Such model could adapt to new tasks and datasets via zero-shot prompting, few-shot learning or fine-tuning, enabling accurate predictions in cold-start and expertise-, compute- or data- constrained settings. More broadly, it would democratize the use of AI in enterprise contexts, where relational data is ubiquitous, by providing non-experts with accessible predictive tools and offering experts a strong initialization for further model development.
Prior work. Traditionally, tasks on relational databases have been solved using tabular models 35), which depend on manual, error- proor-prone, and costly feature engineering (25). The emerging area of relational deep learning (RDL) [15] addresses this challenge by developing end-to-end models that operate directly on relational databases. Prior RDL research has explored graph neural networks
Work done partly as an intern at SAP Labs L.C.. Work done while at Stanford University. Code, models, data: httpt:////tttttte com/ http://aaasssaaffo/reeeeeeeeeeee -nwn-taaaaaoooroooo.aaaaa

---

Published as a conference paper at ICLR 2026
(a) Schema
(c) Architecture
Predicted Cal Vidiuw
Visualizing Relational Attention
Tosk Definitonn
Per-datatyoe decoder nuni) Eact [oo iine
Predict toldl soles over nowt week.
Database
Sale
Tosk
Item
Tlime
Relational Transformer
User
UseriD (P)
Gendar
LiiitDD ItemD (F)
Store
Hemil (P)
Deta
Price
Task Schemo
Label
trepeazed
Uimee
Block
A Full Attention
(F) foreign key
Neighbor Astention
Relctional Attention
Feoture Amenttonn
.ayats
(b) Context Window
User
Sale
Item
Oetals
Tosk
Laball
Colamnn na
Per-datatype encoder frun| text boo tine
V
Task rows provide
Frozen LM embedding
i-cortiect labels
Target row wil- maskid coll
6V Quory
Natention cllowed
Set: of coll val.asa
without bost: ond information
Astentior disa lowad
Figure 1: (a) The schema specifies tables, columns, foreign keys and primary keys. The task definition is used to construct the task table, which includes labels one aims to predict (e.g.. customer churn labels). (b) The context window captures relevant information to predict the label column the target row, which is masked, excluding rows with later timestamps to prevent temporal leakage. See Fig, for a real example. (c) Cells correspond to tokens. Token embedding comprises trainable datatype- specific encoding of cell values and frozen language model (LM) embeddings of table/column names. Relational structure is modeled by our novel Relational Artention layers, where cell attends to (1) cells in the same column (columm attention), (2) cells in the same and F-PP linked rows (feature attention), and (3) P->F linked rows (neighbor attention).
and hybrid models 43 that combine GNNs and large language models (LLMs), leverage the relational structure. However, these architectures remain tightly coupled to specific schemas and fail to generalize to new databases. In contrast, existing tabular foundation models (TFMs) [1] 20 24 30 36 [3] generalize to unseen datasets but cannot capture the rich relational structure. An alternative is serialize relational databases into text formats such as XML, JSON, or CSV for processing with LLMs [44), or to flatten them into single joined table. However, these strategies suffer from scalability issues and distributional mismatch with the pretraining data of LLMs and TFMs. As a result, no existing method provides a viable backbone for building foundation models relational databases.
(GNNs) 34
transformers
Our contribution. In this paper, we introduce the Relational Transformer (RT) (cf. Fig. novel transformer design for relational databases that enables large-scale pretraining and zero-shot generalization across diverse domains and tasks. RT' introduces three key innovations. First, it represents each database cell as token, with embeddings constructed from its value, column name and table name. This cell-level tokenization allows all downstream tasks to be cast masked token prediction, thereby supporting flexible and scalable self-supervised learning. Second, RT augments the input with task-specific context via task table prompting, which enables zero-shot prediction across diverse schemas. While task rows provide "in-context labels", our setting is not few-shot explicit subgraph-label pairs are not required. Finally, in RT we develop our novel Relational Atlention mechanism - column attention to model value distributions within a column, feature attention mix information across cells in the same row and their linked parents. and neighbor attention to propagate signals along frimary-foreign key relationships - along with the standard self-attention for unrestricted pairwise interactions. Together, these mechanisms capture dependencies across cells, rows, and tables, explicitly leveraging the structure of relational databases.
We pretrain RT on relational databases from RelBench [34] and observe strong zero-shot transfer to tasks on unseen datasets. For example, average zero-shot AUROC binary classification tasks is 90.3% full supervised learning AUROC, and rises to 98.1% with continued pretraining on the target dataset (but not the task). For comparison, Gemma3-27B achieves only 83.7% of
2

---

Published as a conference paper at ICLR 2026
Zero Zero-Shot Shot Prompting
Supervised Fine. Tuning
AUROC
Fine tune on target donaset
AUROO vs stops
RT lounst
GNN
Fre trained R1
Fine- turnee
RIT
70
60
66
Gemmma
55
22M
27B
Pre troin with civerse sohemos and tosk definitions
Tetal eoutmppe from dostoett and toak
Fre Tne-traciredd RT gives strong O-ehot reaults
ded torget
Pre-trcined shows high learning officiency
dataee, and took
Figure 2: RT can be pretrained on data with diverse schemas and task definitions. Pretrained RT is accurate on new datasets and tasks with zero-shot prompting. See Fig. for an example zero-shot prompt. Dataset- and lask-specific fine-tuning of pretrained RT shows high learning efficiency.
full supervised learning AUROC with equivalent context information expressed as text, despite taking 10 more inference FLOOPs. Further, pretrained RT shows high learning efficiency, reaching similar performance as the second-best baseline in 10-100x fewer steps / training examples. These findings establish RT as a powerful model capable of robust zero-shot transfer in relational settings, while enabling rapid fine-tuning when supervision is available. Our results provide strong empirical evidence that relational databases, despite spanning diverse applications, share transferable patterns that can be captured through pretraining. Overall, this work marks significant step toward building foundation models for predictive tasks on relational data, moving beyond hand-crafted features and task-specific models to unified approach for relational modeling.
BACKGROUND: PREDICTIVE TASKS ON RELATIONAL DATA
2.1 RELATIONAL DATABASES
A relational database (RDB) is collection of lables linked through inter-table relationships. Each table is composed of rows, where every row is a set of cells, one for each column in the table. We define feature columns as the columns that contain numeric, text, and datetime information, and ID columns then define how rows are uniquely identified and connected across tables. Every table has primary key (P-key), and some include foreign keys (F-keys) referencing primary keys in other tables. This induces graph structure, where connections from foreign keys to primary keys are denoted as F-PP links, and the reverse incoming connections as P-F links (Fig.
Many RDBs are temporal, with timestamp columns that record when rows are created. Temporal information is crucial: if we want to predict whether a user will buy an item at time t, the model must only use information available before i, otherwise it risks temporal leakage. To prevent temporal leakage, modeling is conditioned only
that were created prior to the target the schema of an RDB (Fig. D specifies the tables with their columns and datatypes along with the relational structure. Because schemas vary widely, pretraining requires schema-agnostic architectures that directly incorporate multi-table structure through attention masks.
Finally,
2.2 PREDICTIVE TASKS
Masked token prediction (MTP). We focus on masked token prediction, where the goal is to predict the value of masked cell in the database, conditioned on the rest of the observed database. A broad class of important predictive tasks on RDBs can be framed as MTP, including (1) autocomplete tasks and (2) forecasting tasks.
Autocomplete tasks. Here the missing or masked value belongs feature column that already exists in the database. Consider the e-commerce schema with Users, Items, and Transactions tables. An autocomplete task might involve predicting a user's age in the Users table if the entry is missing, or inferring the category of an item in the Items table from its textual description and price. In both cases, the label comes from an existing feature column.
Forecasting tasks. Here, the goal is to predict something that has not yet happened. For example, in the e-commerce setting, we want to forecast whether given will churn in the next month, or predict the total revenue of a product in the upcoming quarter. Unlike autocomplete, the target values for forecasting do not exist in the original database and must be constructed from future rows.
3

---

Published as a conference paper at ICLR 2026
Task tables. To formalize forecasting tasks, we introduce a new task table. This table stores the forecasting labels, together with foreign keys linking to the relevant entities (c.g., user IDs or item IDs) and timestamp specifying the prediction horizon. For instance, a task table for churn prediction with timestamp showing the cutoff date.
might contain one row per user, indicating whether the user made a purchase within the next 30 days,
2.3 ZERO-SHOP RELATIONAL LEARNING
In domains like language, vision, robotics, biology, etc., foundation models benefit from world knowledge memorized during pretraining, as tasks of interest often share the same underlying reality. In contrast, relational learning is most useful for application-specific predictions on proprictary data unlikely appear pretraining. Thus, we focus on zero-shot relational learning, defined as predicting new targets on a new RDB with a new schema, without weight updates. Information about the new RDB is available only at inference time, solely through the context window. This can include
schema metadata, relevant rows from different tables, and their connectivity (e.g.. Fig. T (b), Fig. 6
and need not consist of explicit input-output pairs as required by most prior work [16]
While predictive tasks on RDBs are ubiquitous, only a small fraction justify the cost of custom model development. Zero-shot relational learning fills this gap, making data-driven predictive modeling accessible to small businesses, schools, individual users, etc., for example, to estimate future in-stock ingredients, flag students at risk of failing, provide autocomplete functionality in database-backer web application. Even at larger organizations, data scientists can quickly prototype task framing and modeling approaches such as feature selection, e.g., does removing location data hurt sales forecasting?. ? Further, high-stakes predictions, e.g., loan defaults, can be prioritized for custom modeling by relegating less critical ones, e.g., targeted offers to zero-shot relational learning.
RELATIONAL TRANSFORMER
We introduce the Relational Transformer (RT, Fig. D. a transformer architecture designed for relational data. The design of RT is guided by three core principles: (i) effectively capture rela- tional structure while preserving the natural symmetries of relational data, (ii) support flexible self-supervised pretraining, and (iii) enable zero-shot generalization across heterogencous schemas.
3.1 INPUT REPRESENTATION
RT introduces two key innovations in input representation: (i) task table prompting, where prediction tasks are represented as additional tables appended to the database, and (ii) cell-level tokenization, where each database cell is modeled as an individual token. Task table prompting augments the database input with task-specific context, enabling zero-shot predictions across diverse schemas. Cell-level tokenization then provides a unified view of relational data, allowing all downstream tasks to be cast as MTP and thereby supporting flexible and scalable self-supervised learning.
Task table prompting. For each task, we attach a dedicated task table the database. Only one task table is active at a time, ensuring that sampling is task-specific. Rows from the task table serve as seed rows for context construction, and from the model's perspective task tables are treated as ordinary relational tables. This allows downstream tasks to be seamlessly expressed in the same input space as pretraining.
Context window for zero-shot prompting. Given a seed row, RT constructs a context window of cells by expanding across primary-foreign key links. Following the intuition that relevant information concentrated in nearby hops, we apply a bounded-we wiith breadth-first search (BFS) with three modifications: (1) all parent rows (F=P) are always included, (2) child rows (P-+F) are subsampled with a wiStw bound w. and (3) rows with timestamps later than the seed row are excluded to prevent temporal leakage. Once row is selected, all of its non-missing feature cells are added to the context. A more detailed discussion of the sampling procedure is provided in App. A
Cell token encoding. A cell is represented by (v, c, t). where v is the cell value, is the column name, and is the table name. The value can be numeric, boolean, datetime, or text; other modalities (e.g., image) can be handled analogously to text.
Numeric/boolennt Normalize to obtain r = (u - pa)/o, E R, where H, and or are the column mean and standard deviation computed on the training split.
4

---

Published as a conference paper at ICLR 2026
Datetime: Convert to seconds and normalize globally: (v- /or, where pr and or RR are the global mean and standard deviation timestamps in the training split.
Text: Embed using frozen encoder glext; Eux Rdd..
Schema semantics are incorporated via a text embedding of the phrase "scolumn.name" of <table.name>" e.g.. "price of product", "age of user", using Erchema, The token embedding is x = War + W Exchemn (c, t), where Wa a is datatype-specifie and W is shared. For masked cells, the value embedding War is replaced with learned mask vector ma-
Permutation invariance for tables, rows and columns. To preserve the natural symmetries of relational data, RT does not incorporate positional encodings (26 22 23:2 2 Relational structure is captured by Relational Attention layers (.2) which invariant to the ordering of tables, rows and columns. This a crucial driver of sample-efficient generalization from limited schemas in pretraining (see App. for an expanded discussion).
3.2 RELATIONAL ATTENTION
The core of RT is novel Relational Attention mechanism in which the fundamental processing unit is the cell token. This formulation enables flexible pretraining via MTP, and stands in contrast Graph Transformers 10 31 46]. which tokenize the row level, but can be viewed as natural extension of Tabular Transformers (9] 20). By operating at the cell level, RT can explicitly model one-to-one dependencies between attributes across rows, columns, and tables, while also supporting zero-shot generalization across schemas. RT follows the standard transformer design, but augments each block with Relational Attention layers that effectively encode relational structure and inductive bias. Other architectural details (normalization, activations, etc.) follow the design choices of I.LaMA 38
The main operation in RT the scaled dot-product attention (SDPA) with masking, given by:
Aij if M.j = Mask (QKT. T; ;M) V. Mask(A; M) = -00 if M. 0 Vd
SDPA K, M: = Soofmmx
Here, Q E Raxd, K E Randd V E Raadd. are the query, key, and value matrices, and n the context length. M E (0, 1) is the attention mask, which controls token-to-token visibility. M[g, k] = 1 means the q-th token can attend to the k-th token, and M qg k] means cannot. For example, auto-regressive language models use causal attention mask, given by Mrausal [g. 1[k < 4). where 1(.} is the indicator function.
Relational Attention masks. Using specialized masks, we define four attention types: column, feature, neighbor, and global. For the cell corresponding to token let Col(t) be its column, Row(i) its row, and OutLinks(i) (i) the set of rows, possibly different tables, which are pointed to by foreign keys of Row(i).
Column attention: For any query token, this layer allows attention only to key-value tokens from the same column, resulting in the mask: Moolumi [g, k] = 1 (Col(k) = Co Column attention helps model the distribution of values in each column.
Feature attention: For any query token, this layer allows attention to key-value tokens from the same row, as well as from F-PP linked rows with the attention mask given by: M [g. k] = 1 (k)o((k) Row(g) v Row(k: OutLinks(q)) Feature attention is equivalent to row-wise attention after joining each table with its parent tables, and enables feature mixing for entities.
Neighbor attention: For any query token, this layer allows attention key-value tokens from P-+F linked rows, defined by the attention mask: Mueighhor [g, k] = 1 (Row(4) E (ktL)nks(k)). Neighbor attention captures information from incoming links an entity, enabling the model to aggregate signals from its child rows. This module acts analogously to message- pasesegaannngggg in GNNs.
Full attention: Finally, standard bidirectional layer allows full pairwise interactions: (f k] 1. Full attention confers the expressive power of standard Transformers, com-
plementing the relationally constrained layers above.
Taken together, the proposed attention layers provide the model with an explicit encoding of database structure with high expressive power (App. (). These layers are implemented with sparse attention masks, and compiled to efficient FlashAttention baasee 6 kernels using FlexAttention g The
proposed transformer block in RT' is summarized in Alg. 2
5

---

Published as a conference paper at ICLR 2026
3.3 OUTPUT DECODING AND TRAINING OBJECTIVE
Cell decoders/ prediction heads. An output token embedding from the transformer backbone is processed by multiple cell decoders (also called prediction heads), one for each datatype, into cell representation r'. The decoder to select for final prediction depends on the task type, or equivalently on the datatype of the masked cell. Binary classification corresponds to the boolean datatype, and regression corresponds to the numeric datatype.
Loss. Having separate decoders for different datatypes allows us to use custom loss functions for each task type. In this work, we only mask cells in boolean or numeric columns as RelBench tasks are either binary classification or regression. For masked cell e with value v, representation (as defined in $ 3.1.. and decoder output r', we apply HuberLoss(r, r") for regression and binary cross-entropy loss (r > 0}, r') for binary classification. The overall loss is the mean over all masked cells in the batch. This formulation is used in both pretraining and fine-tuning, ensuring consistency between objectives and contributing to sample efficiency.
4 RESULTS
4.1 EXPERIMENTAL SETUP
Datasets and tasks. For all our experiments, we use datasets and tasks from RelBench [34 18 RelBench vI contains 7 relational databases from diverse domains, namely, rel-amazon, rei hm, rel-stack ck, rel-ava to, rel-event, rel-trial 1 and rel-f1. Each dataset has multiple forecasting tasks defined on it, which are either binary classification or regression. For example, for rel-amazon, there are forecasting tasks (user-churn, i item-churn, user-ltv and item-ltv), of which the first 2 are binary classification, and the last 2 are regression. We also define autocomplete tasks, both binary classification and regression, on all datasets (App. D) See App. C for detailed characteristics which make RelBench suitable for pretraining and evaluation.
RelBench provides standardized temporal splits for tasks, as well as cutoff timestamps for the databases. We pretrain and fine-tune only on the training splits of the tasks, using database rows only up to the training cutoff timestamps. To pick the best checkpoints, we use the validation task splits and validation cutoff timestamps. We report the test set performance, for both learning curves and tables. We evaluate only on the forecasting tasks, as autocomplete tasks are not part of the standard RelBench benchmark. We skip rel-event, it has been found to have temporal leakage issues.
Leave-one-DB-out pretraining. The strongest setting demonstrate transfer is when both the dataset and task are unseen. We expect cross-dataset transfer despite disparate application domains as databases share structural commonalities, e.g., dimension tables, fact tables, hubs and tripartite structures 4 as well as semantic similarities, e.g., similar user behaviors when reviewing books (rel-ama zon), commenting on posts (rel-stack), attending events (rel-event) purchasing clothes (rel hhm), and clicking on ads (rel-aviton to), which RT could learn to identify and exploit. Due to the limited number of different datahases, we pretrain separately for each target dataset on all tasks from all other datasets. We select the best validation checkpoint [32 separately for each target task as we found significant overfitting during pretraining due to limited pretraining tasks.
Architecture details. We use a 12 layer transformer with hidden dimension 256 and 8 attention heads per layer. We use gated MLPs with SiLU activation, as found in the LLaMA architecture (38), with hidden dimension 1024. For text embeddings, we use the Mvn 44] model from CentenceTransffomees (33,, which produces 384 dimensional embeddings. With this configuration, the architecture has about 22M trainable parameters.
Training details. We pretrain RT for 50k steps at context length of 1024, with a batch size of 256, Adam optimizer with weight decay 0.1, and a peak learning rate of 10-3, with linear warmup from zero for the first 20% of training, and linear decay to zero for the remainder. On each downstream task, we fine-tune for 33k steps with the same context length, batch size and optimizer as above, but with a constant learning rate of 10-4 and no weight decay. One pretraining (fine-tuning) run takes around 2 hours (1.5 hours) on A100 GPUs at BF precision, with a training throughput of around 8 batches/second or 2M tokens/second.
Baselines. We compare RT against tchema-agnostiee methods, which can be pre- pre-trained on diverse databases, and schema-specific ones, which cannot. Schema-agnostic baselines include Griffin 42

---

Published as a conference paper at ICLR 2026
Binary Classification (mean over 5 downstream tasks)
Regression (mean over 4 downstream tasks)
80
00-shot
1--hoot
Test AUROC (96)
75
70.88
Test R2 (%)
Pretrained RT (ours)
65
168.5
Untrained RT (ours)
20
Untrained RDL-GNN
10
60
Pretrained Griffin
55
Untrained Griffin
0
10
10a 10a
10a
10
10a
10a
10a
10a
10a
Fine-tuning steps (batch_Se size
256)
Fine-tuning steps + (batch size 256)
Figure 3: Test set learning curves up to 32k fine-tuning steps (8M training examples, including repetitions). Averaging is done over tasks which do not show overfitting. X-axis is on log-scale. The first point on each curve is the zero-shot performance. Target datasets and tasks are unseen during pretraining. Pretraining data is the same for both RT and Griffin. Pretrained RT is best overall, and untrained RT catches up towards the end.
pretrained and fine-tuned with the same setup RT and matched in parameter count, and LLM, where we prompt Gemma3 [37] models with instructions followed by a text-serialized database subgraph identical to that used for RT [44). Schema-specife baselines include RDL-GNN (4), which encodes rows into node vectors via tabular encoders and aggregates across tables with GNN layers, ReILLM 43 which combines GNN encoders with LLMs in strieval-augmented framework, ReIGNN 4 and ReIGT [II] (latter 2 are in App. F). The non-neural Entity baseline predicts the mean of the past labels for the target entity. See App. [ for more details.
4.2 LEARNING EFFICIENCY
The key promise of pretraining efficient adaptation new datasets and tasks. Adaptation can be via zero-shot prompting, few-shot learning or fine-tuning, Since we pretrain only on a handful of different databases, we focus on efficient fine-tuning in this work. Remarkably, we see the emergence of zero-shot abilities in RT, which we compare with relevant baselines in 4.33
Setup. In Fig. we show learning curves for supervised fine-tuning downstream tasks for RT (pretrained and untrained), RDL-GNN (untrained), and Griffin (pretrained and untrained). We average the curves over tasks which do not show overfitting. For all these models, use the same fine-tuning setup as described above. Full fine-tuning results for all tasks, are in App.
Observations. Our first observation is that RT exhibits strong zero-shot performance, demonstrating effective transfer from pretraining to unseen databases and tasks. While Griffin also benefits from zero-shot transfer, RT consistently outperforms it. The pretrained RT consistently maintains the gap in performance based on its strong initialization and is only caught up by the untrained RT and RDI.-GNN on classification tasks after extensive fine-tuning. Notably, fine-tuning from a pretrained checkpoint shows a small dip in performance in the very early steps, which we conjecture might be due to the model transitioning from shot, in-context-labels- and semantics- a data-driven prediction regime. Finally, while RDL-GNN initializes faster than untrained RT, the latter overtakes it after few training steps (2 on classification and zs 100 regression).
regime to
4.3 ZERO-SHOT PROMPTING
Setup. In Tabs. and 2 we report zero-shot results. The target task is always unseen. We report results for when the target dataset also unseen, as well as after doing some continued pretraining on the target dataset. Columns "No" and "Yes" respectively in Tabs. D2.. Pretraining data for Gemma models is unknown, but likely includes similar public datasets (e.g., Fl, Amazon and StackExchange). hence they are grouped under "Maybe". For RT and Gemma, we construct the input context using our sampling algorithm. For Griffin, we adapt its sampling procedure to include rows from task tables, making it consistent with our task table prompting to enable zero-shot capabilities. For ReILLM we use their own prompt construction. Both ReILLM and Gemma baselines are additionally provided with textual task descriptions and instructions.
Observations. RT demonstrates non-trivial zero-shot performance on all tasks. On classification, attains the best average AUROC and is the only method consistently beat the entity mean baseline.
7

---

Published as a conference paper at ICLR 2026
Target DB E pretraining?
Maybe
No
Yes
Dataset
Task
Gemma
Gemma
Gemma
Entity Mean
Griffin
(ours)
Rel LLM
Grifin
(ours)
RT
Parameter count
4B
128
27B
22M
22M
3B
22M
22M
L.em: churn
62.1
55.0
42.1
73.0
69.0
70.9
64.1
71.9
73.3
re lol--mmaee zon
user-ohorn
58.
54.7
50.5
64.4
62.3
64.0
60.1
64.1
66.1
uaer-olicks
54.6
59.5
59.8
44.7
45.9
59.5
45.9
60.9
cel-avite
user-visits
60.1
57.9
62.7
60.7
60.7
61.8
56.2
62
62.6
rel-fl
driver-dnf
81.2
71.8
57.7
rel-f1
driver-top3
84.6
90.5
91.4
85.0
82.5
89.3
70.6
81.8
89.3
rel-hm
uaer-churn
59.8
47.1
48.7
60.2
62.8
56.0
60.4
63.3
rel steck
badge
79.1
79.8
90.0
66.2
73.5
80.1
62.1
82.3
81.1
rel-szack
uaer-engage
65.9
67.8
78.0
83.5
77.5
75.7
69.5
89.4
86.9
trial
s.udy out 52.6
57.4
57.2
51.8
57.2
51.6
AUROC
66.7
Table I: Zero-shot test AUROC (%) for 10 binary classification tasks. Higher is better. Random/- majority baseline 50.0. For ReILLM we use their own prompt construction. Other baselines have equivalent database subgraphs. Gemma and ReILLM additionally include dataset and task descriptions, well as natural language instructions. The target task is never seen during pretraining.
Dataset
Tisk
MLP-only
Full
Target pretraining"
FT
Eatity Mean
Emhedder (frizen
GNN
RT(U))
RT(P)
(RT)
Datasct L
Task 4
Grif.
(ours)
Grif.
(ours)
rel-amz
item-churn
79.6
79.9
81.6
83.4
rel-amz
user-chura
67.6
67.0
68.5
70.8
zel amazon
54.2
20.1
32.5
20.1
32.2
rel-hm
user-chura
67.5
68.6
69.3
70.5
ael-amazon
user-ltv
19.9
20.6
36.9
24.4
38.3
rel-ack
user-badge
85.5
88.7
sel-avito
ad-ctr
rel-sck
user-engag
89.1
87.8
89.3
90.2
rel-f1
dir ver-pos
38.2
0.7
52.4
4.6
58.7
rel-hm
item-sales
1.8
2.7
14.0
2.5
30.9
Mean AUROC
77.9
78.0
79.1
80.7
rel-stack
post-votes
43.7
27.4
33.9
27.1
35.0
rel-amz iten-ltv
1.6
43.3
56.5
36.8
rel-trial
alte-succ
-6.4
1.4
4.5
2.6
5.1
rel ama usez 1tv
13.6
31.8
46.8
17.9
zel trial
study
8.1
item-sales
45.7
Mean R²
19.3
8.9
22.7
10.1
26.4
rel s.k post-voles
14.1
37.4
42.6
37.1
Table 2: Zero-shot test R² (%) for 8 regression tasks.
R
10.9
32.5
46.2
Higher is better. Global mean baseline is 0.0. Setup is same as Table 1 LLM baselines are poor (App. I.3
Table 3: RT as feature extractor. GNN, RT(U) are untrained, RT(P) is pretrained.
On regression, where LLM baselines fail provide meaningful predictive accuracy, RT is the only model to consistently achieve positive R²2 and surpass the EntityMean baseline on average.
4.4 RELATIONAL FEATURE EXTRACTION
Setup. We use the masked token representation before the decoder layer from a pretrained RT and fine-tune (FT) only 2-layer MLP over it for downstream tasks on unseen datasets. Results are in Tab. along with untrained GNN and RT, as well as full FT for comparison.
Observations. MLP-only FT has 4.3% (abs.) better avg. R² and only 1.6% (abs.) worse avg. AUROC than full FT, despite being orders of magnitude cheaper. Even untrained RT embeddings are sometimes competitive, and much better than untrained GNN embeddings.
4.5 COST-QUALITY TRADE- OFF
Setup. Fig. shows how zero-shot performance varies with test-time compute in terms of time and memory. We vary the context length for RT (a transformer), and fanout:hops for Griffin (a GNN), at inference for models pretrained at max context -02e--024 for RT, 20:2 for Griffin (default in [42].
Observations. RT dominates Griffin. This is despite RT being a cell-level transformer, and Griffin being a row-level GNN, as (1) Flex kernels used RT are highly efficient [8) (2) Griffin incurs cell-level penalty in the encoding step, and, (3) at equal parameter count and similar context sizes, memory cost is similar. RT's performance degrades gracefully, showing a drop of only 4% absolute AUROC and 7% absolute R² with 25% lower time and 20% lower memory.
8

---

Published as a conference paper at ICLR 2026
Avg. 0-shot AUC (%)
Avg. O-shot R (%
20
102a
Avg. D-shot AUC
1024
Avg. O-shot (9)
20
768
512
512
102
20:2
10.2
20:2
66
256
256
10:1
10
10:1
20.2
10:1
10
10:1
10.2
20:2
64
40
10
30
0.2
0.4
0.2
Inference time A100 (ms/batch, batch_ batch_size-3330 size=32)
Inference memory A100 (GB/batch, batch_ batch_size-30 size=32))
(parameterized by context length)
Griffin-22M (parameterized by fanouthhops)
Figure 4: Cost-quality trade-off w.r.t. varying context sizes at inference. Pretraining was done at the max context size. Average is over 10 classification and regression tasks.
80
Zero-shot
Finc-tuned
Test (96)
Ablated
Test AUROC
elf
reg
clf
reg
75
20
nome
70.1
22.8
77.2
33.2
70
names
20.5
77.5
33.2
self labels
53.8
77.1
26.7
109
10
102
102
10"
10
10a
10a
other labels
70.6
22.9
77.4
31.0
Fine-tuning steps
Rine-tuning steps
Table 4: Mean AUROC (%) and R² (%)
With self labels (pretrained) self labels (pretrained)
self labels (untrained)
transfer without self labels
on ablating context window components for classification (clf) and regression (reg)
Figure 5: Pretrained RT shows transfer even without self labels. Setup is same as in Fig.
tasks. Individual numbers are in App. G
ABLATION STUDIES
Here we ablate various context window and architecture components RT, seeking to understand their role in enabling zero-shot transfer and contrasting that with their impact on fine-tuning.
5.1 CONTEXT WINDOW ABLATIONS
Setup. In Table we investigate the impact of shuffling column and table names, removing past task rows which refer to the target entity (self labels in short) and removing task rows from other entities (other labels). We use the same pretrained checkpoints as in We provide breakdown of self and other labels in 1024 cell contexts in Tab. 12
Observations. We find that zero-shot transfer is driven primarily by the model's ability to leverage past labels of the target entity. From EntityMean baseline results in $ we know that RT is learning more complex functions than averaging self labels. Fig. 5 investiggates whether the model learns other transferable patterns besides self label ones. The positive transfer regions in the plots establish that this is indeed the case, but self labels do constitute the dominant share of transfer from pretraining even during initial steps of fine-tuning. Second, we find that the model indeed leverages table/column names for zero-shot transfer, as wrong names lead to consistent degradation. After full fine-tuning, context ablations have little effect, except that removing self labels significantly degrades performance on regression tasks.
5.2 RELATIONAL ATTENTION LAYER ABLATIONS
report the effect of removing column, feature, neighbor, or full attention layers on regression tasks. Classification tasks (App. H) showed minor differences. We maintain the overall parameter count by increasing the number of transformer blocks.
Setup. In Tab.
Observations. Column attention has the highest impact on zero-shot transfer. However, on fine- tuning the impact is less pronounced, and the impact of feature and neighbor attention is more significant. Removing full attention has the least impact in both zero-shot and fine-tuned settings, even significantly improving the results
tasks, perhaps surprisingly.
In App. H.I we also investigate the order relational attention layers, finding that the specific order of relational attention layers has little effect on performance.

---

Published as a conference paper at ICLR 2026
Datasct
Task
Zero-shot
Fine-tuned
Ablated attention
feat
nhr
full
col
fent
nhr
full
rel-amazon
itam-lty
33.2
29.8
36.8
rel-ama zon
user-ity
36.4
6.5
30.4
34.6
35.0
47.4
47.7
45.6
46.3
46.4
xel avito
ad-ctr
-1.4
9.2
8.5
5.3
-7.1
16.6
2.1
10.6
rel-fl
driver-pos
54.7
36,9
37.6
49.4
50.1
51.6
51.4
42.0
48.9
39.5
rel-hm
itam-sales
14.0
11.0
4.9
10.5
13.9
39.0
38.1
34,8
37.0
39.6
rel stack
post
32.4
28.7
36.5
37.2
36.4
rel-trial
site-sune
5.2
4.7
7.9
6.7
4.9
6.4
4.7
8.8
6.7
7.9
rel-trial
study-adv
Mean R
22.8
11.9
18.8
22.0
23.7
33.2
31.2
32.2
31.4
32.7
Table 5: Ablation studies on the attention layers of RT. R² (%) for 8 regression tasks, Higher is better. Global-mean baseline is 0.0. col, feat, nbr, full denote that column-, feature, neighbor- full- attention layers are absent respectively. Total parameter count kept constant by increasing the number of layers. Shading is proportional to difference from the none column. Classification numbers (App. HD show minor differences.
6 RELATED WORK
graph transformers 28 12 and hybrid GNN-LLM models [43 have been proposed for relational deep learning, but these methods are schema-specific and lack transferability across databases. Wydmuch et 44 investigate predictive modeling with LLMs, yet this approach reduces the relational database to sequence and is constrained by small context windows that are not tailored to relational structure. Tabular foundation models [19 30 24 (36] 20 demonstrate the benefits of pretraining and in-context learning, but are confined to single-table settings and cannot capture multi-table relational structure. The most related efforts are KumoRFM [66] and Griffin [42) KumoRFM cannot operate in zero-shot settings, and its technical details remain undisclosed. Griffin aggregates at the row level before GNN-based propagation. In contrast, RT introduces a schema- agnostic, cell-level architecture with Relational Attention masks, enabling pretraining diverse databases and robust zero-shot transfer. See App. B for an expanded discussion.
GNNs 15
DISCUSSION AND CONCLUSION
It is striking that pretraining only 6 datasets yields such strong zero-shot transfer on completely unseen datasets, especially considering that the datasets in RelBench are quite diverse. Our ablation studies in $5 have uncovered the following key components that enable zero-shot transfer, in order of importance: (1) time-series forecasting from past task rows for the target entity, (2) column attention with the help of Relational Attention generalize new column distributions, (3) feature attention, both in the same table and in parent tables, with the help of Relational Attention to generalize to new entity types, (4) schema semantics from table and column names, (5) in-context learning from task rows for other entities.
Limitations of RT include inability handle recommendation link prediction tasks. Further, it does not incorporate the names of primary-key and foreign-key columns, which often carry useful semantics. RT also does not disambiguate between different foreign key columns: for instance, in a product table with both buyer-id and veller-id d as foreign keys to the user table, the model cannot distinguish which user bought and which sold the product. Extending RT handle such cases, and more generally to support link prediction, remains an important direction for future work. Finally, more advanced cell encoders can be explored, including graph positional encodings, to enhance performance for supervised fine-tuning settings in large-data regimes.
To conclude, we introduced the Relational Transformer (RT), general architecture that advances foundation models relational data. RT introduces three key innovations: (i) cell-level tokenization that unifies diverse predictive tasks as masked token prediction, (ii) Relational Attention layers that explicitly capture and generalize column, row, and primary-foreign key structures, and (iii) task table prompting that enables zero-shot prediction across heterogeneous schemas. Through pretraining on diverse databases, RT achieves strong zero-shot transfer, rapid fine-tuning and state-of-the-a art results on classification and regression tasks. These advances demonstrate that relational databases share transferable patterns and position RT a foundation for general-purpose relational modeling.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Matthias Fey, Harshvardhan Aggarwal, Vijay Prakash Dwivedi, Michael Bereket, Marcel Roed, Joshua Robinson, Martin Jurkovic, Fengyu Li, Justin Gu, Zoe Ryan, Sam Thelin, Johannes Hoffart, Maximilian Schambach, Andrew Pouret, Viswa Ganapathy, Tassilo Klein and Mark Li for discussions, feedback and other help.
REFERENCES
[1] Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Zhiyi Li, Gabriela Moisescu-Pareja, Oleksandr Dymov, Samuel Maddrell-Mandder Callum McLean, Frederik Wenkel, Luis Müller, et al. Towards foundational models for molecular learning on large-scale multi-task datasets. In The Twelfth International Conference on Learning Representations, 2024.
[2] Mitchell Black, Zhengchao Wan, Gal Mishne, Amir Nayyeri, and Yusu Wang. Comparing graph transformers via positional encodings. In International Conference Machine Learning, pp. 4p03-43441 PMLR, 2024.
[3] Rishi Bommasani et al. On the opportunities and risks of foundation models, 2022. URL https::/arxiv.omm http:/xwwwvvrrab/////bbbaaa/ 2210 0
[4] Tianlang Chen, Charilaos Kanatsoulis, and Jure Leskovec. ReIGNN: Composite message passing for relational deep learning. In Forty-second International Conference Machine Learning, 2025.
[5] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
[6] Tri Dao, Dan Fu,, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16444-16359, 2022.
[7] Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-411888 2019.
[8] Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels, 2024. URL https://
05496
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[10] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021.
[11] Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico López, Charilaos I. Kanatsoulis, Rishi Puri, Matthias Fey, and Jure Leskovec. Relational graph transformer, 2025. URL
https://arx:t wwwpxxiv..rggaee abs wttpg//a.......a.2 10960
[12] Vijay Prakash Dwivedi, Charilaos Kanatsoulis, Shenyang Huang, and Jure Leskovec. Relational deep learning: Challenges, foundations and next-generation architectures. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 5999-6009, 2025.
[13] Dmitry Eremeev, Gleb Bazhenov, Oleg Platonov, Artem Babenko, and Liudmila Prokhorenkova. Turning tabular foundation models into graph foundation models, 2025. URL actps:/ arxiv. arxiv.org/pdff255500 20906v1
11

---

Published as a conference paper at ICLR 2026
[14] Bahare Fatemi, Jonathan Halerow, and Bryan Perozzi. Talk like graph: Encoding graphs for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[15] Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Position: Relational deep learning graph repre- sentation learning on relational databases. In Forty-first International Conference on Machine Learning, 2024.
[16) Matthias Fey, Vid Koeijan, Federico Lopez, Jan Eric Lenssen, and Jure Leskovec. Kumorfm: A foundation model for in-context learning on relational data, 2025.
[17] Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. In The Twelfth International Conference on Learning Representations, 2024.
[18] Justin Gu, Rishabh Ranjan, Charilaos Kanatsoulis, Haiming Tang, Martin Jurkovic, Valter Hudovernik, Mark Znidar, Pranshu Chaturvedi, Parth Shroff, Fengyu Li, and Jure Leskovec. RelBench v2: A scale benchmark and repository for relational data, 2026. URLhttps
//arxiv.org/eeee abs 602. 12606
[19] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. The Eleventh International Conference Learning Representations, 2023.
[20] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tihor Schirrmeister, and Frank Hutter. Accurate predictions small data with tabular foundation model. Nature, 637(8045):119-32)) 2025.
[21] Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng. Percy S Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. Advances in Neural Information Processing Systems, 36: 13302-16317, 2023.
[22] Yinan Huang, William Lu, Joshua Robinson, Yu Yang. Muhan Zhang, Stefanic Jegelka, and Pan Li. On the stability of expressive positional encodings for graphs. In The Twelfth International Conference Learning Representations, 2024.
[23] Charilaos I Kanatsoulis, Evelyn Choi, Stephanie Jegelka, Jure Leskovec, and Alejandro Ribeiro. Learning efficient positional encodings with graph neural networks. The Thirteenth Interna- tional Conference on Learning Representations, 2025.
[24] Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. Carte: Pretraining and transfer for tabular learning. In International Conference on Machine Learning, pp. 23843-23866. PMLR, 2024.
[25] Hoang Thanh Lam, Beat Buesser, Hong Min, Tran Ngoc Minh, Martin Wistuba, Udayan Khurana, Gregory Bramble, Theodoros Salonidis, Dakuo Wang, and Horst Samulowitz. Auto- mated data science for relational data. In 202/ IEEE 37th International Conference Data Engineering (ICDE), pp. 2689-2692. IEEE, 2021.
[26] Derek Lim, Joshua David Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In The Eleventh International Conference on Learning Representations, 2022.
[27] Aliakbar Nafar, K Brent Venable, and Parisa Kordjamshidi. Learning vs retrieval: The role of in-context examples in regression with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp.
2025.
[28] Jakub Peleška and Gustav Šír. Transformers meet relational databases, 2024. URL https
7/arxiv Taaxii.org/abl... abs 2412. 05218
12

---

Published as a conference paper at ICLR 2026
[29] Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for Ilms, 2024.
URthttps::/ http//axivvvv.. abs/ 2402. 05862
[30] Jingang QU, David Holzmülleer, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. In Forty-second International Conference on Machine Learning, 2025.
[31] Ladislav Rampášck, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:1 14501-14515, 2022.
[32] Rishabh Ranjan, Saurabh Garg, Mrigank Raman, Carlos Guestrin, and Zachary Lipton. Post-hoc reversal: Are we selecting models prematurely? Advances in Neural Information Processing Systems, 37:914600--- 91491, 2024.
[33] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLL-IJCNLP), pp. 3982-3992, 2019.
[34] Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan Eric Lenssen, Yiwen Yuan, Zecheng Zhang, et al. Relbench: A benchmark for deep learning on relational databases. Advances Neural Information Processing Systems, 17:21330-21344 2024.
[35] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-99 90, 2022.
[36] Marco Spinaci, Marek Polewczyk, Johannes Hoffart, Markus C. Kohler, Sam Thelin, and Tassilo Klein. Portal: Scalable tabular foundation models via content-specific tokenization, 2024. URL
https://ase w/tp:v/aaiivvvv/aa 1ttp://wwww...//////...111111111 13516
[37] Gemma Team and Google DeepMind. Gemma technical report, 2025. URL https
7/arxiv. orgo
org/abs/////eeoooooe////e//eaaar abs 2503.
[38] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL httpt::t//a/tttttt www.:/xwwwwr...oooooe/ org/ hbt////a.//22.... 13971
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit. Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[40] Liane Vogel, Benjamin Hilprecht, and Carsten Binnig. Towards foundation models for relational databases [vision paper]. In NeurIPS 2022 First Table Representation Workshop, 2022.
[41] Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2140-2151, 2021.
[42] Yanbo Wang, Xiyuan Wang, Quan Gan, Minjie Wang, Qibin Yang, David Wipf, and Muhan Zhang. Griffin: Towards a graph-centric relational database foundation model. In Forty-second International Conference on Machine Learning, 2025.
[43] Fang Wu, Vijay Prakash Dwivedi, and Jure Leskovec. Large language models are good relational learners, 2025.
05725
[44] Marck Wydmuch. Lukasz Borchmann, and Filip Graliñski. Tackling tional databases with llms, 2024. http:h/ppp..aar...ooo///////1111
Tackink prediction tasks in rela-
13

---

Published as a conference paper at ICLR 2026
[45] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023.
[46] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems. 34:28877-28883. 2021.
[47] Jianan Zhao, Zhaocheng Zhu, Mikhail Galkin, Hesham Mostafa, Michael M Bronstein, and Jian Tang, Fully-inductive node classification on arbitrary graphs. In The Thirteenth International Conference on Leurning Representations, 2025.
[48] Qifang Zhao, Weidong Ren, Tianyu Li, Hong Liu, Xingsheng He, and Xiaoxiao Xu. Graphgpt: Generative pre-trained graph eulerian transformer. In Forty-second International Conference Machine Learning, 2025.
[49] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji, Qiben comprehensive survey on pretrained foundation models: A history from Yan, Lifang He, International Journal of Machine Learning and Cybernetics, pp. 1-65, 2024.
bert to chatgpt.
14

---

Published as a conference paper at ICLR 2026
Sampling the context window of Relational Transformer. We use a modified Breadth-First Search (BFS) algorithm, with accommodations for relational-specific considerations, such as F-P links and temporal constraints.
Algorithm 1:
Input: seed row 8, context length I., width bound w, and, for each row in the database: non-missing feature cells C(r), P->F neighbors (r)- F-PP neighbors NF--p(r) and timestamp T(r)
Output: the set of database cells C' in the context window
C
f}. F 4- {s}
/ F is the frontier of rows to explore
while [C| < LLAF#(0 do
/x select a row to explore; R is the set of candidates x/ // F-P linked rows
R f E F was added via an F-SP link)
if = ( then
L R. arg min HOPDISTANCE(1,.)
// rows closest to s at random
RANDOMSELECT(BR
// pick // remove row from frontier
F + (r)
if r has been visited then continue else mark as visited
/+ visit row */ // add cells to context 1/ add F-P neighbors to frontier // filter P-F neighbors by time // pick S Ww P-F neighbors at random // add P-FF reighbors to frontier
C cuC(r)
F + FUNE-P(()
N + Eq T(q) T(()) NO + RANDOMSAMPLE(N, w)
F FUN
return C
(a) Graph visualization of the context window for an example from rel-hm (dataset) /
(b) Tabular visualization of the same context window
(task)
Bum-sales (aak) (50 rowe x cala)
transactions (21 rows x colo)
Limestareg ericts
1,591
aunarweJ Jr
anide
prla
cale choree J4
9-480)
G article row
154964
38-01
124254
0.44/83
O cuscomar row
Itee-saless row (task)
9690-36-01
1242814
154968
transactions row
2020-31-13
0,000000
Tbrget
article (14
25
articiejid
product cooe
prod_ name
grosppp
garmant_gnnup grtpppaarts
deteil
2267
169223
Curvy Jegpines H..
1090
Thosers Desin
Aeppings wash...
109344
655799
Nep
1035
Jones Panty
Be
Tancy
customer (7 Jome x usa))
customer_j
IN
Active
cMb EMt__member_.e siatia featlon preuuuerc
age
peotil_codee
ACTIVE
40NS
AcTiVE
#hoertCutalsoo
ACTIVE
VONE
24.0
Figure 6: Visualization of a real context window used for zero-shot prompting, sampled by Alg.
with context length L 512 and BFS width bound 128.
A CONTEXT WINDOW FOR ZERO-SHOT PROMPTING
Sampling cells for the context window. In prediction tasks such as forecasting and autocom- pletion are framed as predicting masked cell in the appropriate row (the seed row). We sample the context window independently for each training/testing example, so there is always a unique seed row for context construction. Given the seed row and context length L, suitable algorithm should select the cells most relevant to predicting the masked cell. Since relevance requires strong models to estimate accurately, lies within a few hops of the seed row when following FPPP and P-+F links, and that lower hops are more informative than higher hops.
simple heuristic guided by the intuition that most relevant information
15

---

Published as a conference paper at ICLR 2026
We treat rows as the sampling unit: once row is selected, all its non-missing feature cells (i.e., cells not from primary- or foreign-key columns) are included in the context. After the seed row, other rows are added using bounded-width BFS across F-+P and P-*F links, with the following modifications: (1) F->P links are immediately followed: (2) the traversal stops when the total numher of cells reaches the context length; (3) unvisited rows at the same depth from the seed row sampled uniformly; and (4) rows with timestamps greater than the seed row's timestamp are skipped (temporal constraint). The algorithm is summarized in Alg. [ to prevent on on-the-fly ffy sampling from slowing down training. Fig. 6 shows an example of sampled context window.
We use fast, optimized Rust implementation
In Alg. 1 F-PP and P-> links are treated asymmetrically. The number of F->P links from a row is limited by the number of foreign-key columns in that table, and each parent row typically contains important features (e.g., transaction row links to user and product). Conversely, P-F links can have unbounded degree; informative signals from P- links often arise via aggregation, with diminishing returns from including many children. Thus we prioritize F-PP links (followed immediately without subsampling) and subsample P-F links by enforcing a width bound w (i.e., follow at most w children from any row).
B EXPANDED RELATED WORK
Relational deep learning (RDL). Fey et al. [5] introduced end-to-end framework for predictive modeling on relational databases using neural networks. At its core, RDL represents a database as relational entity graph: a temporal, heterogeneous graph where each table is a node-type, each row an individual node, and every primary-foreign key relationship an edge. Initial approaches applied heterogeneous graph neural networks directly to these relational entity graphs [34,, and more recently, advanced message-passing approaches have been proposed to enhance the efficiency of GNNs on relational data [4. Transformers have emerged as way to improve upon the GNN muessage-passing paradigm. Peleška & Šír [28] and Dwivedi et al. ||| propose transformer-base architectures that achieve better performance than GNNs on relational data. An review of RDL architecture can be found in 112 A key limitation, however, is that these architectures are schema- specific, which prevents pretraining and fine-tuning on diverse database structures, Our Relational Transformer, by design, is schema-agnostic, enabling it to learn from and be directly applied to new, unseen database structures. This design principle allows our architecture to demonstrate foundation model-like capabilities, similar to those recently shown in tabular learning.
Tabular foundation models (TFMs). Recent advancements in tabular foundation models have
demonstrated significant promise, exhibiting capabilities such as in-context learning [19 30 and
efficient fine-tuning (24). These efforts have explored both supervised [9] and self-supervised [36]
24 pretraining on real [24] or synthetic [99 30 data. Extending tabular foundation models to
relational data is non-trivial, because not only are there multiple tables, but rows in one table are linked to rows in another by foreign-primary key links. We take inspiration for the universal cell encoders/decoders from PORTAL (36), which has a similar handling of column names and text/numeric//aateinee data types. We also take inspiration from the TabPFNv2 [20] transformer architecture, which uses stacked layers of row-wise and then column-wise attention, except that we also have all-pair attention and use attention masks to capture foreign-primary key links.
Relational foundation models (RFMs). While tabular foundation models can, in principle, be applied to relational datasets, they fail account for the rich, multi-table structure of real-world data. To address this limitation, recent works have begun to develop dedicated relational foundation models. For instance, Fey et al. [66] propose a relational foundation model based on graph graph-transformer transformnnsssss and in- context learning, demonstrating both in-context learning and fine-tuning capabilities. However, their solution is not open-sourced, and the exact pretraining procedure has not been released. Separately, Wang et al. [42 design novel architecture pretrained on a mixture of tabular and relational datasets, showing that fine-tuning improves downstream task performance. Their model, however, differs significantly from our own; it first aggregates information within each table and then utilizes graph neural networks to propagate that information between tables. In contrast, our model uses a cell-level representation of the entire database and employs attention masks to directly represent the foreign- -ee structure. This allows our approach to reason directly over the relational database in its native, cell-based format, offering more granular and unified understanding.
16

---

Published as a conference paper at ICLR 2026
amazon
avito
[1
aLack
trial
-amazon
1,000
0.000
0.000
0.043
0.000
0.018
rel-avito
0,000
1.000
0.000
0.000
0.018
0.000
rel-ft
0.000
0.000
rel-hin
0.043
0.000
0.000
1,000
0.000
0,000
zel sta-stack
0.000
0.018
0.000
0.000
1.000
0,000
rel-trisl
0,018
0.000
0.022
0,000
0.000
1,000
Table 6: Jaccard similarity matrix showing exact column name matches across RelBench databases. A
value of 1.0 indicates identical column sets, while 0.0 indicates no common columns. The extremely low off-diagonal values demonstrate that column names are domain-spectie with minimal overlap.
Pretrained models for graphs. Pretrained graph learning models have shown strong success in molecular domains. For example, MoleBERT [45] introduces masked atom modeling and triplet- masked contrastive learning to pretrain GNNs for both node-level and graph-level tasks relevant to drug discovery. Beaini al. [ scale pretraining by curating massive multi-task molecular datasets with billions of labels, showing that combining quantum and biological data improves low-resource tasks. Beyond molecules, [21] introduced a novel pretraining framework that leverages prompt- based graph representations enable in-context learning graphs. GraphAny [47] develops zero-shot node classification framework, grounded in linear least-squares principles, that generalizes across graphs with disjoint feature and label spaces by leveraging LinearGNN ensembles and inductive attention. ULTRA [17] targets knowledge graph reasoning. learning universal relational representations that transfer zero-shot to unseen knowledge graphs. Finally, GraphGPT [48] casts graphs as reversible token sequences via Eulerian paths, enabling transformer-he generative pretraining that scales with model size.
Graphs and Large Language Models. A growing line of research investigates how large language models can be adapted to reason over graph-structured and relational data. [14 [2] explore parameter- efficient encoders that converts graphs into soft prompts for frozen L.Mss, showing that performance strongly depends on the choice of graph serialization and structure encoding.
44 investigate
predictive modeling directly on relational databases with LLMs, demonstrating that careful schema- aware prompt design improves over naive text flattening. Building on this idea, 43 propose Rel-LLM, hybrid architecture that combines GNN encoders with LLMs in retrieval augmented generation framework. These works highlight the potential of combining graph or relational encoders with LLMs, but they are often limited by small context windows and are not tailored to relational databases. In contrast, our proposed Relational Transformer directly encodes multi-table structure via attention masks, offering a fully end-to-end solution that can operate either independently or alongside large language models.
C SUITABILITY OF RELBENCH FOR PRETRAINING AND EVALUATION
C.1 DATASET CURATION AND DIVERSITY
RelBench aggregates 7 real-world relational databases from diverse domains including sports (rel-f1), medical (rel-trial) social (rel-stack, rel-event), and e-commerce (rel-amazon, re1, rel rel-avito). The benchmark's 's curation process is intentionally min- imal, limited standardizing formats such as ensuring valid foreign keys, consistent timestamps, and schema validity. Critically, no datasets were modified to align column names, harmonize feature distributions, or unify schema structures. This lack of artificial harmonization ensures that the benchmark does not favor RT or any other schema-agnostic method through dataset engineering.
C.2 SCHEMA HETEROGENEITY
To assess whether RT's zero-shot generalization might be explained by lexical similarity across databases, we analyzed column name overlap using both exact matching (Jaccard similarity) and fuzzy string matching. The results, presented Tables and 7 reveal extremely low schema overlap.
The Jaccard similarity results show near-zero exact matches between databases, with the highest overlap being only 4.3% between rel-amazer and rel-hm. Even fuzzy string matching, which accounts for partial character overlap, yields only moderate similarities ranging from 0.47 to 0.59 across database pairs. These findings indicate that column names are highly domain-specifie, and
17

---

Published as a conference paper at ICLR 2026
avito
[1
aLack
trial
-amazon
1,000
0.587
0.503
0.522
0.515
0.496
rel-avito
0.579
1.000
0.528
0.478
0.586
0.504
rel-ft
rel-hi
0,516
0.474
0.473
1,000
0.474
0.487
rel stack
0.507
0.590
0.583
0.480
1.000
0.531
rel-trisl
0.497
0.504
0.519
0.490
0.526
1,000
Table 7: Fuzzy string similarity matrix for column names across RelBench databases. A value of 1.0 indicates all columns have perfect fuzzy matches, while 0.0 indicates no similar columns. The moderate off-diagonal values (0.47-0.59) reflect some character-level overlap but demonstrate that column names remain domain-specifie.
RT's zero-shot generalization cannot be attributed to lexical similarity. Instead, the model must learn to recognize and exploit structural relational patterns.
C.3 FEATURE DISTRIBUTION HETEROGENEITY
Beyond schema diversity, RelBench databases exhibit substantial heterogeneity in their feature distributions. Numerical features across databases display diverse statistical properties, and these distributions were deliberately not harmonized during curation.
Analysis of numerical feature distributions reveals significant variation:
rel-amazon: Contains numerical columns with highly right-skewed distributions (skewness 72.1 for price).
rel-avito: Contains 17 numerical columns with extremely high kurtosis indicating heavy tails; price exhibits very heavy skew (skewness = 547.6).
rel-f1: Contains 16 numerical columns with moderate skewness; features such as wins, points, and altitude show notable skew.
rel-hm: Contains 15 numerical columns, some with missing values; includes negative skew in graphical appearance features.
rel-stack: Contains 6 numerical columns with moderate to high skewness across several features.
Contains 12 numerical columns with extremely skewed distributions; count feature shows skewness of 615.8, indicating very heavy tails.
rel-trial:
The high skewness and kurtosis values, particularly in rel-tria strrial and re rel-aviton indicate that these datasets contain many outliers and exhibit long-tailed distributions typical of real-world data. This heterogeneity ensures that models cannot rely on iitribution-speciin shortcuts and must learn robust, generalizable representations.
C.4
SCALE AND TEMPORAL DIVERSITY
RelBench datasets vary substantially in scale and temporal coverage:
Schema complexity: Databases contain between 3 and 15 tables.
Scale: Row counts range from 74k 41M across databases.
Dimensionality: Column counts range from 15 to 140.
Temporal span: Forecasting time horizons vary from 2 weeks to 55 years.
This natural heterogeneity in scale, complexity, and temporal dynamics is precisely what makes RelBench challenging and suitable for evaluating pretraining approaches. To succeed on RelBench, RT must learn schema-agnostic relational computations rather than sptaset- shortcuts. The diversity domains, schemas, feature distributions, and temporal patterns ensures that observed zero- shot transfer reflects genuine generalization rather than memorization or dataset-specific overfitting.
18

---

Published as a conference paper at ICLR 2026
Dataset
Table
Column
Pos/NNeg, values
Non-missing
Positive (prop.)
Nogative
rel-amazon
review
verified
N/A
20 040
14493 882 (0.69)
6366 158 (0.31)
rel-avito
Searchinfo
LsUsern.ogyedOnn
2,799
095 (0.32)
(0.68)
rel-stack
postl links
LinkTypeld
103
89.076 (0.86)
(0.14)
studies
has time
t
234 467
79 850 (0.34)
154.677 (0.66)
rel-trial
eligihilities
adult
vs
273 160
251 581 (0.92)
21 579 (0.08)
eligibiltties
child
273 160
51 899 (0.19)
221 261 (0.81)
rel-event
event interest
not hnt_interetted
N/A
15 3989
514 (0.03)
84.88 (0.97)
Table 8 Autocomplete classification tasks with distributions of observed non-missing labels (propor- tions in parentheses). When applicable, the positive/negative value mapping is provided.
Datasct
Table
Colurn
Non-missing
Max
Median
Mean
rel-amazon
review
rating
20 8626 040
0.0
5.0
5.0
4.39
results
position
15207
1.0
33.0
7.0
7.97
rel-fl
qualifying
position
9815
1.0
28.0
11.0
11.24
constructor resalts
points
12290
0.0
66.0
0.0
3.86
construcoo_sstaddingg
position
13051
1.0
22.0
7.0
7.27
hm
transactions
price
15453651
0.03
rel-trial
studies
enrollment
0.0
188814085.00
3975.83
rel-event
users
hirthyear
36715
1900.0
1999.0
1991.0
1988.74
Table 9; Autocomplete regression tasks with summary statistics of observed non-missing labels (rounded to two decimals).
D AUTOCOMPLETE TASKS
Definition. Autocomplete tasks are defined as masked cell prediction feature columns that already exist in the database. Unlike forecasting tasks, they do not require constructing additional task tables. The input sequence to the model is constructed in the same way for forecasting tasks, preserving the relational structure and temporal order, but sampling starts from the masked database row.
Task selection. All autocomplete tasks were selected manually by inspecting the database schema. For each task, also identify potential sources of information leakage and discard these columns the fly when building the input sequence.
Task overview. Tables and and list all classification and regression autocomplete tasks, respectively, together with label distributions (classification) and summary statistics (regression).
E SYMMETRY AND EXPRESSIVE POWER
RT preserves the natural symmetries of relational data. In particular, the architecture is invariant permutations of rows, columns, and tables, providing an inductive bias that improves generalization. This contrasts with LLM-based approaches for relational data, which are often highly sensitive to prompt order and formatting, Permutation invariance has been key driver of success in prior graph neural networks, and respecting such symmetries has also shown benefits for large language models.
We also discuss role of the specialized attention layers in determining the expressive power of RT. For empirical evidence, see 5
Column attention. Removing column attention does not reduce expressivity, since global attention emulate it. In particular, some global heads can learn restrict attention to tokens from the same column by exploiting table and column name embeddings in their query-key construction.
Feature attention. Feature attention is the only mechanism that explicitly groups cells into rows. While neighbor attention provides partial information,celly the same row attend to the same set of neighbors, cannot uniquely disambiguate rows, especially in tables without incoming foreign keys.
19

---

Published as a conference paper at ICLR 2026
rtllamazan/usen-chun
rol-hm/user-chum examples
ral-stack//senrbaadge 3.4M examples
celsstack/usse--eggggmee 1.4M exampliss
colaamazenyitmmmchum
4.7M
2.6M examples
Test AUROC (%)
70
70
80
80
60
60
60
|
rel.avitoolserviitt
sel-avito//sorrdliks 59K examples
relttrialstuuyy-outtomm 12K examples
rel-f3/driver-dnf 11K examples
rel-f1/driver-to33 1.4K examples
87K examples
Test AUROC, (%)
65
70
65
80
60
60
60
55
10
107
101
10a
10
10
10
10a
10'
10a
Fine-tuning steps (batch_site battch_sieeeae
256)
Pretrained RT
Untrained RT
Untrained RDL-GNN
Pretrained Griffin
Untrained Griffin
Figure 7: Per-task test set learning curves on classification tasks for up to 32k fine-tuning steps (8M training examples, including repetitions). X-axis is log-scale.
Neighbor attention. Similar to column attention, neighbor attention is not strictly required for expressivity, as feature and global attention together can simulate its effect. Feature attention exposes F-P links, which global attention can then leverage to infer P-F relationships.
Full attention. Without full attention, information can only propagate hop per layer, limiting expressivity to local message passing. By contrast, full attention enables long-range interactions in fixed number of layers, independent of database or graph diameter.
leave the full theoretical characterization of RT's expressive power to future work.
F SUPERVISED FINE-TUNING RESULTS
In this section, we present detailed supervised fine-tuning results. Figures and g show per-task learning curves for classification and regression tasks, respectively. We then provide additional full fine-tuning results in Section F.I. analyzing the effect of pretraining in high-resource settings and comparing RT against both sppemf- and schema-agnostic baselines.
F.1 SUPERVISED LEARNING IN HIGH-RESOURCE SETTINGS
Setup. In Table 10 report results from full-dataset fine-tuning, using up several million training examples and continuing until convergence (tens of thousands of steps). We compare against schema-specific baselines IRDL-GNN, ReIGNN, and RelGT), which cannot be pretrained, as well as agnost-e baselines (ReILLM and Griffin.. For RT and Griffin, we evaluate both untrained and pretrained initializations to assess the impact pretraining. For methods, selected based on validation set performance. For ReIGNN, ReIGT, and ReILLM, we use the original training setups and hyperparameters. For Griffin, we increase the model size and update the sampling and pretraining procedures to be consistent with RT.
best checkpoint
Observations. The pretrained RT achieves the best performance on average, achieving the highest mean AUROC and R. On classification, it is outperformed on certain tasks by ReIGNN, RelGT and ReILLM, but it is important to note that these methods utilize custom setups for each task, whereas RT uses single unified hyperparameter setup across all experiments. On regression, pretrained RT ranks best on average, substantially outperforming the second- best method across most tasks. Overall, RT matches exceeds the performance of schema-specific baselines while maintaining general, schema-agnostic design, showing that generalization does not come at the expense of fine-tume tine-tuning performance.
20

---

Published as a conference paper at ICLR 2026
mel-hruttem-saes 5.5M examples
rel rel-amazonyuse-ilt 4.7M examples
rolamazann/tem--ty 2.7M examples
rel- rel.stack/ost--otess
2.5M examples
50
50
40
Test Test&? (%)
30
30
20
20
10
10
10
10
rel reltrialisite-success 150K examples
(el-tria adverse 43K examplies
rel-Ilidrieeppesitinn 7.5K examples
rel-avibo/ad-ctr 5.1K examples
10
60
20
30
10
R²
10
-5
10a
10a
10)
10a
101
10a
101
10a
Fine-tuning steps (batch_size 256)
trained
Untrained
Untrained
Pre-trained Griffin
Untrained Griffin
Figure 8: Per-task test set learning curves on regression tasks for up to 32k fine-tuning steps (8M
training examples, including repetitions). X-axis is on log-scale.
Train set size
Cannot be pretrained
Can be pretrained
Datasct
Task
(sorted)
RDI. GNN
Rel
Rel
Rel
Griffin
Griffin
RT
RT
GNN
GT
LLM
(Ours)
(Ours)
pretrained?
No
No
No
Yes
No
Yes
No
Yes
AUROC (%) for 10 binary classification tasks. Higher is better. Random/majority baseline 50.0.
user-churn
4.7M
70.7
71.0
70.4
71.9
70.0
69.4
70.5
70.8
zel hm
user churn
3 3.9M
69.
70.9
69.3
70.5
68.2 3
68.0 0
69.9
70.5
rel-stack
user-badge
88.9
rel-stack
user-engage
1.4M
90.6
90.8
90.5
91.2
89.8
90.4
90.0
90.2
ral-amezoo
ttem-churn
2.6M
82.8
rel-avito
user-visits
87K
66.1
66.2
66.8
67.0
65.0
62.6
65.0
65.2
rel-avito
user-alicka
59K
63.1
68.2
68.3
66.7
63.0
64.7
63.6
59.0
study-out
68.6
71.2
68.6
71.0
68.9
64.6
68.6
68.2
zel El
driver daf
11K
72.5
75.3
75.9
77.2
74.5
66.7
78.7
84.2
driver-top3
1.4K
Mean AUROC +
75.4
77.1
76.2
77.1
75.0
73.2
76.1
77.2
X² (%) for 8 regression tasks. Higher is better. Global-mean baseline is O.O.
zel hm rel aml--mmzonn
LLem sales
5.5M
21.8
22.1
22.6
31.1
30.4
45.7
39.0
user-ltv item-Itv
4.7M
21.9
17.9
17.5
nan
30.7
32.9
47.9
47.4
ral-omazon
2.7M
3.5
3.4
23.4
25.2
31.5
36.8
rai rel-atack
post-votes
2.5M
17.9
12.2
13.1
nan.
41.8
42.7
37.1
36.5
rel-trial
site-succ
-9.5
6.8
2.4
rel-trial
study-adv
43K
18.8
19.7
17.0
11.2
18.2
41.3
43.4
rel-fl
driver-pos
7.5K
20.7
12.4
29.9
33.7
51.6
zel-a80 aviio
ad-etr
5.1K
18.3
15.6
18.4
5.9
8.4
-1.5
1.5
R²
Table 10: Supervised fine-tuning results. Models are trained on the full training set until convergence, with checkpoint selection based validation performance. RT achieves the best mean AUROC and R² across tasks, surpassing both schema-specing specific (cannot be pretrained) and schema-agnostic (can be pretrained) baselines.
21

---

Published as a conference paper at ICLR 2026
Daraset 4
Task
Zero-shot
Fine-tuned
self labels
other labels
self labels
other labels
from context -
none
none
names
names
AUROC (%) for 10 binary classification tasks.
Higher
better.
Rundonvmmjjoiity baseline is 50.0.
xel
item chuzn sser-churn
70.2
71.0
48.1
72.2
83.4
83,3
83.4
83.2
-arazon xel-avito
61.9
63.9
64.2
70.8
70.4
70.8
70.6
user user-olicxs
59.5
58.5
39.0
61.0
60.1
zel-avito
user-visis
61.8
61.2
49.9
62.1
63.2
65.1
64.4
65.3
xel-f1
driver-dnf
rel-f1
driver-top3
89.1
86.5
74_1
89.1
91.9
91.8
89.7
91.7
rei-hm
user-churn
70.5
70.2
70.7
rel stack
badge
80.0
79.2
54.8
82.4
88.7
88.9
88.8
88.8
rel-stacx
user-engage
77.1
80.1
41.9
77.2
90.2
90.1
90.2
90.2
study
54.5
53.2
54.6
68.2
68.9
68.9
R² (%) for 8 regression tasks, Higher better. Global-mean baseline is 0.O.
rell -amason
item-Itv usez 1tv
33.2
33.0
-2.8
33.1
36.8
34.7
28.0
35.1
xel anazon
36.4
47.4
47.2
37.6
zel-evito
ed-ese driver-pos
4.5
5.7
-3.6
4.5
4.5
4.9
-5.2
4.3
rel-fl
54.7
50.3
.31.9
54.7
51.6
52.3
54.7
54.1
rel-hm
Item-sales
14.0
9.2
-2.8
14.7
39.0
39.5
33.6
39.3
rel-stace
post-votes
32.4
27.5
-0.7
32.8
36.5
35.9
34.9
37.2
rel-trial
8.3
rel-trial
study-adv
2.1
1.3
2.1
2.1
43.4
42.4
37.0
38.1
Mean R
22.8
20.5
33.2
33.2
26.7
31.0
Table 11: Ablation study of context construction. To explain the zero-shot performance we remove column names, past labels from the target entity and labels from other entities. To assess how much task-relevant information lost, we repeat the same ablations in the fine-tuning setting. Shading indicates the performance difference relative to the full context (none column).
G CONTEXT WINDOW ABLATIONS
In Section 5.1 we introduced ablations of the context window to analyze the emergence zero-shot performance. Here, provide the full results of that study. Specifically, systematically remove or perturb individual context components and report their effect on both zero-shot transfer and supervised fine-tuning performance.
Setup. Table I we report results when shuffling column and table names, removing past labels from the target entity, removing labels from other entities. For zero-shot evaluation, the ablations are applied directly to the sampled context used as input. For fine-tuning, models are trained to convergence with the same modified contexts. In addition, [able([2[providee statistics the number of label cells (mean + std. dev.) included in a context window of length 1024 under our sampling procedure (Alg. D.
Observations. We find that zero-shot transfer primarily arises from the presence of past labels of the target entity. Removing these labels causes the largest drop in performance, whereas removing labels from other entities has a smaller effect. Shuffling column and table names also harms transfer, highlighting the importance of semantic signals from schema metadata. In the fine-tuning setting, classification performance is largely robust to these ablations, but regression tasks consistently benefit from access past labels of the target entity.
H ARCHITECTURE ABLATIONS
In Section 5.2
we introduced ablations of the relational attention layers to assess their contribution to zero-shot transfer and fine-tuning performance. Here, present the detailed results of that study. Specifically, we remove individual attention layers- feature, neighbor, or full/global and analyze their effect across regression tasks, while classification results (showing minor differences) are provided in App. H
Table 13 shows the full Relational Attention layer ablations. We observe clear patterns in the zero-shot setting, but during finetuning removing any layer results in a decrease in performance, except the user-clicks task, where the model is prone overfit.
22

---

Published as a conference paper at ICLR 2026
Target entity labels
Other entity labels
Unique labeled entities
Binary Classilication Tasks
re rel-amazoon
poer-churr
4 +
1 + 5
2
+ 3 1
rel
user chure
rel-stack
user-badge
7 +
3 =
rel-amszon
item-churr.
1:
2
2 2::00
rel
user engagement
10
rel-avito
user-visits
2 3
0
1
rel-ovito
user-clicks
010
rel-trial
study-out.come
0
0
0
rel
driver dnf
= 141
0
1
rel-fl
driver-top3
+
0
1
Regression Tasks
rel-hm
iten-sales user-ltv Iten-Itv
+: 13
0
1
rel-omazon
4 1
1 1:.5
2 213
rell-maaonn
2
2
rel-stock
post-votes
16 1 10
44
212
rel rel-trial
site-sucoess
1 +
0
1
study adverse
0 0
0=0
0
rel-fl
arsver-positier
14 10
0
1
rel-avito
Table 12: Breakdown of "in-context labels" sampled by Alg.
Context length is 1024 cells. Numbers (mean + std. dev.; both rounded the nearest integer). Target entity, c.g., user, item, etc., the one for which prediction is desired. Labels refer to unmasked cells from the target column. Other entities are reached via graph traversal. Multiple labels are possible for the same entity as the tasks are temporal.
Datasct $
Task
Zero-sbot
Fine-tuned
Ablated attention
AUROC (%)
10 hinary classiiccaiion tasks. Higher better. Ramdom/mmajrrty buseline is 50.0.
re -ommazon
item-churn
70.2
60.8
73.0
70.8
71.5
83.4
83.3
83.1
82.2
83.2
zel amazon
user user-clicks
63.9
63.2
62.9
63.1
70.7
70.4
69.3
70.3
cel-avito
59.5
63.0
62.0
58.7
61.5
59.0
62.1
64.9
62.4
63.3
rel-avito
user-visits
62.7
60.6
65.3
64.6
65.0
rel-E1
driver-drff
82.0
79.4
77.1
81.4
81,9
84.2
83.8
82.2
81.1
82.2
rel-fl
driver-top]
89.1
87.5
85.1
88.0
89.3
91.9
92.0
85.9
90.4
90.2
ral-hm
user-churn
62.8
65.8
65.6
64.4
70.5
69.8
70.1
69.4
70.1
rel-stack
user-badge
80.0
79.4
82.0
79.7
81.2
88.7
89.2
88.9
88.
88.8
zel
usec engage
79.1
sel-trial
study-out
54.5
59.4
54.8
58.6
54.8
68.2
66.6
66.5
68.3
67.1
Mean AUROC
70.1
69.8
70.6
70.8
71.1
77.2
77.3
76.7
76.5
77.0
Table 13: Ablation studies on the attention layers of RT on classification tasks. col, feat, nbr, full denote that column-, feature- neighbor-, full- attention layers absent respectively. Total parameter count is kept constant by increasing the number of layers. Shading is proportional to difference from the none column.
H.1 RELATIONAL ATTENTION ORDER ABLATIONS
Tab, 15 shows the results of ablations on the order of relational attention layers. We observe no clear patterns, suggesting that the specific order of relational attention layers not critical to performance.
I
BASELINE IMPLEMENTATIONS
1.1
ENTITY MEAN
The EntityMean baseline predicts the mean of past target-entity labels. When past labels available for the target entity, falls back to the global mean of the target column on the training set. For computational efficiency, report the metrics subsampled test set.
23

---

Published as a conference paper at ICLR 2026
Datasct
Task
Zero-shot
Fine-tuned
P(mask)
0.2
0.4
0.0
0.2
0.4
NP
(8)
binary classificaiin
ligheer better. Randommmajjrity baseline
rel-ama son
Iten-chorr
70.2
70.5
83.4
83.0
$2.9
$3.2
rel amazon
churc
63.9
62.6
62.9
70.8
70.7
70.6
70.5
rel rel-avito
user-al1 ckk
59.5
61.5
61.1
59.0
62.3
62.2
63.6
rel-avito
user-visita
61.8
63.0
63.2
65.2
65.5
66.5
66.0
rel-f1 F1
driver-dnf
82.0
80.9
76.7
84.2
81.7
77.7
78.7
rel-fl
driver-tap3
91.9
91.0
91.2
rel-hm
user-chorm
62.8
60.5
62.3
70.5
69.9
69.9
69.9
rel-stack
user-badge
80.0
79.1
77.9
88.7
88.3
88.7
rel stack
engage
77.1
75.0
73.6
90.2
90.1
90.0
90.0
rel-trial
study-out
54.5
55.1
65.2
68.2
70.2
68.2
68.6
Mcan AUROC
70.1
69.8
69.3
77.2
77.3
76.7
76.1
R² (5) for regression tacks. Higher is better. Glohal-mean haseline 0.0.
amazon
33.2
36.8
37.0
30.9
31.5
rel-amazon
user-itv ctr
36.4
25.8
16.4
47.4
49.0
49.6
47.9
rel avito
10.8
1.9
rel-fl
driver-pos
51.7
16.8
42.4
51.6
50.1
47.7
33.7
rel-hm
item-sales
14.0
10.0
6.2
39.0
50.7
53.5
45.7
rel-stack
post-votes
32.4
33.5
32.3
36.5
39.9
38.5
37.1
rel-trial
site-suce
5.2
1.4
3.0
6.4
7.1
6.6
8.8
sel-trial
study
13.4
47.3
46.8
Mean R²
22.8
16.9
15.3
33.2
35.6
34.4
28.4
Table 14: Pretraining with multi-cell masking. Masked cells contribute to the loss. The target cell always masked. Other cells are masked with probability P(mask). NP denotes no pretraining. Shading is proportional to difference from the P(mask) 0.0 column.
PCN
FNC
NCF
AUROC (8) for 10 binary classification lasks. Higher betler. Random/nujjriiy baseline 50.0.
lesttorel-amzzon amazon
Itee-churm
70.1
72.4
70.8
69.4
71.1
70.9
70.5
user-churn
63.2
64.3
64.2
63.7
63.9
64.
63.8
rel rel-avii avitto
user-clicks
58.7
60.9
59.6
61.5
58.8
59.3
62.3
user-visits
61.5
62.2
rel fl
drives dnf
79.6
81.5
81.4
80.7
81.2
80.0
79.9
rel-fl
drivez-top3
87.9
88.9
88.1
88.1
90.0
88.6
86.8
rel-hm
user-churn
63.2
62.5
63.4
63.3
63.6
63.5
62.5
rel-stack
user-baage
79.6
79.7
81.2
79.3
79.8
79.6
80.5
slack
engage
80.5
78.5
76.3
73.3
77.0
rel-trial
study-out
55.8
55.3
55.5
54.3
51.6
53.4
47.5
Mean AUROC
70.0
70.6
70.0
69.8
70.2
69.5
69.3
R² (%) regression Higher
Global mean
rel amason
iLem 1Lv
31.6
30.7
30.9
29.9
30.0
30.9
32.2
rel-smezon
user-ltv
35.6
35.1
38.3
37.3
34.5
35.1
38.1
rel-avito
ad-etz
5.7
5.7
6.5
4.9
3.4
4.7
4.9
drivez pos
57.2
53.0
55.8
53.9
rel-hm
item-sales
17.5
16.5
14.6
16.5
18.5
15.9
18.8
rel-stack
post-vates
31.5
29.5
rel-trial
sité-suce
3.9
4.5
3.3
5.2
4.2
8.0
4.0
rel triel
study adv
2.1
2.7
2.7
1.5
1.2
2.5
2.3
Mean R²
23.0
22.5
23.2
22.5
22.1
23.0
23.6
Table 15: Different orders of relational attention layers. C, F, N denote peatuue-.feature-- neighbor- attention layers respectively. The order of letters in the column headers indicates the order of relational attention layers applied sequentially. Parallel denotes that all three attention layers are applied in parallel and their outputs are summed. Results are shown for the zero-shot setting. Experimental setup is same as followed the main paper.
1.2 LLM PROMPT CONSTRUCTION
Large language models (LLMs) are evaluated under the same information regime our relational transformer (RT): input to both is constructed from the same context subgraph produced by our sampling algorithm (Alg. Tp. In this graph, nodes correspond to database rows and edges represent F-PP and F- links, We serialize the sampled entity graph into JSON, which encodes relational structure.
24

---

Published as a conference paper at ICLR 2026
Serialization procedure. We begin with the subgraph produced by the sampler. Serialization starts at the task node, which specifies the prediction timestamp and links directly to the target entity for which the label is to be predicted. From this target entity, we traverse the relational graph using both F-PP and P-F links. Each visited row is merged into the existing record in the case of F-P link or further serialized and appended as new entry the list of linked entities in the case of P->F link.
Prompt components. Each prompt follows fixed four-part structure: (i) short dataset description; (ii) description of the prediction task; (iii) the serialized graph context (a JSON of table-row objects) including the prediction timestamp fo: and (iv) concise instruction specifying the expected output ("yes" or "no"). Dataset and task descriptions are adapted from prior work 34
Full prompt example.
are a strict prediction assistant. Follow instructions exactly. latabase
Names Stack Exchange
Deseription: Stack Exchange network of question-ond-anawer websites different topics, where questiona, answers, and users are susject to a reputation award proceas. The reputation self-moderating The database inoludes detailed of
syatem allowa
sitea activity
raw text), histories, voting, ralated posts. In our benchmark, me use the stats-exshanga site.
including usez biographies, pests comments
Task
Name: user-badge
Descriptions This task 1a to predict If thts user will recetve a new badge in the next 3 montha not.
Input
Ustabase serialization starting from target instance, expanding consext by including rowg
reached via fēp !foreign to prinary) and p2r (primary foreign) relationehipe,
The first timestamp the sequence denotes the prediction time
# Database serialization for the target entity
"timestamp": "2021-01-01100:000000
"UserId": 211098,
"Td": 211058,
"AccountId"; 12827220.0,
"Diisplayamme: "Shashwat Thwary", "Location": muil, "Ezofileinagelz1"; mull,
"websiteerr"; m=l1,
"AboutMe": null,
"CreationDate"" 1201-00111111::::3333......
"add_bedgge""
("ra": 383629, "UaerId"; 211398, "Claas": "Name": "Bditor", "TagRsaed"; false,
"Date": *2019 09 15107:40:23.963000
"add_ =sdd_ussr=-:dda";
("Eimestamp"; 12020-10-01700, 00" "JaerTd": 211098, "wi1 Radgo"; *no"|,
("Limestamp": *2020 04 02T00:00:00 00:00", "UserId": 211098,
"MillGelBadge": "no"|,
("timestamp": "2019-10-03100:00000 "Jserid": 211098, "MillGetRadge": "no"1
Outpptt
Output exactly one word on a single Lines yea or no. No units, no punctuation, spaces, commas, extra text, extra symbols, no lines.
Make your prediction for the target entity eo waing databass serializacion, database description, and task description.
In-context labels. Due to the nature of sampling algorithm, past (unmasked) labels from the target column can remain in the serialized JSON. For example, in the use rbadgee task (see full prompt above), the nested entries under addluser-badge constitute such in-context labels. More details on the occurrence and distribution of these labels are provided in Table [2
1.3 REGRESSION RESULTS WITH LLM BASELINES
In addition to classification, we evaluated zero-shot regression with LLMs of varying sizes under the same RT information regime. Across eight regression tasks, performance was consistently sooreemalerr models even failed to produce stable numerical outputs under strict prompting. We attribute this to unconstrained number generation and context not optimized for LLM regression. Prior work shows that carefully selecting and formatting in-context examples can substantially improve results [27. Given these limitations, do not report detailed regression metrics.
25

---

Published as a conference paper at ICLR 2026
Target DB pretraining?
Maybe
No
Yes
Dataset
Entity Mcan
Griffin
RT
Rel LLM
Griffin
RT
(ours)
(ours)
Parameter count
4B
12B
27B8
0
22M
22M
3B
22M
22M
xel
item 1tv
<
-9
-9
54.2
20.1
32.5
20.1
32.2
rall-amaz0r Por
vser-Ity
<
< -9
<
19.9
20.6
36.9
24.4
38.3
rel-avito
-8.2
3.4
2.4
4.5
8.0
rel-fl
driver-pos
35.2
13.4
52.4
38.2
-0.7
52.4
4.6
58.7
rel-hm
item-sales
<-9
--9
30.9
rel-staack
post-vatee
< -9
< -9
43.7
27.4
33.9
27.1
35.0
rel-trial
site-auco
-6.4
1.4
4.5
rel trial
study adv
<-9
-7.1
-0.5
-255
2.6
-2.5
3.1
R²
9
10.1
Table 16: Zero-shot R (%) for regression tasks. Higher is better. Global mean baseline is 0.0. Setup same as Table]
1.4 GRIFFIN
To ensure fair comparison with Griffin, we scale its hidden dimension from 512 to 728, resulting in a comparable parameter count to RT (22.8M vs. 22.3M). We also match the training setup by adopting both the deave-one-database-n and continued pre-training regimes. However, since the Griffin implementation does not support joint training on forecasting and autocomplete tasks, we restrict it to forecasting tasks only.
J RELATIONAL TRANSFORMER BLOOC
Algorithm illustrates the architecture single Relational Transformer Block. This block consists of series of attention mechanisms: column attention layer, feature attention layer, neighbor attention layer, and a global attention layer, each with its own specific relational inductive bias. These Relational Attention layers are followed by feed- feed-forward network (MLP) for further processing.
Algorithm 2: A transformer block in RT.
Input: input token representations X E Baxd
Output: output token representations X Raxd
X + X + NORM( MHA(X. Moolumi)
X + X + NORM( MHA (X; Mfeature) )
X f- X + NORM( MHA X; Mneighbor)
X X + NORMÍ MHA X; Mfll!)
X + X NORM( MLP(X).
return X
26


Reviews:
[{"review_id": "UdREXkie1B", "reviewer": "Reviewer_fWfy", "summary": "This paper proposes Relational Transformer, a pre-trained model for relational databases. The model treats each cell in a given relational database as a token for the transformer; the proposed transformer holds specialized attention mechanisms that adapts to specific traits of relational databases; the pre-training is carried out with self-supervised objective of masked token prediction. To show the effectiveness of the Relational Transformer, it showcases with RelBench, in which leave one-database out approach is implemented for pre-training and evaluation.", "questions": "-\tWhat are some concrete examples on the usefulness of zero-shot abilities for relational databases?\n-\tIn Algorithm 2, is there a reason for the specific order of different data types?\n-\tHow curated are databases in RelBench?\n-\tHow does the Relational Transformer perform with respect to the computation time?\n-\tCan Relational Transformer be used as a feature extractor (e.g., sentence transformer as in LLMs)?\n-\tWhat does it mean by the sentence ‘While task rows provide “in-context labels”, our setting is not few-shot as explicit subgraph-label pairs are not required.’? If the input of the prediction contains past labels, does this mean that the Relational Transformer calculates the attention between what to predict and the past labels (possibly through column attention?\n- What is the reason behind the choice MiniLMv2 as the language model?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general, the paper is well-written and easy to follow. The Relational Transformer attempts to build a pre-trained model for relational databases, which can be of non-trivial impact towards the foundation models for relational databases.", "weaknesses": "-\tIt would help to clarify the strength of the proposed Relational Transformer with some descriptions of the RelBench databases. This would include not only the basic statistics, but also the overlap of column names across the databases (possibly measuring the similarities of the llm embeddings), how the numerical values are distributed, etc., While diverse the RelBench maybe, I am uncertain as to how much the databases are curated so that they meet the standards to be included in the benchmark, and this may be in favor of having the zero-shot abilities for the Relational Transformer. Moreover, the characteristics of RelBench may give insights on ‘enabling large-scale pretraining’ as the paper claims for Relational Transformer.\n-\tIt would be helpful to include examples that could highlight the importance of zero-shot learning on relational databases.\n-\tOne of the possible extensions could be incorporating meta-data(base) information (possibly through analyzing the encoding steps).\n-\tWhile there could be some space constraints, it would be helpful to see a figure (possibly with an example from Figure 1) on how zero-shot prompting is conducted for understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DCbdmxZhie", "reviewer": "Reviewer_XZvM", "summary": "This work proposes the Relational Transformer (RT), a RDB foundation model with zero-shot prediction capacity.  Facing the core challenge of heterogeneous schemas, graph structures, and functional dependencies, it leverages table metadata and task text definition to adapt to different input data and task. Moreover, it models RDB in cell level and passing messages between cells with relation attention. RT attains strong zero-shot and fine-tuning performance on RelBench.", "questions": "1. RT relies on table metadata and task description, which are not always available in real-world cases. Can RT works in case that the meta data and task text description not available? \n2. RT is a cell-level model, where each cell is a token. Will it take more computation resource than row-level model Griffin?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear method illustration. Figure 1 shows the problem formulation and method, especially relation attention design, clearly. \n2. Strong empirical performance. It outperforms Griffin, a strong baseline. Moreover, to our best knowledge, it is the first RDB model with zero-shot capacity.", "weaknesses": "1. Code is not available.\n2. Tables and Figure in Page 8 looks messy.\n3. Ablation study in Table 3 should further include ablation of task description and schema data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "clbwxv2hgi", "reviewer": "Reviewer_rqLM", "summary": "The paper tackles the open problem of building a schema-agnostic foundation model for relational databases that can transfer across datasets and tasks without task-specific fine-tuning or in-context retrieval. The proposed Relational Transformer operates at the cell level, tokenizing each value together with its column and table names, pretraining with masked-token prediction, and introducing Relational Attention masks that selectively attend across columns, features, foreign-key neighbors, and a global channel. A task table is concatenated to provide task context for zero-shot prediction. On RelBench datasets spanning classification and regression tasks, RT shows strong zero-shot transfer—e.g., the abstract reports ~94% of fully supervised AUROC for binary classification using a 22M-parameter model, and substantially better sample efficiency during fine-tuning—while a 27B LLM under comparable context performs markedly worse at much higher inference cost. The paper presents ablations isolating the effect of self-labels, schema semantics, and attention masks, and discusses current scope limits (e.g., link prediction/recommendation). Overall, the core contributions are: (i) a cell-level tokenization that unifies relational prediction as masked token prediction; (ii) a set of relational attention masks that encode schema structure; and (iii) a zero-shot task-table prompting interface enabling transfer across heterogeneous schemas.", "questions": "1. Do you observe consistent gains as the number/diversity of pretraining databases grows? A simple scaling-law study over #datasets and steps would help calibrate the “foundation model” claim.\n2. Beyond entity-level classification/regression, how would RT fare on link prediction or forecasting formulated without self-labels? Any preliminary results or blockers?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The cell-level tokenization plus Relational Attention masks is a clean, general design that bridges tabular “foundation models” and relational deep learning. The work articulates a concrete, reproducible path to schema-agnostic pretraining over diverse databases and makes a credible zero-shot case relative to text-serialized LLMs and graph-centric foundations.\n2. The empirical suite covers multiple tasks from RelBench, reports zero-shot vs continued pretraining vs fine-tuning, and includes context and attention ablations. The paper contrasts RT with LLM baselines and graph/tabular lines, and provides architectural ablations showing column attention’s disproportionate effect on zero-shot.", "weaknesses": "1. Missing compute, throughput, and memory comparisons vs. graph-centric models. The model uses sparse masks compiled to FlexAttention; some training details are given, but no throughput comparisons vs. Griffin/RelGT or cost-for-quality trade-offs are reported.\n2. Ambiguity in the pretraining exposure conditions (Maybe/No/Yes) undermines interpretability of zero-shot tables. Tables 1–2 have a column “Target dataset ∈ pretraining? → Maybe / No / Yes,” but the meaning of “Maybe” is not clearly defined in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)