Title: ConTextTab: A Semantics-Aware Tabular In-Context Learner
Venue: NeurIPS 2025
Accepted: yes
Keywords: tabular learning

semantics

in-context learning

deep learning
Abstract: Tabular in-context learning (ICL) has recently achieved state-of-the-art (SOTA) performance on several tabular prediction tasks. Previously restricted to classification problems on small tables, recent advances such as TabPFN and TabICL have extended its use to larger datasets. Although current table-native ICL architectures are architecturally efficient and well-adapted to tabular data structures, their exclusive training on synthetic data limits their ability to fully leverage the rich semantics and world knowledge contained in real-world tabular data. At the other end of the spectrum, tabular ICL models based on pretrained large language models such as TabuLa-8B integrate deep semantic understanding and world knowledge but are only able to make use of a small amount of context due to inherent architectural limitations. With the aim to combine the best of both these worlds, we introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework. By employing specialized embeddings for different data modalities and by training on large-scale real-world tabular data, our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark. Code and model checkpoints are available at: https://github.com/SAP-samples/contexttab .


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ConTextTab, a foundation model for tabular data using in-context learning that is pre-trained on real tables and has meta-learned how to handle not only numerical and categorical features but also text, dates, and column names. ConTextTab combines architectural innovations of different models together with mechanisms for handling embeddings of different column types. It is evaluated on multiple benchmarks with a diverse set of baselines and achieves strong results, particularly on the semantically rich CARTE benchmark, where it sets a new state-of-the art. Numerous ablations illustrate the effect of different design decisions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Clarity: The paper is well-written and generally easy to follow. \n- Significance: The paper studies a practically important problem. It provides many interesting insights into the pre-training and architecture of tabular foundation models, therefore I expect that it will be significant for the development of future models. In particular, I appreciate the numerous ablations.\n- The paper includes a diverse set of established baselines.\n- The paper is not overselling too much. In particular, I appreciate that the paper includes multiple benchmarks, both textual and non-textual ones, which is not common for papers introducing textual models.\n\nWeaknesses:\n\n(1) The paper compares foundation models using training+validation data in the forward pass to non-foundation models that (mostly) use a single train-val holdout split (according to Appendix B.2). The classical methods would likely benefit significantly from inner cross-validation (e.g., 8-fold as in AutoGluon, but 5-fold is also common), while the foundation models would probably not benefit from it compared to the current approach. This is a major concern for my score. I see several approaches to address this, listed in order of decreasing quality: (a) Ideally the authors would run every of the non-foundation methods with cross-validation. However, I understand that this is expensive. (One way to illustrate the cost would be to actually report the training+inference costs of different models.) (b) If the authors want to keep the holdout validation, in my experience it would at least be more fair to compare to foundation models that only see the training set. (One might think that this is now unfair to the foundation models as they don't see the validation set at all, but other models can benefit a lot from cross-validation due to the ensembling, while foundation models are already ensembled.) This of course makes the comparison of foundation models to AutoGluon unfair, so it might be best to have a train+val and a train version of the foundation models. (c) Mention this point in the main paper and state it as a limitation.\n\n(2) The runtime (training+inference) of methods is not evaluated, making it unclear how fast ConTextTab is compared to other models.\n\n(3) Minor: Most results come without error bars.\n\n\nOriginality: The proposed method is partially a combination of aspects of different models (dataset from TabuLa, base architecture from TabPFNv2, etc.) with some new aspects in terms of the details and the handling of different datatypes. I see originality as neither a strength nor weakness of the paper.\n\nQUESTIONS:\n- Eq. (2): I find this part somewhat confusing. In TabICL, if I understand correctly, only the final MAB uses Q, so wouldn't it be MAB(C+Q, MAB(I, C))? A diagram as in TabICL might help to illustrate things. Also, in MAB(X, Y), it could be clarified that X are the queries and Y are the keys.\n- Figure 2: Why is the R2 score of CatBoost [HPO] negative on OML-CTR23?\n- How many parameters does the model have, and how does it compare to the other foundation models?\n- For the text embedding models, the authors might want to look into / cite https://arxiv.org/abs/2312.09634\n\n\nMinor comments:\n- l. 69: could mention TabM and ModernNCA as well\n- l. 109: \"then\" -> \"than\"\n- l. 292: It could be nice to state the default context size (8c = 7600 if I understand correctly)\n- l. 536: \"deault\" -> \"default\"\n- l. 580: \".\" missing\n\nLIMITATIONS:\nThe paper states quite a few limitations. Besides the mentioned lack of (inner) cross-validation, there are some other concerns that affect tabular benchmarks in general and could be discussed, like dataset quality, quality of search spaces for baselines, etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce ConTextTab, a new tabular ICL model, following the ideas presented in TabPFN. The core differences are pre-training on real world data as well as integrating semantic understanding (column names, etc.) into the model.\nThe core training architecture of common ICL models (TabPFN, TabICL, Attic) is still implemented.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n- Well written\n- Relevant and interesting research question addressing the problems of current ICL models to adequately be applied to real-world datasets\n- Benchmarks are very large\n\n# Weaknesses\n- Benchmarks are somewhat skewed. It seems as if ConTextTab is outperforming PFN only on the CARTE benchmark whereas TabPFN outperforms ConTextTab on all other benchmarks. \n  - I am not sure how  in Table 1 Naive in the CARTE benchmark can have an average rank > 11 when you compare 11 models? This somehow makes me question the ranks reported for CARTE since they are so different from all other tasks (even TabReD). And why does TabICL not have a rank if you have an accuracy?\n- While the benchmarks are quite large, including TabM [1], Mambular [2] and extending TabICL [3] to further datasets would have been nice.\n- Compute times for different models, especially for TabPFN, TabICL and ConTextTab are missing.\n- While mentioning the dataset sizes of TabPFN and TabICL as limitations, I did not find any information of whether ConTextTab can handle larger datasets any better than the two. The results on the TabReD dataset suggest otherwise.\n\n# General comment\n- Section 3.4 is somewhat poorly named. Currently I would read it as an alternative idea to your presented architecture.\n\n\n\n\n------\n\n[1] Gorishniy, Yury, Akim Kotelnikov, and Artem Babenko. \"TabM: Advancing tabular deep learning with parameter-efficient ensembling.\" The Thirteenth International Conference on Learning Representations.          \n[2] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv preprint arXiv:2408.06291 (2024).        \n[3] Qu, Jingang, et al. \"TabICL: A Tabular Foundation Model for In-Context Learning on Large Data.\" arXiv preprint arXiv:2502.05564 (2025).\n\nQUESTIONS:\n- Please release model code during reviews in an anonymous format.\n- Could you outline the contributions of each task to the total number of datasets.\n  - Provide critical difference diagrams for each benchmark suite separately\n- Have you tried out different preprocessing strategies, especially for numerical, i.e. PLE or PLR?\n- Why do you actually sum the embeddings? Wouldn't at least averaging them be more secure as o avoid overly large values?\n\nIn section 3.2 you write that \"horizontal\" and \"vertical\" blocks have exactly the same weights. Could you elaborate? Do they share the weights?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ConTextTab, a semantics-aware, table-native in-context learning (ICL) model for tabular prediction tasks. The motivation is to bridge the gap between traditional table-specific ICL architectures (e.g., TabPFN, TabICL), which excel at leveraging structural information but neglect table semantics, and large language model-based approaches (e.g., TabuLa-8B), which understand semantics but are inefficient in processing large tabular data due to linear tokenization and context limitations. ConTextTab enhances the table-native ICL backbone by introducing modality-specific embeddings for textual, numerical, and date columns, as well as semantic embeddings for column names and categorical values. The model is pretrained on large-scale real-world tabular datasets (T4), and evaluated on a suite of tabular benchmarks, demonstrating strong results—especially setting a new state-of-the-art on the semantically-rich CARTE benchmark, and exhibiting robust performance in low-data regimes. Comprehensive ablations analyze the impact of model components, context size, and data scale.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. The experimental evaluation is comprehensive, covering 203 datasets across five different benchmarks with proper statistical analysis including critical difference diagrams. The ablation studies are thorough, examining various architectural choices, training data scales, and hyperparameter settings. The authors provide detailed implementation details and plan to release code and model weights.\n\n2. The paper is well-written with clear motivation, methodology, and experimental setup. The figures effectively illustrate the architecture and results. The related work section provides good context for understanding the contribution relative to existing methods.\n\n3. The multi-modal embedding approach is sensible, with appropriate handling of text, dates, and numerical features. The architectural modifications to TabPFN are well-motivated and the training procedure is clearly described.\n\n### **Weaknesses**\n\n1. The paper lacks direct validation of semantic understanding capabilities. There are no experiments demonstrating that the model can leverage semantic similarities between column names, understand categorical label meanings, or transfer knowledge across semantically related tasks. The \"semantic understanding\" claim rests primarily on downstream task performance.\n\n2. While the combination is novel, the core technical contributions are relatively incremental. The approach primarily involves using pre-trained text embeddings and training on real data rather than synthetic data. The architectural modifications are minor adaptations of existing transformer-based designs.\n\nQUESTIONS:\n1. Can you provide experiments that directly test semantic understanding capabilities, rather than relying solely on downstream task performance? For example, can the model correctly identify semantically similar columns across different datasets, or does it show improved performance when categorical labels have semantic meaning versus arbitrary codes? Such experiments would strengthen the core claim about semantic awareness.\n\n2. Your results show that performance plateaus after 100K training tables and that larger models perform worse. This contradicts the intuition that semantic understanding requires large-scale data and model capacity. Can you provide a more detailed analysis of what limits further scaling and whether this indicates fundamental limitations in your approach?\n\nLIMITATIONS:\nThe authors do address limitations in their conclusion section, acknowledging the challenges with scaling to large datasets and the need for more diverse semantic evaluation benchmarks. However, the limitations section could be more comprehensive. Missing discussions include potential data contamination effects, the circular reasoning in using downstream performance to validate semantic understanding, and the practical constraints of the method's applicability. The authors should also acknowledge that their evidence for semantic understanding is primarily indirect and that the method's advantages may be limited to specific data regimes and domains.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes ConTextTab, an in-context learning model pretrained from the real-world T4 dataset. ConTtextTab employs embeddings tailored to different data modalities, in particular incorporating semantic embeddings of column names and categorical values. The experiments were conducted on diverse range of datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
- In general paper is well written and easy to follow.
- The performance comparison across diverse datasets shows competitiveness of ConTextTab.
- The work proposes architectural choices and training schemes that maybe useful for future related works.

Weaknesses:
- It might be useful to include some characteristics of the datasets. For example, are the datasets provided with textual features or are they processed to have only the numbers?
- It would be useful to include the hyperparameter search space.
- It would be interesting to see a comparison with simple random-weights (no pretraining).
- In table2, the word 'fine-tuning' is obscure, as the ablation is carried out on benchmark specific fine-tuning. It could be more clearer with 'benchmark-dependent fine-tuning'.
- What are some expected outcomes with dataset specific fine-tuning?
- It would be interesting to see computation time comparison between the in-context learning models. This could provide some insights on the trade-offs in using larger llms.

QUESTIONS:
Please refer to weaknesses above.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03