Attic: A New Architecture for Tabular In-Context Learning Transformers

Authors: Felix den Breejen, Se-Young Yun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: tabular classification, tabular in-context learning transformers, architecture

Abstract:
Tabular In-Context Learning (ICL) transformers, such as TabPFN and TabForestPFN, have shown strong performance on tabular classification tasks. In this paper, we introduce Attic, a new architecture for ICL-transformers. Unlike TabPFN and TabForestPFN, where one token represents all features of one observation, Attic assigns one token to each feature of every observation. This simple architectural change results in a significant performance boost. As a result, we can confidently say that neural networks outperform tree-based methods like XGBoost.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730485484743

thread_type:
official_review

<thread_object>
creator: reviewer_fvnr

<content>
summary:
Traditional ICL transformers like TabPFN and TabForestPFN use observation tokens, which can be influenced by the order of these features. To overcome these challenges, this paper introduces Attic, which employs cell tokens to represent individual observations x features, allowing the model to handle features without being affected by their order. As a result, the authors claim that Attic significantly outperforms traditional methods like XGBoost, CatBoost, and TabPFN in benchmark tests from WhyTables and Tabzilla. These findings suggest that Attic is a strong candidate, given its superior accuracy on various datasets.

strengths:
- The motivation is clear and well-founded. I like the idea of removing the feature order importance that is placed on classic ICL transformers. This is a nice thing to focus on and is a strength of this paper. 
- I appreciate the author’s writing style and delivery of the material. The paper reads well and the authors provide a nice, logical flow for their arguments. (See below for some suggestions for additional details which will improve the paper even more!)
- The authors’ results are promising and they certainly show their motivation and technical solution (observation x feature tokens) is a promising research direction!
- Evaluating their method against existing benchmarks (Tabzilla and WhyTables) is a very important and necessary step for any new method in the tabular data space. I appreciate the authors' care and consideration here.

weaknesses:
- More details are needed throughout the paper. For example, can the authors provide values instead of variables for architecture details: 
    - Token dictionary size, what is the value of $L$, dimension sizes, what values of $k$ did you test?etc
    - How is the tokenization performed for each observation x feature?
    - What are the model sizes for the -M and -L versions of TabForestPFN?
- You claim in L153: “We changed this because we believe this formulation is more natural, but performance-wise, it has little impact.” Please provide an ablation describing and proving this claim. 
- The authors need to highlight more clearly for the reader that this work and its findings are limited to datasets with 10 or fewer classes. 
- Do the authors re-run TabPFN for the Tabzilla benchmark? or do they take the TabPFN results from the Tabzilla paper/results?
- What model is being reported in Tables 6 and 7 for TabForestPFN when you describe at least 3 different versions you use in your main paper? This actually is an important question/distinction the authors need to make through the paper. It is very hard to evaluate the paper without knowing this since the 3 versions of TabForestPFN are so different and the evaluation of the paper rests on understanding which version is being discussed in each results section. With further clarity, I’ll be able to update my score accordingly. 
- Can the authors confirm the train/val/test splits they use are consistent with what the authors used in WhyTress and Tabzilla? My concern here is that tabular data use cases are not limited in the ways the authors may have selected their subset of Tabzilla. Tabzilla, which quickly has become a gold-standard benchmark in tabular data evaluations, is vast and captures a lot of nuance in the tabular community and use cases. I’m concerned the author’s down-selecting is increasing hiding some flaws of the approach. 
- Can the authors please describe why they omit about half of the Tabzilla benchmark? 
- Can the authors provide details on the computation resources? They provide comparisons to TabForestPFN primarily, but they should also provide comparisons to TabPFN and CatBoost at least — preferably more. Computational complexity is one of the main downsides of this approach, so the authors need to provide more care here for us to evaluate the work appropriately.

questions:
See Weaknesses. I am certain that with satisfying clarifications on the questions above, I will happily raise my score.

Also, a minor typos:
- L221: “Additionally, pretraining Attic on smaller datasets than TabForestPFN favors TabForestPFN”. This typo makes the entirely of the paragraph of L219 hard to comprehend.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730492836190

thread_type:
official_review

<thread_object>
creator: reviewer_56pb

<content>
summary:
The paper is on in context learning (ICL) for tabular prediction problems.
Essentially the complete tabular training data is placed in context and predictions are performed on test data. The main difference to existing work is that in the proposed approach each cell value is represented individually as compared to an entire row.
The approach is evaluated on both classification and regression problems and compared to existing ICL approaches as well as standard ML models such as gradient boosted trees.

strengths:
In theory the idea of representing feature values individually seems very reasonable to me since it is allows a richer representation of the data. They also show that extending the existing architectures (TabPFN/TabForestPFN) accordingly can be performed pretty smoothly (Figure 1). 

The evaluation is performed on more than 100 benchmarks and the authors consider both classification and regression problems.

The performance on classification problems (see Table 2 and Table3) seems to outperform a wide range of existing approaches.

Section 4.6 and 4.7 provide interesting insights into the decision boundaries obtained by the proposed models as well as some indication on which kind of benchmarks the approach does not work so well.

weaknesses:
First, to me there are the following limitations of the proposed approach:
1. Representing each value is sensible, but it does come at a substantial computational cost. The growth is in the number of columns and there are data sets with more than a thousand columns. Now, one can argue that for the ICL setting there will always be limitations (e.g., 1 billion rows and 1k columns is unlikely to happen). However, when looking at Table 2 TabForestPFN is only marginally worse than Attic (ZeroShot) but much more cost effective due to the more abstract representation. 
2. On regression tasks (Figure 6) the performance is by far not as good. And that is actually interesting and not necessarily something negative about this approach. While in theory classification and regression are the same, it does seem to pose issues especially for DL approaches in general and not just the approach presented here.


Moreover, there are some points in the paper that are unclear to me.
For instance, in Section 4.2 it states that 94 out of 176 benchmarks are used from TabZilla.
One question is how those were selected, but to me the more important question is how they are split up into classification and regression tasks (this is explained for the WhyTrees benchmarks but not TabZilla).
Looking at Table 2, it seems that there only 28 classification tasks.This number of benchmarks weakens the results in my opinion - especially given that on regression it is known not to work as well.

In terms of hyper-parameter optimization - what was the experimental setup? For instance, was there a split of train/validation to perform  HPO and then the evaluation on test? To me this is not stated clearly in the paper.
One could also pick a bit more sophisticated HPO approach (e.g., https://github.com/hyperopt/hyperopt-sklearn ), but at least random search is used (and not grid search).

At the end of Section 3 it is stated: "We believe that this dependency on feature order leads to training inefficiencies. The cell-token
ICL-transformer treats each feature the same and learns how to construct relationships between features."
And while I agree with this intuition, why not perform an ablation study on it (e.g., shuffling columns and see if it impacts the performance)?

Similarly, it would be useful to try to explain the results in Table 3 a bit more - is it that the model needs the fine-tuning to embed the categorical values appropriately? The gap between fine-tuned and zero-shot is very large and it would be great to understand the reason for it better.

questions:
In addition to the above questions:

Is it possible to show if the differences in performances shown in Table 2 are statistically significant?

Figure 4 is interesting - but isn't it the case that as the decision boundaries become more and more detailed, the generalization capability of the approach is reduced at the same time?

In terms of computational complexity is there a way to show computational cost (e.g., as a new column in Table 2)?
The question here is of course what the metric in this context could be - maybe the combination of runtime/memory usage/GPU requirement (for GPU use: number of input/output tokens) would already be insightful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730492959885

thread_type:
official_review

<thread_object>
creator: reviewer_qp5e

<content>
summary:
The authors propose a new tabular in-context learning algorithm. They introduce a new dimension in the model input by modeling each feature as one token rather than each sample. This allows for feature order invariance at the cost of extra computation. The proposed model, ATTIC, makes substantial performance gains.

strengths:
- The experimental results of the method is really exciting. As someone very familiar with the baselines, ATTIC seems to address a major performance bottleneck in tabular in-context learning.

- The authors provide an simple intuitive approach, encode each feature seperately to overcome feature order invariance instead of using ensembling, which provides large performance improvements.

- The authors provide results on several hard benchmarks and achieve performance with no hyperparameter tuning. The decision boundaries results provide further evidence for ATTIC's efficacy.

- The authors provide a detailed comparison of Attic compared to TabForestPFN for easy reproducibility.

weaknesses:
Overall, I think some more in-depth analysis into the runtime and memory tradeoffs Attic makes to achieve its superior performance can greatly benefit the paper.

- Because Attic introduces a new dimension over the feature counts, I imagine the runtime costs greatly increase. How much does this increase with respect to the number of features?

- Similar to the previous question. How does the memory costs of Attic rise with respect tothe number of features?

- My understanding is TabPFN overcomes feature order invariance through ensembling across multiple feature shufflings. Could you discuss the tradeoff between using larger TabPFN ensembles and Attic, as both incur additional runtime costs?

- Intuitively, I feel larger feature count datasets would benefit the most from Attic. Empirically, are there trends on what specific datasets Attic most improves upon baseline algorithms?

questions:
See Weaknesses.

Will you open-source your code?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730834725389

thread_type:
official_review

<thread_object>
creator: reviewer_m8oc

<content>
summary:
This paper presents Attic, a novel transformer architecture for tabular in-context learning that uses cell tokens instead of observation tokens. The work demonstrates significant empirical improvements over existing methods and makes a meaningful contribution to the field.

strengths:
* The architectural modification from observation tokens to cell tokens is simple yet effective, showing substantial performance gains across multiple benchmarks
* Thorough empirical evaluation on two major benchmarks (WhyTrees and TabZilla) with comprehensive comparisons against state-of-the-art methods
* Technical contribution backed by clear motivation and intuitive explanations for why cell tokens may work better than observation tokens
* Impressive results showing Attic outperforming tree-based methods and ensemble approaches like AutoGluon on several datasets
* Detailed ablation studies and analysis of computational requirements and decision boundaries

weaknesses:
1. Limited theoretical analysis or formal justification for why cell tokens perform better.
2. The authors note that Attic is significantly slower and more memory-intensive than TabForestPFN for datasets with many features. This scalability issue could limit Attic's practical applicability to large, high-dimensional datasets.
3. Mixed precision training issues are not fully resolved, with float16 training instability remaining unexplained.
4. Initial regression results feel preliminary and could be expanded. 
5. More detailed analysis of failure cases where tree-based methods outperform Attic. Why, in datasets with less than 500 observations, does Attic underperform?
6. The paper does not thoroughly explore potential limitations or failure cases of Attic, which would provide a more balanced view of the method's capabilities.

questions:
1. Any potential approaches to reduce memory and computational requirements of Attic?
2. How does Attic handle missing data or categorical variables? Are there any special preprocessing steps required?
3. Can you elaborate on Attic's performance in regression tasks compared to classification?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733288302843

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Statement

comment:
Dear Reviewers,

As the rebuttal period comes to a close, we would like to express our gratitude for the valuable reviews and feedback we received.

Our paper introduces a new architecture for tabular in-context learning, with the primary modification being a switch from an observation-token representation to a cell-token representation. This change resulted in a significant performance increase under the same pretraining compute budget. Based on the reviews and our subsequent discussions during the rebuttal, we would like to summarize the strengths and weaknesses highlighted by multiple reviewers.

**Strengths**

All reviewers agreed on the strengths of the paper. They praised the clear motivation of the method as well as the logical flow of the work. The token modification was described as intuitive and interesting. Additionally, the reviewers were impressed by the performance and appreciated our use of existing benchmarks in literature.

**Weaknesses** 

One commonly noted weakness is the increased finetuning runtime of our proposed architecture, Attic. We acknowledge this concern. In our rebuttal, we argued that achieving higher accuracy should be the primary focus initially, as future work can address runtime optimizations using the numerous efficiency techniques developed for large language models. Applying these techniques is more effective after establishing a high-performing model, which is why we believe our Attic architecture is an important contribution.

The second point raised was the lack of certain implementation details in the paper. Specifically, two reviewers requested more information on how we implemented the TabZilla benchmark, including details on the train/validation/test splits, hyperparameter settings, and dataset selection from the benchmark. We provided all these details during the rebuttal, which seemed to resolve the concerns.

Once again, we appreciate your time and feedback. Your insights have been invaluable in refining our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1754282160445

thread_type:
public_comment

<thread_object>
creator: ~xunye_tian1

<content>
title:
Open-source code link

comment:
Can I know the open-source code link for this work?
Thanks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Attic, a new architecture for in-context learning on tabular data, replacing the embedding per data point that Tab(Forest)PFN uses by an embedding for every data point and feature. The reviewers agree that the modification makes intuitive sense; the main strength is that results are strong compared to established baselines, on common benchmarks. The reviewers criticize the lack of theoretical justification, complexity (memory and runtime), unresolved instabilities in mixed-precision training, and fairness of evaluation due to the longer training time of Attic. Due to a combination of these issues, I am leaning towards rejection of the current version. The authors have improved the paper over the course of the rebuttal, and I encourage them to resubmit their work to the next venue.