EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network
Authors: Michael Arbel
David Salinas
Frank Hutter
Venue: NeurIPS 2025 poster ()
Keywords: tabular data
equivariance
foundation model
TabPFN
Primary area: deep_learning
TL;DR: We introduce a target-permutation equivariant foundational model for tabular data

Abstract: Recent foundational models for tabular data, such as TabPFN, excel at adapting to new tasks via in-context learning but remain constrained to a fixed, pre-defined number of target dimensions—often necessitating costly ensembling strategies. 
We trace this constraint to a deeper architectural shortcoming: these models lack target-equivariance, so that permuting target-dimension orderings alters their predictions. This deficiency gives rise to an irreducible “equivariance gap,” an error term that introduces instability in predictions. We eliminate this gap by designing a fully target-equivariant architecture—ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism. Empirical evaluation on standard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.

EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network
Michael Arbel"
David Salinas*2.3 Frank Hutter
INRIA
University of Freiburg
$ELLIS Institute Tübingen "rriorLabs
Equal contribution
Abstract
Recent foundational models for tabular data, such as TabPFN, excel at adapting to new tasks via in-context learning, but remain constrained to a fixed, pre-defined number of target dimensions often necessitating costly ensembling strategies. We trace this constraint to deeper architectural shortcoming: these models lack target equivariance, so that permuting target dimension orderings alters their predictions. This deficiency gives rise to irreducible "cquivariance gap," eπror term that introduces instability in predictions. We eliminate this gap by designing a fully target-equivariant architecture ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism. Empirical evaluation on stan- dard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.
-ftee
Introduction
Tabular data, a prevalent format in many real-world applications, has historically presented unique challenges to deep learning due to its lack of inherent structure compared to image Foundation models, such as TabPFN [12], have recently been introduced to tackle classification tasks in tabular domains. These models leverage in-context learning capabilities of transformers [3], to perform both training and prediction in single model evaluation, without requiring any parameter updates, achieving remarkable performance.
data [10].
At the core of these foundational models a pre- tre-training procedure in which a transformer model is trained to predict test targets from test covariates, conditioned training covariate/target pairs all of which are randomly sampled from some well-designed generative model. While it might seem surprising, at first, how model pre- trained on synthetic data could perform well real unseen data, recent work, such as Nagler [26], provides a theoretical study that shades some light this phenomenon. These models, leverage the transformer architecture perform attention over rows-attending to all samples simultaneously to enable cross-sample comparison. Applying attention over rows crucial, enables the model capture higher-order similarities between samples while preserving an inherent symmetry of tabular data-namely, that row order is irrelevant under the common i.i.d. assumption in supervised learning. Conveniently, such mechanism also allows, in theory, handling an arbitrary number of training samples property that enables the model generalize across tasks with varying dataset sizes without architectural modifications.
However, models such as TabPFN [12] are inherently confined to covariate- target pairs of fixed, predefined dimension, thereby limiting their applicability to datasets that match these specifications. This limitation can be alleviated via additional data pre-processing-ee e.g. projecting high-dimensional covariates into lower-dimensional subspace by post-processing model predictions, e-g. ploying hierarchical strategies for classification tasks with many classes [31, 5]. However, the increased computational burden often offsets some of the benefits offered by in-context learning.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent work in [24, 13] have partially addressed these challenges, allowing the model to handle arbitrary number of covariates, albeit, requiring the target dimension to have a fixed pre-defined di- mension. A key insight there, is to exploit another inherent symmetry of tabular data: the arrangement of columns/covariates's dimensions should not influence model predictions. This is achieved through the bi-attention mechanism which alternates between attention over samples/rows and columns, i.e. covariates dimensions, thereby making the model equivariant to feature permutations just as it is equivariant to sample permutations. Nevertheless, these models remain limited to tasks where the target size matches the predefined dimensionality. While covariates are provided for both training and test samples, only training targets are available the model. This asymmetry between training and test samples complicates direct extensions of the above approaches to handle the targets.
In this work, we propose EquiTabPFN, a novel architecture that enforces target equivariance, thus ensuring more robust and consistent predictions while handling targets of arbitrary dimensions. Unlike feature equivariance which can be directly obtained using bi-attention mechanism, we achieve target equivariance by carefully combining three different mechanisms: (1) Bi-attention across covariates/target components and datapoints, (2) Prediction tokens to replace unavailable test targets, and (3) A non-parametric decoder preserving equivariance in predictions. We then establish the importance of target equivariance through theoretical and empirical analyses. In our theoretical study, we show that optimal functions for the pre-training procedure must necessarily be target-equivan eenivvrauivar..nt. Finally, we demonstrate, on real-world datasets, that aarget-quivariann models are beneficial both in terms of classification performance and inference runtime.
2
Related Work
Since the introduction of TabPFN in Hollmann et al. [12], which demon- strates how such a model can be successfully trained on synthetic data, several works have leveraged this architecture for applications such as Bayesian Optimization [23], forecasting [7], learning curve extrapolation [1], and fairness analysis [29]. Aside from Hollmann et al. [13], Müller al. [24] that proposed a covariate equivariant version of the TabPFN models to better capture natural symmetries of tabular data, other works focused on improving scalability and speed of the model. This includes Müller et al. [22] who proposed to pre-train the model for producing the weights on an MLP by in-context learning, so that the resulting MLP performs well on test portion of a particular dataset while achieving much lower latency. Recently, Qu et al. [27] improved the scalability w.r.t. to the sample size by introducing a two-stage architecture that first builds -ideedinniio embeddings of rows, followed by a transformer for efficient in-context learning. All these approaches, still require pre-defined fixed target dimension. The focus is orthogonal to ours, as we specifically analyze and enhance the target's representation of TabPFN family of architectures.
Prior-Fitted Networks.
Modifying Frior-Fitted Networks outputs. To the best of our knowledge, no previous approach has proposed a target-equivariant architecture for foundational tabular models, with the exception of the concurrent work from Koshil et al. [16] which unlike us avoid to process labels with non-linear outputs to focus on interpretability versus accuracy. Several works have also proposed modifications to the output architecture of TabPFN. Müller et al. [24] introduced modification of the output, replacing the linear projection from token embeddings the target with Generalized Additive Models. This approach improves the interpretability of the model by constructing shape functions for each feature, which can be analyzed independently. Margeloiu et al. [19] and Ye et al. [36] explored combining nonpparametric models on top of TabPFN. Both approaches require training model at inference time unlike our method.
Beyond pre-defined target dimensionality. To address TabPFN's limitation to a predefined number of classes, Hollmann et al. [13] propose split the classification problem into smaller ones on which the model can be employed, then to aggregate predictions using a strategy, such as the one based on an error-correctin output codes (ECOC) [5]. In this work, we show that such a strategy results in an increased computational cost compared to using our architecture that is natively target equivariant. Qu et al. [27] proposed to use a hierarchical classification strategy [31] which still incurs an increased computational cost. Recently, Wu and Bergman [35] propose a mechanism to handle arbitrary number of classes without target equivariance and require a different paradigm involving an adversarial training procedure.
2

---

Equivariance beyond tabular methods. Designing equivariant architectures [4] has long been recognized as beneficial, with the most prominent example being convolutional neural networks, which are equivariant to image translation [17]. More recently, research has focused on designing architectures with other symmetries, such as those present in spherical [9], set [37], or graph data [30]. Recent work has also explored incorporating symmetries in Large Language Models. For instance, Egressy and Stühmer [8] proposed a modification to self-attention that produces outputs equivariant to permutations of multiple-choice options, ensuring that the order of choices does not affect the result-a property known to be important in LLM applications such as LLM judges [39].
3 Background on Prior-Fitted Networks
Hollmann et al. [12] introduced a pre-trained model, TabPFN, that leverages the transformer architec- ture [33] to perform in-context learning on unseen tabular datasets for classification tasks without the need for any further training. Specifically, given training and test datasets of input-ouiput pairs (X == (Ins Un) and test samples (X", Y*) ) == (Itm I y "²)===> TabPFN returns prediction Y = fX.Y (X*), where fx.y (X*) is the output of the network when provided with the training collection (X, Y) and test queries X Here the imput vectors In and 2 belong to euclidean space IP, while classes are represented by one-hot vectors Yn E R7.
We now briefly describe the three modules of TabPFN model: an encoder, a backbone and decoder, as it will help identify the main architectural constrains that impose a limit on the number of classes.
Linear Encoder. The encoder module constructs training and test tokens (en) and (et) ) that are provided the transformer backbone assuming the inputs æ and y are vectors of fixed dimensions p and gmoz. Each training token e,, is obtained by linearly embedding both covariate In and target yn into feature space of fixed dimension d and then summing both embeddings, i.e. en = Uz, + Vyn, where U and V are trainable matrices of sizes d x p and d x q. On the other hand, the test token consists only in embedding the test covariate x;, i.e. em U x since the test target ym is not provided to the network. While targets with smaller dimensions can be handled by simple rero-padding procedure (see Hollmann et al. [12]), the encoder cannot easily accommodate target data with dimensions greater than q.
Transformer backbone. The backbone consists of a succession of residual multi-head self-attention layers between all tokens followed by residual feed-forward network applied to each token. In order avoid information leakage from test train data, attention mask ensures that all tokens can only attend to the training tokens. This also ensures that test tokens are processed independently from each other. The residual connections in the backbone preserve the initial feature dimension d, so that each token is still associated to a particular sample while gradually incorporating information from all training tokens.
MLP decoder. The decoder consists of a one-hidden layer MLP that takes each test output token et produced by the transformer backbone and produces prediction vector f.. of dimension q. As the decoder requires a pre-defined target dimension it cannot be used post-hoc on new data with higher target dimensions.
notable property the TabPFN architecture its invariance of the test predictions to the order by which training and test points are provided. This property is desirable since the order of training points is arbitrary and should not influence predictions on test samples. However, the network lacks equivariance w.r.t. the targets' dimensions, meaning that predictions are highly dependent the order by which the training target dimensions are provided, an undesirable property as we show in theoretical analysis in Section 5.
Target equivariant prior-fitted network
We introduce EquiTabPFN, new model architecture for in-context learning on tabular data, that is permutation equivariant w.r.t. the target's components, Our architecture integrates self-attention mechanisms across data points and data components to leverage relationships between datapoints while preserving equivariance by processing individual attributes (such as the targets components). Unlike TabPFN which requires fixed target dimensions for all datasets, EquiTabPFN allows the dimensions of the target to change depending on the dataset as consequence of its equivariant
3

---

Pradiction (x,00*)
Decoder
LX
Self-Attention
Training objective Y0
Altention
Enooder txi Cons.
Self-Attantion across Features
Prodiation IDviees
Linear projection
Tiast
Figure 1: Overview EquiTabPEN's architecture. Data is tokenized via an encoder, processed using self-attention, and decoded to obtain predictions. The encoder maps each covariate single token and embeds target components into tokens via a x 1 convolution. Missing test tokens are replaced by prediction tokens. Self-attention alternates between (1) feature-wise attention, with target tokens attending only to covariate tokens (gray arrows) while covariate tokens attend to all tokens (blue arrows); (2) Data-wise attention, where test tokens attend only training tokens (blue arrows), and training tokens attend to themselves (gray arrows).
architecture. EquiTabbPFN consists of three major modules: a target equivariant encoder, an attention module both across data components and across data points, and non-parametric equivariant decoder, each designed to facilitate the end-to-end learning of components interactions and datapoint relationships, see Figure 1. Below, we elaborate on each module and their interactions.
4.1 Target equivariant encoder
The encoder constructs training and test tokens by applying a linear projection to both covariates and targets so that target equivariance is preserved. Specifically, following [12], each training and test covariate vector In and 2 encoded into a single token of dimension d by applying a linear projection matrix U of size d x p. However, instead of adding a linear projection of the training targets to each corresponding training token, as done in the case of TabPFN (see Section 3), we compute a token for each component j of training target by multiplying them with embedding vector V of dimension for all S S q- This operation amounts to applying 1 x convolution along the components of each target which preserves target equivariance. Since, the validation target Y** is not provided to the model as input, it is replaced by a trainable prediction token W Wpeed of dimension d that is repeated M x times to form an initial guess yo of the target. When considering batch B of B datasets ((X, Y), (X*, Y*)) E same dimensions, all these embeddings along with prediction tokens are collected to form single tensor E of shape (B, N + M, + 1, d). Here, for each batch element ((X, Y), (X*, Y*) of index b, the blocks E...N.I.. E..1:M.1.: correspond to embeddings of X and X*, Eb,.N,1:g:: represents the embedding of Y while 1:,. denotes the initial guess o obtained using the prediction token. This tensor is then processed by the attention modules as described next.
4.2 Self-Attention Mechanisms
The core of the architecture involves two alternating self-attention modules: self-attention across components SelfAtt, and self- self-attention across datapoints SelfAtt, used for transforming the tokens. These alternating self-attention layers allow the model to learn both intra- satrlesample components interactions and sateles relationships. Following standard design choices for transformers, we apply residual connections and layer normalization to ensure stability and robust gradient flow, i.c.:
E - LN E + SelfAttt:/(E))).
E - LN MLP(E)),
where LN denotes the layer normalization layer [2). SelfAtte/b denotes one of the considered self-attention mechanisms and MI.P is a one hidden-layer network acting on each embedding independently. Below, describe both self-attention mechanisms in more detail.

---

Self-attention across components allows interactions among components within each datapoint. It applied independently per samples to preserve equivariance w.r.t the samples. In practice, this is achieved by reshaping the activation into tensor of the shape (B x (N + M) q - 1, d) before applying attention between q + covariate tokens of dimension d. We further employ masking strategy that we found useful empirically: forcing target tokens to attend only to the covariate token, while allowing the covariate token to attend to all tokens.
Self-A Aelf-Attention across datapoints captures relationships between datapoint embeddings, allowing the model aggregate information globally. It is applied between samples and independently per each input dimensions and to preserve equivariance. In practice, this is achieved by reshaping the activation into a tensor of the shape (B x (q 1), .N + .M, d) before applying attention between N + M training and validation tokens of dimension d. Similarly to [12], training and validation tokens only attend to training tokens.
Remark. EquiTabPFN and TabPFN both have linear computational complexity in the number of classes, but EquiTabPPN has larger factor due to self-attention across components. In TabPFN, linear scaling arises from projecting classes into a -iee-diieesiooal space and then mapping back fixed dimensional features classes, while the backbone remains independent of the number of classes. In contrast, EquiTabPFN''s backbone scales linearly with class number, allowing it to handle arbitrary class counts.
4.3 Non-parametric equivariant decoder
The decoder aggregates the processed embeddings to produce prediction Y. This is achieved in two steps: an attention module first computes an intermediate prediction Y = (fm ) in the form of weighted average of training targets Y. then a residual correction is added produce the final prediction. More precisely, the attention module uses the embeddings of the training and validation samples as keys and queries, while the attention values are simply the training targets
Y, i.e. fm E SofoMMaxx (Ei, F.b.n,i......,,...... /V + q)d where b is the batch-index corresponding to the training target Y. The residual correction, in the form of a point-wise MIP, operates independently on each dimension of the attention output Im so that equivariance is preserved while enabling nonlinear interactions between training values.
Without the residual correction and removing the dependence of the keys and queries embeddings on the training targets Y (for instance by setting the weights of the target encoder and pointwise MLP to 0), the decoder becomes linear con-paraettric regression estimator [32, Definition 1.7], which is generalization of Nadaraya- Watson's 's estimator [25, 34]. However, linear estimators are known to be suboptimal compared non-linear ones [6]. This motivates introducing a nonlinear dependence of the estimator to Y, in our setting, to increase the expressiveness of the decoder allowing it to adapt to the prediction task at hand. Experimentally, we also found clear improvement when adding such residual correction and making the embeddings dependent on the targets.
4.4 Pre-training Procedure
EquiTTabPFN can be pre-trained using the same procedure as in Hollmann et al. [12], on arti- ficial datasets sampled from a sophisticated generative model meant to capture the real-world distribution of datasets. More precisely, each artificial dataset consists of training/and test splits (X,Y) Y) := (In, yn) ) and (X*, YY) (=tn, Von MM m=1 sampled according to a conditional distri- bution p(r, ylub) characterized by a latent parameter th. The parameter / characterizes the dataset and itself sampled according to predefined prior p(t). The pre-training procedure requires repeatedly generating artificial datasets and training the model predict test target values Y given corresponding test covariates X as well as training covariates/tarett pairs (X, Y') by minimizing an objective of the form:
[()) == = [e ( (X*), YY*).
(1)
Here, (v.3/) + E(y.v)) I is a wiint-wise loss, typically cross- coss-entropy, and the expectation is over the collections datasets sampled according to the dataset prior. Note that each test query at, is processed independently by the network ʃ so that fx,y (X*) (Sx,y ) Next, we show, under natural conditions, that target equivariant functions constitute the right space of functions when searching for solutions to objectives of the form in Equation (1).
5

---

Target permutation equivariance and prior-fitted networks
We now formally analyze the impact of non-target equivariance on the training objective. We begin by precisely defining target equivariance, then show that optimal solution must be equivariant or otherwise incurs an error quantified by the target equivariance gap. Finally, through empirical analysis of TabPFN training, illustrate how the equivariance gap decreases slowly, highlighting the fundamental challenge non-equivariant architectures face in learning this key data symmetry.
5.1 Optimality of target equivariant networks
When presenting a new unseen dataset of covariate/target pairs (No, Do Vo)n= to a pre-trained model, the order of the component's target is arbitrary. In other words, given target vectors of the form yn ((v))...... (ym ),), these could as well be presented in a different order by applying a permutation a to the components of Yn to obtain a permuted target (vn) = ((Un)o()) transformed dataset (In, (yn)) is still essentially the same as the original dataset up to the permutation as we only changed the order of the target components. For instance, when the target represents a one hot encoding vector of 2 classes: "red" or "blue". it should not matter whether we encode "red" as the first or the second class in the one-hot vector. Consequently, a pre-trained model should be able to provide consistent predictions regardless of the component's order. More formally, the model should satisfy the following equivariance property:
(Un)o(a)) The
Definition 5.1 (Target permutation equivariance). A function f is permutation equivariant in the largets' components iff for any training data (X, (X,Y) and test covariates X*:
Yo E Gg. o- (fx,0r)) (X*)) = fx, (X*),
(2)
where G, denotes the set of all possible permutations of the components of vector of q elements.
is clear from Theorem 5.1 that EquiTabPFN is target equivariant. Even when a model is not target equivariant by construction, it is natural to expect it to learn to be target equivariant, when trained via the objective in Equation (1) over a large class of randomly sampled datasets. To formalize this, define the target equivariance gap, which quantifies the deviation of function from its symmetrized counterpart fequ.
Definition 5.2 Farget-equivaranne gap). The target- enget--quivariince gap Eequi (f) of a function f w.r.t. L is the difference between the objective values at J and its symmetrized version fequi:
(eq == L(f) L(
where fequi is obtained by applying the following averaging operation w.r.t to the uniform distribution E. over all permutations of the target:
JX.Y (x*) = E. (fx,tr) (X+))]
4
By construction, sequi is permutation equivariant w.r.t. the target components. Moreover it can be easily shown that a function f is itself equivariant iff lequi = J, so that the gap vanishes. In general, the equivariance gap can take negative values. However, we establish later that this equivariance gap must be non-negative under the following assumptions on the pointwise loss and the marginal distribution p of the data:
(A) Invariance and convexity of the pointwise loss. The pointwise loss & is strictly convex in its first argument and is invariant to permutations of the components of its arguments, i.e. for any permutation a, it holds that: E(a(v), (I))) = E(y.
(B) Invariance of the data distribution. The marginal distribution of the data is invariant to permutations applied to and Y*, i.e.: p(X, ( (Y), X*, = pXX, X,X*,
Assumption (A) is satisfied for most commonly used losses such as the cross-entropy loss or the quadratic loss. Assumption (B) holds as soon as data can be presented without a preferred ordering, as most i.i.d. tabular datasets. The next proposition, proved in Section A, decomposes the objective into a non-negative equivariance gap and optimality error.

---

Targets permuted o; Targets permuted Targets permuted
6x
TebPFN
4x10-
EQuivariiees
10-1
153600
460800
Number of gradient iterations
Equteenacce eree
0.2
Ex EquiTabPPN
0.1
16
10
Number of clasess
Figure 2: Prediction comparison TabPEN, TabPFN-v2, and our model on the same datasets with three different class orderings (one columa). Models predict on dense grid using 9 distinct training points, marked with dark crosses, each having distinet class.
Figure Equivariance error for TabPEN observed while training (top) and inference with different number of classes and ensembles (bottom).
Proposition 5.3. Under Assumptions (A) and (B), the equivariance gap Eoqui (f) is always non- negative and only equal to 0 when f is equivariant to permutations, so that for any
L()- = f( + & (J) 2 L(
Moreover, if f minimizer of over measurable functions, then f must be target equivariant.
expectation over permutations using Assumptions (A) and (B) on the data and loss: L()) = E, [o (X*), Y+)]- The non-negativity of the equivariance gap is then established using Jensen's 's inequality by convexity of the loss f its first argument (Assumption (A)). Now, assume by contradiction that f not equivariant and note that f*equi a measurable function by construction. follows that goqui (oqui(** > 0, which directly implies that C(f*) > C( f( ), thus contradicting the optimality of f*
Proof sketch. The key step is to express the objective
Theorem 5.3 shows that minimizing the objective L results in target equivariant function. Hence, using a non-equivariant model must employ some of its expressive power solely for being equivariant, which can be wasteful, as we verify empirically in Section 5.2 below.
5.2 Non-equivariance of TabPFNs models
PFNs models, as introduced in Hollmann et al. [12, 13] are not permutation equivariant the target's components. Consequently, they not guaranteed to provide consistent predictions when the target components are permuted, thus affecting their robustness.
Predictions instabilities. To illustrate the implications of non-equivariance on the robustness of PFNs models, we consider toy classification problem in 2 dimensions, where training points are positioned 2-dimensional regular grid, each corresponding to different class inspired by McCarter [20]. Different pretrained models are then used to predict the classes on a regular grid of 402 points. Figure shows the classification boundaries when using the same data but with 3 different orderings for the classes. is clear that the ordering heavily affects the prediction even in this simple example, which strongly impacts robustness of models like TabPFN and TabPFN-v2. The predictions of TabPFN-v2 are particularly noisy due to having only 9 training data points which is not issue in itself, unlike the extreme unstability to the class ordering. Note that the axis are scaled differently for presentation.
Target equivariance gap during training. In Figure 3, we analyse the equivariance error while training TabPFN. Figure 3 (left) shows the equivariance error of TabPFN in terms of percentage of violation of Equation (2), e.g. how frequently the predicted classes (xx+ and a (fx,y (X*)) differ. We sample 512 datasets from the prior and report standard deviation. In Figure (left), the equivariance error is clear and slowly decreases during training. This non-qquivariance 1) induces
7

---

additional enrors for the model as demonstrated in Theorem 5.3 and 2) causes the model to provide surprising predictions given that permuting the output order can change the results as seen in Figure 2.
Mitigation via castly ensembling. Hollmann et al. [12] introduce a mitigation strategy to non- equivariance by averaging several predictions using random permutations of the target dimensions. Averaging over all possible permutations gives an equivariant function as discussed in Equation (4). However, this requires making Olg!) calls to the model, where q is the number of classes. This becomes quickly prohibitive, even for q = 10 as considered in the original study. Randomized estimators, using Nens random permutations can compute a prediction that converge to the averaged one at rate of 1/v/ens. However, the variance of such estimator would typically present a dependence on the dimension that is, unfortunately, challenging to quantify in general. This variance is highest when the function is far from being equivariant, constituting the worst scenarios. We illustrate how fast the model becomes equivariant when using ensembles in Figure 3 (right). While ensembling more permutation helps to make the model more equivariant, many ensembles are required, in particular, when considering more classes. In contrast, the model we propose is fully larget equivariant so that predictions remain the same regardless of the class ordering illustrated in Figure (bottom).
6 Experiments
6.1 Experimental setup
Pretraining. We trained our model, EquiTabPFN on artificial dataset extending the public code of Müller et al. [22] and using the same procedure from Hollmann et al. [12] employed to train TabPFNvI. Crucially, it was trained on classification tasks with less than 10 classes. The total training time took approximately 4 days on a single A 1100 GPU with 80GB of memory. We set our architectures hyperparameters so that they match the number of parameters of previous work, in particular the model we trained contains 255 parameters similar to the basclines we consider and we match most our hyperparameters to the same values as Hollmann et al. [12]. We refer to Section B for further training details and description of the hyperparameters used.
Benchmarks. For evaluation, we consider classification tasks from the TahZilla benchmark [21]. To assess the impact of unseen class counts during evaluation (i.e., exceeding 10), we consider two setups: one 76 multi-class tasks with at most 10 classes, and another on 10 tasks with more than 10 classes. Details of the datasets/tasks used are provided Tables and 2. In all results, except those of Figure 5 (left), we report the results of EquiTabPFN without ensembling as it is not critical for the performance of our method.
Baselines. We consider baselines from the TabZilla benchmark. addition, we compare with TabPFNv2, the state-of-the-art model with open-weights released by [13] and which was shown to out-perform standard baselines, including those appearing the TabZilla benchmark. This model was pretrained using improved prior compared to initial version TabPFNv1. However, the code for the prior and the training is not publicly available. Hence, to allow a fair comparaison, we also include second version, TabPFNv2 using the exact same architecture as TabPFNy2, but which trained on the same publicly available prior and training code of [22] that we used for our model.
6.2 Main results
This section presents the main experimental findings. Additional results, including ablation of the EquiTabPFN architecture and computational cost comparisons are found in Sections C.I and C.2.
EquiTabPEN enables in-context-learning on datasets with unseen class counts. Figure 4 shows accuracies relative to the KNN baseline for all models on 76 datasets with less than 10 classes (left) and 10 datasets with more than 10 classes (right). When considering datasets with more than 10 classes (right), Equi TaNPNN obtains the best median relative accuracy across all datasets. It strongly outperforms TabPFNv2, which performs worse than a linear model or random forests in terms of relative median. These results show that target-equivariance allows EquiTabPFN to seamlessly generalize to larger numbers of classes even though was trained on datasets with less than 10 classes, just like TabPFNv2. Additional evaluations in Tables 5 and 6 of Section C show that such
8

---

Classes
Datasets, Classes
TabPFNv2
EquiTabPFNi-11
EquiTabPFN
-G
TabPFNv1
rtdl *4
XGBoost
TabPFNv1 (0
RandomForest
LinearModeli-
TFV2
RandomForest (1
rtall ResNet
TabPFNv2
LinearModel
TabPFNv2 22
Tree
MLP
DecisionTree (1
rtall
TabNet
KNN
STG:-1
STG
rtdl MLP 62-
TabNet
MLP
VIME
VIME
0%
100%
100%000%%5000
0%
50%
100%
Relative Acc. improvement over KNN (f)
Relative Acc. improvement over KNN (t)
10
10
10
10a
Mean training evaluation time [s]
Mean training + evaluation [s]
Figure 4: Relative improvement KNN for datasets with less than 10 classes (left) and more than
10 classes (right). Red lines are the median metric over datasets after averaging each dataset over 10
splits. The runtime displayed with color on a log scale and is reported on V100 GPU for PFNs.
Model
50-
6.0%
EquibbRFN
Tab
ThePhi22
TabPrho2"
Seeedup (1BP7292 Eou
40
Foreso
XGBoost
ridl Rec/ee
3.0%
Ensembles
Parcto front
2.0%
1.5
16a
Number of Ensembles used for TobPFNv2
Figure 5: Left: Scatter plot of runtime vs % AUC improvement over KNN for different methods. Right: barplot of AUC ratio relatively KNN (blue) and speedup of EquiTaaPPNN (green). both figures, we increase the number of ensembles of TabPFNs variants.
improvement is consistent over various metrics: AUC, Accuracy and F1 score, for datasets with unseen class counts.
Competitive performance on datasets with class counts seen during pre-training. On the 76 datasets with less than 10 classes (Figure 4, left), EquiTabPFN performs comparably to the state-of the art method, TabPFNv2, even though it did not benefit from the improved data prior used in Hollmann et al. [13] as it is not publicly available. To assess the impact of the pre-training prior on performance, pre-trained the same network architecture TabPFNv2 using the same publicly available protocol used for training our method and reported the results for reference (TabPFNv2").
The results show that EquiTabPPN consistently outperforms TabPFNv2* both dataset with less than 10 classes and those with unseen class count. These results suggest that EquiTabPPN would likely benefit from the improved training procedure and prior of TabPFNv2.
Speedup over TabPFNy2 on datasets with unseen class counts. discussed Section 3, PFNs models cannot natively handle an arbitrary number of classes due to its decoder architecture. In order apply TabPFNv2 to problems with more than 10 classes, we employ the error-correcting output codes (ECOC) strategy [5] as recommended in Hollmann et al. [13]. Such approach requires

---

decomposing the classification problem into several smaller classification tasks with up to 10 classes, performing predictions for each sub-task using an ensemble of TabPFNv2 models, least one model per task, then aggregating predictions using ECOC strategy. This incurs an extra computational cost for performing ensembling. All results reported Figure 4 (right) are using the minimal number of ensembles to guarantee coverages of all classes. Despite this sophisticated aggregation strategy, the performance of TabPFNv2 degrades significantly, while still incurring a substantial slow-down compared to EquiTabPFN as shown by the average run-times reported by color in Figure 4 (left), Note that, for datasets with less than 10 classes (Figure 4, left) ensembling is not required anymore by TabPFNv2 and the runtime of both methods are comparable.
EquīTabPFN achieves the best tradeoff between performance and cost on datasets with unseen class counts. To further illustrate the improved trade-off between runtime and accuracy of our method, we show in Figure (left) the improvement in AUC relatively to KNN for all methods when both TvbPFNv2 and Equi' EquITab are allowed to have more ensembles. Ensembling generally improves performance of PFN models, it allows to make more robust predictions by averaging them over transformed version of the datasets, ex: by applying permutations of the labels order discussed in Section 5.2. While ensembling helps improve performance of TabPFNv2, the improvement is marginal compared to EquiTabbPNN without ensembling and comes at considerable computational cost, as also shown in Figure (right) and on the critical difference diagrams in Figures 7 and 8 of Section C. Therefore, amongst top performing methods, EquiTabPPN achieves the best trade-off.
7 Conclusion
Summary. this paper, we introduced EquiTabPFN, an architecture for PFNs that is equivariant to target permutations and enables in-context learning datasets with arbitrary class counts. We established optimality of cquivariant architectures for foundational tabular models and proved that non-equivariant models worsens the pre-training objective with incompressible error term. Finally, we empirically showed the benefits of EquiTabbPNN both in terms of classification performance and inference runtime synthetic and weal-world datasets. We hope this work enables future developments on prior-fitted networks that incorporate such fundamental symmetry of tabular data.
Non-equivarianee In some cases, the data may not be target equivariant, for instance on ordinal data. We found only few of those cases in the benchmarks used (5 of the datasets surveyed). Handling such cases would require to update the prior as it is currently equivariant to target permutation. The method could then be adapted by providing positional embedding or using column indices as input.
Limitations. EquiTabPFN requires quadratic extra-cost with the number of target dimensions to perform self-attention. While many tabular problems have relatively small number of target dimensions, this may become a problem for a very large number of dimensions and future work could consider efficient self-attention address this issue. This may be alleviated by work focused on improving the efficiency of PFNs models [28,38].
The code for training and evaluating our model is available httpa://githut: com/ Mi MichaelArbell Biichee/Arbal//AquTTaarII
Acknowledgments
This work supported by the ANR project BONSAI (grant ANR-23-CE23-0012-00 and was granted access to the HPC resources of IDRIS under the allocation [AD01 10151661 made by GENCI.
References
[1] S. Adriaensen, H. Rakotoarison, S. Müller, and F. Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2023.
[2] I. Ba, J. R. Kiros, and G. F. Hinton. Layer normalization. arXiv preprint arXiv: 1607.6606560 2016.
10

---

[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19001 2020.
[4] S. Cohen and M. Welling. Group equivariant convolutional networks, 2016. URL https:
/ //arxiv.. warxii.org//eooo.oooooooo wwt///g//////eeeeee//eeee
[5] T. G. Dietterich and G. Bakiri. Solving multiclass learning problems via enror-crrrecingg output codes. Journal of artificial intelligence research, 2:263-286, 1994.
[6] D. L. Donoho and I. M. Johnstone. Minimax estimation via wavelet shrinkage. The annals of Statistics, 26(3):879-921, 1998.
[7] S. Dooley, G. S. Khurana, C. Mohapatra, S. V. Naidu, and C. White. Forecastpfn: Synthetically- trained zero-shot forecasting, Advances in Neural Information Processing Systems, 36, 2024.
[8] B. Egressy and J. Stühmer. Set-Ilm: A permutation-invarian 1lm, 2025. URL https://arxiv. org/ /abbs/505.... 15433.
[9] C. Esteves, C. representations with spherical cnns, 2018. URL attpp://arrziv.org/abb. 0r1.0////b//1/107111..1.1
Makadia, and K. Daniilidis. Learning so(3) equivariant
[10] L. Grinsztajn. E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 507-520, 2022.
[11] S. Herbold. Autorank: A python package for automated ranking of classifiers. Journal of Open Source Software, 5(48):2173, 2020. doi: 10.21 101/100ss00217.. URL http:://d/o:...... org/ 10. 211 21105/joss. 02173.
[12] N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. TabPFN; A transformer that solves small tabular classification problems in a second. The Eleventh International Conference on Learning Representations, 2023.
[13] N. Hollmann, S. Müller, L.. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, T. Schirrmeis- ter, and F. Hutter. Accurate predictions on small data with tabular foundation model. Nature, 637((805))3119-3)
2025.
[14] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[16] M. Koshil, M. Feurer, and K. Eggensperger. In-context learning of soft nearest neighbor classifiers for intelligible tabular machine learning. In The 4th Table Representation Learning Workshop at ACL 2025, 2025. URL https: httpe:///ppeeevveeeeeee nww. peet/forummPidvvittppBBao
[17] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4): 541-551, 1989, doi: 10.1 1 162/neco. 1989.1.4.544.
[18] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with restarts. In Proceedings of the International Conference on Learning Representations (ICLR' 2017.
Published online: iclr.cc
[19] A. Margeloiu, A. Bazaga, N. Simidjievski, P. Lio, and M. Jamnik. TabMDA: Tabular manifold data augmentation for any classifier using transformers with in-context subsetting. In ICML 2024 Workshop on In-Context Learning, 2024. URL https: /ttpa/////e//eeeeeeeeeeee net/forum?id= tntVIbDdoD.
[20] C. McCarter. What exactly has tabPFN learned to do? In The Third Blogpost Track at ICLR 2024, 2024. URL http:///ppeeeeeeeeeeeeeeeeeeeeea
11

---

[21] D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White. When do neural nets outperform boosted trees on tabular data? Advances Neural Information Processing Systems, 36:76336-76369, 2023.
[22] A. Müller, C. Curino, and R. Ramakrishnan. Mothernet A foundational hypernetwork for tabular classification. arXiv preprint arXiv:2312.08899,, 2023.
[23] S. Müller, M. Feurer, N. Hollmann, and F. Hutter. PFNs4BO:
In-context learning for Bayesian optimization. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 25470.25470. PMLR, 23-29 Jul 2023. URL http::: htttps:/prroceedingg
mlr. wrw.s.rees/vveeeee hatpa///22mmmmmmlmeeee mmtmmmmmllaaaaa.mmmmm hmmm..mmm.
[24] A. Müller, J. Siems, H. Nori, D. Salinas, A. Zela, R. Caruana, and F. Hutter. Gamformer: In-context learning for generalized additive models. arXiv preprint arXiv:2410.0453 14.006000 2024.
[25] E. A. Nadaraya. On estimating regression. Theory of Probability Its Applications, 9(1): 141-142, 1964.
[26] T. Nagler. Statistical foundations of prior-data fitted networks. In International Conference Machine Learning, pages 25660-25677 PMLR, 2023.
[27] J. Qu, D. Holzmüller, G. Varoquaux, and M. L. Morvan. Tabicl: A tabular foundation model for in-context learning on large data. arXiv preprint 25X2::552..05544 2025.
[28] Qu, D. Holzmüller, G. Varoquaux, and M. L. Morvan. Tabicl: A tabular foundation model for in-context learning on large data, 2025. URL httts:://axiiv http://wawwwwl../.../e/ /25/222 05564.
[29] J. Robertson, N. Hollmann, N. Awad, and F. Hutter. Fairpfn: Transformers can do counterfactual fairness, 2024. URL https: org/ /t////207000733
[30] V. Satorras, E. Hoogeboom, and M. Welling. E(n) equivariant graph neural networks, 2022. URL https: /tttps://arxiv. org/ argg/bbs/2100 09844.
[31] C. N. Silla and A. A. Freitas. A survey of hierarchical classification across different application domains. Data mining and knowledge discovery, 22:31-72, 2011.
[32] A. B. Tsybakov. Introduction to Nonparametrie Estimation. Springer Series in Statistics. Springer- Veringer-Verlag,, 2009. ISBN 978-0-387-79051-0, doi: 10.1007/b13794.
[33] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
[34] G. S. Watson. Smooth regression analysis. Sankhya: The Indian Journal of &Stttssiics, Series A, pages 359-372, 1964.
[35] Y. Wu and D. L.. Bergman. Zero-shot meta-learning for tabular prediction tasks with adversarially pre-trained transformer, 2025. URL htt.s: /ttp///aa/vvv///////// org/ abs/ 2aabb//250.0000 04573..
[36] H.J. Ye, H.-H. Yin. D.-C. Zhan, and W.-L. Chao. Revisiting nearest neighbor for tabular data: deep tabular baseline two decades later. arXiv preprint arXiv:2407.03257, 2024.
[37] M. Zaheer, Kottur, S. Ravanbakhsh, B. Poczos, R. Salakhutdinov, and A. Smola. Deep sets, 2018. URL httph::///// //axxiv.///xxv...// wrgg//aa//03../555555
[38] Y. Zeng, T. Dinh, W. Kang, and A. C. Mueller. Tabflex: Scaling tabular learning millions with linear attention, 2025. URL htttps:////aiiii http://awxvv.rro/..//// absa 2506. 05588..
[39] L.. Zheng, W.LL. Chiang, Y. Sheng, S. Zhuang, 7. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot at//a//////..../////////////e
arena, 2023, URL https: h/opps://aae //arxxaxvvvv///////////ee//aaall org/abs/////eoooooooo..oo.a. /206/ 05685.
12

---

A Proofs
Proof of Theorem 5.3. By Assumption (B) we can express L as an expectation of the form:
(()) = E, [e (xx,o(r) (X*), o({)))
(5)
By Assumption (A), l is invariant to permutations, which allows to further write:
[(f) = 5, [e(o (fx,a(r) (X*)) .Y"]
(6)
We will show that (uq - ( is non-negative and vanishes only when / is equivariant.
This is a direct consequence of Jensen's 's inequality applied to the strictly convex function + E(y, v²) (Assumption (A)). Indeed, for any samples (X, Y, X*, Y*), the following holds:
8 ( (x*), (x))) = { [o-1 (fx (Xx)))
SE,l (o- (fx,ary) (X*)) ,Y").
where the first line follows by definition of foqui while the second line uses Jensen's inequality, Further taking the expectation w.r.t. p shows that goqu(() 2 O. If (f) = 0, then by the above incquality it holds that (IX (x*), xx* = e (fx,y (X*), Y*) almost surely. However, since is strictly convex in its first argument (Assumption (A)), the previous equality is only possible when frqui f almost surely, meaning that f is equivariant.
Finally, to show the final result, note that:
Eoqui (f) C(f) - ((f
Eo 11o- (fx.o(r) (X')) 1x. (X*) + 12. (x*) 1x))-1111) [(gequ))
Eo |10-1 (fx,or) Px
(X+))
Bo [o- (fx,a(x) (X*)) fx (x*))
( (x (x::)
E. |1o- (fx,(()) (X))) 18.1 (x*)12)
Ep (Es [o- (fx,or)) (x*))] IXYY (x+))
( (Xx) (X:)-
=0
Here, the cross cross-product term equals 0 since E, [o-] (fxo(r) (X+))] fx,Y (X*) by definition of pequi Hence, we have shown that:
gauu (f)= |1o-1 (fx.o()) -XX,Y (x+))
Finally, use the invariance of the squared error to permutations, the equivariance of fequi to permutations, and the invariance of p to permutations to get:
gequ (f) E.
[l (X*) (1R9) (xx)) 1P2] [llxx,(()) (X*)= - pequi
E,E.
(X*)- (xx))12)
(xo())
Ep [Ixx,r (X*) 1xy (x+)12)
B Additional Experimental details
Training procedure, We use similar training protocol as in Hollmann al. [12], in which the model is trained on classification datasets generated according to their proposed artificial dataset prior.
13

---

In this protocol, each dataset has fixed size of 1024 and is split into training and test uniformly random. The maximum number of classes is fixed to 10, while the maximum dimension of the covariate vector is fixed to 100. Following Müller et al. [22], we represent the target y as a one-hot encoding vector whose dimension is the number of classes in the dataset. Moreover, we employ the exact same strategy for handling missing values in the covariates. Training is performed using 153600 batches of 72 synthetically generated datasets each, which means the model was exposed to ~ IM artificial datasets during pre-training, similar order of magnitude of datasets used for pre-training TabPFN by Hollmann et al. [12]. The total training time the network lasts approximately 4 days on single A100 GPU with 80GB of GPU memory. The resulting network is then used for all our evaluations without altering its parameters.
We used the Adam optimizer [15] with initial learning rate of 0.0001 and linear- warem- scheduler for the first 10 epochs followed by cosine annealing [18] as in Hollmann et al. [12].
Architecture details. We use an EquiTabPFN network with 12 self-attention layers alternating between both type of attention introduced in Section 6 blocks SelfAtt, and 6 blocks SelfAtt, Each self-attention layer consists of a multi-head attention blocks with 4 heads, embeddings of dimension 512, and hidden layers of dimension 1024. This choice ensures a fair comparison with the models used in Hollmann et al. [12]. Müller et al. [22], since the number of parameters (25.17M) are of the same order when counting them as proposed in Kaplan al. [14].
Datasets. For the datasets with less than 10 classes, we collect the ones with less than 3000 samples and 100 features and retain the datasets that contain the same number of classes accross all folds and splits. For the datasets with more than 10 classes, we filter in addition the ones able to run inference on an 80GB A100 GPU. The datasets obtained given in Table 1 and Table 5.
Inok
name
Classss
Featunes
Samples
Baskid
nome
Classes
Features
Samplles
kr-vs-la
2
36
2556
3729
analcandata-hhlamge
2
3
80
lolvee
16
45
3748
tranoplant
2
3
104
163
3779
fri-c--100-5
ballnce-scale
3797
micaa founeer
3942
pol
bresstww
559
3001
16
mícat karhunen
10
64
1600
3913
kc2
416
18
mfcat- morphelogical
10
5
1600
3917
kcl
2
21
1687
22
micat zermike
10
47
1600
3918
pcl
2
21
887
25
cine
3
9
1177
9946
write
30
455
colic 25
26
294
9957
brodgg
843
27
294
465
552
Seve---evel---e
31
coddit-g
800
9979
cudinnacgraahyy
35
dermatology
292
9984
fertility
2
37
diabetes
8
614
10089
acute indammations
2
6
96
39
soner
60
166
10093
2
1096
80
glass
6
9
170
10101
blood trandusion service- cenier
2
4
598
45
splice
2552
14954
cylinder bands
432
47
120
2256
125020
400
tic-tac-toe
766
125921
LED dispha-domain- Jomain 7digit
heart-h
234
14575/
yeast
53
vehicle
676
145759
brenst-camce canoor
228
54
hepatitis
19
123
145836
2
4
598
59
tris
3
4
120
145547
hill valley
2
100
966
2079
eucalyguus
19
598
145984
konosphree
2
34
280
2967
renii
38
718
146074
lung-cnnee
3
24
3512
synthtic control
450
148033
roth
3
4
128
3540
makaadatt boxingg
monks prublems-
480
2543
Iish
146192
cvalltatation
3549
70
672
146210
posnperrho-pprinttduuu
3560
makaidata dmil
637
146300
MiceProtein
8
864
3561
proth
536
146817
steel stecl-plates- faste
7
27
1552
3602
3
88
146818
Australian
2
14
552
1620
fit-0 100-5
8D
146819
18
432
1647
rabe 265
146821
6
1382
3731
Visullzing-liveetts
104
146822
segment
Table 1: List of the 76 datasets with less than 10 classes used for evaluating EquiTabPFN. The
datasets are extracted from the TabZilla benchmark [21] and have number of classes no greater than
10. Here taskId is the OpenML ID of the task, Classes indicates the number of classes, Features the number of covariates and Samples the number of samples in each dataset.
14

---

taskId
name
Classes
Features
Samples
arrhythmia
12
279
360
audiology
23
69
180
soybean
19
545
3022
vowel
11
792
3481
isolet
26
617
6237
3567
collins
15
21
400
3952
chess
18
6
22444
9956
une-buundreddlaantteeetturr
100
1279
125922
lexture
4400
146032
primary-tumm -tumor
20
271
Table 2: List of datasets used for evaluating EquiTabPFN on unseen number of classes. Datasets
extracted from thellaZilla benchmark [21] and have a number of classes greater than 10 (ranging from 11 to 100). Here taskid is the OpenML. ID of the task, Classes indicates the number of classes, Features the number of covariates and Samples the number samples in each dataset.
C Additional experiment results
C.1
Ablation the different components of EquiTabPPN.
Table 3 reports relative error reduction over TabPFN in two settings: (1) TabPFN bb + Eq dec. using the TabPFN backhone (without bi-attention) and our equivariant decoder, and (2) Bi-attn bh MLP dec. using a bi-attention backbone and a standard MLP decoder from TabPFNv1. Since the encoder a simple linear embedding, is considered part of the backbone: fully connected for TabPFN-style models and Ixl convolution for biattention-bas based ones. The combination of both-bbattention and the equivariant decoder, e-g., the EquiTTabPFN model we propose-yields the largest performance gain. We also tried using TabPFNv2 architecture backbone and modified it to make it target equivariant, then trained it using the publicly available code for the prior used for training TabPFNy This led to improvements compared to TabPFNy2* (the version we retrained ourselves using publicly available training prior). However, we found that using TabPFNv1 's backbone yielded the best performance overall.
EquiTabPFN
TabPFN bb + Eq dec.
Bi-attn bb + MLP dec.
% Error reduction +1.50%
+0.94%
-0.12%
Table 3: Error reduction over TabPFN for different model configurations.
C.2 Computational cost comparisons
Run time comparison. Table 4 shows the run time comparison between EquiTabPFN, TabPFNy1 and T2bPFNv2 on both types of datasets (small or large number of classes). On small number of classes, EquiTabPFN incurs slowdown of 5x compared to TabPFNv1. This is expected as TabPFNvI was optimized handle data with less than 10 classes. On datasets with more than 10 classes, the gap narrows (only 1.3x slowdown) as TabPFNv1 requires ensembling techniques to handle the larger number of classes. A more complete picture accounts for the tradeoff between time cost and performance as in Figure 5 (left) and shows clear advantage of EquiTabbPN in terms of efficiency.
EquiTabPFN
TabPFNy1
TabPFNy2
10 Classes
0.12
0.02
0.16
10 Classes
0.40
0.30
2.80
Table 4: Time comparison (in seconds) across two types of datasets, depending on their class count.
FLOPS comparaison. While EquiTabPFN and TabPFN have similar parameter counts, EquiTabPFN more FLOPS. On A100 GPU with 2,000 samples (100 features, 10 classes), EquiTabPFN
15

---

Input data
Nearest Neighbors
Linear SVM
Random Forest
Neural Net
TabPFN
EquiTaOPFN
88
.93
.90
.95
.95
90
Figure 6: Binary classification decision boundary for 7 methods on 3 datasets. Even without ensembling, the boundary of EquiTabPFN is stable and smooth as opposed to TabPFN.
required 566 GFLOPS vs. 76 GFLOPS for TabPFN (~7.45xx more). However, with more classes (15), the gap narrows due to the ensembling needed for TabPFN to handle more than 10 classes (EquíTabPFN: 820 GFLOPS; TabPFN: 456 GFLOPS), consistent with runtime trends. Overall, those numbers remain small for modern GPU given that single H100 can easily reach 400 TFLOPS on an LLM training workflow for instance.
Memory cost, EquiTabPPN incurs an increase compared to TabPFN which was translated in the use of smaller batch size (first dimension of the activation tensors). However, the context (in terms of the number of samples that can be processed by the model on our devices) was not affected in the experiments. This likely due the moderate size of the contexts used whose ranges are within the recommended limits for TabPFN.
C.3 Binary classification decision boundary
In Figure 6, we show the decision boundary 3 binary classification datasets for multiple baselines. To illustrate the stability of the method, we do not do ensembling for TabPFN and EquiTabPPN.
C.4 Critical difference diagrams and performance metric tables
We show the critical diagram using Autorank implementation [11] in Figure for datasets with more than 10 classes and Figure 7 for datasets with less than 10 classes. Critical diagrams show the average rank of each method (lower is better) and horizontal bar show the methods statistically tied. We set the confidence level to 0.05 and use default hyper-parameters while forcing non-parametric mode ensure stability.
We also give aggregate results for datasets with more than 10 classes in Table 5 and less than 10 classes in Table 6.
26
15 14
376
BoS wine Boseiiu Gouthaor' Sor Tabee KGGoost rol. Pas
DocltioTon
H6.
UnearVade
Figure 7: Critical diagram the 76 real-world datasets with less than 10 classes from Table 1.
16

---

116
XGRomt
Peciit
Taa
Lavourfooi
Figure Critical diagram on the 10 worldw datasets with more than classes from Table
Median relative Acc.
Mean Acc.
Mean AUC
Mean FI
Mcan time (8)
model
PauPPa
27.9
77.0 4/-
1.8
95.2 0.7
75.0
2.0
0,4
79.1
Rel(ResNen)
21.2
80.9 4/- 1.7
91.1 1.1
77.9
1.9
20.5
TabPFNy1
0.8
4f-24
LinearModel
16.8
65.8 +/ 2.5
92. 0.8
63.1 2.7
0,3
RandomForest
TabPENv2
6.0
74.5 +/ 2.6
94.4 0.9
73.6 2.6
2.8
V2.7
KNN
0.0
62.9 +/- 1.9
90.3
1.0
5844 2.1
1.6
DecisionTree
.4.2
51.8 1.8
80.7 1.0
47
TabNet
12.5
51.2 4/ 2.8
77.5
1.7
48.7 2.9
43.4
STG
19.3
46. 1.9
78.7
35.2
ndI(MLP)
20.7
53.9 4/ 2.5
74.1
+1
1.8
44.7
2- 29
13.9
MLP
-22.6
51.0 2.1
73.4
1.6
41.6 2.3
15.5
VIME
-27.4
49.2
2- 2.6
70.5
+/ 1.9
41.3
3.0
39.2
Table 5: Aggregate accuracy, AUC, Fl and runtime for all methods datasets with over
than 10 classes. datasets after averaging
Results are ordered by median relative accuracy improvement w.r. over the 10 different splits. Mean accuracies, AUC and Fl score are averaged over all splits and datasets. Numbers after the symbol -/- refer to the standard eтror of the mean over all splits and datasets.
relative Acc.
model
TabPFNv2
85.7 0,5
90.3 0.5
85.5 +/- 0.6
0,2
EquiTabPEN
6.3
83.7 0.6
88.9 +/- 0.5
83.3 0.6
0.1
TabPFNy1
XGBoost
5.2
82.7 +/ 0.6
88.1
0.5
82.5 0.6
0.4
RandomForest
TabPENv2
4.3
80.8 +/ 0.6
87.1 +.6 0.6
80.1 10.1+ 0.6
0.2
Rel(ResNee)
LincarModel
1.5
77.1 +/-
82.4
4--2
0.0
DecisionTree
76.2 0.6
4f
75-1
0.0
MI.P
73.2 4/ 0.7
76.746
71.0 D&8
6.7
MdI(MPP)
73.4 41-44--00 0.8
4f
71.1
4.8
KNN
.0.0
73.8 +/. 0.6
79.5 0.6
73.0 D.7
0.0
STG
-5.0
66.5
70.6+1055
63.7
0.8
11.0
TabNet
-6.4
68.5 0.6
73.5
67.5
17.2
VIME
-8.1
63.5 63.544000
69.7 +1 0.7
60.8
0.8
10.5
Table 6: Aggregate accuracy, AUC, Fl and runtime for all methods on datasets with less than 10 classes. Results are ordered by the median relative accuracy improvement w.r.t KNN over the 10 datasets after averaging over the 10 different splits. Mean accuracies. AUC and Fl after the symbol +/- refer the standard
averaged all splits and datasets. Numbers the mean over all splits and datascts.
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providded a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
the checklist one of the factors in their evaluation.
reported because it would be too computationally
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims of the paper are that demonstrate theoretically and empirically the impact of not having equivariant models and that this leads to requiring fix number of classes during training. We show the theoretical results in Section 5 and empirical results in Section 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
18

---

Answer: [Yes]
Justification: limitations are discussed in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- lant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical have a full proof and a list of complete assumptions, some proof are detailed in the appendix due to the page limit.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details related to training hyperparameters and evaluation details are described in the text.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soneee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code used to generate evaluations and figures will be released at the camera-ready, we did not get the time to properly anonymized the code by the submission deadline.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
aasionPolicy olicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips.co. ppppuulccuuuids..... /wtoe/eobesssiiiieoeaae sesiiissssoooloo asiionoiicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
20

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: all hyperparameters are given in the main and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper provides standard error of the mean and critical diagrams that are described the text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: we detail the hardware used (A100-80GB GPU) and runtime.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/ap:::///eurips// httt::::/aarrss..cc/pbblt///ee Seppulic//thcssuuiiee ines?
Answer: [Yes]
Justification: the paper follows the ethics guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The method improves general tabular methods at a theoretical level, we do not feel there is need to discuss societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
22

---

Justification: We present and analyse general tabular method as such we do not think there are particular safeguards be put in place.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all dataset source are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www. dataaatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets is being released. If the paper is accepted, we will release our wedel-weights to facilitate reproduction together with our training code.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
23

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not perform crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We did not perform crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy (https htt///eeuripp.pe/ wc//onneeeenccss/22222 for what should or should not be described.
24


Reviews:
[{"review_id": "jxrCKi1T5J", "reviewer": "Reviewer_RkbL", "summary": "This paper introduces an equivariant model for tabular data to address the lack of target equivariance in existing methods. It does this by adding equivariance to the components of a TabPFN model, especially by adding a bi-attention mechanism in the backbone of the architecture. It demonstrates the superiority of the model over existing baselines in the case of unseen class counts at test time.", "questions": "- What is the difference in flops between TabPFN and EquiTabPFN?\n- What is the benefit of the individual equivariance adding components in the encoder, backbone, and decoder? An ablation would be interesting to see here. It is possible that one component is more important than others and can help reduce the complexity of the method.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper addresses an important challenge with an original methodology.\n- It presents a theoretical argument for their approach, and shows that target equivariant models are optimal. It shows that models not inherently designed to be equivariant like so are either less robust or use more computational power to achieve equivariance. \n- It empirically shows that the proposed model is more robust to the number of target classes, performing better than current models in the case that test classes differ from pre-training. \n\n\n\nWeaknesses:\n\nThe paper already addresses the limitation that the attention mechanism adds quadratic complexity. In the interest of fairness, they compare to baselines with similar number of model parameters. But in keeping with LLM literature, it would be a fairer comparison to compare compute flops to baselines, especially given that their method adds more complexity not necessarily reflected in the amount of parameters.", "was_revised": true, "final_justification": "Following the authors' response, I will keep my original score of 5. The paper is interesting with sufficient experiments, but the computational overhead is large enough to not justify a higher score.", "rebuttal": "Thank you for your positive feedback. To address your points, we performed the FLOPS and ablations comparison and summarize the findings below. \n\n**Flops comparaison.** While EquiTabPFN and TabPFN have similar parameter counts, EquiTabPFN uses more FLOPS. On an A100 GPU with 2,000 samples (100 features, 10 classes), EquiTabPFN required 566 GFLOPS vs. 76 GFLOPS for TabPFN (~7.45× more). However, with more classes (15), the gap narrows due to the ensembling needed for TabPFN to handle more than 10 classes (EquiTabPFN: 820 GFLOPS; TabPFN: 456 GFLOPS), consistent with runtime trends. \n\nStill, FLOPS or runtimes need to be traded for performance as in Figure 5 to which we added TabPFNv1 as requested by reviewer gnTN. There, EquiTabPFN offers the best trade-off between time and performance in most cases as summarized in the table below, in particular since TabPFN requires more ensemble to reach good performance. Finally, we note those numbers remain small for a modern GPU given that a single H100 can easily reach 400 TFLOPS on an LLM training workflow for instance.\n\n*Table of time/performance tradeoff:*\n| Pareto points   | 1        | 2        | 3        | 4        | 5        | 6        | 7        |\n|---------|----------|----------|----------|----------|----------|----------|----------|\n| Old Pareto front   | LinearModel     | EquiTabPFN(1)       |___________| EquiTabPFN(4)      | EquiTabPFN(12)      | EquiTabPFN(18)        | ___________ |\n| New Pareto front | TabPFNv1(6)         | EquiTabPFN(1)       | TabPFNv1(18)       | EquiTabPFN(4)       | EquiTabPFN(12)        | EquiTabPFN(18)    | EquiTabPFN*(18)   |\n| Time| 0.269 | 0.439 | 0.789 | 1.978 | 3.230 | 4.907 | 9.979 |\n| Performance| 51.06 | 60.14 | 61.06 | 61.73 | 61.74 | 63.68 | 66.50 |\n\n**Ablations.** Thank you for the suggestion, we will include the following ablation results on different architectures in the revised version. Specifically, we report relative error reduction over TabPFN in two settings: (1) using the TabPFN backbone (without bi-attention) with our equivariant decoder, and (2) using a bi-attention backbone with a standard MLP decoder (as in TabPFNv1). As shown in the table below, the combination of both—biattention and the equivariant decoder, eg the EquiTabPFN model we propose —yields the largest performance gain. The encoder, a simple linear embedding, is considered part of the backbone: fully connected for TabPFN-style models and 1x1 convolution for biattention-based ones.\n\n| Method                                | % Error reduction over TabPFN  |\n|----------------------------------------|-------------|\n| EquiTabPFN (Bi-attention backbone + Equivariant decoder)                          | +1.50%      |\n| TabPFN backbone + Equivariant decoder   | +0.94%      |\n| Bi-attention backbone + MLP decoder                       | -0.12%      |"}, {"review_id": "64qpbOgDUF", "reviewer": "Reviewer_gntN", "summary": "The authors propose a mechanism to make TabPFN-style models equivariant to the ordering of the class labels, something which is often achieved in practiced by randomizing over permutations of those.\nThe authors methods is to essentially tie a vector to each class dimension and perform a cross attention at the last layers to get similarity between the \"validation\" token refined through the forward pass and the class tokens. This also allows them to extend naturally beyond a fixed number of classes which is desirable.", "questions": "See above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Simple architectural improvements that can improve the models are always welcome, I have a few questions and reservations.\n\nQ1: To make sure I understand correctly, your activations vector are of shape $(B, N+M, q+1, d)$? For the instance attention you reshape to $(B*(q+1), N+M, d)$ where batch size is first, then transformer sequence length, then transformer dimension. For the self attention across labels dims/features, it would be $(B*(N+M), q+1, d)$? \n\nThe method overall makes sense, and so does the cross-attention to compute outputs.\nOn the evaluation side, I appreciate the ablations/baselines with variants of TaPFN v1/2 to disentangle the impact of different choices.\n\nQ2: I see that you have tried to be faithful to TabPFN v1 procedure and hyperparameters, so would it be fair to say that using your training code but using TabPFN v1 architecture you would get very similar experimental results to TabPFN v1? I am asking because some small choices (adding a cosine in the prior bank of activations, changing the some optimization parameters...) can affect/improve the performance of those model, and as such I am not totally certain I can use TabPFN v1's performance as a reference point for equitabpfn.\n\nQ3: I am not too sure how a 9 instances predictions problem is really all that insightful but the vizualization is nice. However as TabPFN v1 is typically averaged over class permutation, does this averaging actually lead to decent clusters? \n\nQ4: could you comment on Figure 7 and 8 in appendix? it seems that for <=10 classes equitabpfn has a performance between tabpfn v1 and tabpfn v1 ensemble?\n\nQ5: Apologies if I missed but have you tried your method on top of tabpfn v2? What are the results?\n\nQ6: *Computational efficiency and memory* Finally this is my most important remark and question. If I understand correctly, compared to tabpfn v1, you now deal with a 4D tensor, and as such your activations are much larger (q+1 times) which is bound to limit the context size you can use. Does your method exhibit a reduced context size compared to v1?\nFurthermore, it does also increase the computations if I understand correctly. Where is TabPFN v1 and TabPFN v1-ensemble in the time/performance graph? It seems to me that for the same memory and computational cost you can afford to have the vanilla more efficient architecture+some bagging for free.\n\nLastly, this is a more general open question on which I am interested to have your opinion. I think being able to extend beyond a fixed number of classes is great. I am not convinced baking equivariance in the model is necessary. In the same way that Alphafold 2 simply used data augmentation over many protein orientations instead of baking the equivariance in the model, a sufficiently large model is able to learn invariance/equivariance on its own. Even more that this, there may be some patterns in the training data that could be destroyed by equivariant/invariant transformations. In tabular data, classes are actually often ordinals in disguise (age, number of children etc...) so I tend to believe that just showing real data to a sufficiently powerful model would enable it not only to be invariant to order but also *when* to be.\n\n## Minor points\nLine 276: While indeed O(q!) would be needed for perfect invariance to ordering, in practice the prediction would concentrate with O(1/sqrt(N_ensembles)) which should converge on a given class in actually a not so high number of ensembles.", "was_revised": true, "final_justification": "The work is overall well executed but I still have reservations about the significant compute required by the method when used with the rows-as-tokens architecture as K times more tokens/compute units/memory units are now allocated to the class module.", "rebuttal": "Thank you for your encouraging and thorough feedback. To address your question on time/performance comparison,  we included TabPFN v1 in figure 5 and present the pareto front in table format (below). Below, we address your questions. \n## **Q1 – Tensor shapes** \nYes, the tensor shapes are modified as you say. We will add an explanation using shapes in the text for clarity.  \n## **Q2  – Reproducing TabPFNv1** \nYes, we reused the public reimplementation from the Mothernet paper and we confirm that we could reproduce the results of TabPFNv1 using this setup which resulted in similar performance as the publicly available pre-trained model from the original author paper. We kept the optimization hyperparameters as-is and did not change nor tune them.\n## **Q3  – Averaging TabPFNv1 in figure 2**  \nYes indeed, TabPFN-V1 converges to better clusters when increasing the number of ensembles. However, as shown in Figure 3, the equivariance error decreases only slowly and the equivariance error is still significant when using 16 ensembles for instance.\n## **Q4  –  Critical diagrams (figures 7 & 8)** \n- **Figure 7:** Ensembling mechanically helps increase TabPFNv1 performance, but induces extra cost. Please also refer to the updated time/comparison results  (answer to Q6), which compares TabPFNv1 with different ensembling to EquiTabPFN. \n- **Figure 8:** Similarly, ensembling increases performance of TabPFNv2 at an extra cost. This is also consistent with the performance vs time cost  plot in figure 5 (left).  \nOverall these results are consistent with the claim of the paper: EquiTabPFN deals more efficiently (better performance at lower time cost) with an unseen number of classes thanks to its target equivariance architecture.\n\n## **Q5  –  Using TabPFNv2 backbone** \nWe tried TabPFNv2 architecture backbone and modified it to make it target equivariant, then trained it using the publicly available code for the prior used for training TabPFNv1. This led to improvements compared TabPFNv2* (the version we retrained ourselves using publicly available training prior). However, we found that using TabPFNv1’s backbone yielded the best performance overall. We will clarify that in the text.\n\n## **Q6  –  Computational efficiency and memory**   \n- **Effect of architecture on context size:** Indeed, there is an increase in memory cost, this had an impact on the batch size (first dimension) that we had to reduce during training. However, the context (in terms of the number of samples that can be processed by the model on our devices) was not affected in the experiments. This is likely due to the moderate size of the contexts used whose ranges are within the recommended limits for TabPFN. We agree however, that scaling to larger datasets could cause memory issues. We will clarify this in the text. \n- **Time/performance comparison with TabPFNv1:** Thank you for this comment, it is an excellent point! The focus of the time/performance graph was on TabPFNv2 as the SOTA method. However, as you pointed out, TabPFNv1 is lighter and would benefit from additional ensembling. When adding TabPFNv1 and EquiTabPFN* (the version using TabPFNv2 backbone) to the time/performance graph, we observed that TabPFNv1 indeed contributes (2) points on the Pareto front (2 points), while the points contributed by EquiTabPFN to the front remain unchanged. Hence, EquiTabPFN remains efficient on a wide range of time budgets. As we cannot include the graph here, we present a table summarizing these results. We will add the graph and discussion in the revised version. \n**Time/performance tradeoff**:\n| Pareto points   | 1        | 2        | 3        | 4        | 5        | 6        | 7        |\n|---------|----------|----------|----------|----------|----------|----------|----------|\n| Old Pareto front   | LinearModel     | EquiTabPFN(1)       |___________| EquiTabPFN(4)      | EquiTabPFN(12)      | EquiTabPFN(18)        | ___________ |\n| New Pareto front | TabPFNv1(6)         | EquiTabPFN(1)       | TabPFNv1(18)       | EquiTabPFN(4)       | EquiTabPFN(12)        | EquiTabPFN(18)    | EquiTabPFN*(18)   |\n| Time| 0.269 | 0.439 | 0.789 | 1.978 | 3.230 | 4.907 | 9.979 |\n| Performance| 51.06 | 60.14 | 61.06 | 61.73 | 61.74 | 63.68 | 66.50 |\n\nThe table below shows how the pareto front is affected by adding TabPFNv1 and EquiTabPFN* (the version using TabPFNv2 backbone). The first row is the order of the methods crossing the pareto front from left to right.  Four observations:\n1. Overall, the points forming the pareto and corresponding to EquiTabPFN are unchanged.\n2. The leftmost point on the front is no longer LinearModel and is now TabPFNv1 using an ensemble of size 6 (smallest possible to handle more than 10 classes as discussed in the paper). \n3. There is an additional point obtained using TabPFNv1 with 18 ensembles, that is inserted between EquitabPFN (1 ensemble) and EquiTabPFN (4 ensembles). \n4. An additional point obtained using EquiTabPFN* with 18 ensembles is added at the rightmost side. \n\n## **General question: Equivariance vs no equivariance** \nWe thank you for this great question, we will discuss this point in the paper. Our thoughts:   \n\n- **When to (not) be equivariant:**  We agree that if data is not exchangeable and has additional structure, e.g. ordinal, one should not aim at enforcing equivariance (which would destroy it), and instead leverage the ordinal structure. However, the prior used in TabPFN is equivariant and because of that, the model still ends up learning approximate invariance during training as you pointed out (also shown in Figure 3).\n- **Controlling (non)-equivariance:** It is unclear to us to what extent current models (such as TabPFN) could learn a mechanism to control when to be equivariant given that they are trained with an equivariant prior. However, if equivariance is no longer desirable, one could think of a way for building such a control mechanism in EquiTabPFN to break equivariance while retaining the ability to handle any class number. As pointed out by reviewer Y2PB, a  “positional encoding” computed at inference time could be added to the tokens class to differentiate them from each other. When providing the same positional encoding to all tokens, the representation becomes equivariant, otherwise, it is not. For this, the prior would have to be adapted to include examples of non-equivariant and realistic artificial datasets which we believe is an interesting research direction.\n\n## **Minor points: Estimation by ensembling** \nIndeed, we will clarify that point. The variance of the estimator could present a dependence on the dimension, however it is challenging for a general function. The closer the function is to equivariance the smaller such variance would be. Hence, the worst scenarios happen when the function is far from being equivariant, in which case, the variance is the highest."}, {"review_id": "ipItWK9f6Z", "reviewer": "Reviewer_Y2PB", "summary": "EquiTabPFN is a proposed modification to TabPFN, a transformer model that has been meta-learned to perform in-context learning based tabular classification. In particular, it fixes the lack of output class order equivariance in TabPFN, with evidence from both theoretical analysis and from tabular benchmarks that this produces meaningful improvements.", "questions": "Questions:\n\n(Q1) I see that EquiTabPFN was trained with similar number of parameters as TabPFN for correctness evaluation fairness, so I am guessing that they are probably similar in computation time as well. But could you please comment more on this (and ideally include in the paper)? Did EquiTabPFN produce any slowdown in runtime? Are the computational complexities the same in terms of number of classes?\n\n(Q2) Even though EquiTabPFN is equivariant by construction, it seems to me from Figure 1 that it's not perfectly equivariant. For example the pink and blue classes aren't identical in shape, with the pink boundary curved and the blue boundary pointy. Is this due to numerical imprecision?\n\n(Q3) Are the x-axis and y-axis in Figure 1 equally scaled? It's not clear to me what are the sets of classes that should be identical in decision boundary shape to each other.\n\n(Q4) Related to (W1), is there a way to reengineer non-equivariance back into the model, the same way position encodings allow one to \"learn to bypass\" attention's permutation-invariance?\n\n(Q5) Analysis about which datasets in the benchmark improved or worsened from TabPFN to EquiTabPFN would be interesting. Were there any datasets that have some kind of class-order non-equivariance that TabPFN manages to exploit?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n(S1) [Significance, Originality] The presented problem (lack of equivariance for permutions of classes in TabPFN) and proposed solution (a classification head that is class order equivariant) are relevant, clever, and interesting. \n\n(S2) [Significance, Quality, Originality] The theoretical analysis showing the benefit of equivariance is very nice. Instead of being a box-checking exercise or an impress-the-reviewers exercise, the analysis proves something interesting and contributes to the narrative.\n\n(S3) [Significance, Quality] The results on both the initial analyses and also on the benchmarks are great. It's surprising that ensembling in TabPFN fails so badly to attain (approximate) equivariance, and that the very rough decision boundaries in Figure 1 would be solved by equivariance.\n\n(S4) [Clarity] The paper is very clearly written.\n\nWeaknesses\n\n(W1) [Significance] Code was not provided, but the authors explained that anonymization wasn't ready at time of submission. So we hope that this can be addressed by the rebuttal period.\n\n(W2) [What follows is a subjective, debatable point, and is not a reason for rejection.] Efforts to introduce invariances / equivariances have generally faced the Bitter Lesson. For example, the most recent version of AlphaFold found that dropping rotational equivariance offered better performance. Why should we not suspect that EquiTabPFN won't meet the same fate? For example, one might imagine that in many cases, modelers sort their classes (with say the first one being the \"default\" majority class). As another example, certain classification problemsPerhaps a non-equivariant model will be able to learn and exploit such phenomena while defaulting to equivariance.", "was_revised": true, "final_justification": "Reading the other reviews and the authors' rebuttal strengthened my belief that this submission warrants acceptance into NeurIPS. In particular, the authors' responses allayed my concerns about this work being incompatible with belief in The Bitter Lesson. I am not changing my original score because I already gave it a 5 (accept) with confidence 5 (full confidence). I didn't assign a 6 (strong accept) because I'm not sure about \"groundbreaking impact\", but I think this paper is indeed \"technically flawless\" with likely significant impact.", "rebuttal": "Thank you for your positive feedback. We provide below answers to your questions, hoping they address them. \n## **Code availability**  \nWe unfortunately cannot share the code at the rebuttal as submitting any link is forbidden by the current Neurips policy. We are fully committed to share all the code necessary to reproduce our experiments with the paper including the code of the method and the code to reproduce figures and tables.\n## **Equivariance vs no equivariance** \nThis is a great point that we think should be emphasized in the paper. Here are our current thoughts on the question.  \n- **What does equivariance buy for TabPFN?** It allows the model to handle an arbitrary number of classes, which is one of the key strengths of the model. This property would not be true even if the original non-equivariant was trained to achieve 0 equivariance error (for a fixed maximal class number). That being said, one could retain this property (handling arbitrary class number), while breaking equivariance in a controlled way, if it is not always desirable. For example, by adding a form of positional encoding as you suggested (see also answer to Q4). Hence, equivariance, could be a general guiding principle for building models that naturally adapts to varying sizes and shapes. Once constructed, one could choose to break equivariance in a model using general recipes (ex: positional encodings). \n- **The case of AlphaFold 3:**  It is unclear from the paper whether the performance boost is due to absence of equivariance, or the use of a better modeling approach (diffusion model instead of an equivariant structure model). The authors explained that they do not use an equivariant diffusion model to simplify the architecture, but they did not claim that using an equivariant diffusion model performed worse. They motivate the use of a diffusion model by noting that the complexity of the structure module in AF2 had only a minor effect on performance and that diffusion models avoid the ‘torsion-based parametrization’ and explicit ‘violation losses on the structure’ present in AF2. \n\n## **Q1: Computational cost**\n - **Run time comparison:** Thank you, we will add those and discuss them in more detail based on the table below showing the average times on the datasets considered in the experiments (either more than 10 classes or less than 10 classes). On datasets with a small number of classes, EquiTabPFN incurs a slowdown of 5x compared to TabPFNv1. This is expected as TabPFNv1 was optimized to handle data with less than 10 classes. On datasets with more than 10 classes, the gap narrows (only 1.3x slowdown) as TabPFNv1 requires ensembling techniques to handle the larger number of classes. A more complete picture accounts for the tradeoff between time cost and performance in figure 5, to which we added TabPFNv1 as requested by reviewer gnTN. It shows that, given a time budget,  EquiTabPFN remains the most efficient method most of the time due to the fact that TabPFNv1 typically requires more ensembles to reach good performance likely due to its non-equivariance as illustrated in Figure 3).  \n\n\n| Model        | < 10 Classes | > 10 Classes |\n|--------------|--------------|---------------|\n| EquiTabPFN   | 0.12    | 0.4           |\n| TabPFNv1     | 0.02     | 0.3           |\n| TabPFNv2     | 0.16     | 2.8           |\n\n\n\n\n- **Complexity:** The theoretical computational complexity in the number of classes is linear for both, but with a larger factor for EquiTabPFN. The linear dependence for TabPFN comes from projecting the classes into a fixed dimensional vector and then mapping back a fixed dimensional feature to the number of classes, while the dependence on the class number is constant for its backbone.  For EquiTabPFN, the backbone features scale linearly in the class number, but it enables handling arbitrary class numbers. \n## **Q2 and Q3: On figure 2 (robustness to class ordering)**  \nFrom Q2 and Q3, we understand that our current presentation of the figure suggests there could be some symmetries in the decision boundary between classes that the network should capture. We do not expect EquiTabPFN to exhibit such symmetries, as nothing in the model constrains it, e.g. blue vs pink clusters need not have the same shape as nothing would enforce the shape to capture the Voronoi diagram of the 9 training points). We do expect, however, that EquiTabPFN produces the same decision boundaries across the bottom three figures as a consequence of target equivariance (i.e. after permuting the order of the classes numbers). This is indeed the case, up to numerical precision. Thank you for raising this point, we will make sure to clarify this.\nQ3: Scaling in figure 2: We will clarify that both axes do not have the same scaling. \n## **Q4: Reengineering non-equivariance back into the model** \nThis is a great question! Also related to one from reviewer `gntN`.  Indeed, one could think about breaking equi-variance by providing some form of positional encoding. One would have to adapt the prior to generate non-equivariant but realistic datasets (currently only equivariant datasets are generated). One could then use column index or name, to define the encoding and consider ordinal classes. We believe this can be a promising future work direction. \n## **Q5: Analysis per datasets** \nMost of the datasets have nominal classes which are equivariant. For the 5/86 datasets that are ordinal, we could not see a particular difference for TabPFN performance. As noted above, we believe it is expected as TabPFN is trained with an equivariant prior so it is unlikely it would be able to exploit non-equivariant information as it was not trained to handle it."}, {"review_id": "PrKkfMDKJK", "reviewer": "Reviewer_PBVJ", "summary": "TabPFN is inherently limited to predicting no more than 10 classes, as Hollmann et al. (2025) observe:\n\n> “Thus, natively our method is limited to predicting at most 10 classes.”\n\nThe authors of the current paper argue that this ceiling arises from the model’s lack of equivariance with respect to the target dimension. To overcome it, they introduce an architecture that projects each target dimension into its own token, thereby avoiding the identified weakness. On a large suite of datasets, they empirically demonstrate the resulting performance gains.", "questions": "1. **Figure 2.** I was surprised to see TabPFN-v2 performing worse than the original TabPFN, even though your architecture appears conceptually closer to TabPFN-v2 and achieves results closer to or better than TabPFN. Could you clarify why TabPFN-v2 lags behind TabPFN?\n\n2. **Figure 5.** You refer to a “Pareto front,” but the concept is not explained in the text. Could you elaborate on what a Pareto front represents in this context and any consequences stemming from it?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. **Significance.** The authors propose a relatively straightforward modification that performs exceptionally well in practice.\n2. **Theoretical justification.** By introducing the notion of an *equivariance gap*, the authors show that methods lacking the equivariance property are theoretically bound to underperform.\n3. **Extensive empirical validation.** The paper presents a comprehensive evaluation: both in simulation and on real-world dataset, highlighting key design choices, pointing out potential flaws, and reporting performance alongside computational cost for all methods considered.\n4. **Clarity.** The presentation is clear and well-structured; multiple plots and tables effectively communicate the core ideas.\n\n## Weaknesses\n\n1. **Originality.** The final prediction layer, $\\sum_{n=1}^N y_n \\omega_n$ (without the residual MLP, where weights $\\omega_n$ are computed via attention), together with the comparison to KNN, suggests a close relationship to nearest-neighbor methods. It would be valuable to include a comparison with ModernNCA (Ye et al., 2025), which employs a similar prediction layer, even though it is not a foundation model and must be trained per dataset.\n\n---\n\nYe, Han-Jia; Huai-Hong Yin; and De-Chuan Zhan. “Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later.” *ICLR* (2025).", "was_revised": true, "final_justification": "The authors have answered all my questions during the rebuttal. I am confident their is significant and merits acceptance.", "rebuttal": "## **Comparison with ModernNCA**\nThank you for mentioning this method. One reason why we did not include it in the comparison is that the method is not an ICL method and requires being trained on each dataset. However, we agree that it is relevant given that it proposes using a non-parametric output as in our method and we will include it in the paper.\n\n## **Answers to questions** \n\n- **Figure 2:** We agree with you that the behavior on Figure 2 is surprising for TabPFN-v2. One reason for this behavior is that the model development considered mostly improving point prediction accuracy but has not considered the (non-)equivariant aspect to guide the model development.\n\n- **Figure 5:** Thank you for raising this point. We apologize for using this Pareto front terminology without properly introducing it, we will clarify in the paper. The Pareto front is defined by the methods that achieve the best tradeoffs of time cost vs performance (%AUC improvement) among the compared methods. These best tradeoff-achieving methods are the ones there is no other method that is strictly better both in terms of performance and time cost.\n\nImplication: The fact that EquiTabPFN  + various ensemble sizes, constitute the Pareto front means that: to achieve the highest performance given a fixed time budget, one should use EquiTabPFN + corresponding number of ensembles (closest dot on the Pareto), instead of the other models in the setup we studied."}]

Author rebuttal:

Meta-review: The submitted paper introduces EquiTabPFN, a modification to the TabPFN model, addressing the lack of target equivariance in existing methods classification. The authors propose a bi-attention mechanism  to achieve target equivariance. Empirical results show that EquiTabPFN outperforms existing baselines, particularly in scenarios where the number of test classes differs from training. The paper also highlights the trade-offs between computational cost and performance and relate EquiTabPFN and TabPFN in this regard.

**Strengths of the paper**
1. Significance and originality:
   - The paper addresses a limitation in TabPFN by introducing target equivariance, which is both theoretically justified and practically important.

2. Theoretical contributions:
   - The authors provide a theoretical analysis, building on the concept of an "equivariance gap" and show that models lacking equivariance are suboptimal.

3. Empirical validation:
   - Experiments on real-world and synthetic datasets demonstrate the superiority of EquiTabPFN in terms of robustness and performance.
   - The paper includes ablation studies to disentangle the contributions of different architectural components.

4. Clarity:
   - The paper is well-written, with clear explanations of the methodology, theoretical insights, and experimental results.

5. Rebuttal period improvements:
   - The authors addressed reviewer concerns by adding FLOPS comparisons, ablation studies, and detailed discussions on computational trade-offs.

**Weaknesses of the paper**
1. Computational overhead:
   - EquiTabPFN incurs a significant computational cost compared to TabPFN, particularly for datasets with fewer than 10 classes. For example, EquiTabPFN requires ~7.45× more FLOPS than TabPFN in some scenarios.
   - While the authors argue that the trade-off is justified by performance gains, this overhead may limit the method's scalability to larger datasets.

2. Limited generalization to non-equivariant data:
   - Some reviewers raised concerns about the applicability of target equivariance in cases where class order carries semantic meaning (e.g., ordinal classes).

3. Baselines:
   - The paper does not include a comparison with ModernNCA, a relevant baseline that employs a similar prediction layer. The authors did not include this baseline because it requires training on all datasets.

4. Memory constraints:
   - The increased memory requirements due to the use of 4D tensors may limit the model's applicability in certain settings.

**Discussion**
In the rebuttal period, the authors addressed most reviewer concerns:
- FLOPS and time/performance trade-offs: The authors provided detailed comparisons, showing that EquiTabPFN achieves better performance at higher computational costs. They also clarified that the method becomes similar efficient for TabPFN for datasets with more than 10 classes due to reduced ensembling requirements.
- Ablation studies: The authors demonstrated that both the bi-attention backbone and the equivariant decoder contribute to performance gains, with their combination yielding the best results. They will add the shown results and further to the revised version of the paper.
- Equivariance vs. non-equivariance: The authors discussed the potential to reintroduce non-equivariance through positional encodings.
- Scalability: The authors acknowledged the memory limitations and suggested that recent advances in efficient attention mechanisms could mitigate these issues.

**Recommendation**
The paper is technically solid and has relevant theoretical and empirical contributions. While the computational overhead is a concern, the authors provide convincing justifications for the trade-offs. The work is relevant to the field, particularly in scenarios requiring robustness to varying class counts. The reviewers unanimously recommended acceptance, and I am following their recommendation.
Decision: Accept (poster)