Revisiting Multimodal Transformers for Tabular Data with Text Fields (2024)

Abstract: Tabular data with text fields can be leveraged in applications such as financial risk assessment or medical diagnosis prediction. When employing multimodal approaches to make predictions based on these modalities, it is crucial to make the most appropriate modeling choices in terms of numerical feature encoding or fusion strategy. In this paper, we focus on multimodal classification tasks based on tabular datasets with text fields. We build on multimodal Transformers to propose the Tabular-Text Transformer (TTT), a tabular/text dual-stream Transformer network. This architecture includes a distanceto-quantile embedding scheme for numerical features and an overall attention module which concurrently considers self-attention and crossmodal attention. Further, we leverage the two well-informed modality streams to estimate whether a prediction is uncertain or not. To explain uncertainty in terms of feature values, we use a sampling-based approximation of Shapley values in a bimodal context, with two options for the value function. To show the efficacy and relevance of this approach, we compare it to six baselines and measure its ability to quantify and explain uncertainty against various methods. Our code is available at https://github.com/thomas-bonnier/ TabularTextTransformer.

heading: 1 Introduction
text: Tabular datasets with text fields can be leveraged in various critical applications such as finance or healthcare. In financial risk assessment, numerical, categorical, and text data can be used by a classification model in order to assess companies’ creditworthiness (Nguyen et al., 2021). In the medical field, clinical data and caregiver notes could be employed for diagnosis prediction. Transformers with an attention mechanism have become popular by achieving state-of-the-art performance in natural language processing (Vaswani et al., 2017; Devlin
et al., 2019). In particular, multimodal Transformers have been used with various modalities such as audio, language, and vision (Tsai et al., 2019).
Motivation. Employing Transformers with multimodal tabular/text datasets requires adequate modeling choices. First, relevant encoding schemes should be used for numerical features. The traditional mapping methods used to construct embeddings for these features, e.g. linear functions (Gorishniy et al., 2021), are more difficult to interpret in high-dimensional settings. Numerical features are ordered and can have various distributions including extreme values. This should be expressed through the choice of a relevant embedding scheme. Regarding the attention mechanism, it may be more relevant to consider the whole surroundings of an item from the input sequence rather than exclusively considering the features from the same modality or from the other modality. In other words, the attention weights may be more informative by taking into account the whole context rather than considering partial contexts (e.g. cross-modal attention only). Lastly, when a multimodal model makes predictions based on unlabeled data, it is essential for subject matter experts to understand whether the predictions are trustworthy. If the uncertainty is significant for a given prediction, the source of uncertainty should be explained in terms of feature values. For instance, a data scientist could identify regions of the feature space where the model is uncertain because it is under-specified by the data. Or a doctor might want to know why a diagnosis prediction is uncertain and might then override the prediction thanks to expert knowledge.
Perimeter and contribution. The focus is on multimodal classification tasks based on tabular datasets with text fields in English. These datasets contain numerical and categorical features (together referred to as the tabular modality here) and fields with free-form text (i.e. the text modal-
1481
ity). Numerical features have continuous scalar values while categorical features have discrete values. The latter could be a finite number of unquantifiable categories (e.g. country), boolean values (e.g. True/False flag), or ordinal values (e.g. rating from 1 to 5). Our mains contributions are fourfold:
(1) We propose the Tabular-Text Transformer (TTT), a tabular/text dual-stream Transformer network. This model exploits a specific embedding scheme for each numerical feature value, based on the distances to a set of quantiles of the feature distribution. TTT also employs an overall attention module which simultaneously considers self-attention and cross-modal attention.
(2) The two modality streams of TTT can then be leveraged in order to quantify uncertainty when the model makes predictions based on unseen data.
(3) To explain local predictive uncertainty in terms of feature values, we leverage a samplingbased approximation of Shapley values in a bimodal context, with two options for the value functions: a similarity measure and a supervised learning approach.
(4) To show the relevance of our approach, we (a) assess it on eight classification datasets against six baselines including pretrained models, (b) perform various ablation studies, and (c) measure its ability to quantify and explain uncertainty against various methods.

heading: 2 Related Work
text: Embeddings for tabular features. The quality of feature embeddings is key for the performance of models such as Transformers. The objective is to find a map from a scalar value or a category name to an embedding vector of higher dimension. To mimic word embeddings, a categorical feature can be embedded through a look-up table. Numerical features can naturally be encoded by using linear functions (Gorishniy et al., 2021) or piece-wise linear encoding (Gorishniy et al., 2022).
Multimodal machine learning. For a given task, a multimodal model leverages heterogeneous and connected modalities like text, image, and audio, as inputs. Such a model aims to learn representations of cross-modal interactions by fusing information across diverse modalities (Liang et al., 2022; Xu et al., 2023). With early fusion, cross-modal interactions happen at an early stage (Guo et al., 2020). For a Transformer with early concatenation of two modalities, this means that full pairwise at-
tention will be computed at all layers. At the other end of the spectrum, late fusion of final representations make cross-modal interactions occur at a later stage (Gu and Budhkar, 2021). While vectors can be fused with a simple addition or concatenation, Zadeh et al. (2017) propose the Tensor Fusion Network to represent the inter-modality dynamics by using a Cartesian product of embeddings. The Multimodal Transformer (Tsai et al., 2019) is a multi-stream model that takes advantage of crossmodal attention in order to attend to interactions between multimodal sequences at all layers. In this approach, self-attention is computed separately.
Uncertainty quantification and explanation. There are two types of uncertainty: aleatoric and epistemic (Antoran et al., 2021). The first type is caused by noise in the data, generating for instance class overlap. The second one originates from the model’s parameters being under-specified by the data. In that case, out-of-distribution (OOD) instances can increase uncertainty during inference. To quantify predictive uncertainty, methods such as conformal prediction (Vovk et al., 2005; Papadopoulos et al., 2002; Bonnier, 2023) can produce prediction sets based on a target coverage level. To detect model failure during inference, Corbière et al. (2019) propose a method which estimates the true class probability in image classification tasks. Speaking of explanation methods, Parcalabescu and Frank (2023) present MM-SHAP, a multimodality score based on Shapley values, which helps detect unimodal collapse. To overcome exponential time complexity, Štrumbelj and Kononenko (2010) suggest an efficient samplingbased approximation of Shapley values (Shapley, 2016) and compute the feature contributions to a classifier’s prediction in unimodal contexts. Antoran et al. (2021) propose CLUE, a method based on counterfactuals, which identifies which features are responsible for uncertainty in probabilistic models. They focus on unimodal tabular or image datasets.

heading: 3 Tabular-Text Transformer
text: We consider a dataset D = {(xi, yi)}ni=1, where each input x = (xtext, xcat, xnum) = (xtext, xtab) ∈ X contains text fields, categorical, and numerical features. The true class is y ∈ Y = {1, 2, . . . , C} for C-class classification tasks.

heading: 3.1 Novel Components

heading: 3.1.1 Distance-to-quantile embedding
text: For a random variable X with a cumulative distribution function FX , its quantile with probability α is defined by F−1X (α) = inf{x : FX(x) ≥ α}. The objective of this module is to transform a scalar and standardized feature value vnum ∈ R into an embedding vector znum ∈ R1×d, as a weighted sum of quantile embeddings: znum = ωTS. Here, d is the embedding dimension. S ∈ Rs×d denotes the stack of trainable embeddings of s empirical quantiles q0, .., qs−1 of the corresponding feature distribution {vnum,i : i ∈ I}, where the training dataset is indexed by a set I. The quantiles are for s evenly spaced cumulative probabilities (0/(s − 1), 1/(s − 1), ..., (s − 1)/(s − 1)) for integer s > 1: e.g. (0., 0.25, 0.5, 0.75, 1.) for s = 5. Quantiles are representative of the feature distribution and are used as references for the embeddings. ω ∈ Rs×1 is a normalized vector of weights. The latter are computed with the distances between the feature value vnum and the quantiles: (1) If ∃k ∈ {0, ..., s − 1} : vnum = qk, then ωk = 1 while the other elements ωj,j ̸=k = 0; (2) Otherwise, ωj = 1|vnum−qj | for j ∈ {0, ..., s− 1} . The weights are then normalized by dividing each element by the sum of weights. The resulting embedding vector of vnum will be influenced by closer quantiles. With this approach, an outlier value will be impacted by the embeddings of extreme quantiles, thus the method takes into account exceptional values. Further, high feature values will be more influenced by high level quantiles than low feature values, thus reflecting order. This method can be used to generate embeddings for each numerical feature individually.

heading: 3.1.2 Overall attention
text: α and β are two modalities with respective sequences of embeddings Zα ∈ Rtα×d and Zβ ∈ Rtβ×d. Zαβ = [Zα||Zβ] ∈ R(tα+tβ)×d is the concatenation of the previous two sequences. d denotes the embedding dimension for both modalities and t is the sequence length. For a singlehead overall attention module, we define the Query Qα = ZαW
Qα , Key Kαβ = ZαβWKαβ , and Value Vαβ = ZαβW Vαβ , with the projection matrices WQα ∈ Rd×dk , WKαβ ∈ Rd×dk , and W Vαβ ∈ Rd×dv . Motivated by the original definition of attention (Vaswani et al., 2017), we compute the overall attention for the stream of modality α,
in a bimodal setting:
OverAttα(Zα, Zαβ) = Softmax( Qα(Kαβ)
T
√ dk
)Vαβ
The output of the overall attention module is a sequence of embeddings of dimension tα × dv. Therefore, the modality α is enriched with its own features and the features from another modality β by concurrently considering self-attention and cross-modal attention. Figure 1 is a simple example of overall attention. It shows that a feature from the tabular modality (type="jackets") can attend not only to other tabular features such as the feedback count, but also to elements from the text modality such as the words flattering or cozy. Thus, the embedding vector of type="jackets" will be significantly impacted by the embeddings of those items, which may be more insightful than a limited context.

heading: 3.1.3 Uncertainty quantification
text: True labels are sometimes obtained with a certain cost or lag. Thus, it is key to measure the uncertainty when the model makes predictions based on new data. To optimize the parameters of TTT for a given classification task, we minimize the dual loss: L(l̂text(x), y) + L(l̂tab(x), y), where l̂text(x) and l̂tab(x) are the logits predicted by the text and tabular streams of the model based on input x. L is the cross-entropy loss L(l, y) = −∑Cc=1 log(Softmax(l)c)δcy where l is a vector of logits, δ is the Kronecker delta, and c ∈ Y. Our goal is to obtain optimized logits for each modality stream and employ them for uncertainty quantification. During inference, we can generate prediction sets based on the predictions from each modality stream. If each stream predicts a different label, and thus they disagree (argmaxc∈Y(l̂text,c(x)) ̸= argmaxc∈Y(l̂tab,c(x))), the decision will be uncertain. In that case, the prediction set will contain two predictions. Otherwise, the set will have a unique label. Prediction sets can be useful for subject matter experts who
need to obtain prediction regions rather than one single predicted label for any new input. Most importantly, this technique can be used to monitor uncertain predictions over time, by computing the mean prediction set size for a collection of new inputs and controlling its evolution.

heading: 3.1.4 Explaining uncertainty
text: Our objective is to understand why a given prediction is estimated as uncertain (i.e. the streams disagree) by comparison to a subset of predictions that are estimated as certain (i.e. the streams agree). In other words, we want to display the feature values that contribute to an uncertain outcome. This could be due, for instance, to an OOD input. Let p̂α(x) = Softmax(l̂α(x)) denote the softmax probability vector (i.e. vector of probability estimates) produced by the stream of modality α, with l̂α(x) the vector of logits predicted based on input x.
Constructing the value function. To compute the feature contributions to predictive uncertainty, we first need to define relevant value functions. Due to its binary nature, the stream disagreement indicator is not a tractable value function when estimating uncertainty. Therefore, we suggest two options for the value function in order to assess predictive uncertainty on a more granular scale.
When each stream predicts a different class, their respective softmax probability distribution should be more distant than when they agree. We thus evaluate the degree of uncertainty with the Jensen-Shannon Distance (JSD) (Lin, 1991), de-
fined as: JSD(p, q) = √
KL(p,m)+KL(q,m) 2 , with
here p = p̂text(x) and q = p̂tab(x) for input x. m = p+q2 is a mixture distribution, and KL is the Kullback-Leibler divergence (Kullback, 1997). JSD is a symmetrized version of Kullback-Leibler divergence, bounded by 1 if KL uses base 2 logarithm. When the streams disagree, we expect that this metric will be higher than when they agree.
For the second value function, we fit a classifier f̂ to the test dataset Dtest indexed by Itest, in order to predict the probability for a prediction to be uncertain, given the softmax probability distributions. This model is constructed as f̂ = C({((p̂text(xi), p̂tab(xi)), ui) : i ∈ Itest}), where C denotes any classification algorithm that takes in data indexed by Itest in order to output a classifier fitted on that data. Therefore, the concatenated probability estimates are used as features. The label u is equal to 0 (stream agreement) or 1
(stream disagreement). This classifier should be able to easily identify the degree of uncertainty based on the softmax probability distributions.
Approximating Shapley values for j-th feature. We build on the sampling-based method proposed by (Štrumbelj and Kononenko, 2010) to estimate the Shapley values by randomly and repeatedly selecting a subset of features instead of all possible coalitions. Based on the central limit theorem, the estimator is unbiased and its standard deviation is proportional to 1√
M , where M is the
total number of iterations. We adapt the method to the explanation of uncertainty in a bimodal context. The unlabeled test dataset is first split into certain (Dc) and uncertain (Du) predictions based on stream agreement or disagreement, respectively. Let x = (xtext, xtab) denote an input from Du, where xtext is a sequence of token values and xtab a sequence of tabular feature values. For x and for the j-th feature, we approximate the Shapley value to explain the uncertainty by performing M Monte Carlo iterations. Therefore, we compute the average contribution of a tabular feature with index j or a text feature (i.e. token) with index (i.e. position) j to the uncertainty, by iterating M times:
• We draw a random instance z = (ztext, ztab) from Dc, where ztext is here a sequence of token values and ztab a sequence of tabular feature values. Dc is used as the reference dataset to sample from as we want to understand what distinguishes x (∈ Du) from instances from Dc. For instance, if we consider the second value function, our objective is to uncover what contributes to f̂((p̂text(x), p̂tab(x))) − EDc [f̂((p̂text(X), p̂tab(X)))].
• We select a random subset of tabular feature indices stab and a random subset of token indices stext. If the j-th feature is tabular, we also specify that j /∈ stab; otherwise (j is a token index, i.e. a position in the text sequence): j /∈ stext.
• We construct two new instances. First, x+j is input x where all the tabular feature values in xtab with index in stab are replaced by the corresponding values from ztab, and all the token values in xtext with index in stext are replaced by the value of the padding token [PAD] when these token values are not in ztext (otherwise they are kept unchanged). As [PAD] does not contribute to the gradient, it is used as a mask.
Second, x−j is similar to x+j with one difference: if j is the index of a tabular feature, the value in xtab with index j is also replaced by the corresponding value from ztab; otherwise (i.e. j is a token index), the token value in xtext with index j is replaced by the value of the padding token [PAD] if this token value is not in ztext.
• We compute the marginal contribution for the given feature with each value function. For JSD, we compute the contribution for iteration m:
ϕmj (x) = JSD(p̂ + text, p̂ + tab)− JSD(p̂−text, p̂−tab).
With the the uncertainty classifier, we assess:
ψmj (x) = f̂((p̂ + text, p̂ + tab))− f̂((p̂−text, p̂−tab)),
Where p̂+α = p̂α(x+j) and p̂ − α = p̂α(x−j) for modality α.
Lastly, we compute the average to obtain the approximated Shapley value for each value function: ϕj(x) = 1M ∑M m=1 ϕ m j (x) and ψj(x) = 1 M ∑M m=1 ψ m j (x).
Disagreement between explanation methods. As the explanations provided by different methods may disagree (Krishna et al., 2022), it is essential to include various approaches in order to check the consistency between their outputs. In fact, our sampling-based algorithm makes it easy to compute the Kernel SHAP contributions (Lundberg and Lee, 2017). The "perturbation samples" x+j and x−j are converted into z′ ∈ {0, 1}T , where T is the number of tabular and text features, and z′ = 1 when the feature value from original x is present, and 0 when it is absent. For each z′, we also have the corresponding JSD and uncertainty classifier values that have been previously computed with the sampling-based method. We compute the Kernel SHAP weights (T−1)
( T|z′|)|z′|(T−|z′|) ,
where |z′| is the number of present features with 0 < |z′| < T . We fit a weighted Lasso regression r̂1(z′) = ϕks0 + ∑T j=1 ϕ ks j z ′ j where the target values are the corresponding JSD values. We fit another weighted Lasso regression where the target values are the corresponding uncertainty classifier values: r̂2(z′) = ψks0 + ∑T j=1 ψ ks j z ′ j . Lastly, the coefficients ϕksj and ψ ks j are the resulting Kernel SHAP feature contributions computed for each value function. They can be compared to the ϕj and ψj previously computed with the sampling-based algorithm.

heading: 3.2 TTT Overall Architecture
text: To show how its components are connected, TTT’s architecture is displayed in Figure 2.
Embeddings. The text inputs are transformed into word embeddings (dimension Rttext×d) while categorical features are turned into category embeddings (Rtcat×d). ttext and tcat denote the number of text tokens and categorical features, respectively. These transformations are based on the usual mechanism of look-up tables. The embeddings (Rtnum×d) for the tnum numerical features are constructed by using the distance-to-quantile technique previously described. Categorical and numerical embeddings are concatenated (Rttab×d,with ttab = tcat + tnum). A classification token [CLS] (Devlin et al., 2019) is then added to the beginning of each text and tabular embedding sequence. All these embeddings are learned by the model through the training process. Lastly, fixed positional encoding with sine and cosine waves is added to the text embeddings as suggested by Vaswani et al. (2017). The inputs to the next step are denoted
Z [0] text ∈ R(ttext+1)×d and Z [0] tab ∈ R(ttab+1)×d for the text and tabular streams.
Transformers with overall attention. Each Transformer contains L layers of overall attention blocks. If we consider for instance the tabular stream, each attention block with residual connection starts with the following operation for i = 1, ..., L layers:
OverAtt[i],multtab (LN(Z [i−1] tab ),LN([Z [i−1] tab ||Z [0] text])) + Z [i−1] tab
LN stands for layer normalization (Ba et al., 2016). OverAtt[i],multtab denotes the multi-head version (Vaswani et al., 2017) of OverAtttab at layer i. Z [0]text are used as low-level features from the text modality. After applying layer normalization, the outputs are then fed to a feed-forward neural network. Lastly, we employ a residual connection.
Classification layers. The text and tabular embeddings of the [CLS] tokens (Rd) from each output Z [L]. ∈ R(t.+1)×d are projected through fullyconnected layers to produce the logits for each stream. These are leveraged to quantify uncertainty as previously described. The final prediction is based on the argmax of the average of the text and tabular logits. The performance measurement is based on this final output. For the sake of clarity, the prediction set or the stream predictions denote the output of the uncertainty analysis while the (final) prediction denotes the single label predicted by the model in order to measure its accuracy.

heading: 4 Experiments
text: We empirically test the relevance of our approach on eight classification datasets. In the appendix, we provide further details on the experimental settings and results (e.g. details on datasets, data preprocessing, hyper-parameters, baselines).

heading: 4.1 Settings
text: Datasets. We use various public datasets for multi-class classification tasks, with a number of classes ranging from 5 to 100: airbnb, cloth, petfinder, salary, and wine with the 10/100 most frequent classes (referred to as wine10 and wine100, respectively). We also test our method on two public binary classification datasets: jigsaw and kick. These datasets have been suggested by (Shi et al., 2021) and (Gu and Budhkar, 2021).
Model selection. Each dataset is split into training-validation-test disjoint subsets. The validation subset is used for hyper-parameter tuning and early stopping. For all non pretrained models, we select the best trial via a Bayesian optimization algorithm (Tree-structured Parzen Estimator) by using Optuna library (Akiba et al., 2019). We finetune the pretrained models based on DistilBERTbase-uncased (Sanh et al., 2019). The test dataset (20%) is used for the final evaluation (accuracy, uncertainty). Each use case is run over five different random dataset splits.
Evaluation 1 (low-dimensional embeddings). To reduce computation costs, we first evaluate various multimodal architectures (including TTT) with low-dimensional embeddings (maximum of 64): (1) EarlyConcat: A self-attention Transformer with early concatenation of modalities; (2) LateFuse: A Transformer with late fusion of modalities (Gu and Budhkar, 2021); (3) MulT: The Multimodal Transformer (Tsai et al., 2019) with cross-modal attention; (4) TFN: Tensor Fusion Network (Zadeh et al., 2017) with LSTM. For numerical features, TTT leverages the distance-to-quantile embedding scheme with an arbitrary value of 6 quantiles (0., 0.2, 0.4, 0.6, 0.8, 1.). The embeddings of the baselines are constructed with linear functions (EarlyConcat, MulT) or features are kept as standardized values (LateFuse, TFN).
Evaluation 2 (pretrained models). Secondly, we test TTT-SRP: TTT with sparse random projections (Li et al., 2006) of input embeddings extracted from a pretrained model. TTT-SRP has an intermediate size for the embeddings, with a 240 dimensional space at most. It is initialized with pretrained DistilBERT-base-uncased input embeddings after reducing their dimensions through sparse random projections. We measure it against two pretrained baselines: (1) AllTextBERT: The tabular features, converted to strings, and the text fields are concatenated and input into DistilBERTbase-uncased as text; (2) LateFuseBERT: A dualstream model with late fusion of [CLS] tokens generated by DistilBERT-base-uncased and a tabular Transformer.
Ablation studies. To evidence the contribution of each component, we perform three ablation studies for TTT: (1) Ablation 1: The numerical features are encoded with linear functions instead of the distance-to-quantile embedding scheme; (2) Abla-
tion 2: We use self-attention instead of overall attention; (3) Ablation 3: We minimize the loss based on the average of logits: L((l̂text(x) + l̂tab(x))/2, y), instead of the dual loss. Moreover, to show the effect of the TTT-SRP’s initialization scheme, we measure its performance against TTT-PCA (same as TTT-SRP but with principal component analysis instead of sparse random projections) and a standard approach based on the Kaiming uniform weight initialization (He et al., 2015).
Quantifying and explaining the uncertainty. Our method, used to quantify uncertainty (stream disagreement), is compared to the LAC conformal prediction approach (Sadinle et al., 2019) and the Monte Carlo dropout (MCD) with 50 simulations (Gal and Ghahramani, 2016). For LAC, we compute the mean prediction set size. For MCD, the uncertainty is assessed with the entropy. Lastly, we explain the uncertainty of TTT model’s predictions from wine100 test dataset. The dataset task is to predict grape varieties based on numerical features (e.g. price), categorical variables (e.g. country), and wine tasting descriptions. The uncertainty classifier is a random forest algorithm (Breiman, 2001) with 100 trees. In the sampling-based algorithm, we stop the sampling iterations when the maximum absolute change in the Shapley values computed with the stream disagreement (binary) signal is lower than 0.01. This method is used to speed up the computation as additional iterations would modify the Shapley values only very marginally. We use the binary signal for this purpose as the changes are expected to be more sudden than for the other value functions. Thus, the convergence is
more certain when it occurs with the binary signal.

heading: 4.2 Results
text: Performance. The results from Table 1 show that TTT outperforms the other multimodal approaches on most datasets. Further, the results from Table 2 demonstrate that the lightweight TTT-SRP (8.2M parameters) outperforms LateFuseBERT (81.3M). TTT-SRP also matches AllTextBERT (66.4M) in terms of number of top results. Lastly, it is worth noting that TTT achieves better results than larger architectures from Table 2 on small datasets (i.e. airbnb, petfinder, salary).
Uncertainty quantification. Tables 1 and 3 show that the mean prediction set size is negatively correlated with TTT’s accuracy. The Pearson correlation between the mean prediction set size and model’s performance (across the 40 experiments: 8 datasets × 5 seeds) is slightly more negative (-
0.953) with our approach than with the other baselines: conformal prediction (-0.947) and Monte Carlo dropout (-0.932). The mean prediction set size computed with our technique is most relevant to quantify uncertainty on unlabeled data. Thus, when the streams disagree in their outputs, the final prediction is less reliable, and this leads to lower accuracy (first column of Table 3).
Ablation studies. Table 4 shows that TTT has a solid performance as it ranks first or second across the various datasets. The effect of using overall attention instead of only self-attention is quite obvious (Ablation 2). The dual loss used in TTT is also relevant based on the weaker results of Ablation 3.
The effect of the distance-to-quantile encoding is less obvious for some of the datasets but may remain of relevance for datasets with many numerical features, such as airbnb. Lastly, Table 5 displays the results related to the initialization scheme of input embeddings: TTT-SRP achieves the best accuracy on most datasets.
Explaining the uncertainty. This method can be used on unlabeled datasets. Here, the true labels are only leveraged to audit the method. Figure 3 (left) displays the distributions of the JSD metric and uncertainty classifier’s predicted probability over Dc and Du, respectively. The uncertainty classifier performs better at discriminating between
certain and uncertain predictions, and may be more reliable. Figure 3 (middle) displays the results for an uncertain and incorrect final prediction. Each stream predicts a different label. The instance is related to a country value ("Brazil") which is quite rare in the dataset (i.e. data scarcity), generating an uncertain decision. Further, the term abrasive also contributes to the uncertainty. This word appears in text fields related to several grape varieties from the same region (e.g. Merlot or Tannat varieties). This is a sign of potential class overlap. Both value functions help detect these problems. To confirm the consistency of explanations, we compute the Pearson correlations between the Shapley values obtained with our method and with Kernel SHAP, for both value functions: 0.93 (JSD) and 0.86 (uncertainty classifier). Lastly, we perform a synthetic sensitivity test with an input where the price has been artificially halved so that the decision becomes uncertain (stream disagreement), but remains correct. Figure 3 (right) evidences the origin of uncertainty and the potential inconsistency between the price and year values. With the uncertainty classifier, these positive contributions are largely offset by the negative contributions of specific words (e.g. vegetal). This is why, unlike the tabular stream, the text stream still predicts a correct label. In that case, using the uncertainty classifier produces more reliable outputs as confirmed by the Pearson correlations between the feature contributions generated by our method and by Kernel SHAP: 0.39 (JSD) and 0.88 (uncertainty classifier). This explanation method could be employed on more critical applications such as medical diagnosis prediction by combining a predictive algorithm with human-in-the-loop. For instance, a prediction could be uncertain due to specific words from the caregiver notes. This would help a medical expert understand why the prediction is not certain and make a decision for a more likely diagnosis. Appendix K displays additional examples from wine, cloth, and kick datasets.

heading: 5 Conclusion
text: We presented the Tabular-Text Transformer, a multimodal approach which shows promising results on classification tasks. We demonstrated how uncertainty can be quantified and explained. Future work will focus on pretraining strategies for TTT.

heading: 6 Limitations
text: Datasets for critical tasks. The datasets used in this pilot study are not related to critical tasks. Thus, it would be useful that the scientific community shares tabular/text datasets for medical or financial applications in order to test the efficacy of the approach presented here.
Overall attention with unbalanced sequences. The effect of having sequences with different lengths (e.g. long text sequence and few tabular features) has not been assessed. Therefore, the impact of unbalanced lengths for text and tabular sequences remains to be studied.
Uncertainty quantification and coverage. The size of the prediction sets generated by TTT’s streams is bounded by two. Unlike conformal prediction, it is not possible to specify the desired coverage level (i.e. the proportion of true label values that lie in the prediction sets). The actual coverage depends on the use case and can only be assessed on a validation dataset. Further, the conditional coverage could vary across different dimensions. For instance, it could be different across different population subgroups. Therefore, it is essential to monitor conditional coverage across these dimensions when the true labels are obtained.

heading: 7 Ethical Considerations
text: We presented a new Transformer-based architecture for classification tasks based on tabular datasets with text fields. Our method does not aim to predict or exploit any personal or sensitive information. We do not expect any significant risks related to this multimodal approach.
Given it is based on pretrained models and lowdimensional embeddings, the environmental impact of our study is limited. We reported the computational cost for each model in Appendix I. A largescale test could benefit from employing specific cost-related algorithms for tuning hyper-parameters (Wu et al., 2021).
With regard to transparency and compliance issues, we conducted the experiments on publicly available datasets and implemented the models by exploiting well-recognized pretrained models and Python libraries, e.g. Pytorch (Paszke et al., 2019). The experimental results are obviously specific to these datasets. We ran some of our experiments on a sample of the jigsaw1 public dataset which is
1https://www.kaggle.com/c/
used for detecting toxic comments. We reported the performance results, but did not display any harmful content from this dataset. Lastly, it is important to monitor the performance of TTT over time. In particular, if the use case is related to individuals, practitioners should ensure that the model does not achieve different levels of uncertainty or performance across population subgroups. Artefacts such as the ESG model card can help report these risks and limitations (Bonnier and Bosch, 2023).

heading: A Appendix: Table of Contents
text: B Related Work C Datasets and Sampling D Pre-processing E Hyper-parameter Tuning, Weight Initialization, and Training Settings F Variability in Results G Additional Baseline H Uncertainty Quantification I Implementation Information J Models’ Architectures K Explaining the Uncertainty

heading: B Related Work
text: Transformers for tabular data. With regard to tabular data, most existing works on Transformers have focused on learning the structure and content of tables with pretraining strategies. Those models are usually employed for tasks such as question answering (Yin et al., 2020; Eisenschlos et al., 2021), table based fact-checking (Yang et al., 2022), or table content population (Iida et al., 2021). Such architectures leverage specific positional embeddings and row-wise/column-wise attention. Our work, however, concentrates on employing labeled tabular/text data for multimodal classification tasks. We note that few works address this objective (Badaro et al., 2023; Gu and Budhkar, 2021).

heading: C Datasets and Sampling
text: All the datasets are publicly available with one of these licenses: "CC0: Public Domain", "Competition Data", or "CC BY-NC-SA 4.0". These datasets can be accessed and used for the purpose of academic research. The text fields are in English.
In Table 6, we give more details on the datasets:
• airbnb2: the task is to predict the price range of Airbnb listings. The text fields are listing descriptions.
• cloth3: the goal is to classify the sentiment (represented as a class) of user reviews regarding clothing items. The text fields are customer reviews.
2https://www.kaggle.com/datasets/tylerx/ melbourne-airbnb-open-data
3https://www.kaggle.com/datasets/nicapotato/ womens-ecommerce-clothing-reviews
• jigsaw4: the objective is to detect toxic comments. As advised by the creators of this use case, examples with target value greater or equal to 0.5 will be considered to be in the positive class (toxic). As class imbalance can make the accuracy metric irrelevant, we oversample instances with the toxic label to reach 20% share, while keeping only 100,000 instances from the initial 1.8 million examples in order to reduce the computational cost.
• kick5: the task is to predict whether a proposed project will meet its funding goal. The text fields are project descriptions.
• petfinder6: the goal is to predict the speed range at which a pet is adopted. The text fields are profile write-ups for the pets.
• salary7: the task is to predict the salary range based on data scientist job postings. The text fields are job descriptions.
• wine8: the goal is to predict the variety of wines. The text fields are wine tasting descriptions.
For some of the use cases, we employ the original training dataset as the test dataset does not include the true labels (competition data). In that case, we consider the training dataset as the modeling data which is then randomly split into trainingvalidation-test subsets. For all datasets, the splits
4https://www.kaggle.com/c/ jigsaw-unintended-bias-in-toxicity-classification
5https://www.kaggle.com/datasets/codename007/ funding-successful-projects
6https://www.kaggle.com/competitions/ petfinder-adoption-prediction/data
7https://machinehack.com/hackathons/predict_ the_data_scientists_salary_in_india_hackathon/ overview
8https://www.kaggle.com/datasets/zynicide/ wine-reviews
are constructed as follows: (1) The initial dataset is randomly split into two disjoint temporary (80% share) and test (20% share) subsets, respectively; (2) The temporary dataset is randomly split into two disjoint training (80% share) and validation (20% share) subsets, respectively.

heading: D Pre-processing
text: Feature engineering. When the dataset contains several text fields, these are concatenated in order to obtain a single field. Rows with missing values are dropped. Missing values for categorical variables could be handled by specific embeddings, but this remains to be studied. The list of final features for each dataset is available in the settings.py Python file. We mention here additional features that were created from the raw dataset (detailed in dataset.py):
• airbnb: for this dataset only, we discretize the target variable by employing quantile binning (ten intervals with equal share of data). We also create two new features host_since_year and last_review_year by extracting the year from host_since and last_review respectively.
• kick: we compute the duration to launch (in days) with deadline and launched_at. We also log-transform goal.
• wine: we extract the year from title.
Tabular feature encoding. For numerical features, TTT and TTT-SRP use the distance-toquantile embedding scheme with an arbitrary value of 6 quantiles (0., 0.2, 0.4, 0.6, 0.8, 1.). The embeddings of the baselines are constructed with linear functions (EarlyConcat, MulT, LateFuseBERT) or features are kept as standardized values (LateFuse, TFN). A linear function applies the following transformation to a scalar feature value
x ∈ R: x.Wnum + b where Wnum ∈ Rd and the bias b ∈ Rd. For categorical features, we use the one-hot-encoding technique (LateFuse, TFN) or encode them as category embeddings (EarlyConcat, MulT, TTT, TTT-SRP, LateFuseBERT). In that latter case, the corresponding embedding is computed as eTWcat where e ∈ Rnc×1 is a one-hotvector for the associated categorical feature, nc denote the number of categories for this feature, and Wcat ∈ Rnc×d. Lastly, in AllTextBERT, the tabular features, converted to strings, and the text fields are concatenated and input into DistilBERTbase-uncased.
Text pre-processing for small architectures (EarlyConcat, LateFuse, MulT, TFN, TTT). We keep only words and whitespaces (e.g. we remove numbers and punctuation). We use the basic_english tokenizer from PyTorch which performs lowercasing and basic text normalization for English words. This tokenizer returns a list of tokens after splitting on whitespace. To reduce the dictionary’s size, we remove words that appear only once. For the text sequence maximum length (see Table 6), the value is set as the 0.9 quantile of the text field lengths’ distribution, based on words. We also use the [UNK] token for unknown words, truncation, and padding to the specific maximum length.
Text pre-processing for large architectures (AllTextBERT, LateFuseBERT, TTT-SRP). For these models, we perform the following text preprocessing: we keep words, numbers, and whitespaces. We then use the DistilBERT-base-uncased tokenizer based on WordPiece. For the text sequence length, the value is the 0.9 quantile of the text field lengths’ distribution. We use truncation and padding to the fixed maximum length.
Key padding mask. We use key padding masks in order to specify which text tokens should be ignored (i.e. "padding") for the purpose of attention.

heading: E Hyper-parameter Tuning, Weight Initialization, and Training Settings
text: Small architectures. For all the small architectures (EarlyConcat, LateFuse, MulT, TFN, TTT), the hyper-parameters are tuned on the validation dataset. The objective is to select the best trial, i.e. the trial with the highest accuracy on the validation subset. We run a Bayesian Optimization algorithm (Tree-structured Parzen Estimator) by using Optuna library (Akiba et al., 2019) with a limited time budget for each model/run: 900s for each case where the training dataset contains more than 20,000 samples; 600s otherwise. Table 8 displays the hyper-parameter space used during tuning. We also use the pruning option which enables to early-stop unpromising trials. For faster epochs, we use only 60% of the training samples if the training dataset contains more than 20,000 samples.
The selected model is then trained (on the training dataset) by using early stopping with a patience of 4 for the accuracy on the validation set. An exponential learning rate scheduler with gamma of 0.9 (multiplicative factor of learning rate decay) is also employed. We keep the best model in terms of epochs, i.e. with the highest accuracy on validation data.
Additional information on training settings is provided here: (1) We use the Xavier uniform weight initialization (Glorot and Bengio, 2010) for the parameters in the attention modules and the Kaiming uniform weight initialization (He et al., 2015) for the other weights; (2) Loss: crossentropy; (3) Loss optimization: optimizer (Adam) with weight decay (1e − 5); (4) Maximum number of epochs: 100; (5) Dropout rate (for attention, residual, feed-forward network, fully-connected networks, embeddings): 0.1.
TTT-SRP. For TTT-SRP, the hyper-parameters are tuned on the validation dataset. The objective is to select the best trial, i.e. the trial with the highest accuracy on the validation subset. We run a Bayesian Optimization algorithm (Tree-structured
Parzen Estimator) by using Optuna library with a limited time budget for each model/run: 900s for each case where the training dataset contains more than 20,000 samples; 600s otherwise. Table 9 displays the hyper-parameter space used during tuning. We also use the pruning option which enables to early-stop unpromising trials. For faster epochs, we use only 60% of the training samples if the training dataset contains more than 20,000 samples.
The selected model is then trained (on the training dataset) by using early stopping with a patience of 1 for the accuracy on the validation set. An exponential learning rate scheduler with gamma of 0.9 (multiplicative factor of learning rate decay) is also employed. We keep the best model in terms of epochs, i.e. with the highest accuracy on validation data.
The input embeddings are initialized with pretrained DistilBERT-base-uncased input embeddings after reducing their dimensions through sparse random projections: E = 1√
d AR, where
E ∈ Rv×d denotes the reduced input embeddings from TTT-SRP (v = 30522 is the vocabulary size and d is the embedding dimension selected through hyper-parameter tuning), A ∈ Rv×DDBERT is DistilBERT input embeddings (DDBERT = 768 is the embedding dimension), and R ∈ RDDBERT×d is the projection matrix with elements rij . As per (Li et al., 2006), we have rij = √ s with probability
1 2s , rij = 0 with probability 1− 1s , and rij = −
√ s
with probability 12s . As advised by the authors, we take s = √ DDBERT . Further, we use the Xavier uniform weight initialization for the parameters in the attention modules and the Kaiming uniform weight initialization for the remaining weights.
Lastly, the other elements are similar to small architectures: (1) Loss: cross-entropy; (2) Loss optimization: optimizer (Adam) with weight decay
(1e− 5); (3) Maximum number of epochs: 100; (4) Dropout rate (for attention, residual, feed-forward network, fully-connected networks, embeddings): 0.1.
AllTextBERT, LateFuseBERT. In order to finetune these pretrained models, we follow the guidelines from Devlin et al. (2019); Sanh et al. (2019) with a batch size of 32 and a learning rate of 5e−5. The model is trained (i.e. fine-tuned on the training dataset) by using early stopping with a patience of 1 for the accuracy on the validation set. An exponential learning rate scheduler with gamma of 0.9 (multiplicative factor of learning rate decay) is also employed. We keep the best model in terms of epochs, i.e. with the highest accuracy on validation data.
For the tabular Transformer component of LateFuseBERT, we use the Xavier uniform weight initialization for the parameters in the attention modules and the Kaiming uniform weight initialization for the other weights.
Lastly, we also have: (1) Loss: cross-entropy; (2) Loss optimization: optimizer (AdamW) with weight decay (1e − 5); (3) Maximum number of
epochs: 100; (4) Dropout rate: 0.1.
F Variability in Results
Table 7 displays the variability in performance.

heading: G Additional Baseline
text: We experiment with a different type of baseline: a random forest (100 estimators, maximum depth of 10) based on term frequency-inverse document frequency (TF-IDF) with 1000 text features, one-hotencoding for the categorical features, and the scaled numerical features. This model achieves the same mean accuracy as TTT on 2 small datasets with many tabular features (airbnb and petfinder). However, its accuracy is weaker on the other datasets: airbnb 0.385 | cloth 0.555 | jigsaw 0.806 | kick 0.685 | pet 0.391 | salary 0.372 | wine10 0.67 | wine100 0.468.

heading: H Uncertainty Quantification
text: LAC baseline. The conformity score of the LAC method (Least Ambiguous Set-Valued Classifiers) from (Sadinle et al., 2019) is one minus the probability of the true label. Thus, we compute it for each data point of the validation dataset (size n). To make the methods comparable, the target coverage level (1− α) is set to the one obtained on the validation set with the stream disagreement (SD) method. We then compute the (1 − α)(n + 1)/n quantile of the conformity scores’s distribution. For each data point of the test dataset, the prediction set is then constructed with the labels where one minus the predicted probability is below the previous quantile.
MCD baseline. For each data point of the test dataset, we obtain M = 50 softmax distributions with the Monte Carlo dropout method. We then estimate the uncertainty at each datapoint by computing the entropy u as in (Antoran et al., 2021): u = −∑c∈Y( 1M ∑M i=1 Pi,c) log2( 1 M ∑M i=1 Pi,c), where Pi,c = p̂(c|θi, x) is the predicted probability from TTT for simulation i, model parameters θi, and class c.
Coverage. The prediction regions produced by TTT can be more informative than a single predicted label. For example, TTT’s accuracy is 47% on salary dataset, but Table 10 shows that the prediction sets achieve 57% coverage and 35% of the sets contain two predicted labels. From a frequentist standpoint, if we consider the whole test dataset,
57% of the time, the prediction sets of size 1 or 2 would include the true label. Lastly, Table 10 also shows the coverage achieved by TTT’s prediction sets when the two streams agree or disagree. These coverage values are not equal but remain quite close.
Example. Lastly, Figure 4 displays two examples where the streams disagree and the model fails to predict the correct label.
I Implementation Information
Hardware and computational cost. We run the experiments with a Tesla T4 GPU. Table 11 summarizes the average computational cost for each model. It is worth noting that the computation time for TTT-SRP includes hyper-parameter tuning (between 600 and 900s), which explains why the total computation time is as high as LateFuseBERT’s one.
Python libraries. The implementation is based on Python 3.10 and the following packages: torch 2.1.0+cu118 (Paszke et al., 2019), optuna 3.4.0 (Akiba et al., 2019), transformers 4.35.2 (Wolf et al., 2020), pandas 1.5.3 (Wes McKinney, 2010), numpy 1.23.5 (Harris et al., 2020), sklearn 1.2.2 (Pedregosa et al., 2011), scipy 1.11.4 (Virtanen et al., 2020). These libraries are publicly available with "BSD", "MIT", or "Apache Software" licenses.
For AllTextBERT and LateFuseBERT, we use the ’distilbert-base-uncased’ tokenizer and model from the transformers library. All the baselines are implemented with Pytorch and adapted to tabulartext inputs. The code has been written based on the content of the references cited in Section 4.1.

heading: J Models’ Architectures
text: TTT and TTT-SRP. We detail here the feed-forward module and final fullyconnected layers from Figure 2. The feedforward module can be described as follows: FF(x) = Linear(Dropout(ReLU(Linear(x)))) where the output has an embedding dimension of d.
The final fully-connected layers can be described as follows: FC(x) = Linear(Dropout(ReLU(Linear(x)))) where the output has a dimension of C (number of classes).
TTT-SRP’s architecture is quite similar. As previously detailed, the only differences are related to the weight initialization scheme of the input embeddings and the size of the network.
Baseline 1: EarlyConcat. The model is a onestream Transformer with self-attention applied on the concatenated tabular-text embedding sequence (including [CLS] token). Positional embeddings are used too. The final prediction is performed after projecting the embeddings of the [CLS] token through fully-connected layers.
Baseline 2: LateFuse. Only the text stream goes though a Transformer model with self-attention. The [CLS] output of this Transformer is then concatenated with the standardized numerical features and one-hot-encoded categorical variables. The final prediction is performed after projecting this vector through fully-connected layers.
Baseline 3: MulT. This model has two streams: tabular and text. Each stream contains a Transformer with cross-modal attention followed by a Transformer with self-attention. Positional embeddings are used in this architecture too. We then concatenate the [CLS] text and tabular embeddings from each stream in order to project the resulting vector through fully-connected layers.
Baseline 4: TFN. This is the only model which is not based on a Transformer architecture. This model starts with two streams: an LSTM for the text sequence and a neural network for the tabular features. We compute the Cartesian product from the outputs of these two networks (after adding 1 in each embedding vector) in order to take into account 1-D and 2-D effects. The resulting matrix is flattened and projected through fully connected layers.
Baseline 5: AllTextBERT. The tabular features, converted to strings, and the text fields are concatenated and input into DistilBERT. The final prediction is performed after projecting the embeddings of the [CLS] token through fully-connected layers.
Baseline 6: LateFuseBERT. This model has two streams: tabular and text. Each stream contains a Transformer architecture: DistilBERT is used for the text stream and a Transformer with 3 layers (self-attention with 8 heads) is employed for the tabular stream. We then concatenate the [CLS] text and tabular embeddings from each stream in order to project the resulting vector through fullyconnected layers.

heading: K Explaining the Uncertainty
text: Contribution of special token [CLS]. When we compute the token contributions, the classification token [CLS] is not considered. This token is only used for the classification task.
Lasso parameters in Kernel SHAP. When computing the Kernel SHAP contributions, we fit a weighted Lasso regression with the following settings: L1 regularization penalty: 1e-3, maximum number of iterations: 1000, tolerance for the optimization: 0.0001.
Example 1: OOD instance from wine. The original wine dataset contains more than 100 classes. Thus, we select an instance from a minority class which is not included in wine100. The uncertain OOD instance has the following characteristics:
• Tabular features: country="Italy", year="2005", price=75.0, points=91.
• Text: "This floral wine is bursting with notes of jasmine and honeysuckle backed by sweet fragrances of candied fruit and caramel. The wine is sweet and thick in the mouth but the welldosed acidity keeps it from being cloying or too fat. It would pair well with slightly sweet foods such as honey glazed ham or lobster."
• Label and predictions: y = 611, ŷtext = 21, ŷtab = 43, ŷ = 97.
• JSD(p, q) = 0.82, f̂((p, q)) = 0.84.
The feature contributions to uncertainty are displayed in Figure 5. There are several positive (red) contributions, proving that the input may be unusual. The main source of uncertainty is related to the country along with specific words. This method might be used for novelty detection as well. In that case, using the uncertainty classifier produces more reliable outputs as confirmed by the Pearson correlations between the feature contributions generated by our method and by Kernel SHAP: 0.68 (JSD) and 0.89 (uncertainty classifier).
Example 2: uncertain instance from cloth. In Figure 6 (first 2 charts), the distributions of the JSD metric and uncertainty classifier’s predicted probability evidence that the uncertainty classifier can better distinguish between certain and uncertain instances.
The uncertain instance has the following characteristics:
• Tabular features: Division Name="General", Department Name="Bottoms", Class Name="Shorts", Age=43, Positive Feedback Count=0.
• Text : "Romper rules Overall great romper! i have a long torso and thicker legs so i sized up to a small and i am very happy i did. the shorts are a bit on the shorter side and go upon the sides so another reason to consider sizing up. the quality is great and super soft chambray with a trendy tie up."
• Label and predictions: y = 3, ŷtext = 4, ŷtab = 3, ŷ = 4.
• JSD(p, q) = 0.81, f̂((p, q)) = 1.
The feature contributions to uncertainty are displayed in the last two rows of Figure 6. The model’s decision is incorrect. This is the same prediction as the output of the text stream, whereas the tabular stream predicts the correct label. The uncertainty mainly originates from words (e.g. overall, thicker). In that case, the explanations generated with each value function are quite consistent. Further, the Pearson correlations between the feature
contributions generated by our method and by Kernel SHAP is high for each value function: 0.90 (JSD) and 0.93 (uncertainty classifier).
Example 3: uncertain instance from kick. In Figure 7 (first 2 charts), the distributions of the JSD metric and the uncertainty classifier’s predicted probability evidence that the uncertainty classifier can better discriminate between certain and uncertain instances.
The uncertain instance has the following characteristics:
• Tabular features: country="US", currency="USD", disable communication="False", log goal=4, backers count=447.
• Text : "The Life & Times of a Remarkable Misfit 18 months ago, I canceled a book deal with a major publisher because I realized I had sold out to be chosen. Today I choose me."
• Label and predictions: y = 1, ŷtext = 0, ŷtab = 1, ŷ = 0.
• JSD(p, q) = 0.27, f̂((p, q)) = 1.
The task is to predict whether a proposed project will meet its funding goal. The feature contributions to uncertainty are displayed in the last two rows of Figure 7. If we consider the uncertainty classifier value function, the inconsistency between the high value of backers count and the word canceled seems to make the model’s decision uncertain. The Pearson correlations between the feature contributions generated by our method and by Kernel SHAP is high for each value function: 0.96 (JSD) and 0.99 (uncertainty classifier).

heading: Future Work
text: Future work will focus on pretraining strategies for TTT