Investigating the impact of missing value handling on Boosted trees and Deep learning for Tabular data: A Claim Reserving case study
Authors: Alexander Larionov
Niall M. Adams
Kevin N. Webster
Venue: Accepted by TMLR ()

Abstract: While deep learning (DL) performance is exceptional for many applications, there is no consensus on whether DL or gradient boosted decision trees (GBDTs) are superior for tabular data. We compare TabNet (a DL model for tabular data), two simple neural networks inspired by ResNet (a DL model) and Catboost (a GBDT model) on a large UK insurer dataset for the task of claim reserving. This dataset is of particular interest for its large amount of informative missing values which are not missing completely at random, highlighting the impact of missing value handling on accuracy. Under certain missing value schemes a carefully optimised simple neural network performed comparably to Catboost with default settings. However, using less-than-minimum imputation, Catboost with default settings substantially outperformed carefully optimised DL models, achieving the best overall accuracy. We conclude that handling missing values is an important, yet often overlooked, step when comparing DL to GBDT algorithms for tabular data.

Published in Transactions on Machine Learning Research (12/2024)
Investigating the impact of missing value handling on Boosted trees and Deep learning for Tabular data: A Claim Reserving case study
Alexander Larionov
dlesander.aarioooo15555
Department of Mathematics Imperial College London
Niall Adams
n1. nd.ddams www.emmalmmmmpii.......o.
Department of Mathematics Imperial College London
Kevin N. Webster
kevin, wwt....wbbserr@@perii.....
Department of Mathematics Imperial College London
Reviewed on OpenReview: /ttps: openreviee. net/ forum? id= aV=
Abstract
While deep learning (DL) performance is exceptional for many applications, there no consensus whether DL or gradient boosted decision trees (GBDTs) are superior for tab- ular data. We compare TabNet (a DL model for tabular data), two simple neural networks inspired by ResNet (a DL model) and Calboost (a GBDT model) dataset for the task of claim reserving. This dataset is of particular interest for its large amount of informative missing values which are not missing completely at random, high- lighting the impact of missing value handling on accuracy. Under certain missing value schemes a carefully optimised simple neural network performed comparably to Catboost with default settings. However, using les--than-minimun imputation, Catboost with de- fault settings substantially outperformed carefully optimised DL models achieving the best overall accuracy. We conclude that handling missing values is an important, yet often overlooked, step when comparing DL to GBDT algorithms for tabular data.
large UK insurer
1 Introduction
Many machine learning problems involve regressing classifying with tabular structured data. Since
their introduction, GBDTs have performed well on such tabular data (Friedman 2001 Meanwhile, DL has become the state of the art in many problems that involve unstructured data, e.g. images (He et al. 2016
Simonyan & Zisserman 2015; Tao et al. 2020), audio (Ao et al. 2021), text (Baktha Tripathy 2017
Ziegler et al., [2019] Touvron et al. 2023 and their combinations (Radford et al. 2021 Rombach et. al. 2022 Ramesh al. 2021h.
Naturally, with the rise of research in DL, architectures have been proposed that claim to outperform GBDTs
tabular data (Somepalli et al. 2021 Shavitt & Segal 2018 Huang et al. 2020 Kadra et al. 2021
However, there is lack of consensus on whether these architectures rcally are more accurate. Large studies (Grinsttajn et al.] 22022 McElfresh et al. 2024 Borisov et al.) 2022) have compared DL to GBDTs across many datasets, many tasks and different computational budgets and suggest that GBDTs are on average the more accurate model for tabular data. The proposed reasons for the performance edge of GBDTs in these large studies remains an area of active research. Theories include the ability of GBDTs ignore irrelevant variables and model discontinuous functions (Grinsztajn et al. 2022 Importantly, neither the proposed DT.

---

Published in Transactions on Machine Learning Research (12/2024)
architectures nor the large studies investigate the impact of missing values. As missing values are common in real tabular data (Van Ness et al. 2023), and missing value handling can significantly impact the results of analyses (Jin et al. 2021), this is a significant gap in the literature.
In the remainder of this paper we use a claim reserving application with data from a large UK car insurer to shed light on the comparison of GBDTs to DL, with a particular focus on the impact of missing value handling. We specifically study this real-world dataset because it is characterized by extensive missing values that are explicitly not missing completely random (MCAR) offering a more realistic scenario than injection of MCAR missing values into previously studied complete datasets.
We investigate Catboost (Prokhorenkova et al. 2018) and three DL architectures: TabNet (Arik & Plister 2021) and two ResNet-inspired multi layer perceptrons (MLPs). TabNet was chosen as a specialised tabular DL architecture with prior validation in insurance (MeDonnell et al.) 2023). Catboost and ResNets MLP were chosen as the respective best GBDT and DL model from McElfresh et al. (2024).
In Section 2 we present background on our application as well as describe the car insurance dataset that we analyse. In Section 3 we give a more detailed description of the modelling strategies we compare. In Section we cover three elements of methodology. Section 4.1 covers our approach to hyperparameter tuning, where extra care was taken to avoid bias. Section 4.2 covers the experiments used to investigate the impact of missing value handling. Section 4.3 briefly covers experiments studying injection of MCAR missing values into other previously studied complete datasets. In Section 5 we present and discuss the results on our insurance dataset, not only finding Catboost is the superior model for our data, but also highlighting the importance of missing value handling in model accuracy.
2 Background
Car insurance is an important financial service with a 2024 global value of over 1.9 trillion USD, which is estimated to reach over 2 trillion USD by 2028 (Statista 2024h. Car insurance works on the principle that insurers charge customers a premium in return for obligations to provide financial support in the event of contractually agreed risks. Accurate pricing is vital for both the sustainable profit of the insurer and fair prices for customers. The process of determining a price for prospective customer in car insurance is complex (Olivieri & Pitacco 2015 Werner & Modlin 2010). It comprises of three core steps i) estimating the expected value of payments to the customer over the duration of the contract ii) estimating current liabilities for clains that are reported but not settled (RBNS) and iii) somehow sensibly combining the two prior estimates into a price. The first step typically comprises finding model for claim frequency and a model for claim severity; the latter estimating the cost of a claim conditional on an accident. The second step comprises modelling the cost of claims conditional on them having already occurred and is called claim reserving. The third step combines the claim frequency, claim severity and claim reserve estimates using risk models and business considerations: such as profit margins, legal requirements, risk appetite and operational costs.
The locus within this work will be on the second step: claim reserving. Specifically, we focus on outslanding claim reserve modelling which is the process of predicting costs for claims that have been RBNS.
Typically, outstanding claim reserve modelling is mainly done on a portfolio level. In other words, insurance companies predict the overall reserve requirement for a given time period, say a quarter, across all customers. Importantly, these forms of claim reserve modelling use no individual claim information, instoad using historic portfolio level settlement aggregates. This is done with deterministic algorithms such as run-off triangles, the chain ladder (CL) method and the Bornhuetter-Ferge Fergussn algorithm (Bornhuetter & Ferguson 1972 ; stochastic extensions of said algorithms.
We focus instead on individual claim reserve modelling, or micro-level reserving, an alternative method of reserving. Individual claim reserve modelling predicts portfolio reserves from aggregating estimates per incident. There not yet a consensus that individual claim reserving is more or less accurate than aggregate modelling. Still, the hypothesised benefits of micro-level reserving are: greater insight into exposure profiles within a portfolio; more signal (i.e. relevant covariates) should produce more accurate models; and the
2

---

Published in Transactions on Machine Learning Research (12/2024)
ability to adapt to trends that can be captured by covariates (Blier-Wool Wong et al. 2021 Lopez et al. 2019
Delong & Wüthrich] 2020).
There is literature investigating the use of older machine learning (ML) algorithms such as CART
Breiman ct al., 1984) and generalized linear models for individual claim reserving (Lopez et al., 2019 De Felice &
Moriconi] 2019; Taylor et al. 2008 Wuthrich 2018). Newer ML. methods, such as neural networks (Delong
& Wüthrich 2020; Delong et al.] 2022 Kuo, 2020 and GBDT's (Duval & Pigeon| 2019) have also had
some, limited, research. These works analysing micro-level reserving strategies broadly conchude that their
respective models are either par or better than aggregate CL method, validating miero-level reserving in principle. However, there are only a few such works; their insurance fields vary; they use small sets of covariates and some use simulated data. This makes it difficult to know whether the results are relevant to car insurance micro-level reserving. Furthermore, of considerable practical importance is that missing data is endemic to real insurance data (Fauzan & Murfi, 2018 Hanafy & Ming (021) and none of these works give any special focus to missing data. Finally, these works often report benchmarks against the CL method instead of overall accuracy which complicates the interpretation of results as disagreement with CL could be the consequence of more accurate modelling. To our knowledge, also noted by the survey of Blier- Wong ct al. 2021), none compare modern ML methods directly to cach other on real data. The lack of direct comparison means no conclusion can be drawn about the relative performance of newer ML methods for claim reserving.
Ultimately, both research insurance and ML more broadly paints a blurry picture on the relative merits of GBDTs and DL for micro-level reserving using tabular data. Furthermore, treatment or influence of missing values on accuracy is not investigated when comparing the methods. Although missing value handling has been shown to be important in other fields (Herring et al. 2004) and as such could be important to reserving. This leaves reserving actuaries dealing with tabular data unclear on whether it is worth the investment to investigate and deploy these more modern ML algorithms nor the impact of missing data for said algorithms.
The most relevant work to ours, comparing DL to GBDTs in insurance, is McDonnell et al. (2023). They compare the DL. architecture TabNet. (Arik & Pfister 2021 to the GBDT implementation XGBoost. They model discretised claim severity classification on a dataset with hundreds of thousands of claims. They find TabNet to be comparable to XGBoost, with marginally better F1 score. Although this is modelling claim severily, not claim reserving, we note that claim severity and claim reserving both model costs conditional on an accident occurring. However, claim severity is estimated before the accident occurs and claim reserve after. From the perspective of regression, the only difference between micro-level reserving and claim severity modelling is the number of covariates- as more is known after and accident occurs. In the work of MeDnelll et al.] (2023), although TabNet performs comparatively well XGBoost, the models were evaluated on synthetic data generated using a neural network (So et al. 2021) thus potentially biasing performance towards DL as the model class was more likely to be correct. Furthermore, the casting of the regression problem into discretised classification and lack of missing cata makes the findings less interpretable and transfcrable.
2.1
Data description
The tabular data we model in Section g consists of many hundreds of thousands of insurance claim feature vectors as rows, with hundreds of features as columns. This dataset has never been previously studied. The data combination of information available at policy issue (e.g. make and model of the car) and information available just after the time of claim reporting (c.g. accident date). The settlement value (SV) variable gives how much the insurer paid overall lo settle a claim; inclusive of vehicle, personal and property damage. We aim to accurately predict SV for each claim to build micro-level reserve, as described in Section 2 As we use supervised ML methods we only consider closed claims, i.c. there exists a SV to be used as a label.
Commercial confidentiality prevents us from giving a more detailed description of the data. However, we present the missing data properties the next section. We present other data characteristics and their implications for modelling and data processing in Appendix A.I this includes time varying properties and handling of high cardinality categoricals, such as postcode information.
3

---

Published in Transactions on Machine Learning Research (12/2024)
2.1.1 Missing data
The dataset we study has extensive missing values. Over 50% of features contain missing values, therefore ignoring all features with missing values would drop the number of features by over half. This could drop highly informative features, e.g. details of additional drivers on policy, which are missing in the majority of claims.
Furthermore, due to the interaction of missing values in multiple features there is complete feature vectors, i.e every row has at least one missing value. Therefore if we directly apply strategy such as complete-cas analysis (Little & Rubin 2019 p. 47), where any row with missing values is dropped, the whole dataset would be dropped. Instead, we can first drop features that are missing in more than a certain proportion of cases and then run a complete-case analysis. This latter approach also used to deal with missing values by GGrisstajj al. (2022), one of the broad comparative studies mentioned in Section | We explore this method, along with alternative imputation approaches, calling this missing value handling strategy Drop in Section !
Beyond the extent of missing values, the data presents dependence of the response, SV, on the missing value structure. This can be shown by a large shift in the mean and standard deviation of the SV when using Drop at various missing value proportion thresholds. Smaller proportions of missing values in feature vector are associated with substantially higher SV. This suggests that the data is not MCAR (Little Rubin 2019 p. 13-23). Therefore, fitting a model under a Drop strategy will result in biased predictions, above and beyond any bias introduced by the model training algorithm. This bias also means the accuracy results of model fit Drop not comparable to those of model fit imputed data.
To summarise, missing values represent large portion of our insurance dataset and are not MCAR as the SV varies substantially conditional on the missing value structure. This highlights the importance of investigating and choosing appropriate missing value handling strategies.
3 Models
In this section we start by defining notation, outlining some DL terms and then briefly give background on the models used: i) Catboost, a GBDT model; ii) two ResNets, a general purpose feed forward DL model: ours and that of Gorishmiy et al. (002]); ; and iii) TabNet, a DL architecture specifically designed to accommodate tabular data. Within this section we do not aim to provide comprehensive details. Instead we aim describe methods in sufficient detail to follow the hyperparameters tuned Scction 4.1
3.1 Notation
We denote the dataset D, as a set of tuples, = {xxx y1) ))) where y E R+ denotes the target settlement value, N denotes the number of claims and x denotes a feature vector with D features. Subscripts denote indexing arbitrary ordering of data tuples from the overall dataset.
We seek a model, F(x), to predict the claim settlement value y. The accuracy of this model is measured by some loss function L(y, F(x)), that we wish to minimise.
3.2 Catboost model
Catboost (Prokhorenkova et al. 2018 is GBDT (Friedman 2001 with a special procedure for categorical encoding and gradient estimation. Note that the Catboost algorithm details are complex and have many configurable options. Here we only cover the relevant details of the base GBDT algorithm and briefly mention the core novel concepts proposed by PProkhorenkova al. (2018). For removal of ambiguity, as the default behaviour can vary depending on the execution hardware, we present details and use defaults for running CPU opposed to GPU.
4

---

Published in Transactions on Machine Learning Research (12/2024)
3.2.1 Gradient boosted decision tree
Boosted models learn an additive ensemble of 'weak learner' models. If is the total number of weak learners we want to use, the boosted model would be:
Fr(x) Go(x) + Σ G
(1)
where (x( the tth 'weak learner', 17 weighting factor, and Go(x) is an initial estimate, such as the mean response of the training set.
Usually boosted models are built in sequential fashion, e.g. the ith model incorporating i weak learners would be F; = F; + Gp for i = I. The sequential construction of the model enables the procedure to be terminated early validation performance is not improving i.e. return (x(x) with i <.
For gradient boosting, the summands (x))), {l,..., T), are chosen from within a hypothesis class of functions to approximate 3 (y. F(x)) F.-1" the negative functional derivative of the loss. This negative functional derivative of the loss is also called pseudo-residual and denoted r2-1 x) (Friedman 2001
To evaluate [t-1 x) requires knowledge of both x and y. As is unavailable outside the training set, (y.1 x) can only be evaluated on the training data. However we can approximate r--1 with a given summand (x( and measure of function fit,, L' (rrt-11
G. = Farg min Geg
Σ (x,))ED
(x()-1(x,9), G()).
(2)
In practice finding the true arg min is infeasible so G2 is some approximation learned following standard algorithm to minimise T."
can be different from L as it is used to fit Gx(x) to T--1 (y, F(x)) to enable derivative evaluation on data outside the training set. is the addition of Gy to the ensemble that contributes to the minimisation L given a small enough step size 17.
This results in the boosted ensemble approximating gradient descent of the loss functional (in the space of linear combinations of G) with constant learning rate r7:
OL
Fj F + r/ Fi - aF P
(3)
In the context of GBDTs; the weak learner a docision tree (reimaa et al. 1984.. 1) and desired ensemble size are among the hyperparameters tuned in Section 4.1
The choice of step size
3.2.2 Catboost: Pseudo-residua, calculation and categorical encoding
Catboost aims to improve performance on unseen data by reducing overfitting. The key innovations of Catboost are twofold: i) how the pseudo-residuals, Fz-1, are approximated using G, and ii) how categorical variables are encoded. Although we will describe the core idea of the improvement, there are further tech- nicalities and engineering modifications present in Prokhorenkova et al. (2018), e.g. improve speed, that we do not describe.
For the alteration to G, fitting, the core idea is to fit Ge data excluding the data point for which it will predict rt-1, i.c. to calculate (x(xx) Catboost would fit G, on data (x, : < k}. This excludes the data point X5, and also generates different G, for different data points.
Likewise, Catboost follows this procedure for generating a categorical encoding. For a given data point x Catboost fits a target mean encoding (Pargent et al. 2022) on discrctized target for {x, : < k} that are before the point encoded. Furthermore, when processing categoricals Catboost uses novel algorithm redefine category labels as the algorithm runs ('feature combinations in the original work).
5

---

Published in Transactions on Machine Learning Research (12/2024)
We note that there are further important implementation details regarding the categorical encoding, such as how the target is discretised prior to mean encoding, that are absent from the original publication. Full details can be found in the tool's documentation (Catboost 2024b) and codebase (Catboost 2024a
This work studies Catboost with two different categorical encoding schemes: first with target mean encoding, and second with Catboost's 's novel debiased target encoding, that also employs category redefinition.
3.3 ResNet MLP
A ResNet, short for residual network, MLP is a feed-forward neural network (Murphy 2022 p. 419) with additive residual connections that skip layers. Without additional knowledge about the underlying structure of the data, an MLP is a simple general purpose DL architecture; and skip connections make training more stable MMrphy 2022 p. 445).
We implement ResNet by using residual connections across building blocks, along with skip connections to the output. We use a building block layout of BatchNorm, ReLU and Dense as in He et al. (2016) We add Dropout following the example of (Gorishnyy et al. (2021) which proposed the best performing DL model (a ResNet) from McElfresh et al. (2024). For the sake of clarity, our ResNet MLP is not identical in architect.ure to Gorishniy et al. ), and by extension McElfresh et al. (2024). For details of the differences with Gorishniy al. (2021) see Appendix A.. To contextualise our findings we also present results using the ResNet architecture from Gorishniy et al. 2021 referring to it 'RTDL ResNet'.
Figure 1 shows the layout of our ResNet MLP layers on the lar left, with their combination into a sub-block denoted by A. The ResNet sub-block, A, is repeated in a residual pattern to form a high level block B shown on the right of Figure [ This higher level block B is in turn composed using skip connections into an overall model. Each block has independently trainable parameters. In the context of deep learning, choosing the architecture size (such as number of blocks, size of Dense layers in units etc.) is a part of the broader problem of hyperparameter tuning. Our approach involves choosing the number of B blocks to vary depth; and choosing the number of units used in every Dense layer to vary the width of the network. This tuning is further described in Section 1.1 We use a grid search, described in Section 4.1.3 to select the optimiser used, the initial learning rate of the optimiser, lcarning rate schedule (Murphy weight regularisatio strategy and regularisation intensity.
2022 p. 288), Dense layer
3.4 TabNet model
TabNet (Arik & Pfister 2021 is DL architecture specifically designed for tabular data. TabNet works by learning a step that multiplies subset of features by zero, conditional on the input. Then TabNet uses DL layers on the remaining non-zero parts to produce an intermediate decision vector. The architecture sequentially applies multiple steps. Each step can determine a different subset of features to set to zero so a feature that is set to zero for one step does not need to be zero for the following steps. Tn fact, the hyperparameters described below control how many distinct features can be selected and their potential for reuse across steps. As each step can select different subsets, each step can produce different decision vector. Finally, the decisions from all steps are combined through a DL layer into a prediction.
TabNet is a complex architecture for which we defer the detailed description to the original paper (Ari & Pfister 2021). However, there are some key hyperparameters which we are required to tune.
The number of steps, S, determines the number of different feature subsets that are modelled to produce a prediction. With S = only a single subset of the features used, with more steps resulting in more feature subsets. Intuitively, more steps increases the overall number of features used, but also increases the depth of the network and destabilises training.
The so-called 'relaxation parameter', 2 1, is designed to encourage different feature subsets to be selected at each step. When 7 = 1 TabNet has the special property of being able to prevent reuse of features bet ween steps. As y increases, TabNet is more able to reuse features between steps.
The sparsity regularisation coefficient, X 2 0, is used to encourage more input features to be zeroed out in each step. As X increases the network zero out more features, even if it decreases training accuracy.
6

---

Published in Transactions on Machine Learning Research (12/2024)
Dense
Norc
ReLU
B
A
Dropout
B
Dense
y
Figure 1: ResNet MLP model architecture. The architecture consists of number of blocks, B. The skip connections in the diagram indicate that the outputs from each block B are all summed together and passed through final Dense layer to produce the scalar output j. Each layer consists of three sub-blocks denoted as A the diagram. Each B contains single residual connection so that the block output is produced by summing the outputs of the final two sub-blocks A. Each sub-block A consists of feed forward BatchNorm, ReLU, Dropout and Dense layers.
4 Experimental method
To investigate the impact of preprocessing schemes for handling missing values, first tuned the hyper- parameters of each model. Preliminary analysis, described in Appendix A.5 suggested the best hyperpa- rameters did not vary with preprocessing scheme. Therefore, the hyperparameter tuning process was done independently of later missing value invostigation.
To prevent data leakage, the last 15% of the data was set aside inlo test sel, Diest shown Figure This Diest was always withheld from training validation procedures and only used to report the metrics presented in Section 5
Farly stopping Murphy 2022 p. 448) was applied to improve training speed and prevent overfitting in both hyperparancter tuning and final model training. Early stopping is a form of regularisation where out-of- sample model performance is evaluated at regular intervals on a dataset withheld from training. When the performance on the withheld dataset decreases, the training algorithm is terminated. Preliminary analysis confirmed there was no decrease accuracy from using early stopping.
Section 4.1 describes the luning of hyperparameters discussed in Section Section 4.2 describes the exper- imental setup used to investigate the impact of missing value handling and categorical encoding. Practical commentary on the training speeds of the algorithms can be found in Appendix A.2
4.1 Hyperparameter tuning
Hyperparameter luning on our ResNet and TabNet consisted of grid search optimising for accuracy. For each modelling strategy a hyperparameter (HP) grid was subjectively chosen after initial trial and error. Although grid search may be common practice industry, it is also well known to thcoretically underperform more principled methods of HPO such as Optuna (Akiba et al. 2019). As such, we also reran analyses for our ResNet using Optuna with the same upper and lower bounds numeric hyperparameter
7

---

Published in Transactions on Machine Learning Research (12/2024)
D tratn
D ES
D HP
D test
time
Figure 2: Dataset partitioning strategy for both hyperparameter luning and final evaluation, where DES and Dtrain are shuffled per replication of a given experiment. DES is a split of data used for early stopping. DHP is a split of data used for evaluation of hyperparameters. DHP is sampled randomly in time, Diest is exclusively future data.
values. Those results can be found Appendix but we note that in this instance Optuna hyperparameter optimisation (HPO) did not significantly change performance. For consistency with McElfresh al. (2024)
Opluna was used for HPO on the RTDL ResNet model and presented in Section
Table
A single data subset, DHP was sampled once for accuracy evaluation of all models under any given HP configuration. For clarity, DHP was not a future partition the data, but rather randomly sampled time.
For each node the grid, we randomly split the remaining data (after removing Diest and DHP) into Dirain and DES subsets, illustrated in Figure Training was performed solely on Dirain whilst D was used to trigger carly stopping (ES). Once training completed, the root mcan squared error (RMSE) for given node was evaluated on DHP, This random splitting of Dirain and DES and subsequent evaluation of RMSE on fixed DHP, was independently repeated 10 times for each HP node in the grid. This repeated split sampling is called Monte-Carlo cross-validation (Kuhn et al. 2013
71-72).
The best HPs for a given model were chosen on the basis of the lowest RMSE averaged across the Corto cross-validation samples. This best HP configuration for given model was then used for the model-to-model comparison described in Section
The use of separate data subsets for early stopping, DES and accuracy evaluation, DHP allowed us to remove bias associated with evaluation on data that was indirectly used for training. This is an especially rigorous process in contrast to what is often done in practice, where one validation set would be used for both early stopping and evaluation. However, as we had sufficient data, we opted for separate subsets to minimise potential bias.
Further details of the training algorithm and hyperparameter selection for each modelling strategy follow.
4.1.1 Catboost hyperparameter tuning
Initial manual exploration of the hyperparameters gave no significant improvements in DHP RMSE. The step size, 77, was varied between 0.005 and 0.018. The ensemble size, T,, was varied between 1000 and 10000. Mean squared error was used as the loss function, As no noticcable improvement came from hcuristic tuning, complete grid search was performed and all hyperparameters were left defaults for evaluation of Catboost in model-to-mon used to speed up training, and had no noticeable effect accuracy in the hyperparameter tuning stage.
comparison. The only non-default choice inclusion of early stopping which
4.1.2
TabNet tuning
For TabNet we follow the original paper (Arik & Pfister 2021 in using the Adam optimiser (Kingma & Ba 2014h and exponential decay learning rate schedule with fixed initial learning rate of 0.001. The

---

Published in Transactions on Machine Learning Research (12/2024)
sparsity regularisation coefficient, X; mumber of steps, S: and relaxation parameter y were tuned. Following the recommendations and ablations of Arik & Pfister (202T): À was varied between 0.0001 and 0.01, 7 varied between 1.3 and 2 and S was varied between 3 and 10.
4.1.3 ResNet MLP tuning
Currently, there is no principled way to construct deep learning architecture, for example choosing depth and width, beyond intuition; trial and error; and neural architecture search (Ren et al., 2021 As neural architecture search was prohibitively expensive from computational perspective we instead opted for a heuristic trial and error approach for choosing our ResNet architecture.
We began by arbitrarily choosing some expressive high level block, denoted B Figure We built this block from sub-blocks, denoted A in Figure [ layers as described in Section 3
Sub-blocks utilise ResNet (He et al. 2016) layout of
We then chose the depth: i.e. the number of B blocks composed together prior to Dense layer with single output unit, with no activation, for prediction. A depth of 3 was chosen; subjectively balancing simplicity of the model with expressive power.
We then parameterised the width of a block with d, the number of units in each Dense layer of the ResNet block, denoted Figure We varied d between heuristically identified limits wherein the model exhibited underfitting and the capacity to overfit training data. Underfitting was identified by both the training and validation error being similar and approximately constant per training epoch. Capacity to overfit identified by the ability for the model to keep reducing training error whilst validation error is constant or getting worse. These criteria were considered fulfilled when the validation loss was not improving whilst training loss was still decreasing after 1000 epochs.
Varying d within the bounds of under and overfitting did not noticeably impact accuracy. As such, d was chosen for a total model size of four hundred thousand parameters. This was between the number of parameters which under- or overfitted. After the architecture was heuristically selected, grid search hyperparameter optimisation followed. The optimisers compared were Adam (Kingma & Bal 2014) and RMSProp (Tieleman & Hinton 2012). Both L.1 and 1.2 regularisation of kernel weights were used; coefficients for each were varied between 0.01 and 1. Exponential Decay (Murphy 2022, p. 288) and Cosine Decay [Loshchilov & Hutter 2016)) learning rate schedules were both compared; where initial learning rate was varied between 0.001 and 0.01 for each.
4.2 Investigating different preprocessing schemes
We aimed to investigate the impact of missing value handling through the comparison of each model under different preprocessing schomes.
To enable a pairwise comparison of trained models, the data subsets for these experiments were different from those of the hyperparameter tuning in Section 1.1 Instead of sampling different subsets per node in grid scarch all experiments used the same set of 20 Monte-Carlo cross-validation samples of Dtrain and DÉS These were sampled once prior to running any experiments, enabling pairwise model comparison. As evaluation was performed on Diest, there was no need for another withheld evaluation partition, like D
A given Dirrin subset was used for training all models with all missing value handling schemes, described below, and the corresponding DES was used to trigger early stopping. After the models were trained, RMSE was evaluated on Diest for each of the 20 Dirain and DES samples. The mean and standard error of the RMSE across the 20 Monte-Carlo cross-validation samples for each model and preprocessing scheme are reported in Table I These results are discussed in Section 5
4.2.1 Missing value handling
Four missing value handling strategics were investigated. Two of the mothods, Drop and Binarize, were used in an attempt to disentangle the capability of a model to successfully extract signal from observed covariates and from missing value structure. The other two, LT Min Impute and Mean Impute, were used to
9

---

Published in Transactions on Machine Learning Research (12/2024)
300
0.75
N/A
36
0
Figure 3: Example row undergoing Binarize transformation; where 'N/A' represents missing values. Present covariates map to 1; missing to 0.
investigate model capability to jointly use covariates and missing value structure through imputation. The missing value handling strategies summarised:
Drop: Rows with any missing values were dropped after first removing features with more than 30% missing values described in Section 2.1 From this scheme aimed to isolate the ability of each modelling strategy to extract signal from observed covariates.
The whole dataset was converted inlo a binary representation of whether the covariate was observed or not, see Figure 3 This retained only the signal inherent in the missing value structure. With this scheme isolate the ability of model strategics to regress against missing value structure.
Binarize:
LT Min Impute: Missing values were imputed using less than the minimum of the numeric feature. This is the default missing imputation strategy of Catboost. The use of this strategy allows the weak learner trees, G, in Section 3 to separate the missing values completely from observed values. However, this also means that should the tree split an observed value it will include all missing values on the lesser side of the split.
Mean Impute: Missing values were imputed using the mean of the corresponding numeric feature. This was the approach used in McElfresh et al. (2024).
We also present results using both the Min Impute and Binarize stralegies simultaneously in Appendix finding there is no noticeable benefit to combining the approaches over Min Impute for our dataset.
A.7
4.2.2
Categorical encoding
Categorical encoding was performed using larget mean encoding for the majority of the experiments, this compared the modelling strategies fairly by holding potential confounders in categorical encoding constant.
In the interests of identifying the best performance, we also evaluated the original Catboost categorical encoding with the Catboost. GBDT algorithm, described in Section 3.2 We report the model names in Table I with the suffix(CE) indicating that Catboost encoding used; otherwise mean encoding used.
4.3 Effects of imputation scheme on other datasets
To study the broader relevance of results obtained from our insurance dataset, further analyses were per- formed two other datasets under Drop, Mean Impute and LT Min Impute. The leading two GBDTs and leading two DL algorithms from McElfresh et al. (2024) were compared on two datasels using the TabZilla framework. The datasets did not contain missing values; so MCAR missing values were simulated by removal. As MCAR data are not the focus this case study we present the results in Appendix [A.6] Overall, these preliminary analyses showed lesser impact from missing value handling than our not MCAR application but still demonstrate the relevance of our work.
5 Results and discussion
The results for various preprocessing strategics are presented in Table [ As a benchmark, we include Mean prediction row which shows the performance that is obtained by using the mean SV of given training dataset constant prediction, using feature information. Note the performance evaluated with the
10

---

Published in Transactions on Machine Learning Research (12/2024)
Drop preprocessing scheme is not directly comparable to the performance for other preprocessing schemes, T.e. Drop cannot be compared with other strategies across rows in Table I This is because the missingness mechanism is not MCAR and causes a distributional shift within both the training and evaluation data when dropping rows. However, due to consistency of the training datascts, these results are comparable within modelling stralegies, i.e. one can compare down columns of Table
Dlest RMSE mean and standard error, se, over cross validation partitions. Lower RMSE is better. RMSE is to the nearest integer, is to 2 significant figures. (CE) indicates Catboost encoding used for categorical encoding. t replicates the HPO, architecture and impulation scheme of McEllresh et al.
Table 1:
( corresponds to an Optuna HPO scheme.
Model
Drop
(+see
LT Min
(+8e)
Mean
( (+8
Binarize (+8E)
Impute
Impute
Catboost (CE)
1969
(+10)
1452
(=6.3)
1519
(+17)
2302
(#0.18)
Catboost
2060
(+1.7)
1574
(+2.1)
1524
(+5.2)
2268
(+0.28)
Our ResNet
2046
(+1.2)
1872
(+4.5)
7855
(+1600)
2288
(+1.8)
RIDL ResNett
2439
(+24)
2007
(=20)
2387
(=270)
2282
(#2.4)
TabNet
2702
(#8.1)
2748
(=5.3)
3081
(+180)
2754
(#5.1)
Mean prediction 2813
(+0.34)
2764
(#0.020)
2764
(=0.020)
2764
(=0.020)
The TabNet row of Table shows that TabNet substantially underperforms both Catboost and the ResNet MLP. Notably, TabNet performs on par with the Mean prediction benchmark. This indicates that TabNet is either not suitable for micro-level reserving with this dataset or, at best, that TabNet is very difficult train. This runs counter to published work on the use of TabNet for claim severity modelling (MeDonnell et al. 2023), but in line with surveys of DL for structured data (Grinsztajn et. al., 2022 McFlfresh et al., 2024 where TabNet performed poorly. As mentioned in Section [ this could potentially be explained by the fact that the results of McDonnell et al. (2023) could be biased because the modelled data is itself generated from a neural network.
The relative accuracy of the RTDL ResNet MLP row of Table [ agrees with McElfresh et al. (2024) in outperforming TabNet and underperforming CatBoost. Although for Drop, Min Impute and Binarize the RTDL ResNet has lower accuracy than our ResNet it is interesting to note that under Mean Impute the RTDL RosNet performs substantially better than any other DL algorithm. This highlights the sensitivity of previously studied algorithms to imputation schemes.
However, we note that the performance of Mean Impute is still generally worse than that of LT Min Impute across most models. Mean Impute demonstrates modcrately high standard error for the RTDL ResNet and high mean RMSE and standard error for our ResNet and TabNet. The exceptionally high RMSE for our ResNet appeared to be due to the rare prediction of low value claims many orders of magnitude larger than they were. It is unclear why this would be the case speeifically for Mean Impute and our ResNet. We hypothesise this could be due to Mean Impute making it difficult to distinguish meaningfully missing data, where absence of data is not due to random lack of records but from the reality of the data generating process, e.g. no additional drivers on the policy.
The Drop and Binarize columns of Table [ show that both the covariates and the missing value structure are important in the performance of both Catboost and ResNet, respectively offering approximately a 27% and 18% improvement over Mean prediction benchmark. Interestingly, neither ResNet nor Catboost is practically more accurate than the other when trained on either covariates (Drop) or missing value structure (Binarize) alone. This similarity of performance mentioned in Section [ do not have conclusions that are necessarily transferable to the context of claim reserve modelling. Those studies found a performance edge for GBDTs, notably either ignoring or in the absence of missing data. Whereas we find, for this micro-level reserving dataset, Catboost and ResNet are practically the same in terms of accuracy with missing data ignored, due to Drop.
Drop suggests studies comparing GBDTs and DL,
However, the LT Min Impute and Mean Impute columns of Table Table shows that Catboost with LT Min Impute both achieves the best overall performance and has better accuracy than both Our ResNet and the RTDL.
11

---

Published in Transactions on Machine Learning Research (12/2024)
ResNet under either imputation scheme. We note that this better accuracy is between well-tuned ResNets and a default. Catboost, underscoring the robustness of Catboost. The improved accuracy may be because Catboost better leverages both covariate signal and missing value structure, at least under a LT Min Impute or Mean Impute strategy. The strongest performance being achieved under LT Min Impute could also be due to the fact. that minimum impulation intuitively lends itself to the partitioning stralegies of GBDT algorithm. It is unclear how minimum imputation would interact with ResNets, nor DL more broadly. This discrepancy in performance suggests an indicator of when Catboost may outperform ResNet for micro-level reserving: when there is high proportion of missing values. This indicator is particularly relevant missing values are pervasive in real world datasets, and are currently not studied in their role in micro-level reserving nor their contribution to GBDT and DL performance.
These results do not rule out the potential for ResNets to perform comparably or even better than Catboost given a suitable imputation strategy. However, there does not appear to be a scientific consensus on simple and robust neural network appropriate imputation strategies. Some generative imputation approaches exist (Yoon et. al. 2018 but. these involve considerable extra complication and have not shown much industry adoption.
6 Conclusion
In this paper we investigated four different imputation schemes, described in Section 4.2.1 for handling missing values using gradient boosted decision tree and deep learning models. We investigated the impact of these imputation schemes using large, real world insurance dataset with not MCAR missing values. Under two of the missing value handling schemes there is no noticeable difference between Catboost and the ResNets. However, using imputation, Catboost substantially outperforms the other models. This result highlights the importance of missing value handling in claim reserving.
More broadly, this result adds a case study to the body of evidence that gradient boosted decision trees can outperform deep learning for tabular data, but emphasises the importance of missing values in drawing this conclusion. The significant impact of missing value handling accuracy also suggests that when analysing datasets with missing values, extra care should be taken choosing the missing value handling method and not to just focus on model selection. Furthermore, our results suggest research comparing gradient boosted decision trees to deep learning for tabular data could benefit from including more datasets with missing values, especially missing values that are not MCAR. The importance of including missing value handling in comparisons was corroborated on other datasets using the Tabzilla repository (McElfresh et al.) (022)) by injecting MCAR missing values, described in Appendix A.6 However, with MCAR missing values the effect of missing value handling was lessened.
Based on the analysis in this work we recommend excrcising caution when using deep learning models for claim reserving as they require thorough tuning and their interaction with imputation schemes is not under- stood. Furthermore, using Catboost for claim reserving has some practical advantages, beyond potentially better accuracy with the correct missing value handling. Catboost fast, robust and easy use off-the- shelf. In comparison, deep learning methods require more expertise to deploy successfully: requiring both architecture selection and hyperparameter tuning. Even with the expertise required to select, implement and optimise deep learning models there is no compelling empirical or theoretical evidence that they are likely to produce better results for claim reserving.
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujic Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. Speecht5: mndaied-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.02205, 2021.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Sercan Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8):6679-6668) 2021.
Kiran Baktha and BK Tripathy. Investigation of recurrent neural networks in the field of sentiment analysis. 2017 International Conference on Communication and Signal Procossing, pp. 2047-2050, 2017.
Christopher Blier-Wong, Hélène Cossette, Luc Lamontagne, and Etienne Marceau. Machine learning in P&C insurance: A review for pricing and reserving. Risks, 9(1):4, Jamuary 2021. ISSN 2227-9091. doi: 10.3390/risks9010004.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systoms, pp. 1-21, 2022.
Ronald L Bornhuetter and Ronald E Ferguson. The actuary and IBNR. Casualty Actuarial Society, (9(1) 181-195, 1972.
Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone. Classification and Regression Trees. Wadsworth and Brooks/Cole Monterey, CA, USA, 1984.
Catboost. Codebase Iutorial on Catboost encoding, Feb 2024a. http://p//thtt..oo/////aoo//aaba beob/master/catbooo /ttp:///orotttteeeeeeeeaaaaaaaas ipynb. Accessed on April 2024.
catboost htgpha/sie/e.oooooooooa.caaaaaae
ffaaaeegggrrcclalfaueeeeeeeee features parameters. _eatuures_paaamttersss.....
Catboost. Transforming categorical features to numerical features Catboost, Feb 2024b. https://
tateee--ttttttt-t-ieeee Accessed on April 2024.
Massimo De Felice and Franco Moriconi. Claim watching and individual claims reserving using classification and regression trees. Risks, 7(4):102, December 2019. ISSN 2227-9091. doi: 10.3390/risks7040010
Lukasz Delong and Mario V. Wüthhich. Neural networks for the joint development of individual payments and claim incurred. Risks, 8(2):33, June 2020. ISSN 2227-9091. doi: 10.3390 0020000033
bukasz Delong, Mathias Lindholm, and Mario V. Wüthrich. Collective reserving using individual claims data. Scandinavian Actuarial Journal, 2022(1):1-28) January 2022. ISSN 0336-1238. doi: 10.1080/0344 0034//288... 2021.1921836.
Francis Duval and Mathieu Pigcon. Individual loss reserving using a gradient boosting-based approach. Risks, 7(3):79, September 2019. ISSN 2227-9091. doi: 10.3390/risks7030077
Muhammad Arief Fauzan and Hendri Murfi The accuracy of XGBoost for insurance claim prediction. International Journal of Advances in Soft Computing and its Applications, 10(2):159- 2018.
Jerome Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5), October 2001. ISSN 0090-5364. doi: 10.1214/a0s/ 100////001331111111
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-18944 2021.
Lóo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35:507-522 520, 2022.
Mohamed Hanafy and Ruixing Ming. Machine learning approaches for auto insurance big data. Risks, 9(), 2021. ISSN 2227-9091. doi: 10.3390/Hiskss0020044.. URL http://www..apl.oooo//2/////////
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European Conference Computer Vision, pp. 630 645, 2016.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Amy H. Herring, Joseph G. Ibrahim, and Stuart R. Lipsitz. Non-ignorable missing covariate data in survival analysis: A case-study of an international breast cancer study group trial. Journal of the Royal Statistical
Society Series C: Applied Statistics, 53(2):293 310, 03 2004. ISSN 0035-9254. doi: 10.1046/j.1467-901
2003.05168.x. URL https:///oi. org/ 1.t./.044/.... 1467 9876. 2003. 2003.05168.0.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modcling using contextual embedldnngs arXiv preprint arXiv:2012.06678, 2020.
Liang Jin, Yingtao Bi, Chenqi Hu, Jun Qu, Shichen Shen, Xue Wang, and Yu Tian. A comparative study of evaluating missing value imputation methods in label-free proteomics. Scientific Reports, 11(1):1760, Jamuary 2021. ISSN 2045-2322. doi: 00088/s41598-001-887777
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuedd simple nets excel on tabular 2021.
datasets. Advances in Neural Information Processing Systems,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
(xXiv:1412.6980 2014.
Max Kuhn, Kjell Johnson, al. Applied predictive modeling, volume 26. Springer, 2013.
Kevin Kuo. Individual Claims Forecasting with Bayesian Mixture Density Networks. arXiv preprint HrXiv:2003.02453. March 2020. doi: 2055500/rrXi.2003.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, 3rd Edition. John Wiley & Sons, 2019.
Olivier Lopez, Xavier Milhaud, and Pierre-E Thérond. A tree-based algorithm adapted to microlevel reserv- ing and long development claims. ASTIN Bulletin: The Journal of the International Actuarial Association, 49(3):741 762, 2019.
Tya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 2016.
Kevin McDonnell, Finbarr Murphy, Barry Sheehan, Leandro Masello, and German Castignani. Deep learning in insurance: Accuracy and model interpretability using TabNet. Expert Systems with Applications, 217:
119543, 2023. ISSN 0957-4174. doi: https://ooi.orr/ 10. 10tp://....../..2222.3.....3 119543. URL https://www.
sciencedirect. / htt.:///ww...//oooacaa.aaaaaa//r
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024.
Kevin P. Murphy. Probabilistic machine learning: An introduction. MIT Press, 2022. URL probml. ai
Robert Nisbet, John Elder, and Gary D Miner. Handbook of statistical analysis and data mining applications. Academic press, 2009.
Annamaria Olivieri and Ermanno Pitacco. Introduction to insurance mathematics: technical and financial features of risk transfers. Springer, 2015.
ONS. Postal geographies - office for national statistics, Feb 2024, httpp:///ww.c ons.wwon..noo www...g................ http:/ooss.go..uunnttoooooomoooe www..uuu/nttoooooogy//ooooo/oo// geogrppr//beorrpppeee//roaaggggg Accessed Feb 2024.
Florian Pargent, Florian Pfisterer, Janek Thomas, and Bernd Bischl. Regularized target encoding outper- forms traditional methods in supervised machine learning with high cardinality features. Computational Statistics, (7(5):2671-2692 November 2022. ISSN 1613-9658. doi: 10.1007/6001100002--007777
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31, 2018.
14

---

Accessed on Published in Transactions on Machine Learning Research (12/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning. pp. 8748 8763, 2021.
Aditya Ramosh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-lo-image generation. International Conference on Machine Learning, 139: -831, 18 24 Jul 2021. URL https://proceeddigss mlr, press//119/aamshh1aa.. html
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-yao Huang, Zhihui Li, Xiaojiang Chon, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys, 54(4), may 2021. ISSN 0360-0300. doi: 10.1145/3447588 URL
1rt../0.1/1..11/4776888111 1145/3447582
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjarn Ommer. High-resolution image synthesis with latent diffusson models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
org/ 2t5/abs/22112.1000
Ira Shavitt and Eran Segal. Regularization learning networks: Deep learning for tabular datasets. Advances in Neural Information Processing Systems, 31, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Banghee So, Jean-Philipppe Boucher, and Emiliano A. Valdez. Synthetic dataset generation of driver telem- alics. Risks, 9(4), 2021. ISSN 2227-9091. doi: 10.3399//risk:9040055.. URL
com/
2227-909/9///BBB
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training, arXiv preprint arXiv:2106.01342, 2021.
Statista. Motor vehicle insurance global: Statista market forecast, Feb 2024. http::://wwwssstttiie com/ outlook/fmo//oooooooooo Feb 2024.
Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmenta- tion. arXiv preprint arXiv:2005. 10821, 2020.
Greg Taylor, Gráinne McGuire, and James Sullivan. Individual claim loss reserving conditioned by case estimates. Annals of Actuarial Science, 3(1-2):215-256, 2008.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. Coursera: Neural networks for machine learning, 4(2):26 31, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinct, Maric-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
Mike Van Ness, Tomas M. Bosschieter, Roberto Halpin- Halpin-Gregore Gregorio, and Madeleine Udell. The Missing Indicator Method: From Low High Dimensions. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5004 5015, August 2023. doi: (0.1145/3580005533950
Geoff Werner and Claudine Modlin. Basic ratemaking. Casualty Actuarial Society, 4:1 320, 2010.
Mario V. Wuthrich. Machine learning in individual claims reserving. Scandinavian Actuarial Journal, 2018 (6):465-480) July 2018. ISSN 0346-1238. doi: 10. 101800034461111 1428681.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Jinsung Yoon, James Jordon, and Mihacla der Schaar. GAIN: Missing data imputation using generative adversarial nets. International Conference on Machine Learning, 80:5689-5608 10-15 Jul 2018. URL
mlr. /otp html
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08598; 2019.
A Appendices
A.1
Data properties
This appendix presents further properties of the data.
A.1.1
Missing value concentration
Tables
and
characterisc the concentration of missing values in our datasct.
Fraction f missing
Percentage of columns with given fraction missing
f=
45
<< f < 0.2
17
0.2 f 0.4
0 0.6
11
0 0.8
0 1.0
Table Missingness distribution by column, to nearest %
Fraction f missing
Percentage of rows with given fraction missing
0< f 0.2
46
0.2 < f S 0.4
19
0.4 < f S 0.6
12
0.6 < f S 0.8
0.8 f 1.0
Table 3: Missingness distribution by row, to nearest
A.1.2 Time-varying properties
Figure 4 depicts the mean claim value per month, with claim value and year of incident anonymised for commercial confidentiality. The figure shows that mean claim value is clearly dependent on the time of the claim exhibiting both seasonality and trend. This has implications for both the evaluation of model accuracy and encoding of time data.
When evaluating the accuracy of the ML models, described in Scction g the time series nature of the data requires the definition of the test set data be in the future relative all training and validation data. This is to prevent bias in the estimation of performance, form of data leakage (Nisbet et al.) 2009 p. 742).
When considering how to encode the time variables, we aim to encode in such a way as to make it casier to fit seasonality and trend. To explicitly encode cyclic limestamp properties, we encode month and day-of-month variables separately. Having cyclic values in the input data, like month and say-of-montil intuitively makes it casier fit conditional on cyclic scasons. In order fit overall trend, we chose also encode timestamps as monotonic value, enabling the model to order claims in time from single numeric value. We did this at two resolutions: year and fi) seconds since the epoch, also known as Unir time. The goal of encoding
16

---

Published in Transactions on Machine Learning Research (12/2024)
10-
Nermalzze claim
0.7
0.0
JAn Jen
Sep
Sep
Jan
Jan
Sep
incident
Figure 4: Time series plot of mean standardised claim values against anonymised time. Claim values were aggregated on month. Claim values and years of data are anonymised for confidentiality
in years is to allow a fit to large scale trends such as inflation. The goal of encoding in seconds is to enable more granular ordering in time. Therefore, overall we encode a single timestamp feature into 4 variables: year, month, date and seconds since the epoch.
A.1.3 High-cardinality categorical variables
The dataset contains some high-cardinality categorical variables: insuree car make with hundreds of cat- egories, insuree car model with thousands of categories and first half of insuree postcode with thousands of categories. The typical one-hot encoding (Murphy) 2022, p. 23) of these features would dramatically blow up the dataset size in number of features. Therefore, we applied numeric encoding strategies such impact encoding (Pargent et al. 2022), also known target mean encoding, and Catboost's novel categorical
encoding, described in Section 3.2.2
As the UK has 1.7 million postcodes (ONS 2024), attempting to treat full postcodes as categorical variable would give rise to at least hundreds of thousands of categories. Even target mean encoding such high cardinality categorical could exhibit high bias and high variance due to the low number of samples per category; following the intuition outlined in Prokhorenkova et al. (2018). Only modelling the first. half of postcodes was used to decrease the cardinality of the variable to thousands of categories to potentially reduces the bias and variance of the mean encoding.
A.2 Training speed
Using 12 Intel Xoon 6136 CPUs and approximately 60GB of RAM: Catboost trains on our datasct in the order of 3 minutes per training run and the ResNet. MLP architecture trains in the order of 45-120 minutes per training run. TabNet trains in the order of 300 minutes per training run. Although both Catboost and ResNet exhibited training times low enough be retrained many times a day to keep up with changes in underlying distribution; the significantly faster training time of Catboost enables greater experimentation. This is potentially relevant for other applications it has been shown that the difference
17

---

Published in Transactions on Machine Learning Research (12/2024)
in modelling strategies without hyperparameter tuning is just great the difference within model class with hyperparameter tuning when training on tabular data (Kadra el. al. 2021
A.3 Differences with McElfresh
There are few key differences between Our ResNet MLP and the RTDL ResNet MLP:
Our ResNet MLP HPO tunes more HPs; McElfresh et al. (2024) only tunes learning rate whereas ResNet uses an HPO grid containing learning rate, weight decay, regularisation coefficient, learning rate schedule and optimizer.
Our ResNet MLP does not attempt to embed categorical features, target mean encoding used to convert all categories into scalar representations.
Our ResNet MLP has skip/ hesidrll connections from output of each block to the overall model output; whereas residual connection only skip blocks in McElfresh et al. (2024).
We use three repeating blocks prior to the regression head layer; McElfresh et al. (0044) use two.
Our ResNet MLP does not vary dimensionality of the hidden state we keep constant dimensionality of 256, in comparison to McElffres et al. (2024) who step down to 128 and back up to 256 throughout the network. Ultimately, this and the above point result in our model having approximately 2.25x the parameters of McElfresh et al. (2024).
A.4 Optuna HPO Results
Table f shows a table analogous to Table fable[[comparing results obtained using Optuna for the HPO instead of grid search. It can be seen that there is no substantive difference to the conclusions in Table of the work from swapping to Optuna HPO; as core points are made in comparison of Our ResNet MLP and Catboost (which did not undergo HPO).
Table Results when using Optuna HPO
Model
Drop
(+se)
LT Min
(#se)
Mean
(=
Binarize (+se)
Impute
Impute
Our ResNet MLP Grid
2046
(+1.2)
1872
(+4.5)
7855
(# (+1600)
2288
(+1.8)
Search (ME)
Our ResNet MLP + Op-
2039
(+5.1)
1887
(+.65)
7673
(+16000
2283
(+1.3)
tuna HPO (ME)
A.5 Impact of imputation scheme on HPO
This appendix presents preliminary experimental data used to justify the process of performing HPO in- dependently of imputation scheme; as described in the beginning of Section 4 The closeness in parameter magnitude value and the robustness of HPO performance to varying hyperparameters suggested it was ac- ceptable to reduce the computational complexity of experiments by performing HPO under one imputation scheme and evaluating on all. The LT Min Impute scheme was chosen as initial results suggested would be the scheme with the best performance; and as such chosen in an attempt to give a level playing field for best performance across models.
A.5.1 Grid search based
The optimum hyperparameters obtained using grid search with LT Min Impute and Drop for our ResNet from preliminary analysis are presented in Table 5 Due similarity in the optimal hyperparameters obtained it was concluded from this preliminary analysis that HPO could be performed independent of imputation scheme; keeping overall computational cost down.
18

---

Published in Transactions on Machine Learning Research (12/2024)
Table 5: Best performing hyperparameters under different imputation schemes; following a grid search HPO procedure.
Hyperparameter
LT Min Impute
Drop
Regularisation coellicient.
0.01
0.01
Initial learning rate Learning rate schedule Optimiser Weight Decay
0.01
0.01
CosineDecay RMSProp 0.01
CosineDecay
RMSProp
0.0001
A.5.2 Optuna based
The optimum hyperparameters obtained using Optuna with LT Min Impute and Drop for our ResNet are presented in Table G With corresponding final outcomes presented in Table 7 can be seen from 7 that although tuning with Drop gives different hyperparameters; and gives our ResNet more stable results for Mean Impute; the relative rankings in the final Table would be unaffected.
Table 6: Best performing hyperparameters under different imputation schemes; following an Optuna HPO procedure.
Hyperparameter
LT Min Impute
Drop
Regularisation coefficient
0.017
0.68
Initial learning rate Learning rate schedule Optimiser
0.009
0.006
CosineDecay RMSProp
ExponentialDeccay
RMSProp
Weight Decay
0.0002
0.0002
Table Results when using Optuna HPO
Model
Drop
(+se)
LT Min
(#se)
Mean
(=se)
Binarize (=se)
Impute
Impute
Our RosNet MLP
+
2039
(+5.1)
1887
(+
7673
(=1600)
2283
(+133
LT Min Impute
Opluna
HPO
Our ResNet MLP + Drop
2047
(+443)
1885
(+446)
4548
(+680)
2292
(+2.0)
Optuna HPO
A.6 TabZilla replication
This appendix details the results obtained from comparing LT Min Impute, Mean Impute and Drop on some extra datasets and algorithms from the TabZilla Benchmark (McElfresh et al. 2024 Source code can be found at htttp:://githb..co http://hhw..caaapppppaa////aaaaa taabi/ilt///abalaaaaaaaaaaaaaala
30% of numeric values were randomly replaced with missing values from each dataset and then imputed using the imputation schemes studied. Then the corresponding OpenML task was performed using Catboost, XGBoost, FTTransformer (Gorishniy al. 2021 and the RTDL ResNet. We present the results by dataset in Table g
From Table we can see that the overall impact of imputation schemes is not large and in some cases causes no performance difference between certain imputation schemee.. However, the changes in accuracy rankings of the datasots under different imputation schemes demonstrates that in principle it is possible for the rankings, and therefore results such as those in McElfresh al. (2024), to be inffuenced by imputation scheme. We note that, by design, Table 8 shows results from injected MCAR missing values and such
19

---

Published in Transactions on Machine Learning Research (12/2024)
do not attempt to interpret the absolute performance of any imputation scheme MCAR data is not the focus of our work.
In summary, the TabZilla replication with different imputation schemes demonstrates in principle that han- dling missing values could be important; but the MCAR nature of the injected missing values and low signal value of the numerics the chosen datasets produces less pronounced effect than our work with non MCAR claim reserving. This highlights the importance of analyses using real world large datasets with not MCAR missing value structure in comparing GBDTs and DL models.
Table 8: Relative accuracy and ranking per dataset per algorithm using the TabZilla repository under different imputation schemes. We report TabZilla mean 10-fold cross-validation test accuracy; as extracted from the tuned aggregated results output. Dataset
Model
Mean
Drop
LT Min
Impute
Impute
Accuracy
Rank
Accuracy
Rank
Accuracy
Rank
ada_agnostia
CatBoost
0.857
0.854
2
0.857
1
ada ada_agnostine
FTTransformer
0.844
4
0.845
3
0.846
3
ada_agnostic
RTDL ResNet
0.846
3
0.842
4
0.841
4
ada_ aia_agnostic
XGBoost
0.855
2
0.855
1
0.855
2
LED-display
CatBoost
0.716
2
0.714
3
0.716
3
LED diD-ais
I'Transformmr
0.708
4
0.728
1
0.728
1
LED LiD-display
RTDL ResNet
0.724
1
0.724
2
0.722
LED-display
XGBoost
0.710
3
0.706
4
0.710
A.7 Combining missing value handling strategies
Table 9 shows the performance obtained when combining both the LT Min Impute and Binarize approaches, turning each D dimensional feature vector into 2D dimensional feature vector. For each original feature generate two features: one following LT Min Impute and another that is a binary indicator of whether the data is (Binarize). This could be interpreted as providing both features and 'missingness masks' of present features to the model. Table g also includes results for LT Min Impute and Binarize from Table [ for reference. Although it is a natural question to investigate combinations of compatible methods, we find that for our datasot there is no noticcable difference between combining LT Min Impute and Binarize and using LT Min Impute on its own.
Table 9: Diest RMSE mean and standard error, se, over cross validation partitions. Lower RMSS is better. RMSE to the nearest integer, se is to 2 significant figures. (CE) indicates Catboost encoding was used for categorical encoding. corresponds an Optuna HPO scheme.
Model
LT Min (#se)
Binarize (#te)
LT Min (=se)
Impute
Impute +
Binarize
Catboost (CE)
1452
(6.3
2302
(+0.18)
1449
(#1.6)
Catboost
1574
(+2.1)
2268
(+0.28)
1575
(+2.8)
Our ResNet
1872
(+:45)
2288
(+1.8)
1883
(+:8.)
RTDL ResNett
2007
(+-20)
2282
(+2.4)
1994
(+18.7)
Mean prediction
2764
(:.0.000)
2764
(#00020)
2764
(+00020)
20


Reviews:
[{"review_id": "9Mkp78VDfb", "reviewer": "Reviewer_STkm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses an important issue in the field.\n\nWeaknesses:\n\n- Limited data analysis. Only a single dataset was used, which limits the generalizability of the findings.\n- Writing clarity. The manuscript’s structure and language are often unclear, impacting readability and understanding.\n- Insufficient baseline comparisons. The study evaluates only one ML model and two DL models, which is inadequate for a comprehensive analysis.\n- Limited baselines for missing value handling (Section 4.2.1). Only three methods are considered, while more techniques are commonly used in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper address the issue of missing value in the tabular datasets for deep tabular learning models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Apart from the weaknesses noted above, I would recommend the authors:\n\n- Remove definitions of well-known algorithms: Exclude explanations for commonly understood algorithms like gradient boosting and ResNet. Similarly, consider removing Figures 1 and 2.\n\n- Consider a more suitable venue. Given that the paper exclusively examines an insurance dataset, it may be more appropriate for a journal or conference specializing in that field.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wBV2Ez7yXZ", "reviewer": "Reviewer_rES8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors propose and undertake an interesting and important topic -- missing data values in tabular data. Specifically the angle of the research -- how do missing values impact the GBDT vs deep learning debate -- is important and worthy of study\n- This work is clearly angled towards a specific domain which the authors have deep technical expertise in (or so it seems from reading the work, particularly Section 2). The authors communicate the various aspects of the car insurance marketplace and how the technical interventions are deployed within the larger car insurance pipeline. This is a very nice aspect of this paper.\n- The paper is generally well-wrtitten and generally describes the work the authors did in an appropriate level of depth with nice clarity. \n\nWeaknesses:\n- The main weakness is the limited application of this work just to car insurance. Other work that looks at the GBDT vs DL debate in tabular data, like McElfresh et al., does so across a wide range of datasets and topic areas. This is a major weakness for the generalizability of the findings the authors gave. Particularly in light of the main conclusion from other work (like McElfresh) that the superiority of a given model greatly depends on the dataset. \n- The dataset the authors use is not public. Too little information is given about the dataset. For example, we don't even know how many rows there are, what the variables are, temporal or geographic components, how concentrated the missing values are per column, etc. \n- Could the authors provide more evidence to this claim: \"Preliminary analysis suggested the best hyperparameters did not vary with preprocessing scheme.\" Like what experiments were run and what the results were?\n- Grid search is the most basic and least effective algorithm for hyperparameters tuning. The authors should consider using optuna to perform at least a random search through their search space. They could perform Bayesian methods as well. \n- \"Although there is research on generating architectures following an algorithm these approaches are computationally expensive and give only slight performance improvements (Ren et al., 2021). Since these methods are seldom used in practice, the depth and width of our ResNet MLP was selected heuristically.\" I do not believe the literature supports this claim. Neural Architecture Search has been very productive in a wide range of applications [1,2,3,4], including tabular data. Can the authors provide more details for why they didn't perform any NAS or NAS+HPO experiments? \n\n[1] https://openaccess.thecvf.com/content_ICCV_2019/html/Howard_Searching_for_MobileNetV3_ICCV_2019_paper.html\n\n[2] https://proceedings.neurips.cc/paper_files/paper/2023/file/eb3c42ddfa16d8421fdba13528107cc1-Paper-Conference.pdf\n\n[3] https://arxiv.org/abs/2006.02049\n\n[4] https://arxiv.org/abs/2310.12145", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the ongoing debate between deep learning (DL) models and gradient boosted decision trees (GBDTs) when it comes to handling tabular data. Focusing on a claim reserving application from a large UK insurer, the authors compare three models: TabNet, a specialized DL architecture; a simple neural network inspired by ResNet; and Catboost, a popular GBDT model. They aim to illuminate how missing value handling can significantly impact model accuracy—a topic often overlooked in previous research.\n\nThe findings reveal that with some imputation methods, the optimized simple neural network performed comparably to Catboost. However, other imputation methods revealed Catboost outperformed both DL models, achieving the best accuracy overall. This underscores the critical role that missing value handling plays in model performance. Ultimately, the study suggests that before jumping to conclusions about which algorithm reigns supreme for tabular data, we must first address how missing values are managed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggested edits which would greatly impact my evaluation of the paper:\n- Can the authors consider expanding this work to other datasets as well? I understand the benefit of the data that you currently use may be that it is commercially viable (used by a company in real world setting), but one thing you could do is to replicate the TabZilla project (or part of it) with removing some data. \n- Can the authors incorporate some more standard HPO (and ideally) NAS+HPO methods (like from the [optuna](https://optuna.org)) framework? Using more rigorous and widely-accepted methods of hyperparameter searches would improve the results of the paper significantly. \n\nSuggested edits which would improve the view of the paper:\n- As noted in the weaknesses above, the authors could improve the writing and description of some of the conclusions of the work. In addition to those previously listed, the authors should consider making the tables and figures more helpful for the reader. For example, Figure 2 takes up an entire page, but it does not provide much description besides what is already described in the work and I do not feel advances the reader's understanding of the work. A figure about the hyperparemetters and tuning procedure however would be much more helpful. Additionally, the paper's structure currently reads with the first 6 pages being mostly background and prep material. I'd suggest compressing those sections and building out the last 3 sections with more details and experiments (as suggested above).\n\nVery Minor Note:\n- \"The process of determining price for a prospective customer\" -> \"The process of determining the/a price for a prospective customer\"\n- Missing ref: \"Finally, Section ?? touches on training speeds as a practical consideration.\"", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9XE86oCLVA", "reviewer": "Reviewer_PdDC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n\nThe authors use a new (large) dataset, which has not been used in previous publications (if I understand correctly). As such, it is useful to check the results of previous comparisons, and provide additional information.\n\nStudying the impact of imputation strategy on different models (neural networks, GBDTS..) is underexplored in current comparisons and an interesting avenue of research.\n\nThe paper is well written. In particular, the descriptions of the model provide good short summaries.\n\n# Weaknesses\n\nUsing only one dataset makes it hard to generalize the conclusion of the paper, in particular the discussion on imputation strategy.\n\nThe number of models compared is quite limited. In particular, TabNet is known to perform very poorly in general (as found and noted in the current paper), and the previous paper on insurance severity finding good performance for TabNet is very poorly done. This is not an issue, but it means that only 2 interesting models are compared. Other GBDTS (XGBoost, LightGBM...), and other neural networks architecture (TabR[1], FT-Transformer[2], numerical embeddings[3], RealMLP[4]...) could be investigated. That being said, we find ResNet to be a good choice if a single neural networks is chosen, as, as noted in the current paper, it was the best performing neural network in McElfresh et al.\n\nContinuing on the last point, the issue is that the authors do not follow the guidelines of McElfresh et al. (which probably contribute to the good performance of ResNet in their paper, as ResNet do not performance so well in other comparisons, e.g Grinsztajn et al.) for their ResNet model. In particular, they do not use the same hyperparameter space nor HPO strategy, and do not use the same imputation strategy. I am not sure whether the model implementation is taken from McElfresh et al. I understand that it's likely possible to do better than McElfresh et al. when focusing on 1 dataset, but it would be interesting to compare to the standard strategy of McElfresh et al., to check that the authors' results are indeed better.\n\nThe imputation strategy comparison is quite unfair to the neural networks, as the only imputation strategy tried is \"less-than-minimum\" imputation, CatBoost default strategy. As such, it is perhaps less surprising that CatBoost benefits the most from this strategy. Other imputation strategies could be easier to use for neural networks, such as mean imputation, used in McElfresh et al. Adding missingness masks could also be interesting, and perhaps more useful for neural networks.\n\nThe HPO strategy is perhaps suboptimal, as grid search is often a bad choice compared to random search on a larger set of hyperparameters. Again, I can imagine that the authors have used more manual tuning to restrict the hp space on their one dataset, and found good hp, but it is not clear if the manual effort is fair when comparing different models. For this reason, it would be interesting to compare the performance with more standard hp space and random search or other hpo algorithm.\n\nI wonder if the results would be easier to interpret for \"drop\" if only rows were removed, and sample sizes were equalized (it won't be comparable to other strategies in any cases, but it might be easier to compartmentalise the performance in \"drop\" + \"binarize\").\n\n[1]: Gorishniy, Yury, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. 2023. “TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023.” arXiv:2307.14338.\n\n[2]: Gorishniy, Yury, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2023. “Revisiting Deep Learning Models for Tabular Data.” arXiv. https://doi.org/10.48550/arXiv.2106.11959.\n\n[3]: Gorishniy, Yury, Ivan Rubachev, and Artem Babenko. 2023. “On Embeddings for Numerical Features in Tabular Deep Learning.” arXiv. https://doi.org/10.48550/arXiv.2203.05556.\n\n[4]: David Holzmüller, Léo Grinsztajn, Ingo Steinwart. 2024. \"Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data.\" arXiv. https://arxiv.org/abs/2407.04491", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the problem of claim reserving in car insurance, i.e estimating the cost of insurance claims already reported but not yet settled. In particular, the authors study the performance of neural networks (here a Resnet and TabNet, compared to Gradient Boosting Decision trees (here, CatBoost) with light hyperparameter tuning, on a new (not used in previous publications as far as I understand) commercial dataset of car insurance.\n\nThe paper report very poor performance of TabNet on this dataset, and quite similar performance for ResNet and CatBoost when dropping missing value, or predicting using only the missing data structure. However, using a different missing value encoding (\"less-than-minimum\" encoding) makes CatBoost much better than the ResNet model (though it helps both models), highlighting the importance of missing value imputation strategies when comparing different models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical requested changes\n\nTry different imputation strategies, in particular mean imputation and missingness masks.\n\nShow the performance of a standard pipeline on this dataset for reference. For instance, take the hp spaces from McElfresh et al. for CatBoost and ResNet and do light hpo.\n\nDetails validation strategy with respect to time. Currently it is not clear if the validation sets are also taken in the future.\n\n# Non-critical change\n\nEvaluate more models.\n\nMaybe only drop rows in the \"drop\" strategy.\n\nThe current explanation of the \"binarize\" strategy is a bit hard to understand.\n\n> Finally, Section ?? touches on training speeds\nas a practical consideration.\n\nbroken reference", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the reviewers lean toward acceptance while one leans toward rejection. The reviewers favoring acceptance report that the concerns described above are largely addressed. After reading the reviews, rebuttals, and part of the already revised paper myself, I agree with the two positive reviewers. That said, I have two requests that I ask the authors to at least seriously consider for the final version:
1. In the abstract and introduction, I think the authors could do a better job of justifying their focus on the single car insurance dataset, particularly because of the prevalence and not MCAR nature of its missing values as mentioned earlier. I think the justification is stronger in the conclusion (Section 6), and this messaging could be brought forward as well. More pointers could also be given to the Tabzilla experiment in the appendix to show that the problem appears less severe when missingness is MCAR.
1. Reviewer PdDC suggested "missingness masks" as an additional imputation method. I clarified with the reviewer that this means adding, for each feature with missing values, another binary-valued feature that indicates where the missing values are. As I understand it, this would be a combination of the "Binarize" method with one of the imputation methods. I think the addition of such a combination would further strengthen the work.
Decision: Accept (poster)