Function Basis Encoding of Numerical Features in Factorization Machines
Authors: Alex Shtoff
Elie Abboud
Rotem Stram
Oren Somekh
Venue: Accepted by TMLR ()

Abstract: Factorization machine (FM) variants are widely used for large scale real-time content recommendation systems, since they offer an excellent balance between model accuracy and low computational costs for training and inference. These systems are trained on tabular data with both numerical and categorical columns. Incorporating numerical columns poses a challenge, and they are typically incorporated using a scalar transformation or binning, which can be either learned or arbitrarily chosen. In this work, we provide a systematic and theoretically-justified way to incorporate numerical features into FM variants by encoding them into a vector of function values for a set of functions of one's choice.
 
We view FMs as approximators of segmentized functions, namely, functions from a field's value to the real numbers, assuming the remaining fields are assigned some given constants, which we refer to as the segment. From this perspective, we show that our technique yields a model that learns segmentized functions of the numerical feature spanned by the set of functions of one's choice, namely, the spanning coefficients vary between segments. Hence, to improve model accuracy we advocate the use of functions known to have powerful approximation capabilities, and offer the B-Spline basis due to its well-known approximation power, widespread availability in software libraries and its efficiency in terms of computational resources and memory usage. Our technique preserves fast training and inference, and requires only a small modification of the computational graph of an FM model. Therefore, incorporating it into an existing system to improve its performance is easy. Finally, we back our claims with a set of experiments that include a synthetic experiment, performance evaluation on several data-sets, and an A/B test on a real online advertising system which shows improved performance. We have made the code to reproduce the experiments available at https://github.com/alexshtf/cont_features_paper.

Published in Transactions on Machine Learning Research (12/2024)
Function Basis Encoding of Numerical Features in Factoriza- tion Machines
Alex Shtoff
olex. Blen.shtf@gmail com
Yahoo
Elie Abboud
clabbooud:0000@@gmaaaaa
Department of Computer Science University of Haija
Rotem Stram
motemstram@gmmailccmm
Yahoo
Oren Somekh Yahoo
orensomekh www.ommekk@@yhhhocccoo
Reviewed on OpenReview:
https: openreview. net/ forum? id- M4222IBHihh
Abstract
Factorization machine (FM) variants widely used for large scale real-time content recommendation systems, since they offer excellent balance between model accuracy and low computational costs for training and inference. These systems are trained on tabular data with both numerical and categorical columns. Incorporating numerical columns poses a challenge, and they are typically incorporated using scalar transformation or binning, which can be either learned or arbitrarily chosen. this work, we provide a systematic and pnooreticallyjjusiifie. way incorporate numerical features into FM variants by encoding them into vector of function values for set of functions of one's choice.
We view FMs approximators of segmentized functions, namely, functions from field's value to the real mumbers, assuming the remaining ficlds assigned some given constants, which we refer to as the segment. From this perspective, show that our technique yields a model that learns segmentized functions of the numerical feature spanned by the set of functions of onc's choice, namely, the spanning coefficients vary between segments. Hence, to improve model accuracy we advocate the use functions known to have powerful well-known approximation
approximation capabilities, and offer the B-Spline basis due power, widespread availability in software libraries and its efficiency terms of computational resources and memory usage. Our technique preserves fast training and inference, and requires only small modification of the computational graph of FM model. Therefore, incorporating it into an existing system to improve its performance is easy. Finally, we back our claims with a set of experiments that include a synthetic experiment, performance evaluation on several data-sets, and an A/B test on a real online advertising system which shows improved performance. We have made the code to reproduce the experiments available at ltttts://ggthhee www.aa/westtf//ooo//ooooo/ooeooa features__appett _ppperrppppp
1 Introduction
Traditionally, online content recommendation systems rely on predictive models to choose a set of items to display by predicting the affinity of the user towards a set of candidate items. These models are usually trained on feedback gathered from log of interactions between users and items from the recent past. For systems such as online ad recommenders with billions of daily interactions, speed is crucial. The training

---

Published in Transactions on Machine Learning Research (12/2024)
process must be fast to keep up with changing user preferences and quickly deploy fresh model. Similarly, model inference, which amounts to computing score for each item, must be rapid to select few items to display out of a vast pool of candidate items, all within a few milliseconds. Factorization machine (FM) variants, such as Rendle (2010); Juan et al. ) Pan et al. (2018): Sun et al. (2021). are widely used in these systems due to their ability to train incrementally, and strike a good balance between being able to produce accurate predictions, while facilitating fast training and inference.
The training data consists of past interactions between users and items, and is typically given in tabular form, where the table's columns, fields, have either categorical or numerical features. For example, "gender" "time since last visit" are fields, whereas "Male" and "10 hours" are corresponding features. In recommendation systems that rely on FM variants, each row in the table is typically encoded as a concatenation of field encoding vectors. Categorical fields are usually one-hot encoded, whereas numerical fields are conventionally binned to a finite set of intervals form a categorical field, and onc-hot encoding subsequently applied. A large number of works are devoted the choice of intorvals, e.g. Dougherty et al.) (1995); Peng et al. (2009); Liu et al. ( Gama & Pinto (2006). Regardless of the choice, the model's output slep function of the value of given numerical field, assuming the remaining fields are kept constant, since the same interval is chosen independently of where the value falls in given interval. For example, consider a model training on a data-set with "age", "device type", and "time the user spent on our site". For the segment of 25-years old users using an iPhone the model will learn some step function for different spending time values, whereas for the segment of 37-years old users using a laptop the model may learn (possibly) different step function.
However, the optimal segmentized functions the model aims to learn, which describe the user behavior, aren't necessarily step functions. Typically, such functions continuous or even smooth, and there is gap between the approximating step functions the model learns, and the optimal ones. In theory, potential solution is simply to increase the number of bins. This increases the approximation power of step functions, and given an infinite amount of data, would indeed help. However, the data is finite, and this can lead to a sporsity issuc- as the number of learning samples assigned to each bin diminishos, it becomes increasingly challenging to learn a good representation of each bin, even with large data-sets, especially because we need represent all segments simultaneously. This situation can lead to degradation in the model's performance despite having increased the theoretical approximation power of the model. Therefore, there is limit to the accuracy we can achieve with binning given data-set demonstrated in Figure
Figure 1: For a given segment, learned segmentized step function approximations (orange) of a true function (blue) which was used to generate a synthetic data-sel. On the left. too few bins, "bad" approximation. In the middle a balanced mumber of bins, "moderate" approximation. On the right many bins, but approximation gets even "worse" due a sparsity issuc. Refer to Figure for additional synthetic evaluation results with loss measurements.
In this work, we propose a technique to improve the accuracy of FM variants by reducing the approximation gap without sparsity issues, while preserving training and inference speeds. In other words, on the approximation- estimation balance, we aim reduce both approximation and estimation error. Our technique composod of encoding numerical field using vector of basis functions, and a minor modification to the computational graph of an FM variant. The idea of using nonlinear basis functions model nonlinear functions is, of
2

---

Published in Transactions on Machine Learning Research (12/2024)
course standard practice with linear models. This work is about analyzing the interplay between nonlinear basis functions and the FM family, to yield a surprisingly powerful technique for recommender systems, and tabular datasets in general.
Indeed, we present an elementary Lemma showing that the resulting model learns a segmentized output function spanned by the chosen basis, meaning that spanning coefficients depend on the values of the remaining fields. This is, of course, an essential property for recommendation systems, since indeed users with different demographic or contextual properties may behave differently. For pair of fields, the segmentized functions are spanned by a tensor product of the chosen pair of bases, where the coefficients are learned in factorized form by the model.
Based on the generic observation above, we offer the B-Spline basis (de Boor 2001 pp 87) defined on the unit interval on uniformly spaced preak-points, composed onto a transformation that maps the feature to the unit interval. The number of break-points (a.k.a knots) is a system hyper-paramcter which can be further optimized. The strong approximation power of splines (de Boor 2001, pp 149) ensures that we do not need large number of break-points. Hence, we can closely approximate the optimal segmentized functions, without introducing sparsity issues. Moreover, to make integration of our idea easier in practical production- recommendation system, we present a technique to scamlessly integrate a model trained using our proposed scheme into an existing recommendation system that employs binning, albeit with a controllable reduction in accuracy. Although a significant part of this work considers the B- B-Spline basis, the techniques we present can be used with an arbitrary basis.
Our work builds on the recent work of Rügamer (022)., ; that uses a similar technique in conjunction with FMs. Our work extends these results to wider family of FM variants, and utilizes it from the perspective of recommender systems and a broaded practical scope: application to unbounded domains and interaction with categorical features, deployment in an existing recommender system, and A/B test results on a real-world online advertising product.
To summarize, the main contributions of this work are: (a) Basis encoding We propose roduciggg a slight, modification to the computational graph of FM variants that facilitates encoding of numerical features as vector of basis functions; (b) Spanning properties We show that our modification makes any model from family of FM variants learn segmentized functions spanned by pairwise tensor products of the basis of our choice, and inherits the approximation power of that basis; (c) B-Spline basis We justify the use of the B-Spline basis from both theoretical and practical aspects, and demonstrate their benefits via numerical ovaluation; (d) Ease of integration into an cxisting system We show how to integrate a model trained according to our method into an existing recommender system which currently employs numerical feature binning, to significantly reduce integration costs; (c) Simplified numerical feature engineering the strong approximation power of the cubic B-Spline basis allows building accurate models without investing time and effort to manually tune bin boundaries, and use simple uniform break-points instead.
1.1 Related work
Putting aside the FM variants, there is a large body of work dealing with neural networks training over tabular data (Arik & Pfister 2021:
Badirli et al. 2020 Gorishniy et al. 2021 Huang et al. 2020: Popoy et al. 2020: Somepalli et al., 2022 Song et al. 2019 Hollmann et al. 2022). Neural networks have the potential to achieve high accuracy and can be incrementally trained on newly arriving data using transfer learning techniques. iddalion,,l,, due to the universal opprozimation theorem (Hornik et al. 1989), neural networks are capable of representing a segmentized function from any numerical feature to a real number. However, the time required to train and make inferences using neural networks is significantly greater than that required for factorization machines. Even though some work has been done to alleviate this gap for neural networks by using various embedding techniques (Gorishniy et al. 2022), they have not been able to outperform other model types. As a result, in various practical applications, FMs are preferred over NNs. For this reason, in this work we focus on FMs, and specifically on decreasing the gap between the representation power of FMs and NNs without introducing a significant computational and conceptual complexity.
A very simple but related approach to ours was presented in Covington et al. 2016).. The work uses neural networks and represents a numerical value z as the triplet. (z, 2², v2) in the input layer, which can be seen
3

---

Published in Transactions on Machine Learning Research (12/2024)
variant of our approach using basis of three functions. Another approach which bears similarity to ours also comes from works which use deep learning, such as Cheng (2022); Gorishniy et al. (2021) Song et al. (2019); Guo et al. [202]). In these works, first-order splines are used in the input layer to represent contimuity, and the representation power of neural network compensates for the weak approximation power
of first-order splines. Here we do the opposite we use the stronger approximation power of cubic splines to compensate for the weaker representation power of FMs.
Finally, any comprehensive discussion on tabular data would be incomplete without mentioning gradient boosted decision trees (GBDT) (Chen & Guestrin 2016 Ke et al. 2017 Prokhorenkova et al. 2018), which are known to achieve stato-of-the-art results (Gorishniy et al., 2021 Shwartz-Ziv & Armon (022). However, GBDT models aren't nameds, it challenging and costly to use GBDT models to rank hundreds of thousands of items in a matter of milliseconds.
t. useful in a variety of practical applications, primarily due to siguificantly slower inference
2 Background and problem statement
Factorization machines are formally described models whose input is feature vector representing rows in tabular data-sets. In the context of recommendation systems, the data-set contains past interactions between users and items, whose columns, often named fields, and whose values are features. The columns describe the context, such as the user's gender and age, the time of visit, or the article the user is currently reading, whereas others describe the item, such as product category, or item popularity.
In this section we formally describe how we assume that the data is modeled as the feature vector æ, describe the FM family we shall focus in this paper, and formally state the main problem we aim to solve.
2.1 Feature vector structure
Consider a tabular data-sets with fields (1....; m), each be either numerical categorical field. We denote a row in the dataset with (zi,..., zm), each 2j is feature (value) associated with its corresponding field [(i). Each feature z is mapped to vector of values by applying its corresponding encoding function enc;(-):
enc; (21)
x = enc ( mm)= .m))
encm (zm)
For example, consider three fields: field the user's country, field 2 is the user's device type, and field 3 is the product category. All three are categorical. Then enc1(.) one-hot encodes the country, enc2(-) one-hot encodes the device, and enca(-) one-hot encodes the product category. As a result, in this concrete example, æ is the concatenation of three one-hot encodings.
2.2 Segmentized view of models
In this work we aim to study models learned on tabular data as a function of one or two tabular columns, while the remaining are kept constant. Therefore, we need some compact notation for the concept of vicwing a multivariate function as a function of one or two variables, while keeping the rest constant.
For a given vector y we shall denote by y-k the vector obtained from all its components except for yk- For a function (yy), we shall denote by k the function obtained by assigning the values y- to all its arguments, except for yk. producing function of yı. Formally, we have
sog( Φ; k) + (Bx) = (b)).
This "partial application" concept not new. The mathematical term is currying, coined by Frege 1893 However, we shall use the term segmenized view of for a reason that will become apparent immediately. Similarly to the exclusion of single coordinate k. for pair of coordinates k.j we define y-(k.) and
segl p k, (. (k.))(K))))) (t.](V((k))) (MK) v3) +(v).
4

---

Published in Transactions on Machine Learning Research (12/2024)
Namely, seg[#; (4. assigns values to all inputs of Φ, except for y and yj.
A machine learned model (()) is a function of the encoded feature vector a, but we are interested in studying it as function of the original tabular columns. Thus, our aim is studying the composition
T gew enc.
Given field the function seg|$one,,ff f((=))) is function of z,, given concrete assignment 2-f to the remaining fields. Recalling our three-field example of country, device lype, and product category, the function seg[$enc " "Furniture") characterizes the model as function of the device type for users from the US that interact with furniture products. From the recommender systems perspective, we are characterizing the model for a given segment of users and items, hence the name segmentized view, and the notation seg.
2.3 The factorization machine family
In this work we consider several model variants, which we refer to as the factorization machine family. The family includes the celebrated factorization machine (FM) (Rendle 2010). the field- aware factorization machine (FFM) (Juan et al., 2016), the field-weighted factorization machine (FwFM) (Pan et al. 2018), and the felddmmatrized factorization machine (FmFM) (Sun 2021 Pande 2021), that generalized the former
variants. FMs typically employed for supervised learning tasks dataset. {(rr,, yt) with either binary real-valued labels Vt-
The classical Factorization Machines (FMs), as proposed by Rendle 2010 compute
(F) Wo + (ax, w)
( 2,
The learned parameters are wp E R, w E R", and Ui...., Un E R where is a hyper-parameter. The model can be thought of as way represent the quadratic interaction model
(x) wi + (x, w) +
Σ Σ A.jz,z,
where the coefficient matrix A is represented in factorized form. The vectors Ui...., Un are the feature embedding vectors. Classical matrix factorization recovered when we have only user id and item id fields, whose values are one-hot encoded.
The model dFM does not represent the varying behavior of a feature belonging to some field when interacting with features from different fields. For example, genders may interact with ages differently than they interact with product categories. Initially, to explicitly encode this information into a model, the Field-Aware Factorization Machine (FFM) was proposed by Juan et al. (2016). Each embedding voctor v; is modcled as a concalenation of field-specific embedding vectors for each of the m fields:
Ui
Ui.m_
where Ve,f the embedding vector of feature i when interacting with another feature from field f. As a convention, we denote by f(i) the field whose value is used to encode the 7th component of the feature vector The model computes
(x) = w0 + (a, w)
As pointed out by Pan al. 2018); Juan et al. (2016) the FFM models are prone to over-fitting, since it learns feature embedding vector for each feature x field pair. As remedy, Pan et al. (0188) proposed the
5

---

Published in Transactions on Machine Learning Research (12/2024)
Field- Seidh-eeghted Factorization Machine (FwFM) that models the varying behavior of field interaction using learned scalar field interaction intensities re,f for each pair of fields e, The FwFM computes
(Fw = wa (ar, w) +
(10,)))
A generalization of all the models above has been independently proposed in Sun et al. 2021 Pande 2001
To describe it, let (x, y) p = x² Py denote the "inner product" associated with some matrix P. The FmF model computes
L FFFFM = wo + (z, w) +
(ay 2 23
(1)
+=
where Wp E IR w E R", and v, R are learned parameters. The field-interaction matrices M)) e R (k56) can be either learned or predefined, and may have a special structure. Note that this model family allows a different embedding dimension for each field, since the matrices M do not have to be square.
Let us explain why all the models described here are special cases of FmFMs in equation I terms of their representation power: To recover the classical FMs, take equation ] with MG).(() I. To recover FFMs, let Pj be the matrix that extracts Vi,f from vi, namely, Pyv; vi Then FFMs are also special case of the FmFM model in equation I with Me. PJ Pe. Finally, to recover FwFMs we just need to take equation with M&j = re. Given this property, we use PPMFM in equation [ to describe and prove propertics which should hold for the entire family.
2.4 Linearity of FmFMs and binning
A fundamental property of the FmFM family is that PFMFM is offine in any one coordinate of æ. Formally, FmgM k)(x=)) is an affine function of zg. The explanation stems from separating the summands with 2 and those without my in the FFMFM formula:
(PFFM( = wp (0- w w-i)
Σ
(110) tok
Wi: +
Σ
(Uu, (yb)) Uy M
15izisn
15/5n JFk
157K
= 0c + B3%.
Consequently, an inherent limitation of the FmFM family that it can represent only affine relationships between any one given input and the prediction. Thus, if the encoding function of numerical field f is lincar scaling, such as min-max scaling or standardization, then FMFM 4FmFM o enc will only be able represent affine functions of that field. In most cases in practice, such an affine relationship is not expressive enough.
To overcome this limitation, numerical columns typically undergo binning quantization: the numerical range is partitioned into finite set of disjoint intervals, and ency(-) one-hot encodes the interval its argument belongs to. Binning a numerical field f. this results in seg ( jumps are at the bimning boundaries. This is because the encoding of ž anywhere inside a given interval between two boundaries is constant, and thus the model's output is constant for any 2 in that interval. Similarly, for two numerical fields e < f the segmentized views seg| P e, f((=z(e).)) are piecewise-constant functions of 2y); where the constant-value regions are grid cells formed by the bin boundaries of ficlds and f.
being a step function of 2 whose
FMFMM do require be product. For be true product, the matrix has be square and positive definite

---

Published in Transactions on Machine Learning Research (12/2024)
2.5 Problem statement
When learning a model, our aim for FmFM = 4PmFM o ene to approximate some optimal predictor 4* as a function of the original tabular fields. Thus, may think of segmentized views of dene FMFM as approximations
of the segmentized views of Φ. Bimning, therefore, results in piecewise-constant approximations. Binning with very coarse grid may result in a poor approximation. Conversely, a grid that is too fine might significantly reduce the amount of training data falling into cach interval, resulting in over-fitting. This kind of over-fitting is also known as the data sparsity issue. However, it is nothing more than the classical caapeiiimattio-ssimtttttt balance.
With respect to this problem, our work has two aims thcoretical and practical. Theorctically, we aim to show that the interplay between slightly modified FmFFM model family and the classical well-known encoding schemes for numerical features, such as polynomials and splines, yields a powerful family of models: the segmentized views for each field are spanned by the chosen basis, whereas the views for each pair of fields are spanned by the basis tensor products, with the span coefficients depending on the remaining field. Consequently, the model learns a potentially different segmentized view for each segment, while the segmentized views inherit the approximation power of the chosen basis.
Second, we aim utilize this theoretical insight to show how to improve an existing recommender system driven by models from the FmFM family trained with binned numerical features. Concretely, show how balance by avoiding sparsity for various kinds of
this insight allows improving the approximation--stinneee numerical fields encountered in practice, and propose deployment scheme for our modified models that requires only minor changes to an existing serving part of a recommender system with only minor changes.
3
The basis function encoding approach
To describe our approach we need to introduce a slight modification to the FmFM computational graph formulated in equation [
We can re-write the formula by extracting 2;U; and 2,w; into auxiliary variables:
Tj 11
U
Fn Un
x W1
y =
F'y Wn
PPMFM W0
Ut
U U Ma)M
Here, U.. is a "Python" notation for denoting the i-th row of U. In a sense, the first two lines extract the relevant embedding vectors and linear coefficients, and the last line combines them to produce a score.
Our modification is applied to the first two 'extraction' lines. Before combining the vectors and the lincar coefficients, first apply field-wise reduction to the embedding vectors and linear coefficients of each (a) the identity reduction, which just returns its input; and (b) sum reduction, which sums up the embedding vectors belonging to the field: Σiegy E;U; and, respectively, EvEIT) 2,W4, where I(f) are the indices in ø containing the encoding of field f.
field. We consider two kinds of reductions:
The identity reduction is equivalent to multiplying by the identity matrix on the left, whereas the summation reduction is equivalent to multiplying by a single-row matrix of ones 1/ of the appropriate size f. Thus, each field f has an associated reduction matrix Ry, and the resulting model obtained by pre- pre-mittillyng the original U and y by a block-diagonal reduction matrix that applics the appropriate reduction the embodding
TThe serving part of recommender system is the part that consumes trained models, and uses them recommend items.
7

---

Published in Transactions on Machine Learning Research (12/2024)
V
XC
0
0
Field
ld
X
0
X
FmFM inner products
Field
1
ld
0
(2())
Field 3
B
+
B (X)
B
Figure 2: The computational graph with inntinuous numerical fields. Field 3 is continuous numerical field whose value is 2. The vectors vi...., U2 in the rows of are multiplied by the input vector æ. Then, a reduction is applied to each field. Most fields have the identity (Id) reduction, whose output is identical to its input, whereas field 3 uses the sum reduction. The resulting vectors are then passed to the pairwise interaction module. An analogous process happens with the w vector.
vectors linear weights belonging to each field:
R1
I1 U1
U
Pm
Un
R
2 W1
(2)
y
Pm
Dn Wn.
6
T PmFM = W0
Uk
Un... U M
For every field f, the matrix Ry is either the identity matrix, or a row of ones. Note, that after the reductions, the matrix U and the vector y may have different number of rows, denoted by in the formula above.
As slight abuse of notation, we shall now refer to this slightly modified model in equation Mas "the" FmFM. This is because an FmFM is obtained by choosing all reductions to be the identity. Next, we shall describe how we incorporate basis encoding into this model.
Suppose we would like to encode a continuous numerical field f. We choose set of functions B!....,B/ BB and encode the field as (=) = (z)..... B,. (z))² For that field, we choose the summing reduction, namely, R is a row of ones. When the field f under consideration if clear from the context, we omit it for clarity, and write Bı,..., Bg. The result is that for this field, there one row U that is equal Every) B2(2),, and one row y that equal EiEry) B2(2)u,, Note that each field can have its own sot of basis functions, and in particular, the size of the bases may differ. The computational graph, including matrix is only mathematical
basis encoding, is depicted in Figure 2 We note that the block-diagone formalism, and a reasonable implementation to compute the score is similar to what is illustrated in the figure: extract vectors corresponding only to the non-zero components of æ associated with each field, perform the reductions for each field, and then compute the pairwise inner products.
As final note, observe that binning is obtained as a special case of our basis encoding method. Indoed, suppose we have intervals that partition our numerical range, choosing the basis B Be be just the indicator functions of these intervals yields model gene fae that is equivalent binning.
8

---

Published in Transactions on Machine Learning Research (12/2024)
3.1 Spanning properties
To show why modeling choices improve the approximation power of the model, present two technical lemmas showing that the segmentized functions for one or two basis-encoded tabular columns are spanned by the basis of choice, or the tensor product of the two chosen bases.
Lemma 1 (Spanning property). Let
ency(=) = (B)()), BB(2))²
be the encoding associated with continuous numerical field f, let R2 = 1 and suppose FFMFM is computed according to equation Then, for any z-f there erist 01,...,at. at, R, which depend only on z-f, such thot:
seg
Deee Mm f((=-)) Σ (()
An elementary formal proof can be found in Appendix A.1 but it can be explained intuitively. Looking at equation 2 both the matrix U and the vector y have one row that is linear in the basis .I,..., Be, whereas the other rows are constant. Since FFMF is affine in any one input feature, must be an affine function of the basis.
It becomes even more interesting when looking at segmentized functions for two continuous numerical fields. Lemma 2 (Pairwise spanning property). Let e < I be two continuous numerical fields. Let
enc,(ze) (B.( ... (..)))))..BB(()))))) enc,(zz) = (Ci(2)), Cn(zz))
let R. 17 and R, 12, and suppose PFMFM is computed according to equation 2 Define Bo(z) = Co(2) 1. Then, for every () there erist azj,8 E R for i {0....,() and j {0, ......... R), such that:
one FmFM , e, (o.
an (,(2) + t=U j=
The proof can be found in Appendix A.2 The intuition is similar in nature to the one we have for the spanning property for one field, but now two instcad of one rows of y and U are affine in the bases.
henc It's important to note that the segmentized view seg[ PmPP: e, f](=-(e.)) has O x), but the model does not actually learn Ole-w x) parameters. Instead, it learns O(e + x) parameters in the form of & + embedding vectors. Consequently, the spanning coefficient matrix explicitly, but rather its low-rank factorization is learned in the form of embedding vectors. This extends the observation made in Rügamer (2022) for classical FMs to the broader FmFFM family.
(a for each segment is not learned
3.2 Splines and the B-Spline basis
Spline functions [Schoenberg] 1946) are piece-wise polynomial functions of degree with up to d- continuous derivatives defined on some interval (a, b). The interval divided into disjoint sub-intervals at a set of break-points a to t <... < te-a = b, where each polynomial piece is defined on [ty, (y.) It is well-known that spline functions of degree d defined on L d-d + break-points can be written as weighted sums of the celebrated B-Spline basis (de Boor 2001 pp 87) comprising of exactly É functions. For brevity, we will not elaborate their explicit formula in this paper, and point out that it's available in variety of standard scientific computing packages, e.g., the scipy. interpolate. BSplinn class of the SciPy package [Virtanen et al. 2020).
In this paper we concentrate on the cases of d 2 and d 3, which are quadratic and cubic splines, respectively, with uniformly spaced break-points. At this stage we assume that the values of our numerical field lie in compact interval (a, b), which we assume w.l.o.g is [0, 1). We discuss the more generic cases in the

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
1.0
100
1.0
0.0
0.0
0.0
3/5
0.5
0.0
0.0
6.0
3/5
Figure Cubic B-Spline basis functions defined on five break-points.
following sub-section. A visual depiction of the cubic B-Spline basis with five break-points is illustrated in Figure 3
The B-Spline basis is our proposed candidate for recommender systems for two reasons: computational efficiency, and approximation power. For efficiency, an important property of the B-Spline basis of degree d is that at any point only + d basis functions are non-zero. Thus, regardless the number of basis functions we use, computing the model's 's output remains efficient, since the reduction described in Figure requires a weighted sum of only + vectors, regardless of the size of the basis.
For the approximation power, is known dde Boor 2001 pp 149), that splines of degree d can approximate arbitrary function y with l S + d continuous derivatives up to error bounded by (1) where (8) is the htth derivative of g. The spanning property (Temma 1 ensures that the model's segmentized outputs are splines spanned by the same basis, and therefore are more powerful in approxinating the optimal segmentizze outputs than step functions. Assuming that the functions we aim to approximate are smooth enough and vary "slowly", in the sense that their high-order 1tth derivatives are small, the approximation error goes down the rate of (+),, whereas with bimning the rate is O
A direct consequence is that we obtain theoretically good approximation which is also achievable in practice, since we can be accurate with small number of basis functions, and this significantly decreases the chances of sparsity and over-fitting issues. This is in contrast to binning, where high-rosolution binning is required to for a good theoretical approximation accuracy, but it may not be achievable in practice.
3.3 Continuous numerical fields with arbitrary domain and distribution
Splines approximate functions on a compact interval. Thus, numerical fields with unbounded domains pose a challenge. Moreover, the support of each B-Spline function is only a sub-interval of the domain defined by consccutive knots. Thus, even if numerical field f is bounded in [a, b), highly skewed distribution may cause "starvation" of the support of some basis functions: if P(zy E support(B:)) is extremely small, there will be little training data to effectively learn useful representation of B:
Here we do not invent any wheels, and suggest using common machine-learning practice: a scalar transform + R [0, 1] to the features of field f, such as min-max scaling, or quantile transform We note that standardization less appropriate, since standardized values do not necessarily lie a bounded interval, as required by the B-Spline basis.
The quantile transform idea is useful for conducting benchmarks, but in a real-time recommender system it may be prohibitive due to its computational complexity. A quantile transform is obtained by learning the empirical CDF of the training data, and using it as the transform Ty. But the empirical CDF is just an approximation of the data generating distribution, and we may use others. Simple parametric distributions have closed-fore form and fast to compute CDF formulae, and we found that in practice it works well: just fit a few parametric distributions the training data of a given field f, and use the best-fit as Ty. We found this to work well practice, shown in Section 4.4
3 $1 is vector of 4 For function defined on S, infinity norm is || max [o(x)
5 Replacing zj with eCDF(=y), where eCDF is the empirical CDP learned from the training data
10

---

Published in Transactions on Machine Learning Research (12/2024)
Over-all, the property we want is for P(2 E support(B:)) non-negligible for every basis function B;, namely, the data is spread on the interval such that no region of the interval is "starved". Thus, even though that in the experiments section we use min-max scaling and quantile transform, for a practical large-scale recommender system we recommend using a parametric distribution.
The above discussion shows that our method does not eliminate the need for data-analysis and feature engineering, but only streamlines it: just try out a few simple options to make sure the data not too skewed when mapped to [0, 1]. The rest of the "heavy lifting" is done by the strong approximation power splines, and the computational efficiency facilitated by the B-Spline basis.
3.4 Integration into an existing system by simulating binning
Suppose would like to obtain model which employs binning of the field into large number N of intervals, c.g. N = 1000. As we discussed, in most cases we camnot directly learn such model because of the approximatiion-stimattin balance. However, we can generate such model from another model trained using our scheme to make initial integration easier.
For a given numerical field f, the reduction stage produces only one row of of the post-reduction matrix U, which we shall denote by u. In fact, this row is function of zy: namely:
The idea is best explained by referring, again, to Figure 2 and equation 2
u u(z)) E (.).(T iEI(F)
Consequently, we have a mapping from 2 to corresponding embedding vector. It a curve in the k-dimensional space, parametrized by 2,. Hence, to simulate N "bins" 21))), simply discretize this curve at N points, such as the mid-point of each interval:
ui 11+1+81)
j j=
The resulting model now has an embedding vector for each bin, as we desire. Hence, the serving system sees a binning-baat based model with many bins, even though it was trained with an arbitrary basis of our choice.
4
Evaluation
We divide this section into three parts. First, we use a synthetically generated data-set show that our theory holds the model learns segmentized output functions that resemble the ground truth. Then, we compare the accuracy obtained with binning versus splines on several data-sets. The code to reproduce these experiments is available in the supplemental material. Finally, we report the results of web-scale A/B test. conducted on major online advertising platform serving real traffic.
4.1 Learning artificially chosen functions
We used synthetic toy click-through rate prediction data-set with four fields, and zero-one labels (click / non-click). Naturally, the cross-entropy loss is used to train models on such tasks. We have three categorical fields each having two values each, and one numerical field in the range [o, 40). For each of the eight segment configurations defined by the categorical fields, we defined functions Po....,Ph (see Figure 4 describing the CTR as a function of the numerical field. Then, we generated a data-set of 25,000 rows, such that for cach row we chose a segment configuration s; E {0, ..., 7) of the categorical fields uniformly random, the value of the numerical field 2 Beta- Pitta-Bioomia(411 0.9, 1.2), and label y4 Bernoulli(p,, (2)).
We trained an FFM (Juan et al. 2016) provided by Yahoo-Inc (2023) using the binary cross-cntropy loss on the above data, both with binning of several resolutions and with splines defined on 6 sub-intervals. The numerical field was naïvely transformed to [o, 1] by simple normalization. We plotted the learned curve for every configuration in Figure g Indood, low-resolution binning approximates poorly, higher resolution approximates better, and too-high resolution cannot be learned because of sparsity. However, Splines defined only six sub-intervals approximate the synthetic functions {pr) quite well.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Segment
begmennt
begment =
begmennt
Segmentt
begmment
begmentt
Segmentt
velue
voluee
0.2
Segment
Sogment
Sogment
Segment
Senm
Segment
Segment
9.2
valse
20 Nam. field
20 Num. field
20 Num. feedd
Num.
Feldd
Feldd
held
field
(a) 5-bin segimentized approximation (Lest loss
0.3474) (b) 1 12-bin segmentized approximation (lest loss 0.34442)
segment
Segment
Segment
Segment
Segment
Sog mant
Sogment
Sogmant
9.
value
vake
Wilnoww
0.1
Segment
Scgment
Segment
Scament
Segment
Seng
Segmeent
Sengment
0.2
0.3
volce
0.1
20 Num. Fiedd
20
40
20
40
20
Num. faeid
Num. faddd
Num. firdd
20 Num.
20
20
40
20
Num.
(d) break-point segmentized cubic spline approximation (0.3432)
(c) 120-bin segmentized approximation (test loss 0.3478)
Figure 4: Results of segmentized approximations of four FFM models trained synthetic data. In each plot, family of segmentized functions on the interval [0, 40). plotted in blue, are approximated by the model, plotted in orange. 12 bins are more accurate than 5, but 120 bins are even less accurate than 5 due to sparsity. With splines we achieve best accuracy.
Next, we compared the test cross-cntropy loss on 75,000 samples generated in the same manner with for several numbers of intervals used for binning and cubic Splines. For each number of intervals we performed 15 experiments to neutralize the effect of random model initialization. As is apparent in Figure 5 Splines consistently outperform this theoretical setting. The test loss obtained by both strategies increases if the number of intervals becomes too large, but the effect is much more significant in the binning solution.
4.2 Public tabular data-sets
Since our approach works on any tabular dataset, and isn't specific to recommender systems, we mainly test our approach versus binning several tabular data-sets with abundant numerical features that have strong predictive power: the California housing (Pace & Barry 1997) adult income (Kohavi 1996). Higgs (Baldi et al. 2014) (we use the 98K version from OpenML. (Vanschoren et al. 2014)), and song year prediction |Bertin-Mahieux et al. 2011). For the first two data-sets we used an FFM, whereas for the last two we used an FM, both provided by Yahoo (Yahoo- Inc] 2023). since FFMs are significantly more expensive to train when there are many columns, and even more so with hyper-parameter tuning. The above data-sets were chosen since they were used in a previous line of work by Gorishniy et al. [2021 2022) on numerical features in tabular data-sets. We chose the subset of these data-sets whose labels are either real-valued or binary, since multi-class multi- classification problems require a model that produce scalar for each class. Factorization machines are limited to only one scalar in their output.
We assume that the lask on all data-sels is regression, both with real-valued and binary labels. This is in line with what factorization machines are commonly used for CTR prediction. For binary labels we use the cross-cntropy loss, whereas for real-valued labels we use the L.2 loss. For tuning the step-size, batch-size, the number of intervals, and the embedding dimension we use Optuna (Akiba et al.) 2019). For binning, we also tuned the choice of uniform quantile bins. Tn addition, 20% of the data held out for validation, and
12

---

Published in Transactions on Machine Learning Research (12/2024)
0.352
0.351
Test loss
0.350
Method
bins
0.349
splines
0.348
8
16
32
64
# boundaries
Figure 5: Comparison of the test cross-entropy loss obtained with Splines and bins. Both methods suffer from sparsity issues as the number of intervals grows, but Splines are able to utilize their approximation power with a small number of intervals, before sparsity takes effect. The bands are 90% bootstrap confidence intervals based on multiple experiment repetitions: for each number of boundaries and numerical field type we ran 15 experiments to neutralize the effect of random initialization.
regression targets were standardized. Finally, for the adult income data-set, has a special meaning for columns, and was treated categorical value.
We ran 20 experiments with the tuned configurations to neutralize the effect of random init ializaziii,n and report the mean and standard deviation of the metrics on the test set in Table | where it is apparent that our approach outperforms binning on these datasets. These datasots were chosen since they contain several numerical fields, and are small enough to run many experiments to neutralize the effect choice and random initialization at a reasonable computational cost, or time. They were also used in other works on tabular data, such Gorishniy et al. (2021 2022
Table 1: Comparison of binning vs. splines. Standard deviations are reported as parentheses as of the mean.
Cal. housing
Adult income
Higgs
Song year
#
20640
48842
98050
515345
# cat. fields
0
8
0
0
# num. fields
8
6
28
88
Metric type
RMSE
Cross-Entropp
Cross- Cross-Entropy
RMSE
Binning metric (std%)
0.4730 (0.36%)
0.2990 (0.47%)
0.5637 (0.22%)
0.9186 (0.4%)
Splines metric (std%)
0.4294 (0.57%)
0.2861 (0.28%)
0.5448 (0.12%)
0.8803 (0.2%)
Splines vs. binning lift
9.2%
4.3%
3.36%
4.16%
4.3 The Criteo dataset
To further demonstrate the benefits of our approach over on real-world recommendation dataset, we evaluate it with an FwFM on the Criteo display advertising challenge dataset (Criteo 2014). It has plenty fields, but these fields integers. Before presenting the experiments, it's important to discuss the unique
of numerical
13

---

Published in Transactions on Machine Learning Research (12/2024)
challenges posed by integers, and especially fields representing counts, e.g. the number of visits of the user in the last week.
We note here that modeling integers is not in the scope of this work, but a paper on recommender systems cannot go without experiments with large scale recommender-system datasets. The Criteo dataset was chosen due to its popularity and the abundance of numerical fields, even though they are integers. We point out that other classical public recommendation datasets, such as MovieLens (GorupLens), or Avazu (Avazu 2014), do not contain continuous or similar integer numerical fields, and therefore were incompatible for the analysis of this work"] Thus, what we propose doing with integers in this section is more of a heuristic adaptation of our method to enable conducting an experiment with the Critco data-sct, rather than systematic treatment of integers that can be seen as an integral part of this paper.
Integers possess properties of both discrete and continuous nature. For example, their CDF function is a step function that may have large jumps, and it's not trivial to transform them to [o, using a CDF approximation in a reasonable manner. This is because large jumps produce large gaps in the [0, 1] interval that are not covered by any data. Moreover, features that represent counts, such as the number of times the user interacted with some category of items, pose an additional difficulty stemming from this hybrid discrote-continuons nature. Smaller values are more 'diccrete', while larger values are more 'continnous', i.e. the difference between users who never visited our site and users who visited it once may be large, the difference between one and two visits will be large as well, but the difference in user behavior between 98 and 99 visits is probably small.
The data-set is a log of 7 days of ad impressions and clicks in chronological order. Thus, we split into training, validation, and test in the following manner: the first 5/7 of the data-sct is the training set, the next 1/7 is the validation set for hyper-parameter luning, and the last 1/7 is the lest set. Categorical features with less than 10 occurrences were replaced by a special "INFREQUENT" value for every field When employing binning, we use a similar strategy to the winners of the Criteo challenge- - the bin index is (oor(In((2) for = 2 1. Namely, the bin boundaries are {exp(v/)) for 2 1. Values smaller than 1 are treated as categorical values. The lower and upper bounds are learned from the training set.
For splines, we stand on the shoulders of giants, and use transform that resembles the squared logarithm x -> arcsinh"(z),, followed min-max scaling. The reason is that arcsinh(z) mimics the logarithm, but is also defined for zero. The min-max scaling ensures that the transformed feature stays in [0, 1], and is learned from the training set only. If the validation or the test set contain values above the maximum observed, they are mapped to 1. Negative values are, similarly, treated as categorical values. We note, that this transform is far from being the empirical CDF, and due to the reasons discussed above, the empirical CDF is typically not applicable to integers. The histograms of the transformed columns I_1, ..., I_13 are plotted in Figure G We can see that some of the columns appear to be quite discrete. For example, I_10 has only four distinct values. The columns I 11 and I 12 appear discrete well. Thus, when conducting experiment with cubic splines, we used them for all columns, except for the above three.
We conduct experiments with k E 8, 16...., 64 as embedding dimensions, and each experiment is conducted using 50 trials of Optuna (Akiba et al. 2019) with its default configuration to tune the learning rate and the Le regularization coefficient. The models were trained using the AdamW optimizer (Loshchilov & Hutter 2019). As an ablation study, to make sure that cubic splines contribute to the the improvement we observe, we also conduct. experiments with 9th order splines applied after the above transformation, since it may be the case that the arcsinh transformation itself yields improvement. We remind the readers that 0th order splines are just uniform bins. For splines, we used 20 knots, which is roughly half the number of bins obtained by the strategy employed by the Criteo winners. The obtained test losses are summarized in Figure 7 It is apparent that cubic splines outperform both uniform binning (0th order splines) and the original binning procedure of the Criteo winners for most embedding dimensions. Moreover, we can see that cubic splines perform best when the embedding dimension is slightly larger than the best one for binning. We conjecture that, at least for the Criteo data-set, cubic splines require more expressive power yielded by slightly higher embedding dimension to show their full potential.
We could, in theory, engineer such features using the well-known Target Encoding or Count Encoding techniques, but this is out of the scope of this work.
14

---

Published in Transactions on Machine Learning Research (12/2024)
100
107
1070
10ng
10m1
1.5
1.10
201
207
307
107
10
33-11
10:0
100
3.5
1.0
10
0.5
0.5
0.4
100
10
100
Figure 6: Histograms of columns I_1, I_13 transformed by (y(x) defined by the squared inverse hyperbolic sine, followed by min-max scaling.
Criloo Test Lows
0.45250
experiment
0.45225
floor(In(x)3) bins 0th degree spline
0.45200
3rd degree spline
0.45175
Test loss
0.45150
0.45125
0.45100
0.45075
0.45050
8
16
24
32
40
48
56
64
Embedding dim k
Figure 7: Test LogLoss for the Criteo experiment for various embedding dimensions.
15

---

Published in Transactions on Machine Learning Research (12/2024)
4.4 A/B test results online advertising system
Here we report an online performance improvement measured using an A/B test, serving real traffic of a major online advertising platform. The platform applics a proprictary CTR prediction model that is closely related to FWFM (Sun et al. 2021). The model, which provides CTR predictions for merchant catalogue- based ads, has a recency field that measures the time (in hours) passed since the user viewed a product at the advertiser's site. We compared an implementation using our approach of continuous feature training and high-resolution binning during serving time described in Section 3.4 with fine grained geometric progression of 200 bin break points, versus the "conventional" binned training and serving approach used in the production model at that time. The new model is only one of the ranker that participates in our ad auction. Therefore, a mis-prediction means the ad either unjustifiably wins or loses the auction, both leading to revenue losses.
We conducted an A/B test against the production model at that time, when our new model was serving 40% of the traffic for over six days. The new model dramatically roduced the CTR prediction error, mcasured as Average predicted CTR - 1) on an hourly basis, from average of 21% in the baseline model, to an average of Measured CTR 8% in the new model. The significant increase in accuracy has resulted in this model being adopted as the new production model.
5 Discussion
We presented an easy to implement approach for improving the accuracy of the factorization machine family whose input includes numerical fields. Our scheme avoids increasing the number of model parameters and over-fitting, by relying on the approximation power of splincs. This is explained by the spanning property along with the spline approximation theorems. Moreover, the discretization strategy described in Section 3.4 allows our idea to be integrated into an existing recommendation system without introducing major changes to the production code that utilizes the model to rank items, i.c., inference.
It easy to verify that our idea can be exlended to factorization machine models of higher order (Blondel et al., 2016). This has been shown in Rügamer (2022 for vanilla higher order FMs, but can be extended "feedd-informed" variants of higher order FMMs. In particular, the spanning property in Lemma still holds, and the pairwise spanning property in Lemma 2 becomes g-wise spanning property from machines of order q. However, to keep the paper focused and readable, we keep the analysis out of the scope of this paper.
With many advantages, our approach is not without limitations. We do not eliminate the need for data research and feature engineering, which often required when working with tabular data, since the data still needs to be analyzed to fit a function that roughly resembles the empirical CDF. We believe that feature engineering becomes easier and more systematic, but some work still has to be done.
Finally, we would like to note two drawbacks. First, approach slightly reduces interpretability, since we cannot associate feature with corresponding learned latent vector. Second, our approach may not be applicable to all kinds of numerical fields. For example, consider a product recommendation system with a product price field. Usually higher prices mean a different category of products, leading to a possibly different. trend of user preferences. In that case, the optimal segmentized output as a function the product's price is probably far from having small (higher order) derivatives, and thus cubic splines may perform poorly, and possibly even worse than simple binning.
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
Ussally each auction is conducted among several models (or "rankers"), that rank their ad inventories and compete over the incoming impression.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Sercan Ô. Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAT Conference Artificial Intelligence, 35(8):6679 6687, May 2021. doi: 10.1609 20000/aai.v333 2aaiiv3358.16686
URL attps://ojs.aaai. wtt.:/aaaa..ggggga wwooorrgggeeeeeo..ooooooooeeeeee http://ppppp//aaaararrrr////////
Avazu. Avazu Advertising http::////w....gggg...e//a/aaaaa
2014. [Online;
accessed O1-Apr-2024).
Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya S. Keerthi. Gradient boosting neural networks: Grownet, 2020.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1))4308, 2014.
Thierry 2011.
Daniel PW Ellis, Brian Whitman, and Paul Lamere. The million song dataset.
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. Higher-order factorization machines. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (ods.), Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL htttps:/pproceedings mewpeddiiinnnnnnniis...... cc/ paper / /Btt://2222022222234440002222220
pdf
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp- 785 794, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 2933977 URL https:///oli http:///oooo.or/.../. 1145/2939672. 2939785
Yuan Cheng. Dynamic explicit embedding representation for numerical features in deep ctr prediction. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 3888 (188-389,,00 2022.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. Proceedings of the 10th ACM Conference on Recommender Systems, RecSys '16 pp. 191-198, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450340335 doi: 10. 10.1145 021144555551102255115
URL https::/doi...oo
htt///ii..rrgg// org/10..11101/ /111//959111..... 2959190
Criteo.
Criteo
Display Advertising Challenge.
https://go. https://go.critee.. criteo.net
octiooorrsscrrddroaggaa--aaaaaae whwpeeigeghhalleeeeeeaeeeeeeeeee wataset.ee.m tar g2 cessed O1-Apr-2024)
2014. [Online;
Carl de Boor A Practical Guide to Splines, volume 27 of Applied Mathematical Sciences. Springer-Verlag New York, 2001. ISBN 978-0-387-95366-3.
James Dougherty, Ron Kohavi, and Mchran Sahami. Supervised and unsupervised discretization of continuous features. In Armand Prieditis and Stuart Russell (eds.), Machine Learning Proceedings 1995, pp. 194- 202. Morgan Kaufmann, San Francisco (CA), 1995. ISBN 978-1-55860-377-6 doi: httpps://oi.org/ 10.1016 111006///997--1-55583---------00 3.00000 URL htttps:/www.scieneeee /tt/://wwciinneeeeeettt.t.t htwpecaccectcccoo/cssseeeeerttte B97815586037765006222
Gottlob Frege Grundgesetze der Arithmetik: begrüffsschrüfflich abgeleitet, volume 1. H. Pohle, 1893.
João Gama and Carlos Pinto. Discretization from data streams: Applications to histograms and data mining. In Proceedings of the 2006 ACM Symposium Applied Computing, SAC '06, pp. 662-667, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595931082. doi: 10.1145/111177.1144442 URL https::/doi... org/ 10. 1eeei11111111111 1141429
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning mod- els for tabular data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances Neural Information Processing Systems, volume 34, pp. 18932-18943. Cur- ran Associates, Inc., 2021. URL U www.eeedinnnnnnuiisss.oo neurips.cc/papeee http://ppp...eeeppp//////////aa/ aBcGa8sppF770eebbbbchh377----ppp
17

---

Published in Transactions on Machine Learning Research (12/2024)
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In Advances in Neural Informalion Processing Systems, volume 35, pp. 24991-25000 2022.
GorupLens. GroupLens datasets. Apr-2024].
http:p///goooopa.....rrraaaaaatt [Online; accessed 01-
Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. An embedding learning framework for numerical features in ctr prediction. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2910 2918, 2021.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in second. In NeurIPS 2022 First Table Representation Workshop, 2022. URL htttps://openreviien htt.:/ot.....oorrrrddaaaaaaaaaa
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approx- imators. Neural Networks, 2(5):359 366, 1989. ISSN 0893-6080. doi: https///bbborrr//0000000000990 90020-8. URL
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings, 2020.
Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chib-Jen Tin. Field-aware factorization machines for ctr prediction. In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys '16, pp. 43 50, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450340359, doi: 29591144 URL ptttp:://doi.rrg//// 1145/2959100 2959134
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tic- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Tn T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htttps://proceediiii neuriss..urpp..... wttpsperrcppp.r...pp
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledgee Discovery and Data Mining, KDD'96, pp- 202 202-207. AAAT Press, 1996.
Huan Liu, Farhad Hussain, Chew Lim Tan, and Manoranjan Dash. Discretization: An enabling technique. Data mining and knowledge discovery, 6:393-423, 2002.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/ abs/1711.05107
R Kelley Pace and Ronald Barry, Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):
291-297, 1997.
Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu. Field- weighted factorization machines for click-through rate prediction in display advertising. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pp. 1349 1357, New York, NY, USA, 2018. Association URL https:///oi.orr
for Computing Machinery. ISBN 9781450356398, doi: 10.
10. 1e111//1/1176..... 3186040)
Harshit Pande. Field-embedded factorization machines for click-through rate prediction, 2021.
Liu Peng, Wang Qing, and Gu Yujia. Study on comparison of discretization methods. 2009 International Conference on Artificial Intelligence and Computational Intelligence, volume pp. 380 384, 2009, doi: 10.1109/AIC1.2009.0
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representations, 2020. URL https
18

---

Published in Transactions on Machine Learning Research (12/2024)
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Cathoost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https: httpttpp://ppoceednnggg .eweriie..pppp. cc/paper ceaaepp.ccppppprr files/ /21os//pper//20//ffll/// 14417766aaf/acc42244255553899 .dff
Steffen Rendle. Factorization machines. In 2010 IEEE Internalional Conference on Dala Mining, pp. 995 1000, 2010. doi: N0.1109/ICDM..000.10
David Rügamer. Additive higher-order factorization machines, 2022.
Isaac Jacob Schoenberg. Contributions to the problem of approximation of equidistant data by analytic functions. part b. on the problem of osculatory interpolation. a second class of analytic approximation formulac. Quarterly of Applied Mathematics, 4(2):112-141, 1946.
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84 90, 2022.
Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for labular data via row atlention and contrastive pre-training. In NewrIPS 2022 First Table Representation Workshop, 2022. URL
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang, Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedinggs of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19, pp. 1161 1170, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450369763, doi: 10. 40.1145/8337388433377222 URL https://doi. orggoo///////...
Yang Sun, Junwei Pan, Alex Zhang, and Aaron Flores. Fm2: Field-matrixed factorization machines for recommender systems. In Proceedings of the Web Conference 2021, WWW '21, pp. 2828 2837, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi:
URL https://doi. org/10.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Tan Henriksen, F. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Melhods, 17:261 272, 2020. doi: 10. 10.1038//41159--09---888
Yahoo-Inc. Fully-vectorized weighted field embedding bags for recommender systems. https:// https://github... httg//tttuub..ooo yahoo//eighted fergghttedffldddeeeeeeeeeeeeedsd [eelsd__eeccs____ 2023. [Online; accessed 01 01-May-2023)
19

---

Published in Transactions on Machine Learning Research (12/2024)
A Proof of the spanning properties
A.1 Proof of the spanning property (Lemma
Proof. For some vector q, denote by Qa:b the sub-vector (gar..., 9o). Assume w.l.o.g. that f = 1, and that field 1 has the value By construction in equation 2 we have y1 Ei= (,), while the remaining components y2a do not depend on Thus, the have
wo + (1,
wy ((() + we + (1, (2m)
(3)
1=1
Moreover, by equation depend Thus,
we have that U1 Ei=1 B,Ba(2), whereas the remaining .2....., Un. do not
Σ 2=1 1-11
Un. Up.) M50.10
à + Σ
11, U
(Ui. Uza) M
j=2
1=2 -=
82
(4)
a B,)x U3a)Mi. + +
j=2
Ui Uj.) M3.)
Bi(z) + B2.
1=1
Combining equation 3 and equation
we oblain
seg enn 1)((=1)(21) = > (w. (.) (8) + +.8).
which is the desired form.
A.2 Proof of the pairwise spanning property (Lemma 2)
Proof. Recall that we need to rewrite equation as function of Ze, 2f, which are the values of the fields and Assume w.l.o.g. that 1, f 2. By construction in equation 2 have y1 = (x(),, and 1/ = Em= wxC)(2)), while the remaining components Us. do not depend on 2. Thus, the we have
wo + (1,8)
Σ w. (2)) + WieeC((22) + we + (1, ys.))
31
(5)
a (z1 + B1.
where 0s,0 Wi, Oro.j = witt for all i,j 2 1, for all other values of i,j we set As.j = 0.
Morcover, by equation 2 we have that U1.- Ei B,() and U2. Σ (22)), whereas the remaining Us...... Un. : do not depend 21, 22. First, let rewrite the interaction between 21, #2
20

---

Published in Transactions on Machine Learning Research (12/2024)
specifically:
(U M2,) = Σ B,),( Σ EottOC(())
21.2
(6)
U M (2(2))(((22))
By following similar logic to 4 one can obtain a similar expression when looking at the partial sums that include the interaction between (resp. e) and all other fields except e (resp. f). Observe that the interaction between all other values does not depend on 21, 22. Given all of this, show how to rewrite the interaction function of 21, 22
U1s M
(U... U2
Σ
U1. U,.)
3=3
A
(U2 Mg.)
Σ
(Ui, U M
)=3
j=it1
92
Yi. (z1
a.Bi)z)
(, + 82
where 0g, aj are obtained similarly in equation As final step. Consequently,
Tene seg FMFM 1, (z2](Zz 22)
is affine function of the tensor-product basis (z2))
21


Reviews:
[{"review_id": "vb384PwWoU", "reviewer": "Reviewer_1MX4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n* FM is an important technique, and the problem addressed in the paper is relevant to the field. The authors tackle the problem of effectively incorporating numerical features into FM models.\n* The presentation is clear, with a comprehensive evaluation of the proposed approach. Empirical results suggest benefits of the proposed method. The code and results are also made publicly available, allowing for easy reproduction and verification.\n* The encoding scheme using basis functions has potential benefits if the basis functions are carefully chosen for the specific data. The authors demonstrate that their approach can improve the accuracy of FM models.\n\n[Weaknesses]\n* Motivation: it is not entirely clear why numerical columns could pose an issue for the FM. The authors could provide more context or examples to illustrate the limitations of using numerical columns naively in FM, and how their proposed approach addresses these limitations. \n* Novelty of approach: The proposed approach seems to be a feature pre-processing/transformation step before the FM, rather than a fundamental change to the FM formulation itself. While the use of basis functions may have some benefits, it is not clear whether this approach is significantly different from other feature transformation techniques that have been proposed in the literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new technique for incorporating numerical features into Factorization Machine (FM) variants, which are widely used in large-scale real-time content recommendation systems. The proposed method encodes numerical features into a vector of function values for a set of functions of one's choice, allowing the model to learn segmentized functions of the numerical feature spanned by the chosen basis. The authors demonstrate the effectiveness of their approach through experiments on synthetic and real-world datasets", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Illustrate the issue of using the numerical columns naively in the FM.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6BoU3wA2fz", "reviewer": "Reviewer_oC7P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Positive:**\n- The paper motivates the problem and the proposed approach quite clearly.\n- The datasets used in the experiments seem sufficient to show the advantage of the proposed method.\n\n\n**Negative**\nI am not an expert in the field. In general I liked the presented idea and how the paper is structured. However, I have some concerns.\n- B-splines are extensively used for such function approximations in different fields, e.g., image processing and it is usually one of the first choices. I am a bit surprised that a similar idea was not presented before. Since I am not familiar with the literature, I cannot comment about the novelty but I would strongly advise authors to check the relevant literature and compare with spline based methods. For example, the following reference I found with a quick search seems relevant [1].\n\n[1] Lan et al, Accurate and Interpretable Factorization Machines, AAAI 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the limitations of traditional Factorization Machine (FM) variants in recommendation systems that rely on binning numerical features. FM models are widely used in online content recommendation due to their balance between prediction accuracy, fast training, and inference times; however, the current binning-based approaches whose output is a step fundtion have difficulties encoding numerical features. These step functions often fail to capture the true, often smooth, nature of user behavior, leading to a gap between the model's learned functions and the actual behavior and increasing the number of bins is simply not effective. To deal with the existing issues, the paper propose a method that encodes the numerical features using a vector basis function while introducing a small change in the computation graph. They propose to use B-Splines which are particularly advantageous due to their strong approximation power and the fact that they require relatively few breakpoints (knots), minimizing the risk of sparsity. The expeirments are performed on both synthetic and real tabular datasets and demonstrate remarkable improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would expect to see more discussion about B-spline based methods in the literature.", "broader_impact_concerns": "I don't have any broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "71CBQdvCXZ", "reviewer": "Reviewer_9Z8u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Much recent work on feature preprocessing focuses on tabular ML and/or time-series settings, so examining this problem in the RecSys setting is interesting.\n\n2. As far as I'm aware, use of B-splines for feature preprocessing is original and interesting.\n\n3. The authors provided code, which is very helpful.\n\nWeaknesses:\n\n1. The writing needs major revisions to improve clarity. The paper needs to more clearly delineate between previous work and the new contributions. The background should not assume total familiarity with factorization machines; the FM description in the appendix was necessary to even try to understand the main text. The contributions should state precisely what is being proposed. The implications of the proved spanning properties are not explained -- why do these properties matter? \n\n2. If one is to evaluate a new method on tabular datasets, it is best practice to use a preexisting benchmark (eg OpenML or UCI++), instead of choosing one's own datasets (which can always be cherry-picked). Or, if one has particular reasons to choose particular datasets, the criteria for choosing these ones should be provided. (Granted, the authors seem to choose datasets with large N, which is probably justifiable as a proxy for RecSys scenarios; but this should be explained.) \n\n3. Given the interaction between numerical feature normalization and the discretization that follows, it is not clear whether the empirical benefits of B-splines generalize to other normalizations than the ones used in the paper.\n\n4. Other standard normalization and discretization methods should be compared against. (See below.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses feature preprocessing, particularly discretization of numerical features, to be used with factorization machines in the context of recommendation systems. The authors propose a basis function encoding of features, using B-splines basis functions, and evaluate their proposal on both tabular ML and recommendation system settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A) The content needs to be reorganized and reordered. First describe what exactly you're doing, then later use proofs to argue about the properties of your proposal. If related work is going to just list related papers without explaining the research context, it should be moved towards the end. The experiments section should not meander into discussing how to address integer features; that belongs elsewhere in the paper. \n\nB) Consider whether standard ML formalisms might be preferrable to RecSys terminology. For example, I suspect, but am not sure, whether \"transformation -> segmentation -> field creation\" is exactly equivalent to discretization with one-hot encoding.\n\nC) As mentioned previously, there's a vast literature on feature normalization and discretization, and at least some standard baseline methods and current leading methods should be compared against. For normalization, standard baselines include sklearn's RobustScaler, and for discretization, standard baselines include uniform, quantile and k-means discretization. There is also finite scalar quantization [Mentzer, Fabian, et al. \"Finite scalar quantization: Vq-vae made simple.\" arXiv preprint arXiv:2309.15505 (2023)], which in this setting is would involve applying rounding to normalized features to make them categorical. And there is kernel density integral (KDI) transformation [McCarter, C. The Kernel Density Integral Transformation. TMLR 2023.], which proposes adaptive methods for both normalizing features with skewed distributions and for discretizing the normalized features.  \n\nD) Figure 2 uses \"x\" notation to mean multiplication presumably, but is this a dot product, cross product, element-wise multiplication, or elementwise after broadcasting? This kind of mathematical sloppiness occurs elsewhere.\n\nE) It is claimed but never explained (or empirically shown) why FMs benefit from discretization of numerical features.\n\nF) Figure 4 -- why is there lag for orange vs blue? Is this a problem?\n\nG) Section 4.2 -- please provide more clarity on the problem setup. For example, is this classification or regression?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The revisions made by the authors have improved the paper substantially and addressed the reviewer concerns.

Please make the following changes before submitting the final reivison:
- Proofread the paper, there is still a number of typos in it.
- Make sure that citations use parenthesis when appropriate.
- Define 1_l notation in Lemma 1.
- Remove "why?" from Section 5.
Decision: Accept (poster)