CATEGORICAL FEATURE SELECTION FOR RANKING MODELS

Machine Learning based ranking models are ubiquitous in powering recommendation engines at internet companies. These models typically use a combination of real-valued numerical and categorical features to generate predictions. Feature selection may be a widely encountered problem in this setting, that entails picking the optimal set of features as inputs to these models from a large pool of candidate real-valued and categorical features. A novel feature selection algorithm for categorical features building on stochastic neural networks is provided. It is shown empirically through results, the superiority of this algorithm over existing approaches. Study and proposal of best practices are also provided to practitioners to extract maximum value out of the new feature selection approach.

TECHNOLOGICAL FIELD

Exemplary embodiments of this disclosure relate generally to methods, apparatuses and computer program products for utilizing stochastic neurons for categorical feature selection in neural networks, and in particular, to stochastic neurons that enable selection of a random subset of categorical features for a ranking model in neural networks.

BACKGROUND

Machine Learning based models are at the heart of the engines that power ranking and recommendation services at internet companies. These include ranking videos, posts, images, advertisements (ads) and other kinds of content. For some social networks, the most used ranking models may power several trillion predictions each day. These models help sort among the many pieces of content eligible to be shown to a user, optimizing for various metrics. In applications such as ranking ads, machine learning may play a central role in computing the expected utility of showing a candidate ad to a user.

BRIEF SUMMARY

In search advertising, the input user query is used to first filter candidate ads from a larger set, which are matched to the query either through implicit or explicit means. For some social networks, ads may not be associated with a query, but instead specify demographic and interest targeting. As a result, the volume of eligible ads to be chosen from when a user visits the social network, may be larger than that for search advertising. To deal with this large candidate set, exemplary embodiments may employ multi-stage ranking with each stage filtering down ads and growing in model complexity. As described herein, exemplary embodiments may perform experiments on the last stage click prediction model, that is the model that produces predictions for the final set of candidate ads.

These models typically rely on several hundreds of real-valued and categorical features, which are in turn to be picked from a pool of several thousands of features. The number of features that may be used in a model may be constrained by serving memory and compute constraints. For a simple ranking model architecture such as Deep Factorization-Machine (FM), the computational cost of the input layer may scale quadratically in the number of features participating in interactions. The above drawbacks may necessitate the use, by the exemplary embodiments, of a feature selection algorithm for picking the optimal set of features to be used in a given ranking model.

Ranking models rely on broadly two kinds of features which may differ principally in their input representations namely real-valued (e.g., numerical or dense) and categorical (e.g., sparse) features. Real-valued features may be those which are represented using scalar real values viz., average click-through rate on a given ad. Categorical features may be those which are represented as a one-hot vector of the length of the number of categories. Examples of categorical features may include language of a post, semantic category of an ad, etc. Categorical features may typically be densified by employing an embedding layer before being passed as inputs to the rest of the model. These embedding layers may house a parameter variable of size and number of categories x embedding dimension. Such embedding layers may allow for rich semantic and contextual information to be learned and stored in each of the embedding vectors corresponding to various indices of the categorical feature.

Feature selection methods may be divided into three categories: Filter methods, Wrapper methods, and Intrinsic/Embedded methods. Filter methods typically may not involve a learning component and may work like a pre-processing step. These methods may use a statistical measure of each individual feature to sort features by importance. These measures may include feature variance, correlation to the output variable among other metrics. They typically may fail to account for dependencies across features. On the other hand, Wrapper methods may involve the use a learned classifier to determine the importance of each feature. Wrapper methods may typically work by using a subset of features each time (with one or more features added or removed sequentially) and using the resulting classifier performance as a proxy for feature importance. Because they involve training a classifier for each such feature subset, they can be computationally expensive especially for large feature pools and complex classifiers such as those used for recommendation engines. The third variety, Intrinsic/Embedded methods may be designed to pick the important subset of features during model training itself and may thereby avoid the overhead of Wrapper methods. Examples of these may include usage of decision trees, Least Absolute Shrinkage and Selection Operator (LASSO). While it may seem attractive to extend LASSO to neural networks, gradient descent with an L1 penalty added to the loss in practice may not sparsify the input layer as desired.

The exemplary embodiments may adapt the sparsification method using stochastic gates to the categorical feature selection problem.

Additionally, the exemplary embodiments may demonstrate empirically, the superiority of this method over existing feature selection approaches.

Further, extensive experiments were conducted by exemplary embodiments to enable provision of useful advice for practitioners to best leverage this method in large-scale recommendation models.

DETAILED DESCRIPTION

As defined herein a “computer-readable storage medium,” which refers to a non-transitory, physical or tangible storage medium (e.g., volatile or non-volatile memory device), may be differentiated from a “computer-readable transmission medium,” which refers to an electromagnetic signal.

It is to be understood that the methods and systems described herein are not limited to specific methods, specific components, or to particular implementations. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

Exemplary System Architecture

FIG.1illustrates an example network environment100associated with a social-networking system160. Network environment100includes a user101, a client system130, a social-networking system160, and a third-party system170connected to each other by a network110. AlthoughFIG.1illustrates a particular arrangement of user101, client system130, social-networking system160, third-party system170, and network110, this disclosure contemplates any suitable arrangement of user101, client system130, social-networking system160, third-party system170, and network110. As an example and not by way of limitation, two or more of client system130, social-networking system160, and third-party system170may be connected to each other directly, bypassing network110. As another example, two or more of client system130, social-networking system160, and third-party system170may be physically or logically co-located with each other in whole or in part. Moreover, althoughFIG.1illustrates a particular number of users101, client systems130, social-networking systems160, third-party systems170, and networks110, this disclosure contemplates any suitable number of users101, client systems130, social-networking systems160, third-party systems170, and networks110. As an example and not by way of limitation, network environment100may include multiple client systems130, social-networking systems160, third-party systems170, and networks110.

In particular embodiments, user101may be an individual (human user), an entity (e.g., an enterprise, business, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with or over social-networking system160. In particular embodiments, one or more users101may use one or more client systems130to access, send data to, and receive data from social-networking system160or third-party system170.

In particular embodiments, client system130may be an electronic device including hardware, software, or embedded logic components or a combination of two or more such components and capable of carrying out the appropriate functionalities implemented or supported by client system130. As an example, and not by way of limitation, a client system130may include a computer system such as a desktop computer, notebook or laptop computer, netbook, a tablet computer, e-book reader, Global Positioning System (GPS) device, camera, personal digital assistant (PDA), handheld electronic device, cellular telephone, smartphone, augmented/virtual reality device, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client systems130. A client system130may enable user101to access network110. A client system130may enable its user101to communicate with other users101at other client systems130.

In particular embodiments, social-networking system160may store one or more social graphs in one or more data stores164. In particular embodiments, a social graph may include multiple nodes—which may include multiple user nodes (each corresponding to a particular user101) or multiple concept nodes (each corresponding to a particular concept)— and multiple edges connecting the nodes. Social-networking system160may provide users101of the online social network the ability to communicate and interact with other users101. In particular embodiments, users101may join the online social network via social-networking system160and then add connections (e.g., relationships) to a number of other users101of social-networking system160to whom they want to be connected. Herein, the term “friend” may refer to any other user101of social-networking system160with whom a user101has formed a connection, association, or relationship via social-networking system160.

In particular embodiments, social-networking system160may be capable of linking a variety of entities. As an example and not by way of limitation, social-networking system160may enable users to interact with each other as well as receive content from third-party systems170or other entities, or to allow users to interact with these entities through an application programming interfaces (APIs) or other communication channels.

In particular embodiments, social-networking system160may also include user-generated content objects, which may enhance a user’s interactions with social-networking system160. Content may also be added to social-networking system160by a third-party through a “communication channel,” such as a newsfeed or stream.

FIG.2illustrates an example computer system200. In particular embodiments, one or more computer systems200may perform one or more steps of one or more methods described or illustrated herein. In some example embodiments, the computer system200may be the server162of social-networking system160. In other example embodiments, the computer system200may be the client system130. In particular embodiments, one or more computer systems200may provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems200may perform one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular exemplary embodiments may include one or more portions of one or more computer systems200. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

In particular embodiments, computer system200may include a processor202, memory204, storage206, an input/output (I/O) interface208, a communication interface210, camera module212, stochastic neurons module214, and a bus216. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor202may include hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor202may retrieve (or fetch) the instructions from an internal register, an internal cache, memory204, or storage206; decode and execute them; and then write one or more results to an internal register, an internal cache, memory204, or storage206. In particular embodiments, processor202may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor202including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor202may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory204or storage206, and the instruction caches may speed up retrieval of those instructions by processor202. Data in the data caches may be copies of data in memory204or storage206for instructions executing at processor202to operate on; the results of previous instructions executed at processor202for access by subsequent instructions executing at processor202or for writing to memory204or storage206; or other suitable data. The data caches may speed up read or write operations by processor202. The TLBs may speed up virtual-address translation for processor202. In particular embodiments, processor202may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor202including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor202may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors202. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor. In some example embodiments, the stochastic neurons module214may utilize stochastic neurons to select an optimal set of features (e.g., an optimal feature subset from a full set of features) for one or more ranking models, as described more fully below.

In particular embodiments, memory204may include main memory for storing instructions for processor202to execute or data for processor202to operate on. As an example and not by way of limitation, computer system200may load instructions from storage206or another source (such as, for example, another computer system200) to memory204. Processor202may then load the instructions from memory204to an internal register or internal cache. To execute the instructions, processor202may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor202may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor202may then write one or more of those results to memory204. In particular embodiments, processor202executes only instructions in one or more internal registers or internal caches or in memory204(as opposed to storage206or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory204(as opposed to storage206or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor202to memory204. Bus216may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor202and memory204and facilitate accesses to memory204requested by processor202. In particular embodiments, memory204may include random access memory (RAM). This RAM may be volatile memory, where appropriate Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory204may include one or more memories204, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage206includes mass storage for data or instructions. As an example and not by way of limitation, storage206may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage206may include removable or non-removable (or fixed) media, where appropriate. Storage206may be internal or external to computer system200, where appropriate. In particular embodiments, storage206may be non-volatile, solid-state memory. In particular embodiments, storage206may include read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage206taking any suitable physical form. Storage206may include one or more storage control units facilitating communication between processor202and storage206, where appropriate. Where appropriate, storage206may include one or more storages206. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface208includes hardware, software, or both, providing one or more interfaces for communication between computer system200and one or more I/O devices. Computer system200may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system200. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, camera module212(e.g., a still camera, a video camera), stylus, pointing device, tablet, touch screen, trackball, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces208for them. Where appropriate, I/O interface208may include one or more device or software drivers enabling processor202to drive one or more of these I/O devices. I/O interface208may include one or more I/O interfaces208, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In particular embodiments, bus216includes hardware, software, or both coupling components of computer system200to each other. As an example and not by way of limitation, bus216may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus216may include one or more buses216, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

Exemplary System Operation

The base ranking model architecture utilized by the exemplary embodiments for exposition of the method is described herein. The background and problem setup are then described. Thereafter, a description is provided of the proposed extension to apply the proposed extension to categorical feature selection.

A. Deep Learning Based Recommendation Model (DLRM)

With the advent of deep learning, neural network based models have proliferated into ranking, recommendation and personalization applications in the industry. When practitioners first began to design neural network architectures for these applications, they typically had to contend with the different kinds of features presented to these models vis-a-vis those architectures that had been devised and popularized in early deep learning literature. While numerical (e.g., dense) inputs may be trivial to process through a Multi-Layer Perceptron (MLP) like architecture, it may not be directly evident how best to process categorical inputs which describe high level attributes. While one part of this question may be the input representation for categorical features, it may also be unclear how best to make them interact deeper inside the architecture.

Some social networks may utilize a simple architecture to solve these problems called the Deep Learning for Recommendation Model (DLRM). In this architecture as shown inFIG.3, for the input representation, categorical features may be processed using an embedding layer, while continuous features may be processed using an MLP. Thereafter, second-order interactions may be computed among all pairs of features explicitly. On top of these computed interactions, there is another top MLP which may feed into a sigmoid function to output the probability of the desired event (for example, an ad click).

B. Stochastic Neurons for Feature Selection

Feature selection may be posed as a subset selection problem from the universe of all available features. This problem may be Non-deterministic Polynomial (NP)-hard in the general case owing to the combinatorially large search space, thus forcing choosing between either sub-optimal feature selection methods that make impractical independence assumptions or inefficient computation.

Over the last decade, examples of machine learning aiding the solution of such theoretically hard problems have been identified. This raises the intriguing possibility of whether it may be possible to learn the importances of features, just as parameters are learned in a model end-to-end using gradient descent. Of course, unlike traditional parameter learning, this may be complicated by the discrete nature of the subset selection operation. Typically, such roadblocks have been tackled in literature using algorithms such as REINFORCE (i.e., a known algorithm). But this approach may suffer from high variance and may be computationally expensive given the large model sizes used in production ranking systems.

To motivate the use of stochastic neurons, simpler options are first considered at hand. For example, the exemplary embodiments may gate input features through an element-wise multiplicative layer of binary neurons, and may then incentivize the model to only keep a fraction of these neurons alive. The L0 norm of the vector of gating values may serve to capture this quantitatively. However, minimizing the L0 norm may be unwieldy to gradient-based optimization owing to non-differentiability of the L0 norm. This may motivate turning to stochastically sampling the gating values from distributions, for whose choices the expected L0 norm of the gating vector is differentiable. The simple case of binary gates, which may be sampled from Bernoulli distributions are considered. This time around, the expected L0 norm may be easy to compute, as a sum of the Bernoulli parameters. However, the click-label cross entropy loss now may have dependence on the binary gates and thus it may be hard to minimize without resorting to Straight-through estimator or REINFORCE-like techniques.

To get around this, the stochastic neurons module214may attempt to smooth the discrete gates while also crucially allowing for exact zeros. This may be done using a simple hard-sigmoid rectifier e.g., min(1, max(0, x)) applied to samples from a continuous distribution. The stochastic neurons module214may now choose distributions under which the expected L0 norm is differentiable. While the stochastic neurons module214may have flexibility in this choice, it may choose either the Binary Concrete distribution or the Gaussian distribution. The stochastic neurons module214may learn the parameters of these distributions at each of the feature gating neurons. While some existing techniques may be interested in learning sparse neural networks, the stochastic neurons module214may, in some exemplary embodiments, apply a layer of stochastic neurons only at the input layer for the purpose of selecting features. Crucially, in some existing techniques, only gates that eventually attain a value 0 may be useful for the end goal of sparsity. However, in the case of feature selection, the stochastic neurons module214may interpret the resultant gate distribution parameter values as the importance of various features, and may use these to retain any desired number of features for training the ranking model. The stochastic neurons module214may rank features in this way as meaningful because of the nature of gradient-based learning e.g., features whose gates may need to be eventually pushed to zero may first need to be lowered through that continuum. The stochastic neurons module214may verify this hypothesis empirically.

While the above expositions detail the application of multiplicative gating for dense features, it may be unclear how best to extend this for categorical features which are represented as a one-hot vector in the input layer. Two possible ways to extend this algorithm to categorical features are now described, while also keeping in mind how they are used in the DLRM model.

1) Individual Feature Gating: In this method, the stochastic neurons module214may create one stochastic neuron per categorical feature, and may gate its entire embedding (after embedding layer lookup) to pass through this neuron. The stochastic neurons module214may use such a gate sharing mechanism because a feature may either be present or absent in its entirety, but not partially present.FIG.4illustrates how this is applied in the DLRM model.

The Individual Feature Gating approach of the exemplary embodiments may improve optimization of the feature selection inputs to a Multi-Layer Perceptron architecture associated with a ranking model (e.g., an ads ranking model). For example, there may be 2,000 features determined as being associated with a ranking model. As an example only, these 2,000 features may be associated with all possible interests expressed by users in a social network.

However, as an example, only the top 400 features may be provided as inputs to the ranking model. In this regard, the Individual Feature Gating approach may be implemented by the stochastic neurons module214to determine which of the 2,000 features are the top 400 features for the ranking model.

In this manner, for example, the stochastic neurons module214may determine different categorical features42,44,46associated with a ranking model (as shown inFIG.4). The categorical features42,44,46, etc. may be represented by corresponding embedding layers41,43,45, etc. (as shown inFIG.4). For purposes of illustration and not of limitation, a categorical feature42may relate to which webpages users of a social network may have liked to visit over a time period (e.g., within the last week) which may be associated with the embedding layer41. Another categorial feature44may be associated with the types of plasma televisions (TVs) that users of the social network liked over a time period (e.g., within the last week) based on visiting TV webpages, which may be associated with another embedding layer43. The TV webpage visits may be determined by analyzing the users browsing history. Other categorical features (e.g., preferred rideshare services, road transportation, etc.) may be associated with corresponding embedding layers in a similar manner.

The categorial features associated with the embedding layers may be input to stochastic gates47,48,49(as shown inFIG.4). The stochastic neurons module214may implement the stochastic gates47,48,49such that the stochastic gates47,48,49may determine whether the corresponding categorical feature (e.g., categorial feature42) is provided to the ranking model (or not provided to the ranking model) associated with the MLP 40 (as shown inFIG.4). In this regard, for example, the stochastic neurons module214may determine importance scores for each of the categorical features (e.g., 2,000 features) and may pass the top categorical features (e.g., top 400 features) from the set of features (e.g., the 2,000 features) having non-zero importance scores for example within a score range, as referred to herein as gating values, (e.g., between 0 and 1) to a ranking model associated with the MLP 40. The scoring that may be attributed to a feature(s) may be a deterministic function of the learned parameter value (e.g., through neural network training) of its gate distribution.

2) Gating on Pairwise Interactions: In the DLRM model, the primary mode of consumption of categorical features by the neural network (NN) may be through pairwise interactions of features. These pairwise interactions may allow to evolve semantic meaning to the input features. For example, in the application to ads click prediction, these interactions may capture the synchrony between a certain user characteristic (e.g., stated user interests) and an ad characteristic (e.g., ad category). So, taking this view of the model, it may be reasonable to attribute importances to individual features based on importances of interactions that a feature participates in.

In this extension, it is proposed to apply multiplicative gating on top of the interactions layer output. The stochastic neurons module214may then map learned gating values at the pairs level to individual features by suitable averaging. This design is illustrated inFIG.5. The Gating on Pairwise Interactions approach, implemented by the stochastic neurons module214, may be the same or similar approach to the stochastic neurons module214implementation of the Individual Feature Gating approach described above. The Gating on Pairwise Interactions approach may be another manner of answering/addressing the same question/issue as the Individual Feature Gating approach. In the Gating on Pairwise Interactions approach, the stochastic neurons module214may first compute importances for pairs of features. The stochastic neurons module214may then compute the importance of an individual feature as the average importance of all feature pairs that the individual feature is part of.

Below is a review of some of the work both on feature selection generally and on ranking models specifically.

A. Filter Methods

Filter methods as known may not involve a learning component and may work like a pre-processing step. These methods may use a statistical measure of each individual feature to sort features by importance. These measures may include feature variance, correlation to the output variable among other metrics. The filter methods may typically fail to account for dependencies across features.

Wrapper methods as known may involve the use a learned classifier to determine the importance of each feature. Wrapper methods may typically work by using a subset of features each time (with one or more features added or removed sequentially) and may use the resulting classifier performance as a proxy for feature importance. Because Wrapper methods may involve training a classifier for each such feature subset, they can be computationally expensive especially for large feature pools and complex classifiers such as those used for recommendation engines.

C. Embedded Methods

Embedded methods may be designed to pick the important subset of features during model training itself and may thereby avoid the overhead of Wrapper methods. Examples of these may include usage of decision trees, and Least Absolute Shrinkage and Selection Operator (LASSO). While it may seem attractive to extend LASSO to neural networks, gradient descent with an L1 penalty added to the loss in practice may not sparsify the input layer as desired.

Previously, some researchers have accounted for this by developing ways to use the L0 penalty which may serve better to capture the presence/absence of a feature without penalizing absolute value. In some existing techniques, the notion of using stochastic neurons for inducing sparsity is introduced. In an existing technique, for example, a Binary Concrete distribution may be used to model the gating values. The reparametrization trick may be used to allow for gradient based learning of the distributional parameters. In some other existing approaches, an unsupervised feature selection method using a Concrete layer at the input may be utilized. In other traditional approaches, an input reconstruction loss may be used for driving the parameter training, with an optional supervised extension. In some other traditional approaches, stochastic gates may be applied to real-valued feature selection utilized on numerical features Additionally, some existing approaches tout using Gaussian as the underlying distribution choice as working better than Binary Concrete for feature selection settings. There is also some existing techniques involving re-purposing the Integrated Gradients feature attribution work for the application of feature selection.

However, unlike prior approaches, the exemplary embodiments may be directed to the usage of stochastic gates for categorical feature selection in ranking models. The exemplary embodiments also consider systematic study of design choices that practitioners confront when applying techniques to large scale models.

Experiments

In this section, results from experiments are described regarding the proposed feature selection method. In all experiments, the DLRM architecture may be utilized for the models. An event prediction model may be utilized (for example by stochastic neurons module214) and its corresponding datasets across all experiments to maintain consistency across findings. In this example, the size of the categorical feature pool is 2,000. After a feature selection run is performed on this entire pool, the stochastic neurons module214may rank features by their computed importance values and pick the top features (e.g., the top 400 features) for use in the ranking model. The stochastic neurons module214may use the DLRM architecture for the feature selection model runs, except when stated otherwise. The stochastic neurons module214may also utilize an optimization algorithm and may track the normalized cross entropy (NCE) of the event prediction task as the primary metric of comparison. For each experiment, the stochastic neurons module214may report relative improvements or drops in NCE compared to the baseline. As described herein, the proposed method may be referred to as Categorical Stochastic Neurons (CSN) in the comparisons.

A. Comparing Different Algorithms

The proposed method for categorical feature selection may be compared against multiple strong baselines including the shuffle-based method, the Integrated Gradients method as well as the choice between using stochastic neurons on pairwise interactions versus at an individual feature level. As shown in the data below, it can be seen that the stochastic neurons method outperforms existing approaches significantly (e.g., lower NCE is better). As shown in the data below, it can also be seen that gating at an individual feature level may work better than gating on pairwise interactions and thereafter mapping gating values to individual feature importances.

B. Comparing Choices of Distribution

Two different choices of the underlying distribution governing gating neurons are now described. The first candidate is the Binary Concrete distribution, which is parametrized using the log(a) parameter and beta (which is set to 0.5). The second candidate is the Gaussian distribution, which is parametrized using the mean, with the variance found using hyperparameter tuning. As can be determined from Table II, both perform very similarly across settings. For example, Table II shows that the relative difference between the Binary Concrete distribution and the Gaussian distribution is only 0.003%, which is insignificant.

C. Studying the Effect of Data Volume

A natural question that a practitioner may pose is “What is the right data volume to use for the Feature Selection (FS) run?” It seems intuitive that using more data may help with learning the importance of features better, but is there a saturation point beyond which it may not help with improving downstream ranking model loss minimization? This question was studied by varying the data volume used for the FS run. It was determined (for example, by the stochastic neurons module214) that in the low data regime, there is an improvement in downstream model performance with more data added, but that this advantage may saturate as the volume increases.

One of the most important requirements from the output of the Feature Selection run may be the importance of features thus computed (e.g., by the stochastic neurons module214) provide a total ordering among the input feature pool. With the stochastic neurons approach, the gating values are constrained to be in [0, 1]. For this reason, if several (e.g., greater than a constant K) features are either saturated at an importance of 1.0 or squashed down to 0, then it may not be possible to find the top K features from the feature pool. This provided motivation to study which hyperparameters may have the most significant bearing on the number of importances that settle to the open interval (0, 1). It was hypothesized that the learning rate of the FS run may be a critical hyperparameter. Through experiments, it was determined (for example by the stochastic neurons module214) that the learning rate may give practitioners a powerful way to control for the dynamic range of feature importance. This was measured using a proxy - i.e., the number of features whose importances settle in (0, 1). This metric was measured as a function of the a parameter in an optimization algorithm and the findings are indicated in Table IV.

Conclusion

In this disclosure, the motivation behind devising effective feature selection algorithms for ranking and recommendation models is described. Recent developments involving stochastic neural networks are discussed, along with relevant extensions to the feature selection problem. This idea is then extended to work for categorical feature selection. Advice to practitioners to best utilize this method for their models is also described herein.

As extensions to the stochastic neural network developments provided by the exemplary embodiments, study regarding the effect of performing feature selection of both numerical features and categorical features (and possibly other kinds of features) together in a single FS run may be performed. While the exemplary embodiments may use hyperparameter search for setting optimal values for L0 regularization strength, it may be studied whether there are more systematic ways of estimating optimal values for these parameters. These extensions may also be the subject of future work.