Bayesian Input Variable Selection Using Cross-Validation Predictive Densities and Reversible Jump MCMC

We consider the problem of input variable selection of a Bayesian model. With suitable priors it is possible to have a large number of input variables in Bayesian models, as less relevant inputs can have a smaller effect in the model. To make the model more explainable and easier to analyse, or to reduce the cost of making measurements or the cost of computation, it may be useful to select a smaller set of input variables. Our goal is to find a model with the smallest number of input variables having statistically or practically the same expected utility as the full model. A good estimate for the expected utility, with any desired utility, can be computed using cross-validation predictive densities (Vehtari and Lampinen, 2001). In the case of input selection, there are 2 K input combinations and computing the cross-validation predictive densities for each model easily becomes computationally prohibitive. We propose to use the reversible jump Markov chain Monte Carlo (RJMCMC) method to find out potentially useful input combinations, for which the final model choice and assessment is done using the cross-validation predictive densities. The RJMCMC visits the models according to their posterior probabilities. As models with negligible probability are probably not visited in finite time, the computational savings can be considerable compared to going through all possible models. The posterior probabilities of the models, given by the RJMCMC, are proportional to the product of the prior probabilities of the models and the prior predictive likelihoods of the models. The prior predictive likelihood measures the goodness of the model if no training data were used, and thus can be used to estimate the lower limit of the expected predictive likelihood. These estimates indicate potentially useful models, for which better estimates of the expected utilities can then be computed using the cross-validation predictive densities. If, due to the size of the problem, the RJMCMC has problems of obtaining enough samples in reasonable time to estimate the probabilities of the models well, we propose to use the marginal posterior probabilities of the inputs to select potentially useful inputs. We also discuss why the posterior probabilities of the models (or inputs) given by the RJMCMC should not be used directly for input selection. As illustrative examples we use MLP neural networks and Gaussian processes (GP) in two real world problems. The first problem was a concrete quality prediction problem, where the primary goal was to make models more explainable. Depending on the target variable, six to nine input variables were chosen from a set of 27‐28 input variables. The predictive capability remained statistically the same as the predictive capability of the full model. Additional benefit was the reduced cost of making future measurements. The second problem was a forest scene classification problem, where the primary goal was to decrease computational cost. In the planned application it turned out to be sufficient to use just 10 out of 84 input variables, without classification rate dropping too much. In this case, the input selection was guided by the marginal posterior probabilities of the inputs.