Bayesian Regression on segmented data using Kernel Density Estimation Extending Bayes‘ theorem The challenge of having to deal with dependent variables in classification and regression using techniques based on Bayes' theorem is often avoided by assuming a strong independence between them, hence such techniques are said to be naïve. While analytical solutions supporting classification on arbitrary amounts of discrete and continuous random variables exist, practical solutions are scarce. However, not only can we extend Bayes’ theorem to any number of events, we may also use continuous random variables instead, or any combination of both. Random variables however require a conditional density, which, in practice, is not given. We can estimate that density, by first segmenting the dataset using conditions for discrete variables, or the empirical conditional distribution function. This allows us to practically solve a non- naïve version of Bayes’ theorem where we assume full conditionality between all variables. The full general form for classification is shown below: Centre for Data Intensive Sciences and Applications Data-driven Software and Information Quality Group Sebastian Hönel, Morgan Ericsson, Welf Löwe, Anna Wingkvist Data-driven Software and Information Quality Group – Centre for Data Intensive Sciences and Applications Linnaeus University, 351 95 Växjö, Sweden {sebastian.honel, morgan.ericsson, welf.lowe, anna.wingkvist}@lnu.se 𝑃𝐿= 𝑙𝑋1 = 𝑥1, … , 𝑋𝑛= 𝑥𝑛= 𝑝(𝐿=𝑙) ς𝑚=1 𝑛 𝑝(𝑋𝑚=𝑥𝑚|𝑋𝑚+1=𝑥𝑚+1,…,𝑋𝑛=𝑥𝑛,𝐿=𝑙) ς𝑚=1 𝑛 𝑝(𝑋𝑚=𝑥𝑚|𝑋𝑚+1=𝑥𝑚+1,…,𝑋𝑛=𝑥𝑛) , where each conditional p(𝑋𝑚= 𝑥𝑚| … ) is one of ൝𝑃𝑋𝑚= 𝑥𝑚𝑋𝑚+1 = 𝑥𝑚+1, … , 𝑋𝑛= 𝑥𝑛[, 𝐿= 𝑙]), 𝑖𝑓𝑋𝑚𝑖𝑠𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒, 𝑃𝐷𝐹𝑋𝑚|𝑋𝑚+1=𝑥𝑚+1,… ,𝑋𝑛=𝑥𝑛[, 𝐿=𝑙] 𝑥𝑚, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. And, accordingly, ∀𝑋𝑚∈𝑋segmentation is done by ቊ𝑋𝑚= 𝑥𝑚, 𝑖𝑓𝑋𝑚𝑖𝑠𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒, 𝑋𝑚≤𝑥𝑚, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. If all variables are discrete, the general form yields a probability for L = 𝑙, a likelihood, otherwise. This is still useful, as we can simply determine the most likely label መ𝑙= 𝑎𝑟𝑔𝑚𝑎𝑥𝑃𝐿= 𝑙… ∀𝑙∈𝐿. This is called the maximum a posteriori estimation (MAPe). For each label, the strength of the class membership is also indicated. The general form can be simplified to become a Bayesian network, by waiving conditioning over the desired label. Then also, the evidence (denominator, normalizing constant) would become equal to the likelihood (numerator). If relinquished, the simple form looks as in the following equation: 𝑌′ = 𝑃𝑋1 = 𝑥1, … , 𝑋𝑛= 𝑥𝑛= ෑ 𝑚=1 𝑛 𝑝(𝑋𝑚= 𝑥𝑚|𝑋𝑚+1 = 𝑥𝑚+1, … , 𝑋𝑛= 𝑥𝑛) This simplified form still assumes dependence between all variables. Note that 𝑌′ holds all remaining values after segmenting. This allows us to perform a regression, by estimating a kernel density (KDE). Using that empirical PDF, similar to MAPe, a most-likely value can be obtained by finding the highest mode of the PDF, i.e., ො𝑦= 𝑎𝑟𝑔𝑚𝑎𝑥𝑃𝐷𝐹𝑌′ 𝑦, 𝑦∈𝑌′. The developed models allow for a few hyperparameters. We demonstrate the effect of each w.r.t. the current performance metric, i.e., Accuracy or 𝑅2 (the squared Pearson covariance). In order to be able to utilize the full general form, a label would be required. If our target variable were to be continuous, we can discretize it, and then find the most likely range. That range can be used to obtain a Y′ that also holds the most likely values for 𝑦. Using those remaining values, again, we can estimate a kernel density for regression. Predicting the mean would imply assuming a gaussian distribution. KDE, however, allows to estimate an arbitrary distribution from the remaining samples. While the simplified form does not require a label, it is required for the full form. Hence, such a regression is preceded by a classification. Acc. Impr. Kappa Runt.(s) Method Dataset 0.65 0.00 0.00 0.12 ZeroR breast (Class) 0.65 0.00 0.00 8.00 Simple 0.89 0.24 0.75 43.80 Full 0.38 0.00 0.00 0.12 ZeroR chick (Diet) 0.93 0.55 0.90 5.95 Simple 1.00 0.62 1.00 16.89 Full 0.70 0.00 0.00 0.12 ZeroR credit (Class) 0.69 -0.01 0.06 6.37 Simple 0.70 0.00 0.00