Bayesian Regression & Empirical Probability-Based Prediction
Creators: name: Gupta, Riya
Publisher: name: Zenodo
Year: 2025
Subjects: subject: Naive bayes
subject: Bayession regression
subject: Artificial Intelligence
subject: machine learning
subject: Classification
subject: classification

description: Bayesian regression and probabilistic modeling have long been fundamental tools in machine learning, particularly in scenarios requiring uncertainty quantification and interpretability. Traditional Naive Bayes classifiers, while computationally efficient, rely on a strong and often unrealistic assumption of feature independence, leading to suboptimal performance in cases where features exhibit correlations or complex dependencies. This research introduces a novel empirical Bayesian methodology that refines probability estimation by directly computing the likelihood ratio K = P(B|A)/P(B) from observed data, rather than decomposing it under independence assumptions. The proposed framework consists of two algorithmic components: a base algorithm (traditional Naive Bayes) and a main algorithm (the new empirical method), allowing for a systematic comparison of their predictive accuracy, robustness, and calibration.  

 

The primary innovation of this work lies in the data-driven estimation of K, which serves as a correction factor applied to the prior probability P(A) to compute the posterior P(A|B). Unlike Naive Bayes, which estimates P(B|A) and P(B) separately—often leading to biases in small datasets or when features interact—this approach computes the ratio directly, mitigating errors introduced by incorrect independence assumptions. The methodology is tested across multiple experimental settings, including low-data regimes, high-dimensional datasets, and correlated feature spaces, to evaluate its effectiveness in improving classification performance.  

 

A key advantage of this approach is its flexibility in handling both discrete and continuous data. For discrete variables, K is estimated using frequency counts with smoothing, while for continuous data, kernel density estimation (KDE) or Bayesian regression techniques are employed. This adaptability makes the method particularly useful in real-world applications where input features may follow non-Gaussian or multimodal distributions. Additionally, the interpretability of K provides insights into feature importance, as values significantly greater than 1 indicate strong evidence for class A, while values below 1 suggest opposing evidence.  

 

Empirical evaluations demonstrate that the proposed method outperforms traditional Naive Bayes in scenarios where feature dependencies exist, while maintaining comparable performance when the independence assumption holds. The experiments also reveal that the calibration of predicted probabilities improves, reducing the overconfidence typically observed in Naive Bayes classifiers. Furthermore, the framework is extensible to advanced machine learning models, including neural networks and structured probabilistic graphical models, by replacing the empirical K with a learned function or dynamic updating mechanism.  

 

The implications of this research extend beyond classification, offering a generalized Bayesian updating mechanism applicable to regression, anomaly detection, and causal inference tasks. Future directions include integrating the method with deep learning architectures for large-scale applications and exploring its use in sequential decision-making problems. By bridging the gap between theoretical Bayesian principles and empirical data-driven learning, this work contributes a more robust, adaptable, and interpretable probabilistic modeling framework that enhances predictive accuracy while preserving computational efficiency.