Add Geometric Mixture Classifier (GMC)

### Describe the workflow you want to enable

The Geometric Mixture Classifier (GMC) enables interpretable, efficient classification on multimodal datasets where traditional linear models struggle. Many real-world classification problems involve classes that occupy multiple disjoint regions in feature space (e.g., medical diagnosis with multiple disease subtypes, customer segmentation with distinct behavioral clusters).

**Current scikit-learn offers:**

Linear models (LogisticRegression, LinearSVM): Fast and interpretable but fail on multimodal data

Nonlinear models (RBF SVM, RandomForest): Handle complex boundaries but sacrifice interpretability and computational efficiency

Mixture models (GaussianMixture): Generative approach with different assumptions

GMC fills this gap by providing a discriminative, interpretable classifier that can handle multimodal classes while maintaining transparency and competitive performance.

**Use cases enabled:**

Medical diagnosis where doctors need to understand which "subtype" features drove a prediction

Customer segmentation requiring both accuracy and explainable cluster assignments

Educational applications where students can visualize decision boundaries and plane responsibilities

Resource-constrained environments needing fast inference with interpretable results

### Describe your proposed solution

I propose adding the Geometric Mixture Classifier (GMC) - a discriminative model that represents each class as a mixture of hyperplanes with intrinsic interpretability.

Key Features:
Per-class mixture of hyperplanes: Each class can have multiple linear decision boundaries

Soft-OR pooling: Within-class plane scores aggregated via temperature-controlled log-sum-exp

Geometry-aware initialization: Uses k-means clustering for stable plane initialization

Automatic plane budgeting: Determines optimal planes per class via silhouette scoring

Random Fourier Features support: Optional nonlinear extension while maintaining interpretability

Usage-aware regularization: Prevents dead experts and promotes specialization

Mathematical Formulation:​
  planes:
# Per-plane score
z_cm(x) = w_cm·φ(x) + b_cm

# Within-class soft-OR pooling  
s_c(x) = (1/α) * log(∑ exp(α * z_cm(x)))

# Across-class softmax
p_c(x) = exp(s_c(x)) / ∑_k exp(s_k(x))
Implementation Advantages:
Interpretable: Decisions explained by which plane(s) were responsible

Efficient: O(d′M) inference time (comparable to linear models)

Calibrated: Good probabilistic outputs with optional temperature scaling

Transparent: Visualizable decision boundaries and plane responsibilities

API Design:
python
class GeometricMixtureClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self, n_planes_per_class='auto', alpha=6.0, use_rff='auto',
                 rff_dim=1000, gamma=1.0, lambda_reg=1e-4, beta=0.5,
                 label_smoothing=0.02, max_planes=4, random_state=None):
        ...
Performance Characteristics:
Based on the attached research paper, GMC:

Outperforms linear baselines on multimodal datasets

Competitive with RBF-SVM and Random Forests

Faster inference than kernel methods (scales with planes, not support vectors)

Lower ECE than many alternatives after temperature scaling

Proof of Concept:
I have already developed and tested a complete implementation:

GitHub Repository: [Geometric Mixture Classifier](https://github.com/Abitsfhuusrtyt/-Geometric-Mixture-Classifier-GMC---A-Discriminative-Per-Class-Mixture-of-Hyperplanes)

Research Paper: Comprehensive evaluation on synthetic and real datasets

Working Code: scikit-learn compatible implementation with tests and examples

Inclusion Benefits:
Novelty: First discriminative mixture-of-hyperplanes in scikit-learn

Interpretability: Addresses growing need for explainable AI

Performance: Favorable accuracy/efficiency/interpretability trade-off

Educational Value: Clear geometric interpretation helps ML pedagogy

The implementation follows scikit-learn API conventions, includes comprehensive tests, and maintains the project's quality standards. I'm prepared to contribute the code, documentation, and examples to facilitate review and integration.

### Describe alternatives you've considered, if relevant

Existing scikit-learn classifiers and their limitations for multimodal data:

Linear Models (LogisticRegression, LinearSVM)

✅ Fast, interpretable, convex training

❌ Single global hyperplane cannot capture multimodal class structure

❌ Fail on datasets like "moons", "circles", or any class with disjoint regions

Kernel SVM (SVC with RBF kernel)

✅ Handles nonlinear, multimodal boundaries effectively

❌ Poor interpretability (implicit kernel expansion)

❌ Inference cost scales with number of support vectors

❌ Memory intensive for large datasets

Tree-based Methods (RandomForest, GradientBoosting)

✅ Excellent accuracy on complex boundaries

❌ Limited geometric interpretability

❌ Ensemble of trees provides feature importance but not clear decision geometry

❌ Can be computationally heavy for inference

Gaussian Mixture Models + Bayes

✅ Captures multimodality via density estimation

❌ Generative approach optimizes likelihood, not discriminative performance

❌ Strong distributional assumptions (Gaussianity)

❌ Less calibrated probabilistic outputs

K-Nearest Neighbors

✅ Simple, handles complex boundaries

❌ No explicit model, poor interpretability

❌ Degrades in high dimensions

❌ Irregular decision boundaries

Multi-layer Perceptron

✅ Universal approximation capability

❌ Black-box nature limits interpretability

❌ Requires careful tuning and significant compute

❌ Limited geometric transparency

Why GMC is superior for interpretable multimodal classification:

Transparent geometry: Each prediction can be explained by which local hyperplane was responsible

Predictable cost: Inference scales linearly with number of planes, not data size

Discriminative training: Optimizes classification loss directly

Automatic structure discovery: No need to pre-specify cluster structure

Calibrated probabilities: Good uncertainty quantification out-of-the-box

### Additional context

Research Validation:

Comprehensive evaluation in attached paper across 8 datasets (4 synthetic, 4 real-world)

Consistently outperforms linear baselines by 10-40% on multimodal datasets

Competitive with RBF-SVM and Random Forests while being more interpretable

Single-digit microsecond inference time per example (Table 5 in paper)

Low expected calibration error (0.02-0.04 after temperature scaling)

Implementation Status:

✅ Complete scikit-learn compatible implementation available

✅ Comprehensive test suite passing

✅ Documentation and usage examples

✅ Performance benchmarks vs. existing classifiers

✅ Research paper with theoretical foundations

Code Repository:
https://github.com/Abitsfhuusrtyt/-Geometric-Mixture-Classifier-GMC---A-Discriminative-Per-Class-Mixture-of-Hyperplanes

Key Technical Innovations:

Gate-free mixture of experts: Responsibilities emerge naturally from soft-OR pooling

Geometry-aware initialization: k-means seeding prevents poor local optima

Usage-aware regularization: Automatically prunes unused experts

Automatic model selection: Silhouette scoring determines optimal planes per class

Unified linear/nonlinear framework: RFF extension maintains interpretability

Community Need:

Growing demand for interpretable ML in healthcare, finance, and regulated industries

Educational value for teaching decision boundaries and mixture models

Gap in scikit-learn's model portfolio for transparent multimodal classification

Willingness to Maintain:
I am committed to:

Addressing all code review feedback

Maintaining the implementation long-term

Adding additional features and optimizations

Providing user support and documentation updates

The Geometric Mixture Classifier represents a novel contribution to the interpretable ML landscape and would be a valuable addition to scikit-learn's classifier ecosystem.

[2509.16769v1.pdf](https://github.com/user-attachments/files/23139574/2509.16769v1.pdf)

COMMENT:
Can I work on adding this classifier @lesteve ?

Labels: New Feature, Needs Triage