YXNXNXN/Yotam_Axen_Models

Presentation Video - https://youtu.be/gymV9vCqlwo

Hospital Length of Stay Prediction (Diabetes 130-US Hospitals) 🏥📊

This project predicts hospital length of stay for diabetic patients using demographics, diagnoses, procedures, medications, feature engineering, clustering, regression models, and classification models.

It is based on the Diabetes 130-US Hospitals (1999–2008) dataset from Kaggle.

1. Dataset Overview & Goal

The goal of this project is to understand which patient and hospital features are associated with how long a patient stays in hospital, and to build models that can predict length of stay early in the admission process.

The task is approached in two stages:

1. Regression – predict the exact number of days (time_in_hospital).

2. Classification – predict whether a stay will be:
   • Short (1–3 days)
   • Medium (4–7 days)
   • Long (8–14 days)

Special focus is placed on the long-stay group, which is clinically important but under-represented.

Regression target: time_in_hospital (integer days)
Classification target: stay_length_class (0 = short, 1 = medium, 2 = long)

---

2. Dataset Description

Source: Diabetes 130-US hospitals for years 1999–2008 (Kaggle)
Size: ~100,000 encounters, dozens of features

Includes:

• Demographics: race, gender, age
• Administrative data: admission_type_id, admission_source_id, discharge_disposition_id
• Clinical usage: time_in_hospital, num_lab_procedures, num_procedures, number_diagnoses
• Medications: multiple drug columns (No, Steady, Up, Down)
• Outcome: readmitted

Removed upfront:

• Technical IDs: encounter_id, patient_nbr
• High-missing / low-utility fields: weight, payer_code, medical_specialty, max_glu_serum, A1Cresult

---

3. Exploratory Data Analysis (EDA)

Cleaning

• Replaced "?" with NaN in categorical columns.
• Imputed missing values with mode per column.
• Confirmed numeric ranges and basic distributions with df.describe() and df.info().

Target & Age Distributions

• Age shows clear peaks in older brackets (50–80), typical of a chronic disease population.
• time_in_hospital ranges from 1–14 days, skewed toward shorter stays → a natural motivation for short/medium/long buckets.

Gender & Resource Use

• Boxplots of time_in_hospital, num_lab_procedures, num_procedures, and num_medications by gender.
• Countplots of race and age split by gender.
• Insight: resource use and length of stay are broadly similar across genders, with modest distribution shifts.

Admission & Diagnosis Patterns

• Percentage bar plots for:
  • admission_type_id
  • discharge_disposition_id
  • time_in_hospital
  • number_diagnoses
• A small number of admission/discharge codes dominate, indicating strong potential signal around how and where patients come from/go to.

Correlations

• time_in_hospital correlates most with:
  • num_medications
  • num_lab_procedures
  • number_diagnoses
• No single feature explains everything; multiple weak–moderate relationships suggest the need for non-linear models and feature interactions.

EDA Summary

• The dataset is heterogeneous and imbalanced (especially across IDs and length-of-stay).
• Simple linear trends exist but are not strong enough alone → motivates more sophisticated modeling, feature engineering, and clustering.

---

4. Baseline Regression Model

Features

• Only numeric features (no one-hot encoding, no engineered ratios at this stage).

Model

• Linear Regression (scikit-learn)
• 80/20 train-test split

Performance (Baseline)

• MAE: relatively high
• RMSE: relatively high
• R²: modest (≈ 0.27)

Visualizations

• Scatter plot of Actual vs. Predicted shows wide dispersion around the diagonal.
• Residual plot reveals substantial unexplained variance.
• Takeaway: a plain linear model on raw numeric features is not enough.

---

5. Feature Engineering

To capture clinical and operational complexity, several features were engineered:

• Age Midpoint

  • age_mid: converts age ranges (e.g. [40-50)) into numeric midpoints.

• Visit Intensity

  • total_visits = number_outpatient + number_emergency + number_inpatient.

• Medication Dynamics

  • From medication columns with ≤4 categories:
    • med_changes = count of drugs with Up or Down.
    • num_meds_active = count of drugs not equal to No.

• Readmission & Admission

  • readmit_encoded = {NO: 0, >30: 1, <30: 2}.
  • is_emergency = 1 if admission_type_id == 1, else 0.

• Intensity Ratios
  (leakage-safe)

  • proc_med_ratio = num_procedures / (num_medications + 1)
  • lab_intensity = num_lab_procedures / (number_diagnoses + 1)

• Diagnosis Grouping

  • diag1_group, diag2_group, diag3_group = first character of each diagnosis code.

• Low-Variance Removal

  • Dropped columns with only a single unique value.

Encoding & Scaling

• Categorical columns → OneHotEncoder (handle_unknown='ignore').
• Combined numeric and one-hot features.
• Applied StandardScaler to create:
  • X_train_scaled_df
  • X_test_scaled_df

---

6. Clustering (Patient Profiles)

To incorporate latent structure, I introduced clustering on the scaled feature space.

BONUS: Optimal k with PCA + Silhouette / Elbow

I defined a helper function optimal_k_large() that:

• Subsamples rows (sample size set to 10,000 to conserve RAM).
• Optionally reduces dimensionality using PCA.
• Evaluates multiple values of k using either:
  • Silhouette score (k = 2…10)
  • Elbow method on inertia (k = 1…10)

This procedure was also run once on the full dataset and consistently suggested k = 3 clusters.

• Silhouette analysis (k = 2…10):
  

• Elbow method (k = 1…10):
  

Both curves point to a small optimal k, with a preference around k = 3, which was then used for K-Means.

K-Means Clustering

• Fitted KMeans(n_clusters=3) on X_train_scaled_df.
• Added two new features:
  • cluster_id – cluster assignment
  • cluster_distance – distance to nearest cluster center

PCA Cluster Visualization

• 2D PCA scatter plot colored by cluster_id.
• Clusters show visible separation, indicating coherent patient subgroups.

Cluster Profiles

• For each cluster, the top 10 features (by absolute mean) were plotted.
• Clusters vary by:
  • medication counts
  • lab procedure intensity
  • diagnosis groups
  • administrative codes

These cluster features later appear as non-trivial predictors in feature importance plots.

---

7. Improved Regression Models

Models Trained

Using engineered + clustered features (X_train_scaled_df):

• Enhanced Linear Regression
• RandomForestRegressor
• GradientBoostingRegressor
• SVR (on subset for speed)

Leakage & Fix

• An earlier engineered feature mistakenly divided by time_in_hospital, yielding an unrealistic R² ≈ 0.999.
• Once this leakage feature was removed and replaced with lab_intensity, performance dropped to a plausible level.

Final Regression Performance

Model	MAE	RMSE	R²
Gradient Boosting	~1.64	~2.22	~0.43
Linear Regression	~1.70	~2.28	~0.40
Random Forest	~1.71	~2.31	~0.39
Baseline Linear Reg	~1.90	~2.52	~0.27

Winner: GradientBoostingRegressor

Regression Feature Importances

• Top predictors include:
  • num_medications, num_lab_procedures, discharge_disposition_id
  • Diagnosis dummies (e.g., diag_1_414, diag_1_V57)
  • proc_med_ratio, age_mid, number_diagnoses
  • Clustering features (cluster_distance)

Saved Model

• best_model_regression.pkl

---

8. Regression to Classification

To make predictions more actionable, time_in_hospital was converted into 3 classes:

• Bins: [0, 3, 7, 14]
• Labels:
  • 0 → short stay (1–3 days)
  • 1 → medium stay (4–7 days)
  • 2 → long stay (8–14 days)

Class Distribution

• Short (0): ~48–50%
• Medium (1): ~36–37%
• Long (2): ~15%

Imbalance: long stays are under-represented (~15%) but still numerous → accuracy can be misleading.
Metric choice: macro F1, with particular attention to recall/F1 of class 2 (long stay), since missing long stays is clinically costly.

---

9. Classification Models

Models Evaluated

All trained on X_train_scaled_df / X_test_scaled_df with target stay_length_class:

• Logistic Regression
  • Multinomial, class_weight='balanced'
• RandomForestClassifier
  • n_estimators=200, class_weight='balanced'
• HistGradientBoostingClassifier

Performance Summary

Model	Macro F1	Notes
HistGradientBoosting	~0.58	Best overall & best for class 2
Logistic Regression	~0.53	Strong baseline
Random Forest Classifier	~0.51	Good but weaker on long stays

Key Takeaways

• Best model: HistGradientBoostingClassifier
• It achieves:
  • Highest macro F1
  • Strong accuracy (~0.63)
  • Best F1 for the long-stay class, balancing precision and recall.

Confusion Matrix & Reports

• Short and medium stays are predicted reasonably well.
• Long stays remain more challenging but are detected better than with other models.

Classification Feature Importances

• Similar to regression:
  • num_medications, num_lab_procedures, discharge_disposition_id
  • Key diagnosis dummies, proc_med_ratio, admission_type_id
  • cluster_distance, number_diagnoses, age_mid, med_changes

Saved Model

• best_model_classification.pkl

11. Challenges & Lessons Learned

• Data Leakage:
  A single engineered feature that used the target in its formula inflated R² to 0.999. Detecting and removing it was a critical lesson in validating feature design.

• Imbalanced Classes:
  Accuracy alone looked “good” while long stays were being predicted poorly. Macro F1 and per-class metrics told a much truer story.

• Memory Limits in Colab:
  One-hot encoding, scaling, clustering, and tuning on a large dataset is RAM-heavy. I had to:

  • Delete intermediate objects (del ...) and call gc.collect().
  • Sometimes train/tune on subsets (e.g., 20k rows).
  • Keep hyperparameter grids compact.

• Clustering as Features:
  Unsupervised clusters (cluster_id, cluster_distance) carried useful signal and showed up in feature importance, highlighting the value of mixing supervised and unsupervised steps.

• Regression vs Classification Trade-off:
  Regression provides fine-grained estimates but is noisier; classification yields clearer, more actionable categories (short/medium/long) at the cost of granularity.