Domain: Healthcare

You are the Principal Data Scientist at a municipal transportation authority conducting a **Fleet Decarbonization & Telematics Risk Audit** ahead of the city's annual carbon credit certification review. The engineering team extracted raw OBD-II sensor data from 180+ commercial fleet drivers into a single CSV (`fleet_telematics_raw.csv`), but the data pipeline was unreliable — there are case inconsistencies and whitespace artifacts in categorical fields, sensor errors producing physically impossible readings, and exact duplicate records injected by a faulty message queue.

The dataset is located at `/tmp/files/` with columns: `driver_id`, `vehicle_type`, `road_type`, `trip_distance_km`, `trip_duration_min`, `avg_speed_kmh`, `rpm_variation`, `harsh_braking_count`, `idling_time` (in minutes), `fuel_consumption` (L/100km), `acceleration_smoothness` (0–1 scale), `ambient_temperature_c`, `tire_pressure_psi`, `engine_load_pct`, `co2_emissions_gkm` (g/km), `eco_score` (0–100).

Canonical vehicle types: **Sedan, SUV, Truck, Hybrid, EV**. Canonical road types: **Urban, Highway, Rural, Mixed**.

Perform appropriate data cleaning and answer the following 10 questions using only physically valid, deduplicated records. Round all numeric answers to exactly **2 decimal places** unless otherwise specified.

---

**Q1.** How many clean, valid records remain in the dataset after preprocessing?

**Q2.** Compute the **Temperature-Adjusted Fuel Rate (TAFR)** for each trip:
$$TAFR = fuel\_consumption \times \left(1 + 0.02 \times |ambient\_temperature\_c - 21|\right)$$
What is the mean TAFR for **Truck** vehicles? (Round to **4 decimal places**)

**Q3.** Compute the **Driving Behavior Risk Index (DBRI)** for each trip:
$$DBRI = (harsh\_braking\_count \times 12) + \frac{rpm\_variation}{400} + (1 - acceleration\_smoothness) \times 30 - eco\_score \times 0.08$$
What is the **90th percentile** DBRI value for trips on **Urban** roads? (Round to **4 decimal places**)

**Q4.** Estimate the **total fleet-wide fuel wasted due to idling** in liters. Use vehicle-specific idling consumption rates (L/hr): Sedan = 0.8, SUV = 1.2, Truck = 1.8, Hybrid = 0.3, EV = 0.0.

**Q5.** Compute expected CO₂: $expected = fuel\_consumption \times 23.15$. Calculate residuals: $r = co2\_emissions\_gkm - expected$. A trip is **anomalous** if $|r| > 1.96 \times \sigma_r$ where $\sigma_r$ is the standard deviation of all residuals. What percentage of trips are anomalous?

**Q6.** Bin `avg_speed_kmh` into **5 equal-width bins**. For each bin, compute the mean `eco_score`. Report the **left edge** of the bin with the **highest mean eco_score**.

**Q7.** Run an OLS multiple linear regression: `eco_score` ~ `rpm_variation` + `harsh_braking_count` + `idling_time` + `fuel_consumption` + `acceleration_smoothness`. Report the **R² value** (round to **4 decimal places**) and the predictor with the **largest absolute t-statistic**.

**Q8.** Define a **Green Premium Score (GPS)**:
$$GPS = \frac{eco\_score}{100} \times acceleration\_smoothness \times \frac{100}{fuel\_consumption + 1} \times \frac{1}{1 + harsh\_braking\_count}$$
For drivers with **at least 3 trips** in the cleaned data, which `driver_id` has the **highest median GPS**? Report the driver_id and their median GPS value (round to **4 decimal places**).

**Q9.** Using the DBRI from Q3, segment all trips into three tiers: **Safe** (DBRI ≤ 30th percentile), **Moderate** (30th < DBRI ≤ 70th percentile), **Risky** (DBRI > 70th percentile). For the **Risky** tier only, what is the **Pearson correlation coefficient** between DBRI and `fuel_consumption`? (Round to **4 decimal places**)

**Q10.** For each `vehicle_type × road_type` combination, compute the **Carbon Intensity Index (CII)**:
$$CII = \frac{\text{mean}(co2\_emissions\_gkm)}{\text{mean}(eco\_score)}$$
Which combination has the **highest CII**? Report as `VehicleType-RoadType` and the CII value.


Grading rubric:
criterion: Does the response correctly identify that exactly 2338 clean records remain after preprocessing?
weight: 10
criterion: Is the mean Temperature-Adjusted Fuel Rate  for Truck vehicles reported as 9.9977 ?
weight: 10
criterion: Is the 90th percentile of the Driving Behavior Risk Index for Urban road trips reported as 72.1412?
weight: 10
criterion: Is the total fleet-wide idling fuel waste reported as 391.75 liters?
weight: 10
criterion: Is the CO₂ anomaly percentage reported as 4.83%?
weight: 10
criterion: Is the left edge of the speed bin with the highest mean eco_score reported as 26.64?
weight: 10
criterion: Do the response report the OLS regression R² as  0.9283?
weight: 10
criterion: Does DRV-0108 has a median GPS value reported as  2.1769?
weight: 10
criterion: Is the Pearson correlation coefficient between DBRI and fuel_consumption for the Risky tier  reported as 0.3906?
weight: 10
criterion: Do the vehicle_type × road_type combination with the highest Carbon Intensity Index (CII)  has  a CII value of 4.56?
weight: 10

Golden answer:
`================================================================================
EXPERT SOLUTION — Fleet Decarbonization & Telematics Risk Audit
================================================================================

import pandas as pd
import numpy as np
import statsmodels.api as sm

# ============================================================
# DATA LOADING
# ============================================================
df = pd.read_csv('fleet_telematics_raw.csv')
# Raw dataset: 2545 rows, 16 columns

# ============================================================
# STEP 1: DATA CLEANING
# ============================================================

# 1a. Remove exact duplicate rows (45 duplicates from faulty message queue)
df = df.drop_duplicates().reset_index(drop=True)
# After dedup: 2500 rows

# 1b. Standardize vehicle_type — strip whitespace, normalize case,
#     map to canonical forms: Sedan, SUV, Truck, Hybrid, EV
df['vehicle_type'] = df['vehicle_type'].str.strip().str.lower().map({
    'sedan': 'Sedan', 'suv': 'SUV', 'truck': 'Truck', 'hybrid': 'Hybrid', 'ev': 'EV'
})

# 1c. Standardize road_type — strip whitespace, normalize case,
#     map to canonical forms: Urban, Highway, Rural, Mixed
df['road_type'] = df['road_type'].str.strip().str.lower().map({
    'urban': 'Urban', 'highway': 'Highway', 'rural': 'Rural', 'mixed': 'Mixed'
})

# 1d. Remove physically invalid records:
#     - fuel_consumption <= 0 is physically impossible (66 rows)
#     - co2_emissions_gkm < 0 is a sensor error (97 rows; 1 overlaps with fuel)
df = df[df['fuel_consumption'] > 0].reset_index(drop=True)
df = df[df['co2_emissions_gkm'] >= 0].reset_index(drop=True)
# After all cleaning: 2338 rows remain

# ============================================================
# Q1: How many clean, valid records remain?
# ============================================================
# ANSWER: 2338

print(f"Q1: {len(df)} clean records")
# >>> Q1: 2338 clean records


# ============================================================
# Q2: Temperature-Adjusted Fuel Rate (TAFR) — Mean for Truck
# ============================================================
# TAFR = fuel_consumption × (1 + 0.02 × |ambient_temperature_c − 21|)

df['TAFR'] = df['fuel_consumption'] * (1 + 0.02 * np.abs(df['ambient_temperature_c'] - 21))
truck_mean_tafr = df[df['vehicle_type'] == 'Truck']['TAFR'].mean()

print(f"Q2: Mean TAFR for Truck = {truck_mean_tafr:.4f}")
# >>> Q2: Mean TAFR for Truck = 9.9977

# Breakdown by vehicle type:
# EV        9.9513
# Hybrid    9.6735
# SUV       9.7284
# Sedan     9.8805
# Truck     9.9977


# ============================================================
# Q3: Driving Behavior Risk Index (DBRI) — 90th percentile for Urban
# ============================================================
# DBRI = (harsh_braking_count × 12) + (rpm_variation / 400)
#      + (1 - acceleration_smoothness) × 30 - eco_score × 0.08

df['DBRI'] = (df['harsh_braking_count'] * 12) \
           + (df['rpm_variation'] / 400) \
           + ((1 - df['acceleration_smoothness']) * 30) \
           - (df['eco_score'] * 0.08)

urban_90th = df[df['road_type'] == 'Urban']['DBRI'].quantile(0.90)

print(f"Q3: 90th percentile DBRI for Urban = {urban_90th:.4f}")
# >>> Q3: 90th percentile DBRI for Urban = 72.1412

# Full DBRI percentile distribution for Urban trips:
#   10th: 15.3051
#   25th: 26.8963
#   50th: 41.7037
#   75th: 57.6821
#   90th: 72.1412


# ============================================================
# Q4: Total Fleet-Wide Idling Fuel Waste (liters)
# ============================================================
# CRITICAL: idling_time is recorded in MINUTES, rates are in L/hr.
# Must convert: idling_time_hours = idling_time / 60
# fuel_waste_per_trip = (idling_time / 60) × idling_rate

idling_rates = {'Sedan': 0.8, 'SUV': 1.2, 'Truck': 1.8, 'Hybrid': 0.3, 'EV': 0.0}
df['idling_rate'] = df['vehicle_type'].map(idling_rates)
df['idling_fuel_waste'] = (df['idling_time'] / 60) * df['idling_rate']
total_idling_waste = df['idling_fuel_waste'].sum()

print(f"Q4: Total idling fuel waste = {total_idling_waste:.2f} liters")
# >>> Q4: Total idling fuel waste = 391.75 liters

# Breakdown by vehicle type:
#   Sedan:  count=707, total_idle=8002.0 min → 133.37 hr × 0.8 = 106.69 L
#   SUV:    count=571, total_idle=6450.5 min → 107.51 hr × 1.2 = 129.01 L
#   Truck:  count=377, total_idle=4303.0 min → 71.72 hr × 1.8 = 129.09 L
#   Hybrid: count=467, total_idle=5390.5 min → 89.84 hr × 0.3 = 26.95 L
#   EV:     count=216, total_idle=2324.2 min → 38.74 hr × 0.0 = 0.00 L
#   TOTAL: 391.75 L

# NOTE: If someone forgets the minutes→hours conversion (/60),
# they would incorrectly get 23504.75 liters.


# ============================================================
# Q5: CO₂ Anomaly Detection
# ============================================================
# expected_co2 = fuel_consumption × 23.15
# residual r = co2_emissions_gkm − expected_co2
# Anomalous if |r| > 1.96 × σ_r

df['expected_co2'] = df['fuel_consumption'] * 23.15
df['residual'] = df['co2_emissions_gkm'] - df['expected_co2']
sigma_r = df['residual'].std()   # sample std (ddof=1), σ_r = 7.8656
anomaly_mask = np.abs(df['residual']) > 1.96 * sigma_r
anomaly_pct = anomaly_mask.sum() / len(df) * 100

print(f"Q5: Anomaly percentage = {anomaly_pct:.2f}%")
# >>> Q5: Anomaly percentage = 4.83%
# 113 anomalous trips out of 2338


# ============================================================
# Q6: Equal-Width Speed Bins — Mean eco_score per Bin
# ============================================================
# avg_speed_kmh range: [14.90, 73.60]
# 5 equal-width bins via pd.cut (bin width ≈ 11.74 km/h)

df['speed_bin'] = pd.cut(df['avg_speed_kmh'], bins=5)
bin_eco = df.groupby('speed_bin', observed=False)['eco_score'].mean()

print("Q6: Mean eco_score by speed bin:")
for idx, val in bin_eco.items():
    print(f"  {idx}: {val:.4f}")
best_bin = bin_eco.idxmax()

print(f"Q6 ANSWER: Left edge = {best_bin.left:.2f}")
# >>> Q6: Left edge = 26.64

# Bin breakdown:
#   (14.841, 26.64]:  mean_eco = 46.7790  (count 309)
#   (26.64, 38.38]:   mean_eco = 48.6084  (count 524) ← HIGHEST
#   (38.38, 50.12]:   mean_eco = 47.6659  (count 560)
#   (50.12, 61.86]:   mean_eco = 45.5694  (count 575)
#   (61.86, 73.6]:    mean_eco = 48.5746  (count 370)


# ============================================================
# Q7: OLS Multiple Linear Regression
# ============================================================
X = df[['rpm_variation', 'harsh_braking_count', 'idling_time',
        'fuel_consumption', 'acceleration_smoothness']]
y = df['eco_score']

X_const = sm.add_constant(X)
model = sm.OLS(y, X_const).fit()

print(f"Q7: R² = {model.rsquared:.4f}")
# >>> Q7: R² = 0.9283

# Coefficient table:
#   Variable                  Coef        t-stat       p-value
#   const                    109.93       118.13       0.0000
#   rpm_variation             -0.0096     -59.42       0.0000 ← LARGEST |t|
#   harsh_braking_count       -3.9525     -49.58       0.0000
#   idling_time               -0.5557     -25.57       0.0000
#   fuel_consumption          -4.8973     -55.69       0.0000
#   acceleration_smoothness   33.7732      46.84       0.0000

largest_t_var = model.tvalues[['rpm_variation', 'harsh_braking_count',
    'idling_time', 'fuel_consumption', 'acceleration_smoothness']].abs().idxmax()

print(f"Q7 ANSWER: R² = 0.9283, Predictor with largest |t| = {largest_t_var}")
# >>> Q7: R² = 0.9283, Predictor with largest |t| = rpm_variation


# ============================================================
# Q8: Green Premium Score — Driver with Highest Median GPS (≥3 trips)
# ============================================================
# GPS = (eco_score/100) × acceleration_smoothness × (100/(fuel_consumption+1)) × (1/(1+harsh_braking_count))

df['GPS'] = (df['eco_score'] / 100) * df['acceleration_smoothness'] \
          * (100 / (df['fuel_consumption'] + 1)) * (1 / (1 + df['harsh_braking_count']))

# Filter to drivers with at least 3 trips
driver_trips = df.groupby('driver_id').size()
drivers_3plus = driver_trips[driver_trips >= 3].index   # 180 drivers
df_3plus = df[df['driver_id'].isin(drivers_3plus)]

driver_median_gps = df_3plus.groupby('driver_id')['GPS'].median()
best_driver = driver_median_gps.idxmax()
best_gps = driver_median_gps.max()

print(f"Q8: Best driver = {best_driver}, median GPS = {best_gps:.4f}")
# >>> Q8: Best driver = DRV-0108, median GPS = 2.1769

# Top 5 drivers by median GPS:
#   DRV-0108    2.1769
#   DRV-0090    2.0287
#   DRV-0106    2.0063
#   DRV-0069    1.9771
#   DRV-0116    1.8666


# ============================================================
# Q9: DBRI Segmentation — Pearson Correlation for Risky Tier
# ============================================================
# Safe: DBRI ≤ 30th percentile
# Moderate: 30th < DBRI ≤ 70th percentile
# Risky: DBRI > 70th percentile

p30 = df['DBRI'].quantile(0.30)    # 29.7906
p70 = df['DBRI'].quantile(0.70)    # 54.8213

risky = df[df['DBRI'] > p70]       # 702 trips
corr_risky = risky['DBRI'].corr(risky['fuel_consumption'])

print(f"Q9: Pearson correlation (Risky tier) = {corr_risky:.4f}")
# >>> Q9: Pearson correlation (Risky tier) = 0.3906


# ============================================================
# Q10: Carbon Intensity Index (CII) by vehicle_type × road_type
# ============================================================
# CII = mean(co2_emissions_gkm) / mean(eco_score) for each combination

cii_table = df.groupby(['vehicle_type', 'road_type']).agg(
    mean_co2=('co2_emissions_gkm', 'mean'),
    mean_eco=('eco_score', 'mean'),
    count=('eco_score', 'size')
).reset_index()
cii_table['CII'] = cii_table['mean_co2'] / cii_table['mean_eco']
cii_table = cii_table.sort_values('CII', ascending=False)

worst = cii_table.iloc[0]
print(f"Q10: Highest CII = {worst['vehicle_type']}-{worst['road_type']}, CII = {worst['CII']:.2f}")
# >>> Q10: Highest CII = EV-Mixed, CII = 4.56

# Top 5 CII combinations:
#   EV-Mixed:       CII = 4.56  (mean_co2=194.76, mean_eco=42.73, n=49)
#   Hybrid-Rural:   CII = 4.26  (mean_co2=188.52, mean_eco=44.22, n=65)
#   Truck-Urban:    CII = 4.15  (mean_co2=189.97, mean_eco=45.82, n=139)
#   Sedan-Rural:    CII = 4.10  (mean_co2=187.79, mean_eco=45.82, n=98)
#   EV-Highway:     CII = 4.05  (mean_co2=186.95, mean_eco=46.14, n=57)


# ============================================================
# FINAL ANSWER SUMMARY
# ============================================================
# Q1:  2338 clean records
# Q2:  Mean TAFR for Truck = 9.9977
# Q3:  90th percentile DBRI for Urban = 72.1412
# Q4:  Total idling fuel waste = 391.75 liters
# Q5:  Anomaly percentage = 4.83%
# Q6:  Left edge of best bin = 26.64
# Q7:  R² = 0.9283, Predictor with largest |t| = rpm_variation
# Q8:  Best driver = DRV-0108, median GPS = 2.1769
# Q9:  Pearson correlation (Risky tier) = 0.3906
# Q10: Highest CII = EV-Mixed, CII = 4.56