ali591195/apollo-rl-ems

Apollo: Reinforcement Learning Energy Management System

PPO agents trained to manage energy in a simulated solar-powered microgrid — deciding, hour by hour, whether to charge a battery, discharge it, sit idle, or (grid variant) draw from the utility grid. Trained with Stable-Baselines3 inside a custom Gymnasium environment.

This repo hosts two model variants trained on two different environment versions. The environment code is not included in this repo — see Environment & Code below.

Model Variants

Variant	File	Environment	Action Space	Observation Space	Steps Trained
Base	base/ppo_microgrid_checkpoint_100k.zip	Standalone (no grid)	3 discrete: charge / discharge / idle	4-dim: [solar, battery, demand, hour]	125,000 cumulative
Grid	grid/ppo_microgrid_checkpoint_grid_150k.zip	Grid-enabled, dynamic pricing, stochastic blackouts	4 discrete: charge / discharge / idle / use grid	6-dim: [solar, battery, demand, hour, grid_price, load_shed]	220,000 cumulative

⚠️ Important — Evaluation Checkpoint Note

The formal, statistically-evaluated results below (50-seed comparison against Rule-Based and Random baselines) were measured on earlier checkpoints than the ones hosted in this repo:

• Base results correspond to ppo_microgrid_checkpoint_20k (25,000 cumulative steps)
• Grid results correspond to ppo_microgrid_checkpoint_grid_60k (70,000 cumulative steps)

The checkpoints uploaded here (_100k / _grid_150k) were trained further after that evaluation but have not been independently re-run through the full 50-seed evaluation protocol. Training diagnostics observed diminishing returns beyond ~50K additional steps (only ~7-8% further improvement), so performance is expected to be comparable or modestly better — but this has not been formally confirmed for these exact checkpoints. If you need numbers that are guaranteed to match a specific file 1:1, use the _20k / _grid_60k checkpoints from the GitHub repo instead.

Evaluation Results (measured on _20k / _grid_60k — see note above)

Base Environment

Policy	Avg Reward	Avg Unmet Demand	Avg Waste	Avg Penalized Waste	Reward Std Dev
PPO	-10.709	2.118	2.096	0.001	1.724
Random	-15.930	2.340	2.204	1.744	3.916
Rule-Based	-23.284	1.769	1.872	0.000	4.107

Grid-Enabled Environment

Policy	Avg Reward	Avg Unmet Demand	Avg Waste	Avg Penalized Waste	Reward Std Dev
PPO	-3.831	0.407	2.576	0.109	2.251
Rule-Based	-4.460	0.488	2.297	0.000	3.157
Random	-14.203	2.069	2.801	0.487	3.711

PPO outperforms Rule-Based on reward by ~54% and Random by ~33% in the base environment, and wins on both reward and unmet demand in the harder grid-enabled environment, while remaining the most stable policy (lowest reward variance) in both.

Environment & Code

This model depends on a custom MicrogridEnv (Gymnasium) that is not bundled in this repo. Get it from the source GitHub repository:

github.com/ali591195/apollo-rl-ems

That repo contains the full environment (src/env.py, src/reward.py, src/simulation/), the original training notebook, and the complete write-up this model card summarizes.

How to Use

git clone https://github.com/ali591195/apollo-rl-ems
cd apollo-rl-ems
pip install -r requirements.txt

from huggingface_hub import hf_hub_download
from stable_baselines3 import PPO
from src.env import MicrogridEnv
from src.reward import compute_reward

# Download the desired checkpoint from this repo
checkpoint_path = hf_hub_download(
    repo_id="ali591195/apollo-rl-ems",   # update to your actual HF repo id
    filename="grid/ppo_microgrid_checkpoint_grid_150k.zip"
)

# grid_mode must match the variant you downloaded
env = MicrogridEnv(compute_reward, grid_mode=True)
model = PPO.load(checkpoint_path, env=env)

obs, _ = env.reset()
for _ in range(24):  # one simulated day
    action, _ = model.predict(obs)
    obs, reward, done, truncated, info = env.step(action)

Match grid_mode to the variant you load: False for base/, True for grid/. Base and grid checkpoints were trained on different observation/action space shapes and are not interchangeable.

Training Details

• Algorithm: PPO (Stable-Baselines3 2.9.0)
• Policy: MLP, discrete action space
• Training schedule: staged → diagnostic (2-5K steps) → early validation (10-20K) → extended (50K+), checkpointed after each phase
• Base environment: 5K → 20K → 100K step phases (125,000 cumulative)
• Grid environment: 10K → 60K → 150K step phases (220,000 cumulative)

Reward Function

reward = -(5.0 × unmet_demand)
         -(0.2 × wasted_energy, only if battery had room to store it)
         -(2.0 × battery_misuse, running battery near-empty before low-solar periods)
         -(1.5 × grid_price, only if grid action used and grid is online)

Limitations

• The learned policy depends on the chosen reward function. Although the reward weights were iteratively refined using domain reasoning and empirical evaluation, alternative formulations or weightings may produce different optimization strategies.
• In the base environment, PPO accepts ~20% more unmet demand than the Rule-Based baseline in exchange for better overall reward, a real trade-off, not a strict win on every axis.
• Solar generation, demand, and grid pricing are synthetic models, not real historical weather/tariff data.
• No temperature effects modeled.
• See the GitHub repo for the full limitations and future-work discussion (a storage-scaling and action-awareness upgrade is in progress).

License

Apache-2.0

Author

Ali Hassan — ML Developer/Engineer, Lahore, Pakistan · LinkedIn · GitHub · Hugging Face