Athmapriyan/RLT

The Effect of Optimism in the Face of Uncertainty

A Study of Optimistic Initialization in Multi-Armed Bandits

Project Structure

bandit-project/
├── index.html       ← Main page (hero, theory, simulator, insights)
├── style.css        ← All styles (dark theme, responsive layout)
├── simulation.js    ← Core bandit engine (BanditEnv, EpsilonGreedyAgent, runSimulation)
└── main.js          ← UI logic (Plotly charts, sliders, hero canvas)

How to Run

Just open index.html in any modern browser. No build step, no server needed.

# Option 1: Direct
open index.html

# Option 2: Local server (recommended)
python3 -m http.server 8080
# then visit http://localhost:8080

Algorithm

ε-Greedy with three initialization strategies:

Strategy	Initial Q₀	Exploration drive
Optimistic	+5 to +20	High (natural)
Normal	0	Medium (ε only)
Pessimistic	−5 to −20	Low

Update rule: Q(a) ← Q(a) + [R − Q(a)] / N(a) (incremental sample mean)

Parameters

Parameter	Description	Range
K (arms)	Number of bandit arms	2 – 10
Iterations	Steps per simulation run	100 – 3000
ε (epsilon)	Exploration probability	0.00 – 0.50
Runs	Repetitions averaged together	1 – 50
σ (noise)	Reward standard deviation	0.1 – 3.0
Opt Q₀	Optimistic initial value	+1 to +20
Pess Q₀	Pessimistic initial value	−1 to −20

Metrics Tracked

• Average reward over iterations (smoothed with rolling window)
• % Optimal action selected — how fast the agent finds the best arm
• Cumulative reward — total reward accumulated
• Arm selection heatmap — which arms each strategy visited most

Key Takeaways

1. Optimism = free exploration — even with ε=0, optimistic agents explore all arms
2. Faster convergence — optimistic agents discover the best arm sooner
3. Pessimism is costly — premature commitment to a suboptimal arm
4. ε still matters — for non-stationary environments, keep ε > 0

sanjay

• Plotly.js — interactive charts
• Google Fonts — Syne + Space Mono