TicassloThang/multi-floor-maze-rl

Multi-Floor Maze RL Agents

Trained agents for a custom 3-floor maze environment (keys, doors, monsters, bombs, holes, a shield and a hammer), from the course project multi-floor-maze-rl (Reinforcement Learning, HCM-UTE, team of 3, 2026). There are three kinds of agents: first-visit Monte Carlo control and Q-learning (tables), and Double DQN (PyTorch network).

These agents only work with the environment and the 19-value observation defined in the GitHub repo (src/maze_env.py).

Files

Path	Agent	Trained on
MC1/mctable_final.pkl	Monte Carlo, phase 1	3 floors, fixed map, no soft walls or bombs
MC2/mctable_final.pkl	Monte Carlo, phase 2	3 floors, fixed map, 5 soft walls, up to 2 bombs
MC3/mctable_final.pkl	Monte Carlo, phase 3	3 floors, random map each episode
QL1/best_qtable.pkl	Q-learning, phase 1	Same as MC1
QL2/best_qtable.pkl	Q-learning, phase 2	Same as MC2
QL3/qtable_final.pkl	Q-learning, phase 3	Same as MC3
DQN1/best_dqn.pt	Double DQN, phase 1	1 floor, fixed map, max 100 steps
DQN2/best_dqn.pt	Double DQN, phase 2	2 floors, fixed map, max 200 steps
DQN3/best_dqn.pt	Double DQN, phase 3	3 floors, fixed map, max 250 steps
DQN4/best_dqn.pt	Double DQN, phase 4	3 floors, random map each episode, max 250 steps

Each phase continued training from the previous phase's agent. The tabular agents ran 50,000 episodes per phase and the DQN agents 20,000. All runs used seed 42.

File formats

• Q-learning (.pkl): a Python dict {(state, action): q_value}, where state is the 19-value tuple from MazeEnv and action is 0 to 8.
• Monte Carlo (.pkl): a dict {"version": 1, "default": float, "sums": {(state, action): sum_of_returns}, "nCounts": {(state, action): count}}. Q(s, a) is sums / nCounts.
• Double DQN (.pt): a dict with model_state_dict (online network), target_state_dict, optimizer_state_dict (Adam), epsilon and train_steps. The network is QNetworkMLP from the repo with hidden_dim=128: three branches (12 local cells one-hot over 13 cell types = 156 inputs, 4 status values, 3 context values), each Linear + ReLU, joined into a linear layer with 9 outputs.

The .pkl files are Python pickles, so only load them if you trust the source.

How to use

The repo's Zeppelin notebook loads these files from a FinalModel/ folder:

hf download TicassloThang/multi-floor-maze-rl --local-dir FinalModel

Then the notebook's evaluation cells use, for example, loadQTable("FinalModel/QL2/best_qtable.pkl"), load_mctable("FinalModel/MC2/mctable_final.pkl") and loadDQNCheckpoint(dqn_agent, "FinalModel/DQN3/best_dqn.pt").

Results

Success rate (episodes that reach the goal) near the end of training, with exploration still on:

Phase	Monte Carlo	Q-learning	Double DQN
1	0%	80 to 85%	89%
2	0%	about 50%	87%
3	0%	0%	33% (best 41%)
4			0%

Greedy evaluation in the notebook (1,000 episodes, 3-floor fixed map): Q-learning phase 2 reached the goal in 703 episodes (70.3%), Monte Carlo phase 2 in 0, and Double DQN phase 3 in 0. The DQN result does not match its training log, and the evaluation map may differ from the map it was trained on; we did not confirm the cause.

Limitations

• Tabular agents only know states they have seen. On random maps the Q-table grew to about 3.9 million entries and did not learn to reach the goal.
• The observation only covers 12 nearby cells, so far away keys, doors and stairs are not visible to the agent.
• None of the agents solves random 3-floor maps.

Links

• Code: https://github.com/Ticasslo/multi-floor-maze-rl
• Report (Vietnamese): https://github.com/Ticasslo/multi-floor-maze-rl/blob/main/Nhom08_MultiFloorMaze_BaoCao.pdf

Team

Huỳnh Ngọc Thắng (team lead), Huỳnh Thanh Nhân, Trương Tấn Sang