Wrote up the Taxi domain, but made continuous and actually including the fuel (which some people drop for fun), with additive Gaussian noise, and allowed to change the size of the gridworld and where the walls and landmarks (and number of both) are. Now I have to fix the existing agents to actually take discrete observations.