Some test fixes

Mostly that we need to set the agent location *before* we add it to the
environment.