Remove nulling out in agentsets

The `5.0.x` version of #401 (see discussion on the latter).

Maybe wait after 5.0.5 is released to the external world to merge, though.