Shut down Play threads in dev mode to prevent memory leaks

See commit messages for more info. This is a big part of #1453, but I'm not closing that ticket yet because there's still another leak to resolve.