Bring middleware used closer to what is provided by ActionDispatch::Reloader

This prevents reloading of app code on every request.