Try to ensure that rake tasks are only loaded once

No tests, possibly brittle assertion here, but this should be more helpful than not. If anything, I wanted to start a discussion around what the best solution would be here, rather than letting users continue to run into this problem time and time again.