Try to ensure that rake tasks are only loaded once

No tests, possibly brittle assertion here, but this should be more
helpful than not. If anything, I wanted to start a discussion around
what the best solution would be here, rather than letting users continue
to run into this problem time and time again.