Refactor issuing ConfigurationError on a 404
Disable the custom handler after done