If the key is not in the defaults when adding a handler set the default not the config