Remove debug only handlers from loggers when removing  the handler