Ignore NuSuchElementExceptions when removing channel handlers.