When an async callback receives an error make sure we only call SetException.