Fix possible condition where ResponseHandler doesn't get called

If the stream responds really fast, the ResponseHandler won't be set to be executed until too late.