Do not release payload if we shouldn't retry the request

This _should_ fix "Attempted to read on closed stream" issues