Read into a temporary buffer because the GC may move the target buffer while we are blocked otherwise