dnanexus/dx-toolkit — pull request #73 by kislyuk (2014-08-03 01:46:27+00:00)
Title: PTFM-12727, PTFM-3468 Re-fetch upload URL per file upload attempt

@psung to review. @dkillebrew interest.

- Introduce functionality into dxpy.DXHTTPRequest to accept a callback
for fetching the upload URL and headers.
- Refactor dxpy.DXFile.upload_part to use this functionality instead
of supplying the URL and headers directly.

This accomplishes two goals:
- A short-term workaround for
https://github.com/shazow/urllib3/issues/438, which causes the
Python client to treat any early server disconnect when sending a
large body as a transport error, and prevents it from recognizing
503 Retry-After (blocking PTFM-12727).
- A long-term fix for S3 direct upload (PTFM-3468), for which the
client needs to re-request the pre-authenticated URL before every
upload, since enough time may have passed since the last request to
invalidate the preauth timeout.