Repository: googleapis/google-api-python-client

Issue:
HttpError 400 on uploading 0 bytes files, 416 on downloading 0 bytes files
It seems like the bug #241 and @Aran-Fey's problem happened again when i trying to download and upload 0 bytes file using google drive api.
    
```py
    credentials = google.oauth2.credentials.Credentials(**token)
    services_account = googleapiclient.discovery.build(API_SERVICE_NAME, API_VERSION, credentials=credentials)
    fd= BytesIO()
    media = googleapiclient.http.MediaIoBaseUpload(fd=fd, chunksize=128, mimetype='text/plain', resumable=True)
    body = {'name' : 'foo.txt'}
    services_account.files().insert(body=body, media_body=media, fields='id').execute()
```
throws 
```
    `googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v2/files?fields=id&alt=json&uploadType=resumable
returned "Bad Request">`
```



Discussion:
Hi @9kmmr, could you let us know which version of the API (V2 or V3) you are using?
@busunkim96 The error states Drive V2. (It's generally recommended to use V3)

@9kmmr It looks like the API request is bad. Here are more samples for uploading files:
https://developers.google.com/drive/api/v2/manage-uploads
@busunkim96 It is V2
@grant I tried with no-empty file and it worked
@9kmmr Is using Drive V3 API possible in your environment?
@9kmmr Please re-open this bug if it's still an issue for you.
same error, using python. V3 of the drives, python 3.8 and windows 64 bit
I'm still seeing this issue with google-api-python-client 1.7.10. Example code can be found [here in the Forseti repo](https://github.com/forseti-security/forseti-security/blob/master/google/cloud/forseti/common/gcp_api/storage.py#L326). A `googleapiclient.errors.HttpError` 416 exception is raised on the call to next_chunk() when downloading a 0 byte file. I see several closed issues related to this issue. What's the best way to fix this?
Well, in fact I didn't had to deal with that. For my concrete example, I've decided to avoid zero size files and that's all.
 
 
  El mar., ene. 14, 2020 a 23:27, Gregg Kowalski<notifications@github.com> escribió:   
I'm still seeing this issue with google-api-python-client 1.7.10. Example code can be found here in the Forseti repo. A googleapiclient.errors.HttpError 416 exception is raised on the call to next_chunk() when downloading a 0 byte file. I see several closed issues related to this issue. What's the best way to fix this?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
  

Unfortunately there are cases when we are exporting data from GCP to GCS, and some times those files are 0 bytes in size so it's not something we can avoid.
We are running into this as well. Ignoring 416 does the job, but looks like a hack. We are using v2 though (google-api-python-client-1.11.0), as we can't migrate easily to v3. @SurferJeffAtGoogle Looks like other people are having this issue as well, should we reopen this issue? Or is it fixed in v3 and older ones won't be fixed?
We confirmed that other clients fetch 0-bytes files without error, so this must be an issue with the python client.
The initial `POST` to begin the resumable upload succeeds. The `PUT` fails with invalid argument, and the content-range seems the most suspect at the moment `content-range: bytes 0--1/0`. 

```py
content_range = "bytes %d-%d/%s"  % (self.resumable_progress, chunk_end, size)
```

```
send: b'PUT /upload/drive/v2/files?fields=id&alt=json&uploadType=resumable&upload_id=ABg5....HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-range: bytes 0--1/0\r\ncontent-length: 0\r\nauthorization: Bearer ya...\r\nuser-agent: Python-httplib2/0.18.1 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
sendIng a read()able
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: text/plain; charset=utf-8
header: X-GUploader-UploadID: ABg5...
header: Content-Length: 37
header: Date: Tue, 20 Oct 2020 00:34:55 GMT
header: Server: UploadServer
Traceback (most recent call last):
  File "zero_byte_file.py", line 49, in <module>
    main()
  File "zero_byte_file.py", line 45, in main
    service.files().insert(body=body, media_body=media, fields="id").execute()
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/http.py", line 871, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/http.py", line 1061, in next_chunk
    return self._process_response(resp, content)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/http.py", line 1092, in _process_response
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v2/files?fields=id&alt=json&uploadType=resumable returned "Bad Request">
(env) busunkim@busunkim:~/bugs/api-client-638$ 
```

Resolving patch:
diff --git a/googleapiclient/http.py b/googleapiclient/http.py
--- a/googleapiclient/http.py
+++ b/googleapiclient/http.py
@@ -1026,13 +1026,17 @@ def next_chunk(self, http=None, num_retries=0):
             chunk_end = self.resumable_progress + len(data) - 1
 
         headers = {
-            "Content-Range": "bytes %d-%d/%s"
-            % (self.resumable_progress, chunk_end, size),
             # Must set the content-length header here because httplib can't
             # calculate the size when working with _StreamSlice.
             "Content-Length": str(chunk_end - self.resumable_progress + 1),
         }
 
+        # An empty file results in chunk_end = -1 and size = 0
+        # sending "bytes 0--1/0" results in an invalid request
+        # Only add header "Content-Range" if chunk_end != -1
+        if chunk_end != -1:
+            headers["Content-Range"] = "bytes %d-%d/%s" % (self.resumable_progress, chunk_end, size)
+
         for retry_num in range(num_retries + 1):
             if retry_num > 0:
                 self._sleep(self._rand() * 2 ** retry_num)


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -443,6 +443,33 @@ def test_media_io_base_next_chunk_no_retry_403_not_configured(self):
         request._sleep.assert_not_called()
 
 
+    def test_media_io_base_empty_file(self):
+        fd = BytesIO()
+        upload = MediaIoBaseUpload(
+            fd=fd, mimetype="image/png", chunksize=500, resumable=True
+        )
+
+        http = HttpMockSequence(
+            [
+                ({"status": "200", "location": "https://www.googleapis.com/someapi/v1/upload?foo=bar"}, "{}"),
+                ({"status": "200", "location": "https://www.googleapis.com/someapi/v1/upload?foo=bar"}, "{}")
+            ]
+        )
+
+        model = JsonModel()
+        uri = u"https://www.googleapis.com/someapi/v1/upload/?foo=bar"
+        method = u"POST"
+        request = HttpRequest(
+            http, model.response, uri, method=method, headers={}, resumable=upload
+        )
+
+        request.execute()
+
+        # Check that "Content-Range" header is not set in the PUT request
+        self.assertTrue("Content-Range" not in http.request_sequence[-1][-1])
+        self.assertEqual("0", http.request_sequence[-1][-1]["Content-Length"])
+
+
 class TestMediaIoBaseDownload(unittest.TestCase):
     def setUp(self):
         http = HttpMock(datafile("zoo.json"), {"status": "200"})