Repository: googleapis/google-api-python-client

Issue:
HttpError 400 on uploading 0 bytes files, 416 on downloading 0 bytes files
It seems like the bug #241 and @Aran-Fey's problem happened again when i trying to download and upload 0 bytes file using google drive api.
    
```py
    credentials = google.oauth2.credentials.Credentials(**token)
    services_account = googleapiclient.discovery.build(API_SERVICE_NAME, API_VERSION, credentials=credentials)
    fd= BytesIO()
    media = googleapiclient.http.MediaIoBaseUpload(fd=fd, chunksize=128, mimetype='text/plain', resumable=True)
    body = {'name' : 'foo.txt'}
    services_account.files().insert(body=body, media_body=media, fields='id').execute()
```
throws 
```
    `googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v2/files?fields=id&alt=json&uploadType=resumable
returned "Bad Request">`
```



Discussion:
Hi @9kmmr, could you let us know which version of the API (V2 or V3) you are using?
@busunkim96 The error states Drive V2. (It's generally recommended to use V3)

@9kmmr It looks like the API request is bad. Here are more samples for uploading files:
https://developers.google.com/drive/api/v2/manage-uploads
@busunkim96 It is V2
@grant I tried with no-empty file and it worked
@9kmmr Is using Drive V3 API possible in your environment?
@9kmmr Please re-open this bug if it's still an issue for you.
same error, using python. V3 of the drives, python 3.8 and windows 64 bit
I'm still seeing this issue with google-api-python-client 1.7.10. Example code can be found [here in the Forseti repo](https://github.com/forseti-security/forseti-security/blob/master/google/cloud/forseti/common/gcp_api/storage.py#L326). A `googleapiclient.errors.HttpError` 416 exception is raised on the call to next_chunk() when downloading a 0 byte file. I see several closed issues related to this issue. What's the best way to fix this?
Well, in fact I didn't had to deal with that. For my concrete example, I've decided to avoid zero size files and that's all.
 
 
  El mar., ene. 14, 2020 a 23:27, Gregg Kowalski<notifications@github.com> escribió:   
I'm still seeing this issue with google-api-python-client 1.7.10. Example code can be found here in the Forseti repo. A googleapiclient.errors.HttpError 416 exception is raised on the call to next_chunk() when downloading a 0 byte file. I see several closed issues related to this issue. What's the best way to fix this?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
  

Unfortunately there are cases when we are exporting data from GCP to GCS, and some times those files are 0 bytes in size so it's not something we can avoid.
We are running into this as well. Ignoring 416 does the job, but looks like a hack. We are using v2 though (google-api-python-client-1.11.0), as we can't migrate easily to v3. @SurferJeffAtGoogle Looks like other people are having this issue as well, should we reopen this issue? Or is it fixed in v3 and older ones won't be fixed?
We confirmed that other clients fetch 0-bytes files without error, so this must be an issue with the python client.
The initial `POST` to begin the resumable upload succeeds. The `PUT` fails with invalid argument, and the content-range seems the most suspect at the moment `content-range: bytes 0--1/0`. 

```py
content_range = "bytes %d-%d/%s"  % (self.resumable_progress, chunk_end, size)
```

```
send: b'PUT /upload/drive/v2/files?fields=id&alt=json&uploadType=resumable&upload_id=ABg5....HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-range: bytes 0--1/0\r\ncontent-length: 0\r\nauthorization: Bearer ya...\r\nuser-agent: Python-httplib2/0.18.1 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
sendIng a read()able
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: text/plain; charset=utf-8
header: X-GUploader-UploadID: ABg5...
header: Content-Length: 37
header: Date: Tue, 20 Oct 2020 00:34:55 GMT
header: Server: UploadServer
Traceback (most recent call last):
  File "zero_byte_file.py", line 49, in <module>
    main()
  File "zero_byte_file.py", line 45, in main
    service.files().insert(body=body, media_body=media, fields="id").execute()
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/http.py", line 871, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/http.py", line 1061, in next_chunk
    return self._process_response(resp, content)
  File "/usr/local/google/home/busunkim/bugs/api-client-638/env/lib/python3.8/site-packages/googleapiclient/http.py", line 1092, in _process_response
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v2/files?fields=id&alt=json&uploadType=resumable returned "Bad Request">
(env) busunkim@busunkim:~/bugs/api-client-638$ 
```
The empty upload succeeds once the `Content-Range` header is removed from the `PUT` request. This is also what the Java client appears to do (https://github.com/googleapis/google-api-java-client/blob/326370e0ec4351aaea21da271692b2fb9585a28c/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpDownloader.java#L242-L255)
Thank you so much @busunkim96 ! :pray:
@busunkim96 @SurferJeffAtGoogle Upload works flawlessly now, but downloading still fails with 416 as discussed in this ticket, so looks like it shouldn't be closed. Or am I missing something? 

EDIT: here is the exc https://travis-ci.com/github/iterative/PyDrive2/jobs/403036538#L557
Ah, I missed the download part of this. Re-opening.

Resolving patch:
diff --git a/googleapiclient/http.py b/googleapiclient/http.py
--- a/googleapiclient/http.py
+++ b/googleapiclient/http.py
@@ -745,8 +745,16 @@ def next_chunk(self, num_retries=0):
             if self._total_size is None or self._progress == self._total_size:
                 self._done = True
             return MediaDownloadProgress(self._progress, self._total_size), self._done
-        else:
-            raise HttpError(resp, content, uri=self._uri)
+        elif resp.status == 416:
+            # 416 is Range Not Satisfiable
+            # This typically occurs with a zero byte file
+            content_range = resp["content-range"]
+            length = content_range.rsplit("/", 1)[1]
+            self._total_size = int(length)
+            if self._total_size == 0:
+                self._done = True
+                return MediaDownloadProgress(self._progress, self._total_size), self._done
+        raise HttpError(resp, content, uri=self._uri)
 
 
 class _StreamSlice(object):


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -672,6 +672,26 @@ def test_media_io_base_download_empty_file(self):
         self.assertEqual(0, download._total_size)
         self.assertEqual(0, status.progress())
 
+    def test_media_io_base_download_empty_file_416_response(self):
+        self.request.http = HttpMockSequence(
+            [({"status": "416", "content-range": "0-0/0"}, b"")]
+        )
+
+        download = MediaIoBaseDownload(fd=self.fd, request=self.request, chunksize=3)
+
+        self.assertEqual(self.fd, download._fd)
+        self.assertEqual(0, download._progress)
+        self.assertEqual(None, download._total_size)
+        self.assertEqual(False, download._done)
+        self.assertEqual(self.request.uri, download._uri)
+
+        status, done = download.next_chunk()
+
+        self.assertEqual(True, done)
+        self.assertEqual(0, download._progress)
+        self.assertEqual(0, download._total_size)
+        self.assertEqual(0, status.progress())
+
     def test_media_io_base_download_unknown_media_size(self):
         self.request.http = HttpMockSequence([({"status": "200"}, b"123")])