Repository: googleapis/google-api-python-client

Issue:
Not possible to provide `http` parameter in `next_chunk()` with resumable upload? HttpError 400
Hi,

This ticket is probably pointless, but maybe it hilights the need of a better error message?

The following code works fine with chunk size larger than 1024*1024, but using a chunksize of 1024 ends in a HttpError 400. Why is that so?

```python
from google.oauth2 import service_account
import googleapiclient.discovery as gad
import googleapiclient.http as gah

# Run once at start.
creds = service_account.Credentials.from_service_account_file(creds).with_scopes(['https://www.googleapis.com/auth/drive'])
drive = gad.build('drive', 'v3', credentials=creds)

# Run in a loop / implement both thread-safety and caching in http object.
filename = 'my_file.txt'
cache_path='.cache'
# Using 1024 ends up in an error.
# Using 1024*1024 works fine
# Using default parameter also works fine, but is actually not a resumable upload.
media = gah.MediaFileUpload(file_name, chunksize=1024, resumable=True) # Here
file_metadata = {'name': file_name, 'parents': [folder_id]}
request = self.drive.files().create(body=file_metadata, media_body=media, fields='id')
auth_http = _auth.authorized_http(creds)
auth_http.cache = httplib2.FileCache(cache_path)
response = None
while response is None:
        status, response = request.next_chunk(num_retries=4, http=auth_http)
```
Here is the error message I get when `chunksize=1024`.
```python
HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v3/files?fields=id&alt=json&uploadType=resumable returned "Bad Request">
```
Please, why is that so?

I thank you for your support.
Have a good day,
Bests,


Discussion:
The docs mention that the chunk size needs to be a multiple of 256KB. I think that's what's happening here? https://github.com/googleapis/google-api-python-client/tree/master/docs/media.md#resumable-media-chunked-upload

> Note: Chunk size restriction: There are some chunk size restrictions based on the size of the file you are uploading. Files larger than 256 KB (256 * 1024 B) must have chunk sizes that are multiples of 256 KB. For files smaller than 256 KB, there are no restrictions. In either case, the final chunk has no limitations; you can simply transfer the remaining bytes.

This restriction doesn't seem to be mentioned in the [code](https://github.com/googleapis/google-api-python-client/blob/f3348f98bf91a88a28bf61b12b95e391cc3be1ff/googleapiclient/http.py) anywhere, which is a bit odd.


Hi @busunkim96, I think you are right. I did 2 more tests:
- one with `chunksize=256*1024` : works just fine
- one with `chunksize=128*1024` : same error messagee: `HttpError 400`

Should I leave the ticket open so that maybe you implement a check directly on `chunksize` with a more understandable error message in case it is not a multiple of 256KB?

I let you decide.
Have a good day, and thanks a lot for your support! You're great!

Resolving patch:
diff --git a/googleapiclient/errors.py b/googleapiclient/errors.py
--- a/googleapiclient/errors.py
+++ b/googleapiclient/errors.py
@@ -48,7 +48,11 @@ def _get_reason(self):
         """Calculate the reason for the error from the response content."""
         reason = self.resp.reason
         try:
-            data = json.loads(self.content.decode("utf-8"))
+            try:
+                data = json.loads(self.content.decode("utf-8"))
+            except json.JSONDecodeError:
+                # In case it is not json
+                data = self.content.decode("utf-8")
             if isinstance(data, dict):
                 reason = data["error"]["message"]
                 error_detail_keyword = next((kw for kw in ["detail", "details", "message"] if kw in data["error"]), "")
@@ -59,6 +63,8 @@ def _get_reason(self):
                 reason = first_error["error"]["message"]
                 if "details" in first_error["error"]:
                     self.error_details = first_error["error"]["details"]
+            else:
+                self.error_details = data
         except (ValueError, KeyError, TypeError):
             pass
         if reason is None:


Test patch:
diff --git a/tests/test_errors.py b/tests/test_errors.py
--- a/tests/test_errors.py
+++ b/tests/test_errors.py
@@ -94,7 +94,7 @@ def test_bad_json_body(self):
             b"{", {"status": "400", "content-type": "application/json"}, reason="Failed"
         )
         error = HttpError(resp, content)
-        self.assertEqual(str(error), '<HttpError 400 "Failed">')
+        self.assertEqual(str(error), '<HttpError 400 when requesting None returned "Failed". Details: "{">')
 
     def test_with_uri(self):
         """Test handling of passing in the request uri."""
@@ -106,7 +106,7 @@ def test_with_uri(self):
         error = HttpError(resp, content, uri="http://example.org")
         self.assertEqual(
             str(error),
-            '<HttpError 400 when requesting http://example.org returned "Failure">',
+            '<HttpError 400 when requesting http://example.org returned "Failure". Details: "{">',
         )
 
     def test_missing_message_json_body(self):
@@ -121,15 +121,15 @@ def test_missing_message_json_body(self):
 
     def test_non_json(self):
         """Test handling of non-JSON bodies"""
-        resp, content = fake_response(b"}NOT OK", {"status": "400"})
+        resp, content = fake_response(b"Invalid request", {"status": "400"})
         error = HttpError(resp, content)
-        self.assertEqual(str(error), '<HttpError 400 "Ok">')
+        self.assertEqual(str(error), '<HttpError 400 when requesting None returned "Ok". Details: "Invalid request">')
 
     def test_missing_reason(self):
         """Test an empty dict with a missing resp.reason."""
         resp, content = fake_response(b"}NOT OK", {"status": "400"}, reason=None)
         error = HttpError(resp, content)
-        self.assertEqual(str(error), '<HttpError 400 "">')
+        self.assertEqual(str(error), '<HttpError 400 when requesting None returned "". Details: "}NOT OK">')
 
     def test_error_detail_for_missing_message_in_error(self):
         """Test handling of data with missing 'details' or 'detail' element."""