Repository: googleapis/google-api-python-client

Issue:
Content-ID header over 64 characters gets an extra newline; breaks batch requests
I'm working with batch requests, and adding custom request_id values. With a request_id value of length 28, the total length of the Content-ID value inside the `<` and `>` characters is 64.

Somehow, for >=64 characters, `"\n "` (including trailing space) gets added before the Content-ID value, but after the `"Content-ID: "`, resulting in a malformed Content-ID value which makes the round trip back (response takes the form `Content-ID: response- <uuid+request_id>` instead of `Content-ID: <response-uuid+request_id>`) and throws an exception (a BatchError for a malformed Content-ID value).

I've narrowed this down to this patch of googleapiclient/http.py starting at line 1234:

```
# encode the body: note that we can't use `as_string`, because
# it plays games with `From ` lines.
print(message) #no newline here
fp = StringIO()
g = Generator(fp, mangle_from_=False)
g.flatten(message, unixfrom=False)
body = fp.getvalue()
print(body,[body]) #newline appears here, printing a list shows newline as \n
```

This can be reproduced without requiring an authorised call, using the following test code:

```
from httplib2 import Http
from apiclient.discovery import build
directory = build('admin', 'directory_v1')

def errorTest(id_length):
    req_id = "abcdefghijklmnopqrstuvwxyz0123456789"
    try:
        batch_dir = directory.new_batch_http_request()
        batch_dir.add(directory.groups().list(), request_id=req_id[0:id_length])
        batch_dir.execute(http=Http()) #BatchError, unprintable (separate issue)
    except Exception as e:
        print(e.reason)
        raise

for i in [27,28]:
    print(i,"############################################",i) #readability
    try:
        errorTest(i)
    except AttributeError:
        #only occurs if call not authorised; BatchError occurs first and isn't caught
        pass
```

Batch request for group members results in <BatchError "Invalid value for Content-ID: ...
Occurs in line 1191 of `http.py` - exact message is:

`googleapiclient.errors.BatchError: <BatchError "Invalid value for Content-ID: response- <cf4d6eb8-8f95-4fe1-ad82-2252fa9e8197+my-group@mydomain.com>">

If I insert `header = header.replace('response- ','')` at line 1175 the issue goes away.
Content-ID header over 64 characters gets an extra newline; breaks batch requests
I'm working with batch requests, and adding custom request_id values. With a request_id value of length 28, the total length of the Content-ID value inside the `<` and `>` characters is 64.

Somehow, for >=64 characters, `"\n "` (including trailing space) gets added before the Content-ID value, but after the `"Content-ID: "`, resulting in a malformed Content-ID value which makes the round trip back (response takes the form `Content-ID: response- <uuid+request_id>` instead of `Content-ID: <response-uuid+request_id>`) and throws an exception (a BatchError for a malformed Content-ID value).

I've narrowed this down to this patch of googleapiclient/http.py starting at line 1234:

```
# encode the body: note that we can't use `as_string`, because
# it plays games with `From ` lines.
print(message) #no newline here
fp = StringIO()
g = Generator(fp, mangle_from_=False)
g.flatten(message, unixfrom=False)
body = fp.getvalue()
print(body,[body]) #newline appears here, printing a list shows newline as \n
```

This can be reproduced without requiring an authorised call, using the following test code:

```
from httplib2 import Http
from apiclient.discovery import build
directory = build('admin', 'directory_v1')

def errorTest(id_length):
    req_id = "abcdefghijklmnopqrstuvwxyz0123456789"
    try:
        batch_dir = directory.new_batch_http_request()
        batch_dir.add(directory.groups().list(), request_id=req_id[0:id_length])
        batch_dir.execute(http=Http()) #BatchError, unprintable (separate issue)
    except Exception as e:
        print(e.reason)
        raise

for i in [27,28]:
    print(i,"############################################",i) #readability
    try:
        errorTest(i)
    except AttributeError:
        #only occurs if call not authorised; BatchError occurs first and isn't caught
        pass
```



Discussion:
Thank you for the very clear report!

## By Design: Line Folding ([RFC2822](https://www.ietf.org/rfc/rfc2822.txt)) (Unrecognised by server)
I've found that this line-folding is by design (for lines with length >78), thanks to this [StackOverflow Question](https://stackoverflow.com/questions/9267848/unwanted-line-break-in-email-header-when-using-email-mime).

This ordinarily would not be an issue, but for the Content-ID header, it contains no spaces. As such, when the total length is over 78, line folding will occur at the first available whitespace character, which happens to be immediately after the colon, and the server does not recognise this as folding.

(I am unsure if folding is supposed to occur at the beginning of a header, as that does not appear to be explicitly described by RFC2822, but I would assume the implication is that folding may occur at the beginning of a header value.)

If I insert a space within the header value at an appropriate location, it folds there instead, and is unfolded correctly by the server, however the server does not re-fold it in the response (given an absolute maximum 998 character limit in RFC2822, this is irrelevant).

"Content-ID: <>" is 14 characters, leaving a maximum of 64 within the greater than/less than symbols (angle brackets) before folding will occur. The greater than/less than symbols have no effect on the folding (apart from altering the line length).

The folding only occurs in the MIMEMultipart object, not the MIMENonMultipart object - and only in Python3, not Python2.

## Possible Workaround

My proposal, if the server-side issue cannot be fixed, is to store the request_id values separately, and uniquely identify them with a base 10 integer in the actual request, as follows:

    request_id_lookup = { 0:"my_request_id", 1:"my_second_request" }
    #alternatively, a list, because they are sorted and integer-indexed
    request_id_lookup = ["my_request_id", "my_second_request"]

Then, the header would look something like `Content-ID: <uuid+1234>` - 2^64 is only 20 digits in base 10, which only pushes the line to 71 characters. As a matter of fact, the `order` list already behaves like `request_id_lookup`, so a simple `enumerate` call in creating the request is all that would be necessary (plus a lookup to decode it).

I am willing to write this workaround, but would like confirmation that this is the way to proceed, as it is a workaround and does not address the actual issue.
I'm currently reviewing ways to avoid the folding.
Hi @chrish10e 

Do you think you can provide a bit of code that replicates this error for us?
Yes here you go:

```
GROUPS_API = 'https://www.googleapis.com/auth/admin.directory.group.readonly'

# Create Service

store = file.Storage('my_store.json')
creds = store.get() 

if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('my_client_secrets.json', GROUPS_API)
    creds = tools.run_flow(flow, store)

service = build('admin', 'directory_v1', http=creds.authorize(Http()))

# Batch request members of groups

group_emails = ['groupemail1@mydomain.com', 'groupemail2@mydomain.com']

batch = service.new_batch_http_request()

for email in group_emails:
    batch.add(service.members().list(groupKey=email), request_id=email)

batch.execute()

```
Awaiting credentials from Google for GSuite so I can confirm the issue.
It happens to me too, I'm using this as a workaround:

```python
def fixed_header_to_id(self, header):
    # Workaround for https://github.com/google/google-api-python-client/issues/536
    header = header.replace('response- ', '')
    if header[0] != '<' or header[-1] != '>':
        raise BatchError("Invalid value for Content-ID: %s" % header)
    if '+' not in header:
        raise BatchError("Invalid value for Content-ID: %s" % header)
    base, id_ = header[1:-1].rsplit('+', 1)
    return unquote(id_)
```

```python
BatchHttpRequest._header_to_id = fixed_header_to_id
```
Assigned to @crwilcox at the behest of @JustinBeckwith


Thank you for the very clear report!

## By Design: Line Folding ([RFC2822](https://www.ietf.org/rfc/rfc2822.txt)) (Unrecognised by server)
I've found that this line-folding is by design (for lines with length >78), thanks to this [StackOverflow Question](https://stackoverflow.com/questions/9267848/unwanted-line-break-in-email-header-when-using-email-mime).

This ordinarily would not be an issue, but for the Content-ID header, it contains no spaces. As such, when the total length is over 78, line folding will occur at the first available whitespace character, which happens to be immediately after the colon, and the server does not recognise this as folding.

(I am unsure if folding is supposed to occur at the beginning of a header, as that does not appear to be explicitly described by RFC2822, but I would assume the implication is that folding may occur at the beginning of a header value.)

If I insert a space within the header value at an appropriate location, it folds there instead, and is unfolded correctly by the server, however the server does not re-fold it in the response (given an absolute maximum 998 character limit in RFC2822, this is irrelevant).

"Content-ID: <>" is 14 characters, leaving a maximum of 64 within the greater than/less than symbols (angle brackets) before folding will occur. The greater than/less than symbols have no effect on the folding (apart from altering the line length).

The folding only occurs in the MIMEMultipart object, not the MIMENonMultipart object - and only in Python3, not Python2.

## Possible Workaround

My proposal, if the server-side issue cannot be fixed, is to store the request_id values separately, and uniquely identify them with a base 10 integer in the actual request, as follows:

    request_id_lookup = { 0:"my_request_id", 1:"my_second_request" }
    #alternatively, a list, because they are sorted and integer-indexed
    request_id_lookup = ["my_request_id", "my_second_request"]

Then, the header would look something like `Content-ID: <uuid+1234>` - 2^64 is only 20 digits in base 10, which only pushes the line to 71 characters. As a matter of fact, the `order` list already behaves like `request_id_lookup`, so a simple `enumerate` call in creating the request is all that would be necessary (plus a lookup to decode it).

I am willing to write this workaround, but would like confirmation that this is the way to proceed, as it is a workaround and does not address the actual issue.
I'm currently reviewing ways to avoid the folding.

Resolving patch:
diff --git a/googleapiclient/http.py b/googleapiclient/http.py
--- a/googleapiclient/http.py
+++ b/googleapiclient/http.py
@@ -1171,7 +1171,10 @@ def _id_to_header(self, id_):
     if self._base_id is None:
       self._base_id = uuid.uuid4()
 
-    return '<%s+%s>' % (self._base_id, quote(id_))
+    # NB: we intentionally leave whitespace between base/id and '+', so RFC2822
+    # line folding works properly on Python 3; see
+    # https://github.com/google/google-api-python-client/issues/164
+    return '<%s + %s>' % (self._base_id, quote(id_))
 
   def _header_to_id(self, header):
     """Convert a Content-ID header value to an id.
@@ -1192,7 +1195,7 @@ def _header_to_id(self, header):
       raise BatchError("Invalid value for Content-ID: %s" % header)
     if '+' not in header:
       raise BatchError("Invalid value for Content-ID: %s" % header)
-    base, id_ = header[1:-1].rsplit('+', 1)
+    base, id_ = header[1:-1].split(' + ', 1)
 
     return unquote(id_)
 
@@ -1302,8 +1305,8 @@ def add(self, request, callback=None, request_id=None):
         request id, and the second is the deserialized response object. The
         third is an googleapiclient.errors.HttpError exception object if an HTTP error
         occurred while processing the request, or None if no errors occurred.
-      request_id: string, A unique id for the request. The id will be passed to
-        the callback with the response.
+      request_id: string, A unique id for the request. The id will be passed
+        to the callback with the response.
 
     Returns:
       None


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -635,7 +635,7 @@ def test_media_io_base_download_unknown_media_size(self):
 BATCH_RESPONSE = b"""--batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+1>
+Content-ID: <randomness + 1>
 
 HTTP/1.1 200 OK
 Content-Type: application/json
@@ -645,7 +645,7 @@ def test_media_io_base_download_unknown_media_size(self):
 --batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+2>
+Content-ID: <randomness + 2>
 
 HTTP/1.1 200 OK
 Content-Type: application/json
@@ -657,7 +657,7 @@ def test_media_io_base_download_unknown_media_size(self):
 BATCH_ERROR_RESPONSE = b"""--batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+1>
+Content-ID: <randomness + 1>
 
 HTTP/1.1 200 OK
 Content-Type: application/json
@@ -667,7 +667,7 @@ def test_media_io_base_download_unknown_media_size(self):
 --batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+2>
+Content-ID: <randomness + 2>
 
 HTTP/1.1 403 Access Not Configured
 Content-Type: application/json
@@ -693,7 +693,7 @@ def test_media_io_base_download_unknown_media_size(self):
 BATCH_RESPONSE_WITH_401 = b"""--batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+1>
+Content-ID: <randomness + 1>
 
 HTTP/1.1 401 Authorization Required
 Content-Type: application/json
@@ -704,7 +704,7 @@ def test_media_io_base_download_unknown_media_size(self):
 --batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+2>
+Content-ID: <randomness + 2>
 
 HTTP/1.1 200 OK
 Content-Type: application/json
@@ -716,7 +716,7 @@ def test_media_io_base_download_unknown_media_size(self):
 BATCH_SINGLE_RESPONSE = b"""--batch_foobarbaz
 Content-Type: application/http
 Content-Transfer-Encoding: binary
-Content-ID: <randomness+1>
+Content-ID: <randomness + 1>
 
 HTTP/1.1 200 OK
 Content-Type: application/json
@@ -1211,6 +1211,39 @@ def test_execute_request_body(self):
       header = parts[1].splitlines()[1]
       self.assertEqual('Content-Type: application/http', header)
 
+  def test_execute_request_body_with_custom_long_request_ids(self):
+    batch = BatchHttpRequest()
+
+    batch.add(self.request1, request_id='abc'*20)
+    batch.add(self.request2, request_id='def'*20)
+    http = HttpMockSequence([
+      ({'status': '200',
+        'content-type': 'multipart/mixed; boundary="batch_foobarbaz"'},
+        'echo_request_body'),
+      ])
+    try:
+      batch.execute(http=http)
+      self.fail('Should raise exception')
+    except BatchError as e:
+      boundary, _ = e.content.split(None, 1)
+      self.assertEqual('--', boundary[:2])
+      parts = e.content.split(boundary)
+      self.assertEqual(4, len(parts))
+      self.assertEqual('', parts[0])
+      self.assertEqual('--', parts[3].rstrip())
+      for partindex, request_id in ((1, 'abc'*20), (2, 'def'*20)):
+        lines = parts[partindex].splitlines()
+        for n, line in enumerate(lines):
+          if line.startswith('Content-ID:'):
+            # assert correct header folding
+            self.assertTrue(line.endswith('+'), line)
+            header_continuation = lines[n+1]
+            self.assertEqual(
+              header_continuation,
+              ' %s>' % request_id,
+              header_continuation
+            )
+
   def test_execute_initial_refresh_oauth2(self):
     batch = BatchHttpRequest()
     callbacks = Callbacks()