Repository: oasis-open/cti-taxii-client

Issue:
Funny JSON handling
I first started noticing this while working on the user-agent changes (#45), but this isn't directly related to those changes, so I am making a separate issue for this.

One of the things I added (and forgot to mention in the commit message, oops) was to add parameter documentation to _HTTPConnection.post(), which for some reason had been missing.  For the "data" parameter, I just copied text from the [Requests documentation](http://docs.python-requests.org/en/master/api/#requests.Session.post), since it's passed directly into Session.post().  Notice that JSON content is not listed as a legal value.  That was odd, because aren't the TAXII services all JSON-based?  Note also that there is a "json" parameter you can use specifically for JSON.  Wouldn't it have made more sense to use that?  This got me looking at the JSON handling.

The usage of the "data" parameter within taxii2-client is via the Collection.add_objects() method; the actual value is obtained [here](https://github.com/oasis-open/cti-taxii-client/blob/d4fb580379b7ac0ea079cf8becd2348f4e878c9e/taxii2client/__init__.py?#L526-L530):

```python
        if isinstance(bundle, dict):
            if six.PY2:
                bundle = json.dumps(bundle, encoding="utf-8")
            else:
                bundle = json.dumps(bundle)
```

(I just picked master branch head commit as of this writing, to refer to; the aforementioned PR hasn't changed this part.)  This code snip creates JSON which is subsequently passed via "data" parameter, instead of using the "json" parameter and letting Requests take care of the conversion.  There may be some reinvention of the wheel here.  What of that "encoding" parameter though?  Why is it hard-coded, and why make encoding decisions in the Collection endpoint class anyway?

To start with, here's what I think the encoding kwarg actually does: in case there are any binary values in the passed dict, they must be decoded to strings before being encoded as JSON (JSON is a textual format, after all).  The encoding argument describes how to do that.  I.e. the *en*coding argument is actually about *de*coding.

What may have happened is a misunderstanding: that a decision needed to be made here about what to do with high-value codepoints, and the decision was to encode as utf-8.  Lending credence to this idea is that the TAXII 2.0 spec actually defines the string type as `The string data type represents a finite-length string of valid characters from the Unicode coded character set [ISO10646] that are encoded in UTF-8.`  So the spec actually mentions utf-8.  Perhaps the idea was that a json.dumps() call could do double-duty, as a JSON-encode operation followed by a string encode operation.  On the other hand, this code is written to behave differently depending on python version, and there is no usage of encoding on python3.  If encoding needed to happen for python2, why not python3?  On python3 the API changed: there actually is no "encoding" parameter in json.dumps().  Under this misunderstanding, that would mean that the result must remain text, and text is passed to _HTTPConnection.post(), then to Session.post(), which is invalid according to the documentation.  So even if the misunderstanding were true, the code is incorrect.

Also, the "ensure_ascii" param was left at default (True), which means there can be no high-value codepoints in the resulting JSON anyway.  They will all be escaped using the format "\u1234".  JSON specs I looked at do seem to recognize this as an escape syntax, but do we need to do that?  Why not let them pass?

Here's how I think it should work:
- If we actually expect people to pass text-as-binary data in their JSON-serializable objects, we should also allow people to tell us how that text was encoded, not hard-code an assumption.
- The above bullet is probably not worth worrying about; I can't see why anyone would want to do that.  So I don't see any need to specify an encoding there, and have python-version-dependent code.
- Do we really intend for people to be able to pass anything they want to TAXII server endpoints?  If we want to restrict this to JSON, maybe we should be using the "json" kwarg to Session.post(), and mirror that in _HTTPConnection.post().  If want to allow anything, maybe we should mirror Session.post() and allow either "data=" or "json=" as kwargs.  We would only need "json=" in our endpoint classes.  (Btw, when using "json=", Requests automatically encodes the result as utf-8, so it would be compliant.)
- Don't use funny escape syntax unless you really have to.  No point in layering additional escaping which serves no purpose.


Discussion:

Patch:
diff --git a/taxii2client/__init__.py b/taxii2client/__init__.py
index 36e5929..5536a20 100644
--- a/taxii2client/__init__.py
+++ b/taxii2client/__init__.py
@@ -498,7 +498,7 @@ class Collection(_TAXIIEndpoint):
         expires, or the operation completes.
 
         Args:
-            bundle (str): A STIX bundle with the objects to add.
+            bundle: A STIX bundle with the objects to add (string, dict, binary)
             wait_for_completion (bool): Whether to wait for the add operation
                 to complete before returning
             poll_interval (int): If waiting for completion, how often to poll
@@ -528,13 +528,21 @@ class Collection(_TAXIIEndpoint):
         }
 
         if isinstance(bundle, dict):
-            if six.PY2:
-                bundle = json.dumps(bundle, encoding="utf-8")
-            else:
-                bundle = json.dumps(bundle)
+            json_text = json.dumps(bundle, ensure_ascii=False)
+            data = json_text.encode("utf-8")
+
+        elif isinstance(bundle, six.text_type):
+            data = bundle.encode("utf-8")
+
+        elif isinstance(bundle, six.binary_type):
+            data = bundle
+
+        else:
+            raise TypeError("Don't know how to handle type '{}'".format(
+                type(bundle).__name__))
 
         status_json = self._conn.post(self.objects_url, headers=headers,
-                                      data=bundle)
+                                      data=data)
 
         status_url = urlparse.urljoin(
             self.url,
@@ -875,25 +883,34 @@ class _HTTPConnection(object):
 
         return _to_json(resp)
 
-    def post(self, url, headers=None, params=None, data=None):
+    def post(self, url, headers=None, params=None, **kwargs):
         """Send a JSON POST request with the given request headers, additional
         URL query parameters, and the given JSON in the request body.  The
         extra query parameters are merged with any which already exist in the
-        URL.
+        URL.  The 'json' and 'data' parameters may not both be given.
 
         Args:
             url (str): URL to retrieve
             headers (dict): Any other headers to be added to the request.
             params: dictionary or bytes to be sent in the query string for the
                 request. (optional)
-            data: data to post as dictionary, list of tuples, bytes, or
-                file-like object
+            json: json to send in the body of the Request.  This must be a
+                JSON-serializable object. (optional)
+            data: raw request body data.  May be a dictionary, list of tuples,
+                bytes, or file-like object to send in the body of the Request.
+                (optional)
         """
 
-        merged_headers = self._merge_headers(headers)
+        if len(kwargs) > 1:
+            raise InvalidArgumentsError("Too many extra args ({} > 1)".format(
+                len(kwargs)))
+
+        if kwargs:
+            kwarg = next(iter(kwargs))
+            if kwarg not in ("json", "data"):
+                raise InvalidArgumentsError("Invalid kwarg: " + kwarg)
 
-        resp = self.session.post(url, headers=merged_headers, params=params,
-                                 data=data)
+        resp = self.session.post(url, headers=headers, params=params, **kwargs)
         resp.raise_for_status()
         return _to_json(resp)
 


Test patch:
diff --git a/taxii2client/test/test_client.py b/taxii2client/test/test_client.py
index d80b2a3..0bae132 100644
--- a/taxii2client/test/test_client.py
+++ b/taxii2client/test/test_client.py
@@ -533,6 +533,34 @@ def test_add_object_to_collection_dict(writable_collection):
     assert status.pending_count == 0
 
 
+@responses.activate
+def test_add_object_to_collection_bin(writable_collection):
+    responses.add(responses.POST, ADD_WRITABLE_OBJECTS_URL,
+                  ADD_OBJECTS_RESPONSE, status=202,
+                  content_type=MEDIA_TYPE_TAXII_V20)
+
+    bin_bundle = STIX_BUNDLE.encode("utf-8")
+
+    status = writable_collection.add_objects(bin_bundle)
+
+    assert status.status == "complete"
+    assert status.total_count == 1
+    assert status.success_count == 1
+    assert len(status.successes) == 1
+    assert status.failure_count == 0
+    assert status.pending_count == 0
+
+
+@responses.activate
+def test_add_object_to_collection_badtype(writable_collection):
+    responses.add(responses.POST, ADD_WRITABLE_OBJECTS_URL,
+                  ADD_OBJECTS_RESPONSE, status=202,
+                  content_type=MEDIA_TYPE_TAXII_V20)
+
+    with pytest.raises(TypeError):
+        writable_collection.add_objects([1, 2, 3])
+
+
 @responses.activate
 def test_add_object_rases_error_when_collection_id_does_not_match_url(
         bad_writable_collection):
@@ -817,6 +845,19 @@ def test_collection_missing_can_write_property(collection_dict):
     assert "No 'can_write' in Collection for request 'https://example.com/api1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/'" == str(excinfo.value)
 
 
+def test_conn_post_kwarg_errors():
+    conn = _HTTPConnection()
+
+    with pytest.raises(InvalidArgumentsError):
+        conn.post(DISCOVERY_URL, data=1, json=2)
+
+    with pytest.raises(InvalidArgumentsError):
+        conn.post(DISCOVERY_URL, data=1, foo=2)
+
+    with pytest.raises(InvalidArgumentsError):
+        conn.post(DISCOVERY_URL, foo=1)
+
+
 def test_user_agent_defaulting():
     conn = _HTTPConnection(user_agent="foo/1.0")
     headers = conn._merge_headers({})