Repository: TeskaLabs/cysimdjson (language: python)

Issue:
TypeError: expected bytes, str found
Im not computer science guy (mechanical eng.), so hopefully the question is not too off. My project involves streaming millions of data (speed is very important) via websocket client delivered as string messages containing a system's operational parameters in json/dict format. One pushed message is a list (actual list in string) containing several dictionaries through which we iterate and analyse in real time. A 3 abbreviated separately pushed messages would look sth like this:
```
[{"t":1611064750942,"Signal003":"Q","Signal003_MODE":1,"Signal003_TARGET":11,"Signal003_NEXT":8,"Signal003_IRV":9.05,"Signal003_ONOFF":9.06,"Signal003_TEMP":5,"Signal001_PRES":98, ,"Signal001_ENT":3}]
[{"t":1611064750943,"Signal033":"T","Signal003_MODE":1,"Signal003_TARGET":5170,"Signal001_ONOFF":19,"Signal001_TEMP":91.28,"s":6445,"Signal033_IRV":[12],"Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":2,"Signal003_TARGET":5171,"Signal001_ONOFF":8,"Signal001_TEMP":9.04,"s":100,"Signal033_IRV":[12],"t":1611064750943,"Signal003_ENT":3}]
[{"t":1611064750943,"Signal065":"T","Signal003_MODE":3,"Signal003_TARGET":5172,"Signal001_ONOFF":8,"Signal001_TEMP":9.04,"s":1000,"Signal033_IRV":[12],"Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":5173,"Signal001_ONOFF":8,"Signal001_TEMP":9.04,"s":100,"Signal033_IRV":[12],"t":1611064750943,"Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":7116,"Signal001_ONOFF":12,"Signal001_TEMP":9.04,"s":10,"Signal033_IRV":[14,12,37,41],"t":1611064750943,Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":961,"Signal001_ONOFF":19,"Signal001_TEMP":9.04,"s":4,"Signal033_IRV":[14,12,37,41],"t":1611064750943,Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":962,"Signal001_ONOFF":19,"Signal001_TEMP":9.04,"s":10,"Signal033_IRV":[14,12,37,41],"t":1611064750943,"Signal003_ENT":3}]
```
On your frontpage, I saw the example and if understood it correctly, the message should be in byte format, hence I understand the TypeError Im getting:
```
json_bytes = b'''
{
  "foo": [1,2,[3]]
}
'''
```
Would it be possible to parse a string message within the cysimdjson library as we have no influence on the type of the pushed message from the websocket? I hope the question is not too off, but since websockets libraries in conjunction with json formats are widely used, I was thinking the problem might be worth looking into. Also, I can imagine that one could do the string conversion somehow in python, but C speed would probably be affected. At the moment, I use orjson, which is pretty fast and working well, but looking at your results, it ignites the interest. Regardless of the answer, thank you for the efforts with the library.  


Discussion / hints:
I see. Can you please also provide the whole error output you are getting.
We intentionally avoid any unnecessary type checks since it costs a performance but if this is doable in e.g. exception handler, it could be added easily.
Thanks for the reply. I completely understand the avoidance of type checks, so I don't expect too much, but maybe there's some work around or suggestion, thanks in advance. Here goes the error:
`
Traceback (most recent call last):
  File "/home/oem/Syncthing/Projects/Pump_diagnostics_TWU21/JSN_TEST/json_study.py", line 66, in <module>
    benchmark0(F"{'cysimdjson_parser':>15}", parser.parse) 
  File "/home/oem/Syncthing/Projects/Pump_diagnostics_TWU21/JSN_TEST/json_study.py", line 50, in benchmark0
    jsn = loads(m)
  File "cysimdjson/cysimdjson.pyx", line 359, in cysimdjson.JSONParser.parse
TypeError: expected bytes, str found
`


Fix patch:
diff --git a/README.md b/README.md
index 060014f..67eef0d 100644
--- a/README.md
+++ b/README.md
@@ -40,6 +40,27 @@ _Note: `parser` object can be reused for maximum performance._
 The `json_parsed` is a read-only dictionary-like object, that provides an access to JSON data.
 
 
+## Documentation
+
+`JSONParser.parse(json_bytes)`
+
+Parse JSON `json_bytes`, represented as `bytes`.
+
+
+`JSONParser.parse_in_place(bytes)`
+
+Parse JSON `json_bytes`, represented as `bytes`, assuming that there is a padding expected by SIMDJSON.
+This is the fastest parsing variant.
+
+
+`JSONParser.parse_string(string)`
+
+Parse JSON `json_bytes`, represented as `str` (string).
+
+
+`JSONParser.load(path)`
+
+
 ## Installation
 
 ```
diff --git a/cysimdjson/cysimdjson.pyx b/cysimdjson/cysimdjson.pyx
index aaf7f27..19f2450 100644
--- a/cysimdjson/cysimdjson.pyx
+++ b/cysimdjson/cysimdjson.pyx
@@ -3,10 +3,12 @@
 from libc.stdint cimport int64_t, uint64_t, uint32_t
 from libcpp cimport bool
 from libcpp.string cimport string
+
 from cpython.bytes cimport PyBytes_AsStringAndSize
+from cpython.ref cimport PyObject
+
 from cython.operator cimport preincrement
 from cython.operator cimport dereference
-from cpython.ref cimport PyObject
 
 
 cdef extern from "string_view" namespace "std":
@@ -116,8 +118,10 @@ cdef extern from "jsoninter.h":
 
 	cdef object element_to_py_string(simdjson_element & value) except + simdjson_error_handler
 
-	PyObject * string_view_to_python_string(string_view & sv)
-	string get_active_implementation()
+	cdef PyObject * string_view_to_python_string(string_view & sv)
+	cdef string get_active_implementation()
+
+	cdef const char * PyUnicode_AsUTF8AndSize(object, Py_ssize_t *)
 
 
 cdef class JSONObject:
@@ -237,14 +241,14 @@ cdef class JSONParser:
 		simdjson_parser Parser
 
 
-	def __cinit__(self, max_capacity=None):
+	def __cinit__(JSONParser self, max_capacity=None):
 		if max_capacity is not None:
 			self.Parser = simdjson_parser.simdjson_parser(int(max_capacity))
 		else:
 			self.Parser = simdjson_parser.simdjson_parser()
 
 
-	def parse(self, event):
+	def parse(JSONParser self, event: bytes):
 		cdef Py_ssize_t pysize
 		cdef char * data_ptr
 		cdef int rc = PyBytes_AsStringAndSize(event, &data_ptr, &pysize)
@@ -255,7 +259,7 @@ cdef class JSONParser:
 		return _wrap_element(element, self, event)
 
 
-	def parse_in_place(self, event):
+	def parse_in_place(JSONParser self, event: bytes):
 		'''
 		Skip the reallocation of the input event buffer.
 		This method is little bit faster than parse() but you have to ensure proper padding of the event.
@@ -270,12 +274,21 @@ cdef class JSONParser:
 		return _wrap_element(element, self, event)
 
 
-	def load(self, path):
+	def parse_string(JSONParser self, event: str):
+
+		cdef Py_ssize_t pysize
+		cdef const char * data_ptr = PyUnicode_AsUTF8AndSize(event, &pysize)
+
+		cdef simdjson_element element = self.Parser.parse(data_ptr, pysize, 1)
+		return _wrap_element(element, self, event)
+
+
+	def load(JSONParser self, path):
 		cdef simdjson_element element = self.Parser.load(path)
 		return _wrap_element(element, self, None)
 
 
-	def active_implementation(self):
+	def active_implementation(JSONParser self):
 		return get_active_implementation()
 
 


Test patch:
diff --git a/test/test_document/__init__.py b/test/test_document/__init__.py
index 2ed6bed..4199af0 100644
--- a/test/test_document/__init__.py
+++ b/test/test_document/__init__.py
@@ -46,3 +46,12 @@ def test_parser_failure_01(self):
 		parser = cysimdjson.JSONParser()
 		with self.assertRaises(ValueError) as context:
 			json_parsed = parser.parse(b"Definitively not a JSON")
+
+
+
+	def test_string_01(self):
+
+		parser = cysimdjson.JSONParser()
+
+		with open(os.path.join(THIS_DIR, 'document.json'), 'r') as fo:
+			json_parsed = parser.parse_string(fo.read())