Repository: ultrajson/ultrajson (language: python)

Issue:
Feature Request: Load from bytearray
If the built-in `json` is jsut calling `decode()` under the hood then I can jsut do the same explicitly in my code with `ujson` and that's fine.  But I thought it would be nice if `ujson` would let me skip that step for large JSON objects that happen to be stored as `bytearray`s because even decoding (and the string garbage) isn't free when the objects are really big and you have hundreds of thousands of them. 


### What did you do?
Try to pase a `bytearray` with `ujson`.

```
ujson.loads(bytearray(b'123'))
```

### What did you expect to happen?
Expected same behavior as built-in `json` module:

```
>>> json.loads(bytearray(b'123'))
123
```

### What actually happened?
```
>>> ujson.loads(bytearray(b'123'))
Traceback (most recent call last):
  File "/opt/miniconda3/envs/OptimizedPreprocessing/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
TypeError: Expected String or Unicode
```

### What versions are you using?

* OS:  OS X
* Python: 3.8
* UltraJSON: 5.5.0

Please include **code** that reproduces the issue.
See above.



Discussion / hints:
I'm of the unpopular opinion that, since JSON has its own unicode handling/escape mechanisms,  `ujson.loads()` should only support bytes and that `ujson.dumps()` should write bytes instead of strings.

`ujson.decode(b"100")` already works. All we need to do here is to extend `PyBytes_Check()` called [here](https://github.com/ultrajson/ultrajson/blob/18607247be7c8a2d884396888fda3e514c7db612/python/JSONtoObj.c#L188) to accept arbitrary bytes-like objects rather than strictly bytes.
I'd certainly agree with that if it weren't for compatibility with the stdlib `json`.

We'd need to use the [buffer protocol](https://docs.python.org/3/c-api/buffer.html), specifically `PyObject_GetBuffer` with `PyBUF_C_CONTIGUOUS`, I think. But that wouldn't work with the stable API before Python 3.11. The `PyByteArray_*` APIs exist for `bytearray` specifically, of course, but I don't think there's anything for bytes-like objects in general.
FWIW, stdlib `json` only supports `str`, `bytes`, and `bytearray`, not bytes-like objects.
Ahh yes, let's not forget C contiguity!

I think something like this should give us what we want.
```c

Py_buffer buffer;
if (PyObject_GetBuffer(byteslike_object, &buffer, PyBUF_C_CONTIGUOUS))
    return NULL;

self->buffer_info.buf  // <- this is a char * to the data
```

That seems about right, yeah. And `.len` would be the size.

For `#ifdef Py_LIMITED_API`, I guess we'd want a fallback to the `PyByteArray_*` APIs and a `TypeError` when a bytes-like object that isn't `bytes` or `bytearray` is passed. (Side note: we should add tests for the limited API. I don't think we have any currently.)

By the way, relevant semi-open issue for stdlib `json` support of general bytes-like objects: https://github.com/python/cpython/issues/74379
Huh, I had no idea that we were even trying to support the Python stable API. That complicates things a bit.
I don't know whether we really try to, actually. I just figured we do since such an `#ifdef` exists in the code (#417).
I see that you're both committers here.  It is so amazing to have this level (and this speed) of dialogue on my Issue.  

I have a few comments/questions, though as I am definitely not a committer to this repo I'm happy to be ignored if it leads to fulfilling my original request being fulfilled: 

1. I absolutely love [HPy](https://hpyproject.org/) but if there is a way we can avoid breaking them and mostly ignore the ABI I'd be perfectly happy. 
2. Is there tech-debt we are looking to eliminate here are can we make a minimal change to make this happen?
3. If there are other things we want to support, can we write tests that fail at the point when we want them to be supported in order to force next change?


Wow, HPy looks like one ambitious project - and also one where they'd have to do a really good job to not just [reinvent the same problems in another incompatible way](https://xkcd.com/927/).

I see that they've also been using ujson as a pilot project ([hpyproject/ultrajson-hpy](https://github.com/hpyproject/ultrajson-hpy)). I can't say that [the changes](https://github.com/ultrajson/ultrajson/compare/main...hpyproject:ultrajson-hpy:hpy) look like significant improvements but maybe it's more in the diagnostics.

HPy aside though, I don't see any point in using the stable API for ujson. Our wheels are tiny and releasing for new Python versions is streamlined enough.
Anyone have any idea guesses as to why this codebase consistently uses ints as bools?

Fix patch:
diff --git a/python/JSONtoObj.c b/python/JSONtoObj.c
index fbb3fb6f..208055cb 100644
--- a/python/JSONtoObj.c
+++ b/python/JSONtoObj.c
@@ -36,6 +36,8 @@ Numeric decoder derived from from TCL library
  * Copyright (c) 1994 Sun Microsystems, Inc.
 */
 
+#include <stdbool.h>
+
 #include <Python.h>
 #include <ultrajson.h>
 #include "ujson.h"
@@ -185,25 +187,47 @@ PyObject* JSONToObj(PyObject* self, PyObject *args, PyObject *kwargs)
       return NULL;
   }
 
-  if (PyBytes_Check(arg))
+  Py_buffer buffer;
+  size_t sarg_length;
+  char * raw;
+  bool is_bytes_like = !PyObject_GetBuffer(arg, &buffer, PyBUF_C_CONTIGUOUS);
+  if (is_bytes_like)
   {
-      sarg = arg;
+    #ifdef PYPY_VERSION
+      // PyPy's buffer protocol implementation is buggy: https://foss.heptapod.net/pypy/pypy/-/issues/3872
+      if (!PyBytes_Check(arg) && !PyByteArray_Check(arg)) {
+        PyBuffer_Release(&buffer);
+        PyErr_Format(PyExc_TypeError, "Arbitrary bytes-like objects are not supported on PyPy, Use either string, bytes, or bytearray");
+        return NULL;
+      }
+    #endif
+    raw = buffer.buf;
+    sarg_length = buffer.len;
   }
   else
-  if (PyUnicode_Check(arg))
   {
-    sarg = PyUnicode_AsEncodedString(arg, NULL, "surrogatepass");
-    if (sarg == NULL)
+    PyErr_Clear();
+    if (PyUnicode_Check(arg))
+    {
+      sarg = PyUnicode_AsEncodedString(arg, NULL, "surrogatepass");
+      if (sarg == NULL)
+      {
+        //Exception raised above us by codec according to docs
+        return NULL;
+      }
+      sarg_length = PyBytes_Size(sarg);
+      raw = PyBytes_AsString(sarg);
+    }
+    else
     {
-      //Exception raised above us by codec according to docs
+      #ifdef PYPY_VERSION
+        PyErr_Format(PyExc_TypeError, "Expected string, bytes, or bytearray");
+      #else
+        PyErr_Format(PyExc_TypeError, "Expected string or C-contiguous bytes-like object");
+      #endif
       return NULL;
     }
   }
-  else
-  {
-    PyErr_Format(PyExc_TypeError, "Expected String or Unicode");
-    return NULL;
-  }
 
   decoder.errorStr = NULL;
   decoder.errorOffset = NULL;
@@ -211,14 +235,18 @@ PyObject* JSONToObj(PyObject* self, PyObject *args, PyObject *kwargs)
   decoder.s2d = NULL;
   dconv_s2d_init(&decoder.s2d, DCONV_S2D_ALLOW_TRAILING_JUNK, 0.0, 0.0, "Infinity", "NaN");
 
-  ret = JSON_DecodeObject(&decoder, PyBytes_AsString(sarg), PyBytes_Size(sarg));
+  ret = JSON_DecodeObject(&decoder, raw, sarg_length);
 
   dconv_s2d_free(&decoder.s2d);
 
-  if (sarg != arg)
+  if (!is_bytes_like)
   {
     Py_DECREF(sarg);
   }
+  else
+  {
+    PyBuffer_Release(&buffer);
+  }
 
   if (decoder.errorStr)
   {


Test patch:
diff --git a/tests/test_ujson.py b/tests/test_ujson.py
index 2ea8a8c6..a91332cc 100644
--- a/tests/test_ujson.py
+++ b/tests/test_ujson.py
@@ -1124,6 +1124,44 @@ def test_separators_errors(separators, expected_exception):
         ujson.dumps({"a": 0, "b": 1}, separators=separators)
 
 
+def test_loads_bytes_like():
+    assert ujson.loads(b"123") == 123
+    if hasattr(sys, "pypy_version_info"):
+        with pytest.raises(TypeError, match="PyPy"):
+            ujson.loads(memoryview(b"{}"))
+    else:
+        assert ujson.loads(memoryview(b'["a", "b", "c"]')) == ["a", "b", "c"]
+    assert ujson.loads(bytearray(b"99")) == 99
+    assert ujson.loads('"🦄🐳"'.encode()) == "🦄🐳"
+
+
+@pytest.mark.skipif(
+    hasattr(sys, "pypy_version_info"), reason="PyPy uses incompatible GC"
+)
+def test_loads_bytes_like_refcounting():
+    import gc
+
+    gc.collect()
+    buffer = b'{"a": 99}'
+    old = sys.getrefcount(buffer)
+    assert ujson.loads(buffer) == {"a": 99}
+    assert sys.getrefcount(buffer) == old
+
+    buffer = b'{"a": invalid}'
+    old = sys.getrefcount(buffer)
+    with pytest.raises(ValueError):
+        ujson.loads(buffer)
+    assert sys.getrefcount(buffer) == old
+
+
+def test_loads_non_c_contiguous():
+    buffer = memoryview(b"".join(bytes([i]) + b"_" for i in b"[1, 2, 3]"))[::2]
+    assert not buffer.c_contiguous
+    assert ujson.loads(bytes(buffer)) == [1, 2, 3]
+    with pytest.raises(TypeError):
+        ujson.loads(buffer)
+
+
 """
 The following checks are not part of the standard test suite.
 They can be run manually as follows: