Repository: pandas-dev/pandas
Pull request: BUG: Fixed encoding of pd.NA with to_json
- [x] closes #31615
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
pd.NA is converted to {} instead of null with pd.DataFrame.to_json
#### Code Sample

```python
import numpy as np
import pandas as pd

pd.DataFrame([[np.nan]], columns=["NA"]).to_json(orient="records")
# '[{"NA":null}]'
pd.DataFrame([[None]], columns=["NA"]).to_json(orient="records")
# '[{"NA":null}]'
pd.DataFrame([[pd.NA]], columns=["NA"]).to_json(orient="records")
# '[{"NA":{}}]'
```
#### Problem Description

`pd.NA` should behave as `np.nan` and `None` when converting a DataFrame to JSON.

#### Expected Output

In the resulting JSON the `pd.NA` instances should be converted to JSON's `null`.

Hints from issue discussion:
@pvieito Thanks for the report! That's indeed buggy
json writer is having some issue 
I got this one, some changes are needed in ujson.

Commit message: BUG: Fixed encoding of pd.NA with to_json GH31615

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.2.rst b/doc/source/whatsnew/v1.0.2.rst
index 07a837829c384..182ad7231983e 100644
--- a/doc/source/whatsnew/v1.0.2.rst
+++ b/doc/source/whatsnew/v1.0.2.rst
@@ -25,8 +25,9 @@ Fixed regressions
 Bug fixes
 ~~~~~~~~~
 
--
--
+**I/O**
+
+- Using ``pd.NA`` with :meth:`DataFrame.to_json` now correctly outputs a null value instead of an empty object (:issue:`31615`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/_libs/src/ujson/python/objToJSON.c b/pandas/_libs/src/ujson/python/objToJSON.c
index 0fc146d25459b..8cfc20ffd2c1c 100644
--- a/pandas/_libs/src/ujson/python/objToJSON.c
+++ b/pandas/_libs/src/ujson/python/objToJSON.c
@@ -53,6 +53,7 @@ static PyTypeObject *cls_dataframe;
 static PyTypeObject *cls_series;
 static PyTypeObject *cls_index;
 static PyTypeObject *cls_nat;
+static PyTypeObject *cls_na;
 PyObject *cls_timedelta;
 
 npy_int64 get_nat(void) { return NPY_MIN_INT64; }
@@ -149,6 +150,7 @@ int PdBlock_iterNext(JSOBJ, JSONTypeContext *);
 void *initObjToJSON(void) {
     PyObject *mod_pandas;
     PyObject *mod_nattype;
+    PyObject *mod_natype;
     PyObject *mod_decimal = PyImport_ImportModule("decimal");
     type_decimal =
         (PyTypeObject *)PyObject_GetAttrString(mod_decimal, "Decimal");
@@ -174,6 +176,12 @@ void *initObjToJSON(void) {
         Py_DECREF(mod_nattype);
     }
 
+    mod_natype = PyImport_ImportModule("pandas._libs.missing");
+    if (mod_natype) {
+        cls_na = (PyTypeObject *)PyObject_GetAttrString(mod_natype, "NAType");
+        Py_DECREF(mod_natype);
+    }
+
     /* Initialise numpy API */
     import_array();
     // GH 31463
@@ -1789,6 +1797,10 @@ void Object_beginTypeContext(JSOBJ _obj, JSONTypeContext *tc) {
                      "%R (0d array) is not JSON serializable at the moment",
                      obj);
         goto INVALID;
+    } else if (PyObject_TypeCheck(obj, cls_na)) {
+        PRINTMARK();
+        tc->type = JT_NULL;
+        return;
     }
 
 ISITERABLE:
diff --git a/pandas/tests/io/json/test_pandas.py b/pandas/tests/io/json/test_pandas.py
index 602022a21c4a6..f24a9537a6511 100644
--- a/pandas/tests/io/json/test_pandas.py
+++ b/pandas/tests/io/json/test_pandas.py
@@ -1671,3 +1671,11 @@ def test_to_s3(self, s3_resource):
         assert target_file in (
             obj.key for obj in s3_resource.Bucket("pandas-test").objects.all()
         )
+
+    @pytest.mark.parametrize(
+        "dataframe,expected", [(pd.DataFrame([[pd.NA]]), '{"0":{"0":null}}',)],
+    )
+    def test_json_pandas_na(self, dataframe, expected):
+        # GH 31615
+        result = dataframe.to_json()
+        assert result == expected


Code review comments:
diff_hunk: @@ -1671,3 +1671,11 @@ def test_to_s3(self, s3_resource):
         assert target_file in (
             obj.key for obj in s3_resource.Bucket("pandas-test").objects.all()
         )
+
+    @pytest.mark.parametrize(
+        "dataframe,expected", [(pd.DataFrame([[pd.NA]]), '{"0":{"0":null}}',)],
path: pandas/tests/io/json/test_pandas.py
text: @user1:
Can you add pd.NA to the indices of the DataFrame? There are different code paths for encoding values vs index labels so want to cover both 

You can see NdArray_encodeLabels for how that works for labels

@user1:
Note that we also have a `nulls_fixture` in the top level conftest that I think we could use here and probably just add pd.NA to @user2 

@author:
I've added a test for `pd.NA` as an index label.  It currently encodes `pd.NA` as `"<NA>"` as it uses `NAType.__repr__`.  For reference, `np.nan` encodes as `"nan"` and `None` encodes as `"None"` when they are index labels.  Is this the desired behavior?

I'll have performance test result comparisons by tomorrow.

@user2:
> Note that we also have a nulls_fixture in the top level conftest that I think we could use here and probably just add pd.NA to @user2

That might not be that easy, as `nulls_fixture` is used in many places, which not necessarily all support pd.NA 
(but it would be an interesting exercise to do, but not necessarily in this PR)

> It currently encodes pd.NA as "<NA>" as it uses NAType.__repr__. For reference, np.nan encodes as "nan" and None encodes as "None" when they are index labels. Is this the desired behavior?

I am not sure this is that useful, but at least it is consistent with the other null values.

@user1:
I think it is a "regression" for those in the index labels to not write "null". This was fixed for pd.NaT in #30977 but not the rest

I say regression in quotes because I don't think the behavior is well defined, but in any case a separate issue

@user2:
So for the index, this should also give "null" then?

@user1:
Yea I think so. AFAIK there is no JSON standard for how to represent missing values as object keys (or if you even should), but object keys MUST be strings, so `"null"` is probably the most reasonable accommodation. 

I think in 0.25 these probably just showed as `null` as an object key, which I don't think would be standard JSON anyway