Repository: pandas-dev/pandas

Issue:
add indent support to `to_json` method
Sometimes you just want to have a quick look at your data in IPython and having `to_json` to support `indent` (as both `json.dumps` and `simplejson.dumps` do) would be very handy.

A workaround, which however doesn't always work, is to do `json.dumps(series.to_dict())`, which unfortunately is annoying when you don't have a JSON-serializable type (e.g., `numpy.bool_`).

I'm leaving this here as a food for discussion.



Discussion:
Another workaround: `df_json_pretty = json.dumps(json.loads(df.to_json()), indent=2)`


Is it still on the list to be implemented?
Still open. Feel free to work on it if you want, but the current json isn't the friendliest.
@TomAugspurger Ok, got it! I'll try to figure out if there is a straightforward way to deal with it.
It looks like the [JSON writer](https://github.com/pandas-dev/pandas/blob/011b79fbf73b45313b47c08b4be1fc07dcb99365/pandas/io/json/json.py#L107-L117) uses Pandas' own JSON writer:
https://github.com/pandas-dev/pandas/blob/011b79fbf73b45313b47c08b4be1fc07dcb99365/pandas/io/json/json.py#L7

It looks like eventually it calls `objToJSON` in C here:
https://github.com/pandas-dev/pandas/blob/011b79fbf73b45313b47c08b4be1fc07dcb99365/pandas/_libs/src/ujson/python/objToJSON.c#L2317-L2321

So if I'm not mistaken, to add an `indent` option, you'd need to change the C code?
Yep that sounds right. FYI, I think ripping out that JSON code and
replacing it with something more maintainable
(maybe based on Apache Arrow's JSON support) is on the medium to long-term
agenda.



On Mon, Nov 12, 2018 at 9:28 AM Kyle Barron <notifications@github.com>
wrote:

> It looks like the JSON writer
> <https://github.com/pandas-dev/pandas/blob/011b79fbf73b45313b47c08b4be1fc07dcb99365/pandas/io/json/json.py#L107-L117>
> uses Pandas' own JSON writer:
>
> https://github.com/pandas-dev/pandas/blob/011b79fbf73b45313b47c08b4be1fc07dcb99365/pandas/io/json/json.py#L7
>
> It looks like eventually it calls objToJSON in C here:
>
> https://github.com/pandas-dev/pandas/blob/011b79fbf73b45313b47c08b4be1fc07dcb99365/pandas/_libs/src/ujson/python/objToJSON.c#L2317-L2321
>
> So if I'm not mistaken, to add an indent option, you'd need to change the
> C code?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/12004#issuecomment-437922165>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIiHGfDOPOU98wggIOgkOVEaLVrPgks5uuZOKgaJpZM4HBmld>
> .
>

Is the current JSON writer believed to be much faster than the Python standard library's JSON writer?
Yep. It also handles types not supported by the stdlib's json writer
(notably numpy types).

On Mon, Nov 12, 2018 at 9:55 AM Kyle Barron <notifications@github.com>
wrote:

> Is the current JSON writer believed to be much faster than the Python
> standard library's JSON writer?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/12004#issuecomment-437932241>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIvmwtHI6ob3U9Tb4WZVI97EsrOd8ks5uuZn9gaJpZM4HBmld>
> .
>

I agree on this, JSON is also meant to be human readable. Right now this is just annoying.

Resolving patch:
diff --git a/pandas/_typing.py b/pandas/_typing.py
--- a/pandas/_typing.py
+++ b/pandas/_typing.py
@@ -25,7 +25,7 @@
 FilePathOrBuffer = Union[str, Path, IO[AnyStr]]
 
 FrameOrSeries = TypeVar("FrameOrSeries", bound="NDFrame")
-Scalar = Union[str, int, float]
+Scalar = Union[str, int, float, bool]
 Axis = Union[str, int]
 Ordered = Optional[bool]
 
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -8,6 +8,7 @@
 import re
 from textwrap import dedent
 from typing import (
+    Any,
     Callable,
     Dict,
     FrozenSet,
@@ -61,7 +62,7 @@
 from pandas.core.dtypes.missing import isna, notna
 
 import pandas as pd
-from pandas._typing import Dtype, FilePathOrBuffer
+from pandas._typing import Dtype, FilePathOrBuffer, Scalar
 from pandas.core import missing, nanops
 import pandas.core.algorithms as algos
 from pandas.core.base import PandasObject, SelectionMixin
@@ -2249,17 +2250,18 @@ def to_excel(
 
     def to_json(
         self,
-        path_or_buf=None,
-        orient=None,
-        date_format=None,
-        double_precision=10,
-        force_ascii=True,
-        date_unit="ms",
-        default_handler=None,
-        lines=False,
-        compression="infer",
-        index=True,
-    ):
+        path_or_buf: Optional[FilePathOrBuffer] = None,
+        orient: Optional[str] = None,
+        date_format: Optional[str] = None,
+        double_precision: int = 10,
+        force_ascii: bool_t = True,
+        date_unit: str = "ms",
+        default_handler: Optional[Callable[[Any], Union[Scalar, List, Dict]]] = None,
+        lines: bool_t = False,
+        compression: Optional[str] = "infer",
+        index: bool_t = True,
+        indent: Optional[int] = None,
+    ) -> Optional[str]:
         """
         Convert the object to a JSON string.
 
@@ -2339,6 +2341,11 @@ def to_json(
 
             .. versionadded:: 0.23.0
 
+        indent : integer, optional
+           Length of whitespace used to indent each record.
+
+           .. versionadded:: 1.0.0
+
         Returns
         -------
         None or str
@@ -2349,6 +2356,13 @@ def to_json(
         --------
         read_json
 
+        Notes
+        -----
+        The behavior of ``indent=0`` varies from the stdlib, which does not
+        indent the output but does insert newlines. Currently, ``indent=0``
+        and the default ``indent=None`` are equivalent in pandas, though this
+        may change in a future release.
+
         Examples
         --------
 
@@ -2399,6 +2413,10 @@ def to_json(
             date_format = "iso"
         elif date_format is None:
             date_format = "epoch"
+
+        config.is_nonnegative_int(indent)
+        indent = indent or 0
+
         return json.to_json(
             path_or_buf=path_or_buf,
             obj=self,
@@ -2411,6 +2429,7 @@ def to_json(
             lines=lines,
             compression=compression,
             index=index,
+            indent=indent,
         )
 
     def to_hdf(self, path_or_buf, key, **kwargs):
diff --git a/pandas/io/json/_json.py b/pandas/io/json/_json.py
--- a/pandas/io/json/_json.py
+++ b/pandas/io/json/_json.py
@@ -1,6 +1,8 @@
+from collections import OrderedDict
 from io import StringIO
 from itertools import islice
 import os
+from typing import Any, Callable, Dict, List, Optional, Type, Union
 
 import numpy as np
 
@@ -11,6 +13,7 @@
 from pandas.core.dtypes.common import ensure_str, is_period_dtype
 
 from pandas import DataFrame, MultiIndex, Series, isna, to_datetime
+from pandas._typing import Scalar
 from pandas.core.reshape.concat import concat
 
 from pandas.io.common import (
@@ -31,20 +34,23 @@
 
 TABLE_SCHEMA_VERSION = "0.20.0"
 
+Serializable = Union[Scalar, List, Dict]
+
 
 # interface to/from
 def to_json(
     path_or_buf,
     obj,
-    orient=None,
-    date_format="epoch",
-    double_precision=10,
-    force_ascii=True,
-    date_unit="ms",
-    default_handler=None,
-    lines=False,
-    compression="infer",
-    index=True,
+    orient: Optional[str] = None,
+    date_format: str = "epoch",
+    double_precision: int = 10,
+    force_ascii: bool = True,
+    date_unit: str = "ms",
+    default_handler: Optional[Callable[[Any], Serializable]] = None,
+    lines: bool = False,
+    compression: Optional[str] = "infer",
+    index: bool = True,
+    indent: int = 0,
 ):
 
     if not index and orient not in ["split", "table"]:
@@ -59,7 +65,7 @@ def to_json(
     if orient == "table" and isinstance(obj, Series):
         obj = obj.to_frame(name=obj.name or "values")
     if orient == "table" and isinstance(obj, DataFrame):
-        writer = JSONTableWriter
+        writer = JSONTableWriter  # type: Type["Writer"]
     elif isinstance(obj, Series):
         writer = SeriesWriter
     elif isinstance(obj, DataFrame):
@@ -76,6 +82,7 @@ def to_json(
         date_unit=date_unit,
         default_handler=default_handler,
         index=index,
+        indent=indent,
     ).write()
 
     if lines:
@@ -97,18 +104,19 @@ class Writer:
     def __init__(
         self,
         obj,
-        orient,
-        date_format,
-        double_precision,
-        ensure_ascii,
-        date_unit,
-        index,
-        default_handler=None,
+        orient: Optional[str],
+        date_format: str,
+        double_precision: int,
+        ensure_ascii: bool,
+        date_unit: str,
+        index: bool,
+        default_handler: Optional[Callable[[Any], Serializable]] = None,
+        indent: int = 0,
     ):
         self.obj = obj
 
         if orient is None:
-            orient = self._default_orient
+            orient = self._default_orient  # type: ignore
 
         self.orient = orient
         self.date_format = date_format
@@ -117,6 +125,7 @@ def __init__(
         self.date_unit = date_unit
         self.default_handler = default_handler
         self.index = index
+        self.indent = indent
 
         self.is_copy = None
         self._format_axes()
@@ -133,17 +142,19 @@ def write(self):
             self.date_unit,
             self.date_format == "iso",
             self.default_handler,
+            self.indent,
         )
 
     def _write(
         self,
         obj,
-        orient,
-        double_precision,
-        ensure_ascii,
-        date_unit,
-        iso_dates,
-        default_handler,
+        orient: Optional[str],
+        double_precision: int,
+        ensure_ascii: bool,
+        date_unit: str,
+        iso_dates: bool,
+        default_handler: Optional[Callable[[Any], Serializable]],
+        indent: int,
     ):
         return dumps(
             obj,
@@ -153,6 +164,7 @@ def _write(
             date_unit=date_unit,
             iso_dates=iso_dates,
             default_handler=default_handler,
+            indent=indent,
         )
 
 
@@ -169,12 +181,13 @@ def _format_axes(self):
     def _write(
         self,
         obj,
-        orient,
-        double_precision,
-        ensure_ascii,
-        date_unit,
-        iso_dates,
-        default_handler,
+        orient: Optional[str],
+        double_precision: int,
+        ensure_ascii: bool,
+        date_unit: str,
+        iso_dates: bool,
+        default_handler: Optional[Callable[[Any], Serializable]],
+        indent: int,
     ):
         if not self.index and orient == "split":
             obj = {"name": obj.name, "data": obj.values}
@@ -186,6 +199,7 @@ def _write(
             date_unit,
             iso_dates,
             default_handler,
+            indent,
         )
 
 
@@ -214,12 +228,13 @@ def _format_axes(self):
     def _write(
         self,
         obj,
-        orient,
-        double_precision,
-        ensure_ascii,
-        date_unit,
-        iso_dates,
-        default_handler,
+        orient: Optional[str],
+        double_precision: int,
+        ensure_ascii: bool,
+        date_unit: str,
+        iso_dates: bool,
+        default_handler: Optional[Callable[[Any], Serializable]],
+        indent: int,
     ):
         if not self.index and orient == "split":
             obj = obj.to_dict(orient="split")
@@ -232,6 +247,7 @@ def _write(
             date_unit,
             iso_dates,
             default_handler,
+            indent,
         )
 
 
@@ -241,13 +257,14 @@ class JSONTableWriter(FrameWriter):
     def __init__(
         self,
         obj,
-        orient,
-        date_format,
-        double_precision,
-        ensure_ascii,
-        date_unit,
-        index,
-        default_handler=None,
+        orient: Optional[str],
+        date_format: str,
+        double_precision: int,
+        ensure_ascii: bool,
+        date_unit: str,
+        index: bool,
+        default_handler: Optional[Callable[[Any], Serializable]] = None,
+        indent: int = 0,
     ):
         """
         Adds a `schema` attribute with the Table Schema, resets
@@ -255,6 +272,7 @@ def __init__(
         to know what the index is, forces orient to records, and forces
         date_format to 'iso'.
         """
+
         super().__init__(
             obj,
             orient,
@@ -264,6 +282,7 @@ def __init__(
             date_unit,
             index,
             default_handler=default_handler,
+            indent=indent,
         )
 
         if date_format != "iso":
@@ -315,19 +334,20 @@ def _write(
         date_unit,
         iso_dates,
         default_handler,
+        indent,
     ):
-        data = super()._write(
-            obj,
+        table_obj = OrderedDict((("schema", self.schema), ("data", obj)))
+        serialized = super()._write(
+            table_obj,
             orient,
             double_precision,
             ensure_ascii,
             date_unit,
             iso_dates,
             default_handler,
+            indent,
         )
-        serialized = '{{"schema": {schema}, "data": {data}}}'.format(
-            schema=dumps(self.schema), data=data
-        )
+
         return serialized
 
 


Test patch:
diff --git a/pandas/tests/io/json/test_json_table_schema.py b/pandas/tests/io/json/test_json_table_schema.py
--- a/pandas/tests/io/json/test_json_table_schema.py
+++ b/pandas/tests/io/json/test_json_table_schema.py
@@ -5,6 +5,8 @@
 import numpy as np
 import pytest
 
+from pandas.compat import PY35
+
 from pandas.core.dtypes.dtypes import CategoricalDtype, DatetimeTZDtype, PeriodDtype
 
 import pandas as pd
@@ -20,6 +22,14 @@
 )
 
 
+def assert_results_equal(result, expected):
+    """Helper function for comparing deserialized JSON with Py35 compat."""
+    if PY35:
+        assert sorted(result.items()) == sorted(expected.items())
+    else:
+        assert result == expected
+
+
 class TestBuildSchema:
     def setup_method(self, method):
         self.df = DataFrame(
@@ -234,7 +244,8 @@ def test_build_series(self):
                 ),
             ]
         )
-        assert result == expected
+
+        assert_results_equal(result, expected)
 
     def test_to_json(self):
         df = self.df.copy()
@@ -323,7 +334,8 @@ def test_to_json(self):
             ),
         ]
         expected = OrderedDict([("schema", schema), ("data", data)])
-        assert result == expected
+
+        assert_results_equal(result, expected)
 
     def test_to_json_float_index(self):
         data = pd.Series(1, index=[1.0, 2.0])
@@ -352,7 +364,8 @@ def test_to_json_float_index(self):
                 ),
             ]
         )
-        assert result == expected
+
+        assert_results_equal(result, expected)
 
     def test_to_json_period_index(self):
         idx = pd.period_range("2016", freq="Q-JAN", periods=2)
@@ -372,7 +385,8 @@ def test_to_json_period_index(self):
             OrderedDict([("index", "2016-02-01T00:00:00.000Z"), ("values", 1)]),
         ]
         expected = OrderedDict([("schema", schema), ("data", data)])
-        assert result == expected
+
+        assert_results_equal(result, expected)
 
     def test_to_json_categorical_index(self):
         data = pd.Series(1, pd.CategoricalIndex(["a", "b"]))
@@ -406,7 +420,8 @@ def test_to_json_categorical_index(self):
                 ),
             ]
         )
-        assert result == expected
+
+        assert_results_equal(result, expected)
 
     def test_date_format_raises(self):
         with pytest.raises(ValueError):
@@ -542,7 +557,8 @@ def test_categorical(self):
                 ),
             ]
         )
-        assert result == expected
+
+        assert_results_equal(result, expected)
 
     @pytest.mark.parametrize(
         "idx,nm,prop",
@@ -596,7 +612,8 @@ def test_timestamp_in_columns(self):
         )
         result = df.to_json(orient="table")
         js = json.loads(result)
-        assert js["schema"]["fields"][1]["name"] == 1451606400000
+        assert js["schema"]["fields"][1]["name"] == "2016-01-01T00:00:00.000Z"
+        # TODO - below expectation is not correct; see GH 28256
         assert js["schema"]["fields"][2]["name"] == 10000
 
     @pytest.mark.parametrize(
diff --git a/pandas/tests/io/json/test_pandas.py b/pandas/tests/io/json/test_pandas.py
--- a/pandas/tests/io/json/test_pandas.py
+++ b/pandas/tests/io/json/test_pandas.py
@@ -7,7 +7,7 @@
 import numpy as np
 import pytest
 
-from pandas.compat import is_platform_32bit
+from pandas.compat import PY35, is_platform_32bit
 import pandas.util._test_decorators as td
 
 import pandas as pd
@@ -1647,3 +1647,156 @@ def test_tuple_labels(self, orient, expected):
         df = pd.DataFrame([[1]], index=[("a", "b")], columns=[("c", "d")])
         result = df.to_json(orient=orient)
         assert result == expected
+
+    @pytest.mark.parametrize("indent", [1, 2, 4])
+    def test_to_json_indent(self, indent):
+        # GH 12004
+        df = pd.DataFrame([["foo", "bar"], ["baz", "qux"]], columns=["a", "b"])
+
+        result = df.to_json(indent=indent)
+        spaces = " " * indent
+        expected = """{{
+{spaces}"a":{{
+{spaces}{spaces}"0":"foo",
+{spaces}{spaces}"1":"baz"
+{spaces}}},
+{spaces}"b":{{
+{spaces}{spaces}"0":"bar",
+{spaces}{spaces}"1":"qux"
+{spaces}}}
+}}""".format(
+            spaces=spaces
+        )
+
+        assert result == expected
+
+    @pytest.mark.parametrize(
+        "orient,expected",
+        [
+            (
+                "split",
+                """{
+    "columns":[
+        "a",
+        "b"
+    ],
+    "index":[
+        0,
+        1
+    ],
+    "data":[
+        [
+            "foo",
+            "bar"
+        ],
+        [
+            "baz",
+            "qux"
+        ]
+    ]
+}""",
+            ),
+            (
+                "records",
+                """[
+    {
+        "a":"foo",
+        "b":"bar"
+    },
+    {
+        "a":"baz",
+        "b":"qux"
+    }
+]""",
+            ),
+            (
+                "index",
+                """{
+    "0":{
+        "a":"foo",
+        "b":"bar"
+    },
+    "1":{
+        "a":"baz",
+        "b":"qux"
+    }
+}""",
+            ),
+            (
+                "columns",
+                """{
+    "a":{
+        "0":"foo",
+        "1":"baz"
+    },
+    "b":{
+        "0":"bar",
+        "1":"qux"
+    }
+}""",
+            ),
+            (
+                "values",
+                """[
+    [
+        "foo",
+        "bar"
+    ],
+    [
+        "baz",
+        "qux"
+    ]
+]""",
+            ),
+            (
+                "table",
+                """{
+    "schema":{
+        "fields":[
+            {
+                "name":"index",
+                "type":"integer"
+            },
+            {
+                "name":"a",
+                "type":"string"
+            },
+            {
+                "name":"b",
+                "type":"string"
+            }
+        ],
+        "primaryKey":[
+            "index"
+        ],
+        "pandas_version":"0.20.0"
+    },
+    "data":[
+        {
+            "index":0,
+            "a":"foo",
+            "b":"bar"
+        },
+        {
+            "index":1,
+            "a":"baz",
+            "b":"qux"
+        }
+    ]
+}""",
+            ),
+        ],
+    )
+    def test_json_indent_all_orients(self, orient, expected):
+        # GH 12004
+        df = pd.DataFrame([["foo", "bar"], ["baz", "qux"]], columns=["a", "b"])
+        result = df.to_json(orient=orient, indent=4)
+
+        if PY35:
+            assert json.loads(result) == json.loads(expected)
+        else:
+            assert result == expected
+
+    def test_json_negative_indent_raises(self):
+        with pytest.raises(ValueError, match="must be a nonnegative integer"):
+            pd.DataFrame().to_json(indent=-1)