JSON Date Handling 1.0 Regressions
State: closed

closes #30904

Came across these adding parametrized tests in #30903 which this pulls some elements from but which will also ultimately refactor to avoid duplication; just getting this to work as is

To be clear the regressions were:

1. ISO dates with datetime.date objects was broken

```python
>>> import pandas as pd
>>> import datetime
>>> data = [datetime.date(year=2020, month=1, day=1), "a"]
>>> pd.Series(data).to_json(date_format="iso")
TypeError: Expected datetime object
```

2. `pd.NaT` labels would now yield the sentinel value instead of writing "null"

```python
>>> data = [pd.Timestamp("2020-01-01"), pd.NaT]
>>> pd.Series(data, index=data).to_json()
'{"1577836800000":1577836800000,"-9223372036854":null}'
```

Both have been fixed here

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Hello @WillAyd! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-15 18:26:32 UTC
does this have similar perf to master?
Looks like a mixed bag:

```sh
       before           after         ratio
     [d78c0616]       [25444958]
     <master>         <json-date-regr>
+         115±4ms         231±10ms     2.02  io.json.ToJSON.time_to_json('columns', 'df')
+         103±3ms         161±10ms     1.57  io.json.ToJSON.time_to_json('records', 'df')
+        205±10ms         282±20ms     1.37  io.json.ToJSON.time_to_json_wide('index', 'df_int_floats')
-        265±30ms         194±10ms     0.73  io.json.ToJSON.time_to_json_wide('split', 'df_int_float_str')
-        235±10ms          141±9ms     0.60  io.json.ToJSON.time_to_json_wide('records', 'df')
-         276±7ms          141±4ms     0.51  io.json.ToJSON.time_to_json('index', 'df_td_int_ts')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```

May want to just couple this with #30903 which has a refactor which helps performance and fixes Timedelta issues, at the cost of a little more code movement
@WillAyd it looks like between this and #30903 perf is a mixed bag but there are some unambiguous bug fixes?
#30903 fixes #28256 and an unreported memory leak when using datetimelike labels.

There is overlap between that and this PR. I consider this the bare minimum to fix regressions, though would have a slight inclination towards doing this and #30903 (after rebase) to keep performance the same without jumping through a ton of hoops
> #30903 fixes #28256 and an unreported memory leak when using datetimelike labels.
> 
> There is overlap between that and this PR. I consider this the bare minimum to fix regressions, though would have a slight inclination towards doing this and #30903 (after rebase) to keep performance the same without jumping through a ton of hoops

not averse to doing another patch to fix perf, but is there a minimal patch you can add to this one that doesn't involve timedelta (which i view as a nice-to-have); the datetimes are the regression, yes?
Yea can probably extract something out tonirrow
To keep performance inline I pulled in the encodeLabels refactor from #30903, though not doing any of the Timedelta fixes. 

To help translate the diff, just note that the code now basically checks for up front for a date-like value and converts to its nanosecond resolution (note that NPY_DATETIME includes both numpy datetimes and time deltas). That nanosecond resolution gets checked against NaT and continues further to the appropriate serializer depending on the type of object supplied and other keyword arguments

Here are the benchmarks:

```sh
       before           after         ratio
     [bc9d329b]       [c102f5b9]
     <master>         <json-date-regr>
-        242±20ms          173±2ms     0.72  io.json.ToJSON.time_to_json_wide('columns', 'df_int_floats')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
thanks @WillAyd