ENH: Add `drop_na` keyword to "to_json()" function
State: closed

__What's new__

This PR introduces a new keyword, `drop_na`, to the `to_json()` function. It can be set to either `True` or `False`.

When `drop_na=False`, `to_json()` works just like it did before.

When `drop_na=True`, items with value `null` in the output of `to_json()` will be removed. Using this feature is (currently) only possible in combination with `orient="records"` and when the input object is of type `pd.DataFrame`. Removing the `null` items is done through string substitution with `re.sub()` in a new helper function (`remove_null_items()` in `_normalize.py`).

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

__Notes__

I imagine that, ideally, the `ultrajson`/`ujson` package had a `drop_none` parameter that would be adapted to include `NaN` values from pandas. IMO, this would be a better solution because it 1) is likely much faster and 2) avoids the use of regex. I started implementing such a parameter in `ujson` (see [here](https://github.com/romanhaa/ultrajson/commits/drop_none)), but and wondered whether it would introduce too much conditionality to be compatible with their philosophy (being ultra-fast). That's why I decided to propose a sub-optimal but working solution to `pandas` directly.

__Performance__

I ran some performance tests and compared it to a [workaround](https://stackoverflow.com/a/46955083) intended to achieve the same goal.

The input is a dataframe with 214,066 rows and 12 columns (read from a `.json` file, 58M in size, that can be found [here](http://ftp.ebi.ac.uk/pub/databases/opentargets/genetics/20022712/v2g/)).

Code snippet for testing:

```py
df = pd.read_json('part-00001-7a4fb376-056a-4f34-ad92-248989cd92d4-c000.json', lines=True)

print("export with `drop_na=False`")
start_time = time.time()
df.to_json("~/some_file_1.json", orient="records", lines=False, drop_na=False)
print("--- %s seconds ---" % (time.time() - start_time))

print("export with `drop_na=True`")
start_time = time.time()
df.to_json("~/some_file_2.json", orient="records", lines=False, drop_na=True)
print("--- %s seconds ---" % (time.time() - start_time))

print("export with workaround")
start_time = time.time()
with open("/Users/roman/some_file_3.json", "w", encoding="utf-8") as f:
    json.dump([row.dropna().to_dict() for index, row in df.iterrows()], f)
print("--- %s seconds ---" % (time.time() - start_time))
```

Output:

```
export with `to_json(orient='records', drop_na=False)`
--- 0.4323728084564209 seconds ---
export with `to_json(orient='records', drop_na=True)`
--- 1.6985130310058594 seconds ---
export with workaround
--- 79.73292803764343 seconds ---
```

The implementation I propose is not only (much) faster than the workaround, but is also compatible with generating output in `jsonlines` format.

Labels:
name: Stale

Comments:
was there a discussion on this addition in an issue? adding keywords is not always desirable and a discussion is useful to brainstorm whether other implementation options are available.
I had a look but couldn't find an issue related to this topic. I understand that adding keywords might not be desirable. For me this was primarily a coding exercise. I needed this feature some weeks ago and wondered how it could be implemented. No hard feelings if there is no general interest in this feature (even though I personally think it would be quite valuable).
I agree, @jreback . As I explained in my first post, that would be a much cleaner and faster, overall better solution. I opened a [draft PR](https://github.com/ultrajson/ultrajson/pull/463) for `ultrajson` (where I started implementing this feature in C), but I'm not sure whether such functionality is even in the scope of their package.

As for the usefulness, at least I think it is useful because I often data to JSON format with missing/empty items, which results in unnecessarily large files when those `null` values are all written to the file.
not ujson rather the inline pandas c code itself which is adapted 
Ah, well that's where I started but I figured that it would be easier to first implement a `drop_none` parameter in `ujson` (because `ujson` is only aware of `None`), and then extend it to `NA` values in the adaptation. Otherwise it will be quite some refactoring of the `ujson` code.
I was informed that ujson prefers not to introduce such functionality (see [here](https://github.com/ultrajson/ultrajson/pull/463#issuecomment-859391651)). The option remains to (properly) implement it here, in the adapted ujson code. I could look into that but would only do so if there is a realistic chance that it will get accepted.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but agreed that it would be best to first submit and enhancement request such that the API design and potential implementation can be scoped first. Closing, but happy for you to lead a discussion on this new proposal in an issue.