ENH: GH14883: json_normalize now takes a user-specified separator
State: closed

 - [x] closes #14883
 - [ ] tests added / passed (added 3 tests: `test_simple_normalize_with_{default, user_specified, user_specified_unicode}_separator`)
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry (v0.20.0)

However, this doesn't work, even after making the fixes suggested in https://github.com/pandas-dev/pandas/pull/14891. I thought replacing `'.'` in `meta_keys = ['.'.join(val) for val in meta]` with `sep` would do the trick. It doesn't, so I'm a little puzzled. Happy to take a suggestion.

Labels:
name: Enhancement
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
you don't need to open/close new PR's. simply push new commits to this one.
@jreback Thank you for all the kind comments! Um, can you take a minute or three to look at the actual code in json_normalize and clue me into where I'm going wrong? ('Cause it doesn't work.)
@jowens no idea, you'll have to step thru. my guess is that ``meta_keys`` is used for looking things up AND naming things.
can you update
I have no update, sorry to say. Someone who understands this code better is gonna be necessary to make this fix, I'm afraid.
@jowens I pushed a commit here. 3 issues. 

1) the nested record needed to also be updated with the sep (was .) 
2) you *can* create a columns via a dict, but its pretty useless as the ordering is not guaranteed, so you oftentimes mistmatch vs the values, passing a list is idiomatic
3) since the json ordering is essentially a dict, we are not guaranteed orderings vs our expected (in column space), but we can reindex to compare properly.

I think this might benefit from a deeper nested example as well.

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=h1) Report
> Merging [#14950](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/ec84ae3d6f73633aee5058148ea76fdd79f74ac4?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `85.71%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/14950/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #14950      +/-   ##
==========================================
- Coverage   90.99%   90.97%   -0.03%     
==========================================
  Files         143      143              
  Lines       49420    49422       +2     
==========================================
- Hits        44972    44963       -9     
- Misses       4448     4459      +11
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/normalize.py](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vbm9ybWFsaXplLnB5) | `96.93% <85.71%> (-0.98%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `90.96% <0%> (-0.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.56% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=footer). Last update [ec84ae3...0327dd1](https://codecov.io/gh/pandas-dev/pandas/pull/14950?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
@jowens can you update? (for the questions I had above)
Is the update you want "write a better test for 2 and 3, and add a deeper example"?
yep
can you rebase / add those addtl tests?
I'd like to, I just haven't had the opportunity to write them yet. Sorry.
@jowens np. just moving PR's along. pls ping when you are ready for review.