BUG: non-iterable value in meta raise error in json_normalize
State: closed

- [x] closes #31507 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-11 07:54:57 UTC
Can you add a whatsnew for v1.0.1?
oops, i was editing whatsnew just now 😅 
emm, 
```
self = <pandas.tests.io.json.test_normalize.TestNestedToRecord object at 0x14ab9d208>

    def test_meta_non_iterable(self):
        # GH 31507
        data = """[{"id": 99, "data": [{"one": 1, "two": 2}]}]"""
    
        result = json_normalize(json.loads(data), record_path=["data"], meta=["id"])
        expected_values = [[1, 2, "99"]]
        columns = ["one", "two", "id"]
        expected = DataFrame(expected_values, columns=columns)
>       tm.assert_frame_equal(result, expected)

pandas/tests/io/json/test_normalize.py:761: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise_assert_detail(obj, msg, lobj, robj)
E   AssertionError: DataFrame.iloc[:, 2] (column name="id") are different
E   
E   DataFrame.iloc[:, 2] (column name="id") values are different (100.0 %)
E   [left]:  [99]
E   [right]: [99]
```

is there a bug in `assert_frame_equal`? 🤔  will check it out tomorrow
@WillAyd @jreback I slightly changed the change made in #30145 and think this should be the cleaner solution to fix this bug, please take a look and any feedback is very appreciated!
I still find it pretty confusing; if we want to treat the meta differently than the record_path value why not just have a separate function or at least something to more clearly delineate? 
```
if not isinstance(result, Iterable):
    if pd.isnull(result):
         result = []  # type: ignore
    else:
         raise TypeError(
               f"{js} has non iterable value {result} for path {spec}. "
               "Must be iterable or null.")
```
this patch was added in #30145 in `_pull_field` function, however, this will not only affect when specifying `record_path`, but also `meta`, this is why we have this issue. And we would only like to make this patch for record_path, so my change here is to separate this patch out from `_pull_field` and only added it for record path. 

> if we want to treat the meta differently than the record_path value 

so actually we don't want to treat meta differently, but only treat record_path value differently by having this patch

> why not just have a separate function or at least something to more clearly delineate?

I think this is basically what i did here in this PR? separate the added patch out, and assign it only for record path?

@WillAyd maybe i didn't fully understand your comment, please feel free to comment and let me know what you think, thanks a lot! 👍 


@WillAyd any feedback comment? ^^
> this patch was added in #30145 in `_pull_field` function, however, this will not only affect when specifying `record_path`, but also `meta`

Right so I'm questioning that we even should do this. I think might be clearer if there were separate functions to handle each, which may dispatch to another for shared functionality.
Should this have a 1.0.2 label?
@WillAyd I made some change following your suggestion, does this look slightly clearer?
@WillAyd would appreciate for some feedbacks! ^^
@charlesdong1991, thanks for picking this one, i couldn't validate the transition to pandas 1.0.0 as it's impossible to parse data as it used to be.
Those failures are unrelated so can ignore for now. There is another PR in flight to fix

Sent from my iPhone

> On Mar 4, 2020, at 1:18 PM, Kaiqi Dong <notifications@github.com> wrote:
> 
> ﻿
> @charlesdong1991 commented on this pull request.
> 
> In pandas/io/json/_normalize.py:
> 
> > @@ -226,14 +226,26 @@ def _json_normalize(
>      Returns normalized data with columns prefixed with the given string.
>      """
>  
> -    def _pull_field(js: Dict[str, Any], spec: Union[List, str]) -> Iterable:
> +    def _pull_field_meta(js: Dict[str, Any], spec: Union[List, str]) -> Any:
> emm, i changed to Scalar, but some annotation errors raised in internal/manager.py, and even if i changed back to Any, still see these errors:
> 
> pandas/core/internals/managers.py:1109: error: Unsupported target for indexed assignment
> pandas/core/internals/managers.py:1111: error: Unsupported target for indexed assignment
> pandas/core/internals/managers.py:1145: error: Unsupported target for indexed assignment
> pandas/core/internals/managers.py:1146: error: Unsupported target for indexed assignment
> pandas/core/internals/managers.py:1160: error: Unsupported target for indexed assignment
> pandas/core/internals/managers.py:1161: error: Unsupported target for indexed assignment
> do you have any idea how/why those errors are raised? any hints are very appreciated, thanks! @WillAyd
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

many thanks for your quick response @WillAyd 

I will leave this PR then, and feel free to still take a look and see if anything else you want to see improvements. I will rebase once the fix PR is merged 
any further feedbacks? @WillAyd 

I think this would be nice to be included in `1.0.2` as it seems impact many users based on the responds under the Issue
@TomAugspurger thanks for your quick feedback, however, maybe something went wrong with github, your question about release note is not presented, could you please rewrite your question? thanks!
Strange. Is this fixing a regression? If so, the note should be towards the top of 1.0.2.rst with the other regression fixes.
thanks! @TomAugspurger 
this is not a regression issue fix IMHO, and maybe it is better to keep it in `I/O` section?
Yep, that sounds good.

On Tue, Mar 10, 2020 at 4:00 PM Kaiqi Dong <notifications@github.com> wrote:

> thanks! @TomAugspurger <https://github.com/TomAugspurger>
> this is not a regression issue IMHO, and shall I keep it in I/O section?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/31524?email_source=notifications&email_token=AAKAOIU2NMFUWM5SKIEJVOLRG2TAXA5CNFSM4KOO44P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEONEGEI#issuecomment-597312273>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQGSU6DDXCA7YS27X3RG2TAXANCNFSM4KOO44PQ>
> .
>

Sorry about the mess I made on the type annotation and function naming  @WillAyd 

I have committed a fix following ur suggestion, pls let me know your thoughts, and I will try to update PR asap to get this done before release.
CI is passing, so I'm planning to merge this in an hour or so if there aren't any objections. We can fixup annotations later if needed.
Thanks @charlesdong1991 . Great job seeing this through