Avoids exception when pandas.io.json.json_normalize contains items in…
State: closed

Continued in https://github.com/pandas-dev/pandas/pull/14583

---

When using pandas.io.json.json_normalize to parse a nested json and convert it to a dataframe, the meta parameter can be used to use fields as metadata for each record in resulting table. In some cases, not all items may contain all of the specified meta fields. This change will avoid throwing an error and output np.nan instead.

… meta parameter that don't always occur in every item of the list


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14505?src=pr) is 85.27% (diff: 90.00%)

> Merging [#14505](https://codecov.io/gh/pandas-dev/pandas/pull/14505?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14505   diff @@
==========================================
  Files           140        140          
  Lines         50670      50698    +28   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43205      43233    +28   
  Misses         7465       7465          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [e3d943d...33495bc](https://codecov.io/gh/pandas-dev/pandas/compare/e3d943d1876e8914036e9c323f1876e52da34a04...33495bc5ac2a6b5d8184b543fc4f66934693b936?src=pr)

what exactly is this fixing? would need a test.

```
j={
    "Trades" : [{
            "general" : {
                "tradeid" : 100,
                "trade_version" : 1,
                "stocks" : [{

                        "symbol" : "AAPL",
                        "name" : "Apple",
                        "price" : "0"

                    }, {

                        "symbol" : "GOOG",
                        "name" : "Google",
                        "price" : "0"

                    }
                ]
            },
        }, {
            "general" : {
                "tradeid" : 100,
                "stocks" : [{

                        "symbol" : "AAPL",
                        "name" : "Apple",
                        "price" : "0"

                    }, {
                        "symbol" : "GOOG",
                        "name" : "Google",
                        "price" : "0"

                    }
                ]
            },
        }
    ]
}
json_normalize(data=j['Trades'], record_path=[['general','stocks']], meta=[['general','tradeid'],['general','trade_version']])
```

The above will fail because trade_version is only available in one of the two items in the list, so there is no way to output it if not all elements are exactly the same. With my change it will simply ignore it and output nan instead of throwing an error.

@dickreuter Can you add that as a test and add a release note? See http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-to-the-code-base

This would be the test, but unclear where I should store it. Any suggestions?

from unittest import TestCase
from pandas.io.json import json_normalize

```

class Tester(TestCase):
    def test_json_normalise_fix(self):
        j = {
            "Trades": [{
                "general": {
                    "tradeid": 100,
                    "trade_version": 1,
                    "stocks": [{

                        "symbol": "AAPL",
                        "name": "Apple",
                        "price": "0"

                    }, {

                        "symbol": "GOOG",
                        "name": "Google",
                        "price": "0"

                    }
                    ]
                },
            }, {
                "general": {
                    "tradeid": 100,
                    "stocks": [{

                        "symbol": "AAPL",
                        "name": "Apple",
                        "price": "0"

                    }, {
                        "symbol": "GOOG",
                        "name": "Google",
                        "price": "0"

                    }
                    ]
                },
            }
            ]
        }
        j = json_normalize(data=j['Trades'], record_path=[['general', 'stocks']],
                           meta=[['general', 'tradeid'], ['general', 'trade_version']])
        self.assertEqual(len(j), 4)

```

Looks like those tests are all in https://github.com/pandas-dev/pandas/blob/master/pandas/io/tests/json/test_json_norm.py

You could add it as a test method under `TestJSONNormalize`

Test and documentation is now added.

Added keyword errors {'raise'|'ignore}
Added documentation
Shortened what's new
Removed commas in dictionary for linting compatibility

you need to rebase on master

```
git rebase -i origin/master
```

I did that earlier today. It now says: "This branch is 8 commits ahead of pandas-dev:master.". There should currently be no more conflicts.

maybe you didn't push it
it's not about conflicts

My fork on github seems up to date with what I have locally, so I assume it has been pushed. Are there any further changes you expected me to implement that are not present?

@dickreuter its impossible to see until you rebase on master. this should have just your commits

https://github.com/pandas-dev/pandas/pull/14505/commits

I see, isn't it showing those commits of others only because I did a rebase of my fork (and then a local rebase of my local copy instead of a merge?). If that's a problem I could delete my fork and create a new one, then make the changes again and create a new pull request, unless you have a better suggestion.

you prob just need something like

```
git fetch origin
git rebase -i origin/master
git push yourremote thisbranchname -f
```

git fetch origin --> doesn't fetch anything as my local copy is in
sync with my remote origingit rebase -i origin/master --> noop - no
commits to pick

I think all my changes can be seen
here:https://github.com/pandas-dev/pandas/pull/14505/commits/984883796e0ee6e7355c2d5a746273c9551ead91

What may be confusing is that I also did an automatic reformatting
with pycharm to make the file conform with pep8. But those changes
only concern empty spaces.

On 3 November 2016 at 23:32, Jeff Reback notifications@github.com wrote:

> you prob just need something like
> 
> git fetch origin
> git rebase -i origin/master
> git push yourremote thisbranchname -f
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/pandas-dev/pandas/pull/14505#issuecomment-258305113,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/ABMrfm7iAtn0lL-LYi_wVB5pLLx7lS5Eks5q6m76gaJpZM4Khogo
> .

could also be called `upstream`

```
(pandas) [Thu Nov 03 19:52:46 ~/pandas]$ git remote -v|grep origin
origin  https://github.com/pandas-dev/pandas.git (fetch)
origin  https://github.com/pandas-dev/pandas.git (push)
```

it doesn't matter if your branch is in sync with YOUR upstream, rather it needs to be in sync with pandas master (and on top of it), that's what a rebase is.

you need to rebase to remove all of the merges of master. you shouldn't do that, instead rebase.

This seems to be the problem:
http://stackoverflow.com/questions/40413071/after-rebasing-my-github-fork-commits-from-others-are-in-my-pull-request/40413455#40413455

Will try to fix it, or if it's too complicated just delete and redo.

@jreback Regarding your comment above (https://github.com/pandas-dev/pandas/pull/14505#issuecomment-258305113): pandas-dev repo is typically called 'upstream', and your own fork 'origin' (that's how our contributor guide also says it), so you need `git fetch upstream` and `git rebase -i upstream/master` instead of `git fetch origin` and `git rebase -i origin/master` to rebase properly.

Follow-up in #14583