Repository: pandas-dev/pandas

Issue:
json_normalize gives KeyError in 0.23
#### Code Sample, a copy-pastable example if possible

```python
import json
from pandas import show_versions
from pandas.io.json import json_normalize

print(show_versions())

with open('test.json', 'r') as infile:
    d = json.load(infile)

normed = json_normalize(d)
```

The `test.json` file is rather lengthy, with a structure similar to:

```json
{
  "subject": {
    "pairs": {
      "A1-A2": {
        "atlases": {
          "avg.corrected": {
            "region": null,
            "x": 49.151580810546875,
            "y": -33.148521423339844,
            "z": 27.572303771972656
          }
        }
      }
    }
  }
}
```

This minimal version is enough to show the error below.

#### Problem description

This problem is *new* in pandas 0.23. I get the following traceback:

Traceback:

```pytb
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    normed = json_normalize(d)
  File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 203, in json_normalize
    data = nested_to_record(data, sep=sep)
  File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 88, in nested_to_record
    new_d.update(nested_to_record(v, newkey, sep, level + 1))
  File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 88, in nested_to_record
    new_d.update(nested_to_record(v, newkey, sep, level + 1))
  File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 88, in nested_to_record
    new_d.update(nested_to_record(v, newkey, sep, level + 1))
  File "/Users/depalati/miniconda3/lib/python3.6/site-packages/pandas/io/json/normalize.py", line 84, in nested_to_record
    new_d.pop(k)
KeyError: 'region'
```

Note that running the same code on pandas 0.22 does not result in any errors. I suspect this could be related to #20399.

#### Expected Output

Expected output is a flattened `DataFrame` without any errors.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: 3.5.1
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.2
scipy: None
pyarrow: None
xarray: 0.10.3
IPython: 6.3.1
sphinx: 1.7.2
patsy: None
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
I couldn't reproduce your error with the information provide (was getting others) - can you please update it so the example can be fully copy/pasted to reproduce?
I'm not sure what errors you are getting. Here's a version with the JSON contents directly in the Python file as a dict:

```python
from pandas import show_versions
from pandas.io.json import json_normalize

print(show_versions())

d = {
  "subject": {
    "pairs": {
      "A1-A2": {
        "atlases": {
          "avg.corrected": {
            "region": None,
            "x": 49.151580810546875,
            "y": -33.148521423339844,
            "z": 27.572303771972656
          }
        }
      }
    }
  }
}

normed = json_normalize(d)
print(normed)
```

This results in the same error.
Running his code I have the same error as well.

https://github.com/pandas-dev/pandas/blob/master/pandas/io/json/normalize.py#L79

I think the problem is here.

If I add two print statements:

```
             if not isinstance(v, dict):
                 print("cond1: %s" % (level != 0))
                 print("cond2: %s" % (v is None))

                  if level != 0:  # so we skip copying for top level, common case
                      print(new_d)
                      v = new_d.pop(k)
                      new_d[newkey] = v
                  if v is None:  # pop the key if the value is None
                      print(new_d)
                      new_d.pop(k)
                  continue
```
I get the following printout:

cond1: True
cond2: True
{'region': None, 'x': 49.151580810546875, 'y': -33.148521423339844, 'z': 27.572303771972656}
{'x': 49.151580810546875, 'y': -33.148521423339844, 'z': 27.572303771972656, 'subject.pairs.A1-A2.atlases.avg.corrected.region': None}

So new_d is getting popped twice which causes an error on the second time?



I added a continue in the code:
              
  if level != 0:  # so we skip copying for top level, common case
                    v = new_d.pop(k)
                    new_d[newkey] = v
                    continue

and the code looks to run fine.





Also as mivade pointed out these lines of code were changed in  #20399

https://github.com/pandas-dev/pandas/pull/20399/files#diff-9c654764f5f21c8e9d58d9ebf14de86dR83

Resolving patch:
diff --git a/pandas/io/json/normalize.py b/pandas/io/json/normalize.py
--- a/pandas/io/json/normalize.py
+++ b/pandas/io/json/normalize.py
@@ -80,7 +80,7 @@ def nested_to_record(ds, prefix="", sep=".", level=0):
                 if level != 0:  # so we skip copying for top level, common case
                     v = new_d.pop(k)
                     new_d[newkey] = v
-                if v is None:  # pop the key if the value is None
+                elif v is None:  # pop the key if the value is None
                     new_d.pop(k)
                 continue
             else:


Test patch:
diff --git a/pandas/tests/io/json/test_normalize.py b/pandas/tests/io/json/test_normalize.py
--- a/pandas/tests/io/json/test_normalize.py
+++ b/pandas/tests/io/json/test_normalize.py
@@ -375,3 +375,59 @@ def test_nonetype_dropping(self):
              'info.last_updated': '26/05/2012'}]
 
         assert result == expected
+
+    def test_nonetype_top_level_bottom_level(self):
+        # GH21158: If inner level json has a key with a null value
+        # make sure it doesnt do a new_d.pop twice and except
+        data = {
+            "id": None,
+            "location": {
+                "country": {
+                    "state": {
+                        "id": None,
+                        "town.info": {
+                            "id": None,
+                            "region": None,
+                            "x": 49.151580810546875,
+                            "y": -33.148521423339844,
+                            "z": 27.572303771972656}}}
+            }
+        }
+        result = nested_to_record(data)
+        expected = {
+            'location.country.state.id': None,
+            'location.country.state.town.info.id': None,
+            'location.country.state.town.info.region': None,
+            'location.country.state.town.info.x': 49.151580810546875,
+            'location.country.state.town.info.y': -33.148521423339844,
+            'location.country.state.town.info.z': 27.572303771972656}
+        assert result == expected
+
+    def test_nonetype_multiple_levels(self):
+        # GH21158: If inner level json has a key with a null value
+        # make sure it doesnt do a new_d.pop twice and except
+        data = {
+            "id": None,
+            "location": {
+                "id": None,
+                "country": {
+                    "id": None,
+                    "state": {
+                        "id": None,
+                        "town.info": {
+                            "region": None,
+                            "x": 49.151580810546875,
+                            "y": -33.148521423339844,
+                            "z": 27.572303771972656}}}
+            }
+        }
+        result = nested_to_record(data)
+        expected = {
+            'location.id': None,
+            'location.country.id': None,
+            'location.country.state.id': None,
+            'location.country.state.town.info.region': None,
+            'location.country.state.town.info.x': 49.151580810546875,
+            'location.country.state.town.info.y': -33.148521423339844,
+            'location.country.state.town.info.z': 27.572303771972656}
+        assert result == expected