Repository: pandas-dev/pandas (python)

Issue:
BUG: None value in MultiIndex can't be found
### Pandas version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas.

- [ ] I have confirmed this bug exists on the [main branch](https://pandas.pydata.org/docs/dev/getting_started/install.html#installing-the-development-version-of-pandas) of pandas.


### Reproducible Example

```python
import pandas as pd

indices = [["one"], ["a"], ["yes"]]
df = pd.DataFrame([0], index=indices).T
df["one", None, "yes"] = 1
print(df)
print()
print(df["one"])
print()
print(df["one"][None])
```


### Issue Description

The dataframe has an item at the multiindex of ["one", None, "yes"], but attempting to access it fails. It appears to only happen when I use None/nan values for the middle index that I'm adding, and it only happens if I add the value using the df[] = syntax. 

### Expected Behavior

It should return a value when I query df["one"][None]. If I construct the dataframe with this data in it to begin with, or add it with a df.concat, it works.

### Installed Versions

<details>

INSTALLED VERSIONS
------------------
commit                : d9cdd2ee5a58015ef6f4d15c7226110c9aab8140
python                : 3.9.2.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 4.18.0-513.24.1.el8_9.x86_64
Version               : #1 SMP Thu Mar 14 14:20:09 EDT 2024
machine               : x86_64
processor             : x86_64
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 2.2.2
numpy                 : 1.23.4
pytz                  : 2024.1
dateutil              : 2.9.0.post0
setuptools            : 70.0.0
pip                   : 23.1
Cython                : None
pytest                : 7.4.4
hypothesis            : None
sphinx                : 7.3.7
blosc                 : None
feather               : None
xlsxwriter            : 3.2.0
lxml.etree            : 5.2.2
html5lib              : None
pymysql               : None
psycopg2              : None
jinja2                : 3.1.4
IPython               : 8.18.1
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : 2024.6.0
gcsfs                 : None
matplotlib            : 3.9.0
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : 3.1.2
pandas_gbq            : None
pyarrow               : None
pyreadstat            : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : None
sqlalchemy            : 2.0.12
tables                : None
tabulate              : None
xarray                : None
xlrd                  : 2.0.1
zstandard             : None
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None

</details>


Pull request description:
Fix insertion of None value in MultiIndex
- [x] closes #59003 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Patch:
diff --git a/ci/code_checks.sh b/ci/code_checks.sh
index 41564e6426..0fa61a2465 100755
--- a/ci/code_checks.sh
+++ b/ci/code_checks.sh
@@ -162,7 +162,7 @@ if [[ -z "$CHECK" || "$CHECK" == "docstrings" ]]; then
         -i "pandas.Series.ne SA01" \
         -i "pandas.Series.pad PR01,SA01" \
         -i "pandas.Series.plot PR02" \
-        -i "pandas.Series.pop RT03,SA01" \
+        -i "pandas.Series.pop SA01" \
         -i "pandas.Series.prod RT03" \
         -i "pandas.Series.product RT03" \
         -i "pandas.Series.reorder_levels RT03,SA01" \
diff --git a/doc/source/whatsnew/v3.0.0.rst b/doc/source/whatsnew/v3.0.0.rst
index ec7b8b375a..e06df92a22 100644
--- a/doc/source/whatsnew/v3.0.0.rst
+++ b/doc/source/whatsnew/v3.0.0.rst
@@ -545,6 +545,7 @@ MultiIndex
 ^^^^^^^^^^
 - :func:`DataFrame.loc` with ``axis=0``  and :class:`MultiIndex` when setting a value adds extra columns (:issue:`58116`)
 - :meth:`DataFrame.melt` would not accept multiple names in ``var_name`` when the columns were a :class:`MultiIndex` (:issue:`58033`)
+- :meth:`MultiIndex.insert` would not insert NA value correctly at unified location of index -1 (:issue:`59003`)
 -
 
 I/O
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
index 9d7c7f3e4a..8e3ebc7816 100644
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -3906,8 +3906,11 @@ class MultiIndex(Index):
                 # have to insert into level
                 # must insert at end otherwise you have to recompute all the
                 # other codes
-                lev_loc = len(level)
-                level = level.insert(lev_loc, k)
+                if isna(k):  # GH 59003
+                    lev_loc = -1
+                else:
+                    lev_loc = len(level)
+                    level = level.insert(lev_loc, k)
             else:
                 lev_loc = level.get_loc(k)
 
diff --git a/pandas/core/series.py b/pandas/core/series.py
index ee34240366..0a67ad7e57 100644
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -5020,7 +5020,8 @@ class Series(base.IndexOpsMixin, NDFrame):  # type: ignore[misc]
 
         Returns
         -------
-        Value that is popped from series.
+        scalar
+            Value that is popped from series.
 
         Examples
         --------


Test patch:
diff --git a/pandas/tests/dtypes/test_dtypes.py b/pandas/tests/dtypes/test_dtypes.py
index 252fc484a8..903c135871 100644
--- a/pandas/tests/dtypes/test_dtypes.py
+++ b/pandas/tests/dtypes/test_dtypes.py
@@ -1231,3 +1231,15 @@ def test_multi_column_dtype_assignment():
 
     df["b"] = 0
     tm.assert_frame_equal(df, expected)
+
+
+def test_loc_setitem_empty_labels_no_dtype_conversion():
+    # GH 29707
+
+    df = pd.DataFrame({"a": [2, 3]})
+    expected = df.copy()
+    assert df.a.dtype == "int64"
+    df.loc[[]] = 0.1
+
+    assert df.a.dtype == "int64"
+    tm.assert_frame_equal(df, expected)
diff --git a/pandas/tests/groupby/test_groupby.py b/pandas/tests/groupby/test_groupby.py
index 3506fb55e0..13fb9cfc4c 100644
--- a/pandas/tests/groupby/test_groupby.py
+++ b/pandas/tests/groupby/test_groupby.py
@@ -2985,3 +2985,14 @@ def test_groupby_agg_namedagg_with_duplicate_columns():
     )
 
     tm.assert_frame_equal(result, expected)
+
+
+def test_groupby_multi_index_codes():
+    # GH#54347
+    df = DataFrame(
+        {"A": [1, 2, 3, 4], "B": [1, float("nan"), 2, float("nan")], "C": [2, 4, 6, 8]}
+    )
+    df_grouped = df.groupby(["A", "B"], dropna=False).sum()
+
+    index = df_grouped.index
+    tm.assert_index_equal(index, MultiIndex.from_frame(index.to_frame()))
diff --git a/pandas/tests/test_multilevel.py b/pandas/tests/test_multilevel.py
index 8f661edf0f..e874987420 100644
--- a/pandas/tests/test_multilevel.py
+++ b/pandas/tests/test_multilevel.py
@@ -288,6 +288,13 @@ class TestMultiLevel:
 
         tm.assert_frame_equal(df, expected)
 
+    @pytest.mark.parametrize("na", [None, np.nan])
+    def test_multiindex_insert_level_with_na(self, na):
+        # GH 59003
+        df = DataFrame([0], columns=[["A"], ["B"]])
+        df[na, "B"] = 1
+        tm.assert_frame_equal(df[na], DataFrame([1], columns=["B"]))
+
 
 class TestSorted:
     """everything you wanted to test about sorting"""