Repository: pandas-dev/pandas

Issue:
Clarify that MultiIndex.set_levels() interprets passed values as new components of the .levels attribute
#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.rand(3, 3), columns=pd.MultiIndex.from_tuples([(1, 2), (1, 4), (5, 6)]))
df.columns = df.columns.set_levels([1, 1, 3], level=0)
```
Which raises a ```ValueError```:

```python
ValueError: Level values must be unique: [1, 1, 3] on level 0
```

#### Problem description

Despite a dataframe with non-unique MultiIndex can be created, they cannot be set using ```set_levels()```. Is this behaviour expected? I believe non-unique level values were not allowed for a period of time (#18882), but then they were allowed again (#21423), so I am not sure which is the current convention.

#### Expected Output

```python
          1                   3
          2         4         6
0  0.317669  0.329142  0.056725
1  0.969472  0.340309  0.135204
2  0.242408  0.934748  0.683186
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.1
numpy            : 1.16.4
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.2
setuptools       : 41.0.1
Cython           : 0.29.13
pytest           : 5.1.2
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.8.0
pandas_datareader: None
bs4              : None
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : None
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.7
tables           : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : 1.1.8

</details>



Discussion:
@toobaz 
>  I believe non-unique level values were not allowed for a period of time (#18882), but then they were allowed again (#21423), so I am not sure which is the current convention.

Those issues were about non-unique level _names_, yours about non-unique level _values_. And in this second case the story is simple: yes, they are and always were allowed.

But I think you've misunderstood how ``set_levels`` works (possibly because the docs could and should be improved): you're not assumed to provide a new value for each row, but rather _for each code_ describing values in the level. In other words, you are changing the ``levels`` attribute. And _there_ having duplicates does not make a lot of sense. This said, you _can_ do it if you really want, by passing ``verify_integrity=False``.

In short, this function is probably not what you're looking for. And unfortunately, there are no obvious alternative to do what you want to do.
The simplest way is to recreate the index.
The most efficient way might be to use ``set_levels`` in combination with ``set_codes`` (you can obtain levels and codes from a simple ``MultiIndex`` with 1 level).
Should we at least keep this open to improve the docs then?
> Should we at least keep this open to improve the docs then?

Makes sense!
I could give a try at improving the docs for MultiIndex.set_levels(), if that's okay.
@hweecat sure!

Resolving patch:
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -742,32 +742,47 @@ def set_levels(self, levels, level=None, inplace=False, verify_integrity=True):
         Examples
         --------
         >>> idx = pd.MultiIndex.from_tuples([(1, 'one'), (1, 'two'),
-                                            (2, 'one'), (2, 'two')],
+                                            (2, 'one'), (2, 'two'),
+                                            (3, 'one'), (3, 'two')],
                                             names=['foo', 'bar'])
-        >>> idx.set_levels([['a', 'b'], [1, 2]])
+        >>> idx.set_levels([['a', 'b', 'c'], [1, 2]])
         MultiIndex([('a', 1),
                     ('a', 2),
                     ('b', 1),
-                    ('b', 2)],
+                    ('b', 2),
+                    ('c', 1),
+                    ('c', 2)],
                    names=['foo', 'bar'])
-        >>> idx.set_levels(['a', 'b'], level=0)
+        >>> idx.set_levels(['a', 'b', 'c'], level=0)
         MultiIndex([('a', 'one'),
                     ('a', 'two'),
                     ('b', 'one'),
-                    ('b', 'two')],
+                    ('b', 'two'),
+                    ('c', 'one'),
+                    ('c', 'two')],
                    names=['foo', 'bar'])
         >>> idx.set_levels(['a', 'b'], level='bar')
         MultiIndex([(1, 'a'),
                     (1, 'b'),
                     (2, 'a'),
-                    (2, 'b')],
+                    (2, 'b'),
+                    (3, 'a'),
+                    (3, 'b')],
                    names=['foo', 'bar'])
-        >>> idx.set_levels([['a', 'b'], [1, 2]], level=[0, 1])
+
+        If any of the levels passed to ``set_levels()`` exceeds the
+        existing length, all of the values from that argument will
+        be stored in the MultiIndex levels, though the values will
+        be truncated in the MultiIndex output.
+
+        >>> idx.set_levels([['a', 'b', 'c'], [1, 2, 3, 4]], level=[0, 1])
         MultiIndex([('a', 1),
                     ('a', 2),
                     ('b', 1),
                     ('b', 2)],
                    names=['foo', 'bar'])
+        >>> idx.set_levels([['a', 'b', 'c'], [1, 2, 3, 4]], level=[0, 1]).levels
+        FrozenList([['a', 'b', 'c'], [1, 2, 3, 4]])
         """
         if is_list_like(levels) and not isinstance(levels, Index):
             levels = list(levels)


Test patch: