Repository: pandas-dev/pandas

Issue:
Strange behaviour with MultiIndex and more than one million records
#### Code Sample

```python
r = list(range(999999))
df2 = pd.DataFrame({'a':r, 'b':r}, index=pd.MultiIndex.from_tuples([(x,x) for x in r]))
df2['a'].blabla()
# result as expected: AttributeError: 'Series' object has no attribute 'blabla'

r = list(range(1000000))
df2 = pd.DataFrame({'a':r, 'b':r}, index=pd.MultiIndex.from_tuples([(x,x) for x in r]))
df2['a'].blabla()
#strange result: TypeError: unorderable types: numpy.ndarray() < str()

from scipy.stats.mstats import winsorize
winsorize(df2['a'])
#result: TypeError: unorderable types: numpy.ndarray() < str()
```
#### Problem description

When there are exactly one million records or more with a MultiIndex, I get this TypeError when I call a non-existent method on a column. I get the same error when I try to call the winsorize function from scipy.stats on a column. The type error originates here: File "pandas/index.pyx", line 481, in pandas.index._bin_search

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-98-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: nb_NO.UTF-8
LOCALE: nb_NO.UTF-8

pandas: 0.21.0
pytest: 3.2.2
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.26.1
numpy: 1.13.1
scipy: 0.19.1
pyarrow: None
xarray: 0.9.6
IPython: 2.4.1
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: 2.4.4
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
show versions as the issue template indicates 
Done
so indexing with a million elements is different internally 
so we are prob not catching and reraising the correct exception here
this hits indexing because you are accessing an attribute , which u then try to call 
attribute access can be for columns as well so that’s why indexing is hit

if you would trace would be great - this will end up in cython in index.pyx
 File "/usr/local/lib/python3.5/site-packages/scipy/stats/mstats_basic.py", line 1419, in winsorize
    a = ma.array(a, copy=np.logical_not(inplace))
  File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 6152, in array
    ndmin=ndmin, shrink=shrink, order=order)
  File "/usr/local/lib/python3.5/site-packages/numpy/ma/core.py", line 2762, in __new__
    _baseclass = getattr(data, '_baseclass', type(_data))
  File "/usr/local/lib/python3.5/site-packages/pandas/core/generic.py", line 2742, in __getattr__
    if name in self._info_axis:
  File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1042, in __contains__
    self.get_loc(key)
  File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1643, in get_loc
    loc = self._get_level_indexer(key, level=0)
  File "/usr/local/lib/python3.5/site-packages/pandas/indexes/multi.py", line 1905, in _get_level_indexer
    loc = level_index.get_loc(key)
  File "/usr/local/lib/python3.5/site-packages/pandas/indexes/numeric.py", line 341, in get_loc
    tolerance=tolerance)
  File "/usr/local/lib/python3.5/site-packages/pandas/indexes/base.py", line 2134, in get_loc
    return self._engine.get_loc(key)
  File "pandas/index.pyx", line 132, in pandas.index.IndexEngine.get_loc (pandas/index.c:4433)
  File "pandas/index.pyx", line 140, in pandas.index.IndexEngine.get_loc (pandas/index.c:4075)
  File "pandas/index.pyx", line 481, in pandas.index._bin_search (pandas/index.c:9823)
TypeError: unorderable types: numpy.ndarray() < str()

BTW, the winsorize function works with 999999 records
ok the dundee contains methods in indexes/multi 

a) move way up in the file it should be much closer to the top near other finder methods. (i know get/set state are there as well but leave those)
b) catch TypeError as well as LookupError to return False
c) test all of the cases u gave
can u do a PR?
I'm afraid I don't have the time to look further into it right now, I'm in the process of working around it in my project. 
@olevold ok ; things get fixed by folks contributing

Resolving patch:
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -446,6 +446,17 @@ def _shallow_copy_with_infer(self, values=None, **kwargs):
                               **kwargs)
         return self._shallow_copy(values, **kwargs)
 
+    @Appender(_index_shared_docs['__contains__'] % _index_doc_kwargs)
+    def __contains__(self, key):
+        hash(key)
+        try:
+            self.get_loc(key)
+            return True
+        except (LookupError, TypeError):
+            return False
+
+    contains = __contains__
+
     @Appender(_index_shared_docs['_shallow_copy'])
     def _shallow_copy(self, values=None, **kwargs):
         if values is not None:
@@ -1370,17 +1381,6 @@ def nlevels(self):
     def levshape(self):
         return tuple(len(x) for x in self.levels)
 
-    @Appender(_index_shared_docs['__contains__'] % _index_doc_kwargs)
-    def __contains__(self, key):
-        hash(key)
-        try:
-            self.get_loc(key)
-            return True
-        except LookupError:
-            return False
-
-    contains = __contains__
-
     def __reduce__(self):
         """Necessary for making this object picklable"""
         d = dict(levels=[lev for lev in self.levels],


Test patch:
diff --git a/pandas/tests/indexes/test_multi.py b/pandas/tests/indexes/test_multi.py
--- a/pandas/tests/indexes/test_multi.py
+++ b/pandas/tests/indexes/test_multi.py
@@ -2981,3 +2981,13 @@ def test_nan_stays_float(self):
         assert pd.isna(df0.index.get_level_values(1)).all()
         # the following failed in 0.14.1
         assert pd.isna(dfm.index.get_level_values(1)[:-1]).all()
+
+    def test_million_record_attribute_error(self):
+        # GH 18165
+        r = list(range(1000000))
+        df = pd.DataFrame({'a': r, 'b': r},
+                          index=pd.MultiIndex.from_tuples([(x, x) for x in r]))
+
+        with tm.assert_raises_regex(AttributeError,
+                                    "'Series' object has no attribute 'foo'"):
+            df['a'].foo()