Repository: pandas-dev/pandas

Issue:
setting with enlargement fails for large DataFrames
Setting with enlargement seems to fail for DataFrames longer than `10**6 - 1`
`10**6` seems to be the exact treshold for me. That and anything bigger fails. Anything smaller works.

Example:

```
import pandas as pd

#works 
X = pd.DataFrame(dict(x=range(10**6-1)))
X.loc[len(X)] = 42

#doesn't work
Y = pd.DataFrame(dict(y=range(10**6)))
Y.loc[len(Y)] = 42   

>>> IndexError: index out of bounds
```

pd.show_versions() returns:

```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: fi_FI

pandas: 0.16.2
nose: 1.3.6
Cython: 0.22
numpy: 1.9.2
scipy: 0.15.1
statsmodels: 0.6.1
IPython: 3.0.0
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.6.7
lxml: 3.4.2
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 0.9.9
pymysql: None
psycopg2: None
```



Discussion:
this is the same issue as in #10645 

the cases for len > 1M have different handling and something is amuck.

You know that you are copying the frame on enlargement right? This is extremely inefficient.

@jreback   What is the recommended way to do this?  This exact way is mentioned in the docs and doesn't seem to be discouraged there:  

http://pandas.pydata.org/pandas-docs/stable/indexing.html#setting-with-enlargement

what are you trying to do exactly?

I'm not trying to do anything!  Or maybe you are talking to the OP?  I was actually wondering the same thing, as I would generally use `append()` for this general thing.

But FWIW, these questions do come up at stack overflow with some regularity and if they found "setting with enlargement" in the documentation this is suggested as the way to do it. (or one of the ways, anyway).  And in this case what OP did was pretty much identical to the last example in the "setting with enlargement" doc.

@johne13 sorry, was on my phone.

So enlargement is equivalent of `df.append(Series(..., name=key))`. This creates by definition a copy. A doc note/warning would be nice here.

Actually I didn't know that the df is copied with every enlargement anyway. 
But yeah some warning in  docs would probably be nice to avoid misundersting.

---

About "what are you trying to do exactly?":

I just have a huge DataFrame where I append some information when it's returned from functions etc. I probably have to do some redesigning. I guess the way to go is somehow to preallocate the rows in the main DataFrame or collecting the "stuff to be appended" in some smaller list/df first and then appending all in the end.


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -4727,7 +4727,7 @@ def __contains__(self, key):
         try:
             self.get_loc(key)
             return True
-        except KeyError:
+        except LookupError:
             return False
 
     def __reduce__(self):
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -1040,7 +1040,7 @@ def _convert_to_indexer(self, obj, axis=0, is_setter=False):
         # if we are a label return me
         try:
             return labels.get_loc(obj)
-        except KeyError:
+        except LookupError:
             if isinstance(obj, tuple) and isinstance(labels, MultiIndex):
                 if is_setter and len(obj) == labels.nlevels:
                     return {'key': obj}
@@ -1125,7 +1125,7 @@ def _convert_to_indexer(self, obj, axis=0, is_setter=False):
         else:
             try:
                 return labels.get_loc(obj)
-            except KeyError:
+            except LookupError:
                 # allow a not found key only if we are a setter
                 if not is_list_like_indexer(obj) and is_setter:
                     return {'key': obj}


Test patch:
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -4602,7 +4602,17 @@ def test_indexing_dtypes_on_empty(self):
         assert_series_equal(df2.loc[:,'a'], df2.iloc[:,0])
         assert_series_equal(df2.loc[:,'a'], df2.ix[:,0])
 
+    def test_large_dataframe_indexing(self):
+        #GH10692
+        result = DataFrame({'x': range(10**6)})
+        result.loc[len(result)] = len(result) + 1
+        expected = DataFrame({'x': range(10**6 + 1)})
+        assert_frame_equal(result, expected)
 
+    def test_large_mi_dataframe_indexing(self):
+        #GH10645
+        result = MultiIndex.from_arrays([range(10**6), range(10**6)])
+        assert(not (10**6, 0) in result)
 
 class TestCategoricalIndex(tm.TestCase):