Repository: pandas-dev/pandas

Issue:
bug or unhelpful error message in 0.12 and trunk: IndexError: Out of bounds on buffer access (axis 0)
The following raises the error

```
pandas.Series([0.1, 0.2], index=[1, 2]).loc[[2, 3, 2]]
```

whilst

```
pandas.Series([0.1, 0.2], index=[1, 2]).loc[[2, 2, 3]]
```

does not. Or is there something invalid with this code?



Discussion:
Seem ok in master (they on don't raise because you have least 1 valid value)

```
In [1]: pandas.Series([0.1, 0.2], index=[1, 2]).loc[[2, 3, 2]]
Out[1]: 
2    0.2
3    NaN
2    0.2
dtype: float64

In [2]: pandas.Series([0.1, 0.2], index=[1, 2]).loc[[2, 2, 3]]
Out[2]: 
2    0.2
2    0.2
3    NaN
dtype: float64
```

either I've made a mistake in simplifying or the above scenario has been fixed in the last 2 days of trunk.
I've refreshed from git and can duplicate the problem with:

```
pandas.Series([0.1, 0.2], index=[1, 2]).loc[[3, 2, 3]]

In [69]: pandas.__version__
Out[69]: '0.12.0-399-g1409049'
```

did u make build?

eg regen the cython code 

I just ran:

```
python2 setup.py build_ext --inplace
```

Is that correct?

yep
run make clean
then setup again
does the full test suite pass?

I've deleted and refetched and rebuilt the pandas source, nosetests show only skips, no errors, the above still fails.

can u show
ci/print_versions.py ?

```
INSTALLED VERSIONS
------------------
Python: 2.7.5.final.0
OS: Linux 3.10.10-1-ARCH #1 SMP PREEMPT Fri Aug 30 11:30:06 CEST 2013 x86_64
byteorder: little
LC_ALL: None
LANG: en.US.UTF-8

Cython: 0.19.1
Numpy: 1.7.1
Scipy: Not installed
statsmodels: Not installed
    patsy: Not installed
scikits.timeseries: Not installed
dateutil: 2.1
pytz: 2013d
bottleneck: Not installed
PyTables: Not Installed
    numexpr: Not Installed
matplotlib: Not installed
openpyxl: Not installed
xlrd: Not installed
xlwt: Not installed
sqlalchemy: Not installed
lxml: Not installed
bs4: Not installed
html5lib: Not installed
```

It's running on freshly purpose-built VM with an up-to-date ArchLinux x64, although it also fails on 0.12 in Windows - just that without nightly windows builds and not being able to get pandas to compile with either mingw or Win SDK 7.0 I had to build this VM just for this bug!

Are you having difficulty reproducing?

@ruidc I can't reproduce this either.

Can you reproduce on 0.12 ?

@ruidc yes, it produces an error for me in 0.12.0, but this no longer occurs on current master.  I believe we're putting out either a point release or a new release version very soon.  I am, however, going to add this to the test suite.

Error was:

```
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pandas/core/indexing.py", line 699, in __getitem__
    return self._getitem_axis(key, axis=0)
  File "pandas/core/indexing.py", line 785, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "pandas/core/indexing.py", line 493, in _getitem_iterable
    result = result._reindex_with_indexers(*args, copy=False, fill_value=np.nan)
  File "pandas/core/series.py", line 2649, in _reindex_with_indexers
    new_values = com.take_1d(self.values, indexer, fill_value=fill_value)
  File "pandas/core/common.py", line 537, in take_nd
    func(arr, indexer, out, fill_value)
  File "generated.pyx", line 2560, in pandas.algos.take_1d_float64_float64 (pandas/algos.c:68204)
IndexError: Out of bounds on buffer access (axis 0)
```

can you post the full trace back u r getting?

@jreback see my comment above for traceback from the error I got on 0.12 [@ruidc might be getting something different]

Thanks, I look forward to Windows binaries from head working again, or an RC binary for 0.13.

```
In [1]: import pandas

In [2]: pandas.Series([0.1, 0.2], index=[1, 2]).loc[[3, 2, 3]]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-2-2abb8a62abc9> in <module>()
----> 1 pandas.Series([0.1, 0.2], index=[1, 2]).loc[[3, 2, 3]]

/home/src/pandas/pandas/core/indexing.pyc in __getitem__(self, key)
    933             return self._getitem_tuple(key)
    934         else:
--> 935             return self._getitem_axis(key, axis=0)
    936
    937     def _getitem_axis(self, key, axis=0):

/home/src/pandas/pandas/core/indexing.pyc in _getitem_axis(self, key, axis)
   1025                 raise ValueError('Cannot index with multidimensional key')
   1026
-> 1027             return self._getitem_iterable(key, axis=axis)
   1028         else:
   1029             return self._get_label(key, axis=axis)

/home/src/pandas/pandas/core/indexing.pyc in _getitem_iterable(self, key, axis)
    716                         raise AssertionError("invalid indexing error with non-unique index")
    717
--> 718                     result = result._reindex_with_indexers({ axis : [ new_labels, new_indexer ] }, copy=True, allow_dups=True)
    719
    720                 return result

/home/src/pandas/pandas/core/generic.pyc in _reindex_with_indexers(self, reindexers, method, fill_value, limit, copy, allow_dups)
   1214                 indexer = com._ensure_int64(indexer)
   1215                 new_data = new_data.reindex_indexer(index, indexer, axis=baxis,
-> 1216                                                     fill_value=fill_value, allow_dups=allow_dups)
   1217
   1218             elif baxis == 0 and index is not None and index is not new_data.axes[baxis]:

/home/src/pandas/pandas/core/internals.pyc in reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups)
   2830
   2831         if axis == 0:
-> 2832             return self._reindex_indexer_items(new_axis, indexer, fill_value)
   2833
   2834         new_blocks = []

/home/src/pandas/pandas/core/internals.pyc in _reindex_indexer_items(self, new_items, indexer, fill_value)
   3178     def _reindex_indexer_items(self, new_items, indexer, fill_value):
   3179         # equiv to a reindex
-> 3180         return self.reindex(new_items, indexer=indexer, fill_value=fill_value, copy=False)
   3181
   3182     def reindex_axis0_with_method(self, new_axis, indexer=None, method=None, fill_value=None, limit=None, copy=True):

/home/src/pandas/pandas/core/internals.pyc in reindex(self, new_axis, indexer, method, fill_value, limit, copy)
   3171
   3172         block = self._block.reindex_items_from(new_axis, indexer=indexer, method=method,
-> 3173                                                fill_value=fill_value, limit=limit, copy=copy)
   3174         mgr = SingleBlockManager(block, new_axis)
   3175         mgr._consolidate_inplace()

/home/src/pandas/pandas/core/internals.pyc in reindex_items_from(self, new_ref_items, indexer, method, fill_value, limit, copy)
    234             if self.ndim == 1:
    235                 new_values = com.take_1d(self.values, indexer,
--> 236                                          fill_value=fill_value)
    237             else:
    238

/home/src/pandas/pandas/core/common.pyc in take_nd(arr, indexer, axis, out, fill_value, mask_info, allow_fill)
    551     func = _get_take_nd_function(arr.ndim, arr.dtype, out.dtype,
    552                                  axis=axis, mask_info=mask_info)
--> 553     func(arr, indexer, out, fill_value)
    554     return out
    555

/home/src/pandas/pandas/algos.so in pandas.algos.take_1d_float64_float64 (pandas/algos.c:68204)()

IndexError: Out of bounds on buffer access (axis 0)
```

no this changed from 0.12
this is partial setting where the frame is expanded for the missing values 


Resolving patch:
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -701,6 +701,11 @@ def _reindex(keys, level=None):
                     new_labels[cur_indexer]     = cur_labels
                     new_labels[missing_indexer] = missing_labels
 
+                    # reindex with the specified axis
+                    ndim = self.obj.ndim
+                    if axis+1 > ndim:
+                        raise AssertionError("invalid indexing error with non-unique index")
+
                     # a unique indexer
                     if keyarr_is_unique:
                         new_indexer = (Index(cur_indexer) + Index(missing_indexer)).values
@@ -708,12 +713,15 @@ def _reindex(keys, level=None):
 
                     # we have a non_unique selector, need to use the original indexer here
                     else:
-                        new_indexer = indexer
 
-                    # reindex with the specified axis
-                    ndim = self.obj.ndim
-                    if axis+1 > ndim:
-                        raise AssertionError("invalid indexing error with non-unique index")
+                        # need to retake to have the same size as the indexer
+                        rindexer = indexer.values
+                        rindexer[~check] = 0
+                        result = self.obj.take(rindexer, axis=axis, convert=False)
+
+                        # reset the new indexer to account for the new size
+                        new_indexer = np.arange(len(result))
+                        new_indexer[~check] = -1
 
                     result = result._reindex_with_indexers({ axis : [ new_labels, new_indexer ] }, copy=True, allow_dups=True)
 


Test patch:
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -1436,6 +1436,49 @@ def f():
         p.loc[:,:,'C'] = Series([30,32],index=p_orig.items)
         assert_panel_equal(p,expected)
 
+    def test_series_partial_set(self):
+        # partial set with new index
+        # Regression from GH4825
+        ser = Series([0.1, 0.2], index=[1, 2])
+
+        # loc
+        expected = Series([np.nan, 0.2, np.nan], index=[3, 2, 3])
+        result = ser.loc[[3, 2, 3]]
+        assert_series_equal(result, expected)
+
+        expected = Series([np.nan, np.nan, np.nan], index=[3, 3, 3])
+        result = ser.loc[[3, 3, 3]]
+        assert_series_equal(result, expected)
+
+        expected = Series([0.2, 0.2, np.nan], index=[2, 2, 3])
+        result = ser.loc[[2, 2, 3]]
+        assert_series_equal(result, expected)
+
+        expected = Series([0.3, np.nan, np.nan], index=[3, 4, 4])
+        result = Series([0.1, 0.2, 0.3], index=[1,2,3]).loc[[3,4,4]]
+        assert_series_equal(result, expected)
+
+        expected = Series([np.nan, 0.3, 0.3], index=[5, 3, 3])
+        result = Series([0.1, 0.2, 0.3, 0.4], index=[1,2,3,4]).loc[[5,3,3]]
+        assert_series_equal(result, expected)
+
+        expected = Series([np.nan, 0.4, 0.4], index=[5, 4, 4])
+        result = Series([0.1, 0.2, 0.3, 0.4], index=[1,2,3,4]).loc[[5,4,4]]
+        assert_series_equal(result, expected)
+
+        expected = Series([0.4, np.nan, np.nan], index=[7, 2, 2])
+        result = Series([0.1, 0.2, 0.3, 0.4], index=[4,5,6,7]).loc[[7,2,2]]
+        assert_series_equal(result, expected)
+
+        expected = Series([0.4, np.nan, np.nan], index=[4, 5, 5])
+        result = Series([0.1, 0.2, 0.3, 0.4], index=[1,2,3,4]).loc[[4,5,5]]
+        assert_series_equal(result, expected)
+
+        # iloc
+        expected = Series([0.2,0.2,0.1,0.1], index=[2,2,1,1])
+        result = ser.iloc[[1,1,0,0]]
+        assert_series_equal(result, expected)
+
 if __name__ == '__main__':
     import nose
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],