internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pandas-dev/pandas

Issue:
pandas.read_msgpack(None) returns [78, 111, 110, 101]
It should return an error, as that's what it most likely is. 

If someone is really sending empty msgpacks out there to transmit [78, 111, 110, 101], they could add a special case.  



Discussion:
yep, that should raise. The idea is you can simply pass a string or a file-like object (e.g. `StringIO`). So needs to be a test for a `.read` method on the file-like, and otherwise raise. Want to do a pull-request?

Yes, but it should work for strings, which have no `read()`. For example, the following should succeed: 

 `pandas.read_msgpack('\x87\xa5index\x85\xa5dtype\x07\xa3typ\xa5index\xa5klass\xaaInt64Index\xa4data\xa8\x00\x00\x00\x00\x00\x00\x00\x00\xa4name\xc0\xa4name\xc0\xa3typ\xa6series\xa8compress\xc0\xa5klass\xa6Series\xa5dtype\x07\xa4data\xa8\x01\x00\x00\x00\x00\x00\x00\x00')`

strings are allowed now
you just need to exclude not (strings or file-like)


Resolving patch:
diff --git a/pandas/io/packers.py b/pandas/io/packers.py
--- a/pandas/io/packers.py
+++ b/pandas/io/packers.py
@@ -141,24 +141,28 @@ def read(fh):
 
         try:
             exists = os.path.exists(path_or_buf)
-        except (TypeError,ValueError):
+        except (TypeError, ValueError):
             exists = False
 
         if exists:
             with open(path_or_buf, 'rb') as fh:
                 return read(fh)
 
-    # treat as a string-like
-    if not hasattr(path_or_buf, 'read'):
-
+    # treat as a binary-like
+    if isinstance(path_or_buf, compat.binary_type):
+        fh = None
         try:
             fh = compat.BytesIO(path_or_buf)
             return read(fh)
         finally:
-            fh.close()
+            if fh is not None:
+                fh.close()
 
     # a buffer like
-    return read(path_or_buf)
+    if hasattr(path_or_buf, 'read') and compat.callable(path_or_buf.read):
+        return read(path_or_buf)
+
+    raise ValueError('path_or_buf needs to be a string file path or file-like')
 
 dtype_dict = {21: np.dtype('M8[ns]'),
               u('datetime64[ns]'): np.dtype('M8[ns]'),


Test patch:
diff --git a/pandas/io/tests/test_packers.py b/pandas/io/tests/test_packers.py
--- a/pandas/io/tests/test_packers.py
+++ b/pandas/io/tests/test_packers.py
@@ -93,6 +93,17 @@ def test_iterator_with_string_io(self):
         for i, result in enumerate(read_msgpack(s,iterator=True)):
             tm.assert_frame_equal(result,dfs[i])
 
+    def test_invalid_arg(self):
+        #GH10369
+        class A(object):
+            def __init__(self):
+                self.read = 0
+
+        tm.assertRaises(ValueError, read_msgpack, path_or_buf=None)
+        tm.assertRaises(ValueError, read_msgpack, path_or_buf={})
+        tm.assertRaises(ValueError, read_msgpack, path_or_buf=A())
+
+
 class TestNumpy(TestPackers):
 
     def test_numpy_scalar_float(self):
Repository: pandas-dev/pandas

Issue:
DOC: na_fvalues argument for read_csv not documented in docstring



Discussion:
this is not a public parameter. It is internally computed and is just a passed thru variable. User cannot specify it.

Yep, I also just concluded that, but then we should just remove it from the signature of read_csv? As it is calculated inside `TextFileReader`, this passed value will always be overwritten?

not sure what you mean. its a constructed value passed the the actual reader engine (python/c) which are not public.

See the title, I am talking about `read_csv`:

> In [20]: pd.read_csv?
> Signature: pd.read_csv(filepath_or_buffer, sep=',', dialect=None, compression='infer', doublequote=True, escapechar=None, quotechar='"', quoting=0, skipinitialspace=False, lineterminator=None, header='infer', index_col=None, names=None, prefix=None, skiprows=None, skipfooter=None, skip_footer=0, na_values=None, **na_fvalues=None**, true_values=None, false_values=None, delimiter=None, converters=None, dtype=None, usecols=None, engine=None, delim_whitespace=False, as_recarray=False, na_filter=True, compact_ints=False, use_unsigned=False, low_memory=True, buffer_lines=None, warn_bad_lines=True, error_bad_lines=True, keep_default_na=True, thousands=None, comment=None, decimal='.', parse_dates=False, keep_date_col=False, dayfirst=False, date_parser=None, memory_map=False, float_precision=None, nrows=None, iterator=False, chunksize=None, verbose=False, encoding=None, squeeze=False, mangle_dupe_cols=True, tupleize_cols=False, infer_datetime_format=False, skip_blank_lines=True)

Will do a PR to remove it


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -355,7 +355,6 @@ def parser_f(filepath_or_buffer,
                  skipfooter=None,
                  skip_footer=0,
                  na_values=None,
-                 na_fvalues=None,
                  true_values=None,
                  false_values=None,
                  delimiter=None,
@@ -431,7 +430,6 @@ def parser_f(filepath_or_buffer,
                     prefix=prefix,
                     skiprows=skiprows,
                     na_values=na_values,
-                    na_fvalues=na_fvalues,
                     true_values=true_values,
                     false_values=false_values,
                     keep_default_na=keep_default_na,


Test patch:
Repository: pandas-dev/pandas

Issue:
Add note in docs for interpolate that Only `method=linear` interpolation is supported on MultiIndexes.
Trying to interpolate a DataFrame with MultiIndex:

```
lib/python2.7/site-packages/pandas/core/generic.pyc in interpolate(self, method, axis, limit, inplace, downcast, **kwargs)
   2868 
   2869         if isinstance(self.index, MultiIndex) and method != 'linear':
-> 2870             raise ValueError("Only `method=linear` interpolation is supported "
   2871                              "on MultiIndexes.")
   2872 

ValueError: Only `method=linear` interpolation is supported on MultiIndexes.
```

add a note [in the docs](http://pandas.pydata.org/pandas-docs/dev/generated/pandas.Series.interpolate.html).



Discussion:
@cancan101 thanks for the reports! (and feel free to open PRs for it :-))

@cancan101 link redirects to the Pandas docs front page. 
Here is a link to the doc paged referenced in the issue http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.interpolate.html for anyone working on a PR and wondering which doc page should be amended.


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -2876,6 +2876,9 @@ def interpolate(self, method='linear', axis=0, limit=None, inplace=False,
         """
         Interpolate values according to different methods.
 
+        Please note that only ``method='linear'`` is supported for DataFrames/Series
+        with a MultiIndex.
+
         Parameters
         ----------
         method : {'linear', 'time', 'index', 'values', 'nearest', 'zero',


Test patch:
Repository: pandas-dev/pandas

Issue:
TST: test_read_famafrench fails with HTTP 404
Not look into detail because I'm out, filenames seem to be changed.

http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html



Discussion:
pls report on `pandas-datareader` repo.

@jreback It fails on current master, and it uses `pandas.io.data` yet. Is it OK to remove the test cases from pandas, and fix in datareader?

https://travis-ci.org/pydata/pandas/jobs/71088889

I guess might as well fix in both

ok, this is annoying. Skipping the test for now: https://github.com/pydata/pandas/commit/e023f2f7d7739ab0c509d53f97066b6e60b04917


Resolving patch:
diff --git a/pandas/io/data.py b/pandas/io/data.py
--- a/pandas/io/data.py
+++ b/pandas/io/data.py
@@ -504,7 +504,7 @@ def fetch_data(url, name):
 
 def get_data_famafrench(name):
     # path of zip files
-    zip_file_path = '{0}/{1}.zip'.format(_FAMAFRENCH_URL, name)
+    zip_file_path = '{0}/{1}_TXT.zip'.format(_FAMAFRENCH_URL, name)
 
     with urlopen(zip_file_path) as url:
         raw = url.read()


Test patch:
diff --git a/pandas/io/tests/test_data.py b/pandas/io/tests/test_data.py
--- a/pandas/io/tests/test_data.py
+++ b/pandas/io/tests/test_data.py
@@ -481,8 +481,6 @@ def test_read_famafrench(self):
         for name in ("F-F_Research_Data_Factors",
                      "F-F_Research_Data_Factors_weekly", "6_Portfolios_2x3",
                      "F-F_ST_Reversal_Factor", "F-F_Momentum_Factor"):
-            raise nose.SkipTest('getting 404 errors as of 7/15/15')
-
             ff = DataReader(name, "famafrench")
             self.assertTrue(ff is not None)
             self.assertIsInstance(ff, dict)
Repository: pandas-dev/pandas

Issue:
DOC: limit error traceback to one line for expected exceptions
xref https://github.com/pydata/pandas/pull/10674#issuecomment-126837519

When using `:okexcept:`, limit the traceback to one line (or as a separate option + try to push to ipython upstream afterwards)



Discussion:
Basically, I think just need to add something like:

```
 if is_okexcept and "Traceback" in output:
    output = truncated(output)
```

here: https://github.com/pydata/pandas/blob/master/doc/sphinxext/ipython_sphinxext/ipython_directive.py#L482


Resolving patch:
diff --git a/doc/sphinxext/ipython_sphinxext/ipython_directive.py b/doc/sphinxext/ipython_sphinxext/ipython_directive.py
--- a/doc/sphinxext/ipython_sphinxext/ipython_directive.py
+++ b/doc/sphinxext/ipython_sphinxext/ipython_directive.py
@@ -461,10 +461,6 @@ def process_input(self, data, input_prompt, lineno):
 
         self.cout.seek(0)
         output = self.cout.read()
-        if not is_suppress and not is_semicolon:
-            ret.append(output)
-        elif is_semicolon: # get spacing right
-            ret.append('')
 
         # context information
         filename = self.state.document.current_source
@@ -494,6 +490,16 @@ def process_input(self, data, input_prompt, lineno):
                 sys.stdout.write(s)
                 sys.stdout.write('<<<' + ('-' * 73) + '\n')
 
+        # if :okexcept: has been specified, display shorter traceback
+        if is_okexcept and "Traceback" in output:
+            traceback = output.split('\n\n')
+            output = traceback[-1]
+
+        if not is_suppress and not is_semicolon:
+            ret.append(output)
+        elif is_semicolon: # get spacing right
+            ret.append('')
+
         self.cout.truncate(0)
         return (ret, input_lines, output, is_doctest, decorator, image_file,
                     image_directive)


Test patch:
Repository: pandas-dev/pandas

Issue:
Inconsistency with MovingOLS min_periods
I'm not sure if this is a bug or a typo in the documentation but for the `min_periods` parameter in MovingOLS the documentation claims

```
min_periods : int
       Threshold of non-null data points to require.
       If None, defaults to size of window.
```

However this does not seem to be the case? From line 640-648 we have

```
    if self._is_rolling:
        if window is None:
            raise AssertionError("Must specify window.")
        if min_periods is None:
            min_periods = window
    else:
        window = len(self._x)
        if min_periods is None:
            min_periods = 1
```

So this is not true for expanding window MovingOLS? Would it be appropriate to update the documentation to reflect this?



Discussion:
Yes, certainly welcome to update the docstring.

(and even more welcome to look into migrating this to statsmodels!)

Okay I will look at updating the docs. Probably don't have the bandwidth to take on migrating this to statsmodels, is this something that is actively being discussed though so I can follow along?

Issue that tracks this on the pandas side: https://github.com/pydata/pandas/issues/6077 
and for statsmodels: https://github.com/statsmodels/statsmodels/issues/2302

From the conversation in #6077 it seems like the plan is to move the stats part of pandas into statsmodels and deprecate the functionality in pandas? Is this still the plan? I do not see any deprecation warnings using 0.16.1 however from that thread it seems like that was going to be the idea?

That is indeed the idea. But since the functionality is not yet in statsmodels (at least eg the MovingOLS), we cannot already deprecate it in pandas saying you have to use the statsmodels one.


Resolving patch:
diff --git a/pandas/stats/ols.py b/pandas/stats/ols.py
--- a/pandas/stats/ols.py
+++ b/pandas/stats/ols.py
@@ -614,7 +614,8 @@ class MovingOLS(OLS):
         size of window (for rolling/expanding OLS)
     min_periods : int
         Threshold of non-null data points to require.
-        If None, defaults to size of window.
+        If None, defaults to size of window for window_type='rolling' and 1
+        otherwise
     intercept : bool
         True if you want an intercept.
     nw_lags : None or int


Test patch:
Repository: pandas-dev/pandas

Issue:
inplace operation for Series causes reappearance popped column
Hi, Thanks for all efforts for the great tool!

I've found a strange behavior like this...

``` python
In [1]: a = pd.DataFrame([[1,2,3],[4,5,6]], columns=['A','B','C'], index=['X','Y'])

In [2]: a
Out[2]: 
   A  B  C
X  1  2  3
Y  4  5  6

In [3]: b = a.pop('B')

In [4]: a
Out[4]: 
   A  C
X  1  3
Y  4  6

In [5]: b
Out[5]: 
X    2
Y    5
Name: B, dtype: int64

In [6]: b += 1

In [7]: a # Why does a popped column "B" appear again?
Out[7]: 
   A  C  B
X  1  3  3
Y  4  6  6

In [8]: b
Out[8]: 
X    3
Y    6
Name: B, dtype: int64
```

Is it a bug of pandas?
This behavior can be avoided by using `b = b+1` instead of implace operations. 



Discussion:
I think the cacher is somehow getting updated here. 


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -501,6 +501,11 @@ def pop(self, item):
         """
         result = self[item]
         del self[item]
+        try:
+            result._reset_cacher()
+        except AttributeError:
+            pass
+
         return result
 
     def squeeze(self):
@@ -1094,6 +1099,11 @@ def _set_as_cached(self, item, cacher):
             a weakref to cacher """
         self._cacher = (item, weakref.ref(cacher))
 
+    def _reset_cacher(self):
+        """ reset the cacher """
+        if hasattr(self,'_cacher'):
+            del self._cacher
+
     def _iget_item_cache(self, item):
         """ return the cached item, item represents a positional indexer """
         ax = self._info_axis
@@ -1330,6 +1340,7 @@ def __delitem__(self, key):
             # exception:
             self._data.delete(key)
 
+        # delete from the caches
         try:
             del self._item_cache[key]
         except KeyError:


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -5177,6 +5177,20 @@ def test_pop(self):
         self.assertNotIn('foo', self.frame)
         # TODO self.assertEqual(self.frame.columns.name, 'baz')
 
+        # 10912
+        # inplace ops cause caching issue
+        a = DataFrame([[1,2,3],[4,5,6]], columns=['A','B','C'], index=['X','Y'])
+        b = a.pop('B')
+        b += 1
+
+        # original frame
+        expected = DataFrame([[1,3],[4,6]], columns=['A','C'], index=['X','Y'])
+        assert_frame_equal(a, expected)
+
+        # result
+        expected = Series([2,5],index=['X','Y'],name='B')+1
+        assert_series_equal(b, expected)
+
     def test_pop_non_unique_cols(self):
         df = DataFrame({0: [0, 1], 1: [0, 1], 2: [4, 5]})
         df.columns = ["a", "b", "a"]
Repository: pandas-dev/pandas

Issue:
DOC/API: clarify groupby sorting behavior
clarify / doc / test (explicity) on passing `sort=True|False` to a `groupby/resample`
from [SO](http://stackoverflow.com/questions/26456125/python-pandas-is-order-preserved-when-using-groupby-and-agg/26465555#26465555resample forces a sort)

groupby only sorts if sort is True and the index is NOT monotonic.

just like to test / validate and write a doc not about this



Discussion:

Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -3247,11 +3247,13 @@ def groupby(self, by=None, axis=0, level=None, as_index=True, sort=True,
             index. Only relevant for DataFrame input. as_index=False is
             effectively "SQL-style" grouped output
         sort : boolean, default True
-            Sort group keys. Get better performance by turning this off
+            Sort group keys. Get better performance by turning this off.
+            Note this does not influence the order of observations within each group. 
+            groupby preserves the order of rows within each group. 
         group_keys : boolean, default True
             When calling apply, add group keys to index to identify pieces
         squeeze : boolean, default False
-            reduce the dimensionaility of the return type if possible,
+            reduce the dimensionality of the return type if possible,
             otherwise return a consistent type
 
         Examples


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -5436,6 +5436,32 @@ def test_first_last_max_min_on_time_data(self):
         assert_frame_equal(grouped_ref.first(),grouped_test.first())
         assert_frame_equal(grouped_ref.last(),grouped_test.last())
 
+    def test_groupby_preserves_sort(self):
+        # Test to ensure that groupby always preserves sort order of original
+        # object. Issue #8588 and #9651
+        
+        df = DataFrame({'int_groups':[3,1,0,1,0,3,3,3], 
+                        'string_groups':['z','a','z','a','a','g','g','g'], 
+                        'ints':[8,7,4,5,2,9,1,1],
+                        'floats':[2.3,5.3,6.2,-2.4,2.2,1.1,1.1,5],
+                        'strings':['z','d','a','e','word','word2','42','47']})
+
+        # Try sorting on different types and with different group types
+        for sort_column in ['ints', 'floats', 'strings', ['ints','floats'], 
+                  ['ints','strings']]:
+            for group_column in ['int_groups', 'string_groups', 
+                                 ['int_groups','string_groups']]:
+
+                df = df.sort_values(by=sort_column)
+
+                g = df.groupby(group_column)
+                
+                def test_sort(x):
+                    assert_frame_equal(x, x.sort_values(by=sort_column))
+    
+                g.apply(test_sort)
+
+
 def assert_fp_equal(a, b):
     assert (np.abs(a - b) < 1e-12).all()
Repository: pandas-dev/pandas

Issue:
Broken nunique on Series group by
The following code works in 0.16.2 and not in latest master:

``` python
data = pd.DataFrame(
    [[100, 1, 'Alice'],
     [200, 2, 'Bob'],
     [300, 3, 'Charlie'],
     [-400, 4, 'Dan'],
     [500, 5, 'Edith']],
    columns=['amount', 'id', 'name']
)

expected = data.groupby(['id', 'amount'])['name'].nunique()
```

Going to bisect this today unless someone beats me to it.



Discussion:
https://github.com/pydata/pandas/pull/10894

Can we just revert that change and keep the test?

The only other solution I see is to coerce `val` to `str` if it's an object dtype

if its not `int64` factorize it

```
ipdb> p np.lexsort((pd.factorize(val)[0],ids))
array([0, 1, 2, 3, 4])
```

ok

or rather it has a `TypeError`, though this shouldn't have gotten this far if that was the case....


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2565,7 +2565,17 @@ def nunique(self, dropna=True):
         ids, _, _ = self.grouper.group_info
         val = self.obj.get_values()
 
-        sorter = np.lexsort((val, ids))
+        try:
+            sorter = np.lexsort((val, ids))
+        except TypeError:  # catches object dtypes
+            assert val.dtype == object, \
+                'val.dtype must be object, got %s' % val.dtype
+            val, _ = algos.factorize(val, sort=False)
+            sorter = np.lexsort((val, ids))
+            isnull = lambda a: a == -1
+        else:
+            isnull = com.isnull
+
         ids, val = ids[sorter], val[sorter]
 
         # group boundries are where group ids change


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -5511,6 +5511,22 @@ def test_sort(x):
 
                 g.apply(test_sort)
 
+    def test_nunique_with_object(self):
+        # GH 11077
+        data = pd.DataFrame(
+            [[100, 1, 'Alice'],
+             [200, 2, 'Bob'],
+             [300, 3, 'Charlie'],
+             [-400, 4, 'Dan'],
+             [500, 5, 'Edith']],
+            columns=['amount', 'id', 'name']
+        )
+
+        result = data.groupby(['id', 'amount'])['name'].nunique()
+        index = MultiIndex.from_arrays([data.id, data.amount])
+        expected = pd.Series([1] * 5, name='name', index=index)
+        tm.assert_series_equal(result, expected)
+
 
 def assert_fp_equal(a, b):
     assert (np.abs(a - b) < 1e-12).all()
Repository: pandas-dev/pandas

Issue:
Union of index with itself not invariant, sets frequency attribute
When creating the union of an index with itself, the frequency may end up being reset. I would have expected the call to union in the following example to be invariant and return the original index:

```
In [56]: index = pd.bdate_range('20150101', periods=10)
In [57]: index.freq = None
In [58]: index
Out[58]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2015-01-01, ..., 2015-01-14]
Length: 10, Freq: None, Timezone: None

In [59]: index.union(index)
Out[59]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2015-01-01, ..., 2015-01-14]
Length: 10, Freq: B, Timezone: None
```

This then propagates through to e.g. `combine_first`, which is also not invariant. This might be desirable behavior in some cases but definitely surprising imo.



Discussion:
so the case is an index that has `idx.freq is None` with another will NOT take the fast path, but then both compare equal. ok so far.

Then an infererence happens [here](https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L899)

I suppose if both have incoming `freq is None` then don't need to infer.

Not sure what this would break though.

want to do a PR and see?


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -51,7 +51,7 @@
 _default_encoding = 'UTF-8'
 
 def _ensure_decoded(s):
-    """ if we have bytes, decode them to unicde """
+    """ if we have bytes, decode them to unicode """
     if isinstance(s, np.bytes_):
         s = s.decode('UTF-8')
     return s
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -895,7 +895,8 @@ def union(self, other):
             result = Index.union(this, other)
             if isinstance(result, DatetimeIndex):
                 result.tz = this.tz
-                if result.freq is None:
+                if (result.freq is None and
+                        (this.freq is not None or other.freq is not None)):
                     result.offset = to_offset(result.inferred_freq)
             return result
 


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -2551,6 +2551,15 @@ def test_intersection_bug_1708(self):
         result = index_1 & index_2
         self.assertEqual(len(result), 0)
 
+    def test_union_freq_both_none(self):
+        #GH11086
+        expected = bdate_range('20150101', periods=10)
+        expected.freq = None
+
+        result = expected.union(expected)
+        tm.assert_index_equal(result, expected)
+        self.assertIsNone(result.freq)
+
     # GH 10699
     def test_datetime64_with_DateOffset(self):
         for klass, assert_func in zip([Series, DatetimeIndex],