internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pandas-dev/pandas

Issue:
COMPAT: broken pickle compat from 0.19.2 -> 0.20.2 for UTC tz's
```
In [1]: pd.__version__
Out[1]: '0.19.2'

In [3]: df = pd.DataFrame({'A': [1, 2, 3]}, index=pd.date_range('20130101', periods=3, tz='UTC'))

In [4]: df.to_pickle('v0.19.2.pkl')
```

broken in 0.20.1/2
```
In [2]: pd.read_pickle('../v0.19.2.pkl')
IndexError: tuple index out of range
```

works fine for tz-aware but non-UTC (e.g. US/Eastern is fine).



Discussion:
@jorisvandenbossche @TomAugspurger any objection to doing a 0.20.3 in a few weeks?

Resolving patch:
diff --git a/pandas/compat/pickle_compat.py b/pandas/compat/pickle_compat.py
--- a/pandas/compat/pickle_compat.py
+++ b/pandas/compat/pickle_compat.py
@@ -15,7 +15,7 @@ def load_reduce(self):
     args = stack.pop()
     func = stack[-1]
 
-    if type(args[0]) is type:
+    if len(args) and type(args[0]) is type:
         n = args[0].__name__  # noqa
 
     try:


Test patch:
diff --git a/pandas/tests/io/data/legacy_pickle/0.19.2/0.19.2_x86_64_darwin_3.6.1.pickle b/pandas/tests/io/data/legacy_pickle/0.19.2/0.19.2_x86_64_darwin_3.6.1.pickle
Binary files a/pandas/tests/io/data/legacy_pickle/0.19.2/0.19.2_x86_64_darwin_3.6.1.pickle and b/pandas/tests/io/data/legacy_pickle/0.19.2/0.19.2_x86_64_darwin_3.6.1.pickle differ
diff --git a/pandas/tests/io/generate_legacy_storage_files.py b/pandas/tests/io/generate_legacy_storage_files.py
old mode 100644
new mode 100755
--- a/pandas/tests/io/generate_legacy_storage_files.py
+++ b/pandas/tests/io/generate_legacy_storage_files.py
@@ -1,3 +1,5 @@
+#!/usr/env/bin python
+
 """ self-contained to write legacy storage (pickle/msgpack) files """
 from __future__ import print_function
 from warnings import catch_warnings
@@ -125,7 +127,11 @@ def create_data():
                  mixed_dup=mixed_dup_df,
                  dt_mixed_tzs=DataFrame({
                      u'A': Timestamp('20130102', tz='US/Eastern'),
-                     u'B': Timestamp('20130603', tz='CET')}, index=range(5))
+                     u'B': Timestamp('20130603', tz='CET')}, index=range(5)),
+                 dt_mixed2_tzs=DataFrame({
+                     u'A': Timestamp('20130102', tz='US/Eastern'),
+                     u'B': Timestamp('20130603', tz='CET'),
+                     u'C': Timestamp('20130603', tz='UTC')}, index=range(5))
                  )
 
     with catch_warnings(record=True):
Repository: pandas-dev/pandas

Issue:
DOC: read_excel dtype parameter - str vs. object 
#### Code Sample, a copy-pastable example if possible

```ipython
In [16]: import pandas as pd
    ...: import numpy as np
    ...: pd.DataFrame(['a', 1, np.nan]).to_excel('test.xlsx')
    ...: df_str = pd.read_excel('test.xlsx', dtype=str, names=['col_str'])
    ...: df_str['type_str'] = df_str.col_str.map(type)
    ...: df_obj = pd.read_excel('test.xlsx', dtype=object, names=['col_obj'])
    ...: df_obj['type_obj'] = df_obj.col_obj.map(type)
    ...: pd.concat([df_str, df_obj], axis=1)
    ...: 
Out[16]: 
  col_str       type_str col_obj         type_obj
0       a  <class 'str'>       a    <class 'str'>
1       1  <class 'str'>       1    <class 'int'>
2     nan  <class 'str'>     NaN  <class 'float'>
```
#### Problem description

I imagine <a href="http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.read_excel.html">read_excel</a>'s `dtype` parameter description should just read "Use *object* to preserve and not interpret dtype" and not "Use *str* or *object*".


Discussion:
Yes - looks like I copied the `read_csv` docstring - PR welcome!

Resolving patch:
diff --git a/pandas/io/excel.py b/pandas/io/excel.py
--- a/pandas/io/excel.py
+++ b/pandas/io/excel.py
@@ -85,7 +85,9 @@
 index_col : int, list of ints, default None
     Column (0-indexed) to use as the row labels of the DataFrame.
     Pass None if there is no such column.  If a list is passed,
-    those columns will be combined into a ``MultiIndex``
+    those columns will be combined into a ``MultiIndex``.  If a
+    subset of data is selected with ``parse_cols``, index_col
+    is based on the subset.
 names : array-like, default None
     List of column names to use. If file contains no header row,
     then you should explicitly pass header=None
@@ -96,7 +98,7 @@
     content.
 dtype : Type name or dict of column -> type, default None
     Data type for data or columns. E.g. {'a': np.float64, 'b': np.int32}
-    Use `str` or `object` to preserve and not interpret dtype.
+    Use `object` to preserve data as stored in Excel and not interpret dtype.
     If converters are specified, they will be applied INSTEAD
     of dtype conversion.
 
@@ -116,8 +118,9 @@
     * If None then parse all columns,
     * If int then indicates last column to be parsed
     * If list of ints then indicates list of column numbers to be parsed
-    * If string then indicates comma separated list of column names and
-      column ranges (e.g. "A:E" or "A,C,E:F")
+    * If string then indicates comma separated list of Excel column letters and
+      column ranges (e.g. "A:E" or "A,C,E:F").  Ranges are inclusive of
+      both sides.
 squeeze : boolean, default False
     If the parsed data only contains one column then return a Series
 na_values : scalar, str, list-like, or dict, default None


Test patch:
Repository: pandas-dev/pandas

Issue:
Old read_csv() & EOF character issue back
I recently updated from pandas 0.19.2 to 0.2
I am experiencing the exact same issue as this person did back in 2013
https://github.com/pandas-dev/pandas/issues/5500
"Error tokenizing data. C error: EOF inside string starting at line. 140"

Reverting back to 0.19.2 has fixed the issue. Can run the unit test that was created for this problem?
 https://github.com/gfyoung/pandas/commit/8c4cf85943e647cbb8a0e081ad1e6277dbd309b9


Discussion:
can you show a reproducible example.
I'm having the same problem. Python 3, pandas 0.20.2

reproducible example:


```
import pandas as pd

with open('test.csv', 'wb') as fout:
    fout.write(b'c1,c2\r\n"test \x1a    test", test\r\n')

pd.read_csv('test.csv')

#ParserError: Error tokenizing data. C error: EOF inside string starting at line 1
```

Is this a windows issue? (cannot reproduce your example above on linux)
Yes, it happened on a 64 bit Windows. 
@gfyoung can you verfiy / see if you can fix?

Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1985,7 +1985,8 @@ def __init__(self, f, **kwds):
         self.comment = kwds['comment']
         self._comment_lines = []
 
-        f, handles = _get_handle(f, 'r', encoding=self.encoding,
+        mode = 'r' if PY3 else 'rb'
+        f, handles = _get_handle(f, mode, encoding=self.encoding,
                                  compression=self.compression,
                                  memory_map=self.memory_map)
         self.handles.extend(handles)


Test patch:
diff --git a/pandas/tests/io/parser/common.py b/pandas/tests/io/parser/common.py
--- a/pandas/tests/io/parser/common.py
+++ b/pandas/tests/io/parser/common.py
@@ -1662,6 +1662,21 @@ def test_internal_eof_byte(self):
         result = self.read_csv(StringIO(data))
         tm.assert_frame_equal(result, expected)
 
+    def test_internal_eof_byte_to_file(self):
+        # see gh-16559
+        data = b'c1,c2\r\n"test \x1a    test", test\r\n'
+        expected = pd.DataFrame([["test \x1a    test", " test"]],
+                                columns=["c1", "c2"])
+
+        path = '__%s__.csv' % tm.rands(10)
+
+        with tm.ensure_clean(path) as path:
+            with open(path, "wb") as f:
+                f.write(data)
+
+            result = self.read_csv(path)
+            tm.assert_frame_equal(result, expected)
+
     def test_file_handles(self):
         # GH 14418 - don't close user provided file handles
Repository: pandas-dev/pandas

Issue:
Indexing TimedeltaIndex with np.timedelta64
On pandas 0.20.2, `TimedeltaIndex.get_loc()` worked on np.timedelta64, even though DataFrame.loc fails:
```
In [2]: df = pd.DataFrame({'x': range(3)}, pd.to_timedelta(range(3), unit='days'))

In [3]: df.loc[df.index[0].to_timedelta64()]
TypeError: cannot do index indexing on <class 'pandas.core.indexes.timedeltas.TimedeltaIndex'> with these indexers [0] of <class 'numpy.timedelta64'>

In [5]: df.index.get_loc(df.index[0].to_timedelta64())
Out[5]: 0
```
Now, both raise `TypeError`. (This is similar to the issue in #16896)

I think some xarray users are were likely relying on this (we use `get_loc()` directly), but I think supporting this could also be useful for pandas users.


Discussion:
Question: Should `get_loc`  only work on 'scalar' values?

if that's the case, then it's pretty easy to fix:

replace is_timedelta64_dtype with is_list_like and it should work.
``.get_loc`` by-definition is *only* on scalar values. a string and/or a np.timedelta64 are scalars (e.g. not lists/arrays)

``.get_indexer`` works on list-likes
here you can simply pass scalars to ``Timedelta()`` and then index

Resolving patch:
diff --git a/pandas/core/indexes/timedeltas.py b/pandas/core/indexes/timedeltas.py
--- a/pandas/core/indexes/timedeltas.py
+++ b/pandas/core/indexes/timedeltas.py
@@ -15,7 +15,7 @@
     _ensure_int64)
 from pandas.core.dtypes.missing import isnull
 from pandas.core.dtypes.generic import ABCSeries
-from pandas.core.common import _maybe_box, _values_from_object, is_bool_indexer
+from pandas.core.common import _maybe_box, _values_from_object
 
 from pandas.core.indexes.base import Index
 from pandas.core.indexes.numeric import Int64Index
@@ -682,7 +682,7 @@ def get_loc(self, key, method=None, tolerance=None):
         -------
         loc : int
         """
-        if is_bool_indexer(key) or is_timedelta64_dtype(key):
+        if is_list_like(key):
             raise TypeError
 
         if isnull(key):


Test patch:
diff --git a/pandas/tests/indexes/timedeltas/test_timedelta.py b/pandas/tests/indexes/timedeltas/test_timedelta.py
--- a/pandas/tests/indexes/timedeltas/test_timedelta.py
+++ b/pandas/tests/indexes/timedeltas/test_timedelta.py
@@ -66,6 +66,9 @@ def test_get_loc(self):
         for method, loc in [('pad', 1), ('backfill', 2), ('nearest', 1)]:
             assert idx.get_loc('1 day 1 hour', method) == loc
 
+        # GH 16909
+        assert idx.get_loc(idx[1].to_timedelta64()) == 1
+
         # GH 16896
         assert idx.get_loc('0 days') == 0
Repository: pandas-dev/pandas

Issue:
ERR: missing stacklevel in to_dict UserWarning
#### Code Sample, a copy-pastable example if possible

```python
In [4]: df = pd.DataFrame([[1,2,3]], columns=['a', 'a', 'b'])

In [5]: df
Out[5]:
   a  a  b
0  1  2  3

In [6]: df.to_dict()
C:\<snip>\pandas\core\frame.py:891:
UserWarning: DataFrame columns are not unique, some columns will be omitted.
  "columns will be omitted.", UserWarning)
Out[6]: {'a': {0: 2}, 'b': {0: 3}}
```

#### Problem description

warning should specific stacklevel to warn at calling site, xref #9584



Discussion:
In the future, we'll probably have to go back and find any other hanging warnings like these.
Yep, probably some others, a sufficiently clever grep might be able to turn them up. (basically whenever we call `warn(...)` without a stacklevel passed.
a lot of these prob don't have the correct stack_level; note that on most ``assert_raises_warnings`` we are passing ``check_stacklevel=False`` (so its not checking). certainly these could/should be changed.
That's true, we probably would need to go back and comb through those a bit.

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -972,7 +972,8 @@ def to_dict(self, orient='dict', into=dict):
         """
         if not self.columns.is_unique:
             warnings.warn("DataFrame columns are not unique, some "
-                          "columns will be omitted.", UserWarning)
+                          "columns will be omitted.", UserWarning,
+                          stacklevel=2)
         # GH16122
         into_c = standardize_mapping(into)
         if orient.lower().startswith('d'):


Test patch:
diff --git a/pandas/tests/frame/test_convert_to.py b/pandas/tests/frame/test_convert_to.py
--- a/pandas/tests/frame/test_convert_to.py
+++ b/pandas/tests/frame/test_convert_to.py
@@ -216,6 +216,13 @@ def test_to_dict_errors(self, mapping):
         with pytest.raises(TypeError):
             df.to_dict(into=mapping)
 
+    def test_to_dict_not_unique_warning(self):
+        # GH16927: When converting to a dict, if a column has a non-unique name
+        # it will be dropped, throwing a warning.
+        df = DataFrame([[1, 2, 3]], columns=['a', 'a', 'b'])
+        with tm.assert_produces_warning(UserWarning):
+            df.to_dict()
+
     @pytest.mark.parametrize('tz', ['UTC', 'GMT', 'US/Eastern'])
     def test_to_records_datetimeindex_with_tz(self, tz):
         # GH13937
Repository: pandas-dev/pandas

Issue:
CLN: private impl of OrderedDefaultDict can be removed
https://github.com/pandas-dev/pandas/blob/master/pandas/compat/__init__.py#L376

I think this was leftover from 2.6 compat.


Discussion:
@jreback So the private methods need to be made public? Is that all is required?
no i think it can just be removed
And replaced with an import `from collections import OrderedDict`.

Resolving patch:
diff --git a/pandas/compat/__init__.py b/pandas/compat/__init__.py
--- a/pandas/compat/__init__.py
+++ b/pandas/compat/__init__.py
@@ -21,7 +21,6 @@
   given metaclass instead (and avoids intermediary class creation)
 
 Other items:
-* OrderedDefaultDict
 * platform checker
 """
 # pylint disable=W0611
@@ -373,30 +372,6 @@ def parse_date(timestr, *args, **kwargs):
     parse_date = _date_parser.parse
 
 
-class OrderedDefaultdict(OrderedDict):
-
-    def __init__(self, *args, **kwargs):
-        newdefault = None
-        newargs = ()
-        if args:
-            newdefault = args[0]
-            if not (newdefault is None or callable(newdefault)):
-                raise TypeError('first argument must be callable or None')
-            newargs = args[1:]
-        self.default_factory = newdefault
-        super(self.__class__, self).__init__(*newargs, **kwargs)
-
-    def __missing__(self, key):
-        if self.default_factory is None:
-            raise KeyError(key)
-        self[key] = value = self.default_factory()
-        return value
-
-    def __reduce__(self):  # optional, for pickle support
-        args = self.default_factory if self.default_factory else tuple()
-        return type(self), args, None, None, list(self.items())
-
-
 # https://github.com/pandas-dev/pandas/pull/9123
 def is_platform_little_endian():
     """ am I little endian """
diff --git a/pandas/core/panel.py b/pandas/core/panel.py
--- a/pandas/core/panel.py
+++ b/pandas/core/panel.py
@@ -19,7 +19,7 @@
 import pandas.core.ops as ops
 import pandas.core.missing as missing
 from pandas import compat
-from pandas.compat import (map, zip, range, u, OrderedDict, OrderedDefaultdict)
+from pandas.compat import (map, zip, range, u, OrderedDict)
 from pandas.compat.numpy import function as nv
 from pandas.core.common import _try_sort, _default_index
 from pandas.core.frame import DataFrame
@@ -260,9 +260,11 @@ def from_dict(cls, data, intersect=False, orient='items', dtype=None):
         -------
         Panel
         """
+        from collections import defaultdict
+
         orient = orient.lower()
         if orient == 'minor':
-            new_data = OrderedDefaultdict(dict)
+            new_data = defaultdict(OrderedDict)
             for col, df in compat.iteritems(data):
                 for item, s in compat.iteritems(df):
                     new_data[item][col] = s


Test patch:
Repository: pandas-dev/pandas

Issue:
BUG: SparseSeries from dict inconsistency
#### Code Sample, a copy-pastable example if possible
Two (somewhat related) issues:
```python
>>> pd.Series({1: 1})
1    1
dtype: int64

>>> pd.SparseSeries({1: 1})
---------------------------------------------------------------------------
AttributeError: 'dict' object has no attribute 'index'
```

```py
>>> pd.Series({1: 1}, index=[0, 1, 2])
0    NaN
1    1.0
2    NaN
dtype: float64

>>> pd.SparseSeries({1: 1.}, index=[0, 1, 2])
0    1.0
1     
2     
dtype: float64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([1], dtype=int32)
```
#### Problem description
SparseSeries initialization from dict acts too differently from Series'.

#### Expected Output
```py
>>> pd.SparseSeries({1: 1})
1    1.0
dtype: float64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([1], dtype=int32)

>>> pd.SparseSeries({1: 1.}, index=[0, 1, 2])
0    NaN 
1    1.0 
2    NaN
dtype: float64
BlockIndex
Block locations: array([1], dtype=int32)
Block lengths: array([1], dtype=int32)
```

#### Output of ``pd.show_versions()``

<details>
pandas 0.21.0.dev+223.ga9421af1a
</details>


Discussion:
yep looks like a bug
I think I have a fix for this. I'm testing it now.

Resolving patch:
diff --git a/pandas/core/sparse/series.py b/pandas/core/sparse/series.py
--- a/pandas/core/sparse/series.py
+++ b/pandas/core/sparse/series.py
@@ -146,10 +146,9 @@ def __init__(self, data=None, index=None, sparse_index=None, kind='block',
                 data = data._data
 
             elif isinstance(data, (Series, dict)):
-                if index is None:
-                    index = data.index.view()
+                data = Series(data, index=index)
+                index = data.index.view()
 
-                data = Series(data)
                 res = make_sparse(data, kind=kind, fill_value=fill_value)
                 data, sparse_index, fill_value = res
 


Test patch:
diff --git a/pandas/tests/sparse/test_series.py b/pandas/tests/sparse/test_series.py
--- a/pandas/tests/sparse/test_series.py
+++ b/pandas/tests/sparse/test_series.py
@@ -88,6 +88,24 @@ def setup_method(self, method):
         self.ziseries2 = SparseSeries(arr, index=index, kind='integer',
                                       fill_value=0)
 
+    def test_constructor_dict_input(self):
+        # gh-16905
+        constructor_dict = {1: 1.}
+        index = [0, 1, 2]
+
+        # Series with index passed in
+        series = pd.Series(constructor_dict)
+        expected = SparseSeries(series, index=index)
+
+        result = SparseSeries(constructor_dict, index=index)
+        tm.assert_sp_series_equal(result, expected)
+
+        # Series with index and dictionary with no index
+        expected = SparseSeries(series)
+
+        result = SparseSeries(constructor_dict)
+        tm.assert_sp_series_equal(result, expected)
+
     def test_constructor_dtype(self):
         arr = SparseSeries([np.nan, 1, 2, np.nan])
         assert arr.dtype == np.float64
Repository: pandas-dev/pandas

Issue:
DOC: Clarify .agg status
The documentation for `.agg` is almost hilariously ambiguous:

> agg is an alias for aggregate. Use it.

What is "it" referring to?  Are we being told to use `aggregate` instead of `agg` because it's the fundamental name and `agg` is a mere alias?  Or are we being told that we can and should use `agg` because it's the canonical alias (in the same way that pd is the canonical alias for pandas)?

A quick scan of the codebase suggests that `.agg` is considerably more popular than `.aggregate` among pandas devs, FWIW..


Discussion:
The grammarian says "aggregate" because the last noun before the pronoun is "aggregate"

However, the correct reference is `agg`, which is why you see `.agg` being used more often.

Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -3152,7 +3152,7 @@ def pipe(self, func, *args, **kwargs):
     (e.g., np.mean(arr_2d, axis=0)) as opposed to
     mimicking the default Numpy behavior (e.g., np.mean(arr_2d)).
 
-    agg is an alias for aggregate. Use it.
+    `agg` is an alias for `aggregate`. Use the alias.
 
     Returns
     -------


Test patch:
Repository: pandas-dev/pandas

Issue:
COMPAT: replace import of remove_na
https://travis-ci.org/pandas-dev/pandas/jobs/253985437

xref #16935

this breaks ``seaborn`` which is apparently using this private method.

So let's

- open an issue on seaborn about this (optionally they can simply use ``s[s.notnull()]`` instead of the import), prob would appreciate a PR
- add back the import to ``pandas.core.series`` and deprecate ``remove_na`` (which just can call ``remove_na_arraylike``



Discussion:
cc @gfyoung 
cc @jbrockmendel 
Should I do this in the old PR?
no, new PR for this.

Resolving patch:
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -37,7 +37,6 @@
     maybe_convert_platform,
     maybe_cast_to_datetime, maybe_castable)
 from pandas.core.dtypes.missing import isnull, notnull, remove_na_arraylike
-
 from pandas.core.common import (is_bool_indexer,
                                 _default_index,
                                 _asarray_tuplesafe,
@@ -88,6 +87,17 @@
     versionadded_to_excel='\n    .. versionadded:: 0.20.0\n')
 
 
+# see gh-16971
+def remove_na(arr):
+    """
+    DEPRECATED : this function will be removed in a future version.
+    """
+
+    warnings.warn("remove_na is deprecated and is a private "
+                  "function. Do not use.", FutureWarning, stacklevel=2)
+    return remove_na_arraylike(arr)
+
+
 def _coerce_method(converter):
     """ install the scalar coercion methods """
 


Test patch:
diff --git a/pandas/tests/series/test_missing.py b/pandas/tests/series/test_missing.py
--- a/pandas/tests/series/test_missing.py
+++ b/pandas/tests/series/test_missing.py
@@ -15,6 +15,7 @@
                     MultiIndex, Index, Timestamp, NaT, IntervalIndex)
 from pandas.compat import range
 from pandas._libs.tslib import iNaT
+from pandas.core.series import remove_na
 from pandas.util.testing import assert_series_equal, assert_frame_equal
 import pandas.util.testing as tm
 
@@ -50,6 +51,11 @@ def _simple_ts(start, end, freq='D'):
 
 class TestSeriesMissingData(TestData):
 
+    def test_remove_na_deprecation(self):
+        # see gh-16971
+        with tm.assert_produces_warning(FutureWarning):
+            remove_na(Series([]))
+
     def test_timedelta_fillna(self):
         # GH 3371
         s = Series([Timestamp('20130101'), Timestamp('20130101'), Timestamp(