internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pandas-dev/pandas

Issue:
ASV: occasional asv failures on xlwt
https://travis-ci.org/pandas-dev/pandas-ci/jobs/343616367
this is on current pandas master (this is just the CI job running it).

excel asv's

```
The following asvs benchmarks (if any) failed.
[ 43.02%] ··· Running io.excel.Excel.time_read_excel                  1/3 failed
                   xlwt      failed 
DONE displaying failed asvs benchmarks.
```

I have seen this work as well. maybe a race-condition?


Discussion:
from ``(pandas) bash-3.2$ more asv_bench/benchmarks/io/excel.py ``
```
def setup(...):
      ....
        self.bio_write = BytesIO()
        self.bio_write.seek(0)
        self.writer_write = ExcelWriter(self.bio_write, engine=engine)

    def time_read_excel(self, engine):
        read_excel(self.bio_read)

    def time_write_excel(self, engine):
        self.df.to_excel(self.writer_write, sheet_name='Sheet1')
        self.writer_write.save()
```

I think the ``time_write_excel`` should have the ``.writer_write`` setup I think.

Resolving patch:
diff --git a/asv_bench/benchmarks/io/excel.py b/asv_bench/benchmarks/io/excel.py
--- a/asv_bench/benchmarks/io/excel.py
+++ b/asv_bench/benchmarks/io/excel.py
@@ -25,13 +25,12 @@ def setup(self, engine):
         self.writer_read.save()
         self.bio_read.seek(0)
 
-        self.bio_write = BytesIO()
-        self.bio_write.seek(0)
-        self.writer_write = ExcelWriter(self.bio_write, engine=engine)
-
     def time_read_excel(self, engine):
         read_excel(self.bio_read)
 
     def time_write_excel(self, engine):
-        self.df.to_excel(self.writer_write, sheet_name='Sheet1')
-        self.writer_write.save()
+        bio_write = BytesIO()
+        bio_write.seek(0)
+        writer_write = ExcelWriter(bio_write, engine=engine)
+        self.df.to_excel(writer_write, sheet_name='Sheet1')
+        writer_write.save()


Test patch:
Repository: pandas-dev/pandas

Issue:
Clean: Categorical.fillna NaN in categories checking
We don't allow NaN in the categories anymore, so this block should be unreachable.

https://github.com/pandas-dev/pandas/blob/8bfcddc7728deaf8e840416d83c8feda86630d27/pandas/core/arrays/categorical.py#L1622-L1628

If anyone wants to remove it and test things out.


Discussion:

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -1627,14 +1627,6 @@ def fillna(self, value=None, method=None, limit=None):
 
         values = self._codes
 
-        # Make sure that we also get NA in categories
-        if self.categories.dtype.kind in ['S', 'O', 'f']:
-            if np.nan in self.categories:
-                values = values.copy()
-                nan_pos = np.where(isna(self.categories))[0]
-                # we only have one NA in categories
-                values[values == nan_pos] = -1
-
         # pad / bfill
         if method is not None:
 


Test patch:
Repository: pandas-dev/pandas

Issue:
DOC: is confusing for ddof parameter of sem, var and std functions
https://github.com/pandas-dev/pandas/blob/7bb204a05fee20c3c825e7da39ccaf39fbeb8ca5/pandas/core/generic.py#L7160-L7161

It states that _ddof_ is for degrees of freedom with default value of 1. But actually it is for "Delta degrees-of-freedom" (even the name of the parameter is abbreviation of that definition). The meaning is:  how many degrees of freedom **to adjust** relative to the population size. Default of 1 - means we need to **adjust** by one, not use one as a value of degrees of freedom (thats makes no sense). All functions are worked as expected, but documentation on _ddof_ parameter is confusing.


Discussion:
Yeah it should be fixed according to the latest numpy doc. PR is welcome!

https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.std.html
sure @recontemplator would you like to do a PR
Hi,
Newbie here, would like to take a stab at this.

txs


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -7767,7 +7767,8 @@ def _doc_parms(cls):
     If the axis is a MultiIndex (hierarchical), count along a
     particular level, collapsing into a %(name1)s
 ddof : int, default 1
-    degrees of freedom
+    Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
+    where N represents the number of elements.
 numeric_only : boolean, default None
     Include only float, int, boolean columns. If None, will attempt to use
     everything, then use only numeric data. Not implemented for Series.


Test patch:
Repository: pandas-dev/pandas

Issue:
DEPR: deprecate get_ftype_counts
These are only to provide some inside into a sparse dtype. These should in theory be encoded in the dtypes (in pandas2), but in reality these are just cluttering the API and actually an implementation detail.


Discussion:
As context, you can put a sparse data in a normal dataframe:

```
In [10]: df = pd.DataFrame({'a': [1, 2, 3]})

In [11]: df['b'] = pd.SparseSeries([3, 4, 5])

In [12]: df
Out[12]: 
   a  b
0  1  3
1  2  4
2  3  5

In [13]: df.dtypes
Out[13]: 
a    int64
b    int64
dtype: object

In [14]: df.ftypes
Out[14]: 
a     int64:dense
b    int64:sparse
dtype: object

In [16]: df['b']
Out[16]: 
0    3
1    4
2    5
Name: b, dtype: int64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([3], dtype=int32)
```

The question is: what is the alternative to easily check if a certain column is sparse ?
Happy to add the deprecation warnings for the usage of these. Is there something in particular that they should be replaced with, or just removed in a future version? Also, presuming .ftype is to be left as is?

Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -4677,6 +4677,8 @@ def get_ftype_counts(self):
         """
         Return counts of unique ftypes in this object.
 
+        .. deprecated:: 0.23.0
+
         This is useful for SparseDataFrame or for DataFrames containing
         sparse arrays.
 
@@ -4707,6 +4709,10 @@ def get_ftype_counts(self):
         object:dense     1
         dtype: int64
         """
+        warnings.warn("get_ftype_counts is deprecated and will "
+                      "be removed in a future version",
+                      FutureWarning, stacklevel=2)
+
         from pandas import Series
         return Series(self._data.get_ftype_counts())
 


Test patch:
diff --git a/pandas/tests/series/test_dtypes.py b/pandas/tests/series/test_dtypes.py
--- a/pandas/tests/series/test_dtypes.py
+++ b/pandas/tests/series/test_dtypes.py
@@ -64,8 +64,10 @@ def test_dtype(self):
         assert self.ts.ftypes == 'float64:dense'
         tm.assert_series_equal(self.ts.get_dtype_counts(),
                                Series(1, ['float64']))
-        tm.assert_series_equal(self.ts.get_ftype_counts(),
-                               Series(1, ['float64:dense']))
+        # GH18243 - Assert .get_ftype_counts is deprecated
+        with tm.assert_produces_warning(FutureWarning):
+            tm.assert_series_equal(self.ts.get_ftype_counts(),
+                                   Series(1, ['float64:dense']))
 
     @pytest.mark.parametrize("value", [np.nan, np.inf])
     @pytest.mark.parametrize("dtype", [np.int32, np.int64])
Repository: pandas-dev/pandas

Issue:
`console_encode` doesn't exist 
https://github.com/pandas-dev/pandas/blob/6d610a4d9393c0d0335267dc3252ccabb9e51e43/pandas/io/formats/printing.py#L86

refers to a `console_encoding` util that encodes the current string into the appropriate encoding for the platform or display environment. This util does not exist however


Discussion:
these look commented out. want to do a PR to remove?
```
g(pandas) bash-3.2$ grep -r console_encode pandas
pandas/io/formats/printing.py:# 2) If you need to send something to the console, use console_encode().
pandas/io/formats/printing.py:#    console_encode() should (hopefully) choose the right encoding for you
pandas/tests/io/formats/test_printing.py:# def test_console_encode():
pandas/tests/io/formats/test_printing.py:#     common.console_encode should encode things as utf-8.
pandas/tests/io/formats/test_printing.py:#         result = printing.console_encode(u"\u05d0")
```

Resolving patch:
diff --git a/pandas/io/formats/printing.py b/pandas/io/formats/printing.py
--- a/pandas/io/formats/printing.py
+++ b/pandas/io/formats/printing.py
@@ -81,12 +81,7 @@ def _join_unicode(lines, sep=''):
 #    It will always return unicode text which can handled by other
 #    parts of the package without breakage.
 #
-# 2) If you need to send something to the console, use console_encode().
-#
-#    console_encode() should (hopefully) choose the right encoding for you
-#    based on the encoding set in option "display.encoding"
-#
-# 3) if you need to write something out to file, use
+# 2) if you need to write something out to file, use
 #    pprint_thing_encoded(encoding).
 #
 #    If no encoding is specified, it defaults to utf-8. Since encoding pure


Test patch:
diff --git a/pandas/tests/io/formats/test_printing.py b/pandas/tests/io/formats/test_printing.py
--- a/pandas/tests/io/formats/test_printing.py
+++ b/pandas/tests/io/formats/test_printing.py
@@ -202,19 +202,3 @@ def test_enable_data_resource_formatter(self):
             assert formatters[mimetype].enabled
             # smoke test that it works
             self.display_formatter.format(cf)
-
-
-# TODO: fix this broken test
-
-# def test_console_encode():
-#     """
-#     On Python 2, if sys.stdin.encoding is None (IPython with zmq frontend)
-#     common.console_encode should encode things as utf-8.
-#     """
-#     if compat.PY3:
-#         pytest.skip
-
-#     with tm.stdin_encoding(encoding=None):
-#         result = printing.console_encode(u"\u05d0")
-#         expected = u"\u05d0".encode('utf-8')
-#         assert (result == expected)
Repository: pandas-dev/pandas

Issue:
Mistake in pd.DataFrame.assign() docstring
The docstring for ``df.DataFrame.assign()`` starts with 

```
Assign new columns to a DataFrame, returning a new object
(a copy) with all the original columns in addition to the new ones.
```

... but in fact existing columns can be overridden as well.

Hopefully tomorrow somebody will fix this!

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-5-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.23.0.dev0+463.g94ed47b15
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.7.0
Cython: 0.25.2
numpy: 1.14.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.5.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0dev
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 1.3.0
xlsxwriter: 0.9.6
lxml: 4.1.1
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1


</details>



Discussion:
I would like to take this on as my first open source contribution. 
@HannahFerch good idea! See https://pandas.pydata.org/pandas-docs/stable/contributing.html if you haven't already

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3168,7 +3168,8 @@ def insert(self, loc, column, value, allow_duplicates=False):
     def assign(self, **kwargs):
         r"""
         Assign new columns to a DataFrame, returning a new object
-        (a copy) with all the original columns in addition to the new ones.
+        (a copy) with the new columns added to the original ones.
+        Existing columns that are re-assigned will be overwritten.
 
         Parameters
         ----------


Test patch:
Repository: pandas-dev/pandas

Issue:
Unstack performance regression 
http://pandas.pydata.org/speed/pandas/#reshape.SparseIndex.time_unstack

https://github.com/pandas-dev/pandas/pull/18460#issuecomment-358365042

cc @toobaz 


Discussion:
@toobaz do you have time to take a look at this for 0.23?
I'll try to look at this, and hopefully fix it, next week.

Resolving patch:
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -1476,27 +1476,35 @@ def remove_unused_levels(self):
         changed = False
         for lev, lab in zip(self.levels, self.labels):
 
-            uniques = algos.unique(lab)
-            na_idx = np.where(uniques == -1)[0]
-
-            # nothing unused
-            if len(uniques) != len(lev) + len(na_idx):
+            # Since few levels are typically unused, bincount() is more
+            # efficient than unique() - however it only accepts positive values
+            # (and drops order):
+            uniques = np.where(np.bincount(lab + 1) > 0)[0] - 1
+            has_na = int(len(uniques) and (uniques[0] == -1))
+
+            if len(uniques) != len(lev) + has_na:
+                # We have unused levels
                 changed = True
 
-                if len(na_idx):
+                # Recalculate uniques, now preserving order.
+                # Can easily be cythonized by exploiting the already existing
+                # "uniques" and stop parsing "lab" when all items are found:
+                uniques = algos.unique(lab)
+                if has_na:
+                    na_idx = np.where(uniques == -1)[0]
                     # Just ensure that -1 is in first position:
                     uniques[[0, na_idx[0]]] = uniques[[na_idx[0], 0]]
 
                 # labels get mapped from uniques to 0:len(uniques)
                 # -1 (if present) is mapped to last position
-                label_mapping = np.zeros(len(lev) + len(na_idx))
+                label_mapping = np.zeros(len(lev) + has_na)
                 # ... and reassigned value -1:
-                label_mapping[uniques] = np.arange(len(uniques)) - len(na_idx)
+                label_mapping[uniques] = np.arange(len(uniques)) - has_na
 
                 lab = label_mapping[lab]
 
                 # new levels are simple
-                lev = lev.take(uniques[len(na_idx):])
+                lev = lev.take(uniques[has_na:])
 
             new_levels.append(lev)
             new_labels.append(lab)


Test patch:
Repository: pandas-dev/pandas

Issue:
BUG: Extra color introduced with plot.
User reported in StackOverflow: https://stackoverflow.com/questions/49887076/matplotlib-error-while-coloring-barplot-with-list#49887076
```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame(np.abs(np.random.randn(48, 4)), columns=list("ABCD"))

color_list = plt.cm.gnuplot(np.linspace(0,1,16))

df.A.plot.bar( figsize=(16,7), color=color_list)
plt.axvline(15.5, c="black", lw=1)
plt.axvline(31.5, c="black", lw=1)
plt.plot()
```
#### Problem description
This bug happens when single color is needed for plotting but an array is provided. 
In the end of `_get_standard_colors` function in plotting/_style.py. An extra first item will be added to the color array which caused the circular misbehaved in such case.
![image](https://user-images.githubusercontent.com/659168/38904144-94530962-425d-11e8-82c1-2b5767caba2c.png)

#### Expected Output
The bar 17 should have the same color as the bar 1.
#### 


Discussion:
I can send a PR to fix this.

Resolving patch:
diff --git a/pandas/plotting/_style.py b/pandas/plotting/_style.py
--- a/pandas/plotting/_style.py
+++ b/pandas/plotting/_style.py
@@ -81,7 +81,10 @@ def _maybe_valid_colors(colors):
             # mpl will raise error any of them is invalid
             pass
 
-    if len(colors) != num_colors:
+    # Append more colors by cycling if there is not enough color.
+    # Extra colors will be ignored by matplotlib if there are more colors
+    # than needed and nothing needs to be done here.
+    if len(colors) < num_colors:
         try:
             multiple = num_colors // len(colors) - 1
         except ZeroDivisionError:


Test patch:
diff --git a/pandas/tests/plotting/test_misc.py b/pandas/tests/plotting/test_misc.py
--- a/pandas/tests/plotting/test_misc.py
+++ b/pandas/tests/plotting/test_misc.py
@@ -309,3 +309,21 @@ def test_get_standard_colors_random_seed(self):
         color1 = _get_standard_colors(1, color_type='random')
         color2 = _get_standard_colors(1, color_type='random')
         assert color1 == color2
+
+    def test_get_standard_colors_no_appending(self):
+        # GH20726
+
+        # Make sure not to add more colors so that matplotlib can cycle
+        # correctly.
+        from matplotlib import cm
+        color_before = cm.gnuplot(range(5))
+        color_after = plotting._style._get_standard_colors(
+            1, color=color_before)
+        assert len(color_after) == len(color_before)
+
+        df = DataFrame(np.random.randn(48, 4), columns=list("ABCD"))
+
+        color_list = cm.gnuplot(np.linspace(0, 1, 16))
+        p = df.A.plot.bar(figsize=(16, 7), color=color_list)
+        assert (p.patches[1].get_facecolor()
+                == p.patches[17].get_facecolor())
Repository: pandas-dev/pandas

Issue:
segfault in DataFrame.apply with duplicate column names and multiple aggfuncs
```python
In [1]: import pandas as pd

In [2]: df = pd.DataFrame([['a', 'b'], ['c', 'd']], columns=['a', 'a'])

In [3]: df.agg(['min'])
Fatal Python error: Cannot recover from stack overflow.
```

This was present at least in 0.22.0. Haven't looked further.


Discussion:

Resolving patch:
diff --git a/pandas/core/base.py b/pandas/core/base.py
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -590,9 +590,10 @@ def _aggregate_multiple_funcs(self, arg, _level, _axis):
 
         # multiples
         else:
-            for col in obj:
+            for index, col in enumerate(obj):
                 try:
-                    colg = self._gotitem(col, ndim=1, subset=obj[col])
+                    colg = self._gotitem(col, ndim=1,
+                                         subset=obj.iloc[:, index])
                     results.append(colg.aggregate(arg))
                     keys.append(col)
                 except (TypeError, DataError):
@@ -675,7 +676,6 @@ def _gotitem(self, key, ndim, subset=None):
         subset : object, default None
             subset to act on
         """
-
         # create a new object to prevent aliasing
         if subset is None:
             subset = self.obj
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -5731,7 +5731,12 @@ def diff(self, periods=1, axis=0):
     # ----------------------------------------------------------------------
     # Function application
 
-    def _gotitem(self, key, ndim, subset=None):
+    def _gotitem(self,
+                 key,           # type: Union[str, List[str]]
+                 ndim,          # type: int
+                 subset=None    # type: Union[Series, DataFrame, None]
+                 ):
+        # type: (...) -> Union[Series, DataFrame]
         """
         sub-classes to define
         return a sliced object
@@ -5746,9 +5751,11 @@ def _gotitem(self, key, ndim, subset=None):
         """
         if subset is None:
             subset = self
+        elif subset.ndim == 1:  # is Series
+            return subset
 
         # TODO: _shallow_copy(subset)?
-        return self[key]
+        return subset[key]
 
     _agg_doc = dedent("""
     The aggregation operations are always performed over an axis, either the


Test patch:
diff --git a/pandas/tests/frame/test_apply.py b/pandas/tests/frame/test_apply.py
--- a/pandas/tests/frame/test_apply.py
+++ b/pandas/tests/frame/test_apply.py
@@ -554,6 +554,14 @@ def test_apply_non_numpy_dtype(self):
         result = df.apply(lambda x: x)
         assert_frame_equal(result, df)
 
+    def test_apply_dup_names_multi_agg(self):
+        # GH 21063
+        df = pd.DataFrame([[0, 1], [2, 3]], columns=['a', 'a'])
+        expected = pd.DataFrame([[0, 1]], columns=['a', 'a'], index=['min'])
+        result = df.agg(['min'])
+
+        tm.assert_frame_equal(result, expected)
+
 
 class TestInferOutputShape(object):
     # the user has supplied an opaque UDF where
Repository: pandas-dev/pandas

Issue:
DOC: minor deprecate message documentation on frame.dropna
#### Problem description

Indentation on "axis" incorrect, falling out of deprecated message.

<img width="764" alt="master" src="https://user-images.githubusercontent.com/14131823/40563288-26a7af68-605c-11e8-90d9-ec8e27683bcd.png">

#### Expected Output
<img width="749" alt="pr" src="https://user-images.githubusercontent.com/14131823/40563824-277d44f0-605e-11e8-8743-bcd87a202d1d.png">

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: 0d8e7c976b8e39e1a32053b6e4382fbb5cbe5c3a
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.24.0.dev0+32.g0d8e7c976
pytest: 3.6.0
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.1.0
pyarrow: 0.9.0.post1
xarray: 0.10.4
IPython: 6.4.0
sphinx: 1.7.4
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: 1.5.1
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: 0.4.0
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.5
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: 0.8.1
psycopg2: None
jinja2: 2.10
s3fs: 0.1.5
fastparquet: 0.1.5
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -4177,8 +4177,9 @@ def dropna(self, axis=0, how='any', thresh=None, subset=None,
             * 0, or 'index' : Drop rows which contain missing values.
             * 1, or 'columns' : Drop columns which contain missing value.
 
-            .. deprecated:: 0.23.0: Pass tuple or list to drop on multiple
-            axes.
+            .. deprecated:: 0.23.0
+                Pass tuple or list to drop on multiple axes.
+
         how : {'any', 'all'}, default 'any'
             Determine if row or column is removed from DataFrame, when we have
             at least one NA or all NA.


Test patch: