internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pandas-dev/pandas

Issue:
BUG: accept np.datetime64('NaT') in an object dtyped as a null
from [SO](http://stackoverflow.com/questions/32863674/python-pandas-isnull-does-not-work-on-nat-in-object-dtype)

this should work

```
Out[3]: array([11, 22, 33, nan, numpy.datetime64('NaT')], dtype=object)

In [4]: pd.isnull(ser)
Out[4]: 
0    False
1    False
2    False
3     True
4    False
Name: my_series, dtype: bool
```

This could be updated: https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L711, then `lib.isnullobj` then could use for null detection



Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/frame_methods.py b/asv_bench/benchmarks/frame_methods.py
--- a/asv_bench/benchmarks/frame_methods.py
+++ b/asv_bench/benchmarks/frame_methods.py
@@ -582,7 +582,7 @@ def time_frame_interpolate_some_good_infer(self):
         self.df.interpolate(downcast='infer')
 
 
-class frame_isnull(object):
+class frame_isnull_floats_no_null(object):
     goal_time = 0.2
 
     def setup(self):
@@ -593,6 +593,33 @@ def time_frame_isnull(self):
         isnull(self.df)
 
 
+class frame_isnull_floats(object):
+    goal_time = 0.2
+
+    def setup(self):
+        np.random.seed(1234)
+        self.sample = np.array([np.nan, 1.0])
+        self.data = np.random.choice(self.sample, (1000, 1000))
+        self.df = DataFrame(self.data)
+
+    def time_frame_isnull(self):
+        isnull(self.df)
+
+
+class frame_isnull_obj(object):
+    goal_time = 0.2
+
+    def setup(self):
+        np.random.seed(1234)
+        self.sample = np.array([NaT, np.nan, None, np.datetime64('NaT'),
+                                np.timedelta64('NaT'), 0, 1, 2.0, '', 'abcd'])
+        self.data = np.random.choice(self.sample, (1000, 1000))
+        self.df = DataFrame(self.data)
+
+    def time_frame_isnull(self):
+        isnull(self.df)
+
+
 class frame_iteritems(object):
     goal_time = 0.2
 


Test patch:
diff --git a/pandas/tests/test_common.py b/pandas/tests/test_common.py
--- a/pandas/tests/test_common.py
+++ b/pandas/tests/test_common.py
@@ -179,6 +179,13 @@ def test_isnull_nat():
     exp = np.array([True])
     assert(np.array_equal(result, exp))
 
+def test_isnull_numpy_nat():
+    arr = np.array([NaT, np.datetime64('NaT'), np.timedelta64('NaT'),
+                    np.datetime64('NaT', 's')])
+    result = isnull(arr)
+    expected = np.array([True] * 4)
+    tm.assert_numpy_array_equal(result, expected)
+
 def test_isnull_datetime():
     assert (not isnull(datetime.now()))
     assert notnull(datetime.now())
Repository: pandas-dev/pandas

Issue:
DOC: typo in convert_objects
In the new warning. `to_timestamp` should be `to_timedelta`

```
df.convert_objects()
...
FutureWarning: convert_objects is deprecated.  
Use the data-type specific converters pd.to_datetime, pd.to_timestamp and pd.to_numeric.
```



Discussion:

Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -2595,7 +2595,7 @@ def convert_objects(self, convert_dates=True, convert_numeric=False,
         """
         from warnings import warn
         warn("convert_objects is deprecated.  Use the data-type specific "
-             "converters pd.to_datetime, pd.to_timestamp and pd.to_numeric.",
+             "converters pd.to_datetime, pd.to_timedelta and pd.to_numeric.",
              FutureWarning, stacklevel=2)
 
         return self._constructor(


Test patch:
Repository: pandas-dev/pandas

Issue:
Improve the BigQuery streaming insert failure message
The error message shown when a BigQuery streaming insert fails could be improved by adding the location field. 

Code to reproduce:

```
from datetime import datetime
from pandas import DataFrame
from pandas.io import gbq
import pytz

test_timestamp = datetime.now(pytz.timezone('US/Arizona'))
bad_df = DataFrame({'bools': [False, False], 'flts': [0.0, 1.0], 'ints': [0, '1'], 'strs': ['a', 1],
                    'times': [test_timestamp, test_timestamp]}, index=range(2))
gbq.to_gbq(bad_df, destination_table, project_id, verbose=True,if_exists='replace')
```

Output:

```
Streaming Insert is 100% Complete
Error at Row: 0, Reason: invalid, Message: Could not convert value to string
Error at Row: 1, Reason: invalid, Message: Could not convert value to string
```

Proposed output:

```
Streaming Insert is 100% Complete
Error at Row: 0, Reason: invalid, Location: Field:ints, Message: Could not convert value to string
Error at Row: 1, Reason: invalid, Location: Field:strs, Message: Could not convert value to string
```



Discussion:

Resolving patch:
diff --git a/pandas/io/gbq.py b/pandas/io/gbq.py
--- a/pandas/io/gbq.py
+++ b/pandas/io/gbq.py
@@ -185,7 +185,8 @@ def process_insert_errors(insert_errors, verbose):
             for error in errors:
                 reason = error['reason']
                 message = error['message']
-                error_message = 'Error at Row: {0}, Reason: {1}, Message: {2}'.format(row, reason, message)
+                location = error['location']
+                error_message = 'Error at Row: {0}, Reason: {1}, Location: {2}, Message: {3}'.format(row, reason, location, message)
 
                 # Report all error messages if verbose is set
                 if verbose:


Test patch:
Repository: pandas-dev/pandas

Issue:
Mistake in BinOp.conform?
in BinOp.conform in pandas.computation,pytables there is this code:

```
    def conform(self, rhs):
        """ inplace conform rhs """
        if not com.is_list_like(rhs):
            rhs = [rhs]
        if hasattr(self.rhs, 'ravel'):
            rhs = rhs.ravel()
        return rhs
```

I think this is wrong.  The first test checks `rhs`, but the second checks `self.rhs`.  They should both check one or the other.  I _think_ both should check local `rhs` but I'm not really familiar with all the nuts and bolts here so I'm not sure.

This causes failures when doing HDF5 queries making use of a local variable whose value is a single Numpy value (e.g., a float64).  Because even a single numpy value has `ravel`, but it is not list-like, both `if` blocks execute, meaning that it sets `rhs` to a list and then tries to call `ravel` on it.  This leads to a difficult-to-debug error because the exception is caught at a higher level and replaced with an "Invalid query syntax" error message.



Discussion:
if u can propose a fix gr8.

I already did propose a fix in my bug report above.  It is a less-than-one-line fix: change `self.rhs` to `rhs`.

ok pls submit a pr with tests pls


Resolving patch:
diff --git a/pandas/computation/pytables.py b/pandas/computation/pytables.py
--- a/pandas/computation/pytables.py
+++ b/pandas/computation/pytables.py
@@ -129,7 +129,7 @@ def conform(self, rhs):
         """ inplace conform rhs """
         if not com.is_list_like(rhs):
             rhs = [rhs]
-        if hasattr(self.rhs, 'ravel'):
+        if isinstance(rhs, np.ndarray):
             rhs = rhs.ravel()
         return rhs
 


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -3049,6 +3049,18 @@ def test_select_dtypes(self):
             result = store.select(
                 'df4', where='values>2.0')
             tm.assert_frame_equal(expected, result)
+        
+        # test selection with comparison against numpy scalar
+        # GH 11283
+        with ensure_clean_store(self.path) as store:
+            df = tm.makeDataFrame()
+
+            expected = df[df['A']>0]
+
+            store.append('df', df, data_columns=True)
+            np_zero = np.float64(0)
+            result = store.select('df', where=["A>np_zero"])
+            tm.assert_frame_equal(expected, result)
 
     def test_select_with_many_inputs(self):
Repository: pandas-dev/pandas

Issue:
PERF: checking is_monotonic_increasing/decreasing before sorting on an index
We don't keep the sortedness state in an index per-se, but it is rather cheap to check
- `is_monotonic_increasing` or `is_monotonic_decreasing` on a reg-index 
- MultiIndex should check `is_lexsorted` (this might be done already)

```
In [8]: df = DataFrame(np.random.randn(1000000,2),columns=list('AB'))

In [9]: %timeit df.sort_index()
10 loops, best of 3: 37.1 ms per loop

In [10]: %timeit -n 1 -r 1 df.index.is_monotonic_increasing
1 loops, best of 1: 2.01 ms per loop

In [11]: %timeit -n 1 -r 1 df.index.is_monotonic_increasin^C
KeyboardInterrupt

In [11]: %timeit df.set_index('A').sort_index()
10 loops, best of 3: 175 ms per loop

In [12]: %timeit -n 1 -r 1 df.set_index('A').index.is_monotonic_increasing
1 loops, best of 1: 9.54 ms per loop
```



Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/frame_methods.py b/asv_bench/benchmarks/frame_methods.py
--- a/asv_bench/benchmarks/frame_methods.py
+++ b/asv_bench/benchmarks/frame_methods.py
@@ -930,6 +930,16 @@ def time_frame_xs_row(self):
         self.df.xs(50000)
 
 
+class frame_sort_index(object):
+    goal_time = 0.2
+
+    def setup(self):
+        self.df = DataFrame(randn(1000000, 2), columns=list('AB'))
+
+    def time_frame_sort_index(self):
+        self.df.sort_index()
+
+
 class series_string_vector_slice(object):
     goal_time = 0.2
 
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3157,6 +3157,15 @@ def sort_index(self, axis=0, level=None, ascending=True, inplace=False,
         else:
             from pandas.core.groupby import _nargsort
 
+            # GH11080 - Check monotonic-ness before sort an index
+            # if monotonic (already sorted), return None or copy() according to 'inplace'
+            if (ascending and labels.is_monotonic_increasing) or \
+               (not ascending and labels.is_monotonic_decreasing):
+                if inplace:
+                    return
+                else:
+                    return self.copy()
+
             indexer = _nargsort(labels, kind=kind, ascending=ascending,
                                 na_position=na_position)
 


Test patch:
Repository: pandas-dev/pandas

Issue:
DEPR: google-analytics
not supported / tested in quite some time. this should be spun off into a separate package.

let's deprecate at least in 0.17.1.



Discussion:
+1 to split this off into its own package


Resolving patch:
diff --git a/pandas/io/ga.py b/pandas/io/ga.py
--- a/pandas/io/ga.py
+++ b/pandas/io/ga.py
@@ -4,6 +4,13 @@
 3. Goto APIs and register for OAuth2.0 for installed applications
 4. Download JSON secret file and move into same directory as this file
 """
+
+# GH11038
+import warnings
+warnings.warn("The pandas.io.ga module is deprecated and will be "
+              "removed in a future version.",
+              FutureWarning, stacklevel=2)
+
 from datetime import datetime
 import re
 from pandas import compat


Test patch:
Repository: pandas-dev/pandas

Issue:
May a bug of `Series.plot` in pandas 0.17?
The origin post with pictures is on: http://stackoverflow.com/questions/33559772/is-this-a-bug-of-series-plot-in-pandas-0-17

I always use `ser.plot(xticks=range(0, len(ser), step), ..)` to plot figures with specified xticks. But it doesn't work anymore in pandas 0.17 when the Series with labels. Here are the codes:

``` python
In [1]: from numpy import random as rnd 
In [2]: import pandas as pd
In [3]: pd.__version__
Out[3]: '0.17.0'
In [4]: %matplotlib inline
In [5]: rnd.seed(123)
In [6]: ser = pd.Series(rnd.randn(73).cumsum(), index=['P%02d' % i for i in range(73)])
In [7]: ser.plot(figsize=(9, 6), rot=60, title='Figure without xticks')
Out[7]: <matplotlib.axes._subplots.AxesSubplot at 0x8370198
In [8]: ser.plot(figsize=(9, 6), xticks=list(range(0, 73, 6)), rot=60, title='Figure with xticks')
Out[8]: <matplotlib.axes._subplots.AxesSubplot at 0x83e9940>
```

![image](https://cloud.githubusercontent.com/assets/511719/10991679/4acedd1c-849a-11e5-89d0-f6264b788849.png)
![image](https://cloud.githubusercontent.com/assets/511719/10991682/4d695a16-849a-11e5-9db3-6ff885a22234.png)

The figure with xticks specified doesn't display xticklabes correctly.



Discussion:
Yes, thanks for reporting! This is a regression.


Resolving patch:
diff --git a/pandas/tools/plotting.py b/pandas/tools/plotting.py
--- a/pandas/tools/plotting.py
+++ b/pandas/tools/plotting.py
@@ -985,11 +985,11 @@ def generate(self):
         self._make_plot()
         self._add_table()
         self._make_legend()
+        self._adorn_subplots()
 
         for ax in self.axes:
             self._post_plot_logic_common(ax, self.data)
             self._post_plot_logic(ax, self.data)
-        self._adorn_subplots()
 
     def _args_adjust(self):
         pass


Test patch:
diff --git a/pandas/tests/test_graphics.py b/pandas/tests/test_graphics.py
--- a/pandas/tests/test_graphics.py
+++ b/pandas/tests/test_graphics.py
@@ -1237,6 +1237,13 @@ def test_time_series_plot_color_with_empty_kwargs(self):
             ax = s.plot()
         self._check_colors(ax.get_lines(), linecolors=def_colors[:ncolors])
 
+    def test_xticklabels(self):
+        # GH11529
+        s = Series(np.arange(10), index=['P%02d' % i for i in range(10)])
+        ax = s.plot(xticks=[0,3,5,9])
+        exp = ['P%02d' % i for i in [0,3,5,9]]
+        self._check_text_labels(ax.get_xticklabels(), exp)
+
 
 @tm.mplskip
 class TestDataFramePlots(TestPlotBase):
Repository: pandas-dev/pandas

Issue:
COMPAT: return instead of raising StopIteration in generator
And a couple other changes for [PEP479](https://www.python.org/dev/peps/pep-0479/) to avoid warnings. Will submit a PR if others agree.



Discussion:
sure

@kawochen pr for this?

Ah yes will do tomorrow. 

> On Nov 13, 2015, at 2:01 PM, Jeff Reback notifications@github.com wrote:
> 
> @kawochen pr for this?
> 
> —
> Reply to this email directly or view it on GitHub.

Working on it.  I can trigger the warnings in `SparseArray`.  Will see how to trigger those in `io`. 


Resolving patch:
diff --git a/pandas/sparse/array.py b/pandas/sparse/array.py
--- a/pandas/sparse/array.py
+++ b/pandas/sparse/array.py
@@ -274,7 +274,6 @@ def to_dense(self, fill=None):
     def __iter__(self):
         for i in range(len(self)):
             yield self._get_val_at(i)
-        raise StopIteration
 
     def __getitem__(self, key):
         """


Test patch:
diff --git a/pandas/sparse/tests/test_array.py b/pandas/sparse/tests/test_array.py
--- a/pandas/sparse/tests/test_array.py
+++ b/pandas/sparse/tests/test_array.py
@@ -4,6 +4,7 @@
 import numpy as np
 
 import operator
+import warnings
 
 from pandas.core.series import Series
 from pandas.core.common import notnull
@@ -174,6 +175,18 @@ def _check_roundtrip(obj):
         _check_roundtrip(self.arr)
         _check_roundtrip(self.zarr)
 
+    def test_generator_warnings(self):
+        sp_arr = SparseArray([1, 2, 3])
+        with warnings.catch_warnings(record=True) as w:
+            warnings.filterwarnings(action='always',
+                                    category=DeprecationWarning)
+            warnings.filterwarnings(action='always',
+                                    category=PendingDeprecationWarning)
+            for _ in sp_arr:
+                pass
+            assert len(w)==0
+
+
 if __name__ == '__main__':
     import nose
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],
Repository: pandas-dev/pandas

Issue:
BUG in remove_unused_categories with NaNs in values
From SO: http://stackoverflow.com/questions/33693601/removing-unused-categories-in-series-results-in-duplicated-categories

`removed_unused_categories` gives a wrong result when there is a NaN in the values:

```
>>> s = pd.Series(["A", "B", pd.np.nan]).astype("category")
>>> s.cat.remove_unused_categories()
0      A
1      B
2    NaN
dtype: category
Categories (3, object): [B, A, B]
```

I think the `-1` in the codes (`NaN`) duplicates the last item in the categories



Discussion:

Resolving patch:
diff --git a/pandas/core/categorical.py b/pandas/core/categorical.py
--- a/pandas/core/categorical.py
+++ b/pandas/core/categorical.py
@@ -817,6 +817,9 @@ def remove_unused_categories(self, inplace=False):
         """
         cat = self if inplace else self.copy()
         _used = sorted(np.unique(cat._codes))
+        if _used[0] == -1:
+            _used = _used[1:]
+
         new_categories = cat.categories.take(_ensure_platform_int(_used))
 
         from pandas.core.index import _ensure_index


Test patch:
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -844,6 +844,11 @@ def test_remove_unused_categories(self):
         self.assert_numpy_array_equal(c.categories, exp_categories_dropped)
         self.assertIsNone(res)
 
+        # with NaN values (GH11599)
+        c = Categorical(["a","b","c",np.nan], categories=["a","b","c","d","e"])
+        res = c.remove_unused_categories()
+        self.assert_numpy_array_equal(res.categories, np.array(["a","b","c"]))
+        self.assert_numpy_array_equal(c.categories, exp_categories_all)
 
     def test_nan_handling(self):
Repository: pandas-dev/pandas

Issue:
DatetimeIndex with closed='left' or 'right' drop dates at the boundaries
When either start  or end is at the boundary of a date_range, they get dropped when they shouldn't if we specify the closed argument, see example below:

```
In [1]: pd.date_range('2015-09-12', '2015-10-30', freq='QS-MAR', closed='right')
Out[1]: DatetimeIndex([], dtype='datetime64[ns]', freq='QS-MAR')

In [2]: pd.date_range('2015-09-01', '2015-10-30', freq='QS-MAR', closed='left')
Out[2]: DatetimeInddex([], dtype='datetime64[ns]', freq='QS-MAR')
```



Discussion:

Resolving patch:
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -486,10 +486,9 @@ def _generate(cls, start, end, periods, name, offset,
                 index = index.view(_NS_DTYPE)
 
         index = cls._simple_new(index, name=name, freq=offset, tz=tz)
-
-        if not left_closed:
+        if not left_closed and len(index) and  index[0] == start:
             index = index[1:]
-        if not right_closed:
+        if not right_closed and len(index) and index[-1] == end:
             index = index[:-1]
 
         return index


Test patch:
diff --git a/pandas/tseries/tests/test_daterange.py b/pandas/tseries/tests/test_daterange.py
--- a/pandas/tseries/tests/test_daterange.py
+++ b/pandas/tseries/tests/test_daterange.py
@@ -477,13 +477,37 @@ def test_range_closed(self):
             closed = date_range(begin, end, closed=None, freq=freq)
             left = date_range(begin, end, closed="left", freq=freq)
             right = date_range(begin, end, closed="right", freq=freq)
+            expected_left = left
+            expected_right = right
 
-            expected_left = closed[:-1]
-            expected_right = closed[1:]
+            if end == closed[-1]:
+                expected_left = closed[:-1]
+            if begin == closed[0]:
+                expected_right = closed[1:]
 
             self.assertTrue(expected_left.equals(left))
             self.assertTrue(expected_right.equals(right))
 
+    def test_range_closed_boundary(self):
+        # GH 11804
+        for closed in ['right', 'left', None]:
+            right_boundary = date_range('2015-09-12', '2015-12-01', freq='QS-MAR', closed=closed)
+            left_boundary = date_range('2015-09-01', '2015-09-12', freq='QS-MAR', closed=closed)
+            both_boundary = date_range('2015-09-01', '2015-12-01', freq='QS-MAR', closed=closed)
+            expected_right = expected_left = expected_both = both_boundary
+
+            if closed == 'right':
+                expected_left = both_boundary[1:]
+            if closed == 'left':
+                expected_right = both_boundary[:-1]
+            if closed is None:
+                expected_right = both_boundary[1:]
+                expected_left = both_boundary[:-1]
+
+            self.assertTrue(right_boundary.equals(expected_right))
+            self.assertTrue(left_boundary.equals(expected_left))
+            self.assertTrue(both_boundary.equals(expected_both))
+
     def test_years_only(self):
         # GH 6961
         dr = date_range('2014', '2015', freq='M')
Repository: pandas-dev/pandas

Issue:
COMPAT: drop support for Python 3.3



Discussion:
any objections for 0.18.0 (early next year) here

@jorisvandenbossche @shoyer @cpcloud @TomAugspurger @sinhrks @wesm 

3.4 and 3.5 pretty well supported (and conda no longer building for 3.3)
though I can build/post them

Same as 2.6, mpl is going to officially drop 3.3 support for 2.0, but I do not anticipate mpl using any py3 features which are not in 3.3 as we are still supporting legacy python anyway.

@tacaswell when you are anticipating 2.0?

End of november/early December.  The current plan is for me and @mdboom to sort out the new default around pydata NYC.

3.3 might be 'officially' dropped from 1.5 which should be out it ~week as well.

@tacaswell ok gr8. we are not planning this till 0.18.0 prob early next year.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -180,7 +180,6 @@ def build_extensions(self):
     'Programming Language :: Python :: 3',
     'Programming Language :: Python :: 2.6',
     'Programming Language :: Python :: 2.7',
-    'Programming Language :: Python :: 3.3',
     'Programming Language :: Python :: 3.4',
     'Programming Language :: Python :: 3.5',
     'Programming Language :: Cython',


Test patch:
Repository: pandas-dev/pandas

Issue:
read_excel documentation does not mention names keyword
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html
does not mention the keyword "names"

``` python
import pandas as pd
df = pd.read_excel('yourfile.xls', header=None, names=['Attribut', 'Value'])
```

is equivalent but simpler to

``` python
df = pd.read_excel(file, sheetname='Metadata', header=None)
df.columns = ['Attribut', 'Value']
```

names keyword is defined as in read_csv
names: List of column names to use as column names. To replace header existing in file, explicitly pass header=0.



Discussion:
suppose that's reasonable. a number of parameters are actually passed thru to `read_csv`.

a pull-request to add to the doc-string (and as a named parameter) would be helpful.


Resolving patch:
diff --git a/pandas/io/excel.py b/pandas/io/excel.py
--- a/pandas/io/excel.py
+++ b/pandas/io/excel.py
@@ -71,7 +71,7 @@ def get_writer(engine_name):
         raise ValueError("No Excel writer '%s'" % engine_name)
 
 def read_excel(io, sheetname=0, header=0, skiprows=None, skip_footer=0,
-               index_col=None, parse_cols=None, parse_dates=False,
+               index_col=None, names=None, parse_cols=None, parse_dates=False,
                date_parser=None, na_values=None, thousands=None,
                convert_float=True, has_index_names=None, converters=None,
                engine=None, **kwds):
@@ -116,6 +116,9 @@ def read_excel(io, sheetname=0, header=0, skiprows=None, skip_footer=0,
         Column (0-indexed) to use as the row labels of the DataFrame.
         Pass None if there is no such column.  If a list is passed,
         those columns will be combined into a ``MultiIndex``
+    names : array-like, default None
+        List of column names to use. If file contains no header row,
+        then you should explicitly pass header=None
     converters : dict, default None
         Dict of functions for converting values in certain columns. Keys can
         either be integers or column labels, values are functions that take one


Test patch: