internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pandas-dev/pandas

Issue:
Fix Type Annotations in pandas/_config/config.py
Part of #25882

Current errors are as follows:

```sh
pandas/_config/config.py:60: error: Need type annotation for '_deprecated_options'
pandas/_config/config.py:61: error: Need type annotation for '_registered_options'
pandas/_config/config.py:62: error: Need type annotation for '_global_config'
```

Should add annotations and remove this module from mypy.ini - think this is straightforward so community PR is certainly welcome!


Discussion:
I will take care of this too.
The problem with these three are that they already have a comment describing what these variables do. If I add type comment too the line becomes too long:

    _deprecated_options = {}  # type: Dict[str, str]  # holds deprecated option metdata

How should I split it to multiple lines
This comment doesn’t really add much over code so I’d just delete. In other cases could move to a different line. 

Sent from my iPhone

> On Mar 28, 2019, at 7:15 AM, Vaibhav Vishal <notifications@github.com> wrote:
> 
> The problem with these three are that they already have a comment describing what these variables do. If I add type comment too the line becomes too long:
> 
> _deprecated_options = {}  # type: Dict[str, str]  # holds deprecated option metdata
> How should I split it to multiple lines
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.


Resolving patch:
diff --git a/pandas/_config/config.py b/pandas/_config/config.py
--- a/pandas/_config/config.py
+++ b/pandas/_config/config.py
@@ -51,16 +51,24 @@
 from collections import namedtuple
 from contextlib import contextmanager
 import re
+from typing import Dict, List
 import warnings
 
 DeprecatedOption = namedtuple('DeprecatedOption', 'key msg rkey removal_ver')
 RegisteredOption = namedtuple('RegisteredOption',
                               'key defval doc validator cb')
 
-_deprecated_options = {}  # holds deprecated option metdata
-_registered_options = {}  # holds registered option metdata
-_global_config = {}  # holds the current values for registered options
-_reserved_keys = ['all']  # keys which have a special meaning
+# holds deprecated option metdata
+_deprecated_options = {}  # type: Dict[str, DeprecatedOption]
+
+# holds registered option metdata
+_registered_options = {}  # type: Dict[str, RegisteredOption]
+
+# holds the current values for registered options
+_global_config = {}  # type: Dict[str, str]
+
+# keys which have a special meaning
+_reserved_keys = ['all']  # type: List[str]
 
 
 class OptionError(AttributeError, KeyError):


Test patch:
Repository: pandas-dev/pandas

Issue:
Duplicate xstrtod implementations
In _libs/src/parse_helper.h and libs/src/parser/tokenizer.c

Not identical, comments suggest the tokenizer.c version may be more modern.


Discussion:
The core functionally is the same, differences are the tokenizer.c version handles a thousands separator, the parse_helper.c version has a `maybe_int` out variable, agree they could be combined.
@chris-b1 There are .gitignore and Makefile files in _libs/src/parser.  Are those used for something or remnants from the Old Days?
Yeah, definitely something very old, I think those could be cleaned out too.
It also looks like src/klib/ktypes.h and kvec.h are unused, though this should be checked by someone who understands C namespaces.
Yeah, those are unused as well - for good and bad C doesn't have namespaces, (or modules, or imports, etc) - all it can really do is textual inclusion, so if those aren't `#include`'ed somewhere, they aren't used.
_libs.groupby.kth_smallest_c almost identical to _libs.algos.kth_smallest
@chris-b1 hopefully a softball question: Currently util [gets its](https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/util.pxd#L89) INT32_MAX etc constants from headers/stdint.h.  headers/stdint.h contains special-casing for windows systems.  Is that special casing relevant for UINT64_MAX etc, or can we use the (much more convenient for current purposes) `from libc.stdint cimport ...`?
Yes, that special casing is required - cython definitions look for a `'stdint.h'` which the Windows  toolchain for 2.7 doesn't have, so the build would fail.
https://www.microsoft.com/en-us/download/details.aspx?id=44266


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -558,7 +558,8 @@ def srcpath(name=None, suffix='.pyx', subdir='src'):
     '_libs.lib': {
         'pyxfile': '_libs/lib',
         'include': common_include + ts_include,
-        'depends': lib_depends + tseries_depends},
+        'depends': lib_depends + tseries_depends,
+        'sources': ['pandas/_libs/src/parser/tokenizer.c']},
     '_libs.missing': {
         'pyxfile': '_libs/missing',
         'include': common_include + ts_include,


Test patch:
Repository: pandas-dev/pandas

Issue:
Add ArrayLike an Dtype definitions to pandas._typing
The existing type hints in the codebase suggest a couple of easy additions to the pandas._typing module, including 

- [ ] ArrayLike = `Union[ABCExtensionArray, np.ndarray]` and
- [ ] Dtype = `Union[str, np.dtype, ExtensionDtype, Type]`.

These should be added to the typing module and the matching types replaced with the newly defined ones where they occur.

Crossreferences: #25829 and #25791.


Discussion:

Resolving patch:
diff --git a/pandas/_typing.py b/pandas/_typing.py
--- a/pandas/_typing.py
+++ b/pandas/_typing.py
@@ -1,4 +1,11 @@
 from pathlib import Path
 from typing import IO, AnyStr, Union
 
+import numpy as np
+
+from pandas.core.dtypes.dtypes import ExtensionDtype
+from pandas.core.dtypes.generic import ABCExtensionArray
+
+ArrayLike = Union[ABCExtensionArray, np.ndarray]
+Dtype = Union[str, np.dtype, ExtensionDtype]
 FilePathOrBuffer = Union[str, Path, IO[AnyStr]]
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
--- a/pandas/core/arrays/base.py
+++ b/pandas/core/arrays/base.py
@@ -21,6 +21,7 @@
     ABCExtensionArray, ABCIndexClass, ABCSeries)
 from pandas.core.dtypes.missing import isna
 
+from pandas._typing import ArrayLike
 from pandas.core import ops
 
 _not_implemented_message = "{} does not implement {}."
@@ -338,7 +339,7 @@ def astype(self, dtype, copy=True):
         """
         return np.array(self, dtype=dtype, copy=copy)
 
-    def isna(self) -> Union[ABCExtensionArray, np.ndarray]:
+    def isna(self) -> ArrayLike:
         """
         A 1-D array indicating if each value is missing.
 
diff --git a/pandas/core/arrays/sparse.py b/pandas/core/arrays/sparse.py
--- a/pandas/core/arrays/sparse.py
+++ b/pandas/core/arrays/sparse.py
@@ -4,7 +4,7 @@
 import numbers
 import operator
 import re
-from typing import Any, Callable, Type, Union
+from typing import Any, Callable
 import warnings
 
 import numpy as np
@@ -30,6 +30,7 @@
     ABCIndexClass, ABCSeries, ABCSparseArray, ABCSparseSeries)
 from pandas.core.dtypes.missing import isna, na_value_for_dtype, notna
 
+from pandas._typing import Dtype
 from pandas.core.accessor import PandasDelegate, delegate_names
 import pandas.core.algorithms as algos
 from pandas.core.arrays import ExtensionArray, ExtensionOpsMixin
@@ -79,7 +80,7 @@ class SparseDtype(ExtensionDtype):
 
     def __init__(
             self,
-            dtype: Union[str, np.dtype, ExtensionDtype, Type] = np.float64,
+            dtype: Dtype = np.float64,
             fill_value: Any = None
     ) -> None:
         from pandas.core.dtypes.missing import na_value_for_dtype


Test patch:
Repository: pandas-dev/pandas

Issue:
Fix Type Annotations in pandas/_version.py
Part of #25882 

Current errors are as follows:

    pandas\_version.py:51: error: Need type annotation for 'LONG_VERSION_PY'
    pandas\_version.py:52: error: Need type annotation for 'HANDLERS'




Discussion:

Resolving patch:
diff --git a/pandas/_version.py b/pandas/_version.py
--- a/pandas/_version.py
+++ b/pandas/_version.py
@@ -12,6 +12,7 @@
 import re
 import subprocess
 import sys
+from typing import Callable, Dict
 
 from pandas.compat import PY3
 
@@ -48,12 +49,11 @@ class NotThisMethod(Exception):
     pass
 
 
-LONG_VERSION_PY = {}
-HANDLERS = {}
+HANDLERS = {}  # type: Dict[str, Dict[str, Callable]]
 
 
-def register_vcs_handler(vcs, method):  # decorator
-    def decorate(f):
+def register_vcs_handler(vcs: str, method: str) -> Callable:  # decorator
+    def decorate(f: Callable) -> Callable:
         if vcs not in HANDLERS:
             HANDLERS[vcs] = {}
         HANDLERS[vcs][method] = f


Test patch:
Repository: pandas-dev/pandas

Issue:
Fix performance regression with Series operations
#25743 caused a performance regression with Series operations (mean, sum, etc).  This can be avoided by moving some of the `if`s around so that the additional check is not performed for those operations.


Discussion:

Resolving patch:
diff --git a/pandas/core/groupby/grouper.py b/pandas/core/groupby/grouper.py
--- a/pandas/core/groupby/grouper.py
+++ b/pandas/core/groupby/grouper.py
@@ -522,21 +522,17 @@ def _get_grouper(obj, key=None, axis=0, level=None, sort=True,
     any_arraylike = any(isinstance(g, (list, tuple, Series, Index, np.ndarray))
                         for g in keys)
 
-    try:
+    # is this an index replacement?
+    if (not any_callable and not any_arraylike and not any_groupers and
+            match_axis_length and level is None):
         if isinstance(obj, DataFrame):
-            all_in_columns_index = all(g in obj.columns or g in obj.index.names
-                                       for g in keys)
+            all_in_columns_index = all(g in obj.columns or g in
+                                       obj.index.names for g in keys)
         elif isinstance(obj, Series):
             all_in_columns_index = all(g in obj.index.names for g in keys)
-        else:
-            all_in_columns_index = False
-    except Exception:
-        all_in_columns_index = False
 
-    if (not any_callable and not all_in_columns_index and
-            not any_arraylike and not any_groupers and
-            match_axis_length and level is None):
-        keys = [com.asarray_tuplesafe(keys)]
+        if not all_in_columns_index:
+            keys = [com.asarray_tuplesafe(keys)]
 
     if isinstance(level, (tuple, list)):
         if key is None:


Test patch:
Repository: pandas-dev/pandas

Issue:
Fix Type Annotation in pandas.compat.pickle_compat
Part of #25882 

Current errors are as follows:

>mypy pandas/compat/pickle_compat.py

    pandas\compat\pickle_compat.py:141: error: Name 'pkl._Unpickler' is not defined


Discussion:
PR coming as soon as figure out how to fix this.
Looks like a bug with mypy because `_Unpickler`is certainly a class in module `pickle`.
Similar to https://github.com/python/mypy/issues/6383
Only solution I can find is:

    class Unpickler(pkl._Unpickler):  # type: ignore

@WillAyd what should I do?

Resolving patch:
diff --git a/pandas/compat/pickle_compat.py b/pandas/compat/pickle_compat.py
--- a/pandas/compat/pickle_compat.py
+++ b/pandas/compat/pickle_compat.py
@@ -138,7 +138,7 @@ def load_reduce(self):
 # our Unpickler sub-class to override methods and some dispatcher
 # functions for compat
 
-class Unpickler(pkl._Unpickler):
+class Unpickler(pkl._Unpickler):  # type: ignore
 
     def find_class(self, module, name):
         # override superclass


Test patch:
Repository: pandas-dev/pandas

Issue:
Unable to open an S3 object with # in the URL
```python
import pandas as pd
df = pd.read_csv('s3://bucket/key#1.csv')
df = pd.read_csv('s3://bucket/key%231.csv')

```
#### Problem description

Pandas can't open an object from S3 if it has a # sign in the URL, both in the case where the URL path is percent encoded and not.  The reason is that urllib.parse.urlparse(), which is used in io/s3.py to parse the URL, treats the # sign as the beginning of the URL fragment, and thus it is lost (in the case of not percent encoded).

I see two possible solutions to the problem, but I'm not sure which one is best, since there does not seem to be a 'specification' for the S3 URL scheme (at least that I can find):
1. Use `allow_fragments=False` when calling `urllib.parse.urlparse()`.  This would allow the non-percent-encoded case to work, but seems slightly wrong.
2. Call `urllib.parse.unquote()` on S3 paths before passing to s3fs.  s3fs *seems* to want just a bucket/key as input, so pandas would have to remove the percent encoding.  This would allow the percent-encoded case to work.  It seems a bit more correct, but it might change some existing behavior where users could be loading URLs with % characters in them.


Discussion:
Not certain myself - @TomAugspurger may have thoughts but otherwise if you have a proposal in mind feel free to submit a PR
Based on the analysis by @swt2c , I think the option#1 is the way to go (feel free to correct me if I am wrong).
I can start working on this issue & submit a PR with the changed code soon if that's ok.

Resolving patch:
diff --git a/pandas/io/s3.py b/pandas/io/s3.py
--- a/pandas/io/s3.py
+++ b/pandas/io/s3.py
@@ -10,7 +10,7 @@
 
 def _strip_schema(url):
     """Returns the url without the s3:// part"""
-    result = parse_url(url)
+    result = parse_url(url, allow_fragments=False)
     return result.netloc + result.path
 
 


Test patch:
diff --git a/pandas/tests/io/conftest.py b/pandas/tests/io/conftest.py
--- a/pandas/tests/io/conftest.py
+++ b/pandas/tests/io/conftest.py
@@ -59,6 +59,7 @@ def s3_resource(tips_file, jsonl_file):
         moto = pytest.importorskip('moto')
 
         test_s3_files = [
+            ('tips#1.csv', tips_file),
             ('tips.csv', tips_file),
             ('tips.csv.gz', tips_file + '.gz'),
             ('tips.csv.bz2', tips_file + '.bz2'),
diff --git a/pandas/tests/io/parser/test_network.py b/pandas/tests/io/parser/test_network.py
--- a/pandas/tests/io/parser/test_network.py
+++ b/pandas/tests/io/parser/test_network.py
@@ -198,3 +198,8 @@ def test_read_csv_chunked_download(self, s3_resource, caplog):
             read_csv("s3://pandas-test/large-file.csv", nrows=5)
             # log of fetch_range (start, stop)
             assert ((0, 5505024) in {x.args[-2:] for x in caplog.records})
+
+    def test_read_s3_with_hash_in_key(self, tips_df):
+        # GH 25945
+        result = read_csv('s3://pandas-test/tips#1.csv')
+        tm.assert_frame_equal(tips_df, result)
Repository: pandas-dev/pandas

Issue:
DOC: update docstrings with new Index repr
After #9901 has been merged, we should go through the docstrings and update where there are examples with Index reprs.

The ones that changed the most are the datetime-like. A search for `<class 'pandas.tseries.period.PeriodIndex'>` and alike can help there.



Discussion:

Resolving patch:
diff --git a/pandas/core/arrays/period.py b/pandas/core/arrays/period.py
--- a/pandas/core/arrays/period.py
+++ b/pandas/core/arrays/period.py
@@ -439,19 +439,16 @@ def asfreq(self, freq=None, how='E'):
         --------
         >>> pidx = pd.period_range('2010-01-01', '2015-01-01', freq='A')
         >>> pidx
-        <class 'pandas.core.indexes.period.PeriodIndex'>
-        [2010, ..., 2015]
-        Length: 6, Freq: A-DEC
+        PeriodIndex(['2010', '2011', '2012', '2013', '2014', '2015'],
+        dtype='period[A-DEC]', freq='A-DEC')
 
         >>> pidx.asfreq('M')
-        <class 'pandas.core.indexes.period.PeriodIndex'>
-        [2010-12, ..., 2015-12]
-        Length: 6, Freq: M
+        PeriodIndex(['2010-12', '2011-12', '2012-12', '2013-12', '2014-12',
+        '2015-12'], dtype='period[M]', freq='M')
 
         >>> pidx.asfreq('M', how='S')
-        <class 'pandas.core.indexes.period.PeriodIndex'>
-        [2010-01, ..., 2015-01]
-        Length: 6, Freq: M
+        PeriodIndex(['2010-01', '2011-01', '2012-01', '2013-01', '2014-01',
+        '2015-01'], dtype='period[M]', freq='M')
         """
         how = libperiod._validate_end_alias(how)
 


Test patch:
Repository: pandas-dev/pandas

Issue:
Fix Type Annotation in pandas/core/accessor.py
Part of #25882 

Current errors are:

    pandas\core\accessor.py:14: error: Need type annotation for '_accessors'

PR coming soon


Discussion:

Resolving patch:
diff --git a/pandas/core/accessor.py b/pandas/core/accessor.py
--- a/pandas/core/accessor.py
+++ b/pandas/core/accessor.py
@@ -5,13 +5,14 @@
 that can be mixed into or pinned onto other pandas classes.
 
 """
+from typing import Set
 import warnings
 
 from pandas.util._decorators import Appender
 
 
 class DirNamesMixin:
-    _accessors = frozenset()
+    _accessors = set()  # type: Set[str]
     _deprecations = frozenset(
         ['asobject', 'base', 'data', 'flags', 'itemsize', 'strides'])
 


Test patch:
Repository: pandas-dev/pandas

Issue:
Doc for HDFStore compression unclear on what the default value of None does
The doc for the `HDFStore` class mentions:

```    
complevel : int, 0-9, default None
            Specifies a compression level for data.
            A value of 0 disables compression.
```

That doesn't actually answer the question of what compression level is used when the default (None) is used, though. Is None translated further down to 0? it turns out yes, but you have to dig in the code to actually figure that out. And it could as well have been translated eventually to any other value.

Two options:
1. Actually change the default in the `complevel` argument to be "0". (It's an immutable object, so it's fine as a default value for a function argument.)
2. Just adjust the doc in some way.

When the right solution is decided, I can do a pull request with it. Thanks!


Discussion:
I think either option would be fine
It seems like `complevel=None` is translated into `0`. I will add this to the docstring.
https://github.com/pandas-dev/pandas/blob/cb00deb94500205fcb27a33cc1d0df79a9727f8b/pandas/io/pytables.py#L484

Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -422,7 +422,7 @@ class HDFStore(StringMixin):
             It is similar to ``'a'``, but the file must already exist.
     complevel : int, 0-9, default None
             Specifies a compression level for data.
-            A value of 0 disables compression.
+            A value of 0 or None disables compression.
     complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib'
             Specifies the compression library to be used.
             As of v0.20.2 these additional compressors for Blosc are supported


Test patch:
Repository: pandas-dev/pandas

Issue:
Doc Check Failures
This just started showing up in the CI failures today:

```sh
Traceback (most recent call last):
  File "ci/../scripts/validate_docstrings.py", line 991, in <module>
    args.ignore_deprecated))
  File "ci/../scripts/validate_docstrings.py", line 891, in main
    result = validate_all(prefix, ignore_deprecated)
  File "ci/../scripts/validate_docstrings.py", line 845, in validate_all
    doc_info = validate_one(func_name)
  File "ci/../scripts/validate_docstrings.py", line 801, in validate_one
    errs, wrns, examples_errs = get_validation_data(doc)
  File "ci/../scripts/validate_docstrings.py", line 749, in get_validation_data
    if not doc.see_also:
  File "ci/../scripts/validate_docstrings.py", line 477, in see_also
    in self.doc['See Also'])
  File "ci/../scripts/validate_docstrings.py", line 476, in <genexpr>
    for name, desc, _
ValueError: not enough values to unpack (expected 3, got 2)
```

I think it's an issue with upgrading to numpydoc 0.9 but will look in more detail and confirm


Discussion:
Here's the relevant enhancement on the numpydoc side released in v0.9:

https://github.com/numpy/numpydoc/pull/172


Resolving patch:
diff --git a/scripts/validate_docstrings.py b/scripts/validate_docstrings.py
--- a/scripts/validate_docstrings.py
+++ b/scripts/validate_docstrings.py
@@ -472,9 +472,12 @@ def parameter_desc(self, param):
 
     @property
     def see_also(self):
-        return collections.OrderedDict((name, ''.join(desc))
-                                       for name, desc, _
-                                       in self.doc['See Also'])
+        result = collections.OrderedDict()
+        for funcs, desc in self.doc['See Also']:
+            for func, _ in funcs:
+                result[func] = ''.join(desc)
+
+        return result
 
     @property
     def examples(self):
@@ -731,7 +734,7 @@ def get_validation_data(doc):
             if doc.method_returns_something:
                 errs.append(error('RT01'))
         else:
-            if len(doc.returns) == 1 and doc.returns[0][1]:
+            if len(doc.returns) == 1 and doc.returns[0].name:
                 errs.append(error('RT02'))
             for name_or_type, type_, desc in doc.returns:
                 if not desc:


Test patch:
Repository: pandas-dev/pandas

Issue:
Fix Type Annotation in  pandas.compat
Part of #25882

Type annotations for the following file need to be corrected to remove **pandas.compat** from the mypy blacklist. 

Mypy throws the following error:
```sh
pandas/compat/__init__.py:81: error: Module has no attribute "re"
```

PR to follow.



Discussion:

Resolving patch:
diff --git a/pandas/compat/__init__.py b/pandas/compat/__init__.py
--- a/pandas/compat/__init__.py
+++ b/pandas/compat/__init__.py
@@ -11,9 +11,9 @@
 * platform checker
 """
 import platform
-import re
 import struct
 import sys
+from typing import Pattern
 
 PY36 = sys.version_info >= (3, 6)
 PY37 = sys.version_info >= (3, 7)
@@ -74,13 +74,7 @@ def raise_with_traceback(exc, traceback=Ellipsis):
     raise exc.with_traceback(traceback)
 
 
-# In Python 3.7, the private re._pattern_type is removed.
-# Python 3.5+ have typing.re.Pattern
-if PY36:
-    import typing
-    re_type = typing.re.Pattern
-else:
-    re_type = type(re.compile(''))
+re_type = Pattern
 
 
 # https://github.com/pandas-dev/pandas/pull/9123


Test patch:
Repository: pandas-dev/pandas

Issue:
Fix type annotations in pandas.core.computation
Part of  #25882

The files in pandas.core.computation have the following errors:
```
pandas/core/computation/ops.py:169: error: Read-only property cannot override read-write property
pandas/core/computation/pytables.py:59: error: Read-only property cannot override read-write property
pandas/core/computation/pytables.py:384: error: Incompatible types in assignment (expression has type "Type[pandas.core.computation.pytables.Constant]", base class "BaseExprVisitor" defined the type as "Type[pandas.core.computation.ops.Constant]")
```

Pull request to follow.


Discussion:

Resolving patch:
diff --git a/pandas/core/computation/expr.py b/pandas/core/computation/expr.py
--- a/pandas/core/computation/expr.py
+++ b/pandas/core/computation/expr.py
@@ -7,6 +7,7 @@
 import itertools as it
 import operator
 import tokenize
+from typing import Type
 
 import numpy as np
 
@@ -327,7 +328,7 @@ class BaseExprVisitor(ast.NodeVisitor):
     parser : str
     preparser : callable
     """
-    const_type = Constant
+    const_type = Constant  # type: Type[Term]
     term_type = Term
 
     binary_ops = _cmp_ops_syms + _bool_ops_syms + _arith_ops_syms
diff --git a/pandas/core/computation/ops.py b/pandas/core/computation/ops.py
--- a/pandas/core/computation/ops.py
+++ b/pandas/core/computation/ops.py
@@ -149,10 +149,6 @@ def value(self, new_value):
     def name(self):
         return self._name
 
-    @name.setter
-    def name(self, new_name):
-        self._name = new_name
-
     @property
     def ndim(self):
         return self._value.ndim
diff --git a/pandas/core/computation/pytables.py b/pandas/core/computation/pytables.py
--- a/pandas/core/computation/pytables.py
+++ b/pandas/core/computation/pytables.py
@@ -56,7 +56,8 @@ def _resolve_name(self):
         except UndefinedVariableError:
             return self.name
 
-    @property
+    # read-only property overwriting read/write property
+    @property  # type: ignore
     def value(self):
         return self._value
 


Test patch: