Pick and choose numpy version based on Python 2 or 3.
State: closed

- [x] closes #27435
- [ ] tests added / passed
- [ ] passes `black pandas` (not applicable, `black setup.py` changes too many quotes, polluting the diff).
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @jorisvandenbossche

Labels:
name: Build
description: Library building on various platforms
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks for the PR!

Hmm, there a couple of failures. Not directly related to your change, but related to the fact that we didn't run those tests (at the state of 0.24.2) for some time, and fixes have been made for changed dependencies. 
See eg https://github.com/pandas-dev/pandas/issues/26725, https://github.com/pandas-dev/pandas/pull/26726

There also seem to be some actual failures with numpy 1.17, so an option could be to restrict it to lower versions also for python 3 ?
Thanks for the quick response.

Do you have documented versions of what the maximum versions pandas should have on Python 3?
I haven't looked at the failures, but are they actual, noticeable bugs, or
just failures in the test?

If they're real issues then we can just have pandas 0.24.x require NumPy
<1.17 across the board.

On Thu, Sep 19, 2019 at 12:44 PM Didip Kerabat <notifications@github.com>
wrote:

> Thanks for the quick response.
>
> Do you have documented versions of what the maximum versions pandas should
> have on Python 3?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28511?email_source=notifications&email_token=AAKAOITOQJNDPWJTJBLZRO3QKO26PA5CNFSM4IYDXLM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7EIVHI#issuecomment-533236381>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIUASZO5KC2EJDTIHATQKO26PANCNFSM4IYDXLMQ>
> .
>

Part of them are certainly test failures, but it seems there are also actual bugs (although someone needs to investigate it more in detail to be sure):

<details>

```
______________________ TestRangeIndex.test_numpy_argsort _______________________
[gw1] linux -- Python 3.7.4 /home/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.indexes.test_range.TestRangeIndex object at 0x7ff094eacd50>

    def test_numpy_argsort(self):
        for k, ind in self.indices.items():
>           result = np.argsort(ind)

pandas/tests/indexes/common.py:320: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
<__array_function__ internals>:6: in argsort
    ???
../../../miniconda3/envs/pandas-dev/lib/python3.7/site-packages/numpy/core/fromnumeric.py:1089: in argsort
    return _wrapfunc(a, 'argsort', axis=axis, kind=kind, order=order)
../../../miniconda3/envs/pandas-dev/lib/python3.7/site-packages/numpy/core/fromnumeric.py:61: in _wrapfunc
    return bound(*args, **kwds)
pandas/core/indexes/range.py:323: in argsort
    nv.validate_argsort(args, kwargs)
pandas/compat/numpy/function.py:56: in __call__
    self.defaults)
pandas/util/_validators.py:218: in validate_args_and_kwargs
    validate_kwargs(fname, kwargs, compat_args)
pandas/util/_validators.py:157: in validate_kwargs
    _check_for_default_values(fname, kwds, compat_args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fname = 'argsort', arg_val_dict = {'axis': -1, 'kind': None, 'order': None}
compat_args = OrderedDict([('axis', -1), ('kind', 'quicksort'), ('order', None)])

    def _check_for_default_values(fname, arg_val_dict, compat_args):
        """
        Check that the keys in `arg_val_dict` are mapped to their
        default values as specified in `compat_args`.
    
        Note that this function is to be called only when it has been
        checked that arg_val_dict.keys() is a subset of compat_args
    
        """
        for key in arg_val_dict:
            # try checking equality directly with '=' operator,
            # as comparison may have been overridden for the left
            # hand object
            try:
                v1 = arg_val_dict[key]
                v2 = compat_args[key]
    
                # check for None-ness otherwise we could end up
                # comparing a numpy array vs None
                if (v1 is not None and v2 is None) or \
                   (v1 is None and v2 is not None):
                    match = False
                else:
                    match = (v1 == v2)
    
                if not is_bool(match):
                    raise ValueError("'match' is not a boolean")
    
            # could not compare them directly, so try comparison
            # using the 'is' operator
            except ValueError:
                match = (arg_val_dict[key] is compat_args[key])
    
            if not match:
                raise ValueError(("the '{arg}' parameter is not "
                                  "supported in the pandas "
                                  "implementation of {fname}()".
>                                 format(fname=fname, arg=key)))
E               ValueError: the 'kind' parameter is not supported in the pandas implementation of argsort()
```

</details>

But it might be more a corner case anyway.
Actually, I was looking at the numpy dev build. Most of the other build failures seem to be problems getting the environment installed (missing packages etc), like the python 2.7 build on Azure:

```
Solving environment: ...working... failed

ResolvePackageNotFound: 
  - xlsxwriter=0.5.2
  - pytz=2013b
  - python-dateutil=2.5.0
  - xlwt=0.7.5
```

Note sure how much effort we should do to get those CIs running again ..
Yeah, numpydev can certainly be ignored.

For the rest, perhaps try adding back the `free` channel in each
environment that needs it?

On Thu, Sep 19, 2019 at 1:54 PM Joris Van den Bossche <
notifications@github.com> wrote:

> Actually, I was looking at the numpy dev build. Most of the other build
> failures seem to be problems getting the environment installed (missing
> packages etc), like the python 2.7 build on Azure:
>
> Solving environment: ...working... failed
>
> ResolvePackageNotFound:
>   - xlsxwriter=0.5.2
>   - pytz=2013b
>   - python-dateutil=2.5.0
>   - xlwt=0.7.5
>
> Note sure how much effort we should do to get those CIs running again ..
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28511?email_source=notifications&email_token=AAKAOIRXC2E4C3BPFKJZM5TQKPDFZA5CNFSM4IYDXLM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7EPEMA#issuecomment-533262896>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIV27MQSLBV63ZUB6JLQKPDFZANCNFSM4IYDXLMQ>
> .
>

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=h1) Report
> Merging [#28511](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=desc) into [0.24.x](https://codecov.io/gh/pandas-dev/pandas/commit/1ec04725e41ec072d899343c17bdbe38ce53832a?src=pr&el=desc) will **decrease** coverage by `49.48%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/28511/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           0.24.x   #28511       +/-   ##
===========================================
- Coverage   92.39%   42.91%   -49.49%     
===========================================
  Files         166      166               
  Lines       52430    52430               
===========================================
- Hits        48443    22499    -25944     
- Misses       3987    29931    +25944
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.91% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `35.59% <ø> (-61.28%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [125 more](https://codecov.io/gh/pandas-dev/pandas/pull/28511/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=footer). Last update [1ec0472...278f7dc](https://codecov.io/gh/pandas-dev/pandas/pull/28511?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Also, if you can please merge master into this branch.
I think this is targeting 0.24.x directly, since we don't want these changes on master.
Agree with @TomAugspurger
We still need the linting issues fixed though.

On Mon, Oct 7, 2019 at 11:41 AM Didip Kerabat <notifications@github.com>
wrote:

> Agree with @TomAugspurger <https://github.com/TomAugspurger>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28511?email_source=notifications&email_token=AAKAOIWZ32LSX4LVKAQ5IELQNNRCBA5CNFSM4IYDXLM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEARAMAA#issuecomment-539100672>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQRLK3TQUGZNE7KR2TQNNRCBANCNFSM4IYDXLMQ>
> .
>

Yes, I’ll try to find sometime today to address it.
Where is the CI/CD output that shows failed linting?

This one? https://github.com/pandas-dev/pandas/pull/28511/checks?check_run_id=229464786
yes, but the build is too old and is not available anymore. You can run `./ci/code_checks.sh` and should report where are the tailing spaces.
There are tons of lint problems when running `./ci/code_checks.sh` but I don't see one that pertains to `setup.py`.

Not quite sure what's the problem with the docstrings, but you've also got this error in the CI `pytables needs a release beyong 3.4.4 to support numpy 1.16x`.

Do you mind having a look @didip. We can't merge this until the CI is green.
At this point we don't have any further plans for a 24.x release, so unfortunately I don't think this is something we will address (if anyone disagrees feel free to reopen)
As far as I recall, we said to give the opportunity to interested contributors, so let's wait a bit more to see if @didip has time to update this.
@didip it looks like some xfailed pytables tests need to be updated for the changes in this PR to work.  can you take a look and update?
Upgraded `pytables` to 3.5.2. `sh test_fast.sh` now actually runs end-to-end with 20 failures.
`sh test.sh` ran until the end with 45 failures.

Anything else I need to do? Is this good enough to merge?
> sh test.sh ran until the end with 45 failures.  Anything else I need to do? Is this good enough to merge?

You'll need to get the tests passing.  This can include xfailing if appropriate.
@jbrockmendel They look unrelated though. Example:

```
XFAIL pandas/tests/test_strings.py::TestStringMethods::test_api_per_method[empty0-rsplit-Index-category]
  reason: Raising too restrictively; solved by GH 23167
```

> They look unrelated though

Looking at azure, some of them seem to relate to numpy version and error messages.

For the ones that are XPASS if this PR actually fixes those tests then you can un-xfail them.
Ran `sh ci/code_checks.sh`. All of its failures are DocTest.

All of them failed because the expectation and results are in the wrong order. Examples:

```
4253 Examples
4254 --------
4255 >>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'],
4256 ...                    'mask': ['red', 'purple'],
4257 ...                    'weapon': ['sai', 'bo staff']})
4258 >>> df.to_csv(index=False)
Expected:
    'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n'
Got:
    'mask,name,weapon\nred,Raphael,sai\npurple,Donatello,bo staff\n'
```

or another one:

```
2665         >>> d2 = {'name': ['Alice', 'Bob'],
2666         ...       'score': [9.5, 8],
2667         ...       'employed': [False, True],
2668         ...       'kids': [0, 0]}
2669         >>> df2 = pd.DataFrame(data=d2)
2670         >>> df2
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,3 @@
    -    name  score  employed  kids
    -0  Alice    9.5     False     0
    -1    Bob    8.0      True     0
    +   employed  kids   name  score
    +0     False     0  Alice    9.5
    +1      True     0    Bob    8.0
```
@didip - thanks for the effort so far - do you still want to work on this?
To be honest, I am running out of capacity to continue this work. I am completely unfamiliar with the build system to properly debug further problems.
closing, but still happy for someone to push a PR to close this issue.