BrandonFors/pandas-github-issues huggingface dataset:
API: remove deep keyword from EA.copy
State: closed

Not clear on what type of deprecation cycle we want here

- [x] closes #26995, closes #27024

Labels:
name: API Design
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
cc @TomAugspurger 

Looks good to me. As usual (sorry ...), I would only add `view()` if we actually need it (meaning: use it internally)
> Slight preference for deprecation, but don't have strong thoughts here.

So a user only runs into this when they directly interface with an ExtensionArray themselves (as the uses within pandas get fixed). Further, for EA authors only implementing the EA itself, it's not a backwards incompatible change, as we don't have a problem that the EA still has the keyword, it will just never be called from inside pandas.

It seems that users doing `Series.array.copy(deep=True)` is something very specific, and also only introduced very recently (eg Categorical is longest out there, but has actually no copy keyword). IntegerArray and IntervalArray eg does have one.
Sounds good.

On Thu, Jun 27, 2019 at 3:04 PM Joris Van den Bossche <
notifications@github.com> wrote:

> Slight preference for deprecation, but don't have strong thoughts here.
>
> So a user only runs into this when they directly interface with an
> ExtensionArray themselves (as the uses within pandas get fixed). Further,
> for EA authors only implementing the EA itself, it's not a backwards
> incompatible change, as we don't have a problem that the EA still has the
> keyword, it will just never be called from inside pandas.
>
> It seems that users doing Series.array.copy(deep=True) is something very
> specific, and also only introduced very recently (eg Categorical is longest
> out there, but has actually no copy keyword). IntegerArray and
> IntervalArray eg does have one.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27083?email_source=notifications&email_token=AAKAOIXCWW7BQBAYLMGNM3TP4UMMZA5CNFSM4H37CG42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYYG4YY#issuecomment-506490467>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVQTC74QUOSYMYYFRTP4UMMZANCNFSM4H37CG4Q>
> .
>

lgtm. ping on green.
Thanks!
Removed Panel Kludge from Pickle/Msgpack tests
State: closed

@jreback @jbrockmendel 

Recreated pickle file for 0.20.3 using Python 3.5.2 on Mac. Created msgpack file anew (previously 0.19 was last version tested)

Lots of follow ups can be done here including:

  - Automatically generating one of these files as part of build process
  - Decoupling script from test requirements (fixturization and break off)

Can also add more platforms / versions here as required but this should be a good starting point

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @WillAyd! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-27 20:16:22 UTC
need a note for mspack as well.
thanks @WillAyd
CI: Disable codecov comments and re-enable results in the checks
State: closed

- [x] closes #26896

Depending on how flaky the coverage calculations are, we could consider adding `threshold` to the yaml to specify how far the coverage can drop while posting success.

https://docs.codecov.io/docs/commit-status


Labels:
name: CI
description: Continuous Integration

Comments:
thanks @mroeschke yeah we should figure out what the right coverage report we want.
Add pytest-azurepipelines
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
It seems this plugin does not need any configuration
Looks like the package is not built for Python 3.7, so we'll have to wait or implement this manually. May be we can report it upstream.

What would be nice is to make a test fail in this PR, so we can see if the package is generating the expected result.

Also, if you can please reference the issue in the PR description, that would be great.
duplicated by https://github.com/pandas-dev/pandas/pull/26620
xfail test_missing_required_dependency test
State: closed

See https://github.com/MacPython/pandas-wheels/pull/50. I can't get this test to work reliably.

Labels:
name: CI
description: Continuous Integration

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=h1) Report
> Merging [#26993](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/171615a35574a1fc9c4e8260edca7d1e08e9c302?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26993/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26993      +/-   ##
==========================================
- Coverage   91.99%   91.98%   -0.01%     
==========================================
  Files         180      180              
  Lines       50774    50774              
==========================================
- Hits        46711    46707       -4     
- Misses       4063     4067       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.63% <ø> (ø)` | :arrow_up: |
| #single | `41.85% <ø> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26993/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26993/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=footer). Last update [171615a...2961aa2](https://codecov.io/gh/pandas-dev/pandas/pull/26993?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

cool, thanks @TomAugspurger 
AFAICT this is resolved by using `sys.executable` instead of hard-coding "python"
DO NOT MERGE: Timing test files
State: closed

Checking how long every test file takes to run in the CI. In #26949 I'd like to run tests constraining that a whole file must be run in a single worker (so we can run `single` tests in parallel).

This means that files that will take a lot of time won't be parallelized well, and should be splitted for this to work. So, I want to know how long each takes.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Checking how long it takes every test file to see if some should be splitted for better parallelization I see this, which doesn't look right to me:
```
pandas/tests/io/test_html.py : 21m59.041s
pandas/tests/io/json/test_pandas.py : 8m50.339s
pandas/tests/io/parser/test_common.py : 6m45.860s
pandas/tests/io/excel/test_readers.py : 13m14.864s
```
Didn't check the files individually, I guess in some cases we want to test loading big files, but 22 minutes to test the html I/O sounds very exaggerated.

@pandas-dev/pandas-core is this expected?
Did you try running that single file locally?
No, wanted to check first that this is not expected and I'm missing something. If this doesn't seem right, I'll then try to figure out what's going on.
The second one takes locally 4.6 seconds (and not almost 9 min). There are 2 out of 97 tests skipped (encoding related), of which one is an s3 test
> pandas/tests/io/test_html.py

I'm seeing the two (not marked as slow) test_invalid_url tests as about 150s apiece.
@datapythonista closable?
Yep, let's close this for now, may need to reopen later to research on the timing problem, but I don't have time now.
WIP/CI/TST: Clean up of the tests script
State: closed

Follow up of the conversation in #26932, and fixing some of the things that #24106 tried.

The main change is **call pytest once** instead of once for `single` and once for `multi`. This shouldn't be a problem since I add the parameter `--dist=loadfile` which should prevent tests in the same file running in parallel. I don't think running `serial` tests in parallel with tests in other files should be a problem.

Other things cleaned in the PR:
- Setting always the proxy (shouldn't hurt for not network tests, and makes things simpler)
- Using as many processes as cores, not 2
- Removed hack for when no tests were found (since we don't call only for `single` tests, there will always be tests to run)

So far testing, let's see how does this work in the CI.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Clean

Comments:
Using `dist=loadfile` seems fine, assuming things are reasonable evenly spread across workers. I imagine that some of our files take much longer, so hopefully pytest-xdist is smart about scheduling.
I guess we may need to split some big test files, but I think Jeff was requesting that for some already, and it's probably good anyway. I remember the tests taking longer the last time I tried this.
Looks like couple of builds are indeed significantly longer here than when not using `--dist=loadfile`. I think should be easy to get the list of test files, with the amount time that each takes, and see if it makes sense to split something.

Then, I see that some tests are failing because the pandas module has more attributes than expected, `'hard_dependencies', 'dependency', 'missing_dependencies'`. Not sure how this could be related to the changes here. Did anyone faced this before?
Other PRs are hitting that. I think there’s a test order issue. We should skip that test for now. 

> On Jun 19, 2019, at 17:14, Marc Garcia <notifications@github.com> wrote:
> 
> Looks like couple of builds are indeed significantly longer here than when not using --dist=loadfile. I think should be easy to get the list of test files, with the amount time that each takes, and see if it makes sense to split something.
> 
> Then, I see that some tests are failing because the pandas module has more attributes than expected, 'hard_dependencies', 'dependency', 'missing_dependencies'. Not sure how this could be related to the changes here. Did anyone faced this before?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.

needs rebase
@datapythonista needs rebase
CI is hopefully fixed if you can merge master.
Thanks, but even if the CI is green we can't merge this, since some builds take much longer with the change. Need to find the time to figure out how to solve it. Will leave it red for now so this is not merged by mistake.
getting pretty stale, closing.
Docs: Capitalisation of title line changed to lower case
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Duplicate Report
description: Duplicate issue or pull request

Comments:
There is already somebody working on this: https://github.com/pandas-dev/pandas/pull/26933 
So best to first check how that overlaps (don't have time to check now, just quickly wanted to note that)

Closing in favor of #26950.

@HolgerSpernau : Thanks for your interest in contributing!  Although this time you stumbled upon on issue that was already being worked on, there are plenty of other open issues where your help would be welcomed and appreciated!
@gfyoung : Thanks. I think the work I have done can still be used. I understood in #26933 that it is not only related to the first word in the header, but also the second or third words. That's what I concentrated on. Have a look again on the changed. Thanks.
@HolgerSpernau : That PR is for all words.  In addition, we're going with as much title case (in general) as possible, whereas your PR goes for lowercase.  Thanks for checking though!
DOC: Disallowing search engines in the development docs
State: closed

- [X] closes #26924
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=h1) Report
> Merging [#26931](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d39a6de66803b2a13d6bbb8aa98c56e90f95526c?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26931/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26931      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50716    50716              
==========================================
- Hits        46593    46589       -4     
- Misses       4123     4127       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.45% <ø> (ø)` | :arrow_up: |
| #single | `41.1% <ø> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26931/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26931/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=footer). Last update [d39a6de...dbd3764](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

sure. note sure how we can tell if this works?
The file has been created successfully, that worked: https://dev.pandas.io/robots.txt

I think Google has tools where they give you information on how they crawl your site, but not much idea how that works, and not really looking forward to start working in SEO now. :)

I think we can simply wait some days until Google updates its data, and then we shouldn't see all the results here: https://www.google.com/search?q=site%3Adev.pandas.io
debug CI
State: closed

Still debugging https://github.com/pandas-dev/pandas/pull/26816. Sorry for the noise.

Labels:

Comments:
I'm pretty confused by the test failures: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=12996

```
______________________________ TestPDApi.test_api ______________________________
[gw1] darwin -- Python 3.5.6 /Users/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.api.test_api.TestPDApi object at 0x13ed7a1d0>

    def test_api(self):
    
        self.check(pd,
                   self.lib + self.misc +
                   self.modules + self.deprecated_modules +
                   self.classes + self.deprecated_classes +
                   self.deprecated_classes_in_future +
                   self.funcs + self.funcs_option +
                   self.funcs_read + self.funcs_to +
                   self.deprecated_funcs_in_future +
                   self.deprecated_funcs + self.private_modules,
>                  self.ignored)

pandas/tests/api/test_api.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/api/test_api.py:18: in check
    tm.assert_almost_equal(result, expected)
pandas/util/testing.py:325: in assert_almost_equal
    **kwargs)
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise_assert_detail(obj, '{0} length are different'.format(obj),
E   AssertionError: Iterable are different
E   
E   Iterable length are different
E   [left]:  132
E   [right]: 129
E   [diff]: ['missing_dependencies', 'dependency', 'hard_dependencies']
```

I don't see how these changes could have caused that, which makes me suspect a flaky test? I know at least one other contributor reported some issues with this test.
Resolved by https://github.com/pandas-dev/pandas/pull/26953
Add typing annotation to IntervalIndex.intersection
State: closed

Follow up https://github.com/pandas-dev/pandas/pull/26225#pullrequestreview-242044547

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=h1) Report
> Merging [#26870](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d432f654efa3c9160309bbe17cb44ce863ebc328?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26870/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26870      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50716    50716              
==========================================
- Hits        46594    46590       -4     
- Misses       4122     4126       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.45% <100%> (ø)` | :arrow_up: |
| #single | `41.1% <100%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/26870/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `96.44% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26870/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26870/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=footer). Last update [d432f65...9f485c3](https://codecov.io/gh/pandas-dev/pandas/pull/26870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

thanks @makbigc
Change Scope of xlrd Import Attempt
State: closed

Moved attempt to import xlrd to global scope of module. Simplifies code a bit and keeps xlrd relevant imports within one block

Labels:
name: IO Excel
description: read_excel, to_excel
name: Clean

Comments:
Realized this causes some conflict with top level imports of ExcelFile, ExcelWriter, read_excel, etc... into the API regardless of whether or not xlrd is installed. I don't think worth the effort to change at this time
TST: troubleshot numexpr test
State: closed

seemingly unrelated test failure to changes in PR #26662

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=12414

```
_______ TestAlignment.test_basic_series_frame_alignment[numexpr-python] ________
[gw0] linux -- Python 3.6.8 /home/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.computation.test_eval.TestAlignment object at 0x7f5f831aa5c0>
engine = 'numexpr', parser = 'python'

    def test_basic_series_frame_alignment(self, engine, parser):
        def testit(r_idx_type, c_idx_type, index_name):
            df = mkdf(10, 7, data_gen_f=f, r_idx_type=r_idx_type,
                      c_idx_type=c_idx_type)
            index = getattr(df, index_name)
            s = Series(np.random.randn(5), index[:5])
            if should_warn(s.index, df.index):
                with tm.assert_produces_warning(RuntimeWarning):
                    res = pd.eval('s + df', engine=engine, parser=parser)
            else:
                res = pd.eval('s + df', engine=engine, parser=parser)
    
            if r_idx_type == 'dt' or c_idx_type == 'dt':
                expected = df.add(s) if engine == 'numexpr' else s + df
            else:
                expected = s + df
            assert_frame_equal(res, expected)
    
        # only test dt with dt, otherwise weird joins result
        args = product(['i', 'u', 's'], ['i', 'u', 's'], ('index', 'columns'))
        with warnings.catch_warnings(record=True):
            # avoid warning about comparing strings and ints
            warnings.simplefilter("ignore", RuntimeWarning)
    
            for r_idx_type, c_idx_type, index_name in args:
>               testit(r_idx_type, c_idx_type, index_name)

pandas/tests/computation/test_eval.py:934: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/computation/test_eval.py:925: in testit
    assert_frame_equal(res, expected)
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise_assert_detail(obj, msg, lobj, robj)
E   AssertionError: DataFrame.columns are different
E   
E   DataFrame.columns values are different (100.0 %)
E   [left]:  Index([           0,            1,            2,            3,            4,
E          '0בקבנ899זק', '28ףן4כנךטש', '63צגהק5שץן', '7סף4נלדקסב', 'אגןנםצככד0',
E          'וכ2ףזאמאןא', 'ל8לצאןחצךר'],
E         dtype='object')
E   [right]: Index(['0בקבנ899זק', '28ףן4כנךטש', '63צגהק5שץן', '7סף4נלדקסב', 'אגןנםצככד0',
E          'וכ2ףזאמאןא', 'ל8לצאןחצךר',            0,            1,            2,
E                     3,            4],
E         dtype='object')

pandas/_libs/testing.pyx:178: AssertionError
```

Labels:

Comments:
not reproducing failure. will merge master to check.
closing as not reproducing failure.
Fix broken np_dev CI
State: closed

- [X] closes #26725
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
are the doc / slow failures covered by this?
No those are separate
> are the doc / slow failures covered by this?

no. docs #26723, slow #26714, and timeouts #26716 
k merge away
thanks @WillAyd