BrandonFors/pandas-github-issues huggingface dataset:
CLN: remove fastparquet compat shim
State: closed

Introduced in #19434 Jan 2018.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm.
I guess fastparquet 0.2.1 still has the offending code. Can you add an issue to their tracker? (this is the failing build)
> Can you add an issue to their tracker?

Looks like Tom beat me to it by about 2 weeks
I'll push a fix to fastparquet for the issue.
I'd recommend skipping for fastparquet <0.3.2

On Mon, Jul 1, 2019 at 9:21 PM jbrockmendel <notifications@github.com>
wrote:

> Can you add an issue to their tracker?
>
> Looks like Tom beat me to it by about 2 weeks
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27170?email_source=notifications&email_token=AAKAOIRVMXTMT5OJXVX7ZF3P5K3TRA5CNFSM4H4WEHJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY73LPQ#issuecomment-507491774>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVOGIORGMWR26QZI4LP5K3TRANCNFSM4H4WEHJA>
> .
>

> I'd recommend skipping for fastparquet <0.3.2
> […](#)

we can’t skip as that just papers over; we would need to bump or not merge this patch
Let's leave this for after 0.25.0 then, or at least until there is a new fastparquet release with the fix. It's a bit unfortunate that fastparquet didn't fix that a year ago.
Waiting may be prudent. I'm going to try to do a FP release today.

On Tue, Jul 2, 2019 at 8:50 AM Joris Van den Bossche <
notifications@github.com> wrote:

> Let's leave this for after 0.25.0 then, or at least until there is a new
> fastparquet release with the fix. It's a bit unfortunate that fastparquet
> didn't fix that a year ago.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27170?email_source=notifications&email_token=AAKAOIQD2WDPZ7NB6REHIATP5NMJXA5CNFSM4H4WEHJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZBKXNQ#issuecomment-507685814>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOISOB3NUWNOYTD4YC5DP5NMJXANCNFSM4H4WEHJA>
> .
>

@TomAugspurger should I stick a pin in this for [?] weeks?
yeah I think we need FP release around for a while before we can do this.
Closing to clear the queue, will revisit a release or two from now
io/msgpack: Raise a more informative exception on file not found
State: closed

- [x] closes #27160
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Okay, requested fixes should be in place now:
 - Fixed CI
 - Removed test and instead amended `TestIOCommon`
 - Added `whatsnew` entry
 - Added comment in `packers.py` indicating bug
@smspillaz looks like there are some merge conflicts - can you clean those up?
Can be closed as #27201 was merged which fixes the same issue
Remove sudo tag in travis
State: closed

Travis are now recommending removing the sudo tag.
Check out [this](https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration).

Labels:
name: CI
description: Continuous Integration

Comments:
lgtm, I guess the CI errors are unrelated, you may want to merge master into your branch.
CI should be fixed, if you can merge master & reps.

On Mon, Jul 1, 2019 at 7:34 AM Marc Garcia <notifications@github.com> wrote:

> lgtm, I guess the CI errors are unrelated, you may want to merge master
> into your branch.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27158?email_source=notifications&email_token=AAKAOITHGNQORAN2VVV2PDTP5H2VDA5CNFSM4H4RBXS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY57JSQ#issuecomment-507245770>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOITRXFLJVGDORZUZJB3P5H2VDANCNFSM4H4RBXSQ>
> .
>

@abhinavsagar can you rebase
this was actually removed to fix the CI here: https://github.com/pandas-dev/pandas/pull/27542
Fix build warnings
State: closed

Cleans up a whole bunch. Remaining offenders are:
- Parsers.pyx and tokenizer. Signed/unsigned hell, made worse by python/cython/c mixing.
- interval.pyx, not touched because large PR in progress in #27100.
- np_datetime.c and np_datetime_strings.c. These are linked into 15+ cython modules as extra object files, but they '#include <datetime.h>`, which expects only standalone modules, who properly init to do it. Kinda painful to fix, though it means each function is compiled and kept in memory 15 times.


Labels:
name: Build
description: Library building on various platforms

Comments:
Green and I think I've addressed all the comments.
> np_datetime.c and np_datetime_strings.c. These are linked into 15+ cython modules as extra object files, but they '#include <datetime.h>`, which expects only standalone modules, who properly init to do it. Kinda painful to fix, though it means each function is compiled and kept in memory 15 times.

One more for my edification.  Is this because they are specified in setup.py, or because of cimports from np_datetime.pyx, or [...]?  How costly is the "compile and kept in memory 15 times"?
> One more for my edification. Is this because they are specified in setup.py, or because of cimports 
> from np_datetime.pyx, or [...]? How costly is the "compile and kept in memory 15 times"?

I came across your comment in the file:
https://github.com/pandas-dev/pandas/blob/02b552d280f917c1a181fcfd956a2d01f0bd5462/pandas/_libs/tslibs/src/datetime/np_datetime.c#L339-L341

The issue is that it's supposed to be an independent cython/cpython module, and imported by other cython modules. Then it could call IMPORT, and a few other warnings would go away.

Instead these files are compiled into object files, and linked into each module that uses them. there's no crash because each module does call IMPORT, but the compiler can't see that.

I don't think compilation is an issue, because:
1. its not a large file
2. cython generates monstrosities, so any true c pales
3. On most dev machine, you have ccache, so (maybe, possibly) the compilation result is cached
4. depending on how  distutils compiles things, it's linked 15 times, but the object is the same dependency for all of them in the Makefile sense, so compiled only once/

as for memory waste it's 2019, so no not an issue.

In summary: more of a smell than a perf/resource issue.



ok thanks
BUG: fix KeyError with list of a single, missing, element
State: closed

- [x] closes #27148
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
thanks @toobaz
CLN: separate raising from non-raising parts of method
State: closed

There is lots of code in core.internals inside try/except blocks, in particular calls to _try_coerce_args.  It is tough to reason about these because it is often unclear what the raising cases are.  This simplifies that problem by separating `values, args = self._try_coerce_args(values, args)` into `values = self._coerce_values(values)` and `args = self._try_coerce_args(args)`  (Note the former doesnt have a "try" in the name because it never raises).

Also: removed a never-used `ndim` kwarg from `Block.make_block` and removed a now-redundant `DatetimeTZBlock.copy` method.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
@jorisvandenbossche CI fails look unrelated, ami missing something?
@mroeschke mentioned those should be fixed by a commit in https://github.com/pandas-dev/pandas/pull/27144#issuecomment-507071945
yep, that fixed it, thanks.
lgtm. ping on green.

would really love a typing PR on blocks :->
psuedo-ping.  CI has been in this state for 13 hours
CI finished, seems like Github missed the notification.

FYI @jbrockmendel let me know if you want to be added to the azure team so you can restart builds. Just need an azure account name.
DEPR: DataFrame.get_dtype_counts
State: closed

- [x] xref #18262
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Unfortunately, `.dtype.value_counts()` isn't a perfect replacement for `get_value_counts()` (IMO it's better), so the test replacements calls the internal representation:

```
In [2]: df = pd.DataFrame([[1, 1.5, pd.Timestamp.now(), True]])

In [3]: df
Out[3]:
   0    1                          2     3
0  1  1.5 2019-06-30 11:55:24.380126  True

In [7]: df.get_dtype_counts().index
Out[7]: Index(['float64', 'int64', 'datetime64[ns]', 'bool'], dtype='object')

In [8]: df.dtypes.value_counts().index
Out[8]: Index([datetime64[ns], bool, float64, int64], dtype='object')
```

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-03 02:19:59 UTC
All green.
thanks @mroeschke
ERR/TST: Raise NotImplementedError in to_hdf for extension dtypes in MultiIndex
State: closed

- [x] closes #7775
- [x] closes #21671
- [x] closes #14948
- [x] closes #11926
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Also adds 3 regression tests for fixed timezone issues.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO HDF5
description: read_hdf, HDFStore
name: Timezones
description: Timezone data dtype

Comments:
@jorisvandenbossche this commit in this PR should hoepfully fix the flaky `dtype_str` test https://github.com/pandas-dev/pandas/pull/27144/commits/667a101533b3e6e05612e0510a6b65fab3c7d34e
thanks @mroeschke
CLN: Remove cdate_range, raise_on_error keyword, categories and ordered keywords in astype
State: closed

- [x] xref #6581
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Removed:

- `cdate_range`
- `raise_on_error` keyword in `.where` and `.mask`
- `ordered` and `categories` keywords in `.astype`

Labels:
name: Clean

Comments:
LGTM.
BUG: Fix empty closed window issue with rolling min and max
State: closed

- [x] closes #26005 (#26940 and #26910 solves other issues presented in #26005)
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
thanks @ihsansecer very nice, keep em coming!
BUG: Fix Series divmod #26987
State: closed

- [x] closes #26987 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@TomAugspurger I'm running into issues in this branch calling np.putmask on a SparseArray.  Is there an alternative for that?
merge master here
> merge master here

Still troubleshooting SparseArray and IntegerArray failures locally.
Closing in favor of #27239; SparseArray putmask needs to be figured out before this branch can go anywhere.
ERR: Raise error in to_excel when saving datetimes with timezones
State: closed

- [x] closes #27008
- [x] closes #7056
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Raise a consistent error, regardless of the writing engine, when saving datetimes with timezones in Excel since they are unsupported.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO Excel
description: read_excel, to_excel
name: Timezones
description: Timezone data dtype

Comments:
cc @WillAyd 
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-29 17:08:50 UTC
CLN: Remove older deprecations
State: closed

- [x] xref #6581
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Removed:

- Using `Series` as list-like in `rename_categories`
- `DataFrame/Series.reindex_axis`
- Renaming labels using `rename_axis`

Labels:
name: Clean

Comments:
once merged move the issue as per instructions in #6581  (to the completed list)
PERF: do not instantiate IndexEngine for standard lookup over RangeIndex
State: closed

- [x] closes #16685
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The ``pytest.raises(AttributeError, ...)`` are somewhat superfluous in that they test a limit of the implementation, rather than a guarantee of the API... but they still make the test more robust.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
Hello @toobaz! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/indexes/range.py`](https://github.com/pandas-dev/pandas/blob/e9f6f6ba09fa6c817026e3805731542790bb969d/pandas/core/indexes/range.py):

> [Line 355:80](https://github.com/pandas-dev/pandas/blob/e9f6f6ba09fa6c817026e3805731542790bb969d/pandas/core/indexes/range.py#L355): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (95 > 79 characters)
> [Line 361:80](https://github.com/pandas-dev/pandas/blob/e9f6f6ba09fa6c817026e3805731542790bb969d/pandas/core/indexes/range.py#L361): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (92 > 79 characters)






All green. I'm afraid pep8speaks is resenting the heat.
thanks @toobaz nice patch!
Plotting ExtensionArrays
State: closed

- [x] closes #26173
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

How do I go about adding tests for that rely on an external module?
Would be something like
```python3
df = pd.DataFrame({
        "length" : pd.Series([1,2],dtype = pd.Int16Dtype),
        "width" : PA_([2,3], dtype="pint[m]"),
        "distance" : PA_([2,3], dtype="in"),
        "height" : PA_([2,3], dtype=ureg.m),
        "depth" : PA_.from_1darray_quantity(Q_([2,3],ureg.mm)),
    })
df.plot()
```

Labels:

Comments:
Hello @andrewgsavage! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-30 07:12:34 UTC
I'm a bit confused here, df.select_dtypes('number') will not return columns with bool dtype. However is_numeric_dtype(dtype) will return True for bool. Should is_numeric_dtype be used in select_dtypes?
can you merge master
Nice idea but looks like this has gone stale and I think has a few dependencies that need to be cleaned up. In any case @andrewgsavage if interested in continuing please ping and can reopen
Bump python_requires to 3.5.3
State: closed

Mentioned during sprint today some of the typing constructs we use were not added until Python 3.5.2, specifically typing.Type and typing.DefaultDict and maybe at some point typing.TYPE_CHECKING

https://docs.python.org/3/library/typing.html#typing.Type

This is a bump of the setup files and docs to 3.5.2. 

Labels:
name: Build
description: Library building on various platforms

Comments:
should change

```
ci/deps/azure-35-compat.yaml:  - python=3.5.*
```

to 3.5.2?
Hmm based off error in Linux compat might even need a bump to 3.5.3 per https://github.com/python/typing/pull/267 giving that a look now
TST: Decoupled more xlrd reading tests from openpyxl
State: closed

- [X] closes #27111
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

@simonjayhawkins 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Excel
description: read_excel, to_excel

Comments:
This PR results in an additional 2 warnings. although 8 of the same were introduced in #25092.

```
pandas/tests/io/excel/test_readers.py::TestReaders::test_reader_special_dtypes[openpyxl-.xlsx]
pandas/tests/io/excel/test_readers.py::TestReaders::test_reader_special_dtypes[openpyxl-.xlsm]
pandas/tests/io/excel/test_readers.py::TestReaders::test_reading_all_sheets[openpyxl-.xlsx]
pandas/tests/io/excel/test_readers.py::TestReaders::test_reading_all_sheets[openpyxl-.xlsm]
pandas/tests/io/excel/test_readers.py::TestReaders::test_reading_multiple_specific_sheets[openpyxl-.xlsx]
pandas/tests/io/excel/test_readers.py::TestReaders::test_reading_multiple_specific_sheets[openpyxl-.xlsm]
pandas/tests/io/excel/test_readers.py::TestReaders::test_read_excel_squeeze[openpyxl-.xlsx]
pandas/tests/io/excel/test_readers.py::TestReaders::test_read_excel_squeeze[openpyxl-.xlsm]
pandas/tests/io/excel/test_readers.py::TestExcelFileRead::test_excel_passes_na[openpyxl-.xlsx]
pandas/tests/io/excel/test_readers.py::TestExcelFileRead::test_excel_passes_na[openpyxl-.xlsm]
  C:\Users\simon\Anaconda3\envs\pandas-dev\lib\site-packages\openpyxl\worksheet\_reader.py:296: UserWarning: Unknown extension is not supported and will be removed
    warn(msg)
```
@simonjayhawkins that should be solved by https://github.com/pandas-dev/pandas/pull/27122
DEPR: Change current DeprecationWarnings to FutureWarnings
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

in prep for removal in 1.0 after 0.25.0 is released.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm. merge on green.
DEPR: Series.item and Index.item
State: closed

- [x] xerf #18262
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm. pls add a reference in #6581 (and the let's deprecate issue); merge on green.
BUG: Fix indexing on DatetimeBlock
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Addresses one of the problems behind #26864, but probably doesn't address the statefullness problem there.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
I'm pretty sure this problem also affects other EA blocks, will check and add tests if so
@jreback any idea what’s up with the CI?
```
/home/vsts/work/1/s/doc/source/whatsnew/v0.25.0.rst:786: WARNING: Unknown interpreted text role "method".
/home/vsts/work/1/s/doc/source/whatsnew/v0.25.0.rst:786: WARNING: Unknown interpreted text role "method".
```
method -> meth in the whatnew
thanks @jbrockmendel
DEPR: remove deprecated date casting; closes #21359
State: closed

- [x] closes #21359
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
Not blocking, but in #21359 the discussed plan was to remove in 0.24.0
Reading through https://github.com/pandas-dev/pandas/issues/21359, it seems like @jreback and @jorisvandenbossche preferred a bit longer of a deprecation. I don't have a preference.
this is probably mergable for 1.0, can you rebase
Rebased
Thanks.
DEPR: Series.put, Series.real, Series.imag, Index.dtype_str
State: closed

- [x] xref #https://github.com/pandas-dev/pandas/issues/18262
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Removes:
`Series.put`
`Index.dtype_str`
`Series.real`
`Series.imag`

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-28 21:44:10 UTC
can you add to the list in #6581 as well
Changed to FutureWarnings and green.
thanks @mroeschke
DEPR: deprecate msgpack support
State: closed

closes #27084


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
I’m generally on board with this, but concerned about the fact that msgpack uses internals, so if this becomes “downstream” then that may make it harder to simplify things in internals if we are worried about breaking downstream.
> I’m generally on board with this, but concerned about the fact that msgpack uses internals, so if this becomes “downstream” then that may make it harder to simplify things in internals if we are worried about breaking downstream.

that’s all the more reason to deprecate this actually.
Update visualization.rst
State: closed

closes #26293

Note added to the docs, warning users to be careful when adapting datetime indices generated by pandas.

Labels:
name: Docs
name: Visualization
description: plotting

Comments:
@TomAugspurger any thoughts on this? 
What does this add to the paragraph starting with "In some situations it
may still be preferable or necessary to prepare plots"?

They both seem to be about using Matplotlib directly rather than
DataFrame.plot.

On Mon, Aug 26, 2019 at 1:08 PM William Ayd <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> any thoughts on this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27092?email_source=notifications&email_token=AAKAOIT6I2K3YQWJ3ZDZRSTQGQL2RA5CNFSM4H4E7AZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FFR7I#issuecomment-524966141>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQEGU77MGLETDDPC2LQGQL2RANCNFSM4H4E7AZA>
> .
>

I agree with Jeff and Tom, the note in this PR is just repeating what it's already in the docs. Closing this, and since the issue is still open, if someone has a better idea on how to help the users understand the problem with ticks, can open a new PR.
BUG: Fix handling of ambiguous or nonexistent of start and end times in date_range
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Closing a `FIXME` note in the test suite.

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
lgtm. @mroeschke if you want to add some types great, otherwise pls merge.
Thanks. Will add typing in a future PR.