BrandonFors/pandas-github-issues huggingface dataset:
tostring->tobytes
State: closed

- [x] closes #33238
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Thanks, maybe a few others

```
pandas/core/computation/pytables.py
176:        val = v.tostring(self.encoding)
606:    def tostring(self, encoding) -> str:

pandas/io/sas/sas.pyx
434:                    start + lngt)]).tostring().rstrip(b"\x00 ")

pandas/_libs/writers.pyx
115:    return narr.tostring().decode('utf-8')
```
updated+green
Thanks!
@Meeseeksdev backport to 1.0.x
ENH: provide standard BaseIndexers in pandas.api.indexers
State: closed

- [X] closes #33201
- [X] 1 test modified / 1 passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

This PR adds a forward-looking `pandas.api.indexers.BaseIndexer` subclass implementation and exposes it to users as discussed in #33180 .

### Exposing other indexers

It is also possible to expose a backward-looking indexer simply by exposing `pandas.core.window.indexers.FixedWindowIndexer` through the API. Seems redundant to me as this is basically the same as supplying an integer as the window argument when creating a rolling object but it can be done for consistency. I'd be grateful for a discussion.

### Note on tests

The class is incorporated into `test_rolling_forward_window` in `test_base_indexer.py` and this serves as a test that the new indexer is working correctly.

### Notes on performance
I've tested the implementation and it shows comparable performance to `pandas.core.window.indexers.FixedWindowIndexer`. In my tests the new indexer was slightly faster, either by luck, or because I streamlined the backward-looking indexer a bit when making the forward-looking one.

Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
looks good, if you can rebase and make the change to use the FixedForward indexer. Also I think worthile adding a benchmark.
@TomAugspurger @jorisvandenbossche @jbrockmendel if any comments.
Travis failing has nothing to do with this PR. See #33280 
Waiting for merge.
Could you document this new indexer around here: https://pandas.pydata.org/pandas-docs/stable/user_guide/computation.html#custom-window-rolling
Found a bug in the benchmark, needs a bit more work.

UPDATE: fixed it and tested locally, the new benchmark is working now.
@mroeschke Added documentation to the custom window rolling subsection. Can introduce a new subsection, but I think it fits in. Please tell me what you think.
@jreback 
Rebased, changed the test, added a benchmark to the suite. All green.
LGTM
Small change: had forgotten to include versionadded in the user guide. Included it.
Looks like the doc build fail is being handled in #33309 
@jreback 
All green, pinged you on relevant changes. I think I've addressed everything, but I'm not sure if I understood you correctly on adding refs and the ipython block. Please tell me if I misunderstood anything.
@jreback 
Thanks for the clarification! Sorry for getting confused: first time adding a public class.
1. Added an entry to the `window.rst` so that the docstrings get published as part of API reference output.
2. Made the links in the user guide clickable and tested locally.
3. Replaced the `code-block` with an `ipython` block. Thanks.
4. Added `pytest.raises` to test error-raising behavior. I think having it in the same test works, but please tell me if you'd rather have error-raising behavior be a separate test.
thanks @AlexKirko very nice!
TST: Use try/except block to properly catch and handle the exception
State: closed

Now no more warnings

- [x] closes #32951
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: IO CSV
description: read_csv, to_csv
name: CI
description: Continuous Integration

Comments:
perf impact?
None, i reran the asv benchmark 
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Hmm not sure about this approach though; typically you do want to pass along the error to the caller rather than swallowing

ref https://cython.readthedocs.io/en/latest/src/userguide/language_basics.html#error-return-values does `except NULL` in the signature not solve the same thing?
This fix should be more to your liking. It handles the exception exactly at the level where it is supposed to be handled.
Rerunning the asv benchmarks tells me that there are no significant changes.
@jbrockmendel 
@WillAyd does this address your concern [here](https://github.com/pandas-dev/pandas/pull/33235#issuecomment-608161329)?  If so, LGTM.
Yea this looks good very nicely done. I think good to merge @jreback 
thanks @roberthdevries very nice. don't need a whatsnew note here as this is pretty much internal.
Ods loses spaces 32207
State: closed

- [X] closes #32207
- [X] tests added / passed tests.io.excel.test_reader_spaces
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff` (I had to use git diff master though)
- [ ] whatsnew entry. It's only a bug fix does it need a whatsnew?

I'm not sure how you make xlsxb files so that test wasn't implement when checking for the different types of space.

There's quite possibly many other ways this parser doesn't handle the odf specification.


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Can you also add a whatsnew for 1.1?
Can you run black on the changes? Should fix the code checks failure

The linux 37 failure is probably unrelated
While I was filling out the form I ran black, but forgot to push the commit.

And thanks for uploading a xlsb file, I don't have a copy of excel here at home.
If you can add a whatsnew note for 1.1 this lgtm - thanks!
How's the description?
@jreback care to look?
looks ok, i imagine this would slow down writing a bit?
Hmm fair point. @detrout do you mind run `asv continuous upstream/master -b io.excel.ReadExcel` from the asv_bench directory and posting output here?

Yea in any case fixes a bug so probably not a blocker, but might want to be cognizant of perf impact(s) if any
> looks ok, i imagine this would slow down writing a bit?

There's no OpenDocument write support so I don't think it can slow down writes.
I'll need to figure out how to have a stable test environment for running a benchmark
thanks @detrout
DOC: Fixed examples in `pandas/core/groupby/`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @MomIsBestFriend
REF: sql insert_data operate column-wise to avoid internals
State: closed


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Clean

Comments:
> I don't think we should be using internal functions except in the internals. This is not internal by any stretch (would say that pandas.io is offlimits entirely)

Can you please explain a bit more *why* you don't want a method like `DataFrame._iter_column_arrays` to be used outside of `frame.py` ?

Yes, we want to use as much public functions in the other pandas modules. But we do have a set of "private" methods that we use ourselves, that I would see as "internal developer APIs". For example, we also use `Series._values`, `Series._can_hold_na`, `Index._get_level_values` (this are a few examples currently being used in sql.py), and many others, outside core/series.py or core/indexes/. 
Or do you find that those should also not be used? 

Or, do you want to limit the use here because it is outside pandas/core? (initially, you commented you didn't want private DataFrame methods to be used outside of frame.py, but maybe you meant outside pandas/core, and within all of /core it is fine to use those? In which case the argument would only be about pandas.io, pandas.plotting, pandas.tseries, more or less) 
Just trying to understand your argument.
> Can you please explain a bit more why you don't want a method like DataFrame._iter_column_arrays to be used outside of frame.py ?

Can we please bike-shed this elsewhere?
Clean Up Categorical Test for JSON
State: closed

This is a little wonky in its current state, bolting categoricals onto an existing object / fixture

Instead just localized this to the one test that covers it for now

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks @WillAyd
Fix typos in 09_timeseries.rst
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
one comment, otherwise LGTM pending green
Thanks @rogererens
CLN: Use C-API for datetime.date
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
LGTM pending green
For my own understanding, why do we prefer to use the C API? Is this likely to improve performance?
>  Is this likely to improve performance?

@TomAugspurger Correct! 

---

This is because when we use ```import``` and not ```cimport``` we are going through the python space, which is slower than pure C.

I have made POC so you could see to diffs:

![date_vs_Cdate](https://user-images.githubusercontent.com/50263213/78190573-ef6ebc80-747c-11ea-8cab-fa5fe06856fe.png)

Both are doing the exact same thing, only that the C-API ```datetime.date``` is much faster.
Thanks @MomIsBestFriend
32380 deprecate squeeze in groupby
State: closed

- [x] closes #32380
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I deprecated the squeeze keyword. I think I got every points this keyword was used. 
I hope that I considered every point necessary to deprecate a keyword. I tried to use other points were a keyword was deprecated as example.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-22 13:44:41 UTC
We will still need a whatsnew entry for this (the one you had in a prior commit was good iirc)
@mroeschke  I have added the whats new entry.
Looks good. Just need to merge master to fix the merge conflict and address @WillAyd's comment. 
> 
> 
> Looks good. Just need to merge master to fix the merge conflict and address @WillAyd's comment.

Done
Looks good. Can you update.
> 
> 
> Looks good. Can you update.

Done
Ci failure should be unrelated - can you merge master and repush?
thanks @phofl very nice.
BUG: Block.setitem GH#32395
State: closed

@h-vishal this is based on what you have in #32479.  Can you adapt parts of this to address comments over there?  (that should be merged, not this, as you did the real work)

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jbrockmendel yes, I can do that. I need to add the additional tests as well, I'll make another commit tomorrow.
closing in favor of #32479
Added option to include/ignore file extensions in `scripts/validate_unwanted/patterns.py`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

This is needed for the check in #32942 to ignore cython files.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
sure
DOC: Added check for standard pandas reference
State: closed

- [x] xref #32316
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

As suggested by @datapythonista , I've added checks in `ci/code_checks.sh` to look if the pandas is being referenced in a standardized way i.e pandas, not *pandas* or Pandas

Labels:
name: Docs
name: Clean

Comments:
> This is not working. We're here to help, but please try your changes locally before submitting a PR.

@datapythonista my bad, will try my changes locally from now on.
> This is not working. We're here to help, but please try your changes locally before submitting a PR.

I'm getting `./ci/code_checks.sh: line 243: RET: command not found`, I can't understand why I'm getting this error. All the checks are using it. 

EDIT : Fixed the error mentioned above, pushing the changes
Once again, can you check locally if things are working? This is my 10th time reviewing this PR and it's obvious once more that this is not working. There are more than 100 cases of the work `Pandas` in `doc/`, and your changes are not detecting those. No need for your PRs to be perfect, but can you please stop proposing changes that you didn't try locally even once to see if they run without errors or they make sense? Thank you
@joybhallaa are you still working on this?
take
Closing this looks like @willie3838 is looking
DOC: Fixed examples in pandas/core/indexes/
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
ping @simonjayhawkins 
Thanks @MomIsBestFriend
Add test for #24615 (Accept PandasArray (with correct dtype) in DatetimeArray constructor)
State: closed

issue #24615

I added a test for the issue 24615.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@mroeschke @jorisvandenbossche 
See my comment on the issue (https://github.com/pandas-dev/pandas/issues/24615), the original code snippet from that issue changed behaviour, so this PR is actually only testing that DatetimeArray accepts instances of itself. And that is something that I suppose is already tested.
@gorogoroumaru mind checking if we have a test in our codebase where DatetimeArray accepts itself?
DOC: Fix Error in pandas.Series.last
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

output of `python scripts/validate_docstrings.py pandas.Series.last`:
```
################################################################################
################################## Validation ##################################
################################################################################
```



Labels:
name: Docs

Comments:
Thanks @farhanreynaldo
Added test for #5091 (Missing Periods for some DateOffsets)
State: closed

- [x] closes #5091 



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Ah my mistake. When reviewing that issue I too hastily concluded that the issue was solved.

Any interest in helping solving the bug? Otherwise, definitely recommend other issues that are tagged `good first issue`
Thank you, I will try another issue.
ADMIN: Create separate issue templates for different usecases
State: closed

revival of #31551 originally opened by @jschendel 

----

> I recall this being mentioned on one of the core dev calls a few months back, and took inspiration from [rapidsai/cudf](https://github.com/rapidsai/cudf) for the various templates.
> 
> Added templates for the following use cases:
> 
>     * Bug Report
>       
>       * This is largely the same as the current issue template
> 
>     * Documentation Enhancement
> 
>     * Documentation Error
> 
>     * Feature Request
> 
>     * Submit Question
>       
>       * This attempts to direct users to StackOverflow for usage questions
> 
> 
> We could also add a Blank Template for opening an issue without any template provided, but opted not to do that for now, as I don't want to encourage people to bypass these templates. Could certainly add one if there's a consensus that we want this.
> 
> I've created a [local repo](https://github.com/jschendel/pandas-templates/issues) where you can see these templates in action since I couldn't figure out another way to actually display these. Click the "New Issue" button to see what these changes would look like.
> 
>



Labels:
name: Admin
description: Administrative tasks related to the pandas project

Comments:
thanks @MomIsBestFriend 
it even pre-sets some labels, nice!
TST: cover search_sorted scalar mixed timezones case
State: closed

- [X] closes #30086
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Raises a `ValueError` on version 0.25.1 (reporter's version), as expected:

```
>>> import pandas as pd
>>> pd.__version__
'0.25.1'
>>> ser = pd.Series(date_range("20120101", periods=10, freq="2D", tz="UTC"))
>>> val = pd.Timestamp("20120102", tz="America/New_York")
>>> res = ser.searchsorted(val)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jamescobonkerr/venv/lib/python3.7/site-packages/pandas/core/series.py", line 2694, in searchsorted
    return algorithms.searchsorted(self._values, value, side=side, sorter=sorter)
  File "/Users/jamescobonkerr/venv/lib/python3.7/site-packages/pandas/core/algorithms.py", line 1887, in searchsorted
    result = arr.searchsorted(value, side=side, sorter=sorter)
  File "/Users/jamescobonkerr/venv/lib/python3.7/site-packages/pandas/core/arrays/datetimelike.py", line 666, in searchsorted
    self._check_compatible_with(value)
  File "/Users/jamescobonkerr/venv/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 591, in _check_compatible_with
    own=self.tz, other=other.tz
ValueError: Timezones don't match. 'UTC != America/New_York'
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timezones
description: Timezone data dtype

Comments:
thanks @jamescobonkerr
REF/CLN: test_get_dummies
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
lgtm. merge on green.

> and to be redundant with the test method names (TestGetDummmies.test_basic by TestGetDummmies.test_get_dummies_basic)?

In this case it is redundant, but in many other cases we'll see something like `test_basic` in a class without an informative name. It's also not uncommon to see tests for something other than get_dummies living inside a TestGetDummies class.  I think being explicit is worthwhile here.

> Is there an ongoing effort to replace pd.DataFrame by DataFrame in our code

Not that I'm aware of.  For files testing lower-level code, I like to have all the relevant imports at the top so that it is easy to tell what is tested in this file.  If `pd` is in the namespace, I can't rule anything out.  Not really relevant for this file, just a habit I've gotten into.
updated with typo fixup + green
> (for example, I would personally prefer to always have the module name and not import objects).

Yea @jorisvandenbossche and I agree, but at the same time I think someone needs to champion that cause if we want to establish a standard

Thanks @jbrockmendel 
> > (for example, I would personally prefer to always have the module name and not import objects).
> 
> 
> 
> Yea @jorisvandenbossche and I agree, but at the same time I think someone needs to champion that cause if we want to establish a standard
> 
> 

i have always been in favor of this :)

moving to this will require a good code check though - s may need be that easy
IMO this doesn't necessarily require a code check initially. 
But we could already just stop merging PRs that do changes like `pd.DataFrame` -> `DataFrame` (or ask them to revert that part). @WillAyd gave the comment like https://github.com/pandas-dev/pandas/pull/33184#issuecomment-606957773 on several PRs recently I think, but in the end we always just merged the PR..
I care most about consistency in a single file meaning we shouldn’t mix ways of referencing common things like Series vs pd.Series

I do prefer using import of Series and not pd.Series


> I do prefer using import of Series and not pd.Series

I am confused now. I interpreted the above "i have always been in favor of this" as the opposite (https://github.com/pandas-dev/pandas/pull/33184#issuecomment-606961547), since that is replying to a quote of Will that is basically saying that he, Marc and I prefer to not import objects but modules (so using `pd.Series` instead of `Series`)
>  I think someone needs to champion that cause if we want to establish a standard

I opened a separate issue about it: https://github.com/pandas-dev/pandas/issues/33203. Let's try to decide there which of the two ways has our preference.