BrandonFors/pandas-github-issues huggingface dataset:
CLN: Split pandas/tests/base/test_ops.py
State: closed

As suggested by @jreback in [this PR comment](https://github.com/pandas-dev/pandas/pull/32725#issuecomment-599299873)

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
great, i think we merged something, can you rebase.

cc @jbrockmendel who has an outstanding PR that overlaps a bit here (well inserting some new things)
I dont think any of my PRs have a conflict here; if they do I'll rebase when the time comes
@jreback @jbrockmendel Conflicts resolved and CI green
thanks @SaturnFromTitan
BUG: read_csv: fix wrong exception on permissions issue 
State: closed

Get rid of all error printf's and produce proper Python exceptions

- [x] closes #23784
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
Will add tests and whatsnew later, for now curious if it passes the windows build.
@gfyoung if any comments.
BUG: arithmetic with reindex pow
State: closed

- [x] closes #32685
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
lgtm. needs a rebase
@Meeseeksdev backport to 1.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 be7bfe6ab7ae2cba056f61dea6c3b0226bf80082
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32734: BUG: arithmetic with reindex pow'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32734-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32734 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jbrockmendel do you have time to backport this today?
> do you have time to backport this today?

yes
FIX: series.fillna doesn't shallow copy if len(series) == 0
State: closed

- [x] closes #32543
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
/azp run

<samp>
Commenter does not have sufficient privileges for PR 32733 in repo pandas-dev/pandas<br>

</samp>
@jreback Adapted the isnew section and CI is green
@jreback btw, is there any chance I could get privileges to run `/azp run`?
@SaturnFromTitan added you to pandas-triage

you can now label & close issues (i think u can prob trigger CI not sure about that)
sweet, thanks a lot @jreback!
Great thanks @SaturnFromTitan and congrats on the triage role - well deserved!
Thanks for the kind words @WillAyd :)
REF: simplify json get_values
State: closed

cc @WillAyd trying to get away from _internal_get_values in the json code; this is zero-logic-changing to narrow the scope of what gets passed to _internal_get_values.

AFAICT replacing the _internal_get_values call with calling `__array__` doesnt break any tests, but in general the two are not always equivalent and it isnt obvious to me whether there are untested cases where it matters.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
sure
@WillAyd could use your input on how to handle the last few edge cases.  The relevant cases all involve either CategoricalIndex[dt64tz] or Series[Category[dt64tz]].  Example:

```
dti = pd.date_range('2016-01-01', periods=2, tz='US/Pacific')
ci = pd.CategoricalIndex(dti)
ser = pd.Series(dti)
cser = pd.Series(ci)
ciser = pd.Series(range(2), index=ci)
```

`cser.to_json()` matches `ser.to_json()`, but they get there in different ways.  `get_values` in objToJSON calls `ser._internal_get_values()` and gets a ndarray[dt64_naive], while `cser._internal_get_values()` gets a ndarray[object] of Timestamp objects.  Is going through object-dtype intentional, or would it be OK to match the non-categorical behavior?

If matching the non-categorical behavior is OK, then we can get equivalent behavior by calling `ser.__array__` instead of `ser._internal_get_values`.

`ciser.to_json` currently raises because the call to `ci._internal_get_values` returns a `DatetimeIndex`.  Is raising the desired behavior, or maybe we want to match the series values behavior?
> `get_values` in objToJSON calls `ser._internal_get_values()` and gets a ndarray[dt64_naive], while `cser._internal_get_values()` gets a ndarray[object] of Timestamp objects. Is going through object-dtype intentional, or would it be OK to match the non-categorical behavior?

Shouldn't need an object coercion. Off the top of my head there were some bugs with datetimelike values where coercion to object would have yielded different values in some combination of the index / values, but those should be squashed now (or if not, would want to identify and correct)
OK.  Cleanest option I've identified is to use `obj.values.__array__()` instead of `obj._internal_get_values()`.  The only behavior difference is that CategoricalIndex[dt64tz] (or Series with that for its index) will no longer raise.  Is that acceptable?
CLN: Remove unncessary lambda wrapper
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

A lambda wrapper is unnecessary in these cases as the underlying methods are themselves callable. Moreover, the code does not add any logic in the wrapper to warrant the usage of lambda. 

Labels:

Comments:
[jreback] how are these inecessary?
[skasturi]  
- `max` and `lambda x: max(x)` have the exact functionality. 
- `fmt.format` and `lambda x: fmt.format(x)` have the exact functionality

[jreback] did you time the difference
[skasturi] Thank you for suggesting me to do this. They both seem to have the almost similar timing. Guess it's a macro replacement? 

![image](https://user-images.githubusercontent.com/482709/76707024-25542880-66a9-11ea-9f1b-ab84dd95ab4b.png)

[jreback] these are timing using a lambda function
[skasturi] I am not sure what you meant by this. 

[jreback] using max actually transforms this to another operation
[skasturi] Yes. As I mentioned earlier `max` and `fmt.format` are themselves `Callable`. Therefore, in general it might not be necessary to wrap them in a `lambda`.
> how are these inecessary?
> [skasturi]
> 
> * `max` and `lambda x: max(x)` have the exact functionality.
> * `fmt.format` and `lambda x: fmt.format(x)` have the exact functionality
> 
> did you time the difference
> [skasturi] Thank you for suggesting me to do this. They both seem to have the almost similar timing. Guess it's a macro replacement?
> 
> these are timing using a lambda function
> [skasturi] I am not sure what you meant by this.
> 
> using max actually transforms this to another operation
> [skasturi] Yes. As I mentioned earlier `max` and `fmt.format` are themselves `Callable`. Therefore, in general it might not be necessary to wrap them in a `lambda`.

show the the asv results
Sure. Would you mind telling me how to run the asv_bench? I am unable to find it in the top level readme. 
https://pandas.pydata.org/docs/development/contributing.html#running-the-performance-test-suite
Thanks for the PR but AFAICT the intent of these benchmarks is to actually measure the application of a lambda function. We should already have measurements for the built-in max function, so I think don't want to do this
agree with @WillAyd here, the point of these asv's is to specifically measure when using a lambda. so this change is measuring something else.
Thank you @WillAyd and @jreback for the context.
CLN: Use keep fixture in more places
State: closed

Follow up of #32487 - for [this comment of jreback](https://github.com/pandas-dev/pandas/pull/32487#issuecomment-599153289) in particular

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jreback I did a search for `pytest.mark.parametrize("keep"` in the test suite and these were the only places that popped up
thanks
CLN: Remove Ops Mixin from pandas/tests/base
State: closed

11 PRs later, the moment has finally come: closes #23877 🎉 

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`




Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
thanks @SaturnFromTitan very nice!

ok next up would be to start splitting things out of test_ops to better named modules.
BUG: Fix HDFStore empty keys on native HDF5 file by adding keyword include
State: closed

- [x] closes #29916
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Hello @roberthdevries! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-14 21:02:12 UTC
> looks fine. does this solve the OP? meaning folks simply want to use .keys() to get _all_ the tables, then use .get() to read them?

We could also merge the two sets of tables. I think pandas does not use native HDF5 tables to store DataFrames. In that case the two sets are disjoint and we should be fine returning the joined set.
I tried the variant with returning the combination of keys of pandas tables and HDF5 native tables, and the unit test `pandas/tests/io/pytables/test_store.py::TestHDFStore::test_copy` fails. I think the solution we have now helps the `dask` developers at least to have a work-around solution, that can work on both older and newer pandas versions by checking for the pandas version.
> I tried the variant with returning the combination of keys of pandas tables and HDF5 native tables, and the unit test `pandas/tests/io/pytables/test_store.py::TestHDFStore::test_copy` fails.

Hi there, this is interesting, how did this test pass before the `keys()` method changed, i.e. it used to return all the keys in pandas < 0.24?

> I think the solution we have now helps the `dask` developers at least to have a work-around solution, that can work on both older and newer pandas versions by checking for the pandas version.

Hopefully, thanks for working on it.
> don't you need this _also_ in `.groups()`?

Could be added there as well, although that use case is less typical. Most users are interested in the _tables_ in the file.
However, if this question pops up, I'll be happy to create a PR for that.
@jreback tests should have run fine, except that one of the tests took too long I think. So this is my ping on sort of green
thanks @roberthdevries 
thanks @roberthdevries and @jreback for resolving this issue for keys, it will help my project a lot.

@roberthdevries the source files coming into my system use groups (not created by pandas) so I would need the same fix applying to .groups() as well as .keys() if at all possible.
TST: Fix HDFStore leak in tests/io/pytables/test_store.py
State: closed


- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO HDF5
description: read_hdf, HDFStore

Comments:
thanks!
CLN: Switch to using `is not None` rather than `bool()`
State: closed

- [x] closes #32720
- [ ] tests added / passed - This change seemed trivial to *add* tests and should be covered by existing ones. 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [] whatsnew entry


Labels:
name: Clean

Comments:
Thanks, @skasturi.
DOC: Fix PR01, PR07 in Index.min and Index.max
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

Output for pandas.Index.min:
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found:
        No extended summary found
```

and output for pandas.Index.max:
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found:
        No extended summary found
```

Labels:
name: Docs

Comments:
Great thanks @farhanreynaldo
skip 32 bit linux
State: closed

follow up to #32706

Labels:
name: CI
description: Continuous Integration
name: 32bit
description: 32-bit systems

Comments:
can you also create an issue investigate this.
@TomAugspurger marking this for 1.0.3 if needed with the manual backport tag
Sure opened #32709 to come back to a long term solution
thanks
@MeeseeksDev backport to 1.0.x
CI: Update pipelines config to trigger on PRs
State: closed

- [X] closes #32705



Labels:
name: CI
description: Continuous Integration

Comments:
Thanks for this! Not sure what is up with the failure - just restarted let's see...
hmm, is the 32-bit build borked? we can just merge this and followup if needed later.
Yea I guess need a follow up to get rid of the 32 bit one though; looks to be failing
@TomAugspurger marking for 1.0.3 if needed
This is marked as still needs manual backport. looking at the code changes there is reference to master. What needs to be done here?
Not sure, I guess if we want to run the CI on other branches, the name of the branch (`master` in the diff) should be updated for each branch.

For context, without these changes, pipelines wasn't running.
> For context, without these changes, pipelines wasn't running.

the azure jobs are running on 1.0.x, its the code checks that are not running
i've removed the needs manual backport, but will probably resurface after 1.1.0 is released.
> i've removed the needs manual backport, but will probably resurface after 1.1.0 is released.

cc @TomAugspurger
I think needs to be changed in master so that when the 1.1.x branch is created, backport PRs will be checked
DOC: Add examples to Series operators (#24589)
State: closed

Further work on #24589.  PR #25524 added examples for many operations but not `pandas.Series.eq`, `pandas.Series.ne`, `pandas.Series.gt`, `pandas.Series.ge`, `pandas.series.le`, and `pandas.series.lt`.  This adds examples for those.

`pandas.Series.divmod` is still missing an example as discussed in #25524.


Labels:
name: Docs

Comments:
Thanks for working on this @sullivanbt 

What do you think about having in a variable the creation of the `Series`, which is repeated in every example, and then do something like:
```python
_ne_example_SERIES = _common_examples_SERIES + """
>>> a.ne(b, fill_value=0)
...
```
This would save a decent amount of repeated code.
Thanks for the suggestion @datapythonista - I updated the branch with 926a3cf to include that.
Cool. May be it would make sense to rename `_common_examples_SERIES` to `_common_examples_algebra_SERIES` to be more explicit.

Did you render one of the pages to html? Would be good to render at least one of each (algebra/comparison), to make sure there are not too many or two few blank lines... If you can post couple of screenshots even better. `scripts/validate_docstrings pandas.Series.mul` can also be helpful to identify possible errors.

Good job, nice improvement.


Completely agree that `_common_examples_algebra_SERIES` is a more useful name - updated in ca5111b.  

I've attached two screenshots - one from the algebra examples one from the comparison.  I believe this is how they should be rendering (same as the existing examples) but let me know if not.

Thanks again for your feedback.

![image](https://user-images.githubusercontent.com/30410059/77416959-046f8f80-6d9b-11ea-9721-d9966da058d0.png)
![image](https://user-images.githubusercontent.com/30410059/77416995-194c2300-6d9b-11ea-874e-f110da77a60b.png)

Thanks @sullivanbt!
BUG: Add errors argument to to_csv() call to enable error handling for encoders
State: closed

- [x] closes #22610 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
Cool makes sense. Can you add tests to make sure this works?
@WillAyd Test code is now added, pls. review
There is prior art for the `errors` argument for this purpose in `DataFrame.to_hdf` and `DataFrame:read_hdf`
@roberthdevries this looked fine, can you merge master and ping on green.

@gfyoung if you have comments.
@jreback: Agreed that this looks good.  Just need to merge with master.
Thanks @roberthdevries
BUG: Fix segfault on dir of a DataFrame with a unicode surrogate character in the column name
State: closed

Return a `repr()` version if the column name string is not printable. This also means the the column name is not present in the output of `dir()`

- [x] closes #25509 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Unicode
description: Unicode strings

Comments:
Thanks for the PR! I'm not sure falling back to the repr is the right approach though. A call to print would raise:

```python
>>> colname = "\ud83d"
>>> print(colname)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed
```

So I think should do the same here
I think this column name should be treated like other non-identifier column names. So this means that it gets ignored in `dir()`, but you can see it when you print the data frame.
I have now pushed the check for printable characters up in the failing call chain. This is the only fix that I could think of which did not print any error messages.

I have also made a mini bug reproducing script that just calls the underlying failing pandas call:
```python
import pandas as pd
import numpy as np

a = np.ndarray(shape=(1,), dtype=np.object)
a.fill('\ud83d')
table = pd._libs.hashtable.StringHashTable(2)
print(table.unique(a))
```
can you run the algos asv's (e.g. the things that hit unique) and report the results.
You lost me here, what are the algos asv's?
Would be in regards to this:

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#running-the-performance-test-suite

so from asv_bench folder run

```sh
asv continuous upstream/master HEAD -b algorithms
```
The results are a significant performance decrease for the string cases. (factor 2.3 to 1.4 worse)
I guess I have to figure out to reset the python error when the conversion fails.
With the current patch:
```
       before           after         ratio
     [e72e2dd1]       [a52e59cd]
     <master>         <fix-25509-dir-failure-on-df-with-unicode-surrogates>
-      5.83±0.8μs      4.84±0.02μs     0.83  algorithms.Duplicated.time_duplicated(True, 'first', 'string')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
Another point is that I am not quite sure why the conversion to UTF-8 is needed to compute a hash value. If we can avoid the conversion, the hashing should even be faster.
Thanks @roberthdevries another great PR

> Another point is that I am not quite sure why the conversion to UTF-8 is needed to compute a hash value. If we can avoid the conversion, the hashing should even be faster.

If you see a way to improve performance here would welcome a follow up PR
Track times
State: closed

I implement feature request https://github.com/pandas-dev/pandas/issues/32682 but if someone has more experience with pytables consider these changes.

The h5 files generated with track_times=False has the same md5 hashes, that is covered in the unit test as well as the fact that the hashes are not the same with track_times=True

- [x] closes #32682
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
https://www.pytables.org/usersguide/libref/file_class.html?highlight=create_table#tables.File.create_table this option already defaults to `True`. is the purpose of this PR not to set it? 
> https://www.pytables.org/usersguide/libref/file_class.html?highlight=create_table#tables.File.create_table this option already defaults to `True`. is the purpose of this PR not to set it?

Yes, you are right. The default value is `True` that means that creation time is stored inside h5 (that is reported as unwanted due to change of SHA hash). I exposed the track_times to `put` method of HDFStore. I let default value to `True` in all methods that it goes through to keep current behavior. Finally i pass it to `table._handle.create_table`.

The possibility to set of track_times can be seen in test on line 325. And on lines 340 and 343 you can see the different or the same check-sums after appropriate setup of track_times. 
Hi @jreback . I did changes you requested. But, I found another possibility - not to pass track_times in parameters but put it here: https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html . What do you think?
> Hi @jreback . I did changes you requested. But, I found another possibility - not to pass track_times in parameters but put it here: https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html . What do you think?

i think accepting it like in this pr is ok.
> we already require 3.4.2, so ok to simply bump the minimum to 3.4.3 is fine.

I updated requirements in separate PR and updated this PR accordingly.
Hi, I see that "krysma" already remind this feature. Please revise changes and decide next steps. I synced it with master, so now it is without conflicts.
thanks @rbenes 
Hi, trying to use track_times=False I add to look at all the options used in the [test](https://github.com/pandas-dev/pandas/blob/b1b70c7390e589bbfa0d8896aa76e64bec0cf51e/pandas/tests/io/pytables/test_store.py#L301), so that I found that index=None is needed in order to indeed obtain reproducible files.
Is it expected? If yes it should be said somewhere in the documentation.