BrandonFors/pandas-github-issues huggingface dataset:
fix windows issue pre-commit
State: closed

By not explicitly using `python` in the entry, we would get
```
Executable `python3` not found
```
on Windows

xref https://github.com/pandas-dev/pandas/pull/37023#issuecomment-716607730

Labels:
name: Windows
description: Windows OS
name: Code Style
description: Code style, linting, code_checks

Comments:
sure ok. pls merge on green.

are the windows builds still not very stable? any ideas on this. 
> pls merge on green.

Sure - merging then (following explicit approval - I wouldn't usually do this)
DOC: move release note for 35792
State: closed

follow-on from #37416

Labels:
name: Docs

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 c85fbc72e33f6674c6189f8648bb41e9239eb76f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #37417: DOC: move release note for 35792'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-37417-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #37417 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
This didn't need a backport.
Backport PR #35792 on branch 1.1.x: CLN/BUG: Clean/Simplify _wrap_applied_output
State: closed

Backport PR #35792

https://github.com/pandas-dev/pandas/pull/37305#issuecomment-716498455 cc @jreback 

Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jreback just the conda solve issue on travis-36-cov
since #37417 merged. merging this also and changing milestones on #35792
TST: Set timeoffset as the window parameter
State: closed

- [X] closes #28266
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Hi guys, just added tests for issue #28266. I'm participating in the Hacktoberfest event and would much appreciate it if someone put the label ‘hacktoberfest-accepted’  on this PR. Any modification needed just hit me up :)


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @gabriellm1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-26 01:23:03 UTC
Thanks @gabriellm1 
Thanks!
BUG: Fix inconsistent ordering between left and right in merge
State: closed

- [x] closes #35382
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Passed sort through for right join too. Did not see a reason why not.

On a related note: I think outer joins return a inconsistent order too.

```
df = pd.DataFrame({'a': [1, 0, 1]})

print(df.merge(df, on='a', how='outer', sort=False))
   a
0  1
1  1
2  1
3  1
4  0
```

instead of
```
   a
0  1
1  1
2  0
3  1
4  1
```

like ``left`` and ``right`` now. Would have to add a ``sort`` keyword to outer join. Should I open an issue for that?


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
can you merge master and will look
merged master
Merged master too
thanks @phofl very nice!
this migtht close https://github.com/pandas-dev/pandas/issues/31951, https://github.com/pandas-dev/pandas/issues/22449, https://github.com/pandas-dev/pandas/issues/17257

if any of these, would take a test and we can close
Thx

No they are not related. but #22449 can be closed probably, after we improved the Error Reporting here this should definitely fail as said by @TomAugspurger
DEPR: Add FutureWarning when matching NaN with NaN in merge/join
State: closed

- [x] xref #32306 (and a few others, will add them later
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @WillAyd 

I started with raising a FutureWarning when merging nan on nan. This works, if ``left_index`` and ``right_index`` are not both True.

If both are True, we are dispatching to the Index methods. Should we deprecate this for indexes too? Additionally, should we remove this for ``intersection`` of index in the future too? Will continue after receiving feedback :)

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
not averse to doing this @phofl if you can add a test that shows the warning.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@phofl if you can update or close.
Closing this for now, have to evaluate here. To be consistent this would influence a lot of index methods too, if nan should no longer match nan
CLN: Fix unwanted pattern in unittest
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Introduced with #37090


Labels:
name: CI
description: Continuous Integration
name: Clean

Comments:
cc @jreback @mroeschke 

Fixes Checks failure on master,
Merging as pre-commit and the relevant CI check both pass, and the rest isn't affected - thanks @phofl !
TST: check_comprehensiveness compat for --lf and -k
State: closed

- [x] closes #23930
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
cc @TomAugspurger is this roughly what you had in mind?
Yeah, this seems nice. I want both of

1. Checks for comprehensive tests by default
2. The ability to select a subset without error

so I think this looks about ideal.
CI/CLN: Lint more class references in tests
State: closed


Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
I take it this is a get-it-all-done-at once PR?
> I take it this is a get-it-all-done-at once PR?

It started out as just MultiIndex, then I may have gotten a bit carried away
needs rebase
@jreback This is rebased and almost green if we want to merge and avoid more conflicts
thanks @dsaxton
REGR/PERF: Index.is_ performance regression
State: closed

Fixes https://github.com/pandas-dev/pandas/pull/37321#issuecomment-715878012

The offender is actually #37321. Apparantly is's a bit slow to do the check using a generator like `com.any_none`.

```python
In [1]: from pandas import *
In [2]: idx_large_fast = RangeIndex(100000)
   ...: idx_small_slow = date_range(start="1/1/2012", periods=1)
   ...: mi_large_slow = MultiIndex.from_product([idx_large_fast, idx_small_slow])
   ...:
   ...: idx_non_object = RangeIndex(1)
In[3]: %timeit mi_large_slow.equals(idx_non_object)
3.13 µs ± 79.4 ns per loop  # master
2.09 µs ± 31.1 ns per loop  # This PR
```

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
thanks @topper-123
CLN: nanops
State: closed

This sits on top of #37394.  The cleanup in _maybe_get_mask means that we end up getting a correctly-dtyped result instead of object-dtype, which becomes important in the upcoming nanmean PR.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Clean

Comments:
thanks
BUG: DataFrame.std(skipna=False) with td64 dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type

Comments:
sure
CLN: Deprecate dayofweek/hello day_of_week (#9606)
State: closed

Rename dayofweek properties in Period, Datetime, and PeriodArray classes.
Add day_of_week property to respective tests for each class.

- [x] closes #9606
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: API Design
name: Frequency
description: DateOffsets

Comments:
@arw2019 Thank you for the review. I addressed your comments and added an entry to whatsnew file.
@arw2019 Resolved minor issues. Do you know how to locally test .rst files? In the CI stage of the checks, I am failing things like
`/home/runner/work/pandas/pandas/pandas/core/indexes/period.py:docstring of pandas.PeriodIndex.rst:77: WARNING: autosummary: stub file not found 'pandas.PeriodIndex.day_of_year'. Check your autosummary_generate setting.
2391`
AND
`/home/runner/work/pandas/pandas/doc/source/whatsnew/v1.2.0.rst:31: WARNING: Explicit markup ends without a blank line; unexpected unindent.`

`./ci/code_checks.sh` shows no errors locally
> @arw2019 Resolved minor issues. Do you know how to locally test .rst files? In the CI stage of the checks, I am failing things like
> `/home/runner/work/pandas/pandas/pandas/core/indexes/period.py:docstring of pandas.PeriodIndex.rst:77: WARNING: autosummary: stub file not found 'pandas.PeriodIndex.day_of_year'. Check your autosummary_generate setting. 2391`

I think you (maybe) need to add the new methods to the rst files manually - for example in doc/source/reference/series.rst both    daysinmonth and days_in_month are listed

> `/home/runner/work/pandas/pandas/doc/source/whatsnew/v1.2.0.rst:31: WARNING: Explicit markup ends without a blank line; unexpected unindent.`

This one is a linting issue

@arw2019 The only unit test that is failing on Azure pipeline is `test_binary_arith_ops` which seems to be a known unreliable test (#37328). (The test suite passes on my macOS machine)

Is this good to merge now?
@pandas-dev/pandas-core should we do a DeprecationWarning / FutureWarning on the original attributes? (would be a followon)
@jreback All checks green except aforementioned `test_binary_arith_ops`
thanks @abhishekmangla very nice
PERF: release gil for ewma_time
State: closed

The performance changes for `rolling.EWMMethods` are not stable. I've seen those and no changes in separate runs.
```
+     1.35±0.02ms       2.17±0.5ms     1.61  rolling.EWMMethods.time_ewm('Series', 1000, 'float', 'std')
```
```
-      2.58±0.2ms      1.36±0.01ms     0.53  rolling.EWMMethods.time_ewm('Series', 10, 'float', 'std')
```
```
+     1.39±0.01ms       2.17±0.5ms     1.56  rolling.EWMMethods.time_ewm('DataFrame', 10, 'float', 'std')
-      1.71±0.3ms      1.35±0.01ms     0.79  rolling.EWMMethods.time_ewm('Series', 10, 'float', 'std')
```


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
did you ever get consistent results from the asvs?
> did you ever get consistent results from the asvs?

No, and I have no idea why...
@fangchenli ping when ready here
thanks @fangchenli
TST: split up test_concat.py #37243 - more follows up
State: closed

 * created test_categorical.py

- [x] closes #37243 
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I will finish it tomorrow, i didn't have much free time today. There is not much left, i have to create test_sort and test_invalid and make some small changes.
most of the work should be done, all tests are working, it was 291 passed and 1 xfailed before and it's the same now. 
I haven't touched test_append_common and test_append, but these files looks ok for me.
Before: 
![obraz](https://user-images.githubusercontent.com/66844934/97202340-c19feb00-17b3-11eb-9569-6af850bd1e12.png)

After: 
![obraz](https://user-images.githubusercontent.com/66844934/97202666-3410cb00-17b4-11eb-86ae-f8255e7bc4ab.png)

couple of comments, ping on green.
ok, i will check it in a moment.
@jreback everything done
Can I ask what does "ping on green" mean? Is it the green "review changes" button in "files changed" section? Or is it just typing comment, so you can see new activity?
thanks @laffite56
DOC: Improve clarity and fix grammar
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This PR improves the "Performance considerations" section of the IO tools user guide:
https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#performance-considerations

Labels:
name: Docs

Comments:
Thanks @sahidvelji
replace Appender decorator with doc
State: closed

- [x] ref #31942
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] replaces Appender with doc for NDFrame.compare()


Labels:
name: Docs

Comments:
Don't you need to Series.compare and DataFrame.compare too?
> Don't you need to Series.compare and DataFrame.compare too?

Yes. Done.

I kept the use to of `_shared_docs["compare"]` to keep the PR small but I'd find it more intuitive to define the doc string on `NDFrame.compare()` and then refer to it in the calls to the doc decorator on `Series.compare()` and `DataFrame.compare()`.

> Yes. Done.

Have you checked locally that you get the correctly-rendered docstrings?  When I try it with this PR, pandas cant even be imported.
> > Yes. Done.
> 
> Have you checked locally that you get the correctly-rendered docstrings? When I try it with this PR, pandas cant even be imported.

You were right. The CI is passing now.
thanks @smartvinnetou
TST: Different tests were collected between gw0 and gw1
State: closed

not sure if anyone else if having issue running tests locally. have created a clean pandas-dev env on a different machine so maybe a transient thing.

https://github.com/pytest-dev/pytest-xdist/issues/432

<details>

```
INSTALLED VERSIONS
------------------
commit           : dce547ea2fe7f8c5c51c04d0a87704a2b4abf476
python           : 3.8.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.18362
machine          : AMD64
processor        : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : English_United Kingdom.1252

pandas           : 1.2.0.dev0+908.gdce547ea2f
numpy            : 1.19.2
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.2.4
setuptools       : 49.6.0.post20201009
Cython           : 0.29.21
pytest           : 6.1.1
hypothesis       : 5.37.4
sphinx           : 3.2.1
blosc            : None
feather          : None
xlsxwriter       : 1.3.7
lxml.etree       : 4.6.1
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.18.1
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : 1.3.2
fsspec           : 0.8.4
fastparquet      : 0.4.1
gcsfs            : 0.7.1
matplotlib       : 3.3.2
numexpr          : 2.7.1
odfpy            : None
openpyxl         : 3.0.5
pandas_gbq       : None
pyarrow          : 2.0.0
pyxlsb           : None
s3fs             : 0.4.2
scipy            : 1.5.0
sqlalchemy       : 1.3.20
tables           : 3.6.1
tabulate         : 0.8.7
xarray           : 0.16.1
xlrd             : 1.2.0
xlwt             : 1.3.0
numba            : 0.51.2
```
</details>

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
LGTM
yep lgtm merge when greenish
I'm on windows and these failures seem to be windows specific. So I've done some bisecting, and the first failing PR for me is #35964 (commit ab5b38d56). Haven't checked what precisely it is there, but could be the parametrization over `transformation_kernels` (a set).

These are also fairly old commits, so I'm wondering why they're failing now. Maybe an incompatable update has been done in `xdist` or `pytest`?
@simonjayhawkins , can we merge this? With these changes everythings runs on my machine and evidently also on the CI.

The changes to `_good_arith_ops ` look reasonable to me, so IMO should be kept.
Fixed Metadata Propogation in DataFrame
State: closed

In reference to #28283 

Added finalize reference to append, merge, eval, diff, applymap, and update. Removed the ignore marks from the associated tests. 


Labels:
name: metadata
description: _metadata, .attrs

Comments:
Hello @jwhitty32! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-04 15:29:31 UTC
I'm still failing some checks. Didn't realize committing again in my branch would affect the PR. Sorry! New to this and trying to figure things out.
> I'm still failing some checks. Didn't realize committing again in my branch would affect the PR. Sorry! New to this and trying to figure things out.

I think those are related to the change you made to eval
Ok, I decided the issue with eval is pretty involved so I will take it off this PR. Going to come back to it in a new one. 
Great! Just pushed with merge removed. 
thanks @jwhitty32 very nice
Sorry to ask a stupid question for this PR. I'm using pandas=1.2.4 and I think that the pd.merge should now propagate the metadata.

```
>>> import pandas
>>> df1 = pandas.DataFrame({"a": [1]})
>>> df1.attrs = {"test": "test"}
>>> df2 = pandas.DataFrame({"a": [1]})
>>> df2.attrs = {"test": "test"}
>>> pandas.merge(df1, df2).attrs
{}
```

I wonder if this is expected? Thank you.
TST: Update unreliable num precision component of test_binary_arith_ops (GH37328)
State: closed

- [x] closes #37328 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Seems like a numerical precision issue indeed.

Comparing the values: 16.964677878827604 and 16.964677878827608 I get the following tolerance levels:
Max absolute difference: 3.55271368e-15
Max relative difference: 2.09418281e-16

Therefore I changed: `tm.assert_numpy_array_equal(result.values, expected)` to `tm.assert_almost_equal(result.values, expected, atol=1e-14, rtol=1e-15)`
I see that this issue is not limited to those values, I also get this error on #36838, where there's a mismatch between 0.8129387113107932 and 0.8129387113107933

The proposed change would catch it, but maybe we will need to relax the tolerances more to make it generic. 
@avinashpancham Merging master should fix the linting error
xref #13339 which changed the first assert in `check_binary_arith_op` to `assert_almost_equal`. Make sense we would need to do the same here.
@rhshadrach sorry but I dont understand what you mean. Could you maybe give some more explanation?
@dsaxton thanks that solved it indeed
> @rhshadrach sorry but I dont understand what you mean. Could you maybe give some more explanation?

Now that I read it again it looks more like a general comment, instead of feedback. My bad.
CI failed due to: `RuntimeError: can't start new thread`. Would it be possible for someone to restart the CI?
> CI failed due to: `RuntimeError: can't start new thread`. Would it be possible for someone to restart the CI?

restarted.
cc @jbrockmendel this seems reasonble to me.
@jreback merged master and CI is greenish. Errors are memory/OS errors unrelated to this change, do we want to run this again or is this fine?
thanks @avinashpancham
CI move validate unwanted patterns over to pre-commit
State: closed

This becomes much simpler with pre-commit :smile: As with other PRs of this kind, the motivation for moving checks like these over to pre-commit is that they become cross-platform and that they provide faster feedback to devs

----

example: if I apply
```diff
diff --git a/pandas/io/formats/info.py b/pandas/io/formats/info.py
index 891b3ea7a..6c1075910 100644
--- a/pandas/io/formats/info.py
+++ b/pandas/io/formats/info.py
@@ -479,7 +479,7 @@ class DataFrameTableBuilder(TableBuilderAbstract):
         collected_dtypes = [
             f"{key}({val:d})" for key, val in sorted(self.dtype_counts.items())
         ]
-        self._lines.append(f"dtypes: {', '.join(collected_dtypes)}")
+        self._lines.append(" " f"dtypes: {', '.join(collected_dtypes)}")
 
     def add_memory_usage_line(self) -> None:
         """Add line containing memory usage."""
```
then I get
```console
$ pre-commit run unwanted-patterns-strings-to-concatenate --all
Check for use of not concatenated strings................................Failed
- hook id: unwanted-patterns-strings-to-concatenate
- exit code: 1

pandas/io/formats/info.py:482:String unnecessarily split in two by black. Please merge them manually.
```

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks
ENH/BUG: implement __iter__ for IntegerArray so conversions (to_dict, tolist, etc.) return python native types
State: closed

- [x] closes #29738
- [x] closes #34665
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Reheating #31328 (as it's been stale for a few months)

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
likely also close

Possibly related to: #27616, #25969, #21256

pls indicate this by adding tests (or comment that we don't close these)
Getting a doctest failure after this patch: is it ok to change the doc here?
```
_____________ [doctest] pandas.core.arrays.floating.FloatingArray ______________
240 
241     Returns
242     -------
243     FloatingArray
244 
245     Examples
246     --------
247     Create an FloatingArray with :func:`pandas.array`:
248 
249     >>> pd.array([0.1, None, 0.3], dtype=pd.Float32Dtype())
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,3 @@
     <FloatingArray>
    -[0.1, <NA>, 0.3]
    +[0.10000000149011612, <NA>, 0.30000001192092896]
     Length: 3, dtype: Float32
```
> likely also close
> 
> Possibly related to: #27616, #25969, #21256
> 
> pls indicate this by adding tests (or comment that we don't close these)

#21256 - fixed on current master (I'll add tests)
#27616, #25969 - open & unaffected by this fix (they're bugs in the NumPy-backed data structures). I'll investigate in a separate PR
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale. if you want to continue, pls ping and can re-open.