BrandonFors/pandas-github-issues huggingface dataset:
ENH: add render_links for Styler.to_html formatting
State: closed

This came about as a requirement for converting DataFrame.to_html into Styler.to_html, citing @jorisvandenbossche in favour of the functionality.

Note that the `DataFrame.to_html` `render_links` argument is rather limited to just detecting only a URL in a cell.

The function added for Styler here is a more general pattern search, and it allows the result to be viewed in Jupyter Notebook directly, which the precursor does not.


![Screen Shot 2021-12-24 at 22 20 45](https://user-images.githubusercontent.com/24256554/147372493-5458f18a-4cf3-417a-86a1-7750e430b145.png)




Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Please note I added the LaTeX equivalent, so the signature is now consistent with the `escape` argument (in `{"html", "latex"}`.

Also note I changed the argument name to `hyperlinks` from `render_links`. Comments welcome. 
thanks @attack68 
i was not sure if there was an actual issue for this (aside from the discussion), if so can you link.
PERF: avoid copies in lib.infer_dtype
State: closed

- [x] closes #45022
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

In the `skipna=True` case, instead of doing `values = values[~isnaobj(values)]`, just pass `skipna` to the relevant validator functions.  This causes trouble because those validator functions have different rules about what NA values they allow (analogous to is_valid_na_for_dtype).

I'm down to 1 test case failing locally, fixing it will require changing the StringArray constructor to allow None and np.nan (xref #40839) in addition to just pd.NA (AFAICT pd.array and pd.Series(... dtype="string") will not be affected).  I'm fine with that, but someone more involved with the string code might want to weigh in cc @simonjayhawkins 

```
In [1]: import numpy as np

In [2]: arr = np.arange(10**5).astype(object)

In [3]: from pandas._libs import lib

In [4]: %timeit lib.infer_dtype(arr, skipna=True)
962 µs ± 77 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR
3.81 ms ± 33 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- master


In [7]: %load_ext memory_profiler

In [16]: arr = arr.repeat(100)

In [18]: %memit lib.infer_dtype(arr, skipna=True)
peak memory: 169.24 MiB, increment: 0.00 MiB  # <- PR
peak memory: 265.39 MiB, increment: 95.38 MiB  # <- master

```

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Bad news @mroeschke:

```
    def test_dialect_conflict_except_delimiter(all_parsers, custom_dialect, arg, value):
[...]
E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning('unclosed file <_io.BufferedRandom name=10>'), 'D:\\a\\1\\s\\pandas\\core\\indexes\\base.py', 665)]
```
> Bad news @mroeschke:
> 
> ```
>     def test_dialect_conflict_except_delimiter(all_parsers, custom_dialect, arg, value):
> [...]
> E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning('unclosed file <_io.BufferedRandom name=10>'), 'D:\\a\\1\\s\\pandas\\core\\indexes\\base.py', 665)]
> ```

:(

This looks like it occurred on Windows? The check I added doesn't work there since idk the equivalent `lsof` check.
wow, looks pretty good.
this WIP?
> this WIP?

@lithomas1 says this needs to wait on #41412
nice!
STYLE: upgrade flake8 to version 4+
State: closed

- [x] closes #45051 
- [x] tests passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] bumping version of flake8 and update repo url


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Sorry I'm unfamiliar with the  `pre-commit run --show-diff-on-failure --color=always --all-files` command, which I'm failing. It even fails when i run it locally on the master branch. Can someone explain it to me? Sorry, I'm new to this.
DEPR: Deprecate set and dict as indexers
State: closed

- [x] closes #42825
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

- ``DataFrame.__setitem__`` does not work for unhashable object
- ``iloc`` does not work for unhashable objects for __getitem__ and __setitem__

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
Not 100% sure, but wanted to add explicit checks as followups to raise a controlled error. Something we can reuse for the deprecated cases.
> Not 100% sure, but wanted to add explicit checks as followups to raise a controlled error. Something we can reuse for the deprecated cases.

sgtm
thanks @phofl
TST: implement deserialization tests for older schema versions
State: closed

- [X] closes #44995
- [x] tests added / passed (see #45047)
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks @jmg-duarte
BUG: DataFrame(nested_object_ndarray) match DataFrame(nested_pandas_array)
State: closed

- [x] closes #43986
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
i guess should have a whatsnew note, pls add and ping on green.
ping
DOC: to_hdf() method docstring updated with dropna and index
State: closed

Resolves GitHub Issue #45030

- [x] closes #45030 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
@github-actions pre-commit
@github-actions pre-commit
Hey, thx for tackling this. I think there are a few places where this might be missing too. Should have mentioned in the issue

HDFStore.append and HDFStore.put are candidates. 
> Hey, thx for tackling this. I think there are a few places where this might be missing too. Should have mentioned in the issue
> 
> HDFStore.append and HDFStore.put are candidates.

Cheers, I'm getting a couple of failing checks in the github actions, not entirely sure hwat is going on as I only updated docstrings. Can you lend me some adivce cheers?
They are unrelated
Hello @fanoway! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-27 03:00:08 UTC
> 

Ok cheers. I fixed all the doc strings for the HDFStoe.put() and HDFStore.append() as they were missing quite a few parameters
@fanoway you can install pre-commit locally which will check a few styling things
@fanoway Are you interested in continuing?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing please merge in main, address the comments and we can reopen
REF: dispatch Series.rank to EA
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Test coverage is severely lacking here.

Maybe could implement the general case in terms of argsort?  cc @TomAugspurger @jorisvandenbossche 

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
looks ok to me
Error on bad lines pyarrow
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

cc @lithomas1 Not sure if that was what the TODO was referring to

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
> Not sure if that was what the TODO was referring to

I don't know the specific context of that comment, but many of the `TODO(1.4)` comments have to do with the `pyarrow_skip` mark being changed to pyarrow_xfail
> cc @lithomas1 Not sure if that was what the TODO was referring to

The TODO was to uncomment that line from the set which I see you've done here.
thanks @phofl and @lithomas1 
Can we use pandas' `read_csv` with `pyarrow` engine and skip bad lines like "ArrowInvalid: CSV parse error: Expected 2 columns"?

The default engine's `read_csv` is quite slow while many original args are not supported by `pyarrow` engine, which makes it inconvenient to process large data file.
BUG: dropna changing index when nothing is dropped
State: closed

- [x] closes #41965
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

```
# With nan as last element
# before
# 1.63 ms ± 113 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# after
# 1.68 ms ± 91.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


# without nan
# before
# 1.6 ms ± 68.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# after
# 714 µs ± 63.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
ty!
Fix Zstandard compression unit test
State: closed

Follow-up for #43925

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks @jonashaag
DOC: Run doctests for pandas/plotting
State: closed

- [x] closes #22459




Labels:
name: Docs

Comments:
Failure unrelated. Looks mamba related

```

Updating 'pandas-dev' env from conda env update...
  /usr/share/miniconda/condabin/mamba env update --name pandas-dev --file ci/deps/actions-38-db.yaml
  
  
  Looking for: ['python=3.8', "cython[version='>=0.29.24']", "pytest[version='>=6.0']", "pytest-xdist[version='>=1.31']", "hypothesis[version='>=5.5.3']", "pytest-cov[version='>=2.10.1']", "aiobotocore[version='<2.0.0']", 'beautifulsoup4', 'boto3', "botocore[version='>=1.11']", 'dask', "fastparquet[version='>=0.4.0']", "fsspec[version='>=0.7.4']", "gcsfs[version='>=0.6.0']", 'geopandas', 'html5lib', 'matplotlib', 'nomkl', 'numexpr', 'numpy=1.18', 'odfpy', 'openpyxl', 'pandas-gbq', 'psycopg2', "pyarrow[version='>=1.0.1']", 'pymysql', 'pytables', 'pytorch', 'python-snappy', 'python-dateutil', 'pytz', "s3fs[version='>=0.4.0']", 'scikit-learn', 'scipy', 'sqlalchemy', 'statsmodels', 'xarray', 'xlrd', 'xlsxwriter', 'xlwt', 'brotlipy', 'coverage', 'pandas-datareader', 'pyxlsb']
  
  
  Encountered problems while solving:
    - nothing provides openblas 0.3.5.* needed by libblas-3.8.0-4_openblas
  
  pkgs/main/noarch          
  pkgs/r/linux-64           
  pkgs/r/noarch             
  pkgs/main/linux-64        
  conda-forge/noarch        
  conda-forge/linux-64  
```
BUG: DataFrame(dict_of_series) raising depending on order of dict
State: closed

- [x] closes #44091
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
greenish
Typ parts of python parser
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
looks fine can you rebase again
Done
Thanks @phofl
Set correct missing value indicator in astype for categorical
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
The failure comes from:

``np.nan`` as float is considered as valid na type for str dtype. I think this should return False instead of True?
Question is if we want to support np.str in the first place...
also this should add onto another whatsnew note? or this is separate?
This is a follow up of #44930 and goes with the whatsnews there
nice!
DEPR: special-cased downcasting in DataFrame.where GH#44597
State: closed

- [x] closes #44597
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
hmm lots of failures
hmm timeouts....
> hmm timeouts....

the last one was ArrayManager, now looks like its complaining 'zstandard' isnt installed.
> > hmm timeouts....
> 
> the last one was ArrayManager, now looks like its complaining 'zstandard' isnt installed.

yeah this is a problem on master rn
BLD: remove MSVC workaround
State: closed

- [x] closes #23209
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Shot in the dark to see if the upstream problem behind #23209 is resolved.

Labels:
name: Build
description: Library building on various platforms
name: Windows
description: Windows OS

Comments:
> does this bump up min versions of any windows things?

im not aware of anywhere that we specify those
We're still on VS 2017 on the wheel repo. This needs to wait for MacPython/pandas-wheels#160.
Thanks @lithomas1.  Is there an ETA on that?  i.e. should this be closed or just given the "Blocked" label?
That patch is scheduled to land for the 1.4rc, but this'll probably have to land in 2.0 because we'll have to wait until after 1.4.0 is released to make sure that no issues are reported that will cause it to be rolled back(last time we upgraded to Visual Studio 2019, we had missing DLL issues).
 
> That patch is scheduled to land for the 1.4rc, but this'll probably have to land in 2.0 because we'll have to wait until after 1.4.0 is released to make sure that no issues are reported that will cause it to be rolled back(last time we upgraded to Visual Studio 2019, we had missing DLL issues).

changing milestone to 1.5
@lithomas1 looks like MacPython/pandas-wheels#160 has been merged, does that mean we're good to go here?
I think so.
ok let's give this a go, thanks @lithomas1 

might be worth updating install.rst with the min required MSVC version?
REGR: DataFrame.shift with periods>len(columns) GH#44978
State: closed

- [x] closes #44978
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
can you rebase
hmm one more time pls
PERF/CLN: avoid potential copies in ravel
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> pandas/core/dtypes/missing.py:276: error: Module has no attribute "isnaobj2d"; maybe "isnaobj"?  [attr-defined]
CI: skip segfault tests for array manager
State: closed

- [x] xref #44994
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

cc @jreback 

Labels:
name: CI
description: Continuous Integration
name: Segfault
description: Non-Recoverable Error

Comments:
thanks @phofl merge on green
The segfault is gone, greenish
CI: Pin setuptools for python dev
State: closed

Lets see if this works

- [x] xref #44980 

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
should this

https://github.com/pandas-dev/pandas/blob/d228a781b4d7be6c753cee940ce3ee692e97697d/ci/setup_env.sh#L109

also be pinned?
Yep, thx
Could you leave a `TODO` comment nearby the change referencing https://github.com/pypa/setuptools/issues/2941?
Done
merge when ready
uhh the ArrayManager tests have been going for 4 hours and counting
> uhh the ArrayManager tests have been going for 4 hours and counting

hmm 3hrs on second attempt.. 

Should be allow >=60.0.3 given this release fixes the issue?
I think we should be able to avoid pinning this, if the newest version works?
yes, agreed
Looks like it does not adress all problems, https://github.com/pandas-dev/pandas/runs/4594019360?check_suite_focus=true

Edit: Can reproduce locally
The experimental data manager issue seems unrelated. I can also see this at https://github.com/pandas-dev/pandas/runs/4594019873?check_suite_focus=true. This has the newest setuptools
experimental data manager failure is unrelated, see #44994
other than ArrayManager is this ready?
Yes
> Yes

ok and the AM tests broke on the json changes?
actually can we just unpin this? e.g. if >=60.0.3 its working?
60.0.3 does not work unfortunately. There are still problems. Posted some more info on the issue.
Yes AM case was from the json change
cool thanks @phofl 

ok if you don't mind disabling the failing tests in AM build for the json (and create an issue about it) to get the CI back to green :->
#44999
TYP: Return int instead of changing result type in nanops.py
State: closed

xref #37715

The error was about the lines that set ````result = -1````, which changed the type of result from np.ndarray to int. Changed it to return -1 from the if block since everywhere else its dealing with an np.ndarray

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Could you merge master? We have fixed the ci now
thanks @kashifkhan
DOC fix typo in describe_option
State: closed

I updated pandas/_config/config.py fixing a typo - 
Call with not arguments to get a listing for all registered options to Call with no arguments to get a listing for all registered options.

closes #44970

Labels:
name: Docs

Comments:
@MarcoGorelli I got the above message and do not understand it. Do i need to delete the branch because it has been merged? 
Thanks in advance
hey - it's up to you
BUG: NDArrayBackedExtensionArray.transpose, copy
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Shouldn't be user-facing ATM.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> pandas/core/arrays/datetimelike.py:511: error: Unexpected keyword argument "order" for "copy"  [call-arg]

typing
DEPR: Index.__getitem__[bool] GH#44051
State: closed

- [x] closes #44051
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks the first comment pls followup when you have a chance (list of bools when indexing)
WIP: CI: Pameterize locale tests specifically for Linux
State: closed

Looking to eventually eliminate the local-specific builds by parametrizing in specific tests 


Labels:
name: CI
description: Continuous Integration

Comments:
What are the motivations which have led to considering to remove the locale-specific builds? The reason I ask is because I see two different reasons to test against multiple locales:
 1. Test intended locale-specific behaviour
 2. Prevent unintended locale-specific behaviour

The first category will probably benefit from this PR, as testing multiple locales becomes much faster and thus can be performed more frequently and efficiently. However, the second category will effectively lose coverage.

Specifically, I personally run on the locale `de_CH`, where multiple tests cause false positives. That makes impractical for me to run the test suite locally, so all my contributions are blind. I was about to open a PR to include `de_CH` in the test-set, as a basis to provide contributions to fix those tests. Should I do that anyway, or how would you like me to proceed with that issue?

On another note, just to double-check: The `with_locale` fixture and the `tm.set_locale` context manager it uses are neither thread-safe nor context safe. They set the locale process-wide. This could cause trouble in some parallel testing scenarios, although I believe `pytest-xdist` does spawn separate processes before running any test, so the sub-processes should have independent locales.
See also #44625 and #45013.
Currently pandas only tests a European locale `it_IT` and non-latin locale `zh_CN` on PY38 (in 2 separate builds)

This PR ensures we are testing these locales (also with `en_US`) on all our Linux PY38/39/3.10 builds so locale testing is being _expanded_ not removed.

After this PR it would be easier in theory to tests more locales like `de_CH`. And `pyetest-xdist` runs tests in multiple processes so hopefully the thread safety concerns should not be an issue here.
CI: Avoid flaky hypothesis test with deadline=None
State: closed

xref https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=70928&view=logs&jobId=9dd5e044-6afa-5be6-cb56-860f1ee82b31&j=9dd5e044-6afa-5be6-cb56-860f1ee82b31&t=d3ae057b-d0af-59ce-6e4c-b69ef3739701


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
great, merge on green
Failure due to stalled build