BrandonFors/pandas-github-issues huggingface dataset:
BUG: Parquet does not support saving fp16 GH#44846
State: closed

- [x] closes #44846
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] Adds validation check for float16 column in to_parquet function by raising a ValueError


Labels:
name: IO Parquet
description: parquet, feather

Comments:
Hello @Anirudhsekar96! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-14 04:01:15 UTC
Is this explicit check needed? It seems pyarrow itself already gives a decent error message?

```
In [4]: import pandas as pd
   ...: import numpy as np
   ...: 
   ...: data = np.arange(2, 10, dtype=np.float16)
   ...: df = pd.DataFrame(data=data, columns=['fp16'])
   ...: df.to_parquet('./fp16.parquet')
...
ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: halffloat
```

(adding an explicit check also means we wouldn't automatically get the improvement if this is fixed on pyarrow's side)
In the given example, PyArrow creates a file called 'fp16.parquet' before throwing the error resulting in an empty file. An explicit check before passing to PyArrow would prevent the creation of the empty file.

Additionally, it seems that there is little interest in adding support for float16 in PyArrow at the moment (see ref: https://github.com/apache/arrow/issues/2691). Parquet format also does not seem to have adding extensions for supporting float16 in the road map (ref: https://issues.apache.org/jira/browse/ARROW-7242). 

Another option would be to coerce all the columns with float16 coerce to float32 and adding a warning indicating the conversion. Currently fastparquet handles fp16 using the conversion method. 
> In the given example, PyArrow creates a file called 'fp16.parquet' before throwing the error resulting in an empty file. An explicit check before passing to PyArrow would prevent the creation of the empty file.

That might actually be a bug in the pandas code, as I can't reproduce it with just pyarrow (I do reproduce it with the pandas version above):

```python
import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({'a': np.array([0, 1, 2], dtype="float16")})
pq.write_table(table, "test_halffloat.parquet")
```
So this "creates emtpy file" issue also happens for other data types that aren't supported with Parquet (such as timedelta64, or for any other reason that pyarrow raises an error while writing). I think that is something we should fix in general instead of doing a special case check for float16.
good point @jorisvandenbossche 

ok let's close this issue and open one for the writing partial files
Opened a new bug report for the broader issue:

https://github.com/pandas-dev/pandas/issues/44914
Tests added for dtype comparison with fixture
State: closed

- #43038 closes #xxxx
- 3 tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- Dtype comparsisons added for regression test

Firstly, this is my first contribution to a huge code base therefore I am learning everything the first time, sorry for this inconvenience, again. Secondly, I have another pr for this commit but it involved a lot of other changes (315 file change) by other people and I dont know what to do with those changes. So I made this pr to make my one file change merge to master branch. If you can merge it to master before Sunday, I will be very happy because I need it for my class project. Thank you and kind regards.

My another pull who had 315 file change by other people: https://github.com/pandas-dev/pandas/pull/44618

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can we stick to the original PR please? rebasing is a fundamental part of the git workflow, it's worth learning
@MarcoGorelli My latest commit passed all the tests except one because of this code:

FAILED pandas/tests/io/parser/test_c_parser_only.py::test_buffer_rd_bytes[c_low]
= 1 failed, 141673 passed, 2728 skipped, 1264 xfailed, 2 xpassed, 136 warnings in 1894.99s (0:31:34) =

This is not a test I wrote, so what can i do about it? Thank you. 
it's unrelated, don't worry about it
> it's unrelated, don't worry about it

@MarcoGorelli Do i need to do anything else before merge? 
thanks @fotino21 
Thanks for all of you. I am grateful for this experience ;] @jreback @MarcoGorelli
CI: Pin numpydev wheels
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Honestly, Python 3.10 should be moved to the posix file, xref #43890. I don't have time for that though rn.

Probably not gonna be available tmrw, feel free to push commits to this branch for me.

This pins to a 4 day old wheel.


Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @lithomas1
BUG: IntervalArray.__cmp__(pd.NA)
State: closed

- [x] closes #31882
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Could repurpose #31882 to be about the more general case and not just IntervalArray?

DataFrame cases fail in a way that would be avoided with 2D EAs.

Labels:
name: Bug
name: Interval
description: Interval data type

Comments:
Perf! Have another branch that will move them all to the faster version

On Thu, Dec 9, 2021 at 3:52 PM Jeff Reback ***@***.***> wrote:

> ***@***.**** requested changes on this pull request.
> ------------------------------
>
> In pandas/_testing/__init__.py
> <https://github.com/pandas-dev/pandas/pull/44830#discussion_r766237941>:
>
> > @@ -282,6 +283,10 @@ def to_array(obj):
>          return DatetimeArray._from_sequence(obj)
>      elif is_timedelta64_dtype(dtype):
>          return TimedeltaArray._from_sequence(obj)
> +    elif isinstance(obj, pd.core.arrays.BooleanArray):
> +        return obj
> +    elif isinstance(dtype, IntervalDtype):
> +        return pd.core.arrays.IntervalArray(obj)
>
> can you use is_interval_dtype to be consistent
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/44830#pullrequestreview-828309450>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6HGLKEGVOJT24XOAJTUQE6LRANCNFSM5JXPZ4BA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

if you can add a whatsnew in followon
REGR: resampling DataFrame with DateTimeIndex with empty groups and uint8, uint16 or uint32 columns incorrectly raising RuntimeError
State: closed

- [ ] closes #43329
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version
name: Resample
description: resample method

Comments:
thanks @simonjayhawkins 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have a look at my logs.
PERF: faster dataframe construction from recarray
State: closed

- [x] closes #44826
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

```python
n = int(1e6)
df = pd.DataFrame({"A": [.0]*n})
arr = df.to_records(index=False)

In [1]: %timeit pd.DataFrame(arr)
3.64 s ± 77.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <- master
2.23 ms ± 421 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
How to add test case for performance pr? I have no experience on this.
> How to add test case for performance pr? I have no experience on this.

for this you can just show some `%timeit` results comparing the PR to the status quo
added @jbrockmendel 
thanks @GYHHAHA
BUG: Fix regression in groupby.rolling.corr/cov when other is same size as each group
State: closed

- [x] closes #42915
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Window
description: rolling, ewma, expanding

Comments:
nice ping on greenish
greenish
thanks @mroeschke 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have a look at my logs.
CI: Install more packages from conda-forge
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Let's see how this goes. Not sure if any of those pip installs are there on purpose or just because at the time of writing no conda-forge package was available.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Test failures seem to be unrelated?
Oops I undid my Zstandard removal with the rebase. Will rebase and fix 
thanks @jonashaag
CLN: Split test_fields.py
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jbrockmendel https://github.com/pandas-dev/pandas/runs/4453493254?check_suite_focus=true maybe something changed in numpy?
some of them look like changed exception messages which isn't too bad.  not sure about the reductions
> some of them look like changed exception messages which isn't too bad. not sure about the reductions

great thanks for having a look @jbrockmendel
CI: use pytest-timeout to avoid windows timeouts
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Broken off from #44429, where I haven't seen any of the windows timeouts over quite a few runs.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
thanks, good to have in any event
Clarify regex interpretation in str.contains
State: closed

Fix GH44811

- [x] closes #44811
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry  *Seems unnecesssary*


Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hello @nealmcb! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-08 01:49:44 UTC
thanks @nealmcb better error message are +1
CI: PyPy3 testing
State: closed

- [ ] xref #37653 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: PyPy

Comments:
~~Wow, only 1 failing test here. I guess the crashes/segfaults are probably due to an older numpy then(this needs to be verified though). I guess I'll add more deps in a follow up then.~~
EDIT: This passes w/out deps other than the required ones. I'll add more in follow up
thanks @lithomas1
TST: group by - apply Key Error
State: closed

- [x] closes #44310  
- [X] tests added / passed
- [X] ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
Hello @brunoacpcosta! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-10 18:40:54 UTC
Thanks for the help!
Thanks @brunoacpcosta
TST: split/parametrize
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
great
Close HDFStore when an AttributeError happens
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
I close it because #28429 solution would be more generic.
PERF: avoid going through .values for reindexing both index/columns for ArrayManager
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

This fastpath calls `self.values` and thus is only a fastpath for BlockManager (we only get here with all columns having the same dtype).

Labels:
name: Performance
description: Memory or execution speed performance
name: Stale

Comments:
I've been looking at this code path as, for reasons related to the CI failures, it appears fragile.  e.g. in TestDataFrameSelectReindex.test_reindex_datetimelike_to_object setting `df["b"] = 9` (following `df = ser.unstack()`) breaks similarly.  Another degree of separation xref #42921.
It also seems that our deprecation is not working if you only reindex the rows (and not the rows/columns together):

```
In [1]: arr = date_range("2016-01-01", periods=6).values.reshape(3, 2)
   ...: df = DataFrame(arr, columns=["A", "B"], index=range(3))
   ...: ts = df.iloc[0, 0]
   ...: fv = ts.date()

In [2]: df.reindex(index=range(4), columns=["A", "B", "C"], fill_value=fv)
<ipython-input-2-fdba941b8324>:1: FutureWarning: Using a `date` object for fill_value with `datetime64[ns]` dtype is deprecated. In a future version, this will be cast to object dtype. Pass `fill_value=Timestamp(date_obj)` instead.
  df.reindex(index=range(4), columns=["A", "B", "C"], fill_value=fv)
Out[2]: 
           A          B          C
0 2016-01-01 2016-01-02 2016-01-01
1 2016-01-03 2016-01-04 2016-01-01
2 2016-01-05 2016-01-06 2016-01-01
3 2016-01-01 2016-01-01 2016-01-01

In [3]: df.reindex(index=range(4), fill_value=fv)
...
TypeError: value should be a 'Timestamp' or 'NaT'. Got 'date' instead.
```

The second should also work and trigger the deprecation warning (the tests here are running into this because I am now basically taking the reindex path in separate steps for AM).

Which is probably indeed the same as https://github.com/pandas-dev/pandas/issues/42921
> It also seems that our deprecation is not working

IIRC this code path goes through maybe_promote, which doesn't _quite_ match the DTA behavior.  There was some work a few months ago that brought them closer into alignment, but off the top of my head I don't remember how far that got.
seems reasonable can you merge master
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG/TST: ArrowStringArray skips/xfails
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

This fixes `ArrowStringArray.__eq__(some_list)`, but the Series method already works so i dont think it counts as user-facing.

Labels:
name: Strings
description: String extension data type and string data
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
any user facing changes?
not unless they're using ArrowStringArray directly; the Series method already works
ENH: support np.add.reduce, np.multiply.reduce
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: ufuncs
description: __array_ufunc__ and __array_function__

Comments:
nice!
PERF: avoid calling .values to know the result dtype in eval()
State: closed

Currently, the `pd.eval(..)` expression parser calls `.values` several times just to know the dtype of that array. When you actually need to construct this array (which can be costly), we can know this dtype without actually constructing it.

TODO: I need to update the `as_array_dtype` method to ensure it always returns a `np.dtype` (now it can also return an extension dtype)

Labels:
name: Performance
description: Memory or execution speed performance
name: expressions
description: pd.eval, query
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jorisvandenbossche can you merge master and update
@jorisvandenbossche can you address comments and rebase
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
prob a reasonable change but closing as stale
ENH: add support for reading .tar archives
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

At the moment, reading from `.tar.gz` will decode using `.gzip`, but interprets the `tar` contents as if it were a `csv`. This leads to funky behaviour (due to the way `tar` files are structured, it will interpret filenames as column names), and is generally incorrect.

At the moment, Pandas users can work-around this using `tarfile`:

```python
import pandas as pd
import tarfile

with tarfile.open("file.tar.gz", "r:*") as tar:
    file = tar.getnames()[0]
    return pd.read_csv(tar.extractfile(file))
```

This PR adds this logic into Pandas, similar to how Pandas already supports reading from `.zip` files.

Co-Authored-By: @margarete01

Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Hello @Skn0tt! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-06 09:26:10 UTC
Hey @twoertwein, thank you for the review! Addressed the feedback. I think we're still missing some documentation things here and there.

- [x] TODO: add `tar` as compression option to docs
will need https://github.com/pandas-dev/pandas/pull/43925 and review
Merged in #43925 and added to the shared docs.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@Skn0tt if you can update here
I'm pretty sure I fixed the remaining tests - not so sure on what machines they are, though 😅 I'll look around and see if I can find them
couldn't find it :/ I'll dedicate some time this weekend to do that work again
Tests are green, and all checks apart from one are green too. That last failing check is from a Azure Pipelines timeout - I‘m pretty confident that this goes away when re-run.
cc @twoertwein if you can have a look.
green! :)
Thanks @Skn0tt. If you could follow up with this https://github.com/pandas-dev/pandas/pull/44787#discussion_r867014069= in another PR that would be great
ERR: improve error message for unsupported reductions
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/44737: keeping the change to use "not supported", but removing the class name (end users only know dtypes, not array class names)

Will update the tests if there is agreement on the message.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Stale

Comments:
sure
> Will update the tests if there is agreement on the message.

yes please.
@jorisvandenbossche if you'd fixup and rebase when you can
@jorisvandenbossche can you follow up and rebase
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. Feel free to reopen when you have time to merge main and address the corresponding test.
Simplifying CONTRIBUTING.md for new contributors.
State: closed

Updating the "Contributing to the Codebase" section to make it more beginner friendly.  It is now in a bullet point format with links and commands needed to contribute.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
@jreback no worries.  I had to make a contribution to an OSS project for school.  I'm not familiar enough with Pandas codebase to contribute bug fixes, but perhaps one day
no worries - have a look at the "good first issue" label if you want something accessible to get started with

If you're familiar with mypy / static typing, then there's lots of work to be done in that space

closing for now then