BrandonFors/pandas-github-issues huggingface dataset:
BUG/PERF: DataFrame.to_records inconsistent dtypes for MultiIndex
State: closed

- [x] closes #47263
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.

Fixes a bug/inconsistency in DataFrame.to_records that produced different numpy dtypes for Index vs MultiIndex. 

**NOTE**: one existing test was updated to reflect the change: `test_to_records_index_name`

The fix is also beneficial in terms of performance:

```
import pandas as pd
import numpy as np

n = 5_000_000

mi = pd.MultiIndex.from_arrays(
    [
        np.arange(n),
        pd.date_range('1970-01-01', periods=n, freq='ms'),
    ]
)

df = pd.DataFrame(np.random.randn(n), index=mi)

%timeit df.to_records(index=True)
```

```
# 6.78 s ± 264 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)      <- main
# 90.9 ms ± 3.03 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR
```


Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Double awesome! Thanks @lukemanley
ENH: TDA fields support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Timedelta
description: Timedelta data type
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
Fix typos
State: closed

Some violations raised by the codespell `pre-commit` hook I noticed during #46718.

- [ ] closes
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks - any idea why the check didn't already pick these up?
@MarcoGorelli The hook's config excludes all files that aren't Python, rST or Markdown and not in `pandas/` or `doc/`:

```yml
-   repo: https://github.com/codespell-project/codespell
    rev: v2.1.0
    hooks:
    -   id: codespell
        types_or: [python, rst, markdown]
        files: ^(pandas|doc)/
```

I happened to have one of my commits aborted by this hook and decided to check _all_ files with it.
You could probably get rid of this rule `files: ^(pandas|doc)/`.
> You could probably get rid of this rule `files: ^(pandas|doc)/`.

does this PR fix all the remaining warnings? If so, do you want to remove it this PR please?
> does this PR fix all the remaining warnings? If so, do you want to remove it this PR please?

Almost. There is 1 remaining:

```md
web/pandas/try.md:9: ines ==> lines
```

https://github.com/pandas-dev/pandas/blob/019574dd4c3f2eea6af62e800f13d40bf0778ded/web/pandas/try.md#L9

Not sure that file is actually used? 2 years old and still says `_(links will be added soon)_`. Either delete file or add

```yml
args: [--ignore-words-list, ines]
```

to hook config.

Could the ignore-regex in

https://github.com/pandas-dev/pandas/blob/4ea3efe8fce08f6974f94212fe8ed16e34b9f965/setup.cfg#L161-L163

be modified to ignore this?
Oh, didn't see `ignore-words-list` was already set in `setup.cfg`.
DOC: cross reference similar `Series.unique` and `Series.drop_duplicates` methods
State: closed

- [x] closes #xxxx (Replace xxxx with the Github issue number)
- [N/A] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [N/A] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [N/A] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Personally, I found `unique()` before I found `drop_duplicates()`, even though the latter is closer to what I wanted to achieve. As discussed in https://github.com/pandas-dev/pandas/issues/1923#issuecomment-515465111, cross-referencing these methods would help with discovering the desired method.

Closes #1923

Labels:
name: Docs

Comments:
Docs successfully build locally:

![Series.drop_duplicates() documentation linking to Series.unique() documentation](https://user-images.githubusercontent.com/247555/172433702-9ae431b4-74c0-4eb4-847b-73cde8cb41f6.png)


![Series.unique() method documentation including a link to Series.drop_duplicates()](https://user-images.githubusercontent.com/247555/172433491-a9636ed9-8ddc-4f9c-a212-76ba7e9fee5d.png)


Thanks @tswast
Error raised message: DataFrame eval will not work with 'Timestamp' column name
State: closed

- [ ] closes #44603 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: expressions
description: pd.eval, query

Comments:
> can you add a whatsnew entry (for 1.5, conversion bug fix section)

Have added a whatsnew entry (for 1.5, conversion bug fix section)
thanks @weikhor this is great!
BUG: pd.Timedelta(big_int, unit=W) silent overflow
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
Thanks @jbrockmendel 
According to the release notes of pandas 1.5.0 

> Operations with [Timestamp](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.html#pandas.Timestamp) or [Timedelta](https://pandas.pydata.org/docs/reference/api/pandas.Timedelta.html#pandas.Timedelta) that would previously raise OverflowError instead raise OutOfBoundsDatetime or OutOfBoundsTimedelta where appropriate ([GH47268](https://github.com/pandas-dev/pandas/issues/47268))

However, `pd.Timedelta(seconds=60) / 1e-9` still raises an OverflowError. Is that intended? From the changelog, it seems this should return a `OutOfBoundsTimedelta` instead. 
> still raises an OverflowError. Is that intended

No.  PR to fix cases we missed would be welcome.
ENH: Add numeric_only to window ops
State: closed

Part of #46560 

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Doh - thanks @mroeschke; test added and implementation fixed for Series case.
Awesome thanks @rhshadrach. Was this the final change for https://github.com/pandas-dev/pandas/issues/46560? 
@mroeschke - I believe so, but going to be reviewing the API manually to double check things before closing that issue. Will report there and manually close if done.
add test to check type of df column after loc()
State: closed

- [x] closes #23326
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hey @mroeschke, I'd like to start writing tests for a lot of the existing GH issues. I am learning as I go, so I am not sure if something like this is sufficient and/or in the right place. Any critical feedback would be greatly appreciated so I can start writing better and more complex tests. Thanks!
Awesome! Definitely check out https://pandas.pydata.org/docs/dev/development/contributing_codebase.html#using-pytest for all the details of how tests ideally should be structured and what should be included
Hello @jackgoldsmith4! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-12 18:18:15 UTC
Thanks @jackgoldsmith4
CI: Allow more permissive HTTP retries/timeouts when connecting to conda
State: closed

Hoping this alleviates the following traceback that has been occurring recently

```
     Traceback (most recent call last):
        File "/usr/share/miniconda/lib/python3.9/site-packages/urllib3/response.py", line 438, in _error_catcher
          yield
        File "/usr/share/miniconda/lib/python3.9/site-packages/urllib3/response.py", line 767, in read_chunked
          chunk = self._handle_chunk(amt)
        File "/usr/share/miniconda/lib/python3.9/site-packages/urllib3/response.py", line 711, in _handle_chunk
          value = self._fp._safe_read(amt)
        File "/usr/share/miniconda/lib/python3.9/http/client.py", line 626, in _safe_read
          chunk = self.fp.read(min(amt, MAXAMOUNT))
        File "/usr/share/miniconda/lib/python3.9/socket.py", line 704, in readinto
          return self._sock.recv_into(b)
        File "/usr/share/miniconda/lib/python3.9/ssl.py", line 1241, in recv_into
          return self.read(nbytes, buffer)
        File "/usr/share/miniconda/lib/python3.9/ssl.py", line 1099, in read
          return self._sslobj.read(len, buffer)
      ConnectionResetError: [Errno 104] Connection reset by peer
  
      During handling of the above exception, another exception occurred:
  
      Traceback (most recent call last):
        File "/usr/share/miniconda/lib/python3.9/site-packages/requests/models.py", line 760, in generate
          for chunk in self.raw.stream(chunk_size, decode_content=True):
        File "/usr/share/miniconda/lib/python3.9/site-packages/urllib3/response.py", line 572, in stream
          for line in self.read_chunked(amt, decode_content=decode_content):
        File "/usr/share/miniconda/lib/python3.9/site-packages/urllib3/response.py", line 793, in read_chunked
          self._original_response.close()
        File "/usr/share/miniconda/lib/python3.9/contextlib.py", line 137, in __exit__
          self.gen.throw(typ, value, traceback)
        File "/usr/share/miniconda/lib/python3.9/site-packages/urllib3/response.py", line 455, in _error_catcher
          raise ProtocolError("Connection broken: %r" % e, e)
      urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
```


Labels:
name: CI
description: Continuous Integration

Comments:
Failure will be fixed by https://github.com/pandas-dev/pandas/pull/47259
TST: Fix test warnings 
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Beat me to it. The only one in my branch worth porting is in tests/frame/test_constructors.py

```
-        df.iloc[:, 2] = pd.array([45, 46], dtype=c.dtype)
+        with tm.assert_produces_warning(FutureWarning, match="will attempt to set"):
+            df.iloc[:, 2] = pd.array([45, 46], dtype=c.dtype)
         assert df.dtypes.iloc[2] == c.dtype
```
Failure will be fixed by https://github.com/pandas-dev/pandas/pull/47259
cool
ENH: Provide the error message from the HTTP server when return the HTTPError, #47242
State: closed

ENH: Provide the error message from the HTTP server when return the HTTPError, #47242

Signed-off-by: Andrew Li <orcahmlee@gmail.com>

- [x] closes #47242 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
> I would be -1 on this change. For `urllib.error.HTTPError`, the exception object contains more attributes than just the reason (code, reason, headers). So I think it's more sensible for the user to catch and log/handle as appropriate.

Hi @mroeschke , thanks for your advice.

I did some changes. I only appended the error message from the server and re-raise the `urllib.error.HTTPError`.

The user would see
```sh
urllib.error.HTTPError: HTTP Error 400: Bad Request, message from server: <MORE DETAILED MESSAGES> 
```
instead of 
```sh
urllib.error.HTTPError: HTTP Error 400: Bad Request
```

Also, the users are still able to catch the `urllib.error.HTTPError` if they want to access the attributes of the `HTTPError`.

Please give me some advice on this approach.
> > I would be -1 on this change. For `urllib.error.HTTPError`, the exception object contains more attributes than just the reason (code, reason, headers). So I think it's more sensible for the user to catch and log/handle as appropriate.
> 
> Hi @mroeschke , thanks for your advice.
> 
> I did some changes. I only appended the error message from the server and re-raise the `urllib.error.HTTPError`.
> 
> The user would see
> 
> ```shell
> urllib.error.HTTPError: HTTP Error 400: Bad Request, message from server: <MORE DETAILED MESSAGES> 
> ```
> 
> instead of
> 
> ```shell
> urllib.error.HTTPError: HTTP Error 400: Bad Request
> ```
> 
> Also, the users are still able to catch the `urllib.error.HTTPError` if they want to access the attributes of the `HTTPError`.
> 
> Please give me some advice on this approach.

I still believe that this change is not significant enough to warrant a change in pandas. This exception is wholly provided by the Python stdlib and not necessarily pandas; therefore, HTTPError messaging should be improved in the stdlib. 
> I still believe that this change is not significant enough to warrant a change in pandas. This exception is wholly provided by the Python stdlib and not necessarily pandas; therefore, HTTPError messaging should be improved in the stdlib.

OK, I see. Thanks again for your review.
PERF: delta_to_nanoseconds
State: closed

```
from datetime import timedelta
from pandas._libs.tslibs.timedeltas import *

td = timedelta(days=4)

%timeit delta_to_nanoseconds(td)
1.74 µs ± 25.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- main
300 ns ± 3.38 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- PR
265 ns ± 7.28 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- 1.4.2
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Timedelta
description: Timedelta data type

Comments:
Thanks @jbrockmendel
ENH: Move NumExprClobberingError to error/__init__.py per GH27656
State: closed

- [x] xref #27656. this GitHub issue is being done in multiple parts
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Looks like there are docbuild failures

```
2022-06-06T02:29:12.9065297Z /home/runner/work/pandas/pandas/doc/source/reference/testing.rst:64:<autosummary>:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
2022-06-06T02:29:12.9067287Z /home/runner/work/pandas/pandas/pandas/errors/__init__.py:docstring of pandas.errors.NumExprClobberingError:2: WARNING: Inline interpreted text or phrase reference start-string without end-string.
```
> Looks like there are docbuild failures
> 
> ```
> 2022-06-06T02:29:12.9065297Z /home/runner/work/pandas/pandas/doc/source/reference/testing.rst:64:<autosummary>:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
> 2022-06-06T02:29:12.9067287Z /home/runner/work/pandas/pandas/pandas/errors/__init__.py:docstring of pandas.errors.NumExprClobberingError:2: WARNING: Inline interpreted text or phrase reference start-string without end-string.
> ```

Ah yeah, my bad, illegal use of bad tick lol. I've fixed it. Thank you for the feedback
Making great progress on that issue, thanks again!
CI/TYP: enable reportGeneralTypeIssues for subset of files
State: closed

Second attempt at enabling pyright's most important type checking rule (reportGeneralTypeIssues). Instead of having many file-based ignores as in  #44855, this PR adds a second pyright configuration file just for reportGeneralTypeIssues and lists all excluded file in this one config file.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
If we encounter pyright errors in the future that are difficult to fix but not reported by mypy, we have a few options:

1. add the entire file to the exclude list
2. add a file-based comments `# pyright: reportGeneralTypeIssues = false`
3. add a comment at the offending line `# pyright: ignore` or `# pyright: ignore [reportGeneralTypeIssues]`
4. if it is a mistake in pyright: report it and there is probably a release within less than a week with the fix (the release cycle of pyright is amazingly fast)

As long as the exclude list is long, I would simply add the entire file to the exclude list. Once the exclude list becomes reasonably small, we could transition to using line-based comments (currently we would need to add ~650 line-based comments) and merge the two pyright configurations (pyproject.toml and pyright_reportGeneralTypeIssues.json).
Thanks @twoertwein. Would be great to create an issue to tackle the typing issues in the config file
Add type annotations to the PR checklist
State: closed

It would be great to have an automated check that new arguments/functions/methods have to have type annotations but I'm not sure whether we can enforce that easily. Let's start by encouraging type annotations through the PR template :)

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@Dr-Irv Should probably also add a comment to nudge people to submit follow-up PRs to [pandas-stubs](https://github.com/pandas-dev/pandas-stubs) to add new/changed arguments/methods/functions.
> @Dr-Irv Should probably also add a comment to nudge people to submit follow-up PRs to [pandas-stubs](https://github.com/pandas-dev/pandas-stubs) to add new/changed arguments/methods/functions.

prob need to add to the typing doc-section about this, but +1
thanks @twoertwein 
> > @Dr-Irv Should probably also add a comment to nudge people to submit follow-up PRs to [pandas-stubs](https://github.com/pandas-dev/pandas-stubs) to add new/changed arguments/methods/functions.
> 
> prob need to add to the typing doc-section about this, but +1

Totally agree.  Still need to write up some docs about how to work with the `pandas-stubs` repo.
ENH: Add check_like functionality to assert_series_equal
State: closed

- [x] closes #47247
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @stevenschaerer! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-07 08:16:34 UTC
Awesome, thanks @stevenschaerer
DEPR: groupby.transform not aligning with input index
State: closed

- [x] closes #45648 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel 

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @jreback - changes made
lgtm merge on green
thanks @rhshadrach