BrandonFors/pandas-github-issues huggingface dataset:
ENH: allow start=range_object in RangeIndex constructor
State: closed

- [x] closes #12067
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

simplifies `Index.__new__` a bit

Labels:
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
hah I originally implemented this but then we settled on `.from_range`. I would be +1 in using this. cc @jorisvandenbossche 
I'm +1 on this change. If we call something RangeIndex, it's reasonable that it takes a range in its constructor IMO.
(no strong opinion here)
lgtm. I think ok to also deprecated `.from_range`
In the RangeIndex class doc stringm in the `Parameters` section, I think the line `start : int (default: 0), or other RangeIndex instance` should be changed to `start : int (default: 0), range or other RangeIndex instance`
> In the RangeIndex class doc stringm in the Parameters section, I think the line start : int (default: 0), or other RangeIndex instance should be changed to start : int (default: 0), range or other RangeIndex instance

good idea.  will add this to my upcoming core.indexes TYP branch if thats OK
thanks @jbrockmendel 

followups when you can
CLN: Remove to_xml from format.py to run directly in frame.py
State: closed

- [X] refereces #40131
- [X] tests passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry (not necessary)


Labels:
name: Clean
name: IO XML
description: read_xml, to_xml

Comments:
thanks!
PERF: enable nullable integer for read_sql via parameter
State: closed

Function `pandas.read_sql` read SQL query or database table into a DataFrame. However, for nullable integer columns, once there exists one row with null, pandas will infer this column as float, which may lead to subsequent issues (in our case, we use DataFrame for ETL and the integer converted to float, not suitable for our sink database table schema).
Using nullable integer is one of the solution we adopted, which may cause more memory usage. Therefore, I add default False parameter for `pandas.read_sql` to not enable it as default, once you want to enable it, just set it as True.

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
Hello @zhengfeiwang! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-05 03:26:58 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@zhengfeiwang Can you implement this for all nullable dtypes(IntegerArray, BooleanArray, FloatingArray, and StringArray) as the parameter ```use_nullable_dtypes``` and resolve conflicts?
> @zhengfeiwang Can you implement this for all nullable dtypes(IntegerArray, BooleanArray, FloatingArray, and StringArray) as the parameter `use_nullable_dtypes` and resolve conflicts?

OK, I will try to work on it and finish that ASAP
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
I will finish this perf ASAP, please hold this thread currently.
> I will finish this perf ASAP, please hold this thread currently.

Thanks. @zhengfeiwang . will close to clear the queue. ping when ready to continue.
DEPR: Partial failure in Series.transform and DataFrame.transform
State: closed

- [x] closes #40211
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

In test_frame_transform.test_transform_partial_failure, two of the tests did not actually test for partial failure as I had intended back in #35964. That is fixed here. Also added tests for Series.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
Backport PR #40143 on branch 1.2.x (REGR: reduction operations failing if `min_count` is larger)
State: closed

Backport PR #40143: REGR: reduction operations failing if `min_count` is larger

Labels:
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
> pandas/core/nanops.py:1373: error: Invalid type comment or annotation  [valid-type]

:-< @simonjayhawkins 
we are using an older mypy on 1.2.x so this is expected. will fix up shortly.
DEPS: skip html tests if html5lib is not installed #40198
State: closed

- [x] closes #40198



Labels:
name: CI
description: Continuous Integration
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Dependencies
description: Required and optional dependencies

Comments:
looks like other tests are missing html5lib
thanks @fangchenli and @lithomas1 
@meeseeksdev backport 1.2.x
Check the that SQLAlchemy ResultProxy returns rows before fetching rows
State: closed

- [X] closes #40205 
- [ ] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Stale

Comments:
@jreback to be honest, I'm not totally sure how to go about writing a test for this. I can't think of a command in vanilla Postgres that returns nothing off the top of my head. Even an `INSERT` or `COPY` will return the number of rows (and it looks like `INSERT` is already being tested). `UNLOAD` is Redshift-specific. Any ideas on how to structure a test for this?

I'm also having trouble getting the existing tests to pass locally. I'm using the supplied Dockerfile to create my dev environment, with the only change I made being that I added `psycopg2` to the `environment.yml` so that I could actually connect to my Redshift db. 
![image](https://user-images.githubusercontent.com/29689587/110144948-27ba0180-7d96-11eb-987c-683986239b49.png)

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR but it appears as if this PR has gotten stale. Additionally, pd.`read_sql` is designed to read in data to a `DataFrame` and not designed to execute generic database transactions. So going to close as I think supporting this use case is out of scope.
PERF: BlockPlacement construction
State: closed

xref https://github.com/pandas-dev/pandas/pull/40171#issuecomment-790219422 when disabling fast_apply, this cuts the linked benchmark by 13%

Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
updated with an implementation of `BlockManager._simple_new` that brings the benchmark gains to 19-22%
> pandas/core/internals/managers.py:817: error: Argument 1 to "_simple_new" of "BlockManager" has incompatible type "Tuple[Any, ...]"; expected "Tuple[Block]"  [arg-type]
> pandas/core/internals/managers.py:817: error: Argument 1 to "_simple_new" of "BlockManager" has incompatible type "Tuple[Any, ...]"; expected "Tuple[Block]" [arg-type]

addressed + greenish
the numpy_dev failure could be related
> the numpy_dev failure could be related

im pretty sure that is from #39953 (joris commented on it and im assuming he was correct)
ok cool
STYLE: Inconsistent namespace - arithmetic (#39992)
State: closed

- [x] xref #39992 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @01-vyom
STYLE: Inconsistent namespace - api
State: closed

- [x] xref #39992 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @01-vyom
TST: use env variable instead of pytest option for testing ArrayManager
State: closed

I noticed a problem with the current `--array-manager` pytest option to test the ArrayManager, while checking the window tests. 

The problem with the option is that `conftest.py` files are read first, *before* handling the option in `pytest_sessionstart`. This means that any DataFrame or Series that gets constructed on *import* of a `conftest.py` file, has already been created before the setting is changed, and will thus be created with the wrong manager. 
Example of this is a DataFrame that is not created *inside* a (fixture) function, but created top-level in the conftest.py file (or eg used in parametrization). 

Luckily we don't have too many of those, but one example in window/conftest.py:

https://github.com/pandas-dev/pandas/blob/74a68980d116c843d04b3ccb6ab9368f19637d9b/pandas/tests/window/conftest.py#L352-L367

or one example in the toplevel conftest.py:

https://github.com/pandas-dev/pandas/blob/74a68980d116c843d04b3ccb6ab9368f19637d9b/pandas/conftest.py#L597-L608

So to avoid needing to rewrite those (and continue to ensure such cases don't get added), I don't see another way as using the environment variable approach instead of the pytest option (although the option is much easier to use interactively ..)

cc @jreback 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> Luckily we don't have too many of those, but one example in window/conftest.py:

should we just change / check for these?
> should we just change / check for these?

I think that's a bit fragile to ensure that in the future (and in theory there is also nothing wrong with it, just not working with this option that influences the dataframe creation). Would there be an easy way to check this?
> > should we just change / check for these?
> 
> I think that's a bit fragile to ensure that in the future (and in theory there is also nothing wrong with it, just not working with this option that influences the dataframe creation). Would there be an easy way to check this?

agree its fragile, but would be good to check it, yeah not sure its easy.
If someone comes up with a good approach for keeping it with the pytest option, we can always change back. But for now I want to ensure I am correctly testing ArrayManager ;)
STYLE: Inconsistent namespace - arrays #39992
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Linking: https://github.com/pandas-dev/pandas/issues/39992
@thomasyu888: Welcome to `pandas` !

CI notwithstanding, LGTM!
Thanks @thomasyu888
STYLE: Inconsistent namespace - resample #39992
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Linking: https://github.com/pandas-dev/pandas/issues/39992
CI notwithstanding, LGTM!
Thanks @thomasyu888
BUG: update should try harder to preserve dtypes #4094
State: closed

- [ ] closes #4094
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I have to admit, I am not quite sure about the code in the [initial example](https://github.com/pandas-dev/pandas/issues/4094#issue-16212879). But the rest of the given examples ([1](https://github.com/pandas-dev/pandas/issues/4094#issuecomment-297671925), [2](https://github.com/pandas-dev/pandas/issues/4094#issuecomment-583174514)) are converted to tests and they do pass.

I would have loved to make better use of hypothesis in the tests, but currently [column](https://hypothesis.readthedocs.io/en/latest/numpy.html#hypothesis.extra.pandas.columns) does not support dtype strategies and neither does [from_dtype](https://hypothesis.readthedocs.io/en/latest/numpy.html#hypothesis.extra.numpy.from_dtype) (if I would define the elements of the column and infere the dtype).

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@Horstage closing as stale. ping when ready to continue
STYLE: Inconsistent namespace - scalar #39992
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Linking: https://github.com/pandas-dev/pandas/issues/39992
Thanks @thomasyu888 for doing these - let's leave the rest for other new contributors, but there's plenty of other issues to work on
Thanks @MarcoGorelli!  Excited to contribute more as I use this package quite often - sorry if I did too many!
[ArrayManager] TST: activate a bunch of (passing) tests
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146/

This is on top of #40152, so only the second commit is relevant: https://github.com/pandas-dev/pandas/pull/40217/commits/e7209aaa7b85c206d3efcd446eaa54b34851cb98

But this basically adds the following to CI for ArrayManager:

```
        pytest pandas/tests/api/ --array-manager
        pytest pandas/tests/base/ --array-manager
        pytest pandas/tests/computation/ --array-manager
        pytest pandas/tests/config/ --array-manager
        pytest pandas/tests/dtypes/ --array-manager
        pytest pandas/tests/generic/ --array-manager
        pytest pandas/tests/indexes/ --array-manager
        pytest pandas/tests/libs/ --array-manager
        pytest pandas/tests/plotting/ --array-manager
        pytest pandas/tests/scalar/ --array-manager
        pytest pandas/tests/strings/ --array-manager
        pytest pandas/tests/tools/ --array-manager
        pytest pandas/tests/tseries/ --array-manager
        pytest pandas/tests/tslibs/ --array-manager
        pytest pandas/tests/util/ --array-manager
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
LGTM (just looked at the second commit)

for testing locally, is there a simple command i can use instead of copy/pasting the entire list of files?
> is there a simple command i can use instead of copy/pasting the entire list of files?

What do you mean exactly? To only run the tests that are already activated on CI? 
> What do you mean exactly? To only run the tests that are already activated on CI?

yah, ATM when i want to run the relevant ArrayManager tests i do `pytest pandas/tests/foo panda/test/bar pandas/tests/baz [...] --array-manager` and its starting to involve a lot of copy/pasting (which is a good thing!)

can you rebase
> yah, ATM when i want to run the relevant ArrayManager tests i do `pytest pandas/tests/foo panda/test/bar pandas/tests/baz [...] --array-manager` and its starting to involve a lot of copy/pasting (which is a good thing!)

Yeah, that's a little bit annoying. Most of the time I am focusing on tests in a single directory, and then that's OK. But we're getting closer to get all tests running ;)
BUG: read_excel skips single-column empty rows
State: closed

Ref #39808

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This is the fix proposed in the discussion of #40095

Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
Not sure if this closes the original issue, as the parameter would have to be exposed in order to truly fix. 
I think modifying this function https://github.com/pandas-dev/pandas/blob/fad96e1d3145019a5d865564bb4db008a1b1e5ea/pandas/io/parsers/python_parser.py#L596 might be able to fix the issue, but it could also introduce some side effects for CSV parsing. 
@lithomas1 I'm nervous about breaking the CSV behaviour, because it seems deliberate. I found one test where an input line `,,` is supposed to produce a row of NaNs:

https://github.com/pandas-dev/pandas/blob/154026cc7104f7b58c10a1557564eccd0f71c4a2/pandas/tests/io/parser/common/test_common_basic.py#L412-L414

Maybe this behaviour should be documented better?
@simonjayhawkins @jreback - This is a bit of an odd case, wanted to get some eyes to make sure it's being handled appropriately. This PR fixes the following bug (not a regression, AFAICT) on master: when reading an excel file, blank rows would be skipped if and only if a single column is being read. This PR makes it so that blank rows are never skipped.

However, the previous workaround was to supply `skip_blank_lines=False` in `**kwds`; this argument was removed in 1.1 and so is no longer available. I'm -0 on adding `skip_blank_lines` back in (see https://github.com/pandas-dev/pandas/pull/40214#pullrequestreview-605859867).

Since this is a rather localized case, my thinking is to have this be a bugfix for 1.3 and leave the regression in 1.1.x and 1.2.x unresolved.
> Since this is a rather localized case, my thinking is to have this be a bugfix for 1.3 and leave the regression in 1.1.x and 1.2.x unresolved.

sgtm. does the OP need to change to not close the issue?
@simonjayhawkins, thanks for taking a look at this. My preference is to close #39808 based on

- the OP of https://github.com/pandas-dev/pandas/issues/39808#issuecomment-784366880 prefers the current solution now that we understand how `skip_blank_lines` really works
- IIUC, @rhshadrach and I don't think `skip_blank_lines` should be added to `read_excel` in the future

But I'm happy to edit the PR if everyone prefers to leave #39808 open.
> * IIUC, @rhshadrach and I don't think `skip_blank_lines` should be added to `read_excel` in the future

The issue could be closed once that is agreed as they way forward on the issue itself.
for 1.3
Thanks @ahawryluk! I've left the issue open for now, but will put my thoughts there shortly.
TST/CLN: Move raising apply tests to test_invalid_arg
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
BUG: constructing DTA/TDA from xarray/dask/pandasarray
State: closed

- [x] closes #24539
- [x] closes #24615
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
lgtm, is this user facing at all? or *only* library facing? e.g. seems now can construct ```pd.to_datetime(pd.Series(...., dtype='Int64')))``` might work now?
> e.g. seems now can construct pd.to_datetime(pd.Series(...., dtype='Int64'))) might work now?

this works on master.

> is this user facing at all? or only library facing?

AFAICT only library-facing.
looks like a mypy complaint crept in.  updated #40361 to fix it
DOC: Fix Examples section for some window methods
State: closed

Some of the example sections were not rendering correctly after a doc refactor I did a while back:

https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.core.window.rolling.Rolling.mean.html


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
sure, doc build failing :-<
thanks @mroeschke
DOC: Add Notes and Examples back to GroupBy.apply docs template
State: closed

- [X] xref #37875 (see https://github.com/pandas-dev/pandas/issues/37875#issuecomment-790026815)
- [ ] tests added / passed (n/a)
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry (n/a)


Labels:
name: Docs
name: Groupby

Comments:
I think the remaining check failures are unrelated.
thanks @TheNeuralBit 

pls check https://pandas.pydata.org/pandas-docs/dev/ after this builds for correctness
Thanks @jreback. The [page](https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.core.groupby.GroupBy.apply.html) LGTM:
![image](https://user-images.githubusercontent.com/675055/109998315-f70a9700-7cc5-11eb-91d0-7e3e3e3fdbc2.png)
DOC: Update read_hdf's acceptable path types
State: closed

The docstring for `read_hdf` claims it can read from URLs and file-like objects. Neither of these are supported by the underlying `PyTables` library.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
yeah this was just copy-paste error a while back
REF: remove block access in groupby libreduction Series(Bin)Grouper
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

The `Series(Bin)Grouper` still has some direct block access, which needs to be resolved to let it work for both Block and ArrayManager

Labels:
name: Groupby
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
typing issue on ci / checks
It's the failure that was failing on master yesterday. Will merge master to be sure.
REF: split out to_datetime_with_unit, _to_datetime_with_format
State: closed

The idea is that to_datetime_with_format will eventually be moved into objects_to_datetime64ns, and most of _convert_listlike_datetimes will just be a call to sequence_to_dt64ns

Labels:
name: Refactor
description: Internal refactoring of code
name: Datetime
description: Datetime data dtype

Comments:
doc failure looks like network; ping on green.
hmm can you merge master