BrandonFors/pandas-github-issues huggingface dataset:
Added orient="ndarray" to DataFrame.to_dict()
State: closed

It would be useful to allow Series to be directly converted to numpy arrays instead of having to do this outside.


Labels:

Comments:
Hello @LarsKue! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-08 20:48:26 UTC
is this really that useful, its only a one-line transformation?

```
df = pd.DataFrame({'a': [1,2], 'b': [3,4]})
dict_arr = {k: np.array(v) for k, v in df.to_dict(orient='list').items()}
```
> is this really that useful, its only a one-line transformation?

If you consider this not useful, the entire `to_dict` method should also be considered not useful, since it does exactly this for other types.

Edit: Analogous outside code is
```
df = pd.DataFrame({'a': [1,2], 'b': [3,4]})
dict_arr = {k: v.to_numpy() for k, v in df.items()}
```

To me it seems natural `to_dict` should be able to convert to `np.ndarray` since `pandas` otherwise has great integration for `numpy` arrays, etc.
`dict(my_dataframe)` does this.
> `dict(my_dataframe)` does this.

You are right, good to know. I could not find this in the docs. Closing this PR.
> `dict(my_dataframe)` does this.

No longer true.
GH39992 update inconsistent namespace usage for reductions
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Awesome, thanks @sionedbaker !
[ArrayManager] Enable read_parquet to not create 2D blocks when using ArrayManager
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

I was exploring the Parquet IO, and pyarrow has an option to not created consolidated blocks. If we do this when wanting to create an ArrayManager, we can reduce the memory usage. It's a bit slower, though, because there is still the overhead of creating more blocks (that's something that would need to be changed in pyarrow).

Would still need to add a test that checks the option is honored.


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: IO Parquet
description: parquet, feather

Comments:
small question, LGTM
PERF: get_dummies create array of 0/1s with appropriate layout for internals
State: closed

This ensures that the 2D ndarray that is created in `get_dummies` for a single column is directly in the "right" layout as the internals prefer it (column-major, so it's row-major in the transposed Block.values).

For ArrayManager, this gives a speed-up for `get_dummies` itself (because we still copy the arrays to ensure proper contiguous memory layout for each column, currently):

```
In [7]: %timeit pd.get_dummies(s, sparse=False)
23.2 ms ± 1.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- master
16.6 ms ± 689 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR
```

For the current BlockManager, it makes the actual get_dummies a bit slower, but can make a follow-up operation actually faster (where a copy, or consolidation, or a reduction, or .. would happen):

```
In [4]: %timeit pd.get_dummies(s, sparse=False)
10.7 ms ± 397 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- master
14.2 ms ± 713 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR

In [5]: %timeit pd.get_dummies(s, sparse=False).copy()
22.3 ms ± 260 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- master
17.8 ms ± 1.92 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR
```

The timing is from the following example taken from the `frame_methods.py::GetDummies` ASV case:

```
import string

categories = list(string.ascii_letters[:12])
s = pd.Series(
    np.random.choice(categories, size=1000000),
    dtype=pd.CategoricalDtype(categories),
)
```

So it makes the actual `get_dummies` a bit slower for the BlockManager, and thus I could also explicitly only do this for the ArrayManager. But since it improves follow-up operations, I think it can actually be beneficial overall for BlockManager as well. But either is fine for me.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jreback any further feedback here?
Thanks!
> So it makes the actual `get_dummies` a bit slower for the BlockManager, and thus I could also explicitly only do this for the ArrayManager. But since it improves follow-up operations, I think it can actually be beneficial overall for BlockManager as well. But either is fine for me.

https://pandas.pydata.org/speed/pandas/#reshape.GetDummies.time_get_dummies_1d?python=3.8&Cython=0.29.21&commits=b8a71f33-154096f2
[ArrayManager] TST: enable IO tests
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

This basically just adds IO testing to the set of tests we run for ArrayManager. I already added skips for JSON in the past (and now also for PyTables IO and pickle), so those are still skipped now as well.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Some failing SQL tests use `from_records` and will be fixed by https://github.com/pandas-dev/pandas/pull/40301
looks like array manager tests are reporting failures?
Yes, there are still a few errors I need to skip (parquet test outside of test_parquet.py). But the strange thing is that the github action kept running, although the output shows it finished (6 failed, 7616 passed, 2180 skipped, 280 deselected, 23 xfailed, 3 xpassed in 192.71s (0:03:12)). But after the IO tests, it never continued with the next tests ..
Now it's passing
Going to merge, since this is just adding some skips for ArrayManager
PERF: declare contiguity in libgroubpy
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
this provides the speedup yes?
> this provides the speedup yes?

yes, ported from the \<update\>discussion in\</update\> 1d-version PR and applied to everywhere applicable in this file
thanks!
TYP: Window code
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: Window
description: rolling, ewma, expanding
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Happy to have a subsequent PR will follow up comments.
DOC: fix grammar issues and incorrect link in "Getting Started" guide…
State: closed

… (#40215)

- [x] closes #40215
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @rlukevie
DEPR: Series/DataFrame.transform partial failure except TypeError
State: closed

- [x] closes #40211
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Partial revert of #40238. I did not realize how `TypeError` was handled differently than other errors in other UDF functions (https://github.com/pandas-dev/pandas/issues/40211#issuecomment-792030597). The deprecation is now only for partial failure when something other than TypeError is raised.

In order to make this work, transform needed to passthrough TypeErrors when they occur so we can correctly decide how to act with a list- or dict-like.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
STYLE: Resolved Inconsistent namespace
State: closed

- [x] closes #39992 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi @deepang17 - instead of excluding a file, can we change `test_unique_tuples` so the last argument of the function is `uniques` instead of `unique`?
Sure, will do that.
Thanks @deepang17 , I'll just mark this as draft pending on https://github.com/pandas-dev/pandas/issues/39992#issuecomment-792294708 , will then reopen (depending on what we hear back, you may need to rebase / fix conflicts)
Hi @deepang17 - #40310 is in, do you want to rebase and fix up any remaining errors so we can get this in?
Hello, @MarcoGorelli I have completed work from my side. If you think that changing from `unique` to `uniques` is not a breaking change then this works fine.
You'll also need to change it to `uniques` in
```
    @pytest.mark.parametrize(
        "arr, unique",
```

It's just a variable in a test, so it's not a breaking change

If you could fetch and merge upstream/master and fixup "unique" in `pytest.mark.parametrize` then this should be good to go
Ok, will do that.
![image](https://user-images.githubusercontent.com/47976918/110361148-a0a2ad00-8065-11eb-89e1-e5996f7fbbb3.png)

@MarcoGorelli these new files are producing the error.

![image](https://user-images.githubusercontent.com/47976918/110361228-bdd77b80-8065-11eb-84c6-0d5da79597e6.png)
Shall I convert them to arrays?
I'd suggest renaming to `arr` rather than `arrays`, as here there's only one of them (`uniques` in `test_algos.py` is fine as there's many objects there anyway many objects and arguably might have been better named `uniques` from the start)
`arr` is used at quite a few places, so it will again create conflicts. Please suggest something else. Thank you.
@MarcoGorelli it's ready to be merged. Thank you for your guidance.
Hi @deepang17 - can you fetch and merge upstream/master please?
Great, at that time some other variables were also used with the name `arr` and one of them produced an error for me hence I thought it will produce an error for all. But now as you have done it, it seems fine.
Thanks @deepang17 !
API: remove `.set_tooltips_class` incorpate to `.set_tooltips`
State: closed

`.set_tooltips` and `.set_tooltips_class` are introduced to 1.3.0.

Just realised it seems a bit pointless to have two api functions:

- [x] remove `.set_tooltips_class()` and `._init_tooltips()`
- [x] include the `css_name` and `props` arguments in `.set_tooltips` for same functionality.
- [x] no api break since these functions have not yet been released
- [x] make argument names more consistent with other Styler methods.

Labels:
name: Clean
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68 very nice
STYLE: Inconsistent namespace - tools (pandas-dev#39992)
State: closed

- [x] xref #39992
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @deepang17 !
CI: avoid port-already-in-use errors
State: closed

Should prevent the occasional CI failures e.g. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=55705&view=ms.vss-test-web.build-test-results-tab&runId=2644226&resultId=178989&paneView=debug

Labels:
name: CI
description: Continuous Integration

Comments:
cool
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=55858&view=logs&j=b516280a-8c75-541b-8ba0-ae13561718a7&t=88a240e0-0f0f-5fd5-6f10-d82e025ed5d5

I think this is why these were hard coded in the first place.

let's revert this CR.
DOC: Clarify DataFrame.combine_first and Series.combine_first
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @HeidiNeuhaeuser! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-07 19:40:27 UTC
thanks @HeidiNeuhaeuser if you'd like to make a shared doc-string as suggested by @MarcoGorelli pls open a new PR
Move cpplint check to pre-commit
State: closed

Currently, cpplint runs from https://github.com/pandas-dev/pandas/blob/master/ci/code_checks.sh
However, cpplint have a pre-commit hook which can be used.

- [x] closes https://github.com/MarcoGorelli/pyladies-sprint-March-2021/issues/2
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MarcoGorelli I've changed the order and rebased now
Reorder pre-commit-config and normalize id names
State: closed

- [ ] closes https://github.com/MarcoGorelli/pyladies-sprint-March-2021/issues/21
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry (not sure where to add this?)


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @mgmanzella
STYLE: Inconsistent namespace - plotting
State: closed

- [x] xref #39992
- [x] tests added / passed - plotting
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @AnglinaBhambra
Add examples to documentation for DataFrame.swaplevel
State: closed

- [x ] closes https://github.com/MarcoGorelli/pyladies-sprint-March-2021/issues/13
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @thordisstella! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-11 20:58:01 UTC
> Really nice example @thordisstella , thanks, have just left some minor comments

Thanks for the review @MarcoGorelli! I addressed the comments, but when I ran `black` on the changes I'd made, nothing further was updated (after I'd addressed your other two comments). Please let me know if I've missed something here
Hey @thordisstella , thanks for updating (and happy IWD!)! Sorry, I was unclear - `black` indeed won't modify what you've written, because it doesn't format code inside docstrings (barring some very minor things). I was suggesting to reformat the docstring the way that `black` would - one way to do this would be to paste the docstring into a temporary file (say, `t.py`):
```python
# t.py

"""
        >>> df = pd.DataFrame({'Grade':['A','B','A','C']},
        ...                   index=[['Final exam', 'Final exam', 'Coursework',
        ...                              'Coursework'],
        ...                         ['History','Geography','History','Geography'],
        ...                         ['January','February','March','April']])
"""
```
, running
```python
pip install blackdoc
blackdoc t.py --line-length 80
```
and then pasting the reformatted version back into this docstring of `swaplevel`

Finally, you should run
```console
$ python scripts/validate_docstrings.py pandas.DataFrame.swaplevel
```
and check it doesn't throw any extra errors compared to when run on `master` (in this case you prob. just need to reformat the code in the example and remove the extra blankline at the end of the docstring)
Thanks, and belated happy IWD to you too! 😊 

Thanks for the clarification - I've updated the PR as you suggested so hopefully we should be good now 🤞  
Hi @thordisstella - looks good, CI failures should be unrelated, could you fetch and merge upstream/master please?
Thanks for the review, all the advice and for merging this @MarcoGorelli ! And apologies for the delay in reacting to your last comment. I'll take a look at the follow up PR you suggested either later this week or next week :)
Remove pd_array from tests
State: closed

- [x] closes https://github.com/MarcoGorelli/pyladies-sprint-March-2021/issues/7
- [x] tests passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks for looking @MarcoGorelli !

As for the changes, I understand the one for the pre-commit, but can you explain what `stdlib_array` is and where it is supposed to live? I do not see where it is defined.

When I do the change as you mention, I get this error:
`ImportError: cannot import name 'stdlib_array' from 'array'`
Hey @olgarithms - sorry, I mistyped, I mean to write `from array import array as stdlib_array` (`array` is a standard library module)
Thanks @olgarithms
CLN: BlockManager.get_slice require only slice arg
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

non-slice objects are currently getting past mypy.  This ensures we only get slices, will open up some optimization options.

Also did some docstring/typing cleanup in the neighborhood.

<b>update</b>: updated to implement Index._getitem_slice.  Cuts about 10% off of the benchmark discussed in https://github.com/pandas-dev/pandas/pull/40171#issuecomment-790219422 (as compared to current master, not then-master)

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> this is quite a lot of added code paths in index, are these really all tested?

yes
ok looks fine. we have sufficient asv's to cover? (assume we do)
> we have sufficient asv's to cover? (assume we do)

yes.  this is motivated by the one discussed in joris's libreduction PR
lgtm. can you rebase & ping on green.
Ping
CLN: Remove _axis from apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

The regex `[\W]_axis` is showing this removes the only occurances of the variable `_axis` throughout all of `pandas.core`. It is only set in groupby in a branch where is must be 0, and is only used to check that it is 0.

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
Fixning-Doc-Typos
State: closed

Docs: changes the documentation

Fix docs typos and small grammatical  update 


Labels:

Comments:
Hi @mohamed-khoualed - instead of repeatedly (cf #39933) opening PRs which introduce new grammatical errors, why not try tackling on of the many [good first issue](https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)s? See [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html) for how to contribute, and feel free to reach out on Gitter if you have trouble getting set up

I promise it'll be a more rewarding experience for you :+1:
REF: simplify operating-columnwise dispatch
State: closed

After this we have one usage of split_and_operate left and 2 places where we can _almost_ use `@maybe_split`.  In both cases I'm still trying to untangle things.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
> rebase after we merged the other one, but have some questions here

rebasd + green.  questions?
updated to re-write and re-use split_and_operate
rebase, but need to relook
rebased, green ex two issues on master being addressed elsewhere
ok improves on existing (tangled code), splitting block ops not very clear where its happening, but I am sure future PRs can help address.
PERF/CLN: restrict _isna_ndarraylike to actual ndarray/EA
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

I am looking at some `isna` related benchmarks, and noticing a lot of overhead related to checking of types. This makes `_isna_ndarraylike` stricter to only handle ndarray and EA, and not Series/Index (and therefore also renamed it to `_isna_array`).

Reproducer from the `frame_methods.py::Isnull` benchmark:

```python
N = 10 ** 3
sample = np.array([np.nan, 1.0])
data = np.random.choice(sample, (N, N))
df = DataFrame(data)
df_am = df._as_manager("array")
```

```
In [3]: %timeit df_am.isna()
5.03 ms ± 125 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- master
4.25 ms ± 9.94 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR
```

This gives a small improvement. But it further allows to directly use `_is_array` instead of the general `isna` from Manager.isna, which will give further improvements (but was planning to do this as a follow-up)

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Performance
description: Memory or execution speed performance
name: Clean

Comments:
LGTM pending green
BENCH: add DataFrame.reindex(columns=..) benchmark that selects existing columns
State: closed

I noticed that the current `df.reindex(columns=idx)` benchmark uses an indexer with all-non-existing indices (it uses `range(4000, 7000)` as indexer, but there are only 1000 columns), so basically it is benchmarking to create empty (all-NaN) columns. Therefore added a case of reindexing with an indexer consisting of existing column labels.

For reindexing the rows, it's already using existing row labels.

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
cool
ENH: consistent `Styler.highlight_X` arg signature with `props` 
State: closed

This splits #39821 into separate PRs.

The `See Also` doc strings here reference methods created in those PRs - if we push this through now I will fix it later if those methods don't get approved before 1.3.0.

The tests that were edited contained redundant code and were not testing. 

Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
lgtm. ping on greenish (or green if you had rebased recently)
@jreback green now
thanks @attack68