BrandonFors/pandas-github-issues huggingface dataset:
add test for fillna with column
State: closed

- [X] tests added / passed

I noticed there was no test for pd.Series.fillna using a column, so I added one.

Labels:

Comments:
Hello @hasandiwan! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/series/methods/test_fillna.py`](https://github.com/pandas-dev/pandas/blob/f6059224456f6e287ec43524c6a1286fe398c548/pandas/tests/series/methods/test_fillna.py):

> [Line 24:38](https://github.com/pandas-dev/pandas/blob/f6059224456f6e287ec43524c6a1286fe398c548/pandas/tests/series/methods/test_fillna.py#L24): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 24:41](https://github.com/pandas-dev/pandas/blob/f6059224456f6e287ec43524c6a1286fe398c548/pandas/tests/series/methods/test_fillna.py#L24): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 32:1](https://github.com/pandas-dev/pandas/blob/f6059224456f6e287ec43524c6a1286fe398c548/pandas/tests/series/methods/test_fillna.py#L32): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace


are you sure this is not already solved by #5703 and PR #5705? The tests seem there already.
closing this as stale
TST: Benchmarks for CSV writing for various index variations
State: closed

Benchmarks derived from GH37484 (https://github.com/pandas-dev/pandas/issues/37484). This captures a peculiar difference in timings between a DataFrame originating from `df.set_index(...).head(...)` and one from `df.head(...).set_index(...)`.

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

I've not included a whatsnew entry, as no section seems particularly relevant to internal-only test/benchmark changes. If it should go in, shout and I'll add a note somewhere.

Labels:
name: IO CSV
description: read_csv, to_csv
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
can you show these your PR (e.g. `asv dev ` just see the timings
Yes, here's the benchmark output:

```
$ asv run -e -E existing -b ^io.csv.ToCSVIndexes
· Discovering benchmarks
· Running 3 total benchmarks (1 commits * 1 environments * 3 benchmarks)
[  0.00%] ·· Benchmarking existing-py_home_jbanorthwest.co.uk_danielevans_venvs_pandas_dev_py39_bin_python3.9
[ 16.67%] ··· Running (io.csv.ToCSVIndexes.time_head_of_multiindex--)...
[ 66.67%] ··· io.csv.ToCSVIndexes.time_head_of_multiindex                                                  2.04±0.04s
[ 83.33%] ··· io.csv.ToCSVIndexes.time_multiindex                                                            818±60ms
[100.00%] ··· io.csv.ToCSVIndexes.time_standard_index                                                        645±60ms
```
Hello @DanielFEvans! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-21 16:25:08 UTC
thanks @DanielFEvans
BUG+DOC: Recent Styler Enhancements
State: closed

**DOC**: this edits the `style.ipynb` file to include examples of the 3 recent enhancements to `Styler`:

 - Tooltips
 - External CSS classes to datacells
 - Row and Column styling via table styles

It also addresses 3 minor bugs that I detected when wirting the documentation:

**BUG1** (lines 696-703): `set_td_classes` only attached classes to 'diagonal elements' of the DataFrame due a misaligned comprehension loop:

```
{
r: {
   c: [str(classes.iloc[r, c])] 
   for c, cn in enumerate(classes.columns) if not mask.iloc[r, c]
   }
for r, rn in enumerate(classes.index)
}
```
replaces:
```
{
r: {
    c: [str(classes.iloc[r, c])]
    }
for r, rn in enumerate(classes.index)
for c, cn in enumerate(classes.columns)
if not mask.iloc[r, c]
}
```

**BUG2** (lines 513 to 537): external css class names from the set_td_classes were appended to the HTML id tag, as well as the HTML class tag, due to a premature variable update relating to the `row_dict`. The solution was to re-order so that 'id' tags were generated first and then 'class' was updated with information from the new methods.

**BUG3** (lines 1920-1925): Tooltips were generated for the wrong indexes/columns, since the null rows/columns were dropped and then the interger locations of rows/columns was mapped. The solution was to avoid dropping null rows/columns.




Labels:
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@attack68 can you also add tests for the bugs you have fixed (e.g. simply a check on the generated html might be fine), just to lock down these cases.
@jreback think this is ready now..
thanks @attack68 

pls check the rendered docs when they build: https://pandas.pydata.org/docs/dev/
[DOC] Update PR link in 1.2.0 release notes
State: closed

- [x] closes #39291
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
~whatsnew entry~


Labels:
name: Docs

Comments:
thanks @rs2 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
REF/TST: Move Series/DataFrame apply tests to tests.apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Ref: https://github.com/pandas-dev/pandas/pull/39212#issuecomment-763112628

Just a move. Will be following up with combining Series/DataFrame tests where appropriate.

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks !
REF: Move part of groupby.agg to apply
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

pre-commit adjusted to account for the false positive on `Union[SeriesGroupBy, DataFrameGroupBy]`.

Moves part of groupby.agg implementation from `pandas.core.aggregation` to `pandas.core.apply`. The only difference in the code is that in `apply`, the error message raised is different when the argument is a string (e.g. "sum") and the kwarg axis=1 is provided. Added a test to check for error message.

After this will be resample and rolling, then much of the implementation code in core.aggregation code can be moved into apply.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
CI: fix flaky tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks for identifying and fixing the zip issue, I had created an issue for it #39286. Unfortunately, `ZipFile` has unlike `GZipFile` no `mtime` argument.
CLN: consistently put '|' at start of line in pygrep hooks #39301
State: closed

- [x] closes #39301
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


I moved the `|` symbols from the end of the line to the beginning of the line in pygrep hooks.

I also added the `|` symbol in the `non-standard-imports-in-tests` at the beginning of the line 93, which was missing as @MarcoGorelli pointed.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @gcmaciel 

prob need a precommit meta-hook here (heheh)
[DOC/WEB]: Resolving 'Getting started' URL ambiguity in pandas.io(Issue #32307)
State: closed

The Following goals are being accomplished in this PR:
- [x] Having the navigation bar's 'Getting Started' link point to `/docs/getting_started/index.html` from `/getting_started.html`
- [x] Removing the redundant markup `pandas/web/pandas/getting_started.md`.
<!--- [x] Adding above redirecting information in `pandas/doc/redirect.csv`.-->

#32307 


Labels:
name: Docs

Comments:
> Good start! To follow through on removing the duplicate page:
> 
> * [x]  Have the navigation bar point to `/docs/getting_started/index.html`
> * [ ]  Delete [the old page](https://github.com/pandas-dev/pandas/blob/master/web/pandas/getting_started.md)
> * [ ]  [Add a redirect](https://github.com/pandas-dev/pandas/blob/master/doc/redirects.csv)

@afeld I just noticed that the `Install Pandas Now ` button also leads to `getting_started.html`. In this case would it be valid to carry out the second task, i.e, "Delete [the old page](https://github.com/pandas-dev/pandas/blob/master/web/pandas/getting_started.md)", or should we redirect that link too?
what do we need to do there @jorisvandenbossche @afeld 
I'm probably biased, but I'm -10 here.

I don't see the duplication on those pages. For a user without knowledge of Python, Jupyter, Conda... The page you are removing is a great starting point, and they should be able to get a pandas hello world with further research. The Getting started in the docs does not accomplish that, and assumes you've got a Python working environment, and provides lots of information that is intimidating for someone just starting.

I think we're doing a very poor favor to many our users by removing this page.
Thanks for the PR, but appears that this change needs more discussion on the original issue first before coming to a conclusion. Since discussion here has generally stalled, closing for now
WIP/REF: BlockManager.setitem_blockwise
State: closed

Posting largely for discussion with @phofl.

Motivations:

1) Try to end up with fewer code paths for setitem so that we can ensure consistent inplace/view/copy behavior (xref #38896)
2) Simplify (ATM this does not simplify things)
3) Perf: iterating over blocks instead of columns may save some copies (though doing the Index.intersection calls may go the other way) (havent done any profiling yet)

Non-trivial overlap with #39044.
Some of this would be simplified by #39163.
No logical overlap with #39266, but will require rebase.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hello @jbrockmendel! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/arrays/datetimelike.py`](https://github.com/pandas-dev/pandas/blob/604f796f9a78d572ec784334f862a1a6d2fbc7ee/pandas/core/arrays/datetimelike.py):

> [Line 619:89](https://github.com/pandas-dev/pandas/blob/604f796f9a78d572ec784334f862a1a6d2fbc7ee/pandas/core/arrays/datetimelike.py#L619): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (91 > 88 characters)



* In the file [`pandas/core/frame.py`](https://github.com/pandas-dev/pandas/blob/604f796f9a78d572ec784334f862a1a6d2fbc7ee/pandas/core/frame.py):

> [Line 3222:89](https://github.com/pandas-dev/pandas/blob/604f796f9a78d572ec784334f862a1a6d2fbc7ee/pandas/core/frame.py#L3222): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (104 > 88 characters)












Is there a specific place where I can be of help?
Mostly just want to keep you in the loop on what i have in mind since you're working on similar parts of the code.
@phofl this fails the test added in #39280.  Thoughts on an appropriate fix?

(a branch that only has the edits to `DataFrame._setitem_array` has the same failure)
Not per se a fix, but I had similar difficulties in #39341 

We have to account for duplicate column names in these checks. Maybe using get_indexer_for or something like that if there is a faster alternative. Simply comparing the lenght does not work with duplicates
You could use 
```
if self.columns.is_unique:
    len_key = len(key)
else:
    len_key = len(self.columns.get_indexer_non_unique(key))
if len(value) != len_key:
    raise ValueError("Columns must be same length as key")
```

as a helper functions for both len(value) != len(key) checks. Test will keep failing because the column is now cast to integer, but this is good since this is consistent with other setitem cases. If you like I could implement this in #39341. I am dispatching to the else block there right now, since this does the trick currently.
mothballing
CLN, TYP Remove string type hints
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

For some background: there was some consensus on doing this [here](https://github.com/pandas-dev/pandas/pull/36034#issuecomment-685186742)

Then [here](https://github.com/pandas-dev/pandas/pull/39270#pullrequestreview-571190631) there was the suggestion to use `ast.parse`, rather than regular expressions. I had a go at that, and made this into its own package: https://github.com/MarcoGorelli/no-string-hints , which I've included here as a pre-commit hook

The changes here are just a result of running `pre-commit run no-string-hints -a`, and then adding `from __future__ import annotations` to some files.

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @MarcoGorelli. this is great.
Thanks all for your kind words, much appreciated :pray: !
REF: add skeleton for strategy pattern in describe
State: closed

- [ ] xref #36833
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Added a skeleton for the strategy pattern in ``pandas/core/describe.py``.
Currently the methods ``describe`` simply delegate to the corresponding ``describe_`` functions.
In the next steps I plan on moving the ``describe`` functions into each of the strategy classes.
Finally, these functions can be unified, since a lot there has a common pattern.

Labels:

Comments:
> * [ ]  xref #36833
> * [ ]  tests added / passed
> * [ ]  Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
> * [ ]  whatsnew entry
> 
> Added a skeleton for the strategy pattern in `pandas/core/describe.py`.
> Currently the methods `describe` simply delegate to the corresponding `describe_` functions.
> In the next steps I plan on moving the `describe` functions into each of the strategy classes.
> Finally, these functions can be unified, since a lot there has a common pattern.

can you not unify these w/o using this pattern? this looks like a lot of extra boilerplate
> can you not unify these w/o using this pattern? this looks like a lot of extra boilerplate

It is possible, but I imagine may not be an improvement over the current state.

The idea was to make an abstract strategy with the method ``describe`` like this
```
def describe(self):
    return Series(
        self.array,
        index=self.names,
        name=self.data.name,
        dtype=self.dtype,
    )
```
and then only re-define ``self.array``, ``self.names`` and ``self.dtype`` in each of the concrete strategy classes.
This way it will probably be more readable and maintainable.
Please let me know what you think.

Currently the code is heading somewhat close to xref #36833.
If you consider that this is a wrong direction, then I am OK on closing this PR.
yeah the code looks pretty good as is. i am not sure we need to greatly expand it.
I am closing this as it was decided not to proceed with such type of refactoring.
STYLE: dont use pd api types in tests
State: closed

- [x] closes #39203 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [] whatsnew entry

I didn't add a whatsnew entry since it's only style fixes.
I'll open a new issue regarding @jreback suggestion to ban this with a precommit style rule.
I prefer it to be done in a different PR.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @nofarm3 

> I'll open a new issue regarding @jreback suggestion to ban this with a precommit style rule.
I prefer it to be done in a different PR.

Any chance this could be done in the same PR, so that it can be seen to be fixed? This is analogous to adding tests in the same PR in which one fixes a bug
@MarcoGorelli 
 Yes it make sense. no problem I'll add this to this PR.
sure - do ask if it's not something you're familiar with, I think it should be enough to just add a line to the `non-standard-imports-in-tests` hook
@MarcoGorelli 
I added a hook that validates there is no usage in ```pd.api.types``` under ```pandas/tests/```. 
Do you think it's good enough for now?
I'm finished, lmk if you want to see anything more.
thanks @nofarm3
Default behaviour for return_type = None corrected.
State: closed

- [x] closes #35518
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:

Comments:
Hello @rajat315315! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-24 12:17:47 UTC
'lower' method causing problems when individual values are being read as 'int'
State: closed

- couldn't run tests as using company locked system



Labels:
name: Strings
description: String extension data type and string data

Comments:
Hello @AbhinavRao23! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/tools/datetimes.py`](https://github.com/pandas-dev/pandas/blob/a915f883447599eb739a99eb33b842ce1bd9ce50/pandas/core/tools/datetimes.py):

> [Line 893:29](https://github.com/pandas-dev/pandas/blob/a915f883447599eb739a99eb33b842ce1bd9ce50/pandas/core/tools/datetimes.py#L893): [E261](https://duckduckgo.com/?q=pep8%20E261) at least two spaces before inline comment
> [Line 893:30](https://github.com/pandas-dev/pandas/blob/a915f883447599eb739a99eb33b842ce1bd9ce50/pandas/core/tools/datetimes.py#L893): [E262](https://duckduckgo.com/?q=pep8%20E262) inline comment should start with '# '
> [Line 893:89](https://github.com/pandas-dev/pandas/blob/a915f883447599eb739a99eb33b842ce1bd9ce50/pandas/core/tools/datetimes.py#L893): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (102 > 88 characters)


closing. please open an issue with a representative issue.
Series with dtype=object does unexpected type conversion
State: closed

- [ ] closes #21881
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @ftrihardjo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-28 09:55:31 UTC
thanks @ftrihardjo
Replaced empty series check with Series with columns
State: closed

- [x] closes #39111 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@rajat315315 can you add a test for this
Sure.
Thanks for the PR but it appears that this PR has gotten stale. If you're still interested in working on this, ping us and we'd be happy to reopen.
BUG: DataFrame.__setitem__ raising ValueError when setting multiple values to dup columns
State: closed

- [x] closes #15695
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Setting with a scalar key which is duplicated in columns should be the same as using a listlike key

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hacky, will try to improve
Hm this is a hacky edge case:
```
df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=["a", "b", "a"])

df["a"] = df.index
```

This is tested in a unit test ``test_column_dups_operations``

If this should work, we can't really implement 

```
df["a"] = (1, 2)
```

because here is not clear what to do, interpret (1,2) as a column, which should result in (this is tested with the case above)
```
   a  b  a
0  1  2  1
1  2  5  2
```
or interpret as a row (as described in the op) which would result in 
```
   a  b  a
0  1  2  2
1  1  5  2
```

The loc case interprets this as
```
df.loc[:,"a"] = df.index
   a  b  a
0  0  2  1
1  0  5  1
```

which results in an Error if len(index) != number of "a" in columns

thanks @phofl 
Doesn't this slow down every setting operation with a single key? I think it's fine as a temporary measure, but really we shouldn't be doing multiple lookups for a single operation.
> Doesn't this slow down every setting operation with a single key?

Well no, I guess only with a list of keys. Still, we could check if ``columns`` is unique (which is O(1), right?) before the test involving  ``get_indexer_for``?
That's probably a good idea, yes
REF: share start_time/end_time between Period/PeriodArray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

One thing that might make this undesirable is that the annotations on this are accurate for Period but not for PeriodArray.  Thoughts @simonjayhawkins 


Labels:
name: Period
description: Period data type

Comments:
lgtm. can you show the typing issue?
> can you show the typing issue?

start_time and end_time are both annotated as returning Timestamp, but that is only accurate for Period, whereas for PeriodArray the annotation would be DatetimeArray.  mypy doesnt pick it up for reasons discussed on last week's call
> > can you show the typing issue?
> 
> start_time and end_time are both annotated as returning Timestamp, but that is only accurate for Period, whereas for PeriodArray the annotation would be DatetimeArray. mypy doesnt pick it up for reasons discussed on last week's call

ok
BUG: loc.setitem raising ValueError when df has duplicate columns
State: closed

- [x] closes #38521
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Is there a more elegant way to check if an element occurs only once in an index?
Failure unrelated
thanks @phofl