BrandonFors/pandas-github-issues huggingface dataset:
BUG: Appending empty list to DataFrame #28769
State: closed

- [x] closes #28769
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@simonjayhawkins 
That sounds reasonable. The reason for implementing it to return the same dataframe was based on the suggestion in #28769. I could update the docs to mention that if `other` is an empty list, then it would return the same dataframe. Or maybe explicitly state that passing an empty list would result in an Exception (ValueError?)
I'm not sure. see what others think.

not raising on an empty list is perhaps more pythonic, but without checking, I'm not sure how we handle this scenario in other methods.

It maybe worth adding type annotations to `append` and see what mypy has to offer.
Hello @asishm! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-16 20:08:13 UTC
can you merge master 
Rebased master. I'm not sure if there's an issue with the CI or if I messed something up
> Rebased master. I'm not sure if there's an issue with the CI or if I messed something up

I see this in the Travis logs
https://travis-ci.org/pandas-dev/pandas/jobs/612167009#L2418

```
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
The command "ci/setup_env.sh" failed and exited with 2 during .
```
thanks @asishm 
> @asishm
> 
> shouldn't passing an empty list raise an Exception. but perhaps with a more user friendly message.

That would be quite special behavior:
```
123 + 0 == 123
x = []; x.extend([]); x == []
"a" + "" == "a"
```
empty lists, strings, etc should be neutral in operations instead of causing errors. That saves an unecessary if check for a corner case.
TST: fix 24 xfails in maybe_promote
State: closed

Vague gameplan: I think I've got a handle on the `if not boxed` xfail tests that are left, will incrementally fix them over a few more steps.  Then the `if boxed` cases all correspond to passing `ndarray` for `fill_value`, which never happens in the tests except for dedicated tests designed to do exactly that.  If we can rule that case out, we can remove most of the remaining xfails (and a lot of complexity in the test code) in one swoop.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Edit in internals.concat turns out to be necessary to fix a different test that the maybe_promote edit broke.  The relevant code in internals.concat should really use maybe_promote or something similar.  Will open an issue to that effect.
thanks @jbrockmendel (hope these were orthogonal!)
> hope these were orthogonal

Yep, I'm being pretty careful to keep these orthogonal so we can keep a tight turnaround on these.
troubleshoot CI failing Linux py36_32bit
State: closed

closes #28829

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@TomAugspurger there's probably a reason pytest was pinned. if not successful we should add attrs to dependencies and pin attrs?
ok so the pytest pin makes no difference. pytest 4.0.2 installed.
so pytest is now running. merge on green? @pandas-dev/pandas-core 
this is ok
FYI, these CI fixups should typically be backported. Doing so in https://github.com/pandas-dev/pandas/pull/28999.
CLN: core.generic.NDFrame.astype
State: closed

mypy error: `pandas\core\generic.py:5902: error: "append" of "list" does not return a value`

This does not actually manifest as a bug with the current implementation since `pd.concat` ignores `None`.

```python
>>> import pandas as pd
>>>
>>> pd.__version__
'0.26.0.dev0+498.gaf498fe8f'
>>>
>>> d = {'col1': [1, 2], 'col2': [3, 4]}
>>>
>>> df = pd.DataFrame(data=d)
>>>
>>> df
   col1  col2
0     1     3
1     2     4
>>>
>>> df.astype({'col1': 'int32'}).dtypes
col1    int32
col2    int64
dtype: object
>>>
>>> results = []
>>> results.append(df['col1'].astype('int32'))
>>> results.append(results.append(df['col2']))
>>> results
[0    1
1    2
Name: col1, dtype: int32, 0    3
1    4
Name: col2, dtype: int64, None]
>>>
>>> pd.concat(results, axis=1)
   col1  col2
0     1     3
1     2     4
>>>
```


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Clean

Comments:
thanks @simonjayhawkins 
way to go typing! lol
BUG: Patch for skipping seek() when loading Excel files from urllib3.response.HTTPResponse object
State: closed

- [ x] closes #28825
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Discovered that 0.25.x reintroduces #20437. Proposing the same fix here.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO Excel
description: read_excel, to_excel

Comments:
I just tested with 25.1 (on osx), can't reproduce. I took this from #20437:
```python
import pandas

url = "https://raw.github.com/pandas-dev/pandas/master/pandas/tests/io/data/test1.xls"
df = pandas.read_excel(url)
print(df)
```

I believe there are tests for downloading from a url included in the test suite. 
Thanks @tdamsma ! I actually was not able to reproduce this on master either. @hellbe can you try there?
Oh, slight difference in use actually. The reason is that I downloaded first using `requests`. The following code reproduces:

```
import pandas
import requests
url = "https://raw.github.com/pandas-dev/pandas/master/pandas/tests/io/data/test1.xls"
r = requests.get(url, stream=True)
df = pandas.read_excel(r.raw)
print(df)
```

Throws `UnsupportedOperation: seek`
So in my case I can easily fix my code by passing the url directly instead. However, there might be other situations where extra functionality from `requests` is needed, e.g. for authentication.

https://requests.kennethreitz.org/en/master/user/quickstart/#raw-response-content

So `excel_read()` is fed an [urllib3.response.HTTPResponse](https://urllib3.readthedocs.io/en/latest/reference/#urllib3.response.HTTPResponse) object which has a `read()` but not a a `seek()` method.

So I think the patch is still valid, but maybe there's a nicer way to mitigate?
Updated the issue with the above information.
Updated the PR commit to match above and refer to new issue
Rebased
Surely possible to create a test case specifically for reading urllib3's HTTPResponse objects but I'm not sure if it's meaningful tbh
@ocefpaf would this fall under the umbrella of what you were trying to accomplish in #21504? Wonder if it would be better to address comprehensively in a common func rather than one-off for excel files
> @ocefpaf would this fall under the umbrella of what you were trying to accomplish in #21504? Wonder if it would be better to address comprehensively in a common func rather than one-off for excel files

Probably. Let me check. I still have the old PR here waiting for me. (I'm at a code sprint at the moment, Might tackle it again this week.)
@WillAyd I can confirm that #21504 (or the new incarnation that I'll push in a few hours) will "fix this" b/c pandas will use `requests` and wrap it in a `BytesIO` object, with `seek`, if requests is installed.
Thanks for the PR @hellbe but I think this is getting more comprehensively addressed in #28874 by @ocefpaf so let's track this there
TST: Update test in #28798 to use a Categorical
State: closed

- [x] xref #28798
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I merged #28798 too fast, as the column "Name" wasn't a Categorical, which was to be tested. I've made it a Categorical and for good measure I've added ``ordered_fixture`` to the test.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
I get this error, which doesn't seem to be related to my PR:

```
pytest -m "single and not slow and not network" -n 1 -s --strict --durations=10 --junitxml=test-data-single.xml pandas
Traceback (most recent call last):
  File "/home/vsts/miniconda3/envs/pandas-dev/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 182, in _prepareconfig
    config = get_config()
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 153, in get_config
    pluginmanager.import_plugin(spec)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 522, in import_plugin
    __import__(importspec)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/tmpdir.py", line 25, in <module>
    class TempPathFactory(object):
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/tmpdir.py", line 35, in TempPathFactory
    lambda p: Path(os.path.abspath(six.text_type(p)))
TypeError: attrib() got an unexpected keyword argument 'convert'
##[error]Bash exited with code '1'.
```

This doesn't seem to be related to this PR.
TST: Allow for multiple variables on the same line in docstring validation
State: closed

Allow for multiple variables to share the same type and description in documentation. For example, `i, j, k : int` Would be three variables, i, j, and k, which are all ints.

- [ ] tests added / passed 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry - I'm not sure how to add this, if it needs done could I get a pointer?


Labels:
name: Docs

Comments:
> Can you add test(s) for this change? There is a test directory next to the validation script you can put those in

I put in a test for if spacing is off. Other than that, it's a little hard to test because I can't have a test with more than one wrong parameter. The validation script uses sets, so the order can change between runs.
Can you not put a test in `class GoodDocStrings` that has this behavior to make sure it works?
Oh, yeah. I'll do that real quick. Sorry, didn't cross my mind.
Thanks @Nabel0721 !
TST: Dataframe KeyError when getting nonexistent category (#28799)
State: closed

- [x] closes #28799 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
Added test to check for correct exception when accessing a nonexistent category in a DataFrame. 

Labels:
name: Groupby
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @bravech
DOC/BLD: remove old sphinx pin
State: closed

xref https://github.com/pandas-dev/pandas/issues/26058 and https://github.com/pandas-dev/pandas-sphinx-theme/issues/25

We need a newer sphinx version for the new theme. There were some styling issues with the parameter listings on the docstring API pages, but those probably need to redesigned anyway with the new theme.

Labels:
name: Docs

Comments:
I am going to merge this, and leave the minimum sphinx version for later. With the new theme development, we still need to check somewhat the minimum sphinx requirements for the sphinx features we are using, so let's revisit this a bit later when having a better idea on this (the theme python package should then also enforce such a minimum version, which is not yet the case).
TST: Test pivot_table() with categorical data
State: closed

- [x] closes #21370
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
@jorisvandenbossche @jschendel Just had a go at this - let me know if any changes required
Don't know why CI is failing - not sure error is related to my PR 
```
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/16dbc4a2-96e5-4986-aa53-46b5a447d609.sh
pytest -m "single and not slow and not network" -n 1 -s --strict --durations=10 --junitxml=test-data-single.xml pandas
Traceback (most recent call last):
  File "/home/vsts/miniconda3/envs/pandas-dev/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 182, in _prepareconfig
    config = get_config()
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 153, in get_config
    pluginmanager.import_plugin(spec)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/config/__init__.py", line 522, in import_plugin
    __import__(importspec)
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/tmpdir.py", line 25, in <module>
    class TempPathFactory(object):
  File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.6/site-packages/_pytest/tmpdir.py", line 35, in TempPathFactory
    lambda p: Path(os.path.abspath(six.text_type(p)))
TypeError: attrib() got an unexpected keyword argument 'convert'
##[error]Bash exited with code '1'.
##[section]Finishing: Test
```
thanks @ryankarlos
BUG: Coercing bool types to int in qcut
State: closed

- [x] closes #20303 
- [x] tests added / passed: `pytest pandas/tests/reshape/test_qcut.py pandas/tests/reshape/test_cut.py -v` 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master --name-only -- "*.py" | xargs flake8`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @ryankarlos
TST: Move datetime specific test from common tests
State: closed

This test gets rerun multiple times unnecessarily since it is included in the common tests but is actually datetime specific.  Moved it to the more appropriate `datetimes/test_datetime.py` file. Did a quick audit of the other common tests and the rest look fine.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype
name: Clean

Comments:
thanks
TST: Reorganize IntervalIndex tests
State: closed

There shouldn't be any behavioral changes here: mostly just moved things around and created new files for better organization.  I tried to do separate granular commits, so looking at each commit individually might make things a little more obvious if there are concerns.

Summary:
- Removed `test_interval_new.py`
- Created `test_indexing.py` for tests specific to indexing methods
   - Nearly all of `test_interval_new.py` is here
   - Also moved relevant tests from `test_interval.py` here
- Created `test_base.py` for tests inherited from the common `Base` class
  - Only tests related to common tests should be placed here (i.e. overriding, skipping)
- Created `test_format.py` for formatting related methods

All of the above is generally consistent with how the datetime/period indexing tests are organized.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Interval
description: Interval data type
name: Clean

Comments:
thanks @jschendel always love to see reorg on tests :->
BUG: Keep categorical name in groupby
State: closed

- [x] closes #28787
- [x] tests added / passed
- [x] passes `black pandas`
- [x] whatsnew entry

Fixes an issue where column name information was getting dropped when grouping by a categorical column.  I had to change a couple existing tests which I think were incorrect since they were implicitly assuming this behavior was expected.  Also confirmed that this fixes the problem from #28787:

```python
[ins] In [1]: import pandas as pd 
         ...:  
         ...: df = pd.DataFrame(data=(('Bob', 2),  ('Greg', None), ('Greg', 6)), columns=['Name', 'Items']) 
         ...:  
         ...: df_cat = df.copy() 
         ...: df_cat['Name'] = df_cat['Name'].astype('category') 
         ...: df_cat.groupby('Name', observed=True).agg(pd.DataFrame.sum, skipna=True).reset_index() 
         ...:                                                                                                                                                              
Out[1]: 
   Name  Items
0   Bob    2.0
1  Greg    6.0
```

Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
lgtm. over to @topper-123 
@topper-123 Not sure if it was related to using the fixture, but was getting some test failures due to column order after the update, so added a `check_like` to the assertion
Thanks, @dsaxton!
DOC: Added docs to clarify that MultiIndex.set_levels() interprets passed values as new components of the .levels attribute (#28294)
State: closed

Added documentation on MultiIndex.set_levels to clarify that MultiIndex.set_levels() interprets passed values as new components of the .levels attribute

- [x] closes #28294 - or at least attempts to
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Here's my first PR to pandas - looking forward to feedback on how I can further improve this.

Labels:
name: Docs
name: MultiIndex

Comments:
Hello @hweecat! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-20 01:51:11 UTC
Travis CI is failing at Python 3.8-dev but passes for Python 3.6 and 3.7 - not too sure how to resolve this.

https://travis-ci.org/pandas-dev/pandas/jobs/600207880

Edit: Synced my fork with upstream fork in order to pass Travis CI for Python 3.8.
@hweecat I'm not sure what happened but the git diff on this seems to have pulled in close to 200 files

Are you still fixing that up? It might be easier to start fresh with a new branch if unsure what happened
@WillAyd I think it could be because I tried to sync my fork with the upstream in order to pass Travis CI for Python 3.8 (Travis CI kept failing for Python 3.8 previously: https://travis-ci.org/pandas-dev/pandas/jobs/600207880).

Think I might need to start fresh with a new branch (from the master that is synced with the latest upstream) and add changes to that branch. Does that mean I might have to create a new pull request? If yes, I'll proceed to start fresh with a new pull request.
Yea it might be easiest to just do that at this point
Superseded by #29143
PERF: improve perf. of Categorical.searchsorted
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Improves performance of ``Categorical.searchsorted`` by avoiding expensive data convertions.

```python
>>> n = 100_000
>>> c = pd.Categorical(['a'] * n + ['b'] * n + ['c'] * n)
>>> %timeit c.searchsorted('b')
259 µs ± 2.95 µs per loop  # master
5.5 µs ± 165 ns per loop  # this PR
>>> %timeit c.searchsorted(['b', 'c'])
240 µs ± 4.24 µs per loop  # master
9.9 µs ± 166 ns per loop  # this PR
```

Also, ``CategoricalIndex.searchsorted`` now calls ``self.values.searchsorted`` directly instead of going through ``algorithms.searchsorted``, which always ends up calling ``self.values.searchsorted`` anyway. This ends up getting performance to 5.5 µs instead of 12 µs.


Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
Comments addressed.
thanks @topper-123
DOC: Reference level name as Term of HDFStore.select query (#28791)
State: closed

Add documentation for default level name with HDFStore MultiIndex.

- [x] closes #28791
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
are there any other issues about this (can you do a search)
> are there any other issues about this (can you do a search)

I did not find any other issue specifically about this (I did not even find it on StackOverflow). I guess everybody name its index…

I did find the issue #6208 that speficy that you cant use "index" as an level name (still true on master). I guess it is worth a note in section Storing MultiIndex DataFrames.
thanks @nrebena
TST: Added SA01, SA02, and SA03 error checks to the Travis CI Build
State: closed

These error types no longer show errors on master, so they're ready to be turned on for builds.

- [x] closes #23630
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs

Comments:
lgtm. This doesn't fully close the issue referenced though right? Would SA01 and SA04 still not be required to fully close?
Those weren't part of the errors being run in that issue. If you'd like I can look into if those are passing to, and if not edit the PR so it addresses the issue instead of closing it.
Yea I guess if you can check on those would be great - they are all part of the "See Also" validation so I think would make sense to include if not a lot of effort
Looks like SA01 is clean too, but SA04 has a lot of issues. I'll add SA01 to the checks run for now. We should open a new issue for just SA04, so people can focus down on just that error.
Sounds good to me - if you can open that follow up issue would be appreciated
Sure thing. I'll post the output of the errors with the issue. Thanks for the quick response!
thanks @Nabel0721 I closed the original issue, but if there are still errors, pls open a new one (as @WillAyd indicated).
DOC: Misc typos fixed in docs and code comments
State: closed

Fix some typo

Labels:
name: Docs

Comments:
Small comment, generally looks good
@garanews can you apply the fix suggested by datapythonista?  this should be about ready otherwise
lgtm cc @datapythonista 
Thanks a lot for fixing all those @garanews 

If you want to open an issue for the typo in the csv that would be great. And also if you want to fix it yourself.

Btw, opening PRs from master is not a good practice, you'll have problems if you try to work in something else in parallel while the PR is being reviewed. Better to create a branch for every PR.
BUG: pd.MultiIndex.get_loc(np.nan)
State: closed

MultiIndex.get_loc can find nan when input is NA value(e.g. nan, None)

- [ ] closes #19132
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
Hello @proost! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-10 03:33:36 UTC
@toobaz care to take a look?