BrandonFors/pandas-github-issues huggingface dataset:
29213: Dataframe Constructor from List of List and non-iterables
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/29213
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

---
## What
Whilst the original issue is from the factory method `DataFrame.from_dict(d, orient='index')`, the main issue is the *order* of elements within the list in the main constructor:

For example:
```
pd.DataFrame([[1, 2, 3], 4])  # doesn't work
# TypeError: object of type 'int' has no len()

In [2]: pd.DataFrame([4, [1, 2, 3]]) # works, creates a 1D DataFrame
Out[2]:
           0
0          4
1  [1, 2, 3]
```



## Current Constructor Logic on List Argument
- Current logic looks at the [first element](https://github.com/pandas-dev/pandas/blob/master/pandas/core/frame.py#L462-L465), and *infers* that all other elements are iterables as well.
    - if all elements in the list are iterables, it generates a 2D DataFrame.
    - if any element from index-1 onwards is a non-iterable, it doesn't have a len() method and fails.

## Proposed Solution
- Based on the first element, *try* to infer that all elements are iterables as well.
    - If not all subsequent elements are iterables, then return 1D DataFrame. This would be the same behaviour as it would have been if the first element in the list is non-iterable.
    - This should not have performance degradation as there is noneed to check if all elements in the list are iterables.


## Note
- This does not solve the issue whereby we have iterables of different types (such as lists and strings...)
```
In [2]: pd.DataFrame([[1, 2, 3], 'foobar'])
Out[2]:
   0  1  2     3     4     5
0  1  2  3  None  None  None
1  f  o  o     b     a     r
```

Labels:
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
can you merge master and update
@matttan90 can you address comments?
can you merge master and will look again
Sorry. My bad for disappearing. Will think again on how I can implement a fix on this..
Closing as I think stale, but @matttan90 ping if you want to pick back up and can merge master / address comments
Test added: uint64 multicolumn sort
State: closed

- [x] closes #9918
- [x] tests added / passed
- [x] passes `black pandas`
- [x] added test test_sort_multicolumn_uint64

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @pv8473h12
Add documentation linking to sqlalchemy
State: closed

- [x] closes #23086
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ n/a] whatsnew entry 


Labels:
name: Docs

Comments:
Thanks very much for the PR:

You can do something like:
```
For documentation on pyarrow, see `here
        <https://arrow.apache.org/docs/python/index.html>`__.
```
For reference see:
https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html

To avoid it rendering like so:

![image](https://user-images.githubusercontent.com/16733618/68075205-98b16980-fd9c-11e9-83ab-9f288c446fb2.png)
Disallow non-scalar fill_value in maybe_upcast
State: closed

Along with #29331 and #29332 this allows us to rule out non-scalar fill_value being passed to maybe_promote for _almost_ all cases, the last few of which we'll have to address individually.

The restriction in `Block.shift` could be an API change if you squint and tilt your head.

The assertion in masked_rec_array_to_mgr is based on my read of the numpy docs, but it'd be worth double-checking.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
this is orthogonal to #29331 ?
> this is orthogonal to #29331 ?

correct

> do we need additional tests here?

for maybe_upcast_putmask i dont think so.  for Block.shift maybe; im still a little iffy on the object-dtype case there
thanks, pls followon tests as needed
GH 16051: DataFrame.replace() overwrites when values are non-numeric
State: closed

- [x] closes #16051
- [x] tests added and passed
- [x] passes `black pandas`
- [x] added new test test_replace_commutative

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I have updated it @WillAyd
All green @jreback
Thanks @pv8473h12
TST: Apply method broken for empty integer series with datetime index
State: closed

I added a unit test for an edge case that was failing. Using the apply method on an empty integer series with a datetime index would throw an error.

- [x] closes #21245
- [x] 1 test added
- [x] passes pandas and flake8


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @okz12
ensure consistent structure for groupby on index and column
State: closed

- [x] closes #17681
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
Thanks for the PR! But I think this was already covered in #29124
TST: new test for subset of a MultiIndex dtype
State: closed

Tried unsuccessfully reproducing Issue #20757 on pandas: 0.23.0 and 0.26.0.dev0+734.g0de99558b.dirty. That's why I decided to keep the input data from Issue #20757 unchanged.

- [x] closes #20757
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex

Comments:
thanks @ganevgv
TST: Test type issue fix in empty groupby from DataFrame with categorical
State: closed

TST: Test type issue fix in empty groupby from DataFrame with categorical

closes #9614 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Hello @louishuynh! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-12 19:05:15 UTC
thanks @louishuynh
Pv feature2
State: closed

- [x] closes #9113
- [x] tests added / passed
- [x] passes `black pandas`
- [x] new test added: test_replace_no_cast()

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
lgtm. ping on green.
All green @jreback
Thanks @pv8473h12
BUG: Issue #29128 Series.var not returning the correct result
State: closed

- [ x ] closes #29128
- [ x ] tests added / passed
- [ x ] passes `black pandas`
- [ x ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ x ] whatsnew entry

This PR closes https://github.com/pandas-dev/pandas/issues/29128

Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Hello @HubertKl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-09 00:01:11 UTC
Sorry for the mess, I ran the wrong version of black. 

I updated my PR accordingly to the change requested.
@HubertKl can you rebase
> @HubertKl can you rebase

Done

Thanks @HubertKl
pin black, xref gh-29341
State: closed

xref #29341 

pinning until we can update the codebase

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
cc @TomAugspurger 
We should probably also pin in the pre-commit configuration: https://github.com/pandas-dev/pandas/blob/master/.pre-commit-config.yaml ?
Stable python 3.8.0
State: closed

Python 3.8.0 stable release is now available on travis https://travis-ci.community/t/add-python-3-8-support/5463 we can use it?

The 3.8-dev snapshot seemed to cause some issues here:
https://travis-ci.org/pandas-dev/pandas/jobs/606411398?utm_medium=notification&utm_source=github_status

related - https://github.com/pandas-dev/pandas/issues/26626

Labels:
name: CI
description: Continuous Integration

Comments:
fine if this passes; we currently cannot do much more until conda forge packages are built for our depenedencies.
thanks @alimcmaster1 
thanks @jreback !
GH14422: BUG: to_numeric doesn't work uint64 numbers
State: closed

-closes #14422
-passes `black pandas`
-added test test_downcast_uint64_exception()

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @pv8473h12! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-02 18:41:52 UTC
thanks @pv8473h12 for the PR! - mind addressing the pep8 issues?
thanks @pv8473h12
TST: Adding styler applymap multindex & code test [Ref: #25858]
State: closed

- [x] closes #25858
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Thanks @sofiane87 !
ensure consistency between columns aggregates with missing values present
State: closed

- [x] closes #16832
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @mjarosie
TST: Test for Boolean Series with missing to Categorical dtype
State: closed

- [x] closes #19182
- [x] 1 test added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
thanks !
TST: Test Groupby with Categoricals consistent for Series and DataFrame results
State: closed

- [ ] closes https://github.com/pandas-dev/pandas/issues/20416
- [ ] 1 test added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @keechongtan
TST: add new test - check reindex() and astype() order consistency for Series
State: closed

check reindex() and astype() order consistency for Series

- [x] closes #17444
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @ganevgv
Fix mypy error messages in pandas/test/tseries/offsets/test_offsets.py
State: closed

Activate and fix mypy for test offsets 

- xref #28926
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm @simonjayhawkins 
Great thanks @samuelsinayoko
TST: Adding to_dict numeric consistency test (#22620)
State: closed

- [x] closes #22620
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
this is good, thanks @sofiane87
BUG: Categorical.from_codes should not warn on empty codes
State: closed

Empty codes should not warn about float codes:

```python
>>> pd.Categorical.from_codes([], ['a', 'b', 'c']) 
C:\Users\TP\Miniconda3\envs\pandas-dev\Scripts\ipython:1: FutureWarning: float codes will be disallowed in the future and raise a ValueError
```


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
cool thanks
Remove FrozenNDArray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jbrockmendel this is partially a culprit for the circular references you would get when trying to import `pandas.core.reshape.concat`. To try and frame the issue,

1. pandas.core.base requires concat in some places, but if you try to move this to global imports THEN
2. concat requires an import of pandas.core.index WHICH
3. requires imports from pandas.core.indexes.frozen WHICH
4. require the import of PandasObject from pandas.core.base

But PandasObject is not defined by the time step 4 comes around, hence the circular import

In a follow up could maybe question if FrozenList really needs to inherit from PandasObject, but will leave that to a separate discussion
Not having looked at the test failures yet, two thoughts: 1) im on board for removing this if we can and 2) not needing to subclass PandasObject is also plausible
Closing to clear queue but plan on coming back to this
This seems to have caused the problem described here:

https://stackoverflow.com/questions/61641738/pandas-1-0-cannot-pickle-load-dict-containing-dataframe-with-multiindex

which I'm now also experiencing. I dumped a MultiIndex dataframe containing ndarrays to a pickle on disk under pandas 0.25.x in Python 3.6, and now I'm getting:

`AttributeError: Can't get attribute 'FrozenNDArray' on <module 'pandas.core.indexes.frozen'`

when trying to load it in pandas 1.0.3 (still on Python 3.6). Any suggestions/workarounds? Should I instead open up a new issue?

This solved issue seems related, but is for Python 2.7:

https://github.com/pandas-dev/pandas/issues/31988

This comment in the original rationale for getting rid of `FrozenNDArray` mentions `pandas.compat.pickle_compat.py`, which seems relevant:

https://github.com/pandas-dev/pandas/issues/9031#issuecomment-306659873
Can you open a dedicated issue for that? I think need to define some pickle combat for that particular use case but can discuss on its own thread
BUG: assignment to multiple columns when some column do not exist
State: closed

- [x] closes #13658
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Previous PR: #26534

In particular, the following code now behaves correctly.

```python
import pandas as pd
df = pd.DataFrame({'a': [0, 1, 2], 'b': [3, 4, 5]})
df[['a', 'c']] = 1
print(df)
```
v0.22: error

master: column `'c'` is converted to index `-1`, which causes last column to be overwritten.
```
   a  b
0  1  1
1  1  1
2  1  1
```

After this PR:
```
   a  b  c
0  1  3  1
1  1  4  1
2  1  5  1
```

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jreback any more changes required?
pls rebase as well
@howsiwei some CI failures now, if you have a chance to look.
@TomAugspurger I'm not sure how to fix it. See https://github.com/pandas-dev/pandas/pull/29334#discussion_r364716860
Thanks for the update. I'm not sure either :/
@howsiwei is this still active? Can you fix merge conflicts and try to get CI green?

Also want to move whatsnew to 1.1.0 at this point
@WillAyd I'm not sure how to proceed due to https://github.com/pandas-dev/pandas/pull/29334#discussion_r364716860. I can get the CI green with my previous approach (https://github.com/pandas-dev/pandas/blob/8ccd2b879b1a10beb7760f1f1b43fac36b41ae19/pandas/core/indexing.py#L1731) of having different checks depending on whether the DataFrame is set by `.loc[]` or `[]` but that was advised against by @jreback.
> @WillAyd I'm not sure how to proceed due to [#29334 (comment)](https://github.com/pandas-dev/pandas/pull/29334#discussion_r364716860). I can get the CI green with my previous approach (
> 
> https://github.com/pandas-dev/pandas/blob/8ccd2b879b1a10beb7760f1f1b43fac36b41ae19/pandas/core/indexing.py#L1731
> 
> ) of having different checks depending on whether the DataFrame is set by `.loc[]` or `[]` but that was advised against by @jreback.

you can pass axis
@howsiwei can you merge master to resolve conflicts
@howsiwei can you fix up merge conflict? Otherwise should be good to merge here
thanks @howsiwei lgtm.

@jbrockmendel if any comments.
@howsiwei this looks good. can you merge master and ping on green (just to make sure)
@jreback merged master. It's green now.
thanks @howsiwei very nice
thanks for sticking with it!
CLN: type annotations
State: closed

Separating these out from the non-CLN work going on in/around these functions.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> Just as a general comment I think it would be better to fully annotate some methods rather than partially annotating a lot of methods.

That's reasonable.  How strong is this preference?  i.e. when weighed against "its nice to do these separately from the other PR that makes non-typing changes in the same functions to have a simpler diff"
> > Just as a general comment I think it would be better to fully annotate some methods rather than partially annotating a lot of methods.
> 
> That's reasonable. How strong is this preference? i.e. when weighed against "its nice to do these separately from the other PR that makes non-typing changes in the same functions to have a simpler diff"

I would rather merge quickly on correct changes that are non-controversial, rather block things. so actually happy to type the 'easy' things (of course nice for newcomers to do this too).
FWIW I’m not blocking on this just have reviewed thoroughly. Feel free to merge if happy with it

Sent from my iPhone

> On Nov 2, 2019, at 8:34 AM, Jeff Reback <notifications@github.com> wrote:
> 
> ﻿
> Just as a general comment I think it would be better to fully annotate some methods rather than partially annotating a lot of methods.
> 
> That's reasonable. How strong is this preference? i.e. when weighed against "its nice to do these separately from the other PR that makes non-typing changes in the same functions to have a simpler diff"
> 
> I would rather merge quickly on correct changes that are non-controversial, rather block things. so actually happy to type the 'easy' things (of course nice for newcomers to do this too).
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or unsubscribe.

green
gentle ping.  a couple of upcoming branches are going to have merge conflicts with this otherwise
thanks
REF: implement maybe_promote_scalar
State: closed

Narrowing down the set of cases where we have to worry about a non-scalar fill_value.  I'm pretty sure we can narrow it down further, but saving that for a separate step as it has some complications for EA corner cases.

cc @jreback

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
lgtm. ping on green.
ping
gentle ping.  we're getting close to being done with the maybe_promote saga
Unnecessary following #29416, closing.
WEB: Fix extension of about/index
State: closed

- [✓] closes #29325
- [✓] whatsnew: the about/index.html file is now about/index.md


Labels:
name: Web
description: pandas website

Comments:
@datapythonista 
Here is a screenshot of the page that was generated locally. I just noticed that the URL shows the .html extension at the end. Should it be .md or is it ok as shown?

![Screenshot_index_html](https://user-images.githubusercontent.com/56150991/68067532-5e48bd80-fd05-11e9-8112-90ba3ee8c32c.png)

Thanks, looks perfect now. The extension in the web needs to be html, we convert markdowns to html if they have the right extension.

This is ready, will let someone else from the team review and merged.

Thanks for the fix!
lgtm. merge when ready.
Sorry for the delay @est271 and thanks a lot for your contribution. Hope to see more coming!
@datapythonista No worries. Glad to help.
CLN: fix mypy warnings/errors
State: closed

When I run `mypy pandas` locally I get a handful of complaints.  This fixes about half of them.  The other half are about `FixedOffset` which IIRC we are handling elsewhere.

cc @simonjayhawkins 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
What version of mypy are you using and what are the errors exactly? I'm very hesitant to add `# type: ignore` anywhere, especially if there's no urgency to do so

@angelaambroz fixed the issue with FixedOffset upstream in TypeShed and I think that will make its way into mypy 0.75
0.740
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-02 02:37:11 UTC
thanks @jbrockmendel
Pr09 batch 1
State: closed

fixing PR09 missing period formatting errors in various parameter descriptions. 

Subset of #28602, mainly functions with single parameter's missing periods that didn't make sense to open an entire issue for. 

I'm periodically updating the main issue for PR09 with the current list of errors remaining as they're removed. 

Labels:
name: Docs

Comments:
pandas/core/reshape/tile.py(278,): error PR05: pandas.qcut: Parameter "labels" type should not finish with "."

Thanks @HughKelley
CLN: Consistent imports - linting rules
State: closed

- [x] closes #29272
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @SaturnFromTitan very nice!
fix bugs cause replacer's dtype not respected
State: closed

- [x] closes  #26632
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
I need some assistance here. #26632 
> I need some assistance here. #26632

?

this PR looks good. if you'd merge master and add a whatsnew note
> can we just move 2834 here?

I propose an alternative approach in my previous comment. Does it look better ?
lgtm @peterpanmj can you merge master and ping on green.
Thanks @peterpanmj