BrandonFors/pandas-github-issues huggingface dataset:
BUG: GH12902 fixed coercion of complex values to float when using gro…
State: closed

- [x] closes #12902  
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Complex
description: Complex Numbers

Comments:
thanks!
BUG: TypeError in index coercion
State: closed

- [x] closes #12893
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: MultiIndex
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
small change, lgtm otherwise. ping when green.

looks like a PEP issue. can you fix and push. ping when green.

thanks!
BUG: Enforce parse_dates as bool when scalar
State: closed

Title is self-explanatory.  Closes #5636.


Labels:
name: Datetime
description: Datetime data dtype
name: API Design
name: IO CSV
description: read_csv, to_csv

Comments:
ok, lgtm. ping when greeen.

@jreback : Travis finally got the OSX build done, and everything is green!  Ready to merge.
BUG: sniffing a csv raises with only a header
State: closed

Add test to verify that #7773 is no longer an issue.  Closes #7773.


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
don't show headers that don't give any info as that requires people to click on it. Its much better to have something that is readable.

lgtm. ping when green.

@jreback : Travis has finally given the green light!  Ready to merge if there is nothing else.

ty
Add test to validate resampling GH9915
State: closed

- [x] closes #9915
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Resample
description: resample method

Comments:
This shows `FutureWarning: .resample() is now a deferred operation use .resample(...).mean() instead of .resample(...)` in the IDE ...

lgtm. ping when green.

thanks!
BUG: SparseArray numeric ops misc fixes
State: closed

- [x] no existing issue
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Fixed following 3 issues occurred on the current master.
### 1. addition ignores rhs `fill_value`

```
pd.SparseArray([1., 1.]) + pd.SparseArray([1., 0.], fill_value=0.)
# [2.0, nan]
# Fill: nan
# IntIndex
# Indices: array([0], dtype=int32)

Expected:
# [2.0, 1.0]
```
### 2. mod raises `AttributeError`

```
pd.SparseArray([1, 1]) % pd.SparseArray([1, np.nan])
# AttributeError: 'module' object has no attribute 'sparse_nanmod'
```
### 3. pow outputs incorrect result wiht `1.0 ** np.nan`

```
pd.SparseArray([1., 1.]) ** pd.SparseArray([1., np.nan])
# [1.0, nan]
# Fill: nan
# IntIndex
# Indices: array([0], dtype=int32)

Expected:
# [1.0, 1.0]

# NumPy result
np.array([1., 1.]) ** np.array([1, np.nan])
# array([ 1.,  1.])
```


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Sparse
description: Sparse Data Type

Comments:
whoosh you blew away lots of code. must have been work-arounds built in there for maybe an old cython or something. Just want to be sure (since tests were removed) everything still working (as may not be testing some of the numeric ops as much, though did see you added some tests)

The removed test calls `_nan` funcs which is no longer exists, and remaining logic is tested with `_op_tests` below. I'm willing to add more tests if you have anything.

thanks!
BUG: SparseSeries.shift may raise NameError or TypeError
State: closed

- [x] no existing issue
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Fixed following bugs on current master. Also, moved `TestSparseArrayIndex` to `test_libsparse`
#### 1. shift

```
pd.SparseSeries([1, 2, 3], fill_value=0).shift(1)
# NameError: global name 'kwds' is not defined

pd.SparseSeries([1, 2, 3]).shift(1)
# TypeError: %d format: a number is required, not float
```
#### 2. dtype

```
from pandas.sparse.array import IntIndex
arr = pd.SparseArray([1, 2], sparse_index=IntIndex(4, [1, 2]), dtype=None)
arr.dtype
# dtype('int64')

arr.values
# array([-9223372036854775808,                    1,                    2,
#        -9223372036854775808])

# Expected outputs
arr.dtype
# dtype('float64')

arr.values
# array([ nan,   1.,   2.,  nan])
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Sparse
description: Sparse Data Type

Comments:
thanks!

sparse is really looking nice!
BUG: Subclassed DataFrame slicing may return normal Series
State: closed

- [x] closes #11559
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Failed in codecov, but it's ok as it doesn't touch codebase.

thanks!

FYI I had fixed the codecov issue.
add support for specifying secondary indexes with to_sql
State: closed

Support for creating secondary indexes in `to_sql` via the parameter `indexes`.

The code also tries to avoid creating duplicate indexes when `keys` is specified in `SQLTable`.
(Once the PR for passing the `keys` this would be useful)

This PR introduces a new method in `SQLTable` called `_is_column_indexed` so that subclasses can easily override or change the logic.

Interesting future work will be to also support creating compound indexes.


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
does this have an open issue for this? (IIRC there is)

I didn't find this exact issue.

https://github.com/pydata/pandas/issues/7984 and https://github.com/pydata/pandas/issues/9084 are similar in that they also deal with to_sql / sql indexes but not specifically specifying which columns should be indexes.

I am interested in maybe picking up https://github.com/pydata/pandas/issues/7984 as well after this PR. Together with a PR for specifying multi-column / compound indexes.

hmm @trbs this PR looks like it should close those 2 issues, or is there something that is indicated in those that is not in your PR?

#7984 is about unique indexes... this PR would only add an index to the column not unique. I add that to this PR.

#9084 is basically about adding more parameters to get_schema(...) to that it includes the index=.... and index_label=... parameters that `to_sql` has. (so its about the dataframe index not database indices)

I could do the patch for #9084 as well since it looks kinda trivial. But it's probably better to do that in a separate PR ?

@jorisvandenbossche 

status? @jorisvandenbossche 

can you rebase / update?

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12904?src=pr) is 85.24% (diff: 100%)

> Merging [#12904](https://codecov.io/gh/pydata/pandas/pull/12904?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #12904   diff @@
==========================================
  Files           140        140          
  Lines         50563      50575    +12   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43103      43114    +11   
- Misses         7460       7461     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [54ab5be...a092198](https://codecov.io/gh/pydata/pandas/compare/54ab5be78fac125ae6efcd650d8d3f418e60754d...a0921987d5fc69b2b1b80c7b24988f1d8d3ff530?src=pr)

@jorisvandenbossche status?
btw happy to rebase again if Pandas wants to merge this PR.
closing as stale
Fix ndarray allany
State: closed

- [x] closes #12863
- [ ] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
can you rebase & update

can you rebase / update?

can you update

can you rebase / update?

can you rebase / update

can you rebase / update
closing as stale. if you want to update, pls comment.
BUG: .astype() of a Float64Index to a Int64Index
State: closed

closes #12881
closes #12866 

xref #12309 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
So we allow astyping only to F8 (and now I8). 

```
In [2]: i = pd.Float64Index([0, 1, 2])

In [3]: pd.Float64Index([0, 1, 2]).astype('i8')
Out[3]: Int64Index([0, 1, 2], dtype='int64')

In [4]: pd.Float64Index([0, 1.1, 2]).astype('i8')
Out[4]: Float64Index([0.0, 1.1, 2.0], dtype='float64')

In [5]: pd.Float64Index([0, 1.1, 2]).astype('i4')
TypeError: Setting <class 'pandas.indexes.numeric.Float64Index'> dtype to anything other than float64 or object is not supported
```

Should we allow other integer/floats? (even though we upcast them anyhow), this is [5]

e.g. we allow: 

```
In [6]: Index([0, 1.1, 2],dtype='float32')
Out[6]: Float64Index([0.0, 1.10000002384, 2.0], dtype='float64')
```

Further on [4], should this raise instead?

@sinhrks @TomAugspurger @jorisvandenbossche @shoyer 

On In[4], shouldn't we follow numpy behaviour here?

```
In [3]: np.array([0, 1.1, 2]).astype('i8')
Out[3]: array([0, 1, 2], dtype=int64)
```

ok all fixed up:

```
In [8]:  pd.Float64Index([0, 1, 2]).astype('i8')
Out[8]: Int64Index([0, 1, 2], dtype='int64')

In [9]: pd.Float64Index([0, 1.1, 2]).astype('i8')
Out[9]: Int64Index([0, 1, 2], dtype='int64')

In [10]: pd.Float64Index([0, 1.1, 2]).astype('i4')
Out[10]: Int64Index([0, 1, 2], dtype='int64')

In [11]: pd.Float64Index([0, 1.1, 2]).astype('f4')
Out[11]: Float64Index([0.0, 1.10000002384, 2.0], dtype='float64')
```
BUG: add names parameter to read_excel
State: closed

- [x] closes #12870, xref #11874
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
Currently, set argument `index_col` = `none` can disable taking column names from excel file.
I can also refactor `excel.py` so that old column names can be automatically dropped when `names=names`

**Example**
If we write `DataFrame([[1, 'foo'], [2, 'bar'], [3, 'baz']], columns=['A', 'B'])` to excel and then read,

```
[In] : read_excel(reader, 'DataFrame', names=['A', 'B'])
[out]: 
   A    B
0  a    b
1  1  foo
2  2  bar
3  3  baz
```

```
[In] : read_excel(reader, 'DataFrame', index_col=None, names=['A', 'B'])
[out]: 
   A    B
0  1  foo
1  2  bar
2  3  baz
```

Will remove test files

this PR is about a single issue - don't add other things 

Hey may I ask what is going on with codecov/patch? It pops up some files I didn't touch at all.

yeah, code cov was set a bit high.

thanks for the PR!
PERF: Only do case sensitive check when not lower case
State: closed

- [x] closes #12876
- [ ] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
thanks @RogerThomas
BUG: json invoke default handler for unsupported numpy dtypes
State: closed

- [x] closes #12554
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
- [x] valgrind clean
- [ ] windows tests ok
- [x] vbench ok

Recommend to merge after #12802 has been accepted. (valgrind should be clean then)


Labels:
name: Bug
name: Enhancement
name: IO JSON
description: read_json, to_json, json_normalize
name: Complex
description: Complex Numbers

Comments:
ok so other PR merged, you can rebase

FYI windows test https://ci.appveyor.com/project/Komnomnomnom/pandas/build/1.0.4 . Thanks for the tip!

@Komnomnomnom gr8!. note that appveyor is REALLY SLOW!

@jreback build clean (apart from msgpack & pickle again)
https://travis-ci.org/Komnomnomnom/pandas/builds/126098431

also windows is ok:
https://ci.appveyor.com/project/Komnomnomnom/pandas/build/1.0.8

just remember how to fix the tag issue.

on your local master
`git push yourorigin master --tags`

tags come from your fork (and not from the clone, pandas)

next time you branch you should be good 

thanks!

make an issue about the deprecations for 0.19.0 pls.
_get_new_index always returns index in PeriodIndexResampler
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Cleans up a method that returned different types depending on the length of the index


Labels:
name: Resample
description: resample method

Comments:
was this meant to be a PR?

Yes! Split this out of the other given it wasn't directly impacting the issue. 
Does the point make sense?

can you add some tests for validation, or you can just roll this into the other issue

OK I will roll in. I know you at times prefer things split out where it doesn't directly impact; easier for me to roll it in though

Rolling into #12874
PeriodIndex count & size fix
State: closed

closes #12774 
closes #12868 
closes #12770 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Initial attempt at fixing some of the more urgent issues


Labels:
name: Bug
name: Period
description: Period data type
name: Resample
description: resample method

Comments:
@jreback green

@MaximilianR thanks!

I used your defs of `downsample_methods` and such and found a buggie, similar to what you just fixed, e.g. #12886 

@jreback , @MaximilianR : I'm not sure what happened when this PR was committed to `master`, but it broke Travis AFAICT (even though the tests here in this PR are passing).  Can replicate the failures locally FYI (i.e. no failure on 2.x, but failure on 3.x).  Does it have to do with #12886?

I was testing some more things which happened to fail on py3. Reverted for now, the issue is #12866 which is broken.

@jreback : I presume you mean #12886?  Yeah, I don't quite understand the exceptions caught either that you get with 2.x...confusing... :confused: At least the tests for `test_resample.py` are all passing again locally :smile: 

yeah somethings wrong not sure what

Just saw this. Are there any more details on the breakage?

no this is all fixed though see the linked issue

OK, thanks
DOC: fix code-block ipython highlighting
State: closed

I noticed some warnings in the travis doc build log in the sense of "WARNING: Could not parse literal_block as "python". highlighting skipped." 
This was because some of the `.. code-block:: python` directives contained code that could not be interpreted as python. And, additionally, these code blocks were also not highlighted correctly (the `In [1]` prompt were not recognized by the default python highlighter). 

Therefore, I changed those `.. code-block:: python` directives containing IPython prompt to `.. code-block:: ipython` 


Labels:
name: Docs

Comments:
oh, this fixes it great!

Fixed the merge conflict.

So in the future, when using a `code-block` that includes some IPython `In []`/`Out []` prompts, we should use `ipython`, and otherwise `python` is OK (although actually in that case you can also use just `::` as python is the default)

@jorisvandenbossche lgtm.
TST: Add more Sparse indexing tests
State: closed

- [x] closes #4400
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Sparse
description: Sparse Data Type

Comments:
hah, actually need a rebase on this one!

Rebased, now green.

than you sir!
BUG: empty Series concat has no effect
State: closed

- [x] closes #11082, closes #12695, closes #12696
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Also includes some cleanup for #10723

CC: @AnkurDedania , @IamGianluca


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
thanks @sinhrks awesome cleanups!
BUG: SparseSeries slicing with Ellipsis raises KeyError
State: closed

- [x] closes #9467
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Sparse
description: Sparse Data Type

Comments:
thanks!
BUG: SparseSeries concat results in dense
State: closed

- [x] closes #10536
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
@jreback Added `core/concat.py` and moved all `concat` related utility funcs. Does this meet your intention?

lgtm. any tests to move as well? (from core/common.py)?

actually going to merge this. other things going to depend.
ENH: Add Index.str.get_dummies
State: closed

- [x] closes #10103
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Add `Index.str.get_dummies` which always returns `MultiIndex`.


Labels:
name: Enhancement
name: Strings
description: String extension data type and string data

Comments:
OK, added the doc.

thanks!

Nice!
TST: Add numeric coercion tests
State: closed

- [x] related to #12747 and #12780
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`

Because dtype coercion affects to lots of functions, would like to fix in the following steps. The behavior which looks incorrect are commented with `TODO_GH12747`

1(this PR) Add numeric related tests to clarify the current behaviour (and future change) 
2. Fix numeric coercion.
3. Add datetime-like tests.
4. Fix datetime-like coercion.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@jreback OK, created `test_coercion.py`.

thanks!
BUG: GroupBy with TimeGrouper sorts unstably
State: closed

- [x] closes #7453 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Because it may affect to resample perf, alternative idea is to add `TimeGropuper` path to `GroupBy.first` and `last`.


Labels:
name: Bug
name: Groupby

Comments:
hmm, ok, can you add a tests specifically for this sorting differential?

Yep, added tests using `get_group`.
BUG: SparseSeries.to_frame results in dense
State: closed

- [x] closes #9850
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

This should be after #12831.


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
so this can be rebased

Rebased. Also, forgot to note that I don't fix `to_panel` which is being deprecated.

@sinhrks thanks! you are really closing these sparse issues!

yeah `SparseDataFrame.to_panel()` is no problem
BUG: SparseSeries.value_counts ignores fill_value
State: closed

- [x] closes #6749
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Also fixed an issue which categorical `value_counts` resets name.


Labels:
name: Bug
name: Sparse
description: Sparse Data Type
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Timezones
description: Timezone data dtype
name: Categorical
description: Categorical Data Type

Comments:
Found current master doesn't work for datetimetz. Let me check.

```
values = [pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 10:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 11:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'),
                  pd.Timestamp('2011-01-01 11:00', tz='US/Eastern')]
s = pd.Series(values, name='xxx')

# NG, index must have tz
s.value_counts()
# 2011-01-01 14:00:00    3
# 2011-01-01 16:00:00    2
# 2011-01-01 15:00:00    1
# Name: xxx, dtype: int64
```

Additionally fixing the other `Categorical` issue.

```
pd.CategoricalIndex([1, 2, 3, 1, 2, 3]).value_counts(normalize=True)
# UnboundLocalError: local variable 'counts' referenced before assignment
```

@sinhrks pls rebase when you can

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12835?src=pr) is **84.04%**

> Merging [#12835](https://codecov.io/gh/pydata/pandas/pull/12835?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **+0.07%**

``` diff
@@             master     #12835   diff @@
==========================================
  Files           136        136          
  Lines         49908      49931    +23   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41876      41962    +86   
+ Misses         8032       7969    -63   
  Partials          0          0          
```
1. File `...ndas/core/groupby.py` (not in diff) was modified. [more](https://codecov.io/gh/pydata/pandas/commit/38a75311c5b138f12bb37d03e04925da964571f1/changes?src=pr#70616E6461732F636F72652F67726F757062792E7079) 
   - Misses `-63` 
   - Partials `0` 
   - Hits `+63`

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by 38a7531
BUG 12800 fixed inconsistent behavior of last_valid_index and first_valid_index
State: closed

- [x] closes #12800
- [x] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
pls add a whatsnew note (bug fixes)

@jreback 

I went ahead and added the whats new note and the comments in the bug.
Let me know if there is anything else I have to do and thanks!

thanks!