BrandonFors/pandas-github-issues huggingface dataset:
BUG: GH12824 fixed apply() returns different result depending on whet…
State: closed

- [x] closes #12824 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

In `_wrap_applied_output` for `NDFrameGroupBy`, if the first result of `apply` is `None`, find the first non-`None` result to determine behavior


Labels:
name: Bug
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
can you rebase / update

Rebased and made the changes you suggested but there was a new error with the test. It is introduced by #12743 which now throws an error in `_concat_objects` when any of the values are `None` (see below). The new PR makes a small change to deal with this, can you check if ok?

```
======================================================================
ERROR: test_groupby_apply_none_first (pandas.tests.test_groupby.TestGroupBy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../pandas/tests/test_groupby.py", line 6292, in test_groupby_apply_none_first
    result = test_df.groupby('groups').apply(test_func)
  File ".../pandas/core/groupby.py", line 651, in apply
    return self._python_apply_general(f)
  File ".../pandas/core/groupby.py", line 660, in _python_apply_general
    not_indexed_same=mutated or self.mutated)
  File ".../pandas/core/groupby.py", line 3241, in _wrap_applied_output
    not_indexed_same=not_indexed_same)
  File ".../pandas/core/groupby.py", line 824, in _concat_objects
    values = reset_identity(values)
  File ".../pandas/core/groupby.py", line 809, in reset_identity
    ax = v._get_axis(self.axis)
AttributeError: 'NoneType' object has no attribute '_get_axis'

----------------------------------------------------------------------
```

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12977?src=pr) is **84.12%**

> Merging [#12977](https://codecov.io/gh/pydata/pandas/pull/12977?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #12977   diff @@
==========================================
  Files           138        138          
  Lines         50388      50390     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42386      42388     +2   
  Misses         8002       8002          
  Partials          0          0          
```
1. 4 files (not in diff) in `pandas` were modified. [more](https://codecov.io/gh/pydata/pandas/commit/c630c8b6471468afa0ea6ff5bcb81279902bf0f6/changes?src=pr#70616E646173) 
   - Hits `-4`

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [4e4a7d9...c630c8b](https://codecov.io/gh/pydata/pandas/compare/4e4a7d9ae5b3c05b21dc0a0f3c12648236ec586e...c630c8b6471468afa0ea6ff5bcb81279902bf0f6)

can you squash

thanks @adneu
fix for read_html with bs4 failing on table with header and one column
State: closed

- [x] closes [#9178](https://github.com/pydata/pandas/issues/9178)
- [x] The test is added and passing (while failing before the fix).
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Fix as had been proposed in [PR 9194](https://github.com/pydata/pandas/pull/9194#issuecomment-209849563), but this PR was closed because of tests missing. They are added now.


Labels:
name: Bug
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
`git diff master | flake8 --diff`

Should be OK now.

small comments. ping when green.

Done. 

(I wasn't sure if I can use `(:issue 9178)`, so it's just as a regular comment)

@hnykda ahh I see. we are testing with multiple flavors. I think we default to `lxml` which I have installed so it works now. ok. then. ping on green.

Exactly. 

Everything is green.

thanks @hnykda
ENH: add .resample(..).interpolate() #12925
State: closed

- [x] closes #12925
- [X] tests added / passed in pandas/tseries/tests/test_resample.py: Base.test_resample_interpolate()

These tests simply check that df.resample(...).mean().interpolate() gives the same result than df.resample(...).interpolate()
- [X] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

-> Edit: 24/04/2016 20:25


Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Resample
description: resample method

Comments:
minor doc-change. pls squash as well.

ping when green.

thanks!

@jorisvandenbossche ahh, ok, let me fix that.

Thanks for your help too!
gh9084_get_schema_index_parameter
State: closed

closes #9084 
- new parameters are added to the end of the definition to remain backwards compatible.
- `index` defaults to `False` (not `True` as in `to_sql`) to remain backwards compatible.
- added tests


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jorisvandenbossche status?

can you rebase / update?

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12973?src=pr) is 85.23% (diff: 100%)

> Merging [#12973](https://codecov.io/gh/pydata/pandas/pull/12973?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **0.01%**

``` diff
@@             master     #12973   diff @@
==========================================
  Files           140        140          
  Lines         50563      50568     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43103      43101     -2   
- Misses         7460       7467     +7   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [54ab5be...679a799](https://codecov.io/gh/pydata/pandas/compare/54ab5be78fac125ae6efcd650d8d3f418e60754d...679a79994413378b048717c6592dab52852a53b2?src=pr)

@jorisvandenbossche status of this?
closing as stale
ENH/BUG: Sparse now supports comparison op
State: closed

- [x] no open issue
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
## on current master

```
pd.SparseArray([1, 2, np.nan]) > 0
# [True, True, nan]
# Fill: nan
# IntIndex
# Indices: array([0, 1], dtype=int32)

# Expected
# [True, True, False]
```

```
a = pd.SparseArray([1, 2, 0])
b = pd.SparseArray([0, 1, np.nan])
a > b
# ValueError: operands could not be broadcast together with shapes (3,) (2,) 
```


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Sparse
description: Sparse Data Type

Comments:
Ok, fixed and now green.

thank you sir!
COMPAT: remove NaT comparison warnings with numpy >= 1.11
State: closed

cc @sinhrks 


Labels:
name: Datetime
description: Datetime data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12969?src=pr) is **83.82%**

> Merging [#12969](https://codecov.io/gh/pydata/pandas/pull/12969?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **-0.00%**

``` diff
@@           master   #12969   diff @@
======================================
  Files         136      136          
  Lines       49642    49663    +21   
  Methods         0        0          
  Branches        0        0          
======================================
+ Hits        41617    41632    +15   
- Misses       8025     8031     +6   
  Partials        0        0          
```

[![Sunburst](https://codecov.io/gh/pydata/pandas/pull/12969/graphs/sunburst.svg?src=pr&size=660)](https://codecov.io/gh/pydata/pandas/pull/12969?src=pr)

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by 5b766d2
BUG: Sparse concat may fill fill_value with NaN
State: closed

- [x] closes #9765
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
  
  This also added tests related to #12174 and fixed `concat(axis=1)` issue.

```
# on current master
dense1 = pd.DataFrame({'A': [1, 2, 3, np.nan],
                    'B': [0, 0, 0, 0],
                    'C': [np.nan, np.nan, np.nan, np.nan],
                    'D': [1, 2, 3, 4]})
dense2 = pd.DataFrame({'E': [1, 2, 3, np.nan],
                    'F': [0, 0, 0, 0],
                    'G': [np.nan, np.nan, np.nan, np.nan],
                    'H': [1, 2, 3, 4]})
sparse1 = dense1.to_sparse()
sparse2 = dense2.to_sparse()
pd.concat([sparse2, sparse1], axis=1)
# AttributeError: 'int' object has no attribute 'ravel'
```

One point to be discussed is the logic for return type. Currently, `SparseDataFrame` is returned only when all blocks are all sparse. Because `SparseDataFrame` can't work properly if dense block is contained.

Thus, dense and sparse `concat` with axis=0 resunts in `SparseDataFrame`, and axis=1 results in normal `DataFrame`.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
I think we could return `SparseDataFrame` if _any_ blocks are sparse.

OK. Let me clarify the detail. When returning `SparseDataFrame` if any block is sparse:
- Should dense blocks be converted to sparse?
- Or keep dense blocks as it is (allow mixture of sparse blocks and dense blocks)?
  - If so, slicing dense block(s) should be normal `Series` / `DataFrame`, rather than `SparseSeries` / `SparseDataFrame`?

In an ideal world we wouldn't even _have_ `SparseDataFrame`, just `SparseSeries``. but since we do I wouldn't coerce anything, e.g. leave sparse as sparse and dense as dense. In general we DO preserve these kinds so things will tend to propogate anyhow.

> we wouldn't even have `SparseDataFrame`

Indeed...

Fixed the return type logic and now green. 

thanks!
PERF: some more perf/clean in saslib.pyx
State: closed

more perf

cc @kshedden 


Labels:
name: Performance
description: Memory or execution speed performance
name: IO SAS
description: SAS: read_sas

Comments:
@kshedden if you can run this on your example would be great (it may only be a small speedup)

I'm getting ~2.99 seconds per 100K lines on the same file I have been using in #12656 (I think I was getting 3.1-3.2 seconds with the earlier version).

My test file is not compressed and many of your changes are in the decompression, so it would not have helped. 

good point

do we have a compress file to test on ? 

are those sas7cdat?

`end_of_first_byte` needs to be an int I think (at least 2 bytes wide).  Even though its value is copied out of a uint8 array it needs to be promoted to uint16 or it overflows later on.  We have to be very careful in the decompressors about types.  It's been a few months since I did this but there is a lot of subtle stuff in there and I spent a lot of trial and error getting it to work.  The masking and shifting gets messed up if things are not exactly the right int type (too wide or too narrow both cause problems).  I don't think this is covered well by our tests.  These are very outdated and poorly documented compression algorithms.  Most of the compression codes are rarely used except in larger files.  To keep the test suite fast I created only small test files which don't hit all the compression codes.

Many of the files in our test suite are compressed, but they are small and don't necessarily use all the compression codes.  Compressed SAS files have the same name extension as decompressed ones (sas7bdat).  You might be thinking of sas7bcat which has categorical data labels (sas7bdat only supports floats and strings). 

@kshedden ok I fixed up the byte construction. ideally we would have a test for this :>

will merge on your ok.

@kshedden ok with this?

@kshedden I merged in. lmk if any more issues (or if you have better test cases)
ENH: Python parser now accepts delim_whitespace=True
State: closed

Title is self-explanatory.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
can you confirm with #12686 that all options are covered?

I think doc-string and io.rst need small edits for this (to remove that this is only for c engine)

I don't quite understand your question.  Could you clarify?

Ah, yes, good point.  Need to change what I just added since it's supported for Python now!

I meant we are now listing all of the options in that issue so that we can check them off. And you have them listed in the variables to check for the options, just want a cross check.

bonus points if you can autogenerate a section in the doc-string which explicity lists the unsupported options (by using those variables). Or maybe don't need to auto-generate, maybe just list a section in doc-string & io.rst to at least list the un-described / only certain engine supported options.

Well, they used to be a copy of `_c_parser_defaults`, which I would consider to be the "correct" version until demonstrated otherwise (Python will raise if any of these are passed with a `python` engine specified).  So in fact the list for C-only features in #12686 should be the one I created in this PR, which it is not.

@gfyoung if you can post a correct list I will update

FYI, Travis has just given the green light for my changes, so if you don't have any other code-related complaints, I'll update the documentation with `[ci skip]`.

<pre>
as_recarray
na_filter
compact_ints
use_unsigned
low_memory
memory_map
buffer_lines
error_bad_lines
warn_bad_lines
dtype
decimal
float_precision
</pre>

If any of these are non-default with a `python` engine, the parser will raise.

ok I edited the other issue, pls have a look

Are those uncapitalised headers (e.g. "undocumented in C engine only") sub sections for "Features supported in the C engine only"?  If not, all of the ones that I listed just now should also go under "Features supported in the C engine only"

ty
ENH/COMPAT: update tests for dateutil 2.5.3
State: closed

closes #12944 

so this fixes the compat with all dateutils in the test suite
add the `dayfirst` and `yearfirst` args to `Timestamp`. This make testing and compat a lot
easier. xref discussion in #7599 where we didn't implement it, though the feeling was sort of +0.


Labels:
name: Datetime
description: Datetime data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @jorisvandenbossche @sinhrks 

so we _might_ want to ban 2.5.2 from being accepted in pandas because it has a bug in the dayfirst/yearfirst logic, though it will raise an error (rather than give incorrect results). Though its an edge case.

How about showing a warning?

@TomAugspurger @jorisvandenbossche comments? (or +1 ok too: :)

As I expressed (I think) my feeling about this in the previous discussions, I am (in principle) -1 on the addition of dayfirst/yearfirst to `Timestamp` (it is just not needed and bloats its constructor). 
Is it really a big difference for the tests? I don't directly see how that would be a problem, I would just think that the yearfirst/dayfirst tests do not test for  `Timestamp`?

@jorisvandenbossche IIRC you were not -1, but indifferent. But I dont' see why we are not compat, it just causes even more confusion. Yes this mainly fixes tests, but to be honest. Having `Timestamp` and `to_datetime` be pretty much exactly the same is a good thing.

ok I took `dayfirst/yearfirst` off of the public `Timestamp` constructor. So this is just a compat PR now that fixes up tests mainly.

@jreback sorry, late reply. I was maybe indeed more indifferent, anyway, you already merged now. It would maybe indeed be more easy (implementation wise) that `to_datetime` and `Timestamp` work similar, but my main concern is that I would not want us to encourage people to use `Timestamp` for anything else as manually constructing a timestamp when needed (and then you should always provide a standard string, and not use eg `dayfirst`), so not eg for parsing strings, as for this there is `to_datetime`.

@jorisvandenbossche yeah, I think would have to add `format` & `dayfirst` and `weekfirst` to be really consistent. I actually don't think its a big deal really. People _mis-use_ `pd.to_datetime` all the time. So it gives them more fuel to the fire!

thanks for the comments.
Add defaults to swaplevel() parameters i and j
State: closed

Closes #12934 feature request.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
@brandon-rhodes I just updated master, so you will need to rebase as unrelated issue with python-dateutil will cause a test for fail :< (we are now skipping)

did u mean to close this?

No. I am used to the Phabricator work flow, and this pull request is teaching me the GitHub workflow :) Stand by!

@jreback Okay, I pulled and pushed some stuff, and all the code reappeared. :) I added the issue number, and rebased against your latest master — let me know how things look!

can u update tests for MultiIndex as well

pls squash as well

Oh — GitHub doesn't let you squash as you land the pull request? Yet another difference with the workflow I'm used to. Yes, I'll do the squashing myself, and also try addressing your two comments — thanks!

yes I will squash on merge - since we rebase its usually a bit easier to use logical commits and squash 

Just saw your comment — had already squashed on my end :) Per the Sphinx docs, I went with `versionchanged` instead, since it seemed closer to the intent of a change like this. I also found the only test I could find where the multiindex tests used swaplevel and supplemented it. Let me know what you think! (Also: wow, your integration tests take a while!)

@brandon-rhodes lgtm.

any comments? @sinhrks @shoyer @jorisvandenbossche @TomAugspurger 

This seems reasonable enough to me. :+1: 

👍 

thanks @brandon-rhodes 

Thanks for considering this contribution from me! :)
BUG: provide SparseArray creation in a platform independent manner
State: closed

makes creation w/o specifying a dtype choose `np.int64/float64` regardless of the platform. This is similar to how Series works. This only is exposed on windows (as the default ndarray creation is np.int32) rather than np.int64

maybe _should_ use `pandas.core.series._sanitize_array` (but needs some tweeks I think).

cc @sinhrks 


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
doing this via `_santize_array` add a dependency here, so may not want to do this.
COMPAT: Add Pathlib, py.path support for read_hdf
State: closed

- [*] closes #xxxx
- [*] tests added / passed
- [*] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Closes #11773

Travis-ci test JOB_NAME=27_slow_nnet_LOCALE consistently fails due to locale issue...  zh_CN.GB18030??


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO HDF5
description: read_hdf, HDFStore
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls add a whatsnew note (you can put in Enhancements)

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12930?src=pr) is **83.91%**

> Merging [#12930](https://codecov.io/gh/pydata/pandas/pull/12930?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **+<.01%**

``` diff
@@             master     #12930   diff @@
==========================================
  Files           136        136          
  Lines         49918      49927     +9   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41885      41895    +10   
+ Misses         8033       8032     -1   
  Partials          0          0          
```
1. File `...das/tseries/index.py` (not in diff) was modified. [more](https://codecov.io/gh/pydata/pandas/commit/f8173edf1226abb96d980334e60b5df73f751b38/changes?src=pr#70616E6461732F747365726965732F696E6465782E7079) 
   - Misses `-1` 
   - Partials `0` 
   - Hits `+1`

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by f8173ed

Ok, updates made.  On the whatsnew item, there was already a pathlib item for read_csv down in the bugfixes section so I put the read_hdf item there.

if you can rebase / update

Ok, rebased.  

@quintusdias lgtm. minor comments on docs. pls update / ping when green.

Comments addressed

Changes made.  

thanks @quintusdias
BUG: .asfreq on resample on PeriodIndex/TimedeltaIndex are not
State: closed

closes #12926


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
a tad hacky on this soln, as seems `DatetimeIndex` does lots of manipulations inside the binner to fix fencepost issues, while the other indexes (Period/Timedelta) don't seem to need this.

cc @MaximilianR created a structure to support common testing for resampling (the `Base` test class)

Nice re the tests.

Tangential: I had a go deprecating the `how` keyword to make the Period stuff much simpler. I ran into issues with simple upsampling into super periods; I don't think that is possible without `DatetimeIndex` machinery atm. It's on my list though.

yep - I think we can still use much of the DatetimeIndexResampler machinery - just return the PeriodIndex at the end (and of course handle special cases)
BUG: Fix argument order in call to super
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`

`super` should have arguments (type, object), not (object, type).

I've run `test_fast.sh` using Python 2.7 and it gives [2 failures](https://github.com/pydata/pandas/files/225866/testfail.txt) relating to language locales (I'm using en_GB). Pretty sure these are unrelated to the commit.


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
hah I guess we don't have a test for that

can u create a test that hits that path as well

Sure. I could find a concrete sub-class of `PandasObject` that doesn't implement `memory_usage`, instantiate that, get its `sizeof()` and check it returns something of the correct type? For example, from a quick look, looks like the `GroupBy` class is such a class.

can you rebase / update

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12924?src=pr) is **83.76%**

> Merging [#12924](https://codecov.io/gh/pydata/pandas/pull/12924?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **0.38%**

``` diff
@@             master   #12924   diff @@
========================================
  Files           138      135     -3   
  Lines         50392    49640   -752   
  Methods           0        0          
  Branches          0        0          
========================================
- Hits          42406    41583   -823   
- Misses         7986     8057    +71   
  Partials          0        0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [4b50149...68e4b90](https://codecov.io/gh/pydata/pandas/compare/4b501497a88c118c2bfdcdbc4a5b216b68b1b88c...68e4b9096c4b9fd4098a397335a694add5316ff5)

@jreback Added a test to hit the path.

thanks! nice way of testing this!
Correct out-of-bounds error with large indeces
State: closed

- [x] closes #12527 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
the build is pending for 13 hours now ...

yeah I think that the OSX queue was bogged for a while; these _seem_ to be proceeding now (though still quite a backlog)

@jreback the build is green

thanks!
Add pandas logo to repo
State: closed

- [ ] partially closes #12911
- [x] tests added / passed (N/A)
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
This image has New BSD license, doesn't it? i have set it to cc-by-sa-4.0 [on wikipedia](https://commons.wikimedia.org/wiki/File:Pandas_logo.png)