BrandonFors/pandas-github-issues huggingface dataset:
DOC: improve binary operator docs (fixes #10093)
State: closed

fixes #10093

@jorisvandenbossche this covers `Series`, `DataFrame` and `Panel`, please take a look 


Labels:
name: Docs

Comments:
The resulting text would now be: `Binary operator subtraction.` 

I think that maybe also the 'binary operator' part can be confusing for people. I don't know if this terminology is that familiar for most users? 
Thinking about an alternative, I came up with `Subtraction of series and other (binary operator ``sub``).`

actually the current implementation is such that the doc would be: `Binary operator sub (subtraction)`

Ah, yes, I see the `desc` line! 
But still, my reservation about 'binary operator' stays the same (but to be clear, just adding the full name between parantheses is already a big improvement)

hmm... yeah the wording for this is kind of tricky 

if we go with `Subtraction of series and other (binary operator sub)` maybe it should be `Subtraction of other from series` instead to indicate the order of operation, and so `rsub` would be `Subtraction of series from other`. But I'm not sure if this is actually more clear. 

Proposal:

```
Subtraction of series and other, element-wise (binary operator `sub`).

Equivalent to ``series - other``, but with support to substitute a fill_value for missing data in
one of the inputs.
```

But agree that the "series and other" is maybe better as "other from series", but that will depend a bit from operator to operator how to best specify it.

Can you maybe also add a 'see also' section each time to the reversed/non-reversed version (add -> radd, radd -> add, ...)

So I ended up restructuring the code quite a bit in order to pack more information into the docstring, also added a unit test for it. Please take a look.

Basically the docstrings look like this now:

```
Signature: pd.DataFrame.sub(self, other, axis='columns', level=None, fill_value=None)
Docstring:
Subtraction of dataframe and other, element-wise (binary operator `-`).

Equivalent to ``dataframe - other``, but with support to substitute a fill_value for
missing data in one of the inputs.

Parameters
----------
other : Series, DataFrame, or constant
axis : {0, 1, 'index', 'columns'}
    For Series input, axis to match Series index on
fill_value : None or float value, default None
    Fill missing (NaN) values with this value. If both DataFrame locations are
    missing, the result will be missing
level : int or name
    Broadcast across a level, matching Index values on the
    passed MultiIndex level

Notes
-----
Mismatched indices will be unioned together

Returns
-------
result : DataFrame

See also
--------
DataFrame.rsub
```

Nice!  Thanks a lot.

Noticed a small problem, instead of:
`element-wise (binary operator `-`)`
we should have:
`element-wise (binary operator `sub`)`

it's fixed now

Travis is failing, but seems to be related to another issue

yep will be fixed by #10152

OK, merging then. @mortada Thanks a lot!
v0.16.1 docs
State: closed


Labels:
name: Docs

Comments:
did you merge this? 

No, this are the fixes I did for the docs, but I just applied this to my checkout of the 0.16.1 version tag for building the docs (so everything in here is in the online stable docs already).  
This is still open for review. If everything is OK, it can be merged. Otherwise I can do a new doc build to update the online docs

ok, perfect. Everything updated (pandas.pydata.org/PyPI), so sending out ann soon.
Default values for dropna to "False" (issue 9382)
State: closed

As per discussion in Issue 9382, changes all HDF functions from having default of dropping all rows with NA in all non-index rows.

closes #9382

Followup for #9484


Labels:
name: API Design
name: IO HDF5
description: read_hdf, HDFStore

Comments:
this will need to go into 0.17.0 as its a backward-incompatible change.

Moved.

One question: I just noticed that saving an h5 file in the whatsnew file is creating an orphan file in the doc folder. Is there a standard way to deal with this? Should I import tempfile and use that? Delete after creation?

search for suppress in io.rst to see how to do this

Got it. Amended commit. Thanks!

@jreback updated. 

@nickeubank thanks. This is an API change so being deferred to 0.17.0. We are going to release 0.16.2 next week.

Sounds good!

On Wed, Jun 3, 2015 at 11:27 AM jreback notifications@github.com wrote:

> @nickeubank https://github.com/nickeubank thanks. This is an API change
> so being deferred to 0.17.0. We are going to release 0.16.2 next week.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10097#issuecomment-108561769.

status?

@jreback rebased and read to go I think!

@jreback thanks for comments. Think all updated now. 

I also wanted to see an example with a panel (which was the original case here) 

oh, ok. In the test, in io.rst, in whatsnew, or all of the above?

test and io.rst. and put a link to the docs in whatsnew 

@jreback  think we're good, though test_pytables is failing all over the place at the moment due to an apaprently change in array_to_datetime(). I get `TypeError: array_to_datetime() got an unexpected keyword argument 'require_iso8601'`

you need to `make` as some cython code change

Great! Thanks so much @jreback . All better. :)

ok, minor correction, pls squash.

@jorisvandenbossche ?

@jreback updated and squashed. Thanks. 

Two more comments:
- it seems that dropna is not included in the `to_hdf` docstring?
- can you explicitly add how to restore the previous behaviour in the whatsnew file? (you said it in the docs, but I think it is important that this is also mentioned in the whatsnew)

@jorisvandenbossche 
- added comments in what's new. 
- Not sure how to change to_hdf docstring. There's something doing on in the pytables.py table with docstrings I don't quite follow. Can you point me in the right direction?

I think it's docstring is in core/generic.py or core/frame.py

thanks @jorisvandenbossche , found it and amended the commit. 

thanks @nickeubank
BUG: Timestamp properties may return np.int
State: closed

Closes #10050.

Also added `daysinmonth` and `dayofweek` properties to `NaT` and perform tests.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can you also confirm (maybe add tests?) for `Timedelta` that it does the same

@jreback That's better. Added type check for `Timedelta` and its `.components`

looks good
go ahead and merge on green 

Thanks for the review. Rebased and merged.
BUG: Series.fillna() raises if given a numerically convertible string
State: closed

First reported here but it's been dormant for a long time: https://github.com/pydata/pandas/pull/9043 

I think it'd be good to patch this bug. @jreback please let me know which release this should go to and I can add a release note. Thanks. 


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
this looks ok for 0.16.1.

sounds good, I added a release note 

ok, ping on green.

@jreback green now 

thanks!
PERF: increase performance of str_split when returning a frame
State: closed

Closes #10081. 

This simply removes the unnecessary `Series()` in the creation of the DataFrame for the split values in `str_split`. It vastly improves performance (20ms vs 3s for a series of 20,000 strings as a test) while not changing current behavior.


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
Hmm... this actually causes significant problems when dealing with NaNs.

@cgevans this code path has changed in master a bit (though issue still remains). 

@cgevans can you rebase and see where this is?

The change in the code path means this particular fix would need to be rewritten, but it seems this would need further changes to avoid changing behavior. I've written a bit about the problem in #10081.

One solution is to change the DataFrame constructor such that it handles lists of lists differently, making it consistent with lists of Series.

The other, somewhat hackish method is to take the array that comes from str_split, change all NaNs to [None], construct a DataFrame, and then change all Nones to NaNs.

This new version now tries to handle NaNs in a bit of a roundabout way, but still in a manner that will be fast.

pls add a release note. pls show the results of the benchmark for this (there is a join_split one, but you may have to add one)

you can do (you might want to see which is better perf):

I think this can ONLY? return lists or scalars?

```
if isinstance(list, x): # might need com.is_list_like if this can return list/tuple/ndarray/Series
    # ok
else:
    [ x ]
...
```

```
In [8]: x = [ 'foo' ]

In [9]: pd.lib.isscalar(x) and pd.isnull(x)
Out[9]: False

In [10]: x = 'foo'

In [11]: pd.lib.isscalar(x) and pd.isnull(x)
Out[11]: False

In [12]: x = np.nan

In [13]: pd.lib.isscalar(x) and pd.isnull(x)
Out[13]: True
```

pls also add a note to 0.16.2 performance imprv section.

Here is the benchmark result:

```
Invoked with :
--ncalls: 3
--repeats: 3


-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
strings_join_split_expand                    |  48.8970 | 1554.2669 |   0.0315 |
strings_join_split                           |  36.1670 |  36.3259 |   0.9956 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [5e65f02] : PERF: Increase performance of string split when expand=True
Base   [bc7d48f] : disable some deps on 3.2 build
```

minor whatsnew note change. pls ping when revised and will merge.

Done.

excellent ty!
Extended DataReader to fetch Dividends and Stock Splits from Yahoo!.
State: closed

- Added yahoo-actions data_source to DataReader;
- Added get_data_yahoo_actions that returns a DataFrame containing
  DIVIDEND and SPLIT corporate actions fetched from Yahoo! Finance;
- Added unit tests;
- Added documentation example.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
this is a nice add, pls direct your PR to https://github.com/pydata/pandas-datareader

as that is how updated to the `io/data.py` will be handled going forward.

thanks!
DEPR: Deprecate str.split return_type
State: closed

Closes #9847, Closes #9870. Default value is `expand=False` to be compat with`return_type='series'`. May better to change the default to True in future (and show warning about it)?

CC @jreback @jorisvandenbossche @sreejata


Labels:
name: Strings
description: String extension data type and string data
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jreback, @mortada Thanks, updated.

Looks good! 

Another thing I encountered while reviewing: there is no good explanation of the `n` kwarg of split in the docstring, and the default differs between `str_split`/docstring (default of None) and the signature of `split` itself (default of -1)

@jorisvandenbossche Explanation of `n` is written as notes, thus moved it to main part. Also, modified the docstring to meet the `str.split` without changing the default values. Or may better to change defaults as None and -1 has the same meaning?

I also found current arg names are different from standard `str.split` (`sep` and `maxsplit` ) ...

merged via https://github.com/pydata/pandas/commit/8b8984258cbede113d362c18508f85fcef4eaaf5

@sinhrks thank you!

if you have minor doc changes, pls fee free to make another PR.
Fix for GH9992 - use create_tempfile to remove permission-related errors when running tests
State: closed

Fixes #9992.

I wanted to take a stab at developing pandas. I am a novice at both git/github and I hope I did the merge and PR correctly.

@jreback, please let me know if there is anything I can fix.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO HDF5
description: read_hdf, HDFStore

Comments:
looks fine.

can you add a release note in v0.16.1 

and squash, pls ping on green.

see http://pandas-docs.github.io/pandas-docs-travis/contributing.html for git help.

merged via https://github.com/pydata/pandas/commit/2dc9e0e874f2fbfd0d159bfc74b62c45625c42b0

thanks!
DOC/DEPR: port pandas.rpy to rpy2 guide
State: closed

Addresses the doc part of #9602


Labels:
name: Docs
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks for this. Thoughts on adding a section explaining _why_ we're deprecating it? (no real maintainers, `rpy2` is better... I think there were some others).

Yes, indeed, good point. I was also thinking that, but in the end just wrote that it is "similar" functionality. 

I think the main reason is to "avoid duplication"? (having the same functionality in both pandas and rpy2 is not needed, and anyhow, those who used `pandas.rpy` were in fact already using `rp2`) 
Saying that `rpy2` is better is not yet fully correct .. as there are still some things better in the pandas version to be ported (eg NaN handling, see the issue).

Similar is probably good enough. I just didn't read closely enough. :+1: 

@jorisvandenbossche I just stumbled across this PR in the Travis CI build queue and was wondering how one would convert a pandas dataframe to a R dataframe with `rpy2`. Any chance the docs could contain a minimal how-to for this very frequent use-case?

The `pandas2ri.py2ri` function as mentioned in the new docs in this PR (or even only doing `pandas2ri.activate()` will be enough). But I assumed knowledge of the existing functions, so this is equivalent to the `convert_to_r_dataframe`. 
But it would be nice to also have docs geared towards new users (who didn't know the old way). But let's say that is for another PR.

lgtm. merge when ready.
BUG: Fix typo in parsers.py
State: closed

closes #9266 

np.uin8 should be np.uint8


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
Looks like this code path was never tested then :laughing: @twu interested in writing a test that _does_ cause cause a `ValueError` here?

this is already partially done in #9350 

just need a complete test

pls feel free to lift that code and complete. Ideally we'd like to do this for 0.16.1, releasing in next couple of days.

closing in favor of #9350
BUG: median() not correctly handling non-float null values (fixes #10…
State: closed

closes #10040 


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Timedelta
description: Timedelta data type

Comments:
Note that the existing unit test is actually wrong

```
In [1]: from pandas import Series, Timestamp

In [2]: from datetime import timedelta

In [3]: s = Series([Timestamp('20130101') + timedelta(seconds=i*i) for i in range(10)])

In [4]: td = s.diff()

In [5]: td
Out[5]:
0        NaT
1   00:00:01
2   00:00:03
3   00:00:05
4   00:00:07
5   00:00:09
6   00:00:11
7   00:00:13
8   00:00:15
9   00:00:17
dtype: timedelta64[ns]
```

`td.median()` should really be `00:00:09` instead of `00:00:08` as it is currently asserting. 

pls add a release note for 0.16.1

@jreback it's updated 

ty!

side issue. I briefly looked it and it appears to work, but are there cases where precision drops are an issue? e.g. we have int64 converted to float64 which drops precision, but it DOES seem to work. if you can experiment and see (I don't have a good) example. If you find somethign pls file an issue.

sure I'll play with this and see if I can find examples where precisions become an issue. It does seem like it could be a problem.
BUG: Series.align resets name when fill_value is specified
State: closed

```
import pandas.util.testing as tm

ts = tm.makeTimeSeries()
ts.name = 'ts'
a, b = ts[2:], ts[:5]

# OK (names are preserved)
aa, ba = a.align(b)
aa.name
# ts
ba.name
# ts

# NG, specifying fill_value resets names
aa, ba = a.align(b, fill_value=-1)
aa.name
# None
ba.name
# None
```


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@jreback Could you review this?

lgtm

Thanks!
DOC: add warning for datareader changes in 0.17.0 (GH10004)
State: closed

closes #10004 
![remote data access pandas 0 16 0-249-g8f0f417 documentation](https://cloud.githubusercontent.com/assets/953992/7471821/9a99d0b4-f2f9-11e4-9a34-90b80bf9d4bf.png)


Labels:
name: Docs

Comments:
Looks good to me!

@hayd @dstephens99
BUG: Panel.from_dict does not set dtype when specified
State: closed

currently `Panel.from_dict` is just ignoring the `dtype` argument:

```
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: df = pd.DataFrame(np.array(range(6)).reshape(3, 2))
In [4]: pa = pd.Panel.from_dict({'a': df, 'b': df}, dtype=float)
In [5]: pa.dtypes
Out[5]:
a    int64
b    int64
dtype: object
```

this PR sets the `dtype` and fixes the docstrings in some places 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
ok, guess this is fine for consistency.

put release for 0.17.0 (the file exists in master now)

@jreback release note for this is updated 

thanks
ENH: Series.resample performance with datetime64[ns] #7754
State: closed

It fixes the performance drop.
See %timeit result [here](http://nbviewer.ipython.org/gist/scari/1173a9519d8b89000c08).
closes #7754 

@jreback Would you review this PR? Thanks!


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
hmm, would need a benchmark, e.g. post a sample timeit for before/after (on a good sized test frame)

``` python
import pandas as pd

intSeries = pd.Series(5, pd.date_range(start='2000-01-01', end='2000-01-08', freq='555000U'), dtype='int64')
timeSeries = intSeries.astype('datetime64[ns]')

%timeit intSeries.resample('1S', how='last')
%timeit timeSeries.resample('1S', how='last')
%prun intSeries.resample('1S', how='last')
%prun timeSeries.resample('1S', how='last')
```

### Before

100 loops, best of 3: 18.8 ms per loop
1 loops, best of 3: **17 s** per loop
449 function calls (441 primitive calls) in 0.021 seconds
[snip]
22378318 function calls (21773499 primitive calls) in 24.219 seconds
[snip]

### After

100 loops, best of 3: 19 ms per loop
100 loops, best of 3: **20 ms** per loop
449 function calls (441 primitive calls) in 0.020 seconds
[snip]
418 function calls (410 primitive calls) in 0.020 seconds
[snip]

ok, looks ok, can you add a vbench in `vb_suite/timeseries.py`. you can make similar to the above. but make smaller, so that the slowest time doesn't take too long (max 1s)

Added test. @jreback please review again.

```
$ ./test_perf.sh -b HEAD~1 -t HEAD -r timeseries
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_resample_datetime64               |   3.7426 | 2009.6250 |   0.0019 |

Target [dd6ffa3] : ENH: Series.resample performance with datetime64[ns] #7754
Base   [3128215] : cleanup test for GH10044
```

Updated whatsnew/v0.17.0.txt.

looks good ping when green

on green! :) @jreback 

ty!
BUG: read_hdf modifying passed columns list (GH7212)
State: closed

Closes #7212. If this object is a list, make a copy and then pass to process_axes - otherwise, just pass the object directly (in this case it is most likely none).

A test for this behavior seems complex and somewhat contrived when compared with what exists in test_pytables.py, so I didn't add one --- let me know any thoughts on this.


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
I know it seems somewhat contrived, but we do try to add tests even for things like this.

@shoyer thanks for the feedback! much better solution. i'll get a test together as well.

@ajamian can you add a test for this?

@jreback absolutely, should be getting to it this weekend.

@shoyer @jreback thanks for the feedback --- let me know if you have any other comments

@ajamian one last thing -- can you add a note to the release notes for 0.17? otherwise this looks good to me!

yep lgtm as well - pls add a comment on the test with the issue number

I put the note into doc/source/whatsnew/v0.17.0.txt, I think this is where it belongs...and comment added to test. Thanks guys - let me know if you have any more comments.

Thanks!

thanks Tom!
DOC: Add infer frequency to docs
State: closed

Was there a reason infer_freq and inferred_freq weren't in the docs?  Did I pick the right spot to add it? Does the feature work well?

I set out looking for a feature like this, searching "guess frequency", "auto detect frequency".  


Labels:
name: Docs
name: Frequency
description: DateOffsets

Comments:
minor change, ping when pushed.

@jnmclarty Could you also update the `infer_freq` docstring with:
- it also works for TimedeltaIndex
- it raises a ValueError when not having 3 values? (now it only mentions a TypeError when the the provided values are no DatetimeIndex)

@jorisvandenbossche side issue, we don't appear to have `PeriodIndex` at all in `api.rst`....do we have an issue for this?

Ideally, we should also resolve this one: https://github.com/pydata/pandas/issues/5082

On the PeriodIndex API docs: yes, the issue is here: https://github.com/pydata/pandas/issues/6384

@jorisvandenbossche on #5082 

In reality, I just think we could just fix `infer_freq` & `infered_freq` to return an actual offset object (which reprs to a string). rather than adding addtional methods. These compare correctly to strings anyhow.

@jreback Okay to merge.

thanks!
DOC: add PeriodIndex.asfreq docstring
State: closed

Part of https://github.com/pydata/pandas/issues/6384

I guess PeriodIndex don't show up in the API docs at all. Not sure if I'll get to any more.


Labels:
name: Docs

Comments:
@jorisvandenbossche thanks. Should be good now I think

@TomAugspurger added some more .. :-)

@TomAugspurger can you update?

@jreback done. Merge when you're ready.

@TomAugspurger I think you forgot to add your changes

Seems to be good now.

I realize after we merge index-repr prob need to fix s bunch of doc strings :(

e.g. will be.

```
In [1]: pidx = pd.period_range('2010-01-01', '2015-01-01', freq='A')

In [2]: pidx
Out[2]: PeriodIndex(['2010', '2011', '2012', '2013', '2014', '2015'], dtype='int64', freq='A-DEC')

In [3]: pidx.asfreq('M')
Out[3]: PeriodIndex(['2010-12', '2011-12', '2012-12', '2013-12', '2014-12', '2015-12'], dtype='int64', freq='M')

In [4]: pidx.asfreq('M', how='S')
Out[4]: PeriodIndex(['2010-01', '2011-01', '2012-01', '2013-01', '2014-01', '2015-01'], dtype='int64', freq='M')
```

Indeed. A search for `<class 'pandas.tseries.period.PeriodIndex'>` and alike can help there maybe. I will open an issue for that. -> #10094
ENH Add normalize method to Series dt accessor.
State: closed

_Unclear whether this was intentionally not included in the dt accessor._


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
pls add this into the test `tests/test_series/test_dt_namespace_accessor` as well.

merged via https://github.com/pydata/pandas/commit/7f43252f155734c17b7a08752e4f22614b5b367e

thanks!