BrandonFors/pandas-github-issues huggingface dataset:
add bins argument to Histogram function
State: closed

The primary argument used when plotting histograms is `bins`. This argument is not in the pandas documentation so you do not see it when auto-tabbing for documentation. I have added  `bins=10` as an argument to the dataframe and series histogram function while also adding a line of documentation. `10` is the matplotlib default bin value so this is only making `bins` visible, not adding or changing any plotting functionality.


Labels:
name: Visualization
description: plotting

Comments:
can you move the argument to the end of the arg list pls.

I'll look closer at this later, but the usual convention for our plotting functions is that any keyword arguments used in matplotlib's `plot` or `hist` or `bar`, etc. are passed in `**kwargs`.

Also you may want to use `df.plot(kind='hist')`.

@jreback updated argument list
@TomAugspurger  I think this is import documentation for histograms. You can use `df.plot(kind="hist")` but I typically use `series.hist(bins=20)`.  For historical reasons or otherwise dataframes and series can call `hist` directly so having histogram-specific information there makes sense to me. I can certainly see why you wouldn't want `bins` in `df.plot()`. 

I misspoke earlier. `hist` isn't even a supported kind for `df.plot()`

@TomAugspurger what should we do with this?

I'm happy whatever you decide to do. I just rebased against upstream/master in case its a go.

The downside is that this will be inconsistent with all our other plotting functions, where the arguments that go directly to the maptlotlib function are passed in the **kwargs section. But, it's good for discoverability, so let's merge.

@zachcp could you move the `bins` down in the docstring to match the order in the function signature?

Looks like you had a test fail too.

made a couple minor fixes to padd test (need to pass bins as arg) and added two small tests.

OK thanks.

Don't need a release notes entry since this is really just a doc change. There isn't any change to the API

Oops, looks like we missed a call to hist in `hist_frame`. Not surprising given how spread out the code is. I've got a PR coming

merged https://github.com/pydata/pandas/pull/6935
API: make Series.sort/order arguments in the same order
State: closed

- drop unused `Series.sort` order argument
- added `Series.sort na_position` argument to conform with `Series.order`
- default kind for `Series.order` is now `quicksort`, same as `Series.sort` and numpy default

closes #6847


Labels:
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @unutbu 

any objections here?

@jtratner @cpcloud @jorisvandenbossche @hayd 

fine by me 

:+1: 

Are `sort` and `order` now fully equivalent? (apart from that `sort` is inplace and `order`not?) Maybe this should be added to the docstrings (eg a "See also")

yes, good point

fixed in https://github.com/pydata/pandas/commit/c1440e9b1b206b9c892105ce68f3b88f2436f5a5

as an aside, maybe we should add an `inplace` argument to `sort=True` (so if False it will return `order`)?

And then also in `order` the other way around with `inplace=False`?

I think that is right....seems innocuous enough (famous last words)...will make an issue
Additional documentation for holiday calendars
State: closed

@jreback here are the additional holiday examples.  Let me know if you want something else.


Labels:
name: Docs

Comments:
@rockg sorry I think I canceled your Travis build instead of mine on accident. Could you retriggger the build with

```
git commit --amend -C HEAD
git push origin <yourbranch> -f
```

Sorry!

I'm going to do another commit so no worries.

ping when u r ready to merge 

@jreback  Think I understood what you wanted regarding separate ipython blocks.  Please take a look.  Further, there was one bad piece of code that got in during the merge which I took out.

beautiful! thanks
ENH: Scatter plot now supports errorbar
State: closed

This is enhancement for #5638. I sometimes want to plot scatter with errorbars, thus I've refactored to support it.

![figure_1](https://cloud.githubusercontent.com/assets/1696302/2632517/21edcfd6-be65-11e3-9cd9-97bad65c8129.png)

Also, this includes following fixes. I've also modified tests to check the number of errorbars actually drawn.
- There are some types which expected to work, but results in `ValueError`.
  - `float` (MPL can accept it)
  - `unicode` (in Python 2.x)
  - Other list-like, such as `iterator`
- A bug related to time series.

```
>>> s = Series(rand(6), index=range(6), name='x')
>>> err_df = DataFrame(rand(6, 3) / 10, index=range(6), columns=['x', 'y', 'z'])
>>> s.plot(yerr=err_df)               # This works, errorbar appears using err_df x column

>>> s.index = tm.makeDateIndex(k=6)
>>> s.plot(yerr=err_df)               # But this doesn't, errorbar doesn't appear
```


Labels:
name: Enhancement
name: Visualization
description: plotting

Comments:
can you rebase this on top of master again, then we can review

Sure. Rebased.

@TomAugspurger can you give a look see

@sinhrks need a release note for this pls

Rebased. I've thought release note is not requreid because error bar itself is new function of v0.14, and release note has following description. Should I prepare separate one?

> Added error bar support to the `.plot` method of `DataFrame` and `Series` (:issue:`3796`)

oh just add this note number (or pr numbe) to that one 

OK, added PR number.

@sinhrks @TomAugspurger 

Thanks @sinhrks. Could you rebase off master now that the Area plot is merged? There are a few conflicts in `tools/plotting.py` Other than the we're good.

 @TomAugspurger  Sure, rebased to latest master. Plotting test has been passed in Travis, and looks fail in offset test.

pls rebase 1 more time, master is fixed up

Thanks, rebased.

Thanks.

@jreback a failure in `pandas.io.tests.test_packers.TestNumpy` and `pandas.io.tests.test_packers.TestNumpy` under the allowed failures build. Is this OK?

@TomAugspurger yeh...thouse fail every once in a while on 3.4, not really sure why. if you would like to investigate give it a go!

otherwise this PR is ok

Excellent. Thanks @sinhrks
CLN apply pep257 via docformatter
State: closed

Not really for merging, but I wanted to see how [docformatter](https://github.com/myint/docformatter) handled codebase (e.g. any bugs). Perhaps we can make these changes incrementally.

cc #6248


Labels:

Comments:
I skimmed though the first third and last third line changes... haven't checked the middle section (as there are many)

cc @myint Are the last couple of comments bugs ? Also, the second (and third) commit here is from a second (resp. third) pass of docformatter... so potentially these changes are not correct? After third pass it stabilises.

The sentence splitting is based on heuristic regular expression. It certainly can be faulty.

The underline removal was intended as I purposely used docformatter in the past to get rid of them. That feature probably should removed though.

The indenting seems odd. docformatter is supposed to completely leave the description untouched if it looks at all complicated (has numbered list or has anything that cannot be simply wrapped).

@myint thanks, hmm the regex is tricky, looks like it was just caught by an unlucky word wrap... in fact was manually word wrapped, so I think it's ok to be tripped up by that! Do you want me to post indenting as a sep issue?

Sure.

Thanks

@myint that is actually fixed in master (presumably as a weird case of the underline). :)

There is a docformatter option of pep8radius. Anyway, I'm closing this, as too disruptive, maybe can do after pep8 splurge post 0.14.1.
BUG: fix replace bug where different dtypes in a nested dict would only replace the first value
State: closed

closes #6689


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
ok

merge the plot issue fix first

going to add a release note

@jreback merging on next pass .. ok w/ you?

yep

> closes #6689

Hello,I ve got the same problem how did u fix it?
BUG: adding np.timedelta64 to DatetimeIndex with tz outputs incorrect
State: closed

```
>>> idx = pd.date_range(start='2010-11-02 01:00:00', periods=3, tz='US/Pacific', freq='1H')

>>> idx + offsets.Hour(3)
[2010-11-02 04:00:00-07:00, ..., 2010-11-02 06:00:00-07:00]
Length: 3, Freq: H, Timezone: US/Pacific

>>> idx + datetime.timedelta(hours=3)
[2010-11-02 04:00:00-07:00, ..., 2010-11-02 06:00:00-07:00]
Length: 3, Freq: H, Timezone: US/Pacific

>>> idx + np.timedelta64(3, 'h')          # incorrect
[2010-11-02 11:00:00-07:00, ..., 2010-11-02 13:00:00-07:00]
Length: 3, Freq: H, Timezone: US/Pacific
```


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
looks good

OK, I've added a comment and rebased. There seems to be unnecessary line remains in `release.rst` thus removed it.
DEPR:  Deprecate DateRange
State: closed

Deprecate DateRange in favor of DatetimeIndex, perfromed in commit 6fe2db57.  Deprecation warning was first given in v0.8.  

Related: #6641, #6813 


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jreback Let me know if you need anything else from me to close this issue.  Thanks

@jsexauer thanks!
BUG: DataFrame._reduce was converting integers to strings in mixed-type case.
State: closed

closes #6806


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
pls add a release note and good 2 go

@jreback g2g

k....in general also nice to rebase when adding something (don't worry in this case though). as it makes merging very easy.

I thought I did rebase and squash. Well, it should be rebased from an hour or two ago.

I had just pushed something.....:)

thanks!

lol. I had to wait for travis to ping green! /me searches for some github lock command. 

what I usually do is see that the auto-merge is indicated, if not, pull, fix, and repush
BUG/ENH: Fix to_excel representation of inf values
State: closed

Fixes #6782 by converting inf values to string (default is `'inf'`).
This still round trips successfully. According to Stack Overflow post I
found, there isn't a specific representation of `INF` values -
http://stackoverflow.com/questions/15704538/how-can-i-express-nan-values-in-excel
so we're going to make the decision _for_ the writers. This also
(implicitly) fixes an issues where round-tripping an Excel file with an
inf value would cause an `OverflowError` and fail to read back.

cc @jmcnamara and @arthurgerigk-rocket


Labels:

Comments:
should also handle -inf

@jreback good call - any preference between `not np.isfinite(val)` and `abs(some_value) == np.inf`?

I think u should use isinfinite maybe has better platform compat (or may be the same)
the preprnd a minus to inf_rep  if negative?

Yeah, realized that after I said it haha

okay, that's done.

ok looks fine

side note - this formatting to excel space should be done by blocks - would make it much faster (but separate issue)

that's a good point. - let's accept this for now (along with the extra tests to spec out inf) and then we can go back to that later on.
CLN: replace pandas.compat.scipy.scoreatpercentile with numpy.percentile
State: closed

PR to fix #5824.
- Replaces `compat.scipy.scoreatpercentile` with `numpy.percentile`.
- Sets `axis=0` in a few tests, because `axis=None` by default on `numpy.percentile`, which returns a scalar (i.e., operates on a flattened version of the array).
- Fixes a test that fails after the switch due to differences in how fractions are computed. (Created a `np.timedelta64` directly since `to_timedelta` doesn't support enough precision.)

Let me know if you'd like to see any changes..


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls add a release note referinc the issue (do it in API changes, even though just a compat change).

side note, I am trying to figure out why @wesm included this in the first place, as I think `np.percentile` has been around a while? any idea

@jreback, sorry for the delay, but I've added a second commit to this PR with:
- added to release notes
- `Series.quantile` on datetime[ns] series changed to return a `Timestamp` object (with an explicit test)
- added a test for `Series.quantile` on timedelta64 series, which didn't require any
  special handling, though the test is skipped at numpy 1.6.x, because
  numpy.percentile throws the following exception with timedelta64 objects
  pre-1.7:
  
  ```
      TypeError: ufunc 'multiply' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe'
  ```
- removed timedelta64 handling (introduced by #6558) in `Series.quantile`,
  because `not np.isscalar` is always false and, in my testing,
  numpy.percentile properly returns timedelta64 objects on numpy 1.7 & up.

No idea why np.percentile wasn't used originally, but will note that scipy.stats.scoreatpercentile existed first. Perhaps, when the scipy dependency was removed from pandas, the difference in rounding made copying scoreatpercentile the easiest way forward.  Will also note that scipy has deprecated scoreatpercentile in favor of np.percentile.

@gdraps looks good

just a couple of minor comments

timedeltas are thoroughly broken under numpy < 1.7
so I gave up on trying to make them work a while ago

@jreback, updated the last commit of this PR based on your feedback -- let me know if you see anything else

@gdraps thanks for the work on this gr8!
BUG: parser_source as a filename with multibyte characters in Windows(non utf-8 filesystem)
State: closed

fopen() in Windows doesn't accept utf-8 encoded filename with multibyte characters, so need to convert it to filesystem encoding.
Set 'utf-8' as default in case sys.getfilesystemencoding() return None.
sys.getfilesystemencoding() will return 'mbcs' in Windows, and will 'utf-8' or user setting in other systems.


Labels:
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv
name: Windows
description: Windows OS

Comments:
thanks

can u write a test that fails without this change and works with it ?

@hshimizu77 tests?

I have added a test but I'm wondering it's right place or not.

that's the correct place

@TomAugspurger can you test this on OSX?

@hshimizu77 ok...let me test on windows; I think we may put this in w/o the test (but don't change the test ATM).

as I don't think pandas will install on some systems because of the encoding of the filename (in theory it should always work, but don't want to bork older systems).

The test errors

```
(pandas-dev) ~/E/p/l/p/s/p/p/i/tests (hshimizu77-bug-fix=) nosetests test_parsers:ParserTests.test_read_csv_example_in_windows_filesystem
E
======================================================================
ERROR: test_parsers.ParserTests.test_read_csv_example_in_windows_filesystem
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/admin/Envs/pandas-dev/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/admin/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/io/tests/test_parsers.py", line 1827, in test_read_csv_example_in_windows_filesystem
    self.read_csv(path)
  File "/Users/admin/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/io/tests/test_parsers.py", line 55, in read_csv
    raise NotImplementedError
NotImplementedError

----------------------------------------------------------------------
Ran 1 test in 0.067s

FAILED (errors=1)
```

Seems to work from the interpreter though:

``` python
In [1]: from test_parsers import *

In [2]: path = tm.get_data_path(u'日本語ファイル名テスト_read_csv_in_win_filesystem.csv')

In [3]: pd.read_csv(path)
Out[3]: 
     A    B    C
0  100  200  300
1  aaa  bbb  ccc

[2 rows x 3 columns]
```

I think you want `read_csv` not `self.read_csv`

@hshimizu77 

for this

```
path = tm.get_data_path(u'日本語ファイル名テスト_read_csv_in_win_filesystem.csv')
```

use `u(.....)` instead of the a literal u (this is translated by the compat module as needed for py2/3

merged via 79df67a7e7e034c40f5560dc8de31ecf3da14bff

thanks!
BUG/API: disallow boolean arithmetic operations
State: closed

closes #6762


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite

Comments:
cool...merge on green

crap just realized this will fail on py3 ... `operator.div` isn't there anymore only `truediv`

less trivial than i thought ... will finish in a bit

@cpcloud @jreback I know this is already merged some time, but just stumbled on this. Is there a good reason to disallow this? Because it is an API change (so people could rely on this behaviour), _and_ it does also work in numpy this way (at least for + and *).

no this only disallowing silly things

True + True
True \* True works but you should just use &

Is this silly?

```
In [18]: pd.__version__
Out[18]: '0.13.0'
In [19]: s = pd.Series(range(5))
In [20]: (s > 1) * (s < 3)
Out[20]:
0    False
1    False
2     True
3    False
4    False
dtype: bool
```

I know we say: just use & instead. And no problem at all that is the recommended way. 
But above code now fails, while it was working previously. So potentially people could rely on this. Certainly given it also works this way in numpy:

```
In [21]: (s.values > 1) * (s.values < 3)
Out[21]: array([False, False,  True, False, False], dtype=bool)
```

So I was just wondering if there is a good reason to disallow this, apart from "people should use &"

it breaks Numexpr and so will simply raise
if u r doing boolean ops you should use boolean operators

it will fail loudly

it didn't work before if the frame was big enough (and hit Numexpr)

this was disallowed because numexpr doesn't have the op codes to deal with ops like 

``` haskell
Bool OP Bool -> Bool
```

where `OP` is a (non-bitwise) arithmetic operator (e.g., `+`, `-`, `*`, `/`). It should either fail across the board or work across the board (the board being either numexpr is used or it's not) but since numexpr doesn't work with those ops (IMHO rightly so) we have to make it fail. It shouldn't work for objects < some size threshold and then not work for larger objects
BLD: spring cleaning on Makefile
State: closed


Labels:
name: Build
description: Library building on various platforms

Comments:
I have I idea how to test this
but looks ok

manual testing ... really only a dev and power user feature ... most ppl prob not using make

@cpcloud seems ok....I guess rebase and put it in...we'll see

well this caused a kitten to die

https://travis-ci.org/pydata/pandas/jobs/22389411

how ironic that that was a bug i fixed

that test works on py27 mpl 1.3.1, trying with 1.3.0

hm well it appears `pip install matplotlib==1.3.0` doesn't work

oh fudge the `random.choice` probably only choice a single value, therefore creating a single plot and thus failure ... i'll make it not random

I think 1.3.0 got removed somehow....1.3.1 is current

odd though because I rebuilt the wheels just recently....
BUG: fix metacharacter replacement bug in DataFrame.replace()
State: closed

closes #6777


Labels:
name: Bug
name: API Design
name: Strings
description: String extension data type and string data

Comments:
merge in green

on

blah yahoo junk is failing again

@jreback any idea why this doesn't say green even tho https://travis-ci.org/cpcloud/pandas/builds/22297556 is all green?

it turns green after the build on pydata/pandas runs (and not cpcloud/pandas)

essentially it's run twice (I think this is a Travis thing)

just wait a bit

ah ok thanks ... green now bombs away
DOC: Issue #6763 - Corrected MERGE example
State: closed

Corrected the example provided in the MERGE function docstring.

Closes #6763 


Labels:
name: Docs

Comments:
@jorisvandenbossche looks ok to me...merge when you are ok

Yes, certainly OK. Thanks @SplashDance for the quick fix! Your first contribution is a fact

Well, I'll admit -- it does feel pretty good having made my first contribution to pandas... Thanks for your help!
BUG: Series.iteritems should be lazy
State: closed

Closes #6760


Labels:

Comments:
this seems fine
ok when green 

A minor nitpick: the test won't catch lists in py3, because `getslice` is deprecated since py2.6 and is removed in py3:

``` python
In [3]: sys.version
Out[3]: '3.3.3 |Continuum Analytics, Inc.| (default, Nov 25 2013, 17:28:04) \n[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]'

In [4]: list()
Out[4]: []

In [5]: _4.__getslice__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-ff32cb569e0e> in <module>()
----> 1 _4.__getslice__

AttributeError: 'list' object has no attribute '__getslice__'
```

This is probably not an issue within this PR, since it was fixing a py2-specific inconsistency, but worth keeping an eye on.

Thanks. I didn't realize it was deprecated. I'll switch to checking for `reverse`.
DOC: documented that .apply(func) executes func twice on the first time
State: closed

Related issues are #2656, #2936 and #6753.


Labels:
name: Docs

Comments:
good..thanks!

@anton-d An advice: try to not mix tabs and spaces for indentation.

And something else, the alignment of your code example was not fully correct so it wouldn't render well (the `..ipython::` should start at the same indentation as the text of the warning), but that are tricky rst things to get right, so no problem (I corrected it here: https://github.com/pydata/pandas/commit/95a562cb01d6f0954cdc2021e02d4d8234ae5ba3). And thanks for the contribution!

is this deployed ?
i still got the issue(first group called twice): with version=0.24.2
d = pd.DataFrame({"a":["x", "y"], "b":[1,2]})
```
def identity(df):
    print (df)
    return df

d.groupby("a").apply(identity)

```

outputs:

   a  b
0  x  1
   a  b
0  x  1
   a  b
1  y  2
Quick shift fixes
State: closed

Adjustments to the quick fix PR. 

https://gist.github.com/dalejung/9731798#file-vb_suite-log

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_shift_c_order_axis1                    |  14.4524 |  52.8037 |   0.2737 |
frame_shift_c_order_axis0                    |  16.9516 |  46.5320 |   0.3643 |
frame_shift_axis0                            |  15.2656 |  16.3440 |   0.9340 |
frame_shift_axis1                            |  22.2600 |  22.0567 |   1.0092 |
```

Note, that the speed up was really in sending a C order array to `np.take`. You'll notice that master is slow for `frame_shift_c_order_` which are tests where the `block.values` is c_contiguous. Thus current master always transposes. 

The original example had it's speed up because it `np.roll`ed `df.values` which is C ordered (fast) while the `.blocks` were were (slow).


Labels:

Comments:
good catch on this
thanks!