BrandonFors/pandas-github-issues huggingface dataset:
ENH:  Series between inclusive pair (GH: 12398)
State: closed

- [ X  ] closes #12398
- [ X ] tests added / passed
- [ X ] passes `git diff upstream/master | flake8 --diff`
- [ X ] whatsnew entry

Amends pandas.Series.between method to set different inclusive/exclusive endpoints.

inclusive : Boolean to indicate whether or not to include both the left
            and right endpoints (True: a <= series <= b, False: a < series < b),
            or an iterable boolean pair to set them separately, e.g
            inclusive=(False, True) for a < series <= b.


Labels:
name: Enhancement
name: API Design

Comments:
I guess. pls add some doc-string examples

use `closed='left'|'right'|None` like we do for `DatetimeIndex`

```
closed : string or None, default None
    Make the interval closed with respect to the given frequency to
    the 'left', 'right', or both sides (None)
```

up pls update for 0.18.1. pls also create a new issue to deprecate `.between` (and i'll mark for 0.19.0)

closed in favor of #13027
BUG: Return UTC DateTimeIndex when specified in to_datetime
State: closed

Title is self-explanatory.  Closes #11934.


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
1) Will add "what's new" entry as soon as Travis gives me a green light.
2) Where should I put a test for this bug?  There is no `test_tools.py` file AFAICT.

tseries/test_timeseries

look for similar tests

@jreback : All tests are passing.  Should be good to merge then.

@jreback : All tests are passing.  Should be good to merge then.

thanks!
TST: skip bool_ops_scalar tests for numexpr if numpy >= 1.11
State: closed

numpy >= 1.11failing on these


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
closing as https://github.com/numpy/numpy/pull/7285 will likely be merged to revert this behavior
Use correct random choice function in code snippet
State: closed

A hidden code snippet uses `tm.choice()` when it should use `np.random.choice()`


Labels:
name: Docs
name: Groupby
name: Clean

Comments:
cc @ellisonbg

@carroux thanks!

so I think this was for numpy < 1.7 compat.

pls remove this: https://github.com/pydata/pandas/blob/master/pandas/util/testing.py#L173

as well (its possible there are some other references as well), pls search the docs; in the code I think something will fail.

thanks!

oh, saw @TomAugspurger merged this, ok. then.

Ha, sorry bout that. FYI

``` bash
pandas/tests/frame/test_query_eval.py
99:        a = tm.choice(['red', 'green'], size=10)
100:        b = tm.choice(['eggs', 'ham'], size=10)
152:        a = tm.choice(['red', 'green'], size=10)
153:        b = tm.choice(['eggs', 'ham'], size=10)
246:        a = tm.choice(['red', 'green'], size=10)
978:        a = Series(tm.choice(list('abcde'), 20))

pandas/tests/test_graphics_others.py
638:            gender = tm.choice(['male', 'female'], size=n)
712:            gender_int = tm.choice([0, 1], size=n)

pandas/tests/test_graphics.py
63:            gender = tm.choice(['Male', 'Female'], size=n)
64:            classroom = tm.choice(['A', 'B', 'C'], size=n)
3864:            gender = tm.choice(['male', 'female'], size=n)

pandas/tools/tests/test_merge.py
239:            df = DataFrame({'a': tm.choice(['m', 'f'], size=3),
241:            df2 = DataFrame({'a': tm.choice(['m', 'f'], size=10),
248:            df = DataFrame({'a': tm.choice(['m', 'f'], size=3),
251:            df2 = DataFrame({'a': tm.choice(['m', 'f'], size=10),
257:            df = DataFrame({'a': tm.choice(['m', 'f'], size=3),
259:            df2 = DataFrame({'a': tm.choice(['m', 'f'], size=10),

vb_suite/groupby.py
146:obj = tm.choice(list('ab'), size=n).astype(object)
```

are the other occurrences, if we want to open a cleanup issue.
API: correctly provide __name__ for cum functions
State: closed

closes #12021

```
(py3.5)[Tue Apr 05 08:16:18 ~/miniconda/envs/py3.5/pandas]$ ipython
Python 3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:24:55) 

In [1]: pd.Series.sum.__qualname__
Out[1]: 'Series.sum'

In [2]: pd.Series.cumsum.__qualname__
Out[2]: 'Series.cumsum'

In [3]: pd.Series.cumsum.__module__
Out[3]: 'pandas.core.series'

In [4]: pd.Series.cumsum.__name__
Out[4]: 'cumsum'
```


Labels:
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @mrocklin

0.18.0

```
In [1]: Series.sum.__qualname__
Out[1]: '_make_stat_function.<locals>.stat_func'
```

this PR

```
In [1]: Series.sum
Out[1]: <function pandas.core.series.Series.sum>

In [2]: Series.sum.__qualname__
Out[2]: 'Series.sum'

In [3]: Series.cumsum.__qualname__
Out[3]: 'Series.cumsum'
```

cc @mrocklin
this doesn't solve the `cloudpickle` issue, but I think makes progress

cc @mrocklin did you have a chance to try this? 

Ah, sorry for letting this slip
BUG #12336: Fix init categorical series with scalar value
State: closed

closes #12336 

Changed the tests to match the other ones in `test_fromValue`.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
normally we don't need to open a new PR, just push to the same one in the future. ok for now

@jreback Sorry about opening a new PR; got confused about rebasing.
Tests are green now.

thanks!
fmt in TimeFormatter is now changeable
State: closed

`datetime.time` objects are useful as index values, e.g. in a data frame
containing data of an experiment lasting a few minutes or hours. When
plotting such data frame, the x-axis ticks are formatted as `'%H:%M:%S'`
as standard. In most cases, either `'%H:%M'` or `'%M:%S'` contains enough
information, and the `'%H:%M:%S'` format is superfluous.

The xticks were formatted based on the fmt variable in the `__call__` method. In this commit, the tick format is saved in the `TimeFormatter.fmt`
variable. This is set in the `__init__` method to `'%H:%M:%S'`, but can
later be changed, e.g.:

```
ax = df.plot()
ax.xaxis.get_major_formatter().fmt = '%M:%S'
```


Labels:

Comments:
is this meant to solve a particular issue? which one?

what is the purpose here?

pls comment on the purpose.

I think I explained the purpose in my initial comment.

The purpose is to be able to change the format used by the TimeFormatter class for formatting time when plotting time data. Currently this is only possible by somehow providing another formatting class, which is cumbersome. Due to this change, it is possible to provide a new format using just one line of code.

pls show a complete example, which would form the basis of a tests then.
docs: add hyperlinks to str.get_dummies
State: closed

The description used in the summary table is the first sentence of
str_get_dummies()'s docstring, edited to match the style of other
summaries.

I tried to build this branch locally – the _Method Summary_ section doesn't render with this change and I can't see why, it looks syntactically correct to me…


Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
thanks!
BUG: Fix bug where sort_index if unsorted levels.
State: closed

Closes #12261


Labels:

Comments:
Not sure if this is the best way to solve this, but it looks like the bug in #12261 is because the labels are sorted, but the levels aren't.

It seems like the better way is `is_lexsorted` should return `False` if the levels aren't sorted because you could get the potential mismatch, but that's a layer below this issue so I didn't want to touch it.

this is not the right way to go about this.

Ok... Does this help you see what this issue is enough (you didn't seem to have an idea from the issue*) to recommend what the appropriate path would be?
- Seems to be the level sorting issue unless it's a red herring.

yeh I haven't really delved into it much. 

Ok...
BUG: Common NumericIndex.__new__, with fixed name handling
State: closed

closes #12309 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls add tests for the name propogation

Ping?
(in general, are you postponing non-0.18.0-tagged PRs to after the 0.18.0 release?)

@toobaz yes

though non-api changes or straightforward stuff I will merge (so pinging is good)

@toobaz if you'd like to rebase / update, pls reopen.

Rebased, updated, addressed your comments, except a couple of them which were unclear to me. I don't think I am allowed to reopen.

While extending `est_ensure_copied_data` as you suggested, a bug came out in `DateTimeIndex`, now fixed.

(not sure why the code isn't being updated... shall I open a new PR?)

@toobaz yes, you will have to open a new PR. That's something with github that you cannot reopen a PR once the branch has been updated while the PR is closed (so we should have reopened it before you pushed updated to the branch ..). Anyway, no problem to open a new one! :-)
FIX: pivot_table dropna=False drops index level names
State: closed

closes #12133 
level names are lost on `MultiIndex.from_arrays(cartesian_product(table.index.levels))`


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
pls add a whatsnew entry (0.18.0 bug fixes)
you also have linting errors

pls rebase/update

closed in favor of #12650 

I'll do my best to learn how to squash commits ASAP.)
DOC/DEPR: remove remaining occurences of mpl_style from docs (GH12190)
State: closed

@TomAugspurger There were some remaining usages of `options.display.mpl_style` in the docs, so removed those + also removed the reference to it in the text.


Labels:
name: Docs
name: Visualization
description: plotting
name: Deprecate
description: Functionality to remove in pandas

Comments:
:+1:, thanks for catching those.
CLN: change getargspec -> signature
State: closed

Change signature in compat to be able to return defaults, varargs,
kwargs

Closes #12171


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I've almost messed up committing to the master, but I've tried to make everything clear.

Regarding the changes: the signature in the compat was needed to return defaults, varargs, kwargs in the decarators.py.

```
def foo(a,b=2,*args,**kwargs):
    pass

signature(foo)
```

was returning ['a','b'] and ['a','b','args','kwargs'] in Python 2 and Python 3, respectively.

Now 

```
def foo(a,b=2,*args,**kwargs):
    pass

signature(foo).args
```

returns ['a','b'] in both Python 2 and 3.

small changes. ping when green.

@jreback added reference to Django issue (does it suffice?) and put import at the top.

@troglotit thanks!
DOC: some rewording of highlights
State: closed

@jreback I proofread the highlights and made some edits (but language is always a bit subjective). Comments?


Labels:
name: Docs

Comments:
trivial change. lgtm. merge away (I have a small doc fix w.r.t. the float indexers section but orthogonal)

I fixed the float indexers code block issue already on master, a colon was missing

oh ok, then i'll rebase on top
BUG: Prevent abuse of kwargs in stat functions
State: closed

Addresses issue #12301 by filtering `kwargs` argument in stat functions to prevent the passage of clearly invalid arguments while at the same time maintaining compatibility with analogous `numpy` functions.


Labels:
name: API Design
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Suggestions on where I could place tests to make sure such an issue doesn't crop up again?  Tried finding a place, but no one place really stood out to me.

tests/frame/test_analytics.py

or maybe better in test_generic.py (if you place it in Generic then it will be run for each of the structures)

@jreback : Off topic, but I noticed that there's a MASSIVE queue on Travis right now.  There seem to be a ton of duplicate builds that could be cancelled.

its actually progressing, travis was down most of today. 

your builds occur independently of pandas FYI

Ah, I see.  Also, I like the suggestion for `test_generic.py`.  I'll go with that one.

lgtm. ping when green.

pls add a whatsnew entry (you can put in API changes)

@jreback : Travis is happy again.  Good to merge if there's nothing else.
BUG: GH12290 where tz_convert used values of uninitialized arrays
State: closed

closes #12290


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
aha!

@kawochen : Also, you should put "Closes gh-12306" in your commit message so that the issue gets automatically closed once the PR is merged.

the closes directive for the actual PR Is created at merge time

updated according to @gfyoung's comment

thanks @kawochen and @gfyoung
BUG: Fix subtraction with timezone Series
State: closed

Addresses issue in #12290 with timezone `Series` subtraction in which single element `Series`
objects containing tz-aware objects would return a timedelta of zero, even though it visually could not be the case.

The bug was traced to the conversion of the contained timezones to UTC, in which the method call was somehow returning `NaT`, even though attempts to replicate that behaviour were unsuccessful. This new
method call fixes the issue and is in some ways more intuitive given the comment above the conversions.


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
fixed by #12306 

ah sorry didn't realize there was already a PR.  thought I was quick!

ahh, but I think @kawochen  soln fixes the root of the problem.

Closing this PR because #12306 provides a better solution to the issue.
NaT.isoformat() returns 'NaT'.
State: closed

NaT.isoformat() returning 'NaT' allows users to use Timestamp to
rehydrate any _Timestamp object from its isoformat. This means that it
is safe to use _Timestamp.isoformat() as a serialization format for
timestamps where NaT is a valid value.


Labels:
name: Datetime
description: Datetime data dtype
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
I am not sure if this is actually a good change. NaT is not defined by iso so idk if this is misleading. I just wanted to raise the issue.

this would change the way JSON serializes I think. IIRC its the only thing that uses `.isoformat` internally at least.

@llllllllll this looks fine. pls rebase and well put in.

build is passing, thanks!

thanks @llllllllll
Improvements to str_cat
State: closed

closes #11435 
closes #11334.

Modified behavior of str_cat to ignore NaN values in the Series by default.

Added some doc examples to make the behavior with nan more clear, and catch an error condition when a sep is passed instead of an others (array) as an argument and print a more intuitive error message.

This is a retry of PR #12000.

Before, if NaN values were present in the Series, calling `cat()` on it would return NaN, unlike the rest of the `Series.str.` API. This behavior has been amended to ignore NaN values by default.
(#11435)

A new, friendlier ValueError was also added to protect against the mistake of supplying the `sep` as an arg, rather than a kwarg.
(#11334)

``` python

    >>> Series(['a','b',np.nan,'c']).str.cat(sep=' ')
    'a b c'

    >>> Series(['a','b',np.nan,'c']).str.cat(sep=' ', na_rep='?')
    'a b ? c'

    >>> Series(['a','b',np.nan,'c']).str.cat(' ')
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-3-338c379049e9> in <module>()
    ----> 1 Series(['a','b',np.nan,'c']).str.cat(' ')

    [...]

    ValueError: Did you mean to supply a `sep` keyword?
```


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Strings
description: String extension data type and string data

Comments:
looks good.

some small comments and you have linting errors:

```
linting -> pandas/core
pandas/core/strings.py:4:80: E501 line too long (84 > 79 characters)
linting -> pandas/io
linting -> pandas/stats
linting -> pandas/compat
linting -> pandas/sparse
linting -> pandas/tools
linting -> pandas/tseries
linting -> pandas/tests
pandas/tests/test_strings.py:2463:24: E231 missing whitespace after ','
pandas/tests/test_strings.py:2463:28: E231 missing whitespace after ','
pandas/tests/test_strings.py:2463:32: E231 missing whitespace after ','
pandas/tests/test_strings.py:2471:5: E303 too many blank lines (2)
linting -> pandas/computation
linting -> pandas/util
```

ping when green.

@hack-c thanks, I make the flake fixes.

Hey @jreback so is this all done?

yep all done. thanks!

just trying to get the 0.18.0RC1 out the door :)

Ok, great @jreback!  

How can I help? I want to do more :)

@hack-c great!

just troll issues, you can select the difficulty labels as you want!
Fix #12292 : error when read one empty column from excel file
State: closed

Fix #12292 
also fix #9002 


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
Just made my changes only inside excel.py itself to avoid mess up the html parser.

@jreback please take a look

@BranYang does this also fix #9002 

@BranYang this looks good.

@jreback I can confirm that I captured the StopIteration and return an empty DataFrame. So I believe this also fix #9002 

thanks @BranYang
BUG: Refine validation of parameters to RangeIndex.__init__
State: closed

closes #12288


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
I updated the issue.

thanks. added some more tests, pls duplicate (or could refactor) for testing on other numeric indexes w.r.t. name propogation. Best would be to move this to `Base` as then will test for all indexes.
BUG: Series.plot() doesn't work with CustomBusinessDay frequency
State: closed

Closes #7222
Missing freqstr 'C' in `_period_code_map`
I found `pandas/tseries/tests/test_frequencies.py:TestFrequencyCode.test_freq_code` cover this change.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
tests!

`pandas/tseries/tests/test_frequencies.py:TestFrequencyCode.test_freq_code` cover `_period_code_map` change itself.
however, I think it's also good idea to add a test on `test_graphics.py` since the original issue was related on plotting.

be back soon.

added a test.

thanks @scari
BUG: provide chunks with progressively numbered (default) indices
State: closed

closes #12185 

Notice the test I fix was indeed wrong - I had written that line as a workaround waiting for this fix.


Labels:
name: Bug
name: API Design
name: IO CSV
description: read_csv, to_csv

Comments:
needs to use the parser data directly. If not available, then needs to be exposed. This makes it less complicated from a future reader perspective.

can you rebase/update

can you rebase / update

can you rebase and i'll have a look

Rebased. Certainly two problems left (the one you pointed out - we should already use the available info inside the parser - and the one concerning the empty chunk I commented above): I can take care of them but I don't know when, so if you want the fix in 0.19.0 I suggest to postpone them to future PRs.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12289?src=pr) is 85.23% (diff: 100%)

> Merging [#12289](https://codecov.io/gh/pydata/pandas/pull/12289?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #12289   diff @@
==========================================
  Files           140        140          
  Lines         50415      50423     +8   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42970      42977     +7   
- Misses         7445       7446     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [98c5b88...381e3b3](https://codecov.io/gh/pydata/pandas/compare/98c5b88d6f9f7bb0afa4fbae49a045d93a1cb33f...381e3b39268fb0a7da9df43622306b998dd98045?src=pr)

OK, test on `index_col` added, ready as far as I can tell.

Hope the sub-section is fine... I was unable to generate the docs (probably my `IPython`'s fault).

@toobaz The whatsnew explanation looks good I think! But can you move it a bit below to the API changes section? (it can be a subsection of that)

@toobaz Can you also take a look in the io.rst docs on the explanation about chunksize to see if there is something that should/could be changed? (edit: took a quick look in the io-chunking section http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking, and I think this is good)

@jorisvandenbossche : thanks for the comments, should have fixed everything. I also checked `io.rst` everywhere `chunksize` appears, and the index is never mentioned.

Ahem... please kill https://travis-ci.org/pydata/pandas/builds/147442885 ...

@jreback this looks good to me

thanks @toobaz
BUG: Fixes KeyError when indexes don't overlap.
State: closed

Closes #10291 


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Can you edit your first post to include "Closes #10291, Closes #12133" so that Github will auto-close those issues when we merge?

Also would need a release note in `doc/source/whatsnew/v0.18.0` under bug fixes and maybe add a note to the docstring of `crosstab` explaining that this is the outcome for non-overlapping indices. Thanks!

you need a test to cover #12133 

pls add a whatsnew as well

Thought it may be easier in the near term to reduce the scope of this change to fixing the issue associated with #10291 and not attempt a fix at #12133.

I've added #12133 to my todo list so I'll try to get back to it.  Hopefully it's as simple as adding a test case.

thanks @tshauck 

`if values_passed and not values_multi and not table.empty:`
Excuse me, but how does this cover #12133? index names are dropped earlier in `if not dropna:` block. We need to pass names to `from_arrays`:

```
m = MultiIndex.from_arrays(cartesian_product(table.index.levels), names=table.index.names)
...
m = MultiIndex.from_arrays(cartesian_product(table.columns.levels), names=table.columns.names)
```

UPD: sorry, i thought that `it's as simple as adding a test case` means that it covers #12133 but tests are needed
UPD2: #12327

if u read the comments about it does not cover 12133
Hdf warning
State: closed

In accordance with #12268 this is now a separate PR.

I think that this warning could be useful either way for no non-daily hdf user:
- Telling that string method is preferred
- Adding a warning


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
@dacoex idea being that you separate the changes into 2 PRs, one containing the `pd.` changes, the other the doc-changes w.r.t. hierarchical keys. (there are 2 PR's now, but they both contain all the changes)

Sorry, my git skills do not make it.
I will update them during the next days.

hahha, no prob. you will want to rebase on master, then delete that commit.

e.g.

`git rebase -i master`

then force push

pls rebase on master

I hope I got it right now ;-)

anything holding this one back?

thanks!
Fix #12169 - Resample category data with timedelta index
State: closed

closes #12169 


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timedelta
description: Timedelta data type

Comments:
pls add a whatsnew entry as well.

Can you also add a simpler example (as well) that just hits the construction path; put it in the `series/test_constructors`

@jreback added tests

thanks @BranYang 

one tip, when you add a bug fix put it in a blank spot not at the end. that way you won't get merge conflicts (that's why I randomly put blank lines there)