BrandonFors/pandas-github-issues huggingface dataset:
added support for selecting multiple nth values
State: closed

I'm not 100% sure this is the best way to implement this, but I think it'd be nice for GroupBy.nth() to support selecting multiple values. For instance if we have

```
df = DataFrame(1, index=pd.date_range(start='1/1/2013', end='6/30/2014', freq='B'), columns=['a', 'b'])
```

and currently we can easily get the nth entry for each month by:

```
df.groupby((df.index.year, df.index.month)).nth(0)
```

however we can't easily get, for instance, the first 5 entries for each group without having a more complex loop in the user code. If nth() accepts a list of integers we would be able to simply do

```
df.groupby((df.index.year, df.index.month)).nth(range(5))
```

This PR makes the above work when there's no dropna. I can't quite figure out the section where dropna is handled, and therefore I'm currently requiring that no dropna be passed when n is a list of values. I'm happy to improve this and also enhance the tests. Please let me know your feedback. Thanks. 


Labels:
name: Enhancement
name: Groupby
name: API Design

Comments:
FYI this is prob useful

but u know you can do .head(5) for first 5 entries

Ah indeed ... head() and tail() can cover some of these cases. This is probably still useful for the general case though. 

can you add an example of this to: http://pandas-docs.github.io/pandas-docs-travis/groupby.html#taking-the-nth-row-of-each-group (maybe right after this). Also ok with forming a sub-sectin of the the 'nth' usages

Good idea - I added an example to groupby.html, also added a few more unit tests. Thanks. 

@mortada thanks for this. I forgot to have you put in a release note.

can you do a quick PR with a release note for this (use the PR number), in the enhancements section. and add the example as well. thanks

sure happy to do that
Remove from start/end dates if tz is not None (#7901, #7835)
State: closed

Below fixes date_range when input dates are localized.  In that case inferred_freq is not None and so the dates do not have their tzinfo removed.  This causes a fixed offset to be applied when the range is created.  If tzinfo is removed, they will be correctly localized.  

Fixes #7901 
Fixes #7835.


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
pls add a release note (with both issues)

@jreback Any reason for the weird spacing in v0.15.0?  https://github.com/pydata/pandas/blob/master/doc/source/v0.15.0.txt 

yep

it makes it so that he can edit a line and not have a merge conflict if someone else does a different line 

used to be that a merge would cause everyone pr to have to rebase which was a bit annoying

just pick a random line and put in the note

this looks fine

@rockg ready to go?

@sinhrks comments?

It's ready.

thanks!
TST/BUG: Rename html encoding test files.
State: closed

html test_encode fails on OSX 10.9.4 due to missing dash in `utf32` encoding
string.

The test encoding string is derived from a split of the test file name.

Only utf32 is affected but this commit changes the utf8 and utf16 containing filenames as well for the sake of consistency.

``` ======================================================================
ERROR: test_encode (pandas.io.tests.test_html.TestReadHtmlEncodingLxml)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jmorris/Code/pandas/pandas/io/tests/test_html.py", line 624, in test_encode
    from_string = self.read_string(f, encoding).pop()
  File "/Users/jmorris/Code/pandas/pandas/io/tests/test_html.py", line 619, in read_string
    return self.read_html(fobj.read(), encoding=encoding, index_col=0)
  File "/Users/jmorris/Code/pandas/pandas/io/tests/test_html.py", line 607, in read_html
    return read_html(*args, **kwargs)
  File "/Users/jmorris/Code/pandas/pandas/io/html.py", line 851, in read_html
    parse_dates, tupleize_cols, thousands, attrs, encoding)
  File "/Users/jmorris/Code/pandas/pandas/io/html.py", line 714, in _parse
    raise_with_traceback(retained)
  File "/Users/jmorris/Code/pandas/pandas/io/html.py", line 708, in _parse
    tables = p.parse_tables()
  File "/Users/jmorris/Code/pandas/pandas/io/html.py", line 178, in parse_tables
    tables = self._parse_tables(self._build_doc(), self.match, self.attrs)
  File "/Users/jmorris/Code/pandas/pandas/io/html.py", line 527, in _build_doc
    parser = HTMLParser(recover=False, encoding=self.encoding)
  File "/Users/jmorris/anaconda/envs/py27/lib/python2.7/site-packages/lxml/html/__init__.py", line 1664, in __init__
    super(HTMLParser, self).__init__(**kwargs)
  File "parser.pxi", line 1598, in lxml.etree.HTMLParser.__init__ (src/lxml/lxml.etree.c:100669)
  File "parser.pxi", line 792, in lxml.etree._BaseParser.__init__ (src/lxml/lxml.etree.c:93393)
LookupError: unknown encoding: 'utf32'
```

Test failure was not resolved by building the most recent lxml with static dependencies and most recent versions of libxml2 and libxslt.

I contacted the lxml mailing list. 

http://mailman-mail5.webfaction.com/pipermail/lxml/2014-July/007239.html

It was suggested that problem may be in OSX libiconv though `iconv -l`doesn't show the bug on my system.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Funny, my branch builds fine. Is friday night a bad time for FRED?

hah

rebase on master 

I had fixed this issue already 

done.

@cpcloud looks ok to me

weird issue! @jmorris0x0 thanks for fixing!
BUG: do not assume 0 is in index of potential Series
State: closed

closes #7857... this time it checks also for a Series (and it passes all tests).


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
fyi, you don't normally need to close and reopen the PR, just force push, see here: https://github.com/pydata/pandas/wiki/Using-Git

Thanks; OK for the force push in the future (just feared it was unpolite), but the page you linked doesn't mention it!

no unless you are working with someone else then almost always better to force push (and if you are then someone needs to periodically rebase), just much cleaner that way IMHO.
BUG: ewm*() interpretation of min_periods is off by one
State: closed

Closes https://github.com/pydata/pandas/issues/7884.

This addresses only the `ewm*()` functions' off-by-one interpretation of `min_periods`.
It doesn't address the inconsistency between the `ewm*()` functions and the `rolling_*()` functions in the meaning of `min_periods`.


Labels:
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
are there test for min_periods 1 and 0?
if not can u add pls 

thxs

OK, will add tests for min_periods 0 and 1.

In the process of adding tests for min_periods = 0, 1, I noticed the following, which is really messing up the testing. I think all these functions should return `NaN` for a single value.

I just noticed that `ewmvar`, `ewmstd`, `ewmvol`, `ewmcov`, `rolling_var`, `rolling_std`, returns `0.0` for a single value (assuming `min_periods=0`); whereas `Series.std`, `Series.var`, `ewmcorr`, `expanding_cov`, `expanding_corr`, `expanding_std`, `expanding_vol`, and `expanding_var`, `rolling_cov`, and `rolling_corr` all return `NaN` for a single value.

I've updated the code to add tests for min_periods = 0, 1, 2, leaving the single-value behavior noted above unchanged. If want to, can address that in a separate issue/PR: https://github.com/pydata/pandas/issues/7900.

The Travis CI build passed.

thanks!
BUG: fixing ewma() for adjust=False and ignore_na=False
State: closed

The previous code in https://github.com/pydata/pandas/pull/7603 wasn't quite right for the case that both `adjust=False` and `ignore_na=False`. This PR fixes the handling of that case. Note that `ignore_na=False` behavior is new in v0.15.0, so this bug and its fix don't affect any prior behavior.


Labels:
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
side issue can you add a couple of examples in the doc-string of `ewma`? Including one with adust
thxs

Happy to give it a shot, though to be honest I'm not quite sure what to put there. Can you point me to some other doc strings that have such examples? I can add more verbose notes about `adjust` (which to my mind isn't documented very clearly) and `ignore_na` to `_ewm_notes` (which is used to construct the doc strings for all the `ewm*()` functions), though I don't see numerical examples being illuminating.

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.xs.html?highlight=xs#pandas.DataFrame.xs

(just showing some examples, these are for users to copy-paste). if it doesn't add then its ok. but a bit more docs are always better.

I don't think examples will add much, as it's just setting `adjust=True` or `False` and producing different numbers, but they won't be intuitive either way. I will augment `_ewm_notes` to give a better explanation of what `adjust` and `ignore_na` do, and will update this PR.

I've updated `_ewm_notes` to give a much clearer description of `adjust` and `ignore_na`. Let me know what you think.

Actually, let me make a slight edit...

that looks good. does this need to also be on the other `ewma_*` doc-strings?

(you can simply put this in a variable and use Appender to propgate this)

I put it in `_ewm_notes`, which I think is incorporated into all the `ewm*` doc strings automatically. I'm making a slight edit...

OK, I'm done editing.
Yep, all the `ewm*` functions begin with `@Substitution(..., _ewm_notes)`, so these updated notes should be reflected in all the `ewm*` doc strings.

The Travis CI build passed.
API: add 'level' kwarg to 'Index.isin' method
State: closed

closes #7890

This PR adds `level` kwarg for `Index` objects as discussed (briefly in #7890):

``` python

Index.isin(iterable, level=None)
```
### Summary
- for non-`MultiIndex` classes:
  - no functional changes in the actual membership test
  - valid `level` values are `None`, `0`, `-1` and `self.name`
  - if `level` doesn't match the name, it's a `KeyError` (was `AssertionError` before)
- for `MultiIndex` classes:
  - if `level=None`, elements of `self.values` (tuples) are used
  - otherwise elements of corresponding level are used
  - valid `level` values are the same as in `MultiIndex.get_level_values` method, `-self.nlevels..(self.nlevels - 1)` plus all unique index names
### TODO
- [x] add to documentation
- [x] optimize MultiIndex case


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: MultiIndex

Comments:
Also, probably related to #3268

Performance patch is in, here's a synthetic benchmark:

``` python
In [7]: idx = pd.MultiIndex.from_product([np.arange(10000), ['a', 'b', 'c']])

In [8]: timeit idx.isin(['a'], level=1)
10000 loops, best of 3: 104 µs per loop

In [9]: timeit idx.get_level_values(1).isin(['a'])
1000 loops, best of 3: 1.37 ms per loop

In [10]: timeit idx.isin([1], level=0)
1000 loops, best of 3: 1.02 ms per loop

In [11]: timeit idx.get_level_values(0).isin([1])
100 loops, best of 3: 2.84 ms per loop


```

looks good....doc and then can merge

FYI, if you want to revise/update isin section gr8! (I would also make it a top-level, rather than below boolean indexing)

re: doc

other than that I've already added?

I think it makes sense to combine those 2 sections (isin of Index objects) and isin of PandasObject as they are really the same (and confusing to look in 2 sections). http://pandas-docs.github.io/pandas-docs-travis/indexing.html#indexing-with-isin

I would simply change the isin section (that I am pointing), make it a top-level (e.g. same as Boolean Indexing), them put the examples there

I'm a bit concerned about introducing functionality of classes that themselves have not yet been described, because `Index` & `MultiIndex` sections are way below the "isin" section on containers while they probably should be introduced with the rest in `dsintro.rst`.

Although, I see some of that happening for docs on `where` that do mention `Multiindex`, so you may be ok with that, but just to be sure, are you?

I agree with all of that

index/MultiIndex should have an intro at the top of indexing (side issue)

but I think splitting isin up (as it is now) is a bigger problem

so I would consolidate and out as a big bullet after boolean indexing 

feel free to do a short intro if u would like as well (for index/mi)

looks good...

@jorisvandenbossche anything else?

no, looking good!

thanks

let's check docs after it's built and review for correctness
PERF: groupby / frame apply optimization
State: closed

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_frame_apply_overhead                 |   9.1400 |  70.3743 |   0.1299 |
groupby_frame_apply                          |  43.5640 | 186.6427 |   0.2334 |
groupby_apply_dict_return                    |  39.6016 |  80.1926 |   0.4938 |
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
was introduced earlier in 0.15.0, now going back to the prior perf
ENH add level argument to set_names, set_levels and set_labels (GH7792)
State: closed

Closes #7792

New `level` argument:

```
set_names(self, names, level=None, inplace=False)
set_levels(self, levels, level=None, inplace=False, verify_integrity=True)
set_labels(self, labels, level=None, inplace=False, verify_integrity=True)
```

e.g.
    set_names('foo',level=1)
    set_names(['foo','bar'],level=[1,2])
    set_levels(['a','b','c'],level=1)
    set_levels([['a','b','c'],[1,2,3]],level=[1,2])


Labels:
name: API Design
name: MultiIndex

Comments:
Added tests for the reverse (len(levels) == 2 but level =1)

look good

thanks! this is a nice enhancement
BUG: Fixed incorrect string length calculation when writing strings in Stata
State: closed

Strings were incorrectly written using 244 character irrespective of the actual
length of the underlying due to changes in pandas where the underlying NumPy
datatype of strings is always np.object_, and never np.string_. Closes #7858


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
@bashtage I would generate some files in stata in both formats and add to the repo as well. Is this a bug fix or an API change?

@jreback I'm not sure a Stata-generated file would add much - the bug is in `StataWriter` which no longer correctly identifies string sizes, and so defaults to the maximum size 244 for all strings. 

I think I've decided this is a bug since the code does not appear to do what is intended (even though it is functional).

ok, that's fine then

Hold off a while on the merge - I need to check what happens for longer strings (> 244, which are a problem)

I c, it was being actually set before. never saw that.

@bashtage np. lmk

you truncate / warn / raise for too long strings

Just want to make sure it is still working correctly, and be certain there is a test for that case.

Definitely not working for longer strings, so more work needed.  The files generated actually crash Stata 12...

looks like you were truncating before (which is fine I think) (maybe with a warning thougH)

I have raised it as an error in the PR.  Not completely sure that it the correct behavior though.

stata has  max of 244, right? so then that is ok

I just checked 0.14.1 - attempting to write a 500 character string produced no warning or error, and outputs a corrupt stata file (that segfaults Stata)

ok gr8! so raising is good (put this fix in API changes, even though its really a bug -fix). in theory its more visible there.

@jreback I think this is ready, subject to doc/other checks

@jreback Hopefully ready now

gr8 ping when green

@bashtage awesome as usual!
BUG/API: allow get_indexer to work with nans
State: closed

- [ ] clear the FIXME in Categorical in #7820 (comment)

closes #7820


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
looks ok conceptually...but failing tests.

yeah a few kinks to work out 

@cpcloud can you circle back to this when you have a chacne?

sure

If this is fixed, the workaround in categorical.py should be removed:

```
        # FIXME: the following can be removed after https://github.com/pydata/pandas/issues/7820
        # is fixed.
        # float levels do currently return -1 for np.nan, even if np.nan is included in the index
        # "repair" this here
        if com.isnull(rvalue).any() and com.isnull(self.levels).any():
            nan_pos = np.where(com.isnull(self.levels))
            lindexer[lindexer == -1] = nan_pos
```

[This is in #8007, so not yet merged]

@cpcloud ?

@cpcloud ?

@cpcloud status?

@cpcloud can you update?

closing
ENH: tz_localize(None) allows to reset tz
State: closed

Closes #7812. Allow `tz_localize(None)` for tz-aware `Timestamp` and `DatetimeIndex` to reset tz.
CC @rockg, @nehalecky 

```
t = pd.Timestamp('2014-01-01 09:00', tz='Asia/Tokyo')

# tz_localize resets tz holding current tz representation
t.tz_localize(None)
#2014-01-01 09:00:00

# tz_convert reset tz using UTC representation (no changes by this PR)
t.tz_convert(None)
#2014-01-01 00:00:00

idx = pd.date_range(start='2011-01-01 09:00', freq='H', periods=10, tz='Asia/Tokyo')

idx.tz_localize(None)
# <class 'pandas.tseries.index.DatetimeIndex'
# [2011-01-01 09:00:00, ..., 2011-01-01 18:00:00]
# Length: 10, Freq: H, Timezone: None

#  (no changes by this PR)
idx.tz_convert(None)
# <class 'pandas.tseries.index.DatetimeIndex'>
# [2011-01-01 00:00:00, ..., 2011-01-01 09:00:00]
# Length: 10, Freq: H, Timezone: None
```


Labels:
name: Enhancement
name: API Design
name: Timezones
description: Timezone data dtype

Comments:
`idx.tz_localize(None)` is the functionality I've been looking for. Just making sure I understand `tz_convert(None)`, this would be equivalent to a chained `.tz_convert('UTC').tz_localize(None)`?

@nehalecky 

```
In [2]: t = pd.Timestamp('2014-01-01 09:00', tz='Asia/Tokyo')

In [3]: t
Out[3]: Timestamp('2014-01-01 09:00:00+0900', tz='Asia/Tokyo')

In [4]: t.tz_convert(None)
Out[4]: Timestamp('2014-01-01 00:00:00')

In [5]: t.tz_convert('UTC')
Out[5]: Timestamp('2014-01-01 00:00:00+0000', tz='UTC')

In [6]: t.tz_convert('UTC').tz_localize(None)
Out[6]: Timestamp('2014-01-01 00:00:00')

In [7]: t.tz_convert('UTC').tz_convert(None)
Out[7]: Timestamp('2014-01-01 00:00:00')

In [8]: t.tz_localize(None)
Out[8]: Timestamp('2014-01-01 09:00:00')
```

`tz_localize(None)` essentially just drops the tz (but leaving the time as is), while convert
converts to UTC (first)

@jreback thanks! Just like I understood. :)

This also closes #3653 
Nice to see a solution for this! (I initially asked it on stackoverflow in that issue)

@sinhrks @jreback This is ready?

@sinhrks pls rebase and ping when green

good 2 go

@jreback rebased, and now green

thanks @sinhrks
BUG: fix greedy date parsing in read_html
State: closed

Also, `infer_types` now has no effect _for real_ :)

closes #7762
closes #7032


Labels:
name: Bug
name: API Design
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Deprecate
description: Functionality to remove in pandas

Comments:
Weird that Travis failed again. The file is says it can't find sure seems to be there....

Yeah not totally  sure what happened. I'm gonna take a look tokorrow morning. I tried to reproduce but I had to move on to other things today. 

Ah, I think this is because I forgot to add it to the data files in `setup.py`. I'll just rename to `*.html`, that way it gets picked up

@jreback @danielballan any (more) comments?

i'll add a release note

cc @clarkfitzg

Looks ready to merge to me. I didn't know about `kwargs.setdefault`; that's nice.

looks ok

Merging
BUG/VIS: rot and fontsize are not applied to timeseries plots
State: closed

In some plots, `rot` and `fontsize` arguments are not applied properly.
- timeseries line / area plot: `rot` is not applied to minor ticklabels, and `fontsize` is completely ignored. (Fixed to apply to xticklabels)
- kde plot: `rot` and `fontsize` are completely ignored. (Fixed to apply to xticklabels)
- scatter and hexbin plots: `rot` and `fontsize` are completely ignored. (Under confirmation)

```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(10, 4), index=pd.date_range(start='2014-07-01', freq='M', periods=10))
df.plot(rot=80, fontsize=15)
```
### Current Result

![figure_ng](https://cloud.githubusercontent.com/assets/1696302/3709379/7841977e-1454-11e4-8396-fc42e943d54a.png)
### After Fix

![figure_ok](https://cloud.githubusercontent.com/assets/1696302/3709380/7cc1037a-1454-11e4-9456-a9335907473b.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
The problem affects to kde, scatter, and hexbin plots. These all ignores `rot` and `fontsize` now.

In kde, I think it is natural to use `rot` and `fontsize` for `xticklabels`. But in `scatter` and `hexbin`, maybe better to apply to both `xticklabels` and `yticklabels`? Or only to `xticklabels`?

@sinhrks thanks....
ENH: allow MultiIndex column selection from HDFStore
State: closed

Needs some more testing


Labels:
name: Enhancement
name: IO HDF5
description: read_hdf, HDFStore

Comments:
badness :)

@jreback Is this actually more efficient, i.e., does the selection happen _after_ reading in every column? If so, then there's really no point in this PR as I can just select after reading in. What I'm interested in doing is actually reading in only 3 or so columns from about 60; there's about 26M rows, so I don't want to read in 60 columns just to eventually select out 3 of them.

Just stepping through it looks like things _are_ read in first, but my test only has a single dtype, and my actual use case has 3 or so dtypes, (the 3 columns I want to select have 1 object and the others are either float or int dtype)

all columns are read (e.g. the entire row), then reindexed. So it won't be more efficient (need a column store for that). 

However, you _can_ do `select_column` for a small number of columns (then concat together). I think their is an issue out their to do that. (in combination with a `select_as_coordindates` which creates the row indexer that you pass as the `where`). So on a wide table this would be MUCH more efficient.

Blah okay :) I'll see if I can't work something up for that issue.

though this _does_ enhance the selection syntax a bit :)

@cpcloud can you update? forgot what this was for

closing, but feel free to revisit
BUG: Fixed failure in StataReader when reading variable labels in 117 
State: closed

Stata's implementation does not match the online dta file format description.
The solution used here is to directly compute the offset rather than reading
it from the dta file.  If Stata fixes their implementation, the original code
can be restored.
closes #7816


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
anyway to include a test file? (small one)

I haven't added a test yet, but will do.

@jreback Test added, so if no typos and all pass, should be ready

I think it is ready now.

ok, ping when green

Strange, a failure in a file I didn't change.

I think that was fixed
are you rebased against master?

Rebased

these look like small test files you created, yes? (which is good)

@bashtage thanks for the fix!