BrandonFors/pandas-github-issues huggingface dataset:
ENH: column label filtering via regexes to work for numeric names
State: closed

Simple fix to allow regex filtering to work for numeric column labels, e.g. df.filter(regex="[12][34]")

closes #10506 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
can you add some tests?

For search(x) -> search(str(x))?

Any advice on what to add or where? I don't see any existing tests for this function...

look in `pandas/tests/test_frame` for `test_filter`

Thanks Jeff! Added the test. Let me know what you think...

add a not in whatsnew/0.17.0. Put in Other Enhancements section

What would this do in 0.16.2 (if you passed the regex), not fitler anything? or raise?

Done! In 0.16.2 `re.search` will raise if a column name is numeric...

when you are all done, pls rebase/squash see contributing docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

I'm having trouble with squashing the commits. I don't have a ton of experience with git, so I'm not sure what to do next. Below is the message. Seems to have to do with a merge conflict in test_frame? Any advice?

```
error: could not apply ac90352... Add test for regex filter on numeric column names

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
```

contributing docs are here: http://pandas.pydata.org/pandas-docs/stable/contributing.html

you have a conflict and need to fix it

Hmm, when I rebase it detects conflicts, then I resolve them using `git mergetool`, and commit. Doesn't seem to change anything. When I run `git merge master` I get that everything is up-to-date. I'm probably missing something simple?

FYI, you don't normally need to add an issue if you just create a PR (like you did), but no biggie.

I rebase you: https://travis-ci.org/jreback/pandas/builds/69631109

FYI don't use merge master. This is not pandas standard practice. This makes rebasing much more difficult.

merged via https://github.com/pydata/pandas/commit/bfe5a7f0c1d7250be68f1d2bceca2138d561014d

thanks!
Allow passing other arguments to interpolation functions
State: closed

Closes #10378


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
can you add some tests?

Pls update docstring in `generic.py`.

Okay. I added tests and updated the docstring. In looking more closely I'm actually not sure what other interpolating methods should be passed the kwargs. Potentially `piecewise_polynomial_interpolate` but I don't really understand it (#10365). I think `interp1d` is getting all of the relevant variables.

Anything else to get this merged? And wow, travis-ci is SLOW at running these tests.

see here: http://pandas.pydata.org/pandas-docs/stable/contributing.html

pls squash.

and travis seems quite chipper lately. We have a quite complicated setup, so takes a bit.

Ok, I'll update the docs and then squash. I usually wait until code is :+1: before squashing to make reviewing easier.

Squashed and docs.

Anything else remaining to get this merged?

thanks @cancan101
Check for size=0 before setting item
State: closed

This is a second try at fixing #10193; the first try is in #10194. There is some useful discussion in that PR, so I didn't want to clobber that- not sure of the etiquette of multiple PRs for the same bug??

The discussion around setting values in views on #10194 is separate from the bug itself- I find that setting an item on a newly-constructed empty series with a frequency (thus, no views are involved) raises the same error. 

I couldn't cherry pick from my old branch easily as some commits contained more code than is necessary for this set of changes, so I have done a new one.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
can you add a release note

@jreback Release note added

this is fine. ping on green.

@jreback It is green

@rekcahpassyla thanks for the fix!

Most welcome!
ENH: Enable ExcelWriter to construct in-memory sheets
State: closed

Add support for StringIO/BytesIO to ExcelWriter
Add vbench support for writing excel files
Add support for serializing lists/dicts to strings
Fix bug when reading blank excel sheets

closes #8188
closes #7074
closes #6403
closes #7171


Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: IO Excel
description: read_excel, to_excel

Comments:
@jreback The vbench for openpyxl is very slow ~ around 300s on my computer. This is around 10x slower than the next slowest test (sql).  Should I reduce the df size for this test? 

yes, a vbench should be < 1s if at all possible, so reduce the size on these.

I think it is finished once green - I snuck in one more Excel-related issue (testing xlwt on Py34).

Openpyxl is very slow compared to xlsxwriter (5x slower)

ahh, the `xlwt` just got ported to py3. finally! excellent.

yes, `openpyxl` does have this perf issue (and why `xlsxwriter` is the default). Though the author is working on making it much better.

ping when green.

@jreback ready

minor doc comment. ping after you have pushed for merging.

@jreback Added those.

Rebased

@bashtage thanks for this!
ERR: GH9513 NaT methods now raise ValueError
State: closed

This is to close #9513. `NaT` methods now raise `ValueError` or return `np.nan`.


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Just a question: isn't it also a possibility to let it return `np.nan` ? Because now you get different results depending on if you access the date on a full index/series, or on a single element of that index/series

That makes sense. So leave `weekday` unchanged, and have `date` and `time` return `nan`? Unsure about `toordinal` though. 

@jreback tests are green. 

I think I would still prefer to raise for date and time. It's one thing to return nan for methods/attributes that return numbers, but nobody expects that for a method that is supposed to return these specific types. 

@shoyer and what with the attributes that don't return a number but a specific date/time object?

hmm, we _could_ return `NaT` itself for `date/time` e.g.

```
def date(self):
    return self
```

```
In [8]: Timestamp('20130101').date()
Out[8]: datetime.date(2013, 1, 1)

In [9]: Timestamp('NaT').date()
Out[9]: NaT
```

The idea being is that you asked for a `date/time` you get a python object (as pandas does'nt have a specific date/time object specifically), but you have a missing value, so you get an object that represents a missing value (just typed to the missing value)

That makes perfect sense for `date`. But `time` is a different type, so I think users can still be surprised by `NaT`.

Technically, datetime is a subclass of date, not the other way around. It think it's better just to raise an error as opposed to guessing an appropriate sentinel value.

Most likely updating tonight.  Any verdict on raising vs returning `nan` or `NaT`?

right

I think things that have date should be NaT eg
date,now,today

I am ok with time be Nat as well -even though it pushes the semantic boundaries 

number like should be nan (eg weekday)
ordinal should prob raise 

> I think things that have date should be NaT eg
> date,now,today

Agreed. NaT is a datetime instance and also a date instance.

> I am ok with time be Nat as well -even though it pushes the semantic boundaries

I disagree. NaT is not a time instance. We should not return it from a method which promises to return a time.

> number like should be nan (eg weekday)
> ordinal should prob raise

Agreed.

ok, so the suggestion is to raise for `time`. Ok not adverse to that.

updated

updated

ok lgtm

@jorisvandenbossche @shoyer comments?

Lgtm 

Updated again. Moved the list concatenation outside the loop.

lgtm, @shoyer if ok to you, pls merge.

Merged via c87fa18cc22c15d37c023581fad9a46fff31534d
BUG: drop_duplicates drops name(s).
State: closed

Closes #10115. Closes #10116.

Based on @seth-p  and @jreback 's fix, added tets for datetime-likes.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
@jreback could you review this, as this will be the base of #10236 and #10150 (adding `names` to all test cases)

lgtm. ideally let's try to group tests that work across ALL sub-classes in `test_index`, and leave timeseries specific ones in `tseries/test_base`

merge when ready

Thank for review:)
CI: run doc-tests always
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
@jreback This does not run the doctests, only those doctests in the test modules (and I don't think we have such?) See http://nose.readthedocs.org/en/latest/plugins/doctests.html#options

I am in favor of running doctests (with the only purpose of guaranteeing the docs are correct/up to date), but I don't think they will pass at the moment (they are currently not written to pass)

hmm u might be right
thought this was actually running them 
ok then
BUG: frequencies.get_freq_code raises an error against offset with n != 1
State: closed

Fixed a `frequencies.get_freq_code` raises `ValueError` against offset instance with n!=1. This is needed for #7832.

```
# OK
>>> import pandas.tseries.frequencies as f
>>> f.get_freq_code(('D', 3))
(6000, 3)
>>> import pandas.tseries.offsets as offsets
>>> f.get_freq_code(offsets.Day())
(6000, 1)

# NG!
>>> f.get_freq_code(offsets.Day(3))
ValueError: Unknown freqstr: 3D
```

Also, added docstrings and tests which make frequencies behavior understandable.


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
rebase against master 
I just merged #10352 wasn't running doc tests before

merge on green

Thanks, rebased and now green. Merging.
BUG: df.to_html(index=False) renders index.name
State: closed

Closes #10344.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@jreback can u take a look?

lgtm.

Thanks for review.
BUG: Check complib values
State: closed

Add check for complib when opening a HDFStore

closes #4582
closes #8874


Labels:
name: Bug
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Using multiple compressors, e.g. `blosc:zlib`, resulted in corruption for me - also not sure why. So it seems safer to just prevent them being used and to enforce what the docs say are supported compressors.

ok that's fine, move to a class variable though `_compressors`? to make maintenance easier in the future

ping when green. I am going to cut 0.16.2 soon, so this will be last one :)

merged via https://github.com/pydata/pandas/commit/3cdd1df36bdd7a49d629aaae70b65616ea702771

thanks!
BUG #10228: segfault due to out-of-bounds in binning
State: closed

Closes #10228.   I also deleted some duplicated code while I was at it.

So, I wasn't sure if I should be including a unit test for this.  This issue was a segfault, which was happening when you do:

```
s = pd.Series([], index=pd.DatetimeIndex([]), dtype=np.object)
s.resample('d', how='count')
```

so I suppose I could add this code into a test, and just verify it doesn't segfault, but that seemed like a bad idea.  It would be testing for something that's undefined behavior and therefore be a non-deterministic test.


Labels:
name: Bug
name: Resample
description: resample method

Comments:
of course you need a test, otherwise how can you tell if your fix works? The test can be as simple as asserting that it runs.

@jreback, sure I'll write a test.  I have a piece of code that reliably segfaults for me:

```
ss = pd.Series([], index=pd.DatetimeIndex([]), dtype=np.object)
ss.resample('d', how='count')
```

That being said, this doesn't mean it'll reliably segfault for other machines since out-of-bounds access are undefined behavior.  One way would be to re-enable the bounds-checking on these Cython functions just for the unit tests.  (I actually posed [this question](http://programmers.stackexchange.com/a/286726/134794) on Programmers Stack Exchange and they seemed surprised that the disabling of bounds-checking was hard-coded rather than something that can be switched on and off)

@Garrett-R then that is a reasonable test. Even if it doesn't segfault on some platforms that is ok. Note that you _can_ change the directive at run-time, see [here](http://docs.cython.org/src/reference/compilation.html#how-to-set-directives), e.g. you would have a flag that is passed in to re-enable this. But to be honest it is overkill, and not necessary to tests with that. Since you have an example that fails, that can serve as a smoke tests (e.g. the test is simply running that code, if it doesn't segfault then the test passes). 

@Garrett-R No objection to a separate PR for adding a warning to `generated.pyx` (and more importantly prob a note in `internals.rst` about how/what to change. thanks!

@jreback, good call about adding more comprehensive testing.  This helped me find some more out-of-bounds bugs (which were in `algos.pyx`).

I'll send a PR soon to supply a "DO NOT EDIT THIS GENERATED FILE...." warning for other fellow noobs.

merged via https://github.com/pydata/pandas/commit/25fc49d975d74c5cb969089dcb3efeda15c3262f

thanks!

nice fix.

btw if you'd like to update the docs w.r.t. the `generate_code.pyx` would be gr8. pls put in `internals.rst`
DOC: Remove auto_close option form read_hdf
State: closed

Remove docstring indicating auto_close can be used in read_hdf.
This value is always ignored.
Also removes unreachable code.

xref #9327


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO HDF5
description: read_hdf, HDFStore
name: Clean

Comments:
hmm, this should be hit in the iterator, or if you pass an open store to `read_hdf`. Maybe not tested.

It doesn't have a `return` so I don't think it can't do anything.  Maybe I'm missing the point.  Tests pass with it removed.  

There is also another problem if a buffer is passed - there is no code path for a buffer since the `if` only checks for `basestring` and there is no `else`

I could probably improve it to handle an open store (although why one would want to use `read_hdf` if they have a store open is not obvious).  The path to handle a buffer should be fairly straightforward (for example, if reading a HDF file from the internet).  Presumably no one has ever tried to do this using `read_hdf`

I _think_ it should handle an open store (and not close it), that is the key. But of its not tested. So for now, let's put a test that hits that path (and you can raise `NotImplemented` or make it work), though the object has to be an actual `HDFStore` (or then its a `ValueError`).

Since this hasn't been hit in real code, either it works (maybe) or raises some weird error. Either way its not tested.

It now checks and also raises if a generic buffer type is passed.  

Added test for each.

lgtm. minor comment ping when green.

merged via https://github.com/pydata/pandas/commit/51047d4992b650bd60c47119ca0f373c61e82fce
BUG/ENH: GH10319 added higher_precision argument to rolling_mean/sum
State: closed

This is not specifically for #10319, but should close it for most cases, should users pass in `higher_precision=True`. #10319 is not really a bug in the Cython code, just precision loss when you do finite-precision arithmetic naively. 

```
>>> sum([0.00012456, 0.0003, -0.00012456, -0.0003])
-5.421010862427522e-20
```

This is `2**-64` and quite negligible, although it can be made much bigger with carefully chosen numbers.  I think rolling_\* functions that require summation could use some more precision, since doing everything in one pass accumulates rounding error, even if the rolling window is narrow.
The only reason why [0.00012456, 0.0003, 0.0, 0.0] currently returns the 'correct' result while [0.00012456, 0.0003, -0.0, -0.0] doesn't is because of explicit negative entry counting. This is only done in rolling_mean but not rolling_sum, and the lack of symmetry doesn't seem desirable. Perhaps we could remove the counting, or add it to rolling_sum as well? 


Labels:

Comments:
This is going to kill performance. I also think an option to do this is really a bad API. I would only be not -1 if this could be done by another non-slow algo (e.g. like we did using Welford method for variance) .

Point taken. The algo itself doesn't seem that slow to me, though my implementation certainly is. I will do everything C-style and remove the option (no branching) and measure the performance. 

I agree with @jreback, I don't think this is right API. Double doubles are fine (in theory), but the right way to use them should be to cast your column to a double double explicitly, e.g., via astype. But I would also want to see this `float128` type in numpy first rather than trying to implement it in pandas.

actually float128 DO exist in numpy (though not supported on some systems) - so better is to add them to the code generators (but these would have to be protected by try/excepts as the signature doesn't exist) - maybe a better soon for that is to use ifdef though 

@shoyer My concern is that the current rolling_sum/mean algorithm, although fast, introduces a lot more numerical instability than the most naive algorithm without the users being aware. The last sum, even if the window size is 1, takes 2*n computations.

@kawochen separately you could short-circuit rolling functions if the `window==1` (though only for `sum/mean`, `var/std` etc are all `nan` at that point.
Bug in to_json causing segfault with a CategoricalIndex (GH #10317)
State: closed

Fixed GH #10317
Stole tests from #10321


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
oh, see you did already!

creating the tests was the hardest part actually :)

I agree! Glad I didn't have to.

thanks @evanpw !

FYI, https://github.com/pydata/pandas/commit/da3e0d1c9778ac39d1daec779d6defa322ed37b4

windows didn't like the redeclaration; gcc was ok with it...weird
BUG: Bug in to_json with certain orients and a CategoricalIndex would segfault, closes #10317
State: closed

xref #10317 

```
In [17]: df = DataFrame({ 'A' : pd.Series([3,2,2],index=pd.Categorical([1,2,3],categories=[1,2,3])), 'B' : pd.Categorical(list('aab')) })

In [18]: df
Out[18]: 
   A  B
1  3  a
2  2  a
3  2  b

In [19]: df.index
Out[19]: CategoricalIndex([1, 2, 3], categories=[1, 2, 3], ordered=False, dtype='category')

In [20]: df.dtypes
Out[20]: 
A       int64
B    category
dtype: object

In [31]: def f(orient):
    print "orient->%s" % orient
    print df.to_json(orient=orient)
   ....:     

In [32]: f('columns')
orient->columns
{"A":{"1":3,"2":2,"3":2},"B":{"1":"a","2":"a","3":"b"}}

In [33]: f('index')
orient->index
{"1":{"A":3,"B":"a"},"2":{"A":2,"B":"a"},"3":{"A":2,"B":"b"}}

In [34]: f('split')
orient->split
{"columns":["A","B"],"index":[1,2,3],"data":[[3,"a"],[2,"a"],[2,"b"]]}

In [35]: f('records')
orient->records
[{"A":3,"B":"a"},{"A":2,"B":"a"},{"A":2,"B":"b"}]

In [36]: f('values')
orient->values
[[3,"a"],[2,"a"],[2,"b"]]
```


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
closing in favor of #10322
BUG: bug in setitem where type promotion is applied to entire block
State: closed

closes https://github.com/pydata/pandas/issues/10280

on master:

``` python
>>> df
   foo  bar  baz
a    0    1    2
b    3    4    5
>>> df.dtypes
foo    int64
bar    int64
baz    int64
dtype: object
>>> df.loc['a', 'bar'] = 3.14
>>> df.dtypes
foo    float64
bar    float64
baz    float64
dtype: object
```

on branch:

``` python
>>> df.dtypes
foo    int64
bar    int64
baz    int64
dtype: object
>>> df.loc['a', 'bar'] = 3.14
>>> df.dtypes
foo      int64
bar    float64
baz      int64
dtype: object
>>> df
   foo   bar  baz
a    0  3.14    2
b    3  4.00    5
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can you update

i added more tests

ok thanks. ping when green

a `HTTPError: HTTP Error 404: Not Found` error; otherwise all green

thanks!
BUG: bug in json serialization when frame has mixed types
State: closed

closes https://github.com/pydata/pandas/issues/10289

on master:

``` python
>>> df
   1st  2nd  3rd  4th   5th
a   10    1  foo  0.1  0.01
b   20    2  bar  0.2  0.02
c   30    3  baz  0.3  0.03
d   40    4  qux  0.4  0.04

>>> read_json(df.to_json(orient='index'), orient='index', convert_axes=False)
   1st  2nd  3rd  4th   5th
a   40    4  qux  0.4  0.04
```

on branch:

``` python
>>> read_json(df.to_json(orient='index'), orient='index', convert_axes=False)
   1st  2nd  3rd  4th   5th
a   10    1  foo  0.1  0.01
b   20    2  bar  0.2  0.02
c   30    3  baz  0.3  0.03
d   40    4  qux  0.4  0.04
```


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks you sir!
BUG: PeriodIndex.order doesnt preserve freq
State: closed

Closes #10295 (This is based on #10292). Changed `lib.maybe_indices_to_slice` to detect slice with `step` and also check upper bound of index.

However, above change affects to the behavior of `Index.slice_locs`. Internally, the behavoir of `Index.get_slice_bound` seems to assume `lib.maybe_indices_to_slice` returns slice only when its step is "+1". Options are:
1. Change `Index.get_slice_bound` to meet updated `lib.maybe_indices_to_slice` logic.
2. Leave `lib.maybe_indices_to_slice` as it is. Let `DatetimeIndex` to use separate logic, maybe `lib.maybe_indices_to_slice_with_step`

To avoid any unexpected result, I think option 2 is preferable? 


Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type
name: Frequency
description: DateOffsets
name: Period
description: Period data type

Comments:
BTW, `__array_finalize__` is never called because it is no longer a subclass of ndarray.

This should be a separate PR, but can I change this to `__finalize__` and let the logic to handle metadata (properties listed in `_attributes`) ?

The fix should work, but changed the title until we can decide to change `lib.maybe_indices_to_slice` or prepare separate func.

This is ready for review. I think the point is whether to change `maybe_indices_to_slice` (which is used in base `Index` also), or prepare separate function for datetime-likes.

`__array_finalize__` is being handled in #10638. 

I have not carefully considered this, but my inclination is that option (1) is preferable (changing `maybe_indices_to_slice`). Which did you choose?

@shoyer Currently, option 1 (fix `maybe_indices_to_slice`).

@sinhrks this looks reasonable. you added a huge number of tests (which is gr8!). so will take a bit to look.

minor comment 
lgtm to me otherwise
merge when ready

Thanks, fixed the release note and rebased. Will merge when green.