BrandonFors/pandas-github-issues huggingface dataset:
ENH: Standardized timeseries accessor names, #9606
State: closed

In accordance with https://github.com/pydata/pandas/issues/9606, I added `day_of_week` and `day_of_year` accessors to DatetimeIndex, Timestamp, and Period. Since https://github.com/pydata/pandas/issues/9606 didn't mention `weekofyear`, and there's already `week` as a replacement for that, I didn't add a `week_of_year`, but I'm happy to.

I updated the tests and docs, but this is my first PR to pandas involving actual code, so let me know if I missed anything.


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
For `dayofweek`/`day_of_week` we also already have `weekday`, so this is adding a third alias for that one .. 
So I am a bit torn on this one (although I like the fact that we try to standardize the namings)

@jorisvandenbossche I think we should deprecate the original ones in 0.18 anyhow (eg daysinmonth)

Maybe we could also remove these from the dir to avoid all the duplicates in tab completion?

there is a test in test_series for dt accessor that needs updating as well

yep @jorisvandenbossche agreed

I think choosing one prefered name per attribute, and remove all others from the docs (only mentioning them somewhere as being aliases) and tab completion, is maybe a less 'aggresive' way for striving for consistency instead of really deprecating them

sure that's fine too 

@emilydolson you will need to define

def _dir_deletions()

which returns a set of the attributes to remove

I've got the tab completion fixed for DatetimeIndex and PeriodIndex, but defining `_dir_deletions()` in Timestamp doesn't seem to affect the return value of `__dir__()`. Is there something about Cython that might interfere with _dir_deletions? 

so the issue here is that `Timestamp` needs to inherit from `PandasObject`, which is in `pandas.core.base`. But you can't simply include that as it would have a circular dependency.

So here is what we need to do.

Move from `pandas.core.base`, `StringMixin` and `PandasObject` to a new module, call it `pandas.core.object`. import this to `pandas.core.base` (you _may_ want to change imports of `PandasObject` elsewhere in the codebase). Then you can import this as a mix-in for `Timestamp` (in tslib.pyx).

long-story-short this will then work.

can you rebase / update

also let's deprecate the old names (e.g. w/o the underscores).

You will need to put the warning in the accessor (and probably create a specific function/property for them in order to do this as they are defined by the `_field_accessor` ATM)

can you update

@emilydolson can you rebase/update according to comments

@emilydolson can you rebase/update according to comments

closing, but pls reopen if you'd like to update, which would be great.
Fix #10770 by adding days_in_month to docs
State: closed

Fix https://github.com/pydata/pandas/issues/10770 by adding days_in_month to Time/Date Components. Am I right in thinking `days_in_month` (rather than `daysinmonth`) is the right name to use going forward? Is there anywhere else in the documentation that it should be added that I missed?


Labels:
name: Docs

Comments:
I think this is good! 
In the API reference it is already added: http://pandas.pydata.org/pandas-docs/stable/api.html#datetimelike-properties

thanks!

interested in working on #9606 ?

Yeah, sure!
BUG: Holiday observance rules could not be applied
State: closed

Closes #11477
Closes #11533 

There were some other bugs here that I added tests for.  For example, `holiday.dates` for MLK day was returning all holidays from the holiday start date up to the end date rather than just between the range.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
@jreback Changes incorporated.  Holiday.dates() does return a DatetimeIndex rather than a list, but I'm thinking that's okay.

@rockg looks good.

pls rebase / minor edits. ping when green.

@jreback Green

thanks!
CLN: move routines for filling missing data from core/common.py to core/missing.py
State: closed

This PR moves the routines used for filling missing data into their own module as suggested in PR #11445.

Thoughts on re-naming `interpolate_1d()` and `interpolate_2d()`? They way they are currently named makes them sound like they're the same functions for arrays of differing dimensions. However, `interpolate_1d()` is used for interpolating with scipy interp methods whereas `interpolate_2d()` is used for forward- or back-filling.


Labels:
name: Clean

Comments:
ok looks good
ping when green

For internals.py and generic.py I switched to the `import ... as` technique. Both index.py and and categorical.py only import one method, so I left those as `from ... import`. Will ping when green.

@stahlous thanks! this was great!

let's revisit the naming scheme for these in your other PR.
implemented fix for groupby date bug, #11324
State: closed

This is a fix for issue #11324 in which grouping when datetime fields are involved raises an exceptoin


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Groupby

Comments:
pls add a whatsnew comment (bug fix section), and squash.

Added comments to test, made an entry in v0.17.1.txt, and squashed commits.

merged via https://github.com/pydata/pandas/commit/5df693fafa7eedf732d414cc9eb8310b5391dfae

thanks!
ENH: tilde expansion for write functions, #11438
State: closed

closes #11438 
Added calls to `_expand_user()` in multiple Formatter classes so that writer functions, such as `to_csv()` and `to_html()`, can perform tilde expansion on file paths beginning `~/`. This makes the writer functions consistent with the reader functions, such as `read_csv()`. I also changed the `__init__()` for `CSVFormatter` to conform to the pep8 width limit.


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
tests pls

actually on 2nd thought.

I think you these should ONLY be used in `core/frame.py/DataFrame.to_csv and to_html`

I don't really like this here, as all of the other IO functions are localized in `pandas/io/*` but that's how it is for now.

I had the same thought, but then I assumed that would cause problems for `Series.to_csv()`. But, having now checked the code, I see that they are the same function, with each Series being temporarily converted to a DataFrame.

Should I put the `_expand_user()` calls in frame.py?

Regarding the tests: let me start out by saying that I am oh-so-close to being out of my depth here. The current suite seems to run OK (various `DeprecationWarnings` and `RuntimeWarnings` notwithstanding), but I'm a bit unclear as to how (or whether) I should create a new test function. That function would need to write and read some temporary files, but it looks like it would be difficult, if not impossible, to use `testing.ensure_clean()` to generate expanded and non-expanded file paths.

you might have a look at these: https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_common.py

yeh this might be quite tricky to test 

can you add a release note and we'll stick in 0.17.1

@jreback looks like I might have caused some trouble editing the most recent version of the release notes file...

merged via https://github.com/pydata/pandas/commit/ea758cd2467f3b9dbfe4aced1ce6666145c502c6

thanks!
CLN/PERF: remove used functions; use C skip list for rolling median
State: closed

removes some unused code
reverts this commit https://github.com/pydata/pandas/commit/a40226e5c4c3d29e5e2383433c5f0b94c3c0ecc1 

performance consideration

```
import pandas
import numpy
arr = numpy.random.rand(1000000)
%timeit pandas.rolling_median(arr, 1000)
```

master

```
1 loops, best of 3: 4.94 s per loop
```

branch

```
1 loops, best of 3: 821 ms per loop
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
you _may_ be able to insert `nogil` (at the top-level), and make almost all of `skiplist.pyx` `nogil` (obviously where you raise you have re-acquire).

```
In [2]: import pandas
import numpy
arr = numpy.random.rand(1000000)

In [4]: from pandas.util import testing

In [5]: @testing.test_parallel(2)
def g():
    pandas.rolling_median(arr, 1000)
   ...:     

In [6]: @testing.test_parallel(1)
def f():
    pandas.rolling_median(arr, 1000)


In [7]: %timeit f()
1 loops, best of 3: 752 ms per loop


In [8]: %timeit g()
1 loops, best of 3: 972 ms per loop
```

can you add an asv benchmark? not sure we have much for rolling in general......

OK.  might as well release the GIL on roll_*

yep - same or another

Added asv benchmarks for the gil release.  But I can't get any of those to show in asv bench at all.  Have I written the tests incorrectly?
I do `asv continuous upstream/master HEAD -b gil` while on my branch.

A simple `timeit` does show the improvement, e.g.
for `rolling_mean`

```
import pandas
import numpy
from pandas.util.testing import test_parallel
arr = numpy.random.rand(1000000)
@test_parallel(num_threads=2)
def f():
     pandas.rolling_mean(arr, 100)
%timeit f()
```

branch

```
100 loops, best of 3: 12.5 ms per loop
```

master 

```
10 loops, best of 3: 19.5 ms per loop
```

for `rolling_kurt`
branch 

```
10 loops, best of 3: 65.5 ms per loop
```

master

```
10 loops, best of 3: 98.8 ms per loop
```

the `time_rolling_median` in `stat_ops.py` shows the same 5x improvement I get using `timeit`.

thanks!
DOC: Improvements in panel apply docstring
State: closed

Not sure what the bar is for a PR here - this is a mild but meaningful improvement - in particular re the option to supply numbers & a tuple - by no means perfect though


Labels:
name: Docs

Comments:
@jreback green

@jorisvandenbossche 
Thanks - I edited. I think we also can't just do `panel.apply`, but instead need the full expression `pd.Panel(pd.np.random.rand(4,3,2)).apply(pd.np.sqrt)`. Tell me if I'm wrong though

@maximilianr In theory, a full expression is not needed as this code is at the moment never run in order to build the docs (as is the case with the tutorials), but it is always nice to have full examples so people can copy paste it.

For layout, I think something like this is best (once define the variable, and then use it instead repeating the creation):

```
Returns a Panel with the square root of each element

>>> p = pd.Panel(np.random.rand(4,3,2))
>>> p.apply(np.sqrt)

Equivalent to p.sum(1), returning a DataFrame

>>> p.apply(lambda x: x.sum(), axis=1)

Equivalent to previous:

>>> p.apply(lambda x: x.sum(), axis='minor')

....
```

@jorisvandenbossche updated

@jorisvandenbossche I accidentally PEP8-ed the whole file. Shall I undo, or is that actually helpful?

@maximilianr yeh, normally want to do PEP stuff in separate PR's as confusing what is actually changed, but ok.
BUG: fix Panel.fillna() ignoring axis parameter (re-submission)
State: closed

closes #3570 
closes #8251

This is a re-submission of PR #8395 and addresses issue #8251. My apologies for the duplicate PR, but despite rebasing to pydata/pandas master, the original PR would not update with my new commits and I couldn't get rid of the `LooseVersion()` errors in the Travis build.

This PR may need some fleshing out still, but I wanted to submit this to see what the thoughts are on this approach. For this PR I've created a new interpolation mechanism at the block level to implement filling across 3 or more dimensions. This avoids the fiddly problems of trying to implement filling of 3+ dimensions at the frame level. However, it isn't possible with this technique to fill across blocks of different dtypes, although that seems like that should be a rare occurrence. 

Incidentally, this will also address #3570, which is one of the issues referenced in #9862.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
I merged the previous `interpolate_2d()` and `interpolate_nd()` functions into a single function and then renamed it simply `pad()`. I renamed `interpolated_1d()` to simply `interpolate()`. 

Looking through the code, I see a number of places where the terms "pad", "fill", and "interpolate" are used semi-interchangably and with different meanings. I think it would clear things up if, at least internally, these terms had these fixed definitions:

**fill**: fill in missing values with a static value
**pad**: fill in missing values with the most recent non-missing value
**interpolate**: interpolate in the `scipy.interpolate` sense

I'm not suggesting any changes to the public API, but I would be willing to work on a new PR to try and makes things internally consistent. I don't want to confuse this PR by trying to roll it in here.

In any event, this fix seems to work OK. Not sure if it needs more documentation other than mention of bugs fixed in whatsnew.

updated

Implemented most of the suggestions - just need clarification on what is meant by about the ndims being the same.

@stahlous do you want to rebase / update. fyi, you might want to squash as well.

I squashed the commits. Sorry for the delay - had other things going on the past couple weeks. Can you take a look and let me know if I've addressed everything?

closing, we are going to deprecate `Panel` in 0.18.0
DOC:Remove supression from ipython directive to make object visible
State: closed

Fix for https://github.com/pydata/pandas/issues/11223
@TomAugspurger should these two Ipython blocks be merged into one?

(this is what it looks like at the mo)
![pandas-11223](https://cloud.githubusercontent.com/assets/4982858/10770424/2013049a-7ce3-11e5-9f64-f8a66c98b961.png)


Labels:
name: Docs

Comments:
That'd be fine, or maybe put a note before each saying something like "First some setup" and "with the setup out of the way...". Either way, it'll be an improvement :+1: 

Added some 'verbal sugar' on either side of the Ipython directive blocks. Hope that's ok.

Looks good!

I just put some comments, but that is actually more about the current content than your change :-)

@Winterflower Do you have some time to update this?

yep, @jorisvandenbossche will do tonight!

@Winterflower ping when pushed & green

@jreback pushed and green!

thanks!
BUG: fix UnicodeEncodeError with to_sql and unicode column names (GH11431)
State: closed

Closes #11431

Still have to add tests


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jorisvandenbossche if you want to add some tests can do for 0.17.1

I was just looking at it! :-)

lgtm, just needs a whatsnew note. merge when ready.
BUG: vectorized DateOffset match non-vectorized
State: closed

closes #11370 

Also added a bit of documentation on the semantics for anchored offsets

currently blocked by #11406 - either need to change the vectorized or non-vectorized implementation of `QuarterBegin`  As discussed in that issue, my opinion is the vectorized version is the "right" (or at least consistent) one.


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
@chris-b1 can you rebase / update

@jreback - can you take another look at #11406?  Based on that I still need to change `QuarterBegin` for either the vectorized or non-vectorized case.

#11777 merged! pls update

@jreback - updated

lgtm! thanks!
BUG: implement .sort_index(...inplace=True) for #11402
State: closed

closes #11402

Adds inplace sort_index() for Series.


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls also add a whatsnew note for 0.17.1 (put in API changes even though this is a bug-fix, as it technically is an api change)

Updated and running green.

merged via https://github.com/pydata/pandas/commit/1135ce34fc2fb995e239a7d36aee1b687ef69dd9

thanks!
ENH: namedtuple's fields as columns
State: closed

Resolves https://github.com/pydata/pandas/issues/11181

Is this testing OK? Or do we need to test with differing lengths of tuples etc?


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback green

couple comments....ping when green

ok, ping on green.!

@jreback green

@maximilianr thanks!
CLN: period boundscheck warnings
State: closed

follow up to fix warnings in https://github.com/pydata/pandas/pull/11263#issuecomment-149641296


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
gr8. ping on green.

rebase/repush as conda had some issues

@jreback - this is passing now

thanks!

@chris-b1 if you'd do one more ....thanks!

```
cythoning pandas/tslib.pyx to pandas/tslib.c
warning: pandas/tslib.pyx:4257:25: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4258:26: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4260:56: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4261:22: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4275:25: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4276:26: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4278:56: Use boundscheck(False) for faster access
warning: pandas/tslib.pyx:4279:22: Use boundscheck(False) for faster access
```
BUG: using .ix with a multi-index indexer
State: closed

closes #11372 

Add an optional argument to _NDFrameIndexer to indicate
if the indexer is from a MultiIndex


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
Updated with comments from review. I have consolidated the tests `test_ix_setitem_multiindex` and `test_loc_setitem_multiindex` into `test_setitem_multiindex`, running the union of the cases for both `ix` and `loc`.

pls add a whatsnew note (bug fixes), looks good otherwise.

Thanks - rebased on master, added whatsnew and pushed. 

Renamed to `multiindex_indexer` and added docstring for `_align_series`

@rekcahpassyla thanks! 

that was a nice deep dive. lots of other indexing issues if you'd like :smile: 

Welcome! In the short term, a bit pressed for time but we are testing our entire library against `0.17.0` and if anything else comes out of that, I'll be happy to have a crack at those too.
ENC: better pandas typesetting in ipython nbconvert --to latex (updated)
State: closed

New pull request with tests and comments. See https://github.com/pydata/pandas/pull/9821 for original PR.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO LaTeX
description: to_latex

Comments:
closing, pls reopen when updated

@lmarti you are being way too terse here and making me work too much.

when you are finally ready for review, pls ping.
BLD: add CFLAGS `-fgnu89-inline` on FreeBSD 10+
State: closed

closes #10510 


Labels:
name: Build
description: Library building on various platforms

Comments:
can you add a whatsnew note as well.

pls confirm that this works for you as well. (e.g. download in a clean env and build). pls try on 2.7/3.5

Hi @jreback !
The whatsnew note added.

I ran the testing for 2.7 and 3.5 .... and i got some failures
- python3.5: 1 error; 5 failures
- python 2.7: no error; 5 failures

More detail is on [gist](https://gist.github.com/iblis17/8b74bab02ee26d48c243)

I also tried this:  In py35 venv, switch to master branch, set CC to `gnu48` , build pandas, and run the testing.
Unfortunately, it failed again. The failure report is almost same as previous one.

these are the same failures that another user has been getting see #11388 

can you give a squash. ping when green. (and confirm that this works on FreeBSD!)

@jreback The commits squashed. Most of features works fine on FreeBSD. :smile: 

There is few test cases falied on FreeBSD, but i think it's another issues.

@iblis17 thanks.

When we release 0.17.1 (prob 2 weeks). pls comment if any remaining issues.
BUG: GH11344 in pandas.json when file to read is big
State: closed

closes #11344
taken from esnme/ultrajson#145

test code 

```
import json
import pandas
from pandas.compat import zip, range
SIZE = 5*10**7
FILENAME = 'generated.json'
with open(FILENAME, 'w') as fileh:
    json.dump(dict(zip(range(SIZE), range(SIZE))), fileh)
with open(FILENAME) as fileh:
    pandas.json.load(fileh)
```


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
note that Travis is borked ATM 

@kawochen repush, Travis is fixed.

easy way to put in a test for this? (you can mark it slow, but still shouldn't take forever, or blow up memory, not sure if that's possible as Travis has a limited amount)

sorry about the noise - I had pushed again precisely when Travis was throwing another tantrum.
I can't think of a good way to write a test, but locally, you could change `SIZE_MAX` to like `10` and check that even if that path is taken, there will be no seg fault.  That matters for 32-bit environments way more than for 64-bit. 
But that path shouldn't have been taken in OP's case. 

@kawochen yeh, let me give this a try on windows.

can you put the test case generation code here for refernce

added to top

ok dind't crash windows (though didn't finish as don't have a lot of memory and was swapping)...oh well

thanks for the fix @kawochen
BUG: .loc assignment of datetime with tz is coercing to naive #11365
State: closed

closes #11365 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
sorry - i realize the fix is incomplete. will close and submit a new pull request soon.

you don't need to close
just push more commits in this one
interpreting index name in min_itemsize specification #11364, #10381
State: closed

closes #10381 

The fix replaces named index columns in [`Table.index_axes` with their names](https://github.com/pydata/pandas/compare/master...DSLituiev:master#diff-1b15d1477da3a0548d2dd72a5d023d00L3323)  and [corrects `min_itemsize` if it contains 'index' item which is not in `Table.index_axes`](https://github.com/pydata/pandas/compare/master...DSLituiev:master#diff-1b15d1477da3a0548d2dd72a5d023d00R3093)


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
this is where to make the change: https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L3248

I updated the test, but sorry I do not see how to handle it by changing https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L3248 as it works  on `data_columns` and the issue concerns the index columns.

tests should be actually in test_pytables.py

and just slot into the existing infrastructure
you are reinventing the wheel 

I am sorry to admit I am unfamiliar with your infrastructure and conventions. I am not a professional software developer and it is first time I am trying it. Does it need to come in a special place? Anyway, I do not get your idea with implementation.

guidlines are [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

no worries. just trying to help / get people to contribute.

closing, but if you'd like to update, pls reopen

No problem. I anyway did not get how you want it to be solved. 
As I wrote  https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L3248  works on data_columns and the issue concerns the index columns. 

I think the references get moved. (as the file was updated). no worries. I was looking for a general fix ,rather than special casing here.
fixed pathlib tests on windows
State: closed

as requested here: https://github.com/pydata/pandas/issues/11033#issuecomment-148924778

i can’t test on windows unfortunately.


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
use this

```
In [6]: os.path.join('',*['foo','bar'])
Out[6]: 'foo/bar'
```

much easier: `os.path.join('foo', 'bar')`

@flying-sheep right!

ok ping on green.

passes on windows

ping!

thanks!