BrandonFors/pandas-github-issues huggingface dataset:
DOC: Remove quasi-non-public parts of Index API from API docs
State: closed

We added some notes on Index to the API Reference that probably
shouldn't be there at all, since they're not meant for public
consumption or not documented.  I've slimmed down the methods to set
ops, copy, metadata (i.e. names) and a few of the DatetimeIndex methods.


Labels:
name: Docs

Comments:
@jorisvandenbossche look okay to you? Anything you think ought to be there?

I think h ought to leave the datetime selectors, eg day,hour,etc

also tolist

I would really prefer these to stay. Many times I have had to go to the index objects themselves to perform operations.  These methods should be documented for somebody who requires more control over the index objects. The disclaimer at the beginning of the index section alludes to the fact that many of the index operations should be done at the dataframe and series level and I would really prefer to keep it that way. It will save people a lot of time looking for documentation rather than going to the source code. To make it more concrete we could put an asterix by the private methods. Please keep this around. 

The datetime components should indeed certainly stay I think. For the other methods I have a less strong opinion as I don't use them myself. But methods to select values in the index, like `isin` seems certainly useful for users to me. @jtratner Can you explain why you think index methods should in general not be public?

And by the way, removing them from the api.rst pages won't remove the docs itself, as they are all linked to at the Index docstring page (http://pandas.pydata.org/pandas-docs/dev/generated/pandas.Index.html#pandas.Index)

Just for clarification, I found the lack of DatetimeIndex documentation frustrating and that's why I put it in.  Personally, I find that DatetimeIndex is used by itself much more than Index as Index has standard replacements such as range, xrange but DatetimeIndex does not and that's why functions like date_range and the DatetimeIndex class are so important.  And since DatetimeIndex subclasses Index it made sense to separate those parts of the API.  Having the class specific documentation that @jorisvandenbossche points to is great.  I still think some of the methods that are removed should remain (all the DatetimeIndex stuff, Index->Selecting section, tolist, order, argsort).

@jorisvandenbossche IIRC you arlready fixed much of this? close this one?

@jtratner closing this.....api been changed quite a bit....pls reopen if needed
RLS: first release candidate for v0.13
State: closed


Labels:

Comments:
hold off a bit on actually building
I want to put the revised SettingwithCopy in

ok np. just say when

ok ready for rc1

there are a couple of open but minor issues which can be put in later I think

ok cool, let me kick off the windows tests just to make sure, then i'm going to merge this and tag the rc1

gr8

ok tests are going, probably going to take an hour to run through all the versions.

gr8 this is on shining panda?

Great! (and seems like we don't need to maintain a separate 0.13 branch now)

@jtratner FYI should out up an issue for 0.14 (as soon as 0.13 final is out) to change the version to 0.14-dev

no not yet, we're going to get that sorted out asap. jenkins still for now.

On Fri, Nov 29, 2013 at 11:30 AM, jreback notifications@github.com wrote:

> gr8 this is on shining panda?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5617#issuecomment-29533699
> .

don't we need to set `ISRELEASED=False` right after this so we can tell the actual version number people are running?
Expand groupby dispatch whitelist (GH5480)
State: closed

(and create a groupby dispatch blacklist)

closes #5480.


Labels:

Comments:
looks fine to me - @jreback any objections?  I think this needs to end up in 0.13

my problem with this is we have a whitelist; by definition anything not on it is blacklist. not a big fan of listing all kinds of methods (bad enough that have it on whitelist), but going to be a problem as there are easy way to maintain this

so why is there a blacklist? and not just disallow not on whitelist - I thought this pr was just going to add some methods to whitelist which is ok

I however do like the idea of testing all methods on the whitelist (and hard coding in the test suite) so that if something is changed the test will fail (also should compare the whitelist to the defined whitelist) and then also test that not on whitelist will fail (with a sample of methods)

remember that the worst case is something is left off of whitelist - you can still do it via apply not just automatic dispatch - as long as most common are on their it is fine 

@jreback, a blacklist was created to block access to some methods while deprecating access to others (my interpretation of the comments in #5480).  In summary, the changes in this PR are:
1. add the following methods to the groupby whitelist:
   
   ```
   'dtypes', 'value_counts', 'mad', 'any', 'all', 'irow', 'take', 'shift', 'tshift', 'ffill', 'bfill', 'pct_change', 'skew', 'corr', 'corrwith', 'cov'
   ```
2. create a blacklist to block access to:
   `'eval', 'query', to_*`
3. change AttributeError to a DeprecationWarning when accessing methods not in either whitelist or blacklist

Re: testing, I added a new commit to this PR with improved whitelist testing (the full whitelist is hard-coded in the test suite now and compared to the defined whitelist).  Since some methods in the whitelist are unique to Series or DataFrame, the whitelist needed to be refactored a bit, but that should clean up erroneous tab completion results (i.e., dtype on a DataFrameGroupBy object).

Let me know if I should back out changes 2 & 3, or if there are any other suggestions.

1. is fine

I am -1 on  blacklist; this is going to be impossible to maintain. users will see the error and see that they are using an unsupported access method. If a method which is not on the list and should be then it can be added (as you are doing in 1), otherwise apply can be used.

so I would back out 2 & 3 (but leave in tests for non-whitelist methods)

Ok, thanks @jreback.  I've backed out 2 & 3, rebased, and squashed the commits.

looks ok to me...@jtratner?

can u  add `cumcount` to list as well (since its a defined method it bypasses this anyhow, but for consistency we have these methods in the whitelist ), this is new method in 0.13

np. updated the pr to include cumcount in the whitelist. 

this looks ok..>@jtratner?
BUG: replace with a scalar works like a list of to_replace for compat with 0.12 (GH5319)
State: closed

closes #5319

Essentitally this is `ffill`

```
In [1]: Series([1,np.nan,2]).replace([np.nan])
Out[1]: 
0    1
1    1
2    2
dtype: float64

In [2]: Series([1,np.nan,2]).replace(np.nan)
Out[2]: 
0    1
1    1
2    2
dtype: float64
```

A bit confusing on a Frame, so raise an error

```
In [3]: DataFrame(randn(5,2)).replace(np.nan)
TypeError: cannot replace [nan] with method pad on a DataFrame
```

This was the exception before this PR on a Frame

```
TypeError: If "to_replace" and "value" are both None and "to_replace" is not a list, then regex must be a mapping
```


Labels:

Comments:
@cpcloud can you take a look?
Add image for whatsnew docs
State: closed

This should have been in PR #5550, but the directory is gitignored, and I missed it.


Labels:

Comments:
I've checked, and this appears as it should in the latest docs build. Thanks @jreback .

yep looks good

one thought - does it make sense to show not just the first max_rows
but instead 1/2 of max_rows from the front and 1/2 from the end?
kind of like series shoes a long display

(use iloc to select the rows out)

should be a simple change?

I guess that would be more consistent with Series and numpy arrays. It's not trivial to get it right for text and HTML representations, including multiindexed frames, though - a certain amount of refactoring would be needed, I think.

ok
why don't u open a separate issue then
and if u can work on it?
can try to get it in 0.13

OK, I've made #5603.
ENH: Add wide_to_long convenience function
State: closed

closes #4920

I thought I submitted this PR long ago.


Labels:

Comments:
looks fine, can you add a mention in release notes / v0.13.0 (example if you want)...thxs

does this close an issue?

As a remark from somebody who isn't familiar with the function(ality) and is just looking at the docstring, it is not really clear for me what it does. What is meant with wide panel/long format? What is a subobservation? What are stub names? Terminology that I don't really have encountered yet (in pandas).
In first sight, it seems a little bit specific (or the terminology). But is is possible it comes from STATA? (I searched for "stub names" on the internet, and the only relevant thing I found was something from reshape in STATA).

@jseabold - you proposed it here - #4920 :)

We'd discussed putting this as a keyword arg to melt in #4920 - thoughts on that?

@jorisvandenbossche Yes, it's inspired by stata, but wide format panel data is fairly common in longitudinal datasets in economics. There are many stack overflow questions on how to do this in pandas.

Stub names are what I'm calling the common roots of the variable names. Stub because it's the shorter part of the variable name that you want. E.,g. for "A\d*", it's "A". See the example given in the docstring.

@jratner I guess I wouldn't mind if the functionality were in melt, but sometimes convenience functions are nice. Especially if you haven't used 'melt' in R, then you wouldn't look there. I certainly didn't. (I also find the melt docstring to be sparse, though the examples help immensely.)

C.f., and others http://stackoverflow.com/questions/17688155/complicated-for-me-reshaping-from-wide-to-long-in-pandas

can you add a release notes entry reffing the issue? otherwise good to go

Will try to remember when I get home tonight. Y'all need to hook something like this up. Life saver.

https://github.com/statsmodels/statsmodels/blob/master/tools/github_stats.py

I tried this function to solve the stackoverflow question you linked to, but I get an error: http://nbviewer.ipython.org/github/jorisvandenbossche/scipy_notebooks/blob/master/pandas-pr-wide_to_long.ipynb#Test-example-of-stackoverflow (it's due to the fact that the second part of the column names (after the stub) are no ints but strings)

Yes, this is a convenience function intended for proper panel (longitudinal) data where the stub variables are wide format timeseries data. I suppose there's nothing stopping this from taking a stub ending regex that defaults to ints though. I'll have a look.

@jseabold merge this? hold off to look at handling ints? (or just raise as NotIMplemented for now)?

Hold off, if you can. I should be able to get to it this weekend.

ping @jseabold 

Fixed the int conversion. Someone could likely make this more general. Also rebased.

pls squash down and can merge

also if you want to put your docstring example in reshape.rst (if you think its needed), you can put a mention in v0.13.0 as well (up2u)

thanks @jseabold
DOC: styling clean-up of docstrings (part 1, frame.py)
State: closed

Some stylistic clean-up of the docstrings in frame.py.

Things changed are mainly:
- missing spaces aroung the `:` in `param : type`
- explanation of param on second line (instead of on the same line -> there should be the type info)
- add bullets to enumerations of possible values of a keyword (so they are rendered as a list in the html docstring instead of one continuous block of text) Eg the `method` keyword explanation in http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.corr.html


Labels:

Comments:
:+1: 

@jorisvandenbossche is this ready to be merged? (I'm going to look through one more time, but just wanted to make sure you aren't planning to do anything else)

@jtratner Yes, ready to merge! 
I just only changed the bool -> boolean now, and as I said, for the rest more some numpydoc-style changes, not really content changes.

Looks good - thanks!
TST: fix cumcount test on 32-bit env
State: closed

Fix `test_groupby.test_cumcount_empty()` on 32-bit platforms by specifying expected dtype of 'int64'.

```
======================================================================
FAIL: test_cumcount_empty (pandas.tests.test_groupby.TestGroupBy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gmd/github_shadow/pandas/pandas/tests/test_groupby.py", line 2579, in test_cumcount_empty
    assert_series_equal(e, ge.cumcount())
  File "/home/gmd/github_shadow/pandas/pandas/util/testing.py", line 423, in assert_series_equal
    assert_attr_equal('dtype', left, right)
  File "/home/gmd/github_shadow/pandas/pandas/util/testing.py", line 407, in assert_attr_equal
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "/home/gmd/github_shadow/pandas/pandas/util/testing.py", line 390, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int32') != dtype('int64')
```


Labels:

Comments:
We should add a note about using int64 instead of int. Not that it will
necessarily resolve this completely.

Thanks for catching this! I'm hoping we can get an automatic 32bit build from Travis at some point. 

Thanks Jeff..  For completeness, here's the open issue tracking 32-bit env support in Travis: https://github.com/travis-ci/travis-ci/issues/986
TST: correctly predict locale setability
State: closed

Don't join the lang and encoding when performing the trial set, because

```
locale.setlocale(locale.LC_ALL, ('it_CH', 'utf-8'))
```

succeeds, but

```
locale.setlocale(locale.LC_ALL, 'it_CH.utf-8')
```

throws locale.Error.

Closes #5537.


Labels:

Comments:
Thanks! (and it's great that Travis builds PRs now)
PERF faster head, tail and size groupby methods
State: closed

Try again with #5518.

Massive gains in groupby head and tail, adds more tests for these, slight speed improvement in size (not as much as I'd hoped, basically iterating through grouper.indices is slow :( ).

_As mentioned before, I added a helper function to prepend as_index to index (if that makes **any** sense), I think it could be faster, and also using it in apply may fix some bugs there..._


Labels:

Comments:
@jreback I fixed this up. A slight api change is that head now respects original frame order (it doesn't if you do an apply).

can you add some tests where the groups vary in size and where the number you are asking for (e.g. head(3)) is > than the number in some/all groups.

Added tests for <=0 and > max group size. One already for between group sizes.

(though it's a small example, I think it covers...)

Added some comments to the docstrings.

I have another question regarding the docstring style:
- You changed the `"""` to ``'''`. Is this recommended, or is there some 'policy' on this in pandas? Because I always use `"""` (and this is also used in PEP8: http://www.python.org/dev/peps/pep-0008/#documentation-strings).  
  It doesn't matter at all for me what we use, but maybe we should try to be consistent, and I had the impression that `"""` is used the most in the pandas source code

@jorisvandenbossche  Thanks for comments, will update.

Fixed these. Mentioned the ascending arg to cumcount, which I've purposely made kwarg only for now.

_Noticed a weird related thing with nth on a DataFrame (it just doesn't work, and is kinda undefined), will make sep issue though._ #5552
StataWriter: Replace non-isalnum characters in variable names by _ inste...
State: closed

...ad of integral represantation of replaced character. Eliminate duplicates created by replacement.


Labels:

Comments:
this looks fine, can you add a release notes entry? 

Done.

I had to revert this commit

https://github.com/pydata/pandas/commit/a5a2c9210b9399d84df69826c089095b3920f190

This fails on win32 with python2.6, because of the unicode in the source file

you can do something like this (but then the test breaks)

https://github.com/pydata/pandas/tree/stata_fix

pls submit a new PR with the fixed version
BUG/TST: Fixes isnull behavior on NaT in array. Closes #5443
State: closed

I added a test case test_isnull_nat() to test_common.py and a check for NaT in lib.isnullobj. pd.isnull(np.array([pd.NaT])) now yields the correct results ([True]). 

closes #5443


Labels:

Comments:
this needs to be profiled via test_perf.sh

I ran test_perf.sh on the patch. Here is the vb_log: https://gist.github.com/commonlisp/7488695

Ah, yes, I did have the check in util._checknull at first, but this creates a circular dependency between util and tslib where NaT is defined. Is that ok? Or perhaps NaT should be factored out. 

actually...this whole change is trivial....just change `_checknull` to `checknull` in `isnullobj`; this already checks for NaT. and then profile again (it prob will be fine because `isnullobj` is only called if its object type in the first place)

Great call. Using checknull now and added the new explicit test case. Here is a fresh test_perf.sh: https://gist.github.com/commonlisp/7492647. Going to run it again just to make sure there aren't any aberrations. 

I added a gist https://gist.github.com/commonlisp/7547232 with another run of test_perf.sh. It appears to agree with the previous run.  

The test_perf.sh runs for modify isnullobj directly and using checknull do differ somewhat (1.22 vs. 1.87). The question is whether this is within test_perf.sh's margin of error.  

that's definitly out of bounds, pls see if its something simple..

I've run a good number of performance tests. It seems that util.is_float_object, util.is_complex_object, util.is_datetime64_object, and util.is_timedelta64_object, though declared inline functions in numpy_helper.h, are contributing considerably to the runtime (~1.26-1.5x individually). We could factor out yet another version of lib.checknull that skips those checks to support the lib.isnullobj case.  

ok I guess having a special version of checknull_object might be useful for object arrays where u can skip certain checks (but still need some that _checknull does not do)

Any reason not to get this merged in for v0.13?

@cancan101 have to nail down the perf issue, this code is used everywhere

That makes sense. I wasn't sure if a perf fix was worth waiting for.

New test_perf.sh for this factoring of lib.checknull: https://gist.github.com/anonymous/8017235

I think this is ready for review again. Please let me know if anyone has feedback.

can you squash the commits: https://github.com/pydata/pandas/wiki/Using-Git

and post a perf run...just to check 

thanks

checkout this: https://github.com/pydata/pandas/wiki/Using-Git

you don't want to merge in other commits (nor merges)

so `git rebase -i origin/master`...then delete everything but your commit

then

`git push myfork thisbranchname -f`

Commits squashed, not merging any other commits
Recent test_perf.sh https://gist.github.com/commonlisp/8290153

you need to take out all but your commit

do this:

`git commit -C HEAD --amend`

then force push

this will make it build again

then'll I merge it....

@commonlisp 
thanks!

first time working thru the process is painful......next time should be easier!

@jreback Thank you!
CLN: Add more methods to whitelist (for now)
State: closed

No test cases because goal is just to make sure this is not blocked in
0.13. Can do more tweaks later. If I have time I will try to pull
together something to cover what we were discussing in #5480.


Labels:

Comments:
"for now" ?

As in, do it for 0.13 and then figure out about adding test cases and
whatever else later on when we have more time to discuss (since both you
and @gdraps had issues with things not being whitelisted that were supposed
to be).

ok, sweet. I wonder if these are low hanging fruit for vector / cythoning.

@jtratner what's the status of thsi?

I have a branch with a tentative PR that I can submit later today.

One hitch found while testing is that the whitelist is used for groupby tab
completion but some methods in it are specific to Series or DataFrame
(dtype, corrwith, value_counts). Any thoughts on tailoring the tab
completion based on groupby object type, or should the whitelist be limited
to common methods?
On Nov 27, 2013 10:34 AM, "jreback" notifications@github.com wrote:

> @jtratner https://github.com/jtratner what's the status of thsi?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5514#issuecomment-29408564
> .

I'd be inclined to just leave it for both...once you try to use it you'll
learn that it's not available anyways. That way we can keep things more
simple.

The other option is to add it in the Series/DataFrameGroupby class rather than to the whitelist.

ok, I put a quick fix PR out there: #5604.  I used @jtratner's approach of leaving Series/DataFrame-specific methods in the whitelist (dtype, dtypes, boxplot, corrwith, and value_counts).  Feel free to use it, or not.

closing in favor of #5604
ENH add cumcount groupby method
State: closed

closes #4646

Happy to hear ways to make this faster, but I think this is a good convenience function as is.

Update: made significantly faster...


Labels:

Comments:
As an aside, I was sure you used to do be able to do `g.value_counts()`... guess I was mistaken.

@hayd that's an open issue and being re-enabled. You could reincorporate it here if necessary to make this PR works.

I really there ought to be tests for dupe index too. In which case this implementation fails (!)

@jreback Is this a wider bug in apply alignment / or is there a cheeky way to do this without a sort:

```
In [52]: df = pd.DataFrame([['a'], ['a'], ['a'], ['b'], ['b'], ['a']], columns=['A'], index=[0] * 6)

In [53]: g = df.groupby('A')

In [54]: pd.concat(pd.DataFrame(np.arange(len(v)), v) for v in g.indices.values()).sort_index().set_index(df.index)
Out[54]: 
   0
0  0
0  1
0  2
0  0
0  1
0  3

In [55]: g.apply(lambda x: pd.Series(np.arange(len(x)), x.index))  # not correct 
Out[55]: 
0    0
0    1
0    2
0    3
0    0
0    1
dtype: int64
```

Refactored to work with dupes, also faster:

```
In [1]: import pandas as pd; import numpy as np

In [2]: df = pd.DataFrame(np.random.randint(0, 2, 1000), columns=['A'], index=[0] * 1000); g = df.groupby('A')

In [3]: %timeit g.apply(lambda x: pd.Series(np.arange(len(x)), x.index))
1000 loops, best of 3: 786 µs per loop

In [4]: %timeit pd.concat(pd.DataFrame(np.arange(len(v)), v) for v in g.indices.values()).sort_index().set_index(df.index)
1000 loops, best of 3: 568 µs per loop

In [5]: %timeit g.cumcount()
10000 loops, best of 3: 41.6 µs per loop


In [8]: df = pd.DataFrame(np.random.randint(0, 100, 1000), columns=['A'], index=[0] * 1000); g = df.groupby('A')

In [9]: %timeit g.apply(lambda x: pd.Series(np.arange(len(x)), x.index))
100 loops, best of 3: 7.35 ms per loop

In [10]: %timeit pd.concat(pd.DataFrame(np.arange(len(v)), v) for v in g.indices.values()).sort_index().set_index(df.index)
100 loops, best of 3: 10.7 ms per loop

In [11]: %timeit g.cumcount()
1000 loops, best of 3: 215 µs per loop

```

@jreback / @cpcloud too late for 0.13 ?

@hayd I think this is ok fine for 0.13. maybe a doc mention in groupby.rst (could be a separate PR)

Updated those to int64, good spot.

I'm not sure what/where to add in the groupby.rst (there is nothing about cumsum and friends either)... ["Other useful features"](http://pandas.pydata.org/pandas-docs/dev/groupby.html#other-useful-features) ?

maybe create another section after `Flexible apply`, which `apply` examples?

Just appended a small mention of it at the end of groupby.rst. ...cumsum etc. are covered in dispatched (though could do more in the future). :S

going to go ahead an merge this :)