BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix nested dict replace
State: closed

closes #6342


Labels:

Comments:
see u r hacking the biggest hack in internals, putmask
because of the dtype changes that are possible

yes this is definitely a hack. `np.putmask` will cycle values after a false is found if the size of the array and replacement values are not equal but it will only use the next value in the replacements if the `True`s are contiguous ... highly counterintuitive behavior for that if u ask me ... is there a more optimal way that i'm overlooking?

internals is kind of fun .... always have to clear out my weekend when a bug involves internals.py :smile: 

fyi `copyto` raises in the case that the sizes are unequal, but only available in numpy >= 1.7

I avoided `copyto` for that reason

yeh...`putmask` has a lot of 'special' cases....which I think really are just weird things numpy does

`replace` is turning into a god function ... worth opening an issue?

what do you mean?

It's just really, really large ...

hahah so true!

next project...you can effectively replace `update` with replace!

@jreback What do u think about this? I'll run a vbench to make sure this doesn't cause a hit.

sure....looks fine otherwise.....not even sure how often that path is actually hit (can you put a case that hits it up?)

Nested dicts hit it if they contain keys that aren't in the frame/series

``` python
df = DataFrame({'col': range(1, 5)})
df.replace({'col': {-1: 'a', 1: 'b', 2: 'c'}})
```

ok...then prob ok....not hit a lot...but do a perf check anyhow

Perf looks good:

```
packers_write_pack                           |   0.9952 |   0.9129 |   1.0901 |
stats_rank_average                           |  23.8746 |  21.2846 |   1.1217 |
timeseries_asof                              |   7.9971 |   7.0529 |   1.1339 |
join_dataframe_index_single_key_small        |   6.4060 |   5.5654 |   1.1510 |
timeseries_asof_nan                          |   8.0025 |   6.8281 |   1.1720 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

Random network failure unrelated to this

https://travis-ci.org/cpcloud/pandas/jobs/18993144

merging
DOC: clarify docstring of rolling/expanding moments
State: closed

- document center argument
- add note about the result set at right edge by default
- clarified freq keyword a little bit

Closes #5504


Labels:

Comments:
Two more things:
- there is some more room in avoiding duplication of the docstrings (more sharing)
- there seems to be a problem with the `@Appender` decorator picking up the docstring of the underlying cython function (see eg `expanding_median` at http://pandas.pydata.org/pandas-docs/dev/api.html#standard-expanding-window-functions)

related is #4950

yes I saw that now. The PRs are not overlapping, but that will be a more painfull rebase for one of us then .. (the other PR also solves a little bit the first point of avoiding duplication)

Merging this as at the other PR there is not direct action.
BLD/TST: fix bool block failures when strings are passed to replace list
State: closed

closes #6353


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Oh darn this will not pass. This allows different size lists thru

gotta run will get to this tonight after work

@jreback This line (from `TestSeries.test_replace`) is masking the same failure, which is a `TypeError` and it in fact should be a `TypeError`, but not because it's trying to index a `bool` object.

``` python
ser = Series(tm.makeTimeSeries().index)
self.assertRaises(TypeError, ser.replace, range(1, 3), [np.nan, 0])
```

I think we should raise a `TypeError` on these kinds of replacements. It makes no sense to do something like

``` python
s.replace({'asdf': 'fda'})
```

on a `bool` `Series`. I would argue that even something like

``` python
s = Series([True, False])
s.replace({'asdf': 'fda', True: 'abc'})
```

should raise as well, because only part of the replacement is valid. @jreback What do you think?

Darn, that's not really symmetric with other types though.

Problem is that the time series behavior is throwing typeerrors for the wrong reasons, but i don't see a general purpose way to fix both this issue and that one without special case one of them

But since we have to maintain the time series behavior for back compat and the bool behavior is now being defined, i think we should raise a typeerror on any of these kinds of conversions...doing that doesn't break any existing code, but makes it a bit more confusing for users to figure out what works .... e.g.,

``` python
s = Series([True, False])
s.replace({'a': 'b'})
```

will raise, but 

``` python
s.replace('a', 'b')
```

will not

can you make an entire pass thru on boolean? I think it should just return directly (and if they actually try something like:

```
Series([True,False]).replace(True : 'b')
```

maybe just raise

but all other's will simply pass thru...
?

should the possibly compare stuff (and raising) just be in mask_missing?

Tried that already, it breaks things

ok then

merge when ready

I'm going to write a bit of docs and release notes first ... just to clarify these somewhat dark corners of the replace API

github looks back to form now......pls merge when you can....(of course docs good too!)

looks cool...thanks!

got that other replace fix coming soon to a pandas repo near you
ENH: add method='dense' to rank
State: closed

This adds a `method='dense'` option to `rank` which does the same thing as the one in `scipy.stats.rankdata`: the first ranked group gets 1, the second 2, etc.  Basically it ports the approach used in `scipy.stats._rankdata_fused` to the pandas rankdata implementation.

It also adds a few tests that the various methods do what they're supposed to, which I couldn't find a test for before.

closes #6333


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
(1) The tests I couldn't find were lurking in `test_stats.py`, so I should add to those instead of tossing them in `test_series.py`.

(2) I'm not sure what I'm doing which is breaking everything -- the code passes every test for me.  Any obvious goofs I'm missing?

nothing obvious....

Ah, got it.   When I added `total_tie_count`, what was formerly `dups=0` became `dups, total_tie_count=0` and `dups` was no longer being initialized.  That explains why even the cases which shouldn't have been affected at all started breaking, and why my tests passed even though Travis was barfing.

It must have worked for me by fluke.  Will fix.

Managed to get it working last night but got the branch into a weird inconsistent state.  Will make a new one instead.
BUG: fix bool no-op replace calls
State: closed

closes #6332


Labels:

Comments:
@jreback look okay to u?

why does this work?

For the same reason I mentioned in #6332. `np.atleast_1d` is a call to `array` plus some insurance that you get at least a 1D array. Using the SO example, that will create an array of one string. `S` dtypes cannot be safely cast to `bool` therefore I pass in the array of strings which will ultimately raise `ValueError` in `mask_missing` which will convert them to `object` dtype. Then `mask_missing` procedes to compare each non NA value with those in `arr`. Since no strings are equal to `True` or `False`, the mask is an array of `False`s. Does that suffice?

`dict`s work too

```
In [18]: s
Out[18]:
0     True
1    False
2     True
dtype: bool

In [19]: s.replace({'asdf': 'asdb', True: 'a'})
Out[19]:
0        a
1    False
2        a
dtype: object

In [20]: s.replace({False: 'asdb', True: 'a'})
Out[20]:
0       a
1    asdb
2       a
dtype: object
```

If there's no test I'll add one

One thing that won't work as "expected" is `df.replace({True: False, False: True})`. The keys are iterated over in an arbitrary order and you'll end up with either all `True`s or all `False`s. I don't really think this is that common.

merge away

neat trick with atleast1 d
BUG: #6175 removed assert_  statements 
State: closed


Labels:

Comments:
we need to update and/or create a new section to enumerate the methods that should be used for testing going forward

https://github.com/pydata/pandas/wiki/Testing

can u write up something ? (u might be able to directly edit this page but not sure)

It looks like I should be able to edit the testing wiki. There is an open issue for improving test coverage, and as part of that we are talking about documenting testing practices and cases (#6251).

I will take a look at starting a new section. Though the most helpful resource at this point is likely docs.python.com, as testing.TestCase doesn't contain many specific methods. Though, I have a feeling that will change as we get rid of the assert_ statements.

Which version of Python are tests expected to run on?

all that Travis tests 

@KTAtkinson 
Python 2.6, 2.7, 3.2 and 3.3

@KTAtkinson did you incorporate this PR elsewhere? close this one? or rebase

@jreback I will rebase. I am currently trying to figure out how to get python 1.6 working on my machine, with all the necessary deps.

1.6 would be hard! assume you mean 2.6!

use a virtual env......(install python2.6 from apt-get)

I have 2.6 installed, but I am having issues getting numpy working on 2.6. Though it works on 2.7.

hmm....just install using pip/easy_install..

I did. So I originally installed numpy/nose with only 2.7 installed. When I installed 2.6 numpy didn't seem to work, so I uninstalled it and re-installed to see if that would make it available in 2.6. Unfortunately it didn't.  

@jreback Ended up using virtualenv for python. It seems to work nicely.

gr8.....pls rebase this...I just merged in a PR which touched test_bases

Any ideas what is the issue with the most recent failure? I don't know what the issue is with Python 2.6. I added the method assertIsInstance to the tm.TestCase, but it doesn't seem to be able to find it.

I still don't have 2.6 running on my own machine, which is also making this very hard to troubleshoot.

@jreback Looks like The PR you submitted last night fixed all the issues with this test. I am closing this PR.

This may be a mistake on my part, but I do not see the addition of assertIsInstance to testing.py. I see (in the "Files Changed" tab at the top of this page) changes to three files: test_base, test_categorical, and test_testing. If that is the case, 2.6 would be failing because the assertIsInstance method was added in 2.7: http://docs.python.org/2/library/unittest.html#unittest.TestCase.assertIsInstance
ENH date_range accepts timedelta as freq
State: closed

fixes #6307


Labels:
name: Enhancement
name: Timedelta
description: Timedelta data type
name: Frequency
description: DateOffsets

Comments:
Can you also update the date_range docstring?

you need to fail this if u r on numpy < 1.7 (use pd._np_under_p17); raise that it is not supported (see pd.to_timedelta) for what to do

and skip tests if the same (np.timdelta64 has a diff signature under 1.6)

whoops, I had made docstring, but forgot to save that file.

Thanks, will fix up for <1.7.

looks like datetime timedelta don't have a total_seconds attribute in 2.6.

yep see my comment in the pr - just pull out the days, seconds and microseconds directly 

@hayd any thoughts about not having this always return micro/nano and instead a higher freq if possible?

Yeah, will have another look at this tomorrow.

It begs question should we _always_ be simplifying ticks?

```
Second(30) + Second(30) == Second(6)  # not Minute(1)
```

hmm offhand I'll say that the resample code doesn't care whether it's simplified or not (and doesn't make a difference) - 

can u give a look. see to see if it matters?

resample won't _care_, will be in the freq repr for the index...

datetime.timedelta does some simplifying think we probably should. sigh, to micros being timedeltas smallest unit, this means the delta attribute does different things for nano and everything else:

In [21]: Nano(1000) == Micro(1)
Out[21]: False

Will fix as part of this...

@hayd ok with merging this like this (after rebase)....then maybe create another issue about the simplification stuff?

yea, I refactored this a little already (but not pushed and away atm) so future change should be straightforward. Will do next week.

ping!

ping

moving to 0.15

related #5963.

I think the pending part of this was comparison of Tickers (previously `Nanos(1000) != Micro(1)`). Think this should be fixed...

if u want to go down the rabbit hole
ok (1.6 is really broken though) but do after 0.14

Note: docstring doesn't require changing as it already suggests that a timedelta should work.

pls change release notes to v0.14.1 thanks!

added and green (on my travis).

I think it makes sense to simply if its unambiguous

agree, done. Also removed some "horrible-ness".

Hmmm I thought this had already been merged in.

rebase this, and I think you have a 2.6 failing issue

@hayd ?

@hayd let's revist

Sigh, it's only 2.6 that's at issue here (should probably remove the  `if not _np_version_under1p7` in tests...), I just need to install/build against 2.6 (ahem!)

@hayd ?

@hayd ?

closing as stale. @hayd if you'd like to update, pls reopen
FIX: hdfstore queries of the form where=[('date', '>=', datetime(2013,1,...
State: closed

```
FIX: hdfstore queries of the form where=[('date', '>=', datetime(2013,1,1)), ('date', '<=', datetime(2014,1,1))] were broken
- modified Expr.parse_back_compat to check for tuples, in w, and unpack into w, op, value
- modified Expr.__init__ to modify the where list/tuple with the parsed result
```


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
hmm

did u check master with this; I thought I had fixed this already

would also need some tests for this 

It broke for me in master, I'll add some tests though.  But first - is this the right way to fix it?  I don't know the pandas codebase that well

where u made the change is fine
their are already a bunch of tests for the back compat formats
so just add them their (u can add as another test function)
need to make sure that fix doesn't break anything 

this is fine...can you add a release note (in bug section of 0.14; reference this PR number)

@hhuuggoo can you add a release note, otherwise looks ready to go

can you rebase...

and add a test that does an `assert_produces_warning`...that validates that you are in fact showing the warning int he correct place

thanks

I did it - however something wierd happens.  The test is fine if ran with others, however if I run it on it's own using the following:

`nosetests  -s pandas/io/tests/test_pytables.py:TestHDFStore.test_backwards_compat_without_term_object`

`AssertionError: Caused unexpected warning(s): ['PendingDeprecationWarning'].`

I see some comments about this in the test as well, so it seems like it's a persistent problem?

@jreback btw, do you know why the query syntax changed?  Is there some discussion of this online I can read?

that deprecation warning is a weird thing...I have seen it using `assert_produces_warning` - and I can't narrow down why it actually happens

what do you mean why the query syntax changed? The original 'syntax' was just a hack...you could do anything beyond anding and too much verbiage to do anything...

is their a problem?

there was a whole very long discussion on this...search for 'eval' in the issues....

@jreback, no problem for me, but personally I prefer the old syntax, because it's easier to programatically build up queries using lists and tuples, rather than doing string operations.  Not a problem, I can always pull out the backwards compat code in my application if I really need to.

anyways I think this PR is done?

@hhuuggoo thanks!

its not that difficult to put in a combiner to handle built up queries (of say list of sub-queries)...
you may need a wrapper class (which basically does what the older code did, but ends up stringifying)

I think that is done now (but maybe only if you pass Terms...can't remember)

interested?
CLN: Convert last of assert_(x == y) to assertEqual
State: closed

Work on #6175. Fixes last of vanilla assert_(x == y) conversions in pandas/tests and pandas/*/tests. Tests still pass locally.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
The following returns seven results (when run from pandas), but none are in the assert_(x == y) category.

grep -nH --recursive -e 'assert_(.*==' . | grep -v 'all()' | grep -v 'any()'

I will handle the easy '!=' cases and then move to refactoring some of the other ideas discussed in #6175.

In your next PR can you make an entry in release notes (improvements section), referencing these previous PR's to say that the testing statement were updated (just so their is a record of what changed)

@jreback, thank you, I did this in the PR I submitted a moment ago.
Add example section to groupby.rst and one example
State: closed

following this question on  [StackOverFlow](https://stackoverflow.com/questions/21584434/group-by-value-of-sum-of-columns-with-pandas) and [this issue](https://github.com/pydata/pandas/issues/6288).
I opened an "Examples" section in the doc of `groupby()` and add an example about how a way to regroup columns of a DataFrame according to their sum, and to sum the aggregated ones.

closes #6288


Labels:

Comments:
thanks!!

if you want to put some more examples, you love to expand this section!
DOC: clarified to_excel and read_excel (GH6260)
State: closed

as discussed in issue #6260
- io.rst: added a paragraph explaining the difference in behaviour from 0.12
- excel.py: added docstring for has_index_names

thanks to jmcnamara

ps: sorry about the pollution with multiple commits, I'm still learning how to rebase/squash


Labels:

Comments:
Thanks for the contribution! I just added some small comments.

There also seems something wrong with your history of commits (undo and redo again). You can clean this up by rebasing interactively (https://github.com/pydata/pandas/wiki/Using-Git#wiki-fetch-and-then-rebase-interactively-to-squash-reword-fix-otherwise-change-some-commits) and then squash the 5 commits in one.

@jorisvandenbossche  thanks for the education. Please let me know if everything is correct now.

That looks very good now! Thanks again
DOC: clean-up docstrings of sql + fix doc build
State: closed

@mangecoeur I don't think I changed some real content, just some adjustments for the numpy docstring standards. Only for has_table I corrected the parameters.

And the second commit should fix the doc builds (see http://pandas-docs.github.io/pandas-docs-travis/io.html#io-sql)


Labels:

Comments:
nothing disruptive, so going to merge.
ENH: All args and kwargs to generic expanding/rolling apply.
State: closed

Been bugging me lately. Simple enough fix I think. Example usage

```
def mean(x, const):
    return np.mean(x) + const

df = pd.DataFrame(np.random.rand(20, 3))
pd.expanding_apply(df, mean, args=(20,)

pd.expanding_apply(df, mean, kwargs=dict(const=20))
```


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
sure

Fixed failing tests. Maybe hack-ish.

this loooks indepedent of #5071 yes?

Yeah, I didn't find an open issue for this. 

not sure of the test coverage here; but are their any tests for a function which doesn't take any arguments? (e.g. does it still work)? I assume cython raises some kind of error if the function take args but they are not supplied?

Yeah, it will raise. That's why I added the args=(), kwargs={} to the Python signatures, so that there are always defaults. AFAIK, the Cython function isn't used outside of the wrappers in moments.py, so it should be enough to ensure that the wrapper functions always call algos.roll_generic with some default args.

merged via https://github.com/pydata/pandas/commit/1e9c0a630db9b316edfaa86ad7b8f34f5d462b89

thanks!
BUG: Fix interpolate with inplace=True
State: closed

Closes https://github.com/pydata/pandas/issues/6281

I didn't have the non inplace return in an else block. And I clearly didn't test this :/


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
@jreback sorry I didn't see the failure earlier (Yahoo related). Green now.

ok....then doing the downcast change as another PR then....
PERF: micro optimizations
State: closed

PERF: enhance tseries getitem indexing
PERF: micro optimzations in inference
API/CLN: core.internals.apply now only takes kwargs; makes code a bit simpler
CLN: wrap core/common/isnull into more direct Block.isnull
PERF: fast path series block operations

closes #6264

original timings

```
In [11]: %timeit com.is_integer(13)
100000 loops, best of 3: 1.29 µs per loop

In [12]: %timeit com.is_integer(np.int32(2))
100000 loops, best of 3: 3.32 µs per loop

In [13]: %timeit com.is_integer(np.int64(2))
100000 loops, best of 3: 1.86 µs per loop

In [14]: %timeit com.is_integer(14L)
100000 loops, best of 3: 1.29 µs per loop

In [15]: %timeit com.is_integer("1")
100000 loops, best of 3: 2.2 µs per loop
```

with this PR

```
In [2]: %timeit com.is_integer(13)
10000000 loops, best of 3: 72.5 ns per loop

In [3]: %timeit com.is_integer(np.int32(2))
1000000 loops, best of 3: 620 ns per loop

In [4]: %timeit com.is_integer(np.int64(2))
1000000 loops, best of 3: 315 ns per loop

In [5]: %timeit com.is_integer(14L)
10000000 loops, best of 3: 72.4 ns per loop

In [6]: %timeit com.is_integer("1")
10000000 loops, best of 3: 78.8 ns per loop
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@y-p side issue when you have a chance

for some reason the cython caching is too aggressive here

https://travis-ci.org/jreback/pandas/jobs/18271060

this is a new branch with a cherry-picked commit (as the original branch had the same caching issue)
BLD: add support for pytables 3.1
State: closed

BLD: update 3.3 build for numexpr 2.3

closes #5861


Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: CI
description: Continuous Integration

Comments:
@y-p when you have a chance

I am not sure how to update whl for numexpr 2.3 and pytables 3.1
(or do you have whl for them?)

pls update your windows configs as well.thanks

Jeff you can 

Ping me whenever you need some wheels for Travis 

@cpcloud if you could make a whl of pytables 3.1 and numexpr 2.3 would be gr8! 

Also if you want I can put your pub ssh key on the open shift server in case I get trampled in a wildebeest stampede

thanks...just e-mailed to you

Cool thanks I'm on my way to work. I can set you up when I get in. 

I've updated the windows CI 3.4.0 build with numexpr 2.3 and pytables 3.1 final,
but the laptop has to go back to it's owner (back from abroad) in a few weeks...
no more windows CI.

hmm ok

i could run the windows ci

can u send me how to setup?

Sure.

Just checking: I don't have any issues currently with pytables 3.1 and pandas 0.13.1 (or numexpr 2.3) on ubuntu 64bit -> is that correct or am i at risk and should I downgrade any library? (couldn't really find anything about this in the normal docs)

oh it works
it just isn't in the test suite yet (their were a few small issues but are resolved)

just note that the file caching works slightly differently in older hdf versions (you will get an error message if it's too old)
i

Thanks!