BrandonFors/pandas-github-issues huggingface dataset:
BUG: GH4080, int conversion of underlying series to float needs updating in parent frame
State: closed

closes #4080

essentially a variation on #3970

here block consolidation and cache is not the issue,
rather the dtype of the series is int, and as a result of the update (directly on the series)
changes dtype to float (which is fine)

BUT the parent still has this in an Int Block

this PR essentially creates an Observer Pattern thru a weakref
that allows notification of chanes (propogated back to the BlockManager)
which can then take action if needed


Labels:

Comments:
@wesm reprise of #3970, no other way around this as far as I can see

Sigh, more signs that pandas is due for a major internal refactor

yep....you want to include this or wait till 0.13 (where even after Series refactor I think this still might be needed)?

@wesm ?

Hold on just a minute here. The Series `dtype` is changing _in place_? 

I'm seeing worse behavior on my machine:

```
In [8]: df
Out[8]: 
       val
a b c     
1 1 1    0
2 2 2    0
3 3 3    0

In [9]: df['val'].update(s)

In [10]: df
Out[10]: 
                       val
a b c                     
1 1 1                    0
2 2 2  4607182418800017408
3 3 3                    0

In [11]: df['val'].values
Out[11]: array([                  0, 4607182418800017408,                   0])
```

Don't approve of hacking around this issue or modifying an ndarray's dtype in place.

your example shows the current behavior (this with PR)

```
In [1]: df = DataFrame(dict((c, [1,2,3]) for c in ['a', 'b', 'c']))

In [2]: df.set_index(['a', 'b', 'c'], inplace=True)

In [3]: s = Series([1], index=[(2,2,2)])

In [4]: df['val'] = 0

In [5]: df
Out[5]: 
       val
a b c     
1 1 1    0
2 2 2    0
3 3 3    0

In [6]: df['val'].update(s)

In [7]: df
Out[7]: 
       val
a b c     
1 1 1    0
2 2 2    1
3 3 3    0

In [8]: df['val'].values
Out[8]: array([ 0.,  1.,  0.])
```

in place is ok for int->float, but doesn't work if its an object array (see #3217), but that's not an issue here

I don't understand why it needs to upcast? 

in this particular case it doesn't in thery, but

and this is contrived too....if we could avoid the direct series updateing (`df['val'].update(...)`)
then I think would be ok

```
In [1]: df = DataFrame(dict((c, [1,2,3]) for c in ['a', 'b', 'c']))

In [2]: df.set_index(['a', 'b', 'c'], inplace=True)

In [3]: df['val'] = 0

In [4]: s2 = Series([1.5],index=[(2,2,2)])

In [5]: df['val'].update(s2)

In [6]: df['val']
Out[6]: 
a  b  c
1  1  1                      0
2  2  2    4609434218613702656
3  3  3                      0
Name: val, dtype: int64

In [7]: pd.__version__
Out[7]: '0.11.0'

```

@wesm I guess let's push to 0.13....assume you want to get 0.12 out the door....:)

ok i'll push to 0.13/0.12.1

closing in favor of #3482 which addresses this
TST: more cleaning of yahoo/google testing
State: closed

fixes yet another bug with yahoo/google testing


Labels:

Comments:
that basically invalidates the test
these should be definite values assuming no network issue (which should raise)
or a parsing error (raise also) or just no data for some reason (I'll bet this is it)

the point is that you can't know if the network fails. it seems pointless to parse the warnings. the best solution would be to put a nan there.

alright i broke down and am now checking the warnings for the symbols

closing. going to reopen cleaning of data.py

https://travis-ci.org/jreback/pandas/jobs/8570082

?

the issue of no html. just now, lxml will read and try to parse but then no elements so it throws this error. before was the equally cryptic `AttributeError` when trying to call methods on `None`

should probably revert back the call to `getroot()` and check for `None`ity

ok....so this is an error then (when/how you catch it up 2 you)....

the return of this function in this case (since you got NO data) should be to raise some kind of error, maybe DataError?

(as opposed to getting some data, in which case you could return nan for the ones you don't get)

oh gosh i just found a list in a default parameter...>:O
Fix for #3970 cache invalidation bug
State: closed

Also fixed stable sorting test failure that presents on some platforms


Labels:

Comments:
@jreback can you review and merge pls? also
- PEP8, man (especially line length) =) 
- Can you configure your text editor to strip trailing whitespace? in emacs i have

`(add-hook 'before-save-hook 'delete-trailing-whitespace)`

thanks!

sure...no prob

when BlockManager is revisited....easiest prob to move the cache inside BM, then consolidate can just invalidate easily

merged in...thanks for the debug..
TST/BUG: fix 2to3 import rewrite of import pickle
State: closed

In pandas.io.pickle the following statement will not allow python to import 
anything due to the following rewrite by 2to3

import pickle -> from . import pickle

This makes the import try to import itself ad infinitum and thus fails.

closes #4062


Labels:

Comments:
@jreback even though i said 0.13 for this one i'd like to get this in for 0.12 would be nice to be able to run tox on py3.x...

ok by me

btw i tried w/ numpy 1.5.1 and works fine

I think it's time to ditch tox_prll.sh and move to [detox](https://pypi.python.org/pypi/detox), 
tox_prll was a hack from a time when detox was having growing pains. 

is this a dropin replacement?
TST/BUG: fix failing data.py tests for good
State: closed

closes #4028, #3982. for good.


Labels:

Comments:
:+1: 

thanks. i'm going to do a couple more rehashes + push to make sure nothing is falling through the cracks. 4-5 seems to be the magic number to get at least one fail in the build matrix. i think i've done 2 so far

Could you cc me when you merge this one?

@woodb sure thing

@jreback look ok? the last 4 git rehashes and force pushes have passed on travis. running another one just in case. when that succeeds i wiil merge

ok by me

bombs away................

cc @woodb
TST: remove test_yahoo options test skipping
State: closed

closes #4028

just a reversal of the warnings - seem ok now


Labels:

Comments:
merge?

2 to go!

yerp

This is still failing for me:

``` python
======================================================================
FAIL: test_options (pandas.io.tests.test_yahoo.TestYahoo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../pandas/pandas/util/testing.py", line 769, in network_wrapper
    return t(*args, **kwargs)
  File "../pandas/pandas/io/tests/test_yahoo.py", line 120, in test_options
    assert len(calls)==5
AssertionError
```

this works when I run it at the command line.....any reason you can think why this sometimes fails?

@jreback pull down my `data-reader-test` branch. you should be able to repro the failure there if you do `git reset --hard HEAD~1` and then run the test suite. it fails because there's a call to `dropna` without the `all` argument and the bid and asking price columns are all nans.

i've separated all of the tests so that failures are more isolated and we can tell wtf is going on. before that, everything was bundled together in just a few test functions, maybe i'll write something on the wiki about isolating tests as much as possible
TST: fix failing data.py tests FOR GOOD
State: closed

closes #4028, #3982


Labels:

Comments:
this needs to be in 0.12 since the context manager i just put in fails with this test in python3. fixing now

it's possible that this could fix #3982 but i'm not sure. connection reset seems to only happen on python 3 boxes

hm in `can_connect` there's a call to urlopen that isn't closed...will add a context manager. shouldn't matter though since it's calling out to a different url, but best not to leave the object open anyway

cc @jtratner

@cpcloud yes, that's absolutely my bad. I actually didn't realize that `urlopen` needed to be closed (and actually when you were commenting on another issue, I realized I hadn't done that).  Should be a really simple fix, no?

yep it's running on travis now

i'm going to `grin` through and see if there are any other lurking file-likes that need a context manager

Sorry about that! Definitely a "A few days ago I learned -- You have to
close urlopen results" :P

On Wed, Jun 26, 2013 at 6:59 PM, Phillip Cloud notifications@github.comwrote:

> yep it's running on travis now
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4047#issuecomment-20086374
> .

well that's all the calls to `urlopen`. still getting intermittent failures about nonexistent data. i'll see what i can do about that

ok well this is passing but i think i'm going to do a couple git rehashes to see if the changes are robust

this also fixes the `test_fred` suite not running on travis because its tests are marked slow and network

this fixes this as well? (this was running full suite locally)

```
======================================================================
FAIL: Throws an exception when DataReader can't get a 200 response from
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/util/testing.py", line 851, in wrapper
    return t(*args, **kwargs)
  File "/mnt/home/jreback/pandas/pandas/io/tests/test_fred.py", line 31, in test_fred
    16004.5)
AssertionError: DATE
2013-01-01    15984.1
Name: GDP, dtype: float64 != 16004.5
```

yes

closing. changed the branch name, opening a new pr
DOC: Minor correction to DataReader data_source options
State: closed

Documentation for DataReader had noted that Fama/French was "ff" for data_source, rather than "famafrench". 

Modified DataReader slightly to accept "ff" anyway.


Labels:

Comments:
this needs a test. caveat: you would need to write the `DataReader` test suite since one does not currently exist. i'm cleaning up `data.py` and i've added a bare bones test suite for `DataReader`, so i can merge this after the cleanup..or can merge this first and add to the new test suite...@jreback?

If you want to merge your test suite for DataReader, I can pull that and update it to include a test for the "ff"/"famafrench" option.

#4047 ... i'll wait till it passes then merge, then u should rebase and then we can merge your pr

also please add a blurb to release notes (they are in `doc/source/release.rst`) and copy it to `doc/source/v0.12.0.txt` maybe under the "Enhancement to existing features" heading(s)

I won't have time until next week to get more into this commit, but this should get us started. 

I think it's good enough for this PR, and more advanced testing (i.e. value checking) should probably be another tracked issue.

I'm going to close this one for now, I'll reopen a PR with the changes after the dust settles on data.py. Cheers!
CLN/DOC: update version numbers
State: closed


Labels:

Comments:
@cpcloud complete side note.....prob should add the `version added` directive wherever possible in the docs...course one would have to go back over the whatsnews/release notes and manually add...maybe add this as a good first pr project?

indeed maybe we should be more strict about PRs using as well, going forward?

well..its only in the main docs....I always forget about it....its a nice thing to do though

ok, not super important so will do if i remember from now on maybe i will add a shell script to run when i write a commit message that tells me about it

why don't you rebase and try to get this in...?

damn i really screwed that up. i thought there would be a bunch of merge conflicts so i did `git rebase -s ours upstream/master`. horrible idea since of course after RTFM i see that that discards the patch from my branch.

git reflog is your friend :)

On Wed, Jun 26, 2013 at 12:14 PM, Phillip Cloud notifications@github.comwrote:

> damn i really screwed that up. i thought there would be a bunch of merge
> conflicts so i did git rebase -s ours upstream/master. horrible idea
> since of course after RTFM i see that that discards the patch from my
> branch.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4038#issuecomment-20060178
> .

@jtratner oh gosh i feel like such a git n00b, thanks!

and ....we're back

looks reasonable
ENH: add rmod methods to frame and series
State: closed


Labels:

Comments:
your sure this is necessary

#3590 fixed these mod issues

as it stands mod follows `series % df == df % series` which is wrong IMO

this pr fixes that

so the broadcasting is wrong?

passes....i'll wait a bit to merge to make sure i didn't forget anything

I tried your fix, didn't realize this was an issue...its essentially not broadcasting..so good to go
Add layout keyword to dataframe.hist()
State: closed

closes #4050. The current version defaults to a very sane default, but sometimes for presentation I prefer them all to be all in a row, or all in a column, or some non-balanced mix of the two.  This keyword gives the flexibility for all of those options.


Labels:

Comments:
is this a 'standard' option name? it seems intuitive, but does mp use this in a similar context (or some other keyword)? just asking, seems logical

fyi..in future, pls create an issue, then reference it in the PR, we can label issues so categorizing is easier..thanks!

Sorry about that, was unaware about the issue preference.  Made issue #4050, added issue to release notes and original commit message.

As for the name, I just copied the name/interface of "layout" from plotting.grouped_hist(...).  I don't know too much about the collision with matplotlib, other than when matplotlib is talking about layouts they are referring to the same type of thing, and it doesn't appear to me that matplotlib actively uses this keyword.  However, I am perfectly happy to change it, I have no strong connections to this name.

I think perhaps the best alternative would be to break this into two keywords: rows/cols?

this looks good
can you hook up Travis?
then recommit?

git commit --amend -C HEAD

Alright, Travis is hooked up.  One particular unrelated test is a bit flakey, but otherwise the tests are all passing: https://travis-ci.org/danbirken/pandas

@danbirken flaky tests should be resolved...can u rebase and push again? thx

There we go.  Still had a transient failure, but after resetting that test, we are fully green.

@danbirken thanks much!
BUG: Removed import nose on top of testing.py script
State: closed

I'm having this error in master branch:

> > > import pandas
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "pandas/**init**.py", line 33, in <module>
> > >     from pandas.util.testing import debug
> > >   File "pandas/util/testing.py", line 17, in <module>
> > >     import nose
> > > ImportError: No module named nose


Labels:

Comments:
this is not necessary; how are you building master?

I did this:
python setup.py build_ext --inplace

then run:
python setup.py develop

Does pip install pandas will also install nose?

no...you need to install nose separately (and cython too)

Okay I might be missing something, but pandas/**init**.py  is importing from pandas.util.testing import debug wouldn't this error out on fresh installs once there's a new release of pandas or nose is part of setup.py requirements?

hmm...i just looked at my installed version and it IS different; apparently this was added recently
so your PR is good! (and we don't test this on travis....nose is automatically installed)

can you hook up travis in any event?

Yeah exactly what I thought and perhaps all developers have nose installed already. Okay let me configure travis I don't now why it's not building since I already hook it on my first commit to pandas.

if you can try the build (w/o) nose installed would be good as well....good catch!

thanks for the catch!
Remove deprecation warning about 'display.height' option.
State: closed

Hi,

See the related issue at https://github.com/pydata/pandas/issues/4026

Best regards,
Damien


Labels:

Comments:
this is ok, but for 0.12 (I know could be done here, but trying to avoid API changes),

also can you hookup travis?

Fine for me for 0.12. I see "Milestone 0.13", close to the pull-request title, was it updated?

"hookup Travis": did you mean that I should activate the Travis hook for my pandas repo ? I sign in. See https://travis-ci.org/garaud/pandas

I'm a "Travis-CI" newbie ;-) I don't know how I can tell to Travis to build this pull-request or this branch, i.e. "display_format". Should I be more patient? I quickly read the doc but I don't find a clearly answer.

0.11.1 was just changed to. 0.12 (as its now a major release
so this will be in next one

look at contributing in the main pandas dir
shows how to hookup up to Travis; u have to hook up
turn it on then push again

lmk if u need help

OK. Looks fine See https://travis-ci.org/garaud/pandas/builds/8501905

Note that when I pushed a few hours ago, I saw that the latest commit of the branch master wasn't passed. It deals with the pull request about color_map.

See you.
Damien

yeh....some intermittent network failures on a couple of unreleated tests

#4391 should silence this, without reintroducing (the useless) height options.
Thanks @garaud for the fix, I thought it best to solve this that way.

Note that 2.7 silences deprecationwarnings by default, which is why I (and I suspect
the other developers) didn't need to scratch this itch ourselves earlier.

might do well for devs/travis to turn this on to get a better feel for latent issues. (cc @cpcloud)

yes i test on 2.7 ... so i don't see these very often and when i do i ignore them

@y-p good idea to turn on `DeprecationWarning` reporting for testing.
BUG: Make secondary_y work properly for bar plots GH3598
State: closed

As far as I'm aware this seems to work properly, but I'm not sure if this was done the previous way for a reason I don't understand.

Fixes: https://github.com/pydata/pandas/issues/3598


Labels:

Comments:
can u add release notes?

Done

@jreback looks ok 2 me, thoughts?

is mark_right a _public_ option?

In a theoretical sense they should be public, in the spirit of this option being similar to "stacked" for bar plots or the same properly "mark_right" for a LinePlot; it is a perfectly reasonable publicly editable properly of a plot.

However, since via standard use of (series|frame).plot(kind='xyz'), you never actually get to interact with the XyzPlot object (and nowhere else in the pandas codebase are these plot objects ever interacted with) I don't think it matters all that much, and since there already is a previous convention I believe sticking with that is reasonable.

that's all fine
didn't know if they belong in the doc string

I just added it to the docstring of plot_frame() [along with fixing minor mistake in secondary_y].  It is getting a little verbose, but more documentation is probably better than less in general.  Up to you.

that looks good thxs

thanks!
BUG: allow series to use gcf-style figures
State: closed

cc @fonnesbeck

here's the original issue: https://github.com/pydata/pandas/commit/51cc9d9d21554701f70ef24392bd78a3da189335#commitcomment-3494266


Labels:

Comments:
@fonnesbeck can u try this out? @jreback any thoughts?

@jreback couple of things here: possible issue is the new-ish API change that if `figure` and `by` are given an error will be raised saying that you cannot pass `figure` and `by` since the `figure` will be discarded with grouped hist plots; i feel like this is something @wesm might give a big :-1: to...personally i would prefer to get an error telling me my passed figure won't be used. OTOH i could certainly make `_subplots` take a figure argument.

@jreback can we get this in to 0.12?

is what `Series.plot` does (take from the current figure)?

yep

looks reasonable

actually what i said was slightly wrong: nothing is done with the figure unless the current axis has ticks on the right yaxis.

this looks fine to me

will merge after failing test fix
FIX groupby name without multiindex GH4014
State: closed

Fixes #4014.


Labels:

Comments:
Just checked the fix, thanks a lot. I should get into some pandas dev, too. What's the best way to get started?

FYI, the fix is for GH4014, in the comment on 1418 in pandas/tests/test_groupby.py you specified GH4104 (as in the heading here). Just for future referencing...

Hehe thanks! (I'll fix it thanks, apparently I can't type.)

Perhaps a good way to get started is to check out the ["Good as first PR" tag](https://github.com/pydata/pandas/issues?direction=desc&labels=Good+as+first+PR&page=1&sort=updated&state=open) :)
CLN: Added *.dta to the .gitignore. Fixed typo in comment. Removed import.
State: closed

CLN: Removed import after flake8 said it wasn't needed.


Labels:

Comments:
3 trivialities I noticed while working on offsets.py.

I think this just failed trivially...

`git --amend -C HEAD`

then force push again

Seems to be failing due to a problem fetching data from yahoo. Not sure what to do about that. Run it again?

go and push force push again...should be ok now

thanks!
Basic JSON normalization/flattening
State: closed

closes #1067. 

This isn't a fast function but has quite a bit of functionality. Take a look at the unit tests. Comments on the API welcome and this could use a few iterations. Various ad hoc things like concatenating "key paths" to metadata


Labels:

Comments:
@nehalecky can i get some feedback?

@wesm, hey! Yes, I will roll this out to a direct use case I have and let you know. I was hoping to do so today, so I'll get back soon. ty.

_Really_ busy yesterday, I am going to try and get to this today. :)

@nehalecky did you get a chance to try out?

we adding this to 0.12?

Hey @jreback. I am finishing one last deadline (due today). My _very_ next project will benefit much from this functionality, so I will be testing it out. I have a deadline for this early next week, and will certainly have reviewed this by then. If that works, great and either way I will let you know my thoughts. :) THANKS!

maybe push to 0.13? @wesm?

yes let's push to 0.13 so it can see some work

**Warning**: _cute_.

``` python
def jqj(data,rule):
     from sh import jq
     from sh import ErrorReturnCode
     import json   
     jq = jq.bake('-M')
     try:
         return json.loads(jq(rule,_in=json.dumps(data)).stdout)
     except ErrorReturnCode as e:
         raise RuntimeError("Jq exited with an error\n%s" % str(e))
     data = dict(dwarves={'names':['dancer','prancer','grinch']},tale='rambo II')
     rule = '[{name: .dwarves.names[],tale}]'
     jqj(data,rule)

[{u'name': u'dancer', u'tale': u'rambo II'},
 {u'name': u'prancer', u'tale': u'rambo II'},
 {u'name': u'grinch', u'tale': u'rambo II'}]
```

[jq](http://stedolan.github.io/jq/manual/)

That's pretty nifty 

@nehalecky @cpcloud @wesm try to merge to 0.13?

@nehalecky have a look?

@wesm, merge this?

Sure, it's good to go by me. 

ok..let me rebase....and can do

Sorry all, life + work is all really busy. In startup's requirements change quickly! Regarding the PR, glad to see it's been merged and included in `0.13`. I'm eager to work with it, and when I do, I will provide feedback.

Thank for all your efforts!

@nehalecky gr8...try it out....it may not be super fast....(optimization PR's always welcome)...b.ut should be good on moderately sized sets

xref: http://stackoverflow.com/questions/21494030/create-a-pandas-dataframe-from-deeply-nested-json

@y-p  thanks for that snippet above

I've looked at this again recently for other reasons and found plenty of existing,
dedicated languages for doing this. Basically, XML has XSLT for doing transformations
and lots of people came up with the idea of doing the same for JSON.

If you're interested google: jaql, jsonpath, jsoniq or jsont.

Appreciate your care for attribution.