BrandonFors/pandas-github-issues huggingface dataset:
Use psutil to set process affinity (test_perf.py) + minor pylint-friendliness for ._ix
State: closed

please test test_perf.py -- I have rebased against master and had to resolve conflicts

Cheers,


Labels:

Comments:
can you hook up to travis?

hm -- I thought that pull requests should also be travis'ed for pandas regardless where they are coming from... meanwhile enabled travis for neurodebian/pandas

I think you need to set the github hook up (then login to travis), then recommit it

see contributing.md !

perfect....merge?

and fyi....built successfuly on the sparc (very slow......!)

Would have been fine to abolish affinity completely in favor of psutil.
TST: raise an error json serialization of floats that cannot be accurate represented
State: closed

related to #4042

```
In [1]: DataFrame([[1e-16,'foo',1e-8]],columns=list('ABC')).to_json()

ValueError: ujson currently cannot accurately format float data less
than 1e-15. A work-around is to multiply the data by
a large positive factor and divide on deseriliazation
```


Labels:

Comments:
seems covered by #4299
BUG: fix data.py regression
State: closed

get_data_famafrench was not returning any data because of an incorrect slice 
and read + splitlines != readlines for zipfiles


Labels:

Comments:
@jreback look ok?

this bug was actually caught by a doc build

interesting...guess it could be worth it to add in doc builds to travis?

Yep the docs function nicely as smoke tests. 

loook ok to me
BLD/TST/CI: add script to show skipped tests at the end of a build
State: closed

closes #4261.


Labels:

Comments:
also shows skips grouped by class where applicable, for easier reading

nice!

now of course need a sscript to go find `nose.SkipTest` that don't have any explanation!

complete aside.....I think it might be nice to show off the travis build scripts to the travis guys...maybe they will put up as some examples...(e.g. with the wheel stuff and version printing and such)

this shows the skips _and_ their message, and the ones that don't have a message have empty space next to the colon so no need for another script for that!

agree about the travis showcase...wonder if @y-p might want to do that?

i also need to make sure this isn't skipping function tests

Not me, I prefer the lack of syndication. But feel free to get in touch with them,
they're really nice guys and their blog has picked up in recent months, they'll
probably be glad to point out the use of wheel files to reduce build time. 

I think other pydata projects are hurting there, and making others aware that this is both 
simple and effective would be a good service to the community. 

Should give mad kudos to the pypa guys and the people behind the new PEPs who are
giving python packaging a much needed kick in the backside on several fronts.

@jreback i think my exception parsing script would tell you where there are empty `SkipTest` exceptions. you can pass a regex to search for a `raise` statement argument

@jreback u ok with this? it's ready to go

@cpcloud - this is great, I was mulling over the same thing, glad you added
it!  Would it be possible to change this into a nose plugin?  Not sure how
much work that would be, but it'd be great to use this for other projects
too. Might also make it less fragile (e.g., if location of tempfile
changes, etc)

On Tue, Jul 16, 2013 at 4:22 PM, Phillip Cloud notifications@github.comwrote:

> @jreback https://github.com/jreback u ok with this? it's ready to go
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4262#issuecomment-21070704
> .

looks ok to me

@jtratner you're welcome to do that if you want! I don't really have time to do it ATM, could revisit later if you haven't done it by then.

nice.

@y-p thanks!
TST/BLD: add proper skips for bad bs4 version
State: closed

Make sure things are being properly skipped because of a bad bs4 version.

Also adds a 4.2.0 known bad version of bs4 to 26 travis build so that we can verify that tests are being skipped when the version is bad.


Labels:

Comments:
O_o travis DL speeds are double digits...so slow...

@jreback this ok?

just adding bs4 and html5lib to travis python26 build so that we can verify 4.2.0 tests are in fact being skipped. also adds skip messages for things not found

yes...this looks ok

i see 0.12 on the horizon :sunrise:
ENH: Dataframe isin2
State: closed

fixes #4211, an alternative (on top of) #4237

DataFrame isin method:

```
In [11]: df = pd.DataFrame([['a', 'a', 'c'], ['b', 'e', 'a'], ['c', 'a', 'f']], columns=['A', 'A', 'B'])

In [12]: df
Out[12]:
   A  A  B
0  a  a  c
1  b  e  a
2  c  a  f

In [13]: df.isin(['a'])
Out[13]:
       A      A      B
0   True   True  False
1  False  False   True
2  False   True  False

In [14]: df.isin({'A': ['a']})
Out[14]:
       A      A      B
0   True   True  False
1  False  False  False
2  False   True  False

In [15]: df.isin({0: ['a']}, iloc=True)
Out[15]:
       A      A      B
0   True  False  False
1  False  False  False
2  False  False  False
```

cc @TomAugspurger 


Labels:

Comments:
Looks good!   The `for i, ind in enumerate(self.columns)` is what keeps the columns in order when using a dict right?

Yes, we grab out the columns (as a DataFrame!) by integer location and when they're concat-ed back they stay in order :) , slightly changed notation in second commit (to be less ambiguous / ix like):

```
concat((self.iloc[:, [i]].isin(values[i]) for i, col in enumerate(self.columns)), axis=1)  # iloc
concat((self.iloc[:, [i]].isin(values[col]) for i, col in enumerate(self.columns)), axis=1) # label
```

yep... @hayd method is also correct

a selector like `.loc/[]` will get you more than one column (in the order you specify) if there are dups (which is what you generally want), but in this case you are just doing a transform so need to address by `.iloc`

Cool.  Thanks for all the hand-holding through this.

@jreback 0.13 ?

if u want to shove in 0.12 ok 
trying to stop adding but seems like a nice feature
@cpcloud ?

i really like this, but i think we should maybe just focus on getting the rest of the bugs fixed for 0.12 and add this very soon after 0.12 is released.

OTOH i haven't had a chance to play with this so if @jreback and @hayd think it's ok and won't delay the release at all, it's ok by me

here's the issue

once you release it the API shouldn't change
if its good then that is fine

you could mark it experimental if u want

I think the api is pretty good (and will be stable)... but I'm biased!

merged into 0.13 :)

but oops should have moved the release note...

(pushed release note fix for that direct to master.)
CLN: Remove unused and cleanup imports throughout.
State: closed

Cleans up imports throughout pandas, removing unused and reordering to
better match PEP8 where reasonable. Thanks to PyCharm for making this much
easier. Doesn't change any .pyx or .pyd files. (inspired by import issue
in #4249)


Labels:

Comments:
this is nice....but let's wait till 0.13 (can be first one)...I know prob hard to generate..buts a big change, though needed

@jreback that's fine (and what I expected) - just wanted to remember to do this after the issue that @michaelaye mentioned. Looking forward to 0.13 to merge some of these pending PRs :)

maybe make this on of the first right after 0.13 starts as it touches almost everything....just rebase and speak up after @wesm releases

@jreback maybe it would be better to let other pending PRs happen first (not sure how many there are), so that others don't have to handle rebase conflicts with this relatively trivial-but-useful change? Then after all the pending 0.13 PRs finish up, can redo this with all those changes included.

I don't think there is actually that much pending....its not a big deal really; most of your changes will be fast-forwarded anyhow with little conflict

@jreback got it.

doesn't make sense to do this until we've put in many of the major changes for 0.13. Otherwise just creates headaches for little gain...
BUG: remove six import
State: closed

closes #4249


Labels:

Comments:
@y-p or @jreback can one of u merge when this passes if possible? i won't have access to a computer for the rest of the night :scream: 

Forgot to say thanks! 

sorry.....was out last night.....never really sure why `six` was needed...

i must've lost my mind for some reason i thought it was a dependency. turns out just later versions of dateutil install it
DOC: Fixed erroneous method calls in 10min.rst. 'pd.' was missing - Issue #4244
State: closed

Added 'pd.' to the examples in 10min.rst (10 Minutes to Pandas).


Labels:

Comments:
if you would also fix the Excel example to use the new format (`pd.read_excel`) would be appreciated as well

This is probably _everywhere_ (and worth fixing at some point) :)

why doesn't this show up in a doc build?

there is a 

from pandas import \* at the top of each doc file

then why is this an issue? i don't see these errors in the docs

it's only an issue if you copy and paste from the docs
technically then it won't work 

that's a legit concern but the issue says there are errors, i can't find them. @jreback do u get the errors?

Errors show up when you evaluate the code. Sorry if didn't explain that
well enough.
On Jul 15, 2013 5:35 PM, "Phillip Cloud" notifications@github.com wrote:

> that's a legit concern but the issue says there are errors, i can't find
> them. @jreback https://github.com/jreback do u get the errors?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4245#issuecomment-21011673
> .

ok. just FYI any time you try to evaluate an object that hasn't been defined you'll get a `NameError`.

I've updated the issue. The error occurs when evaluating sample code. The pull request includes updated sample code. https://github.com/pydata/pandas/issues/4244

closed via 527db3818117c2eaaa8e85a6292f94b37a03feb5
BLD: use wheels and newer scipy bc of py33 scipy issue
State: closed

closes #4239 #4194

using scipy 0.10 for 2.7 builds and 0.12 for 3.2 and 3.3 builds


Labels:

Comments:
@jreback also seems like it might be running just the slightest bit faster as well :smiley: 

running in around 10-11 minutes
TST: remove fudgy ujson test #4223
State: closed

Removed ambiguous ujson test, see #4223, #4186.


Labels:

Comments:
Hmm looks like there is an ongoing build issue (for numpy `ImportError: libatlas.so.3gf: cannot open shared object file: No such file or directory`). 

Let me know if I need to push again later to re-run travis...

you should rebase against master to update 

going to merge this in any event

thanks!

woops, forgot to rebase before pushing, my bad. thanks @jreback!
BUG: Boolean indexing on an empty series loses index names
State: closed

#4235


Labels:

Comments:
can u put tests in to check the resulting dtype
this actually may be an API change

Added a check on dtype.

@jreback this looks reasonable to me. Think it's a bug fix rather than API change.

@mtkni Do you mind adding a Note in the release notes (`docs/source/release.rst`) mentioning 4235 (you'll probably have to rebase off current master first) ?

I think this might be fixed 
as am pretty sure that in propogate names (but maybe not on empty)

The first test is already working in master, names propagate (however can't hurt to add test?).

This one isn't though:

```
        result = lib.infer_dtype(np.array([], dtype='i4'))
-        self.assertEqual(result, 'empty')
+        self.assertEqual(result, 'integer')
```

I get (in master):

```
In [10]: pd.lib.infer_dtype(np.array([], dtype='i4'))
Out[10]: 'empty'
```

maybe @mtkni has some thoughts.

actually the inference of 'empty' I don't think is actually used anywhere. I would ALWAYS return the type (and mixed) if you fall thru the loop, empty is definitly weird

if n == 0, then just return an inferred dtype (you have to map them, prob via the kind)

e.g.

```
if dtype.kind == 'i':
   return 'integer'
elif dtype.kind == 'f':
   return 'floating'
......
```

@jreback well apparently not, since moving just those 3 lines seems to do the trick. merge then?

@hayd ok then!

This passes on a rebase https://travis-ci.org/hayd/pandas/builds/10534501

@jreback  Will wait on release note? Or I could pop it in to my branch.

@hayd whatever works for you!

ok, adding commit to my rebased branch and will close these.
BUG: initialize DatetimeIndex with array of strings (#4229)
State: closed

Possibly a fix for #4229.

The rationale is that an array of strings should be handled in the same way as a list of strings (after applying `np.asarray(list)`), and I supposed that the list was handling the timezone correctly.
## 

More in detail:
- handling of the list of strings: https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L196 => returning `data`
- handling of an array of strings: https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L211 =>  returning directly `subarr`

Because in the case of an array `subarr` is set as a DatetimeIndex (instead of as `.values` attribute of it => array of datetime64), the array is not localized to UTC (https://github.com/pydata/pandas/blob/master/pandas/tseries/index.py#L249) but assumed to be UTC. So, in fact, the `tz` keyword is ignored.
## 

Something else: it is not completely clear what the `tz` keyword is supposed to do (it is not documented in the DatetimeIndex docstring). But I assumed it is to say that the given values to DatetimeIndex are in that timezone (and so the behaviour with a list is correct). However, it is also a little bit strange that eg `to_datetime` has not such of a keyword then.
## 

PS: the PR is not ready (I want to add some tests, release notes, and there is too much whitespace in the commit), but I just wanted to submit it already to see if this makes sense.


Labels:

Comments:
change looks good
FYI usually I write the test first to make sure that a) test breaks as expected and b) code change fixes the problem 

@jreback I know that is the better order to do things, but I used the issue as a test

Regarding to adding the test, it seems like it could go in either `test_timeseries.py` (together with all other tests on DatetimeIndex construction tests) or `test_timezones.py` (together with all different tests dealing with timezones). Any preference to where I put it?

your choice (just put the issue number as a comment)

I try to group the tests with like tests - emphasis on try :)

@jreback OK, I added test + release note.

@jorisvandenbossche can u rebase? thanks

@cpcloud Done I think

thanks @jorisvandenbossche 

Thank you very much @jorisvandenbossche and @jreback
BUG: Index removes name when sliced
State: closed

closes #4226


Labels:

Comments:
can u make sure no perf hit with a test_perf ?

running now

whenever I touch indexing I make sure to do that
seemingly innocuous changes sometimes have impact ( that said there IS some variation in perf stats so even a 15% reduction may not mean much and could change in retest)

good to know. seems like only a small constant will be added here since getslice just passes to getitem, but i'll be sure to do that whenever i touch indexing

i haven't really had time to play with `test_perf.sh` but is there a way to look at all `test_perf.sh` runs one has done? e.g., i could run overnight 1000 times and look at variability to get a reasonable estimate 

well this is annoying

```
index_datetime_union                         |  51.5563 |  10.3513 |   4.9806 |
index_datetime_intersection                  |  51.4587 |  10.3063 |   4.9929 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

twice in a row it's been around ~5

@jreback what to do here...`__getslice__` is defined by `ndarray` in C code so...gonna be a little hard to get around this.

array_finalize should set name (I thought it did)?
this is called on array creation after a numpy func

hm ok my fix looks unecessary, there's something weird in `DatetimeIndex` since `__array_finalize__` is not being called

oh nvm i c. no name attr there nice, easy fix

perf back to fluctuating around 1.0! yay!

@jreback any more thoughts here?

so this actuallly ends up being a trivial change....ok...merge away
CLN/TST: clean up and raise on bs4 version and no tables
State: closed

closes #4214


Labels:

Comments:
@jreback u ok with this?

does 4.2.0 work _sometimes_ eg it will occasioanlly parse things
???

I would maybe steer clear and have it just raise on this version always
it's essentially a known failure and there are options to upgrade/downgrade

but ok either way

trivial to test that way, so i think i'll do that

@jreback shall i merge?

on second thought maybe we should wait for @yarikoptic?

go ahead

he will test against master
BUG: Display the legend when calling plot with kind='density'. GH4216
State: closed

See issue https://github.com/pydata/pandas/issues/4216


Labels:

Comments:
@garaud thanks for the contribution! can you add a test and release notes please. thanks

Thanks to remind me to add a test and release notes. See my two latest commits.

See you,
Damien

you'll also need to rebase on top of `master`, github is warning about merge conflicts, probably just a release notes conflict

Should be good now.

@jreback look ok?

yep

@garaud thanks!
BLD: Use 'git describe' to generate the version string for dev versions
State: closed

closes https://github.com/pydata/pandas/issues/4046

```
λ sudo python ./setup.py develop
...
Installed /home/user1/src/pandas
Processing dependencies for pandas==v0.12.0rc1-48-g9442eac
Searching for numpy==1.7.1
Best match: numpy 1.7.1
Adding numpy 1.7.1 to easy-install.pth file

λ python -c 'import pandas; print pandas.version.version'
v0.12.0rc1-48-g9442eac

```


Labels:

Comments:
To be merged right after the final 0.12 release, in case this breaks something,
learn that fact in master rather then have users discover it when upgrading.

Please mention me if this merge creates issues.
PERF: add ix scalar get benchmark
State: closed


Labels:

Comments:
@y-p question about test_perf. I just added a benchmark in the vb_suite. Which of course didn't exist if I want to compare to a commit in say 0.10.1. How do I make this work?

```
*** Running benchmarks for baseline [8a79d0d]
Running 6 benchmarks for revision 8a79d0d
indexing_frame_get_value_ix
indexing_dataframe_boolean
indexing_dataframe_boolean_rows
indexing_dataframe_boolean_rows_object
indexing_panel_subset
indexing_frame_get_value
```

```
./test_perf.sh -b 8a79d0d3b4d3992a955066ed6085a732eb0041c6 -t 7b2eaa4983162511868ce4ef7e567c97cae89fe3 -r indexing

-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
indexing_dataframe_boolean                   |   8.2850 |  36.8630 |   0.2248 |
indexing_dataframe_boolean_rows_object       |   0.4253 |   0.3800 |   1.1192 |
indexing_panel_subset                        |   0.4750 |   0.4147 |   1.1455 |
indexing_dataframe_boolean_rows              |   0.2220 |   0.1824 |   1.2170 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [7b2eaa4] : Merge pull request #4207 from jreback/indexing_perf
Base   [8a79d0d] : RLS: Version 0.10.1

```

```
git checkout v0.10.1
git checkout upstream/master vb_suite # bring in vb_suite from the future
```

then run test_perf and save the results to a pickled df. The vbenches are reasonably decoupled from the rest of the pandas codebase.

I added the option to compare two pickle files a while back, I think.

@y-p that worked...thanks

I didn't see the compare option...but for this case didn't matter ayhow....
BUG: (GH4192) Fixed buglet in the broadcasting logic in Series.where
State: closed

Fixed #4192


Labels:

Comments:
```
======================================================================
FAIL: test_where (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.12.0rc1-py2.7-linux-x86_64.egg/pandas/tests/test_series.py", line 1139, in test_where
    self.assertRaises(ValueError, s.__setitem__, mask, ([0]*5,))
AssertionError: ValueError not raised
```

fyi...travis runs on 32-bit (not that this caused this error)

in the future, you can use `vagrantup.com` to setup a machine in any configuration you want...

everything passes for me now (on 32/64) bit

look good, merge?

I'll just take out the print statements.....

so was a one-line change! hah

thank you sir!

Oops, sorry about the print statements. I had to hurry quite a bit due to some other work and didn't realize that I left some debug stuff in there. I'll take greater care next time.

np......
FIX disallow MultiIndex in Series constructor GH4187
State: closed

fixes #4187 (at least the segfault part).

Constructing Series with a MultiIndex raises NotImplementedError, the option (to implement this) is to return `.to_series()` ?


Labels:

Comments:
anyone have an opinion on this (whether `pd.Series(a_multiindex)` should return `a_multiindex.to_series()`).

If anyone says "yes", I'll update.

@hayd do it internally (e.g. don't use **new**) (you can just do a_multiindex.to_series() in **init**)

I hate `__new__` so much...

so I can just more this same couple of lines into the init?

sorry....leave it there for now....that's ok (when refactor series will fix it)

@jreback so should I change it to m..to_series() or shall we merge as NotImplemented (should fix this before 0.12, noone wants a segfault...).

just leave as not implemented
TST: fix ujson tests failures on 32-bit
State: closed

As pointed out by @lodagro [here](https://github.com/pydata/pandas/commit/a8d15bd4c6dae2c65ed4aa4b2f76c029b6358113#commitcomment-3521549) and [here](https://github.com/pydata/pandas/commit/a8d15bd4c6dae2c65ed4aa4b2f76c029b6358113#commitcomment-3521581) some of the ujson tests fail on 32-bit due to precision errors.

The changes below amend the tests so they pass.


Labels:

Comments:
looks good...merge?

Thanks @jreback 

32b failures resolved, will need to keep this in mind in case of upgrading to a new ujson version (as long as tests are not changed there)