BrandonFors/pandas-github-issues huggingface dataset:
BUG: allow single element bool queries
State: closed

closes #6148


Labels:

Comments:
I see a tiny yellow dot :cry: what happened to the big yellow/red/green banners? I liked those.

where's the dot?

It was next to the commit hash, now it's gone ....

Not totally sure how it's supposed to work.

I see them now...I think they just take time to show up

Ok I'm just watching travis
BUG/API: df['col'] = value and df.loc[:,'col'] = value are now completely equivalent
State: closed

previously the .loc would not necessarily coerce the dtype of the resultant series
closes #6149


Labels:

Comments:
this refues to pass py2.6 and I cannot duplicate this error with the exact same virtual env....if
i can't fix it by tomorrow...will move to 0.14

this exposed an odd bug, which is fairly tricky to fix...so moving to 0.14
BUG: allow lex string comparisons
State: closed

closes #6155


Labels:

Comments:
gr8!

thanks!

np trying to do more ... got  the single element bool coming ur way in 5...4...3...

do you know how the new github inteface shows if travis has passed?

i was getting a yellow box for a while while it was running

but u merged before i had a change to see the green here (was green by email)

oh...mine not updating...I actually looked on travis

FYI...I think travis runs each one under your id AND pydata.....this doesn't turn green until the pydata one does

hm ok good 2 know thx
TST: rec arrays don't support datetimes in creation on certain platforms, related (GH6140)
State: closed

related #6140, #6142


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
fixes the most recent windows issue too #6152 which I causes

luv http://scatterci.github.io/pydata/pandas

:100: @y-p 

Thanks, I do manage to do some good ocassionaly. Very pleased it's useful to you.
PERF: perf regression index construction from series (GH6150)
State: closed

closes #6150


Labels:

Comments:
@hayd easy enough....

back to the old perfs

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
index_from_series_ctor                       |   0.0233 |   7.7777 |   0.0030 |
dtindex_from_series_ctor                     |   0.0143 |   4.1296 |   0.0035 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [20e3fcc] : PERF: perf regression index construction from seris (GH6150)
Base   [6d889d4] : BLD/TST: one more skip_errno for @network

```

Awesome!
ENH: Add unique and nunique to GroupBy whitelist
State: closed

Series.unique and Series.nunique are useful methods for querying the
unique elements (or number of unique elements) in each group when used
in conjunction with DataFrame.GroupBy.  This commit addresses
closes pydata/pandas#6146.


Labels:

Comments:
pls add in `tests/test_groupby.py` as well (for the whitelist tests)

Ok, I added the tests as well.  Let me know if everything looks ok.

thanks for this!
BUG: Bug in DataFrame construction with recarray and non-ns datetime dtype (GH6140)
State: closed

closes #6140


Labels:

Comments:
```
===============================================================================
ERROR: test_from_records_with_datetimes (pandas.tests.test_frame.TestDataFrame)
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/pandas-py3_x-wheezy-sparc/build/venv/lib/python3.2/site-packages/pandas-0.13.0_420_ga67c3cd-py3.2-linux-sparc64.egg/pandas/tests/test_frame.py", line 3970, in test_from_records_with_datetimes
    recarray = np.core.records.fromarrays(arrdata, dtype=dtypes)
  File "/usr/lib/python3/dist-packages/numpy/core/records.py", line 566, in fromarrays
    _array[_names[i]] = arrayList[i]
ValueError: Must be a datetime.date or datetime.datetime object
```

Best thing I ever did.

funny thing when I was doing this was thinking that their is a bug that caused all of the other sparc issues

namely that for some reason dtype comparisons don't work on date time types (but u can compare then as strings) - weird because it works on Linux in general

but getting off topic !

and I put in that exact comparison

ok 1.5 hrs to compile / test

2.5h average delay +1.5 run actually. But at least you know there's a problem the same day
you commited, not 3 weeks after.
[DOC] install.rst: numexpr and anaconda warnings
State: closed

- Anaconda currently provides `pandas` v0.13, `numpy` 1.7.1 and `numexpr` 2.2.2 as the latest versions (#6056).
- There seem to exist rare but evil lurking bugs when using `numexpr` <= 2.2.2 and `numpy` >= 1.8.0 (#6119).


Labels:

Comments:
can you post output of  `ci/print_versions.py` 

this does not occur on any of our tests machines...so must be machine specific

``` sh
INSTALLED VERSIONS
------------------
Python: 2.7.6.final.0
OS: Linux
Release: 3.0.38-0.5-default
Processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

pandas: 0.13.0
Cython: 0.20
Numpy: 1.8.0
Scipy: 0.13.2
statsmodels: 0.5.0
    patsy: 0.2.1
scikits.timeseries: Not installed
dateutil: 2.2
pytz: 2013.9
bottleneck: 0.7.0
PyTables: 3.0.0
    numexpr: 2.2.2
matplotlib: 1.3.1
openpyxl: 1.6.2
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: 0.4.9
sqlalchemy: 0.9.1
lxml: 3.3.0
bs4: 4.2.1
html5lib: 1.0b3
bigquery: Not installed
apiclient: Not installed
```

the only different I can tell with your setup and the test machins the linux kernel (travis uses 2.6)..

I think has to be an odd view thing. Since this doesn't seem to affect anyone else, going to hold off on a doc change
(and 2.3 is really not out for numexpr in any event; its still in dev AFAICT)
PERF: fix JSON performance regression from 0.12 (GH5765)
State: closed

This fixes the main JSON performance regression  in v0.13 (closes #5765). The main bottleneck was the use of intermediate NumPy scalars (from PR #4498). I introduced code to avoid the use of NumPy scalars. Also:
- checks current locale so locale fudging is only done if necessary
- added numpy 1.6 preprocessor switches as it still requires the use of scalars due to its weird datetime handling
- reorgranised some of the if/else checks in `objToJSON.c` based on likelihood.
- added some JSON benchmarks to vbench to try and avoid future regressions.

0.12

``` python
In [1]: import pandas as pd, numpy as np

In [2]: df = pd.DataFrame(np.random.rand(100000,10))

In [3]: %timeit df.to_json(orient='split')
10 loops, best of 3: 119 ms per loop

In [4]: pd.__version__, np.__version__
Out[4]: ('0.12.0', '1.8.0')
```

This PR

``` python
In [1]: import pandas as pd, numpy as np

In [2]: df = pd.DataFrame(np.random.rand(100000,10))

In [3]: %timeit df.to_json(orient='split')
10 loops, best of 3: 119 ms per loop

In [4]: pd.__version__, np.__version__
Out[4]: ('0.13.0-406-ga18e5e6', '1.8.0')
```

While this solves the main performance issue, using vbench this PR still appears to be a bit slower than v0.12 (~ 2 - 5%) but it's unclear where the slowdown is coming from.

@jreback do you still have a windows box / vm that you could test on?


Labels:

Comments:
```

-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
packers_write_json                           |  23.3610 |  42.2543 |   0.5529 |
packers_write_json_date_index                |  31.3267 |  45.5254 |   0.6881 |
packers_read_json                            |  39.0206 |  39.3570 |   0.9915 |
packers_read_json_date_index                 |  39.2287 |  39.2920 |   0.9984 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [0975572] : Merge branch 'json-0.13-slowdown' of https://github.com/Komnomnomnom/pandas into Komnomnomnom-json-0.13-slowdown
Base   [464c1f9] : Add Scatter-CI link to README.md
```

looks good....tested fine on windows , fyi you can watch this: http://scatterci.github.io/ScatterCI-Pandas/

for all builds (linux/windows/sparc builds too)

Awesome! That looks pretty sweet, thanks.
DOC: clarify that ExcelFile is not deprecated but only moved (#5435)
State: closed

closes #5435. Clarified that ExcelFile is not deprecated but just moved. I don't use Excel myself, so can somebody check if it makes sense what I wrote? (@

For now, I didn't add ExcelFile to api.rst (but ExcelFile.parse is already in there, so it is already in some way mentioned), as I am not sure if all it's methods (`mro`?) should be public (this has possibly to be cleaned up first)


Labels:

Comments:
I think this makes sense, atm i like to read in Excel (if I really have to!) as a dict with sheet names:

```
xl = ExcelFile('foo.xls')
{sheet: xl.parse(sheet) for sheet in xl.sheet_names}
```

Note sure there is a way to do this without ExcelFile.

@jorisvandenbossche looks good to me ...bombs away
ENH get_dummies str method
State: closed

fixes #3695.

```
In [8]: s = pd.Series(['a|b', 'a|c', np.nan], list('ABC'))

In [9]: s.str.get_dummies('|')
Out[9]: 
   a  b  c
A  1  1  0
B  1  0  1
C  0  0  0
```

_iterates over each tag filling in with str.contains._

Edit: This works pretty fast with "few" tags (since I'm iterating over each tag), which I think is the main usecase, but I'm sure some more perf can be eeked out:

```
In [6]: %timeit genres.str.get_dummies('|')
10 loops, best of 3: 20.9 ms per loop

In [7]: %timeit genres.str.split('|').apply(lambda x: pd.Series(1., x)).fillna(0)
1 loops, best of 3: 623 ms per loop
```


Labels:

Comments:
atm this doesn't work with floats (other than NaN), I have a fix makes NaN rows go to 0... this may be preferable? In fact, it would be more in line with get_dummies itself...

@jreback This could actually sneak into 0.13.1 ??

sure....

pls dd release note / v0.13.1 (if you want) and to API docs 

@jreback Added in release and some docs.

Also, fixed for floats/ints, so you can do:

```
In [7]: s = pd.Series(['a|b', 'a|c', 1], list('ABC'))

In [8]: s.str.get_dummies('|')
Out[8]: 
   1  a  b  c
A  0  1  1  0
B  0  1  0  1
C  1  0  0  0
```

tested for but purposely not in docs, as I don't think it's what people should be doing! ...For one thing it won't play nice with 3 (int) and 3. (float) in an object Series.

Ahhhhhhh, I'm confused just retested the timings is now slower. Confusing (since it was just as fast when I pushed).

meh. timings seem eratic. It's still 40ms vs 500, so it's pretty good. I just thought I had this down to 20 (for Wes' movie dataset).

Ok, well refactored it to be nicer, and perhaps faster...

Definitely erratic though:

```
In [2]: s = pd.read_pickle('foo')

In [3]: %timeit -n 200 s.str.get_dummies('|')
200 loops, best of 3: 26.8 ms per loop

In [4]: s = pd.read_pickle('foo')

In [5]: %timeit -n 200 s.str.get_dummies('|')
200 loops, best of 3: 16.5 ms per loop

In [6]: s = pd.read_pickle('foo')

In [7]: %timeit -n 200 s.str.get_dummies('|')
200 loops, best of 3: 16.6 ms per loop
```

and does this go in the API docs automagically? or need to define in api.rst?

@jreback Ah, good point. I will add it to the api rst.

Will push to master assuming the function ref is correct.

gr8 thanks! 

seems your tests are breaking windows: http://scatterci.github.io/ScatterCI-Pandas/

[pushed](https://github.com/pydata/pandas/commit/f0d2c249178b9dea0b7ba049ba10af16915f4f36) (along with your fix), ooops about Windows. Sorry! Thanks for the fix!

(Edit: looks fixed in scatter)
ENH: allow legend='reverse' in df.plot() GH6014
State: closed

![reverse](https://f.cloud.github.com/assets/1820866/2007133/009ee032-8732-11e3-920c-7bb53c754ed7.png)

works for line plots, bar plots, and kde plots. scatter plots don't seem to have a legend.

As usual, the plotting code is a tangled mess, I have no idea if this breaks something,
testing seems like a herculean task and I'm scared to death that by being brave and merging
this, I'm setting myself up to touch the plotting code again in the future, at least once. quite
the deterrent.
#6014


Labels:

Comments:
I'm sure something, somewhere will break because someone dared to
touch the plotting module.
CLN: Eliminate handful of test docstrings, use nose -v on travis
State: closed

I'm being forced to do unspeakable thing in aid of pandas and these docstring 
outliers are getting in the way.


Labels:

Comments:
Travis builds every PR twice. Once in the submitter repo and once in the submitee repo. Weird.
I guess they don't mind 100s/1000s of unnecessary builds every day.

cc @travis-ci
TST: add tests for old version of numexpr
State: closed

closes #5857
closes #5855
closes #5845


Labels:

Comments:
think you need to add numexpr < 2.0 to py2.6 build?

yep i didn't forget ... i'm doing it now :)

look ok to me

cool i just want to move `skip_if_no_ne` to testing ... DRY and all ... and it'll give a more informative msg on skip

cool

hm getting a timeout from the redhat server ... rehashed ... let me see if that passes

nvm

@jreback stepping out for a bit can u merge if green? thx

sure

Ok I'll merge. @jreback any last words?

sorry stepped out too

bombS away

Crap I just undid @y-p's nosetests v pr I'm on my phone right now can't get to it ... Can do it later tonight or if someone wants to fix thats cool too

I'll fix it.

Thanks. Sorry about that. Don't mean to create moar werk :/

we all clobber at one point or another. np.
BUG: in HTMLFormatter._write_header(), str() fails on column names in unicode
State: closed

Issue #6098


Labels:

Comments:
Sorry, didn't see this before fixing myself in  #6112

Note that we use the GH____ convention for different reasons in commit subjects. when you want 
to cross-reference an issue, you should use the #___ notation, that way GH will pop up an item in 
the linked issue.
BUG: array_equivalent was not passing its own doctest. 
State: closed

The examples in the docstring showed lists being passed to array_equivalent. `array_equivalent` expects `ndarrays` only. (It raises an `AttributeError` when passed a list.)

Also, the use of `pd.isnull` can be removed from `array_equivalent`. My thinking here was muddled. `pd.isnull` is only needed if `array_equivalent` were to accept objects other than ndarrays. `np.array_equal` works as desired for all `ndarrays` except those of float or complex dtype.


Labels:

Comments:
thanks....nice following up!
BUG: not converting scalars properly to M8/m8 on assignment (GH6079)
State: closed

partial fix of #6079


Labels:

Comments:
I don't even bother measuring compile time :)

```
Ran 4657 tests in 2032.428s
```

this was a PITA

@y-p pls check status of the sparc builds maybe in next day or 2 and let me know

that said...I did find some actual bugs (which for some reason show up on sparc but not on other platforms)....

I think because dtype coercions are slightly different (mainly datetime related)

:+1: 

The buildbot builds once every 5 hours btw, because of the long runtime.

about 45min compile, 30min to run tests! sheesh

I'm way too young to give you a lecture about punch cards, but, you know...

before my time too 

hahha
BLD/DOC: report context on warnings and add :okwarning: to ipython_directive
State: closed

I plan to get this upstream soon, unless joris find bugs.

Reports context on warnings just like recently added context for raised exceptions.
adds :okwarning: option to ipython directive to suppress them.

```
reading sources... [  6%] basics                                                                        

>>>-------------------------------------------------------------------------
Warning raised in /home/user1/src/pandas/doc/source/basics.rst at line 734:
----------------------------------------------------------------------------
/home/user1/src/pandas/pandas/core/frame.py:2805: FutureWarning: TimeSeries broadcasting along DataFrame index by default is deprecated. Please use DataFrame.<op> to explicitly broadcast arithmetic operations along the index
  FutureWarning)

<<<-------------------------------------------------------------------------
```

Let's get rid of the noise.


Labels:

Comments:
@y-p This is really fantastic! I tested it and it seems to work perfectly. Up to a completely warning/error free doc build.

Did you already introduce this in the source code of the docs? Hopefully not (I hadn't seen it), because I have a little PR ready.

I was hoping that by giving you tools to painlessly handle it, you will.
Hey, it worked!  ;)
BUG: fix broken BigqueryError reference
State: closed

`BigqueryError` wasn't imported directly, and so has to be referred to via the module.


Labels:

Comments:
was this another cython bug finder?

Yeah, I have a handful of them.

Is there a way to get Travis to try again?

git commit --amand -C HEAD

force push 

there are rebuild buttons as well on the the of the Travis screen

@jreback: thanks, pushing the amended version seems to work. 

thanks for the catch!
DOC/BUG: Fix documentation for `infer_datetime_format` #6073
State: closed

Fix formatting typos and ensure the "foo.csv" ipython processing works.

Fixes #6073


Labels:

Comments:
@jorisvandenbossche @y-p ?

Checking.

Thanks! Looking good

@y-p Ah, didn't see your comment, but I think it's OK

no problem.

That was fast.  Thanks guys.  Two minutes to spare on my 30 minute inflight internet pass!
Pr ipython directive context
State: closed

```
Running Sphinx v1.1.3
loading pickled environment... not yet created
loading intersphinx inventory from http://statsmodels.sourceforge.net/devel/objects.inv...
loading intersphinx inventory from http://docs.python.org/objects.inv...
building [html]: targets for 2 source files that are out of date
updating environment: 2 added, 0 changed, 0 removed
reading sources... [ 50%] dsintro                                                                       

>>>-------------------------------------------------------------------------
Exception in /home/user1/src/pandas/doc/source/dsintro.rst at line 33: # <<<< This is new
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-4-05c9758a9c21> in <module>()
----> 1 1/0

ZeroDivisionError: integer division or modulo by zero
<<<-------------------------------------------------------------------------

```

No more excruciating "guess where" on errors.


Labels:

Comments:
:joy: 

Nice!

Furthermore, if you want to look it up in the html docs (and not in the source code for which you need line numbers), then you can also look at the `<ipython-input-4-05c9758a9c21> in <module>()` which means the error happend at ipython prompt 4 (`In [4]`)
TST: Fix test_eval.py
State: closed

Fixes typos found in #GH 6069.  Does not change the tests to ensure those branches are exercised.


Labels:

Comments:
Looks like u have an extra commit in there

Urf.  You're right.  If I squash 'em, is there a way to associate the new PR with this one or should I close and open a new one?

No no don't do that. Hold on a sec I'll show u how to just get that commit out

Just peel it off like so:

``` sh
git reset HEAD~1
git stash
git reset --hard upstream/master
git stash pop
git add .
git commit -m'TST: fix test_eval typos'
```

alternatively you could probably cherry pick the commit into a new branch, reset hard on upstream/master and then merge the branch u just created back into the one u just reset

something like this

(say you're on a branch called `orig_branch`)

``` sh
git checkout -b tmp
git cherry-pick 42e3a76
git checkout orig_branch
git reset --hard upstream/master
git merge tmp
git branch -D tmp
git push --force
```

Or if you want I can merge this :)

Hmm.  I followed the cherry-pick instructions, but I'm not sure what the effect was; that branch still seems to show two commits, and the result of `git push --force` was `eacc354..1020b51  master -> master` with no reference to the `fix-test_eval` branch.  

now that i think about it i don't think it makes sense to cherry pick here ... the first instructions are what i use when i do this ... you can just mutate the branch rather than having to make a new one

bam!

Thanks for the assistance!  I'm afraid my brain is pre-git, and I can't afford the upgrade.

:) It's okay, my git chops are getting rusty hope cherry picking wasn't cherry bombing instead

Thx
DOC: Clarify that DataFrame string formatters must return strings
State: closed

I am not 100% sure that string formatters must really return _unicode_ strings
(non-unicode strings seem to work, too), but I am leaving that because it's
what is already documented.
related #6052


Labels:
name: Docs

Comments:
Disagree this "fixes" 6052, but it does help! I wonder if there is a good place in the docs to add more about this?

I think probably we should catch bad input and re-raise it with better message... accepting string e.g. `'%0.f'` would also be nice.

ok...will leave open #6052 

@shoyer thanks for the investigation
BUG: Regression in .get(None) from 0.12 (GH5652)
State: closed

closes #5652


Labels:

Comments:
This is just back-compat, right? because the correct thing here is... debatable.

yep, though the user can easily get the 'correct' behavior by just regular indexing, e.g. `meat[None]` will raise

`get` as in `dict.get(NoSuchKey)`. Right.
ENH add show_dimensions display config
State: closed

see http://stackoverflow.com/questions/21293983/disabling-dataframe-dimensions-printing

If you've already done this @DSM054 apologies, just saw your comment... :s


Labels:

Comments:
Next time I'll call dibs.  In any case, I found an error in the test suite while I was doing it, so it wasn't all a waste.

What do you think about the reference in `_repr_html_`?  I switched it too in mine.

anyone want to provide a doc on 'how to configure your display', showing what/how the options interact? (for the main docs)

@dsm054 _repr_html is a good spot... Added!

@jreback definitely needed.

looks good to me, @y-p ?

I was thinking, would it be an option to only have it displayed if your dataframe is truncated? Or does that make it to complicated? (so you would have three options: always, if truncated, never)

@jreback sure, ok.

Should update the FAQ section on customizing display with this.
http://pandas.pydata.org/pandas-docs/dev/faq.html#how-do-i-control-the-way-my-dataframe-is-displayed

@hayd I would move that current section to the main docs (and update it), then provide a link back into the FAQ
(or link back to the main docs ok too......)

we have this 'orphaned' section here: http://pandas.pydata.org/pandas-docs/dev/basics.html#console-output-formatting which should either include the display formatting (more prominently), or added to the FAQ

@hayd can you make those updates to the FAQ (and can fix the full docs at a later time)....though if you could move/link console-formatting would be great

@hayd , we're ramping up to release 0.13.1, are you on this or would you like me to take care of it?

Added to FAQ. If it looks ok, will squish.

@y-p Happy if you can fix the console-formatting docs or we can have as sep issue.. It needs some love.

Cleaned up and merged as 9ab2385.

#3104 is still open for the other stuff.

@jorisvandenbossche , on/off seems good enough.