BrandonFors/pandas-github-issues huggingface dataset:
BUG/VIS: fix Series.hist so that users can create hist subplots without the mpl API
State: closed

closes #4408.
closes #4402.


Labels:

Comments:
~~ahhh nice, `matplotilb` has an image comparison decorator i'll use that~~

it's too specific to `matplotlib`'s testing suite.

@jreback u ok with this? fixes a couple of issues with overlapping subplots when ppl want to use the matlab-like pylab stuff

also passing both color and style when style has a color symbol is not valid anymore, not sure it ever was, since the ts plot tests pass but that's b/c the error is somewhere deep in the mpl stack and it isn't propagated up to pandas

so not an API change since i don't think it ever worked, but pandas won't try to create a plot if those two args are invalid, it will raise instead

ok w me
BUG: Fix an issue with the csv cparser when usecols is used
State: closed

closes #3192

This pull request fixes a problem arising from using the csv parser with the usecols option. In that case, all the dtypes will be converted to object. This code fixes that issue, resulting in the correct dtypes.

Thanks,
Tiago


Labels:

Comments:
This pull request fixes issue #3192  .

that's what I like someone who fixes an issue they see!
- can you put an entry in release notes? (edit doc/source/release.rst)...follow the format
- can you turn on travis (so it builds on various confifgurations), see contributing.md for help

thanks!

looks like you need to rebase on master

```
git checkout master
git branch <this branch> --set-upstream master
git checkout <this branch>
git pull --rebase
```

fix any conflicts (which are most likely in release.rst)

also might want to rebase down to 1 commit

```
git rebase -i <commit before your first one>
```

then `s` or `f`

then

`git push origin <this branch> -f`

see here

https://github.com/pydata/pandas/wiki/Tips-&-Tricks

Should be ok now.

thanks!

This PR created a regression, #5766.
BUG: Fix .iat indexing with a PeriodIndex GH4390
State: closed

closes #4390

Interesting fixed this as well (only affects `get`)

```
In [1]: s = Series(randn(10),date_range('20130101',periods=10))

In [2]: s[0]
Out[2]: 1.1755210838393764

In [3]: s
Out[3]: 
2013-01-01    1.175521
2013-01-02    0.007910
2013-01-03   -1.235595
2013-01-04    0.478245
2013-01-05   -0.140616
2013-01-06   -2.317585
2013-01-07    0.609555
2013-01-08    1.204914
2013-01-09   -0.589893
2013-01-10   -2.017237
Freq: D, dtype: float64

In [4]: s[-1]
Out[4]: -2.0172365222181257

In [5]: s[-5]
Out[5]: -2.3175847263447964
```

This was before this PR returning `None` (while `s[-5]` worked)

```
In [6]: s.get(-5)
```


Labels:

Comments:
@jreback waiting for @jtratner b4 merging this?

yep
ENH: add git names to test_perf.py
State: closed

Let users pass nice git names so that they don't have to manually pass sha1 
hashes to test between commits.


Labels:

Comments:
with this you can do things like `./test_perf.sh -b upstream/master -t HEAD~2` or whatever `git` allows to specify commits

:+1: 

haha...this explains the weird errors I was getting when I tried to do this :+1:
ENH: suppress display.height DeprecationWarning
State: closed

related https://github.com/pydata/pandas/issues/4026?source=cc

The `display.height` option is dperecated. It's value is interogated by
`get_conosole_size()`, whose only caller in the codebase summarily ignores
that bit of the `(width,height)` retval.
Another ugly ricochet from the `repr()` cluster***\* of 0.11. a tar pit if ever I saw one.

rather then reprecate (?) the depecrated option (As in #4027), just silence the deprecation 
warning at the root and quickly look away.


Labels:

Comments:
i like reprecate
BUG: to_json should raise exception for non-unique index / columns (#4359)
State: closed

Issue #4359

to_json will raise ValueErrors where non-unique indices or columns would cause invalid JSON to be produced. Updated docstring and added some tests.


Labels:

Comments:
thanks!

fixing #4377 now

Why don't `to_dict`  can return a list of values for non Unique indexes being the key in dictionary? Came here from issue #22801.
BUG: Fixed passing keep_default_na=False when na_values=None (GH4318)
State: closed

closes #4318


Labels:

Comments:
Excellent. I would suggest changing "would be replaced" to "is matched to" or "is read as" or "is parsed as". Good thing to doc the float and integer behaviour.

done...thanks...will be merged in soon (not today though...waiting on a big PR to be merged first)
TST: Make network tests run twice
State: closed

Might help with issues like #4242

cc @cpcloud


Labels:

Comments:
looks good, @jreback ?

this is just a try it out..right...still don't really know why these tests occasionally fail?

yep the latest from #4357 is very weird...so yeah maybe wait to merge...should really try to get to the bottom of this

its a nice fix ...maybe it will work....up2u

i like this, but i don't think it will fix #4357 (a separate but related issue) ... would certainly help a lot with _consistent_ errors tho so i'm going to merge.

thanks @jtratner!

even tho #4357 is consistent from the OP's end, everyone else is having trouble reproducing it...so fyi by consistent i mean reproducible by at least 2 people ... maybe that's too strict...

that's definitely _not_ saying that there isn't an issue there, there certainly is and it needs fixing

just wanted to clarify my statement above
World Bank I/O docs and minimal improvement
State: closed

- DOC: Simple example of how to use Pandas' World Bank IO functions. 
- ENH: Return a DataFrame full of floats instead of full of strings

RE: https://github.com/pydata/pandas/issues/4354


Labels:

Comments:
@jreback will the `astype()` call work here? 

https://github.com/vincentarelbundock/pandas/commit/cf586b9b9648d271c8ee73a659feb2ff5e2bd7a1

Cool. I rebased/fixuped for cleaner history. Now using convert_objects

looks good...do you have travis enabled?

I didn't, but now I flipped the switch and made a minor push. I dont' have a lot of experience with travis, so I don't know if this will work.

`git ci --amend -C HEAD` will basically repush the last commit with a new hash, but what you did was fine.....

can you rebase to 1 commit? otherwise seems ready to go

done

going to merge in a few.....

side issue

should these types of Data I/O things (which are really remote data readers) as opposed to local data formats
be put in a separate doc section entirely?

maybe `Data Readers`?

@cpcloud ?

That sounds good, what about "Remote Data Access"?

sure.....will throw it in....

@jreback the code chunks don't show up in the live docs: http://pandas.pydata.org/pandas-docs/dev/remote_data.html

What did I do wrong?

your directives were `code`, while we use `code-block` in the rest of the code. must be a older version of sphinx that wesm builds with; I built locally and worked fine. i just changed them and it worked fine. weird.

fixed here: 19c8874dbfaa2eef06f2470762f002bbf786ed53

give a once onver after 5pm est again...thanks!
BUG: Fix brittle pivot margins
State: closed

closes #3334  Brittle pivot margins

Issue was that pivot tables that use all columns of the original DataFrame in rows and cols failed on the margin computation. This is a special case for margins: one should use the index itself as the value.

```
>> df = DataFrame({'Response' : ['Y', 'N' ,'N', 'Y', 'Y', 'N'],
            'Type' : ['A', 'A', 'B', 'B', 'B', 'C']})

>> pivot_table(df, rows='Response',cols='Type',aggfunc=len,margins=True)
>> Type      A  B   C  All
>> Response               
>> N         1  1   1    3
>> Y         1  2 NaN    3
>> All       2  3   1    6
```

This is my first PR to Pandas, so apologies for anything out of the ordinary.


Labels:

Comments:
pls move the release notes to 0.13.thanks! (you prob need to rebase too)

I moved release note to v0.13.0.txt and rebased the branch. 

can you squash into 1-2 commits?

`git rebase -i <commit 1 prior to yours>`...then s/f

sorry..should have mentioned that before

Fixed

also...need to rebase on master (what happens is there are conflict on release notes....no biggie)....just rebase and i'll merge you right away

woops! Turns out my origin master was out of date. Should have used upstream. Hang on, will fix.

@jreback I made a new branch off master and cherry-picked my commits into it. I picked up a change in a test file
https://github.com/pydata/pandas/blob/master/pandas/tools/tests/test_pivot.py#L11
that makes my tests fail. Any idea what could have happened there?

you need to rebase this

something like

updates master to current

```
git checkout master
git pull
```

```
git branch my-branch --set-upstream master
git checkout my-branch
git pull --rebase
```

will bring you up to current

Fixed - sorry.

@jreback for some reason I made the decision to never use pull (I like to use fetch/merge fast-forward/rebase so I don't get in sticky situations) but now I look again it looks like you can do that with pull too, cooool

```
git pull --ff-only
```

_not sure I'll change._

@jreback Travis is green. Any idea when this can get merged in?

@cpcloud can u take a look?

@guyrt thanks! merged in

via 69f559447bc2ac8dc216aad3569dbaf6f46280eb
TST/BUG/CLN: make stata IO tests use temporary files for writing
State: closed

closes #4353


Labels:

Comments:
@mitya57 can you confirm that this works for you?

Yes, it works for me. Thanks!

@mitya57 Do all test pass for you? There are quite a few other places where file writing occurs where an explicit path is passed.

I do get some other failures locally (i.e. https://gist.github.com/mitya57/5910851), but they are unrelated to permissions issue and don't happen on Jenkins, so you may ignore those.

second one is an old bug that has been fixed...not sure about the other two
DOC/BUG: fix doc build error because a frame was being redefined
State: closed


Labels:

Comments:
not sure if this qualifies as one that i should have pushed directly to `master`.

might be worth noting to be a bit more careful when redefining variables in the ipython snippets since there's a global namespace shared....

also maybe reminder to build the docs from scratch after you make changes

yep
easy enough to push to master
BUG: fill_value is ignored when reindexing empty series
State: closed

Here's a snippet depicting the, probably, buggy behaviour:

``` python
>>> import pandas as pd
>>> s = pd.Series()
>>> s.reindex([1,2,3])
1   NaN
2   NaN
3   NaN
dtype: float64
>>> s.reindex([1,2,3], fill_value=0.0)
1   NaN
2   NaN
3   NaN
dtype: float64
>>> pd.version.version
'0.12.0.dev-d7c6eb1'
```

I'd expect the last output to contain `0.0` values instead of `NaN`.


Labels:

Comments:
pls add a release notes entry as well..thanks

Done (also rebased the PR against post-0.12.0 master)

@immerr this works in master now

but would like to have your tests in any event

can you rebase and change so just your tests are in?

closed via cef13dbccaf155067ded9d9f2739149e46f2e7d2

@immerrr thanks!
SQL alchemy file structure
State: closed

partial solutions to #4163

Merge this into the sql branch so we can start incorporating bug fixes into the new file structure of `sql.py` / `sql_legacy.py`.


Labels:

Comments:
failed travis should be fixed with a rebase :)

Yeah, thanks. It's running now. What happened there, out of curiosity?

was a travis config issue my fault

Good to go. We should merge #4304 into sql.py, legacy_sql.py, test_legacy_sql.py. test_sql.py is empty so far.

not sure, but i think #4163  will be closed when this is merge just note to self reopen it if it closes :smile: 

I think the issue only closes if you say 'closes' before the issue (I one had 'fixes' and nothing happened)

[official docs for that](https://help.github.com/articles/closing-issues-via-commit-messages) (sort of) it's about commit message closing

@danielballan minor quibble....do we need sqlalchemy in all builds? things are so fast now almost doesn't matter

I'm not sure. Are there docs on what builds do what? The versions of Python are obvious, but what is _LOCALE doing?

no docs really, i'll open an issue to doc them in the wiki, but the build stuff changes occasionally so need to remember to update that.

here's the short version

2.6 runs tests marked not slow WITHOUT full dependencies
2.7_LOCALE runs slow tests WITH full deps in a different locale (Chinese i think)
2.7 runs not slow WITH full deps (we unofficially call this the production build)
3.2 runs not slow WITH full deps (minus a few deps that just aren't py3k compat)
3.3 runs not slow WITH full deps same as above ^^^^

generally full deps are on
2.7,3.2 locale does the plotting stuff
2.6,3,3 are pretty bare
look at the print versions to see what is installed 

@jreback 3.2, 3.3 are now running full deps

3.3 runs full deps?

yep

oh ok great

I can remove sqlalchemy from 2.6 for consistency. Might as well run it to be sure we don't introduce 2.6 compat issues? Your call.

We could make it a soft dependency i.e. skip tests if SQLalchemy (or MySQL etc.) installed or if there is a connection faiiure (with anything other than SQLite)?

At the moment:
- test_sql is empty, I'll quickly tweak the test_sql I put together a couple of weeks ago (to use SQLite BUT I won't yet use SQLAlchemy notation (but that needs to be done to avoid writing lots of dialect specific tests), but that should be done very shortly). Could you then cherry-pick before we merge into master? I don't think we can put this into master completely untested.
- should update _0.13_ release notes that sql has been moved to sql_legacy, and sql will be used for sqlalhemy.

don't merge quite yet
waiting for Wes to cut the release

@hayd Yes, I agree sqlalchemy should be a soft dependency, just as MySQLdb is now. Can we merge all these commits into the sql branch but none into the master?

yes.....can merge to sql branch no prob (@hayd)

Updated sql branch, go for it!

closing in favour of merging sql branch later
DOC: Update computation.rst
State: closed

A wider span means a smaller alpha, which means each x_t has less impact on the ewma.


Labels:

Comments:
The proposed change to the documentation

y_t = (1 - \alpha) y_{t-1} + \alpha x_t 

would break this test: http://stackoverflow.com/a/17954898/190597

@unutbu are the docs wrong or the computation?

I believe the documention is correct as it stands.

@vt100 this change would make the code and docs inconsistent

can u confirm?

Closing request. Issue solved by Pull Request #4499
Arguably better handling of input data in constructor for DataFrame (fix for #4297)
State: closed

The problem was basically an odd restriction to `list` ,`tuple`, `dict`, or `numpy.ndarray` for data structure given as input to any constructor for DataFrame. Much cruder than a possible issue around buffer protocols I thought (the whole issue report went off on a tangent anyway ;-) ).

The source did not have much (any, actually) comments in that region, and I could not find unit tests documenting the behaviour, but I am guessing that the restriction stemmed from the fact that a string is a sequence in Python while the constructors should consider one string passed as data a scalar rather than a sequence.

The change made goes the other way around and accepts any sequence (not just `tuple`, `list`, or `numpy.ndarray`), except `str` or `unicode` (the later only applies to Python 2.x). `bytes` and `bytearray` are allowed (but I am not sure they should).

The pull request was tested (only `tests/test_frame.py` was run) on Python 2.7.4 and 3.3.1


Labels:

Comments:
fix for #4297 

ok...you will need to hook up to travis (see contributing.md) and push again

also...use `com._is_sequence()` instead of your direct try: except around `len`

@jreback : I can track `_is_sequence()` to be in `pandas/core/common.py`:

``` python
def _is_sequence(x):
    try:
        iter(x)
        len(x) # it has a length
        return not isinstance(x, basestring) and True
    except Exception:
        return False
```

I think that there is a (silent) possible issue with the logic at the following line:

```
19dc2847 (Wes McKinney     2012-11-18 23:59:31 -0500 1618)         return not isinstance(x, basestring) and True
```

(`and True` does not bring anything).

@jreback 
Hook to Travis seems to be in place.

@lgautier 

other thing is this is messing with the constructor, so need perf testing

`.test_perf -b <base_commit> -t <last commit of this PR>`

post (only care about > 1.15 ratio)...(some random things will appear, that's ok)

@jreback 

Isn't this leaning a little toward hairsplitting ?

The only effective code difference is:

``` diff
-            elif isinstance(v, (list, tuple, np.ndarray)):
+            elif com._is_sequence(v):
```

the rest is comments, or editing the message string associated with raising exceptions when a problem occurs...

@lgautier I think it would also be nice to incorporate #3478/#3479 in this same PR

and really no hurry on this....0.13 will be brewing for a while

closing in favor of #4829
Added ind and bw_method kwargs to KdePlot
State: closed

Adds ability to set the bandwidth used for the kde and the indices at which it is evaluated. The first is passed to the bw_method kwarg for scipy.stats.gaussian_kde (for scipy >= 0.11.0); the second is passed to gkde.evaluate. For the case when scipy < 0.11.0 but bw_method is specified, bw_method is ignored (not passed to scipy.stats.gaussian_kde), but a warning is issued to alert the user.

https://github.com/pydata/pandas/issues/4298


Labels:

Comments:
looking good....pls add a mention in v0.13.0.txt and doc/source.release.rst (in the 0.13 section)

great...looks mergable (as soon as we start the 0.13 ones)...@cpcloud?

@khgitting this looks good, can you rebase on master, then good to go?

@jreback done. 

@khgitting can you squash to 1 commit?

(like a rebase, but use 's' on the other commits)

@jreback Rebased, squashing, on top of a fresh master from fetching upstream. 

@khgitting thanks!
TST: make sure test passes even if components link isn't present
State: closed

closes #4309


Labels:

Comments:
@jreback i think this might be broken...

```
In [1]: from pandas.io import data as web

In [2]: web
Out[2]: <module 'pandas.io.data' from 'pandas/io/data.pyc'>

In [3]: web.get_components_yahoo('^NDX')
Out[3]:
      exchange   name
@^NDX      N/A  @^NDX

In [4]: web.get_components_yahoo??

In [5]: web.get_components_yahoo('^DJI')
Out[5]:
      exchange               name
AA        NYSE  Alcoa Inc. Common
AXP       NYSE  American Express
BA        NYSE  Boeing Company (T
BAC       NYSE  Bank of America C
CAT       NYSE  Caterpillar, Inc.
CSCO  NasdaqNM  Cisco Systems, In
CVX       NYSE  Chevron Corporati
DD        NYSE  E.I. du Pont de N
DIS       NYSE  Walt Disney Compa
GE        NYSE  General Electric
HD        NYSE  Home Depot, Inc.
HPQ       NYSE  Hewlett-Packard C
IBM       NYSE  International Bus
INTC  NasdaqNM  Intel Corporation
JNJ       NYSE  Johnson & Johnson
JPM       NYSE  JP Morgan Chase &
KO        NYSE  Coca-Cola Company
MCD       NYSE  McDonald's Corpor
MMM       NYSE  3M Company Common
MRK       NYSE  Merck & Company,
MSFT  NasdaqNM  Microsoft Corpora
PFE       NYSE  Pfizer, Inc. Comm
PG        NYSE  Procter & Gamble
T         NYSE          AT&T Inc.
TRV       NYSE  The Travelers Com
UNH       NYSE  UnitedHealth Grou
UTX       NYSE  United Technologi
VZ        NYSE  Verizon Communica
WMT       NYSE  Wal-Mart Stores,
XOM       NYSE  Exxon Mobil Corpo
```

i don't think `NDX` is a valid symbol anymore for whateve reason

yep
just comment out the whole test

if i figure this out i'll submit another pr

figured it out

http://finance.yahoo.com/q?s=%5ENDX

the link to components doesn't work so that test should be totally removed or test failure

or what i could do is check for `len(df)>1` and continue the test if it is, because this was valid at some point and i'm guessing might be in the future

that would work

is this the last bug for release 0.12? what's going on with the docs stuff?
ENH: add integer sheetname support in read_excel
State: closed

closes #4301


Labels:

Comments:
I guess a sheet names as '0' would have to be specified as as actual string to avoid confusion

yep. i should also add something to the docs acknowledging that and an example

i'll add a test for that to see how it's handled. not sure how `xlrd` deals with that

you could try to get a stringified first then fall back to positional if it is an integer
not sure if that is more confusing

so essentially prefer named sheets to indexing where possible

on 2nd thought I think passing an int try for index otherwise try named (like u have it)
no ambiguity

@jreback ot what is the Bounty label for?

Bounty label?

there's an issue label called "Bounty", just wondering what that's for...

no idea
maybe a standard label that git creates?
like bounty on stackoverflow

or it could be the quicker picker upper :)

:laughing:

@cpcloud @jreback it's the label for issues with bounties on Bountysource ;) (https://www.bountysource.com/trackers/50755-pydata-pandas)

@ccvergara oh cool didn't know about that thanks!

@jreback any last words?

I think u need to rebase as release notes 
looks older

@jreback good 2 go?
CLN: let URLError fall through in pandas.io.html._read
State: closed


Labels:

Comments:
cc @jtratner
#4303 

@cpcloud handling network IO is clearly tricky. Were you able to recreate the conditions for the error that was initially raised with the original URL?

nope. just skipping for now if raising `IOError` (this is what the `network` decorator does), but there's something going on with a bogus url that i'm passing in tests with unicode not sure what yet

marking for 0.13 because i'm not sure if i'll be able to get this by this weekend

oh phooey i put in 0.12 release notes :expressionless:
BUG/TST: Fix io.sql.write_frame replace bug and complete test coverage of if_exists functionality
State: closed

This fixes #2971 and #4110.  This also provides test coverage for the different values for the if_exists argument for io.sql.write_frame.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
related to #4163

Rebased and added test coverage for mysql flavor

@hayd you have this included in SQL ?

@jreback thanks, I'd missed this. Will squish and add in to legacy and sqlalchemy.

@hayd just keeping it on the radar

#2971 is about pandas dropping tables and losing data. That's... not cool.
We should get it fixed for 0.14, regardless of #5950. 

@hayd, we pushed 0.13.1 to next week, can you give a push on this for 0.13.1
by any chance?

squished and rebased this branch, will see if it passes/what needs fixing. Agree this was bad, apologies on dropping the ball here. Should make sure this is also fixed in the new stuff.

@jorisvandenbossche  @hayd should this be incorporated / closed 
does it fit the new schema? 

This PR was rebased by @hayd in #6164 and this PR is merged, so closing.
ENH: expose ujson precise_float argument on decode
State: closed

The new version of ujson takes a boolean `precise_float` argument when decoding, this should be exposed when decoding pandas objects.


Labels:

Comments:
I can update the docs about `precise_float` (when you say ready for merge)....

how shall I describe this?

Doh, I never think about the docs, ujson describe it as 

```
precise_float

Set to enable usage of higher precision (strtod) function when decoding string to double values. Default is to use fast but less precise builtin functionality
```

perfect...let's get everything merged in and i'll fix the docs....let me know what to merge and in which order (prob doesn't matter)

testing on windows now

both PR passed on windows (amd-64)

Order shouldn't matter (I think) as they both touched different files. I re-based both onto master before pushing.

I'm happy for this to be merged now and if / when people are happy with #4299 that can be merged too.

going (only this one until you tell me #4299) ready....in 3...2....1

Cool, thanks

thank you!