BrandonFors/pandas-github-issues huggingface dataset:
ENH: drop_level argument for xs
State: closed

As discussed [here](http://stackoverflow.com/questions/17552997/how-to-update-a-subset-of-a-multiindexed-pandas-dataframe), xs usually remove the level which you are accessing. This commit allows you to explicitly say you don't want that (and want to keep the same levels in the result).

```
In [4]: df = DataFrame({'day': {0: 'sat', 1: 'sun'},
                                   'flavour': {0: 'strawberry', 1: 'strawberry'},
                                   'sales': {0: 10, 1: 12},
                                   'year': {0: 2008, 1: 2008}}).set_index(['year','flavour','day'])

In [5]: df
Out[5]:
                     sales
year flavour    day
2008 strawberry sat     10
                sun     12

In [6]: df.xs('sat', level='day')
Out[6]:
                 sales
year flavour
2008 strawberry     10

In [7]: df.xs('sat', level='day', drop_level=False)
Out[7]:
                     sales
year flavour    day
2008 strawberry sat     10

In [8]: df.xs([2008, 'sat'], level=['year', 'day'], drop_level=False)
Out[8]:
                     sales
year flavour    day
2008 strawberry sat     10
```


Labels:

Comments:
oh dear, screwed something up in this, need to fix.

@hayd should be marked for 0.13?

@jreback merge? or do you have a better arg name?

hmmm.....and we don't want to make `drop_level` a public function....

could we do this thru `drop`? maybe like this: (same impl)

`df.drop('sat', level='day')`

? 

This makes xs more like a filter (atm the behaviour is to drop the selected level, this adds option not too).

this is adding a modification option `xs`, something it doesn't do now; that's my hesitation

what about adding `drop_level`, be very explicit?

ah ok, yeah drop_level is better

Hmmm giving def _drop_levels a drop_level was stupid, this is a bad way to do this. Need to reconsider.

no...I meant in core/generic.py make a method called `drop_levels` with same args as `xs`, label, axis, level and just do what you are doing now

ah, true and this isn't really a cross section either... I dunno actually I think this makes sense as an option for xs...just viewing the data in slightly different way, with superfluous index level.

I guess you can hack select to do this:

```
In [10]: df.select(lambda x: x[2] == 'sat')
Out[10]:
                     sales
year flavour    day
2008 strawberry sat     10
```

Maybe could just add this api to select? (add level argument)

```
df.select(self, crit, axis=0, level=None)
```

looks to see if crit is callable, if is current behaviour, if not then looks at level whatever and does xs without dropping level (the result we're talking about).

would shy away from select, its a linear search...let's go back to xs...

how about ''drop=...'' ? (rather than `drop_selected_level`?

I liked you drop_level (as a compromise :p), I think it's clearer what you're dropping then. Ok, have to clean up this code though, it looks like I just threw this together without thinking about back-end semantics. I think it's much easier than my awful hack.

@jreback logic is now much improved.

looks good..
CLN: write the attributes in a HDFStore as strings
State: closed

These strings are converted to bytes, previously these were written as unicode


Labels:

Comments:
no....that has to do with the _format_ of the attributes and such (will bump if we ever need to actually change it); I think I did once between 0.10 and 0.10.1

This pr just fixed a display issue I think. was actually writing the unicode (rather than bytes) for pandas_type,pandas_version (because I decode them and the variable was being passed around). It actually doesn't make any difference and you would not even be able to know except if you do `ptdump -av` the (and you are not using PyTable 3.0.0), then these attribues look like string (e.g. 'appendable_frame'), while in PyTables 3.0.0 (even if you are not using it, but have the ptdump installed), they are bytes (b'appendable_frame')

somehow they were getting pickled (rather than written as strings), but this was all transparent (just a bit inconcistent that's all). You should be able to read/write something from an older version with a newer version (of PyTables) and vice-versa

if you try out 0.12rc1 you can see these (as compared to master)...install PyTables 3.0.0, write a store and do ptdump on it (then install master) and do the same thing
TST: Do not skip MySQL tests on Travis.
State: closed

A first step for #4163


Labels:

Comments:
The build [failed](https://travis-ci.org/danielballan/pandas/jobs/8892378) because of something in `pandas.io.tests.test_data.TestYahoo`. Should I just try pushing again? Previous test on the [exact same code passed](https://travis-ci.org/danielballan/pandas/builds/8891299).

The error is small, and moreover it's unrelated to my changes.

```
AssertionError: 
Arrays are not almost equal to 5 decimals expected 904.91000 but got 904.90000
```

rebase off of master and try again, i just merged a commit to fix that

Full speed ahead.

[2.7 build](https://travis-ci.org/danielballan/pandas/jobs/8895481) sulking a little:

```
/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.12.0rc1-py2.7-linux-x86_64.egg/pandas/io/tests/test_sql.py:391: Warning: Unknown table 'test_table'
  cur.execute(drop_sql)
/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.12.0rc1-py2.7-linux-x86_64.egg/pandas/io/tests/test_sql.py:408: Warning: Unknown table 'test_table2'
  cur.execute(drop_sql)
./home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.12.0rc1-py2.7-linux-x86_64.egg/pandas/io/tests/test_sql.py:300: Warning: Unknown table 'test'
  cur.execute(drop_sql)
```

Is this running all the tests now? Merge?

Merge.

does this also correct skip when mysql not installed?

Yes.
BUG: wrong parsing of microseconds with format arg (#4152)
State: closed

Closes #4152.

`fraction` was already in microseconds, so no need to multiply it with 1000000, which caused faulty parsing of dates with microseconds.

With this commit I get:

```
pandas version: 0.12.0.dev
Value to parse: 01-Apr-2011 00:00:01.978
datetime.strptime        : 2011-04-01 00:00:01.978000
to_datetime, w/out format: 2011-04-01 00:00:01.978000
to_datetime, w/ format   : 2011-04-01 00:00:01.978000
```

for 

```
import datetime as dt
import pandas as pd
val = '01-Apr-2011 00:00:01.978'
print 'pandas version:',pd.__version__
print 'Value to parse:',val
format = '%d-%b-%Y %H:%M:%S.%f'
print 'datetime.strptime        :',dt.datetime.strptime(val, format)
print 'to_datetime, w/out format:',pd.to_datetime(val)
print 'to_datetime, w/ format   :', pd.to_datetime(val, format=format)
```


Labels:

Comments:
I added a test based on the reported bug in the issue, but I don't have experience with writing tests. Is it OK like this?

~~Nearly, not sure this is going to work since now to_datetime returns a Timestamp object rather than a Datatime...~~

Maybe better to do (will also give nicer error message):

```
self.assertEquals(str(result),  '2011-04-01 00:00:01.978000')
```

that is fine

there is another issue in processing to_datetime
thanks for this catch 

a Timestamp will compare to a datetime ok

@jreback my bad!

OK!

@jreback What do you mean with another issue?

`to_datetime` had some convoluted logic and I came up with a case where the format was being interpreted correctly; the existing test was not good enough

but unrelated to the bug you are fixing.....

going to merge yours and then fix the other issue

you can reproduce if you pass a list of values (rather than a single value)

it will ignore the format

@jorisvandenbossche I take that back....looks like your PR fixes this....

can you add a release notes mention? (then squash together if you can)...thanks

OK, I added something to the release notes. With 'squaqh together', do you mean putting it together in one commit?

I looked at the PR #4167 you mentioned, but I doubt that this PR fixes the test you added there ... (this one only changed something that had to do with microseconds)

@jreback OK, I also put it in one commit. 

lovely! Excellent work!

@jorisvandenbossche 2nd that!

this really does fix the error, my PR just makes the test more consistent (and is indepdent of what you did)

thanks again

this introduced another bug it seems, see #4179 

@cpcloud Strange, it is exactly the test that I introduced in this PR and which should be fixed by this PR that is failing. Did you rebuild the cython files?

oh gosh duh. sorry. closing the issue
ENH: add correct versions of numpy installs
State: closed

closes #4142.

should be able to finish up tomorrow, just need to put things on the server and
test


Labels:

Comments:
@y-p can u have a look-see?

some notes:
- for some reason the locale build won't override the lxml install so i'm uninstalling it so that the wheel gets used and that works
- i'm installing build deps for numpy which i don't think was being done before now: i couldn't get numpy to compile on the vagrant box unless i installed numpy's build deps, it doesn't add a significant amount of time to the build

anyway it seems to be working so lmk if u think i should change anything

here's a travis build:
https://travis-ci.org/cpcloud/pandas/builds/8825273

Very strange about the LOCALE build refusing lxml, there shouldn't be any difference at all
between it and the other 2.7 build.

Adding a JOB_TAG envar to each job in travis.yml and using that to select the requirements file
and url would make more sense IMO then what you opted for.

re numpy build deps, what packages specifically? I've had no problems building numpy 
with the stock precise64.box + the packages that the original build script installed.

All in all, If it works reliably, it's good enough.

- [x] `numpy` build-dep was a false alarm, rebuilding packages now...
- [x] `JOB_TAG` stuff

just a note to self: distribute 0.6.35 doesn't work with py3

The amount of pain and suffering python packaging inflicts on kind, honest devs..

note that latest pip stable use 0.6.34 I believe, and 0.6.35 is the minimum version required
by pip beta (which Is why I named that specifically) , distribute itself has released many versions 
since, although now it's being merged into setuptools. It's a longshot but you may try one of the 
latest versions.

oh, they just released setuptools 0.8 final and pip 1.4rc3, perhaps
everything just works with that.

Would changing the versions of pip and using setuptools require a rebuild of packages? if not, great, i'll try it, if so i'm going to stick with what works which is pip installing whatever version of distribute it does for py3 and 0.6.35 for py2

It should absolutley _not_ make a difference. Obviously, you install packages from pypi without 
taking notice of what the packager had installed.
I would have thought that the default distribute for py3 would make pip yell, but if you have
a working setup, I'd stick with it too.

not sure what the lxml issue is and not worth tracking down for now...leaving the uninstall in

@y-p any last words before I merge (after passing)?

yes: thanks for taking care of this!

no problem! happy to grow a few gray hairs helping out pandas!

quick recap of changes:
- travis dependencies are now controlled by individual job specific requirements files available in `pandas/ci`
- py26 now now uses numpy 1.6.1
- py27 locale now uses numpy 1.6.1
- py27 uses numpy 1.7.1 and scikits timeseries (still full deps)
- py32 now uses numpy 1.6.2
- py33 now uses numpy 1.7.1 and runs with full dependencies

bombs away
BUG: (GH4145/4146) Fixed bugs in multi-index selection with column multi index duplicates
State: closed

closes #4145, #4146

```
In [1]: data = """h1 main  h3 sub  h5
   ...: 0  a    A   1  A1   1
   ...: 1  b    B   2  B1   2
   ...: 2  c    B   3  A1   3
   ...: 3  d    A   4  B2   4
   ...: 4  e    A   5  B2   5
   ...: 5  f    B   6  A2   6"""

In [2]: df = pd.read_csv(StringIO(data),sep='\s+',index_col=0)

In [3]: df2 = df.set_index(['main', 'sub']).T.sort_index(1)

In [4]: df2.loc[:,('A','A1')]
Out[4]: 
main  A
sub  A1
h1    a
h3    1
h5    1

In [5]: df2[('A','A1')]
Out[5]: 
main  A
sub  A1
h1    a
h3    1
h5    1

In [6]: df2['A']['A1']
Out[6]: 
   A1
h1  a
h3  1
h5  1
```

This should NOT work, as this is tryingo select 2 different columns ('A' is ok, but 'A1' fails which is correct)

```
In [7]: df2[['A','A1']]
KeyError: "['A1'] not in index"
```

This is ok though

```
In [8]: df2[[('A','A1')]]
Out[8]: 
main  A
sub  A1
h1    a
h3    1
h5    1
```


Labels:

Comments:
@hayd cc @jtratner

tell me if you think this covers it....

Looks good to me. Is df2['A']['B2'] behaviour already tested elsewhere?

here are the cases

2 level index
unique - unique
unique - non_unique

select top level
select top level and 2nd level unique
select top level and 2nd level non_unique

maybe should put a comprehensive test
pretty sure all tested - maybe move all to a single test method?
ENH melt uses column name if available
State: closed

Currently melt allows you to manually specify the val_name and var_name, this changes the default behaviour to grab the column name if one is available.

```
In [1]: df = pd.DataFrame({'a': {0: 1.0676829999999999, 1: -1.3214049999999999, 2: -0.80733299999999997}, 'b': {0: -1.110463, 1: 0.36891499999999999, 2: 0.082979999999999998}, 'c': {0: 0.20866999999999999, 1: -1.055342, 2: -0.87336100000000005}})

In [2]: df.columns.name = 'foo'

In [3]: df
Out[3]:
foo         a         b         c
0    1.067683 -1.110463  0.208670
1   -1.321405  0.368915 -1.055342
2   -0.807333  0.082980 -0.873361

In [4]: pd.melt(df)  # before this would have columns ['variable', 'value']
Out[4]:
  foo     value
0   a  1.067683
1   a -1.321405
2   a -0.807333
3   b -1.110463
4   b  0.368915
5   b  0.082980
6   c  0.208670
7   c -1.055342
8   c -0.873361

Also, cleaned up all the tests (and added some more) in test_reshape.
```


Labels:

Comments:
isn't this already in with #3649, or this is just a cleanup of that?

oh i c, you name is more 'automatic'...nice!

@hayd there is a release notes/whatsnew associated with 3649, why don't you add on to it with this change?

I was just pondering whether I needed to bother :) will do

As an aside, wonder if melt should work with multiindexes...

merge? (Have another WIP for MultiIndexes)

look good
BLD: add SITE_PKG_DIR back in and fix numpy 1.6.1 install
State: closed

closes #4142 .


Labels:

Comments:
I pushed the same SITE_PKG_DIR fix a few minutes ago, and just cherry picked the lxml
fix a moment ago as well.

where is numpy 1.7.1 being installed?

https://travis-ci.org/cpcloud/pandas/jobs/8785597

there's nothing in the travis docs about numpy being already installed

[these](http://about.travis-ci.org/docs/user/languages/python/#Pre-installed-packages) are the only pre-installed packages.

i think that the other deps that rely on np need another wheel compiled against 1.6.1 if we want to use it in travis

@y-p should i close? we can move the discussion over to #4142. see my previous comment

ok.
BLD: install pip serially
State: closed

closes #4132.


Labels:

Comments:
this looks like it fixes it?

are some of these suppopsed to install numpy 1.6.1????

https://travis-ci.org/pydata/pandas/jobs/8782276

and statsmodels not install ing either.....?

https://travis-ci.org/pydata/pandas/jobs/8782274

not sure...but all those question marks just about gave me a heart attack :)

sory....creating an issue

i believe all compiled artifacts are coming from @y-p's openshift server, so i think installing 1.6.1 is going to require some help from him if we want to add that...statsmodels i believe compiles only a few things so maybe either we could add to @y-p's server if he thinks it's a good idea or if it's not really worth it since there aren't a whole bunch of things compiled then we'll fix it without that

oh i c the 1.6.1 on the server

The server I put up hosts just the packages that require long compilation (numpy, botlleneck, 
pytables, etc'). If a package is not found on the openshift server pip is supposed to fallback to pypi.
so you can specify any package and be no worse off then before the change. I did make 
multiple versions available on the server, numpy 1.6.1 included.

I'm looking into @jreback's notes.
BUG: Make ExcelFile.parse pass kwds to TextParser
State: closed

Fixes #4131


Labels:

Comments:
test2.xlsx is a single column with the following cells:

```
Test
NA
1
NA
apple
rabbit
```

this looks ok to merge after #4141

@jtratner can u push again...thanks

rebased + pushed again.

On Fri, Jul 5, 2013 at 6:39 PM, Phillip Cloud notifications@github.comwrote:

> @jtratner https://github.com/jtratner can u push again...thanks
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4139#issuecomment-20542620
> .

Oops forgot update to release.rst

@hayd do you want me to make a new PR for that?

actually what we could use here is a method in io.parsers that give u back all of the default args (and values)
to avoid explicitly listing them in more than one place

used in io.parsers obviously but also here

note - maybe be a smaller subset that io.parser
fix typo - change conn to con
State: closed

Noticed a typo in the documentation of `write_frame`.

Method def has `con` while documentation had `conn`.

`def write_frame(frame, name, con, flavor='sqlite', if_exists='fail', **kwargs):`


Labels:

Comments:
Convention generally is to call connection `conn` and not `con`, so maybe the fix should go in the other direction. e.g. [in the Python docs for sqlite3](http://docs.python.org/2/library/sqlite3.html).

I agree and just made the change, however now that I pushed the change I went back and looked the rest of the file and it seems to use `con` for everything.

So I think I'll push back only the documentation update, however I'd be happy to revise the entire file... 

good call

On Thu, Jul 4, 2013 at 1:07 PM, Trent Hauck notifications@github.comwrote:

> I agree and just made the change, however now that I pushed the change I
> went back and looked the rest of the file and it seems to use con for
> everything.
> 
> So I think I'll push back only the documentation update, however I'd be
> happy to revise the entire file, but until somebody with more clout weighs
> in I'll leave it as the simple doc fix.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4127#issuecomment-20486555
> .

@tshauck Thanks for finding/fixing this!

Do you mind deleting the last two commits or rebase squashing the last three? so either

```
git reset --hard HEAD~2
```

or

```
git rebase -i HEAD~3
```

@hayd Happy to do it... I just pushed the change.

Awesome, thanks!
CLN: Small fix to clean_pyc task
State: closed

The previous `clean_pyc` wasn't actually removing `*.pyc` files - apparently each branch of find needs its own exec (I guess?). Anyways this works now.


Labels:

Comments:
Hm. OK didn’t realize there was a notion of precedence here. u could try the -delete option but don't spend too much time here if this works then great.
BUG: make sure fig is not doubled when passing by to series.hist
State: closed

closes #4112.


Labels:

Comments:
@jreback any issues here?

if this isn't a critical bug then i would suggest we wait to _really_ fix the issue here in 0.13

is kind of hacky...up 2 u

k let's wait then. if anyone raises a bug then can show them how to put it into a frame and do ti

closing...i'll raise an issue.

Reopening because the plotting API will take some time to fix and double figures should not be happening in the meantime

@jreback ok 2 merge?

sure...if you think ready...i don't really get into the plotting too much

alright let me put it through the ringer with some of my god-awful plotting code and see if i find any edge cases
CLN: use how="all" in data.py so that things that are not nan in other columns are kept
State: closed


Labels:

Comments:
This is how i feel right now.
![everytime-you-break-the-build1](https://f.cloud.github.com/assets/417981/743286/ad704630-e3e4-11e2-826e-01e92cc3f916.png)
ENH: allow hist to take a string "by" argument
State: closed

closes #4103.


Labels:

Comments:
@jreback ok 2 merge? this adds the ability to specify a string column in the by argument of `hist`

ok

i htink i'm going to wait here until dup plotting is fixed i don't want to complicate things more than they already are

closing because this feature shouldn't be added until the plotting API is more unified .. it's just not essential right now
ENH: allow layout for grouped DataFrame histograms
State: closed

Also remove the slow decorator for tests that take < 100 ms time

closes #4102.

closes #4194.


Labels:

Comments:
@jreback any comments b4 i merge? nothing drastic here just supporting the new layout argument with the by argument

oh release notes

i think this could be merged once it passes...nothing super drastic, just allows the layout keyword to work when the by keyword is passed

eh i'll wait...not critical

@jreback shoudl this be marked for 0.12 to close the kde plot issue? or i can mark that one for 0.13...even though it is probably fixed, if we do 0.12.1 it could go there instead as a bug fix along with this...

up2u

up2u

k i'll mark #4194 for 0.13/0.12.1, just want to get those html bugs squashed!

@jreback anything to add here b4 i merge?

ok
TST: test_fast.sh should not run network tests
State: closed


Labels:

Comments:
The convention until recently was that all '@network' tests are marked '@slow' as well,
by the nature of things.
Most/all of the counterexample I looked at are from code last touched in recent weeks.

if tests are marked `@network` and `@slow` they won't run on any of the travis runs except python 2.6

if

A = set of all pandas tests not marked `@slow`, `-A 'not slow'`
B = set of all pandas tests marked `@slow` but not marked `@network`, `-A 'slow and not network'`
A intersect B = the empty set (cannot be both slow and not slow)

thus no runs of things marked both `@slow` and `@network` (2.6 doesn't have any env args so i'll run things, but not really since no full deps so many (if not all) `@network` tests will not be run)

so either the travis config must be changed (maybe better in the long run, at the cost of using more time) or use this for now

this is why e.g., the test_fama_french wasn't failing even though it should have

closing in favor of using detox, works as a drop in replacement for `tox_prll.sh`, thanks @y-p
ENH: add the beginnings of an exception hierarchy specific to pandas
State: closed


Labels:

Comments:
can you hold off on this for a few days? I've been working on cleaning up Exceptions already and that will make this easier to do (because it will be clearer which items actually need custom exceptions).

sure thing

Thanks :) I agree with you on the breakage aspect - most of the PRs I've
started recently have come from mucking about in pandas' exceptions (e.g.,
the weird issue with naming, etc.).

On Sat, Jun 29, 2013 at 12:12 PM, Phillip Cloud notifications@github.comwrote:

> sure thing
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/4083#issuecomment-20232489
> .

i removed that comment because making invalidindexerror a subclass of keyerror leads to 630 broken tests

so i just removed and everything went swimmingly

but i'm sure your pr will be excellent

heh - well, more along the lines of "Hey, why doesn't removing X make anything fail?" Oh...that's because of other Y thing."

`InvalidIndexError` is an _internal_ (to indexing) exception...more like a signal to break out of deeply nested stuff, rather than a _real_ exception

right, i discovered that using the empirical method :)

I think there is a similar one in groupby

@jtratner i think #3954 can subsume this....should i close?

Absolutely. I need to revisit that soon and start getting more off of the
list...