BrandonFors/pandas-github-issues huggingface dataset:
API: default value for read_excel sheet
State: closed

closes #6573

Implements a default sheetname of 0 for `read_excel` #6573. 

Before: `read_excel(io, sheetname, **kwds)`

After: `read_excel(io, sheetname=0, **kwds)`


Labels:
name: API Design
name: IO Excel
description: read_excel, to_excel

Comments:
can u add a test to validate this?
eg write a frame to a workbook with the name Sheet1 and another with the name 0
read back should work
pls follow the examples in the test_excel

Sure, will do.

On Sat, Mar 8, 2014 at 3:49 AM, jreback notifications@github.com wrote:

> can u add a test to validate this?
> eg write a frame to a workbook with the name Sheet1 and another with the
> name 0
> read back should work
> pls follow the examples in the test_excel
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/6576#issuecomment-37095732
> .

@jreback Added this within the `roundtrip` test.

@jreback Docs revised.

all for clarity in the docs....but have to squeeze lots of things into a limited number of examples.

If you can make it more clear, by all means (but at the same time try to show as much as possible)

you can squash down  to 1 commit right before we merge

@jreback Ok, I changed those examples back to how they were.

ok...pls squash....

you can look at the built docs (now and a few minutes after the merged)

http://pandas-docs.github.io/pandas-docs-travis/

if you want to do a follow-up PR to fix/clarify docs...ok by me

the excel sectino COULD use some TLC!

That's good to know about the built docs. Could we put that link in this page - http://pandas.pydata.org/developers.html#contributing-to-the-documentation

I'll squash when I get home tonight. Should I be using `rebase -i`? I don't want to make a mess for the maintainers to clean up like [last time](https://github.com/pydata/pandas/pull/6008)...

yeh....theirs been some work on add all of this info to the main docs, see here: http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-to-the-documentation

interactive rebase `-i` is usually best

@jreback how does this rebase look?

looks good!

thanks...excellent PRing!

do some more

Glad to hear! I've got a much better handle on how to PR. Thanks @jreback for walking me through it.
ENH/BUG groupby nth now filters, works with DataFrames
State: closed

fixes #5552

partial for #5264

```
In [101]: df = pd.DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=['A', 'B'])

In [102]: g = df.groupby('A')

In [103]: g.nth(0)
Out[103]:
   A   B
0  1 NaN
2  5   6

In [104]: g.nth(1)
Out[104]:
   A  B
1  1  4

In [105]: g.nth(-1)
Out[105]:
   A  B
1  1  4
2  5  6

In [106]: g.nth(0, dropna='any')  # old behaviour-like
Out[106]:
   B
A
1  4
5  6

In [107]: g.nth(1, dropna='any')  # old behaviour-like
Out[107]:
    B
A
1 NaN
5 NaN
```


Labels:
name: Bug
name: Groupby
name: API Design

Comments:
Also note old behaviour was not stable/correct for negative (now fixed with PR&dropna):

```
In [9]: g.nth(-3,)
Out[9]:
               B
A
1  2.144760e-314
5  2.124748e-314

In [10]: g.B.nth(-3,)
Out[10]:
A
1    2.144760e-314
5    2.144337e-314
Name: B, dtype: float64
```

If you get around to it; I suspect the new method is MUCH faster than the old, so maybe add a vbench

will append a vbench. Is much faster except when applying to dataframe with dropna (old-style) it's a little slower, but that was previously borked.

Um, obviously there is overlap with first and last methods, they be got with nth(0) and nth(-1) but not tested the differences yet... you reckon these should change too?

yes I think you should blow away first/last code and just alias them to `nth(0)` and `nth(-1)`.

reminds me that pls put some tests that deal with different types (because first/last have this convert arg..though not sure why)

though maybe `nth(0)` (first) and (iloc[0]) deserver a fast-path as it doesn't need the machinery of cumcount

there was/is a weird test for types of first/last/nth, I tweaked it a little but is still there...

I can iterate tests over a few of the same df (but with different column types), is that what you mean?

Yea, re fast path (will see how they compare the the cumcount for now)...

Added vbench, is about 40 times faster with not-including the setup of the groupby (which is included in the bench)

awesome!
ENH: Allow timestamp and data label to be set when exporting to Stata
State: closed

closes #6545

Added code which allows the time stamp and the data label to be set using
either StataWriter or to_stata.  Also simplified reading these values using
StataReader by removing null bytes from the string values read.

Added basic test for both.

Also fixed one small bug where variables could be stored using Stata reserved
words.


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Addresses the issues in #6545.  Also one very small fix for variables that have name conflicts with reserved words.

@bashtage gr8! FYI keep in mind if you need/want to break back compat should be brought up (not that this change does, and your previous change was essentially a bug-fix)

@jreback I am conscious that breaking backward compat is generally a bad thing that should be avoided if possible.

On the upside, I finally figured out how to rebase correctly, so that I only get 1 commit when I want only 1 commit.  

BTW, is there something i did to not get a TravisCI build?   It failed the first time on the 3.x versions due to a silly leading-0 issue, but I wanted to check that it all passes.

https://travis-ci.org/pydata/pandas

I _think_ that you have to enable travis to access your github account to directly look on travis (e.g. then you can look up all of your builds). This is a 'default' build in that it automatically builds all PR's.

I don't think this fail is me... looks like random network timeout.  Does it need to run until it passes?  And if do, is there a way to do this would a fake commit.

nope...that is a random failure....minor typo other wise looks good

can you rebase on current master and force push....had some issue with the caching of the deps...should be resolved now

Rebased

thanks!
BUG: fix fancy indexing with empty list
State: closed

This should fix #6536.

There's a side issue though with loc/ix indexers that somehow discard column names for empty indexers.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
hmm...the names of the non-indexed axis should be ok

```
In [34]: df
Out[34]: 
          A  B                   C
0  0.067509  0 2013-01-01 00:00:00
1  0.872840  1 2013-01-01 00:01:00
2  0.379634  2 2013-01-01 00:02:00
3  0.552827  3 2013-01-01 00:03:00
4  0.996150  4 2013-01-01 00:04:00

[5 rows x 3 columns]

In [35]: df.loc[[]]
Out[35]: 
Empty DataFrame
Columns: [A, B, C]
Index: []

[0 rows x 3 columns]

In [36]: df.loc[:,[]]
Out[36]: 
Empty DataFrame
Columns: []
Index: [0, 1, 2, 3, 4]

[5 rows x 0 columns]

```

I'm talking about names of the axes theirselves:

``` python

In [1]: pd.DataFrame(columns=pd.Series(list('abc'), name='foobar'))
Out[1]: 
Empty DataFrame
Columns: [a, b, c]
Index: []

[0 rows x 3 columns]

In [2]: pd.DataFrame(np.arange(9).reshape(3,3), columns=pd.Series(list('abc'), name='foobar'))
Out[2]: 
foobar  a  b  c
0       0  1  2
1       3  4  5
2       6  7  8

[3 rows x 3 columns]

In [3]: _2.iloc[:,[]]
Out[3]: 
Empty DataFrame
Columns: []
Index: [0, 1, 2]

[3 rows x 0 columns]

In [4]: _2.iloc[:,[]].columns.names
Out[4]: FrozenList([u'foobar'])

In [5]: _2.loc[:,[]].columns.names
Out[5]: FrozenList([None])

```

ahh..that could be (and I'll be it isn't checked in tests).....you can put it in here if you want or separate PR...(or just create a new issue for reference later)

Ok, as you might've seen, I think it's better to address separately and created a separate issue #6552.  Is there anything else I have to do here?

it's good
Travis barfing for some reason
will merge soon
put a note (after this is merged) on the fixing names issues to remove those skips

thanks!
BUG: informative error message with Series ValueError
State: closed

Fixes issue #6532.


Labels:

Comments:
@jreback Running `nosetests pandas` locally produces the same output before and after the change:

```
Ran 5048 tests in 574.965s
FAILED (SKIP=180, errors=4)
```

Thanks!

ps "All is well" on travis, so failing tests must be something on your system... :s
API change in groupby head and tail
State: closed

BUG/API groupby head and tail act like filter, since they dont aggregage fixes column selection

Breaking API change to groupby head and tail, these were never aggregated output, so should not have the group labels as the index, it should always filter. (It was a legacy from when this was `g.apply(lambda x: x.head(n))`.)

```
        as_index : boolean, default True
            For aggregated output, return object with group labels as the
            index. Only relevant for DataFrame input. as_index=False is
            effectively "SQL-style" grouped output
```

Also fixes the column selection for head and tail

part of #5755 
related #6524 
part of #5264.

cc @jreback @TomAugspurger 


Labels:
name: Groupby

Comments:
Note: definitely need to change some documentation here if we were to do this, but thought should put change out there. / discuss

@hayd this is good

why don't you add doc change in v0.14.0 (and in groupby.rst) for this change,

then in #5264 can expand that?

I edited the top to show this closes #5755 (only), others are related, right?

@TomAugspurger

@jreback is a partial close 5755 (I'll make a checklist on that, some of it's also up for discussion, should clean it up in one-hit IMO - this coming release). Ok, cool, will trawl through the docs to see what needs changing and add twice in release.

Yeah this looks good. I'm looking forward to more consistency in groupby :)

Updated... Something has broken the build (first commit passed travis and second only changes docs)!

hmm..travis seems flaky right now....wait before merging this...let's see if can fix travis

looks like whl are not being transported properly....(though I just accessed the server ok)....maybe a network issue from travis?

@hayd 

shouldn't this level be named? (e.g. its the original index of the frame)

http://stackoverflow.com/questions/22210865/python-assign-values-to-first-observation-of-each-group-in-dataframe/22210998#22210998

maybe make a `first_index()` function?

@hayd I realize that this is a case where `as_index=False` actually is useful!

@jreback haha, but you won't need to with this PR :) Ah, the unnamed level is from the original index (which had no name), no?

Is pretty easy to write with cumcount:

```
g.first_index == df[g.cumcount() == 0] ?
```

(s390x ?!!)

I thought he said mainframe. that's something the sparc guys can't even get numpy to run on, yet they want pandas to run ...meh

@jreback you want me to rebase/amending? I want to do the nth stuff on top of this so would be good to merge. (I tried amending before but it failed)

go ahead.....you can prob just merge this... (not sure why travis is acting up stilll)

we'll see...

yeah, build still failing.
BUG/TST: fix several issues with slice bound checking code
State: closed

This patch removes more of what's left after removing pandas-specific customizations to slice indexing.

BUG/TST: `obj.iloc[:,-4:-10]` should return empty slice (because slice.step > 0 and slice.start > slice.stop)
There's no way to describe this by setting stop to a value in  `[0; len]` interval, it should remain negative (see [this ticket in python bugtracker](http://bugs.python.org/issue11842)]. Proposed fix to this issue is to remove the custom logic altogether. A side-effect of that is correct handling of negative steps in presence of out-of-bounds slices, e.g. `obj.iloc[:,10:4:-1]` would've incorrectly returned empty slice.

BUG: fix exception raised by `Series([1,2,3]).iloc[10:]`.
After I've cleaned up suspicious code that handled empty-sequence keys in #6440, this piece of code started causing trouble:

``` python
if start >= l:
    return self._getitem_axis(tuple(),axis=axis)
```

CLN: #6299 & #6443 rendered `core.indexing._check_slice_bounds` function useless and it's now gone.  Next step would be to drop unused `raise_on_error` parameter in slice-related functions.

Not sure, how do I put down all these relations into changelog, advice on that is welcome.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
you can drop the raise_on_error parameter as it was just something to 'fix' the slicing issues (from somewhere in 0.12-0.13 transition).

Just put this all in a single line in API changes (its not really an API change, but more prominent), list all of the related issues.

> Just put this all in a single line in API changes (its not really an API change, but more prominent), list all of the related issues.

Did I get you correctly: https://github.com/immerrr/pandas/commit/c7e634596b32df9151710a9cb43a0813c5a307d1#diff-5a809a5e093be4fe1396cbadd5bbeccaR110 ?

tep

yep

Ok, folding/rebasing it then.

minor change

I think you can move _slice from frame/panel to generic (but series still overrides)

Whoops, apparently I've accidentally deleted wrong gh branch when cleaning up local ones, sorry about that.

Also, `_slice` is now in ndarray as requested.

can you rebase on currentmaster and push again.thxs

Ok, done.

thanks as always!

if you are interestdd #6536

Sure, that was a low hanging fruit :)
BUG: fix DataFarame column slice indexer
State: closed

The code pulled in in #6123 didn't account for slice indexers which return views, not copies in `take_ref_locs` function. As a result, slice across columns corrupted `_ref_locs` of the parent block (provided `_ref_locs` existed). This PR should fix that.

``` python
In [1]: df = pd.DataFrame(np.random.rand(5,3))

In [2]: df.dtypes
Out[2]: 
0    float64
1    float64
2    float64
dtype: object

In [3]: df.iloc[:,2:]
Out[3]: 
          2
0  0.524160
1  0.072608
2  0.289047
3  0.494401
4  0.235275

[5 rows x 1 columns]

In [4]: _3.dtypes
Out[4]: 
2    float64
dtype: object

In [5]: df.dtypes
Out[5]: 
0    float64
1    float64
2       None
dtype: object

In [6]: pd.__version__
Out[6]: '0.13.1-346-g30d3cba'
```


Labels:

Comments:
perfect...sometimes if i fix a bug I would just add on the issue to the previous one....but this is fine too
thanks
DOC: Add common error message to byte-ordering gotcha.
State: closed

A friend of mine got tripped up by [this gotcha](http://pandas.pydata.org/pandas-docs/stable/gotchas.html#byte-ordering-issues). We document it well, but Googling the error message doesn't turn up our documentation, so it took him a lot of digging to get there.

This commit adds an example error message that byte-ordering issues raise, in hopes of making this easier to Google.


Labels:

Comments:
thanks! these are really helpful examples
Allow the use of buffers and None in to_hdf and read_hdf.
State: closed

This patch basically enables the expression

```
 pd.read_hdf(BytesIO(df.to_hdf(key="/data")), key="/data") == df
```


Labels:
name: Enhancement
name: IO HDF5
description: read_hdf, HDFStore

Comments:
This would need extensive testing, as in-memory stores are not supported on older versions of  PyTables, nor under some versions of the underyling library.

Furthermore they are quite a bit slower than just keeping the data in actual memory.

So I think a cookbook entry of how to do this would be fine. But I don't think this should be in the main code.

Pls submit a doc PR if you'd like.

thanks

This is not intended for in-memory stores (that's why I did not implement it inside of `HDFStore`) but for sending HDF data over the wire.

hmm
for what purpose?

unlike other protocols (json,csv) partial results are completely useless

Maybe, but doing a `socket.send(df.to_hdf(key="/data"))` and `pd.read_hdf(socket, key="/data")` is quite performant in my experience (feel free to show me an alternative) as it doesn't require (too much) parsing on the reading end and allows for exact replication of the transferred frame.

try using msgpack

I'll have a look at this again

this would need some tests

and I don't think would work the way you have it

eg if append=True this will fail

It does work for what I do (I'm using this in production ;)), i.e. sending dataframes around in a local network. I'll have a look into msgpack, but IIRC the documentation clearly stated that the support was experimental.

msgpack is experimental just because it's new
haven't gotten much feedback on it
this would need some tests 
if u want to see this included

@filmor this needs some tests. going to close for now. pls reopen when you have time to add some.
ENH: add method='dense' to rank
State: closed

Addresses #6333.

(Let's see how it goes on Travis; last time there was some strange behaviour which I think turned out to be because of an uninitialized variable which was behaving differently on my system than on the buildbots.)


Labels:

Comments:
looks good
pls add a release note entry (and same in v0.14 if u want)

then good. 2 go

are their frame tests at all? 

Yeah, `test_rank_2d_tie_methods` loops over every rank method and checks the frame behaviour.  Useful, too-- one of my early versions was resetting the count at the wrong place, and so it was ranking the whole flattened frame rather than each column.

ahh ok makes sense

perfect pls just add release notes and can merge

So does this go under "Improvements to existing features" for 0.14.0?

that works

It is my considered opinion that git is sufficiently complicated that having a tutorial reference open while using it is useless, because there is always some state I'm in which prevents the command from working as it's supposed to.  And this is a fact which beginners have no way to know until they've done it and made things worse.

Right now the last the last three log events in my local branch add_dense_rank_v5 are (1) the original commit, (2) 7ec7662's commit, and (3) a merge which was introduced when trying to follow instructions to avoid the "Updates were rejected because a pushed branch tip is behind its remote" error.

hmmm...I avoid merging (except when actually merging other people commits)...

my workflow is something like this:

update master 

```
git checkout master
git pull --rebase
```

```
git checkout branch
git pull --rebase
```

fix conflicts (if needed)

```
git add --all
git rebase --continue
```

add commits

rebase when needed to reorder/squash

```
git rebase -i <commit before this branch>
git push myfork thisbranch -f
```

you can just delete that merge commit FYI (I am going to remove it when I merge you in a moment anyhow)

merged via https://github.com/pydata/pandas/commit/61b141bc934c7e234f9675234a578f043f3bbdd1

thanks as always!
remove semicolon from CREATE TABLE legacy template
State: closed

It's not necessary and causes some DB engines to choke (e.g., Impala)


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jorisvandenbossche ?
@hayd 

is this covered by new sql stuff?

In principle we don't want to add support for more databases to the legacy code (that was the purpose of the new sqlalchemy based functions), but for such a small change this is OK for me.

ok...prob untested right? go ahed and merge
DOC: show users how to emulate R c function with iloc slicing and r_
State: closed


Labels:
name: Docs

Comments:
Nice comparison! 
Just a remark, you didn't really give the pandas alternative for your first case (to access columns by name), so maybe add that? (`df[, c("a", "c", "e")]` -> `df[["a", "c", "e"]]`)

@jorisvandenbossche Thanks! I'll add that.

@jorisvandenbossche Any more comments?

Nope, thanks! Maybe just check tomorrow the rendered dev doc to see if everything looks good

FYI the docs builds almost immediately after it's merged

http://pandas-docs.github.io/pandas-docs-travis/
ENH: Allow custom frequencies
State: closed

Allows custom frequencies other than business day in testing utilities. Keeps the old default.

closes #4541


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jseabold you are closing #4541 with this! excellent!

merge when ready

Ok, I don't like to merge with pandas usually to make sure other eyes get on it, but I'll do this one it's simple enough. I'll edit the commit message to close that issue.

I left custom frequencies for PeriodIndex and friends for another PR...

gr8
BUG: Fixes error in DataReader when getting "F-F_Momentum_Factor" data
State: closed

Hopefully I didn't screw-up using git and github. This PR fixes the minor issue outlined in `GH6460`. I didn't write any new tests but I added the "F-F_Momentum_Factor" data request to the existing tests of the famafrench data reader.

closes #6460


Labels:
name: Bug

Comments:
loooks great! can you add a release note in doc/source/releast.rst, reffing the issue 6460.thanks

I think I can do that.

Updated, I think

merged via https://github.com/pydata/pandas/commit/0eb38184f18639089f6a41fa21812eff17c0b672

thanks!

FYI in the future I usually rebase the commits and squash them together if they are small

but no biggie
DOC: Added entries into ecosystem.rst
State: closed

Added some entries to ecosystem.rst.  Basically just searched Nullege for pandas.  There was a lot more, but thought this made a good start.


Labels:
name: Docs

Comments:
Had to rebase to avoid unrelated test failures in Travis.

@jorisvandenbossche what do you think?

@jreback I don't know any of these packages, so difficult to assess if they should be included in the list (if they are 'substantial' as noted in the text). 

cc @dartdog any thoughts on these packages?

closing...pls put on the mailing list and see if these are common; if so, reopen and will reconsider