BrandonFors/pandas-github-issues huggingface dataset:
CLN: added io.api for i/o importing functions
State: closed

closes most of #3411 (all except the deprecation of `from_csv`)
- moved excel functionaility out of io.parsers to io.excel
  added read_excel top-level function
  aliases from pandas.io.excel
- added read_stata top-level function, to_stata DataFrame method
  aliases from pandas.io.stata
  removed read_dta (replace by read_stata)
- added read_sql   top-level function, to_sql DataFrame method
  aliases from pandas.io.sql

DOC: doc updates for all the above and intro section to io.rst


Labels:

Comments:
@wesm pls give this a once over

@y-p @wesm ok on this?

IIRC, users rely on ExcelWriter directly to write mutliple tabs to the same file, 
these changes would break (at least) that, no?

nope...its still there (and still exported by default)

I must be missing something, how is this not a breaking change?

``` python
In [1]: from pandas.io.parsers import ExcelWriter

In [2]: from pandas.io.parsers import ExcelFile

In [3]: pd.__version__
Out[3]: '1.11.0.dev-6d08f1e'
```

(Note, I just pushed a change to master turning 0.12.0->0.11.1)

This PR:

``` python
In [1]: from pandas.io.parsers import ExcelFile
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-a9154a289dc3> in <module>()
----> 1 from pandas.io.parsers import ExcelFile

ImportError: cannot import name ExcelFile

In [2]: from pandas.io.parsers import ExcelWriter
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-39a586c0fd73> in <module>()
----> 1 from pandas.io.parsers import ExcelWriter

ImportError: cannot import name ExcelWriter

In [3]: pd.__version__
Out[3]: '0.12.0.dev-f7ad3c3'
```

yes, that is an API change, let me see if I can do a warning on that, or let it break?

in any event, the top-level namespace

```
import pandas as pd
pd.ExcelWriter
```

succeeds

I updated to give a warning if you try to use an `ExcelWriter/ExcelObject` imported from pandas.io.parsers

do you know if its possible to do this on import though?

e.g.

```
from pandas.io.parsers import ExcelWriter
ExcelWriter('foo.xls')
```

the warning won't print till the `ExcelWriter` line; any way to actually have it warn on that particular import?
some kind of import hook?

Scratch that, Obviously everyone will get the object from the toplevel namespace. 

actually, I think you have a valid point (as the docs _did_ originally have you import from `pandas.io.parsers`)

If you think so, then this doesn't belong in a minor release. 
The semver contract is "if you upgrade a minor release, your code won't break", not "you'll get a warning".

Other then that, a welcome cleanup of an increasingly crowded space.

ahh...but I have made that so, you WILL get a warning (and your code wont' break)

ok....will proceed then...

@wesm any comments?
ENH: enhance set_option syntax
State: closed

closes #3667.


Labels:

Comments:
who says my schemes are evil :)
well...maybe a little

k removed it sorry for the unnecessary stuff

@y-p this ok?

rebased

a7d404ca9dc1a2869b4c49bb46e5c5385452955f is the latest

the original reason if you're interested was because [`ast.literal_eval`](http://docs.python.org/2/library/ast.html#ast.literal_eval) was exactly what was needed: a safe eval that only evaluates basic python objects

The commit subject for a7d404c says it's concerned with the test suite, which Is why review is confusing.
with squashing, look fine to me.

oh sorry ok i will be more careful with my commit messages

squashed

@jreback this has been squashed and is ready to go.
API: deprecate DataFrame.interpolate
State: closed

closes #3582.


Labels:

Comments:
can you also add a deprecation if `interpolate` is called? (done a slightly different way, but look in core.categorical how Factor was deprecated; and in the warning message point to fillna/replace)

thxs

great....i'll merge this....pls add a new issue (as a reminder to do this), and i'll tag it

txs

shall i add the appropriate issue to v0.11.1.txt and RELEASE.rst?

oh wait don't merge yet i had to `grsh` and now i must rewrite the notes. :|

yep...rel notes (see how I did for tupleize_cols).....and v0.11.1 announcing that we are deprecating (in the API changes section)

perfect this is basically just adding the warning and the v0.11.1, letting everyone know its going to change

this ready right?

yup
scatter_matrix bug
State: closed

Currently scatter_matrix passes all unmatched keyword arguments to both scatter and line subplots (off-diagonal plots are scatter, diagonal plots are discrete or continuous histograms). I needed to color the points on the scatter plots but could not pass a "c" argument or pandas would try to pass the "c" argument to the line plots and barf. This change scratches my itch and doesn't break backward compatibility.


Labels:

Comments:
Great. This will close #3267.

Please add a note to RELEASE.rst, will merge for 0.11.1
Rework display logic again.
State: closed

v0.11.0 introduced last-minute changes to the way the dataframe display logic works,
and they haven't proven themselves. The churn is unforunate, but if it's broken,
fix it. 

`height` is deprecated.
max_rows solely determines the amount of rows displayed.

rebased on top of #3657.

Merging the previous PR and subsequent last minute fixes so close to the release 
contributed to the problems only being discovered after the release.
So please test and report if this works for you, in terminal, qtconsole or notebook,
since the 0.11.1 release is so close.

cc  @hayd , @lodagro ,@jreback 


Labels:

Comments:
I did this in an xterm

```
df = DataFrame(randn(100,20))
pd.set_option('display.max_rows',200)
```

worked

but then

```
pd.set_option('display.max_columns',10)
pd.set_option('display.max_info_columns',10)
```

still displayed the full columns, should this have display the info display?
(all other options were default, and screen width was 300)

as an aside, maybe I just don't understand but why do we have both `max_columns` and
`max_info_columns?`

is there a time when you would have them different?

@y-p the deprecated message for `display.height` should be `max_rows`?

@jreback , try now?

max_info_columns is a limit for the summary view to prevent excessive
delay. max_columns (+ expand_frame_repr) controls whether summary view is invoked.
you might have max_columns =20, but max_info_columns = 1e6
in theory.

Update:
`width` is only used to determine the wrapping when `expand_frame_repr` style is used.

seemed to work, I had `display.expand_frame_repr` set to the default (`True`), which essentially `overrides?`  `max_columns`/`max_info_columns` in a terminal? (IOW, I couldn't force summary view until I set `expand_frame_repr` to `False` (along with `max_columns=10,max_info_columns=10`)

2 feature requests:

1) `pd.set_option('display.max_columns=10,display.max_info_columns=10`)` to set multiple options? or too cumbersome
2) I think a doc outlining several common use-cases (and how to achieve) them may be in order

re expand_frame_repr:

```
In [13]: pd.describe_option('expand_frame')
display.expand_frame_repr: [default: True] [currently: False]
: boolean
        Whether to print out the full DataFrame repr for wide DataFrames
        across multiple lines.
        If False, the summary representation is shown.
```

doesn't that match the description?

re pd.set_option, option_context accepts option_context(pat,val,pat,val,...) we can add support for that,
open an issue for 0.12?

Would welcome your help on examples for the doc.

@y-p sorry for slowness on my end, i'll take a look soon at this

Verified that this fixed the "brokenness" I was experiencing in the IPython notebook (i.e. not being able to get the HTML view unless line_width was increased). Going to merge now and close the loop

Thanks again, team, as always
BUG: Non-unique indexing via loc and friends fixed when slicing (GH3659_)
State: closed

closes #3659

This is if you try a non_monotonic selection on a non_unique index (a mouthful)!
The reason is we cannot determinate a proper start/end point on what to include

```
In [11]: df = DataFrame({'A' : [1,2,3,4,5,6], 'B' : [3,4,5,6,7,8]}, 
           index = [0,1,0,1,2,3])

In [18]: df
Out[18]: 
   A  B
0  1  3
1  2  4
0  3  5
1  4  6
2  5  7
3  6  8

In [12]: df.loc[1:]
KeyError: 'cannot perform a slice operation on a non-unique non-monotonic index'
```

On a non_unique, but monotonic index, however, slicing works normally
(notice, since we are using loc, that both endpoints ARE included)

```
In [13]: df = DataFrame({'A' : [1,2,3,4,5,6], 'B' : [3,4,5,6,7,8]}, 
       index = [0,1,0,1,2,3]).sort(axis=0)

In [14]: df
Out[14]: 
   A  B
0  1  3
0  3  5
1  2  4
1  4  6
2  5  7
3  6  8

In [15]: df.loc[1:]
Out[15]: 
   A  B
1  2  4
1  4  6
2  5  7
3  6  8

In [16]: df.loc[1:2]
Out[16]: 
   A  B
1  2  4
1  4  6
2  5  7
```


Labels:

Comments:
@wesm pls give this a once over

I was making this more complicated that it was. Solution is sort of trivial (just needed to raise an exception when get_loc returns a boolean array rather than a slice, indicating index is non-monotonic and hence not slice indexable)

that is a mouthful
ENH: special case HTML repr behaviour on ipnb GH3573
State: closed

#3573, and SO question mentioned in #3541


Labels:

Comments:
@y-p Thanks! Do you have an Amazon Whishlist, where I can send you something?

Thank you, @JanSchulz, that's really not necessary. When I can fix a bug and
help out a pandas user, I'm good. No further thanks needed.
DOC: add doc for reading from DataFrame.to_html
State: closed


Labels:

Comments:
ok to merge?

just added more notes, so yep

dont' use assert_frame_equal in the v0.11.1...instead print out the table
I would not use `df_list,` maybe `alist` or something (you note  that the return type is a list is good and would leave that in though)
e.g. `alist[0]` (or `alist[0].head())` if its too long

perfect!

fyi....since its ipython, the `print` is not necessary....

cool. i figured. pre-ipdb debugging habits die hard!

hmm....can you rebase on top of master?

Yeah sorry. Will do when I get to a machine... 

@jreback rebased.

something failed......

https://www.travis-ci.org/cpcloud/pandas/jobs/7363117

yep saw it i'm on it, i'm getting a different error locally about recursion depth...

prolly cause i changed the default of infer types...and thus everything is evil object arrays

@jreback any changes to assert_frame_equal recently? not made by me...

ugh why does `assert a == b, "%s != %s" % (a, b)` not cause a recursion error but `assert a == b, "{0} != {1}".format(a, b)` does?!

I think @y-p changed something...didn't seem to affect me

(btw you can NOT have it compare dtypes, there is an option)

but I think you should have infer_dtypes=True by default, it makes sense
(or at least maybe do floats/ints by default), even dates really
people can always turn it off

ok...the example of reading in the output of `df.to_html()` will fail because `DataFrame(['a']).convert_objects(convert_dates='coerce')` thinks `'a'` is a date. i will just use soft conversion then

yeh...dates are tough (though coerce will only convert if at least 1 item is a date)......(if all are not dates then it will leave it)

wait but why is `'a'` considered a date?

what is in the series? `a` (alone) shouldn't trigger it

if u just do

``` python
s = Series(['a']).convert_objects(convert_dates='coerce')
```

`s` now has a date. I think this is a bug...

evil! I can see 'today' being parsed to today, but `a`????

```
(Pdb) import dateutil
(Pdb) dateutil.parser.parse('a')
datetime.datetime(2013, 5, 21, 0, 0)
```

@jreback travis is now all green

ah yes when parsers become too powerful and take over the world! mwahaha!

it's because `'a'` is a token for `'am'`

and also the default of just returning the current time kicks in since the parse of `'a'` returns `None`.

kind of insane if u ask me...

`a,m,p,t` are the offenders (I check them all now).....so odd, you would think you could turn this option off

see #3671

ok...merging.....after my PR you can decide if you want to turn convert_dates back on (or even make it a separate option to `read_html`)

@cpcloud if u have a chance
can u do a pr for read/write html in main docs (your read examples in v0.11.1 are prob good start)
at some point want to go thru the options that u allow a bit more in a doc example

and I don't think any docs for to_html

thxs

@jreback btw should the `to_*` collection of methods allow `str` in the `buf` argument (a file name)? i'm surprised that no one has raised this yet...
ENH: Allow for custom variable/value column names when melt()'ing
State: closed

ENH, CLN: When melt()'ing, allow for specification of custom variable and value names of the resulting DataFrame. Also fix indentation in melt() example to reflect actual output.


Labels:

Comments:
can u add a test and release notes mention (doc string example in v0.11.1 if u want too)
thxs

can you connect travis? see CONTRIBUTING.md for instructions....

and for a couple of the tests can you do something like:

```
result = melt......
expected = ......
tm.assert_frame_equal(result,expected)
```

to actually compare values and indicies (in this case its actually important as you are provide
some new functionaility that changes the column names, need to have a test for that)

similar to how test_pairs tests things (but doesn't have to be extensive....just a couple
of cases to make sure that melt is doing what you ask in various scenarios)

e.g. empty frame
incorrect frame
etc..

thanks

Thanks for your helpful feedback! I'm completely new to Python, testing, and Git(hub) :) I will try to improve my contribution. Stay tuned!

@hoechenberger see if you can connect travis on this....thxs

I will look into this tomorrow, thanks! :)

Now I'm wondering whether, as part of a good practice, I should create my test DataFrames manually or via methods from pandas.util.testing? I see both methods used across the pandas testing code...

you do either - 

whatever works for testing te feature u r doing
you also might find makeCustomDataFrame useful (in testing as well)
since u r testing reshaping

I think I sort of srewed the branching here. But eventually got things merged back into the master branch. Travis-CI is currently testing the code, I'm excited.

to avoid pulling in the merge to this branch, do this:

```
git br this_branch --set-upstream master
```

then

```
git checkout master
git pull
```

updates master

```
git checkout this_branch
git pull --rebase
```

will pull in any changes w/o creating a merge commit
you might need to fix any conflicts 

separately, pls rebase this to 1 or more commits (and you can eliminate the merge commit too)

```
git checkout this_branch
git rebase -i <commit prior to your branch>
```

remove the line of the merge branches
you can `s` to meld commits together

when done

```
git push origin this_branch -f
```

will replace all of the commits on this branch with the new ones
(with out the f it appends the newest commits)

Thanks! I think it's finally in a useful shape, no? :)

yes...that looks good.....once travis finishes will merge

thanks

git is an animal...once you tame it, its your friend :)

Great, thanks! I'm crossing my fingers ;)
ENH: add Series.str iterator
State: closed

closes #3638.


Labels:

Comments:
release notes!

ah crap my bad i always forget about RELEASE.rst :s

done

oh bollocks `assertIsInstance` is not a method in 2.6 :(

@jreback this can be merged. quite a few warnings in `RELEASE.rst` about unknown targets. want me to fix those up?

are these about incorrect gh numbers that don't map to issues? (e.g. are wrong)?

maybe do a separate pr for that

issues r not wrong they r just not mapped to a link, e.g., `GH3504_` doesn't have the corresponding `.. _GH3504 https://github.com/pydata/pandas/issues/3504`.

further motivation for me 2 write the darn sphinx ext

ne way i will clean up `RELEASE.rst`, i.e., get rid of non spurious warnings. hope this will not cause 2 much git merge conflict headaches...

merged..thanks!
single example for release notes
State: closed


Labels:

Comments:
@jreback the requested example for release notes

@cpcloud that must have been a typo, I meant _simple_.....just put whatever you think a user would want to know

@jreback there are two examples in the rls notes now: 1 regex and 1 string (both from the linked-to docs), does that suffice? i also fixed up some typos in the examples and added a more thorough test for nested dicts of regexs

@cpcloud this looks good.....i'll leave open a day or 2...(I always tend to think of something to change 1 second after its merged)....:)

Sounds good.
On May 17, 2013 8:58 PM, "jreback" notifications@github.com wrote:

> @cpcloud https://github.com/cpcloud this looks good.....i'll leave open
> a day or 2...(I always tend to think of something to change 1 second after
> its merged)....:)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3639#issuecomment-18092823
> .
BUG: Reindex data if plotting time/period index (GH3601)
State: closed

Just wanted to push this fix to see if it looked OK and let the Travis build start.  

If it looks ok, i will add something in the release notes and try to write a proper test.


Labels:

Comments:
Added a test and update release notes.  squashed into one commit

@jreback, ok to merge this?

@dieterv77 looks good to me

Since the datetimeindex supplies the labels here, why it it mandatory
to plot it in sorted order? at the very least, what about ascending/descending order?

Hardcoding a sort fixes one problem (GH3061), but creates another.
Previously, the user could opt-in (by sorting himself), but now he can't opt-out.
This looks like a regression to me.

Also, @dieterv77 , please mark issues by the miletsone they go into, if you're merging
your own PRs.
ENH: read-html fixes
State: closed

Some updates and bug fixes. See release notes for more details.
- ~~`vbench` stuff~~ sort of pointless right now since we don't really have control over the speed of the parsing library
- ~~Figure out why `lxml` chooses to ignore things~~ reported a bug w/ example to lxml people
- ~~Figure out why `bs4`'s `thead.find_all(['th', 'td'])` parses differently than `lxml`'s `thead.xpath('.//thead//th|.//thead//td')` even when `lxml` is the `bs4` backend.~~ same as above


Labels:

Comments:
let me know when you need merging on any PR's

this closes #3606, right?

yep, that is fixed already. i might be able to get to the rest of this today, i know the 0.11.1 rls is due today...the annoyances of the parsing may have to wait tho or i might just open up the flavor argument to allow one of `'lxml', 'bs4', 'html5lib'` since `html5lib` is a bit more WYSIWIG than the others even if it's drastically slower.

the main issue is the import errors.....

that is also fixed in this.

u r talking about #3605/#3607 right?

yep...

take your time...btw

ok thanks. i'm working on a cmdline interface to store neurophys data and it's due tmrw so pandas may have to wait...

@jreback @y-p what do u think about removing the _pure_ `lxml` option (then options would then be `lxml` and `html5lib` which correspond to the backend of `bs4` to use)? i think i was a little gung-ho about `lxml` being fast, but in reality the best is `bs4` + `html5lib`. Even though it's very slow, it gives correct output where `lxml` does not.

@cpcloud I would rather see correct and slow then wrong but fast!

let's see **premature optimization is evil**

Can always add it back in 0.12 (or after) if you discover how to fix it. And you have the flavor option, so sort of 'easy' to add it. (course have to edit stuff to take it out...docs,install docs,docstrings...)

of course if there are cases where lxml can do better (and is correct), but bombs on other cases, then you could always raise on those (but that may be more trouble than its worth)

i think the xpath implementation of lxml might be broken... :( 

@jreback can i leave the code for `lxml`/`bs4` + `lxml` in `html.py`? i would just make only `html5lib` available until i either figure out the issue or decide that it's not worth it...

ok 

@jreback this ready 2 go as soon as travis passes.

that is odd. travis is not running arg

ah there we go

@cpcloud thanks...this is great...

I edited the v0.11.1 a bit (as this is new, just announcing it). I think an example is warranted. Maybe take a df, `df.to_html`, them read it in ? just to give an example of how to do it (i do this with read_csv a little lower in the file)

do a separate PR

see this:

https://www.travis-ci.org/pydata/pandas/jobs/7320947

I don't think travis was actually testing html5lib stuff....(I just added it in)
its taken out right now

add in ci/install.sh (right after bs4)....and test

going to put in a separate issue

ok.
ENH add date to DatetimeIndex
State: closed

Allows the date to be pulled out from a DatetimeIndex easily/efficiently.

Similar to #3180, which was for the time-part to be extracted.

_From SO question: http://stackoverflow.com/questions/16563552/pandas-fancy-indexing-a-dataframe_.

~~I haven't added this to release notes, as maybe too late for 11.1?~~


Labels:

Comments:
ok for 0.11.1 I think

Added in release notes (and fixed the immediate merge conflict!) :)
Categorical cleanup
State: closed

Make categorical a little more user-friendly and add some documentation.


Labels:

Comments:
Added a describe method as well.

looks good to me, marked for 0.11.1

To be consistent (and not have the deprecate warnings internally), a couple of minor changes

```
./doc/source/groupby.rst:
  609 : Categorical variables represented as instance of pandas's ``Factor`` class can
./pandas/core/frame.py:
 5427 :     ndarray corresponding to Factor levels
./pandas/core/panel.py:
   85 :     time_factor = Factor.from_array(time)
   86 :     panel_factor = Factor.from_array(panels)
./pandas/core/reshape.py:
  813 :     """ given a tuple of shape and a list of Factor lables, return the expanded label indexer """
./pandas/tools/merge.py:
 1193 :             levels = [Factor.from_array(zp).levels for zp in zipped]
 1231 :             factor = Factor.from_array(concat_index)
```

Should I just change everything to Categorical? My reasoning for the deprecation is that Factor conflicts with the use of the term in patsy where a factor is not a "factor" in the R sense. I believe @wesm and @njsmith decided that Categorical was unambiguous and preferred, and I tend to agree but don't have a really strong feeling about this.

I say yes (its not much to change), leave in `Factor` for 0.11.1 with your deprecate, and can remove in 0.12?

Ok, should be better. No deprecation warnings internally now, though I left the term factor in a few places where it's clear from context what it's referring to (I think).

@jseabold can you add a release notes & v0.11.1 mention?

thank you sir!
BUG:  Fix integer modulo and division to make integer and float dtypes work similarly for invalid values
State: closed

closes #3590

This is a numpy oddity that treats them differently.

```
In [131]: p = DataFrame({ 'first' : [3,4,5,8], 'second' : [0,0,0,3] })

In [132]: p % 0
Out[132]: 
   first  second
0    NaN     NaN
1    NaN     NaN
2    NaN     NaN
3    NaN     NaN

In [133]: p % p
Out[133]: 
   first  second
0      0     NaN
1      0     NaN
2      0     NaN
3      0       0

In [134]: p / p
Out[134]: 
   first    second
0      1       inf
1      1       inf
2      1       inf
3      1  1.000000

In [135]: p / 0
Out[135]: 
   first  second
0    inf     inf
1    inf     inf
2    inf     inf
3    inf     inf
```

Numpy does this (on integers), floats are as like above

```
In [3]: x
Out[3]: 
array([[3, 0],
       [4, 0],
       [5, 0],
       [8, 3]])

In [4]: x % 0
Out[4]: 
array([[0, 0],
       [0, 0],
       [0, 0],
       [0, 0]])

In [5]: x % x
Out[5]: 
array([[0, 0],
       [0, 0],
       [0, 0],
       [0, 0]])

In [6]: x / x
Out[6]: 
array([[1, 0],
       [1, 0],
       [1, 0],
       [1, 1]])

In [7]: x / 0
Out[7]: 
array([[0, 0],
       [0, 0],
       [0, 0],
       [0, 0]])
```


Labels:

Comments:
@jreback Darn u beat me 2 it :) This is nice, thanks.