BrandonFors/pandas-github-issues huggingface dataset:
Switch default colormap for scatterplot to 'Greys'
State: closed

In #7780, I added colormap support to scatter, but choose to override the
default rainbow colormap (https://github.com/pydata/pandas/pull/7780#issuecomment-49533995).

I'm now regreting choosing 'RdBu' as the default colormap. I think a simple
black-white colormap is a more conservative and better default choice for
coloring scatter plots -- binary colormaps really only make sense if the data
is signed. 'Greys' is also the default colormap set by the seaborn package,
and @mwaskom has done far more thinking about colormaps than I have.

If possible, I would love to get this in before 0.15 is released, so we don't
have to worry about changing how anyone's plots look.

CC @sinhrks


Labels:
name: Visualization
description: plotting

Comments:
:+1: 

Fixed my failing test which was checking for the old default colormap -- everything is green now.

No objection here.

@shoyer gr8 thanks!
ENH: Add type conversion to read_stata and StataReader
State: closed

Added upcasting of numeric types which will default numeric data to either
int64 or float64.  This option has been enabled by default.

closes #8527


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design
name: IO Stata
description: read_stata, to_stata

Comments:
@jreback The big question remaining is the name: `preserve_dtypes=False` or `convert_dtypes=True`.

Also, the default behavior has changed, which might not be desirable (it has changed in the current PR so that it will convert to _preferred_ pandas types, e.g. any `float` to `float64` and any `int` to `int64`).

I like `preserve_dtypes=True` (the default will need to be this for 0.15.1).

@jreback Travis errored, but passes all tests  

I think it is ready if you don't see anything else.

I restarted that job
ping me when green and can merge

thanks
Added footer to read_html
State: closed

read_html neglected table footers. Although rare, some sites use the footer for data. First time pull request so hopefully not messed up.


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Needs a test.

Add it in `tests/test_html.py`.

It might be good to have a test that exercises the `<footer>` parsing on a very simple example (maybe even hand written HTML) and then another test with an example of some HTML from the wild that contains `<footer>`s.

Should probably also test an empty footer like `<footer></footer>`.

Ok thanks! I was going to add my local test to tests/test_html.py but found an error which meant I hadn't tested this modification properly (and now getting NaNs where there shouldn't be). Will keep working on it, and getting my head around how to use git.

I was getting `NaN` as my test (and real world example) used `<td>` rather than `<th>` inside `<tfoot></tfoot>` so it wasn't parsing correctly.

@mjsu 
- pls add a release note documenting this in enhancements in v0.15.2 (reference this issue)
- pls squash

@cpcloud will review after

Hmm, I think I have messed this up! (still very novice at git/github)

@mjsu ?

Yep?

@mjsu just pinging in u. This needs a rebase / squash and to add those items mentioned above.

I thought I did it. I wasn't quite sure what I was doing so may have done it wrong?

see here https://github.com/pydata/pandas/wiki/Using-Git

try

```
git rebase -i origin/master
```

squash to 1 commit

then

```
git push myfork thisbranchname -f
```

That done it? (slowly getting my head around this stuff, thanks for the help!)

good.

@cpcloud ?

@cpcloud pls have a look and merge if ok (you prob have to rebase it)

@cpcloud ?

looks good to me, needs a rebase looks like there's some merge conflicts.

@mjsu thanks!
ENH: Move any/all to NDFrame, support additional arguments for Series. GH8302
State: closed

closes #8302
- Move any/all implementations from DataFrame to NDFrame, adding support for Series
- Special case single dimension case in NDFrame, to handle numpy.any/all specific arguments
- Add assorted NotImplementedErrors in cases where arguments are ignored (including in preexisting _reduce functions)
- Move the other any/all implementations from IndexOpsMixin to Index, outside of Series’ inheritance tree


Labels:
name: Enhancement
name: API Design

Comments:
Hi, here is the separate PR for the _reduce changes: https://github.com/pydata/pandas/pull/8592

Ok, I think I've addressed the pending items.

Some tests are currently disabled pending merge of the _reduce changes I separated into the other PR.

pls add a release note v0.15.1 (put in API section for now, though that's getting reorganized). This is not really an API change, more of an internal change.

pls rebase/squash and I'll have another look.

Sure, there are some minor dependencies on the _reduce PR we've also been discussing (some tests are disabled). Since it sounds like we're closed to finishing up that PR, I'll plan to rebase/squash this one once the other is in master.

@staple ideally I'd like to see a SINGLE any/all that goes in NDFrame AND Index (so we have argument consistency). This is tricky of course. because you have multi-dim objects. I think this CAN be done though.

Ok, updated. Go ahead and take a look.

Ok, I made the requested changes.

@staple FYI this PR is really good. I am just being nit-picky! on some details.

@jreback, sure no worries!

can you fixup and rebase as discussed. e.g. remove _args,_*kwargs to Index funcs (and remove keep_dims tests). change test_logical_compat to test the functions are actually disabled on the various Indexes. thxs

Ok, see new commit.

Ok

@staple thanks! This was a bigger rabbit hole, but you got it done!
PERF: Slowness in multi-level indexes with datetime levels (part 2)
State: closed

Corrects some deficiencies noted here: https://github.com/pydata/pandas/pull/8544#issuecomment-58909397. Adds a test case and an additional vbench test.


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
@mtkni thanks!
BUG: inconsistent and undocumented option "converters" to read_excel
State: closed

Issue #8212 (first part): pandas.read_excel accepts an optional argument "converters"  (which is passed down to PythonParser) to convert single cells in columns with a conversion function. I documented this feature and added a try/except block to make it work in case some cells contain NaNs.

What's still missing is the full "dtype" argument, a la read_csv. That patch is somewhat orthogonal because it only works with the C parser, so I plan to implement it in a second step.


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO Excel
description: read_excel, to_excel

Comments:
usually start off a PR with tests first.

I added a test for this functionality.

ok, this looks fine
- need a release note in v0.15.0 (refere the issue you highlited above). as you are now allowing the converters argument to work with missing values. maybe adjust the docs a bit in doc/source/io.rst (excel section).

I added some docs there, not sure it's enough, open for feedback.

Hi, what's the current status on this? I think I implemented all you requested, do you think it's ready and can be merged?
No pressure, I am just trying to join loose ends...

ok, pls edit the docs a bit, rebase and squash.

Ok, edited docs, squashed, rebased, pushed (d0597b8).

looks ok to me

@jorisvandenbossche ?

Thanks!

Thanks! I'm happy since it's my first PR and I made a bit of a mess. I'll try to work on the full dtype integration, using the C parser.
PERF: Performance improvement for MultiIndexes with a DatetimeIndex levels
State: closed

Addresses #8543. The benefits are most pronounced when there are a small number of distinct datetimes relative to the number of rows in the index.

vbench results with `-r multi`:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
multi_with_datetime_level                    |   1.5767 |  46.8633 |   0.0336 |
reindex_multiindex                           |   1.1919 |   1.2113 |   0.9840 |
groupby_transform_multi_key4                 | 121.5626 | 123.5036 |   0.9843 |
frame_multi_and_st                           |  33.3813 |  33.5990 |   0.9935 |
groupby_multi_series_op                      |  11.2731 |  11.3403 |   0.9941 |
groupby_multi_count                          |   7.2227 |   7.2653 |   0.9941 |
groupby_multi_different_functions            |   9.5117 |   9.5397 |   0.9971 |
groupby_multi_size                           |  19.4443 |  19.4850 |   0.9979 |
frame_multi_and_no_ne                        |  55.7680 |  55.8747 |   0.9981 |
groupby_multi_different_numpy_functions      |   9.5197 |   9.5037 |   1.0017 |
groupby_transform_multi_key1                 |  65.6123 |  65.4763 |   1.0021 |
read_table_multiple_date                     | 148.9107 | 148.2257 |   1.0046 |
join_dataframe_index_multi                   |  15.1207 |  15.0447 |   1.0051 |
stat_ops_level_frame_sum_multiple            |   6.7097 |   6.6660 |   1.0065 |
groupby_multi_cython                         |  12.6956 |  12.5817 |   1.0091 |
frame_multi_and                              |  19.6610 |  19.4677 |   1.0099 |
read_table_multiple_date_baseline            |  68.1753 |  67.3580 |   1.0121 |
multiindex_from_product                      |  10.0873 |   9.9500 |   1.0138 |
groupby_transform_multi_key2                 |  45.2770 |  44.5527 |   1.0163 |
groupby_multi_python                         | 122.5337 | 119.1190 |   1.0287 |
stat_ops_level_series_sum_multiple           |   5.2770 |   5.1050 |   1.0337 |
groupby_transform_multi_key3                 | 788.8023 | 736.1453 |   1.0715 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
thanks!

Thanks for the quick response. Unfortunately I found two issues with my change:

1) Incorrect handling of missing values (nan vs NaT)
2) Possible performance degradation when dealing with a small subset of the index (e.g. index[:10].values).

I've addressed those in https://github.com/mtkni/pandas/commit/acb9b3390c8c7df8275b7a7463f34bc640a52fba and I'll submit a another PR shortly.
BUG: Plot with ``label`` would overwrite index name
State: closed

Closes https://github.com/pydata/pandas/issues/8494

I took the easy solution and just copied the series.

Alternatively we could attach a label property to the MPLObject, and check that wherever we check the index name.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
pls rebase and change the location of the release note

Rebased. I'll merge when Travis is green.
BUG: Dont add None to plot legend.
State: closed

Closes https://github.com/pydata/pandas/issues/8491

Building the docs now to check that everything looks ok.


Labels:
name: Visualization
description: plotting

Comments:
Seems like the default for the `legend` argument to `Series.plot` got switched in that plotting cleanup PR that merged stuff between DataFrame.plot and Series.plot.

Maybe the whatsnew is not really needed, as this was not a bug in the previous released version (or add the issue number to the whatsnew entry of the clean-up)

Right again. I removed the release notes entry.

thanks @TomAugspurger

@TomAugspurger I just noticed it isn't solved for all plots. See eg the the third plot on the doc page: http://pandas-docs.github.io/pandas-docs-travis/visualization.html#basic-plotting-plot

hmm, no None label for me.  This is that 3rd plot from a local build.

![df_plot_xy](https://cloud.githubusercontent.com/assets/1312546/4802379/05393e10-5e43-11e4-98db-887ca07626d8.png)

Do you get the None label locally?

Yes, I also get it locally (both on linux -> official docs are built there, as now on WIndows):

```
pandas: 0.15.0-16-g7012d71
numpy: 1.9.0
IPython: 2.1.0
matplotlib: 1.4.0
```

Ok, well I have a windows machine at work now. I'll check it out. Thanks.
DOC: Clean up Sphinx Issues, fix tabs, in Cookbook
State: closed

This is a follow on PR as a result of [seeing travis and sphinx render ~30 in line examples for the cookbook](https://github.com/pydata/pandas/pull/8288#issuecomment-58734824).

Continues to address #6918

Initial PR: https://github.com/pydata/pandas/pull/8288

PS - apologies for wanting to rush the first PR.  My evening opened up unexpectedly.  Feel free to merge this latest, or provide more feedback for me to work on later. I'm going off grid for 2+ days starting oct 10th 11 PM EST.

The latest rendering is [here](https://cloud.githubusercontent.com/assets/6062071/4601203/f3dcdc76-50f0-11e4-8466-4db698bdc3b4.png):

Edit : Removed inline view of humongous rendering, still available for viewing at link above.


Labels:

Comments:
[Latest rendering](https://cloud.githubusercontent.com/assets/6062071/4608241/4ee22ff2-5279-11e4-9169-e40b42834cb3.png), as of 42cde88

![cookbook5png](https://cloud.githubusercontent.com/assets/6062071/4608241/4ee22ff2-5279-11e4-9169-e40b42834cb3.png)

Go ahead and merge.  Unless somebody else speaks up, I'm done.  I want to see how github.io renders it.  I'll check it all, ONCE more, after that.  The finer details between sphinx versions and python versions are proving annoying for something like this.

ok thanks!

@jnmclarty Thanks a lot for this!

A question, with your expercience now of trying to do this, do you have specific points of attention regarding the issues you had with sphinx/python versions (differences between your local and the travis build)? If so, we could list them in the docs/wiki on building the docs.

@jorisvandenbossche w.r.t to your question, I'll see if I can put some notes in the wiki.

But, it might be worth editing the template for the pandas docs, where it says "Built with Sphinx X.X.X..." in the footer, to instead link to a page that renders something akin to:

``` python
import sys
import platform
import pandas
import sphinx

print('Version      : %s' % str(platform.python_version()))
print('Version tuple: %s' % str(platform.python_version_tuple()))
print('Compiler     : %s' % str(platform.python_compiler()))
print('Build        : %s' % str(platform.python_build()))

print('Environment Paths:')
for p in sys.path:
    print(p)

print('Pandas version: %s' % str(pandas.__version__))

print('Sphinx version: %s' % str(sphinx.__version__))
print('More: %s' % str(sphinx.version_info))

print ('Other Imported Modules:')
for m,p in sys.modules.items():
    print("%s - %s" % (m,str(p)))
```

@jorisvandenbossche  I put some stuff in the wiki.
BUG: fix CategoricalBlock pickling
State: closed

This should fix #8518.

Apparently, `NonConsolidatableMixIn.__init__` was copied from a pre-`BlockPlacement` version of `SparseBlock.__init__` which was incompatible with placement being a slice.  The current one is, so I moved it to NonConsolidatableMixIn class almost verbatim.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Categorical
description: Categorical Data Type

Comments:
can u add a 0.14.1 pickle as well for back compat testing (and add to the legacy pickle generation script) thxs

can you also add an explicity pickle of a `pd.Categorical` from 0.14.1, see here: https://github.com/pydata/pandas/blob/master/pandas/tests/test_index.py#L273
needs to go in `tests/data/pickle_name.pickle` to be picked up (better name clearly :)

and put this in a test in `tests/test_categorical.py`

@immerrr I pushed a commit to provide 014.1 pickle compat (didn't have a setstate for some reason). Need a few tests for this though. https://github.com/immerrr/pandas/pull/2

Yeah, thanks, it will probably have to wait for several of hours...

np

important that this is right

was an oversight 

@immerrr ?

On it. Had a blackout at home last night so couldn't do anything useful. 

~~I've added a test and cherry-picked your commit on top. I get a weird error:~~
~~...~~
~~I wonder if this happens because Index is no longer a subclass of ndarray...~~

_EDIT: disregard that, I've forgotten to use `pd.read_pickle` instead of `pickle.load`._

~~deleted~~

you can't use pickle.load(), see here: http://pandas-docs.github.io/pandas-docs-travis/io.html#io-pickle

use pd.read_pickle()

Squashed and added some final touches, have a look.

@immerrr looks good 2 me. ping when green.

@immerrr thanks! gr8 as usual!
ENH: axvlines - boolean option to parallel_coordinates plot
State: closed

Determines whether vertical lines will be added to parallel_coordinates plot. I was using parallel coordinates to make a seasonality chart - plotting years on top of each other (days/months of year on x-axis). With that many xticks, the vertical lines were overwhelming. It's possible to remove the lines after the fact, but without specific line identifiers, it isn't a straight forward process. However, preventing them from being added is a one-liner...


Labels:
name: Visualization
description: plotting

Comments:
@TomAugspurger ?

@TomAugspurger ?

Sorry I meant to reply earlier.  I'll take a look soonish. I'm a bit hesitant just because our parrallel coordinates habdling isn't the best. But I'll get back on this

@bjacobowski ok, sorry it took me so long to get to this.

Could you post some code with small example of what you were doing? Also a before and after picture would be great.

We'll also need a test in `pandas/tests/test_graphics.py`. Just something simple like the length of the `.lines` attribute on the returned axes being equal to the number of lines (so no extra `vlines`). Also a test that the vlines are added (by default).

That said pandas is trying to move its focus away from plotting since there are so many great python vis libraries. This is so minor that I'm happy to accept it. But if you're interested you could see if seaborn or bokeh or ggplot would accept a PR for parallel coordinate support.

@TomAugspurger Here's a basic example:
![parallel coordinates - axvlines example](https://cloud.githubusercontent.com/assets/7805755/4845197/4ca2e9c6-6042-11e4-92ed-a696d38aa952.png)
[seasonality.py](https://gist.github.com/bjacobowski/6c6d1ad0af8d50694765#file-seasonality-py)

@TomAugspurger what are we doing with this?

Let's get it merged with the understanding that any proper implementation of parallel coordinates should go into seaborn or ggplot or bokeh.

@bjacobowski would you mind adding a release note to doc/source/whatsnew/v0.15.2?

@TomAugspurger done

@bjacobowski can you try to rebase against master and squash this down to one commit. For some reason it's trying to commit the entire whatsnew/v0.15.2.txt. Checkout [the wiki](https://github.com/pydata/pandas/wiki/Contributing) or post if you have any trouble.

@TomAugspurger that work?

Perfect. Just going to wait until Travis is green.

Thanks @bjacobowski
DOC: Update ecosystem.rst: Add IDE, IPython, qgrid, Spyder, API, quandl, Out-of-core, Blaze
State: closed


Labels:
name: Docs

Comments:
I added a 'Graphical User Interfaces' heading.
- https://bitbucket.org/spyder-ide/spyderlib/pull-request/31/implementation-of-pandas-dataframe-issue/diff
- TODO: Add https://github.com/quantopian/qgrid

https://github.com/quandl/Python also pulls data into DataFrames. Maybe an (data) API section?

Done.
- GUI -> IDE
- IPython > qgrid
- APIs > quandl

Should be ready for review

Should be ready for review (pending build completion) ... `rst2html.py ecosystem.py` seems to work okay.

Moved IPython into IDE section and added a blurb per https://github.com/pydata/pandas/pull/8507#discussion_r18586499

... Don't know where to put Blaze: http://blaze.pydata.org/docs/latest/backends.html#backends

maybe make a new out-of-core section ..

fyi would move Vincent to the end of that section

lmk when pushed

Pushed.

May need to merge under https://github.com/pydata/pandas/blob/master/doc/source/ecosystem.rst#xray

rebase to master and repush pls

rebased and pushed

thanks!

pls review the built docs (once Travis is finished with them)

Thanks! Will do

https://pandas-docs.github.io/pandas-docs-travis/

https://pandas-docs.github.io/pandas-docs-travis/ecosystem.html
Raise more specific exception on concat([None])
State: closed

More specific exception allows for a safer, more explicit catch by a caller.
ConcatenationError follows the pattern of MergeError which is used in the same file when something is wrong with a caller's inputs to  merge().


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jorisvandenbossche you think this is an API change? seems completely back-compat 

Yeah, as far as I understand it correctly, this should be back compat (as if a user is checking for an Exception now, this will still work for ConcatenationError).

However, I do follow @shoyer here in the question if it is not more useful to provide eg ValueError. 
I know it would be more consistent with the rest of the file to use a custom error, but is that pattern used a lot elsewhere in pandas? (@jreback?) If we think something like ValueError is better, I think we should do that regardless of consistency with `merge`. 
And I personally find ValueError possibly more informative than ConcatenationError, as there are more things that could go wrong in the concatenation function than wrong input values.

I think `MergeError` could be changed to `ValueError`. Which is what this really is, either the arguments are incorrect (e.g. can't specify multiple conflicting options), or your data is not-unique.

So IMHO, I think let's make `MergeError` inherit from `ValueError` (I think this is back-compat), but put in a note that we are getting rid of it in 0.16? (or just drop it)?
and make this bare Exception into a `ValueError`

Updated to remove ConcatenationError and make ValueError more central, as suggested. I'll leave the deprecation of MergeError to you folks if that's OK. Not sure how you'd like to document/go forward with that kind of thing.

Can you sqaush your commits?

@jorisvandenbossche done!

@thatneat can you add a release note to v0.15.0 API changes (reference this issue, saying a `MergeError` will now specialize as a `ValueError` or something like that).

squash and I think good to go

merged via https://github.com/pydata/pandas/commit/ce79c80578708c32a161b9e109c8804ebbf5cea5

thanks!
DOC: Another semicolon fix v0.15
State: closed

Sorry for basically duplicate pr, noticed this one after.


Labels:
name: Docs

Comments:
np. ok, why don't I leave open if you have any more changes. pls ping when ready.

@jreback I went in and added ticks to a lot of identifiers. I tried to follow the convention of the rest of the changelog. Let me know if there are any major mistakes.

Also, do I need to squash these 2 commits? I got my commits down to 1 commit, but am unsure if I need to do anything else in git.

merged via https://github.com/pydata/pandas/commit/685eb0d73863562d56b1bf119a18709615e2739f

thanks!
DOC/BUG:Added Missing Semicolon to Release Note v0.15
State: closed

Missed a semi-colon, here's the fix.


Labels:

Comments:
yep, thanks

Dang, I just noticed a bunch more. I'll go ahead and fix them all and send it up in another.

ok, sure.
see here for how to rebase/squash: https://github.com/pydata/pandas/issues/8502