ENH: Conditional HTML Formatting
State: closed

closes https://github.com/pydata/pandas/issues/3190
closes #4315

Not close to being done, but I wanted to put this here before the meeting. Maybe someone will have a chance to check it out.

~~http://nbviewer.ipython.org/github/TomAugspurger/pandas/blob/638bd3e361633a4c446ee02534e07b8a9332258a/style.ipynb~~

~~https://github.com/TomAugspurger/pandas/blob/stylely/style.ipynb~~

[latest example notebook](http://nbviewer.ipython.org/gist/TomAugspurger/82a53e11993469cb29a1)


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@TomAugspurger what's the status on this (looks interesting). do you have some issue references?

how generic is this

Haven't put any time in to it.

This is something that I could start in another library and be pulled in to pandas if people think it's useful.

> On Oct 11, 2015, at 10:44, Jeff Reback notifications@github.com wrote:
> 
> @TomAugspurger what's the status on this (looks interesting). do you have some issue references?
> 
> how generic is this
> 
> —
> Reply to this email directly or view it on GitHub.

I've updated the notebook here: http://nbviewer.ipython.org/github/pydata/pandas/blob/0f0ba429180c27b4ae3cb21f6b343db46af69397/style.ipynb

Would be curious to hear people's thoughts before I put too much more time into it. Specifically on
1. The API: I'm using method chaining, as suggested by Jeff. That helps with the keyword args.
2. The scope: Does is belong in pandas, or as a separate repo?

No need to read into the messy implementation yet.

@TomAugspurger I think also adding some text formatting things would be nice (e.g. float formatting) in the example

> I think also adding some text formatting things would be nice

Sure. That could be a keyword argument to the Styler constructor. Something like

```
s = Styler(df, formatter=lambda x: "{:>.2f}".format(x)).color_bg_range()
```

gives

![screen shot 2015-10-24 at 11 33 56 am](https://cloud.githubusercontent.com/assets/1312546/10711510/28fcc5fc-7a43-11e5-97c0-a2d27f7d270f.png)

Doing that at the constructor level separates the two types of styling being done. python's formatting for the string vs. the CSS property: values pairs.

actually I realize that should just use .round(2).style

but was thinking that an alignment filter might be nice in that case

eg

.round(2).style.align_right()

We can actually support non-conditional formattings (like `align` in this case) a bit better than having method per CSS property. Instead we'll just have a `.set_properties` method so you can do

``` python
(df.round(2)
    .style
    .color_bg_range(cmap='viridis')
    .highlight_null(null_color='red')
    .set_properties(**{"color": "#f8f8fa", "text-align": 'center', "width": '10em'})
    .render())
```

![screen shot 2015-10-25 at 6 33 59 pm](https://cloud.githubusercontent.com/assets/1312546/10718706/0117fce6-7b47-11e5-80b8-cd90c7d4c107.png)

We _might_ want to support a `selection` argument to that (which would be an `IndexSlice`), in case you want to style a subset.

I don't think u need the .render() any longer
as the _repr_html_ does this

course this would be straightforward to hook up to some ipython widgets .....

eg maybe a row and column slider for colors
s couple of radio buttons ....

Just pushed a quick update. Notebook: http://nbviewer.ipython.org/github/pydata/pandas/blob/a7d730f30d87027464cc1747b38a844ba8d09fe3/style.ipynb
Newest additions are `.set_properties` for setting static (non data-dependent) properties and hooking into our `pd.options.display.precision` for controlling rounding. Both of those are pretty nice to use.

Currently working on
1. supporting slices somehow / applying certain styles to certain columns / rows
2. Table styling.

Supporting slices is tricky because our state is a dict mapping `(row_position, col_postion) -> [styles]`. Whenever you do a selection you lose that position information. We can go from label to position with something like `get_indexer`, but that will only work reliably for Indexes with no duplicates (I think). I'll keep thinking on this.

Table styling is just additional styling applied to the table as a whole, instead of individual cells. Pretty easy to do, it'll probably just be a dictionary you pass in when making the `Style` object, or when you `.render`.

Should have some time on Saturday.

This is awesome. It'll be a step change for us in using IPython Notebooks to show data, rather than writing out to Excel and doing formatting there.

To put some weight behind the slices - there are a couple of concepts there:
- Which set of data to use as comparisons for a certain value. For example, in the [image](https://cloud.githubusercontent.com/assets/1312546/10718706/0117fce6-7b47-11e5-80b8-cd90c7d4c107.png) above, should the '1.0' value in the top left be at the bottom end of the range or the top, depending on whether it's compared to its table or its column.
- Having different styles for different slices. I think this would be really useful. Re the label -> position issue, just my $0.02, but this working only on unique indexes may be fine - generally if you're looking at output you want unique labels anyway.

@maximilianr thanks for the comments.

Re the first one, that coloring is done relative to the column. That's how `color_bg_range` (which will be renamed) is written. But we or the user could pretty easily write their own function

```
def color_table(df):
    # normalized df
    ...
    # fill colors
    colors = [ ]
    return DataFrame(colors, index=df.index, columns=df.columns)
```

and use that with `df.style.applymap(color_table)`. Or we could include that, maybe as an option on `color_bg_range`, maybe as a separate method.

> Having different styles for different slices. I think this would be really useful. 

Agreed, I'd say it's necessary. At the very least, I _have_ to implement it so that errors aren't thrown if you have a DataFrame with a non-numeric column :) I'm also leaning towards just punting on non-unique indexes for now.

I'll try to get some documentation relatively soon. I think things are to the point where the overall interaction is stable.

This is really cool.  My first impression is that the colors play too prominent a role in the styles that I see here.  They tend to dominate over the numerical information.  I wonder if more subtle shading / using less of the dynamic range would provide the same visual hints while not dominating the overall effect.

However, as a disclaimer, IANAD (I am not a designer)

Thanks. Agreed entirely about the colors. I'm 99% sure that there's a parameter you can pass to the matplotlib color converter or color map to control what percent of the range it uses (`vmin` and `vmax`). My first guess didn't work, and I just haven't done researched further yet.

It was a minor nit relative to the technical merits here.  Please do ignore for now.

I think you bring up a good point though. The One Thing I want to get right here is providing a flexible framework for people to build on. I'm hopeful that another library will be built that does more / better actual styling. Our defaults need to just be not awful :)

This is really great to see.  I have been doing a lot of this by hand.

Some thoughts:
- It would be good to color the headings so that categorical information can be identified.  
- The Styler and DataFrame can export to a dataframe than can be used for data visualization.  The visual attributes of the table and visualization can align with less code.
- Create the css from python objects where the key is a css property.
- Use data attributes instead of classes in the table.  This will make the table easier to reuse with client side tools.
  
  ```
   <td id="id" class="data row6 col1">0.121668</td>
   <td id="id" class="data" data-row="6" data-col="1">0.121668</td>
  ```

The DataFrame editor in Spyder IDE may be of interest as far as styling: https://github.com/spyder-ide/spyder/blob/master/spyderlib/widgets/variableexplorer/dataframeeditor.py

![arrayeditor](https://cloud.githubusercontent.com/assets/2096628/10825731/57c6387e-7e34-11e5-97b5-80c8e7980354.png)

Ok, now I'm really going to get documentation written soon, now that there's broader interest.

> It would be good to color the headings

I want this too. From a technical standpoint it will be easy to add. I haven't found a nice way to do this from a user's point of view yet. It's on my TODO list, but will be a little latter.

> The Styler and DataFrame can export to a dataframe than can be used for data visualization. The visual attributes of the table and visualization can align with less code

Can you expand on this? I'm not sure what you mean.

> Use data attributes instead of classes in the table

Good idea, I'll check out how that works.

Got slicing to work, but I've had to give up support for duplicates in the Index. I think that's a fair trade for now. ¯_(ツ)_/¯

[notebook](http://nbviewer.ipython.org/github/TomAugspurger/pandas/blob/27606e607242680453183e434962c06bee4bf1c3/style.ipynb)

And hooking up to widgets is easy and fun!

![out](https://cloud.githubusercontent.com/assets/1312546/10837410/29f94712-7e87-11e5-8f57-7ac5ed555991.gif)

Nice one!  It's cool to think about this eventually playing nicely with [pivottable](https://github.com/nicolaskruchten/pivottable).

Potentially, I’m not familiar with that. This will at least give you classes like the row or column number that you can select and style.

> On Oct 29, 2015, at 10:01 PM, Tony Fast notifications@github.com wrote:
> 
> Nice one! It's cool to think about this eventually playing nicely with pivottable https://github.com/nicolaskruchten/pivottable.
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/pydata/pandas/pull/10250#issuecomment-152397330.

Hi everyone, I've uploaded a first pass of the docs as a notebook here: http://nbviewer.ipython.org/gist/TomAugspurger/841a9dc2816853824ce1

I'll be iterating on that quite a bit, but it gives a good overview of what's capable. You should at least skim to the end :)

---

Speaking of docs, I think we'll need to do some tinkering with sphinx to get this working.

And my next biggish item to fix is styling of Indexes / column headers. My current best idea is dedicated methods `.apply_columns` and `.apply_index` to indicate that the result should be directed at the index / column. I'm very open to alternative suggestions though.

@TomAugspurger I think it would be intuitive to simply do

df.style.index.apply(...) or whatever (might need a slight modification to make this work, maybe a subclass) 

This is a fun one:

<img width="601" alt="screen shot 2015-11-03 at 7 32 13 am" src="https://cloud.githubusercontent.com/assets/1312546/10909213/0d4d625a-81fd-11e5-9317-46f6e5aea7d1.png">

@jreback what's the plan for this before PyData? Do you want to merge it into master (assuming I have it ready) or work off a branch?

I've got a few outstanding issues
- [x] gracefully ignoring non-numeric data with the builtin-stylers or more magically in apply/applymap/tee
- [ ] Think about how to render / decline to render large DataFrames. Right now if you call `.style` on a 100x100 DataFrame you'll wait a couple seconds, 1000x100 is 10-20 seconds.
- [x] tests, tests, tests
- [x] documentation, which could be hard. Anyone have ideas on the best way to get the HTML output of an object? Is there a way to trick the IPython directive to think it's in a notebook (cough @jorisvandenbossche  :smile:)? I'll do some research tonight. It looks like Bokeh does it with a custom directive and [sourcing another python file](http://bokeh.pydata.org/en/latest/_sources/docs/user_guide/layout.txt).

I am away this week
I have his branch setup as a kernel (neat trick actually) so will just update and run your notebook 

are you coming to pydatanyc?

so really just to have a cool notebook would be great!

going to show this as a future item in whatsnew in pandas

> are you coming to pydatanyc?

Unfortunately not. [This](https://github.com/jreback/PyDataNYC2015/blob/master/whats-new-in-pandas/style.ipynb) is your notebook? I'll check it out and make sure I don't break any of it / alert you if I do.

Ah, I see that's an older version. I'll put together a version that's less wordy / documentation and more just cool examples.

oh I am just going to update to your current branch and copy your example notebook in 

with whatever u have latest

it's just show and tell :)

Works for me!

> On Nov 3, 2015, at 8:32 AM, Jeff Reback notifications@github.com wrote:
> 
> oh I am just going to update to your current branch and copy your example notebook in
> 
> with whatever u have latest
> 
> it's just show and tell :)
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/pydata/pandas/pull/10250#issuecomment-153372186.