ENH: Add custom descriptors (such as dtype, nunique, etc.) to Styler output
State: closed


- [x] closes: #43875 


![Screen Shot 2021-10-24 at 08 52 28](https://user-images.githubusercontent.com/24256554/138583994-6a8debd6-6b8a-4fbb-a697-536b5a6b3590.png)

<details>
<summary>Former example...</summary>

![Screen Shot 2021-10-05 at 23 37 31](https://user-images.githubusercontent.com/24256554/136106494-c770db25-5598-4a6b-8708-1d2654eac8e1.png)
</details>



Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Hello @attack68! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-01-28 20:34:09 UTC
umm aren't these just the names of a multi-index on the columns?

> umm aren't these just the names of a multi-index on the columns?

No, the method is creating additional header rows, which return values based on the column data. The columns index is ["A", "B"]. Or was that not your question?
@jreback what do you think here? I think this is quite a useful feature for data exploration, and I can see how users might adapt to using it rather than the more generic `df.describe()` which often doesn't contain what you want and has some stats you don't need.
![Screen Shot 2021-11-16 at 18 24 27](https://user-images.githubusercontent.com/24256554/142034998-cd76dea6-2ed1-4f8e-a057-0bbd5e31db17.png)



I c so this is creating *arbitrary* summarizations actually.
cc @pandas-dev/pandas-core any thoughts here.
Feels like it should be aligned with `describe` a lot more closely.  The implementation feels quite clunky, hard to discover, and requiring a relatively high amount of code (e.g., with multiple lambdas to get a few rows).  Why not let rows from `describe` be prepended rather than relying exclusively on UDFs?
> Feels like it should be aligned with `describe` a lot more closely. The implementation feels quite clunky, hard to discover, and requiring a relatively high amount of code (e.g., with multiple lambdas to get a few rows). Why not let rows from `describe` be prepended rather than relying exclusively on UDFs?

I'd suggest some of the described clunkiness might be fitting the calculated values into the rendering process/dict that Styler uses to parse. Even if values were taken from `describe` they would still need to be fit into this structure similarly?

The core part of the requests/issue put forward here is that `describe` contains either more info than wanted or not enough, and you can't customise it. And since Styler is a visualisation tool you want the results to display consistently with the other Styler settings. (take a look here https://github.com/pandas-dev/pandas/issues/44751#issuecomment-986287632)

If you are suggesting expanding `describe` to be more customisable, and then reference that from Styler, that could work, if we pass through the kwargs and retrieve the values and then render them. But that will take more time and effort to go through all review processes, imo.
> I'd suggest some of the described clunkiness might be fitting the calculated values into the rendering process/dict that Styler uses to parse. Even if values were taken from describe they would still need to be fit into this structure similarly?

It would be a more user friendly API if users could pass the summary measures by name for any remotely common name, e.g., [`"mean", "std dev", "skewness", "kurtosis"] is more user friendly than writing and passing 4 lambdas. 

> The core part of the requests/issue put forward here is that describe contains either more info than wanted or not enough, and you can't customise it.

Maybe the solution is to customize `describe` and then to use the customized `describe` prepended on a `DataFrame`?

> If you are suggesting expanding describe to be more customisable, and then reference that from Styler, that could work, if we pass through the kwargs and retrieve the values and then render them. But that will take more time and effort to go through all review processes, imo.

Ultimately this is just a prepender to a styled `DataFrame` that applies some particular markup to the prepended values. It should be easy to prepend any second `DataFrame` to an existing `DataFrame` and then to style the output.  That DataFrame could come from `describe` or any other function that returns a `DataFrame` with the correct columns.  Limiting the interface to only lambdas makes it very bespoke to me. 

> It would be a more user friendly API if users could pass the summary measures by name for any remotely common name, e.g., [`"mean", "std dev", "skewness", "kurtosis"] is more user friendly than writing and passing 4 lambdas.

> Limiting the interface to only lambdas makes it very bespoke to me.

I think you only reviewed one example. Strings, as you describe, are also accepted - you dont have to use lambdas. Shown in docs in original post https://github.com/pandas-dev/pandas/pull/43894#issue-1016904508 and also as response to the issue https://github.com/pandas-dev/pandas/issues/44751#issuecomment-986287632
> Ultimately this is just a prepender to a styled `DataFrame` that applies some particular markup to the prepended values. It should be easy to prepend any second `DataFrame` to an existing `DataFrame` and then to style the output. That DataFrame could come from `describe` or any other function that returns a `DataFrame` with the correct columns. Limiting the interface to only lambdas makes it very bespoke to me.

Prepending and styling the result will break all the exsiting functions, like builtin highlighters since there is no way to separate the summary statistics from the data. It would also make the hiding functions more unwieldy, and the formatting might be more difficult too, since columns might lose dtype consistency.

I think potentially using the values from the `describe` DataFrame could work, but only within the Styler rendering process.


This has stalled, some of the feedback was based on the misunderstanding that one had to use lambdas when this is not true. Appreciate another look.
@mroeschke this stalled a while back. As the original reviewer, care to revive it?
I can't provide a review but I can say this functionality would be very useful. My work currently gets functionality like this through extending the styler with our own subclass and jinja template. This would save us a ton of work.

Really appreciate all the styler work you've done @attack68 
@jreback what you think here? OK to merge here on green? I can extend the docs as an add on?
no this still has questions 
@bashtage I believe the questions jeff is referring to are yours, do you have time to review?
I am still -1 here. This is basically completely changing the way things work. We *already* have .describe() to do exactly this. If you want to add `.describe()` and then allow it to be more generic ok there. BUT then this adds these as header summary rows, again completely changing things.

I don't have a good recommendation for how to reconcile this. I support if allow a 'header' table would be more ameanable. This just feels tacked on with a lot of bespoke things going on.
> I am still -1 here. This is basically completely changing the way things work. We _already_ have .describe() to do exactly this. If you want to add `.describe()` and then allow it to be more generic ok there. BUT then this adds these as header summary rows, again completely changing things.
> 
> I don't have a good recommendation for how to reconcile this. I support if allow a 'header' table would be more ameanable. This just feels tacked on with a lot of bespoke things going on.

I think I get what you are saying. I think generalizing `.describe()` would be a smart move, but keep this api, have the actual stylization call the new `.describe()`. I'm of the opinion that the addition of the data descriptors to the header does belong on `styler`, but I get your concern of having this _extra_ describe functionality be under `styler` rather than under `.describe()` (if that's indeed what you are talking about).

What I don't think should been done is have these generalized functionalities moved to `.describe()` and then have api be that the end user call `.describe()`  and be forced to pass the resultant  `DataFrame`/`Series`  to some kwarg on `styler`.
@bashtage @jreback  @mroeschke, this PR stalled due to difficulty in finding a way forward. I have tried to refactor it addressing some concerns above in an alternate #46105. If you have time please comment on the alternative.
I'd be interested in hearing a more detailed response to the opposition of this. The styler is an end use feature, so I'm not sure I understand the issue with there being data descriptors in the header or in a separate footer. 

I understand and agree to the issue with duplicating computation logic but not with adding extra rows to the header. Again it's end use, where further computation isnt the primary purpose. 
api considerations are always paramount - the goal is consistency and simplicity 
not convering every possible case

will look soon
There is a Separation of Concerns argument here: 1) the part to generate the descriptor data & combine with the original data, 2) the part to stylize the result. Without this feature today it's maybe roughly equivalent to (?):

```
result = pd.concat([original_data.astype(object), descriptor.astype(object)])
output = result.style.(...)
```

Maybe there should be a better way to do 1 (that's why `describe()` was brought up, `pivot_table` has similar functionality) 
@mroeschke solution, with the concatenation, might work but it then becomes harder to index the original data, export the styler for more reuseable styling, and apply the builtin formatters since you then have to exclude the concatenated rows from your subset.

I have used his solution before for a "Total" row and it is cumbersome but worked.
I've some to the point where the LOC is now small enough that it is worth including for the flexibility it brings.  The hard part is always expanding the API as pandas as become more mature, as this requires some effort (although I would rate this as highly stable, and so low maintenance).

Ideally, there would be a way to find out if this feature was in fact widely desirable before committing to it for a fairly long period of time. 
> I've some to the point where the LOC is now small enough that it is worth including for the flexibility it brings. The hard part is always expanding the API as pandas as become more mature, as this requires some effort (although I would rate this as highly stable, and so low maintenance).
> 
> Ideally, there would be a way to find out if this feature was in fact widely desirable before committing to it for a fairly long period of time.

Thanks @bashtage for the review and I agree with all your points, but I think Jeff was very keen on completely separating the calculation and thereby reducing the LOC much more, which I also agree with, but I was struggling to think of a way.

Finally, I recently had another thought and managed to completely reengineer this with the execution into less than 20 LOC, with better formatting and HTML styling built in, so I'm closing to close this and leave the superior, #46105, open. (thanks also @mroeschke for review)