PERF: to speed up rendering of styler
State: closed

see https://github.com/pandas-dev/pandas/issues/19917#issuecomment-646209895

- [x] closes #19917
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Seems to be failing some tests. Is this supposed to produce an equivalent output?
Yes it does.
Test cases seems expect empty values because of the original code's 'misbehavior' that caused the slowdown.

For example:
https://github.com/pandas-dev/pandas/blob/3c959fc7f1feee185c32cc38d5d31bb8163ce807/pandas/tests/io/formats/test_style.py#L986-L1013

This entry can be removed from the expected test result in above code:
```
        (1, 0): [""],
```
The best way is to typically write benchmarks and run them in the performance suite.

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#running-the-performance-test-suite
As long as the new implementation produces roughly equivalent HTML then we can consider updating intermediate representation expected in the tests.
It does produce exactly same HTML. As you can see, the only things missed in intermediate steps are the empty strings.
Adding empty strings did not change HTML. The only chance that can be different (I have not actually seen anything like this but) is  only if the original code produced ```style=""``` which, even when omitted. will have no effect. 
OK, thanks. Can you update the tests then and add an ASV?
I can update tests but what is an ASV?
@jihwans : ASV refers to https://github.com/airspeed-velocity/asv

This directory is where you add such a test: https://github.com/pandas-dev/pandas/tree/master/asv_bench/benchmarks
Thank you all.
I guess it's going to take some time to figure out these.
I will make changes to test cases first.
I have no idea why this error occured with travis-ci/pr:
```
$ ci/run_tests.sh
xvfb-run pytest -m "(not slow and not network and not clipboard)" -n auto --dist=loadfile -s --strict --durations=30 --junitxml=test-data.xml pandas
ImportError while loading conftest '/home/travis/build/pandas-dev/pandas/pandas/conftest.py'.
pandas/__init__.py:34: in <module>
    raise ImportError(
E   ImportError: C extension: 'dtypes' from partially initialized module 'pandas._libs.tslibs' (most likely due to a circular import) (/home/travis/build/pandas-dev/pandas/pandas/_libs/tslibs/__init__.py) not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.
The command "ci/run_tests.sh" exited with 4.
```
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Commenter does not have sufficient privileges for PR 34863 in repo pandas-dev/pandas<br>

</samp>
ASV files was added
I'm sorry -- I clicked the request review button by accident.
Here's what ASV says:
```
       before           after         ratio
     [1ce1c3c1]       [8b1825a0]
     <v1.0.4^0>       <jihwans-patch-1>
-            178M             157M     0.88  io.style.RenderApply.peakmem_render(36, 1200)
-      68.5±0.7ms       19.0±0.1ms     0.28  io.style.RenderApply.time_render(12, 12)
-         645±3ms          149±2ms     0.23  io.style.RenderApply.time_render(12, 120)
-      6.54±0.04s       1.49±0.05s     0.23  io.style.RenderApply.time_render(12, 1200)
-         132±3ms       29.6±0.4ms     0.22  io.style.RenderApply.time_render(24, 12)
-       191±0.9ms       41.5±0.5ms     0.22  io.style.RenderApply.time_render(36, 12)
-      12.4±0.04s       2.34±0.08s     0.19  io.style.RenderApply.time_render(24, 1200)
-      1.24±0.02s          230±2ms     0.19  io.style.RenderApply.time_render(24, 120)
-      1.88±0.09s          321±3ms     0.17  io.style.RenderApply.time_render(36, 120)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

It's funny - I don't know why one of the case did not appear in the message (19s -> 3.22s)

```
jihwans-patch-1:
              ====== ============ ========= ============
              --                     rows               
              ------ -----------------------------------
               cols       12         120        1200    
              ====== ============ ========= ============
                12    19.0±0.1ms   149±2ms   1.49±0.05s 
                24    29.6±0.4ms   230±2ms   2.34±0.08s 
                36    41.5±0.5ms   321±3ms   3.22±0.03s 
              ====== ============ ========= ============
```
```
v1.0.4:
              ====== ============ ============ ============
              --                      rows                 
              ------ --------------------------------------
               cols       12          120          1200    
              ====== ============ ============ ============
                12    68.5±0.7ms    645±3ms     6.54±0.04s 
                24     132±3ms     1.24±0.02s   12.4±0.04s 
                36    191±0.9ms    1.88±0.09s   19.0±0.3s  
              ====== ============ ============ ============
```
It's also interesting to see that it actually consumed less memory.

I'm just curious - why does ASV compile pandas package again and again every time I run between "asv run" and "asv continuous", even though the versions I chose remains to be the same?
I cannot figure out why this error occured:
```
Oh no! 💥 💔 💥
1 file would be reformatted, 1106 files would be left unchanged.
Checking black formatting DONE
flake8 --version
3.7.9 (flake8-comprehensions: 3.2.3, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.8.3 on Linux
Linting .py code
Linting .py code DONE
Linting .pyx and .pxd code
Linting .pyx and .pxd code DONE
Linting .pxi.in
Linting .pxi.in DONE
flake8-rst --version
0.7.0 (flake8-comprehensions: 3.2.3, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.8.3 on Linux
Linting code-blocks in .rst documentation
Linting code-blocks in .rst documentation DONE
Linting .pyx code for spacing conventions in casting
Linting .pyx code for spacing conventions in casting DONE
Linting .c and .h
Linting .c and .h DONE
Check for use of not concatenated strings
Check for use of not concatenated strings DONE
Check for strings with wrong placed spaces
Check for strings with wrong placed spaces DONE
isort --version-number
4.3.21
Check import format using isort
Check import format using isort DONE
##[error]Process completed with exit code 1.
```
Run black on that file

```
would reformat /home/runner/work/pandas/pandas/asv_bench/benchmarks/io/style.py
```
I checked linter locally but did not get that...
And it does not say what to reformat..?
ASV result with improved benchmark code:
```
[ 50.00%] ··· Running (io.style.RenderApply.time_render--).
[ 50.00%] · For pandas commit 69c2bc2a <jihwans-patch-1> (round 2/2):
[ 50.00%] ·· Benchmarking virtualenv-py3.6-Cython0.29.16-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 56.25%] ··· io.style.RenderApply.peakmem_apply                              ok
[ 56.25%] ··· ====== ====== ====== ======
              --             rows        
              ------ --------------------
               cols    12    120    1200 
              ====== ====== ====== ======
                12    101M   101M   101M 
                24    101M   101M   101M 
                36    101M   101M   101M 
              ====== ====== ====== ======

[ 62.50%] ··· io.style.RenderApply.peakmem_render                             ok
[ 62.50%] ··· ====== ====== ====== ======
              --             rows        
              ------ --------------------
               cols    12    120    1200 
              ====== ====== ====== ======
                12    101M   102M   120M 
                24    101M   104M   139M 
                36    101M   105M   157M 
              ====== ====== ====== ======

[ 68.75%] ··· io.style.RenderApply.time_apply                                 ok
[ 68.75%] ··· ====== ============ ============ ============
              --                      rows                 
              ------ --------------------------------------
               cols       12          120          1200    
              ====== ============ ============ ============
                12    11.7±0.2μs   12.1±0.3μs   12.1±0.3μs 
                24    11.5±0.2μs   11.9±0.1μs   11.7±0.5μs 
                36    11.6±0.1μs   11.5±0.4μs    336±6ns   
              ====== ============ ============ ============

[ 75.00%] ··· io.style.RenderApply.time_render                                ok
[ 75.00%] ··· ====== ============ ========= ============
              --                     rows               
              ------ -----------------------------------
               cols       12         120        1200    
              ====== ============ ========= ============
                12    19.2±0.1ms   154±3ms   1.53±0.02s 
                24    30.9±0.8ms   237±5ms   2.36±0.08s 
                36     40.8±2ms    325±9ms   3.13±0.1s  
              ====== ============ ========= ============

[ 75.00%] · For pandas commit 1ce1c3c1 <v1.0.4^0> (round 2/2):
[ 75.00%] ·· Building for virtualenv-py3.6-Cython0.29.16-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt......
[ 75.00%] ·· Benchmarking virtualenv-py3.6-Cython0.29.16-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 81.25%] ··· io.style.RenderApply.peakmem_apply                              ok
[ 81.25%] ··· ====== ====== ====== ======
              --             rows        
              ------ --------------------
               cols    12    120    1200 
              ====== ====== ====== ======
                12    100M   100M   100M 
                24    100M   100M   100M 
                36    100M   100M   100M 
              ====== ====== ====== ======

[ 87.50%] ··· io.style.RenderApply.peakmem_render                             ok
[ 87.50%] ··· ====== ====== ====== ======
              --             rows        
              ------ --------------------
               cols    12    120    1200 
              ====== ====== ====== ======
                12    101M   102M   127M 
                24    101M   105M   153M 
                36    101M   107M   178M 
              ====== ====== ====== ======

[ 93.75%] ··· io.style.RenderApply.time_apply                                 ok
[ 93.75%] ··· ====== ============= ============ =============
              --                       rows                  
              ------ ----------------------------------------
               cols        12          120           1200    
              ====== ============= ============ =============
                12    11.6±0.04μs   11.5±0.1μs    12.1±0.2μs 
                24     11.5±0.3μs   11.8±0.2μs   11.7±0.06μs 
                36      11.8±1μs    11.5±0.1μs     334±1ns   
              ====== ============= ============ =============

[100.00%] ··· io.style.RenderApply.time_render                                ok
[100.00%] ··· ====== ========== ============ ============
              --                     rows                
              ------ ------------------------------------
               cols      12         120          1200    
              ====== ========== ============ ============
                12    69.4±1ms    679±20ms    6.44±0.01s 
                24    133±1ms    1.23±0.06s   13.1±0.2s  
                36    197±3ms    1.82±0.02s   18.4±0.06s 
              ====== ========== ============ ============

       before           after         ratio
     [1ce1c3c1]       [69c2bc2a]
     <v1.0.4^0>       <jihwans-patch-1>
-            178M             157M     0.88  io.style.RenderApply.peakmem_render(36, 1200)
-        69.4±1ms       19.2±0.1ms     0.28  io.style.RenderApply.time_render(12, 12)
-      6.44±0.01s       1.53±0.02s     0.24  io.style.RenderApply.time_render(12, 1200)
-         133±1ms       30.9±0.8ms     0.23  io.style.RenderApply.time_render(24, 12)
-        679±20ms          154±3ms     0.23  io.style.RenderApply.time_render(12, 120)
-         197±3ms         40.8±2ms     0.21  io.style.RenderApply.time_render(36, 12)
-      1.23±0.06s          237±5ms     0.19  io.style.RenderApply.time_render(24, 120)
-      1.82±0.02s          325±9ms     0.18  io.style.RenderApply.time_render(36, 120)
-      18.4±0.06s        3.13±0.1s     0.17  io.style.RenderApply.time_render(36, 1200)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
Hello @jihwans! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-27 12:35:35 UTC
Thank you Tom, Will and Young, for your helps - I learned a lot along the way.
Hopefully this commit may help a few people who might had same situation as mine.

jreback - I submitted the code again with that test removed -- It's really not necessary since there really is no impact time wise. It does have memory impact.
Time wise impact is only with render, not with apply.

Thanks for your valuable comment.
whatsnew added in 1.1 performance
Hmm... are we sure this won’t break someone’s existing application?

On Thu, Jun 25, 2020 at 7:03 PM Jeff Reback <notifications@github.com>
wrote:

> *@jreback* commented on this pull request.
> ------------------------------
>
> In pandas/io/formats/style.py
> <https://github.com/pandas-dev/pandas/pull/34863#discussion_r445884680>:
>
> > @@ -561,11 +561,15 @@ def _update_ctx(self, attrs: DataFrame) -> None:
>              Whitespace shouldn't matter and the final trailing ';' shouldn't
>              matter.
>          """
> -        for row_label, v in attrs.iterrows():
> -            for col_label, col in v.items():
> -                i = self.index.get_indexer([row_label])[0]
> -                j = self.columns.get_indexer([col_label])[0]
> -                for pair in col.rstrip(";").split(";"):
> +        rows = [(row_label, v) for row_label, v in attrs.iterrows()]
>
> ok i actually like your code above a little better, its very idiomatic and
> easy to understand. push it up and ping on green.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/34863#discussion_r445884680>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABXKQVXJRCGJSMII3YHZQCTRYPJU5ANCNFSM4OCB3SGQ>
> .
>

@jihwans we have tests ; if we get a report then we can further update but ok pushing this change
@Jeff I tried to come up with code that may break with the new patch we discussed but I could not. I guess I will update it with the new one.
thanks @jihwans very nice