CI: mark window online test slow
State: closed

Seems like the new online tests added increase test runtime a lot, making failures like https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=61437&view=logs&j=404760ec-14d3-5d48-e580-13034792878f&t=f81e4cc8-d61a-5fb8-36be-36768e5c561a and 
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=61437&view=logs&j=404760ec-14d3-5d48-e580-13034792878f&t=f81e4cc8-d61a-5fb8-36be-36768e5c561a
more common. Slowest durations showing this are below, have marked these as slow as a potential solution
```
============================ slowest 30 durations =============================
99.91s call     pandas/tests/resample/test_base.py::test_resample_interpolate[period_range-pi-_index_start1-_index_end1]
47.56s call     pandas/tests/resample/test_period_index.py::TestPeriodIndex::test_weekly_upsample[end-D-FRI]
47.36s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-True-True-obj1]
45.55s teardown pandas/tests/window/test_rolling.py::test_rolling_zero_window
30.25s call     pandas/tests/tslibs/test_timezones.py::test_cache_keys_are_distinct_for_pytz_vs_dateutil[America/Argentina/San_Luis]
28.37s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-True-True-obj0]
23.03s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-True-True-obj1]
23.00s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-False-True-obj1]
20.95s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-False-True-obj0]
20.30s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-True-True-obj0]
19.94s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-True-False-obj1]
19.81s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-True-False-obj0]
19.37s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-False-False-obj1]
19.25s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-False-False-obj0]
18.13s call     pandas/tests/test_sorting.py::TestSorting::test_int64_overflow_moar
18.10s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-True-True-False-obj0]
18.07s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-True-False-obj1]
17.17s call     pandas/tests/window/test_numba.py::TestEngine::test_numba_vs_cython_apply[True-False-False-False-True]
17.16s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-False-True-obj1]
16.84s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[False-True-True-False-True-obj1]
16.75s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-False-True-obj0]
16.48s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[False-True-True-True-True-obj1]
16.47s setup    pandas/tests/io/test_fsspec.py::test_from_s3_csv
16.36s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[False-True-False-True-True-obj1]
15.93s call     pandas/tests/io/test_compression.py::test_with_missing_lzma
15.83s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[False-True-False-False-True-obj1]
15.13s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-False-False-obj1]
14.46s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[False-True-True-True-True-obj0]
14.14s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[True-True-False-False-False-obj0]
14.00s call     pandas/tests/window/test_online.py::TestEWM::test_online_vs_non_online_mean[False-True-False-True-False-obj0]
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
cc @mroeschke 
@stuartarchibald any pointers on how I could speed up our numba tests?

Generally I have a parameterized test like this:

https://github.com/pandas-dev/pandas/blob/8c6e8653d4621d08eb6aad55963fac8233f71ae3/pandas/tests/window/test_online.py#L28

Where I am parameterizing over `nogil=True|False, parallel=True|False, nopython=True|False` (among other parameters)

A typical numba function that is being jitted under the hood is like this:

https://github.com/pandas-dev/pandas/blob/8c6e8653d4621d08eb6aad55963fac8233f71ae3/pandas/core/window/online.py#L37

The size of the data in the tests is fairly small, but jitting a new version of these functions for each parameter combination seems to take quite a bit of time.
@mroeschke prob ok to just have a couple of predefined numba options rather than doing them all (as not testing numba per se here)
@mroeschke @jreback I agree, I think cutting down the options is probably a good idea. Numba's test suite is massive and independently tests these features (and permutations of) so it's probably best in view of runtime to focus on testing the things that are important to these Pandas features.

* The `nogil` option is just switching on/off the GIL, it just impacts code gen and can probably be ignored unless you are specifically testing to assert that the GIL is not present?
* The `parallel` option is probably worth testing as it's one of the most complex transforms in Numba.
* The `nopython=True` option permits the "compile to run without the cpython API or fail" behaviour. I think you probably always want this as these functions really need the performance and should always be compiled to run without the cpython API?

If you agree with the above, I think it would cut the test space down to `nogil=False (default), nopython=True(default), parallel=True/False`, i.e. just testing the variation in the `parallel` kwarg. 
if there isn't a better fix on the horizon i think marking as slow is reasonable.  This test accounts for about a quarter of the runtime with --skip-slow locally
Thanks for the insight @stuartarchibald. I'll make those changes in a follow up PR.

As for this PR, I think marking as slow is still a good idea as @jbrockmendel mentioned
thanks @mzeitlin11 

@mroeschke ok to PR the limiting of the numba combinations as @stuartarchibald