Repository: ibis-project/ibis

Issue:
BENCH: cleanup errors around benchmarks
we are showing some errors in the benchmark suite: https://github.com/ibis-project/ibis/pull/2451/checks?check_run_id=1220781799

would be nice to have these run fully w/o errors.


Discussion:
asv doesn't fail when there are exceptions, see #2403
/take
I've been looking into this. Quick question - should I be able to replicate the errors that occur during benchmarking locally with asv? I'm not familiar with asv so I'm looking into both how that works and the ibis implementation. 

Thanks!
You can install asv with `pip install asv` and simply run:
```python
asv machine --yes && asv dev
```
Thanks @datapythonista

For the impala compile errors I think it's due to there being ImportError of ibis.impala.api in ibis/__init__.py because impyla is missing. 

Would we add impyla as dependency for benchmarking or is there another approach?
For the PandasBackend errors it looks like the functions low_card_grouped_rolling_udf and high_card_grouped_rolling_udf weren't implemented in the PandasBackend class. 

Do they simply need to be implemented, if so what type of function should be implemented? Or is there something else going on?

Thanks!
> Would we add impyla as dependency for benchmarking or is there another approach?

If the benchmarks are testing against impala, then we can't get them working until #2388 is fixed. The benchmarks job is not using the impala docker image, and can't use it until that problem is fixed.
Okay looks like there are people working on that already - will check in there.  What about for the pandas backend errors i referenced above? Do `low_card_grouped_rolling_udf` and `high_card_grouped_rolling_udf` need to be implemented?

thx!
> Do `low_card_grouped_rolling_udf` and `high_card_grouped_rolling_udf` need to be implemented?

Not as part of this issue I would say. Not so familiar with those functions, but if they are not implemented for the pandas backend, I'd simply have an `if` to not run them for that backend. May be @jreback has more info on why that is failing.
@jreback checking on the above.  do you have preference for adding those functions (`low_card_grouped_rolling_udf` and `high_card_grouped_rolling_udf` to pandas backend vs simply not running them?
ideally running them

Resolving patch:
diff --git a/benchmarks/benchmarks.py b/benchmarks/benchmarks.py
--- a/benchmarks/benchmarks.py
+++ b/benchmarks/benchmarks.py
@@ -138,7 +138,7 @@ def time_impala_large_expr_compile(self):
 
 class PandasBackend:
     def setup(self):
-        n = 30 * int(2e5)
+        n = 30 * int(2e4)
         self.data = pd.DataFrame(
             {
                 'key': np.random.choice(16000, size=n),
@@ -268,10 +268,10 @@ def time_high_card_grouped_rolling(self):
         self.high_card_grouped_rolling.execute()
 
     def time_low_card_grouped_rolling_udf(self):
-        self.low_card_grouped_rolling_udf.execute()
+        self.low_card_grouped_rolling_udf_mean.execute()
 
     def time_high_card_grouped_rolling_udf(self):
-        self.high_card_grouped_rolling_udf.execute()
+        self.high_card_grouped_rolling_udf_mean.execute()
 
     def time_low_card_window_analytics_udf(self):
         self.low_card_window_analytics_udf.execute()


Test patch: