[ArrowStringArray] PERF: use pyarrow native functions for str.get_dummies
State: closed

draft as performance is not yet better than object fallback

```
       before           after         ratio
     [09f3bf80]       [ec8f4b2d]
     <master>         <get_dummies>
+         647±4ms          761±6ms     1.18  strings.Dummies.time_get_dummies('arrow_string')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```

```
Timer unit: 1e-06 s

Total time: 0.778161 s
File: /home/simon/pandas/pandas/core/arrays/string_arrow.py
Function: _str_get_dummies at line 937

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   937                                               def _str_get_dummies(self, sep="|"):
   938         1          2.0      2.0      0.0          if pa_version_under4p0:
   939                                                       return super()._str_get_dummies(sep)
   940                                           
   941         1      12415.0  12415.0      1.6          result = pc.split_pattern(self._data, pattern=sep)
   942         1       8084.0   8084.0      1.0          uniques = pc.list_flatten(result).unique()
   943         1        143.0    143.0      0.0          tags = uniques.take(pc.array_sort_indices(uniques)).to_pylist()
   944         1         53.0     53.0      0.0          dummies = np.empty((len(result), len(tags)), dtype=np.int64)
   945        63        160.0      2.5      0.0          for i, tag in enumerate(tags):
   946        62      20660.0    333.2      2.7              match_equal = pc.equal(self._data, tag)
   947        62     186099.0   3001.6     23.9              match_mid = pc.match_substring(self._data, sep + tag + sep)
   948       124     112610.0    908.1     14.5              match_start = pc.match_substring_regex(
   949        62        346.0      5.6      0.0                  self._data, "^" + tag + re.escape(sep)
   950                                                       )
   951        62     383985.0   6193.3     49.3              match_end = pc.match_substring_regex(self._data, re.escape(sep) + tag + "$")
   952       124        435.0      3.5      0.1              match = pc.or_(
   953        62       1111.0     17.9      0.1                  pc.or_(pc.or_(match_start, match_end), match_mid), match_equal
   954                                                       )
   955        62       5989.0     96.6      0.8              casted = match.cast(pa.int64())
   956        62       2072.0     33.4      0.3              filled = pc.fill_null(casted, 0)
   957        62      43997.0    709.6      5.7              dummies[:, i] = filled
   958         1          0.0      0.0      0.0          return dummies, tags
```

plenty more ideas to try. so parking here for now.

I think we should assume that in most cases the number of rows will be much greater then the number of dummies, but taking advantage of the split rows may be beneficial.

for now outer loop is dummies (tags).

match_substring is faster than match_substring_regex so if I can append a separator to the start and end efficiently in pyarrow land that should help.

match_equal is faster still, so maybe can operate on the pyarrow list array.
 

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
latest commit

```
       before           after         ratio
     [a246270e]       [f75bda18]
     <master>         <get_dummies>
-         636±7ms          269±1ms     0.42  strings.Dummies.time_get_dummies('arrow_string')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

will leave as draft till have checked that the object fallback cannot be improved, otherwise not a fair comparison.
> plenty more ideas to try. so parking here for now.

> match_equal is faster still, so maybe can operate on the pyarrow list array.

I have tried a few approaches checking on the tag being in a list of the splits. Have a working implementations in numpy but far too slow. In pyarrrow, didn't get far, checking using equals on the flattened array, but may need to work on reconstructing the list array from the list_parent_indices to get much further.
> will leave as draft till have checked that the object fallback cannot be improved, otherwise not a fair comparison.

There's a decent speedup here, but will close for now to help clear the queue.