PERF: StataReader is slow due to excessive lookups for missing vales
State: closed

Previous versions of StataReader did not correctly check for missing values.
This was fixed in a previous PR, but these checks had previously been
implemented on a value-by-value basis.  This has now been changed to a
vectorized version that is orders of magnitude faster.

Additionally, a benchmark was added to monitor performance issues in the
future.

Closes #8040


Labels:
name: Performance
description: Memory or execution speed performance
name: IO Stata
description: read_stata, to_stata

Comments:
can u post the vbench run from say 0.13.1 till now?

`./test_perf.sh -b v0.13.1 -t HEAD -r packers`

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
packers_write_stata                          | 1639.5844 | 9598.6373 |   0.1708                                                                      |
packers_read_pickle                          |  41.2370 | 220.7290 |   0.1868 |
packers_write_stata_with_int                 | 2100.9600 | 10837.9307 |   0.1939                                                                      |
packers_read_stata_with_int                  | 4009.2154 | 4531.7237 |   0.8847                                                                      |
packers_read_stata                           | 3411.1613 | 3739.0094 |   0.9123                                                                      |
packers_read_csv                             | 126.4450 | 133.1240 |   0.9498 |
packers_write_hdf_store                      |  62.1700 |  64.2070 |   0.9683 |
packers_write_pickle                         |  70.9056 |  72.8696 |   0.9730 |
packers_write_hdf_table                      |  46.8840 |  46.7546 |   1.0028 |
packers_write_csv                            | 1265.7423 | 1257.0117 |   1.0069                                                                      |
packers_write_json_date_index                |  91.4153 |  90.7796 |   1.0070 |
packers_write_json                           |  78.6374 |  77.9803 |   1.0084 |
packers_read_hdf_table                       |  25.4953 |  24.7576 |   1.0298 |
packers_write_pack                           |  21.2376 |  20.4477 |   1.0386 |
packers_read_pack                            |  67.0931 |  61.1893 |   1.0965 |
packers_read_json                            | 146.3947 | 133.4081 |   1.0973 |
packers_read_json_date_index                 | 147.1183 | 132.3210 |   1.1118 |
packers_read_hdf_store                       |  27.9624 |  23.9660 |   1.1668 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

Faster now it would appear, although about 10% slower on the example in #8040 in some ad-hoc testing.

hmm, this is against 0.13.1? 

I copied and pasted your command.

```
Target [ca00ebd] : PERF: StataReader is slow due to excessive lookups for missin                                                                     g vales
Previous versions of StataReader did not correctly check for missing values.
This was fixed in a previous PR, but these checks had previously been
implemented on a value-by-value basis.  This has now been changed to a
vectorized version that is orders of magnitude faster.
Adds support for preserving Stata missing values when importing data using
the convert_missing argument.  When True, missing values will be returned as
StataMissingValue objects.  Test cases for missing value conversion have been
included.
Additionally, a benchmark was added to monitor performance issues in the
future.
Closes #8040

Base   [d10a658] : RLS: set released to True. v0.13.1

```

ok.

Seems these are very slow? any thoughts

The code to read and write is very loopy - It is essentially looping over cols and rows, calling `pack` or `unpack` element by element, and reading/writing as it goes.  IMO 2 - 4 seconds for is not a problem in most applications - it has similar performance to CSV.

Cython would probably an alternative solution since the data must be written row-by-row _and_ rows have heterogeneous data types.

I don't know if it would be possible to vectorize `pack`, and even then it is still necessary to loop across columns when assembling data.

ok, as long as you are happy that the perf issues from before are fixed (they look like it). you have 5-6x speedup (of course that does not use validation).

ok, anything else? merge (do you want/need docs for convert_missing?)

It is better than is was in 0.13.1, which was the _fast_ version, so I think this is solved.  The reading code spends almost all of its time in 

```
    def _next(self):
        typlist = self.typlist
        if self.has_string_data:
            data = [None] * self.nvar
            for i in range(len(data)):
                if type(typlist[i]) is int:
                    data[i] = self._null_terminate(
                        self.path_or_buf.read(typlist[i])
                    )
                else:
                    data[i] = self._unpack(
                        typlist[i], self.path_or_buf.read(self._col_size(i))
                    )
            return data
        else:
            return lmap(
                    lambda i: self._unpack(typlist[i],
                                           self.path_or_buf.read(
                                               self._col_size(i)
                                           )),
                    range(self.nvar)
            )
```

which is clever, but probably far from optimal from a performance point-of-view.  Would probably be far faster 
1. Build a composite format string for the entire file since all rows are homogeneous like `format = "<f<f<b<b"`
2. Read an entire row using `struct.unpack(format,  self.path_or_buf.read(total_column_size))`

And the rest of the code could be unchanged.  

Might try some experiments, but should be different PR.

ok, sounds fine, ping me when green

@jreback Ready to go.  I have a further performance patch ready when this is in.

thanks!

better perf always appreciated!