PERF: StataWriter is slow
State: closed

StataWriter wrote data using scalar operations.  This has been replaced using
numpy's internal binary writer (tofile).

Changes needed to improve performance include:
- Vectorized pandas date to Stata date conversion
- Vectorized null padding
- Conversion to record array

When data contain strings, the old writing paths are still used.


Labels:
name: Performance
description: Memory or execution speed performance
name: IO Stata
description: read_stata, to_stata

Comments:
Benchmark numbers:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
packers_write_stata                          |  30.4874 | 9833.6830 |   0.0031 |
packers_write_stata_with_validation          |  60.2467 | 11051.8630 |   0.0055 |
packers_read_stata_with_validation           |  46.3657 | 4559.7486 |   0.0102 |
packers_read_stata                           |  40.0550 | 3760.9890 |   0.0107 |
packers_read_pickle                          |  41.5884 | 216.3510 |   0.1922 |
packers_write_pack                           |  21.2956 |  22.7673 |   0.9354 |
packers_write_pickle                         |  70.2637 |  72.9007 |   0.9638 |
packers_read_csv                             | 126.1433 | 130.7390 |   0.9648 |
packers_read_pack                            |  63.5297 |  65.0126 |   0.9772 |
packers_write_csv                            | 1261.9394 | 1273.5144 |   0.9909 |
packers_write_hdf_table                      |  46.8566 |  47.1373 |   0.9940 |
packers_write_hdf_store                      |  64.9150 |  64.7027 |   1.0033 |
packers_write_json_date_index                |  91.8417 |  91.3290 |   1.0056 |
packers_write_json                           |  79.2526 |  78.5484 |   1.0090 |
packers_read_hdf_table                       |  25.3557 |  24.7653 |   1.0238 |
packers_read_json_date_index                 | 149.3617 | 130.3113 |   1.1462 |
packers_read_json                            | 154.5029 | 133.5477 |   1.1569 |
packers_read_hdf_store                       |  27.9467 |  23.9844 |   1.1652 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [660f900] : PERF: StataWriter is slow
Vectorized pandas date to Stata date conversion

Base   [d10a658] : RLS: set released to True. v0.13.1
```

Thanks - I'm sure I was missing some tricks.

The real problem I am having arises when I have a string column, since using `df.to_records()` produces a rec array containing objects, which cannot be written using `tofile`  Any idea how to convert to a fixed width string.  For example, if I have

``` python
df = DataFrame([[1,'apple'],[2,'banana']])
df.to_records(index=False)
```

Gives me:

```
rec.array([(1, 'apple'), (2, 'banana')],
      dtype=[('0', '<i8'), ('1', 'O')])
```

have a look at io/pytables.py/_write_chunk (well you may need to go before that actually)

all of he data is converted to rec arrays with fixed with dtypes (eg S133) for example

just write a frame with different dtypes to a table format and step thru
see how the conversions are done, see also set_atom_string iirc 

I think this is close - @jreback could you see if the changes I made agree with your suggestions?  I've tested the date conversion code now, including the "slow" path.  

I added `is_datetime64_array` which is suprisingly slow.  Writing the vbench test without dates takes around 10ms, while a single call to `is_datetime64_array` takes 600ms on its own.

```
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.667    0.667    0.667    0.667 {pandas.lib.is_datetime64_array}
        5    0.019    0.004    0.019    0.004 internals.py:227(set)
        1    0.011    0.011    0.011    0.011 {method 'tofile' of 'numpy.ndarray' objects}
       19    0.011    0.001    0.011    0.001 {method 'copy' of 'numpy.ndarray' objects}

```

use com.is_datetime64_dtype 

that is an internal routine you are using which is only called in certain circumstances

lib.infer_dtypes does call it - but it is carefully controlled exactly how/when

With these changes I think I'm about ready, assuming no bad practices.

Changing that check improved timing (was around 220ms for writing in vbench):

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
packers_write_stata                          |  31.4674 | 9872.9297 |   0.0032 |
packers_write_stata_with_validation          |  70.9974 | 11082.5813 |   0.0064 |
packers_read_stata_with_validation           |  46.5394 | 4598.1929 |   0.0101 |
packers_read_stata                           |  40.3523 | 3781.3640 |   0.0107 |
packers_read_pickle                          |  41.6083 | 217.9286 |   0.1909 |
packers_read_pack                            |  61.8817 |  65.1450 |   0.9499 |
packers_read_csv                             | 126.3011 | 131.8997 |   0.9576 |
packers_write_pickle                         |  70.9850 |  72.7733 |   0.9754 |
packers_write_json_date_index                |  90.9127 |  91.7447 |   0.9909 |
packers_write_json                           |  78.7477 |  79.1103 |   0.9954 |
packers_write_hdf_store                      |  64.9327 |  65.2117 |   0.9957 |
packers_write_hdf_table                      |  46.9437 |  47.1194 |   0.9963 |
packers_write_csv                            | 1286.0013 | 1280.6387 |   1.0042 |
packers_read_hdf_table                       |  25.2563 |  24.6787 |   1.0234 |
packers_read_json_date_index                 | 149.1230 | 130.2010 |   1.1453 |
packers_write_pack                           |  23.2460 |  20.1383 |   1.1543 |
packers_read_json                            | 154.8327 | 133.6617 |   1.1584 |
packers_read_hdf_store                       |  28.2524 |  23.9980 |   1.1773 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

I removed the test against `np.string_` which cannot occur anymore since strings are always `np.object`

`max_len_string_array` returns 0 for arrays containing arbitrary objects, so it isn't useful for testing for strings or unicode.

Should I remove unreachable code?  

There is a lot in the two `_write` routines that are no longer used, as well as the scalar date conversion functions (around 10% of the file, at least when measured using coverage which goes from 81% with them to 91% without).

well if it's not reached by any of the test code then I would take it out

can't vouch  for what we can't test

I think when it gets green that it will be ready.  I've remove the unreachable code.

It could use a good CLN for pep8 issues, but that should be in another PR.

@jreback Not my changes, a Yahoo fail.  Should I force to get a new commit, or is it ok to leave?

nah its ok, (feel free to track down those yahoo fails though!)

thanks for this!