pydata/pandas — pull request #2819 by stephenwlin (2013-02-08 05:39:04+00:00)
Title: ENH: Optimize _merge_blocks, take_*; improve non-NA fill_value support

(Depends on #2708, but I am submitting this for now in case anyone has any feedback; I will rebase after #2708 is merged)

Optimizes/improves the `common.take_1d`, `common.take_2d`, `common.take_2d_multi`, and `common.take_fast` in the following ways:
* No intermediate buffer is required anymore to convert between two different dtypes (i.e. when upcasting is necessary): the data is taken and upcasted in one step. Common upcasts are optimized in Cython and more can be added if desired: all others resort to a generic (but relatively efficient) Python implementation.
* Better support for `fill_value` parameters other than NaN: previously, all upcasting was done assuming NaN as the `fill_value`: now, the correct upcast will be chosen based on the type of the `fill_value`. (Ex. Integers stay as integers with integer `fill_value` if the value can fit in the existing size without overflow, integers and floats upcast to complex with complex `fill_value`, all numerics upcast to objects with boolean or string `fill_value`, etc.)
* Added option of `fill_value` of `None`, which means that the corresponding entries in the output are left unchanged (only really makes sense when `out` is also provided)

The last option is used to optimize `internals._merge_blocks` (i.e. the function internally called to do block consolidation): instead of `vstack`-ing the arrays together in an intermediate buffer and then rearranging into the output with a separate operation, the output buffer is allocated uninitialized and directly filled from the inputs using one call to `common.take_fast` per input block, even when they are not contiguous. (However, the special case of contiguous and already properly ordered blocks is still handed with `vstack`, since no other operation is required after that)

Here are some of the resulting performance improvements:
```
t_head t_baseline ratio
name
frame_boolean_row_select 0.3822 0.7733 0.4942
frame_reindex_both_axes 0.5104 0.8807 0.5795
frame_reindex_axis0 1.9940 3.2788 0.6081
frame_reindex_both_axes_ix 0.6137 0.9587 0.6401
panel_from_dict_equiv_indexes 34.3422 48.6950 0.7053
frame_reindex_axis1 3.9934 5.3896 0.7410
groupby_frame_median 12.4208 15.6912 0.7916
panel_from_dict_same_index 34.2735 42.2966 0.8103
groupby_multi_different_functions 23.9548 29.4110 0.8145
groupby_first_float32 6.8324 8.3060 0.8226
groupby_multi_different_numpy_functions 24.1576 29.2740 0.8252
groupby_multi_cython 27.2160 32.8312 0.8290
frame_fancy_lookup_all 29.2068 34.4993 0.8466
groupby_first 6.8660 7.9957 0.8587
timeseries_large_lookup_value 0.0240 0.0269 0.8906
concat_series_axis1 70.0536 78.6340 0.8909
frame_insert_500_columns 105.2279 117.7092 0.8940
frame_reindex_columns 0.4184 0.4599 0.9098
```

Please let me know if you have any feedback...thanks!