PERF: DataFrame.transpose with dt64tz
State: closed

Does what it says on the tin: `DatetimeBlock.values` is always `DatetimeArray`, and `dt64tzblock.shape == dt64tzblock.values` in all cases.  Similarly `TimedeltaBlock.values` is always `TimedeltaArray`.

Notes:
- It is straightforward to extend this to work for PeriodDtype (i have a branch).  Haven't tried it, but I expect it would be similarly easy to do the same for CategoricalDtype.
 
Things that im not yet fully happy with: 
- fillna method on 2D (I think @simonjayhawkins commented on this in another branch recently), 
- nargminmax with 2D and mask.any()
- pytables kludge

ASVs: run repeatedly (vs master from yesterday) with --record-samples --append-samples so im pretty confident these are stable (but still include some nonsense xref #40066)

```
       before           after         ratio
     [f4b67b5e]       [65792836]
     <master>         <ref-hybrid-3>
+        10.1±3ms         13.9±3ms     1.38  eval.Eval.time_add('python', 'all')
+     2.06±0.02ms      2.40±0.06ms     1.16  hash_functions.NumericSeriesIndexingShuffled.time_loc_slice(<class 'pandas.core.indexes.numeric.Int64Index'>, 1000000)
+         227±2μs          263±2μs     1.15  groupby.GroupByMethods.time_dtype_as_field('datetime', 'head', 'transformation')
+         228±2μs          261±2μs     1.15  groupby.GroupByMethods.time_dtype_as_field('datetime', 'head', 'direct')
+         238±2μs          272±2μs     1.14  groupby.GroupByMethods.time_dtype_as_field('datetime', 'tail', 'transformation')
+         248±6μs          282±5μs     1.14  groupby.GroupByMethods.time_dtype_as_field('datetime', 'tail', 'direct')
+     3.92±0.03ms      4.37±0.01ms     1.11  rolling.Engine.time_rolling_apply('DataFrame', 'float', <function Engine.<lambda> at 0x7fb1c0b40670>, 'cython', 'median')
+     2.83±0.02ms      3.14±0.06ms     1.11  io.hdf.HDFStoreDataFrame.time_store_info
-         275±4μs          248±4μs     0.90  groupby.GroupByMethods.time_dtype_as_field('datetime', 'shift', 'direct')
-     1.41±0.05ms      1.27±0.01ms     0.90  stat_ops.FrameOps.time_op('sum', 'int', 1)
-     1.13±0.06ms      1.02±0.07ms     0.90  arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.int64'>, 5.0, <built-in function ne>)
-         271±2μs          242±2μs     0.89  groupby.GroupByMethods.time_dtype_as_field('datetime', 'shift', 'transformation')
-         188±3μs          167±1μs     0.89  algos.isin.IsIn.time_isin_empty('datetime64[ns]')
-         192±2μs          170±2μs     0.89  algos.isin.IsIn.time_isin_mismatched_dtype('datetime64[ns]')
-         227±2μs          200±2μs     0.88  groupby.GroupByMethods.time_dtype_as_field('datetime', 'any', 'direct')
-         226±2μs          199±1μs     0.88  groupby.GroupByMethods.time_dtype_as_field('datetime', 'all', 'transformation')
-         227±2μs          199±1μs     0.88  groupby.GroupByMethods.time_dtype_as_field('datetime', 'any', 'transformation')
-        895±60μs         785±80μs     0.88  arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.float64'>, 3.0, <built-in function ge>)
-      10.2±0.3ms       8.93±0.7ms     0.88  algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.int64'>, 19, 'inside')
-         235±4μs          204±4μs     0.87  groupby.GroupByMethods.time_dtype_as_field('datetime', 'all', 'direct')
-     3.26±0.03μs      2.83±0.03μs     0.87  frame_methods.ToNumpy.time_to_numpy_tall
-     3.28±0.03μs      2.82±0.02μs     0.86  frame_methods.ToNumpy.time_to_numpy_wide
-      9.77±0.2ms       8.40±0.2ms     0.86  indexing.NumericSeriesIndexing.time_loc_slice(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-     2.94±0.05μs      2.52±0.02μs     0.86  frame_methods.ToNumpy.time_values_tall
-     2.95±0.03μs      2.52±0.02μs     0.85  frame_methods.ToNumpy.time_values_wide
-     2.09±0.02ms      1.77±0.01ms     0.85  groupby.FillNA.time_df_ffill
-     2.09±0.02ms      1.77±0.01ms     0.85  groupby.FillNA.time_df_bfill
-         204±3μs          168±2μs     0.82  arithmetic.OffsetArrayArithmetic.time_add_series_offset(<Day>)
-         170±2μs          137±3μs     0.81  groupby.GroupByMethods.time_dtype_as_field('datetime', 'count', 'direct')
-         170±2μs          137±4μs     0.80  groupby.GroupByMethods.time_dtype_as_field('datetime', 'count', 'transformation')
-        29.1±3ms       22.9±0.4ms     0.79  algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.uint64'>, 20, 'outside')
-        26.0±1ms         19.2±2ms     0.74  algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.int64'>, 20, 'inside')
-      26.2±0.2ms      18.0±0.07ms     0.69  index_object.SetOperations.time_operation('date_string', 'symmetric_difference')
-      11.6±0.1ms      7.32±0.08ms     0.63  reshape.ReshapeExtensionDtype.time_stack('datetime64[ns, US/Pacific]')
-      40.2±0.5μs       25.0±0.3μs     0.62  ctors.SeriesDtypesConstructors.time_dtindex_from_index_with_series
-     3.77±0.03ms      2.08±0.03ms     0.55  reshape.ReshapeExtensionDtype.time_unstack_slow('datetime64[ns, US/Pacific]')
-      32.1±0.5μs       17.0±0.2μs     0.53  ctors.SeriesDtypesConstructors.time_dtindex_from_series
-     1.11±0.03ms          408±7μs     0.37  categoricals.Constructor.time_datetimes
-      14.1±0.1μs      1.26±0.02μs     0.09  attrs_caching.SeriesArrayAttribute.time_extract_array_numpy('datetime64')
-      13.7±0.1μs      1.04±0.03μs     0.08  attrs_caching.SeriesArrayAttribute.time_extract_array('datetime64')
-      13.0±0.2μs         455±10ns     0.04  attrs_caching.SeriesArrayAttribute.time_array('datetime64')
-        73.8±1ms      1.66±0.03ms     0.02  reshape.ReshapeExtensionDtype.time_unstack_fast('datetime64[ns, US/Pacific]')
-      64.3±0.9ms          258±2μs     0.00  reshape.ReshapeExtensionDtype.time_transpose('datetime64[ns, US/Pacific]')
```

IIRC the groupby.GroupByMethods.time_dtype_as_field were heavily influenced by constructor overhead, which motivated #40054.  Still need to try out @jorisvandenbossche's suggestion of non-cython optimization there.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
> It is straightforward to extend this to work for PeriodDtype (i have a branch). Haven't tried it, but I expect it would be similarly easy to do the same for CategoricalDtype.

I would leave that out for now, as IMO that requires more discussion (DatetimeBlock / TimedeltaBlock are already consolidated blocks, so changing the underlying array doesn't change consolidation status with its potential impact on actual behaviour changes)
@jorisvandenbossche any ideas on how to troubleshoot the docbuild?  the traceback isn't telling me much about where the problem is
You can see the full log in the "Build documentation" section of the build. For example, it seems that an example in `source/getting_started/comparison/includes/time_date.rst` is failing.
updated to revert `DatetimeTZBlock._can_consolidate` to `False`.  Side-benefit of making a kludge in the pytables code unnecesary.  The remaining things I need to address before considering this ready are a) fillna with 2d (handled by #40294) and b) unhandled/kludgy case in nanargminmax
> can you do pre-cursors that add things in steps.

ive been quietly doing that for a while, but can try to scrape the bottom of the barrel some more.  #38134 will clear out some of the diff
> can you do pre-cursors that add things in steps.

#40331
mentioned on call this morning i made a PR in fastparquet.  until that is merged (and released, and our min version bumped), this implements api.BlockManager that wedges in the needed reshaping
> some comments, this is a very large change

#40456 is a nicely-scoped subset of this intended to cut this down


> > some comments, this is a very large change
> 
> #40456 is a nicely-scoped subset of this intended to cut this down

ahh
updated with a much less invasive fastparquet shim.  There are a few things that _could_ be split off into separate PRs, but very much hitting diminishing returns
Can you give an updated summary / top post of what this PR does? (I don't think the title still reflects that, since that's already the case on master)
> Can you give an updated summary / top post of what this PR does? (I don't think the title still reflects that, since that's already the case on master)

Updated the title.

The main thing this PR does is makes 2D DatetimeTZBlocks be backed by 2D DatetimeArrays.  This allows for some simplification in internals, which this does.  The main outside-internals thing this does is makes DataFrame.transpose take advantage of this which significantly improves performance.

That last bit could still be broken off and done in a follow-up PR.
@jreback would it help to split off the perf-improving part of this for a follow-up to further trim the diff?
> @jreback would it help to split off the perf-improving part of this for a follow-up to further trim the diff?

sure
>> @jreback would it help to split off the perf-improving part of this for a follow-up to further trim the diff?

> sure

hmm this is looking more involved than i expected.  Can try if it makes a difference, basically would split off everything in frame.py
let me look again

i think if u can reduce what is added to frame would be good
broken in as close to half as possible (not that close) in #41082
fairly small diff now
looks fine. can you rebase. pls add a whatsnew note (as this is a non-trivial perf increse). ping on green.
ping
thanks!