REF: remove Block access in the JSON writing code
State: closed

Closes #27164

This is a small refactor of the JSON writing code (`objToJSON.c`) to remove the access of the Blocks. Serializing to JSON was already done column-by-column (also when accessing the blocks), but currently we were iterating over the different "columns" of the 2D array per block in the C code. While with this PR, the C code directly iterates over all the columns (instead of per block).
This gives a less invasive change as the previous attempt of https://github.com/pandas-dev/pandas/pull/27166

To be able to iterate over the column arrays in C, I added the `column_arrays` attribute on the manager, for this. 

I ran the JSON ASV benchmarks with this branch:

`$ asv continuous -b json -f 1.01 upstream/master HEAD`

```diff
       before           after         ratio
     [692bba57]       [4142a532]
     <master>         <refactor-json-blocks>
+         162±2ms          192±2ms     1.19  io.json.ToJSON.time_to_json_wide('values', 'df_td_int_ts')
+         164±2ms          191±2ms     1.16  io.json.ToJSON.time_to_json_wide('split', 'df_td_int_ts')
+      96.2±0.3ms        111±0.8ms     1.15  io.json.ToJSON.time_to_json('values', 'df')
+         192±3ms          221±3ms     1.15  io.json.ToJSON.time_to_json_wide('records', 'df_td_int_ts')
+       106±0.3ms          121±1ms     1.15  io.json.ToJSON.time_to_json('split', 'df')
+        96.4±2ms          110±2ms     1.15  io.json.ToJSON.time_to_json('values', 'df_date_idx')
+         192±4ms          219±4ms     1.14  io.json.ToJSON.time_to_json_wide('index', 'df_td_int_ts')
+         177±4ms          198±3ms     1.12  io.json.ToJSON.time_to_json_wide('columns', 'df_td_int_ts')
+         112±1ms          124±2ms     1.11  io.json.ToJSON.time_to_json('records', 'df')
+       129±0.6ms          143±2ms     1.11  io.json.ToJSON.time_to_json('index', 'df')
+         138±1ms          152±1ms     1.10  io.json.ToJSON.time_to_json('index', 'df_date_idx')
+       168±0.9ms          182±1ms     1.09  io.json.ToJSONLines.time_floats_with_int_idex_lines
+         168±2ms          182±1ms     1.08  io.json.ToJSONLines.time_floats_with_dt_index_lines
+       117±0.9ms          127±1ms     1.08  io.json.ToJSON.time_to_json_wide('split', 'df_date_idx')
+         157±2ms          165±2ms     1.05  io.json.ToJSON.time_to_json_wide('split', 'df_int_floats')
-            177M             175M     0.99  io.json.ToJSON.peakmem_to_json_wide('split', 'df_int_float_str')
-         185±5ms          172±3ms     0.93  io.json.ToJSON.time_to_json('index', 'df_int_floats')
-         127±8ms        110±0.6ms     0.87  io.json.ToJSON.time_to_json('split', 'df_td_int_ts')
```

</details>

There is a slight slowdown in some of the benchmarks for wide dataframes, but only 20%. 
So we can decide to keep the original block-iteration code for BlockManager for now, instead of fully replacing it as I did now, if we don't want this slight slowdown.

This also fixes some remaining cases for ArrayManager (xref https://github.com/pandas-dev/pandas/issues/39146)

Labels:
name: Refactor
description: Internal refactoring of code
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
It might be that with a bit of optimization of `BlockManager.column_arrays`, that could reduce the slowdown.
+1 on deleting the block code
With the last commit (adding some extra python code to more efficiently get a list of 1D ndarrays for BlockManager), I now get hose ASV results (for the same `asv` call as above):

```diff
       before           after         ratio
     [692bba57]       [3c9aa994]
     <master>         <refactor-json-blocks>
+       114±0.6ms          122±2ms     1.07  io.json.ToJSON.time_to_json_wide('split', 'df_date_idx')
+         152±1ms          162±2ms     1.06  io.json.ToJSON.time_to_json_wide('records', 'df')
+         148±2ms          156±2ms     1.05  io.json.ToJSON.time_to_json_wide('columns', 'df')
-      79.8±0.2ms       78.8±0.2ms     0.99  io.json.NormalizeJSON.time_normalize_json('values', 'df_int_floats')
```

So that looks quite good I think
@jreback do you want to review this more in depth?
lgtm. need a rebase
Failure is a flaky resourcewarning.