PERF: 5x speedup for read_json() with orient='index' by avoiding transpose
State: closed

The `.T` operator can be quite slow on mixed-type `DataFrame`s due to the creation of `object` dtype columns. In comparison to direct construction with `DataFrame.from_dict()` can generally be much more efficient.

Making that swap inside `pd.read_json()` yields a `~5-6x` speedup for the `orient='index'` case: 
```
       before           after         ratio
     [d47fc0cb]       [b0fd99ec]
     <read_json_speedup~1>       <read_json_speedup>
-      5.37±0.03s          907±5ms     0.17  io.json.ReadJSON.time_read_json('index', 'int')
-      5.27±0.01s          804±3ms     0.15  io.json.ReadJSON.time_read_json('index', 'datetime')
```

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
cc @TomAugspurger would this be related to https://github.com/pandas-dev/pandas/pull/24387 at all?
Ignore previous comment was too focused on the constructor and not the transposition. This makes sense to me
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=h1) Report
> Merging [#26773](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d47fc0cba3cf94ebd289ad3776bf7ff3fe60dfb8?src=pr&el=desc) will **decrease** coverage by `50.5%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26773/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #26773       +/-   ##
===========================================
- Coverage   91.71%   41.21%   -50.51%     
===========================================
  Files         178      178               
  Lines       50771    50771               
===========================================
- Hits        46567    20926    -25641     
- Misses       4204    29845    +25641
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.21% <0%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `63.17% <0%> (-30.07%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/plotting/\_matplotlib/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19tYXRwbG90bGliL19faW5pdF9fLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.37%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.16%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.1%)` | :arrow_down: |
| [pandas/core/sparse/scipy\_sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3NjaXB5X3NwYXJzZS5weQ==) | `10.14% <0%> (-89.86%)` | :arrow_down: |
| ... and [133 more](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=footer). Last update [d47fc0c...b0fd99e](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=h1) Report
> Merging [#26773](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d47fc0cba3cf94ebd289ad3776bf7ff3fe60dfb8?src=pr&el=desc) will **decrease** coverage by `50.5%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26773/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #26773       +/-   ##
===========================================
- Coverage   91.71%   41.21%   -50.51%     
===========================================
  Files         178      178               
  Lines       50771    50771               
===========================================
- Hits        46567    20926    -25641     
- Misses       4204    29845    +25641
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.21% <0%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `63.17% <0%> (-30.07%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/plotting/\_matplotlib/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19tYXRwbG90bGliL19faW5pdF9fLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.37%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.16%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.1%)` | :arrow_down: |
| [pandas/core/sparse/scipy\_sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL3NjaXB5X3NwYXJzZS5weQ==) | `10.14% <0%> (-89.86%)` | :arrow_down: |
| ... and [133 more](https://codecov.io/gh/pandas-dev/pandas/pull/26773/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=footer). Last update [d47fc0c...b0fd99e](https://codecov.io/gh/pandas-dev/pandas/pull/26773?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

nice! @qwhelan mind taking a look at the test cases ( looks like this changes the order of the index ) https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=12630

```
>   raise_assert_detail(obj, msg, lobj, robj)
E   AssertionError: DataFrame.columns are different
E   
E   DataFrame.columns values are different (100.0 %)
E   [left]:  Index(['A', 'B', 'C', 'D'], dtype='object')
E   [right]: Index(['D', 'C', 'B', 'A'], dtype='object')
```
@alimcmaster1 Given that this only fails on 3.5, I'm guessing this is a dict-orderedness issue in `from_dict()`
lgtm, can you add a note in Performance for 0.25.0, ping on green.
thanks @qwhelan