ENH: Add orient=tight format for dictionaries
State: closed

- [x] xref #4889 (see below)
- [x] tests added / passed
  - tests.frame.methods.test_to_dict.test_to_dict_orient_tight()
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The issue in #4889 contains examples of how the JSON format doesn't support `MultiIndex` for the index or the columns, independent of the `orient` chosen.  Also, none of the orientations support index names (or `MultiIndex` names).  Finally, if you want to have a JSON format that is "tight" (or "compact"), the `split` is the closest thing, but it is incomplete due to the indexing issues.

As a first step to addressing this, I have created a `orient='tight'`  option to `DataFrame.to_dict()` and `DataFrame.from_dict()` .  If we agree that this is a reasonable representation, the next step (which I'd prefer to do in a second PR) would be to also support it with JSON.  The challenge there is that `to_json()` is currently implemented in C.  Right now, one can just use `json.dumps(df.to_dict(orient='tight'))` to get the needed JSON as a workaround.

I'm open to changing the word `'tight'` to something else, but I wanted it to start with a different letter than the other orientations.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Hmm +/- 0 on adding anything to to_dict here; we already have a lot of formats that kind of do the same things but not really and it's not always spelled out. While this could have some utility I think it adds to that confusion

> The challenge there is that `to_json()` is currently implemented in C.

If the ultimate goal is to include in the table schema most of that is written in Python, so maybe not as much of a concern?


> Hmm +/- 0 on adding anything to to_dict here; we already have a lot of formats that kind of do the same things but not really and it's not always spelled out. While this could have some utility I think it adds to that confusion
> 
> > The challenge there is that `to_json()` is currently implemented in C.
> 
> If the ultimate goal is to include in the table schema most of that is written in Python, so maybe not as much of a concern?

@WillAyd Here are two important comments in the original issue that this PR addresses:

https://github.com/pandas-dev/pandas/issues/4889#issuecomment-614798694  states (and I agree with this observation) that the `table` format is not compact when it comes to JSON.  The size of the 'table' format is not as compact as possible when you need to push JSON around as the payload in a REST API.

https://github.com/pandas-dev/pandas/issues/4889#issuecomment-645323225  states the issue with not preserving the `index` correctly.

@jreback before I make the change you suggested, when we had the dev sprint on 9/4/20, we discussed this PR, and it was suggested that I add an argument such as `preserve_names` to `to_dict()` that would apply to the `"split"` format rather than introduce the `"tight"` option.

So what is your opinion about introducing `orient="tight"` versus `orient="split", preserve_names=False` (where the latter would be used to preserve compatibility with prior versions)?
@Dr-Irv what's the status here?
> @Dr-Irv what's the status here?

I've been waiting for @jreback to respond to this comment:  https://github.com/pandas-dev/pandas/pull/35292#issuecomment-699072742

I would like to introduce `orient="tight"`, but I'm not sure there is agreement on that.  If we can agree, I'll move forward with it.
looks ok, can you merge master and will look
@jreback have merged with master and all checks passed.  Changed it from draft status.
thanks @Dr-Irv