API: Honor copy for dict-input in DataFrame
State: closed

Closes https://github.com/pandas-dev/pandas/issues/32960.

~Still need a whatsnew. The tl/dr is~

1. We can't honor no-copy for a dict with multiple values of the same dtype: `DataFrame({"A": np.array([1, 2]), "B": np.array([1, 2])})` as long as we have consolidation.
~2. Currently, this is an API breaking change for something like `pd.DataFrame({"A": np.array([1, 2])})`. To resolve that, I think we'll need something like a default of `copy=None`, and then require `copy=False` to enable no-copy for dict inputs (but keep no copy by default for 2d ndarrays and data frames).~

Unsure how we're going to reconcile that with the behavior of EAs.

EDIT: I've changed the default `copy` to `None`, which means

1. no copy for ndarray / dataframe input (`copy=True` to copy)
2. copy for dict input (`copy=False` to not copy).

Labels:
name: API Design
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Stale

Comments:
So it seems that, at least for sparse, we had a test asserting that we did *not* copy `DataFrame({"A": sparse_array})` by default. So I don't think we can restore the pre-1.0 behavior of copying.

So my recommendation is to just always honor `copy` for dict-inputs when we can.
Ahh going back and forth on this. Right now this is API breaking for a dict of ndarrays... Will need to think on it more. I don't like it but I think we'll need to treat EAs and ndarrays differently, at least by default.
OK, in the latest version we should be backwards compatible. By default we will

1. copy ndarrays
2. not copy EAs

```python
In [6]: a = pd.array([1, 2])

In [7]: b = np.array([1, 2])

In [8]: df = pd.DataFrame({"A": a, "B": b})

In [9]: df.iloc[0, 0] = 0

In [10]: df.iloc[0, 1] = 0

In [11]: a
Out[11]:
<IntegerArray>
[0, 2]
Length: 2, dtype: Int64

In [12]: b
Out[12]: array([1, 2])
```

specifying `copy=True/False` will make things always copy or not, unless you have multiple values consolidated into a single block, in which chase there's an unavoidable copy.
Are people OK with https://github.com/pandas-dev/pandas/pull/34872#issuecomment-646850479? https://github.com/pandas-dev/pandas/pull/34872/files#diff-1e79abbbdd150d4771b91ea60a4e1cc7R361 has a description of the proposed behavior.
What is the reason for excepting EAs contained in a dict? I think would be nice for them to behave the same was as numpy arrays
> What is the reason for excepting EAs contained in a dict?

Excepting from a copy by default? That's the existing behavior for dict of EAs. This PR is making it so that an explicit `copy=True/False` will / won't copy.
Maybe misunderstanding something but I was asking about the default situation, i.e. when a user doesn't provide copy explicitly. What's the reason for the behavior to diverge between EAs / ndarrays in that case?
> What's the reason for the behavior to diverge between EAs / ndarrays in that case?

I don't know the original reason (perhaps consolidation?), but that's the existing behavior. IMO it's not worth changing at this point, but we can consider it.
same here with this PR, this should wait for 1.2
Then we'll need to figure out if there are any API breaking changes reported in https://github.com/pandas-dev/pandas/issues/32960. At a glance, I don't think there are any. I think https://github.com/pandas-dev/pandas/pull/32831 was identified as the reason for the new behavior (which is a bugfix) so we should be OK leaving this till 1.2, but would appreciate a confirmation.
I think this is OK holding for 1.2 if necessary (though I think it's also fine to do now). There won't be a great way to avoid the issue reported at https://github.com/pandas-dev/pandas/issues/32960. Users will need to copy the values prior to passing them to pandas.
I still -1 on this. I think this introduces very odd behavior. I would be ok doing this for 2.0, but this is breaking a lot.
Can you check again? It's not making any breaking changes. 
> We can't honor no-copy for a dict with multiple values of the same dtype: DataFrame({"A": np.array([1, 2]), "B": np.array([1, 2])}) as long as we have consolidation.

That consolidation happens in create_block_manager_from_arrays.  i think we need the copy keyword to propagate down to there and not consolidate when copy=False.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
i think this is worth doing, and that something like #36894 should be incorporated
@TomAugspurger can you resolve merge conflicts and move release note to 1.2
@TomAugspurger if you dont plan on finishing this anytime soon, mind if i take it up?  i think itd be nice to fix and have [opinions](https://github.com/pandas-dev/pandas/pull/36894) about how it should be implemented
Yeah, that'd be great, thanks.

I'll leave this branch around in case you want to take the tests.