PDEP-11: Change default of dropna to False
State: closed

cc @pandas-dev/pandas-core 

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Deprecate
description: Functionality to remove in pandas
name: Stale
name: PDEP
description: pandas enhancement proposal

Comments:
Will need to give this some thought.  The "even further" possibility is to deprecate the argument entirely and tell users to do `obj.whatever().dropna()` to get the old behavior
> Will need to give this some thought. The "even further" possibility is to deprecate the argument entirely and tell users to do `obj.whatever().dropna()` to get the old behavior

Wouldn't this not work for certain functions. If you sum over values with na and then drop you get nothing, whilst if the sum function knows to drop you will return a value. This would be important in the groupby.
@attack68 (https://github.com/pandas-dev/pandas/pull/53094#issuecomment-1536552388): Consider also `obj.dropna(...).whatever()`

> The "even further" possibility is to deprecate the argument entirely and tell users to do `obj.whatever().dropna()` to get the old behavior

@jbrockmendel - I also think further deprecation might be desirable for some of these methods. However they would then act as `dropna=False`, and deprecating an argument _and_ changing the behavior to the non-default option is not a great user experience. So I think we should first change the default and then deprecate if desired. But even if we don't deprecate the argument, I would like to see the default change. So these are independent.
My point being that this doesn't work:
![Screenshot 2023-05-06 at 16 09 33](https://user-images.githubusercontent.com/24256554/236629175-976e9166-e28e-413c-904c-4b44bf870ad2.png)

If you want to get the answer 3, you can currently do `df.sum(skipna=True).sum()`, where the dropna is within the method. Or, in this trivial case you could wrangle it, `df.unstack().dropna().sum()`, but that *might* not work generally?
This should take the ``skipna`` keyword into consideration, right now we are ignoring NAs consistently, this PDEP would make things kind of inconsistent (not that this is a bad thing, but it should at least get mentioned)
@attack68 - `skipna` is not part of the PDEP, I'll make this more clear in an update. And I don't think anyone is thinking it should be deprecated in methods like sum. 

Also, deprecating any `dropna` is not part of the PDEP.
I am generally in support of this PDEP. I think NA information is valuable in many processes and should not be dropped by default.
I understand `skipna` is not part of this PDEP and nothing is being deprecated, but it is worth mentioning as @phofl says.

I've just raised the points to try to discover if there are any processes / calculation chains that might be made more difficult by the change, including any user defined functions and/or in groupbys.

Thanks @jbrockmendel @attack68 @mroeschke @phofl @Dr-Irv - I've updated the PDEP based on your feedback.
I'm +1 on this change.

Do we have a rough timeline for Pandas 3.0? I'm a bit worried about noise from this when working interactively, could we consider implementing the deprecation late in the v2.x release cycle rather than early? Or is the 2.x cycle so short that timing this will not matter?
I am +1 on this proposal as well. 

> Do we have a rough timeline for Pandas 3.0? 

We would like to release 3.0 in 1 year from 2.0, so April 3rd 2024 ideally. 2.1 should be released in August 2.1 where this deprecation could be introduced
@mroeschke: Is there then not going to be a pandas 2.2?

@topper-123: I'm happy to wait as long as possible in 2.x. The deprecation will be straightforward.
Thanks for the PDEP!

I am +1 on the proposal for the groupby-related methods (the variations of `groupby` and `value_counts`).

I am less sure that we should also do this for other methods, or at least I think those merit some specific case-by-case discussion (AFAIK the linked previous discussions were all about the groupby/value_counts-related cases?)

* `mode()` is a tricky one, because on the one hand it is an aggregation (and all our other aggregations skip NAs by default, although the keyword _name_ here is `dropna` and not `skipna`), while on the other hand it is very much related to "counting values" (and for `value_counts` I think it makes sense to change the default to `dropna=False`)
  * Some related discussion about this at https://github.com/apache/arrow/issues/34173
  * There wasn't much discussion about the keyword name (`skipna` vs `dropna`) when this was introduced ([#20779](https://github.com/pandas-dev/pandas/pull/20779)), but from the issue it seems it was modeled after `value_counts`
  * It might be worth checking some other data tools what their behaviour is for "mode" with nulls.

* `nunique()` is a bit similar as `mode()`, since it is also an aggregation function, but also related to counts. But for example, SQL ``COUNT(DISTINCT col)`` also skips nulls.

* `pivot_table()`: it seems that the `dropna` keyword is used for multiple things? It is passed through to the underlying `groupby` call (and here it of course makes sense to follow the change in default), but it is also used to post-process the result and to calculate the margins (the docstring says: *"If True, rows with a NaN value in any column will be omitted before computing margins."*). Have we evaluated what the impact of the change is here? 

* For `stack()` I was going to say that I thought this was only about dropping NAs that were _introduced_ because of the reshaping operations (and not NAs that already existed before), but it seems to be a bit more complicated than just that. The last example in the [docstring](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.stack.html) illustrates the effect of `dropna`, and it's also dropping at least one pre-existing NA value. In that case, it seems this is a good change to preserve those original NAs in the output.  
  But if you are stacking multiple levels at once, you can write an example where the NAs that you get with `dropna=False` are actually _new_ NAs, and for this case I am not sure this is necessarily the best behaviour:

    ```python
    >>> df = pd.DataFrame(np.arange(6).reshape(2,3), columns=pd.MultiIndex.from_tuples([('A','x'), ('A','y'), ('B','z')], names=['Upper', 'Lower']))
    >>> df
    Upper  A     B
    Lower  x  y  z
    0      0  1  2
    1      3  4  5
    >>> df.stack(level=[0,1])
       Upper  Lower
    0  A      x        0.0
              y        1.0
       B      z        2.0
    1  A      x        3.0
              y        4.0
       B      z        5.0
    dtype: float64
    
    >>> df.stack(level=[0,1], dropna=False)
       Upper  Lower
    0  A      x        0.0
              y        1.0
              z        NaN
       B      x        NaN
              y        NaN
              z        2.0
    1  A      x        3.0
              y        4.0
              z        NaN
       B      x        NaN
              y        NaN
              z        5.0
    dtype: float64
    ```

  * There has been some discussion on this topic several years ago, see https://github.com/pandas-dev/pandas/issues/8851, https://github.com/pandas-dev/pandas/pull/9023#issuecomment-72812666 (and comments below that)
  * I think I would be in favor of changing `dropna=True` to `dropna=False`, _if_ this would only about the pre-existing NAs, and not about not dropping the NAs introduced by product-wise combinations of MultiIndex levels (I would say that even `dropna=True` should _only_ drop those introduced NAs, and not the pre-existing ones, in which case we could keep the current default, but effectively get the desired default behaviour)
> @mroeschke: Is there then not going to be a pandas 2.2?

I think there should be a 2.2 release in December with the every-4-months for a minor release
Thanks @jorisvandenbossche - for now I'd like to move discussions on the behavior of `dropna` for `stack` (#53515) and `pivot_table` (#53521) to those issues as I think they will require more lengthy discussion. If at the end they do not sufficiently address your concerns, we can bring them back here.

> I am less sure that we should also do this for other methods, or at least I think those **merit some specific case-by-case discussion**

Emphasis mine. Certainly no objection to such discussions, but I do think there is high value in having a consistent default for `dropna` across pandas. It leads to less surprises and is less taxing on our users to have a consistent default.

> `mode()` is a tricky one, because on the one hand it is an aggregation... It might be worth checking some other data tools what their behaviour is for "mode" with nulls.

I agree this could either have `dropna` defaulting to False or `skipna` defaulting to True. I'm okay either way here and can see arguments for both. At the end of the day I think we just have to make a call as there doesn't seem to be a standard. I'd lean toward changing to `skipna` so as not to change default behavior but have consistency in default arguments. Other languages/libraries:
 
 - Those that do not implement mode: R, SQL, [GSL](https://www.gnu.org/software/gsl/doc/html/) (C/C++)
 - Those that implement mode and include nulls by default: polars, SciPy
 - Those that implement mode and skip nulls: pyarrow, pyspark, scikit-learn

> `nunique()` is a bit similar as `mode()`, since it is also an aggregation function, but also related to counts. But for example, SQL `COUNT(DISTINCT col)` also skips nulls.

pandas impelments `count` and this skips NA values (with no option to not skip). That is the method I think should be compared to SQL `COUNT`, not `nunique`. In addition, I think it is important that `ser.nunique() == len(ser.unique())` and that is currently not the case.

Thanks @lithomas1! I indeed missed checking the top namespace - will add them to the PDEP. Some thoughts on each:

 - `crosstab`: The `dropna` argument is only used via passing to `pivot_table`, so I think we can combine this with the discussion on `pivot_table` (#53521).
 - `lreshape`: The docs here on `dropna` are wrong; this is equivalent to calling `DataFrame.dropna(how='any')` prior to the call to `lreshape`. I think we can deprecate `dropna` here outright.
 - `HDFStore.put|append|append_to_multiple`: These already default to `dropna=False`.

This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@rhshadrach what's the status here? was this announced to solicit feedback from the community?
Thanks for the ping here @simonjayhawkins - I've been meaning to post an update. I put the brakes on this from the feedback in https://github.com/pandas-dev/pandas/pull/53094#issuecomment-1567937468; namely the behavior of `dropna` in `pivot_table` and `crosstab`. I tried to summarize these issues in #53521.

In addition, I think we should decide to have `mode` with the default `dropna=False` or `skipna=True`. I've argued for the latter, but would be happy with either option. Finally, some opposition was voiced against `nunique` defaulting to `dropna=False`; I've argued against this in https://github.com/pandas-dev/pandas/pull/53094#issuecomment-1578462799 but have not heard if others find my arguments compelling.

In my mind, there are two ways forward:

 - change the default to `dropna` in all methods except for `pivot_table` and `crosstab` and clean those methods up in the future; or
 - wait to change the default of `dropna` until `pivot_table` and `crosstab` are fixed.

I am okay with either way, but would lean toward the first.

cc @jorisvandenbossche 


+1

From a user perspective working in data and business analytics, the default behavior in `.groupby` has been the source of several headaches. My team has varied backgrounds including SAS, SQL, and R, so we come in with different assumptions about default behavior.

A few of the issues it's caused:

1. To get around the problem, some users used `.fillna("n/a")` which broke type consistency and made downstream code more complicated. Later on, when trying to resolve this in a refactor, we were tripped up by the fact that rows were now being dropped.
2. On upgrading to pandas 2.0, the treatment of NA values in some areas changed. This again led to rows being dropped unexpectedly.
3. We were merging in reference data, then doing a groupby that included the new columns. When the source data changed to include a value that wasn't present in our reference table, we weren't aware of it and were under-reporting some details.

In most cases, the default behavior doesn't matter because there are not and would never be null values. However, the impact of incorrectly using the default value was strong enough that I wrote a custom flake8 linting rule to enforce explicitly defining the `dropna` argument.
@rhshadrach maybe reject this as withdrawn by the author based in your comments? You can open a new PDEP when you settle on how to move forward. That's what I did with PDEP-9
Sounds good @datapythonista. I plan on tackling `pivot_table` and `crosstab` as part of 4.0 development, and will reopen later.
I'm ok with closing, but what we've been doing so far is merge PDEPs with status rejected, with a short summary of the reasons for withdrawing it. Personally I think this is useful since there is a decent amount of work in writing the PDEP and discussing it, that will be lost if we close it, but will have a link to it in our roadmap page, and useful for future discussions on the topic.

But as said, up to you, if you prefer just closing it, that's surely fine with me.
I'll be reopening this, I don't follow what will be lost.
Ah, sorry, I read yoiur last message too fast, I thought this wouldn't be reopened. You can ignore my last comment, sorry for the noise.