API: Deprecate regex=True default in Series.str.replace
State: closed

From some old discussion it looks like there was interest in changing the default value of regex from True to False within Series.str.replace. I think this makes sense since it would align this method with others within pandas along with the standard library, and would also make fixing https://github.com/pandas-dev/pandas/issues/24804 slightly less disruptive. I'm assuming this would warrant a deprecation note in the next whatsnew?

I think this part of the docs will need to be updated: https://pandas.pydata.org/pandas-docs/stable/user_guide/text.html#splitting-and-replacing-strings

https://github.com/pandas-dev/pandas/pull/24809

Labels:
name: replace
description: replace method
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
There are many tests that are using this (and are now raising a warning) that would need updating.

While I agree that a default of `regex=False` would probably be better ideally, warning about this *in all cases* also seems quite annoying .. 
Would it somehow be possible to detect in advance if the pattern is regex-like or rather a plain string (in which case replacing it with regex=False would not give a different result). Probably not that straightforward given the complexity of regexes .. (but maybe some simple cases, like only characters, could be detected)
> There are many tests that are using this (and are now raising a warning) that would need updating.
> 
> While I agree that a default of `regex=False` would probably be better ideally, warning about this _in all cases_ also seems quite annoying ..
> Would it somehow be possible to detect in advance if the pattern is regex-like or rather a plain string (in which case replacing it with regex=False would not give a different result). Probably not that straightforward given the complexity of regexes .. (but maybe some simple cases, like only characters, could be detected)

Yeah this is a good point, I think looking for regex special characters and only then warning could make the most sense
> Can you also add a whatsnew note?

Would this be an update for 1.1.4 or 1.2?
> Would this be an update for 1.1.4 or 1.2?

Something that is not a regression or critical bug fix should in principle never be added to a micro bug fix release, and we generally see new warnings also as new behaviour. So 1.2 as you already did ;)
Should we also say in the v1.2.0 deprecation note that single character regexes will no longer be treated as literal strings once the default value changes, since this is long-standing documented behavior (even if arguably a bug)?
> Should we also say in the v1.2.0 deprecation note that single character regexes will no longer be treated as literal strings once the default value changes, since this is long-standing documented behavior (even if arguably a bug)?

yes, you are excluding them from the warning now though; can we provide a more specific warning message for single char regexes?
> yes, you are excluding them from the warning now though; can we provide a more specific warning message for single char regexes?

Added an additional warning for single character regexes:

```
[ins] In [3]: s.str.replace(".", "")
<ipython-input-3-ee1a585145f8>:1: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.
  s.str.replace(".", "")
Out[3]: 
0    a
1    b
dtype: object

[ins] In [4]: s.str.replace("^.", "")
<ipython-input-4-3940e91d28a6>:1: FutureWarning: The default value of regex will change from True to False in a future version.
  s.str.replace("^.", "")
Out[4]: 
0    
1    
dtype: object
```
> looks good. doc comment & pls add a deprecation note in the whatsnew (also add to the deprecation master issue) #30228

Updated the OP in that issue
merge on most builds green (some are giving issues now).