BUG: fix str.replace('.','') should replace every character
State: closed

- [ ] closes #24804
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-13 00:02:24 UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=h1) Report
> Merging [#24935](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/83eb2428ceb6257042173582f3f436c2c887aa69?src=pr&el=desc) will **decrease** coverage by `49.48%`.
> The diff coverage is `28.57%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24935/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24935       +/-   ##
===========================================
- Coverage   92.38%   42.89%   -49.49%     
===========================================
  Files         166      166               
  Lines       52388    52392        +4     
===========================================
- Hits        48397    22472    -25925     
- Misses       3991    29920    +25929
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.89% <28.57%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `32.81% <28.57%> (-65.78%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [124 more](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=footer). Last update [83eb242...8061fda](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@d66da60`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `20%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24935/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master   #24935   +/-   ##
=========================================
  Coverage          ?   41.12%           
=========================================
  Files             ?      179           
  Lines             ?    50709           
  Branches          ?        0           
=========================================
  Hits              ?    20854           
  Misses            ?    29855           
  Partials          ?        0
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `41.12% <20%> (?)` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/reshape/melt.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZWx0LnB5) | `12.6% <0%> (ø)` | |
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/24935/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `37.76% <25%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=footer). Last update [d66da60...6267aae](https://codecov.io/gh/pandas-dev/pandas/pull/24935?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

any follow-up code change requests and reviews? @jreback @TomAugspurger 
yes, indeed will be a kind of api change. shall I open another issue to make this change if other maintainers agree with this? Or do you want to do this change in this PR? @jreback 
So as an alternate what if we simply detected the case where the pattern provided is a single / special regex character with `regex=True` and raise a FutureWarning when that happens, noting that we will change the behavior to evaluate that as a regular expression regardless in the future
@WillAyd based on your description, i think it's kind of what this PR is doing, right?
I'm saying don't even update the signature of the function, simply do something like:

```python
if regex=True and pater in re_char_blacklist:
    raise FutureWarning("Single character patterns will be evaluated as regexes in the future")
```

Or something to the effect. Would minimize a lot of the volatility going on here
@WillAyd the `raise` in your pseudocode should probably be a `warnings.warn`?

I (think?) then there is no way to get the desired future behavior without warning. IIUC, your prosposal would warn on

```python
pd.Series(['aa']).str.replace('.', 'b', regex=True)
```

I think we'll need a period of

1. `regex=None` (default): warning and output `aa`
2. `regex=True`: no warning, output `bb`
3. `regex=False`: no warning, output `aa`
Yea sorry meant warn instead of raise. Why do we need regex=None at all? I was hoping to avoid that API change
Does the rest of 
https://github.com/pandas-dev/pandas/pull/24935#issuecomment-468353395 answer your question? The short version is (IIUC), your proposal wouldn't allow for option 2 (`regex=True` interprets `.` as a regex, and doest *not* warn).
Hmm unless I'm mistaken my proposal would only affect option two. So something like:

```python
if regex=True and pater in {'.', '^', '$', '*', '+', '?', '}'}:
   warnings.warn("You've passed a single special character in with regex=True which is evaluated as a single character. This will change in the future to match regex behavior")
```

Warns the user about this behavior in the future but doesn't impact their current code, allowing for a smoother deprecation in the future.

To your point though the result of option 2 would still stay 'aa' as is today just with the warning
Right, so I think we understand each other.

My argument is that we shouldn't add a warning without a concrete action a
user can take to avoid it. Else it'll just annoy them :)

On Thu, Feb 28, 2019 at 11:34 AM William Ayd <notifications@github.com>
wrote:

> Hmm unless I'm mistaken my proposal would only affect option two. So
> something like:
>
> if regex=True and pater in {'.', '^', '$', '*', '+', '?', '}'}:
>    warnings.warn("You've passed a single special character in with regex=True which is evaluated as a single character. This will change in the future to match regex behavior")
>
> Warns the user about this behavior in the future but doesn't impact their
> current code, allowing for a smoother deprecation in the future.
>
> To your point though the result of option 2 would still stay 'aa' as is
> today just with the warning
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24935#issuecomment-468364468>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIhcVxvePpsXFK79RPv9m5d1NITnsks5vSBMugaJpZM4aTbKn>
> .
>

Ah I gotcha. So in my mind if a user is passing a single special character and regex=True (whether explicitly or via the default argument) then it is a current "misuse" of that parameter. 

If the warning said something like "explicitly pass regex=False to maintain current behavior" would that be better?
Then we'd need to wait to actually fix the bug. Since we have a PR now, I'd
prefer to go with the (pretty-well established) pattern of using None to
indicate
that the user needs to take explicit action to avoid the warning.

On Thu, Feb 28, 2019 at 11:48 AM William Ayd <notifications@github.com>
wrote:

> Ah I gotcha. So in my mind if a user is passing a single special character
> and regex=True (whether explicitly or via the default argument) then it is
> a current "misuse" of that parameter.
>
> If the warning said something like "explicitly pass regex=False to
> maintain current behavior" would that be better?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24935#issuecomment-468369265>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIq6HY69PTaFuoP0sGH9qW_vUS3w5ks5vSBZjgaJpZM4aTbKn>
> .
>

Many thanks for your feedback and discussions @TomAugspurger @WillAyd 
My English is not very good, and i might get distracted by your discussions, could you give me any hint on what kind of change I need to make (or give a general direction to make me a bit clearer)? as far as i understand from your discussions, seems warning messages should be changed a bit? @WillAyd @TomAugspurger  many thanks and sorry for asking this.
No worries. I'd recommend waiting, I don't think a decision has been made
yet.

On Thu, Feb 28, 2019 at 2:07 PM Kaiqi Dong <notifications@github.com> wrote:

> Many thanks for your feedback and discussions @TomAugspurger
> <https://github.com/TomAugspurger> @WillAyd <https://github.com/WillAyd>
> My English is not very good, and i might get distracted by your
> discussions, could you give me any hint on what kind of change I need to
> make (or give a general direction to make me a bit clearer)? as far as i
> understand from your discussions, seems warning messages should be changed
> a bit? @WillAyd <https://github.com/WillAyd> @TomAugspurger
> <https://github.com/TomAugspurger> many thanks and sorry for asking this.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24935#issuecomment-468418684>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIuj6ifkRxKYOEfYoCbJ-HchcPieaks5vSDbvgaJpZM4aTbKn>
> .
>

@WillAyd can you summarize your aversion to the (typical) use of None as a backwards compatible sentinel value for when we should warn?
Not terribly against it if that’s our process

Sent from my iPhone

> On Mar 3, 2019, at 5:08 AM, Tom Augspurger <notifications@github.com> wrote:
> 
> @WillAyd can you summarize your aversion to the (typical) use of None as a backwards compatible sentinel value for when we should warn?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

can you merge master
Hi, @jreback I just merged master and fix conflict!
something still failing
@jreback i just merged the master and fix test error
This one has been around for a while. Just so we are all on the same page - is the intention here definitely to force users down the road to explicitly provide regex=True or regex=False? I know that was mentioned in comments above just not sure we agreed to it.

An easy alternative would just be to raise a DeprecationWarning when a single special character gets passed and actually change behavior at a later date, which I think is what @TomAugspurger suggests in https://github.com/pandas-dev/pandas/issues/24804#issuecomment-457315905
I don’t see a reason to make the regex flag required.

> On Apr 9, 2019, at 23:54, William Ayd <notifications@github.com> wrote:
> 
> This one has been around for a while. Just so we are all on the same page - is the intention here definitely to force users down the road to explicitly provide regex=True or regex=False? I know that was mentioned in comments above just not sure we agreed to it.
> 
> An easy alternative would just be to raise a DeprecationWarning when a single special character gets passed and actually change behavior at a later date, which I think is what @TomAugspurger suggests in #24804 (comment)
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

ok, re-reading: https://github.com/pandas-dev/pandas/pull/24809#discussion_r248670917

that looks like a good idea, not sure where that is in implementation.
@jreback thanks for your comment! could you explain a little bit more? I think the change I made in this PR was following @TomAugspurger suggestion in 24809 
@charlesdong1991 can you merge master and we'll see where this is
@jreback I merged to master and resolved conflicts, pls take a look
I just reread through the issue, and my thinking has changed a bit.

Do we think that `.str.replace('.', '...')` replacing every character is actually the most *useful* thing for users? Or is the change primarily motivated by the inconsistency?

If we think that most users will expect `.str.replace(".")` to only replace periods, then perhaps we just document that we don't auto-regex single-character `pat` (`.`, `^`, `$`, `|`) and tell users to pass `re.compile(pat)` if they want that?