BUG: Fix regression on DataFrame.replace for regex
State: closed

The commit ensures that the replacement for regex is not confined to the 
beginning of the string but spans all the characters within. The 
behaviour is then consistent with versions prior to 0.24.0.

- [x] closes #25259
- [x] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
this seems highly related to https://github.com/pandas-dev/pandas/issues/24804 and a corresponding PR #24935 

can you have a look
i just took a quick look... seems the problems don't look the same... shall i move whatsnew in my PR to 0.24.2 from 0.25? @jreback  
@jreback and @charlesdong1991 Hi, thanks for taking a look at this. I have very little knowledge about `pandas` internals but, from the description of #24804, it does not seem to be the same problem. I will try to include a test specifically for the case where the regex pattern is inside the string and not at the beginning. The current change already passes all the tests in `tests/frame/test_replace.py`.
Hello @PetitLepton! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on February 28, 2019 at 03:16 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=h1) Report
> Merging [#25266](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4d44a2aff08c3075d9472e361bd7448a0472494e?src=pr&el=desc) will **increase** coverage by `0.21%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25266/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25266      +/-   ##
==========================================
+ Coverage   91.72%   91.93%   +0.21%     
==========================================
  Files         173      166       -7     
  Lines       52831    52187     -644     
==========================================
- Hits        48457    47979     -478     
+ Misses       4374     4208     -166
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `100% <ø> (+9.71%)` | :arrow_up: |
| #single | `42.13% <0%> (+0.4%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `94.83% <0%> (ø)` | :arrow_up: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `99.34% <0%> (ø)` | :arrow_up: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `95.45% <0%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.16% <0%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `90.14% <0%> (ø)` | :arrow_up: |
| ... and [131 more](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=footer). Last update [4d44a2a...1b05715](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=h1) Report
> Merging [#25266](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c9863865c217867583e8f6592ba88d9200601992?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25266/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25266      +/-   ##
==========================================
- Coverage   91.74%   91.74%   -0.01%     
==========================================
  Files         173      173              
  Lines       52923    52923              
==========================================
- Hits        48555    48554       -1     
- Misses       4368     4369       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.31% <100%> (ø)` | :arrow_up: |
| #single | `41.73% <25%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `94.83% <100%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25266/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.57% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=footer). Last update [c986386...89e8ae5](https://codecov.io/gh/pandas-dev/pandas/pull/25266?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback I added a test for the change and an entry in `whatsnew`. Please let me know if it matches `pandas` standard.
ideally also replicate the test from the OP as well (this could be an alternative to changing the existing test)
@PetitLepton can you address comments and merge master?
@WillAyd @jreback My apologies for the delay. I tried to follow the recommendations, reverting the initial test to its genuine form and added a simple test referring to the issue above.
Two tests are failing but I don't have access to the error messages.
Hmm should have access but in any case here's the error:

```sh
##[error]./pandas/core/internals/managers.py(556,48): error E128: continuation line under-indented for visual indent
```

Also if you merge master instead of rebasing shouldn't need to force push which is preferable with maintaining GitHub comment history
Thanks @PetitLepton!
@WillAyd and @jreback Thank you very much for your patience and guidance, it is my first (small) contribution to an open source project and I am really happy to have been able to help `pandas`. It is an amazing tool.

@TomAugspurger Thank you for approving.
Just spent ages confused by this behaviour, thanks for fixing it @PetitLepton !

For those who need a quick-fix the following will work around the issue:
``` python3
# add this to the start of to_replace: r"(.*?)"
# add this to the start of replacement value: r"\1"
# e.g:
df = pd.DataFrame({'first': ['abc', 'bca', 'cab']})
df.replace({r"(.*?)"+'a':  r"\1"+'.'}, regex=True)

```