Repository: pandas-dev/pandas

Issue:
Series.str.replace() is not actually the same as str.replace()
#### Code Sample

```python
In [1]: import pandas as pd

In [2]: series = pd.Series(['a', '(b)'])

In [3]: series.str.replace('a', '[a]')
Out[3]: 
0    [a]
1    (b)
dtype: object

In [4]: series.str.replace('(b)', '[b]')  # unexpected behavior
Out[4]: 
0        a
1    ([b])
dtype: object

In [5]: series.str.replace('\(b\)', '[b]')  # need to escape
Out[5]: 
0      a
1    [b]
dtype: object

In [6]: '(b)'.replace('(b)', '[b]')   # Python str.replace is different, uses literal string
Out[6]: '[b]'

```
#### Problem description

The documentation for ``Series.str.replace`` says that it takes a "string or compiled regex" ... "String can be a character sequence or regular expression." ... "When _repl_ is a string, every _pat_ is replaced as with str.replace()"

However, that's not what is happening - it appears it's interpreting a string as a regex, so you need to escape characters like parentheses.

#### Expected Output

I would expect that for vanilla strings, it works like regular Python str.replace() - using literal strings instead of regexes.

Alternatively the documentation could be updated, but I think the Python str.replace() behavior is what most users would expect.

#### Output of ``pd.show_versions()``

<details>
# Paste the output here pd.show_versions() here
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-83-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.20.2
pytest: 3.1.2
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.0
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
from the doc-string

> Signature: Series.str.replace(self, pat, repl, n=-1, case=None, flags=0)
Docstring:
Replace occurrences of pattern/regex in the Series/Index with
some other string. Equivalent to :meth:`str.replace` or
:func:`re.sub`.

it is a bit ambiguous, though so is a pattern like ``(b)``, is it a regex or not? This is almost always a regex. So I suppose a case could be made for:

- adding a ``.sub`` function which is by-definition a regex and making ``.replace`` strict
- adding a ``regex=True`` arg to control whether to interpret this as a regex.
I'm fine with either solution, not sure which is more intuitive or less disruptive to existing practice. I too had  considered suggesting an additional "regex" param but had forgotten it when I went to write the issue up; if we go that way I'd argue for defaulting to regex=False.

For what it's worth, a user might not know they are passing a string that "looks like a regex". Maybe you got a result from some other operation and didn't inspect it, but still passed it in to series.str.replace(). I stumbled upon this issue doing something like the following: imagine an item description with a parenthetical remark - e.g. "Men's shirt (blue)" - and you try and replace all instances of that with some other text, maybe just "Men's clothing". When some of your replacements fail to take, it's pretty surprising.

```python
In [1]: import pandas as pd

In [2]: series = pd.Series(['No parens', 'Some text (with parens)'])

In [3]: series.str.replace('No parens', 'Replaced')
Out[3]: 
0                   Replaced
1    Some text (with parens)
dtype: object

In [4]: series.str.replace('Some text (with parens)', 'Replaced') # fails to replace
Out[4]: 
0                  No parens
1    Some text (with parens)
dtype: object

In [5]: series.str.replace(series.max(), 'Replaced') # nothing gets replaced!
Out[5]: 
0                  No parens
1    Some text (with parens)
dtype: object

```

We just hit this issue. At this point the problem goes back at least to 0.16 and is still present in the current 0.23 dev. Problem is at 

```    
use_re = is_compiled_re or len(pat) > 1 or flags or callable(repl) 
```

Thus any pattern len>1 is interpreted as a regex, whether it actually can be parsed as a regex or not.

At this point, any change that made it non-default to regex would be breaking, so I think the options are:

1) Update the documentation only. This function does not support replacing "character sequences", it supports replacing a single character OR a regex if multiple characters.

2) Add regex=True and make the above line something like

```
use_re = is_compiled_re or (regex and (len(pat) > 1 or flags)) or callable(repl)
```

Any thoughts? Current behavior really is inconsistent with the doc.
+1 for adding `regex=True`. We have this elsewhere in, e.g. `.str.contains`.

Interested in making a PR @Liam3851?
I can do a PR. Question on API, though, before I write. Suppose someone gives us
```
ser.str.replace(re.compile('[abc]'), 'd', regex=False)
```
Should that raise or should we just silently use regex?

Also, while 
```
use_re = is_compiled_re or (regex and (len(pat) > 1 or flags)) or callable(repl)
```
maintains back compat, it does mean that `regex=True` with a single character always basically implies `regex=False`. I suppose it could be confusing for someone to explicitly write
```
ser.str.replace('.', 'a', regex=True)
```
and have it only replace periods and not all characters. But maybe that's such a rare request anyway we can let it go for the sake of back compat?



Resolving patch:
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -306,7 +306,7 @@ def str_endswith(arr, pat, na=np.nan):
     return _na_map(f, arr, na, dtype=bool)
 
 
-def str_replace(arr, pat, repl, n=-1, case=None, flags=0):
+def str_replace(arr, pat, repl, n=-1, case=None, flags=0, regex=True):
     r"""
     Replace occurrences of pattern/regex in the Series/Index with
     some other string. Equivalent to :meth:`str.replace` or
@@ -337,25 +337,50 @@ def str_replace(arr, pat, repl, n=-1, case=None, flags=0):
     flags : int, default 0 (no flags)
         - re module flags, e.g. re.IGNORECASE
         - Cannot be set if `pat` is a compiled regex
+    regex : boolean, default True
+        - If True, assumes the passed-in pattern is a regular expression.
+        - If False, treats the pattern as a literal string
+        - Cannot be set to False if `pat` is a compiled regex or `repl` is
+          a callable.
+
+        .. versionadded:: 0.23.0
 
     Returns
     -------
     replaced : Series/Index of objects
 
+    Raises
+    ------
+    ValueError
+        * if `regex` is False and `repl` is a callable or `pat` is a compiled
+          regex
+        * if `pat` is a compiled regex and `case` or `flags` is set
+
     Notes
     -----
     When `pat` is a compiled regex, all flags should be included in the
-    compiled regex. Use of `case` or `flags` with a compiled regex will
-    raise an error.
+    compiled regex. Use of `case`, `flags`, or `regex=False` with a compiled
+    regex will raise an error.
 
     Examples
     --------
-    When `repl` is a string, every `pat` is replaced as with
-    :meth:`str.replace`. NaN value(s) in the Series are left as is.
+    When `pat` is a string and `regex` is True (the default), the given `pat`
+    is compiled as a regex. When `repl` is a string, it replaces matching
+    regex patterns as with :meth:`re.sub`. NaN value(s) in the Series are
+    left as is:
+
+    >>> pd.Series(['foo', 'fuz', np.nan]).str.replace('f.', 'ba', regex=True)
+    0    bao
+    1    baz
+    2    NaN
+    dtype: object
 
-    >>> pd.Series(['foo', 'fuz', np.nan]).str.replace('f', 'b')
-    0    boo
-    1    buz
+    When `pat` is a string and `regex` is False, every `pat` is replaced with
+    `repl` as with :meth:`str.replace`:
+
+    >>> pd.Series(['f.o', 'fuz', np.nan]).str.replace('f.', 'ba', regex=False)
+    0    bao
+    1    fuz
     2    NaN
     dtype: object
 
@@ -397,6 +422,7 @@ def str_replace(arr, pat, repl, n=-1, case=None, flags=0):
     1    bar
     2    NaN
     dtype: object
+
     """
 
     # Check whether repl is valid (GH 13438, GH 15055)
@@ -404,27 +430,33 @@ def str_replace(arr, pat, repl, n=-1, case=None, flags=0):
         raise TypeError("repl must be a string or callable")
 
     is_compiled_re = is_re(pat)
-    if is_compiled_re:
-        if (case is not None) or (flags != 0):
-            raise ValueError("case and flags cannot be set"
-                             " when pat is a compiled regex")
-    else:
-        # not a compiled regex
-        # set default case
-        if case is None:
-            case = True
-
-        # add case flag, if provided
-        if case is False:
-            flags |= re.IGNORECASE
-
-    use_re = is_compiled_re or len(pat) > 1 or flags or callable(repl)
-
-    if use_re:
-        n = n if n >= 0 else 0
-        regex = re.compile(pat, flags=flags)
-        f = lambda x: regex.sub(repl=repl, string=x, count=n)
+    if regex:
+        if is_compiled_re:
+            if (case is not None) or (flags != 0):
+                raise ValueError("case and flags cannot be set"
+                                 " when pat is a compiled regex")
+        else:
+            # not a compiled regex
+            # set default case
+            if case is None:
+                case = True
+
+            # add case flag, if provided
+            if case is False:
+                flags |= re.IGNORECASE
+        if is_compiled_re or len(pat) > 1 or flags or callable(repl):
+            n = n if n >= 0 else 0
+            compiled = re.compile(pat, flags=flags)
+            f = lambda x: compiled.sub(repl=repl, string=x, count=n)
+        else:
+            f = lambda x: x.replace(pat, repl, n)
     else:
+        if is_compiled_re:
+            raise ValueError("Cannot use a compiled regex as replacement "
+                             "pattern with regex=False")
+        if callable(repl):
+            raise ValueError("Cannot use a callable replacement when "
+                             "regex=False")
         f = lambda x: x.replace(pat, repl, n)
 
     return _na_map(f, arr)
@@ -1596,9 +1628,9 @@ def match(self, pat, case=True, flags=0, na=np.nan, as_indexer=None):
         return self._wrap_result(result)
 
     @copy(str_replace)
-    def replace(self, pat, repl, n=-1, case=None, flags=0):
+    def replace(self, pat, repl, n=-1, case=None, flags=0, regex=True):
         result = str_replace(self._data, pat, repl, n=n, case=case,
-                             flags=flags)
+                             flags=flags, regex=regex)
         return self._wrap_result(result)
 
     @copy(str_repeat)


Test patch:
diff --git a/pandas/tests/test_strings.py b/pandas/tests/test_strings.py
--- a/pandas/tests/test_strings.py
+++ b/pandas/tests/test_strings.py
@@ -530,6 +530,27 @@ def test_replace_compiled_regex(self):
         exp = Series(['foObaD__baRbaD', NA])
         tm.assert_series_equal(result, exp)
 
+    def test_replace_literal(self):
+        # GH16808 literal replace (regex=False vs regex=True)
+        values = Series(['f.o', 'foo', NA])
+        exp = Series(['bao', 'bao', NA])
+        result = values.str.replace('f.', 'ba')
+        tm.assert_series_equal(result, exp)
+
+        exp = Series(['bao', 'foo', NA])
+        result = values.str.replace('f.', 'ba', regex=False)
+        tm.assert_series_equal(result, exp)
+
+        # Cannot do a literal replace if given a callable repl or compiled
+        # pattern
+        callable_repl = lambda m: m.group(0).swapcase()
+        compiled_pat = re.compile('[a-z][A-Z]{2}')
+
+        pytest.raises(ValueError, values.str.replace, 'abc', callable_repl,
+                      regex=False)
+        pytest.raises(ValueError, values.str.replace, compiled_pat, '',
+                      regex=False)
+
     def test_repeat(self):
         values = Series(['a', 'b', NA, 'c', NA, 'd'])