Repository: pandas-dev/pandas
Pull request: [ENH] Add "fullmatch" matching mode to Series.str [#32806]
- [X] closes #32806
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

This is my first PR against this project, so apologies if I've missed any steps in the process. I'm assuming that I'm supposed to fill in the checklist above myself.

This pull request adds the `fullmatch` regular expression matching mode to the other modes already present under the `Series.str` namespace. For example:

```python
>>> s = pd.Series(["foo", "bar", "foobar"])
>>> s.str.fullmatch("foo")
0     True
1    False
2    False
dtype: bool
```

The `fullmatch` matching mode restricts matches to those that only match the *entire* string. Note the differences from `match`:
```python
>>> s = pd.Series(["foo", "bar", "foobar"])
>>> s.str.fullmatch("foo")
0     True
1    False
2    False
dtype: bool
>>> s.str.match("foo")
0     True
1    False
2     True
dtype: bool
```

I've also added regression tests and a "what's new" entry.

I have also opened issue #32806 to cover this new feature.

Problem statement:
ENH: Allow regex matching in `fullmatch` mode
#### Problem description

`Series.str` contains methods for all the regular expression matching modes in the `re` package except for `re.fullmatch()`. `fullmatch` only returns matches that cover the entire input string, unlike `match`, which also returns matches that start at the beginning of the string but do not cover the complete string.

One can work around the lack of `fullmatch` by round-tripping to/from numpy arrays and using `np.vectorize`, i.e.

```python
>>> s = pd.Series(["foo", "bar", "foobar"])
>>> my_regex = "foo"
>>> import re
>>> import numpy as np
>>> compiled_regex = re.compile(my_regex)
>>> regex_f = np.vectorize(lambda s: compiled_regex.fullmatch(s) is not None)
>>> matches_array = regex_f(s.values)
>>> matches_series = pd.Series(matches_array)
>>> matches_series
0     True
1    False
2    False
dtype: bool
```

but it would be more convenient for users if `fullmatch` was built in.

The `fullmatch` method was added to the `re` package in Python 3.4. I think that the reason this method wasn't in previous versions of Pandas was that older versions of Python don't have `re.fullmatch`. As of Pandas 1.0, all the supported versions of Python now have `fullmatch`.

I have a pull request ready that adds this functionality. After my changes, the `Series.str` namespace gets a new method `fullmatch` that evaluates `re.fullmatch` over the series. For example:
```python
>>> s = pd.Series(["foo", "bar", "foobar"])
>>> s.str.fullmatch("foo")
0     True
1    False
2    False
dtype: bool
```

[Edit: Simplified the workaround]

Hints from issue discussion:

Commit message: Add type hints to fullmatch and match

Patch to review:
diff --git a/doc/source/user_guide/text.rst b/doc/source/user_guide/text.rst
index 2e4d0fecaf5cf..f67945fc1d52a 100644
--- a/doc/source/user_guide/text.rst
+++ b/doc/source/user_guide/text.rst
@@ -641,21 +641,34 @@ You can check whether elements contain a pattern:
 .. ipython:: python
 
    pattern = r'[0-9][a-z]'
-   pd.Series(['1', '2', '3a', '3b', '03c'],
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
              dtype="string").str.contains(pattern)
 
 Or whether elements match a pattern:
 
 .. ipython:: python
 
-   pd.Series(['1', '2', '3a', '3b', '03c'],
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
              dtype="string").str.match(pattern)
 
-The distinction between ``match`` and ``contains`` is strictness: ``match``
-relies on strict ``re.match``, while ``contains`` relies on ``re.search``.
+.. ipython:: python
+
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
+             dtype="string").str.fullmatch(pattern)
+
+.. versionadded:: 1.1.0
+
+The distinction between ``match``, ``fullmatch``, and ``contains`` is strictness:
+``fullmatch`` tests whether the entire string matches the regular expression;
+``match`` tests whether there is a match of the regular expression that begins
+at the first character of the string; and ``contains`` tests whether there is
+a match of the regular expression at any position within the string. The
+corresponding functions in the ``re`` package for these three match modes are
+``re.fullmatch``, ``re.match``, and ``re.search``, respectively.
 
-Methods like ``match``, ``contains``, ``startswith``, and ``endswith`` take
-an extra ``na`` argument so missing values can be considered True or False:
+Methods like ``match``, ``fullmatch``, ``contains``, ``startswith``, and
+``endswith`` take an extra ``na`` argument so missing values can be considered
+True or False:
 
 .. ipython:: python
 
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 0d3a9a8f969a4..90b24a6b566f4 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -69,6 +69,7 @@ Other enhancements
 - `OptionError` is now exposed in `pandas.errors` (:issue:`27553`)
 - :func:`timedelta_range` will now infer a frequency when passed ``start``, ``stop``, and ``periods`` (:issue:`32377`)
 - Positional slicing on a :class:`IntervalIndex` now supports slices with ``step > 1`` (:issue:`31658`)
+- :class:`Series.str` now has a `fullmatch` method that matches a regular expression against the entire string in each row of the series, similar to `re.fullmatch` (:issue:`32806`).
 - :meth:`DataFrame.sample` will now also allow array-like and BitGenerator objects to be passed to ``random_state`` as seeds (:issue:`32503`)
 -
 
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
index fbc87b1fdac04..9853aec07b8b3 100644
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -2,7 +2,7 @@
 from functools import wraps
 import re
 import textwrap
-from typing import TYPE_CHECKING, Any, Callable, Dict, List, Type, Union
+from typing import TYPE_CHECKING, Any, Callable, Dict, List, Type, Union, Pattern
 import warnings
 
 import numpy as np
@@ -10,7 +10,7 @@
 import pandas._libs.lib as lib
 import pandas._libs.missing as libmissing
 import pandas._libs.ops as libops
-from pandas._typing import ArrayLike, Dtype
+from pandas._typing import ArrayLike, Dtype, Scalar
 from pandas.util._decorators import Appender
 
 from pandas.core.dtypes.common import (
@@ -787,9 +787,15 @@ def rep(x, r):
         return result
 
 
-def str_match(arr, pat, case=True, flags=0, na=np.nan):
+def str_match(
+    arr: ArrayLike,
+    pat: Union[str, Pattern],
+    case: bool = True,
+    flags: int = 0,
+    na: Scalar = np.nan,
+):
     """
-    Determine if each string matches a regular expression.
+    Determine if each string starts with a match of a regular expression.
 
     Parameters
     ----------
@@ -808,6 +814,7 @@ def str_match(arr, pat, case=True, flags=0, na=np.nan):
 
     See Also
     --------
+    fullmatch : Stricter matching that requires the entire string to match.
     contains : Analogous, but less strict, relying on re.search instead of
         re.match.
     extract : Extract matched groups.
@@ -823,6 +830,50 @@ def str_match(arr, pat, case=True, flags=0, na=np.nan):
     return _na_map(f, arr, na, dtype=dtype)
 
 
+def str_fullmatch(
+    arr: ArrayLike,
+    pat: Union[str, Pattern],
+    case: bool = True,
+    flags: int = 0,
+    na: Scalar = np.nan,
+):
+    """
+    Determine if each string entirely matches a regular expression.
+
+    Parameters
+    ----------
+    pat : str
+        Character sequence or regular expression.
+    case : bool, default True
+        If True, case sensitive.
+    flags : int, default 0 (no flags)
+        Regex module flags, e.g. re.IGNORECASE.
+    na : default NaN
+        Fill value for missing values.
+
+    .. versionadded:: 1.1.0
+
+    Returns
+    -------
+    Series/array of boolean values
+
+    See Also
+    --------
+    match : Similar, but also returns `True` when only a *prefix* of the string
+        matches the regular expression.
+    extract : Extract matched groups.
+    """
+    if not case:
+        flags |= re.IGNORECASE
+
+    regex = re.compile(pat, flags=flags)
+
+    dtype = bool
+    f = lambda x: regex.fullmatch(x) is not None
+
+    return _na_map(f, arr, na, dtype=dtype)
+
+
 def _get_single_group_name(rx):
     try:
         return list(rx.groupindex.keys()).pop()
@@ -2762,6 +2813,12 @@ def match(self, pat, case=True, flags=0, na=np.nan):
         result = str_match(self._parent, pat, case=case, flags=flags, na=na)
         return self._wrap_result(result, fill_value=na, returns_string=False)
 
+    @copy(str_fullmatch)
+    @forbid_nonstring_types(["bytes"])
+    def fullmatch(self, pat, case=True, flags=0, na=np.nan):
+        result = str_fullmatch(self._parent, pat, case=case, flags=flags, na=na)
+        return self._wrap_result(result, fill_value=na, returns_string=False)
+
     @copy(str_replace)
     @forbid_nonstring_types(["bytes"])
     def replace(self, pat, repl, n=-1, case=None, flags=0, regex=True):
diff --git a/pandas/tests/test_strings.py b/pandas/tests/test_strings.py
index 6abf174aa7fd2..a654e7ff64e00 100644
--- a/pandas/tests/test_strings.py
+++ b/pandas/tests/test_strings.py
@@ -41,6 +41,7 @@ def assert_series_or_index_equal(left, right):
     ("join", (",",), {}),
     ("ljust", (10,), {}),
     ("match", ("a",), {}),
+    ("fullmatch", ("a",), {}),
     ("normalize", ("NFC",), {}),
     ("pad", (10,), {}),
     ("partition", (" ",), {"expand": False}),
@@ -1176,9 +1177,9 @@ def test_match(self):
         exp = Series([True, np.nan, False])
         tm.assert_series_equal(result, exp)
 
-        values = Series(["fooBAD__barBAD", np.nan, "foo"])
+        values = Series(["fooBAD__barBAD", "BAD_BADleroybrown", np.nan, "foo"])
         result = values.str.match(".*BAD[_]+.*BAD")
-        exp = Series([True, np.nan, False])
+        exp = Series([True, True, np.nan, False])
         tm.assert_series_equal(result, exp)
 
         # mixed
@@ -1208,6 +1209,21 @@ def test_match(self):
         exp = Series([True, np.nan, np.nan])
         tm.assert_series_equal(exp, res)
 
+    def test_fullmatch(self):
+        values = Series(["fooBAD__barBAD", "BAD_BADleroybrown", np.nan, "foo"])
+        result = values.str.fullmatch(".*BAD[_]+.*BAD")
+        exp = Series([True, False, np.nan, False])
+        tm.assert_series_equal(result, exp)
+
+        # Make sure that the new string arrays work
+        string_values = Series(
+            ["fooBAD__barBAD", "BAD_BADleroybrown", np.nan, "foo"], dtype="string"
+        )
+        result = string_values.str.fullmatch(".*BAD[_]+.*BAD")
+        # Result is nullable boolean with StringDtype
+        string_exp = Series([True, False, np.nan, False], dtype="boolean")
+        tm.assert_series_equal(result, string_exp)
+
     def test_extract_expand_None(self):
         values = Series(["fooBAD__barBAD", np.nan, "foo"])
         with pytest.raises(ValueError, match="expand must be True or False"):
@@ -3384,6 +3400,9 @@ def test_match_findall_flags(self):
         result = data.str.match(pat, flags=re.IGNORECASE)
         assert result[0]
 
+        result = data.str.fullmatch(pat, flags=re.IGNORECASE)
+        assert result[0]
+
         result = data.str.findall(pat, flags=re.IGNORECASE)
         assert result[0][0] == ("dave", "google", "com")
 


Code review comments:
diff_hunk: @@ -641,21 +641,34 @@ You can check whether elements contain a pattern:
 .. ipython:: python
 
    pattern = r'[0-9][a-z]'
-   pd.Series(['1', '2', '3a', '3b', '03c'],
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
              dtype="string").str.contains(pattern)
 
 Or whether elements match a pattern:
 
 .. ipython:: python
 
-   pd.Series(['1', '2', '3a', '3b', '03c'],
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
              dtype="string").str.match(pattern)
 
-The distinction between ``match`` and ``contains`` is strictness: ``match``
-relies on strict ``re.match``, while ``contains`` relies on ``re.search``.
+.. ipython:: python
+
+   pd.Series(['1', '2', '3a', '3b', '03c', '4dx'],
+             dtype="string").str.fullmatch(pattern)
+
+.. versionadded:: 1.1.0
path: doc/source/user_guide/text.rst
text: @user1:
Can you move this before the fullmatch example?