Repository: pandas-dev/pandas

Issue:
API: Series.str.match == extract?
This is the whatsnew example. I think for compat, the 3rd entry should be `(np.nan, 3)` ?

```
In [1]: Series(['a1', 'b2', '3']).str.match('(?P<letter>[ab])?(?P<digit>\d)')
Out[1]: 
0       (a, 1)
1       (b, 2)
2    (None, 3)
dtype: object
```



Discussion:
Actually, that should say `str.extract`. (My bad!) See #5099.

Extract does NaN, not None, so that entry will become NaN with this edit. IMO we should leave `match` alone and let it follow the conventions of `re`. The point of `extract` is to give a more pandorable\* alternative to `match`.

*Am I doing it right, @hayd?

is there a reason to not make `match == extract`?  (maybe deprecate in 0.13 and change in 0.14)?

I merged #5099

That makes sense to me.

ok...why don't you do a PR to put that deprecation in 

(also I think you changed the docs in basics.rst, not just in v0.13.0 (#5099)

@jtratner brings up a good point.....

is there a different in `match` and `extract`? 

will it break compat if we just call `extract` match?

@jtratner questions

but if you can use the output of extract with match, then I don't see the point of breaking backwards compatibility by scuttling the old name
only question I have is this:

could you previously do something like:

```
ser[ser.str.match(some_regex)]
```

and will that stop working now?
(and if it used to output tuples, then it would never have been useful)
maybe match should accept strings without match groups and use a different subroutine?
ie.., without match groups returns bools

@hayd @danielballan can you comment on this?

Although match kinda == extract, I much prefer the name extract (as that describes more precisely what it's going on... pandorable), and I think it's ok to be consistent with re but match is vague... I agree current output IS kinda not that useful.

The bool thing for groupless is an interesting idea (and would distinguish extract and match), and seems "expected" (that's what "match" **means** in English!).

Also if the output NaN instead of [] for missing that would be better.

```
In [1]: s = pd.Series(list('abcde'))

In [2]: s.str.match('(a|b)')
Out[2]: 
0    (a,)
1    (b,)
2      []
3      []
4      []
dtype: object
```

atm using `ser[ser.str.match(some_regex)]` throws if you don't match for that reason. If output NaN rather than [] for no match, then you could use `s.str.match('a|b').notnull()` to filter... though actually you ought to be using filter to filter!!

```
In [3]: s.filter(regex='a|b')  # erm this gives me TypeError: buffer size mismatch??
```

okay. I was just hoping we could not add a method then deprecate an old one if we could just put it together under one method.

At the moment what is the best way to match an entire regex? I guess you can use...

```
In [22]: s.str.match('(a|b)$')
```

I can't think of another reason to use match, so if there's another way to do the above (and I expect there is) then I think am happy to depreciate match... (but -1 on renaming extract to match!).

so why don't you

`extract(regex,flags,as_list=False)`

and `as_list=True` will do what match does not

then match is just a wrapper around extract (or can just remove it and point to extract)?

@hayd I think it's worth it to have a way to convert a Series of strings into a boolean indexer (which you might use for filter, but you could also use for, e.g., making an indexer to use with something else).

@jreback I'd like to add extract, and turn match into something that converts str --> bool (and I guess leaves nan?), because I think that's much clearer. No reason to overload extract. (so basically sugar for `ser.apply(lambda x:  re.match(x))` with better nan handling and maybe avoiding the apply machinery if there's a faster path.

@danielballan @hayd @jtratner 

what was decided here? 

Let's add extract and then I'll put in a PR to change match to turn into a
method that returns a bool Series suitable for indexing. We can put that
new behavior under a keyword argument and deprecate the original behavior
for 0.14.

ok...you could deprecate match in 0.13 (and just direct to using the kw with extract).....

Yeah, that's what I'm thinking, maybe something like bool_only = False and
then go from there.
On Oct 11, 2013 8:49 AM, "jreback" notifications@github.com wrote:

> ok...you could deprecate match in 0.13 (and just direct to using the kw
> with extract).....
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/5075#issuecomment-26134330
> .

`as_indexer=False`?

I took a stab at it. See #5224.


Resolving patch:
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -7,7 +7,7 @@
 import pandas.compat as compat
 import re
 import pandas.lib as lib
-
+import warnings
 
 def _get_array_list(arr, others):
     if isinstance(others[0], (list, np.ndarray)):
@@ -169,6 +169,10 @@ def str_contains(arr, pat, case=True, flags=0, na=np.nan):
 
     regex = re.compile(pat, flags=flags)
 
+    if regex.groups > 0:
+        warnings.warn("""This pattern has match groups. To actually get the 
+groups, use str.extract.""", UserWarning)
+
     f = lambda x: bool(regex.search(x))
     return _na_map(f, arr, na)
 
@@ -303,35 +307,70 @@ def rep(x, r):
         return result
 
 
-def str_match(arr, pat, flags=0):
+def str_match(arr, pat, case=True, flags=0, na=np.nan, as_indexer=False):
     """
-    Find groups in each string (from beginning) using passed regular expression
+    Deprecated: Find groups in each string using passed regular expression.
+    If as_indexer=True, determine if each string matches a regular expression.
 
     Parameters
     ----------
     pat : string
-        Pattern or regular expression
+        Character sequence or regular expression
+    case : boolean, default True
+        If True, case sensitive
     flags : int, default 0 (no flags)
         re module flags, e.g. re.IGNORECASE
+    na : default NaN, fill value for missing values.
+    as_indexer : False, by default, gives deprecated behavior better achieved
+        using str_extract. True return boolean indexer.
+
 
     Returns
     -------
-    matches : array
+    matches : boolean array (if as_indexer=True)
+    matches : array of tuples (if as_indexer=False, default but deprecated)
+    
+    Note
+    ----
+    To extract matched groups, which is the deprecated behavior of match, use
+    str.extract.
     """
+
+    if not case:
+        flags |= re.IGNORECASE
+
     regex = re.compile(pat, flags=flags)
 
-    def f(x):
-        m = regex.match(x)
-        if m:
-            return m.groups()
-        else:
-            return []
+    if (not as_indexer) and regex.groups > 0:
+        # Do this first, to make sure it happens even if the re.compile
+        # raises below.
+        warnings.warn("""In future versions of pandas, match will change 
+to always return a bool indexer.""", UserWarning)
+
+    if as_indexer and regex.groups > 0:
+        warnings.warn("""This pattern has match groups. To actually get the 
+groups, use str.extract.""", UserWarning)
+
+    # If not as_indexer and regex.groups == 0, this returns empty lists
+    # and is basically useless, so we will not warn.
+
+    if (not as_indexer) and regex.groups > 0:
+        def f(x):
+            m = regex.match(x)
+            if m:
+                return m.groups()
+            else:
+                return []
+    else:
+        # This is the new behavior of str_match.
+        f = lambda x: bool(regex.match(x))
 
     return _na_map(f, arr)
 
+
 def str_extract(arr, pat, flags=0):
     """
-    Find groups in each string (from beginning) using passed regular expression
+    Find groups in each string using passed regular expression
 
     Parameters
     ----------
@@ -358,7 +397,7 @@ def str_extract(arr, pat, flags=0):
         def f(x):
             if not isinstance(x, compat.string_types):
                 return None
-            m = regex.match(x)
+            m = regex.search(x)
             if m:
                 return m.groups()[0] # may be None
             else:
@@ -368,7 +407,7 @@ def f(x):
         def f(x):
             if not isinstance(x, compat.string_types):
                 return empty_row
-            m = regex.match(x)
+            m = regex.search(x)
             if m:
                 return Series(list(m.groups())) # may contain None
             else:
@@ -668,13 +707,13 @@ def wrapper(self):
     return wrapper
 
 
-def _pat_wrapper(f, flags=False, na=False):
+def _pat_wrapper(f, flags=False, na=False, **kwargs):
     def wrapper1(self, pat):
         result = f(self.series, pat)
         return self._wrap_result(result)
 
-    def wrapper2(self, pat, flags=0):
-        result = f(self.series, pat, flags=flags)
+    def wrapper2(self, pat, flags=0, **kwargs):
+        result = f(self.series, pat, flags=flags, **kwargs)
         return self._wrap_result(result)
 
     def wrapper3(self, pat, na=np.nan):


Test patch:
diff --git a/pandas/tests/test_strings.py b/pandas/tests/test_strings.py
--- a/pandas/tests/test_strings.py
+++ b/pandas/tests/test_strings.py
@@ -5,6 +5,7 @@
 import operator
 import re
 import unittest
+import warnings
 
 import nose
 
@@ -392,10 +393,14 @@ def test_repeat(self):
                       u('dddddd')])
         tm.assert_series_equal(result, exp)
 
-    def test_match(self):
+    def test_deprecated_match(self):
+        # Old match behavior, deprecated (but still default) in 0.13
         values = Series(['fooBAD__barBAD', NA, 'foo'])
 
-        result = values.str.match('.*(BAD[_]+).*(BAD)')
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            result = values.str.match('.*(BAD[_]+).*(BAD)')
+            assert issubclass(w[-1].category, UserWarning)
         exp = Series([('BAD__', 'BAD'), NA, []])
         tm.assert_series_equal(result, exp)
 
@@ -403,7 +408,10 @@ def test_match(self):
         mixed = Series(['aBAD_BAD', NA, 'BAD_b_BAD', True, datetime.today(),
                         'foo', None, 1, 2.])
 
-        rs = Series(mixed).str.match('.*(BAD[_]+).*(BAD)')
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            rs = Series(mixed).str.match('.*(BAD[_]+).*(BAD)')
+            assert issubclass(w[-1].category, UserWarning)
         xp = [('BAD_', 'BAD'), NA, ('BAD_', 'BAD'), NA, NA, [], NA, NA, NA]
         tm.assert_isinstance(rs, Series)
         tm.assert_almost_equal(rs, xp)
@@ -411,10 +419,52 @@ def test_match(self):
         # unicode
         values = Series([u('fooBAD__barBAD'), NA, u('foo')])
 
-        result = values.str.match('.*(BAD[_]+).*(BAD)')
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            result = values.str.match('.*(BAD[_]+).*(BAD)')
+            assert issubclass(w[-1].category, UserWarning)
         exp = Series([(u('BAD__'), u('BAD')), NA, []])
         tm.assert_series_equal(result, exp)
 
+    def test_match(self):
+        # New match behavior introduced in 0.13
+        values = Series(['fooBAD__barBAD', NA, 'foo'])
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            result = values.str.match('.*(BAD[_]+).*(BAD)', as_indexer=True)
+            assert issubclass(w[-1].category, UserWarning)
+        exp = Series([True, NA, False])
+        tm.assert_series_equal(result, exp)
+
+        # If no groups, use new behavior even when as_indexer is False.
+        # (Old behavior is pretty much useless in this case.)
+        values = Series(['fooBAD__barBAD', NA, 'foo'])
+        result = values.str.match('.*BAD[_]+.*BAD', as_indexer=False)
+        exp = Series([True, NA, False])
+        tm.assert_series_equal(result, exp)
+
+        # mixed
+        mixed = Series(['aBAD_BAD', NA, 'BAD_b_BAD', True, datetime.today(),
+                        'foo', None, 1, 2.])
+
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            rs = Series(mixed).str.match('.*(BAD[_]+).*(BAD)', as_indexer=True)
+            assert issubclass(w[-1].category, UserWarning)
+        xp = [True, NA, True, NA, NA, False, NA, NA, NA]
+        tm.assert_isinstance(rs, Series)
+        tm.assert_almost_equal(rs, xp)
+
+        # unicode
+        values = Series([u('fooBAD__barBAD'), NA, u('foo')])
+
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            result = values.str.match('.*(BAD[_]+).*(BAD)', as_indexer=True)
+            assert issubclass(w[-1].category, UserWarning)
+        exp = Series([True, NA, False])
+        tm.assert_series_equal(result, exp)
+
     def test_extract(self):
         # Contains tests like those in test_match and some others.
 
@@ -966,7 +1016,10 @@ def test_match_findall_flags(self):
 
         pat = pattern = r'([A-Z0-9._%+-]+)@([A-Z0-9.-]+)\.([A-Z]{2,4})'
 
-        result = data.str.match(pat, flags=re.IGNORECASE)
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            result = data.str.match(pat, flags=re.IGNORECASE)
+            assert issubclass(w[-1].category, UserWarning)
         self.assertEquals(result[0], ('dave', 'google', 'com'))
 
         result = data.str.findall(pat, flags=re.IGNORECASE)
@@ -975,7 +1028,10 @@ def test_match_findall_flags(self):
         result = data.str.count(pat, flags=re.IGNORECASE)
         self.assertEquals(result[0], 1)
 
-        result = data.str.contains(pat, flags=re.IGNORECASE)
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            result = data.str.contains(pat, flags=re.IGNORECASE)
+            assert issubclass(w[-1].category, UserWarning)
         self.assertEquals(result[0], True)
 
     def test_encode_decode(self):