Repository: pandas-dev/pandas

Issue:
ENH: More convenient regex?
How about circumventing annoying match objects by providing an additional, somewhat redundant string method?

```
s.str.extract(pattern)
```

as a shortcut for

```
s.str.match(pattern).str.get(0)
```

If the pattern contains multiple groups, a DataFrame should be returned. Thoughts? Maybe @hayd would be into this one.



Discussion:
I _do_ hate match objects (is this like a get groups thing?)... this sounds useful. I think should be able to extract using group name (and the moment get is by integer location only), not sure how that would work with mulitple.

_I also dislike how findall returns a series of lists..._

This is how I think single and multiple groups should work.

```
>>> Series(['a1', 'b2', 'c3']).str.extract('[ab](\d)')
0    1
1    2
2    NaN
dtype: object

>>> Series(['a1', 'b2', 'c3']).str.extract('([ab])(\d)')
     0    1
0    a    1
1    b    2
2  NaN  NaN
```

What are you thinking for group names?

Would this / does this only work for series? Also, how would you handle
named match groups? (looks like '(?<name>pattern)')

I think it would only work neatly for Series. As for named match groups, what about this?

```
>>> Series(['a1', 'b2', 'c3']).str.match('(?P<letter>[ab])(?P<digit>\d)')
     letter  digit
0    a       1
1    b       2
2    NaN   NaN
```

As above, if there is no match, the whole row is NaN. If there is a _optional_ group and that group is absent from a given entry, only the absent group is NaN.

```
>>> Series(['a1', 'b2', '3']).str.match('(?P<letter>[ab])?(?P<digit>\d)') # first group optional
     letter  digit
0    a       1
1    b       2
2    NaN     3
```

That's a good way to do it.

Yep, that was exactly what I was getting at with group names, this would be a great enhancement.

_The stupid thing I was also thinking was multiple optional things, but actually this doesn't appear to be supported in re anyway (nor does it really work with unnamed stuff):_

```
In [11]: re.match('((?P<letter>[ab]))*','ab').groupdict()
Out[11]: {'letter': 'b'}
```

Two examples that work in the PR referenced above. Will write tests out of these, and more.

```
In [1]: Series(['a1', 'b1', '1', 'a']).str.extract('([ab])?(?P<digit>\d)')
Out[1]: 
     1 digit
0    a     1
1    b     1
2  NaN     1
3  NaN   NaN

In [2]: Series(['a1', 'b1', '1', 'a']).str.extract('([ab])?')
Out[2]: 
0      a
1      b
2    NaN
3      a
dtype: object
```

Notice that, in the column names, I follow the re module's convention of labeling any unnamed groups with 1. It's actually `'1'`. Should it be `1`? Which is less confusing?

Oh sorry, just commented about that same thing. Not sure what's better. Probably should follow their (weird) convention.

Ha. Saw your comment and changed it. I'm really not sure which is better.


Resolving patch:
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -3,6 +3,7 @@
 from pandas.compat import zip
 from pandas.core.common import isnull, _values_from_object
 from pandas.core.series import Series
+from pandas.core.frame import DataFrame
 import pandas.compat as compat
 import re
 import pandas.lib as lib
@@ -328,6 +329,59 @@ def f(x):
 
     return _na_map(f, arr)
 
+def str_extract(arr, pat, flags=0):
+    """
+    Find groups in each string (from beginning) using passed regular expression
+
+    Parameters
+    ----------
+    pat : string
+        Pattern or regular expression
+    flags : int, default 0 (no flags)
+        re module flags, e.g. re.IGNORECASE
+
+    Returns
+    -------
+    extracted groups : Series (one group) or DataFrame (multiple groups)
+
+
+    Note
+    ----
+    Compare to the string method match, which returns re.match objects.
+    """
+    regex = re.compile(pat, flags=flags)
+
+    # just to be safe, check this
+    if regex.groups == 0:
+        raise ValueError("This pattern contains no groups to capture.")
+    elif regex.groups == 1:
+        def f(x):
+            if not isinstance(x, compat.string_types):
+                return None
+            m = regex.match(x)
+            if m:
+                return m.groups()[0] # may be None
+            else:
+                return None
+    else:
+        empty_row = Series(regex.groups*[None])
+        def f(x):
+            if not isinstance(x, compat.string_types):
+                return empty_row
+            m = regex.match(x)
+            if m:
+                return Series(list(m.groups())) # may contain None
+            else:
+                return empty_row
+    result = arr.apply(f)
+    result.replace({None: np.nan}, inplace=True)
+    if regex.groups > 1:
+        result = DataFrame(result) # Don't rely on the wrapper; name columns.
+        names = dict(zip(regex.groupindex.values(), regex.groupindex.keys()))
+        result.columns = [names.get(1 + i, i) for i in range(regex.groups)]
+    else:
+        result.name = regex.groupindex.get(0)
+    return result
 
 def str_join(arr, sep):
     """
@@ -675,8 +729,12 @@ def __iter__(self):
             g = self.get(i)
 
     def _wrap_result(self, result):
-        return Series(result, index=self.series.index,
-                      name=self.series.name)
+        assert result.ndim < 3
+        if result.ndim == 1:
+            return Series(result, index=self.series.index,
+                          name=self.series.name)
+        else:
+            return DataFrame(result, index=self.series.index)
 
     @copy(str_cat)
     def cat(self, others=None, sep=None, na_rep=None):
@@ -764,6 +822,7 @@ def rstrip(self, to_strip=None):
     endswith = _pat_wrapper(str_endswith, na=True)
     findall = _pat_wrapper(str_findall, flags=True)
     match = _pat_wrapper(str_match, flags=True)
+    extract = _pat_wrapper(str_extract, flags=True)
 
     len = _noarg_wrapper(str_len)
     lower = _noarg_wrapper(str_lower)


Test patch:
diff --git a/pandas/tests/test_strings.py b/pandas/tests/test_strings.py
--- a/pandas/tests/test_strings.py
+++ b/pandas/tests/test_strings.py
@@ -415,6 +415,94 @@ def test_match(self):
         exp = Series([(u('BAD__'), u('BAD')), NA, []])
         tm.assert_series_equal(result, exp)
 
+    def test_extract(self):
+        # Contains tests like those in test_match and some others.
+
+        values = Series(['fooBAD__barBAD', NA, 'foo'])
+        er = [NA, NA] # empty row
+
+        result = values.str.extract('.*(BAD[_]+).*(BAD)')
+        exp = DataFrame([['BAD__', 'BAD'], er, er])
+        tm.assert_frame_equal(result, exp)
+
+        # mixed
+        mixed = Series(['aBAD_BAD', NA, 'BAD_b_BAD', True, datetime.today(),
+                        'foo', None, 1, 2.])
+
+        rs = Series(mixed).str.extract('.*(BAD[_]+).*(BAD)')
+        exp = DataFrame([['BAD_', 'BAD'], er, ['BAD_', 'BAD'], er, er,
+                         er, er, er, er])
+        tm.assert_frame_equal(rs, exp)
+
+        # unicode
+        values = Series([u('fooBAD__barBAD'), NA, u('foo')])
+
+        result = values.str.extract('.*(BAD[_]+).*(BAD)')
+        exp = DataFrame([[u('BAD__'), u('BAD')], er, er])
+        tm.assert_frame_equal(result, exp)
+
+        # no groups
+        s = Series(['A1', 'B2', 'C3']) 
+        f = lambda: s.str.extract('[ABC][123]')
+        self.assertRaises(ValueError, f)
+
+        # only non-capturing groups
+        f = lambda: s.str.extract('(?:[AB]).*')
+        self.assertRaises(ValueError, f)
+
+        # one group, no matches
+        result = s.str.extract('(_)')
+        exp = Series([NA, NA, NA])
+        tm.assert_series_equal(result, exp)
+ 
+        # two groups, no matches
+        result = s.str.extract('(_)(_)')
+        exp = DataFrame([[NA, NA], [NA, NA], [NA, NA]])
+        tm.assert_frame_equal(result, exp)
+
+        # one group, some matches
+        result = s.str.extract('([AB])[123]')
+        exp = Series(['A', 'B', NA])
+        tm.assert_series_equal(result, exp)
+
+        # two groups, some matches
+        result = s.str.extract('([AB])([123])')
+        exp = DataFrame([['A', '1'], ['B', '2'], [NA, NA]])
+        tm.assert_frame_equal(result, exp)
+
+        # named group/groups
+        result = s.str.extract('(?P<letter>[AB])(?P<number>[123])')
+        exp = DataFrame([['A', '1'], ['B', '2'], [NA, NA]], columns=['letter', 'number'])
+        tm.assert_frame_equal(result, exp)
+        result = s.str.extract('(?P<letter>[AB])')
+        exp = Series(['A', 'B', NA], name='letter')
+        tm.assert_series_equal(result, exp)
+
+        # mix named and unnamed groups
+        result = s.str.extract('([AB])(?P<number>[123])')
+        exp = DataFrame([['A', '1'], ['B', '2'], [NA, NA]], columns=[0, 'number'])
+        tm.assert_frame_equal(result, exp)
+
+        # one normal group, one non-capturing group
+        result = s.str.extract('([AB])(?:[123])')
+        exp = Series(['A', 'B', NA])
+        tm.assert_series_equal(result, exp)
+
+        # two normal groups, one non-capturing group
+        result = Series(['A11', 'B22', 'C33']).str.extract('([AB])([123])(?:[123])')
+        exp = DataFrame([['A', '1'], ['B', '2'], [NA, NA]])
+        tm.assert_frame_equal(result, exp)
+
+        # one optional group followed by one normal group
+        result = Series(['A1', 'B2', '3']).str.extract('(?P<letter>[AB])?(?P<number>[123])')
+        exp = DataFrame([['A', '1'], ['B', '2'], [NA, '3']], columns=['letter', 'number'])
+        tm.assert_frame_equal(result, exp)
+
+        # one normal group followed by one optional group
+        result = Series(['A1', 'B2', 'C']).str.extract('(?P<letter>[ABC])(?P<number>[123])?')
+        exp = DataFrame([['A', '1'], ['B', '2'], ['C', NA]], columns=['letter', 'number'])
+        tm.assert_frame_equal(result, exp)
+
     def test_join(self):
         values = Series(['a_b_c', 'c_d_e', np.nan, 'f_g_h'])
         result = values.str.split('_').str.join('_')