Repository: pandas-dev/pandas

Issue:
PERF: speed up certain string operations
from [SO](http://stackoverflow.com/questions/30113499/python-pandas-split-columns-to-slow-anything-to-get-more-performance)

on big enough strings this might be quite useful for a number of string ops.

```
import pandas as pd
import random
import numpy as np
from StringIO import StringIO

def make_ip():
    return '.'.join(str(random.randint(0, 255)) for n in range(4))

df = pd.DataFrame({'ip': [make_ip() for i in range(20000)]})

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = df.ip.str.split('.', return_type='frame')
#1 loops, best of 3: 3.06 s per loop

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = df['ip'].apply(lambda x: pd.Series(x.split('.')))
#1 loops, best of 3: 3.1 s per loop

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = \
    pd.read_table(StringIO(df['ip'].to_csv(None,index=None)),sep='.')
#10 loops, best of 3: 46.4 ms per loop
```



Discussion:
Interestingly, my initial thought that this was because Pandas' split just iterates through in Python was wrong: Using the obvious and pure Python `[ x.split('.') for x in list(df['ip'])]` is actually 400 times faster than using Pandas. Thus there's something else that's causing the problem.

It seems like vast amounts of time are spent in `maybe_convert_objects` and `_possibly_cast_to_datetime`, amongst other things. I'll have to look into it further.

it seems that this operation creates a Series then passes to the DataFrame constructor. No need to do this, the list-like operation should effectively be this.

```
In [33]: %timeit pd.DataFrame([ x.split('.') for x in list(df['ip']) ])
100 loops, best of 3: 13.4 ms per loop
```

simple change fixes this.

```
In [8]: %timeit df['ip'].str.split('.',return_type='frame')
100 loops, best of 3: 20 ms per loop

In [9]: quit()
[jreback-~/pandas] git diff
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
index 6e603f6..b8ea27f 100644
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -723,7 +723,7 @@ def str_split(arr, pat=None, n=None, return_type='series'):
             regex = re.compile(pat)
             f = lambda x: regex.split(x, maxsplit=n)
     if return_type == 'frame':
-        res = DataFrame((Series(x) for x in _na_map(f, arr)), index=arr.index)
+        res = DataFrame([x for x in _na_map(f, arr)], index=arr.index)
     else:
         res = _na_map(f, arr)
     return res
```

It seems like avoiding the list creation could be done as well, though it may not make much difference, and I'm not quite sure how Pandas handles iterators internally:

```
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
index 6e603f6..8fb7f10 100644
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -723,7 +723,7 @@ def str_split(arr, pat=None, n=None, return_type='series'):
             regex = re.compile(pat)
             f = lambda x: regex.split(x, maxsplit=n)
     if return_type == 'frame':
-        res = DataFrame((Series(x) for x in _na_map(f, arr)), index=arr.index)
+        res = DataFrame((x for x in _na_map(f, arr)), index=arr.index)
     else:
         res = _na_map(f, arr)
     return res
```

Ah; it just converts to a list. Oh well.

Because this changes current behavior (non-str values are all converted to str), it may be an option to prepare a shortpath for all-string values. On my environment, numpy's string method is faster than above workaround.

```
%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = \
    pd.read_table(StringIO(df['ip'].to_csv(None,index=None)),sep='.')
# 10 loops, best of 3: 80.9 ms per loop

%timeit pd.DataFrame(np.core.defchararray.split(df['ip'].values.astype(str), '.'))
# 10 loops, best of 3: 29.9 ms per loop
```

sinhrks: the CSV write/read method is a horrible hack. I don't think jrevack's legitimate solution changes current behavior, and it is significantly faster than the hack, likely in line with numpy's performance.

jreback: while I'd be happy to do a PR for this if necessary, I assume you have it dealt with?

pull requests are welcome on this 

@cgevans Thanks for your cooperation :) What I meant in `current behavior` is:

```
s = pd.Series([1.1, '2.2'])

# current behavior (non-strings are left as NaN)
s.str.split('.', expand=True)
#      0    1
# 0  NaN  NaN
# 1    2    2

# numpy (non-strings are forced to be converted)
pd.DataFrame(list(np.core.defchararray.split(s.values.astype(str), '.')))
#    0  1
# 0  1  1
# 1  2  2

# cant use numpy method without changing dtype
pd.DataFrame(list(np.core.defchararray.split(s.values, '.')))
# TypeError: string operation on non-string array
```

I think [numpy funcs](http://docs.scipy.org/doc/numpy/reference/routines.char.html) can be used when the values are all string or unicode (maybe regular case). One idea is to use `pandas.lib.is_string_array` for check and use numpy logic if possible. 


Resolving patch:
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -1,7 +1,7 @@
 import numpy as np
 
 from pandas.compat import zip
-from pandas.core.common import isnull, _values_from_object, is_bool_dtype
+from pandas.core.common import isnull, _values_from_object, is_bool_dtype, is_list_like
 import pandas.compat as compat
 from pandas.util.decorators import Appender, deprecate_kwarg
 import re
@@ -1090,7 +1090,11 @@ def _wrap_result_expand(self, result, expand=False):
         else:
             index = self.series.index
             if expand:
-                cons_row = self.series._constructor
+                def cons_row(x):
+                    if is_list_like(x):
+                        return x
+                    else:
+                        return [ x ]
                 cons = self.series._constructor_expanddim
                 data = [cons_row(x) for x in result]
                 return cons(data, index=index)
diff --git a/vb_suite/strings.py b/vb_suite/strings.py
--- a/vb_suite/strings.py
+++ b/vb_suite/strings.py
@@ -35,6 +35,7 @@ def make_series(letters, strlen, size):
 strings_match = Benchmark("many.str.match(r'mat..this')", setup)
 strings_extract = Benchmark("many.str.extract(r'(\w*)matchthis(\w*)')", setup)
 strings_join_split = Benchmark("many.str.join(r'--').str.split('--')", setup)
+strings_join_split_expand = Benchmark("many.str.join(r'--').str.split('--',expand=True)", setup)
 strings_len = Benchmark("many.str.len()", setup)
 strings_findall = Benchmark("many.str.findall(r'[A-Z]+')", setup)
 strings_pad = Benchmark("many.str.pad(100, side='both')", setup)


Test patch: