EHN: (First commit) --  Began implementation of new regex methods for Se...
State: closed

...ries class, performing element-wise regex matching (Series.str_find) and substitution (Series.string_sub).


Labels:

Comments:
Out of curiosity, Why not use already existing `series.map()` and expose subsitution as something like `pandas.func.str_sub`? I was going to suggest to use `apply()` for `str_find` but realized that it's a `DataFrame` method only. Perhaps it makes sense to add it to Series for consistency as well? (currently you can only do `series[str_find(series)]` if you exposed np.vectorized find). This way there's no need to modify series interface for every specialized type operation such as string matching.

(Edit: sorry, erroneously commented on your commit first) 

Cool, this a good start.

1) Have you looked at pandas.core.strings and the utility module that @takluyver started writing?
2) I assume you're familiar with stringr in R? We should just port all those functions with a few name tweaks / making it more Pythonic
3) Having NA data in string array is super common so you would just want to silently propagate NAs (and definitely not put an empty string). Having print statements like that is also not something I would do personally. You could also have an option to raise errors or just put NAs in the result

Thanks for the comments, dtcaciuc. As this is my first pull request to the pandas repo, (and on Github), I'd just like to say thanks to all of the developers for pandas -- it is truly an incredible package!

Your feedback is a helpful starting point for discussion regarding pattern matching:

> This way there's no need to modify series interface for every specialized type operation such as string matching.

I agree, however, believe this then lends to a general discussion to understand the overall goal for pandas. For me, one of the incredibly useful feature of pandas is the ability to easily perform filtering via complex boolean expressions of numerical data. A good thing, as I imagine that a majority of panda users work primarily with numerical data (myself included). 

Still, there are many who work with non-numeric data, including, for example, alpha-numeric strings (such as element names in a finite-element grid), words and/or entire phrases. As it is now, there doesn't exist any simple methods for searching or analyzing such types of data (unless you are searching for an exact string match via `==` boolean expression). This type of functionality exists in major scripting languages, such as PERL, and also statistical packages like MATLAB. Even while being primarily a numbers kinda guy, this was, for me, a noticeable omission. The ability to parse and/or group data by regex matching is incredibly useful.

Pandas includes an incredibly well-developed set of tools for performing both filtering and calculations of numerical data in pandas. Pattern matching should be thought of as the analogous operation on _non-numerical_ data, and subsequently, I believe should have its own interface to do so, and the reason why I began developing both functions. Even in the short time I have been using them, I have found them very helpful for parsing/analyzing large data sets containing such non-numerical data. Working with the FEC data set, in a single line of code, for instance,
    `fec[fec.contbr_nn.str_find('friends_last_name')]`
and found a friend's multiple donations to Ron Paul (and lol'd). That's not only entertaining, it's really useful.

With that, if the community decides that these features are important, how they could best be implemented is another discussion, and one I have been looking forward to. As per your thoughts regarding this:

> Why not use already existing `series.map()`

Good question. Actually, I initially implemented both functions with `np.vectorize()`, but after exploring the source more, rewrote them to directly call the internal cython-based method `lib.map_infer`, which is what the `Series.map()` function is based on. As I was curious to see the benefits, I compared them both and, to my surprise, the performance of `lib.map_infer` was slightly poorer than with `np.vectorize()`, perhaps by about 5-10%. As to why this is, well, I am hoping that someone here can point me to an answer.

>  I was going to suggest to use `apply()` for `str_find` but realized that it's a `DataFrame` method only. Perhaps it makes sense to add it to Series for consistency as well? (currently you can only do `series[str_find(series)]` if you exposed `np.vectorized` find).

It was also my first thought, as I had been using `apply()` for some other calculations. I have to look at the source when functions can be called with  `apply()` vs. `applymap()`. And also, I agree that it would be extremely beneficial to see `apply()` work across both `DataFrame` and `Series` classes. 

Whatever happens, the `str_find()` and `string_sub()` methods should simply be viewed as convenience functions, allowing for an easy way to filter and parse non-numeric data. I think this is substantial.

Thanks again for the feedback.

Thanks, Wes. Pandas is great.

> 1) Have you looked at pandas.core.strings and the utility module that @takluyver started writing?

I had not, somehow totally missed this. But will check. 

> 2) I assume you're familiar with stringr in R? We should just port all those functions with a few name tweaks / making it more Pythonic

This is exactly what I would like to see, with the expose to such string functions in a form like: 
'Series.str.func(args)`

> 3) Having NA data in string array is super common so you would just want to silently propagate NAs (and definitely not put an empty string). Having print statements like that is also not something I would do personally. You could also have an option to raise errors or just put NAs in the result

Agreed, and had already implemented this in my local version. Will commit this soon.

(Edit: Poor formatting)

Hey @nehalecky-- I am going to defer this work to a branch most likely to allow me to work on it also after the 0.8.0 release, to allow the API to settle before releasing it to the masses. I am working this week on closing the remaining 0.8.0 issues to prepare for release

Hi @wesm, thanks for the note. This makes sense and I think is a good strategy for moving forward. I've been busy lately, but will hope to be able to contribute more very soon! Quick question regarding git workflow: when updating an repository fork with current upstream changes, what is the best method? I imagine that, if the fork hasn't yet been made public, rebasing after each commit would work well before each public push (and subsequent pull request). After push, however, is merging with upstream the only option? Comments from all are appreciated regarding this.

You can rebase your branch onto upstream/master if absolutely necessary, but it's generally a faux pas to merge master into a branch. In this case, I'm not sure you need to do either of those things, but in general you may need to. 

hey folks,

I had a crack at making a comprehensive set of string methods exposed like `series.str.findall(pat)` at PR #1621. I'll definitely take features and unit tests from this PR and adapt them to that API assuming it doesn't change too much. The main goal of my PR was to make it easier to implement these kinds of methods. Some options that you have are still needed like ignoring case, etc. but that should be easy