twitter/twitter-text-rb — pull request #34 by keitaf (2012-01-30 21:46:59+00:00)
Title: Refactor twitter-text to allow punctuation characters before URL.

Currently twitter-text doesn't allow punctuation characters like " or . before URL. But if we change Regex to allow those characters before URL Autolink.auto_link() will break because it'll auto-link the URLs added by auto_link_hashtags/usernames_and_lists(). To solve it, this modifies Autolink to use Extractor to extract all entities first, then use them to auto-link the text.

This also includes performance improvement (which doubles the speed of Autolink/Extractor on my laptop) as well as other minor refactoring.

Here is the list of changes in this branch:

* Add new method Extractor.extract_entities_with_indices(), which extracts all entities (hashtag, mention, URLs) from a given text.
* Modify Rewriter.rewrite*() to use Extractor to extract entities instead of applying Regex.
* Allow punctuation characters like '"', '.' or '!' before URL.
* Consolidate AUTO_LINK_USERNAMES* and EXTRACT_MENTIONS into VALID_MENTION_OR_LIST
* Improve performance of Extractor.extract*() by scanning the text before applying complex regex patterns.
* Add an option in Extractor.extract_urls*() to enable/disable extracting URLs without protocol.