twitter/twitter-text-rb — pull request #14 by klingerf (2011-11-14 21:46:17+00:00)
Title: Stricter extraction rule for t.co URLs

I'm updating Twitter::Extractor to properly extract t.co URLs with trailing characters. The extractor needs to be t.co aware because t.co supports fewer valid path characters than those matched by the valid_general_url_path_chars regular expression. This causes legitimate URLs to be mistakenly identified on read once we swap them out for t.co URLs. For instance:

> Twitter::Extractor.extract_urls("I like twitter.com's website")
=> ["twitter.com"]

If we replace the URL with a t.co URL, then we end up with:

> Twitter::Extractor.extract_urls("I like http://t.co/asdf's website")
=> ["http://t.co/asdf's"]

This is because ' is considered a valid path character, but it's not a valid domain character. This patch strips out any non-word characters following the slug for t.co URLs.