segmentio/inbound — pull request #2 by asolove (2013-03-30 18:25:40+00:00)
Title: Support all tlds for multi-tld providers

The existing matchers either directly search for a .com url or look for a sld by trying to match a string in the hostname. I would like to add automatic support for matching a given sld under all its tlds. This pull request currently demonstrates one way to do this for yahoo properties.

*Why?*

While some services (e.g. Twitter) seem to always be hosted at a .com, regardless of the user's language, most web properties are hosted on multiple tlds. Yahoo is at search.yahoo.com and search.yahoo.co.jp. The easiest way to match this is to do a substring search: ```referrer.host.indexOf("search.yahoo")```. But this is potentially misleading, if there is ever a hostname like "search.yahoo.somethingelse.com". So instead, we need to actually parse apart the tld, sld, and subdomains, and look for exact matches against those.

*Why not?*

The tldextract npm module that this requires takes a long time to start up, because it has to load all extent tlds from a text file. On my machine, just adding this dependency changes the tests from running in 50ms to taking nearly a second. But this appears to be almost all a startup cost: it adds very little to each subsequent query. So it's probably worth it if you want more accurate data and do this often enough to spread the startup cost among lots of hits.

*What's next?*

If this seems like a good idea, I would like to add the tld parsing to the parsedHref and parsedReferrer passed into every matcher so that it is easier to use. Then I can migrate matchers on multi-tld properties to use this rather than string lookups.