joeguo/tldextract — pull request #4 by raybejjani (2013-08-06 22:02:41+00:00)
Title: Whitespace handling && wildcard logic

Hello,
This pull includes 2 changes. The first is trivial, it make the code reading the tld.dat cache file mirror the operations done when downloading. The second is more complex.
Wildcard handling could get confused by setups like
*.tld
*.2ld.tld
where a query like a.b.2ld.tld would return {Root: "b", Tld:"2ld.tld"}. This incorrect because we indicate that .2ld.tld is a tld, so the return should be {Root: "a", Tld:"b.2ld.tld"}. Note that publicsuffix.org states:
"If a hostname matches more than one rule in the file, the longest matching rule (the one with the most levels) will be used."
So, in short, we should prefer longer matches over shorter ones. This patch allows that to happen though there are still situations where our logic is flawed (see PS). The code now suppresses that it saw a * when it has a more specific match and will descend to the matching trie node. Previously, starfound would have been set to true already allowing a short-circuit to occur in the logic. By suppressing it we must find the star and descend to it's node after we follow the specific match.

Thanks!
Ray

PS There is a flaw in our logic. We are doing a depth first search but with no backtracking. If we had
*.tld
*.2ld.tld
and tried to extract on "2ld.tld" we would return a fail. This is because we would look at the node for 2ld in the trie if it's children did not net a match we would never return to try the *.tld rule. In this case, it may be desired but may be more of a problem with *.*.tld rules etc.