mojombo/chronic — pull request #254 by davispuh (2014-03-25 23:12:57+00:00)
Title: Tokenize

So here's my next PR :)

Main changes:
* `could_be_year?` and others accept 2nd argument for specifying width/length of string which was used, so for example if we parse number `010` we can determine that it can't be year as no one writes it this way.
* Add ScalarWide tag to match numbers like `2010` which could mean either year or time.
* Start migrating to new Tokenizer (`Tokenizer#tokenize`). This tokenizer will parse each char and group them based on character types (eg. digit group, letter group). This allows to properly parse various formats than trying to Regexp them. In this PR it's not fully implemented, but only splitting between spaces like was before to pass tests without need to change much other code.
* `Tag#scan_for` updated for special tag matching based on types. Symbol's will be matched exactly (case-sensitive), but String's will be matched case-insensitive and also they'll match outside of current tag. It allows to match `a.` even if current tag is just `a` letter but it's followed by `.`. Also reduces need to use Regexp for simple cases like `/next/i`