vim-ruby/vim-ruby — pull request #138 by amarshall (2013-03-11 04:34:56+00:00)
Title: Correctly highlight identifiers with unicode characters

[Ruby 2.0’s default encoding is UTF-8](http://www.ruby-lang.org/en/news/2013/02/24/ruby-2-0-0-p0-is-released#label-2), so identifiers with unicode characters are now allowed by default. E.g.

```
@∞
@foo―
```

Some sanity-checking of my regex modifications is welcome, especially the symbol ones as there’s a lot going on there that I don’t fully understand. I basically blanket-replaced `\h\w` with `\%(\h\|[^x00-x80]\)\%(\w\|[^\x00-\x80]\)`.

I did verify that all unicode codepoints are valid in an instance variable (and assume that the others follow suite). I did spot-checking of class variables, global variables, symbols, and heredocs. I used this script to do automated verification:

```ruby
unicode_codepoints = 0x80..0x10FFFF
invalid_codepoints = unicode_codepoints.reject do |codepoint|
str = [codepoint].pack 'U'
begin
eval "@#{str}"
rescue SyntaxError
false
else
true
end
end
```

The only codepoints that were invalid was the entire range 0xD800 to 0xDFFF, but that’s only because they [aren’t valid unicode anyway](https://en.wikipedia.org/wiki/UTF-8#Invalid_code_points), so they shouldn’t be enterable at all.