tenderlove/racc — Issue #3: Tracking/accessing additional token information from the lexer
Author: zdennis
Created: 2011-10-17 13:38:03+00:00
State: open

I'd like to track additional information about each token that the lexer provides but racc requires each token to be a 2-element Array. I was writing a lexer to push on a third element which contained custom token attributes in a Hash, but Racc doesn't like that:

/Users/zdennis/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/racc/parser.rb:99:in `_racc_do_parse_c': next_token() returned wrong size of array (3 for 2) (ArgumentError)
from /Users/zdennis/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/racc/parser.rb:99:in `do_parse'
from rscript.y:151:in `parse'
from rscript.tab.rb:343:in `

'

Is there a more appropriate way to store and retrieve this information or would it be possible to allow each token to be 3 elements with the 3rd being optional attributes?