antlr/antlr4 — pull request #306 by HSorensen (2013-07-21 07:57:49+00:00)
Title: RFC Include file support v1

This request is still just an RFC hack. There are a number of things to sort out, error handling is absent, the actual include name need to be done properly. The current implementation is only for the Lexer, parser integration will come later.
I have a grammar that supports including files into the scanning stream. I have hacked Lexer.java to supports this, but I am looking for some guidance from someone with a better understanding of the internals of antlr.

The basic idea is to have a stack of lexer states, and each time a request for include is recognized by the lexer, the current lexer state gets pushed to the stack, and the input stream will continue from the newly recognized include request.

In the nextToken(), the check on EOF will check if the state stack is empty, if it is empty EOF is returned; if not then the previous lexer state gets restored and the scanning will continue from before the include request.

To support this I added a new constructor to Lexer.java with an additional argument IncludeStrategy. The caller of the lexer need to provide the strategy for dealing with include names, e.g. added file extensions and looking in different directories, etc.

In the lexer grammar with the include command is met a lexer action need to have a call to {setIncludeStream(getText());}. This will ensure the nextToken() invocation will read from the new file.

There are still more work to be done, but I would like to hear some comments to this idea.
Thanks
Henrik