Is the Token{...} production explained in the manual?  If not, it should.  Radu, can you please add a quick description of it, as a reminder that we should describe it thoroughly eventually?

Grigore


________________________________
From: Radu Mereuta [notifications@github.com]
Sent: Tuesday, March 04, 2014 3:12 PM
To: kframework/k
Subject: Re: [k] New parser. Adding NullabilityCheck. (First pull request). (#267)


In src/javasources/KTool/src/org/kframework/parser/concrete2/NullabilityCheck.java:

> +            do {
> +                changed = false;
> +                for (Entry<NonTerminal, Queue<State>> ee : dependencies.entrySet()) {
> +                    // for each non terminal set
> +                    State st = null;
> +                    while ((st = ee.getValue().poll()) != null) {
> +                        if (st instanceof ExitState) {
> +                            nullable.add(ee.getKey());
> +                            dependencies.remove(ee.getKey());
> +                            changed = true;
> +                            break;
> +                        } else { // nextable states
> +                            assert st instanceof NextableState : "Expected nextable states here!";
> +                            if (st instanceof PrimitiveState) {
> +                                PrimitiveState ss = (PrimitiveState) st;
> +                                Set<MatchResult> matchResults = ss.matches("", 0, null);


Even right now you may have Regular Expressions that have follow restrictions. Even Ids have follow restrictions.
When you write
syntax Id ::= Token{[a-zA-Z][a-zA-Z0-9]*}
The front end will generate automatically a follow restriction (lookahead) that checks to see if the current location is not followed by any of the [a-zA-Z0-9].

The problem in this location is when you have such a lookahead for an empty string: "" -/- [a-z]
Something like that.

—
Reply to this email directly or view it on GitHub<https://github.com/kframework/k/pull/267/files#r10274839>.