Hi Sam,

I am sorry to say that I disagree with your below statement that "no changes should be required to the ANTLR 4 Java Runtime or code generation templates" when implementing other targets.

This is a highly desirable goal, but it can't be carved in stone. 

Firstly, the reason for the change is a bug in the tool, which doesn't show in Java/C# because those languages happen to share with AntLR the syntax for parameters (type name). Yes the fix is required for Python, but the bug is in the tool itself. I agree that for this particular bug fix, no change was required in the runtime, but changes were required in the templates. Other language targets may reveal other bugs, again impacting all targets. The only ways I can think of to avoid impacting other targets would be to either have the tool use different ST prototypes depending on the target (thus violating altogether the concept of a target), or gradually move some of the generation logic into the target (as already the case for bad words and string serialization). I'm not sure any of these would be desirable.

Secondly, this statement is making the assumption that all target languages have enough in common with Java/C# to not require a change in the tool and runtime model. While Java and C# are indeed very close to each other (single inheritance, multiple interfaces, method overloading... we all know where C# comes from), we might find in the future that other target languages will require changes in the underlying runtime model. For example, Python has no support for interfaces, which breaks the Listener and Visitor model. It also has no support for method overloading, which breaks the ContextRuleListGetterDecl/ContextRuleListIndexedGetterDecl concept. Fortunately, the former was easy to workaround, and the second being unnecessary, was simply skipped. But with the amount of code moved, in antlr4, from the tool into the runtime, I wouldn't be surprised if such incompatibilities are encountered in the future. I'm starting to work on the Javascript target, which you know has no support for classes. They can be mimicked, but that may or not fit with the current model and again might require changes. As an extreme example, would it be possible to implement an asm language target using the current model?

Back to what happened, again the break comes from a bug in the tool, and the fix required a change in the ST prototypes (RuleFunction, LeftRecursiveRuleFunction etc...), which are invoked by the tool. As a result, this necessarily impacts all targets, which I believe are broken if they don't implement the new prototype.

Now how do you address this kind of change/fix in a robust way with a growing number of targets?
I don't understand the concept of "loose guarantee" you refer to in the below, for me those words are antinomic.

In theory, we'd all love to see 0 impact. In practice however, we have no way to guarantee that.
Because if tool n+1 is used with target n, it might produce code that is not supported by runtime n
And if runtime n+1 is used with code produced by tool/target n, it might also break.

In a perfect (and wealthy) world, people would get paid to write and run all the required tests to detect such breaks, and adjust versioning accordingly.
In the real world, our test suite does not yet cover all use cases (as discovered during the Python implementation).
Who is going to write what would be required to provide the guarantee you propose, i.e. go back to tool n-1, n-2, n-3... and run the tests again runtime n, for all targets?

Why are you interested in "ensuring that a .class file produced by javac, from a grammar generated by and compiled against ANTLR 4.1, will continue to work with future ANTLR runtime releases" ?
Is this theoretical, or do we actually know of developers who have that requirement as part of their implementation ? This is not the Java world (or the .Net one) where thousands of jars/assemblies are out there and cannot be recompiled for the 1.7 jvm (or the 4.5 clr). The entire chain, from the g4 files to the listener implementation, is under the developer's control. We're not asking them to rewrite their grammar, just to recompile it with the tool that matches the runtime. Apart from bug fixes, not impacted by the version checking, why would one upgrade the runtime and not the tool? 

Finally, RuntimeMetaData.checkVersion does not prevent the code from running, (except for this particular release which requires RuntimeMetaData class to be present)  it simply outputs an error message in the console, which warns the developer against potential incompatibilities.

I have -really- no doubt that in the future this will save both the developers and the antlr team a tremendous amount of time and energy.

Eric


Le 23 mai 2014 à 00:03, Sam Harwell <notifications@github.com> a écrit :

> In tool/resources/org/antlr/v4/tool/templates/codegen/Java/Java.stg:
> 
> > @@ -214,6 +222,8 @@ Parser(parser, funcs, atn, sempredFuncs, superClass) ::= <<
> >  Parser_(parser, funcs, atn, sempredFuncs, ctor, superClass) ::= <<
> >  @SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast"})
> >  public class <parser.name> extends <superClass> {
> > +    static { RuntimeMetaData.checkVersion("<file.ANTLRVersion>"); }
> Hi Eric,
> 
> First and foremost, you should know that my view on the overall issues surrounding these changes is minimize breaking changes with the requirement that ANTLR 4 support the creation, distribution, and maintenance of reliable implementations of other language targets. We will find a way to make the Python target work, even if I request a few changes before merging the related pull requests.
> 
> This particular line is part of the generated code for the Java target, and is completely unrelated to the Python target. In addition, I believe you have misinterpreted the desired compatibility goals. I'm interested in ensuring that a .class file produced by javac, from a grammar generated by and compiled against ANTLR 4.1, will continue to work with future ANTLR runtime releases. Since this applies only to the generated code in combination with the runtime library, issues with the Java target again do not impact any other target.
> 
> Grammars generated by and compiled against ANTLR 4.x.y1 should continue to work with ANTLR runtime 4.x.y2, provided that y2 >= y1.
> Where possible, grammars generated by and compiled against ANTLR 4.x1 should continue to work with ANTLR Runtime 4.x2, provided that x2 >= x1. This guarantee is loose, but points of incompatibility are documented in the breaking changes section of the release notes and we strive to limit them to cases where users are overriding the default parsing behavior and/or using embedded actions in the grammar. We started work on this issue with ANTLR 4.1, and have done an exceptional job of preserving compatibility through 4.2.2.
> In addressing compatibility problems related to developing the Python target (or other targets), no changes should be required to the ANTLR 4 Java Runtime or code generation templates. If changes to the Tool do require changes be made to the Java code generation templates, they should be structural only and not impact the actual code generated by the templates (i.e. behavior-preserving refactoring operations only). I fully expect changes to be made to the ANTLR 4 Tool, but plan to work to minimize their overall impact. As always, any potentially breaking changes will be tracked and documented in the release notes.
> 
> —
> Reply to this email directly or view it on GitHub.
>