nikodemus/esrap — pull request #16 by scymtym (2013-01-05 07:18:50+00:00)
Title: Add GRAMMAR structure and dependencies between grammars

This is a prototype implementation of package-like grammars. It is prototypical in the sense that much cleanup and some refactoring is necessary. However, I wanted to try and get some feedback before polishing this any further. Thanks for taking the time to consider this.

Commit message:
```
* Partly based on menschenkindlein's
implementation
(Menschenkindlein/esrap@63baec4e20591d3c1517fae88a364de8a39ebb12).

* Grammar objects are a bit like CL packages:

* they have a name and optional :DOCUMENTATION
* they can :USE other grammars
* they act as containers for rules
* redefinition is allowed and preserves existing rules

* Changed interface:

* GRAMMAR-NOT-FOUND-{ERROR,WARNING} [condition]
* RULE-NOT-FOUND-{ERROR,WARNING} [condition]

* GRAMMAR-NAME [function]
* GRAMMAR-DEPENDENCIES [function]
* GRAMMAR-RULES [function]

* DEFGRAMMAR [macro]
* IN-GRAMMAR [macro]

* MAKE-GRAMMAR [function]
* FIND-GRAMMAR [function]
* (SETF FIND-GRAMMAR) [function]

* DEFGRAMMAR, DEFRULE and PARSE signal full warnings at compile time
when constant references to grammars or rules cannot be resolved.

* IMPORTANT: In addition, all user-visible -rule functions take a
grammar designator as their first argument.

* Changed implementation:

* DEFGRAMMAR, IN-GRAMMAR and DEFRULE produce code which is evaluated
at compile-, load- and runtime. This is necessary in order to be
able to define and use a grammar in the same file.

* FIND-RULE traverses used grammars to find the requested rule.

* The new functions UPDATE-RULE-DEPENDENCIES, RECOMPILE-CELL are
called after adding/removing rules/used grammars to update
dependencies and recompile rules.

* Rule dependencies are expressed in terms of referencing rule
cells, not symbols.

* New tests for grammar objects and related lookup and redefinition
issues have been added.

* Documentation has been extended.

* Examples:

* example-sexp.lisp now uses a separate grammar called SEXP

* example-sexp-xml.lisp contains a new grammar called XML which uses
the SEXP grammar to demonstrate grammar dependencies (originally
written by Menschenkindlein)
```

There are design decision, for which I could use some advice:

1. Is backwards compatibility to the previous
```cl
(parse RULE STRING)
```
interface required?

2. Rule names are currently processed using ``STRING``. This allows clients to write
```cl
(parse '#:some-grammar '#:some-rule TEXT)
```
. It also frees grammar writers from exporting rule names as symbols. However, it could lead to clashes between rule names in the ``USE``-closure of some grammar. It also prevents "private" rules.

3. Should operations on grammars accept grammar objects and grammar
designators or should the idiom instead always be
```cl
(grammar-name (find-grammar GRAMMAR-DESIGNATOR))
```
?

4. Should operations on rules accept grammar and rule designators as in
```cl
(rule-dependencies GRAMMAR-DESIGNATOR RULE-DESIGNATOR)
```
or should these operations accept rule objects as in
```cl
(rule-dependencies (find-rule GRAMMAR-DESIGNATOR RULE-DESIGNATOR))
```
or even
```cl
(rule-dependencies (find-rule (find-grammar GRAMMAR-DESIGNATOR) RULE-DESIGNATOR))
```
?
5. Should circular dependencies between grammars be allowed? Currently, they signal an error.

6. The ``*non-existent*`` tests use some compilation/signal-related trickery which can probably be improved.

7. Redefinition of whole grammars (via ``(setf (find-grammar ...) nil)`` and then ``(defgrammar ...)``) after ``(parse ...)`` calls have been compile-time optimized can lead to an old definition of a rule being used.