CoreFiling/reviki — pull request #21 by Barrucadu (2014-07-28 11:08:45+00:00)
Title: ANTLR-based parser for WikiCreole

This pull request implements a mostly-backwards-compatible (more on that later) parser for the wiki syntax, where wikis are first rendered into an AST before being turned into HTML: the idea being that having a richer structure allows more interesting manipulation of pages. I believe the success of this is shown in [bef6a5d].

## Dependencies

ANTLR 4 is now a build- and run-time dependency.

## Building

The Creole.g4 and CreoleTokens.g4 files must have Java generated from them,

java -jar /path/to/antlr4.jar -package net.hillsdon.reviki.wiki.renderer.creole -visitor Creole.g4 CreoleTokens.g4

## API Changes

- The `RenderNode` architecture is completely gone.
- It is no longer possible to add extra nodes which are rendered (or remove things which would have been rendered), except by transforming the resultant AST.
- All parsing and tokenisation is handled by ANTLR in a standard, predefined, fashion.
- `CreoleRenderer` could be simplified because of this.
- It only provides static methods
- Information which was given to specific `RenderNode`s (such as link handlers) is now given directly to `CreoleRenderer::render`.
- The `ResultNode` architecture has been replaced with `ASTNode`s
- `ASTNode`s are largely similar, providing similar abstractions such as `TaggedNode`.
- There is one `ASTNode` subclass per element type, so much more context is available.
- This is taken advantage of by the `ASTVisitor` class to provide a useful interface.
- Link handling has been shuffled around.
- The link handling classes (such as `LinkResolutionContext`) have been moved into their own package.
- The `LinkContentSplitter` interface was removed.
- The methods in `CreoleLinkContentsSplitter` were made static.
- You can now get the `LinkResolutionContext` of a `LinkPartsHandler`.

## Behaviour changes

- Strikethrough is now rendered with ``, not ``
- Empty links (eg `[[]]` and `{{}}`), commonly used to break up WikiWords, are no longer included in the output (but serve the same purpose)
- URL encoding is now a bit better
- URL detection is now more reliable, requiring plain-text links (ie, outside of `[[]]`) to start with `http://`, `https://`, `file:/`, or `mailto:`
- Attachment detection is now very reliable, with plain-text attachment links (ie, outside of `[[]]`) only rendered as links if the attachment exists.
- Syntax-highlighted code is now displayed inside a `

` or a ``, as appropriate, with the wiki-content class.
- All but the final item of a list can contain line breaks (but not paragraph breaks).
- Whitespace is now different, but that has no effect on the visible output, so it doesn't matter.

## Bugs fixed

- REVIKI-374: Bullet points after unformatted text.
- REVIKI-377: Inline (i.e. not block) pre formatted section cannot be at the start of a line.
- REVIKI-388: Parser for WikiCreole syntax.
- REVIKI-412: Reviki rendering of [] content is fragile if content starts inline.
- REVIKI-453: NoWiki markup doesn't work for pipe in table cells
- REVIKI-466: Extra newline causes formatting to implode?
- REVIKI-514: Can't create mailto: links.

## Backwards compatibility

I tested against a few of the CFL wikis, and of 1972 articles, there are only 21 which need editing, and of those the majority are using slightly weird markup (presumably to overcome issues in the old renderer). There are also 8 pages which were broken in the old renderer, and are still broken in (a possibly different way) the new renderer. Finally, there are 81 pages which render slightly differently, but in a way which I believe still acceptable.

A full list of deviations is available on [pdds:RevikiANTLRParser].

## Notes

- The `ASTNode::expandMacros` and `TextNode::append` methods look somewhat complicated and use reflection to construct new nodes, rather than just mutating themselves. This is to make all `ASTNode`s immutable, which I generally hold to be a Good Thing™.
- The lexer is not very straightforward. This is unfortunate, but I couldn't think of a better way to handle things. I have tried to keep complexity to a minimum.

[bef6a5d]: https://github.com/Barrucadu/reviki/commit/bef6a5d8333272b9629cfea08148e51260c45e49
[pdds:RevikiANTLRParser]: https://wiki.int.corefiling.com/pdds/RevikiANTLRParser