evilhackerdude/syntux — Issue #1: Choosing a parser
Author: felixge
Created: 2011-10-25 06:39:05+00:00
State: open

So, I've been thinking about this and I don't think any of the current parsers will fit the bill. That is, I think we want syntux to work "incrementally" (use as many or little options as you'd like), and not mess with code formatting unless you specifically ask it to.

For this to work we need a parser that builds an AST including all whitespace details, and the parser must also not normalize any tokens. For example the following two strings would need a different representation in the AST, even so they hold the same value:

```js
var a = 'foo ';
var b = 'foo \
';
```

Currently ZeParser seems to be the closest to what we need, as it already includes offset information. However, it doesn't handle comments and it also uses a generic 'whitespace' token regardless of the nature of the whitespace (newline, space, tab, ...).

So I think our best option initially is to fork ZeParser, implement a AST.stringify() for it, and then test parsing and re-assembling of some large JavaScript files (jQuery, our own stuff, etc.) and make sure they are identical.

Once we can take apart and put together complex JS files, we should be in good shape to modify the AST in between.

Let me know what you think!

--fg