[BUG #6788] two new attempts at prettyp

- formatter_1.py: external visitor-based formatter
  - the idea was to put commen pre-/post-processing in generic 'visit' method
  - not working well for comments, as they have to be infix sometimes
- formatter_2.py: direct-to-string parsing
  - like jsbeautifier does
  - looks more promising
[BUG #6788] yield whitespace from tokenizer

- to support an 'identical' prettyp (that just reproduces the input)
- might also be interesting in order to respect existing formatting
[BUG #6788] experimenting with both formatter and formatter_2

- using comments/commentsAfter/commentsIn, to carry over comments provided by
  tokens not represented in the ast (']', '}', ',', ';' ...)
  - this still looks really tedious, e.g. for expressions like '[/*foo*/]] you
    need more and more machinery
- direct-to-string parsing in formatter_2 looks interesting, working on this
[BUG #6788] simple way of re-creating the original formatting
[BUG #6788] manage indent from curly braces
[BUG #6788] commiting an intermediate state

- to save this work
[BUG #6788] need to adjust state.currColumn when modifying self.line
[BUG #6788] added more config features
[BUG #6788] maintain currColumn as a property on Formatter class

- the intent is to have currColumn automatically return the length of Formatter.line
[BUG #6788] some smaller improvements
[BUG #6788] started a Concrete Syntax Tree based pretty-printer
[BUG #6788] more work on treegenerator_3
[BUG #6788] finished treegenerator_3 and started with formatter_3

- it seems that formatter rules become very simple with the
  treegenerator_3's CST
- the integration with CodeGenerator/compile.py is still very coarse
[BUG #6788] fixed expression list parsing in condition part of for(;;) control expression

- also streamlined parsing of init part
[BUG #6788] more work on formatter_3 (loops, etc.)

- add helpers prefix_keyword and prefix_kw_optarg
[BUG #6788] more work on formatter_3
[BUG #6788] tried to accumulate output text in the 'state' object (.line, .output)

- not working currently, as return values are replaced by calls to
  state.add()
- but still needed in upstream format() functions
- an external visitor might be better here
[BUG #6788] changed formtter_3 to use an accumulator (in state), and added formatter_4
[BUG #6788] switched back to formatter_3

- it seems important to be able to control the end of construct (like 'statement'),
  so recursion is better than iteration
- i still try to retain the benefits from formatter_4
- trying to do the text_width checking in FormatterState.add(), as a central
  place for adding to the output stream
  (that's what i created the iterative formatter_4 for in the first place)
- hence forwarding the current token to add(), for making decisions
- (i guess i could also keep a backlog of past 10 tokens, for lookbehind's, if that
  turns out to be helpful)

- formatter_4 is working on a primitive basis (token generator from tree is set up,
  central 'format' routine works, and basic output text is produced)
- what would be missing would be more specific format_* routines, to handle
  special tokens
- but the case of 'function' showed the problem to detect end-of-construct
  (to e.g. add a newline or semi)

- in treegenerator_3, the toListG() methods are geared towards only yielding
  lexical tokens, and skip synthetic tokens (that might or might not be
  handy for consumers other than formatter_4)
[BUG #6788] more work on formatter_3

- fixed TextWidth support
- various white space issues (too much, too little)
- last thing i want to look at is the 'mixed' mode for opening curlies