sorting, formatting
added support for pretty-printing manually generated tokens
finished juniper preprocessing