tweak shell scripts
test stuff
add doc/lexing.md
add test driver files