Add charactes position to tokens, clean up handling of nonblank info.


git-svn-id: https://fricas.svn.sourceforge.net/svnroot/fricas/trunk@1575 b0c55286-4f34-0410-a049-a1e7e93b0762