proper eof handling, closing open tokens and handle the last one