refactored. methods outsourced to a separate service
enhance tests. add unexpected tokens.