refactored code to return tokens with properties and metrics after text analysis; this can double function as api