Add tokenization with irregular expressions and associated test