Simplified tokenisation regexp, since no punctuation or whitespace is desired except as part of a grouping bounded by square brackets.