Warn instead of die on json parsing errors
Was not copying hierarchy
Tokenizing was skipping the trailing punctuation, which is bad for short documents