SakuraSound/MADden — Issue #1: POS Tagger having issues with unicode
Author: SakuraSound
Created: 2012-02-22 06:39:41+00:00
State: open

On running cgrant_postag(text) in the database, some tweets were having issues with handling unicode. I looked at the text in the database, and clearly the data was loaded properly, but the postag function seems to lose the encoding. My hope is that there is no need to recall the codecs package to deal with this.

Test Cases to see if this works...
Run pos_tag on tweets with id numbers:
115274198347292672
115274300042379264

these are confirmed to have unicode in them...