# Deep Learning for Information Triage on Twitter
## 3 Preliminary Study: Types of Information Found on Twitter
### Data Preprocessing  
Since with our information triage method, we aimed at providing help primarily in Japan (although making other language versions is possible later), we focused on developing the method firstly on data in the Japanese language.  
The datasets used in this research (see Section 3) were in Japanese, which posed several challenges. Firstly, in transcription of the Japanese language, spaces (\" \") are not used. Therefore, we needed to preprocess the dataset and make the sentences separable  
Figure 8: Tweets which appeared in each day till the day of election.  
Figure 9: Outline of the system.  
into elements for feature extraction. We used MeCab ([[http://taku910.github.io/mecab/](http://taku910.github.io/mecab/)]([http://taku910.github.io/mecab/](http://taku910.github.io/mecab/)), accessed on 23 June 2021), a Japanese morphological analyzer, and CaboCha ([[http://taku910.github.io/cabocha/](http://taku910.github.io/cabocha/)]([http://taku910.github.io/cabocha/](http://taku910.github.io/cabocha/)), accessed on 23 June 2021), a Japanese dependency structure analyzer to preprocess the dataset in the following ways (performance of MeCab is reported around 95-97% (20), and Cabocha around 90% (21) for normal language. Although we acknowledge that in some cases the language used on Twitter could cause errors in POS tagging and word segmentation, we did not want to retrain the basic tools to fit our data because we wanted the method to work using widely available resources, so it was easily reproducible. Additionally, we assumed that even if such errors occur, as long as they are systematic, they will not cause trouble):  
* **Tokenization:** All words, punctuation marks, and so forth are separated by spaces (later: TOK).
* **Lemmatization:** Like the above, but the words are represented in their generic (dictionary) forms, or \"lemmas\" (later: LEM).
* **Parts of speech:** Words are replaced with their representative parts of speech (later: POS).
* **Tokens with POS:** Both words and POS information is included in one element (later: TOK + POS).
* **Lemmas with POS:** Like the above, but with lemmas instead of words (later: LEM + POS).
* **Tokens with Named Entity Recognition:** Words encoded together with with information on what named entities (private name of a person, organization, numericals, etc.) appear in the sentence. The NER information is annotated by CaboCha (later: TOK + NER).
* **Lemmas with NER:** Like the above but with lemmas (later: LEM + NER).
* **Chunking:** Larger sub-parts of sentences separated syntactically, such as noun phrase, verb phrase, predicates, etc., but without dependency relations (later: CHNK).
* **Dependency structure:** Same as above, but with information regarding syntactical relations between chunks (later: DEP).
* **Chunking with NER:** Information on named entities is encoded in chunks (later: CHNK + NER).
* **Dependency structure with Named Entities:** Both dependency relations and named entities are included in each element (later: DEP + NER).  
Five examples of preprocessing are represented in Table 5. Theoretically, the more generalized a sentence is, the less unique and frequent patterns it will contain, but the produced patterns will be more frequent (e.g., there are more ADJ N patterns than \"pleasant day\").  
We compared the results for different preprocessing methods to find out whether it is better for information triage to represent sentences, such as more generalized or more specific.