Package: nlp-id
Kumparan's NLP Servicesnlp-idis a collection of modules which provides various functions for Natural Language Processing for Bahasa Indonesia. This repository contains all source code related to NLP services.InstallationTo installnlp-id, use the following command:$ pip install nlp-idUsageDescription on how to use the lemmatizer, tokenizer, POS-tagger, etc. will be explained in more detail in this section.LemmatizerLemmatizer is used to get the root words from every word in a sentence.from nlp_id.lemmatizer import Lemmatizer 
lemmatizer = Lemmatizer() 
lemmatizer.lemmatize('Saya sedang mencoba') 
# saya sedang cobaTokenizerTokenizer is used to convert text into tokens of word, punctuation, number, date, email, URL, etc.
There are two kinds of tokenizer in this repository,standard tokenizerandphrase tokenizer.
Thestandard tokenizertokenizes the text into separate tokens where the word tokens are single-word tokens.
Tokens that started withku-or ended with-ku,-mu,-nya,-lah,-kahwill be split if it is personal pronoun or particle.from nlp_id.tokenizer import Tokenizer 
tokenizer = Tokenizer() 
tokenizer.tokenize('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# ['Lionel', 'Messi', 'pergi', 'ke', 'pasar', 'di', 'daerah', 'Jakarta', 'Pusat', '.']

tokenizer.tokenize('Lionel Messi pergi ke rumahmu di daerah Jakarta Pusat.') 
# ['Lionel', 'Messi', 'pergi', 'ke', 'rumah', 'mu', 'di', 'daerah', 'Jakarta', 'Pusat', '.']Thephrase tokenizertokenizes the text into separate tokens where the word tokens are phrases (single or multi-word tokens).from nlp_id.tokenizer import PhraseTokenizer 
tokenizer = PhraseTokenizer() 
tokenizer.tokenize('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# ['Lionel Messi', 'pergi', 'ke', 'pasar', 'di', 'daerah', 'Jakarta Pusat', '.']POS TaggerPOS tagger is used to obtain the Part-Of-Speech tag from a text.
There are two kinds of POS tagger in this repository,standard POS taggerandphrase POS tagger.
The tokens instandard POS Taggerare single-word tokens, while the tokens inphrase POS Taggerare phrases (single or multi-word tokens).from nlp_id.postag import PosTag
postagger = PosTag() 
postagger.get_pos_tag('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# [('Lionel', 'NNP'), ('Messi', 'NNP'), ('pergi', 'VB'), ('ke', 'IN'), ('pasar', 'NN'), ('di', 'IN'), ('daerah', 'NN'),  
  ('Jakarta', 'NNP'), ('Pusat', 'NNP'), ('.', 'SYM')]

postagger.get_phrase_tag('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# [('Lionel Messi', 'NP'), ('pergi', 'VP'), ('ke', 'IN'), ('pasar', 'NN'), ('di', 'IN'), ('daerah', 'NN'), 
  ('Jakarta Pusat', 'NP'), ('.', 'SYM')]Description of tagset used for POS Tagger:No.TagDescriptionExample1ADVAdverbs. Includes adverb, modal, and auxiliary verbsangat, hanya, justru, boleh, harus, mesti2CCCoordinating conjunction. Coordinating conjunction links two or more syntactically equivalent parts of a sentence. Coordinating conjunction can link independent clauses, phrases, or words.dan, tetapi, atau3DTDeterminer/article. A grammatical unit which limits the potential referent of a noun phrase, whose basic role is to mark noun phrases as either definite or indefinite.para, sang, si, ini, itu, nya4FWForeign word. Foreign word is a word which comes from foreign language and is not yet included in Indonesian dictionaryworkshop, business, e-commerce5INPreposition. A preposition links word or phrase and constituent in front of that preposition and results prepositional phrase.dalam, dengan, di, ke6JJAdjective. Adjectives are words which describe, modify, or specify some properties of the head noun of the phrasebersih, panjang, jauh, marah7NEGNegationtidak, belum, jangan8NNNoun. Nouns are words which refer to human, animal, thing, concept, or understandingmeja, kursi, monyet, perkumpulan9NNPProper Noun. Proper noun is a specific name of a person, thing, place, event, etc.Indonesia, Jakarta, Piala Dunia, Idul Fitri, Jokowi10NUMNumber. Includes cardinal and ordinal number9876, 2019, 0,5, empat11PRPronoun. Includes personal pronoun and demonstrative pronounsaya, kami, kita, kalian, ini, itu, nya, yang12RPParticle. Particle which confirms interrogative, imperative, or declarative sentencespun, lah, kah13SCSubordinating Conjunction. Subordinating conjunction links two or more clauses and one of the clauses is a subordinate clause.sejak, jika, seandainya, dengan, bahwa14SYMSymbols and Punctuations+,%,@15UHInterjection. Interjection expresses feeling or state of mind and has no relation with other words syntactically.ayo, nah, ah16VBVerb. Includes transitive verbs, intransitive verbs, active verbs, passive verbs, and copulas.tertidur, bekerja, membaca17ADJPAdjective Phrase. A group of words headed by an adjective that describes a noun or a pronounsangat tinggi18DPDate Phrase. Date written with whitespaces1 Januari 202019NPNoun Phrase. A phrase that has a noun (or indefinite pronoun) as its headJakarta Pusat, Lionel Messi20NUMPNumber Phrase.10 juta21VPVerb Phrase. A syntactic unit composed of at least one verb and its dependentstidak makanStopwordnlp-idalso provide list of Indonesian stopword.from nlp_id.stopword import StopWord 
stopword = StopWord() 
stopword.get_stopword() 
# [{list_of_nlp_id_stopword}]Stopword Removal is used to remove every Indonesian stopword from the given text.from nlp_id.stopword import StopWord 
text = "Lionel Messi pergi Ke pasar di area Jakarta Pusat" # single sentence
stopword = StopWord() 
stopword.remove_stopword(text)
# Lionel Messi pergi pasar area Jakarta Pusat  

paragraph = "Lionel Messi pergi Ke pasar di area Jakarta Pusat itu. Sedangkan Cristiano Ronaldo ke pasar Di area Jakarta Selatan. Dan mereka tidak bertemu begini-begitu."
stopword.remove_stopword(text)
# Lionel Messi pergi pasar area Jakarta Pusat. Cristiano Ronaldo pasar area Jakarta Selatan. bertemu.Training and EvaluationOur model is trained using stories from kumparan as the dataset. We managed to get ~93% accuracy on our test set.Citation