Package: connlp
connlpA bunch of python codes to analyze text data in the construction industry.Mainly reconstitute the pre-exist python libraries for Natural Language Processing (NLP).Project InformationSupported by C!LAB (@Seoul Nat'l Univ.)ContributorsSeonghyeon Boris Moon (blank54@snu.ac.kr,https://github.com/blank54/)Sehwan Chung (hwani751@snu.ac.kr)Jungyeon Kim (janykjy@snu.ac.kr)InitializeSetupInstallconnlpwithpip.pipinstallconnlpInstallrequirements.txt.cdWORKSPACE
wget-Orequirements_connlp.txthttps://raw.githubusercontent.com/blank54/connlp/master/requirements.txt
pipinstall-rrequirements_connlp.txtTestIf the code below runs with no error,connlpis installed successfully.fromconnlp.testimporthellohello()# 'Helloworld'PreprocessPreprocessing module supports English and Korean.NOTE: No plan for other languages (by 2021.04.02.).NormalizerNormalizernormalizes the input text by eliminating trash characters and remaining numbers, alphabets, and punctuation marks.fromconnlp.preprocessimportNormalizernormalizer=Normalizer()normalizer.normalize(text='I am a boy!')# 'i am a boy'EnglishTokenizerEnglishTokenizertokenizes the input text in English based on word spacing.The ngram-based tokenization is in preparation.fromconnlp.preprocessimportEnglishTokenizertokenizer=EnglishTokenizer()tokenizer.tokenize(text='I am a boy!')# ['I', 'am', 'a', 'boy!']KoreanTokenizerKoreanTokenizertokenizes the input text in Korean, and is based on either pre-trained or unsupervised approaches.You are recommended to use pre-trained method unless you have a large size of corpus. This is the default setting.If you want to use a pre-trained tokenizer, you have to select which analyzer you want to use. Available analyzers are based on KoNLPy (https://konlpy.org/ko/latest/api/konlpy.tag/), a python package for Korean language processing. The default analyzer isHannanumfromconnlp.preprocessimportKoreanTokenizertokenizer=KoreanTokenizer(pre_trained=True,analyzer='Hannanum')If your corpus is big, you may use an unsupervised method, which is based onsoynlp(https://github.com/lovit/soynlp), an unsupervised text analyzer in Korean.fromconnlp.preprocessimportKoreanTokenizertokenizer=KoreanTokenizer(pre_trained=False)trainIf yourKoreanTokenizerare pre-trained, you can neglect this step.Otherwhise (i.e. you are using an unsupervised approach), theKoreanTokenizerobject first needs to be trained on (unlabeled) corpus. 'Word score' is calculated for every subword in the corpus.fromconnlp.preprocessimportKoreanTokenizertokenizer=KoreanTokenizer(pre_trained=False)docs=['코퍼스의 첫 번째 문서입니다.','두 번째 문서입니다.','마지막 문서']tokenizer.train(text=docs)print(tokenizer.word_score)# {'서': 0.0, '코': 0.0, '째': 0.0, '.': 0.0, '의': 0.0, '마': 0.0, '막': 0.0, '번': 0.0, '문': 0.0, '코퍼': 1.0, '번째': 1.0, '마지': 1.0, '문서': 1.0, '코퍼스': 1.0, '문서입': 0.816496580927726, '마지막': 1.0, '코퍼스의': 1.0, '문서입니': 0.8735804647362989, '문서입니다': 0.9036020036098448, '문서입니다.': 0.9221079114817278}tokenizeIf you are using a pre-trainedKoreanTokenizer, the selected KoNLPy analyzer will tokenize the input sentence based on morphological analysis.fromconnlp.preprocessimportKoreanTokenizertokenizer=KoreanTokenizer(pre_trained=True,analyzer='Hannanum')doc=docs[0]# '코퍼스의 첫 번째 문서입니다.'tokenizer.tokenize(doc)# ['코퍼스', '의', '첫', '번째', '문서', '입니다', '.']If you are using an unsupervisedKoreanTokenizer, tokenization is based on the 'word score' calculated fromKoreanTokenizer.trainmethod.For each blank-separated token, a subword that has the maximum 'word score' is selectd as an individual 'word' and separated with the remaining part.fromconnlp.preprocessimportKoreanTokenizertokenizer=KoreanTokenizer(pre_trained=False)doc=docs[0]# '코퍼스의 첫 번째 문서입니다.'tokenizer.tokenize(doc)# ['코퍼스의', '첫', '번째', '문서', '입니다.']StopwordRemoverStopwordRemoverremoves stopwords from a given sentence based on the user-customized stopword list.Before utilizingStopwordRemoverthe user should normalize and tokenize the docs.fromconnlp.preprocessimportNormalizer,EnglishTokenizer,StopwordRemovernormalizer=Normalizer()eng_tokenizer=EnglishTokenizer()stopword_remover=StopwordRemover()docs=['I am a boy!','He is a boy..','She is a girl?']tokenized_docs=[]fordocineng_docs:normalized_doc=normalizer.normalize(text=doc)tokenized_doc=eng_tokenizer.tokenize(text=normalized_doc)tokenized_docs.append(tokenized_doc)print(docs)print(tokenized_docs)# ['I am a boy!', 'He is a boy..', 'She is a girl?']# [['i', 'am', 'a', 'boy'], ['he', 'is', 'a', 'boy'], ['she', 'is', 'a', 'girl']]The user should prepare a customized stopword list (i.e.,stoplist).Thestoplistshould include user-customized stopwords divided by '\n' and the file should be in ".txt" format.a
is
amInitiate theStopwordRemoverwith appropriate filepath of user-customized stopword list.If the stoplist is absent at the filepath, the stoplist would be ramain as a blank list.fpath_stoplist='test/thesaurus/stoplist.txt'stopword_remover.initiate(fpath_stoplist=fpath_stoplist)print(stopword_remover)# <connlp.preprocess.StopwordRemover object at 0x7f163e70c050>The user can count the word frequencies and figure out additional stopwords based on the results.stopword_remover.count_freq_words(docs=tokenized_docs)# ========================================# Word counts#   | [1] a: 3#   | [2] boy: 2#   | [3] is: 2#   | [4] i: 1#   | [5] am: 1#   | [6] he: 1#   | [7] she: 1#   | [8] girl: 1After finally updating thestoplist, useremovemethod to remove the stopwords from text.stopword_removed_docs=[]fordocintokenized_docs:stopword_removed_docs.append(stopword_remover.remove(sent=doc))print(stopword_removed_docs)# [['i', 'boy'], ['he', 'boy'], ['she', 'girl']]The user can check which stopword was removed withcheck_removed_wordsmethods.stopword_remover.check_removed_words(docs=tokenized_docs,stopword_removed_docs=stopword_removed_docs)# ========================================# Check stopwords removed#   | [1] BEFORE: a(3) ->#   | [2] BEFORE: boy -> AFTER: boy(2)#   | [3] BEFORE: is(2) ->#   | [4] BEFORE: i -> AFTER: i(1)#   | [5] BEFORE: am(1) ->#   | [6] BEFORE: he -> AFTER: he(1)#   | [7] BEFORE: she -> AFTER: she(1)#   | [8] BEFORE: girl -> AFTER: girl(1)EmbeddingVectorizerVectorizerincludes several text embedding methods that have been commonly used for decades.tfidfTF-IDF is the most commonly used technique for word embedding.The TF-IDF model counts the term frequency(TF) and inverse document frequency(IDF) from the given documents.The results included the followings.TF-IDF Vectorizer (a class of sklearn.feature_extraction.text.TfidfVectorizer')TF-IDF MatrixTF-IDF Vocabularyfromconnlp.preprocessimportEnglishTokenizerfromconnlp.embeddingimportVectorizertokenizer=EnglishTokenizer()vectorizer=Vectorizer()docs=['I am a boy','He is a boy','She is a girl']tfidf_vectorizer,tfidf_matrix,tfidf_vocab=vectorizer.tfidf(docs=docs)type(tfidf_vectorizer)# <class 'sklearn.feature_extraction.text.TfidfVectorizer'>The user can get a document vector by indexing thetfidf_matrix.tfidf_matrix[0]# (0, 2)    0.444514311537431# (0, 0)    0.34520501686496574# (0, 1)    0.5844829010200651# (0, 5)    0.5844829010200651Thetfidf_vocabreturns an index for every token.print(tfidf_vocab)# {'i': 5, 'am': 1, 'a': 0, 'boy': 2, 'he': 4, 'is': 6, 'she': 7, 'girl': 3}word2vecWord2Vec is a distributed representation language model for word embedding.The Word2vec model trains tokenized docs and returns word vectors.The result is a class of 'gensim.models.word2vec.Word2Vec'.fromconnlp.preprocessimportEnglishTokenizerfromconnlp.embeddingimportVectorizertokenizer=EnglishTokenizer()vectorizer=Vectorizer()docs=['I am a boy','He is a boy','She is a girl']tokenized_docs=[tokenizer.tokenize(text=doc)fordocindocs]w2v_model=vectorizer.word2vec(docs=tokenized_docs)type(w2v_model)# <class 'gensim.models.word2vec.Word2Vec'>The user can get a word vector by.wvmethod.w2v_model.wv['boy']# [-2.0130998e-03 -3.5652996e-03  2.7793974e-03 ...]The Word2Vec model provides thetopn-most similar word vectors.w2v_model.wv.most_similar('boy',topn=3)# [('He', 0.05311150848865509), ('a', 0.04154288396239281), ('She', -0.029122961685061455)]word2vec (update)The user can update the Word2Vec model with new data.new_docs=['Tom is a man','Sally is not a boy']tokenized_new_docs=[tokenizer.tokenize(text=doc)fordocinnew_docs]w2v_model_updated=vectorizer.word2vec_update(w2v_model=w2v_model,new_docs=tokenized_new_docs)w2v_model_updated.wv['man']# [4.9649975e-03  3.8002312e-04 -1.5773597e-03 ...]doc2vecDoc2Vec is a distributed representation language model for longer text (e.g., sentence, paragraph, document) embedding.The Doc2vec model trains tokenized docs with tags and returns document vectors.The result is a class of 'gensim.models.doc2vec.Doc2Vec'.fromconnlp.preprocessimportEnglishTokenizerfromconnlp.embeddingimportVectorizertokenizer=EnglishTokenizer()vectorizer=Vectorizer()docs=['I am a boy','He is a boy','She is a girl']tagged_docs=[(idx,tokenizer.tokenize(text=doc))foridx,docinenumerate(docs)]d2v_model=vectorizer.doc2vec(tagged_docs=tagged_docs)type(d2v_model)# <class 'gensim.models.doc2vec.Doc2Vec'>The Doc2Vec model can infer a new document.test_doc=['My','name','is','Peter']d2v_model.infer_vector(doc_words=test_doc)# [4.8494316e-03 -4.3647490e-03  1.1437446e-03 ...]AnalysisTopicModelTopicModelis a class for topic modeling based on gensim LDA model.It provides a simple way to train lda model and assign topics to docs.Before using LDA topic modeling, the user should install the following packages.pipinstallpyldavis==2.1.2TopicModelrequires two instances.a dict of docs whose keys are the tagthe number of topics for modelingfromconnlp.analysis_ldaimportTopicModelnum_topics=2docs={'doc1':['I','am','a','boy'],'doc2':['He','is','a','boy'],'doc3':['Cat','on','the','table'],'doc4':['Mike','is','a','boy'],'doc5':['Dog','on','the','table'],}lda_model=TopicModel(docs=docs,num_topics=num_topics)learnThe user can train the model withlearnmethod.
Unless parameters being provided by the user, the model trains based on default parameters.Afterlearn,TopicModelprovidesmodelinstance that is a class of <'gensim.models.ldamodel.LdaModel'>parameters={'iterations':100,'alpha':0.7,'eta':0.05,}lda_model.learn(parameters=parameters)type(lda_model.model)# <class 'gensim.models.ldamodel.LdaModel'>coherenceTopicModelprovides coherence value for model evaluation.The coherence value is automatically calculated right after model training.print(lda_model.coherence)# 0.3607990279229385assignThe user can easily assign the most proper topic to each doc usingassignmethod.Afterassign, theTopicModelprovidestag2topicandtopic2taginstances for convenience.lda_model.assign()print(lda_model.tag2topic)print(lda_model.topic2tag)# defaultdict(<class 'int'>, {'doc1': 1, 'doc2': 1, 'doc3': 0, 'doc4': 1, 'doc5': 0})# defaultdict(<class 'list'>, {1: ['doc1', 'doc2', 'doc4'], 0: ['doc3', 'doc5']})NamedEntityRecognitionBefore using NER modules, the user should install proper versions of TensorFlow and Keras.pipinstallconfig==0.4.2gensim==3.8.1gpustat==0.6.0GPUtil==1.4.0h5py==2.10.0JPype1==0.7.1Keras==2.2.4konlpy==0.5.2nltk==3.4.5numpy==1.18.1pandas==1.0.1scikit-learn==0.22.1scipy==1.4.1silence-tensorflow==1.1.1soynlp==0.0.493tensorflow==1.14.0tensorflow-gpu==1.14.0The modules might require the module ofkeras-contrib.The user can install the module by following the below.gitclonehttps://www.github.com/keras-team/keras-contrib.gitcdkeras-contribpythonsetup.pyinstallLabelsNER_Modelis a class to conduct named entity recognition using Bi-directional Long-Short Term Memory (Bi-LSTM) and Conditional Random Field (CRF).At the beginning, appropriate labels are required.The labels should be numbered with start of 0.fromconnlp.analysis_nerimportNER_Labelslabel_dict={'NON':0,#None'PER':1,#PERSON'FOD':2,}#FOODner_labels=NER_Labels(label_dict=label_dict)CorpusNext, the user should prepare data including sentences and labels, of which each data being matched by the same tag.The tokenized sentences and labels are then combined viaNER_LabeledSentence.With the data, labels, and a proper size ofmax_sent_len(i.e., the maximum length of sentence for analysis),NER_Corpuswould be developed.Once the corpus was developed, every data of sentences and labels would be padded with the length ofmax_sent_len.fromconnlp.preprocessimportEnglishTokenizerfromconnlp.analysis_nerimportNER_LabeledSentence,NER_Corpustokenizer=EnglishTokenizer()data_sents={'sent1':'Sam likes pizza','sent2':'Erik eats pizza','sent3':'Erik and Sam are drinking soda','sent4':'Flora cooks chicken','sent5':'Sam ordered a chicken','sent6':'Flora likes chicken sandwitch','sent7':'Erik likes to drink soda'}data_labels={'sent1':[1,0,2],'sent2':[1,0,2],'sent3':[1,0,1,0,0,2],'sent4':[1,0,2],'sent5':[1,0,0,2],'sent6':[1,0,2,2],'sent7':[1,0,0,0,2]}docs=[]fortag,sentindata_sents.items():words=[str(w)forwintokenizer.tokenize(text=sent)]labels=data_labels[tag]docs.append(NER_LabeledSentence(tag=tag,words=words,labels=labels))max_sent_len=10ner_corpus=NER_Corpus(docs=docs,ner_labels=ner_labels,max_sent_len=max_sent_len)type(ner_corpus)# <class 'connlp.analysis_ner.NER_Corpus'>Word EmbeddingEvery word in theNER_Corpusshould be embedded into numeric vector space.The user can conduct embedding with Word2Vec which is provided inVectorizerofconnlp.Note that the embedding process ofNER_Corpusonly requires the dictionary of word vectors and the feature size.fromconnlp.preprocessimportEnglishTokenizerfromconnlp.embeddingimportVectorizertokenizer=EnglishTokenizer()vectorizer=Vectorizer()tokenized_sents=[tokenizer.tokenize(sent)forsentindata_sents.values()]w2v_model=vectorizer.word2vec(docs=tokenized_sents)word2vector=vectorizer.get_word_vectors(w2v_model)feature_size=w2v_model.vector_sizener_corpus.word_embedding(word2vector=word2vector,feature_size=feature_size)print(ner_corpus.X_embedded)# [[[-2.40120804e-03  1.74632657e-03  ...]#   [-3.57543468e-03  2.86567654e-03  ...]#   ...#   [ 0.00000000e+00  0.00000000e+00  ...]] ...]Model InitializationThe parameters for Bi-LSTM and model training should be provided, however, they can be composed of a single dictionary.The user should initialize theNER_ModelwithNER_Corpusand the parameters.fromconnlp.analysis_nerimportNER_Modelparameters={# Parameters for Bi-LSTM.'lstm_units':512,'lstm_return_sequences':True,'lstm_recurrent_dropout':0.2,'dense_units':100,'dense_activation':'relu',# Parameters for model training.'test_size':0.3,'batch_size':1,'epochs':100,'validation_split':0.1,}ner_model=NER_Model()ner_model.initialize(ner_corpus=ner_corpus,parameters=parameters)type(ner_model)# <class 'connlp.analysis_ner.NER_Model'>Model TrainingThe user can train theNER_Modelwith customized parameters.The model automatically gets the dataset from theNER_Corpus.ner_model.train(parameters=parameters)# Train on 3 samples, validate on 1 samples# Epoch 1/100# 3/3 [==============================] - 3s 1s/step - loss: 1.4545 - crf_viterbi_accuracy: 0.3000 - val_loss: 1.0767 - val_crf_viterbi_accuracy: 0.8000# Epoch 2/100# 3/3 [==============================] - 0s 74ms/step - loss: 0.8602 - crf_viterbi_accuracy: 0.7000 - val_loss: 0.5287 - val_crf_viterbi_accuracy: 0.8000# ...Model EvaluationThe model performance can be shown in the aspects of confusion matrix and F1 score.ner_model.evaluate()# |--------------------------------------------------# |Confusion Matrix:# [[ 3  0  3  6]#  [ 1  3  0  4]#  [ 0  0  2  2]#  [ 4  3  5 12]]# |--------------------------------------------------# |F1 Score: 0.757# |--------------------------------------------------# |    [NON]: 0.600# |    [PER]: 0.857# |    [FOD]: 0.571SaveThe user can save theNER_Model.The model would save the model itself ("<FileName>.pk") and the dataset ("<FileName>-dataset.pk") that was used in model development.Note that the directory should exist before saving the model.fromconnlp.utilimportmakedirfpath_model='test/ner/model.pk'makedir(fpath=fpath_model)ner_model.save(fpath_model=fpath_model)LoadIf the user wants to load the already trained model, just call the model and load.fpath_model='test/ner/model.pk'ner_model=NER_Model()ner_model.load(fpath_model=fpath_model,ner_corpus=ner_corpus,parameters=parameters)PredictionNER_Modelcan conduct a new NER task on the given sentence.The result is a class ofNER_Result.fromconnlp.preprocessimportEnglishTokenizervectorizer=Vectorizer()new_sent='Tom eats apple'tokenized_sent=tokenizer.tokenize(new_sent)ner_result=ner_model.predict(sent=tokenized_sent)print(ner_result)# Tom/PER eats/NON apple/FODWeb CrawlingTheconnlpcurrently provides web crawling for Naver news articles.QueryThe user should prepare the proper queries first.A single text file(.txt) should include every information of the query as below.Date StartDate EndKeywordsThe web crawler utilizes the keywords separated with '\n\n' in the same time.Meanwhile, the web crawler utilizes the keywords separated with '\n' as a different queries.For example, if the queries are determined as below, the web crawler would search the articles with six queries: "smart+construction+safety at 20210718", "smart+construction+management at 20210718", "smart+construction+safety at 20210719", ...20210718
20210720

smart

construction

safety
managementTheNewsQueryParserparses the queries into appropriate formats.fromconnlp.web_crawlingimportNewsQueryParserquery_parser=NewsQueryParser()fpath_query='FILEPATH_OF_YOUR_QUERY'query_list,date_list=query_parser.parse(fpath_query=fpath_query)URLsFor the second step, the web crawler parses the web page that shows the list of news articles.NaverNewsListScraperprovides the function of parsing the list page.The user is recommended to save the url lists and load them later.fromconnlp.web_crawlingimportNaverNewsListScraperlist_scraper=NaverNewsListScraper()fordateinsorted(date_list,reverse=False):forqueryinquery_list:url_list=list_scraper.get_url_list(query=query,date=date)ArticlesThe last step is to parse the article page and get information from the article.NaverNewsArticleParserreturns a class ofArticlefor a given article.Remember to extend the query list of the article.fromconnlp.web_crawlingimportNaverNewsArticleParserarticle_parser=NaverNewsArticleParser()query_list,_=query_parser.urlname2query(fname_url_list=fname_url_list)forurlinurl_list:article=article_parser.parse(url=url)article.extend_query(query_list)StatusNewsStatusprovides the status of the crawled corpus for given directories.fromconnlp.web_crawlingimportNewsStatusnews_status=NewsStatus()fdir_queries='DIRECTORY_FOR_QUERIES'fdir_url_list='DIRECTORY_FOR_URLS'fdir_article='DIRECTORY_FOR_ARTICLES'news_status.queries(fdir_queries=fdir_queries)news_status.urls(fdir_urls=fdir_url_list)news_status.articles(fdir_articles=fdir_article)VisualizationVisualizerVisualizerincludes several simple tools for text visualization.Install the following packages.pip install networkx wordcloudnetworknetworkmethod provides a word network for tokenized docs.fromconnlp.preprocessimportEnglishTokenizerfromconnlp.visualizeimportVisualizertokenizer=EnglishTokenizer()visualizer=Visualizer()docs=['I am a boy','She is a girl']tokenized_docs=[tokenizer.tokenize(text=doc)fordocindocs]word_network=visualizer.network(docs=tokenized_docs,show=True)The word network is amatplotlib.pyplotobject.The user can save the figure by.savefig()method.word_network.savefig(FILEPATH)wordcloudwordcloudmethod provides a word cloud for tokenized docs.fromconnlp.preprocessimportEnglishTokenizerfromconnlp.visualizeimportVisualizertokenizer=EnglishTokenizer()visualizer=Visualizer()docs=['I am a boy','She is a girl']tokenized_docs=[tokenizer.tokenize(text=doc)fordocindocs]wordcloud=visualizer.wordcloud(docs=tokenized_docs,show=True)The wordcloud is amatplotlib.pyplotobject.The user can save the figure by.savefig()method.wordcloud.savefig(FILEPATH)Extracting TextTextConverterTextConverterincludes several methods that extract raw text from various types of files (e.g. PDF, HWP) and/or converts the files into plain text files (e.g. TXT).hwp2txthwp2txtmethod converts a HWP file into a plain text file.
Dependencies: pyhwp packageInstall pyhwp (you need to install the pre-release version)pip install --pre pyhwpExamplefromconnlp.text_extractimportTextConverterconverter=TextConverter()hwp_fpath='/data/raw/hwp_file.hwp'output_fpath='/data/processed/extracted_text.txt'converter.hwp2txt(hwp_fpath,output_fpath)# returns 0 if no error occursGPU UtilsGPUMonitorGPUMonitorgenerates a class to monitor and display the GPU status based on nvidia-smi.Refer to "https://github.com/anderskm/gputil" and "https://data-newbie.tistory.com/561" for usages.InstallGPUtilsmodule withpip.pip install GPUtilWrite your code between the initiation of theGPUMonitorandmonitor.stop().fromconnlp.utilimportGPUMonitormonitor=GPUMonitor(delay=3)# >>>Write your code here<<<monitor.stop()# | ID | GPU | MEM |# ------------------# |  0 |  0% |  0% |# |  1 |  1% |  0% |# |  2 |  0% | 94% |