Package: texcptulz
Text Clean and Preprocess Tools (texcptulz)textcptulz is a text preprocessing framework to transform raw ingested text into a form that is ready for computation and modeling.Table of ContentsInstallationsProject MotivationModulesInstructionsLicenseInstallationsDependenciestextcptulz requires:Python (>=3.5)NumPyscikit-learnNLTKspaCyspacy-langdetectgensimInstallationYou can install texcptulz usingpippip install texcptulzThat easy.Project Motivationtexcptulz aims to be a wrapper for all that processes involved in the wrangling part of an ETL pipeline for text analysis.
In addition, it includes other useful tools such as the detection of the language of a document. These are some of the
supported languages:EnglishSpanishFrenchGermanItalianModulesThe modules included in texcptulz are:normalizerThenormalizermodule includesTextNormalizerclass that can perform the text normalization process, taking a text
as input and returning a list of lemmatized tokens.Special characters, HTML tags, urls, etc. will be removed, using theclean_textfunction, then, the text will be
converted to lowercase and will be splitted into tokens and lemmatized.clustererTheclusterermodule has only one classKMeansClustersthat performs k-means clustering. The distance measure used
is cosine distance. Thekparameter is the number of clusters, the default value is 7.similarityThesimilaritymodule has only one class,Similaritythat creates amxmsimilarity matrix for a group of documents.
In this matrix, columns and rows represents the index of each document the elements of the matrix are similarity between
documents.The range of values will be between 0 and 1, where 0 means totally different and 1 means that the content is identical.vectorizerThevectorizermodule has only one class,OneHotVectorizerthat performs the text conversion into a vector
representations of categorical variables as binary vectors (one hot encoding)utilsThis module includes theLangDetectorclass to detect the language of a documentInstructionsNormalize textfromtxtools.normalizerimportTextNormalizertext='Python is a programming language that lets you work quickly'\' and integrate systems more effectively'normalizer=TextNormalizer()tokens=normalizer.normalize(text,clean=True)print(tokens)# output# ['python', 'programming', 'language', 'let', 'work', 'quickly', 'integrate', 'system', 'effectively']TextNormalizerclass implements theTransformerinterface, this allows us to add this class into a scikit-learn pipeline.Language detectionutilsmodule includes theLangDetectorclass intended to detect the language in which a document was written. The
detection is done with a certain confidence. You can provide a minimum percentage of confidence for a detection process:fromtxtools.utilsimportLangDetectordocuments=['El aprendizaje automático tiene una amplia gama de aplicaciones, incluyendo motores de búsqueda, diagnósticos médicos, detección de fraude en el uso de tarjetas de crédito','Machine learning tasks are classified into several broad categories. In supervised learning, the algorithm builds a mathematical model from a set of data','L\'apprendimento automatico viene impiegato in quei campi dell\'informatica nei quali progettare e programmare algoritmi espliciti è impraticabile','Selon les informations disponibles durant la phase d\'apprentissage, l\'apprentissage est qualifié de différentes manières.']lang_detector=LangDetector()foridx,documentinenumerate(documents):print('Document{}is written in{}'.format(idx,lang_detector.lang(document)))# output# Document 0 is written in Spanish# Document 1 is written in English# Document 2 is written in Italian# Document 3 is written in FrenchAlso, you can get the ISO 639-1 code:foridx,documentinenumerate(documents):print('ISO 639-1 code for document{}is{}'.format(idx,lang_detector.iso_639_1_code(document)))# output# ISO 639-1 lang code for document 0 is es# ISO 639-1 lang code for document 1 is en# ISO 639-1 lang code for document 2 is it# ISO 639-1 lang code for document 3 is frClustering documentsfromtxtools.clustererimportKMeansClustersdocuments=[]# this must contains a lot of documentsk_means=KMeansClusters(k=10)clusters=k_means.transform(documents)Compute document similaritySimilaritycreates creates amxmsimilarity matrix for a corpus wheremis the number of documents.fromtxtools.similarityimportSimilaritydocuments=['Psycho is a 1960 American psychological horror film directed and produced by Alfred Hitchcock, and written by Joseph Stefano.','North by Northwest is a 1959 American thriller film directed by Alfred Hitchcock, starring Cary Grant, Eva Marie Saint and James Mason.','The Birds is a 1963 American horror-thriller film directed and produced by Alfred Hitchcock.','Rear Window is a 1954 American Technicolor mystery thriller film directed by Alfred Hitchcock and written by John Michael Hayes based on Cornell Woolrich\'s 1942 short story "It Had to Be Murder".']similarity=Similarity()sims=similarity.transform(documents)print(sims)# output# [[0.99999994 0.         0.13075474 0.02665992]#  [0.         1.         0.00812627 0.00331377]#  [0.13075474 0.00812627 1.         0.0069054 ]#  [0.02665992 0.00331377 0.0069054  1.        ]]Using a scikit-learn pipelineTextNormalizer,OneHotVectorizer,KMeansClustersandSimilarityimplement theTransformerinterface, so we can
add them to a scikit-learn pipeline.fromsklearn.pipelineimportPipelinefromtxtools.normalizerimportTextNormalizerfromtxtools.similarityimportSimilaritydocuments=['Psycho is a 1960 American psychological horror film directed and produced by Alfred Hitchcock, and written by Joseph Stefano.','North by Northwest is a 1959 American thriller film directed by Alfred Hitchcock, starring Cary Grant, Eva Marie Saint and James Mason.','The Birds is a 1963 American horror-thriller film directed and produced by Alfred Hitchcock.','Rear Window is a 1954 American Technicolor mystery thriller film directed by Alfred Hitchcock and written by John Michael Hayes based on Cornell Woolrich\'s 1942 short story "It Had to Be Murder".']model=Pipeline([('norm',TextNormalizer()),('sim',Similarity())])sims=model.fit_transform(documents)Cleaning textYou can clean text with theclean_textfunction included in thenormalizermodulefromtxtools.normalizerimportclean_texttext='<p><i><b>2001: A    Space Odyssey</b></i>\n\nis a 1968 <a href="/wiki/Epic_film" title="Epic '\'film">epic</a> <a href="/wiki/Science_fiction_film" title="Science fiction film">science fiction film</a> '\'produced and directed by\t<a href="/wiki/Stanley_Kubrick" title="Stanley Kubrick">Stanley Kubrick</a>. '\'The screenplay was written by Kubrick and <a href="/wiki/Arthur_C._Clarke" title="Arthur C. '\'Clarke">Arthur C. Clarke</a>, and was inspired by Clarke\'s short story ""<a href="/wiki/The_Sentinel_('\'short_story)" title="The Sentinel (short story)">The Sentinel</a>" and other short stories by Clarke. A '\'<a href="/wiki/2001:_A_Space_Odyssey_(novel)" title="2001: A Space Odyssey (novel)">novelisation of the '\'film</a> released after the film\'s premiere was in part written concurrently with the screenplay. The '\'film, which follows a voyage to <a href="/wiki/Jupiter" title="Jupiter">Jupiter</a> with the <a '\'href="/wiki/Sentience" title="Sentience">sentient</a> computer <a href="/wiki/HAL_9000" title="HAL '\'9000">HAL</a> after the discovery of a <a href="/wiki/Monolith_(Space_Odyssey)" title="Monolith (Space '\'Odyssey)">featureless alien monolith</a> affecting human evolution, deals with themes of <a '\'href="/wiki/Existentialism" title="Existentialism">existentialism</a>, <a href="/wiki/Human_evolution" '\'title="Human evolution">human evolution</a>, technology, <a href="/wiki/Artificial_intelligence" '\'title="Artificial intelligence">artificial intelligence</a>, and the possibility of <a '\'href="/wiki/Extraterrestrial_life" title="Extraterrestrial life">extraterrestrial life</a>.</p> 'print(clean_text(text))# output# 2001: A Space Odyssey is a 1968 epic science fiction film produced and directed by Stanley Kubrick. The screenplay# was written by Kubrick and Arthur C. Clarke, and was inspired by Clarke's short story "The Sentinel" and other# short stories by Clarke. A novelisation of the film released after the film's premiere was in part written# concurrently with the screenplay. The film, which follows a voyage to Jupiter with the sentient computer HAL after# the discovery of a featureless alien monolith affecting human evolution, deals with themes of existentialism,# human evolution, technology, artificial intelligence, and the possibility of extraterrestrial life.LicenseCopyright (c) 2019 Cisco DelgadoPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.