Package: tidyX
tidyXtidyXis a Python package designed for cleaning and preprocessing text for machine learning applications,especially for text written in Spanish and originating from social networks.This library provides a complete pipeline to remove unwanted characters, normalize text, group similar terms, etc. to facilitate NLP applications.To deep dive in the package visit ourwebsiteInstallationInstall the package using pip:pipinstalltidyXMake sure you have the necessary dependencies installed. If you plan on lemmatizing, you'll needspaCyalong with the appropriate language models. For Spanish lemmatization, we recommend downloading thees_core_news_smmodel:python-mspacydownloades_core_news_smFor English lemmatization, we suggest theen_core_web_smmodel:python-mspacydownloaden_core_web_smTo see a full list of available models for different languages, visitSpacy's documentation.FeaturesStandardize Text Pipeline: Thepreprocess()method provides an all-encompassing solution for quickly and effectively standardizing input strings, with a particular focus on tweets. It transforms the input to lowercase, strips accents (and emojis, if specified), and removes URLs, hashtags, and certain special characters. Additionally, it offers the option to delete stopwords in a specified language, trims extra spaces, extracts mentions, and removes 'RT' prefixes from retweets.fromtidyXimportTextPreprocessorastp# Raw tweet exampleraw_tweet="RT @user: Check out this link: https://example.com 🌍 #example 😃"# Applying the preprocess methodcleaned_text=tp.preprocess(raw_tweet)# Printing the cleaned textprint("Cleaned Text:",cleaned_text)Output:Cleaned Text: check out this linkTo remove English stopwords, simply add the parametersremove_stopwords=Trueandlanguage_stopwords="english":fromtidyXimportTextPreprocessorastp# Raw tweet exampleraw_tweet="RT @user: Check out this link: https://example.com 🌍 #example 😃"# Applying the preprocess method with additional parameterscleaned_text=tp.preprocess(raw_tweet,remove_stopwords=True,language_stopwords="english")# Printing the cleaned textprint("Cleaned Text:",cleaned_text)Output:Cleaned Text: check linkFor a more detailed explanation of the customizable steps of the function, visit the officialpreprocess() documentation.Stemming and Lemmatizing: One of the foundational steps in preparing text for NLP applications is bringing words to a common base or root. This library provides bothstemmer()andlemmatizer()functions to perform this task across various languages.Group similar terms: When working with a corpus sourced from social networks, it's common to encounter texts with grammatical errors or words that aren't formally included in dictionaries. These irregularities can pose challenges when creating Term Frequency matrices for NLP algorithms. To address this, we developed thecreate_bol()function, which allows you to create specific bags of terms to cluster related terms.Remove unwanted elements: such asspecial characters,extra spaces,accents,emojis,urls, tweetermentions, among others.Dependency Parsing Visualization: Incorporates visualization tools that enable the display of dependency parses, facilitating linguistic analysis and feature engineering.Much more!TutorialsStemming and Lemmatizing Texts EfficientlyWord CloudTopic ModellingContributingContributions to enhancetidyXare welcome! Feel free to open issues for bug reports, feature requests, or submit pull requests. If this package has been helpful, please give us a star :D