Package: tokaafrika
Toka (tokaafrika)Justice for African LanguagesThis package helps with utility functions to provide african languageStopWordsand helps in generating the newStopWordswith ease and help in cleaning the text dataset, We are working on improving it and help you get reliableStopWordsandQuality Aligned Datasets.Currently we support the following languagesThroughout our application we will use the language code or description of the language interchangiably, this mean we will use the description of language code consistently. Refer to the table below.Language Code / ISO CODELanguage NameengEnglishsepSepediafrAfrikaanstsnSetswananblisiNdebelesswSiswatixhoisiXhosavenTshivendazulisiZulutsoXitsongasotSesothonuuN|uuInstallationDependenciesWe are using type hinting on this project.Toka-api requirespython>=3.9.13To get started started and install the package execute the following.>>>pipinstalltokaafrika==0.0.1To start using the package, follow the steps belowGetStopWords- (Prebuild Stopwords )At the moment theStopWordsare based onSouth African LanguagesincludingN|uu>>>fromtoka.tokaimportTokaAPI>>>api=TokaAPI()>>>stopwords=api.get_stopwords('tshivenda')# use fullname>>>print(stopwords)frozenset({'a','vha','u','na','tshi','nga','ya','ndi',...'o','khou','ni','uri','hu','ha','kha','i',...'zwi','tsha','ri','yo','wa','ho','vho','musi',...'ḽa','zwa','ḓo','amba','nahone','no'})>>>stopwords=api.get_stopwords('ven')# use shotname/code>>>print(stopwords)frozenset({'a','vha','u','na','tshi','nga','ya','ndi',...'o','khou','ni','uri','hu','ha','kha','i',...'zwi','tsha','ri','yo','wa','ho','vho','musi',...'ḽa','zwa','ḓo','amba','nahone','no'})To Clean SymbolsThis helps in cleaning symbols ensuring your data is clean and free of symbols>>>fromtoka.tokaimportTokaAPI>>>toka_object=TokaAPI()>>>clean_text=\>>>...toka_object.clean_symbols('Hello! This is an example\>>> ... text with numbers like 123 ')>>>print(clean_text)>>>hellothisisanexampletextwithnumberslikeGet Frequent WordsThis helps in quickly getting the frequent words and how many times is appears from given text>>>fromtoka.tokaimportTokaAPI>>>toka_object=TokaAPI()>>>english=toka_object.get_frequent_words('Hello test')>>>print(english){'hello':1,'test':1}ComputeStopWordsThis helps with in computing the stop words from given documents or text, it is accurate when using long text or big document>>>fromtoka.tokaimportTokaAPI>>>api=TokaAPI()>>>stopwords=api.compute_stopwords(..."the the are are the are on the on",3)>>>print(stopwords)['the','are','on']Load modelThis Assummes you have vectorizer pickle and model that is already trained and are both pickle files>>>fromtoka.tokaimportTokaAPI>>>api=TokaAPI()>>>model='model.pkl'>>>vector='vector.pkl'>>>clf,vector=api.load_model_from_pickle(model,...vector)DevelopmentWe welcome new contribution and if you have spotted a bug or have ideas around the improvement leave then in issues or fork the repo and develop the feature and create PR to merge the changes to the repository, ensure tests are written with edge cases.CitationsIf you usedtokaafrikapackage and it helped you a lot we would appreciate citations.@article{tokaafrika,
  title={tokaafrika: African Languages - Machine Learning Package},
  author={Ofentswe Lebogo},
  howpublished={\url{https://pypi.org/project/tokaafrika/}},
  year={2024}}