Package: randomlib
randomlibrandomlibis a python-based natural language processing library focused on the Indian languageMarathi. It provides an easy interface for NLP features like sentiment analysis, named entity recognition, hate speech detection, etc. exclusively for Marathi text.randomlib, the author of this library aims to bring Marathi to the forefront of IndicNLP. Our vision is to make Marathi a resource-rich language and promote AI for Maharashtra!Github RepoDemonstration with examplesFeatures:This library is customised to be used by a basic programmer and an ML practitioner.1. Basic Usage:This mode of access is designed from a basic programmer point of view and follow simpler way to perform the desired tasks. It provides the following features:Datasets:Provides the functionality to load the datasetAutocomplete:Text predictionPreprocess:Data cleaningTokenizer:Tokenizes textTagger:Named entity recognisionMaskFill:Predicts the masked tokensHate:Detects hate speechSentiment:Sentiment analysisSimilarity:Detects similarity2. Advanced Usage:This way of accessing the library is designed from an ML Practitioner's point of view and has more flexibility to choose a model for the desired task.MaskFill Model:Predicts the masked tokensGPT Model:Text predictionHate Model:Detects hate speechNER Model:Named entity recognisionSentiment Model:Sentiment analysisSimilarity Model:Detects similaritySome of the mentioned models have sub models within them that can be seen using thelistModels()function.Installation:pip install randomlib==[version]Eg.: pip install randomlib==0.6or we can simply use:pip install randomlibFew Examples:1. Tagger (from basic usage point of view)Stepwise execution:import
from randomlib.mask_fill import MaskPredictorcreate an object
model = MaskPredictor()It provides one functionalitypredict_mask:Predicts the masked tokenExample:pass the string with the word to be predicted replaced with '[MASK]':text = 'मी महाराष्ट्रात [MASK].'English Translation:
'I in Maharashtra [MASK]'model.predict_mask(text)The output will contain some predictions like:मी महाराष्ट्रातआहे.मी महाराष्ट्रातराहणार.मी महाराष्ट्रातनाही.मी महाराष्ट्रातच.मी महाराष्ट्रातराहतो.There are some optional parameters:details(minimum, medium, all) in string - Default: minimumUsed to pass the detailedness to be consideredas_dict(True, False) in boolean - Default: FalseUsed to define the print typeExample:model.predict_mask(text9, 'all', True)Output:
[{'score': 0.46560075879096985, 'token': 1155, 'token_str': 'आहे', 'sequence': 'मी महाराष्ट्रात आहे.'},
{'score': 0.07969045639038086, 'token': 92222, 'token_str': 'राहणार', 'sequence': 'मी महाराष्ट्रात राहणार.'},
{'score': 0.07400081306695938, 'token': 1826, 'token_str': 'नाही', 'sequence': 'मी महाराष्ट्रात नाही.'},
{'score': 0.050422605127096176, 'token': 1617, 'token_str': '##च', 'sequence': 'मी महाराष्ट्रातच.'},
{'score': 0.04373728483915329, 'token': 62560, 'token_str': 'राहतो', 'sequence': 'मी महाराष्ट्रात राहतो.'}]2. Sentiment (from advance usage point of view)Stepwise execution:import
from randomlib.model_repo import SentimentModellist the available modelsmodelSentiment.list_models()Output:sentiment models: MarathiSentiment : randomlib-pune/MarathiSentimenttagger models: marathi-ner : randomlib-pune/marathi-nerautocomplete models: marathi-gpt : randomlib-pune/marathi-gptsimilarity models: marathi-sentence-similarity-sbert : randomlib-pune/marathi-sentence-similarity-sbert
marathi-sentence-bert-nli : randomlib-pune/marathi-sentence-bert-nlimask_fill models: marathi-bert-v2 : randomlib-pune/marathi-bert-v2
marathi-roberta : randomlib-pune/marathi-roberta marathi-albert : randomlib-pune/marathi-alberthate models: mahahate-bert : randomlib-pune/mahahate-bert
mahahate-multi-roberta : randomlib-pune/mahahate-multi-robertaThe library lists down the models available for all the models. These can be changed by the user.To change the default model:Pass the name of the model as the argument:
modelSentiment = SentimentModel('name of model')
Eg.: modelSentiment = SentimentModel('MarathiSentiment')Sentiment provides one functionalityget_polarity_score:Gives the polarity score of words in a sentence along with the tokens (Neutral, Positive, Negative)Example:
text = 'दिवाळीच्या सणादरम्यान सगळे आनंदी असतात.'English Translation:
'Everyone is happy during Diwali festival.'modelSentiment.get_polarity_score(text)Output:
label: Positive
score: 0.995338Entire working of randomlib is explained in thisdemo file. Please have a look at it to get a more better idea!Thank you
Team randomlib