Package: textgenie
TextGenieTextGenie is a text data augmentations library that helps you augment your text dataset and generate similar kind of samples, thus generating a more robust dataset to train better models. It also takes care of labeled datasets while generating similar samples keeping their labels in memory.It uses various Natural Language Processing methods such as paraphrase generation, BERT mask filling and converting text to active voice if found in passive voices. This library currently supportsEnglishLanguage.Installationpip install textgenieExamplefromtextgenieimportTextGenietextgenie=TextGenie("hetpandya/t5-small-tapaco","bert-base-uncased")# Augment a list of sentencessentences=["The video was posted on Facebook by Alex.","I plan to run it again this time",]textgenie.magic_lamp(sentences,"paraphrase: ",n_mask_predictions=5,convert_to_active=True)# Augment data in a txt filetextgenie.magic_lamp("sentences.txt","paraphrase: ",n_mask_predictions=5,convert_to_active=True)# Augment data in a csv file with labelstextgenie.magic_lamp("sentences.csv","paraphrase: ",n_mask_predictions=5,convert_to_active=True,label_column="Label",data_column="Text",column_names=["Text","Label"],)Examples can be found in the examplesnotebook.UsageInitializing the augmentor:textgenie = TextGenie(paraphrase_model_name='model_name',mask_model_name='model_name',spacy_model_name="model_name",device="cpu")Parameters:paraphrase_model_name:The name of the T5 paraphrase model.A list of pretrained model for paraphrase generation can be foundheremask_model_name:BERT model that will be used to fill masks. This model is disabled by default. But can be enabled by mentioning the name of the BERT model to be used. A list of mask filling models can be foundherespacy_model_name:Name of the Spacy model. Available models can be foundhere. The default value is set toen_core_web_sm.device:The device where the model will be loaded. The default value is set tocpu.Methods:augment_sent_mask_filling():Generate augmented data using BERT mask filling.Parameters:sent:The sentence on which augmentation has to be applied.n_mask_predictions:The number of predictions, the BERT mask filling model should generate. The default value is set to5.augment_sent_t5():Generate augmented data using T5 paraphrasing model.Parameters:sent:The sentence on which augmentation has to be applied.prefix:The prefix for the T5 model input.n_predictions:The number of number augmentations, the function should return. The default value is set to5.top_k:The number of predictions, the T5 model should generate. The default value is set to120.max_length:The max length of the sentence to feed to the model. The default value is set to256.convert_to_active():Converts a sentence to active voice, if found in passive voice. Otherwise returns the same sentence.Parameters:sent:The sentence that has to be converted.magic_once():This is a wrapper method foraugment_sent_mask_filling(),augment_sent_t5()andconvert_to_active()methods. Using this, a sentence can be augmented using all the above mentioned techniques.Since this method can operate on individual text data, it can be merged with other packages.Parameters:sent:The sentence that has to be augmented.paraphrase_prefix:The prefix for the T5 model input.n_paraphrase_predictions:The number of number augmentations, the function should return. The default value is set to5.paraphrase_top_k:The number of predictions, the T5 model should generate. The default value is set to120.paraphrase_max_length:The max length of the sentence to feed to the model. The default value is set to256.n_mask_predictions:The number of predictions, the BERT mask filling model should generate. The default value is set toNone.convert_to_active:If the sentence should be converted to active voice. The default value is set toTrue.magic_lamp():This method can be used for augmenting whole dataset. Currently accepted dataset formats are:txt,csv,tsvandlist.If the dataset is inlistortxtformat, a list of augmented sentences will be returned. Also, atxtfile with the namesentences_aug.txtis saved containing the output of the augmented data.If a dataset is incsvortsvformat with labels, the dataset will be augmented along with keeping in memory the labels for the new samples and a pandas dataframe of the augmented data will be returned. Atsvfile will be generated with the augmented output with nameoriginal_file_name_aug.tsvParameters:sentences:The dataset that has to be augmented. This can be aPython List, atxt,csvortsvfile.paraphrase_prefix:The prefix for the T5 model input.n_paraphrase_predictions:The number of number augmentations, the function should return. The default value is set to5.paraphrase_top_k:The number of predictions, the T5 model should generate. The default value is set to120.paraphrase_max_length:The max length of the sentence to feed to the model. The default value is set to256.n_mask_predictions:The number of predictions, the BERT mask filling model should generate. The default value is set toNone.convert_to_active:If the sentence should be converted to active voice. The default value is set toTrue.label_column:The name of the column that contains labeled data. The default value is set toNone. This parameter is not required to be set if the dataset is in aPython Listor atxtfile.data_column:The name of the column that contains data. The default value is set toNone. This parameter too is not required if the dataset is aPython Listor atxtfile.column_names:If thecsvortsvdoes not have column names, a Python list has to be passed to give the columns a name. Since this function also acceptsPython Listand atxtfile, the default value is set toNone. But, ifcsvortsvfiles are used, this parameter has to be set.ReferencesPassive To Activelicensed under the Apache License 2.0LinksPlease find an in depth explanation about the libraryon my blog.LicensePlease checkLICENSEfor more details.