Package: gensum
gensum - Generative Summarization for Data AugmentationIntroductionImbalanced class distribution remains a classic common problem in ML. Undersampling combined with oversampling are two methods of attempting to address this issue.
Techniques such as SMOTE and MLSMOTE have been proposed, but the high dimensional nature of numerical vectors created from text makes other data augmentation approaches preferable.gensum is an NLP library based onabsumthat uses generative summarization to perform data augmentation by oversampling under-represented classes in text classification datasets. Recent advancements in generative models such as ChatGPT make this approach optimal in achieving realisticbut uniquedata for the augmentation process.It usesChatGPTby default, but is designed in a modular way to allow you to use any large language models capable of generative summarization.gensumis format agnostic, expecting only a DataFrame containing a text and classifier column.InstallationVia pippipinstallgensumFrom sourcegitclonehttps://github.com/aaronbriel/gensum.git
pipinstall[--editable].orpipinstallgit+https://github.com/aaronbriel/gensum.gitUsagegensum expects a DataFrame containing a text column which defaults to 'text', and another classifier column which defaults to 'classifier'. All available parameters are detailed in the Parameters section below. Be sure to set theOPENAI_API_KEYenvironmental parameter prior to running the code.importpandasaspd
fromgensumimportAugmentorcsv='path_to_csv'df=pd.read_csv(csv)augmentor=Augmentor(df,text_column='text',classifier='intent')df_augmented=augmentor.gen_sum_augment()# Store resulting dataframe as a csvdf_augmented.to_csv(csv.replace('.csv','-augmented.csv'),encoding='utf-8',index=False)NOTE: The output dataframe contains only the augmented rows.ParametersNameTypeDescriptiondf(:class:pandas.Dataframe,required, defaults to None)Dataframe containing text and one-hot encoded features.text_column(:obj:string,optional, defaults to "text")Column in df containing text.classifier(:obj:string,optional, defaults to "classifier")Classifier to augment data for.classifier_values(:obj:string,optional, defaults to None)Specific classifier values to augment data for.min_length(:obj:int,optional, defaults to 10)The min length of the sequence to be generated. Between 0 and infinity. Default to 10.max_length(:obj:int,optional, defaults to 50)The max length of the sequence to be generated. Between min_length and infinity. Default to 50.num_samples(:obj:int,optional, defaults to 20)Number of samples to pull from dataframe with specific feature to use in generating new sample with Generative Summarization.threshold(:obj:int,optional, defaults to mean count for all classifier values)Maximum ceiling for each feature, normally the under-sample max.prompt(:obj:string,optional, defaults to "Create SUMMARY_COUNT unique, informally written sentences similar to the ones listed here:")The prompt to use for the generative summarization. If you change the prompt, please be sure to keep the SUMMARY_COUNT string in it somewhere as this is expected and replaced based on the append count calculated for said classifier value.llm(:obj:string,optional, defaults to 'chatgpt')The generative LLM to use for summarization.model(:obj:string,optional, defaults to 'gpt-3.5-turbo')The specific model to use.temperature(:obj:int,optional, defaults to 0)Determines the randomness of the generated sequences. Between 0 and 1, where a higher value means the generated sequences will be more random.debug(:obj:bool,optional, defaults to True)If set, prints generated summarizations.CitationPlease referencethis libraryif you use this work in a published or open-source project.