Package: databalancer
DatabalancerDatabalancer is the python library using in machine learning applications to balance the imbalanced text classification datasets before the model training.FeaturesDatabalancer is able to balance any imbalanced text classification datasetsIf the given dataset is imbalanced then while balancing no existing data is removed, but new data will be generated and added to the datasetFor a particular class the newly generated data will be the paraphrases of the existing data in that particular classBy default, these paraphrases are generated using theramsrigouthamg/t5_paraphrasermodel (You can read more about the model fromHuggingface official documentation)The current version can generate the sentence paraphrases using multiple methods such as T5 models, NLPAUG and TextattackThe user can select the balance method by passing thebalance_methodparameter while calling thebalanceDatasetmethod such asbalance_method=1forramsrigouthamg/t5_paraphraserT5 model based balancing (Default) ( For more info checkt5_paraphraser)balance_method=2forramsrigouthamg/t5-large-paraphraser-diverse-high-qualityT5 model based balancing (For more info checkt5-large-paraphraser-diverse-high-quality)balance_method=3fornlpaugbased balancing (For more info checknlpaug)balance_method=4fortextattackbased balancing (For more info checktextattack)Themodelargument in thebalanceDatasetmethod is only applicable whenbalance_methodis set as3, through which user can pass the transformer model name fromHuggingfaceto generate paraphrases using NLPAUG .If the user enablequantize=TrueinbalanceDatasetthen the T5 models(balance_method==1andbalance_method=2) will go through the quantization process usingfastT5before inference, so that the model inference time will be reduced.By defaultquantizeparameter is set asFalsebecause quantization requires more RAM and more CPU Processing powerDatabalancer also provides another method calledclassCountVisualizationto show the dataset class count distributionInstallationInstall thedatabalancerpackage withpippip install databalancerCompatibilityDatabalancer is only compatable with python 3.6.9 or above.Quick StartThe library databalancer provides two different functionalities.1 - classCountVisualization2 - balanceDatasetclassCountVisualization#Import the classCountVisualization from the 'databalancer' modulefromdatabalancerimportclassCountVisualization#Pass the required datasetname(here traindata.csv) to the functionclassCountVisualization("traindata.csv")OutputbalanceDataset#Import the balanceDataset from the 'databalancer' modulefromdatabalancerimportbalanceDataset#Pass the dataset name which is to be balanced(here traindata.csv) to the balanceDataset functionbalanceDataset("traindata.csv",balance_method=1)The above code will balance the dataset and store the balanced dataset('balanced_data.csv') in the local machine.balanceDataset with model quantization#Import the balanceDataset from the 'databalancer' modulefromdatabalancerimportbalanceDataset#Pass the dataset name which is to be balanced(here traindata.csv) to the balanceDataset function with balance_method=2 and enable quantizationbalanceDataset("traindata.csv",balance_method=2,quantize=True)The above code will balance the dataset using balance_method=2 with quantization and store the balanced dataset('balanced_data.csv') in the local machine.To show the balanced dataset class count distribution, run the code below.fromdatabalancerimportclassCountVisualizationclassCountVisualization("balanced_data.csv")