Package: rarity
Rarityis a diagnostic library for tabular data with minimal setup to enable deep dive into datasets identifying features
that could have potentially influenced the model prediction performance. It is meant to be used at post model training phase to ease the understanding on
miss predictions and carry out systematic analysis to identify the gap between actual values versus prediction values. The auto-generated gap analysis
is presented as a dash web application with flexible parameters at several feature components.The inputs needed to auto-generate gap analysis report withRarityare solely depending on features, yTrue and yPred values. Rarity is therefore a model anogstic package and can be used to inspect miss predictions on tabular data generated by any model framework.Supported Analysis TypeRarity currently supports tasks related toRegressionBinary ClassifciationMulticlass ClassificationIt can also be used to conductbimodal analysis. As it is used to inspect miss predictions in details down to the granularity at each data index level, multiple modal analysis won't be ideal for repetition at individual data index for each model. Therefore, the package supports upto 2 model miss prediction gap analysis for side-by-side comparison benefiting more during the post model training and final phase of model fine-tuning stage.Core Feature ComponentsThere are five core feature components covered in the auto-generated gap analysis report byRarity:General Metrics: covers general metrics used to evaluate model performance.Miss Predictions: presents miss predictions scatter plot by index numberLoss Clusters: covers clustering info on offset values (regression) and logloss values (classification)xFeature Distribution: distribution plots ranked by kl-divergence scoreSimilarities: tabulated info listing top-n data points based on similarities in features with reference to data index specified by userCounter-Factualsis also included underSimilaritiescomponent tab for classification task to better compare data points with most similar features but show different prediction outcomes. For futher details on how the feature components are displayed in the web application, please checkout more examples under sectionFeature Introductionin the package documentation.InstallationRarityhas been tested onPython 3.6+. It is advisable to create aVirtual Environmentto use along withRarity.For details guide on how to create virtual environment, you can refer to thisuser guide.
After creation of the virtual environment, activate it and proceed with one of the following steps.From PyPIpip install rarityFrom Sourcegit clone https://github.com/aimakerspace/rarity.git
cd rarity
pip install -e .Quick StartRarityis developed with minimum setup in mind in order for users to focus more on actual modelling works and decide appropriate fine-tuning strategy after
quick overview on the state of miss-prediction at post model training phase. Users can compile the required inputs and prediction outcomes via one of the following mentioned methods and spin up the dash web application to see the final gap analysis report produced byRarity.Simplest mode with mininum codeAfter cloningRarityrepository, placexFeatures,yTrueandyPred filesintoconfigs/csv_datafolder as illustrated in example belowrarity
    ├── configs
    │   ├── csv_data
    │   │   ├── <example_user_xFeatures>.csv
    │   │   ├── <example_user_yTrue>.csv
    │   │   ├── <example_user_yPred_model_x>.csv
    │   │   └── <example_user_yPred_model_y>.csv
    │   │
    │   └── configs.py
    ...
    │ 
    ├── auto_gap_analysis.pyThen openconfigs.pyfile and update the first section to define the required meta data.# *******************************************************************************************# To be updated by user accordingly if to run this script using saved csv files# Replace 'example_xxx.csv' with user's file namexFeature_filename='example_user_xFeatures.csv'yTrue_filename='example_user_yTrue.csv'# yPred_file can be single file or max 2 files, wrap in a listyPred_filename=['example_user_yPred_model_x.csv','example_user_yPred_model_y.csv']# model name must be listed according to the same sequence of the yPred_filename list above# can be single model or max bi-modal, wrap in a listMODEL_NAME_LIST=['example_model_x','example_model_y']# Supported analysis type: 'Regression', 'Binary Classification', 'Multiclass Classification'ANALYSIS_TYPE='Regression'ANALYSIS_TITLE='example_Customer Churn Prediction'# Defaults to 8000, user can re-define to a new port number of choicePORT=8000# *******************************************************************************************After uploading files toconfigs/csv_datafolder and updatingconfigs/configs.pyfile, open terminal and make sure you are in theRarityproject root folder. A file namedauto_gap_analysis.pyis already in the root folder upon installation. Then run the following line of code in the terminalpython auto_gap_analysis.pyA window will be open in the web browser and you will see the gap analysis report is generated for you byRarity. Below is an example generated for Bimodal analysis on Regression task.Using CSVDataLoaderAfter installation, open terminal and run the following codes :fromrarityimportGapAnalyzerfromrarity.data_loaderimportCSVDataLoader# define the file pathsxFeatures_file='example_xFeatures.csv'yTrue_file='example_yTrue.csv'yPred_file_list=['example_yPreds_model_xx.csv','example_yPreds_rf.csv']model_names_list=['model_xx','model_yy']# specify which port to use, if not provided, default port is set to 8000preferred_port=8866# collate all files using dataloader to transform them into the input format# that can be processed by various internal function calls# example : '<analysis_type>' => 'Regression'# example : '<analysis_title>' => 'Customer Churn Prediction'data_loader=CSVDataLoader(xFeatures_file,yTrue_file,yPred_file_list,model_names_list,'<analysis_type>')analyzer=GapAnalyzer(data_loader,'<analysis_title>',preferred_port)analyzer.run()with additional adjustments as follows :! replacement of example files to your own file names! define `model_names_list`, `analysis_type` and `analysis_title` accordinglyUsing DataframeLoaderTo useDataframeLoader, it is assumed that you already have some inital dataframes tap-out in earlier runs in the terminal and would like to continue analysing the miss-predictions after model training. TheDataframeLoaderapi call is meant for inline analysis if you prefer not to collate base info using csv files. You may collate all thexFeatures,yTrueandyPredsdataframes into the right input format usingDataframeLoaderas demonstrated below :fromrarityimportGapAnalyzerfromrarity.data_loaderimportDataframeLoader# define the file pathsxFeatures_df=<xfeatures_stored_in_pd.DataFrame>yTrue_df=<yTrue_stored_in_pd.DataFrame/Series>yPred_df_model_xx=<yPred_generated_by_model_xx_stored_in_pd.DataFrame>yPred_df_model_yy=<yPred_generated_by_model_yy_stored_in_pd.DataFrame>yPred_list=[yPred_df_model_xx,yPred_df_model_yy]model_names_list=['model_xx','model_yy']# specify which port to use, if not provided, default port is set to 8000preferred_port=8866# collate all files using dataloader to transform them into the input format# that can be processed by various internal function calls# example : '<analysis_type>' => 'Regression'# example : '<analysis_title>' => 'Customer Churn Prediction'data_loader=DataframeLoader(xFeatures_df,yTrue_df,yPred_list,model_names_list,'<analysis_type>')analyzer=GapAnalyzer(data_loader,'<analysis_title>',preferred_port)analyzer.run()SupportsPackage documentation : Full version of the package documentation can be foundHERECommunity : For interest on AI Singapore program, kindly head to our AISG community channelsHEREAcknowledgementThe developer team would like to express our gratitude to the program office for the support given to the development and release of this data dianogstic library#. Special thanks toAI Singaporemaking the development works of the package feasible.Program OfficeThis project is supported by theNational Research Foundation (NRF), Singaporeunder itsAI Singapore Programme(AISG-RP-2019-050).DevelopersYap Siew Lin (Core developer), Jeanne Choo (ex-AISG), Chong Wei Yih (ex-AISG)# Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.