Package: automs
CIAMS - Clustering Indices based Automatic classification Model SelectionThe code for CIAMS is packaged under the nameAutoMS.AutoMS (Automatic Model Selection Using Cluster Indices)is amachine learning model recommendationanddataset classifiability assessmenttoolkit.Find the documentationhere.Table of ContentsOverviewInstalling AutoMSConfiguring AutoMSRunning AutoMS on a datasetStep 1: Downloading the datasetStep 2: Creating the dataset configuration fileStep 3: Predicting Classification Complexity and Estimating F1 scores for the datasetCommand-line InterfacePython InterfaceDocumentationAuthorsAcknowledgmentsOverviewAutoMS estimates themaximum achievable f1 scorescorresponding to various classifier models for a givenbinary classification dataset. These estimated scores help make informed choices about the classifier models to experiment on the dataset, and also speculate what to expect from each of them. AutoMS also predicts theclassification complexityof the dataset which characterizes the ease with which the dataset can be classified.AutoMS extractsclustering-based metafeaturesfrom the dataset and uses fitted classification and regression models to predict the classification complexity and estimate the maximum achievable f1-scores corresponding to various classifier models for the dataset.Note:f1-scorein all discussions pertaining to AutoMS refers to a variant ofweighted average f1-scorefor binary datasets fromclass imbalance learningliterature that weights the f1-scores of classes inversely proportional to their proportions in the dataset.where,Ris the class imbalance ratio, which is the fraction of number of samples in the majority class to the number of samples in the minority class.Installing AutoMSWe recommend installingautomsinto avirtual environment.$sudopipinstallvirtualenv$virtualenv--python=python3.6automs-venv$sourceautoms-venv/bin/activate$pipinstallautomsTip:If youencounter errorsin installing AutoMS,installpython3.6-devsystem package (which contains the header files and static library for Python) and, then attempt installingautomsagain.$sudoapt-getinstallpython3.6-dev$pipinstallautomsConfiguring AutoMSThedefault configurationswith which to runautomscan be configured using theAutoMS Configuration Wizardwith:$automs-configThe configured defaults can be overriden for each invocation ofautomsby suppling appropriate arguments to the command-line or python interface.Running AutoMS on a datasetStep 1: Downloading the datasetDownload abinary classification datasetof choice (in csv, libsvm or arff format) from the web. In this illustration, we will be using theConnectionist Bench (Sonar, Mines vs. Rocks) Data Set. Download the dataset in csv format fromherewith:$wgethttps://archive.ics.uci.edu/ml/machine-learning-databases/undocumented/connectionist-bench/sonar/sonar.all-dataChange the current working directory to the directory into which the dataset was downloaded.Rename the dataset file to have a '.csv' extension.$mvsonar.all-datasonar.csvNote:AutoMS infers the data format of a dataset file from its filename extension. Therefore, you must rename the dataset file to have a filename extension that corresponds to its data format. Supported filename extensions (and data formats) are '.csv', '.libsvm' and '.arff'.Step 2: Creating the dataset configuration fileThe configuration file for the dataset encodes information about the structure of the dataset file.Create a dataset configuration filefor the datasetin the same directory as the dataset file, withfilename same as the dataset filename suffixed with a '.config.py' extension(i.e., in this casesonar.csv.config.py).$echo-e"from automs.config import CsvConfig\nconfig = CsvConfig()">sonar.csv.config.py$catsonar.csv.config.pyFor examples of theconfiguration file contentcorresponding to variety of dataset files, refer to theexamplessection in documentation.Note:For the dataset filesonar.csv, the contents of the dataset configuration filesonar.csv.config.pyis:fromautoms.configimportCsvConfigconfig=CsvConfig()Since, the dataset file in this case is aligned with the default values of the arguments toCsvConfigclass, no arguments have beenexplicitlypassed toCsvConfigclass in the creation of theconfigobject. However, you may need to override some of the default values of the arguments to your data format specific dataset configuration class in the creation of theconfigobject, to suit to your dataset file.For information about the dataset configuration classes corresponding to the various data formats and the arguments they accept, refer toAPI documentation of Dataset Configuration Classes.Step 3: Predicting Classification Complexity and Estimating F1 scores for the datasetCommand-line Interface$automssonar.csv--oneshot--truef1--resultsonar_resultsFor the more information about theoneshotandsubsamplingapproaches, refers toWhat are the oneshot and sub-sampling appeoaches ?andWhen should I use the oneshot and sub-sampling approaches ?in theFAQsection in documentation.The predicted classification complexity, estimated f1-score and true f1-score results for the dataset should be available in thesonar_resultsfile after the completion of execution of the program.$catsonar_resultsNote:The predictedclassification complexitybooleanvalue indicates if the dataset can be classified with af1-score > 0.6using any of the classification methods.Trueindicates that the dataset ishard to to classifyandFalseindicates that the dataset iseasy to classify.Theestimated f1-scorescorresponding to various classifier models should help identify thecandidate top performing classification methodsfor the dataset, and help reduce the search space of classification algorithms to be experimented on the dataset.For more information about the AutoMS command line interface and the arguments it accepts, refer toAPI Documentation for AutoMS command line interface.$automs--helpPython Interface>>>fromautoms.automsimportautoms>>>is_hard_to_classify,estimated_f1_scores,true_f1_scores=automs('sonar.csv',oneshot=True,return_true_f1s=True)>>>print(f"IS HARD TO CLASSIFY ={is_hard_to_classify}")>>>print(f"Estimated F1-scores ={estimated_f1_scores}")>>>print(f"True F1-scores ={true_f1_scores}")For more information about the AutoMS python interface and the arguments it accepts, refer toAPI Documentation for AutoMS python interface.>>>fromautoms.automsimportautoms>>>help(automs)Tip:Inspect the configured (or specified)warehouse sub-directorycorresponding to thelast run of AutoMSfor result filesresults.xlsx,predicted_classification_complexity,estimated_f1_scoresandtrue_f1_scores, and the intermediate data subsample files in itsbags/sub-directory.$ls<PathtoconfiguredAutoMSwarehouse>$cd<PathtoconfiguredAutoMSwarehouse>/sonar.csv/$tail-n+1predicted_classification_complexityestimated_f1_scorestrue_f1_scores$xdg-openresults.xlsxDocumentationThe AutoMS documentation is hosted athttps://automs.readthedocs.io/.AuthorsSudarsun Santhiappan, IIT Madras & BUDDI.AINitin Shravan, BUDDI.AIAcknowledgmentsMukesh Reghu, BUDDI.AIJeshuren Chelladurai, IIT Madras & BUDDI.AI