Package: asreview-makita
ASReview MakitaASReviews' Makita (MAKeITAutomatic) is a workflow generator for simulation studies using the command line interface ofASReview
LAB. Makita can be used to effortlessly generate the framework and code for your simulation study.A simulation involves mimicking the screening process for a systematic review of a human in interaction with an Active learning model (i.e., a combination of a feature extractor, classifier,
balancing method, and a query strategy). The simulation reenacts the screening process as if a researcher were using active learning. The performance of one or multiple model(s) can then be
measured by performance metrics, such as the Work Saved over Sampling, recall at a given point in the screening process, or the average time to discover a relevant record.Using Makita templates, different study structures can be generated to fit the needs of your very own study. If your study requires a unique template, you can create a new one and use it
instead.WithASReview LAB, you cansimulatewith theweb
interface, thePython API, or
theCommand Line Interface (CLI). Makita makes use of the CLI.What Makita does:Setting up a workflow for running a large-scale simulation studyPreparing a Github repositoryAutomating the many lines of code neededCreating a batch script for running the simulation study with just one line of codeMaking your research fully reproducibleAllowing you to add templates to accommodate your own specific research questionWhat Makita does not do:Executing jobs or tasks itselfBeing a black-boxWriting your paperFor a tutorial on using Makita we refer to theExercise on Using the ASReview Simulation Mode.InstallationInstall the Makita extension with pip:pipinstallasreview-makitaFor upgrading, use:pipinstall--upgradeasreview-makitaAfter installing the extension, ASReview should automatically detect Makita. If installed correctly,asreview --helpshould list Makita as an option.Getting startedYou can create the framework and code for your own simulation study in 4 steps.Create an project folder on your computer.Create a subfolder nameddataand fill it using one or more datasets.Using your preferred command line tool,cdinto the project folder.Create a simulation study from a template found in thelist of templatesviaasreview makita template NAME_OF_TEMPLATEwhereNAME_OF_TEMPLATEis one of the templates (e.g.asreview makita template arfi).Your simulation study is now properly set up and ready for use. To start the simulations, execute the following shell script in the project folder:sh jobs.shOr on Windows:jobs.batThejobsscript runs all jobs in the project folder.Platform supportBy default, ASReview Makita renders job files for the platform of rendering. It is possible to render jobs for other platforms as well. Use the argument--platformwith values "Windows", "Linux", or "Darwin" (MacOS) to change the output.asreview makita template basic --platform WindowsBy default, the job file depends on the platform. Windows users will see ajobs.batfile, while other users will seejobs.sh. You can overwrite this withasreview makita template basic --job_file my_jobs_file.my_extTemplatesThe following table gives an overview of the available templates.
SeeGetting startedfor instructions on usage.Note: If no seed is set with the template command, the default seed is used.
While this is important for the reproducibility of the results, it may lead to
long-term bias. To avoid seed-related bias across different simulation
studies, a seed should be for the prior records and models.Basic templatecommand:basicThe basic template prepares a script for conducting a simulation study with one run using the default model settings, and two randomly chosen priors (one relevant and one irrelevant record).optional arguments:-h, --help                                show this help message and exit--job_file JOB_FILE, -f JOB_FILE          The name of the file with jobs.                 Default jobs.bat for Windows, otherwise jobs.sh.-s DATA_FOLDER                            Dataset folder-o OUTPUT_FOLDER                          Output folder--init_seed INIT_SEED                     Seed of the priors.                             Seed is set to 535 by default.--model_seed MODEL_SEED                   Seed of the models.                             Seed is set to 165 by default.--template TEMPLATE                       Overwrite template with template file path.--platform PLATFORM                       Platform to run jobs: Windows, Darwin, Linux.   Default: the system of rendering templates.--n_runs N_RUNS                           Number of runs.                                 Default: 1.--no_wordclouds                           Disables the generation of wordclouds.--classifier CLASSIFIER                   Classifier to use.                              Default: nb.--feature_extractor FEATURE_EXTRACTOR     Feature_extractor to use.                       Default: tfidf.--query_strategy QUERY_STRATEGY           Query strategy to use.                          Default: max.--balance_strategy BALANCE_STRATEGY       Balance strategy to use.                        Default: double.--instances_per_query INSTANCES_PER_QUERY Number of instances per query.                  Default: 1.--stop_if STOP_IF                         The number of label actions to simulate.        Default 'min' will stop simulating when all relevant records are found.ARFI templatecommand:arfiThe ARFI template (All relevant, fixed irrelevant) prepares a script for running a simulation study in such a way that for every relevant record 1 run will be executed with 10 randomly chosen irrelevant records which are kept constant over runs. When multiple datasets are available the template orders the tasks in the job file per dataset.optional arguments:-h, --help                                show this help message and exit--job_file JOB_FILE, -f JOB_FILE          The name of the file with jobs.                 Default jobs.bat for Windows, otherwise jobs.sh.-s DATA_FOLDER                            Dataset folder-o OUTPUT_FOLDER                          Output folder--init_seed INIT_SEED                     Seed of the priors.                             Seed is set to 535 by default.--model_seed MODEL_SEED                   Seed of the models.                             Seed is set to 165 by default.--template TEMPLATE                       Overwrite template with template file path.--platform PLATFORM                       Platform to run jobs: Windows, Darwin, Linux.   Default: the system of rendering templates.--n_priors N_PRIORS                       Number of priors.                               Default: 10.--no_wordclouds                           Disables the generation of wordclouds.--classifier CLASSIFIER                   Classifier to use.                              Default: nb.--feature_extractor FEATURE_EXTRACTOR     Feature_extractor to use.                       Default: tfidf.--query_strategy QUERY_STRATEGY           Query strategy to use.                          Default: max.--balance_strategy BALANCE_STRATEGY       Balance strategy to use.                        Default: double.--instances_per_query INSTANCES_PER_QUERY Number of instances per query.                  Default: 1.--stop_if STOP_IF                         The number of label actions to simulate.        Default 'min' will stop simulating when all relevant records are found.Multimodel templatecommand:multimodelThe multiple model template prepares a script for running a simulation study comparing multiple models for one dataset and a fixed set of priors (one relevant and one irrelevant record; identical across models).optional arguments:-h, --help                                show this help message and exit--job_file JOB_FILE, -f JOB_FILE          The name of the file with jobs.                 Default jobs.bat for Windows, otherwise jobs.sh.-s DATA_FOLDER                            Dataset folder-o OUTPUT_FOLDER                          Output folder--init_seed INIT_SEED                     Seed of the priors.                             Seed is set to 535 by default.--model_seed MODEL_SEED                   Seed of the models.                             Seed is set to 165 by default.--template TEMPLATE                       Overwrite template with template file path.--platform PLATFORM                       Platform to run jobs: Windows, Darwin, Linux.   Default: the system of rendering templates.--n_runs N_RUNS                           Number of runs.                                 Default: 1.--no_wordclouds                           Disables the generation of wordclouds.--query_strategy QUERY_STRATEGY           Query strategy to use.                          Default: max.--balance_strategy BALANCE_STRATEGY       Balance strategy to use.                        Default: double.--instances_per_query INSTANCES_PER_QUERY Number of instances per query.                  Default: 1.--stop_if STOP_IF                         The number of label actions to simulate.        Default 'min' will stop simulating when all relevant records are found.--classifiers CLASSIFIERS                 Classifiers to use                              Default: ['logistic', 'nb', 'rf', 'svm']--feature_extractors FEATURE_EXTRACTOR    Feature extractors to use                       Default: ['doc2vec', 'sbert', 'tfidf']--impossible_models IMPOSSIBLE_MODELS     Model combinations to exclude                   Default: ['nb,doc2vec', 'nb,sbert']If you want to specify certain combinations of classifiers and feature
extractors that should and should not be used, you can use the--classifiers,--feature_extractors, and--impossible_modelsoption. For instance, if you
want to exclude the combinations ofnbwithdoc2vecandlogisticwithtfidf, use the following command:asreview makita template multimodel --classifiers logistic nb --feature_extractors tfidf doc2vec --impossible_models nb,doc2vec logistic,tfidfAdvanced usageCreate and use custom templatesIt is possible to overwrite the internal templates. This can be useful for simulation studies with different needs.Select an existing template that looks similar to your needs. For example, you want to run ARFI with a different model, then you pick theARFI template.Download the template you selected in step 1 from theGithub repository. Template files have the following
structuretemplate_*.txt.template. For the ARFI example, the template istemplate_arfi.txt.template.Save the downloaded template somewhere on your computer. The template is a so-called "Jinja" template. The template consists ofASReview command line
commandscombined with jinja syntax. The Jinja syntax is very intuitive. See thisCheatsheet.Edit the Jinja template to your needs.Run the custom template with the command line option--template PATH_TO_MY_TEMPLATE.txt.template. For the ARFI example, this would beasreview makita template arfi --template PATH_TO_MY_TEMPLATE.txt.template. Please keep in mind that you follow the usual steps for running a template.Ajobs.shfile should be in the your folder.Please contribute your templates back to the project by making a Pull Request. Then, we can integrate it in the core of the makita package.Add and use scriptsMakita can add scripts to your repository. The scripts are mainly pre- and postprocessing scripts. These scripts are not (yet) available in any existing ASReview software. Therefore, they can be added manually
withasreview makita add-script NAME_OF_SCRIPT.For example, the results fromASReview datatoolsare merged via the scriptmerge_descriptives.py(ormerge_metrics.pyforASReview insights), using:Collect statistics (with template)Runasreview makita add-script merge_descriptives.pyRunpython scripts/merge_descriptives.pyUse-s(source) and-o(output) to tweak paths.Some scripts are added automatically to the folder, as they are part of the
template. For example, theget_plot.pyscript is added to the generated folder
when using any template, as it is used to generate the plots.Still,get_plot.pycan be used on its own, as it is a standalone script. To use it,
use-s(source) and-o(output) to tweak paths.Adding a legend to the plot can be done with the-lor--show_legendflag,
with the labels clustered on any of the following:'filename', 'model', 'query_strategy', 'balance_strategy', 'feature_extraction', 'n_instances', 'stop_if', 'n_prior_included', 'n_prior_excluded', 'model_param', 'query_param', 'feature_param', 'balance_param'Available scriptsThe following scripts are available:get_plot.pyget_settings_from_state.pymerge_descriptives.pymerge_metrics.pymerge_tds.pysplit_data_with_multiple_labels.py [DEPRECATED]Time to Discovery TablesThe 'merge_tds.py' script creates a table of the time to discovery (TD) values for each dataset, with each row corresponding to each record ID of the relevant records in a dataset, and the columns correspond to each simulation run (e.g, for the Multimodel template each column corresponds to a simualtion run with each active learning model). Additionally, the tables includes the average-record-TD values (the average of the TD values for a record across multiple simulation runs), and the average-simulation-TD values (the average of the TD values across all records for a single simulation run).Run Makita via DockerTo run Makita template with Docker use the following command:dockerrun-v$PWD:/appghcr.io/asreview/asreviewmakita<YOURCOMMAND>LicenseThis extension is published under theMIT license.ContactThis extension is part of the ASReview project (asreview.ai). It is maintained by the maintainers of ASReview LAB. SeeASReview LABfor
contact information and more resources.