Package: reseval
Reproducible Subjective Evaluation (ReSEval)ReSEval is a framework for quickly building subjective evaluation and
annotation tasks that are deployed on crowdworker platforms likeAmazon Mechanical Turk. ReSEval currently supports
A/B, ABX, MOS, MUSHRA, and Word Selection tests on audio, image, text, and video data.While our code is free to use, performing crowdsourced subjective
    evaluation is not free.We are not responsible for costs incurred
    while using our code.CitationIf you use ReSEval in an academic publication, please citeour paper.IEEEM. Morrison, B. Tang, G. Tan, and B. Pardo, "Reproducible Subjective Evaluation," ICLR Workshop on ML Evaluation Standards, April 2022.BibTex@inproceedings{morrison2022reproducible,
    title={Reproducible Subjective Evaluation},
    author={Morrison, Max and Tang, Brian and Tan, Gefei and Pardo, Bryan},
    booktitle={ICLR Workshop on ML Evaluation Standards},
    month={April},
    year={2022}
}Table of contentsInstallationDeploying locallyConfigurationAdding filesABABXMOSMUSHRAWordSelectCredentialsHerokuAmazon Web ServicesAmazon Mechanical TurkUsageCommand-line interfaceCreateMonitorResultsPayDestroyExtendApplication programming interfaceAdvanced usageCLIAPIAdditional monitoringInstallationFirst, install the Python module. ReSEval requires Python 3.9 or higher.pip install resevalNext,download Node.js. You can check that your installation is correct by runningnode --version. ReSEval uses Node.js version 18.16.1 and is not guaranteed to work on all versions. If needed, Linux and OS X users can usento change their version of Node.js, and Windows users can useNVM for Windows.# Linux or OS X
sudo npm install -g n
sudo n 18.16.1

# Windows
# Must be run with administrator privileges
nvm install 18.16.1
nvm use 18.16.1Note- You must restart your terminal after changing versions of node for the change to take effectDeploying locallyTo be able to preview your subjective evaluation locally, you mustsetup a local MySQL database serverand create a user with database creation privileges.# Linux installationsudoaptinstallmysql-server# MySQL setupsudomysql_secure_installation# Login to MySQL as rootsudomysql-uroot-- Create a user (change new_user and new_password)mysql>CREATEUSER'new_user'@'localhost'IDENTIFIEDBY'new_password';-- Give user database creation privilegesmysql>GRANTALLPRIVILEGESON*.*TO'new_user'@'localhost';Run the following to store the username and password inreseval.CACHE / '.env'.python-mreseval.credentials\--mysql_local_user<mysql_user>\--mysql_local_password<mysql_local_password>The.envfile is used to set local environment variables and is not pushed to
GitHub or uploaded to any remote storage.ConfigurationAll configuration is performed in a YAML configuration file. Seeexamples/*.yamlfor examples and documentation of parameters.Adding filesThe files to be evaluated must be organized in a directory structure according
to the type of test being run. The directory structures for each test are as
follows. Examples of valid directories of evaluation files can be found inexamples/.ABab
├── <condition-1>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-2>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...ABXabx
└── reference
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-1>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-2>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...MOSmos
├── <condition-0>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-1>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-2>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-3>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
└── ...MUSHRAmushra
├── <condition-0>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-1>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-2>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
├── <condition-3>
│   ├── <file-0>
│   ├── <file-1>
│   ├── <file-2>
│   ├── ...
└── ...WordSelectwordselect
├── <file-0>
├── <words-0>
├── <file-1>
├── <words-1>
├── <file-2>
├── <words-2>
├── ...<words-x>is<file-x>with-words.txtextension.CredentialsAPI keys are required to use the third-party services that ReSEval depends on.
These are not required for local development. Do not share these API keys.Amazon Web ServicesSign up for an AWS account. Go toSecurity Credentials. UnderAccess keys, clickCreate New Access Key.Note that this gives you a root access key. You can alternatively use the Identity & Access Management (IAM) system to setup more restrictive permissions for a user.If you have never used AWS Elastic Beanstalk, one more step is required. Elastic Beanstalk instances created from Python (as opposed to the AWS console) do not instantiate the default instance profile IAM role. The solution is to either create (and then delete) an Elastic Beanstalk instance from the console, or to create the required IAM role manually. The IAM role is calledaws-elasticbeanstalk-ec2-roleand contains three permissions: (1)AWSElasticBeanstalkWebTier, (2)AWSElasticBeanstalkMulticontainerDocker, and (3)AWSElasticBeanstalkWorkerTier.Amazon Mechanical TurkFollow the instructionsherefor setting up MTurk and connecting it to your AWS account.Heroku (Optional if you don't want to use AWS)Sign up for a Heroku account. Go toAccount Settings. At the bottom of the page in theAPI Keysection is aRevealbutton.You will also need to enable billing. You can do sohere.UsageOnce you have your configuration file and a properly formatted directory of evaluation files, you are ready to deploy a subjective evaluation. Example configuration files and corresponding evaluation files can be found inexamples/.If you are not deploying locally, add your API keys.# AWS credentialspython-mreseval.credentials\--aws_api_key<aws_api_key>\--aws_api_secret_key<aws_api_secret_key># (Optional) Heroku credentialspython-mreseval.credentials\--heroku_api_key<heroku_api_key>API keys are saved inreseval.CACHE / '.keys'. The.keysfile is used to set local environment variables and is not pushed to GitHub or uploaded to any remote storage.Command-line interfaceArguments for the following command-line interfaces are as follows, unless otherwise specified.<config>- The configuration file<directory>- The directory of evaluation files<name>- The name of the evaluation given in the configuration fileCreateCreate a subjective evaluation either locally, in remote development mode (e.g., MTurk Sandbox), or in production mode.Note-reseval.createis not currently thread-safe. Wait until the first call has finished before calling it again. Seethis GitHub issue.# Local developmentpython-mreseval.create<config><directory>--local# Remote developmentpython-mreseval.create<config><directory># Productionpython-mreseval.create<config><directory>--productionMonitor# Monitor all subjective evaluationspython-mreseval.monitor# Monitor one subjective evaluation# The name of the evaluation can be found in its configuration filepython-mreseval.monitor--name<name>Note- By default, the monitor updates once every minute. You can update the monitor more or less often by providing an update interval in seconds.# Update the monitor once every ten secondspython-mreseval.monitor--interval10Results# Get the results of a subjective evaluation.# Results are stored in <directory>/<name>.# <directory> defaults to the current directory.python-mreseval.results<name>--directory<directory>Pay# Pay participantspython-mreseval.pay<name>Destroy# Destroy the compute resources of a subjective evaluation (e.g., any cloud# storage, databases, or servers)python-mreseval.destroy<name># Destroy a subjective evaluation even if it is still active.# Participants who have taken the test so far will be paid.python-mreseval.destroy<name>--forceExtend# Add <participants> additional participants to a finished evaluationpython-mreseval.extend<name><participants>Application programming interfaceDocumentation for our API can be foundhere.Advanced usageOnce you feel comfortable with using ReSEval step-by-step from the
command-line and after you have added your credentials withreseval.credentials, you can use the CLI or API to run your evaluation with
only a single command.CLI# Local developmentpython-mreseval<config><directory>--local# Remote developmentpython-mreseval<config><directory># Productionpython-mreseval<config><directory>--productionAPIimportreseval# Local developmentreseval.run(config,directory,local=True)# Remote developmentreseval.run(config,directory)# Productionreseval.run(config,directory,production=True)Additional monitoringAWS S3To monitor, edit, or delete AWS S3 storage buckets, or see storage costs, use theAWS S3 console.AWS Elastic BeanstalkTo monitor, edit, or delete the server compute, use theAWS Elastic Beanstalk console.AWS Relational Database ServiceTo monitor, edit, or delete the database, use theAWS RDS console.MTurkHITs not created on the MTurk dashboard are not visible on the MTurk dashboard.
You can use the MTurk CLI to monitor, edit, or delete HITs. MTurk costs appear
on theAWS billing dashboardat the end of the billing period.HerokuTo monitor, edit, or delete Heroku databases and servers, use theHeroku application dashboard. You can see
any costs on thebilling dashboard.