Package: amplify-bench
Fixstars Amplify BenchmarkWhat is Amplify Benchmark?Fixstars Amplify Benchmark is a framework for benchmarking the performances of solvers for quadratic unconstrained binary optimization problems (QUBO). It provides a command line interface to perform benchmarking and a definition of the benchmark problem.TheFixstars Amplify SDKis used as the backend, allowing benchmarks to be run with many solvers such as quantum annealing machines, Ising machines, and mathematical optimization solvers. Benchmarks are run based on a job set file that defines the target problems, the number of runs, and the solvers to be used, making it easy to automate the process from benchmark execution to analyzing results.The results of this library run can be loaded into theAmplify Benchmark Viewerto visualize the results in a web browser. A demo of the benchmark results for Amplify AE ishere.FeaturesEasy to runParallel executionAutomatic evaluation and analysisBenchmark result viewer is providedCustomizable solver and problem parametersFormulations for several benchmark sets are pre-definedUser-define problems can be addedPre-defined benchmark sets:Traveling Salesperson Problem:TSPLIBQuadratic Assignment Problem:QAPLIBMax-CUT Problem:GsetCapacitated Vehicle Routing Problem:CVRPLIBQuadratic Problem:QPLIBSudoku (logic-based combinatorial number-placement puzzle)Supported solvers powered by Amplify SDK:Fixstars Amplify AED-Wave AdvantageFujitsu Digital Annealer 3/4Toshiba SQBM+Gurobi Optimizer(Seesupported solvers of Amplify SDK)GalleryObjective value for execution timeTime To Solution (TTS)Probability of obtaining a feasible solutionProbability of obtaining the best solution RateTable DATAGetting startedInstallationAmplify benchmark is provided as a Python (>=3.8) library. It can be installed with pip as follows:$pipinstallamplify-benchRun example job setAfter installation, theamplify-benchcommand is enabled.$amplify-bench--help
Usage:amplify-bench[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:downloadDownloadallsupportedinstancefilesinthespecified...runQUBOBenchmarkstatsGenerateQUBObenchmarkstatsdata.To run a benchmark, you have to create a benchmark definition ("job set") file. Theexample/benchmark.ymlfile contains a sample job set file.example/benchmark.ymljobs:-problem:class:Tspinstance:eil51client:-FixstarsClient-token:INPUT_API_TOKENparameters:timeout:3000num_samples:2The benchmark job set file contains a list of benchmark jobs in YAML or JSON file format. The jobs consist of the number of runs, a list of problems to solve, and parameter values passed to theClient classto run. For this job set, it consists of the following benchmark jobs:target problem:TSPLIB:eil51instancenumber of runs: 2FixstarsClienttoken: INPUT_API_TOKENparameter.timeout: 3000Now to start the benchmark using this job set file, run theamplify-benchcommand with therunsubcommand with the path to the job set file.NoteReplaceINPUT_API_TOKENwith your API token. If you do not have an API token, go toAmplify WEB siteandcreate an account.$amplify-benchrunbenchmark.yml
input_json:benchmark.yml
label:20230803_223440
output:None
parallel:1aws_profile:None
dry_run:False
cli_benchmark_impl()20230803_2234402023-08-0322:34:41,308[pid:94470][INFO]:542.49msinamplify_bench.cli.parser.parse_input_data2023-08-0322:34:41,309[pid:94470][INFO]:makemodelofeil512023-08-0322:34:41,519[pid:94470][INFO]:209.08msinamplify_bench.problem.tsp.make_tsp_model
totaljobs:2successjobs:2errorjobs:0Jobsnotyetstarted:0When execution completes a JSON file is output as the result of the execution in the same directory. The file name is appended with the execution time by default. The output file path can be changed with the--output <path>option.Open the result with Amplify Benchmark ViewerThe results can be visualized using theAmplify Benchmark Viewer. To analyze for the viewer, give thestatssubcommand with the path to a directory or a result file to theamplify-benchcommand.$amplify-benchstatspreset_20230803_223440.jsonBy default, areport/data.jsonfile is created in the current directory. The path directory of the output file can be changed with the--outputoption.Then, drag and drop the createdreport/data.jsonfile into the Amplify Benchmark ViewerGitHub pagesto visualize the results.NoteThe data is analyzed only on the browser and is not stored on the external server.Drag and Dropdata.jsonfileShow the evaluated problem listThe detailed evaluation for each problemAdvanced usagejob set file in detailsPresetObjectA job set file consisted of JSON objects with the following keys. The schema of the job set file is described inamplify_bench/cli/schemas.keytypedescriptionjobsarray[JobObject]list of benchmark jobsvariablesobjectdefinitions of variables used injobs(Optional)importsarray[string]User-defined problem file path (Optional)Strings in the file that begin with$are treated as variable names. Variables are first expanded by the environment variables at runtime, then the variable definitions given in thevariableskey are referenced injobs. This is useful, for example, to specify a setting that is commonly used in multiple jobsvariables:CLIENT:-FixstarsClient-parameters:timeout:3000jobs:-problem:class:Tspinstance:eil51client:$CLIENTnum_samples:2-problem:class:Tspinstance:burma14client:$CLIENTnum_samples:1importsspecifies the list of paths to the user-defined problem file. Enter the path as a relative path from a job set file, a relative path from the current directory, or an absolute path. For details on user-defined question files, seeCreate your own benchmark problems.JobObjectkeytypedescriptionnum_samplesintthe number of runsclientarrayclient name and parametersproblemProblemObjectthe problem definitionmatrixobject[array]the definitions of variable patterns (Optional)Ifnum_samplesis an integer greater than 1, it will be run multiple times with the same settings. Thematrixkey is given with the patterns of variables explained later.Theclientkey is given an array of length 2. The first element of the array is the name of the client class, and the second element is an object with the property values of the client. For example, the following client configuration in the Amplify SDKfromamplify.clientimportFixstarsClientclient=FixstarsClient()client.token="INPUT_API_TOKEN"client.parameters.timeout=1000should be specified as follows in the job set file.jobs:-client:-FixstarsClient-token:INPUT_API_TOKENparameters:timeout:1000NoteSee thedocumentationfor the available properties for eachClientclass.ProblemObjectTheproblemkey has an object consisting of the following keys:keytypedescriptionclassstringThe name of the problem classinstancestringInstance nameparametersobjectConstructor parameters of the problem classTheclassis the name of the problem class contained inamplify_bench/problem. The predefined problem classes areTsp(TSPLIB),Qap(QAPLIB),Cvrp(CVRPLIB),MaxCut(GSET),SudokuandQplib(QPLIB). Theinstanceis the name of the instance in the problem set corresponding to each problem class. Seeamplify_bench/problem/datafor details. Problem classes may have formulation parameters that can be passed to the constructor, which can be specified byparameterkey.Using a matrix for your jobsMultiple jobs can be automatically generated for all combinations of multiple variable patterns given in a single job definition. For example, to run for all combinations of multiple instances and multiple runtimes, the following job set file can be used.variables:NUM_SAMPLES:100FIXSTARS:-FixstarsClient-token:INPUT_API_TOKENparameters:timeout:$TIMEOUTjobs:-problem:class:Qapinstance:$INSTANCEclient:$FIXSTARSnum_samples:$NUM_SAMPLESmatrix:INSTANCE:-esc32a-sko56TIMEOUT:-10000-30000Thematrixis an array of values with the variable names as keys. In this case, jobs withtimeoutof10000and30000will be created foresc32aandsko56respectively. Note that you can refer to the variables you pass tomatrixin the variables defined invariables.NoteVariables are referenced recursively, but an infinite loop will fail.Create your own benchmark problemsUser-defined formulations can be added as benchmark problems that are recognized in Amplify Benchmark.The following example runs a benchmark against theMyTspclass defined in themytsp.pyfile. Setting a list of Python file paths to theimportskey will load additional probem classes defined in the files. A file path must be specified as an absolute path or relative to the job set file or the current directory.imports:-mytsp.pyjobs:-problem:class:MyTspinstance:random8NoteUser-defined class names should not duplicate the built-in problem classes.The problem class must extend theProblemclass and implement the constructor (__init__) and the methodsmake_modelandevaluate. Themake_modelmethod formulates the problem in Amplify SDK and theevaluatemethod evaluates the formulated model with the solution as input.The following code snippet is an example of aMyTspproblem class.mytsp.pyclassMyTsp(Problem):def__init__(self,instance:str,constraint_weight:float=1.0,seed:int=0,path:Optional[str]=None,):super().__init__()self._instance:str=instanceself._problem_parameters["constraint_weight"]=constraint_weightifinstance.startswith("random"):self._problem_parameters["seed"]=seedself._symbols=Nonencity,distances,locations,best_known=self.__load(self._instance,seed,path)self._ncity=ncityself._distances=distancesself._locations=locations# not usedself._best_known=best_knowndefmake_model(self):symbols,model=make_tsp_model(self._ncity,self._distances,self._problem_parameters["constraint_weight"])self._symbols=symbolsself._model=modeldefevaluate(self,solution:SolverSolution)->Dict[str,Union[None,float,str]]:value:Optional[float]=Nonepath:str=""ifsolution.is_feasible:spins=solution.valuesvariables=np.array(self._symbols.decode(spins))# type: ignoreindex=np.where(variables==1)[1]index_str=[str(idx)foridxinindex]value=calc_tour_dist(list(index),self._distances)path=" ".join(index_str)else:passreturn{"label":"total distances","value":value,"path":path}...Constructordef__init__(self,instance:str,**kwargs)->NoneThe constructor must accept at least aninstance: strargument. Otherwise, if you addconstraint_weight: floatto the constructor arguments for example, theparameterskey inProblemObjectcan have aconstraint_weight.make_modelmethoddefmake_model(self)->NoneThemake_modelmethod is responsible for formulating and storing an instance of theamplify.BinaryQuadraticModelclass inself._model.evaluatemethoddefevaluate(self,solution:amplify.SolverSolution)->Dict[str,Union[None,float,str]]Theevaluatemethod takes and evaluates aamplify.SolverSolution. The return value can be any key and value asDict[str, Union[None, float, str]]. The return value is output to theobjective_valuekey in the JSON file of the benchmark result.ContributingAmplify Benchmark is an open source project. Contributions are welcome, including bug reports, feature additions, documentation improvements, etc.Developed byThe Amplify benchmark project ties together:Fixstars Amplify SDKFixstars Amplify Annealing EngineFixstars Amplify Benchmark Viewer