Package: teneva-bm
teneva_bmDescriptionBenchmarks library, based on the software productteneva, for testing multidimensional approximation and optimization methods. Our benchmarks include both multidimensional data arrays and discretized functions of many variables.InstallationThe package can be installed via pip (it requires thePythonprogramming language of the version 3.8 or 3.9):pipinstallteneva_bm==0.8.6The package can be also downloaded from the repositoryteneva_bmand be installed bypython setup.py installcommand from the root folder of the project.Some benchmarks require additional installation of specialized libraries. The corresponding instructions are given in the description of each benchmark. Fast installation of all required libraries for all benchmarks can be done with the scriptinstall_all.pyfor existinganacondaenvironmentENV_NAME(except the colab platform, whereenvflag should not be used):condacreate--nameENV_NAMEpython=3.8-y
wgethttps://raw.githubusercontent.com/AndreiChertkov/teneva_bm/main/install_all.py
pythoninstall_all.py--envENV_NAME--silentPlease note that the collectionagentrequires a rather complicated installation process of thegymandmujocoframeworks and related packages, so the scriptinstall_all.pyis rather complicated. You can find mode details for using this script in the header of the file. If you have problems downloading the script via wget, you can download it manually from the root folder of the repositoryteneva_bm.To run benchmark optimization examples (seedemo/opti_*.pyfolder), you should also install thePROTESoptimizer:pipinstallprotes==0.3.5Documentation and examplesAll benchmarks inherit from theBmbase class (teneva_bm/bm.py) and are located in the subfolders (collections of benchmarks) ofteneva_bmfolder. The corresponding python files contain a detailed description of the benchmarks. You can get detailed information on the created benchmark using theinfomethod:fromteneva_bmimport*bm=BmQuboMaxcut()bm.prep()print(bm.info())We prepare some demo scripts with benchmark usage and optimization examples in thedemo_optifolder. We also present some examples in thiscolab notebook.Available benchmarksagent- a collection of problems fromgymframework, includingmujoco agentsbased on the physics enginemujocofor faciliatating research and development in robotics, biomechanics, graphics and animation. The collection includes the following benchmarks:BmAgentAnt,BmAgentCheetah,BmAgentHuman,BmAgentHumanStand,BmAgentLake,BmAgentLander,BmAgentPendInv,BmAgentPendInvDouble,BmAgentReacher,BmAgentSwimmer.Within the framework of this collection, explicit optimization of the entire set of actions (discrete or continuous) may be performed (ifdirectpolicy name is set) or discrete Toeplitz policy may be used (iftoeplitzpolicy name is set; it is the default value for almost all agents); you can also pass your own custom policy as an instance of the correct class (seeteneva_bm/agent/policy.pyfor details).decomp- a collection of low-rank tensor networks. The collection includes the following benchmarks:BmDecompHt,BmDecompPeps,BmDecompTt.func- a collection of analytic functions of a real multidimensional argument having an arbitrary dimension (default is7). The collection includes the following benchmarks:BmFuncAckley,BmFuncAlpine,BmFuncChung,BmFuncDixon,BmFuncExp,BmFuncGriewank,BmFuncMichalewicz,BmFuncPathological,BmFuncPinter,BmFuncPowell,BmFuncQing,BmFuncRastrigin,BmFuncRosenbrock,BmFuncSchaffer,BmFuncSchwefel,BmFuncSalomon,BmFuncSphere,BmFuncSquares,BmFuncTrid,BmFuncTrigonometric,BmFuncWavy,BmFuncYang.For almost all functions, the exact global minimum ("continuous x point", not multi-index) is known (seebm.x_min_realandbm.y_min_real). For a number of functions (BmFuncAlpine,BmFuncChung,BmFuncDixon,BmFuncExp,BmFuncGriewank,BmFuncMichalewicz,BmFuncPowell,BmFuncQing,BmFuncRastrigin,BmFuncRosenbrock,BmFuncSchwefel,BmFuncSphere,BmFuncSquares,BmFuncTrid,BmFuncTrigonometric,BmFuncWavy,BmFuncYang), abm.build_cores()method is available that returns an exact representation of the function on the discrete grid in the tensor train (TT) format as a list of 3D TT-cores. Note also that we apply small shift of the grid limits for all functions, to make the optimization problem more difficult (because many functions have a minimum at the center point of the domain).func_fix- a collection of analytic functions of a real multidimensional argument having the fixed dimension. The collection includes the following benchmarks:BmFuncFixBiggs(d = 5),BmFuncFixCantrell(d = 4),BmFuncFixColville(d = 4),BmFuncFixDolan(d = 5),BmFuncFixPiston(d=7).hs- theHock & Schittkowskicollection of benchmark functions, containing continuous analytic functions of small dimensions (2-5for almost all problems), some of which have given constraints (= 0and/or> 0). The collection includes the following benchmarks:BmHsFunc001,BmHsFunc002, ...,BmHsFunc128(exceptBmHsFunc082,BmHsFunc094,BmHsFunc115,BmHsFunc121,BmHsFunc123,BmHsFunc125which do not exist).odeoc- a collection of optimal control problems described by ordinary differential equations (ODEs), some of the problems have explicit restrictions on the elements of the control vector. The collection includes the following benchmarks:BmOdeocSimple,BmOdeocSimpleConstr.qubo- a collection of quadratic unconstrained binary optimization (QUBO) problems having an arbitrary dimension (default is100); all benchmarks are discrete and have a mode size equals2. The collection includes the following benchmarks:BmQuboKnapQuad,BmQuboMaxcut,BmQuboMvc.qubo_fix- a collection of quadratic unconstrained binary optimization (QUBO) problems having the fixed dimension. The collection includes the following benchmarks:BmQuboKnap10(d=10),BmQuboKnap20(d=20),BmQuboKnap50(d=50),BmQuboKnap80(d=80),BmQuboKnap100(d=100).various- a collection of heterogeneous benchmarks that are not suitable for any other collection (note that in this case, we do not use the name of the collection in the name of the benchmarks, unlike all other sets). The collection includes the following benchmarks:BmMatmul,BmTopopt(draft!),BmWallSimple.Note that you can use the functionteneva_bm_getto obtain a list of all benchmark classes. Also, this function supports various filters (for example, callteneva_bm_get(is_func=True, is_opti_max=False)will return all benchmarks that are continuous functions and relates to minimization task).UsageA typical scenario for working with any benchmark from our package is as follows.Benchmark initializationFirst, we create an instance of the desired benchmark class and manually call theprepmethod (optionally, we can also print detailed information about the benchmark):importnumpyasnpfromteneva_bmimport*bm=BmFuncAckley()bm.prep()print(bm.info())We especially note that all imported classes fromteneva_bmhave theBmprefix, that is, using an asterisk when importing will not pollute the namespace.The class constructor of all benchmarks has the following optional arguments:d- dimension of the benchmark's input (non-negative integer). For some benchmarks, this number is hardcoded (or depends on other specified auxiliary arguments), if another value is explicitly passed, an error message will be generated (e.g., the dimension for benchmarkvarious.bm_matmulis determined automatically by the values of auxiliary argumentssize,rankandonly2). By default, some correct value is used (specified in the benchmark description).n- number of possible discrete values for benchmark input variables, i.e., the mode size of the related tensor / multidimensional array (non-negative integer if all mode sizes are equal or a list of non-negative integers of the lengthd). For some benchmarks, this number is hardcoded (or depends on other specified auxiliary arguments), if another value is explicitly passed, an error message will be generated (e.g., inqubocollection all benchmarks should haven = 2). By default, some correct value is used (specified in the benchmark description).seed- the random seed (default is42). Note that we useRandom Generatorfrom numpy (i.e.,numpy.random.default_rng(seed)) and for a fixed value of the seed, the behavior of the benchmark will always be the same, however, not all benchmarks depend on a seed.name- the display name for benchmark. By default, the class name withoutBmprefix is used....other arguments...- some benchmarks have additional optional arguments, which are described in the corresponding python files (and in the info text).Setting advanced optionsBefore calling thebm.prep()method, you can optionally set a number of additional benchmark options:bm.set_budget(m=None, m_cache=None, is_strict=True)- optional method to set the computation bugetm. If the number of requests to the benchmark (from calls tobm.getandbm.get_poimethods) exceeds the specified budget, thenNonewill be returned. If the flagis_strictis disabled, then the request for the last batch will be allowed, after which the budget will be exceeded, otherwise this last batch will not be considered. Note that when the budget is exceeded,Nonewill be returned both when requesting a single value and a batch of values. Also, in a similar way, you can set a limit on the use of the cache bym_cacheparameter.bm.set_constr(penalty=1.E+42, eps=1.E-16, with_amplitude=True)- if the benchmark has a constraint, then using this function you can setpenalty(for the requested points that do not satisfy the constraint, the valuepenalty * constraint_valuewill be returned) andeps(threshold value to check that the constraint has been fulfilled). Note that we set the constraints as a function (bm.constr/bm.constr_batch) that returns the valueconstraint_value(amplitude) of the constraint, and if the constraint is met, then the value must be non-positive, otherwise, the objective function is not calculated and a value proportional to the amplitude of the constraint is returned (if you disable flagwith_amplitude, then just the value of the penalty will be returned). Note that for the case of maximization task you should set negativepenaltyvalue.bm.set_grid_kind(kind='cheb')- by default, we use the Chebyshev grid (kind = 'cheb') for benchmarks corresponds to a function of a continuous argument, but you can alternatively set it manually to use a uniform grid (kind = 'uni').bm.set_max(i=None, x=None, y=None)- if necessary, you can manually set the multi-index, the corresponding continuous point (for benchmarks, which relate to functions of a continuous argument), and the corresponding value for the exact global maximum of the function. The corresponding values will be further available in the benchmark asbm.i_max_real,bm.x_max_realandbm.y_max_realrespectively. When the benchmark is initialized, this function is called automatically if the optimum is known.bm.set_min(i=None, x=None, y=None)- the same as in the previous point, but for the global minimum.bm.set_log(log=False, cond=None, step=1000, prefix='bm', with_min=None, with_max=None)- when calling this function with theTrueargumentlog, the log will be printed while requests to benchmark. You may set the log coditioncond(min,max,max-minorstep; e.g., in the caseminthe log will be presented each time theminvalue is updated), the log step (for conditionstep) and a stringprefixfor the log. You can also disable the display of current minimum values (with_min) or maximum values (with_max) in the log string. Note that you can provide aslogargument some print-like function, e.g.,log=print, in this case, printing will occur not to the console, but to the corresponding function. Ifcond,with_minandwith_maxare not set, then the logs will correspond to an improvement in the optimization result, taking into account the type of benchmark (max/min).bm.set_cache(with_cache=False, cache=None, m_max=1.E+8)- when calling this function with theTrueargumentwith_cache, the cache will be used, that is, all the values requested from the benchmark will be saved and when the same multi-indices are accessed again, the values will be retrieved from the cache instead of explicitly calculating the objective function. Additionally, you can optionally pass as an argumentcachean already existing cache in the form of a dictionary (the keys are multi-indices in the form of tuples, and the values are the corresponding values of the objective function). We especially note that the cache is only used when querying benchmark values in discrete multi-indices; for requested continuous points, no cache will be used. It is also important to note that no cache will be used for matching multi-indices in the same requested batch of values. Optionally, you can setm_maxargument that specifies the maximum cache size. If the size is exceeded, the cache will be cleared and a corresponding warning will be displayed in the log. Note that when thebm.initmethod is called, the cache is always reset to zero.bm.set_opts(...)- for some benchmarks, this function may be called to set additional benchmark-specific options (please see the names and description of options in the info text). Note that the arguments to this function must be named (e.g.,opt_a=42).Computing benchmark valuesNow the benchmark is ready to go, and we can calculate its value in any requested discrete multi-index (a real number will be returned) or a list of its values for any requested batch of discrete multi-indices (1D numpy array of real numbers will be returned):# Get value at multi-index i:i=np.ones(bm.d)print(bm[i])# you can use the alias "bm.get(i)"# Get values for batch of multi-indices I:I=np.array([i,i+1,i+2])print(bm[I])# you can use the alias "bm.get(I)"Note that thegetmethod can be used instead of[ ]notation, for example, if it is necessary to pass somewhere a function that calculates benchmark values.Since the considered benchmark (BmFuncAckley) corresponds to a function of a continuous argument, above we calculated the values for the discretization of the function on an automatically selected grid (seebm.set_gridmethod). Additionally, we can calculate values at continuous points by analogy:# Get value at point x:x=np.ones(bm.d)*0.42print(bm(x))# you can use the alias "bm.get_poi(x)"# Get values for batch of points X:X=np.array([x,x*0.3,x*1.1])print(bm(X))# you can use the alias "bm.get_poi(X)"Dataset generationFor convenience, the benchmark also has functions that allow you to generate training and test data sets on a discrete grid:# Generate random train dataset (from LHS):# I_trn is array of [500, bm.d] and y_trn is [500]I_trn,y_trn=bm.build_trn(500)# Generate random test dataset (from random choice):# I_tst is array of [100, bm.d] and y_tst of [100]I_tst,y_tst=bm.build_tst(100)Note that, by default, the spent computational budget (seebm.set_budgetfunction) and time spent does not change when the test dataset is generated (this is controlled by the defaultskip_processflag value inbm.build_tst(m=0, seed=None, skip_process=True)), but is consumed when generating the training dataset (i.e.,bm.build_trn(m=0, seed=None, skip_process=False)). Also note that if the seed is not given as an argument to these functions, then the global seed of the benchmark will be used (in this case, the same data sets will be generated when the function is called again).Request history analysisDuring requests to the benchmark, that is, when calling functionsbm[](orbm.get),bm()(orbm.get_poi),bm.build_trn(if the flagskip_processis not set in the function arguments; it has a valueFalseby default) andbm.build_tst(ifskip_processis not set; it isTrueby default for this function), the following useful class parameters are updated:bm.m- the total number of performed calculations of the benchmark value (if a cache is used, then the values taken from the cache are not taken into account in this variable).bm.m_cache- the total number of cache hits performed instead of explicitly calculating benchmark values (if no cache is used, it is0).bm.time_calc- total time in seconds spent on calculating the benchmark values (the time spent on cache accesses is also taken into account).bm.time_full- benchmark lifetime in seconds from the moment of initialization (i.e., the call tobm.initmethod, which is called automatically when creating a benchmark or can then be called manually to reset the query history).bm.y_list- a list of all sequentially calculated benchmark values (results of cache accesses are not added to the list).bm.y_list_full- a list of all sequentially calculated benchmark values including the cache requests.bm.i_max,bm.x_max,bm.y_max- a discrete multi-index, a continuous multi-dimensional point, and benchmark's value corresponding to the maximum of all requested values. Note that for the case of a discrete function, the value ofx_maxwill beNone, and for the case of a continuous function, the values ofi_maxandx_maxwill correlate, while if requests were made for continuous points, thenx_maxwill correspond to the exact position of the point, andi_maxwill be the nearest multi-index of the used discrete grid.bm.i_min,bm.x_min,bm.y_min- same as in the previous point, but for the minimum value.bm.i,bm.x,bm.y- the last requested multi-index / point and the related computed benchmark's value.The functionprint(bm.info_history())may be used to print the corresponding values from the history of requests.Extracting information from the benchmarkFor various purposes, including saving calculation results to a file, the following dictionaries can be used:bm.args- the main benchmark arguments that are set during initialization (dimension, mode size, random seed, etc.);bm.opts- additional benchmark options (if available; set using functionset_opts);bm.prps- various properties of the benchmark (function limits, computational budget, etc.);bm.hist- values associated with the history of requests to the benchmark (list of requested values, found optimum, etc.).You can get all these dictionaries within a single higher level dictionary with corresponding key names via propertybm.dict.NotesFor some benchmarks (e.g., for all benchmarks fromagentcollection) the methodshow(save thepngimage with the current state, the state for provided input or the final state for the best found input) andrender(save themp4animation for the current solution, the solution from the provided input or for the solution from the best found input) are available.AuthorsAndrei ChertkovGleb RyzhakovIvan OseledetsContributorsAnastasia BatshevaArtem BasharinWe are happy to invite you to become a contributor, especially if you have interesting benchmarks ;) Please see detailed instructions for developers inworkflow.mdfile.✭__🚂  The stars that you give toteneva_bm, motivate us to develop faster and add new interesting features to the code 😃