Package: otbenchmark
otbenchmarkWhat is it?The goal of this project is to provide benchmark classes for OpenTURNS.
It provides a framework to create use-cases which are associated with
reference values.
Such a benchmark problem may be used in order to check that a given
algorithm works as expected and to measure its performance in terms
of accuracy and speed.Two categories of benchmark classes are currently provided:reliability problems, i.e. estimating the probability that
the output of a function is less than a threshold,sensitivity problems, i.e. estimating sensitivity indices,
for example Sobol' indices.Most of the reliability problems were adapted from the RPRepo :https://rprepo.readthedocs.io/en/latest/This module allows you to create a problem, run an algorithm and
compare the computed probability with a reference probability:problem = otb.RminusSReliability()
event = problem.getEvent()
pfReference = problem.getProbability() # exact probability
# Create the Monte-Carlo algorithm
algoProb = ot.ProbabilitySimulationAlgorithm(event)
algoProb.setMaximumOuterSampling(1000)
algoProb.setMaximumCoefficientOfVariation(0.01)
algoProb.run()
resultAlgo = algoProb.getResult()
pf = resultAlgo.getProbabilityEstimate()
absoluteError = abs(pf - pfReference)Moreover, we can loop over all problems and run several methods on these
problems.AuthorsMichaël BaudinYoussef JebrounElias FekhariVincent ChabridonInstallationTo install the module, we can use the "git clone" command:git clone https://github.com/mbaudin47/otbenchmark.git
cd otbenchmark
python setup.py installGetting helpThe code has docstrings. Hence, using the "help" statement will help. Another way of getting help is to read the examples, which are presented in the next section.Overview of the benchmark problemsThe simplest use cas of the library is inAnalysis of the R-S case, which shows how to use this problem with two variables to estimate its failure probability. In theBenchmark the G-Sobol test functionproblem, we show how to estimate sensitivity indices on the G-Sobol' test function. When using a reliability problem, it is convenient to create a given algorithm, e.g. Subset sampling, based on a given problem: theReliability factoriesshows how to do this for Monte-Carlo, FORM, SORM, Subset and FORM-importance sampling.The library currently has:26 reliability problems,4 sensitivity problems.One of the most useful feature of the library is to perform a benchmark that is, loop over the problems. InBenchmark on a given set of problems, we run several algorithms on all the problems. The associated statistics are gathered in table, presented inBenchmark the reliability solvers on the problems. InCheck reference probabilities with Monte-Carlo, we compare the exact (reference) probability with a Monte-Carlo estimate with a large sample.It is often useful to draw a sensitivity or reliability problem. Since many of these problems have dimensions larger than two, this raises a number of practical issues.Event:Draw eventsshows how to draw an multidimensional event,Function:Draw cross cut of functionsshows how to draw cross cuts of functions,Distribution:Draw cross cuts of distributionsshows how to draw cross cuts of distributions andDraw conditional distributionsplots conditional distributions.The "DrawEvent" class that the module provides typically plots the following figure for the RP57.The following figure plots cross-cuts of the limit state function for the RP8.TheConvergence of Monte-Carlo to estimate the probability in a reliability problemexample might be interesting for those who want to plot convergence graphics.We provide inBBRCan example which shows how to use the "evaluate" function that the module provides to evaluate a function which is available in the remote BBRC server.TheExamplesdirectory has many other examples: please read the notebooks and see if one of the examples fits your needs.TODO-ListThe FORM algorithm does not perform correctly on: RP75, RP111
and Four-branch serial system.
An explanation would be required for this.The computeCDF() method does not perform correctly on many problems.
An explanation would be required for this.