Package: mlperf-loadgen-cb
Overview {#mainpage}Note:A compiled html version of this document is hosted onlinehere.IntroductionThe LoadGen is areusablemodule thatefficientlyandfairlymeasures
the performance of inference systems.It generates traffic for scenarios as formulated by a diverse set of experts
in theMLPerf working group.The scenarios emulate the workloads seen in mobile devices,
autonomous vehicles, robotics, and cloud-based setups.Although the LoadGen is not model or dataset aware, its strength is in its
reusability with logic that is.Integration Example and FlowThe following is an diagram of how the LoadGen can be integrated into an
inference system, resembling how some of the MLPerf reference models are
implemented.Benchmark knows the model, dataset, and preprocessing.Benchmark hands dataset sample IDs to LoadGen.LoadGen starts generating queries of sample IDs.Benchmark creates requests to backend.Result is post processed and forwarded to LoadGen.LoadGen outputs logs for analysis.Useful Links[FAQ](@ref ReadmeFAQ)[LoadGen Build Instructions](@ref ReadmeBuild)[LoadGen API](@ref LoadgenAPI)[Test Settings](@ref LoadgenAPITestSettings) -
A good description of available scenarios, modes, and knobs.MLPerf Inference Code-
Includes source for the LoadGen and reference models that use the LoadGen.MLPerf Inference Rules-
Any mismatch with this is a bug in the LoadGen.MLPerf WebsiteScope of the LoadGen's ResponsibilitiesIn ScopeProvide a reusableC++ library with python bindings.Implementthe traffic patterns of the MLPerf Inference scenarios and
modes.Recordall traffic generated and received for later analysis and
verification.Summarizethe results and whether performance constraints were met.Target high-performancesystems with efficient multi-thread friendly
logging utilities.Generate trustvia a shared, well-tested, and community-hardened
code base.Out of ScopeThe LoadGen is:NOTaware of the ML model it is running against.NOTaware of the data formats of the model's inputs and outputs.NOTaware of how to score the accuracy of a model's outputs.NOTaware of MLPerf rules regarding scenario-specific constraints.Limitting the scope of the LoadGen in this way keeps it reusable across
different models and datasets without modification. Using composition and
dependency injection, the user can define their own model, datasets, and
metrics.Additionally, not hardcoding MLPerf-specific test constraints, like test
duration and performance targets, allows users to use the LoadGen unmodified
for custom testing and continuous integration purposes.Submission ConsiderationsUpstream all local modificationsAs a rule, no local modifications to the LoadGen's C++ library are allowed
for submission.Please upstream early and often to keep the playing field level.Choose your TestSettings carefully!Since the LoadGen is oblivious to the model, it can't enforce the MLPerf
requirements for submission.e.g.:target percentiles and latencies.For verification, the values in TestSettings are logged.To help make sure your settings are spec compliant, use
TestSettings::FromConfig in conjunction with the relevant config file provided
with the reference models.Responsibilities of a LoadGen UserImplement the InterfacesImplement the SystemUnderTest and QuerySampleLibrary interfaces and pass
them to the StartTest function.Call QuerySampleComplete for every sample received by
SystemUnderTest::IssueQuery.Assess AccuracyProcess themlperf_log_accuracy.jsonoutput by the LoadGen to determine
the accuracy of your system.For the official models, Python scripts will be provided by the MLPerf model
owners for you to do this automatically.For templates of how to do the above in detail, refer to code for the demos,
tests, and reference models.