Project: MongoDB (App Definition and Development / Database)
File: suites.md

Resmoke stores test suites represented as `.yml` files in the `buildscripts/resmokeconfig/suites` directory. These `.yml` files allow users to spin up a variety of configurations to run tests against. This represents the type of tests that are running in this suite. Some examples include: jstest, cppunittest, cppintegrationtests, benchmarktest, fsmworkloadtest, etc. You can see all available options in the `SELECTORREGISTRY` at `mongo/buildscripts/resmokelib/selector.py`. Ex: ```yaml testkind: jstest ``` The selector determines test files to include/exclude in the suite. Ex: ```yaml selector: roots: jstests/aggregation//*.js exclude_files: jstests/aggregation/extras/*.js jstests/aggregation/data/*.js excludewithany_tags: requirespipelineoptimization ``` File path(s) of test files to include. If a path without a glob is provided, it must exist. File path(s) of test files to exclude. If a path without a glob is provided, it must exist. Exclude test files by tag name(s). To see all available tags, run `./buildscripts/resmoke.py list-tags`. Configuration for the test execution framework. Ex: ```yaml executor: archive: config: hooks: fixture: ``` Upon failure, data files can be uploaded to s3. A failure is when a `hook` or `test` throws an exception. Data files will be archived in the following situations: Any `hook` included in this section throws an exception. If `tests: true` and any `test` in the suite throws an exception. Ex: ```yaml archive: hooks: Hook1 Hook2 tests: true ``` This section contains additional configuration for each test. The structure of this can vary significantly based on the `test_kind`. For specific information, you can look at the implementation of the `test_kind` of concern in the `buildscripts/resmokelib/testing/testcases` directory. Ex: ```yaml config: shell_options: global_vars: TestData: defaultReadConcernLevel: null enableMajorityReadConcern: "" nodb: "" gssapiServiceName: "mockservice" eval: >- var testingReplication = true; load('jstests/libs/overridemethods/setreadandwrite_concerns.js'); load('jstests/libs/overridemethods/enablecausalconsistencywithoutreadpref.js'); ``` Above is an example of the most common `testkind` -- `jstest`. `jstest` uses `shelloptions` to customize the mongo shell when running tests. `global_vars` allows for setting global variables. A `TestData` object is a special global variable that is used to hold testing data. Parts of `TestData` can be updated via `resmoke` command-line invocation, via `.yml` (as shown above), and during

The global `TestData` object is merged intelligently and made available to the `js_test` running. Behavior can vary on key collision, but in general this is the order of precedence: (1) resmoke command-line (2) [suite].yml (3) runtime/default. The mongo shell can also be invoked with flags & named arguments. Flags must have the `''` value, such as in the case for `nodb` above. `eval` can also be used to run generic javascript code in the shell. You can directly include javascript code, or you can put it in a separate script & `load` it. All hooks inherit from the `buildscripts.resmokelib.testing.hooks.interface.Hook` parent class and can override any subset of the following empty base methods: `beforesuite`, `aftersuite`, `beforetest`, `aftertest`. At least 1 base method must be overridden, otherwise the hook will not do anything at all. During test suite execution, each hook runs its custom logic in the respective scenarios. Some customizable tasks that hooks can perform include: _validating data, deleting data, performing cleanup_, etc. You can see all existing hooks in the `buildscripts/resmokelib/testing/hooks` directory. Ex: ```yaml hooks: class: CheckReplOplogs class: CheckReplDBHash class: ValidateCollections class: CleanEveryN n: 20 class: MyHook param1: something param2: somethingelse ``` The hook name in the `.yml` must match its Python class name in the `buildscripts/resmokelib/testing/hooks` directory. Parameters can also be included in the `.yml` and will be passed to the hook's constructor (the `hook_logger` & `fixture` parameters are automatically included, so those should not be included in the `.yml`). This represents the test fixture to run tests against. The `class` sub-field corresponds to the Python class name of a fixture in the `buildscripts/resmokelib/testing/fixtures` directory. All other sub-fields are passed into the constructor of the fixture. These sub-fields will vary based on the fixture used. Ex: ```yaml fixture: class: ShardedClusterFixture num_shards: 2 mongos_options: bindipall: "" set_parameters: enableTestCommands: 1 mongod_options: bindipall: "" set_parameters: enableTestCommands: 1 periodicNoopIntervalSecs: 1 writePeriodicNoops: true ``` For inspiration on creating a new test suite, you can check out a variety of examples in the `buildscripts/resmokeconfig/suites` directory.