Project: OpenSearch (Observability and Analysis / Observability)
File: README.md

Certain tests require specific settings to be applied to the OpenSearch instance in order to pass. You should run OpenSearch as follows: bin/opensearch -Enode.attr.testattr=test -Epath.repo=/tmp -Erepositories.url.allowed_urls='http://snapshot.*' A YAML test file consists of: an optional `setup` section, followed by an optional `teardown` section, followed by one or more test sections For instance: setup: do: .... do: .... teardown: do: .... "First test": do: ... match: ... "Second test": do: ... match: ... A `setup` section contains a list of commands to run before each test section in order to setup the same environment for each test section. A `teardown` section contains a list of commands to run after each test section in order to setup the same environment for each test section. This may be needed for modifications made by the test that are not cleared by the deletion of indices and templates. A test section represents an independent test, containing multiple `do` statements and assertions. The contents of a test section must be run in order, but individual test sections may be run in any order, as follows: run `setup` (if any) reset the `response` var and the `stash` (see below) run test contents run `teardown` (if any) delete all indices and all templates Dot notation is used for (1) method calls and (2) hierarchical data structures. For instance, a method call like `cluster.health` would do the equivalent of: client.cluster.health(...params...) A test against `_tokens.1.token` would examine the `token` key, in the second element of the `tokens` array, inside the `response` var (see below): $val = $response->{tokens}[1]{token} # Perl syntax roolz! If one of the levels (eg `tokens`) does not exist, it should return an undefined value. If no field name is given (ie the empty string) then return the current $valused for testing the whole response body. Use \\. to specify paths that actually contain . in the key name, for example in the `indices.get_settings` API. If a test section should only be run on certain versions of OpenSearch, then the first entry in the section (after the title) should be called `skip`, and should contain the range of versions to be skipped, and the reason why the tests are skipped. For instance: "Parent": skip: version: "0.20.1 - 0.90.2" reason: Delete ignores the parent param do: ... test definitions ... All tests in the file following the skip statement should be skipped if: `min <= current <= max`. The `version` range can leave either bound empty, which means "open ended". For instance: "Parent": skip: version: "1.0.0.Beta1 - " reason: Delete ignores the parent param do: ... test definitions ... The skip section can also be used to list new features that need to be supported in order to run a test. This way the up-to-date runners will run the test, while the ones that dont support the feature yet can temporarily skip it, and avoid having lots of test failures in the

Once all runners have implemented the feature, it can be declared supported by default, thus the related skip sections can be removed from the tests. "Parent": skip: features: regex do: ... test definitions ... The `features` field can either be a string or an array of strings. The skip section requires to specify either a `version` or a `features` list. Runner supports `catch: unauthorized` on a `do` operator. This test can only run if the cluster is running with the distributions default number of shards. The Java test runner introduces randomness and sometimes overrides the default number of shards to `2`. If the default number of shards is changed, test marked with this feature should not run. The runner is able to set per request headers on the `do` operation. Indicates the runner can parse `node_selector` under the `do` operator and use its metadata to select the node to perform the `do` operation on. Allows you to use a stashed value in any key of an object during a `match` assertion set: {nodes.$clustermanager.http.publishaddress: host} match: $body: { "nodes": { $host: { ... stuff in here ... } } } Allows a stashed value to be referenced in path lookups as a single token. E.g: path.$stash.value Allows a stashed key to appear anywhere in the path (note the placeholder needs to be within curly brackets too in this case): field1.e${placeholder}ments.element1 Used only in the doc snippet tests. Allow you to do ease replacements using a special `$_path` marker. // TESTRESPONSEs/somevalue/$body.${_path}/ to mean "replace somevalue with whatever is the response in the same position." The runner can assert the warnings headers returned by OpenSearch through the `warning:` assertations under `do:` operations. The runner is able to send and receive `application/yaml` and perform all assertions on the returned data. Asserts an array of object contains an object with a property set to a certain value. e.g: contains: { nodes.$cluster_manager.plugins: { name: painless-allowlist } } Asserts the plugins array contains an object with a `name` property with the value `painless-allowlist` Supports the `transformandset` operator as described in this document. Allows you to stash an arbitrary key from a returned map e.g: set: nodes.arbitrarykey: nodeid This means: Stash any of the keys returned under `nodes` as `$node_id` The `do` operator calls a method on the client. For instance: do: cluster.health: level: shards The response from the `do` operator should be stored in the `response` var, which is reset (1) at the beginning of a file or (2) on the next `do`. If the arguments to `do` include `catch`, then we are expecting an error, which should be caught and

For instance: do: catch: missing get: index: test type: test id: 1 The argument to `catch` can be any of: <table> <colgroup> <col style="width: 15%" /> <col style="width: 85%" /> </colgroup> <tbody> <tr class="odd"> <td><p><code>bad_request</code></p></td> <td><p>a 400 response from ES</p></td> </tr> <tr class="even"> <td><p><code>unauthorized</code></p></td> <td><p>a 401 response from ES</p></td> </tr> <tr class="odd"> <td><p><code>forbidden</code></p></td> <td><p>a 403 response from ES</p></td> </tr> <tr class="even"> <td><p><code>missing</code></p></td> <td><p>a 404 response from ES</p></td> </tr> <tr class="odd"> <td><p><code>request_timeout</code></p></td> <td><p>a 408 response from ES</p></td> </tr> <tr class="even"> <td><p><code>conflict</code></p></td> <td><p>a 409 response from ES</p></td> </tr> <tr class="odd"> <td><p><code>request</code></p></td> <td><p>a 4xx-5xx error response from ES, not equal to any named response above</p></td> </tr> <tr class="even"> <td><p><code>unavailable</code></p></td> <td><p>a 503 response from ES</p></td> </tr> <tr class="odd"> <td><p><code>param</code></p></td> <td><p>a client-side error indicating an unknown parameter has been passed to the method</p></td> </tr> <tr class="even"> <td><p><code>/foo bar/</code></p></td> <td><p>the text of the error message matches this regular expression</p></td> </tr> </tbody> </table> If `catch` is specified, then the `response` var must be cleared, and the test should fail if no error is thrown. If the arguments to `do` include `warnings` then we are expecting a `Warning` header to come back from the request. If the arguments dont include a `warnings` argument then we dont expect the response to include a `Warning` header. The warnings must match exactly. Using it looks like this: do: warnings: '[index] is deprecated' quotes are not required because yaml but this argument is always a list, never a single string no matter how many warnings you expect get: index: test type: test id: 1 If the arguments to `do` include `allowed_warnings` then matching `Warning` headers do not fail the request. Unlike the `warnings` argument, these arent expected so much as "allowed". This usually comes up in backwards compatibility testing. Using it looks like this: do: allowed_warnings: some warning this argument is also always a list, never a single string no matter how many warnings you expect get: index: test type: test id: 1 If the arguments to `do` include `nodeselector` then the request is only sent to nodes that match the `nodeselector`. It looks like this: "test id": skip: features: node_selector do: node_selector: version: " - 6.9.99" index: index: test-weird-index- type: weird.type id: 1 body: { foo: bar } If you list multiple selectors then the request will only go to nodes that match all of those selectors. The following selectors are supported: `version`: Only nodes whos version is within the range will receive the request. The syntax for the pattern is the same as when `version` is within `skip`. `attribute`: Only nodes that have an attribute matching the name and value of the provided attribute match. Looks like: node_selector: attribute: name: value For some tests, it is necessary to extract a value from the previous `response`, in order to reuse it in a subsequent `do` and other tests. For instance, when testing indexing a document without a specified ID: do: index: index: test type: test set: { id: id } # stash the value of `response.id` as `id` do: get: index: test type: test id: $id # replace `$id` with the stashed value match: { id: $id } # the returned `response.id` matches the stashed `id` The last response obtained gets always stashed automatically as a string, called `body`. This is useful when needing to test apis that return text rather than json

cat api), as it allows to treat the whole body as an ordinary string field. Stashed values can be used in property names, eg: do: cluster.state: {} set: { clustermanagernode: cluster_manager } do: nodes.info: metric: [ transport ] istrue: nodes.$clustermanager.transport.profiles Note that not only expected values can be retrieved from the stashed values (as in the example above), but the same goes for actual values: match: { $body: /^.+$/ } # the returned `body` matches the provided regex if the body is text match: { $body: {} } # the returned `body` matches the JSON object if the body is JSON The stash should be reset at the beginning of each test file. For some tests, it is necessary to extract a value and transform it from the previous `response`, in order to reuse it in a subsequent `do` and other tests. Currently, it only has support for `base64EncodeCredentials`, for unknown transformations it will not do anything and stash the value as is. For instance, when testing you may want to base64 encode username and password for `Basic` authorization header: do: index: index: test type: test transformandset: { logincreds: "#base64EncodeCredentials(user,password)" } # stash the base64 encoded credentials of `response.user` and `response.password` as `logincreds` do: headers: Authorization: Basic ${logincreds} # replace `$logincreds` with the stashed value get: index: test type: test Stashed values can be used as described in the `set` section The specified key exists and has a true value (ie not `0`, `false`, `undefined`, `null` or the empty string), eg: is_true: fields.foo # the foo key exists in the fields hash and is "true" The specified key doesnt exist or has a false value (ie `0`, `false`, `undefined`, `null` or the empty string), eg: isfalse: fields.source # the _source key doesn't exist in the fields hash or is "false" Used to compare two variables (could be scalars, arrays or hashes). The two variables should be identical, eg: match: { _source: { foo: bar } } Supports also regular expressions with flag X for more readability (accepts whitespaces and comments): match: $body: > /^ epoch \s+ timestamp \s+ count \s+ \n \d+ \s+ \d{2}:\d{2}:\d{2} \s+ \d+ \s+ \n $/ Note: `$body` is used to refer to the last obtained response body as a string, while `''` refers to the parsed representation (parsed into a Map by the Java runner for instance). Having the raw string response is for example useful when testing cat APIs. Compares two numeric values, eg: lt: { foo: 10000 } # the `foo` value is less than 10,000 Compares two numeric values, eg: lte: { foo: 10000 } # the `foo` value is less than or equal to 10,000 This depends on the data type of the value being examined, eg: length: { id: 22 } # the `id` string is 22 chars long length: { tokens: 3 } # the `tokens` array has 3 elements length: { source: 5 } # the `source` hash has 5 keys