Package: ibm-service-validator
ibm-service-validator 0.5.0ExtendsSchemathesisto testIBM API Handbookcompliance and consistency between an API implementation and its OpenAPI definition.OverviewThis tool takes an OpenAPI definition, a valid API endpoint, and any necessary API credentials and verifies that the API implementation complies with its OpenAPI definition and theIBM API Handbook.Python VersionThe minimum python version supported is 3.5.8InstallThe Service Validator is distributed through PyPI, and you may install usingpip.pip3 install ibm-service-validatorUseRunTheruncommand runs a suite of tests against the service using an API definition.ibm-service-validator [env] run <API definition> --base-url <base URL of API> [run options][env]The following options may be used to set environment variables used to obtain a bearer token when --with-bearer used:--set-api-key: set the IBM_CLOUD_SERVICE_VALIDATOR_API_KEY environment variable to the API key needed to obtain a bearer token.--set-iam-endpoint: set the IBM_CLOUD_SERVICE_VALIDATOR_IAM_ENDPOINT to the URL of the API at which the bearer token may be obtained.Example Usage:ibm-service-validator --set-api-key=YOUR_API_KEY --set-iam-endpoint=https://iam.test.cloud.ibm.com/identity/token run path/to/schema -b https://api.com --with-bearerrun options-a (--auth): provide server username and password in the formusername:password.-A (--auth-type): authentication mechanism. May be "basic" or "digest" (default is "basic").-b (--base-url): base url of the service to be tested.-c (--checks): comma-separated list of checks to run. Example:--checks=not_a_server_error,response_schema_conformance.Seeconfigurationfor full list of checks-x (--exitfirst): flag to exit and report on the first error or test failure.-H (--header): custom header to include in all requests. Example:-H Authorization:Bearer\ 123.-v (--verbosity): increase the verbosity of the report using the repetition of options. Examples:-v,-vv,-vvvin order of increasing verbosity. We only use one level of verbosity but this is passed to schemathesis which may utilize more levels of verbosity.-B (--with-bearer): obtains a bearer token and includes it in tests. Usesenvironment variablesto obtain the bearer token.--show-errors-tracebacks: flag to show error tracebacks for internal errors.--store-request-log: name of yaml file in which to store logs of requests made during testing. Example:--store-request-log=logs.yaml.--hypothesis-deadline: number of milliseconds allowed for the server to respond (default is 500). Example:--hypothesis-deadline=300.--hypothesis-phases: determines how test data will be generated.The default value,explicit, indicates test data will only be generated from examples in the OpenAPI definition.Example:--hypothesis-phases=explicit,generatewill use explicit OpenAPI examples and generate test data.explicit: test data generated from examples. Recommended.reuse: reuse old test data.generate: test data generated from schema definition.target: mutate test data for targeting.shrink: shrink test data.--hypothesis-verbosity: control how much information is reported. Example:--hypothesis-verbosity=verbosequietnormalverbosedebugFilter tests by endpoints, methods, and/or tags that match a given endpoint/method/tag pattern.-E (--endpoint) Example:-E /users/create, will only test endpoints containing/users/create-M (--method) Example:-M GET, will only test endpoints with GET requests-T (--tag) Example:-T custom_tag, will only test endpoints with the schema defined tag ofcustom_tag-O (--operation-id) Example:-O custom_operation_id, will only test the endpoint with thecustom_operation_idoperation_idOptions when generating test data for schema definitions without schema defined examples. When there is no example defined in the API definition, these options will determine how the mock-data is generated when testing that endpoint.--hypothesis-derandomize: this flag is used to generate test data deterministically instead of generating random, valid test data.--hypothesis-max-examples: this value determines the maximum number of tests to generate for each method. Example:--hypothesis-max-examples=50.--hypothesis-seed: provide a seed from which random test data will be generated.ReplayThereplaycommand runs a snapshot of tests and compares new results to the original results. This command takes a file with logs from a previous run of tests.ibm-service-validator replay path/to/logs.yaml [options]Note: To capture these logs, run the service validator with the--store-request-logoption.Example:ibm-service-validator run path/to/mock_server.yaml --base-url https://localhost:5000 --store-request-log=request_log.yamlibm-service-validator replay request_log.yaml --status=FAILUREThis will re-run the service validator on the previous run but only on the endpoints that returned a FAILURE and will report the new result for each endpoint.replay options--id: run a specific request by providing the request id (--id 1).--status: run only the requests with a specific status code (--status SUCCESS).SUCCESSERRORFAILURE--uri: run the requests that contain the given uri (--uri path1/resources)--method: run only the requests that use the given HTTP method (--method GET)ConfigurationConfiguration FileRules may be configured using aibm-service-validator-config.yamloribm-service-validator-config.jsonfile.We look for a configuration file in the current working directory from which the service validator is invoked, and we search up the directory until we find the first matching config file. It is recommended to create this configuration file in the root directory of your project.Rules may be on, off, or warn. An example of the configuration file is given below:ibm_cloud_api_handbook:
    allow_header_in_405: 'on'
    content_location: warn
    get_with_request_body: 'on'
    invalid_accept_header: 'on'
    invalid_request_content_type: 'on'
    location_201: 'on'
    no_422: 'on'
    no_accept_header: 'on'
    no_content_204: 'on'
    www_authenticate_401: 'on'
schemathesis_checks:
    not_a_server_error: 'on'
    status_code_conformance: 'warn'
    content_type_conformance: 'warn'
    response_schema_conformance: 'on'Create Default Configuration FileTo initialize a default configuration file in the current working directory, use:ibm-service-validator init [options]init options-o (--overwrite): if config file already exists, overwrite the existing config file with default values.-j (--json): write the default config file as json.Add Case RulesFor some rules, we send an additional request to the API to target specific behavior. We call these rulesadd_caserules. Theinvalid_request_content_typerule, for example, tests how the API responds when it receives a request with a request body and an invalidContent-Typeheader. To make this rule effective, we need to send a request to the API with an invalidContent-Typeheader. Forinvalid_request_content_typeand all otheradd_caserules listed below, we create an additional request to exercise the behavior the rule is testing. When anadd_caserule is disabled, the additional request is also disabled.add_caseRules:get_with_request_bodyinvalid_accept_headerinvalid_request_content_typeIncluding Examples in API DefinitionOften, a service's requirements are stricter than its schema. For example, anaccount_idmay have schema,type: string. However, a validaccount_idis restricted to the set of strings associated with an account. For this reason, the default way to generate requests is to useOpenAPI examplesin the API definition. Notice examples may be provided using theexampleandexampleskeywords. The service validator supports bothexampleandexamples.Important Note on Providing an Object ExampleExamples for object properties should be provided in an example object instead of individual property examples. Hence, for the schema:schema:
    type: object
    properties:
    id:
        type: integer
    name:
        type: stringProvide an example object:schema:
    type: object
    properties:
    id:
        type: integer
    name:
        type: string
    example:
        id: 10
        name: Jessica SmithInstead of individual property examples:schema:
    type: object
    properties:
    id:
        type: integer
        example: 10
    name:
        type: string
        example: Jessica Smith