Package: pytest-oar
OAR PyTest PluginSummaryThe OAR PyTest plugin allows for an intuitive declarative interface for OAR with configurations to connect to any
OAR instance. This will allow for both offline OAR results or uploading OAR results to an OAR instanceQuickstartHere is a walkthrough of a new PyTest project with OAR:InstallationTo start, install the OAR PyTest plugin in your Python interpreter:pip install pytest-oarNext we need to install the plugin in PyTest, this guide assumes a fresh project.Create a new file calledconftest.pyand add the following line:pytest_plugins = ["oar.plugin"]OAR is now installed!Creating a testOAR is designed to be as much or as little as you want it to be. Here is a minimal test that will be reported
to OAR:class TestExample:

    def test_one_equals_one(self, oar_test):
        assert 1 == 1The inclusion of theoar_testfixture is the key here, this will both mark this test to
have its results reported, and yield a test object to enrich during test runtime.Let's run the test and see what we get:pytest -sResult:collected 1 item                                                                                                                                               

test_example.py .

====================================================================== 1 passed in 0.00s =======================================================================Not necessarily the most helpful on its own.In the current working directory, there is a new folder, by default titledoar-results:.
├── __pycache__
├── conftest.py
├── oar-results
├── test_example.py
└── venvEach time OAR runs, by default, it will store an offline JSON result. Ours is:{
    "start_time": "2023-02-21 04:03:26.406457",
    "completed_time": "2023-02-21 04:03:26.441054",
    "tests": [
        {
            "id": 0,
            "summary": "test_one_equals_one",
            "outcome": "Passed",
            "analysis": "TrueNegative",
            "resolution": "NotNeeded"
        }
    ]
}Notice how the "id" is set to 0? When running OAR in offline mode, unless you manually set an ID,
the ID will be 0. If running in online mode, a new ID will be returned corresponding to the ID in the
Postgres DB.ConfigurationLet's make OAR a bit more useful and connect our test to an instance of OAR. To do this, we must
configure OAR.Configuration is handled through Pydantic if you are familiar with that, if not, no worries.Configuration can be done through a.tomlfile, a.jsonfile, or environment variables.By default, it will look for a.tomlor.jsonfile located in:CWD/<OAR_CONFIG_PATH>(defaults to look foroar-config.toml)Let's configure OAR with a.tomlfile:Here is our tree now:.
├── __pycache__
├── conftest.py
├── oar-config.toml
├── oar-results
├── test_example.py
└── venvTo connect OAR to an instance, we need to set two variables:hostandsend_results.Here is ourconfig.toml:host = "http://localhost:8080"
send_results = trueNote: In this example, I have a local OAR instance running, accessible on my localhost's port 8080Thesend_resultsenvironment variable is by default False because you might not want to just upload
any local test results, makes it a bit more deliberate.When running the test again:collected 1 item                                                                                                                                               

test_example.py .OAR Test Result ID: 59Notice, in this result, we have result IDs > 0, this is because it is returning the DB ID now. After each OAR test uploaded,
the ID will also be logged.Note: Because configuration is done with Pydantic, environment variable configuration can be done with the same
config variable names with the "OAR_" prefix prepended.So for example, if I wanted to have an equivalent configuration via environment variables, I could do the following:OAR_HOST='http://localhost:8080' OAR_SEND_RESULTS=True pytest -s'Here is a current list of environment variables (with their defaults):host: str = "oar-service:8080"  # Base URL of the OAR instance to send results to
send_results: bool = False  # This is what will control sending the results to the OAR instance
store_results: bool = True  # This will enable the `oar_results` fixture, will not prevent sending results to OAR
output_file: bool = True  # This will output a JSON results file with name `oar-results-<utc-timestamp>.json`
output_dir: str = "oar-results"  # Controls where JSON results files will be stored. Relative to CWDEnriching TestsLet's go back to our tests to see how we can make the result more useful:Let's say that we have to test an API endpoint/user, and our simplified test looks like this:class TestExample:

    def test_users_endpoint_returns_user(self, oar_test):
        request_body = {"id": 1}
        response = simulate_users_endpoint(request_body)
        assert response["username"]
        assert "@" in response["email"]Theoar_testfixture also serves as an object that can be enriched with runtime data.Let's start by adding some test metadata, like an owner and app name of this test case:class TestExample:

    def test_users_endpoint_returns_user(self, oar_test):
        oar_test.owner = "Ryan"
        oar_test.app = "user-service"

        request_body = {"id": 1}
        response = simulate_users_endpoint(request_body)
        assert response["username"]
        assert "@" in response["email"]OAR tests can be enriched like normal Python objects. All fields added to theoar_testwill be placed
in the dynamicdoccolumn of the OAR table.Great! Later we want to analyze results, we can now query by these fields.Let's add some runtime data too, that can help us potentially debug the test:class TestExample:

    def test_users_endpoint_returns_user(self, oar_test):
        oar_test.owner = "Ryan"
        oar_test.app = "user-service"

        request_body = {"id": 1}
        oar_test.request_body = request_body

        response = simulate_users_endpoint(request_body)
        oar_test.response_body = response

        assert response["username"]
        assert "@" in response["email"]Even better! Now when we go to debug this test, all information will be included onto it. When we analyze results later,
we can look for patterns in request/response bodies that can help us debug and find weaknesses in tests and apps.Let's take a look at our generated JSON test report now:{
    "start_time": "2023-02-21 04:41:21.064272",
    "completed_time": "2023-02-21 04:41:21.119699",
    "tests": [
        {
            "id": 62,
            "summary": "test_users_endpoint_returns_user",
            "outcome": "Passed",
            "analysis": "TrueNegative",
            "resolution": "NotNeeded",
            "app": "user-service",
            "request_body": {
                "id": 1
            },
            "response_body": {
                "username": "aUser",
                "email": "user@mail.com"
            },
            "owner": "Ryan"
        }
    ]
}Our test is also available in the OAR DB now:62,test_users_endpoint_returns_user,Passed,TrueNegative,NotNeeded,2023-02-21 04:41:21.113327,2023-02-21 04:41:21.113327,"{""app"": ""user-service"", ""owner"": ""Ryan"", ""request_body"": {""id"": 1}, ""response_body"": {""email"": ""user@mail.com"", ""username"": ""aUser""}}"If you see by default, our test Summary takes the name of the PyTest test, we can overwrite any default OAR attribute by
just defining it in our test:oar_test.summary = "Ensures the users endpoint returns a valid username/email"You can manually set Outcome, Analysis, and Resolution within a test too to implement your own custom pass/fail conditions,
automatic analysis, and even resolution.Custom Test TypesThis previous example demonstrated how we can add information to a test dynamically, but, as tests scale,
there might be a desire to re-use custom types.With the previous test as an example, let's say we wanted to make a formalAPITesttype to ensure certain
fields get set as a way to harden data and reduce reporting errors.Theoar_testfixture yields an object of the typeoar.Test, this is a Pydantic model. We can inherit
and extend this model. Here is an example:class APITest(oar.Test):
    owner: str
    app: str
    request_body: dict[str, typing.Any]
    response_body: dict[str, typing.Any]With this created, to enforce this type on our test, all we have to do is add the type hint:class TestExample:

    def test_users_endpoint_returns_user(self, oar_test: APITest):
        oar_test.summary = "Ensures the users endpoint returns a valid username/email"
        oar_test.owner = "Ryan"
        oar_test.app = "user-service"

        request_body = {"id": 1}
        oar_test.request_body = request_body

        response = simulate_users_endpoint(request_body)
        oar_test.response_body = response

        assert response["username"]
        assert "@" in response["email"]Now if we don't define a field for this testcase, an error will be raised, and the test will NOT be reported until it is
fixed.Note: It is not advised to add methods to custom types, they will not be available at a test's runtime, as all
tests start as aoar.Testand will only be converted to the specific type after the test is over.Let's see the result of this test:{
    "start_time": "2023-02-21 04:55:50.615502",
    "completed_time": "2023-02-21 04:55:50.668846",
    "tests": [
        {
            "id": 65,
            "summary": "Ensures the users endpoint returns a valid username/email",
            "outcome": "Passed",
            "analysis": "TrueNegative",
            "resolution": "NotNeeded",
            "owner": "Ryan",
            "app": "user-service",
            "request_body": {
                "id": 1
            },
            "response_body": {
                "username": "aUser",
                "email": "user@mail.com"
            },
            "type": "APITest"
        }
    ]
}Notice that, for tests of a specific type, a "type" field will automatically be added.ConclusionThat's about it! If you made it this far, you know how to use OAR in PyTest. This is only the
beginning of the process, pay attention to tests that need analysis/resolution after each run if you
would like to follow the process.Also enrich what is valuable, if it would be helpful for debugging or querying, add it! If it is not,
it is probably best to leave it out.