Package: dtest-framework
dtestA library to facilitate the testing of data inside data pipelines. Results are pushed to a messaging queue of some sort for consumption by applications, persistence, etc.Supported messaging queues / streaming platformsRabbitMQMQTTRedisKafkaKinesisSupported secrets managersAWS Secrets ManagerHashicorp VaultInstallationpip3 install dtest-frameworkUnit TestsTesting is set up using PytestInstall Pytest withpip3 install -U pytestRun the tests withpytestin the root directory.Quick Startfrom dtest.dtest import Dtest
from hamcrest import *


# If publishing to a RabbitMQ queue, specify 'queue' \
# If publishing to a key-value store, specify 'kv-store' \
# Or specify both

connectionConfig = {
    "queue": {
        "host": "localhost",
        "username": "guest",
        "password": "guest",
        "exchange": "test.dtest",
        "exchange_type": "fanout"
    },
    "kv-store": {
        "api_url": "localhost:8080/api/",
        "retrieve_path": "getKeyValue/",
        "publish_path": "postKeyValue/"
    }
}
metadata = {
    "description": "This is a test of the assertCondition",
    "topic": "test.dtest",
    "ruleSet": "Testing some random data",
    "dataSet": "random_data_set_123912731.csv"
}

dt = Dtest(connectionConfig, metadata)

dsQubert = [0,1]

dt.assert_that(dsQubert, has_length(2))
// True

dt.publish()
// Publishes test suite to MQ server


////////////////////////////////////////
// Store value in KV store for later use
dt.publishKeyValue('some-descriptor-dsQubert-length', len(dqQubert))

// Retrieve value from KV store to compare other files against
count = dt.retrieveKeyValue('some-descriptor-dsQubert-length')

dt.assert_that(dsQubert, has_length(count))Connection configurationThere are two options for providing the connection configuration for the publisher - the default way described above and by storing your configuration in a secrets manager. To utilize a secrets manager, use a connectionConfig similar to:connectionConfig = {
    "queue": {
        "vault": {
                    "provider": "aws_secrets_manager",
                    "secret_name": "secret_name_here",
                    "region": "us-east-1"
                }
    }
}Here we are giving the provider nameaws_secrets_manager, the key to use to retrieve the secretsecret_name, and the region in which Secrets Manager is hosted.secret_nameandregionare passed toboto3directly.region_nameis provided when initializing theboto3session andsecret_nameis provided to theboto3.secretsmanager.get_secret_value()function asSecretId.Custom handlersIt is possible to create custom message queue and key value store handlers. Implement a class that inherits fromdtest.handler.MqHandlerordtest.handler.KvHandlerdepending on your needs.class MqHandler:

    @classmethod
    def version(self): return "1.0"

    @abstractmethod
    def connect(self): raise NotImplementedError

    @abstractmethod
    def publishResults(self): raise NotImplementedError

    @abstractmethod
    def closeConnection(self): raise NotImplementedError


class KvHandler:

    @classmethod
    def version(self): return "1.0"

    @abstractmethod
    def retrieve(self): raise NotImplementedError

    @abstractmethod
    def publish(self): raise NotImplementedErrorPackage dependenciesI did not want to require that all dependencies of every module need to be installed. As such, the following packages need to be installed viapipif you would like to utilize the specified functionalityPackageDependent module/functionalitypandasLocal testing withpytestCI/CDUse the standardecslabeled Jenkins agentPerforms tests on master commits and PRsDoes not deploy to PyPI automatically