Package: ddapm-test-agent
Datadog APM test agentThe APM test agent is an application which emulates the APM endpoints of theDatadog agentwhich can be used for testing Datadog APM client libraries.See thefeaturessection for the complete list of functionalities provided.See theHTTP APIsection for the endpoints available.See theDevelopmentsection for how to get the test agent running locally to add additional checks or fix bugs.InstallationThe test agent can be installed from PyPI:pip install ddapm-test-agent

ddapm-test-agent --port=8126or from Docker:# Run the test agent and mount the snapshot directory
docker run --rm\
        -p 8126:8126\
        -e SNAPSHOT_CI=0\
        -v $PWD/tests/snapshots:/snapshots\
        ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:latestor from source:pip install git+https://github.com/Datadog/dd-apm-test-agentor a specific branch:pip install git+https://github.com/Datadog/dd-apm-test-agent@{branch}FeaturesTrace invariant checksMany checks are provided by the test agent which will verify trace data.
All checks are enabled by default and can be manually disabled.See theconfigurationsection for the options.Check descriptionCheck nameTrace count header matches number of tracestrace_count_headerClient library version header included in requestmeta_tracer_version_headerTrace content length header matches payload sizetrace_content_lengthReturning dataAll data that is submitted to the test agent can be retrieved.Traces can be returned via the/test/tracesendpoint documentedbelow.Helpful loggingTheINFOlog level of the test agent outputs useful information about the requests the test agent receives. For traces this includes a visual representation of the traces.INFO:ddapm_test_agent.agent:received trace payload with 1 trace chunk
INFO:ddapm_test_agent.agent:Chunk 0
[parent]
├─ [child1]
├─ [child2]
└─ [child3]
INFO:ddapm_test_agent.agent:end of payload ----------------------------------------ProxyThe test agent provides proxying to the Datadog agent.
This is enabled by passing the agent url to the test agent either via the--agent-urlcommand-line argument or by theDD_TRACE_AGENT_URLorDD_AGENT_URLenvironment variables.When proxying is enabled, the response from the Datadog agent will be returned instead of one from the test agent.At the trace-level, proxying can also be disabled by including theX-Datadog-Agent-Proxy-Disabledheader with a value oftrue. This will disable proxying after a trace
is handled, regardless of whether an agent URL is set.Snapshot testingThe test agent provides a form ofcharacterization testingwhich
we refer to as snapshotting.
This allows library maintainers to ensure that traces don't change unexpectedly when making unrelated changes.This can be used to write integration tests by having test cases use the tracer to emit traces which are collected by the test agent and compared against reference traces stored previously.To do snapshot testing with the test agent:Ensure traces are associated with a session token (typically the name of the test case) by either:Calling the/test/session/startwith the token endpoint before emitting the traces; orAttaching an additional query string parameter or header specifying the session token on/vX.Y/tracerequests (see below for
the API specifics). (Required for concurrent test running)Emit traces (run the integration test).Signal the end of the session and perform the snapshot comparison by calling the/tests/session/snapshotendpoint
with the session token. The endpoint will return a400response code if the snapshot failed along with a plain-text
trace of the error which can be forwarded to the test framework to help triage the issue.Snapshot outputThe traces are normalized and output in JSON to a file. The following transformations are made to the input:Trace ids are overwritten to match the order in which the traces were received.Span ids are overwritten to be the DFS order of the spans in the trace tree.Parent ids are overwritten using the normalized span ids. However, if the parent is not a span in the trace, the parent id is not overwritten. This is necessary for handling distributed traces where all spans are not sent to the same agent.Span attributes are ordered to be more human-readable, with the important attributes being listed first.Span attributes are otherwise ordered alphanumerically.The span meta and metrics maps if empty are excluded.ConfigurationThe test agent can be configured via command-line options or via environment variables.Command lineddapm-test-agentddapm-test-agentis command used to run a test agent.Please refer toddapm-test-agent --helpfor more information.ddapm-test-agent-fmtddapm-test-agent-fmtis a command line tool to format or lint snapshot json files.# Format all snapshot json filesddapm-test-agent-fmtpath/to/snapshots# Lint snapshot json filesddapm-test-agent-fmt--checkpath/to/snapshotsPlease refer toddapm-test-agent-fmt --helpfor more information.Environment VariablesPORT[8126]: Port to listen on.ENABLED_CHECKS[""]: Comma-separated values of checks to enable. Valid values can be found intrace invariant checksLOG_LEVEL["INFO"]: Log level to use. DEBUG, INFO, WARNING, ERROR, CRITICAL.LOG_SPAN_FMT["[{name}]"]: Format string to use when outputting spans in logs.SNAPSHOT_DIR["./snapshots"]: Directory in which snapshots will be stored.
Can be overridden by providing thedirquery parameter on/snapshot.SNAPSHOT_CI[0]: Toggles CI mode for the snapshot tests. Set to1to
enable. CI mode does the following:When snapshots are unexpectedlygeneratedfrom a test case a failure will
be raised.SNAPSHOT_IGNORED_ATTRS["span_id,trace_id,parent_id,duration,start,metrics.system.pid,metrics.process_id,metrics.system.process_id,meta.runtime-id"]: The
attributes to ignore when comparing spans in snapshots.DD_AGENT_URL[""]: URL to a Datadog agent. When provided requests will be proxied to the agent.DD_APM_RECEIVER_SOCKET[""]: When provided, the test agent will listen for traces on a socket at the path provided (e.g.,/var/run/datadog/apm.socket)DD_SUPPRESS_TRACE_PARSE_ERRORS[false]: Set to"true"to disable span parse errors when decoding handled traces. When disabled, errors will not be thrown for
metrics incorrectly placed within the meta field, or other type errors related to span tag formatting/types. Can also be set using the--suppress-trace-parse-errors=trueoption.SNAPSHOT_REMOVED_ATTRS[""]: The attributes to remove from spans in snapshots. This is useful for removing attributes
that are not relevant to the test case.Note that removingspan_idis not permitted to allow span
ordering to be maintained.DD_POOL_TRACE_CHECK_FAILURES[false]: Set to"true"to pool Trace Check failures that occured within Test-Agent memory. These failures can be queried later using the/test/trace_check/failuresendpoint. Can also be set using the--pool-trace-check-failures=trueoption.DD_DISABLE_ERROR_RESPONSES[false]: Set to"true"to disable Test-Agent<Response 400>when a Trace Check fails, instead sending a valid<Response 200>. Recommended for use with theDD_POOL_TRACE_CHECK_FAILURESenv variable. Can also be set using the--disable-error-responses=trueoption.HTTP API/test/tracesReturn traces that have been received by the agent. Traces matching specific trace ids can be requested with the options
below.[optional]?trace_ids=[optional]X-Datadog-Trace-IdsSpecify trace ids as comma separated values (eg.12345,7890,2468)/test/session/startInitiate asynchronoussession. All subsequent traces received will be
associated with the required test token provided.[optional]?agent_sample_rate_by_service=Sample rates to be returned by the agent in response to trace v0.4 and v0.5 requests.Example:"{'service:test,env:staging': 0.5, 'service:test2,env:prod': 0.2}"(note the JSON has to be URL-encoded).[optional]?test_session_token=[optional]X-Datadog-Test-Session-TokenTest session token for a test case.Ensure this value is unique to avoid conflicts between sessions./test/session/snapshotPerform a snapshot generation or comparison on the data received during the session.Snapshots are generated when the test agent is not in CI mode and there is no snapshot file present. Otherwise a
snapshot comparison will be performed.[optional*]?test_session_token=[optional*]X-Datadog-Test-Session-TokenTo run test cases in parallel this HTTP header must be specified. All test
cases sharing a test token will be grouped.* Required for concurrent tests. Either via query param or HTTP header.[optional]?ignores=Comma-separated list of keys of which to ignore values for.The default built-in ignore list is:span_id,trace_id,parent_id,duration,start,metrics.system.pid,metrics.process_id,metrics.system.process_id,meta.runtime-id.[optional]?dir=default:./snapshots(relative to where the test agent is run).Override the directory where the snapshot will be stored and retrieved from.This directory must already exist.This value will override the environment variableSNAPSHOT_DIR.Warning: it is an error to specify bothdirandfile.[optional]?file=[optional]X-Datadog-Test-Snapshot-FilenameAn absolute or relative (to the current working directory of the agent) file
name where the snap will be stored and retrieved.Warning: it is an error to specify bothfileanddir.Note: the file extension will be appended to the filename._tracestatswill be appended to the filename for trace stats requests.[optional]?removes=Comma-separated list of keys that will be removed from spans in the snapshot.The default built-in remove list does not remove any keys./test/session/requestsReturn all requests that have been received by the agent for the given session token.[optional]?test_session_token=[optional]X-Datadog-Test-Session-TokenReturns the requests in the following json format:[{"headers":{},"body":"...","url":"http...","method":"GET"}]bodyis a base64 encoded body of the request./test/session/tracesReturn traces that have been received by the agent for the given session token.[optional]?test_session_token=[optional]X-Datadog-Test-Session-Token/test/session/statsReturn stats that have been received by the agent for the given session token.[optional]?test_session_token=[optional]X-Datadog-Test-Session-TokenStats are returned as a JSON list of the stats payloads received./test/session/responses/config (POST)Create a Remote Config payload to retrieve in endpoint/v0.7/config[optional]?test_session_token=[optional]X-Datadog-Test-Session-Tokencurl -X POST 'http://0.0.0.0:8126/test/session/responses/config/path' -d '{"roots": ["eyJ....fX0="], "targets": "ey...19", "target_files": [{"path": "datadog/2/ASM_DATA/blocked_users/config", "raw": "eyJydWxlc19kYXRhIjogW119"}], "client_configs": ["datadog/2/ASM_DATA/blocked_users/config"]}'/test/session/responses/config/path (POST)Due to Remote Config payload being quite complicated, this endpoint works like/test/session/responses/config (POST)but you should send a path and a message and this endpoint builds the Remote Config payload.The keys of the JSON body arepathandmsg[optional]?test_session_token=[optional]X-Datadog-Test-Session-Tokencurl -X POST 'http://0.0.0.0:8126/test/session/responses/config/path' -d '{"path": "datadog/2/ASM_DATA/blocked_users/config", "msg": {"rules_data": []}}'/test/trace_check/failures (GET)Get Trace Check failures that occured. If a token is included, trace failures for only that session token are returned unless used in conjuction withreturn_all, which can be used to return all failures regardless of inputted token.  This method returns a<Response 200>if no Trace Check failures are being returned and a<Response 400>if Trace Check failures are being returned. Trace Check failures are returned as a content type of text, with failure messages concatenated in the response body. Optionally, set theuse_jsonquery string parameter totrueto return Trace Check failures as a JSON response in the following format:response = { 
  "<FAILING_CHECK_NAME>" : ["<FAILURE_MESSAGE_1>", "<FAILURE_MESSAGE_2>"]
}NOTE: To be used in combination withDD_POOL_TRACE_CHECK_FAILURES, or else failures will not be saved within Test-Agent memory and a<Response 200>will always be returned.[optional]?test_session_token=[optional]X-Datadog-Test-Session-Token[optional]?use_json=[optional]?return_all=curl -X GET 'http://0.0.0.0:8126/test/trace_check/failures'/test/trace_check/clear (GET)Clear Trace Check failures that occured. If a token is included, trace failures for only that session token are cleared unless used in conjuction withclear_all. This argument can be used to clear all failures (regardless of inputted session token).[optional]?test_session_token=[optional]X-Datadog-Test-Session-Token[optional]?clear_all=curl -X GET 'http://0.0.0.0:8126/test/trace_check/clear'/test/trace_check/summary (GET)Get Trace Check summary results. If a token is included, returns summary results only for Trace Checks run during the session.  Thereturn_alloptional query string parameter can be used to return all trace check results (regardless of inputted session token). The method returns Trace Check results in the following JSON format:summary = { 
  "trace_content_length" : {
    "Passed_Checks": 10,
    "Failed_Checks": 0,
    "Skipped_Checks": 4,
  }  ...
}[optional]?test_session_token=[optional]X-Datadog-Test-Session-Token[optional]?return_all=curl -X GET 'http://0.0.0.0:8126/test/trace_check/summary'/test/session/integrations (PUT)Update information about the current tested integration.[optional]?test_session_token=[optional]X-Datadog-Test-Session-Tokencurl -X PUT 'http://0.0.0.0:8126/test/session/integrations' -d '{"integration_name": [INTEGRATION_NAME], "integration_version": [INTEGRATION_VERSION],
"dependency_name": [DEPENDENCY_NAME], "tracer_language": [TRACER_LANGUAGE], "tracer_version": [TRACER_VERSION]}'/test/integrations/tested_versions (GET)Return a csv list of all tested integrations received by the agent. The format of returned data will be:tracer_language,tracer_version,integration_name,integration_version,dependency_name.[optional]?test_session_token=[optional]X-Datadog-Test-Session-Tokencurl -X GET 'http://0.0.0.0:8126/test/integrations/tested_versions'/v0.1/pipeline_statsMimics the pipeline_stats endpoint of the agent, but always returns OK, and logs a line everytime it's called./tracer_flare/v1Mimics the tracer_flare endpoint of the agent. Returns OK if the flare contains the required form fields, otherwise400.Logs a line everytime it's called and stores the tracer flare details in the request under"_tracer_flare"./test/session/tracerflaresReturn all tracer-flares that have been received by the agent for the given session token.[optional]?test_session_token=[optional]X-Datadog-Test-Session-TokenReturns the tracer-flares in the following json format:[{"source":"...","case_id":"...","email":"...","hostname":"...","flare_file":"...",}]flare_fileis the base64 encoded content of the tracer-flare payload.If there was an error parsing the tracer-flare form, that will be recorded undererror.DevelopmentPrerequisitesA Python version of 3.8 or above andriotare required. It is recommended to create
and work out of a virtualenv:python3.11 -m venv .venv
source .venv/bin/activate
pip install -e '.[testing]'Running the testsTo run the tests (in Python 3.8):riot run -p3.11 testNote: if snapshots need to be (re)generated in the tests set the environment variableGENERATE_SNAPSHOTS=1.GENERATE_SNAPSHOTS=1 riot run --pass-env -p3.8 test -k test_trace_missing_receivedLinting and formattingTo lint, format and type-check the code:riot run -s flake8
riot run -s fmt
riot run -s mypyDockerTo build (and tag) the dockerfile:dockerbuild--tagtestagent.Run the tagged image:dockerrun--rm-v${PWD}/snaps:/snapshots--publish8126:8126agentRelease notesThis project followssemverand so bug fixes, breaking
changes, new features, etc must be accompanied by a release note. To generate a
release note:riotrunrenonew<short-description-of-change>document the changes in the generated file, remove the irrelevant sections and
commit the release note with the change.ReleasingCheckout themasterbranch and make sure it's up to date.gitcheckoutmaster&&gitpullGenerate the release notes and usepandocto format
them for Github:riotrun-srenoreport--no-show-source|pandoc-frst-tgfm--wrap=noneCopy the output into a new release:https://github.com/DataDog/dd-apm-test-agent/releases/new.Enter a tag for the release (followingsemver) (eg.v1.1.3,v1.0.3,v1.2.0).Use the tag without thevas the title.Save the release as a draft and pass the link to someone else to give a quick review.If all looks good hit publish