Package: lumigo-opentelemetry
Lumigo OpenTelemetry Distro for Python :stars:The Lumigo OpenTelemetry Distribution for Python is a package that provides no-code distributed tracing for containerized applications.The Lumigo OpenTelemetry Distribution for Python is made of several upstream OpenTelemetry packages, with additional automated quality-assurance and customizations that optimize for no-code injection, meaning that you should need to update exactly zero lines of code in your application in order to make use of the Lumigo OpenTelemetry Distribution.
(See theNo-code activationsection for auto-instrumentation instructions)Note:If you are looking for the Lumigo Python tracer for AWS Lambda functions,lumigo-traceris the package you should use instead.SetupAdding the Lumigo OpenTelemetry Distro for Python to your application is a three-step process:Add the Lumigo OpenTelemetry Distro for Python as dependencyProvide configurations through environment variablesActivate the tracer, which can also be achieved through environment variablesAdd lumigo_opentelemetry as dependencyThelumigo_opentelemetrypackageneeds to be a dependency of your application.
In most cases, you will addlumigo_opentelemetryas a line inrequirements.txt:lumigo_opentelemetryOr, you may usepip:pipinstalllumigo_opentelemetryEnvironment-based configurationConfigure theLUMIGO_TRACER_TOKENenvironment variable with the token value generated for you by the Lumigo platform, underSettings --> Tracing --> Manual tracing:LUMIGO_TRACER_TOKEN=<token>Replace<token>below with the token generated for you by the Lumigo platform.It is also strongly suggested that you set theOTEL_SERVICE_NAMEenvironment variable with, as value, the service name you have chosen for your application:OTEL_SERVICE_NAME=<service name>Replace<service name> with the desired name of the service.Note:While you are providing environment variables for configuration, consider also providing the one needed forno-code tracer activation:-)Tracer activationThere are two ways to activate thelumigo_opentelemetrypackage: one based on importing the package in code (manual activation), and the other via the environment (no-code activation).
Theno-code activationapproach is the preferred one.No-code activationNote:The instructions in this section are mutually exclusive with those provided in theManual activationsection.Set the following environment variable:AUTOWRAPT_BOOTSTRAP=lumigo_opentelemetryManual activationNote:The instructions in this section are mutually exclusive with those provided in theNo-code activationsection.Importlumigo_opentelemetryat the beginning of your main file:importlumigo_opentelemetryConfigurationOpenTelemetry configurationsThe Lumigo OpenTelemetry Distro for Python is made of several upstream OpenTelemetry packages, together with additional logic and, as such, the environment variables that work with "vanilla" OpenTelemetry work also with the Lumigo OpenTelemetry Distro for Python. Specifically supported are:General configurationsBatch span processor configurations: The Lumigo OpenTelemetry Distro for Python uses a batch processor for sending data to Lumigo.Lumigo-specific configurationsThelumigo_opentelemetrypackage additionally supports the following configuration options as environment variables:LUMIGO_TRACER_TOKEN: [Required] Required configuration to send data to Lumigo; you will find the right value in Lumigo underSettings -> Tracing -> Manual tracing.LUMIGO_DEBUG=true: Enables debug loggingLUMIGO_DEBUG_SPANDUMP: path to a local file where to write a local copy of the spans that will be sent to Lumigo; this option handy for local testing butshould not be used in productionunless you are instructed to do so by Lumigo support.LUMIGO_SECRET_MASKING_REGEX=["regex1", "regex2"]: Prevents Lumigo from sending keys that match the supplied regular expressions. All regular expressions are case-insensitive. By default, Lumigo applies the following regular expressions:[".*pass.*", ".*key.*", ".*secret.*", ".*credential.*", ".*passphrase.*"].We support more granular masking using the following parameters. If not given, the above configuration is the fallback:LUMIGO_SECRET_MASKING_REGEX_HTTP_REQUEST_BODIES,LUMIGO_SECRET_MASKING_REGEX_HTTP_REQUEST_HEADERS,LUMIGO_SECRET_MASKING_REGEX_HTTP_RESPONSE_BODIES,LUMIGO_SECRET_MASKING_REGEX_HTTP_RESPONSE_HEADERS,LUMIGO_SECRET_MASKING_REGEX_HTTP_QUERY_PARAMS,LUMIGO_SECRET_MASKING_REGEX_ENVIRONMENT.LUMIGO_SWITCH_OFF=true: This option disables the Lumigo OpenTelemetry distro entirely; no instrumentation will be injected, no tracing data will be collected.LUMIGO_REPORT_DEPENDENCIES=false: This option disables the built-in dependency reporting to Lumigo SaaS. For more information, refer to theAutomated dependency reportingsection.LUMIGO_AUTO_FILTER_EMPTY_SQS: This option enables the automatic filtering of empty SQS messages from being sent to Lumigo SaaS. For more information, refer to theFiltering out empty SQS messagessection.LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX='["regex1", "regex2"]': This option enables the filtering of client and server endpoints through regular expression searches. Fine-tune your settings via the following environment variables, which work in conjunction withLUMIGO_FILTER_HTTP_ENDPOINTS_REGEXfor a specific span type:LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_SERVERapplies the regular expression search exclusively to server spans. Searching is performed against the following attributes on a span:url.pathandhttp.target.LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENTapplies the regular expression search exclusively to client spans. Searching is performed against the following attributes on a span:url.fullandhttp.url.For more information check outFiltering http endpoints.Execution TagsExecution Tagsallow you to dynamically add dimensions to your invocations so that they can be identified, searched for, and filtered in Lumigo.
For example: in multi-tenanted systems, execution tags are often used to mark with the identifiers of the end-users that trigger them for analysis (e.g.,Explore view) and alerting purposes.Creating Execution TagsIn the Lumigo OpenTelemetry Distro for Python, execution tags are represented asspan attributesand, specifically, as span attributes with thelumigo.execution_tags.prefix.
For example, you could add an execution tag as follows:fromopentelemetry.traceimportget_current_spanget_current_span().set_attribute('lumigo.execution_tags.foo','bar')Notice that, using OpenTelemetry'sget_current_span()API, you do not need to keep track of the current span, you can get it at any point of your program execution.In OpenTelemetry, span attributes can bestrings,numbers(double precision floating point or signed 64 bit integer),booleans(a.k.a. "primitive types"), and arrays of one primitive type (e.g., an array of string, and array of numbers or an array of booleans).
In Lumigo, booleans and numbers are transformed to strings.IMPORTANT:If you use theSpan.set_attributeAPI multiple timeson the same spanto set values for the same key multiple values, you may override previous values rather than adding to them:fromopentelemetry.traceimportget_current_spanget_current_span().set_attribute('lumigo.execution_tags.foo','bar')get_current_span().set_attribute('lumigo.execution_tags.foo','baz')In the snippets above, thefooexecution tag will have in Lumigo only thebazvalue!
Multiple values for an execution tag are supported as follows:fromopentelemetry.traceimportget_current_spanget_current_span().set_attribute('lumigo.execution_tags.foo',['bar','baz'])Tuples are also supported to specify multiple values for an execution tag:fromopentelemetry.traceimportget_current_spanget_current_span().set_attribute('lumigo.execution_tags.bar',('baz','xyz',))The snippets above will produce in Lumigo thefootag having bothbarandbazvalues.
Another option to set multiple values is settingexecution Tags in different spans of an invocation.Execution Tags in different spans of an invocationIn Lumigo, multiple spans may be merged together into one invocation, which is the entry that you see, for example, in theExplore view.
The invocation will include all execution tags on all its spans, and merge their values:fromopentelemetryimporttracetrace.get_current_span().set_attribute('lumigo.execution_tags.foo','bar')tracer=trace.get_tracer(__name__)withtracer.start_as_current_span('child_span')aschild_span:child_span.set_attribute('lumigo.execution_tags.foo','baz')In the examples above, the invocation in Lumigo resulting from executing the code will have bothbarandbazvalues associated with thefooexecution tag.
Which spans are merged in the same invocation depends on the parent-child relations among those spans.
Explaining this topic is outside the scope of this documentation; a good first read to get deeper into the topic is theTracesdocumentation of OpenTelemetry.
In case your execution tags on different spans appear on different invocations than what you would expect, get in touch withLumigo support.Execution Tag LimitationsUp to 50 execution tag keys per invocation in Lumigo, irrespective of how many spans are part of the invocation or how many values each execution tag has.Thekeyof an execution tag cannot contain the.character; for example:lumigo.execution_tags.my.tagis not a valid tag. The OpenTelemetrySpan.set_attribute()API will not fail or log warnings, but that will be displayed asmyin Lumigo.Each execution tag key can be at most 50 characters long; thelumigo.execution_tags.prefix doesnotcount against the 50 characters limit.Each execution tag value can be at most 70 characters long.Programmatic ErrorsProgrammatic Errorsallow you to customize errors, monitor and troubleshoot issues that should not necessarily interfere with the service.
For example, an application tries to remove a user who doesn't exist. These custom errors can be captured by adding just a few lines of additional code to your application.Programmatic Errors indicating that a non-fatal error occurred, such as an application error. You can log programmatic errors, track custom error issues, and triggerAlerts.Creating a Programmatic ErrorProgrammatic errors are created by addingspan eventswith a custom attribute being set with the key namelumigo.type.For example, you could add a programmatic error as follows:fromopentelemetry.traceimportget_current_spanget_current_span().add_event('<error-message>',{'lumigo.type':'<error-type>'})Supported runtimescpython: 3.7.x, 3.8.x, 3.9.x, 3.10.x, 3.11.xSupported packagesInstrumentationPackageSupported Versions3.73.83.93.103.11botocoreboto31.17.22~1.33.131.17.22~1.34.491.17.22~1.34.491.17.22~1.34.491.17.22~1.34.49djangodjango3.2.1~3.2.243.2.1~3.2.243.2.1~3.2.243.2.1~3.2.243.2.1~3.2.243.24.0.1~4.2.104.0.1~4.2.104.0.1~4.2.104.0.1~4.2.103.23.25.0.1~5.0.25.0.1~5.0.24.04.03.23.24.0.a14.0.a14.04.04.0.b14.0.b14.0.a14.0.a14.0.rc14.0.rc14.0.b14.0.b14.14.14.0.rc14.0.rc14.1.a14.1.a14.14.14.1.b14.1.b14.1.a14.1.a14.1.rc14.1.rc14.1.b14.1.b14.1rc14.1rc14.1.rc14.1.rc14.24.24.1rc14.1rc14.2.a14.2.a14.24.24.2.b14.2.b14.2.a14.2.a14.2.rc14.2.rc14.2.b14.2.b14.2rc14.2rc14.2.rc14.2.rc14.2rc14.2rc15.05.05.0rc15.0rc1fastapifastapi0.56.1~0.100.00.56.1~0.100.00.56.1~0.100.00.56.1~0.100.00.56.1~0.100.00.100.0b2~0.103.20.100.0b2~0.110.00.100.0b2~0.110.00.100.0b2~0.110.00.100.0b2~0.110.0uvicorn0.11.3~0.22.00.11.3~0.22.00.11.3~0.22.00.11.3~0.22.00.12.0~0.22.00.24.0~0.27.10.24.0~0.27.10.24.0~0.27.10.24.0~0.27.1flaskflask2.0.0~2.2.52.0.0~2.2.52.0.0~2.2.52.0.0~2.2.52.0.0~2.2.5grpciogrpcio1.45.0~1.62.0rc11.45.0~1.62.0rc11.45.0~1.62.0rc11.45.0~1.62.0rc11.49.0~1.62.0rc1kafka_pythonkafka_python2.0.0~2.0.22.0.0~2.0.22.0.0~2.0.22.0.0~2.0.22.0.0~2.0.2pikapika1.0.01.0.01.0.01.0.01.0.01.0.1~1.3.01.0.1~1.3.01.0.1~1.3.01.0.1~1.3.01.0.1~1.3.01.3.0rc5~1.3.21.3.0rc5~1.3.21.3.0rc5~1.3.21.3.0rc5~1.3.21.3.0rc5~1.3.2psycopgpsycopg3.1.1~3.1.183.1.1~3.1.183.1.1~3.1.183.1.1~3.1.183.1.1~3.1.183.13.13.13.13.1psycopg-binary3.1.1~3.1.183.1.1~3.1.183.1.1~3.1.183.1.1~3.1.183.1.4~3.1.183.13.13.13.1psycopg2psycopg2-binary2.7.5~2.9.92.8.1~2.9.92.8.1~2.9.92.8.1~2.8.62.9.5~2.9.92.82.82.82.9.5~2.9.92.92.92.92.8psycopg22.7.5~2.9.92.8.1~2.9.92.8.1~2.9.92.8.1~2.8.62.9.5~2.9.92.82.82.82.9.5~2.9.92.92.92.92.8pymongopymongo3.1.1~3.3.13.1.1~3.3.13.1.1~3.3.13.1.1~3.3.13.1.1~3.3.13.5.0~3.13.03.5.0~3.13.03.5.0~3.13.03.5.0~3.13.03.5.0~3.13.04.0.1~4.6.24.0.1~4.6.24.0.1~4.6.24.0.1~4.6.24.0.1~4.6.23.13.13.13.13.13.23.23.23.23.24.04.04.04.04.0pymysqlpymysql0.9.0~0.10.10.9.0~0.10.10.9.0~0.10.10.9.0~0.10.10.9.0~0.10.11.0.0~1.0.31.0.0~1.0.31.0.0~1.0.31.0.0~1.0.31.0.0~1.0.31.1.0~1.1.0rc21.1.0~1.1.0rc21.1.0~1.1.0rc21.1.0~1.1.0rc21.1.0~1.1.0rc2redisredis4.1.1~4.2.04.1.1~4.2.04.1.1~4.2.04.1.1~4.2.04.1.1~4.2.04.2.1~4.6.04.2.1~4.6.04.2.1~4.6.04.2.1~4.6.04.2.1~4.6.05.0.0~5.1.0a15.0.0~5.1.0b35.0.0~5.1.0b35.0.0~5.1.0b35.0.0~5.1.0b3Automated dependency reportingTo provide better support and better data-driven product decisions with respect to which packages to support next, the Lumigo OpenTelemetry Distro for Python will report to Lumigo SaaS on startup the packages and their versions used in this application, together with the OpenTelemetry resource data to enable analytics in terms of which platforms use which dependencies.The data uploaded to Lumigo is a set of key-value pairs with package name and version.
Similar is available through the tracing data sent to Lumigo, except that this aims at covering dependencies for which the Lumigo OpenTelemetry Distro for Python does not have instrumentation (yet?).
Lumigo's only goal for these analytics data is to be able to give you the instrumentations you need without you needing to tell us!The dependencies data is sent only when aLUMIGO_TRACER_TOKENis present in the process environment, and it can be opted out via theLUMIGO_REPORT_DEPENDENCIES=falseenvironment variable.Baseline setupThe Lumigo OpenTelemetry Distro will automatically create the following OpenTelemetry constructs provided to aTraceProvider.Resource attributesSDK resource attributesThe attributes from the default resource:telemetry.sdk.language:pythontelemetry.sdk.name:opentelemetrytelemetry.sdk.version: depends on the version of theopentelemetry-sdkincluded in thedependenciesThelumigo.distro.versioncontaining the version of the Lumigo OpenTelemetry Distro for Python as specified in theVERSION fileProcess resource attributesThe followingprocess.runtime.*attributes as specified in theProcess Semantic Conventions:process.runtime.descriptionprocess.runtime.nameprocess.runtime.versionA non-standardprocess.environresource attribute, containing a stringified representation of the process environment, with environment variables scrubbed based on theLUMIGO_SECRET_MASKING_REGEXconfiguration.Amazon ECS resource attributesIf the instrumented Python application is running on the Amazon Elastic Container Service (ECS):cloud.providerattribute with valueawscloud.platformwith valueaws_ecscontainer.namewith the hostname of the ECS Task containercontainer.idwith the ID of the Docker container (based on the cgroup id)If the ECS task uses the ECS agent v1.4.0, and has therefore access to theTask metadata endpoint version 4, the following experimental attributes as specified in theAWS ECS Resource Attributesspecification:aws.ecs.container.arnaws.ecs.cluster.arnaws.ecs.launchtypeaws.ecs.task.arnaws.ecs.task.familyaws.ecs.task.revisionKubernetes resource attributesk8s.pod.uidwith the Pod identifier, supported for both cgroups v1 and v2Span exportersIf theLUMIGO_TRACER_TOKENenvironment variable is set: aBatchSpanProcessor, which uses anOTLPSpanExporterto push tracing data to LumigoIf theLUMIGO_DEBUG_SPANDUMPenvironment variable is set: aSimpleSpanProcessor, which uses anConsoleSpanExporterto save to file the spans collected. Do not use this in production!SDK configurationThe followingSDK environment variablesare supported:OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMITOTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT** If theOTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMITenvironment variable is not set, the span attribute size limit will be taken fromOTEL_ATTRIBUTE_VALUE_LENGTH_LIMITenvironment variable. The default size limit when both are not set is 2048.Advanced use casesAccess to the TracerProviderThe Lumigo OpenTelemetry Distro provides access to theTracerProviderit configures (see theBaseline setupsection for more information) through thetracer_providerattribute of thelumigo_opentelemetrypackage:fromlumigo_opentelemetryimporttracer_provider# Do here stuff like adding span processorsEnsure spans are flushed to Lumigo before shutdownFor short-running processes, theBatchProcessorconfigured by the Lumigo OpenTelemetry Distro may not ensure that the tracing data are sent to Lumigo (see theBaseline setupsection for more information).
Through the access to thetracer_provider, however, it is possible to ensure that all spans are flushed to Lumigo as follows:fromlumigo_opentelemetryimporttracer_provider# Do some logictracer_provider.force_flush()# Now the Python process can terminate, with all the spans closed so far sent to LumigoConsuming SQS messages with Boto3 receive_messageMessaging instrumentations that retrieve messages from queues tend to be counter-intuitive for end-users: when retrieving one or more messages from the queue, one would naturally expect that all calls doneusing data from those messages, e.g., sending their content to a database or another queue, would result in spans that are children of the describing the retrieving of those messages.Consider the following scenario, which is supported by theboto3SQSreceive_messageinstrumentation of the Lumigo OpenTelemetry Distro for Python:fromopentelemetryimporttracetracer=trace.get_tracer(__name__)response=client.receive_message(...)# Instrumentation creates a `span_0` spanformessageinresponse.get("Messages",[]):# The SQS.ReceiveMessage span is active in this scopewithtracer.start_as_current_span("span_1"):# span_0 is the parent of span_1do_something()Without the scope provided by the iterator overresponse["Messages"],span_1would be without a parent span, and that would result in a separate invocation and a separate transaction in Lumigo.Filtering out empty SQS messagesA common pattern in SQS-based applications is to continuously poll an SQS queue for messages,
and to process them as they arrive.
In order not to clutter the Lumigo platform with empty SQS polling messages, the default behavior is to filter them
out from being sent to Lumigo.You can change this behavior by setting the boolean environment variableLUMIGO_AUTO_FILTER_EMPTY_SQStofalse.
The possible variations are:LUMIGO_AUTO_FILTER_EMPTY_SQS=truefilter out empty SQS polling messagesLUMIGO_AUTO_FILTER_EMPTY_SQS=falsedo not filter out empty SQS polling messagesNo environment variable set (default): filter out empty SQS polling messagesFiltering http endpointsYou can selectively filter spans based on HTTP server/client endpoints for various components, not limited to web frameworks.Global filteringSet theLUMIGO_FILTER_HTTP_ENDPOINTS_REGEXenvironment variable to a list of regex strings. Spans with matching server/client endpoints will not be traced.Specific FilteringFor exclusive server (inbound) or client (outbound) span filtering, use the environment variables:LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_SERVERLUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENTNotes:the environment variable must be a valid JSON array of strings, so if you want to match endpoint with the hostnamegoogle.comthe environment variable value should be["google\\.com"].If we are filtering out an HTTP call to an opentelemetry traced component, every subsequent invocation made by that
component won't be traced either.Examples:Filtering out every incoming HTTP request to the/loginendpoint (will also match requests such as/login?user=foo,/login/bar))):LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_SERVER=["\\/login"]Filtering out every outgoing HTTP request to thegoogle.comdomain (will also match requests such asgoogle.com/foo,bar.google.com):LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENT=["google\\.com"]'Filtering out every outgoing HTTP request tohttps://www.google.com(will also match requests such ashttps://www.google.com/,https://www.google.com/foo)LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX_CLIENT=["https:\\/\\/www\\.google\\.com"]Filtering out every HTTP request (incoming or outgoing) with the wordlogin:LUMIGO_FILTER_HTTP_ENDPOINTS_REGEX=["login"]ContributingFor guidelines on contributing, please seeCONTRIBUTING.md.