Package: troncos
🪵TroncosCollection of Python logging, tracing and profiling toolsEtymologyInstallationTracingProfilingLoggingEtymology"Troncos" is the plural of the spanish word "Tronco", which translates to "trunk" or "log".Installation#Withpippip install troncosTracingTroncos is designed to take advantage ofddtracemade by DataDog.The ddtrace docs can be foundhere.Best practices for tracesis a good guide to get started.Span vs resource attributesAspan attributeis a key/value pair that provides context for its span.Aresource attributeis a key/value pair that describes the context of how the span was collected.For more information, read theAttribute and Resourcesections in the OpenTelemetry specification.Enabling the tracerConfigure ddtrace as usual and runconfigure_tracerto send spans to Tempo.This is typically done insettings.pyof you want to profile a Django application,
or in__init__.pyin the root project package.TRACE_HOSTis usually the host IP of the K8s pod,TRACE_PORTis usually 4318
when the Grafana agent is used to collect spans using HTTP.importddtracefromtroncos.tracingimportconfigure_tracer# Configure tracer as described in the ddtrace docs.ddtrace.config.django["service_name"]='SERVICE_NAME'# These are added as span attributesddtrace.tracer.set_tags(tags={"key":"value",})# Patch third-party modulesddtrace.patch_all()# Configure the ddtrace tracer to send traces to Tempo.TRACE_HOST="127.0.0.1"# Usually obtained from env variables.TRACE_PORT="4318"configure_tracer(enabled=False,# Set to True when TRACE_HOST is configured.service_name='SERVICE_NAME',endpoint=f"http://{TRACE_HOST}:{TRACE_PORT}/v1/traces",resource_attributes={"app":"app","component":"component","role":"role","tenant":"tenant","owner":"owner","version":"version",})ddtrace also uses env variables to configure the service name, environment and version etc.Add the following environment variables to your application.DD_ENV="{{ environment }}"DD_SERVICE="{{ app }}"DD_VERSION="{{ version }}"#tracecontext/w3cisusuallyusedtopropagatedistributedtracesacrossservices.DD_TRACE_PROPAGATION_STYLE_EXTRACT="tracecontext"DD_TRACE_PROPAGATION_STYLE_INJECT="tracecontext"Debugging during developmentBy setting the environment variableOTEL_TRACE_DEBUG=Trueyou will enable traces
to be printed tostdoutvia the ConsoleSpanExporter as well as through http/grpc.
Also specifyingOTEL_TRACE_DEBUG_FILE=/some/file/pathwill output traces to the
specified file path instead of the console/stdout.Using the GRPC span exporterUsing the GRPC span exporter gives you significant performance gains.
If you are running a critical service with high load in production,
we recommend using GRPC.TRACE_PORTis usually 4317 when the Grafana agent is used to collect
spans using GRPC.poetry add troncos -E grpcor[tool.poetry.dependencies]troncos={version="?",extras=["grpc"]}fromtroncos.tracingimportconfigure_tracer,ExporterTRACE_HOST="127.0.0.1"# Usually obtained from env variables.TRACE_PORT="4317"configure_tracer(enabled=False,# Set to True when TRACE_HOST is configured.service_name='SERVICE_NAME',endpoint=f"http://{TRACE_HOST}:{TRACE_PORT}",exporter=Exporter.GRPC)Setting headers for the exporterfromtroncos.tracingimportconfigure_tracer,Exporter,ExporterTypeTRACE_HOST="127.0.0.1"# Usually obtained from env variables.TRACE_PORT="4317"configure_tracer(enabled=False,# Set to True when TRACE_HOST is configured.service_name='SERVICE_NAME',endpoint=f"http://{TRACE_HOST}:{TRACE_PORT}",exporter=Exporter(ExporterType.GRPC,headers={"my","header"}),)Instrument your codeManual instrumentation of your code is described in theddtrace docs.Add tracing context to your logAdding the tracing context to your log makes it easier to find relevant traces in Grafana.
Troncos include a Structlog processor designed to do this.importstructlogfromtroncos.contrib.structlog.processorsimporttrace_injection_processorstructlog.configure(processors=[trace_injection_processor,],)Logging of major actions in your applicationFinding relevant traces in Grafana can be difficult. One way to make finding the relevant traces
easier it to log every major action in your application. This typically means logging every
incoming HTTP request to your server or every task executed by your Celery worker.The structlog processor above needs to be enabled before logging your major actions is relevant.ASGI middlewareLog ASGI requests.fromstarlette.applicationsimportStarlettefromtroncos.contrib.asgi.logging.middlewareimportAsgiLoggingMiddlewareapplication=AsgiLoggingMiddleware(Starlette())Django middlewareLog Django requests. This is not needed if you run Django with ASGI and use the
ASGI middleware.MIDDLEWARE=["troncos.contrib.django.logging.middleware.DjangoLoggingMiddleware",...]Celery signals`
Log Celery tasks. Run the code bellow when you configure Celery.fromtroncos.contrib.celery.logging.signalsimport(connect_troncos_logging_celery_signals,)connect_troncos_logging_celery_signals()ProfilingEnabling the continuous py-spy profilerStart the profiler by running thestart_py_spy_profilermethod early in your application. This is
typically done insettings.pyof you want to profile a Django application, or in__init__.pyin the root project package.fromtroncos.profilingimportstart_py_spy_profilerstart_py_spy_profiler(server_address="http://127.0.0.1:4100")Enabling the ddtrace profilerStart the profiler by importing the profiler module early in your application. This is
typically done insettings.pyof you want to profile a Django application, or in__init__.pyin the root project package.importtroncos.profiling.autoSetup profile endpointUse one of the methods bellow based on your selected framework.DjangoAdd the profile view to the url config.fromdjango.urlsimportpathfromtroncos.contrib.django.profiling.viewsimportprofiling_viewurlpatterns=[path("/debug/pprof",profiling_view,name="profiling"),]StarletteAdd the profile view to your router.fromstarlette.routingimportRoutefromtroncos.contrib.starlette.profiling.viewsimportprofiling_viewroutes=[Route("/debug/pprof",profiling_view),]ASGIMount the generic ASGI profiling application. There is no generic way to do this,
please check the relevant ASGI framework documentation.fromtroncos.contrib.asgi.profiling.appimportprofiling_asgi_app# FastAPI examplefromfastapiimportFastAPIapp=FastAPI()app.mount("/debug/pprof",profiling_asgi_app)Verify setupYou can verify that your setup works with thepprofcli:pprof -http :6060 "http://localhost:8080/debug/pprof"Enable scrapingWhen you deploy your application, be sure to use the custom oda annotation for scraping:annotations:phlare.oda.com/port:"8080"phlare.oda.com/scrape:"true"LoggingTroncos is not designed to take control over your logger. But, we do include logging
related tools to make instrumenting your code easier.Configure StructlogTroncos contains a helper method that lets you configure Structlog.First, runpoetry add structlogto install structlog in your project.You can now replace your existing logger config withfromtroncos.contrib.structlogimportconfigure_structlogconfigure_structlog(format="json",level="INFO")Adding tracing context to your logTroncos has a Structlog processor that can be used to add thespan_idandtrace_idproperties to your log. More information can be found in theTracingsection in this document. This is used by theconfigure_structloghelper method
by default.Request logging middlewareFinding the relevant traces in Tempo and Grafana can be difficult. The request logging
middleware exist to make it easier to connect HTTP requests to traces. More information
can be found in theTracingsection in this document.