Package: gke-logging
gke-loggingUtilities for interacting with logging facilities in GKE workloadsInstallationRequirementsPython 3.7+Poetry(for development only)Install from PyPI (recommended)pip install gke-loggingInstalling from Githubpip install git+https://github.com/StationA/gke-logging.git#egg=gke-loggingInstalling from sourcegit clone https://github.com/StationA/gke-logging.git
cd gke-logging
poetry installUsagegke_logging.GKELoggingFormatterOne of the core components is theGKELoggingFormatter, which is an implementation of the built-inlogging.Formatterprotocol that translates alogging.LogRecordinto a JSON format that GKE's
logging infrastructure can understand. At a minimum, this enables any software running on GKE to
integrate structured logging simply by applying this formatter for your loggers, e.g.:importloggingfromgke_loggingimportGKELoggingFormatterLOGGER=logging.getLogger(__name__)h=logging.StreamHandler()h.setFormatter(GKELoggingFormatter())LOGGER.addHandler(h)LOGGER.setLevel(logging.INFO)# ...LOGGER.info("Look at me! I can haz GKE structured logging!")# Prints out: {"time": "2022-01-13T23:22:26.336686+00:00", "severity": "INFO", "message": "Look at me! I can haz GKE structured logging!", "logging.googleapis.com/sourceLocation": {"file": "test_log.py", "line": "14", "function": "<module>"}, "logging.googleapis.com/labels": {}}Furthermore, this formatter allows you to set app-level metadata to be sent along with each log
message, which is useful in order to better organize collected log data:# ...h.setFormatter(GKELoggingFormatter(default_labels=dict(app_id="my-cool-app",version="0.1.0")))# ...Also the formatter also allows you to add HTTP metadata to any logs that occur during the course of
a request. This enhances logs that are emitted during request-handling logic in APIs with additional
data. This functionality is primarily utilized in the includedGKELoggingMiddlewarein order to
provide basic access logs.gke_logging.asgi.GKELoggingMiddlewaregke_logging.asgi.GKELoggingMiddlewareis an ASGI middleware that emits basic access logs in
"common log format", with a default behavior that integrates with theGKELoggingFormatterto write
the access logs in a format that GKE's logging infrastructure better understands. By implementing
per the ASGI spec, this means it can work with any ASGI-compatible server, including FastAPI,
starlette, and ASGI implementations:fromfastapiimportFastAPIfromgke_logging.asgiimportGKELoggingMiddlewareapp=FastAPI()app.add_middleware(GKELoggingMiddleware)@app.get("/")defget_it()->str:return"OK"Additionally, because this middleware integrates withgke_logging.contextbindings, it enables any
logger used during the course of handling a request to emit logs that also contain request-time
data, e.g. request URL, user-agent, response latency, etc.importloggingfromfastapiimportFastAPIfromgke_loggingimportGKELoggingFormatterfromgke_logging.asgiimportGKELoggingMiddlewareapp=FastAPI()app.add_middleware(GKELoggingMiddleware)root_logger=logging.getLogger()h=logging.StreamHandler()h.setFormatter(GKELoggingFormatter())root_logger.setLevel(logging.INFO)root_logger.addHandler(h)@app.get("/")defget_it()->str:# Any log records created during request-handling will be enriched with other HTTP request dataroot_logger.info("TEST")return"OK"gke_logging.contextIn order to control additional metadata labels for log records that correspond to one logical
operation, e.g. an HTTP request, a batch job operation, etc., you should use the helper functions
exported ingke_logging.context:importloggingfromcontextvarsimportcopy_contextfromgke_loggingimportGKELoggingFormatterfromgke_logging.contextimportset_labelsLOGGER=logging.getLogger(__name__)h=logging.StreamHandler()h.setFormatter(GKELoggingFormatter())LOGGER.addHandler(h)LOGGER.setLevel(logging.INFO)# ...defrun_job(job_id:str):set_labels(job_id=job_id)LOGGER.info("TEST")ctx=copy_context()foriinrange(10):ctx.run(run_job,f"{i+1}")BecauseContextVars bind natively to Python'sasyncio, you can use these same helper
functions within asynchronous tasks in a similar fashion.Additional examplesAdditional usage examples can be found inexamples/ContributingWhen contributing to this repository, please follow the steps below:Fork the repositorySubmit your patch in one commit, or a series of well-defined commitsSubmit your pull request and make sure you reference the issue you are addressing