Reference example While there are several signals to consider, in practice, the three most commonly used are logs, metrics, and traces. In this chapter, we will use a running reference example, a simple Go program called the echo service, exposing an HTTP interface, to see the three common signal types in action. Change to the ch02/base/ directory, where you find the main.go program shown in listing 2.1. As we progress, we will instrument our reference example with different signals with the goal to consume and interact with the signals using open source tooling, including Grafana, Prometheus, and Jaeger. package main import ( "fmt" "log" "math/rand" "net/http" "time" ) func handleEcho(w http.ResponseWriter, r *http.Request) { message := r.URL.Query().Get("message") if rand.Intn(100) > 90 { Listing 2.1 Example Go program to be instrumented Retrieves the payload to echo from the query parameter message, with the result of being able to interact with the service by issuing an HTTP GET to /echo?message Simulates a 10% error case 16 CHAPTER 2 Signal types w.WriteHeader(http.StatusInternalServerError) return } fmt.Fprintf(w, message) } func main() { rand.Seed(time.Now().UnixNano()) http.HandleFunc("/echo", handleEcho) log.Fatal(http.ListenAndServe(":4242", nil)) } First, let’s build the echo service, as shown in the following listing. $ go build -o echosvc . Now, you can run the service with ./echosvc in the same directory. To invoke it via its HTTP API, you would use the following code (in a separate terminal session): $ curl localhost:4242/echo?message=test test By default, I will use Grafana as the destination, since it allows us to visualize and interact with a wide array of data sources in a uniform manner. This is will allow you to reproduce everything for free with open source. Now that we’ve laid out our reference example, let’s talk about a way to assess the costs of instrumentation, before we move to the instrument itself. 2.2 Assessing instrumentation costs To assess the costs of instrumentation per signal type, we will use a simple measure: the business logic to instrumentation (B2I) ratio. Listing 2.2 Building our example Go program Tip To see the signals going forward, in Grafana, you will need to put some load onto the echo service. One option is to do something like the following directly from the shell (calling the endpoint every second): while true ; do curl -s -o /dev/null localhost:4242/echo ; sleep 1 ; done Alternatively, you can use something more fancy, such as a full-blown testing tool like Salvo (https://github.com/tarekziade/salvo) or even K6 (https://k6.io/docs/testingguides/api-load-testing/). Return the exact payload we retrieved in 90% of the cases Register handler for the /echo path Launches the HTTP server that will serve on port 4242 on all available network interfaces Builds the executable, which is then available in the current directory as ./echosvc 2.3 Logs 17 There are two things worth mentioning concerning B2I:  It’s up to you to decide whether to consider telemetry libraries such as an OpenTelemetry SDK (https://opentelemetry.io/docs/instrumentation/) in your B2I ratio calculation. From a pragmatic point of view, focusing on your own code (excluding these dependencies) might make the most sense. This is what I will do in the remainder of the chapter.  The B2I ratio is a simple measure and should by no means serve as an indicator of how well your code is instrumented or tell you if your code is over-instrumented. Goodhart’s law (see Strathern and Marilyn, 1997, “Improving Ratings: Audit in the British University System,” European Review, John Wiley & Sons) states that when a measure becomes a target, it ceases to be a good measure—this applies here. Use the B2I ratio as a rough guide in a regression validation, rather than an absolute North star. With the theory and overview out of the way, let’s now review the three basic signal types: logs, metrics, and traces. In the remainder of the chapter, we will do the following for each signal type: 1 Define the signal characteristics and semantics. 2 Discuss how to instrument our example echo service. 3 Set up the telemetry to collect and route the signal. 4 Analyze costs (in terms of resource usage and efforts) and benefits of the signal. 5 Explain how the respective signal can contribute to an observability solution. 2.3 Logs Let’s begin by focusing on logs. Logs are signals with a textual payload that capture an event. An event is an umbrella term for anything of interest that happens in the code, be it on the happy path (a completed upload) or unexpected or erroneous behavior (“failed to store items in shopping basket”). The B2I ratio To calculate the B2I ratio, determine the number of lines of code (LOC) before adding an instrumentation (adding code for emitting signals for a signal type), and then determine the LOC after the instrumentation. The B2I ratio is then B2I = LOC_AFTER_INSTRUMENTATION/LOC_BEFORE_INSTRUMENTATION In an ideal world, the B2I ratio would be 1, representing zero instrumentation costs in the code. However, in reality, the more LOC you dedicate to instrumentation, the higher the B2I ratio is. For example, if your code has 3800 LOC and you added 400 LOC for instrumentation (say, to emit logs and metrics), then you’d end up with a B2I ratio of 1.105, from (3800 + 400) / 3800. 18 CHAPTER 2 Signal types Logs are timestamped and should be structured, ideally including some context (e.g., labels) that can be used for correlation with other signals. Usually, humans consume logs, either directly, by searching for an event, or indirectly, via correlation. NOTE There are many ways to refer to a single log event, including log lines, log items, log entries, and perhaps others I have not come across yet. To keep things simple, we will consistently use the term event here, but these terms are, by and large, interchangeable. In the context of logs, we usually differentiate between levels, which can represent the nature of a log event. Different programming languages, libraries, and frameworks have, over time, established conventions; however, the following are some common semantics:  Fine-grained logs events that are relevant for developing or fixing a bug (DEBUG)  Production-type log events, including the following: – Expected behavior (usually INFO) – Unexpected behavior, such as WARNING or ERROR There are several log standards and formats available, including line-oriented formats, such as syslog RFC 5424 (https://tools.ietf.org/html/rfc5424) and the Apache common log format (CLF; http://mng.bz/lWg8); JSON log formats (supported by most modern libraries); and binary formats, such as the journal file format (http://mng .bz/Bmw8), found in the context of systemd. To learn more, check out Graylog’s Logs Format Guide (http://mng.bz/d1nO). Some good practices in the context of logs include the following:  Consider what to log, with the goal of providing actionable insights (context is king).  Log to stdout rather than to a fixed location, such as a file. This enables log agents to collect and route your log events. See also the entry in the 12-factor methodology (https://12factor.net/logs) for some background.  Use labels to enhance the value of a log event. For example, you can use the component name or type, the kind of source, or the type of operation as labels to aid filtering for events. TIP If you’re wondering how you can “future proof” your strategies around logs, I recommend reading Ben Sigelman’s (cofounder of Lightstep, known for his work on OpenTelemetry, Google Dapper, and Monarch) 2001 Twitter thread, “Why Tracing Might Replace (Almost) All Logging” (http://mng.bz/ rWzD), in which he argues that in cloud-native systems, tracing will replace logging for the brunt of the workload via instrumentation about the transactions themselves (as opposed to logging about infrastructure or other resources). This makes sense, at least for a large number of use cases for logs in a transactional setup, and I want to encourage you to consider this kind of thinking and apply it to your environment, if you’re in a position to do so. Naturally, many environments are more on the brown field than green field 2.3 Logs 19 end of the spectrum, so it can be difficult or take a lot of time and energy to establish this tracing-centered (as opposed to logs-centered) approach. 2.3.1 Instrumentation Let’s now see logs in action. For this, we instrument our echo service so that it emits log events. The following listing shows our echo service instrumented with logs. package main import ( "fmt" "math/rand" "net/http" "time" log "github.com/sirupsen/logrus" ) func handleEcho(w http.ResponseWriter, r *http.Request) { message := r.URL.Query().Get("message") log.WithFields(log.Fields{ "service": "echo", }).Info("Got input ", message) if rand.Intn(100) > 90 { log.WithFields(log.Fields{ "service": "echo", }).Error("Something really bad happened :(") w.WriteHeader(500) return } fmt.Fprintf(w, message) } func main() { rand.Seed(time.Now().UnixNano()) log.SetFormatter(&log.JSONFormatter{}) http.HandleFunc("/echo", handleEcho) log.Fatal(http.ListenAndServe(":4242", nil)) } To build the service, you also need to define its dependencies via a file called go.mod (shown in the following listing). module co11yia/ch2/logs go 1.20 require github.com/sirupsen/logrus v1.8.1 Listing 2.3 Example Go program, instrumented to emit structured logs Listing 2.4 Example Go program dependencies file go.mod No matter what, always emit an INFO log line here that confirms the input value we got via the query parameter message. Note the "service": "echo" in the line above (and again in the error case below): this provides a context for the event, allowing you to pinpoint the service. Emit an ERROR log line in error case. Configure the logger to emit log lines formatted in JSON. Also, note that the logger we’re using here, logrus (https://github.com/Sirupsen/ logrus), defaults to output to stderr, something we have to consider when we consume the logs, later. 20 CHAPTER 2 Signal types Now, to pull in the echo service dependencies, run the following command: $ go mod vendor Complete the build by executing go build, as explained in listing 2.2. 2.3.2 Telemetry We are now in the position to execute the echo service locally and send the log events it emits during its execution to certain destinations, such as to the screen and into a file: $ ./echosvc 2>&1 | \ tee echo.log Next, we will use some open source tools, provided by Grafana Labs, to route and collect the logs. We capture the logs of our echo service with Promtail (http://mng .bz/V1J0) and ship them to Loki (https://grafana.com/docs/loki/latest/) as a destination (as a data source in Grafana). Using Grafana as the frontend, you can then view and query the logs as you desire. To set up our log scenario, use the Docker Compose manifest file shown in the following listing. It launches everything needed, from the destination to the frontend. version: "3.5" networks: loki: services: loki: image: grafana/loki:2.4.0 ports: - "3100:3100" command: -config.file=/etc/loki/local-config.yaml networks: - loki promtail: image: grafana/promtail:2.4.0 volumes: - ./promtail-config.yaml:/etc/promtail/config.yml - ./echo.log:/var/log/echo.log command: -config.file=/etc/promtail/config.yml networks: - loki grafana: image: grafana/grafana:latest ports: - "3000:3000" networks: - loki Listing 2.5 Using docker-compose.yaml to stand up Promtail, Loki, and Grafana Execute the echosvc and redirect its stderr output to stdout. Use the tee command to show the log lines on the terminal, at the same time appending the log lines to a file called echo.log. We will use this file as an input to ship to a destination in the next step. Defines the location of the Promtail config file. We map the host local file promtail-config.yaml into the container to /etc/promtail/config.yml, where Promtail expects to find it. Maps the host local file echo.log (which our echo services logs to) into the container to /var/log/echo.log. We will use that in listing 2.6 to define the log processing and routing pipeline. 2.3 Logs 21 TIP While using Docker Compose to develop locally or test apps is certainly handy, for a production setup, Kubernetes is much more useful. For example, you can install the Promtail, Loki, and Grafana stack via Helm charts or using Tanka (https://tanka.dev/). Now that we have set up the framework for launching the logs destination (Loki), the telemetry agent (Promtail), and the frontend (Grafana) via our Docker Compose file (listing 2.5) we need one more thing in place: configuring Promtail, our log router/ ingester. The Docker Compose manifest references (in line 15) this Promtail configuration file, which we call promtail-config.yaml, as shown in the following listing. server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: app static_configs: - targets: - localhost labels: job: echosvc __path__: /var/log/echo.log pipeline_stages: - json: expressions: level: level payload: msg timestamp: time service: service - labels: level: service: Assuming you stored the content of listing 2.5 in a file called docker-compose.yaml, it’s straightforward to bring up the stack using the following code: $ docker-compose -f docker-compose.yaml up Creating network "logs_loki" with the default driver Pulling loki (grafana/loki:2.4.0)... 2.4.0: Pulling from grafana/loki ... Status: Downloaded newer image for grafana/promtail:2.4.0 Creating logs_loki_1 ... done Listing 2.6 Promtail configuration file promtail-config.yaml This defines the input for Promtail. Here, we tell it to ingest logs from a file called /var/log/echo.log (which we mounted from the host via a volume in line 16 of the Docker Compose manifest). This is the first part of the Promtail pipeline (http://mng.bz/x48e). It maps the input JSON fields that our echo service emits (e.g., service or level, the right-hand side here, the values) to labels we want to filter for in Loki; for example, service or level (the same names here, but you can, and often must, normalize keys, and that’s a good place to do this). Here, we define the labels we want to index or expose. 22 CHAPTER 2 Signal types Creating logs_promtail_1 ... done Creating logs_grafana_1 ... done Attaching to logs_loki_1, logs_grafana_1, logs_promtail_1 ... Now, you can head over to http:/ /localhost:3000/ in your browser to log into Grafana. Use the following defaults (be sure to skip the password change request after you enter the credentials):  Username: admin  Password: admin

Q- like this exmple by i want more python oriented step by step tutor that do the same think herre