cwielenb/CNCF-Dataset huggingface dataset:
Project: SkyWalking (Observability and Analysis / Observability)
File: How-to-bump-up-zipkin.md

Because SkyWalking embeds Zipkin Lens UI as a part of the SkyWalking UI, and Zipkin Lens UI contains a lot of other front-end dependencies that we also distribute in SkyWalking binary tars, so we have to take care of the dependencies' licenses when we bump up the Zipkin Lens dependency. Make sure to do the following steps when you bump up the Zipkin Lens dependency: Clone the Zipkin project into a directory. ```shell ZIPKIN_VERSION=<the Zipkin version you want to bump to> git clone https://github.com/openzipkin/zipkin && cd zipkin git checkout $ZIPKIN_VERSION cd zipkin-lens ``` Create `.licenserc.yaml` with the following content. ```shell cat > .licenserc.yaml << EOF header: license: spdx-id: Apache-2.0 copyright-owner: Apache Software Foundation dependency: files: package.json licenses: name: cli-table version: 0.3.1 license: MIT name: domutils version: 1.5.1 license: BSD-2-Clause name: rework version: 1.0.1 license: MIT EOF ``` Create license template `LICENSE.tpl` with the following content. ``` {{ range .Groups }} ======================================================================== {{ .LicenseID }} licenses ======================================================================== The following components are provided under the {{ .LicenseID }} License. See project link for details. {{- if eq .LicenseID "Apache-2.0" }} The text of each license is the standard Apache 2.0 license. {{- else }} The text of each license is also included in licenses/LICENSE-[project].txt. {{ end }} {{- range .Deps }} https://npmjs.com/package/{{ .Name }}/v/{{ .Version }} {{ .Version }} {{ .LicenseID }} {{- end }} {{ end }} ``` Make sure you're using the supported NodeJS version and NPM version. ```shell node -v npm -v ``` Run the following command to generate the license file. ```shell license-eye dependency resolve --summary LICENSE.tpl ``` Copy the generated file `LICENSE` to replace the `zipkin-LICENSE` in SkyWalking repo. Note: if there are dependencies that license-eye failed to identify the license, you should manually identify the license and add it to the step above in `.licenserc.yaml`.
Project: Grafana Mimir (Observability and Analysis / Observability)
File: compaction-planner.md

You can use the compaction planner tool to troubleshoot the Grafana Mimir compactor. The compaction planner tool loads a bucket index from the object storage (for example, Google Cloud Storage) and shows compaction jobs that would be generated by the split-and-merge compaction strategy. Unlike the compactor, which scans the bucket for blocks, the tool informs itself from the bucket index. This means that the jobs listed by the compaction planner tool might not reflect the current state of the bucket. The compactor updates the bucket index every `-compactor.cleanup-interval`. Configure the compaction planner tool with at least the following parameters: The tenant ID (via the `-user` flag) Which storage backend to use (via the `-backend` flag) Those required by the chosen storage backend (for example, the `gcs` backend requires the `-gcs.bucket-name` flag) In addition, you might need to adjust shard count (via `-shard-count`) and split groups (via `-split-groups`) to match the corresponding split-and-merge compactor parameters. The compaction planner tool cannot automatically deduce these parameters. You can modify the order of the job via the `-sorting` flag. The output in the following example lists compaction jobs that the Grafana Mimir compactor would work on, provided that the compactor configuration is equivalent to the compaction planner tool. The compactor would work on the jobs in the order they are listed. ``` $ ./compaction-planner -backend=gcs -gcs.bucket-name=blocks-bucket -user=10428 -split-groups=1 -shard-count=4 2022-02-04 09:52:05.506428 I | Using index from 2022-02-04T08:38:27Z 2022-02-04 09:52:05.533720 I | Filtering using *compactor.NoCompactionMarkFilter Job No. Start Time End Time Blocks Job Key 1 2022-02-04T02:00:00Z 2022-02-04T04:00:00Z 4 0@6672437747845546250-merge-2of4-1643940000000-1643947200000 2 2022-02-04T02:00:00Z 2022-02-04T04:00:00Z 4 0@6672437747845546250-merge-4of4-1643940000000-1643947200000 3 2022-02-04T04:00:00Z 2022-02-04T06:00:00Z 4 0@6672437747845546250-merge-1of4-1643947200000-1643954400000 4 2022-02-04T04:00:00Z 2022-02-04T06:00:00Z 4 0@6672437747845546250-merge-2of4-1643947200000-1643954400000 5 2022-02-04T04:00:00Z 2022-02-04T06:00:00Z 4 0@6672437747845546250-merge-3of4-1643947200000-1643954400000 6 2022-02-04T04:00:00Z 2022-02-04T06:00:00Z 4 0@6672437747845546250-merge-4of4-1643947200000-1643954400000 ```
Project: Grafana Mimir (Observability and Analysis / Observability)
File: query-blocking.md

title: "0001: Query blocking" description: "Query blocking" draft: true Author: @wilfriedroset Date: 05/2023 Sponsor(s): @dimitarvdimitrov Type: Feature Status: Accepted Related issues/PRs: https://github.com/grafana/mimir/pull/5609 The `query-frontend` is a component that is responsible for query acceleration through splitting, sharding, caching, and other techniques. It is also responsible for enforcing limits on queries such as total query time range. In certain situations, you may not be able to control the queries being sent to your Mimir installation. These queries may be intentionally or unintentionally expensive to run, and they may affect the overall stability or cost of running your service. Prevent harmful queries to be processed by Mimir's read path. Rewrite queries in a non harmful way. Optimize queries on the fly. Blocked remote read. Block similar queries with a single definition. Blocking `metricname{label1=value1, label2=value2}` does not automatically block `metricname{label2=value2, label1=value1}`. Lint or re-format queries. Current tenants isolation through `shuffle-sharding` are sufficient to keep the read path healthy. Mimir's operators can use limits such as `maxfetchedchunksperquery`, `maxquerierspertenant` or `storegatewaytenantshard_size` to limits tenants processing capabilities. Pros: Nothing to be done Cons: Harmful queries are still processed by Mimir but in a limited part of the cluster Based on the configuration, Mimir's ) ran by the `query-frontend` would be responsible for blocking queries. Operators would be able to block queries either for all tenants or on a per-tenant basis via the runtime configuration. ```yaml overrides: "tenant-id": blocked_queries: pattern: 'sum(rate({env="prod"}[1m]))' pattern: '="\.\"' # all queries with a label value `.` regex: true ``` The order of patterns is preserved, so the first matching pattern will be used. PromQL queries might contains special characters used in regexp (`{},-`) or regexp. As such, a list of blocked queries, either regex or non-regex, is preferred over a single regex. This would avoid a eventual escaping problems. To accommodate multi-lines promQL queries, Mimir's operators can use YAML multi-line strings. Blocked queries must be logged, as well as counted in the `cortexqueryfrontendblockedqueries_total` metric on a per-tenant basis. Pros: Mimir's operators have a new way to ensure the read path stability. Blocked queries can be defined either at the cluster level or tenant level. Cons: Depending on the configuration assessing if a query should be blocked or not could be expensive, however the cost saving out weight this cons. Any changes made to the queries would allow the tenants to escape the protection. Loki has a .
Project: OpenTelemetry (Observability and Analysis / Observability)
File: llm-semconv.md

This workgroup aims to identify and define a standardized vocabulary and specification for observability of Large Language Model (LLM) based applications. The effort would explore various LLM-based application APIs, LLM tasks and LLM workflows for multistage reasoning. Key areas include: Evaluating existing vendor-specific conventions (which may not be currently standardized but implementation specific) Developing a standardized vocabulary for observability specific semantic conventions for single stage and multistage LLM workflows Initial scope focused on metrics and traces (which could later be expanded to other telemetry data signals) Example Use Cases: Debugging: Standardizing log data and error reporting for efficient error resolution in LLM applications. Evaluation: Collecting signal for evaluation of LLM performance, including response accuracy, context retention, and user feedback. Cost Tracking: Monitoring operational costs like computation time and API calls for resource optimization. AI Safety: Monitoring for ethical LLM use, including privacy compliance, bias detection, and secure data handling. For Spans: Instrumentation and debugging guidance for multi-step LLM applications, with recommendations on user privacy and intellectual property security, marking sensitive attributes and events as opt-in. For Metrics: Tracking model performance by collecting and correlating data like model names, versions, parameters, latency, and evaluation scores. Semantic conventions for LLMs, allowing flexibility for evolution in this fast-paced field. Reference prototypes demonstrating the application of these conventions in data collection, processing, and analysis. Recommended practices for monitoring LLM applications, including implementation strategies. We will gather further staff as the project is established. We need to make sure there is enough representation across both vendors and customers who will benefit from these semantic conventions. Project Leads: @drewby @nirga Sponsoring Members: @lmolkova @alolita Engineers: @nirga @galkleinman @sudivate @cartermp @gyliu513 Positions open for more engineers Maintainers: To be determined Alternating weekly meetings to accommodate different time zones: Week 1: Wednesdays at 8 AM PST Week 2: Wednesdays at 4 PM PST Meeting Calendar: Meeting Notes: Slack: Initial Draft: February 2024 Review and Refinement: April 2024 Prototypes for .NET, Python, JS: June 2024 Initial Merge Target: July 2024 Ongoing Review for Stability: Date TBD `LLM` Issue: PR: PR: - Closed/Replaced with 825 Project Board:
Project: Kiali (Observability and Analysis / Observability)
File: WORKING_WITH_OSSMC.md

:information_source: There is a . The OpenShift Service Mesh Console is a Webpack Plugin that integrates Kiali into the OpenShift Console. The official title of the project is "OpenShift Service Mesh Console" but you may see this abbreviated in documentation and code as "OSSMC", "ossmconsole", or "OSSM Console". The main component is a plugin based on OpenShift Console framework. Installing and enabling the plugin will add OpenShift Service Mesh support into the OpenShift Console. The new "Service Mesh" menu item and tabs allow you to interact with your mesh via the Kiali user interface. Note that OSSMC may also work with upstream Istio installed (as opposed to OpenShift Service Mesh). The main installation mechanism is the Kiali Operator. These are the things you need before developers can start working with the OpenShift Service Mesh Console: OpenShift cluster with OpenShift ServiceMesh or Istio installed. Kiali Server deployed in the cluster `oc` client available in the path `podman` or `docker` client available in the path The OpenShift Service Mesh Console will be installed by end users using the Kiali Operator. More details can be found in the . Developers can use the make targets found in the kiali repo (see below). Here is a summary for developers on how to get the Kiali Operator, Kiali Server, and OSSMC installed in your OpenShift cluster. First run `make cluster-status` to expose the internal image registry and get the docker or podman command needed to log into the internal image registry. Run the podman or docker login command that will log into the internal image registry. Build and push the OSSMC plugin image. You must do this from within your local repo of https://github.com/kiali/openshift-servicemesh-plugin - so clone that repo locally, and from your clone's root directory on your local machine, run `make cluster-push` and that will do what needs to be done. Build, push, and deploy the operator, Kiali, and plugin by running `make cluster-push operator-create kiali-create ossmconsole-create` When you are finished and you want to uninstall the operator, Kiali, and plugin, run `make operator-delete`.
Project: Netdata (Observability and Analysis / Observability)
File: methods.md

<!-- title: "Installation methods" description: "Netdata can be installed as a DEB/RPM package, a static binary, a docker container or from source" customediturl: https://github.com/netdata/netdata/edit/master/packaging/installer/methods/methods.md sidebar_label: "Installation methods" learn_status: "Published" learnrelpath: "Installation/Installation methods" sidebar_position: 30 --> Netdata can be installed: The picks the most appropriate method out of the first three for any system and is the recommended installation method, if you don't use containers. `kickstart.sh` can also be used for , suitable for air-gapped systems.
Project: Vector (Observability and Analysis / Observability)
File: 2020-03-12-papertrail-sink.md

date: "2020-04-13" title: "New Papertrail Sink" description: "Sink logs to the Papertrail logging service" authors: ["binarylogic"] pr_numbers: [1835] release: "0.9.0" hideonrelease_notes: true badges: type: "new feature" domains: ["sinks"] sinks: ["papertrail"] Tried, true, and simple. Our new lets you send logs to the logging service.
Project: Centreon (Observability and Analysis / Observability)
File: poller_pull_configuration.md

We are showing how to configure gorgone to manage that architecture: ```text Central server <- Distant Poller ``` In our case, we have the following configuration (need to adatp to your configuration). Central server: address: 10.30.2.203 Distant Poller: id: 6 (configured in Centreon interface as zmq*. You get it in the Centreon interface) address: 10.30.2.179 rsa public key thumbprint: nJSH9nZN2ugQeksHif7Jtv19RQA58yjxfX-Cpnhx09s The Distant Poller is already installed and Gorgone also. We configure the file /etc/centreon-gorgone/config.d/40-gorgoned.yaml: ```yaml name: distant-server description: Configuration for distant server gorgone: gorgonecore: id: 6 privkey: "/var/lib/centreon-gorgone/.keys/rsakey.priv.pem" pubkey: "/var/lib/centreon-gorgone/.keys/rsakey.pub.pem" modules: name: action package: gorgone::modules::core::action::hooks enable: true command_timeout: 30 whitelist_cmds: true allowed_cmds: ^sudo\s+(/bin/)?systemctl\s+(reload|restart)\s+(centengine|centreontrapd|cbd)\s*$ ^(sudo\s+)?(/usr/bin/)?service\s+(centengine|centreontrapd|cbd|cbd-sql)\s+(reload|restart)\s*$ ^/usr/sbin/centenginestats\s+-c\s+/etc/centreon-engine/centengine\.cfg\s*$ ^cat\s+/var/lib/centreon-engine/[a-zA-Z0-9\-]+-stats\.json\s*$ ^/usr/lib/centreon/plugins/.*$ ^/bin/perl /usr/share/centreon/bin/anomalydetection --seasonality >> /var/log/centreon/anomalydetection\.log 2>&1\s*$ ^/usr/bin/php -q /usr/share/centreon/cron/centreon-helios\.php >> /var/log/centreon-helios\.log 2>&1\s*$ ^centreon ^mkdir ^/usr/share/centreon/www/modules/centreon-autodiscovery-server/script/runsavediscovered_host ^/usr/share/centreon/bin/centreon -u \"centreon-gorgone\" -p \S+ -w -o CentreonWorker -a processQueue$ name: engine package: gorgone::modules::centreon::engine::hooks enable: true command_file: "/var/lib/centreon-engine/rw/centengine.cmd" name: pull package: "gorgone::modules::core::pull::hooks" enable: true target_type: tcp target_path: 10.30.2.203:5556 ping: 1 ``` The Central server is already installed and Gorgone also. We configure the file /etc/centreon-gorgone/config.d/40-gorgoned.yaml: ```yaml ... gorgone: gorgonecore: ... externalcomtype: tcp externalcompath: "*:5556" authorized_clients: key: nJSH9nZN2ugQeksHif7Jtv19RQA58yjxfX-Cpnhx09s ... modules: ... name: register package: "gorgone::modules::core::register::hooks" enable: true config_file: /etc/centreon-gorgone/nodes-register-override.yml ... ``` We create the file /etc/centreon-gorgone/nodes-register-override.yml: ```yaml nodes: id: 6 type: pull prevail: 1 ```
Project: Netdata (Observability and Analysis / Observability)
File: megacli_pd_predictive_failures.md

This alert indicates that one or more physical disks attached to the MegaCLI controller are experiencing predictive failures. A predictive failure is a warning that a hard disk may fail in the near future, even if it's still working normally. The failure prediction relies on the self-monitoring and analysis technology (S.M.A.R.T.) built into the disk drive. Make sure you have taken necessary backup steps before performing any action. Netdata is not liable for any loss or corruption of data, databases, or software. Identify the problematic drives: Use the following command to gather information about your virtual drives in all adapters: ``` megacli LDInfo -Lall -aALL ``` Determine the virtual drive and adapter reporting media errors. Consult the MegaRAID SAS Software User Guide [1]: Refer to Section 2.1.16 to check for issues with your drives. Refer to Section 7.18 to perform any appropriate actions on drives. Focus on Sections 7.18.2, 7.18.6, 7.18.7, 7.18.8, 7.18.11, and 7.18.14. Consider replacing the problematic disk(s) to prevent imminent failures and potential data loss.
Project: Grafana Mimir (Observability and Analysis / Observability)
File: how-integration-tests-work.md

Mimir integration tests are written in Go and based on a running Mimir and its dependencies in Docker containers and using the Go package for assertions. Integration tests run in CI for every PR, and can be easily executed locally during development (it just requires Docker). When integration tests run in CI, we build the Mimir Docker image based on the PR code and then run the integration tests against it. When running tests locally you should build the Mimir Docker image first: ``` make ./cmd/mimir/.uptodate ``` This will locally build the `grafana/mimir:latest` image used by integration tests. Whenever the Mimir code changes (`cmd/`, `pkg/` or vendors) you should rebuild the Mimir image, while it's not necessary to rebuild it while developing integration tests. Once the Docker image is built, you can run integration tests: ``` go test -v -tags=requires_docker ./integration/... ``` If you want to run a single test you can use a filter. For example, to only run `TestChunksStorageAllIndexBackends`: ``` go test -v -tags=requires_docker ./integration -run "^TestChunksStorageAllIndexBackends$" ``` When running all integration tests, the test process may time out before the tests complete. If this happens, you can increase `go test`'s default 10 minute timeout to something longer with the `-timeout` flag, for example: ``` go test -v -tags=requires_docker -timeout=20m ./integration/... ``` `MIMIR_IMAGE`<br /> Docker image used to run Mimir in integration tests (defaults to `grafana/mimir:latest`) `MIMIRTOOL_IMAGE`<br /> Docker image used to run `mimirtool` in integration tests (defaults to `grafana/mimirtool:latest`) `MIMIR_CHECKOUT_DIR`<br /> The absolute path of the Mimir repository local checkout (defaults to `$GOPATH/src/github.com/grafana/mimir`) `E2E_TEMP_DIR`<br /> The absolute path to a directory where the integration test will create an additional temporary directory to store files generated during the test. `E2E_NETWORK_NAME`<br /> Name of the docker network to create and use for integration tests. If no variable is set, defaults to `e2e-mimir-test`. Integration tests have `requiresdocker` tag (`// +build requiresdocker` line followed by empty line on top of Go files), to avoid running them unintentionally as they require Docker, e.g. by running `go test ./...` in main Mimir package. Each integration test runs in isolation. For each integration test, we do create a Docker network, start Mimir and its dependencies containers, push/query series to/from Mimir and run assertions on it. Once the test has done, both the Docker network and containers are terminated and deleted.
Project: Centreon (Observability and Analysis / Observability)
File: dbcleaner.md

This module aims to maintain the Gorgone daemon database by purging entries cyclically. The module is loaded by default. Adding it to the configuration will overload daemon default configuration. | Directive | Description | Default value | | : | :-- | : | | purgesessionstime | Time in seconds before deleting sessions in the `gorgone_identity` table | `3600` | | purgehistorytime | Time in seconds before deleting history in the `gorgone_history` table | `604800` | ```yaml name: dbcleaner package: "gorgone::modules::core::dbcleaner::hooks" enable: true purgesessionstime: 3600 purgehistorytime: 604800 ``` | Event | Description | | :- | :-- | | DBCLEANERREADY | Internal event to notify the core | No API endpoints.
Project: Checkmk (Observability and Analysis / Observability)
File: 14229.md

key | value | date | 2024-04-10T09:56:58+00:00 version | 2.4.0b1 class | fix edition | cre component | wato level | 1 compatible | yes Normally, ListOf elements embedded in other ListOf elements should contain a different magic value, otherwise errors will occur when processing forms. This werk here allows the use of identical magic values.
Project: Promscale (Observability and Analysis / Observability)
File: vacuum.md

The Promscale connector has a background job called the vacuum engine. The goal of the vacuum engine is to help prevent transaction id wraparound which can be a problem for high transaction workloads. Once a chunk is compressed, it is unlikely to be modified. Therefore, a compressed chunk should only need to be vacuumed once and never again. We want to vacuum a compressed chunk as soon as possible in hopes of freezing all the pages for the chunk. We want to freeze so that we don't risk burning through transaction ids too quickly under heavy ingest. The vacuum engine periodically wakes up. If it can grab an advisory lock (only one vacuum engine runs at a time per database regardless of the number of connectors), it will look for compressed chunks that are not fully frozen. If chunks needing freezing are found, it uses the maximum transaction age of the chunks in the set to determine how many workers to use to vacuum the chunks. The closer the maximum transaction age is to autovacuumfreezemax_age the more workers are used (up to a maximum number). This is done to throttle how much database CPU is consumed by vacuuming. We ignore chunks that have been vacuumed in the last 15 minutes. We ignore chunks with a transaction id age less than vacuumfreezemin_age. We grab a list of chunks and then let workers pull from this list. Vacuuming may take a while, so there is a decent chance that the postgres autovacuum engine may vacuum a chunk before we get to it. We get the autovacuum count when we produce the list. Just before vacuuming a chunk, we check the autovacuum count again to see if it has increased. If it has, the autovacuum engine beat us to the chunk, and we skip it. Additionally, we have seen instances in which compresses chunks are missing statistics. Autovacuum ignores tables that are missing statistics. Analyzing these tables does not help since we rarely modify chunks after they are compressed. Therefore, these chunks are ignored until they pass the vacuumfreezemax_age. This can be bad for performance. So, the vacuum engine also looks for these chunks and vacuums them. We only use one worker for these.