cwielenb/CNCF-Dataset huggingface dataset:
Project: Gravitee.io (Orchestration & Management / API Gateway)
File: purge-outdated-storage-container.md

This simple script will purge the storage container older than a specified date. First, be sure you have CLI installed. To run this script you need to provide 2 env vars: CONNECTION_STRING: The connection string to the storage account LIMIT_DATE: The limit date to purge the storage container (everything older than this date will be deleted), in the format `YYYY-MM-DDTHH:MM:SS+00:00`, for instance `2022-02-01T00:00:00+00:00` Then, you can run the script with the following command: `CONNECTIONSTRING="" LIMITDATE="2022-02-01T00:00:00+00:00" zx ./purge-storage-container.md` It will first get the env variables: ```javascript const connectionString = process.env.CONNECTION_STRING; const limitDate = process.env.LIMIT_DATE || '2022-02-01T00:00:00+00:00'; if(!connectionString) { console.error("You need to provide the following env vars: CONNECTION_STRING"); process.exit(1); } ``` Then call the Azure API to list all the containers matching the lastModified date: ```javascript const query = `[?properties.lastModified<="${limitDate}"].{name:name}`; const containersToDeleteAsString = await $`az storage container list --connection-string=${connectionString} --query ${query}`; const containersToDelete = JSON.parse(containersToDeleteAsString); ``` Then call the Azure API to delete all the containers: ```javascript console.log(`Deleting ${containersToDelete.length} containers`); for(const container of containersToDelete) { console.log(`Deleting container ${container.name}`); await $`az storage container delete --name ${container.name} --connection-string=${connectionString}` console.log(` Deleted container ${container.name}`); } ``` Et voila!
Project: Koordinator (Orchestration & Management / Scheduling & Orchestration)
File: 20220615-qos-manager.md

title: QoS Manager authors: "@stormgbs" reviewers: "@eahydra" "@hormes" "@jasonliu747" "@zwzhang0107" creation-date: 2022-06-15 last-updated: 2022-06-15 <!--ts--> * * * * <!--te--> QoS Manager is a new plugin framework for `dynamic plugins` in , `static plugins` will be move into . Currently, plugins from resmanager in Koordlet are mixed together, they should be classified into two categories: `static` and `dynamic`. Static plugins will be called and run only once when a container created, updated, started or stopped. However, for dynamic plugins, they may be called and run at any time according the real-time runtime states of node, such as CPU suppress, CPU burst, etc. This proposal only focuses on refactoring dynamic plugins. Take a look at current plugin implementation, there are many function calls to resmanager's methods directly, such as collecting node/pod/container metrics, fetching metadata of node/pod/container, fetching configurations(NodeSLO, etc.). In the feature, we may need a flexible and powerful framework with scalability for special external plugins. Define a new framework named `qos-manager` to manage dynamic plugins. Define new interfaces to abstract and simplify management of plugins. Refactor and move the built-in dynamic plugins into `qos-manager`, such as `CPU suppress`, `CPU brust`, `CPU evict` and `memory evict`. [Future Plan] Framework design with scalability for external plugins. [Future Plan] Refctor `resource-update-executor`. The below is directory tree of qos-manager inside koordlet, all existing dynamic plugins(as built-in plugins) will be move into sub-directory `plugins`. ``` pkg/koordlet/qosmanager/ manager.go context.go // plugin context /plugins/ // built-in plugins /cpubrust/ /cpusuppress/ /cpuevict/ /memoryevict/ ``` The struct `QoSManager` wraps `resmanager` to provide necessary interface methods for all plugins, here we just put `StateInformer`, `MetricsCache` and `Executor` into a `PluginContext` as a parameter of `InitPlugin` function during plugin initialization. Each plugin should implement interface `Plugin` which contains some basic methods, such as `Name()`, `Run()`, `Stop()`, etc. On startup, each plugin should call `StateInformer.RegisterCallbacks` for `NodeSLO` policy updates. Additionally, QoSManager is also responsible for managing plugin feature-gates. The `Executor` is a very important component in `Koordlet` which is responsible for resolving conflicts of rules produced by `qos-manager` and `runtime-hooks`, adaptations for various platforms (kernel, arch, etc.), it also should have a configuration cache to avoid redundant parameters updates. ```go // QoSManager type QoSManager struct { *resmanager // ... } type PluginContext interface { StatesInformer MetricCache Executor } // Plugin Initialization func InitPlugin(ctx PluginContext) Plugin { // ... } type Plugin interface { Name() string Run(stop <-chan struct{}) () Stop() error } ```
Project: Kubernetes (Orchestration & Management / Scheduling & Orchestration)
File: man_docs.md

Generating man pages from a cobra command is incredibly easy. An example is as follows: ```go package main import ( "log" "github.com/spf13/cobra" "github.com/spf13/cobra/doc" ) func main() { cmd := &cobra.Command{ Use: "test", Short: "my test program", } header := &doc.GenManHeader{ Title: "MINE", Section: "3", } err := doc.GenManTree(cmd, header, "/tmp") if err != nil { log.Fatal(err) } } ``` That will get you a man page `/tmp/test.3`
Project: 3Scale (Orchestration & Management / API Gateway)
File: why.md

3scale provided downloadable NGINX configuration for many years and it worked great for our customers. However, some things could be improved. When NGINX configuration is generated from the Admin Portal, it has all the configuration embedded. That means changes to the code will result in conflicts when configuration changes and needs to be regenerated. Splitting code and configuration makes the code shareable. Now everyone can run the same code and just download new configuration (in JSON format) when needed. That also allows doing customizations to the code without worrying about configs when new configuration needs to be aplied. The configuration is just JSON file. That also allows contribuing those customizations back the the project, so they can be used by everyone. All code is open-source on GitHub. Splitting code and configuration allows thorough testing of each component individually. We are trying hard to cover every feature and every fix by a regression test. Using wonderful framework for high level integration tests allows us to run every test several times in random order to ensure there are no random failures. Low level unit testing is done in Lua testing framework . That allows us to run low level tests for edge cases easily and run them in isolation. We would like to make proxy modular, so you can load a plugin that adds some behaviour.
Project: BFE (Orchestration & Management / Service Proxy)
File: mod_geo.md

mod_geo creates with values depending on the client IP address, using the GEO databases. conf/modgeo/modgeo.conf | Config Item | Description | | | - | | Basic.GeoDBPath | String<br>Path of geo db file | | Log.OpenDebug | Boolean<br>Whether enable debug logs<br>Default False | mod_geo supports GeoDB in MaxMind format which can be downloaded from https://dev.maxmind.com/geoip/geoip2/geolite2/ ```ini [Basic] GeoDBPath = mod_geo/geo.db ```
Project: Gloo (Orchestration & Management / API Gateway)
File: glooctl_edit_settings_ratelimit_server-config.md

title: "glooctl edit settings ratelimit server-config" weight: 5 Add rate-limit descriptor settings (Enterprise) This allows using lyft rate-limit server configuration language to configure the rate-limit server. For more information see: https://github.com/lyft/ratelimit Note: do not add the 'domain' configuration key. This is a Gloo Enterprise feature. ``` glooctl edit settings ratelimit server-config [flags] ``` ``` -h, --help help for server-config ``` ``` -c, --config string set the path to the glooctl config file (default "<home_directory>/.gloo/glooctl-config.yaml") --consul-address string address of the Consul server. Use with --use-consul (default "127.0.0.1:8500") --consul-allow-stale-reads Allows reading using Consul's stale consistency mode. --consul-datacenter string Datacenter to use. If not provided, the default agent datacenter is used. Use with --use-consul --consul-root-key string key prefix for for Consul key-value storage. (default "gloo") --consul-scheme string URI scheme for the Consul server. Use with --use-consul (default "http") --consul-token string Token is used to provide a per-request ACL token which overrides the agent's default token. Use with --use-consul -i, --interactive use interactive mode --kube-context string kube context to use when interacting with kubernetes --kubeconfig string kubeconfig to use, if not standard one --name string name of the resource to read or write -n, --namespace string namespace for reading or writing resources (default "gloo-system") -o, --output OutputType output format: (yaml, json, table, kube-yaml, wide) (default table) --resource-version string the resource version of the resource we are editing. if not empty, resource will only be changed if the resource version matches --use-consul use Consul Key-Value storage as the backend for reading and writing config (VirtualServices, Upstreams, and Proxies) ``` - Configure rate limit settings (Enterprise)
Project: Flomesh Service Mesh (FSM) (Orchestration & Management / Service Mesh)
File: ROADMAP.md

The FSM team uses the to track short term (1-3 months) and medium term (3-6 months) efforts. Once a broad objective (from the "Research & Scoping" column) is further scoped into a GitHub issue that has a clear objective, background and motive, it is moved into the "Planned & Scoped" column. From there, the team plans monthly releases via . At this time, the FSM community is working on: Hardening of the project with a rigorous testing environment. Implementing SMI Traffic Metrics and build tooling to leverage it. Iterating on SMI Traffic Split and Traffic Access implementations. In the medium to long term future, the FSM community will tackle the following: Iterate on the current Ingress/Egress functionality, experience, workflow. Iterate on existing tools for debugging the FSM control plane and data plane. Leverage xDS v3 protocol when limited by SMI to provide a "no cliffs" design. Extend FSM to support VMs, hybrid and multicluster scenarios. Please file to propose features and identify bugs. See more information about how to get involved with the community .
Project: Gloo (Orchestration & Management / API Gateway)
File: installation.md

title: Installation description: How and where to deploy Gloo Edge Federation weight: 15 Learn how and where to deploy Gloo Edge Federation. Gloo Edge Federation is primarily an additional Kubernetes controller that runs alongside Gloo Edge controllers. It is composed of Kubernetes Custom Resource Definitions (CRDs) and a controller pod that watches the custom resources and executes actions. The controller deployment and CRDs are created in an "administrative" cluster. Note that the Gloo Edge Federation controller can also be deployed in an existing cluster that is already running Gloo Edge. {{< tabs >}} {{% tab name="Dedicated Admin cluster" %}} {{% /tab %}} {{% tab name="Shared cluster" %}} {{% /tab %}} {{< /tabs >}} You can choose between two standard deployment models for Gloo Edge Federation: Alongside Gloo Edge Enterprise Standalone mode By default, Gloo Edge Federation is installed alongside Gloo Edge Enterprise. After installation, check that the following deployments are ready. ``` kubectl -n gloo-system get deploy NAME READY UP-TO-DATE AVAILABLE AGE gloo-fed 1/1 1 1 130m gloo-fed-console 1/1 1 1 130m ``` If you can't see these deployments, install or upgrade the Gloo Edge Helm chart with the following Helm values. ```yaml gloo-fed: enabled: true ``` To deploy Gloo Edge Federation in a standalone mode, you must install the Gloo Edge Federation Helm chart. ```shell helm install gloo-fed gloo-fed/gloo-fed --version $GLOOVERSION --set licensekey=$LICENSE_KEY -n gloo-system --create-namespace ``` Next, the Gloo Edge instances.
Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: deploy-multi-volcano-schedulers-without-using-selector.md

Currently the single scheduler can not satisfy the high throughput requirement in some scenarios. Besides the performance optimization against the single scheduler, another choice is to deploy multiple volcano schedulers to improve the overall scheduling throughput. Previously we use label to divide the cluster nodes to multiple sections and each volcano scheduler is responsible for one section and then specify the schedulerName in the Pod Spec and submit it. It is inconvenient in some conditions especially for large clusters. This doc privides a another option for user to deploy multiple scheduler which needs less modification for workload and nodes. A statefulset is used to deploy the volcano scheduler. The Job and Node are assigned to scheduler automatically based on the hash algorithm. ``` kind: StatefulSet apiVersion: apps/v1 metadata: name: volcano-scheduler namespace: volcano-system labels: app: volcano-scheduler spec: replicas: 3 selector: matchLabels: app: volcano-scheduler serviceName: "volcano-scheduler" template: metadata: labels: app: volcano-scheduler spec: serviceAccount: volcano-scheduler containers: name: volcano-scheduler image: volcanosh/vc-scheduler:ae78900d21dce8522eb04b6817aac66c9abd01e2 args: --logtostderr --scheduler-conf=/volcano.scheduler/volcano-scheduler.conf -v=3 2>&1 imagePullPolicy: "IfNotPresent" env: name: MULTISCHEDULERENABLE value: "true" name: SCHEDULER_NUM value: "3" name: SCHEDULERPODNAME valueFrom: fieldRef: fieldPath: metadata.name volumeMounts: name: scheduler-config mountPath: /volcano.scheduler volumes: name: scheduler-config configMap: name: volcano-scheduler-configmap apiVersion: v1 kind: Service metadata: name: volcano-scheduler labels: app: volcano-scheduler spec: ports: port: 80 name: volcano-scheduler clusterIP: None selector: app: volcano-scheduler ``` Notes: MULTISCHEDULERENABLE env is used to enable or disable multi-scheduler. SCHEDULER_NUM indicates the numbers of volcano schedulers which you are planning to launch. ``` kubectl apply -f <volcano-statefulset.yaml> ```
Project: Gloo (Orchestration & Management / API Gateway)
File: event_service_config.proto.sk.md

title: "eventserviceconfig.proto" weight: 5 <!-- Code generated by solo-kit. DO NOT EDIT. --> [#not-implemented-hide:] Configuration of the event reporting service endpoint. ```yaml "grpcService": .solo.io.envoy.config.core.v3.GrpcService ``` | Field | Type | Description | | -- | - | -- | | `grpcService` | | Specifies the gRPC service that hosts the event reporting service. | <!-- Start of HubSpot Embed Code --> <script type="text/javascript" id="hs-script-loader" async defer src="//js.hs-scripts.com/5130874.js"></script> <!-- End of HubSpot Embed Code -->
Project: APISIX (Orchestration & Management / API Gateway)
File: consumer-group.md

title: Consumer Group keywords: API gateway Apache APISIX Consumer Group description: Consumer Group in Apache APISIX. <!-- --> Consumer Groups are used to extract commonly used configurations and can be bound directly to a . With consumer groups, you can define any number of plugins, e.g. rate limiting and apply them to a set of consumers, instead of managing each consumer individually. The example below illustrates how to create a Consumer Group and bind it to a Consumer. Create a Consumer Group which shares the same rate limiting quota: :::note You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command: ```bash adminkey=$(yq '.deployment.admin.adminkey[0].key' conf/config.yaml | sed 's/"//g') ``` ::: ```shell curl http://127.0.0.1:9180/apisix/admin/consumergroups/companya \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "plugins": { "limit-count": { "count": 200, "time_window": 60, "rejected_code": 503, "group": "grpcompanya" } } }' ``` Create a Consumer within the Consumer Group: ```shell curl http://127.0.0.1:9180/apisix/admin/consumers \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "username": "jack", "plugins": { "key-auth": { "key": "auth-one" } }, "groupid": "companya" }' ``` When APISIX can't find the Consumer Group with the `group_id`, the Admin API is terminated with a status code of `400`. :::tip When the same plugin is configured in , , and , only one configuration is in effect, and the consumer has the highest priority. Please refer to . If a Consumer already has the `plugins` field configured, the plugins in the Consumer Group will effectively be merged into it. The same plugin in the Consumer Group will not override the one configured directly in the Consumer. ::: For example, if we configure a Consumer Group as shown below: ```json { "id": "bar", "plugins": { "response-rewrite": { "body": "hello" } } } ``` To a Consumer as shown below. ```json { "username": "foo", "group_id": "bar", "plugins": { "basic-auth": { "username": "foo", "password": "bar" }, "response-rewrite": { "body": "world" } } } ``` Then the `body` in `response-rewrite` keeps `world`.
Project: Traefik (Orchestration & Management / API Gateway)
File: replacepathregex.md

title: "Traefik ReplacePathRegex Documentation" description: "In Traefik Proxy's HTTP middleware, ReplacePathRegex updates paths before forwarding requests, using a regex. Read the technical documentation." Updating the Path Before Forwarding the Request (Using a Regex) {: .subtitle } <!-- TODO: add schema --> The ReplaceRegex replaces the path of a URL using regex matching and replacement. ```yaml tab="Docker & Swarm" labels: "traefik.http.middlewares.test-replacepathregex.replacepathregex.regex=^/foo/(.*)" "traefik.http.middlewares.test-replacepathregex.replacepathregex.replacement=/bar/$$1" ``` ```yaml tab="Kubernetes" apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: test-replacepathregex spec: replacePathRegex: regex: ^/foo/(.*) replacement: /bar/$1 ``` ```yaml tab="Consul Catalog" "traefik.http.middlewares.test-replacepathregex.replacepathregex.regex=^/foo/(.*)" "traefik.http.middlewares.test-replacepathregex.replacepathregex.replacement=/bar/$1" ``` ```yaml tab="File (YAML)" http: middlewares: test-replacepathregex: replacePathRegex: regex: "^/foo/(.*)" replacement: "/bar/$1" ``` ```toml tab="File (TOML)" [http.middlewares] [http.middlewares.test-replacepathregex.replacePathRegex] regex = "^/foo/(.*)" replacement = "/bar/$1" ``` The ReplacePathRegex middleware will: replace the matching path with the specified one. store the original path in a `X-Replaced-Path` header. !!! tip Regular expressions and replacements can be tested using online tools such as or the . When defining a regular expression within YAML, any escaped character needs to be escaped twice: `example\.com` needs to be written as `example\\.com`. The `regex` option is the regular expression to match and capture the path from the request URL. The `replacement` option defines the replacement path format, which can include captured variables. !!! warning Care should be taken when defining replacement expand variables: `$1x` is equivalent to `${1x}`, not `${1}x` (see ), so use `${1}` syntax.
Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: how_to_use_mpi_plugin.md

MPI plugin is designed to optimize the user experience when running MPI jobs, it not only allows users to write less yaml, but also ensures the normal operation of MPI jobs. The MPI plugin will do three things: Open ports used by MPI for all containers of the job Force open `ssh` and `svc` plugins add `MPI_HOST` environment variable for master pod, this environment variable includes the worker's domain name, It is used by the `--host` parameter of `mpiexec` If `master` or `worker` is configured, please ensure that the tasks corresponding to their values exist, and the roles of these tasks correspond to the meaning of the parameters If `port` is configured, make the port value of `sshd` the same as the value of the parameter. If the `gang` plugin is enabled, then make sure that the value of `minAvailable` is equal* to the number of `replicas of the worker`. | ID | Name | Type | Default Value | Required | Description | Example | | - | | | - | -- | - | | | 1 | master | string | master | No | Name of MPI master | --master=mpimaster | | 2 | worker | string | worker | No | Name of MPI worker | --worker=mpiworker | | 3 | port | string | 22 | No | The port to open for the container | --port=5000 | ```yaml apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: lm-mpi-job spec: minAvailable: 1 schedulerName: volcano plugins: mpi: ["--master=mpimaster","--worker=mpiworker","--port=22"] ## MPI plugin register tasks: replicas: 1 name: mpimaster policies: event: TaskCompleted action: CompleteJob template: spec: containers: command: /bin/sh -c | mkdir -p /var/run/sshd; /usr/sbin/sshd; mpiexec --allow-run-as-root --host ${MPIHOST} -np 2 mpihello_world; image: volcanosh/example-mpi:0.0.3 name: mpimaster workingDir: /home restartPolicy: OnFailure replicas: 2 name: mpiworker template: spec: containers: command: /bin/sh -c | mkdir -p /var/run/sshd; /usr/sbin/sshd -D; image: volcanosh/example-mpi:0.0.3 name: mpiworker workingDir: /home restartPolicy: OnFailure ```
Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: bug_report.md

name: Bug Report about: Report a bug encountered while operating Volcano labels: kind/bug <!-- This form is for bug reports ONLY! If you're looking for a help then check our or have a look at our --> What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Volcano Version: Kubernetes version (use `kubectl version`): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. `uname -a`): Install tools: Others:
Project: Open Service Mesh (Orchestration & Management / Service Mesh)
File: automerge.md

title: "Automerge Pull Requests" description: "How to use labels and commands to automerge/autorebase your pull request." type: docs OSM uses to automatically merge (automerge) and automatically squash and merge (autosquash) pull requests. A pull request will be automerged via a merge commit if it meets the following criteria: Has the `automerge` label Does not have the `wip` label Does not have the `do-not-merge/hold` label Successfully completed all checks Has at least 2 maintainer approvals Base branch is either main or a release branch If the pull request has an `automerge` label, the OSM-PR-bot will also autorebase the pull request if the PR branch goes out-of-date. Note: Pull requests that are paths-ignore cannot be merged automatically. A pull request will be autosquashed (commits will be squashed then merged) if it meets the following criteria: Has the `automerge-squash` label Does not have the `wip` label Does not have the `do-not-merge/hold` label Successfully completed all checks Has at least 2 maintainer approvals Base branch is main branch If the pull request has an `automerge-squash` label, the OSM-PR-bot will also autorebase the pull request if the PR branch goes out-of-date. Note: Pull requests that are paths-ignore cannot be merged automatically. Some Mergify commands can also be triggered via comments on the pull request. `@Mergifyio refresh` will re-evaluate the rules `@Mergifyio rebase` will rebase this PR on its base branch `@Mergifyio backport <destination>` will this PR on <destination> branch