cwielenb/CNCF-Dataset huggingface dataset:
Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: general-setting.md

DolphinScheduler supports two types of built-in language which include `English` and `Chinese`. You could click the button on the top control bar named `English` and `Chinese` and change it to another one when you want to switch the language. The entire DolphinScheduler page language will shift when you switch the language selection. DolphinScheduler supports two types of built-in theme which include `Dark` and `Light`. When you want to change the theme of DolphinScheduler, all you have to do is click the button named `Dark`(or `Light`) on the top control bar and on the left of to control button. DolphinScheduler support time zone setting. Server Time Zone The default time zone is UTC when using `bin/dolphinshcedulerdaemon.sh` to start the server, you could update `SPRINGJACKSONTIMEZONE` in `bin/env/dolphinschedulerenv.sh`, such as `export SPRINGJACKSONTIMEZONE=${SPRINGJACKSONTIME_ZONE:-Asia/Shanghai}`.<br> If you start server in IDEA, the default time zone is your local time zone, you could add the JVM parameter to update server time zone, such as `-Duser.timezone=UTC`. Time zone list refer to User Time zone The user's default time zone is based on the time zone which you run the DolphinScheduler service.You could click the button on the right of the button and then click `Choose timeZone` to choose the time zone you want to switch. All time related components will adjust their time zone according to the time zone setting you select.
Project: Fluid (Orchestration & Management / Scheduling & Orchestration)
File: image_pull_secrets.md

If the images of fluid runtime is in private docker registry, it's necessary to set image pull secret. For related knowledge of image pull secret, please refer to [Pull an Image from a Private Registry ](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) fluid supports setting image pull secrets when you deploy fluid service using helm charts you can set the image pull secrets to `values.yaml` like the following example ```yaml image: imagePullSecrets: [] image: imagePullSecrets: name: test-1 name: test-2 ``` After setting `values.yaml` image pull secrets, when you have deployed the fluid service, you can see that controller itself already uses image pull secrets alluxio controller yaml intercepts information ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: alluxioruntime-controller namespace: fluid-system spec: template: spec: containers: image: fluidcloudnative/alluxioruntime-controller: name: manager dnsPolicy: ClusterFirst imagePullSecrets: name: test-1 name: test-2 ``` fluid also supports passing the image pull secrets through the controller to the runtime service that the controller pulls up If you want to specify the image pull key for the runtime, you can do so by editing the environment variables in the deployment associated with the runtime controller that you have created. Here is a brief YAML configuration example. When properly configured, the deployment will apply the image pull secret you specified. ```yaml name: IMAGEPULLSECRETS value: test-1,test-2 ``` In the above YAML file, `IMAGEPULLSECRETS` is the name of the environment variable, and `test-1,test-2` are the image pull keys you wish to specify, separated by commas. alluxio runtime master yaml intercepts information ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: demo-master namespace: test spec: template: spec: containers: image: fluidcloudnative/alluxio:release-2.8.1-SNAPSHOT-0433ade imagePullPolicy: IfNotPresent name: alluxio-master imagePullSecrets: name: test1 name: test2 ``` In conclusion, image pull secrets setting successfully
Project: Gloo (Orchestration & Management / API Gateway)
File: pipe.proto.sk.md

title: "pipe.proto" weight: 5 <!-- Code generated by solo-kit. DO NOT EDIT. --> Pipe upstreams are used to route request to services listening at a Unix Domain Socket. Pipe upstreams can be used to proxy any kind of service, and therefore contain a ServiceSpec for additional service-specific configuration. Unlike upstreams created by service discovery, Pipe Upstreams must be created manually by users ```yaml "path": string "serviceSpec": .options.gloo.solo.io.ServiceSpec ``` | Field | Type | Description | | -- | - | -- | | `path` | `string` | The Unix Domain Socket path. | | `serviceSpec` | | An optional Service Spec describing the service listening at this address. | <!-- Start of HubSpot Embed Code --> <script type="text/javascript" id="hs-script-loader" async defer src="//js.hs-scripts.com/5130874.js"></script> <!-- End of HubSpot Embed Code -->
Project: Gloo (Orchestration & Management / API Gateway)
File: connection.proto.sk.md

title: "connection.proto" weight: 5 <!-- Code generated by solo-kit. DO NOT EDIT. --> Fine tune the settings for connections to an upstream ```yaml "maxRequestsPerConnection": int "connectTimeout": .google.protobuf.Duration "tcpKeepalive": .gloo.solo.io.ConnectionConfig.TcpKeepAlive "perConnectionBufferLimitBytes": .google.protobuf.UInt32Value "commonHttpProtocolOptions": .protocol.options.gloo.solo.io.HttpProtocolOptions "http1ProtocolOptions": .protocol.options.gloo.solo.io.Http1ProtocolOptions ``` | Field | Type | Description | | -- | - | -- | | `maxRequestsPerConnection` | `int` | Maximum requests for a single upstream connection (unspecified or zero = no limit). | | `connectTimeout` | | The timeout for new network connections to hosts in the cluster. | | `tcpKeepalive` | | Configure OS-level tcp keepalive checks. | | `perConnectionBufferLimitBytes` | | Soft limit on size of the clusters connections read and write buffers. If unspecified, an implementation defined default is applied (1MiB). For more info, see the . | | `commonHttpProtocolOptions` | | Additional options when handling HTTP requests upstream. These options will be applicable to both HTTP1 and HTTP2 requests. | | `http1ProtocolOptions` | | Additional Options when handling HTTP requests upstream. These options will be applicable only to HTTP1 requests. | If set then set SO_KEEPALIVE on the socket to enable TCP Keepalives. see more info here: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#config-core-v3-tcpkeepalive ```yaml "keepaliveProbes": int "keepaliveTime": .google.protobuf.Duration "keepaliveInterval": .google.protobuf.Duration ``` | Field | Type | Description | | -- | - | -- | | `keepaliveProbes` | `int` | Maximum number of keepalive probes to send without response before deciding the connection is dead. | | `keepaliveTime` | | The number of seconds a connection needs to be idle before keep-alive probes start being sent. This is rounded up to the second. | | `keepaliveInterval` | | The number of seconds between keep-alive probes. This is rounded up to the second. | <!-- Start of HubSpot Embed Code --> <script type="text/javascript" id="hs-script-loader" async defer src="//js.hs-scripts.com/5130874.js"></script> <!-- End of HubSpot Embed Code -->
Project: Volcano (Orchestration & Management / Scheduling & Orchestration)
File: custom-plugin.md

Until now, plugins like `binpack`, `drf`, `gang` are provided by official. But some users may want to implement the plugin by themselves. So if scheduler can dynamically load plugins, that will make it more flexible when handling different business scenarios. If use `alpine` as base docker image, you should use `musl-gcc` to build scheduler. ```bash wget http://musl.libc.org/releases/musl-1.2.1.tar.gz tar -xf musl-1.2.1.tar.gz && cd musl-1.2.1 ./configure make && sudo make install make images CC=/usr/local/musl/bin/musl-gcc SUPPORT_PLUGINS=yes ``` If use `ubuntu` as base docker image ```bash make images SUPPORT_PLUGINS=yes ``` ```go // magic.go package main // note!!! package must be named main import ( "volcano.sh/volcano/pkg/scheduler/framework" ) const PluginName = "magic" type magicPlugin struct {} func (mp *magicPlugin) Name() string { return PluginName } func New(arguments framework.Arguments) framework.Plugin { // `New` is PluginBuilder return &magicPlugin{} } func (mp magicPlugin) OnSessionOpen(ssn framework.Session) {} func (mp magicPlugin) OnSessionClose(ssn framework.Session) {} ``` Because the default `vc-scheduler` base image is `alpine`, which only has `musl-libc`, so we should use `musl-gcc` to build the plugin. ```bash docker run -v `pwd`:/work golang:1.14-alpine sh -c "cd /work && apk add musl-dev gcc && go build -buildmode=plugin magic.go" ``` Or build the plugin in local. ```bash wget http://musl.libc.org/releases/musl-1.2.1.tar.gz tar -xf musl-1.2.1.tar.gz && cd musl-1.2.1 ./configure make && sudo make install CC=/usr/local/musl/bin/musl-gcc CGO_ENABLED=1 go build -o plugins/magic.so -buildmode=plugin magic.go ``` If want to use `ubuntu` as base image, you can use `gnu-libc` to build the plugin. Since most Linux OS have `gnu-libc`, you can just build the plugin in local. ```bash CGO_ENABLED=1 go build -o plugins/magic.so -buildmode=plugin magic.go ``` Your can build your docker image ```dockerfile FROM volcanosh/vc-scheduler:latest COPY plugins plugins ``` ``` docker build -t volcanosh/vc-scheduler:magic-plugins . ``` Or just use `pvc` to mount these plugins ```yaml ... containers: name: volcano-scheduler image: volcanosh/vc-scheduler:magic-plugins args: --logtostderr --scheduler-conf=/volcano.scheduler/volcano-scheduler.conf --enable-healthz=true --enable-metrics=true -v=3 --plugins-dir=plugins # specify plugins dir path 2>&1 ``` Add your custom plugin name in configmap ``` kubectl edit cm volcano-scheduler-configmap -n volcano-system ``` ```yaml apiVersion: v1 kind: ConfigMap metadata: name: volcano-scheduler-configmap namespace: volcano-system data: volcano-scheduler.conf: | actions: "enqueue, allocate, backfill" tiers: plugins: name: priority name: gang name: conformance name: magic # activate your custom plugin plugins: name: drf name: predicates name: proportion name: nodeorder name: binpack ``` Plugins should be rebuilt after volcano source code modified. Plugin package name must be main.
Project: Avro (Orchestration & Management / Remote Procedure Call)
File: new-committer-david-mollitor.md

title: "New committer: David Mollitor" linkTitle: "New committer: David Mollitor" date: 2021-10-05 <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> The Project Management Committee (PMC) for Apache Avro has invited David Mollitor to become a committer and we are pleased to announce that he has accepted. Since 2017, David has raised and fixed many issues in the Java SDK. Recently he's been finding and providing fixes for subtle performance issues. His work is always high-quality and he is reactive and pleasant to talk with on code reviews and JIRA. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should enable better productivity. It's great to have you as part of the team, David!
Project: Kusk Gateway (Orchestration & Management / API Gateway)
File: working-with-extension.md

Kusk Gateway comes with an `x-kusk` to allow an OpenAPI definition to be the source of truth for both operational and functional aspects of your APIs. The describes all available properties and the following guides are available to help you make the most of them. - How to specify CORS settings. - How to rate-limit requests to your API. - How to configure routing of API requests. - How work with automatic request validation. - How to set request timeouts. - How to set authentication of your API. - How to mock all or parts of your API. - How to cache your API responses. `x-kusk` extension can be applied at (not exclusively): Top level of an OpenAPI definition: ```yaml openapi: 3.0.2 info: title: Swagger Petstore - OpenAPI 3.0 x-kusk: hosts: "example.org" disabled: false cors: ... ``` Path level: ```yaml openapi: 3.0.2 info: title: Swagger Petstore - OpenAPI 3.0 paths: /pet: x-kusk: disabled: true # disables all /pet endpoints post: ... ``` Method (operation) level: ```yaml openapi: 3.0.2 info: title: Swagger Petstore - OpenAPI 3.0 paths: /pet: post: x-kusk: upstream: # routes the POST /pet endpoint to a Kubernetes service service: namespace: default name: petstore port: 8000 ... ``` The `x-kusk` extension at the operation level takes precedence, or overrides, what is specified at the path level, including the `disabled` option. Likewise, the path level settings override what is specified at the global level. If settings aren't specified at a path or operation level, they will be inherited from the layer above, (Operation > Path > Global).
Project: Gloo (Orchestration & Management / API Gateway)
File: http_path.proto.sk.md

title: "http_path.proto" weight: 5 <!-- Code generated by solo-kit. DO NOT EDIT. --> Same as HTTP health checker, but allows a custom path per endpoint The http path to use can be overriden using endpoint metadata. The endpoint specific path should be in the "io.solo.healthcheckers.httppath" namespace, under a string value named "path". ```yaml "httpHealthCheck": .solo.io.envoy.config.core.v3.HealthCheck.HttpHealthCheck ``` | Field | Type | Description | | -- | - | -- | | `httpHealthCheck` | | Http health check. | <!-- Start of HubSpot Embed Code --> <script type="text/javascript" id="hs-script-loader" async defer src="//js.hs-scripts.com/5130874.js"></script> <!-- End of HubSpot Embed Code -->
Project: Karmada (Orchestration & Management / Scheduling & Orchestration)
File: rescheduling_test.md

| Test Case | E2E Describe Text | Comments | ||--|-| | Test if the deployment will rescheduled to other available clusters when some clusters are unjoined | deployment reschedule testing | | | Test when the ReplicaSchedulingType is Duplicate and join some clusters then recovered | when the ReplicaSchedulingType of the policy is Duplicated, reschedule testing | | | Test when the ReplicaScheduling of the policy is nil and join some clusters then recovered | when the ReplicaScheduling of the policy is nil, reschedule testing | | | Test if the deployment will rescheduled when the labels match or not match(PropagationPolicy) | change labels to testing deployment reschedule(PropagationPolicy) | | | Test if the deployment will rescheduled when the labels match or not match(ClusterPropagationPolicy) | change labels to testing deployment reschedule(ClusterPropagationPolicy) | |
Project: Traefik (Orchestration & Management / API Gateway)
File: quick-start.md

title: "Traefik Getting Started Quickly" description: "Get started with Traefik Proxy and Docker." A Use Case Using Docker {: .subtitle } Create a `docker-compose.yml` file where you will define a `reverse-proxy` service that uses the official Traefik image: ```yaml version: '3' services: reverse-proxy: image: traefik:v3.0 command: --api.insecure=true --providers.docker ports: "80:80" "8080:8080" volumes: /var/run/docker.sock:/var/run/docker.sock ``` That's it. Now you can launch Traefik! Start your `reverse-proxy` with the following command: ```shell docker-compose up -d reverse-proxy ``` You can open a browser and go to `http://localhost:8080/api/rawdata` to see Traefik's API rawdata (you'll go back there once you have launched a service in step 2). Now that you have a Traefik instance up and running, you will deploy new services. Edit your `docker-compose.yml` file and add the following at the end of your file. ```yaml version: '3' services: ... whoami: image: traefik/whoami labels: "traefik.http.routers.whoami.rule=Host(`whoami.docker.localhost`)" ``` The above defines `whoami`: a web service that outputs information about the machine it is deployed on (its IP address, host, and others). Start the `whoami` service with the following command: ```shell docker-compose up -d whoami ``` Browse `http://localhost:8080/api/rawdata` and see that Traefik has automatically detected the new container and updated its own configuration. When Traefik detects new services, it creates the corresponding routes, so you can call them ... let's see! (Here, you're using curl) ```shell curl -H Host:whoami.docker.localhost http://127.0.0.1 ``` Shows the following output: ```yaml Hostname: a656c8ddca6c IP: 172.27.0.3 ``` Run more instances of your `whoami` service with the following command: ```shell docker-compose up -d --scale whoami=2 ``` Browse to `http://localhost:8080/api/rawdata` and see that Traefik has automatically detected the new instance of the container. Finally, see that Traefik load-balances between the two instances of your service by running the following command twice: ```shell curl -H Host:whoami.docker.localhost http://127.0.0.1 ``` The output will show alternatively one of the following: ```yaml Hostname: a656c8ddca6c IP: 172.27.0.3 ``` ```yaml Hostname: s458f154e1f1 IP: 172.27.0.4 ``` !!! question "Where to Go Next?" Now that you have a basic understanding of how Traefik can automatically create the routes to your services and load balance them, it is time to dive into and let Traefik work for you! {!traefik-for-business-applications.md!}
Project: KEDA (Orchestration & Management / Scheduling & Orchestration)
File: autorest.md

``` yaml title: EventGridPublisherClient description: Azure Event Grid client generated-metadata: false clear-output-folder: false go: true input-file: https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/eventgrid/data-plane/Microsoft.EventGrid/stable/2018-01-01/EventGrid.json license-header: MICROSOFTMITNO_VERSION openapi-type: "data-plane" output-folder: ../publisher override-client-name: Client security: "AADToken" use: "@autorest/go@4.0.0-preview.52" version: "^3.0.0" slice-elements-byval: true remove-non-reference-schema: true directive: from: swagger-document where: $["x-ms-parameterized-host"] transform: $.parameters = "client" from: client.go where: $ transform: return $.replace(/\[\]CloudEvent/g, "[]messaging.CloudEvent"); from: client.go where: $ transform: return $.replace(/func \(client \Client\) PublishCloudEventEvents\(/g, "func (client Client) internalPublishCloudEventEvents("); from: swagger-document where: $.definitions.CloudEventEvent transform: $["x-ms-external"] = true from: client.go models.go response_types.go options.go where: $ transform: return $.replace(/Client(\w+)((?:Options|Response))/g, "$1$2"); from: models.go where: $ transform: return $.replace(/\/\/ (SubscriptionDeletedEventData|SubscriptionValidationEventData|SubscriptionValidationResponse).+?\n}/gs, "") from: models_serde.go where: $ transform: | return $ .replace(/\/\/ MarshalJSON implements the json.Marshaller interface for type (SubscriptionDeletedEventData|SubscriptionValidationEventData|SubscriptionValidationResponse).+?\n}/gs, "") .replace(/\/\/ UnmarshalJSON implements the json.Unmarshaller interface for type (SubscriptionDeletedEventData|SubscriptionValidationEventData|SubscriptionValidationResponse).+?\n}/gs, ""); from: models.go client.go response_types.go options.go where: $ transform: return $.replace(/CloudEventEvent/g, "CloudEvent"); from: models.go models_serde.go client.go response_types.go options.go where: $ transform: return $.replace(/EventGridEvent/g, "Event"); from: client.go where: $ transform: | return $.replace( /(func \(client \*Client\) publishCloudEventsCreateRequest.+?)return req, nil/s, '$1\nreq.Raw().Header.Set("Content-type", "application/cloudevents-batch+json; charset=utf-8")\nreturn req, nil'); ```
Project: BFE (Orchestration & Management / Service Proxy)
File: mod_access.md

mod_access writes request logs and session logs in the specified format. conf/modaccess/modaccess.conf | Config Item | Description | | -- | | | Log.LogFile | String<br>Set file path of log for saving to a single file without rotation | | Log.LogPrefix | String<br>Filename prefix for log | | Log.LogDir | String<br>Directory of log files | | Log.RotateWhen | String<br>Interval to rotate log file | | Log.BackupCount | Integer<br>Max number of rotated log files | | Template.RequestTemplate | String<br>Template of request log | | Template.SessionTemplate | String<br>Template of session log | ```ini [Log] LogPrefix = access LogDir = ../log RotateWhen = NEXTHOUR BackupCount = 2 [Template] RequestTemplate = "REQUESTLOG $time clientip: $remoteaddr serverip: $serveraddr host: $host product: $product useragent: ${User-Agent}reqheader status: $statuscode error: $error" SessionTemplate = "SESSIONLOG $time clientip: $sesclientip starttime: $sesstarttime endtime: $sesendtime overhead: $sesoverhead readtotal: $sesreadtotal writetotal: $seswritetotal keepalivenum: $seskeepalivenum error: $ses_error" ``` ```ini [Log] LogFile = /dev/stdout [Template] RequestTemplate = "REQUESTLOG $time clientip: $remoteaddr serverip: $serveraddr host: $host product: $product useragent: ${User-Agent}reqheader status: $statuscode error: $error" SessionTemplate = "SESSIONLOG $time clientip: $sesclientip starttime: $sesstarttime endtime: $sesendtime overhead: $sesoverhead readtotal: $sesreadtotal writetotal: $seswritetotal keepalivenum: $seskeepalivenum error: $ses_error" ```
Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: hive.md

| Datasource | Description | |-|| | Datasource | Select HIVE. | | Datasource name | Enter the name of the DataSource. | | Description | Enter a description of the DataSource. | | IP/Host Name | Enter the HIVE service IP. | | Port | Enter the HIVE service port. | | Username | Set the username for HIVE connection. | | Password | Set the password for HIVE connection. | | Database name | Enter the database name of the HIVE connection. | | Jdbc connection parameters | Parameter settings for HIVE connection, in JSON format. | NOTICE: If you wish to execute multiple HIVE SQL in the same session, you could set `support.hive.oneSession = true` in `common.properties`. It is helpful when you try to set env variables before running HIVE SQL. Default value of `support.hive.oneSession` is `false` and multi-SQLs run in different sessions. NOTICE: If Kerberos is disabled, ensure the parameter `hadoop.security.authentication.startup.state` is false, and parameter `java.security.krb5.conf.path` value sets null. If Kerberos is enabled, needs to set the following parameters in `common.properties`: ```conf hadoop.security.authentication.startup.state=true java.security.krb5.conf.path=/opt/krb5.conf login.user.keytab.username=hdfs-mycluster@ESZ.COM login.user.keytab.path=/opt/hdfs.headless.keytab ``` Yes, could use this datasource by default.
Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: ssh.md

This data source is used for RemoteShell component to execute commands remotely. Data Source: SSH Data Source Name: Enter the name of the data source Description: Enter the description of the data source IP Hostname: Enter the IP to connect to SSH Port: Enter the port to connect to SSH Username: Set the username to connect to SSH Password: Set the password to connect to SSH Public Key: Set the public key to connect to SSH