cwielenb/CNCF-Dataset huggingface dataset:
Project: etcd (Orchestration & Management / Coordination & Service Discovery)
File: arm64-infra.md

The infrastructure to build for arm64 is provided by via the . Previously, several maintainers were responsible for managing two bare-metal machines with a self-hosted runner installed. This was a manual process, and side effects could be left over from previous builds. As part of a joint program between Ampere and the CNCF, is providing managed Arm64 builds. To use the new infrastructure, add the following to your workflow: ```yaml runs-on: actuated-arm64-8cpu-8gb ``` The vCPUs and RAM are customizable, i.e. `actuated-arm64-8cpu-16gb` or `actuated-arm64-8cpu-32gb`. For urgent support, contact @alexellis or the .
Project: APISIX (Orchestration & Management / API Gateway)
File: service.md

title: Service <!-- --> A Service is an abstraction of an API (which can also be understood as a set of abstractions). It usually corresponds to an upstream service abstraction. The relationship between Routes and a Service is usually N:1 as shown in the image below. As shown, different Routes could be bound to the same Service. This reduces redundancy as these bounded Routes will have the same and configurations. For more information about Service, please refer to . The following example creates a Service that enables the `limit-count` Plugin and then binds it to the Routes with the ids `100` and `101`. :::note You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command: ```bash adminkey=$(yq '.deployment.admin.adminkey[0].key' conf/config.yaml | sed 's/"//g') ``` ::: Create a Service. ```shell curl http://127.0.0.1:9180/apisix/admin/services/200 \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "plugins": { "limit-count": { "count": 2, "time_window": 60, "rejected_code": 503, "key": "remote_addr" } }, "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:1980": 1 } } }' ``` create new Route and reference the service by id `200` ```shell curl http://127.0.0.1:9180/apisix/admin/routes/100 \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "methods": ["GET"], "uri": "/index.html", "service_id": "200" }' ``` ```shell curl http://127.0.0.1:9180/apisix/admin/routes/101 \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "methods": ["GET"], "uri": "/foo/index.html", "service_id": "200" }' ``` We can also specify different Plugins or Upstream for the Routes than the ones defined in the Service. The example below creates a Route with a `limit-count` Plugin. This Route will continue to use the other configurations defined in the Service (here, the Upstream configuration). ```shell curl http://127.0.0.1:9180/apisix/admin/routes/102 \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "uri": "/bar/index.html", "id": "102", "service_id": "200", "plugins": { "limit-count": { "count": 2000, "time_window": 60, "rejected_code": 503, "key": "remote_addr" } } }' ``` :::note When a Route and a Service enable the same Plugin, the one defined in the Route is given the higher priority. :::
Project: SRPC (Orchestration & Management / Remote Procedure Call)
File: docs-03-server.md

Each server corresponds to one port Each server corresponds to one specific network communication protocol One service may be added into multiple Servers One Server may have one or more Services, but the ServiceName must be unique within that Server Services from different IDLs can be added into the same Server You can follow the detailed example below: Follow the above `ExampleServiceImpl` Service First, create an RPC Server and determine the proto file. Then, create any number of Service instances and any number of Services for different protocols, and add these services to the Server through the `add_service()`interface. Finally, use `start()` or `serve()` to start the services in the Server and handle the upcoming rpc requests through the Server. Imagine that we can also derive more Serivce from `Example::Service`, which have different implementations of rpc `Echo`. Imagine that we can create N different RPC Servers on N different ports, serving on different network protocols. Imagine that we can use `addservice()` to add the same ServiceIMPL instance on different Servers, or we can use `addservice()` to add different ServiceIMPL instances on the same server. Imagine that we can use the same `ExampleServiceImpl`, serving BRPC-STD, SRPC-STD, SRPC-Http at three different ports at the same time. And we can use `add_service()` to add one `ExampleServiceImpl` related to Protobuf IDL and one `AnotherThriftServiceImpl` related to Thrift IDL to the same SRPC-STD Server, and the two IDLs work perfectly on the same port! ~~~cpp int main() { SRPCServer server_srpc; SRPCHttpServer serversrpchttp; BRPCServer server_brpc; ThriftServer server_thrift; TRPCServer server_trpc; TRPCHttpServer servertrpchttp; ExampleServiceImpl impl_pb; AnotherThriftServiceImpl impl_thrift; serversrpc.addservice(&impl_pb); serversrpc.addservice(&impl_thrift); serversrpchttp.addservice(&implpb); serversrpchttp.addservice(&implthrift); serverbrpc.addservice(&impl_pb); serverthrift.addservice(&impl_thrift); servertrpc.addservice(&impl_pb); servertrpchttp.addservice(&implpb); server_srpc.start(1412); serversrpchttp.start(8811); server_brpc.start(2020); server_thrift.start(9090); server_trpc.start(2022); servertrpchttp.start(8822); getchar(); servertrpchttp.stop(); server_trpc.stop(); server_thrift.stop(); server_brpc.stop(); serversrpchttp.stop(); server_srpc.stop(); return 0; } ~~~
Project: Azure Service Fabric (Orchestration & Management / Scheduling & Orchestration)
File: SECURITY.md

<!-- BEGIN MICROSOFT SECURITY.MD V0.0.8 BLOCK --> Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include , , , , , and . If you believe you have found a security vulnerability in any Microsoft-owned repository that meets , please report it to us as described below. Please do not report security vulnerabilities through public GitHub issues. Instead, please report them to the Microsoft Security Response Center (MSRC) at . If you prefer to submit without logging in, send email to . If possible, encrypt your message with our PGP key; please download it from the . You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at . Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) Full paths of source file(s) related to the manifestation of the issue The location of the affected source code (tag/branch/commit or direct URL) Any special configuration required to reproduce the issue Step-by-step instructions to reproduce the issue Proof-of-concept or exploit code (if possible) Impact of the issue, including how an attacker might exploit the issue This information will help us triage your report more quickly. If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our page for more details about our active programs. We prefer all communications to be in English. Microsoft follows the principle of . <!-- END MICROSOFT SECURITY.MD BLOCK -->
Project: Easegress (Orchestration & Management / API Gateway)
File: 1.1.Quick-Start.md

The basic usage of Easegress is to quickly set up a proxy for the backend servers. Easegress can be installed from pre-built binaries or from source. For details, see . Then we can execute the server: ```bash $ easegress-server 2023-09-06T15:12:49.256+08:00 INFO cluster/config.go:110 config: advertise-client-urls: ... ... ``` By default, Easegress opens ports 2379, 2380, and 2381; however, you can modify these settings along with other arguments either in the configuration file or via command-line arguments. For a complete list of arguments, please refer to the `easegress-server --help` command. After launching successfully, we could check the status of the one-node cluster. ```bash $ egctl get member ... $ egctl describe member ... ``` Assuming you have two backend HTTP services running at `127.0.0.1:9095` and `127.0.0.1:9096`, you can initiate an HTTP proxy from port 10080 to these backends using the following command: ```bash $ egctl create httpproxy demo --port 10080 \ --rule="/pipeline=http://127.0.0.1:9095,http://127.0.0.1:9096" \ --rule="/prefix*=http://127.0.0.1:9097" ``` Then try it: ```bash $ curl -v 127.0.0.1:10080/pipeline ``` The request will be forwarded to either `127.0.0.1:9095/pipeline` or `127.0.0.1:9096/pipeline`, utilizing a round-robin load-balancing policy. ```bash $ curl -v 127.0.0.1:10080/prefix/123 ``` The request will be forwarded to `127.0.0.1:9097/prefix/123`. The `egctl create httpproxy` command mentioned above is actually syntactic sugar; it creates two Easegress resources under the hood: `HTTPServer` and `Pipeline`. Now let's create them using yaml files: ```bash $ echo ' kind: HTTPServer name: demo port: 10080 https: false rules: paths: path: /pipeline backend: demo-0 pathPrefix: /prefix backend: demo-1 ' | egctl create -f - ``` More details about . ```bash $ echo ' name: demo-0 kind: Pipeline flow: filter: proxy filters: name: proxy kind: Proxy pools: servers: url: http://127.0.0.1:9095 url: http://127.0.0.1:9096 loadBalance: policy: roundRobin name: demo-1 kind: Pipeline flow: filter: proxy filters: name: proxy kind: Proxy pools: servers: url: http://127.0.0.1:9097' | egctl create -f - ``` More details about . You can also modify, update, or delete these resources using `egctl`; for more details, refer to the guide.
Project: Apache bRPC (Orchestration & Management / Remote Procedure Call)
File: lalb.md

Locality-aware load balancing(LALB) is an algorithm that can send requests to downstream servers with the lowest latency timely and automatically. The algorithm is originated from the DP system and is now added to brpc! Problems that LALB can solve: Round-robin and random policy is not good at scheduling requests since configurations and latencies of downstream servers are different. Downstream servers, offline servers and other servers are deployed hybridly and it is hard to predict performance. Automatically schedule most of the requests to the server in the same machine. When a problem occurs, try the server acrossing the machines. Visit the server in the same server room first. When a problem occurs, try another server room. ... The most common algorithm to redirect requests is round-robin and random. The premise of these two methods is that the downstream servers and networks are similar. But in the current online environment, especially the hybrid environment, it is difficult to achieve because: Each machine runs a different combination of programs, along with some offline tasks, the available resources of the machine are continuously changing. The configurations of machines are different. The latencies of network are different. These problems have always been there, but are hidden by machine monitoring from hard-working OPs. There are also some attempts in the level of frameworks. For example, in UB redirects requests based on cpu usage of downstream machines and obviously it cannot solve the latency-related issues, or even cpu issues: since it is implemented as regularly reloading a list of weights, one can imagine that the update frequency cannot be high. A lot of requests may timeout when the load balancer reacts. And there is a math problem here: how to change cpu usage to weight.
Project: Apache Mesos (Orchestration & Management / Scheduling & Orchestration)
File: 2014-06-24-mesos-community-survey-2014-results.md

layout: post title: 2014 Mesos Community Survey Results permalink: /blog/mesos-community-survey-2014-results/ published: true post_author: display_name: Dave Lester twitter: davelester tags: Community, Survey Below are results of a community survey I led in May to get the pulse of the Apache Mesos community and improve our understanding of how others are using the software. We received a total of 55 responses. While I think the responses below speak for themselves, I wanted to highlight a few things: A majority of responses came from adopters who have been using Mesos for more than 2 months. Mesos had a of 41.5. Documentation was the #1 community request for improving Mesos, followed by integrated Docker support. Unedited responses have been published as a . 36% of adopters are running Mesos in a cluster of 6-20 machines. Almost half of Mesos adopters are running in private data centers. A majority of adopters running in the public cloud use AWS. Hadoop, Marathon, and Chronos were the most-popular frameworks that are running in production. This is the first time we've run a community survey like this; feedback is welcome on how we may improve it when we run it again. (34 responses for "No, but please refrain from adding us at this time", 8 responses for "No, please add us", and 15 responses for "Yes, we're on the list!") The unedited responses to this question have been published as a .
Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: PULL_REQUEST_TEMPLATE.md

<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! --> <!--(For example: This pull request adds checkstyle plugin).--> <!--(for example:) Add maven-checkstyle-plugin to root pom.xml --> <!--(Please pick either of the following options)--> This pull request is code cleanup without any test coverage. (or) This pull request is already covered by existing tests, such as (please describe tests). (or) This change added tests and can be verified as follows: <!--(example:) Added dolphinscheduler-dao tests for end-to-end. Added CronUtilsTest to verify the change. Manually verified the change by testing locally. --> (or) If your pull request contain incompatible change, you should also add it to `docs/docs/en/guide/upgrede/incompatible.md`
Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: datafactory.md

In the world of big data, raw, unorganized data is often stored in relational, non-relational, and other storage systems. is a managed cloud service that's built for these complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects. DolphinScheduler DataFactory functions: Create Azure DataFactory task enable DolphinScheduler to schedule DataFactory pipelines, keep query pipeline execution status until finish the execution. ResourceGroup: Azure owns a resource group DataFactory: Azure owns a data factory under the resource group Pipeline: Azure owns pipelines under the assigned resource group and data factory Application: Azure owns an application has permission to visit data factory, enables using SDK to call data factory functions ApplicationClientSecret: To apply for client secret under Application `Certificates & secrets` AZURE-CLI: Install Azure auth application AZURE-CLI on machine, please refer to First, visit the `Subscription` page and choose `Access control (IAM)`, then click `Add role assignment` to the authorization page. After that, select `Contributor` role which satisfy functions calls in data factory. Then click `Members` page, and click `Select members`. Search application name or application `Object ID` to assign `Contributor` role to application. Config Azure configurations, modify `azure` related configurations in `common.properties` resource.azure.client.id: Azure Application Application (client) ID resource.azure.client.secret: Azure Application client secret under `Certificates & secrets` resource.azure.subId: Data factory subscription ID resource.azure.tenant.id: Azure Active Directory tenant ID ```yaml resource.azure.client.id=minioadmin resource.azure.client.secret=minioadmin resource.azure.subId=minioadmin resource.azure.tenant.id=minioadmin ``` Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the DAG editing page. Drag from the toolbar <img src="../../../../img/tasks/icons/datafactory.png" width="15"/> task node to canvas. Please refer to `Default Task Parameters` section for default parameters. Here are some specific parameters for the DataFactory plugin: factoryName: data factory name resourceGroupName: resource group name of the data factory pipelineName: pipeline name under the resource group and data factory The following shows the task plugin example:
Project: Tengine (Orchestration & Management / Service Proxy)
File: ngx_http_spdy_module.md

NOTE This module has been updated to official nginx SPDY/3.1 module, the document is available here: http://nginx.org/en/docs/http/ngxhttpspdy_module.html This document only applies to Tengine-2.1.0 and its old version. But the listen option `spdy_detect` can still be used with SPDY/3.1. Tengine-2.2.0 or later provides support for HTTP/2 and supersedes the SPDY module. ngx\_http\_spdy\_module Tengine added SPDY/3 support to this module. The new directives are listed below. Syntax: spdy\_version [2|3] Default: 3 Context: http, server Specify the version of current SPDY protocol. Syntax: spdy\_flow\_control on|off Default: on Context: http, server Turn on or off with SPDY flow control. Syntax: spdy\_init\_recv\_window\_size size Default: 64k Context: http, server Specify the receiving window size for SPDY. By default, it's 64K. It will send a WINDOW UPDATE frame when it receives half of the window size data every time. Syntax: listen address[:port] [spdy_detect] [ssl] Default: Context: listen directive Server can work for SPDY and HTTP on the same port with this directive. Note that the server will examine the first byte of one connection and determine whether the connection is SPDY or HTTP based on what it looks like (0x80 or 0x00 for SPDY). Server will listen on port 80 for SPDY and HTTP, for example: listen 80 spdy_detect; Server will detect whether SPDY or HTTP is used without using a TLS extension (NPN), for example: listen 443 ssl spdy_detect; Server can detect whether SPDY or HTTP is used directly, and also it can negotiate with client via NPN, for example: listen 443 ssl spdy_detect spdy;
Project: Contour (Orchestration & Management / Service Proxy)
File: profiling-contour.md

Contour exposes the handlers for `go tool pprof` and `go tool trace` by default on `127.0.0.1:6060`. This service is useful for profiling Contour. To access it from your workstation use `kubectl port-forward` like so, ```bash $ CONTOUR_POD=$(kubectl -n projectcontour get pod -l app=contour -o name | head -1) $ kubectl -n projectcontour port-forward $CONTOUR_POD 6060 ```
Project: Gloo (Orchestration & Management / API Gateway)
File: glooctl_create_virtualservice.md

title: "glooctl create virtualservice" weight: 5 Create a Virtual Service A virtual service describes the set of routes to match for a set of domains. Virtual services are containers for routes assigned to a domain or set of domains. Virtual services must not have overlapping domains, as the virtual service to match a request is selected by the Host header (in HTTP1) or :authority header (in HTTP2). When using Gloo Enterprise, virtual services can be configured with rate limiting, oauth, apikey auth, and more. ``` glooctl create virtualservice [flags] ``` ``` --display-name string descriptive name of virtual service (defaults to resource name) --domains strings comma separated list of domains --enable-rate-limiting enable rate limiting features for this virtual service -h, --help help for virtualservice --name string name of the resource to read or write -n, --namespace string namespace for reading or writing resources (default "gloo-system") --rate-limit-requests uint32 requests per unit of time (default 100) --rate-limit-time-unit string unit of time over which to apply the rate limit (default "MINUTE") ``` ``` -c, --config string set the path to the glooctl config file (default "<home_directory>/.gloo/glooctl-config.yaml") --consul-address string address of the Consul server. Use with --use-consul (default "127.0.0.1:8500") --consul-allow-stale-reads Allows reading using Consul's stale consistency mode. --consul-datacenter string Datacenter to use. If not provided, the default agent datacenter is used. Use with --use-consul --consul-root-key string key prefix for for Consul key-value storage. (default "gloo") --consul-scheme string URI scheme for the Consul server. Use with --use-consul (default "http") --consul-token string Token is used to provide a per-request ACL token which overrides the agent's default token. Use with --use-consul --dry-run print kubernetes-formatted yaml rather than creating or updating a resource -i, --interactive use interactive mode --kube-context string kube context to use when interacting with kubernetes --kubeconfig string kubeconfig to use, if not standard one -o, --output OutputType output format: (yaml, json, table, kube-yaml, wide) (default table) --use-consul use Consul Key-Value storage as the backend for reading and writing config (VirtualServices, Upstreams, and Proxies) ``` - Create a Gloo resource
Project: gRPC (Orchestration & Management / Remote Procedure Call)
File: grpc-cq.md

Author: Sree Kuchibhotla (@sreecha) - Sep 2018 Code: This document gives an overview of completion queue architecture and focuses mainly on the interaction between completion queue and the Polling engine layer. Completion queue has two attributes Completion_type: GRPCCQNEXT: grpccompletionqueuenext() can be called (but not grpccompletionqueuepluck()) GRPCCQPLUCK: grpccompletionqueuepluck() can be called (but not grpccompletionqueuenext()) GRPCCQCALLBACK: The tags in the queue are function pointers to callbacks. Also, neither next() nor pluck() can be called on this Polling_type: GRPCCQNONPOLLING: Threads calling completionqueue_next/pluck do not do any polling GRPCCQDEFAULTPOLLING: Threads calling completionqueue_next/pluck do polling GRPCCQNON_LISTENING: Functionally similar to default polling except for a boolean attribute that states that the cq is non-listening. This is used by the grpc-server code to not associate any listening sockets with this completion-queues pollset ``` C++ grpccompletionqueue_next(cq, deadline)/pluck(cq, deadline, tag) { while(true) { \\ 1. If an event is queued in the completion queue, dequeue and return \\ (in case of pluck() dequeue only if the tag is the one we are interested in) \\ 2. If completion queue shutdown return \\ 3. In case of pluck, add (tag, worker) pair to the tag<->worker map on the cq \\ 4. Call grpcpollsetwork(cqs-pollset, deadline) to do polling \\ Note that if this function found some fds to be readable/writable/error, \\ it would have scheduled those closures (which may queue completion events \\ on SOME completion queue - not necessarily this one) } } ``` ``` C++ grpccqend_op(cq, tag) { \\ 1. Queue the tag in the event queue \\ 2. Find the pollset corresponding to the completion queue \\ (i) If the cq is of type GRPCCQNEXT, then KICK ANY worker \\ i.e., call grpcpollsetkick(pollset, nullptr) \\ (ii) If the cq is of type GRPCCQPLUCK, then search the tag<->worker \\ map on the completion queue to find the worker. Then specifically \\ kick that worker i.e call grpcpollsetkick(pollset, worker) } ```