cwielenb/CNCF-Dataset huggingface dataset:
Project: Apache Spark (App Definition and Development / Streaming & Messaging)
File: streaming-kafka-integration.md

layout: global title: Spark Streaming + Kafka Integration Guide license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. is publish-subscribe messaging rethought as a distributed, partitioned, replicated commit log service. Please read the thoroughly before starting an integration using Spark. At the moment, Spark requires Kafka 0.10 and higher. See <a href="streaming-kafka-0-10-integration.html">Kafka 0.10 integration documentation</a> for details.
Project: Shifu (Provisioning / Automation & Configuration)
File: CHANGELOG-v0.36.0.md

Update Readme picture to show shifu's architecture by @saiyan86 in update shifu's messaging to "Kubernetes native IoT gateway" by @saiyan86 in Bump golang.org/x/crypto from 0.17.0 to 0.18.0 by @dependabot in Bump golang.org/x/net from 0.19.0 to 0.20.0 by @dependabot in Bump github.com/DATA-DOG/go-sqlmock from 1.5.1 to 1.5.2 by @dependabot in Bump github.com/Azure/azure-sdk-for-go/sdk/ai/azopenai from 0.4.0 to 0.4.1 by @dependabot in Bump github.com/onsi/ginkgo/v2 from 2.13.2 to 2.14.0 by @dependabot in Bump sigs.k8s.io/controller-runtime from 0.16.3 to 0.17.0 by @dependabot in Full Changelog: https://github.com/Edgenesis/shifu/compare/v0.35.0...v0.36.0
Project: Shifu (Provisioning / Automation & Configuration)
File: CHANGELOG-v0.7.0.md

remove the extra subfolder checkin by @kris21he in https://github.com/Edgenesis/shifu/pull/405 update-readme by @rachelzhang0922 in https://github.com/Edgenesis/shifu/pull/402 smaller gifs by @rachelzhang0922 in https://github.com/Edgenesis/shifu/pull/403 Update README.md by @rachelzhang0922 in https://github.com/Edgenesis/shifu/pull/404 Full Changelog: https://github.com/Edgenesis/shifu/compare/v0.6.0...v0.7.0
Project: Lagoon (App Definition and Development / Application Definition & Image Build)
File: project-default-users-keys.md

When a Lagoon project is created, by default an associated SSH "project key" is generated and the private key made available inside the CLI pods of the project. A service account `default-user@project` is also created and given `MAINTAINER` access to the project. The SSH "project key" is attached to that `default-user@project`. The result of this is that from inside the CLI pod of any environment it is possible to SSH to any other environment within the same project. This access is used for running tasks from the command line such as synchronizing databases between environments \(e.g. drush `sql-sync`\). There is more information on the `MAINTAINER` role available in the documentation. It is possible to specify an SSH private key when creating a project, but this is not recommended as it has security implications.
Project: KICS (Provisioning / Security & Compliance)
File: 7d05ca25-91b4-42ee-b6f6-b06611a87ce8.md

title: Indefinitely Databricks Token Lifetime hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 7d05ca25-91b4-42ee-b6f6-b06611a87ce8 Query name: Indefinitely Databricks Token Lifetime Platform: Terraform Severity: <span style="color:#ff7213">Medium</span> Category: Insecure Defaults URL: Token has an indefinitely lifetime<br> ```tf title="Positive test num. 1 - tf file" hl_lines="1" resource "databricks_token" "positive" { provider = databricks.created_workspace comment = "Terraform Provisioning" } ``` ```tf title="Negative test num. 1 - tf file" resource "databricks_token" "negative" { provider = databricks.created_workspace comment = "Terraform Provisioning" // 100 day token lifetime_seconds = 8640000 } ```
Project: KubeVela (App Definition and Development / Application Definition & Image Build)
File: check-metrics.eg.md

```yaml apiVersion: core.oam.dev/v1beta1 kind: Application metadata: name: canary-demo annotations: app.oam.dev/publishVersion: v2 spec: components: name: canary-demo type: webservice properties: image: wangyikewyk/canarydemo:v2 ports: port: 8090 traits: type: scaler properties: replicas: 5 type: gateway properties: domain: canary-demo.com http: "/version": 8090 workflow: steps: name: 200-status-percent-2-phase type: check-metrics timeout: 3m properties: query: sum(irate(nginxingresscontrollerrequests{host="canary-demo.com",status="200"}[5m]))/sum(irate(nginxingresscontrollerrequests{host="canary-demo.com"}[2m])) promAddress: "http://prometheus-server.o11y-system.svc:9090" condition: ">=0.95" duration: 2m ```
Project: SPIRE (Provisioning / Key Management)
File: plugin_agent_nodeattestor_k8s_psat.md

Must be used in conjunction with the server-side k8s_psat plugin The `k8s_psat` plugin attests nodes running inside of Kubernetes. The agent reads and provides the signed projected service account token (PSAT) to the server. In addition to service account data, PSAT embeds the pod name and UID on its claims. This allows SPIRE to create more fine-grained attestation policies for agents. The server-side `k8s_psat` plugin will generate a SPIFFE ID on behalf of the agent of the form: ```xml spiffe://<trustdomain>/spire/agent/k8spsat/<cluster>/<node_UID> ``` The main configuration accepts the following values: | Configuration | Description | Default | |||| | `cluster` | Name of the cluster. It must correspond to a cluster configured in the server plugin. | | | `token_path` | Path to the projected service account token on disk | "/var/run/secrets/tokens/spire-agent" | A sample configuration with the default token path: ```hcl NodeAttestor "k8s_psat" { plugin_data { cluster = "MyCluster" } } ``` Its k8s volume definition: ```yaml volumes: name: spire-agent projected: sources: serviceAccountToken: path: spire-agent expirationSeconds: 600 audience: spire-server ``` And volume mount: ```yaml volumeMounts: mountPath: /var/run/secrets/tokens name: spire-agent ``` A full example of this attestor is provided in . This attestor is based on two Kubernetes beta features (since k8s v1.12): TokenRequest and TokenRequestProjection. TokenRequest exposes the ability to obtain finely scoped service account tokens from the Kubernetes API Server. TokenRequestProjection facilitates the automatic creation and mounting of such a token into a container.
Project: StarRocks (App Definition and Development / Database)
File: ADMIN_SET_REPLICA_STATUS.md

displayed_sidebar: "English" This statement is used to set the status of the specified replicas. This command currently only used to manually set the status of some replicas to BAD or OK, allowing the system to automatically repair these replicas. :::tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in to grant this privilege. ::: ```sql ADMIN SET REPLICA STATUS PROPERTIES ("key" = "value", ...) ``` The following attributes are currently supported: "table_id': required. Specify a Tablet Id. "backend_id": required. Specify a Backend Id. "status": required. Specify the status. Currently only "bad" and "ok" are supported. If the specified replica does not exist or its status is bad, the replica will be ignored. Note: Replicas set to Bad status may be dropped immediately, please proceed with caution. Set the replica status of tablet 10003 on BE 10001 to bad. ```sql ADMIN SET REPLICA STATUS PROPERTIES("tabletid" = "10003", "backendid" = "10001", "status" = "bad"); ``` Set the replica status of tablet 10003 on BE 10001 to ok. ```sql ADMIN SET REPLICA STATUS PROPERTIES("tabletid" = "10003", "backendid" = "10001", "status" = "ok"); ```
Project: BigchainDB (App Definition and Development / Database)
File: write-docs.md

<! Copyright 2020 Interplanetary Database Association e.V., BigchainDB and IPDB software contributors. SPDX-License-Identifier: (Apache-2.0 AND CC-BY-4.0) Code is Apache-2.0 and docs are CC-BY-4.0 > If you're writing code, you should also update any related docs. However, you might want to write docs only, such as: General explainers Tutorials Courses Code explanations How BigchainDB relates to other blockchain things News from recent events You can certainly do that! The docs for BigchainDB Server live under ``bigchaindb/docs/`` in the ``bigchaindb/bigchaindb`` repo. There are docs for the Python driver under ``bigchaindb-driver/docs/`` in the ``bigchaindb/bigchaindb-driver`` repo. There are docs for the JavaScript driver under ``bigchaindb/js-bigchaindb-driver`` in the ``bigchaindb/js-bigchaindb-driver`` repo. The source code for the BigchainDB website is in a private repo, but we can give you access if you ask. The (one for each spec version) are in the ``bigchaindb/BEPs`` repo. You can write the docs using Markdown (MD) or RestructuredText (RST). Sphinx can understand both. RST is more powerful. ReadTheDocs will automatically rebuild the docs whenever a commit happens on the ``master`` branch, or on one of the other branches that it is monitoring.
Project: KICS (Provisioning / Security & Compliance)
File: c38d630d-a415-4e3e-bac2-65475979ba88.md

title: Body Parameter With Wrong Property hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: c38d630d-a415-4e3e-bac2-65475979ba88 Query name: Body Parameter With Wrong Property Platform: OpenAPI Severity: <span style="color:#5bc0de">Info</span> Category: Structure and Semantics URL: The Body Parameter Object should only have the following properties defined - 'name', 'in', 'description', 'required', and 'schema'<br> ```json title="Positive test num. 1 - json file" hl_lines="43 19" { "swagger": "2.0", "info": { "title": "Simple API Overview", "version": "1.0.0" }, "paths": { "/": { "get": { "parameters": [ { "name": "limit2", "in": "body", "description": "max records to return", "required": true, "schema": { "type": "string" }, "desc": { "type": "string" } } ], "operationId": "listVersionsv2", "summary": "List API versions", "responses": { "200": { "description": "200 response" } } } } }, "parameters": { "limitParam": { "name": "limit", "in": "body", "description": "max records to return", "required": true, "schema": { "type": "string" }, "desc": { "type": "string" } } } } ``` ```yaml title="Positive test num. 2 - yaml file" hl_lines="20 30" swagger: "2.0" info: title: Simple API Overview version: 1.0.0 paths: "/": get: operationId: listVersionsv2 summary: List API versions responses: "200": description: 200 response parameters: name: limit2 in: body description: max records to return required: true schema: type: string desc: type: string parameters: limitParam: name: limit in: body description: max records to return required: true schema: type: string desc: type: string ``` ```json title="Negative test num. 1 - json file" { "swagger": "2.0", "info": { "title": "Simple API Overview", "version": "1.0.0" }, "paths": { "/": { "get": { "parameters": [ { "name": "limit2", "in": "body", "description": "max records to return", "required": true, "schema": { "type": "integer" } } ], "operationId": "listVersionsv2", "summary": "List API versions", "responses": { "200": { "description": "200 response" } } } } }, "parameters": { "limitParam": { "name": "limit", "in": "body", "description": "max records to return", "required": true, "schema": { "type": "integer" } } } } ``` ```yaml title="Negative test num. 2 - yaml file" swagger: "2.0" info: title: Simple API Overview version: 1.0.0 paths: "/": get: operationId: listVersionsv2 summary: List API versions responses: "200": description: 200 response parameters: name: limit2 in: body description: max records to return required: true schema: type: object parameters: limitParam: name: limit in: body description: max records to return required: true schema: type: object ```
Project: KICS (Provisioning / Security & Compliance)
File: da905474-7454-43c0-b8d2-5756ab951aba.md

title: KMS Key With Vulnerable Policy hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: da905474-7454-43c0-b8d2-5756ab951aba Query name: KMS Key With Vulnerable Policy Platform: CloudFormation Severity: <span style="color:#bb2124">High</span> Category: Insecure Configurations URL: Checks if the policy is vulnerable and needs updating.<br> ```json title="Positive test num. 1 - json file" hl_lines="9" { "Resources": { "RSASigningKey": { "Type": "AWS::KMS::Key", "Properties": { "Description": "RSA-3047 asymmetric CMK for signing and verification", "KeySpec": "RSA_3072", "KeyUsage": "SIGN_VERIFY", "KeyPolicy": { "Version": "2012-10-17", "Id": "key-default-1", "Statement": [ { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "kms:*", "Resource": "*" } ] } } } } } ``` ```yaml title="Positive test num. 2 - yaml file" hl_lines="8" Resources: RSASigningKey: Type: AWS::KMS::Key Properties: Description: RSA-3047 asymmetric CMK for signing and verification KeySpec: RSA_3072 KeyUsage: SIGN_VERIFY KeyPolicy: Version: '2012-10-17' Id: key-default-1 Statement: Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: '*' Action: kms:* Resource: '*' ``` ```json title="Positive test num. 3 - json file" hl_lines="5" { "Resources": { "RSASigningKey2": { "Type": "AWS::KMS::Key", "Properties": { "Description": "RSA-3047 asymmetric CMK for signing and verification", "KeySpec": "RSA_3072", "KeyUsage": "SIGN_VERIFY" } } } } ``` <details><summary>Positive test num. 4 - yaml file</summary> ```yaml hl_lines="4" Resources: RSASigningKey2: Type: AWS::KMS::Key Properties: Description: RSA-3047 asymmetric CMK for signing and verification KeySpec: RSA_3072 KeyUsage: SIGN_VERIFY ``` </details> ```json title="Negative test num. 1 - json file" { "Resources": { "RSASigningKey": { "Type": "AWS::KMS::Key", "Properties": { "Description": "RSA-3047 asymmetric CMK for signing and verification", "KeySpec": "RSA_3072", "KeyUsage": "SIGN_VERIFY", "KeyPolicy": { "Version": "2012-10-17", "Id": "key-default-1", "Statement": [ { "Sid": "Allow administration of the key", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:role/Admin" }, "Action": [ "kms:Create*", "kms:Describe*", "kms:Enable*", "kms:List*", "kms:Put*", "kms:Update*", "kms:Revoke*", "kms:Disable*", "kms:Get*", "kms:Delete*", "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion" ], "Resource": "*" } ] } } } } } ``` ```yaml title="Negative test num. 2 - yaml file" Resources: RSASigningKey: Type: AWS::KMS::Key Properties: Description: RSA-3047 asymmetric CMK for signing and verification KeySpec: RSA_3072 KeyUsage: SIGN_VERIFY KeyPolicy: Version: '2012-10-17' Id: key-default-1 Statement: Sid: Allow use of the key Effect: Allow Principal: AWS: arn:aws:iam::111122223333:role/Developer Action: kms:Sign kms:Verify kms:DescribeKey Resource: '*' ```
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 04_basic_pcollections.md

Prompt: What is a `PCollection` in Apache Beam? Response: A `PCollection` in Apache Beam is an unordered bag of elements. Each `PCollection` is a potentially distributed, homogeneous data set or data stream, and is owned by the specific `Pipeline` object for which it is created. Its the primary data structure used in Apache Beam pipelines to handle large-scale data processing in batch and streaming modes. The following example shows how to create a `PCollection` using the `Create` transform: ```python import apache_beam as beam with beam.Pipeline() as pipeline: pcollection = pipeline | beam.Create([...]) # Create a PCollection ``` A `PCollection` can either be bounded or unbounded, making it versatile for different . Bounded `PCollection` objects represent a finite data set, such as files or databases, ideal for batch processing. Unbounded `PCollection` objects, on the other hand, represent data streams that continuously grow over time, such as real-time event logs, suitable for stream processing. Beams computational patterns and transforms are focused on situations where distributed data-parallel computation is required. Therefore, a `PCollection` has the following key characteristics: All elements must be of the same type (with support of structured types). Every `PCollection` has a coder, which is a specification of the binary format of the elements. Elements are immutable. They can't be altered after creation. Random access to individual elements of the collection isn't allowed. Beam encodes each element for distributed processing. For more information about `PCollection` objects in Apache Beam and usage patterns, see .
Project: Meshery (Provisioning / Automation & Configuration)
File: mesheryctl-components.md

layout: default title: mesheryctl-components permalink: reference/mesheryctl/components redirect_from: reference/mesheryctl/components/ type: reference display-title: "false" language: en command: components subcommand: nil View list of components and detail of components View list of components and detailed information of a specific component <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl components [flags] </div> </pre> To view the number of components present in Meshery <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl exp components --count </div> </pre> To view list of components <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl exp components list </div> </pre> To view a specific component <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl exp components view [component-name] </div> </pre> To search for a specific component <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl exp components search [component-name] </div> </pre> <pre class='codeblock-pre'> <div class='codeblock'> -c, --count (optional) Get the number of components in total -h, --help help for components </div> </pre> <pre class='codeblock-pre'> <div class='codeblock'> --config string path to config file (default "/home/runner/.meshery/config.yaml") -v, --verbose verbose output </div> </pre> Go back to , if you want to add content manually to the CLI documentation, please refer to the for guidance.
Project: Chef Habitat (App Definition and Development / Application Definition & Image Build)
File: origins.md

+++ title = "Create an Origin" description = "Create an Origin on Chef Habitat Builder" gh_repo = "habitat" [menu] [menu.habitat] title = "Create an Origin" identifier = "habitat/origins Create an Origin" parent = "habitat/origins" weight = 10 +++ An origin is a space on Chef Habitat Builder where you can store, share, and build packages. It is a unique namespace within Chef Habitat Builder, and while you can delete or transfer an origin, you can't rename an origin after it is created. One example of an origin is the "core" origin, which is the set of foundational packages managed and versioned by the core Chef Habitat maintainers. You can join existing origins by invitation and you can create your own origins. For more on invitations, see . To create an origin, select the Create origin button on the My Origins page which opens the Create New Origin form. (Chef Habitat Builder > My Origins ) First, enter a unique name that you want to associate with your packages. Chef Habitat will only let you create an origin with a unique name. Some examples that you'll see in Chef Habitat Builder are team names, user names, and abstract concepts. Next, choose a privacy setting to set as the default for new packages. You can override this setting when uploading individual packages from the CLI or by connecting a plan file that declares a package as private. The difference between public and private packages is: Anyone can find and use public packages Only users with origin membership can find and use private packages When you select Save and Continue, Chef Habitat Builder: Creates your origin Creates an Redirects Chef Habitat Builder to the origin page Use the commands to manage your origins from the command line. Create an origin from the command line with the command ```bash hab origin create <origin> ``` The results of this command differ slightly from creating an origin on the Chef Habitat Builder site. The CLI command: Creates an origin on the Chef Habitat Builder site Does not generate an origin key pair For more information, see the CLI documentation.
Project: Apache Hadoop (App Definition and Development / Database)
File: CHANGELOG.0.3.2.md

<! --> | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | log4j changes for hadoopStreaming | Major | . | Michel Tourn | Doug Cutting | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | running without the hadoop script causes warnings about log4j not being configured correctly | Major | . | Owen O'Malley | Owen O'Malley | | | add FAQ to Wiki | Major | documentation | Doug Cutting | Doug Cutting | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | AllTestDriver has incorrect class name for DistributedFSCheck test | Major | . | Konstantin Shvachko | | | | namenode should not log failed mkdirs at warning level | Minor | . | Hairong Kuang | Hairong Kuang | | | Data nodes cannot re-join the cluster once connection is lost | Blocker | . | Konstantin Shvachko | Hairong Kuang | | | Race condition in Configuration.getLocalPath() | Major | . | p sutter | Sameer Paranjpye | | | the datanode crashes if it starts before the namenode | Critical | . | Owen O'Malley | Owen O'Malley | | | Datanodes need to catch SocketTimeoutException and UnregisteredDatanodeException | Major | . | Konstantin Shvachko | Konstantin Shvachko | | | hadoop dfs commands should not output superfluous data to stdout | Minor | . | Yoram Arnon | Owen O'Malley | | | dfs client error retries aren't happening (already being created and not replicated yet) | Major | . | Owen O'Malley | Owen O'Malley | | | job fails because of "No valid local directories in property: " exception | Major | . | Yoram Arnon | Owen O'Malley | | | dfs timeout on open | Major | . | Owen O'Malley | Owen O'Malley |
Project: Project Syn (Provisioning / Automation & Configuration)
File: 02_feature_request.md

name: Feature request about: Suggest an idea for this project labels: enhancement <!-- Please let us know what you are trying to do and how you would want to do it differently? Is it something you currently cannot do? Is this related to an issue/problem? --> <!-- Can you achieve the same result doing it in an alternative way? Is the alternative considerable? -->