cwielenb/CNCF-Dataset huggingface dataset:
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 35_coders_data_encoding.md

Prompt: What is a coder in Apache Beam? Response: In Apache Beam, a `Coder` is an object responsible for specifying how elements in a given `PCollection` are encoded and decoded into byte strings. The `Coder` objects provide type consistency and safety during distributed processing between workers, ensuring the correct interpretation of data based on its original types. Every `PCollection` in an Apache Beam pipeline requires a corresponding `Coder`. While the Apache Beam SDKs can automatically infer a `Coder` for a `PCollection` based on its element type or the producing transform, there are cases where explicit specification or custom `Coder` development is necessary. It is important to note that multiple `Coder` objects can exist for a single data type. The Apache Beam SDKs use various mechanisms to automatically infer the `Coder` for a `PCollection`. Each pipeline object contains a `CoderRegistry` object representing a mapping of language types to the default coder for those types. In the Apache Beam SDKs for Python and Java, the `Coder` type provides the necessary methods for encoding and decoding data. The SDKs offer various `Coder` subclasses working with standard Python and Java types, available in the `apache_beam.coders` package for Python and the `Coder` package for Java. By default, the Beam SDKs use the typehints (Python) or the type parameters (Java) from the transform's function object (such as `DoFn`) to infer the `Coder` for elements in a `PCollection`. For example, in Apache Beam Python SDK, a `ParDo` annotated with the typehints `@beam.typehints.withinputtypes(int)` and `@beam.typehints.withoutputtypes(str)` indicates that it accepts `int` inputs and produces `str` outputs. The Python SDK automatically infers the default `Coder` for the output `PCollection`, in this case, `BytesCoder` (based on the default `CoderRegistry`). You can use the `CoderRegistry` to look up the default coder for a given type or register a new default coder. The following tables show the default mappings of coders to standard types for any pipeline created using the Beam SDK for Java and Python: | Java Type | Default Coder | ||-| | Double | DoubleCoder | | Instant | InstantCoder | | Integer | VarIntCoder | | Iterable | IterableCoder | | KV | KvCoder | | List | ListCoder | | Map | MapCoder | | Long | VarLongCoder | | String | StringUtf8Coder | | Python Type | Default Coder | |--|-| | int | VarIntCoder | | float | FloatCoder | | str | BytesCoder | | bytes | StrUtf8Coder | | Tuple | TupleCoder |
Project: Harbor (Provisioning / Container Registry)
File: 1-03-DB-user-update-password.md

Test 1-03 - User update password (DB Mode) ======= To verify that a non-admin user can update password when users are managed locally by Harbor (DB mode). User guide This test requires that a Harbor instance is running and available. Harbor is set to authenticate against a local database. ( auth_mode is set to db_auth* .) The user data is stored in a local database. A linux host with Docker CLI installed (Docker client). NOTE: Use a non-admin user for this test case. Admin user has other test cases. A non-admin user logs in to the UI by username. The user changes his/her own password. The user logs out. The same user logs in to the UI by email using the new password. The user can log in using `docker login` command using the new password. The user goes to the page to change his/her own password. Provide invalid values of input to see if validation works: old password is incorrect password input does not compliant to password rule two passwords do not match Password can be changed in Step 2. User can log in using new password in Step 4. In Step 6, the user cannot change password due to various errors. Proper error message should be displayed. None
Project: Chef InSpec (Provisioning / Security & Compliance)
File: google_compute_subnetwork_iam_policy.md

+++ title = "googlecomputesubnetworkiampolicy resource" draft = false gh_repo = "inspec" platform = "gcp" [menu] [menu.inspec] title = "googlecomputesubnetworkiampolicy" identifier = "inspec/resources/gcp/googlecomputesubnetworkiampolicy.md googlecomputesubnetworkiampolicy resource" parent = "inspec/resources/gcp" +++ A `googlecomputesubnetworkiampolicy` is used to test a Google Subnetwork Iam Policy resource ```ruby describe googlecomputesubnetworkiampolicy(project: "project", region: "region", name: "name") do it { should exist } end googlecomputesubnetworkiampolicy(project: "project", region: "region", name: "name").bindings.each do |binding| describe binding do its('role') { should eq 'roles/editor'} its('members') { should include 'user:testuser@example.com'} end end ``` Properties that can be accessed from the `googlecomputesubnetworkiampolicy` resource: `iambindingroles` : The list of roles that exist on the policy. `bindings` : Associates a list of members to a role. `role` : Role that is assigned to members. For example, roles/viewer, roles/editor, or roles/owner. `members` : Specifies the identities requesting access for a Cloud Platform resource. `audit_configs` : Specifies cloud audit logging configuration for this policy. `service` : Specifies a service that will be enabled for audit logging. For example, `storage.googleapis.com`, `cloudsql.googleapis.com`. `allServices` is a special value that covers all services. `auditlogconfigs` : The configuration for logging of each type of permission. `log_type` : The log type that this config enables. For example, ADMINREAD, DATAWRITE or DATA_READ `exempted_members` : Specifies the identities that do not cause logging for this type of permission. Ensure the is enabled for the current project.
Project: kpt (Provisioning / Automation & Configuration)
File: 03-editing-a-package.md

kpt does not maintain any state on your local machine outside of the directory where you fetched the package. Making changes to the package is accomplished by manipulating the local filesystem. At the lowest-level, editing a package is simply a process that either: Changes the resources within that package. Examples: Authoring new a Deployment resource Customizing an existing Deployment resource Modifying the Kptfile Changes the package hierarchy, also called package composition. Examples: Adding a subpackage. Create a new dependent subpackage. At the end of the day, editing a package will result in a Git commit that fully specifies the package. This process can be manual or automated depending on your use case. We will cover package composition later in this chapter. For now, let's focus on editing resources within a package. Before you make any changes to package, you should first initialize and commit the pristine package: ```shell $ git init; git add .; git commit -m "Pristine wordpress package" ``` As mentioned earlier, you can manually edit or author KRM resources using your favorite editor. Since every KRM resource has a known schema, you can take advantage of tooling that assists in authoring and validating resource configuration. For example, [Cloud Code] extensions for VS Code and IntelliJ provide IDE features such as auto-completion, inline documentation, linting, and snippets. For example, if you have VS Code installed, try modifying the resources in the `wordpress` package: ```shell $ code wordpress ``` Oftentimes, you want to automate repetitive or complex operations. Having standardized on KRM for all resources in a package allows us to easily develop automation in different toolchains and languages, as well as at levels of abstraction. For example, setting a label on all the resources in the `wordpress` package can be done using the following function: ```shell $ kpt fn eval wordpress -i set-labels:v0.1 -- env=dev ``` [Chapter 4] discusses different ways of running functions in detail.
Project: YDB (App Definition and Development / Database)
File: cms.md

The following command will let you get the current settings for a cluster or tenant. ```bash ydbd -s <endpoint> admin console configs load --out-dir <config-folder> ``` ```bash ydbd -s <endpoint> admin console configs load --out-dir <config-folder> --tenant <tenant-name> ``` First, you need to pull the desired config as indicated above and then prepare a protobuf file with an update request. ``` Actions { AddConfigItem { ConfigItem { Cookie: "<cookie>" UsageScope { TenantAndNodeTypeFilter { Tenant: "<tenant-name>" } } Config { <config-name> { <full-config> } } } } } ``` The UsageScope field is optional and is needed to use settings for a specific tenant. ```bash ydbd -s <endpoint> admin console configs update <protobuf-file> ```
Project: Polaris (Provisioning / Security & Compliance)
File: checks.md

meta: name: description content: "Fairwinds Polaris | Documentation on setting checks by severity " Each check can be assigned a `severity`. Only checks with a severity of `danger` or `warning` will be validated. The results of these validations are visible on the dashboard. In the case of the validating webhook, only failures with a severity of `danger` will result in a change being rejected. Polaris validation checks fall into several different categories: To change the default severity levels, or to turn checks on or off, you can create your own `config.yaml`: ```yaml checks: tagNotSpecified: ignore runAsRootAllowed: danger pullPolicyNotAlways: warning ```
Project: OAuth2 Proxy (Provisioning / Key Management)
File: github.md

id: github title: GitHub Create a new project: https://github.com/settings/developers Under `Authorization callback URL` enter the correct url ie `https://internal.yourcompany.com/oauth2/callback` The GitHub auth provider supports two additional ways to restrict authentication to either organization and optional team level access, or to collaborators of a repository. Restricting by these options is normally accompanied with `--email-domain=*`. Additionally, all the organizations and teams a user belongs to are set as part of the `X-Forwarded-Groups` header. e.g. `org1:team1,org1:team2,org2:team1` NOTE: When `--github-user` is set, the specified users are allowed to log in even if they do not belong to the specified org and team or collaborators. To restrict by organization only, include the following flag: ```shell --github-org="" # restrict logins to members of this organisation ``` To restrict within an organization to specific teams, include the following flag in addition to `-github-org`: ```shell --github-team="" # restrict logins to members of any of these teams (slug), separated by a comma ``` If you would rather restrict access to collaborators of a repository, those users must either have push access to a public repository or any access to a private repository: ```shell --github-repo="" # restrict logins to collaborators of this repository formatted as orgname/repo ``` If you'd like to allow access to users with read only access to a public repository you will need to provide a for a user that has write access to the repository. The token must be created with at least the `public_repo` scope: ```shell --github-token="" # the token to use when verifying repository collaborators ``` To allow a user to log in with their username even if they do not belong to the specified org and team or collaborators, separated by a comma ```shell --github-user="" #allow logins by username, separated by a comma ``` If you are using GitHub enterprise, make sure you set the following to the appropriate url: ```shell --login-url="http(s)://<enterprise github host>/login/oauth/authorize" --redeem-url="http(s)://<enterprise github host>/login/oauth/access_token" --validate-url="http(s)://<enterprise github host>/api/v3" ```
Project: TiKV (App Definition and Development / Database)
File: deploy.md

We provide multiple deployment methods, but it is recommended to use our TiUP deployment for production environment. Ensure SSH connectivity between tiup's machine and several target nodes topo.yaml: ```yaml global: user: "ubuntu" ssh_port: 22 deploy_dir: "whatever" data_dir: "whatever" pd_servers: host: x.x.x.x host: y.y.y.y host: z.z.z.z tikv_servers: host: x.x.x.x host: y.y.y.y host: z.z.z.z monitoring_servers: host: a.a.a.a grafana_servers: host: b.b.b.b alertmanager_servers: host: c.c.c.c ``` `$ tiup cluster deploy foobar v4.0.0 ./topo.yaml --user ubuntu -i ~/.ssh/id_rsa` `$ tiup cluster start foobar` You're all set!
Project: Druid (App Definition and Development / Database)
File: libraries.md

title: Community and Third Party Software layout: simple_page canonical: 'https://druid.apache.org/libraries' <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at * http://www.apache.org/licenses/LICENSE-2.0 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Edit this page in the repo. To see your changes, navigate to them from the .
Project: Podman (App Definition and Development / Application Definition & Image Build)
File: podman-farm-list.1.md

% podman-farm-list 1 podman\-farm\-list - List the existing farms podman farm list [options] podman farm ls [options] List all the existing farms. Change the default output format. This can be of a supported type like 'json' or a Go template. Valid placeholders for the Go template listed below: | Placeholder | Description | | | | | .Connections | List of all system connections in the farm | | .Default | Indicates whether farm is the default | | .Name | Farm name | | .ReadWrite | Indicates if this farm can be modified using the podman farm commands | List all farms: ``` $ podman farm list Name Connections Default ReadWrite farm1 [f38 f37] false true farm2 [f37] true true ``` , July 2023, Originally compiled by Urvashi Mohnani (umohnani at redhat dot com)
Project: Open Policy Agent (OPA) (Provisioning / Security & Compliance)
File: rego-test-assertions.md

title: rego-test-assertions subtitle: Helper functions for unit testing Rego labels: category: library inventors: anderseknert code: https://github.com/anderseknert/rego-test-assertions tutorials: https://github.com/anderseknert/rego-test-assertions/blob/main/README.md docs_features: policy-testing: note: | The library contains various assertion functions, which will print the expected result vs. the outcome to the console on failure. debugging-rego: note: | The library is designed to make debugging Rego tests easier. A Rego library providing helper functions for unit testing. The library primarily contains various assertion functions, which will print the expected result vs. the outcome to the console on failure. This allows you to quickly grasp what went wrong in your unit tests, resulting in a faster test iteration process!
Project: Backstage (App Definition and Development / Application Definition & Image Build)
File: generate-client.md

id: generate-client title: Generate a client from your OpenAPI spec description: Documentation on how to create a client for a given OpenAPI spec Set your OpenAPI file's `info.title` to your pluginID like so, ```yaml info: title: catalog ``` Find or create a new plugin to house your new generated client. Currently, we do not support generating an entirely new plugin and instead just generate client files. Run `yarn backstage-repo-tools package schema openapi generate client --client-package <directory>`. This will create a new folder in `<directory>/src/generated` to house the generated content. You should use the generated files as follows, `apis/DefaultApi.client.ts` - this is the client that you should use. It has types for all of the various operations on your API. `models/*` - These are the types generated from your OpenAPI file, ideally you should not need to use these directly and can instead use the inferred types from `apis/DefaultApi.client.ts`. everything else is directory specific and shouldn't be touched.
Project: KICS (Provisioning / Security & Compliance)
File: 70cb518c-d990-46f6-bc05-44a5041493d6.md

title: User With Privilege Escalation By Actions 'iam:AttachUserPolicy' hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 70cb518c-d990-46f6-bc05-44a5041493d6 Query name: User With Privilege Escalation By Actions 'iam:AttachUserPolicy' Platform: Terraform Severity: <span style="color:#ff7213">Medium</span> Category: Access Control URL: User with privilege escalation by actions 'iam:AttachUserPolicy' and Resource set to '*'. For more information see https://rhinosecuritylabs.com/aws/aws-privilege-escalation-methods-mitigation/.<br> ```tf title="Positive test num. 1 - tf file" hl_lines="1" resource "awsiamuser" "cosmic" { name = "cosmic" } resource "awsiamuserpolicy" "testinline_policy" { name = "testinlinepolicy" user = awsiamuser.cosmic.name policy = jsonencode({ Version = "2012-10-17" Statement = [ { Action = [ "iam:AttachUserPolicy", ] Effect = "Allow" Resource = "*" }, ] }) } resource "awsiampolicy_attachment" "test-attach" { name = "test-attachment" users = [awsiamuser.cosmic.name] roles = [awsiamrole.role.name] groups = [awsiamgroup.group.name] } ``` ```tf title="Negative test num. 1 - tf file" resource "awsiamuser" "cosmic2" { name = "cosmic2" } resource "awsiamuserpolicy" "inlinepolicyruninstances2" { name = "inlinepolicyrun_instances" user = awsiamuser.cosmic2.name policy = jsonencode({ Version = "2012-10-17" Statement = [ { Action = [ "ec2:Describe*", ] Effect = "Allow" Resource = "*" }, ] }) } ```
Project: Checkov (Provisioning / Security & Compliance)
File: Contribute New OpenAPI Policies.md

layout: default published: true title: Contribute New OpenAPI configuration policy nav_order: 5 In this example, we'll add support for a new OpenAPI configuration check. Go to `checkov/openapi/checks/resource`, go to v2|v3|generic according to the OpenAPI version you check and add `GlobalSecurityFieldIsEmpty.py`: v2 - OpenAPI 2.0. v3 - OpenAPI 3. generic - for both OpenAPI 2 and 3. In case the check is for OpenAPI version 2.0, use parent class `BaseOpenapiCheckV2` and override check method `scanopenapiconf` In case the check is for OpenAPI version 3, use parent class `BaseOpenapiCheckV3` and override check method `scanopenapiconf` In case the check is a generic check for OpenAPI version 2.0 and 3, use parent class `BaseOpenapiCheck` and override check method `scanentityconf` ```python from future import annotations from typing import Any from checkov.common.models.enums import CheckResult, CheckCategories from checkov.common.checks.enums import BlockType from checkov.openapi.checks.baseopenapicheck import BaseOpenapiCheck class GlobalSecurityFieldIsEmpty(BaseOpenapiCheck): def init(self) -> None: id = "CKVOPENAPI4" name = "Ensure that the global security field has rules defined" categories = [CheckCategories.API_SECURITY] supported_resources = ['security'] super().init(name=name, id=id, categories=categories, supportedentities=supportedresources, block_type=BlockType.DOCUMENT) def scanentityconf(self, conf: dict[str, Any], entity_type: str) -> tuple[CheckResult, dict[str, Any]]: security_rules = conf.get("security") if security_rules: return CheckResult.PASSED, security_rules return CheckResult.FAILED, conf check = GlobalSecurityFieldIsEmpty() ``` follow the examples in `tests/openapi/test_runner.py` and add a test to the new check So there you have it! A new check will be scanned once your contribution is merged!
Project: ShardingSphere (App Definition and Development / Database)
File: label-relabel-compute-node.en.md

+++ title = "LABEL|RELABEL COMPUTE NODES" weight = 5 +++ The `LABEL|RELABEL COMPUTE NODES` syntax is used to label `PROXY` instance. {{< tabs >}} {{% tab name="Grammar" %}} ```sql LableRelabelComputeNodes ::= ('LABEL' | 'RELABEL') 'COMPUTE' 'NODE' instance_id 'WITH' labelName instance_id ::= string labelName ::= identifier ``` {{% /tab %}} {{% tab name="Railroad diagram" %}} <iframe frameborder="0" name="diagram" id="diagram" width="100%" height="100%"></iframe> {{% /tab %}} {{< /tabs >}} needs to be obtained through syntax query `RELABEL` is used to relabel `PROXY` instance Label `PROXY` instance ```sql LABEL COMPUTE NODE "0699e636-ade9-4681-b37a-65240c584bb3" WITH label_1; ``` Relabel `PROXY` instance ```sql RELABEL COMPUTE NODE "0699e636-ade9-4681-b37a-65240c584bb3" WITH label_2; ``` `LABEL`, `RELABEL`, `COMPUTE`, `NODES`, `WITH`
Project: ArangoDB (App Definition and Development / Database)
File: set_has_exception.md

+++ title = "`static void sethas_exception(Impl &&, bool) noexcept`" description = "Sets whether the implementation has an exception. Constexpr, never throws." categories = ["modifiers"] weight = 330 +++ Sets whether the implementation has an exception by setting or clearing the relevant bit in the flags. Constexpr where possible. Requires: Always available. Complexity: Constant time. Guarantees: Never throws an exception.
Project: YugabyteDB (App Definition and Development / Database)
File: alter_procedure,subprogram_signature,arg_decl,special_fn_and_proc_attribute,alterable_fn_and_proc_attribute.grammar.md

```output.ebnf alterprocedure ::= ALTER PROCEDURE subprogramname ( [ subprogram_signature ] ) { specialfnandprocattribute | alterablefnandprocattribute [ ... ] [ RESTRICT ] } subprogramsignature ::= argdecl [ , ... ] argdecl ::= [ argname ] [ argmode ] argtype specialfnandprocattribute ::= RENAME TO subprogram_name | OWNER TO { role_name | CURRENT_ROLE | CURRENT_USER | SESSION_USER } | SET SCHEMA schema_name | [ NO ] DEPENDS ON EXTENSION extension_name alterablefnandprocattribute ::= SET configuration_parameter { TO value | = value | FROM CURRENT } | RESET configuration_parameter | RESET ALL | [ EXTERNAL ] SECURITY { INVOKER | DEFINER } ```
Project: YDB (App Definition and Development / Database)
File: TESTING.md

Testing ======= Zstandard CI testing is split up into three sections: short, medium, and long tests. Short Tests -- Short tests run on CircleCI for new commits on every branch and pull request. They consist of the following tests: Compilation on all supported targets (x86, x86_64, ARM, AArch64, PowerPC, and PowerPC64) Compilation on various versions of gcc, clang, and g++ `tests/playTests.sh` on x86_64, without the tests on long data (CLI tests) Small tests (`tests/legacy.c`, `tests/longmatch.c`) on x64_64 Medium Tests Medium tests run on every commit and pull request to `dev` branch, on TravisCI. They consist of the following tests: The following tests run with UBsan and Asan on x86_64 and x86, as well as with Msan on x86_64 `tests/playTests.sh --test-large-data` Fuzzer tests: `tests/fuzzer.c`, `tests/zstreamtest.c`, and `tests/decodecorpus.c` `tests/zstreamtest.c` under Tsan (streaming mode, including multithreaded mode) Valgrind Test (`make -C tests test-valgrind`) (testing CLI and fuzzer under `valgrind`) Fuzzer tests (see above) on ARM, AArch64, PowerPC, and PowerPC64 Long Tests Long tests run on all commits to `release` branch, and once a day on the current version of `dev` branch, on TravisCI. They consist of the following tests: Entire test suite (including fuzzers and some other specialized tests) on: x86_64 and x86 with UBsan and Asan x86_64 with Msan ARM, AArch64, PowerPC, and PowerPC64 Streaming mode fuzzer with Tsan (for the `zstdmt` testing) ZlibWrapper tests, including under valgrind Versions test (ensuring `zstd` can decode files from all previous versions) `pzstd` with asan and tsan, as well as in 32-bits mode Testing `zstd` with legacy mode off Entire test suite and make install on macOS