cwielenb/CNCF-Dataset huggingface dataset:
Project: FOSSA (Provisioning / Security & Compliance)
File: rust.md

The rust buildtool ecosystem is nearly exclusive to `cargo`, the package manager that ships with rust distributions. There are rare, but known cases of generic buildtools being used, like `make`, `cmake`, and `ninja`. These cases are not handled here. | Strategy | Direct Deps | Transitive Deps | Edges | Tags | Container Scanning | | -- | | | | -- | | | cargo | :whitecheckmark: | :whitecheckmark: | :whitecheckmark: | Environment | :x: | Find all files with a `Cargo.toml` file, where no direct or indirect parent also contains a `Cargo.toml` file. For example, given the following tree, we would only examine `root/Cargo.toml`. ``` root src1 subpackage Cargo.toml ... main.rs src2 subpackage Cargo.toml ... main.rs Cargo.toml ``` However, in the following tree, we examine `root/pkg1/Cargo.toml` and `root/pkg2/Cargo.toml`, since neither has a parent directory with a `Cargo.toml` file. ``` root pkg1 Cargo.toml main.rs pkg2 Cargo.toml main.rs ``` First, we invoke `cargo generate-lockfile` to trigger the lockfile build. This downloads the dependency graph info and almost nothing else. Then, we read the output of the `cargo metadata --format-version 1` command (format version 1 is JSON, and is currently the only allowed formatting scheme). This will arrange the data into a slightly simpler scheme. NOTE: If we do not attempt to regenerate the lockfile, we may try to download every dependency in the tree. This can take upwards of 10 minutes even for small projects, which is wasteful and unnecessary. However, if the lockfile has recently been generated, it will try to update the package index. This usually takes about 2-10 seconds, which is potentially wasteful, but far less so than the 10 minutes that it would take to download entire crates. We can interrogate the JSON output for direct and transitive dependency info, but it is non-trivial to do so. For more info, see the metadata schema found The analysis of the JSON is slightly complex, but is roughly as follows: Check `.workspace_members` for a list of packages which represent the current package and any sub packages. For now we'll call these local crates. Examine `.resolve` (a flattened dependency graph), for each of the local crates to fetch the list of direct dependencies of that crate. Call this a dependency node. For each dependency node, we are able to retreive some info directly, like its version, whether it is platform-specific, what build category it's included in (build, dev, or standard), and what features it compiles with. For any further information, we have to compare this package and version with the corresponding entry in `.packages`.
Project: StarRocks (App Definition and Development / Database)
File: hours_add.md

displayed_sidebar: "English" Adds hours to a date or datetime. ```Haskell DATETIME hours_add(DATETIME|DATE date, INT hours); ``` `date`: Base date or datetime. `hours`: Hours to be added. Returns a DATETIME value. NULL is returned when any of the input parameters is NULL. ```Plain Text select hours_add('2022-01-01 01:01:01', 2); +-+ | hours_add('2022-01-01 01:01:01', 2) | +-+ | 2022-01-01 03:01:01 | +-+ select hours_add('2022-01-01 01:01:01', -1); +--+ | hours_add('2022-01-01 01:01:01', -1) | +--+ | 2022-01-01 00:01:01 | +--+ select hours_add('2022-01-01', 1); +-+ | hours_add('2022-01-01', 1) | +-+ | 2022-01-01 01:00:00 | +-+ ```
Project: YDB (App Definition and Development / Database)
File: staticfold.md

``` StaticFold(obj:Struct/Tuple, initVal, updateLambda) StaticFold1(obj:Struct/Tuple, initLambda, updateLambda) ``` Left fold over struct/tuple elements. The folding of tuples is done in order from the element with the lower index to the element with the larger one; for structures, the order is not guaranteed. `obj` - object to fold `initVal` - (for StaticFold) initial fold state `initLambda` - (for StaticFold1) lambda that produces initial fold state from the first element `updateLambda` - lambda that produces the new state (arguments are the next element and the previous state) `StaticFold(<|key1:$el1, key2:$el2, ..., keyn:$eln|>, $init, $f)` transforms into: ```yql $f($eln, ...$f($el2, $f($init, el_1))...) ``` `StaticFold1(<|key1:$el1, key2:$el2, ..., keyn:$eln|>, $f0, $f)`: ```yql $f($eln, ...$f($el2, $f($f0($init), el_1))...) ``` `StaticFold1(<||>, $f0, $f)` returns `NULL`. Works with tuples in the same way.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: klio.md

title: "Klio" icon: /images/logos/powered-by/klio.png hasNav: true cardDescription: "Developed at Spotify and built on top of Apache Beam for Python, Klio is an open source framework that lets researchers and engineers build smarter data pipelines for processing audio and other media files, easily and at scale." <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Developed at Spotify and built on top of Apache Beam for Python, Klio is an open source framework that lets researchers and engineers build smarter data pipelines for processing audio and other media files, easily and at scale.
Project: secureCodeBox (Provisioning / Security & Compliance)
File: 12_glossary.md

title: "Glossary" sidebar_label: "Glossary" sidebar_position: 12 API: An (API) is a computing interface which defines interactions between multiple software intermediaries. Charts: Template files for to configure the deployment on . CI: . Container: Concept for . CRD: Acronym for . Custom Resource Definition (CRD): This is a to define custom resource types additionally to the provided ones from Kubernetes. GUID: Acronym for . Hook: A hook is a concept in secureCodeBox to give the opportunity to react on detected findings. Hooks are invoked after parsing and storing the Findings. There are two kinds of Hooks: ReadHooks and ReadWriteHooks. See the for more details. Job: A Job creates one or more Pods and will continue to retry execution of the Pods until a specified number of them successfully terminate. (see .) Engine: This is the central component of the secureCodeBox which controls the business logic. It implements the operator pattern. Finding: This is one single possible security issue found by a scanner which is stored as . K8S: Acronym for . Lurker: This sidecar is a generic component responsible to "lurk" the results spit out by scanners and stores it into the storage for further processing by the parser. Operator: The is a common pattern in Kubernetes to extend Kubernetes. Parser: This job is a scanner specific component which reads the raw stored findings from the lurker and transforms them into the for secureCodeBox findings. Each scanner needs its own parser implementation. Raw finding: In contrast to findings these are the raw results from the scanner before they were converted into the for findings. SCB: Acronym for . Scanner: This is the main component to actually perform a scan. This is simply a container which encapsulates a scanner tool (e.g. nmap, nuclei etc.) and invokes it, parameterized by the custom resource definition for this scan type. Scanner tool: These are the command line tools we integrate into secureCodeBox to perform the actual scan (e.g. Nmpa, Nuclei, Nikto, ZAP etc.). Scan type: Own custom resource definition for the secureCodeBox scanners. Sidecar : This is a for Kubernetes. It is a container with one responsibility which extends a container. We use this pattern for the lurker. Security Operations Center (SOC): A unit, team or department responsible for monitoring and handle security issues and vulnerabilities in an organisation. Storage: The component we use to store raw and parsed findings.
Project: ClickHouse (App Definition and Development / Database)
File: v21.7.7.47-stable.md

sidebar_position: 1 sidebar_label: 2022 Backported in : - Fix uninitialized memory in functions `multiSearch` with empty array, close . (). Backported in : Fix `distributedgroupbynomerge=2`+`distributedpushdownlimit=1` or `optimizedistributedgroupbyshardingkey=1` with `LIMIT BY` and `LIMIT OFFSET`. (). Backported in : Fix `Cannot find column` error for queries with sampling. Was introduced in . Fixes . (). Backported in : Fixed incorrect validation of partition id for MergeTree tables that created with old syntax. (). Fix parallel execution of integration tests (). Fix mysqlkillsyncthreadrestore_test (). Wait for self datasource to be initialized in testjdbcbridge (). Better integration tests (). Implement `legacycolumnnameoftuple_literal` in a less intrusive way (). Update PVS checksum ().
Project: RabbitMQ (App Definition and Development / Streaming & Messaging)
File: 3.9.15.md

RabbitMQ `3.9.15` is a maintenance release in the `3.9.x` release series. Please refer to the Upgrading to 3.9 section from if upgrading from a version prior to 3.9.0. This release requires at least Erlang 23.2, and supports Erlang 24. has more details on Erlang version requirements for RabbitMQ. Release notes are kept under . Contributors are encouraged to update them together with their changes. This helps with release automation and a more consistent release schedule. Stream delivery rate could drop if concurrent stream consumers consumed in a way that made them reach the end of the stream often. GitHub issue: If a cluster that had streams enabled was upgraded with a jump of multiple patch releases, stream state could fail an upgrade. GitHub issue: Significantly faster queue re-import from on subsequent node restarts. Initial definition import still takes the same amount of time as before. GitHub issue: When a policy contained keys unsupported by a particular queue type, and later updated or superseded by a higher priority policy, effective optional argument list could become inconsistent (policy would not have the expected effect). Kudos to @jaroslaw-bochniak for providing very detailed reproduction steps. GitHub issue: Priority queues could run into an exception in some cases. Contributed by @luos and @olikasg. GitHub issue: Maintenance mode could run into a timeout during queue leadership transfer. Contributed by @felixhuettner. GitHub issue: Prometheus collector could run into an exception early on node's schema database sync. GitHub issue: Connection data transfer rate units were incorrectly displayed when rate was less than 1 kiB per second. Contributed by @zgabi. GitHub issue: `rabbitmqadmin` now correctly loads TLS-related keys from its configuration file. Contributed by @fwolfsjaeger. GitHub issue: Corrected a help message for node memory usage tool tip. Contributed by @etc-crontab. GitHub issue: More Erlang 24.3's `eldap` library compatibility improvements. GitHub issue: `ra` upgraded `prometheus` upgraded from `eetcd` upgraded from To obtain source code of the entire distribution, please download the archive named `rabbitmq-server-3.9.15.tar.xz` instead of the source tarball produced by GitHub.
Project: Podman (App Definition and Development / Application Definition & Image Build)
File: podman-tag.1.md

% podman-tag 1 podman\-tag - Add an additional name to a local image podman tag image[:tag] [target-name[:tag]...] [options] podman image tag image[:tag] [target-name[:tag]...] [options] Assigns a new image name to an existing image. A full name refers to the entire image name, including the optional tag after the `:`. If there is no tag provided, then Podman defaults to `latest` for both the image and the target-name. Print usage statement Tag specified image with an image name defaulting to :latest. ``` $ podman tag 0e3bbc2 fedora:latest ``` Tag specified image with fully specified image name. ``` $ podman tag httpd myregistryhost:5000/fedora/httpd:v2 ``` Tag specified image with multiple tags. ``` $ podman tag mymariadb mycontainerregistry.io/namespace/mariadb:10 mycontainerregistry.io/namespace/mariadb:10.11 mycontainerregistry.io/namespace/mariadb:10.11.12 ``` December 2019, Update description to refer to 'name' instead of 'alias' by Sascha Grunert <sgrunert@suse.com> July 2017, Originally compiled by Ryan Cole <rycole@redhat.com>
Project: KICS (Provisioning / Security & Compliance)
File: 8d29754a-2a18-460d-a1ba-9509f8d359da.md

title: IAM Access Analyzer Not Enabled hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 8d29754a-2a18-460d-a1ba-9509f8d359da Query name: IAM Access Analyzer Not Enabled Platform: CloudFormation Severity: <span style="color:#edd57e">Low</span> Category: Best Practices URL: IAM Access Analyzer should be enabled and configured to continuously monitor resource permissions<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="3" AWSTemplateFormatVersion: "2010-09-09" Description: A sample template 2 Resources: myuseeer: Type: AWS::IAM::Group Properties: Path: "/" LoginProfile: Password: myP@ssW0rd ``` ```json title="Positive test num. 2 - json file" hl_lines="4" { "AWSTemplateFormatVersion": "2010-09-09", "Description": "A sample template 2", "Resources": { "myuseeer": { "Type": "AWS::IAM::Group", "Properties": { "Path": "/", "LoginProfile": { "Password": "myP@ssW0rd" } } } } } ``` ```yaml title="Negative test num. 1 - yaml file" AWSTemplateFormatVersion: 2010-09-09 Resources: Analyzer: Type: "AWS::AccessAnalyzer::Analyzer" Properties: AnalyzerName: MyAccountAnalyzer Type: ACCOUNT Tags: Key: Kind Value: Dev ArchiveRules: # Archive findings for a trusted AWS account RuleName: ArchiveTrustedAccountAccess Filter: Property: "principal.AWS" Eq: "123456789012" # Archive findings for known public S3 buckets RuleName: ArchivePublicS3BucketsAccess Filter: Property: "resource" Contains: "arn:aws:s3:::docs-bucket" "arn:aws:s3:::clients-bucket" ``` ```json title="Negative test num. 2 - json file" { "AWSTemplateFormatVersion": "2010-09-09", "Resources": { "Analyzer": { "Type": "AWS::AccessAnalyzer::Analyzer", "Properties": { "AnalyzerName": "MyAccountAnalyzer", "Type": "ACCOUNT", "Tags": [ { "Key": "Kind", "Value": "Dev" } ], "ArchiveRules": [ { "RuleName": "ArchiveTrustedAccountAccess", "Filter": [ { "Property": "principal.AWS", "Eq": [ "123456789012" ] } ] }, { "RuleName": "ArchivePublicS3BucketsAccess", "Filter": [ { "Property": "resource", "Contains": [ "arn:aws:s3:::docs-bucket", "arn:aws:s3:::clients-bucket" ] } ] } ] } } } } ```
Project: Numaflow (App Definition and Development / Streaming & Messaging)
File: README.md

Each SDK is supposed to implement: A `flatmap` UDF example, which splits the input message with `,`, and return a list. Build a docker image `quay.io/numaio/numaflow-${language}/map-flatmap:latest` and push to `quay.io`; A `log` UDSink example, which prints out original message in pod logs. Build a docker image `quay.io/numaio/numaflow-${language}/sink-log:latest` and push to `quay.io`. cluster make start kubectl -n numaflow-system port-forward svc/numaflow-server 8443 ``` Import the App component the run the package as follows: CJS ```javascript import App from 'numaflow-ui-test/dist/cjs/src/components/plugin/NumaflowMonitorApp/App'; ``` ESM ```javascript import App from 'numaflow-ui-test/dist/esm/src/components/plugin/NumaflowMonitorApp/App'; ``` Pass the cluster hostname and namespace to get the UI running ```javascript <App hostUrl={hostUrl} namespace={namespace} /> ```ices: name: in source: generator: rpu: 5 duration: 1s name: cat udf: # A user-defined function builtin: # Use a built-in function as the udf name: cat # cats the message name: out sink: log: {} edges: from: in to: cat from: cat to: out ``` Below we have a simple variation on the above example that takes input from an http endpoint. ```yaml apiVersion: numaflow.numaproj.io/v1alpha1 kind: Pipeline metadata: name: http-pipeline spec: vertices: name: in source: http: service: true name: cat udf: builtin: name: cat # A built-in UDF which simply cats the message name: out sink: log: {} edges: from: in to: cat from: cat to: out ``` Let's modify the UDF in the first example to pass-through only messages with an id less than 100 ```yaml apiVersion: numaflow.numaproj.io/v1alpha1 kind: Pipeline metadata: name: filter-pipeline spec: vertices: name: in source: generator: rpu: 5 duration: 1s name: filter udf: builtin: name: filter kwargs: expression: int(json(payload).id) < 100 name: out sink: log: {} edges: from: in to: filter from: filter to: out ```
Project: KICS (Provisioning / Security & Compliance)
File: 7af43613-6bb9-4a0e-8c4d-1314b799425e.md

title: S3 Bucket Access to Any Principal hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 7af43613-6bb9-4a0e-8c4d-1314b799425e Query name: S3 Bucket Access to Any Principal Platform: Terraform Severity: <span style="color:#ff0000">Critical</span> Category: Access Control URL: S3 Buckets must not allow Actions From All Principals, as to prevent leaking private information to the entire internet or allow unauthorized data tampering / deletion. This means the 'Effect' must not be 'Allow' when there are All Principals<br> ```tf title="Positive test num. 1 - tf file" hl_lines="4" resource "awss3bucket_policy" "positive1" { bucket = awss3bucket.b.id policy = <<POLICY { "Version": "2012-10-17", "Id": "MYBUCKETPOLICY", "Statement": [ { "Sid": "IPAllow", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:*", "Resource": "arn:aws:s3:::mytftest_bucket/*", "Condition": { "IpAddress": {"aws:SourceIp": "8.8.8.8/32"} } } ] } POLICY } ``` ```tf title="Positive test num. 2 - tf file" hl_lines="12" module "s3_bucket" { source = "terraform-aws-modules/s3-bucket/aws" version = "3.7.0" bucket = "my-s3-bucket" acl = "private" versioning = { enabled = true } policy = <<POLICY { "Version": "2012-10-17", "Id": "MYBUCKETPOLICY", "Statement": [ { "Sid": "IPAllow", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:*", "Resource": "arn:aws:s3:::mytftest_bucket/*", "Condition": { "IpAddress": {"aws:SourceIp": "8.8.8.8/32"} } } ] } POLICY } ``` ```tf title="Negative test num. 1 - tf file" resource "awss3bucket_policy" "negative1" { bucket = awss3bucket.b.id policy = <<POLICY { "Version": "2012-10-17", "Id": "MYBUCKETPOLICY", "Statement": [ { "Sid": "IPAllow", "Effect": "Deny", "Action": "s3:*", "Resource": "arn:aws:s3:::mytftest_bucket/*", "Condition": { "IpAddress": {"aws:SourceIp": "8.8.8.8/32"} } } ] } POLICY } ``` ```tf title="Negative test num. 2 - tf file" module "s3_bucket" { source = "terraform-aws-modules/s3-bucket/aws" version = "3.7.0" bucket = "my-s3-bucket" acl = "private" versioning = { enabled = true } policy = <<POLICY { "Version": "2012-10-17", "Id": "MYBUCKETPOLICY", "Statement": [ { "Sid": "IPAllow", "Effect": "Deny", "Action": "s3:*", "Resource": "arn:aws:s3:::mytftest_bucket/*", "Condition": { "IpAddress": {"aws:SourceIp": "8.8.8.8/32"} } } ] } POLICY } ```
Project: YugabyteDB (App Definition and Development / Database)
File: apache-flink.md

title: Apache Flink linkTitle: Apache Flink description: Apache Flink menu: preview_integrations: identifier: apache-flink parent: data-integration weight: 571 type: docs is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink provides various DataStream connectors including one for JDBC to write data to various databases. You can use this connector to write date from Flink to YugabyteDB. The following example based on the describes the Flink connectivity to a YugabyteDB cluster using a JDBC sink connector. Your YugabyteDB cluster should be up and running. Refer to . Note that the YugabyteDB cluster you are connecting to is running on localhost. Create a table named books using as follows: ```sql create table books (id bigint, title varchar, authors varchar, year int); ``` Compile and run the following program: ```java package com.yugabyte; import org.apache.flink.connector.jdbc.JdbcConnectionOptions; import org.apache.flink.connector.jdbc.JdbcExecutionOptions; import org.apache.flink.connector.jdbc.JdbcSink; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; public class JdbcSinkExample { static class Book { public Book(Long id, String title, String authors, Integer year) { this.id = id; this.title = title; this.authors = authors; this.year = year; } final Long id; final String title; final String authors; final Integer year; } public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.fromElements( new Book(101L, "Stream Processing with Apache Flink", "Fabian Hueske, Vasiliki Kalavri", 2019), new Book(102L, "Streaming Systems", "Tyler Akidau, Slava Chernyak, Reuven Lax", 2018), new Book(103L, "Designing Data-Intensive Applications", "Martin Kleppmann", 2017), new Book(104L, "Kafka: The Definitive Guide", "Gwen Shapira, Neha Narkhede, Todd Palino", 2017) ).addSink( JdbcSink.sink( "insert into books (id, title, authors, year) values (?, ?, ?, ?)", (statement, book) -> { statement.setLong(1, book.id); statement.setString(2, book.title); statement.setString(3, book.authors); statement.setInt(4, book.year); }, JdbcExecutionOptions.builder() .withBatchSize(1000) .withBatchIntervalMs(200) .withMaxRetries(5) .build(), new JdbcConnectionOptions.JdbcConnectionOptionsBuilder() .withUrl("jdbc:postgresql://localhost:5433/yugabyte") .withDriverName("org.postgresql.Driver") .withUsername("yugabyte") .withPassword("yugabyte") .build() )); env.execute(); System.out.println("Done"); } } ```
Project: KubeDB by AppsCode (App Definition and Development / Database)
File: kubedb-operator.md

title: Kubedb-Operator menu: docs_{{ .version }}: identifier: kubedb-operator name: Kubedb-Operator parent: reference-operator weight: 0 menuname: docs{{ .version }} sectionmenuid: reference url: /docs/{{ .version }}/reference/operator/ aliases: /docs/{{ .version }}/reference/operator/kubedb-operator/ ``` --default-seccomp-profile-type string Default seccomp profile -h, --help help for kubedb-operator --use-kubeapiserver-fqdn-for-aks if true, uses kube-apiserver FQDN for AKS cluster to workaround https://github.com/Azure/AKS/issues/522 (default true) ``` - Launch KubeDB Provisioner - Prints binary version number.
Project: ShardingSphere (App Definition and Development / Database)
File: show-default-shadow-algorithm.en.md

+++ title = "SHOW DEFAULT SHADOW ALGORITHM" weight = 4 +++ The `SHOW DEFAULT SHADOW ALGORITHM` syntax is used to query default shadow algorithm for specified database. {{< tabs >}} {{% tab name="Grammar" %}} ```sql ShowEncryptAlgorithm::= 'SHOW' 'SHADOW' 'ALGORITHM' ('FROM' databaseName)? databaseName ::= identifier ``` {{% /tab %}} {{% tab name="Railroad diagram" %}} <iframe frameborder="0" name="diagram" id="diagram" width="100%" height="100%"></iframe> {{% /tab %}} {{< /tabs >}} When `databaseName` is not specified, the default is the currently used `DATABASE`. If `DATABASE` is not used, `No database selected` will be prompted. | Column | Description | |--|--| | shadowalgorithmname | Shadow algorithm name | | type | Shadow algorithm type | | props | Shadow algorithm properties | Query default shadow algorithm for specified database. ```sql SHOW DEFAULT SHADOW ALGORITHM FROM shadow_db; ``` ```sql mysql> SHOW DEFAULT SHADOW ALGORITHM FROM shadow_db; +-+-+--+ | shadowalgorithmname | type | props | +-+-+--+ | useridmatchalgorithm | VALUEMATCH | column=user_id,operation=insert,value=1 | +-+-+--+ 1 row in set (0.00 sec) ``` Query default shadow algorithm for current database. ```sql SHOW SHADOW ALGORITHM; ``` ```sql mysql> SHOW SHADOW ALGORITHM; +-+-+--+ | shadowalgorithmname | type | props | +-+-+--+ | useridmatchalgorithm | VALUEMATCH | column=user_id,operation=insert,value=1 | +-+-+--+ 1 row in set (0.00 sec) ``` `SHOW`, `DEFAULT`,`SHADOW`, `ALGORITHM`, `FROM`