cwielenb/CNCF-Dataset huggingface dataset:
Project: KICS (Provisioning / Security & Compliance)
File: 2134641d-30a4-4b16-8ffc-2cd4c4ffd15d.md

title: DOCDB Cluster Encrypted With AWS Managed Key hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 2134641d-30a4-4b16-8ffc-2cd4c4ffd15d Query name: DOCDB Cluster Encrypted With AWS Managed Key Platform: Terraform Severity: <span style="color:#edd57e">Low</span> Category: Encryption URL: DOCDB Cluster should be encrypted with customer-managed KMS keys instead of AWS managed keys<br> ```tf title="Positive test num. 1 - tf file" hl_lines="16" provider "aws" { region = "us-east-1" } data "awskmskey" "test" { key_id = "alias/aws/rds" } resource "awsdocdbcluster" "test2" { cluster_identifier = "my-docdb-cluster-test2" engine = "docdb" master_username = "foo" master_password = "mustbeeightchars" skipfinalsnapshot = true storage_encrypted = true kmskeyid = data.awskmskey.test.arn } ``` ```tf title="Negative test num. 1 - tf file" provider "aws" { region = "us-east-1" } data "awskmskey" "test2" { key_id = "alias/myAlias" } resource "awsdocdbcluster" "test22" { cluster_identifier = "my-docdb-cluster-test2" engine = "docdb" master_username = "foo" master_password = "mustbeeightchars" skipfinalsnapshot = true storage_encrypted = true kmskeyid = data.awskmskey.test2.arn } ```
Project: KICS (Provisioning / Security & Compliance)
File: 300a9964-b086-41f7-9378-b6de3ba1c32b.md

title: GKE Legacy Authorization Enabled hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 300a9964-b086-41f7-9378-b6de3ba1c32b Query name: GKE Legacy Authorization Enabled Platform: Ansible Severity: <span style="color:#bb2124">High</span> Category: Insecure Configurations URL: Kubernetes Engine Clusters must have Legacy Authorization set to disabled, which means the attribute 'legacy_abac.enabled' must be false.<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="18" name: create a cluster google.cloud.gcpcontainercluster: name: my-cluster initialnodecount: 2 master_auth: username: cluster_admin password: my-secret-password node_config: machine_type: n1-standard-4 disksizegb: 500 location: us-central1-a project: test_project auth_kind: serviceaccount serviceaccountfile: "/tmp/auth.pem" state: present legacy_abac: enabled: yes ``` ```yaml title="Negative test num. 1 - yaml file" name: create a cluster google.cloud.gcpcontainercluster: name: my-cluster initialnodecount: 2 master_auth: username: cluster_admin password: my-secret-password node_config: machine_type: n1-standard-4 disksizegb: 500 location: us-central1-a project: test_project auth_kind: serviceaccount serviceaccountfile: /tmp/auth.pem state: present legacy_abac: enabled: no ```
Project: KICS (Provisioning / Security & Compliance)
File: c010082c-76e0-4b91-91d9-6e8439e455dd.md

title: Cloud Storage Bucket Is Publicly Accessible hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: c010082c-76e0-4b91-91d9-6e8439e455dd Query name: Cloud Storage Bucket Is Publicly Accessible Platform: Terraform Severity: <span style="color:#ff7213">Medium</span> Category: Access Control URL: Cloud Storage Bucket is anonymously or publicly accessible<br> ```tf title="Positive test num. 1 - tf file" hl_lines="17 4" resource "googlestoragebucketiammember" "positive1" { bucket = googlestoragebucket.default.name role = "roles/storage.admin" member = "allUsers" condition { title = "expiresafter20191231" description = "Expiring at midnight of 2019-12-31" expression = "request.time < timestamp(\"2020-01-01T00:00:00Z\")" } } resource "googlestoragebucketiammember" "positive2" { bucket = googlestoragebucket.default.name role = "roles/storage.admin" members = ["user:john@example.com","allAuthenticatedUsers"] } ``` ```tf title="Negative test num. 1 - tf file" resource "googlestoragebucketiammember" "negative1" { bucket = googlestoragebucket.default.name role = "roles/storage.admin" member = "user:jane@example.com" } resource "googlestoragebucketiammember" "negative2" { bucket = googlestoragebucket.default.name role = "roles/storage.admin" members = ["user:john@example.com","user:john@example.com"] } ```
Project: Koperator (App Definition and Development / Streaming & Messaging)
File: developer.md

Koperator (formerly called Banzai Cloud Kafka Operator) is built on the project. To build the operator and run tests: Run `make` If you make changes and would like to try your own version, create your own image: `make docker-build IMG={YOUR_USERNAME}/kafka-operator:v0.0.1` `make docker-push IMG={YOUR_USERNAME}/kafka-operator:v0.0.1` `make deploy IMG={YOUR_USERNAME}/kafka-operator:v0.0.1` Watch the operator's logs with: `kubectl logs -f -n kafka kafka-operator-controller-manager-0 -c manager` Alternatively, run the operator on your machine: `export $KUBECONFIG` `make install` `make run` Create CR and let the operator set up Apache Kafka in your cluster (you can change the `spec` of `Kafka` for your needs in the yaml file): Remember you need Zookeeper server to run Apache Kafka `kubectl create -n kafka -f config/samples/simplekafkacluster.yaml` Minikube does not have a load balancer implementation, thus our envoy service will not get an external IP and the operator will get stuck at this point. A possible solution to overcome this problem is to use https://github.com/elsonrodriguez/minikube-lb-patch. The operator will be able to proceed if you run the following command: ```go kubectl run minikube-lb-patch --replicas=1 --image=elsonrodriguez/minikube-lb-patch:0.1 --namespace=kube-system ```
Project: ClickHouse (App Definition and Development / Database)
File: v20.10.6.27-stable.md

Backported in : Fix performance of reading from `Merge` tables over huge number of `MergeTree` tables. Fixes . (). Backported in : Fixed uncontrolled growth of TDigest. (). Backported in : Return number of affected rows for INSERT queries via MySQL protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes . (). Backported in : TODO. (). Backported in : Fix optimization of group by with enabled setting `optimizeaggregatorsofgroupby_keys` and joins. Fixes . (). Backported in : Fix order by optimization with monotonous functions. Fixes . (). Backported in : Avoid unnecessary network errors for remote queries which may be cancelled while execution, like queries with `LIMIT`. (). Backported in : Bug fix for funciton fuzzBits, related issue: . (). Backported in : Fixed crash on `CREATE TABLE ... AS sometable` query when `sometable` was created `AS table_function()` Fixes . (). Backported in : Fix ColumnConst comparison which leads to crash. This fixed . (). Backported in : Fix . When clickhouse-copier handle non-partitioned table, throws segfault error. (). Backported in : Fix set index invalidation when there are const columns in the subquery. This fixes . (). Backported in : Fix crash while reading from `JOIN` table with `LowCardinality` types. Fixes . (). Backported in : Fix duplicates after `DISTINCT` which were possible because of incorrect optimization. Fixes . (). (). Backported in : Fix `ORDER BY` with enabled setting `optimizeredundantfunctionsinorder_by`. (). Backported in : Fix bug when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. (). Backported in : Fix alter query hang when the corresponding mutation was killed on the different replica. Fixes . (). Backported in : When clickhouse-client is used in interactive mode with multiline queries, single line comment was erronously extended till the end of query. This fixes . (). Backported in : In might be determined incorrectly if cluster is circular- (cross-) replicated or not when executing `ON CLUSTER` query due to race condition when `pool_size` > 1. It's fixed. (). Backported in : Fixed `Function not implemented` error when executing `RENAME` query in `Atomic` database with ClickHouse running on Windows Subsystem for Linux. Fixes . (). Backported in : Fixed problem when ClickHouse fails to resume connection to MySQL servers. (). Backported in : Do not restore parts from WAL if `inmemorypartsenablewal` is disabled. (). Backported in : Update embedded timezone data to version 2020d (also update cctz to the latest master). ().
Project: Shifu (Provisioning / Automation & Configuration)
File: telemetry-service-TDEngine.md

Telemetry Service is a standalone service that allow `deviceShifu` send its collected telemetry to designated endpoints. This design aims to add `TDengine` endpoint support to current Telemetry Service. Let telemetry service support pushing telemetries to TDengine endpoints. Let telemetry service support all timeseriesDB endpoint. In order to add `TDengine` as an endpoint, we need to add connection related settings telemetry service's CRD, just like what we did for `MQTT`. ```yaml SQLConnectionSetting: description: SQLConnectionSetting defines SQL specific settings when connecting to SQL endpoint properties: DBServerAddress: type: string DBUserName: type: string DBSecret: type: string DBName: type: string DBTable: type: string DBType: type: string type: object ``` We also will add the corresponding connection setting as a go struct and will add it to `TelemetryRequest` ```go type SQLConnectionSetting struct { DBServerAddress *string `json:"dbserveraddress,omitempty"` DBUserName *string `json:"db_username,omitempty"` DBSecret *string `json:"db_secret,omitempty"` DBName *string `json:"db_name,omitempty"` DBTable *string `json:"db_table,omitempty"` DBType *DBType `json:"db_type,omitempty"` } type DBType string type TelemetryRequest struct { RawData []byte `json:"rawData,omitempty"` MQTTSetting *MQTTSetting `json:"mqttSetting,omitempty"` SQLConnectionSetting *SQLConnectionSetting `json:"sqlConnectionSetting,omitempty"` } ``` We use the name `SQLConnection` instead of `TDengineConnection` because we want to it to be a generic DB connection setting for scalability concerns. We want to use it to connect to various DBs instead of TDengine alone. Telemetry Service will push the data to TDengine: ```mermaid graph LR; deviceShifu -->|TelemetryRequest| TelemetryService; TelemetryService -->|RawData| TDengine; ``` After we enabled `TDengine` endpoint ,we need to extract corresponding endpoint settings from `TelemetryRequest`. ```go func HandleTelemtryRequest(request *TelemetryRequest) err { // http part // mqtt part if (request.SQLConnectionSetting != nil) { sqlcs := request.SQLConnectionSetting } } func sendToTDengine(rawData []byte, sqlcs *v1alpha1.SQLConnectionSetting) err { // Send rawData to TDengine // sample code: taosUri := constructTDengineUri(sqlcs) taos, err := sql.Open("taosSql", taosUri) if err != nil { klog.Error("failed to connect TDengine, err:", err) return err } defer taos.Close() ... } func constructTDengineUri(sqlcs *v1alpha1.SQLConnectionSetting) (string, error) { } ```
Project: ArangoDB (App Definition and Development / Database)
File: implicit_error_exception_converting_constructor.md

+++ title = "`basic_outcome(S &&, P &&)`" description = "Implicit `errortype` + `exceptiontype` constructor. Available if `predicate::enableerrorexceptionconvertingconstructor<S, P>` is true. Constexpr, triviality and noexcept propagating." categories = ["constructors", "implicit-constructors", "converting-constructors"] weight = 202 +++ Implicit `errortype` + `exceptiontype` constructor. Calls {{% api "void onoutcomeconstruction(T *, U &&, V &&) noexcept" %}} with `this`, `S` and `P`. Requires: `predicate::enableerrorexceptionconvertingconstructor<S, P>` is true. Complexity: Same as for `errortype`'s and `exceptiontype`'s copy or move constructor. Constexpr, triviality and noexcept of underlying operations is propagated.
Project: Chef InSpec (Provisioning / Security & Compliance)
File: iptables.md

+++ title = "iptables resource" draft = false gh_repo = "inspec" platform = "linux" [menu] [menu.inspec] title = "iptables" identifier = "inspec/resources/os/iptables.md iptables resource" parent = "inspec/resources/os" +++ Use the `iptables` Chef InSpec audit resource to test rules that are defined in `iptables`, which maintains tables of IP packet filtering rules. There may be more than one table. Each table contains one (or more) chains (both built-in and custom). A chain is a list of rules that match packets. When the rule matches, the rule defines what target to assign to the packet. {{< readfile file="content/inspec/reusable/md/inspec_installation.md" >}} This resource first became available in v1.0.0 of InSpec. A `iptables` resource block declares tests for rules in IP tables: describe iptables(rule:'name', table:'name', chain: 'name', ignore_comments: true) do it { should have_rule('RULE') } end where `iptables()` may specify any combination of `rule`, `table`, or `chain` `rule:'name'` is the name of a rule that matches a set of packets `table:'name'` is the packet matching table against which the test is run `chain: 'name'` is the name of a user-defined chain or one of `ACCEPT`, `DROP`, `QUEUE`, or `RETURN` `ignore_comments: true` is a boolean flag that ignores comments in a rule. `have_rule('RULE')` tests that rule in the iptables list. This must match the entire line taken from `iptables -S CHAIN`. The following examples show how to use this Chef InSpec audit resource. describe iptables do it { should have_rule('-P INPUT ACCEPT') } end describe iptables(table:'mangle', chain: 'INPUT') do it { should have_rule('-P INPUT ACCEPT') } end describe iptables do it { should have_rule('-A INPUT -p tcp -m tcp -m multiport --dports 5432 -m comment --comment "postgres" -j ACCEPT') } end describe iptables(ignore_comments: true) do it { should have_rule('-A INPUT -p tcp -m tcp -m multiport --dports 5432 -j ACCEPT') } end Note that the rule specification must exactly match what's in the output of `iptables -S INPUT`, which will depend on how you've built your rules. {{< readfile file="content/inspec/reusable/md/inspecmatcherslink.md" >}} This resource has the following special matchers. The `have_rule` matcher tests the named rule against the information in the `iptables` file: it { should have_rule('RULE') }
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 29_advanced_side_inputs.md

Prompt: What are side inputs in Apache Beam? Response: In Apache Beam, side inputs are additional inputs that can be provided to a , in addition to the main input . These side inputs can be used by the `ParDo` transform's `DoFn` to access and read additional data during the processing of each element in the main input `PCollection`. Side inputs are particularly useful when a Beam pipeline needs to ingest additional data dynamically at runtime when processing each element, rather than relying on preset or hard-coded values. By leveraging side inputs, Apache Beam can determine additional data based on the main `PCollection` data or even from a different branch of your pipeline. For example, you can leverage side inputs for your streaming analytics use cases, enriching streaming events with additional data retrieved from a lookup table. In the Apache Beam Python SDK, you can provide side inputs as extra arguments in the `DoFn`'s `process` method or the `Map`/`FlatMap` transforms. The Python SDK supports optional, positional, and keyword arguments. ```python class MyDoFn(beam.DoFn): def process(self, element, side_input): ... ``` In the Apache Beam Java SDK, side inputs are provided to the `ParDo` transform by using the `.withSideInputs` method. To access a side input from within a `DoFn`, the `DoFn.ProcessContext.sideInput` method is invoked. ```java PCollection<Integer> input = ...; PCollectionView<Integer> sideInput = ...; PCollection<Integer> output = input.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(ProcessContext c) { Integer sideInputValue = c.sideInput(sideInput); ... } }).withSideInputs(sideInput)); ``` Side inputs can also be used to process windowed data. Apache Beam uses the windows for the main input elements to look up the appropriate windows for the side input elements. Apache Beam projects the main inputs window to the side input's window set and uses the side input from the resulting window. Main and side inputs might have identical or different windowing strategies. For example, if the main input `PCollection` is windowed into ten-minute windows, and the side input is windowed into one-hour windows, Apache Beam projects the main input window set and selects the value from the corresponding one-hour side input window. For more information, refer to . For common patterns and code samples to implement side inputs in Apache Beam pipelines, refer to .
Project: StarRocks (App Definition and Development / Database)
File: SHOW_DELETE.md

displayed_sidebar: "English" Queries historical DELETE operations that were successfully performed on Duplicate Key, Unique Key, and Aggregate tables in a specified database. For more information about data deletion, see . Note that this command cannot be used to query DELETE operations that were performed on Primary Key tables. ```sql SHOW DELETE [FROM <db_name>] ``` `db_name`: the database name, optional. If this parameter is not specified, the current database is used by default. Return fields: `TableName`: the table from which data is deleted. `PartitionName`: the partition from which data is deleted. If the table is a non-partitioned table, `*` is displayed. `CreateTime`: the time when the DELETE task was created. `DeleteCondition`: the specified DELETE condition. `State`: the status of the DELETE task. Show all historical DELETE operations of `database`. ```sql SHOW DELETE FROM database; ++++--+-+ | TableName | PartitionName | CreateTime | DeleteCondition | State | ++++--+-+ | mail_merge | * | 2023-03-14 10:39:03 | name EQ "Peter" | FINISHED | ++++--+-+ ```
Project: Apache Spark (App Definition and Development / Streaming & Messaging)
File: sql-ref-syntax.md

layout: global title: SQL Syntax displayTitle: SQL Syntax license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Spark SQL is Apache Spark's module for working with structured data. The SQL Syntax section describes the SQL syntax in detail along with usage examples when applicable. This document provides a list of Data Definition and Data Manipulation Statements, as well as Data Retrieval and Auxiliary Statements. Data Definition Statements are used to create or modify the structure of database objects in a database. Spark SQL supports the following Data Definition Statements: * * * * * * * * Data Manipulation Statements are used to add, change, or delete data. Spark SQL supports the following Data Manipulation Statements: * Spark supports <code>SELECT</code> statement that is used to retrieve rows from one or more tables according to the specified clauses. The full syntax and brief description of supported clauses are explained in section. The SQL statements related to SELECT are also included in this section. Spark also provides the ability to generate logical and physical plan for a given query using statement. * * * * * * * * * * * * * * * * * * * * * * * * * * * *
Project: Scylla (App Definition and Development / Database)
File: object_storage.md

First one need to describe endpoints where sstables can be stored. This is done in a yaml file with the following format: ```yaml endpoints: name: $endpointaddressordomainname port: $port_number https: optional True or False aws_region: optional region name, e.g. us-east-1 awsaccesskey_id: optional AWS access key ID awssecretaccess_key: optional AWS secret access key awssessiontoken: optional AWS session token ``` The last three items must be all present or all absent. When set the values are used by the S3 client to sign requests. If not set requests are sent unsigned which may not always accepted by the server. By default Scylla tries to read it from the `object_storage.yaml` file located in the same directory with the `scylla.yaml`. Optionally, the `--object-storage-config-file $path` option can be specified. Currently the object-storage backend works if `keyspace-storage-options` is listed in `experimental_features` in `scylla.yaml`. like: ```yaml experimental_features: keyspace-storage-options ``` It can also be enabled with `--experimental-features=keyspace-storage-options` command line option when launchgin scylla. Sstables location is keyspace-scoped. In order to create a keyspace with S3 storage use `CREATE KEYSPACE` with `STORAGE = { 'type': 'S3', 'endpoint': '$endpoint_name', 'bucket': '$bucket' }` parameters, where `$endpoint_name` should match with the corresponding `name` of the configured endpoint in the YAML file above. In the following example, an endpoint named "s3.us-east-2.amazonaws.com" is defined in `object_storage.yaml`, and this endpoint is used when creating the keyspace "ks". in `object_storage.yaml`: ```yaml endpoints: name: s3.us-east-2.amazonaws.com port: 443 https: true aws_region: us-east-2 awsaccesskeyid: EXAMPLEACCESSKEYID awssecretaccesskey: EXAMPLESECRETACCESSKEY ``` and when creating the keyspace: ```cql CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'replication_factor' : 1 } AND STORAGE = { 'type' : 'S3', 'endpoint' : 's3.us-east-2.amazonaws.com', 'bucket' : 'bucket-for-testing' }; ```
Project: ShardingSphere (App Definition and Development / Database)
File: subscribe.en.md

+++ title = "Subscribe Guide" weight = 1 chapter = true +++ When using or contributing to ShardingSphere, if you encounter any problem, or you have any suggestions or new ideas, you can use the Apache mailing-list to participate in the community building, or join the ShardingSphere to connect with other community members. To join the mailing list: Send an e-mail to subscribe. Use your mailbox to send an e-mail to with any subject or content. Receive and reply the confirmation e-mail. After Step 1, you will receive a confirmation e-mail from (if you have not received the email, please check RSS feeds, junk e-mail or other items). Reply to that e-mail directly or click the link in the e-mail to reply, with any subject or content. Receive the welcome e-mail. After finishing the two steps above, an e-mail with the subject of `WELCOME to dev@shardingsphere.apache.org` will be sent to your e-mail address. You'll have then succeeded in subscribing to the Apache ShardingSphere mailing-list. You now can interact with community by your subscribed email or track email conversations by or .