cwielenb/CNCF-Dataset huggingface dataset:
Project: SeaTunnel (App Definition and Development / Streaming & Messaging)
File: quick-start-spark.md

sidebar_position: 4 Before starting, make sure you have downloaded and deployed SeaTunnel as described in Please first(required version >= 2.4.0). For more information you could see Configure SeaTunnel: Change the setting in `config/seatunnel-env.sh`, it is base on the path your engine install at . Change `SPARK_HOME` to the Spark deployment dir. Edit `config/seatunnel.streaming.conf.template`, which determines the way and logic of data input, processing, and output after seatunnel is started. The following is an example of the configuration file, which is the same as the example application mentioned above. ```hocon env { parallelism = 1 job.mode = "BATCH" } source { FakeSource { resulttablename = "fake" row.num = 16 schema = { fields { name = "string" age = "int" } } } } transform { FieldMapper { sourcetablename = "fake" resulttablename = "fake1" field_mapper = { age = age name = new_name } } } sink { Console { sourcetablename = "fake1" } } ``` More information about config please check You could start the application by the following commands spark 2.4.x ```bash cd "apache-seatunnel-${version}" ./bin/start-seatunnel-spark-2-connector-v2.sh \ --master local[4] \ --deploy-mode client \ --config ./config/v2.streaming.conf.template ``` spark3.x.x ```shell cd "apache-seatunnel-${version}" ./bin/start-seatunnel-spark-3-connector-v2.sh \ --master local[4] \ --deploy-mode client \ --config ./config/v2.streaming.conf.template ``` See The Output: When you run the command, you could see its output in your console. You can think this is a sign that the command ran successfully or not. The SeaTunnel console will prints some logs as below: ```shell fields : name, age types : STRING, INT row=1 : elWaB, 1984352560 row=2 : uAtnp, 762961563 row=3 : TQEIB, 2042675010 row=4 : DcFjo, 593971283 row=5 : SenEb, 2099913608 row=6 : DHjkg, 1928005856 row=7 : eScCM, 526029657 row=8 : sgOeE, 600878991 row=9 : gwdvw, 1951126920 row=10 : nSiKE, 488708928 row=11 : xubpl, 1420202810 row=12 : rHZqb, 331185742 row=13 : rciGD, 1112878259 row=14 : qLhdI, 1457046294 row=15 : ZTkRx, 1240668386 row=16 : SGZCr, 94186144 ``` For now, you are already take a quick look about SeaTunnel with Spark, you could see to find all source and sink SeaTunnel supported. Or see if you want to know more about SeaTunnel Run With Spark. SeaTunnel have an own engine named `Zeta` and `Zeta` is the default engine of SeaTunnel. You can follow to configure and run a data synchronization job.
Project: KICS (Provisioning / Security & Compliance)
File: 0a494a6a-ebe2-48a0-9d77-cf9d5125e1b3.md

title: Redshift Cluster Without VPC hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 0a494a6a-ebe2-48a0-9d77-cf9d5125e1b3 Query name: Redshift Cluster Without VPC Platform: Terraform Severity: <span style="color:#edd57e">Low</span> Category: Insecure Configurations URL: Redshift Cluster should be configured in VPC (Virtual Private Cloud)<br> ```tf title="Positive test num. 1 - tf file" hl_lines="1" resource "awsredshiftcluster" "positive1" { cluster_identifier = "tf-redshift-cluster" database_name = "mydb" master_username = "foo" master_password = "Mustbe8characters" node_type = "dc1.large" cluster_type = "single-node" logging { enable = true bucket_name = "nameOfAnExistingS3Bucket" } } ``` ```tf title="Negative test num. 1 - tf file" resource "awsredshiftcluster" "negative1" { cluster_identifier = "tf-redshift-cluster" database_name = "mydb" master_username = "foo" master_password = "Mustbe8characters" node_type = "dc1.large" cluster_type = "single-node" logging { enable = true bucket_name = "nameOfAnExistingS3Bucket" } vpcsecuritygroup_ids = [ awssecuritygroup.redshift.id ] clustersubnetgroupname = awsredshiftsubnetgroup.redshiftsubnetgroup.id } ```
Project: Datree (Provisioning / Security & Compliance)
File: DEVELOPER_GUIDE.md

This guide explains how to set up your environment for developing on Datree. This guide was written for macOS and Linux machines. Go version 1.19 Git We use to build our programs. The simplest way to get started is: ``` $ make build ``` This will build the executable file and place it in the project root. One way to run datree locally is to use the newly created executable file: ``` $ ./datree test ./internal/fixtures/kube/k8s-demo.yaml ``` To run all the tests: ``` $ make test ``` Make sure you have read and understood the main guide: We use as our Command Line Interface framework. The available commands can be found under the `cmd` directory; each folder represents a datree command. To add a command, add a folder with the cobra command, and use it in the `cmd/root.go` file. Datree requires an internet connection to connect to our backend API. While developing locally, API requests will reach our staging environment and be visible on the . All available API requests can be found under `pkg/cliClient` It's best to use fixtures for manual testing, which are found under `internal/fixtures` To add a test for a given file, add a file with a `_test` suffix. For example: for the file `./reader.go` add a test file `./reader_test.go` For bug fixes: add a test that covers the bug fixed For features: add tests for the feature The main branch is the home of the current development candidate. We accept changes to the code via GitHub Pull Requests (PRs). One workflow for doing this is as follows: Fork the repository and clone it locally. Create a new working branch `git checkout -b "ISSUE#195someshort_description"` When you are ready for us to review your changes, push your branch to GitHub, and then open a new pull request to the `main` branch. For Git commit messages, please follow our . For example: `git commit -m "feat: add windows support"` We follow the - simply use your IDE's auto-formatter to make your code beautiful.
Project: Kui (App Definition and Development / Application Definition & Image Build)
File: kui_shell_core.ReplEval.md

/ / ReplEval .ReplEval Apply the given evaluator to the given arguments name: `string` apply<`T`, `O`\>(`commandUntrimmed`, `execOptions`, `evaluator`, `args`): `T` \| `Promise`<`T`\> | Name | Type | | : | :-- | | `T` | extends <`any`\> | | `O` | extends | | Name | Type | | :-- | :- | | `commandUntrimmed` | `string` | | `execOptions` | | | `evaluator` | <`T`, `O`\> | | `args` | <`O`\> | `T` \| `Promise`<`T`\>
Project: Tarantool (App Definition and Development / Database)
File: how_to_add_new_test.md

Firstly, for a new test should be prepared a `fill.lua` file. Here should be code to populate needed spaces, create xlogs, snapshots. `Fill.lua` should be run in an old Tarantool version, upgrade from which is going to be tested. This will produce `.xlog` and `.snap` files. They should be saved into `xlog/upgrade/<version>/<test_name>/` folder. Snapshot is usually enough. Version is the old Tarantool version. Test name is a name of test file, which will test recovery from these xlogs. It should obey to the same rules as other test file names. Secondly, it is necessary to add a `<test_name>.test.lua` file to `xlog/` folder, to all the other tests. It should start a new instance of Tarantool with workdir set to `xlog/upgrade/<version>/<test_name>/`. Then this file should ensure, that `box.upgrade()` works fine, and all data is recovered correctly. The following directory structure should be in the result: ``` xlog/ | +- <test_name>.test.lua | +- upgrade/ | +- <version>/ | +- <test_name>/ | +- fill.lua +- *.snap +- *.xlog ```
Project: StarRocks (App Definition and Development / Database)
File: array_distinct.md

displayed_sidebar: "English" Removes duplicate elements from an array. ```Haskell array_distinct(array) ``` `array`: the array from which you want to remove duplicate elements. Only the ARRAY data type is supported. Returns an array. The elements of the array that is returned may be sorted in a different order than the elements of the array that you specify. The elements of the array that is returned are of the same data type as the elements of the array that you specify. In this section, the following table is used as an example: ```plaintext mysql> select * from test; +++ | c1 | c2 | +++ | 1 | [1,1,2] | | 2 | [1,null,null] | | 3 | NULL | | 4 | [null] | +++ ``` Remove duplicate values from column `c2`. ```plaintext mysql> select c1, array_distinct(c2) from test; ++-+ | c1 | array_distinct(`c2`) | ++-+ | 1 | [2,1] | | 2 | [null,1] | | 3 | NULL | | 4 | [null] | ++-+ ```
Project: Distribution (Provisioning / Container Registry)
File: ROADMAP.md

The Distribution project aims to support the following use cases A library to support building highly scalable and reliable container registries, that can be customised for different backends and use cases. This is used by many of the largest registry operators, including Docker Hub, GitHub, GitLab, Harbor and Digital Ocean. A reference implementation of the OCI registry standards, and an easy way to experiment with new propsals in the registry space as these standards change. Distributed registry tools, such as caching registries and local registries that can be used within clusters for performance and locality use cases. As every container application needs at least one registry as part of its infrastructure, and more cloud native artifacts are using registries as the basis of their distribution, having a widely used and supported open source registry is important for innovation.
Project: Redpanda (App Definition and Development / Streaming & Messaging)
File: 20190312_rpk.md

Feature Name: Red Panda Keeper Status: draft Start Date: 2018-02-09 Authors: RFC PR: (PR # after acceptance of initial draft) Issue: (one or more # from the issue tracker) Red Panda Keeper is a binary that will be a Swiss Army Knife CLI application to interact with Red Panda queue. In order to provide high productivity and be distinguishable, installation, configuration and operation of the system must be clear, easy and user friendly. That is why Red Panda needs state of the art ops tool. In order to achieve that, it is planned to use modern CLI framework that will handle multiple configuration sources and deliver great user experience. The CLI system will help users and developers to interact with Red Panda in structured, understandable and easy way. The RPK is intedented to be the single application used to interact with Red Panda low latency queue. It will allow operators to: Download & install chosen version of Red Panda In future Setup developer cluser in docker-compose (same as Cockroach DB does) Tune the system Configure the queue Interact with the cluster Read the WAL files TBD The Red Panda Keeper CLI is going to be developed in Go using Cobra/Viper CLI development framework. The framework provides easy way to build CLI applications with support for flags, multiple configuration sources and user friendly output. TBD TBD TBD Oclif - Node Js based CLI framework based on NodeJS Knack - Python based CLI framework
Project: ThreatMapper (Provisioning / Security & Compliance)
File: other.md

title: Other Platforms ThreatMapper can perform compliance posture scanning on linux hosts and Kubernetes master and worker nodes. Scanning is done directly, using a local rather than by using the Cloud Scanner task employed by the cloud platform integrations. The sensor agent has direct visibility into the configuration of the base operating system, and can detect a wide range of compliance deviations that are not visible through an API. ThreatMapper can apply general and specific compliance benchmarks, including PCI, HIPAA, and NIST (Kube-master and Kube-slave). These benchmarks each select from a library of controls that cover best practices for Linux, Docker, Kubernetes (master and slave nodes) and well-known services. When you run a compliance scan, you can select which benchmarks you wish to measure against, and ThreatMapper will then evaluate the appropriate controls and present the results, by benchmark, once the scan has completed. For full information, refer to . :::tip Maximizing Coverage For maximum coverage, you can use both Cloud Scanner and Sensor Agent compliance scans together. You could scan your cloud infrastructure using Cloud Scanner, and scan selected VMs deployed within that infrastructure using the Sensor Agent. :::
Project: Vitess (App Definition and Development / Database)
File: java-packages.md

We publish binary packages for our . To do so, we use the http://oss.sonatype.org/ repository. New packages must be uploaded there ("deployed") and will be automatically published ("released"). Once they are released there, they will be automatically synchronized with Maven Central. The synchronization takes only several minutes, but the update on http://search.maven.org may take up to two hours. Then you must be added as member to our `io.vitess` namespace. Therefore, file a JIRA ticket with Sonatype to get added (). Follow . Install `gpg-agent` (needed below) e.g. on Ubuntu via: `sudo apt-get install gnupg-agent`. for Mac you need to install 'gnupg' via 'brew install gnupg' Create the `settings.xml` in the `$HOME/.m2/` directory as described in their .
Project: Meshery (Provisioning / Automation & Configuration)
File: mesheryctl-pattern-list.md

layout: default title: mesheryctl-pattern-list permalink: reference/mesheryctl/pattern/list redirect_from: reference/mesheryctl/pattern/list/ type: reference display-title: "false" language: en command: pattern subcommand: list List patterns Display list of all available pattern files. <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl pattern list [flags] </div> </pre> list all available patterns <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl pattern list </div> </pre> <pre class='codeblock-pre'> <div class='codeblock'> -h, --help help for list -v, --verbose Display full length user and pattern file identifiers </div> </pre> <pre class='codeblock-pre'> <div class='codeblock'> --config string path to config file (default "/home/runner/.meshery/config.yaml") -t, --token string Path to token file default from current context </div> </pre> Usage of mesheryctl pattern list Go back to , if you want to add content manually to the CLI documentation, please refer to the for guidance.
Project: KICS (Provisioning / Security & Compliance)
File: a0f1bfe0-741e-473f-b3b2-13e66f856fab.md

title: S3 Bucket Allows Put Action From All Principals hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: a0f1bfe0-741e-473f-b3b2-13e66f856fab Query name: S3 Bucket Allows Put Action From All Principals Platform: Ansible Severity: <span style="color:#ff0000">Critical</span> Category: Access Control URL: S3 Buckets must not allow Put Action From All Principals, as to prevent leaking private information to the entire internet or allow unauthorized data tampering / deletion. This means the 'Effect' must not be 'Allow' when the 'Action' is Put, for all Principals.<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="6" name: Bucket amazon.aws.s3_bucket: name: mys3bucket state: present policy: Version: "2020-10-07" Statement: Effect: Allow Action: PutObject Principal: "*" ``` ```yaml title="Negative test num. 1 - yaml file" name: Bucket amazon.aws.s3_bucket: name: mys3bucket state: present policy: Version: '2020-10-07' Statement: Effect: Allow Action: PutObject Principal: NotAll ```
Project: Harbor (Provisioning / Container Registry)
File: 12-01-LDAP-usergroup-add.md

Test 12-01 LDAP Usergroup Add ======= To verify admin user can add an LDAP group User guide This test requires that a Harbor instance is running and available. An LDAP server is running and available, and enabled memberof overlay feature LDAP group config parameter are configured. ldapgroupbasedn ldapgroupfilter ldap_gid ldapgroupscope Login UI as admin user. In `Administration->User Group` page, Add an LDAP with a valid group DN with group name. The user group should be created with specified name. In `Administration->User Group` page, Add an LDAP with a non-exist group DN The user group can not be created In `Administration->User Group` page, Add an LDAP with a group DN which already exist, but with different name. The user group is renamed to new user group name. In `Administration->User Group` page, Add an LDAP with a valid group DN without group name. The user group is created and named with the same name in LDAP. Change the configure parameter ldapgroupbasedn to another DN, so that the LDAP user group is outside the base DN. In `Administration->User Group` page, Add an LDAP with a valid group DN but outside the base DN. The user group can not be created Change ldapgroupscope from 2 to 0, so that the LDAP group can not be found with the current scope. In `Administration->User Group` page, Add an LDAP with a valid group DN but can not be searched. The user group can not be created Change ldapgroupfilter to with a specified filter, so that it can filter out the current group DN. In `Administration->User Group` page, Add an LDAP with a valid group DN but this group DN is filtered the user group can not be created. Change ldap_gid with another attribute other than cn In `Administration->User Group` page, Add an LDAP with a valid group DN, check the user group name. The user group is created, the group name is named by specified attributed.
Project: ShardingSphere (App Definition and Development / Database)
File: convert-yaml-configuration.en.md

+++ title = "CONVERT YAML CONFIGURATION" weight = 15 +++ The `CONVERT YAML CONFIGURATION` syntax is used to convert `YAML` configuration to DistSQL RDL statements. {{< tabs >}} {{% tab name="Grammar" %}} ```sql convertYamlConfiguration ::= 'CONVERT' 'YAML' 'CONFIGURATION' 'FROM' 'FILE' filePath filePath ::= string ``` {{% /tab %}} {{% tab name="Railroad diagram" %}} <iframe frameborder="0" name="diagram" id="diagram" width="100%" height="100%"></iframe> {{% /tab %}} {{< /tabs >}} The `CONVERT YAML CONFIGURATION` syntax only reads the YAML file and converts the configuration into DistSQL statements without affecting the current metadata; When `dataSources` in YAML is empty, `rules` conversion will not be performed. ```sql mysql> CONVERT YAML CONFIGURATION FROM FILE '/xxx/configshardingdb.yaml'; +--+ | dist_sql | +--+ | CREATE DATABASE sharding_db; USE sharding_db; REGISTER STORAGE UNIT ds_0 ( URL='jdbc:mysql://127.0.0.1:3306/demods0?serverTimezone=UTC&useSSL=false&allowPublicKeyRetrieval=true', USER='root', PASSWORD='123456', PROPERTIES('maxPoolSize'='10') ), ds_1 ( URL='jdbc:mysql://127.0.0.1:3306/demods1?serverTimezone=UTC&useSSL=false&allowPublicKeyRetrieval=true', USER='root', PASSWORD='123456', PROPERTIES('maxPoolSize'='10') ); CREATE SHARDING TABLE RULE t_order ( STORAGEUNITS(ds0,ds_1), SHARDINGCOLUMN=orderid, TYPE(NAME='mod', PROPERTIES('sharding-count'='4')), KEYGENERATESTRATEGY(COLUMN=order_id, TYPE(NAME='snowflake')) ); | +--+ 1 row in set (0.02 sec) ``` `CONVERT`, `YAML`, `CONFIGURATION`, `FROM`, `FILE`
Project: ClickHouse (App Definition and Development / Database)
File: v22.10.5.54-stable.md

sidebar_position: 1 sidebar_label: 2023 Backported in : Updated normaliser to clone the alias ast. resolves Implementation: Updated QueryNormalizer to clone alias ast, when its replaced. Previously just assigning the same leads to exception in LogicalExpressinsOptimizer as it would be the same parent being inserted again. * This bug is not seen with new analyser (allowexperimentalanalyzer), so no changes for it. I added a test for the same. (). Backported in : Fix race for backup of tables in Lazy databases. (). Backported in : Bring sha512 sums back to the building step. (). Backported in : Kill stress tests after 2.5h in case of hanging process. (). Backported in : In rare cases, we don't rebuild binaries, because another task with a similar prefix succeeded. E.g. `binarydarwin` didn't restart because `binarydarwin_aarch64`. (). Backported in : Retry the integration tests on compressing errors. (). Backported in : . (). Backported in : Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting. (). Backported in : Fix datetime schema inference in case of empty string. (). Backported in : Fix sumMap() for Nullable(Decimal()). (). Backported in : - Fix using multipart upload for large S3 objects in AWS S3. (). Backported in : Fixed `ALTER ... RESET SETTING` with `ON CLUSTER`. It could be applied to one replica only. Fixes . (). Backported in : Fix undefined behavior in the `quantiles` function, which might lead to uninitialized memory. Found by fuzzer. This closes . (). Backported in : Prevent `ReadonlyReplica` metric from having negative values. (). Backported in : Fix possible crash in case function `IN` with constant arguments was used as a constant argument together with `LowCardinality`. Fixes . (). Yet another fix for AggregateFunctionMinMaxAny (). Use only PRs to our repository in pr_info on push (). Fix tags workflow (). Assign release PRs (). Fix another bug in AggregateFunctionMinMaxAny (). Bump libdivide (to gain some new optimizations) (). Add check for submodules sanity (). Implement a custom central checkout action (). Fix crash on delete from materialized view (). Do not check read result consistency when unwinding ().
Project: Vault (Provisioning / Key Management)
File: feature_request.md

name: Feature request about: Suggest something! title: '' labels: '' assignees: '' Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] Describe the solution you'd like A clear and concise description of what you want to happen. Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. Explain any additional use-cases If there are any use-cases that would help us understand the use/need/value please share them as they can help us decide on acceptance and prioritization. Additional context Add any other context or screenshots about the feature request here.
Project: ClickHouse (App Definition and Development / Database)
File: ExternalDistributed.md

slug: /en/engines/table-engines/integrations/ExternalDistributed sidebar_position: 55 sidebar_label: ExternalDistributed title: ExternalDistributed The `ExternalDistributed` engine allows to perform `SELECT` queries on data that is stored on a remote servers MySQL or PostgreSQL. Accepts or engines as an argument so sharding is possible. ``` sql CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] ( name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1], name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2], ... ) ENGINE = ExternalDistributed('engine', 'host:port', 'database', 'table', 'user', 'password'); ``` See a detailed description of the query. The table structure can differ from the original table structure: Column names should be the same as in the original table, but you can use just some of these columns and in any order. Column types may differ from those in the original table. ClickHouse tries to values to the ClickHouse data types. Engine Parameters `engine` The table engine `MySQL` or `PostgreSQL`. `host:port` MySQL or PostgreSQL server address. `database` Remote database name. `table` Remote table name. `user` User name. `password` User password. Supports multiple replicas that must be listed by `|` and shards must be listed by `,`. For example: ```sql CREATE TABLE testshards (id UInt32, name String, age UInt32, money UInt32) ENGINE = ExternalDistributed('MySQL', `mysql{1|2}:3306,mysql{3|4}:3306`, 'clickhouse', 'testreplicas', 'root', 'clickhouse'); ``` When specifying replicas, one of the available replicas is selected for each of the shards when reading. If the connection fails, the next replica is selected, and so on for all the replicas. If the connection attempt fails for all the replicas, the attempt is repeated the same way several times. You can specify any number of shards and any number of replicas for each shard. See Also