cwielenb/CNCF-Dataset huggingface dataset:
Project: RabbitMQ (App Definition and Development / Streaming & Messaging)
File: README.md

This tiny HTTP server serves CA certificates from a user-specified local directory. It is meant to be used with in its test suite and as an example. `/`: serves a list of certificates in JSON. The format is `{"certificates":[{"id": <id>, "path": <path>}, ...]}` `/certs/<file_name>`: access for PEM encoded certificate files `/invlid`: serves invalid JSON, to be used in integration tests ``` <id> = <filename>:<filemodification_date> <path> = /certs/<file_name> <file_name> = name of a PEM file in the listed directory ``` To rebuild and run a release (requires Erlang to be installed): ``` gmake run CERT_DIR="/my/cacert/directory" PORT=8080 ``` To run from the pre-built escript (requires Erlang to be installed): ``` gmake CERTDIR="/my/cacert/directory" PORT=8080 ./rel/truststorehttprelease/bin/truststorehttprelease console ``` To start an HTTPS server, you should provide ssl options. It can be done via Erlang `.config` file format: ``` [{truststorehttp, [{ssl_options, [{cacertfile,"/path/to/testca/cacert.pem"}, {certfile,"/path/to/server/cert.pem"}, {keyfile,"/path/to/server/key.pem"}, {verify,verify_peer}, {failifnopeercert,false}]}]}] ``` This configuration can be added to `rel/sys.config` if you're running the application from source `make run` Or it can be specified as an environment variable: ``` CERTDIR="/my/cacert/directory" PORT=8443 CONFIGFILE=myconfig.config ./rel/truststorehttprelease/bin/truststorehttprelease console ``` Port and directory can be also set via config file: ``` [{truststorehttp, [{directory, "/tmp/certs"}, {port, 8081}, {ssl_options, [{cacertfile,"/path/to/testca/cacert.pem"}, {certfile,"/path/to/server/cert.pem"}, {keyfile,"/path/to/server/key.pem"}, {verify,verify_peer}, {failifnopeercert,false}]}]}] ```
Project: Databend (App Definition and Development / Database)
File: README.md

This script tests whether a newer version databend-query can read fuse table data written by a older version databend-query. ```shell tests/fuse-compat/test-fuse-compat.sh <old_ver> tests/fuse-compat/test-fuse-forward-compat.sh <old_ver> ``` E.g. `tests/fuse-compat/test-fuse-compat.sh 0.7.151` tests if the fuse-table written by databend-query-0.7.151 can be read by current version databend-query. `tests/fuse-compat/test-fuse-forward-compat.sh 1.2.307` tests if the fuse-table written by current can be read by databend-query-0.7.151 version databend-query. Current version of databend-query and databend-meta must reside in `./bins`: `./bins/current/databend-query` `./bins/current/databend-meta` Since building a binary takes several minutes, this step is usually done by the calling process, e.g., the CI script. Suite `tests/fuse-compat/compat-logictest/fusecompatwrite` writes data into a fuse table via an old version query. Suite `tests/fuse-compat/compat-logictest/fusecompatread` reads the data via current version query. Fuse table maintainers update these two `logictest` scripts to let the write/read operations cover fuse-table features.
Project: Backstage (App Definition and Development / Application Definition & Image Build)
File: discovery.md

id: discovery title: Gerrit Discovery sidebar_label: Discovery description: Automatically discovering catalog entities from Gerrit repositories The Gerrit integration has a special entity provider for discovering catalog entities from Gerrit repositories. The provider uses the "List Projects" API in Gerrit to get a list of repositories and will automatically ingest all `catalog-info.yaml` files stored in the root of the matching projects. As this provider is not one of the default providers, you will first need to install the Gerrit provider plugin: ```bash yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gerrit ``` Then add the plugin to the plugin catalog `packages/backend/src/plugins/catalog.ts`: ```ts / packages/backend/src/plugins/catalog.ts / import { GerritEntityProvider } from '@backstage/plugin-catalog-backend-module-gerrit'; const builder = await CatalogBuilder.create(env); / ... other processors and/or providers ... */ builder.addEntityProvider( GerritEntityProvider.fromConfig(env.config, { logger: env.logger, scheduler: env.scheduler, }), ); ``` To use the discovery processor, you'll need a Gerrit integration . Then you can add any number of providers. ```yaml catalog: providers: gerrit: yourProviderId: # identifies your dataset / provider independent of config changes host: gerrit-your-company.com branch: master # Optional query: 'state=ACTIVE&prefix=webapps' schedule: frequency: { minutes: 30 } timeout: { minutes: 3 } backend: host: gerrit-your-company.com branch: master # Optional query: 'state=ACTIVE&prefix=backend' ``` The provider configuration is composed of three parts: `host`: the host of the Gerrit integration to use. `branch` (optional): the branch where we will look for catalog entities (defaults to "master"). `query`: this string is directly used as the argument to the "List Project" API. Typically, you will want to have some filter here to exclude projects that will never contain any catalog files.
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: windows_.md

A comma delimited set of window references. Windows can be local or modular ```tremor win, mymodule::onesec, mymodule::fivesec ``` The identifers refer to a window definition and can be used in operators to define a temporally bound set of events based on the semantics of the window definition. In a tilt frame - or set of windows, the output of a window can is the input the next window in a sequence. This is a form of temporal window-driven event compaction that allows memory be conserved. At 1000 events per second, a 1 minute window needs to store 60,000 events per group per second. But 60 1 second windows can be merged with aggregate functions like `dds` and `hdr` histograms. Say, each histogram is 1k of memory per group per frame - that is a cost of 2k bytes per group. In a streaming system - indefinite aggregation of in memory events is always a tradeoff against available reosurces, and the relative business value. Often multiple windows in a tilt frame can be more effective than a single very long lived window.
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: Patterns.md

```rust self.pipelines.retain(|item| item != to_delete) ``` the loop: ```rust for (idx, in_port) in outgoing { self.stack.push((*idx, in_port.clone(), event.clone())) } ``` for (things that deref to) slices becomes: ```rust if let Some((lastoutgoing, otheroutgoing)) = outgoing.split_last() { // Iterate over all but the last `outgoing`, cloning the `event` for (idx, inport) in otheroutgoing { self.stack.push((*idx, in_port.clone(), event.clone())) } // Handle the last `outgoing`, consuming the `event` let (idx, inport) = lastoutgoing; self.stack.push((*idx, in_port, event)) } else { // `outgoing` was empty } ``` or for iterators over non-slices becomes: ```rust if outgoing.is_empty() { // `outgoing` was empty } else { let len = outgoing.len(); // Iterate over all but the last `outgoing`, cloning the `event` for (idx, in_port) in outgoing.iter().take(len - 1) { self.stack.push((*idx, in_port.clone(), event.clone())) } // Handle the last `outgoing`, consuming the `event` let (idx, in_port) = &outgoing[len - 1]; self.stack.push((*idx, in_port.clone(), event)) } ``` ```rust self.metrics[idx].outputs.entry(inport.clone()).orinsert(0) += 1; ``` becomes where the clone is limited to only happen when the element doesn't exist. ```rust if let Some(count) = self.metrics[*idx].outputs.getmut(inport) { *count += 1; } else { self.metrics[*idx].outputs.insert(in_port.clone(), 1); } ``` https://docs.rs/downcast-rs/1.1.0/downcast_rs/ allows downcasting on traits. An example can be found in `TremorAggrFn`. ```rust pub trait TremorAggrFn: DowncastSync + Sync + Send { //... fn merge(&mut self, src: &dyn TremorAggrFn) -> FResult<()>; //... } impl_downcast!(sync TremorAggrFn); ``` With `impl_downcast` we can can pass src as `dyn TremorAggrFn` instead of `dyn Any` locking down
Project: Hazelcast IMDG (App Definition and Development / Database)
File: README.md

Files in this folder were generated using `createkeymaterialopenssl` script. If you don't have access to it, you can generate files using combination of `openssl` and `keytool` command. You should use `pkcs12` key stores. Only unusual file is the `localhost.p12` file. It must contain client certificate and then the private key.
Project: Numaflow (App Definition and Development / Streaming & Messaging)
File: filter.md

A `filter` is a special-purpose built-in function. It is used to evaluate on each message in a pipeline and is often used to filter the number of messages that are passed to next vertices. Filter function supports comprehensive expression language which extends flexibility write complex expressions. `payload` will be root element to represent the message object in expression. Filter expression implemented with `expr` and `sprig` libraries. These function can be accessed directly in expression. `json` - Convert payload in JSON object. e.g: `json(payload)` `int` - Convert element/payload into `int` value. e.g: `int(json(payload).id)` `string` - Convert element/payload into `string` value. e.g: `string(json(payload).amount)` `Sprig` library has 70+ functions. `sprig` prefix need to be added to access the sprig functions. E.g: `sprig.contains('James', json(payload).name)` # `James` is contained in the value of `name`. `int(json(sprig.b64dec(payload)).id) < 100` ```yaml spec: vertices: name: in source: http: {} transformer: builtin: name: filter kwargs: expression: int(json(payload).id) < 100 ```
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: bytespart.md

The part may take the following general form ```ebnf SimpleExprImut ':' 'int' '/' Ident ``` Where: The `SimpleExprImut can be a literal or identifier to the data being encoded. A optional size in bits, or defaulted based on the data being encoded. An optional encoding hint as an identifier The size must be zero or greater, up to and including but no larger than 64 bits. |Ident|Description| ||| |`binary`|Encoded in binary, using network ( big ) endian| |`big-unsigned-integer`|Unsigned integer encoding, big endian| |`little-unsigned-integer`|Unsigned integer encoding, little endian| |`big-signed-integer`|Signed integer encoding, big endian| |`little-signed-integer`|Signed integer encoding, little endian|
Project: SchemaHero (App Definition and Development / Database)
File: README.md

The files in this directory are an example of using SchemaHero to manage a PostgreSQL database using credentials managed by HashiCorp Vault. Deploy a PostgreSQL and Vault instance to a new namespace. The following manifests were taken from the PostgreSQL and Vault Helm Charts. The vault chart is in dev mode and should be used for this tutorial only -- do not use for production. ``` kubectl create ns schemahero-vault kubectl apply -f ./postgresql/postgresql-11.8.0.yaml kubectl apply -f ./vault/vault.yaml ``` Next, enable the database secret engine: ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault secrets enable database ``` Now, we need to create a Vault role and config: ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault write database/roles/schemahero \ db_name=airlinedb \ creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \ GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \ revocation_statements="ALTER ROLE \"{{name}}\" NOLOGIN;"\ default_ttl="1h" \ max_ttl="24h" ``` ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault write database/config/airlinedb \ plugin_name=postgresql-database-plugin \ allowed_roles="*" \ connection_url="postgresql://{{username}}:{{password}}@postgresql:5432/airlinedb?sslmode=disable" \ username="postgres" \ password="password" ``` The following command will request a new username and password for our database. This is just confirming that Vault it working and has permissions. ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault read database/creds/schemahero ``` ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault auth enable kubernetes ``` FROM YOUR COMPUTER (not in the vault pod): ``` kubectl -n schemahero-vault exec $(kubectl -n schemahero-vault get pods --selector "app.kubernetes.io/instance=vault,component=server" -o jsonpath="{.items[0].metadata.name}") -c vault -- \ sh -c ' \ VAULT_TOKEN=root vault write auth/kubernetes/config \ tokenreviewerjwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ kuberneteshost=https://${KUBERNETESPORT443TCP_ADDR}:443 \ kubernetescacert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' ``` Create the policy ``` tee -a /tmp/policy.hcl > /dev/null <<EOT path "database/creds/schemahero" { capabilities = ["read"] } path "database/config/airlinedb" { capabilities = ["read"] } EOT ``` ``` kubectl -n schemahero-vault cp /tmp/policy.hcl vault-0:/tmp/policy.hcl ``` ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault policy write schemahero /tmp/policy.hcl ``` ``` kubectl exec -n schemahero-vault -it vault-0 -- env VAULT_TOKEN=root vault write auth/kubernetes/role/schemahero \ boundserviceaccount_names=schemahero \ boundserviceaccount_namespaces=schemahero-vault \ policies=schemahero \ ttl=1h ``` Deploy the serviceaccount: ``` kubectl apply -f ./vault/sa.yaml ```