cwielenb/CNCF-Dataset huggingface dataset:
Project: Trivy (Provisioning / Security & Compliance)
File: db.md

| Scanner | Supported | |:-:|::| | Vulnerability | | | Misconfiguration | | | Secret | | | License | | The vulnerability database and the Java index database are needed only for vulnerability scanning. See for the detail. If you want to skip downloading the vulnerability database, use the `--skip-db-update` option. ``` $ trivy image --skip-db-update python:3.4-alpine3.9 ``` <details> <summary>Result</summary> ``` 2019-05-16T12:48:08.703+0900 INFO Detecting Alpine vulnerabilities... python:3.4-alpine3.9 (alpine 3.9.2) =================================== Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0) +++-+-++--+ | LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION | TITLE | +++-+-++--+ | openssl | CVE-2019-1543 | MEDIUM | 1.1.1a-r1 | 1.1.1b-r1 | openssl: ChaCha20-Poly1305 | | | | | | | with long nonces | +++-+-++--+ ``` </details> You can also ask `Trivy` to simply retrieve the vulnerability database. This is useful to initialize workers in Continuous Integration systems. ``` $ trivy image --download-db-only ``` `Trivy` could also download the vulnerability database from an external OCI registry by using `--db-repository` option. ``` $ trivy image --db-repository registry.gitlab.com/gitlab-org/security-products/dependencies/trivy-db ``` !!!note Trivy automatically adds the `trivy-db` schema version as a tag if the tag is not used: `trivy-db-registry:latest` => `trivy-db-registry:latest`, but `trivy-db-registry` => `trivy-db-registry:2`. The same options are also available for the Java index DB, which is used for scanning Java applications. Skipping an update can be done by using the `--skip-java-db-update` option, while `--download-java-db-only` can be used to only download the Java index DB. !!! Note In mode, `Java index DB` is currently only used on the `client` side. Downloading the Java index DB from an external OCI registry can be done by using the `--java-db-repository` option. ``` $ trivy image --java-db-repository registry.gitlab.com/gitlab-org/security-products/dependencies/trivy-java-db --download-java-db-only ``` !!!note Trivy automatically adds the `trivy-java-db` schema version as a tag if the tag is not used: `java-db-registry:latest` => `java-db-registry:latest`, but `java-db-registry` => `java-db-registry:1`. The `--reset` flag removes all caches and databases. ``` $ trivy image --reset ```
Project: Chef InSpec (Provisioning / Security & Compliance)
File: crontab.md

+++ title = "crontab resource" draft = false gh_repo = "inspec" platform = "linux" [menu] [menu.inspec] title = "crontab" identifier = "inspec/resources/os/crontab.md crontab resource" parent = "inspec/resources/os" +++ Use the `crontab` Chef InSpec audit resource to test the crontab entries for a particular user on the system. It recognizes special time strings (@yearly, @weekly, etc). {{< readfile file="content/inspec/reusable/md/inspec_installation.md" >}} This resource first became available in v1.15.0 of InSpec. A `crontab` resource block declares a user (which defaults to the current user) and the details to be tested, such as the schedule elements for each crontab entry or the exact commands themselves: describe crontab do its('commands') { should include '/some/scheduled/task.sh' } end {{< note >}} The `include` matcher in this context specifies the entire list of commands that the crontab should include and not a particular substring that should be included by a command. The `include` matcher always matches a complete command invocation, including options and arguments. {{< /note >}} The path to the system crontab can also be supplied via: describe crontab(path: '/etc/cron.d/some_crontab') do its('commands') { should include '/path/to/some/script' } end Note that only the path or the user (and not both) should be supplied as arguments to the resource. The following examples show how to use this Chef InSpec audit resource. describe crontab('root') do its('commands') { should include '/path/to/some/script -option arg' } end describe crontab('username').commands('/home/username/build.sh') do its('hours') { should cmp '*' } its('minutes') { should cmp '*' } end ```ruby describe crontab.where({'hour' => '', 'minute' => ''}) do its('entries.length') { should cmp '0' } end ``` ```ruby describe crontab.where { command =~ /a partial command string/ } do its('entries.length') { should cmp 1 } end ``` describe crontab.commands('/root/annual_report.sh') do its('hours') { should cmp '0' } its('minutes') { should cmp '0' } its('days') { should cmp '1' } its('months') { should cmp '1' } end describe crontab.commands('/root/reboot.sh') do its('hours') { should cmp '-1' } its('minutes') { should cmp '-1' } end describe crontab do its('minutes') { should cmp '0' } its('hours') { should cmp '0' } its('days') { should cmp '1' } its('weekdays') { should cmp '1' } its('user') { should include 'username'} its('commands') { should include '/some/scheduled/task.sh' } end Chef InSpec will automatically interpret crontab-supported special time strings. For example, a crontab entry set to run `@yearly` can be tested as if the entry was manually configured to run on January 1, 12 AM. {{< readfile file="content/inspec/reusable/md/inspecmatcherslink.md" >}}
Project: kube-hunter (Provisioning / Security & Compliance)
File: KHV006.md

vid: KHV006 title: Insecure (HTTP) access to Kubernetes API categories: [Unauthenticated Access] severity: high The API Server port is accessible over plain HTTP, and therefore unencrypted and potentially insecured. Ensure your setup is exposing kube-api only on an HTTPS port. Do not enable kube-api's `--insecure-port` flag in production.
Project: DevSpace (App Definition and Development / Application Definition & Image Build)
File: devspace_status.md

title: "Command - devspace status" sidebar_label: devspace status Show the current status ``` ``` ``` -h, --help help for status ``` ``` --config string The devspace config file to use --debug Prints the stack trace if an error occurs --kube-context string The kubernetes context to use -n, --namespace string The kubernetes namespace to use --no-warn If true does not show any warning when deploying into a different namespace or kube-context than before -p, --profile string The devspace profile to use (if there is any) --silent Run in silent mode and prevents any devspace log output except panics & fatals -s, --switch-context Switches and uses the last kube context and namespace that was used to deploy the DevSpace project --var strings Variables to override during execution (e.g. --var=MYVAR=MYVALUE) ```
Project: Open Policy Agent (OPA) (Provisioning / Security & Compliance)
File: gatekeeper.md

title: OPA Gatekeeper subtitle: Rego Policy Controller for Kubernetes labels: type: poweredbyopa layer: configuration code: https://github.com/open-policy-agent/gatekeeper software: kubernetes tutorials: https://open-policy-agent.github.io/gatekeeper/website/docs/howto videos: https://youtu.be/RMiovzGGCfI?t=1049 https://youtu.be/6RNp3m_THw4?t=864 docs_features: go-integration: note: | OPA Gatekeeper is written in Go and uses the to evaluate policies loaded from Custom Resources. kubernetes: note: | OPA Gatekeeper integrates with and also uses Custom Resources and the Kubernetes API server to store policy state. allowmissingimage: true Manage Rego Kubernetes admission policies using Custom Resources.
Project: Tetragon (Provisioning / Security & Compliance)
File: verify.md

title: "Verify installation" weight: 5 description: "Verify Tetragon image and software bill of materials signatures" aliases: ["/docs/tutorials/verify-tetragon-image-signatures", "/docs/tutorials/software-bill-of-materials"] Learn how to verify Tetragon container images signatures. You will need to . Since version 0.8.4, all Tetragon container images are signed using cosign. Let's verify a Tetragon image's signature using the `cosign verify` command: ```shell cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com <Image URL> | jq ``` {{< note >}} If you are using cosign < v2.0.0, you must set `COSIGN_EXPERIMENTAL=1` environment variable to allow verification of images signed in KEYLESS mode. To learn more about keyless signing, please refer to . {{< /note >}} Download and verify the signature of the software bill of materials A Software Bill of Materials (SBOM) is a complete, formally structured list of components that are required to build a given piece of software. SBOM provides insight into the software supply chain and any potential concerns related to license compliance and security that might exist. Starting with version 0.8.4, all Tetragon images include an SBOM. The SBOM is generated in format using the tool. If you are new to the concept of SBOM, see . The SBOM can be downloaded from the supplied Tetragon image using the `cosign download sbom` command. ```shell cosign download sbom --output-file sbom.spdx <Image URL> ``` To ensure the SBOM is tamper-proof, its signature can be verified using the `cosign verify` command. ```shell COSIGN_EXPERIMENTAL=1 cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com --attachment sbom <Image URL> | jq ``` It can be validated that the SBOM image was signed using Github Actions in the Cilium repository from the `Issuer` and `Subject` fields of the output.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 18_io_kafka.md

Prompt: Is Apache Kafka supported in Apache Beam? Response: is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It is horizontally scalable, fault-tolerant, and fast. Apache Beam has a built-in KafkaIO connector for Apache Kafka that enables Apache Beam pipelines to read data from and write data to Kafka. Apache Kafka is supported in both batch and streaming pipelines and can serve as both a source and a sink. For more information, refer to the KafkaIO connector documentation in the following languages: To get started with Apache Kafka and Apache Beam, refer to the . Here is an example of Apache Beam pipeline code for reading data from a Kafka topic: ```python with beam.Pipeline(options=options) as p: output = (p | "Read from Kafka" >> ReadFromKafka( consumer_config={'bootstrap.servers': 'localhost:9092'}, topics=['my_topic'], with_metadata=False) | "Log Data" >> Map(logging.info)) ``` This code reads data from a Kafka topic `my_topic` on `localhost:9092` and logs the data to the console. For a detailed demonstration of using the KafkaIO connector, refer to the in the Apache Beam GitHub repository.
Project: Hazelcast Jet (App Definition and Development / Streaming & Messaging)
File: README.md

This website was created with . Make sure all the dependencies for the website are installed: ```sh $ yarn ``` Run your dev server: ```sh $ yarn start ``` Run Markdown linter ```sh npm run lint:markdown ``` Run Link Checker ```sh npm run link-check ``` Your project file structure should look something like this ``` my-docusaurus/ docs/ doc-1.md doc-2.md doc-3.md website/ blog/ 2016-3-11-oldest-post.md 2017-10-24-newest-post.md core/ node_modules/ pages/ static/ css/ img/ package.json sidebars.json siteConfig.js ``` Edit docs by navigating to `docs/` and editing the corresponding document: `docs/doc-to-be-edited.md` ```markdown title: This Doc Needs To Be Edited Edit me... ``` For more information about docs, click Edit blog posts by navigating to `website/blog` and editing the corresponding post: `website/blog/post-to-be-edited.md` ```markdown title: This Blog Post Needs To Be Edited Edit me... ``` For more information about blog posts, click Create the doc as a new markdown file in `/docs`, example `docs/newly-created-doc.md`: ```md title: This Doc Needs To Be Edited My new content here.. ``` Refer to that doc's ID in an existing sidebar in `website/sidebars.json`: ```javascript // Add newly-created-doc to the Getting Started category of docs { "docs": { "Getting Started": [ "quick-start", "newly-created-doc" // new doc here ], ... }, ... } ``` For more information about adding new docs, click Make sure there is a header link to your blog in `website/siteConfig.js`: `website/siteConfig.js` ```javascript headerLinks: [ ... { blog: true, label: 'Blog' }, ... ] ``` Create the blog post with the format `YYYY-MM-DD-My-Blog-Post-Title.md` in `website/blog`: `website/blog/2018-05-21-New-Blog-Post.md` ```markdown author: Frank Li authorURL: https://twitter.com/foobarbaz authorFBID: 503283835 title: New Blog Post Lorem Ipsum... ``` For more information about blog posts, click Add links to docs, custom pages or external links by editing the headerLinks field of `website/siteConfig.js`: `website/siteConfig.js` ```javascript { headerLinks: [ ... / you can add docs / { doc: 'my-examples', label: 'Examples' }, / you can add custom pages / { page: 'help', label: 'Help' }, / you can add external links / { href: 'https://github.com/facebook/docusaurus', label: 'GitHub' }, ... ], ... } ``` For more information about the navigation bar, click Docusaurus uses React components to build pages. The components are saved as .js files in `website/pages/en`: If you want your page to show up in your navigation header, you will need to update `website/siteConfig.js` to add to the `headerLinks` element: `website/siteConfig.js` ```javascript { headerLinks: [ ... { page: 'my-new-custom-page', label: 'My New Custom Page' }, ... ], ... } ``` For more information about custom pages, click . Full documentation can be found on the .
Project: Apache Storm (App Definition and Development / Streaming & Messaging)
File: IConfigLoader.md

title: IConfigLoader layout: documentation documentation: true IConfigLoader is an interface designed to allow dynamic loading of scheduler resource constraints. Currently, the MultiTenant scheduler uses this interface to dynamically load the number of isolated nodes a given user has been guaranteed, and the ResoureAwareScheduler uses the interface to dynamically load per user resource guarantees. The following interface is provided for users to create an IConfigLoader instance based on the scheme of the `scheduler.config.loader.uri`. ``` ConfigLoaderFactoryService.createConfigLoader(Map<String, Object> conf) ``` ``` public interface IConfigLoader { Map<?,?> load(); }; ``` load is called by the scheduler whenever it wishes to retrieve the most recent configuration map. The loaders are dynamically selected and dynamically configured through configuration items in the scheduler implementations. ``` scheduler.config.loader.uri: "artifactory+http://artifactory.my.company.com:8000/artifactory/configurations/clusters/mycluster/raspools" scheduler.config.loader.timeout.sec: 30 ``` Or ``` scheduler.config.loader.uri: "file:///path/to/my/config.yaml" ``` There are currently two implemenations of IConfigLoader org.apache.storm.scheduler.utils.ArtifactoryConfigLoader: Load configurations from an Artifactory server. It will be used if users add `artifactory+` to the scheme of the real URI and set to `scheduler.config.loader.uri`. org.apache.storm.scheduler.utils.FileConfigLoader: Load configurations from a local file. It will be used if users use `file` scheme. scheduler.config.loader.uri: For `ArtifactoryConfigLoader`, this can either be a reference to an individual file in Artifactory or to a directory. If it is a directory, the file with the largest lexographic name will be returned. For `FileConfigLoader`, this is the URI pointing to a file. scheduler.config.loader.timeout.secs: Currently only used in `ArtifactoryConfigLoader`. It is the amount of time an http connection to the artifactory server will wait before timing out. The default is 10. scheduler.config.loader.polltime.secs: Currently only used in `ArtifactoryConfigLoader`. It is the frequency at which the plugin will call out to artifactory instead of returning the most recently cached result. The default is 600 seconds. scheduler.config.loader.artifactory.base.directory: Only used in `ArtifactoryConfigLoader`. It is the part of the uri, configurable in Artifactory, which represents the top of the directory tree. It defaults to "/artifactory".
Project: Numaflow (App Definition and Development / Streaming & Messaging)
File: nats.md

A `Nats` source is used to ingest the messages from a nats subject. ```yaml spec: vertices: name: input source: nats: url: nats://demo.nats.io # Multiple urls separated by comma. subject: my-subject queue: my-queue # Queue subscription, see https://docs.nats.io/using-nats/developer/receiving/queues tls: # Optional. insecureSkipVerify: # Optional, where to skip TLS verification. Default to false. caCertSecret: # Optional, a secret reference, which contains the CA Cert. name: my-ca-cert key: my-ca-cert-key certSecret: # Optional, pointing to a secret reference which contains the Cert. name: my-cert key: my-cert-key keySecret: # Optional, pointing to a secret reference which contains the Private Key. name: my-pk key: my-pk-key auth: # Optional. basic: # Optional, pointing to the secret references which contain user name and password. user: name: my-secret key: my-user password: name: my-secret key: my-password ``` The `auth` strategies supported in `nats` source include `basic` (user and password), `token` and `nkey`, check the for the details.
Project: YDB (App Definition and Development / Database)
File: backup-and-recovery.md

Backups protect against data loss by letting you restore data. {{ ydb-short-name }} provides multiple solutions for backup and recovery: Backing up data to files and restoring it using the {{ ydb-short-name }} CLI. Backing up data to S3-compatible storage and restoring it using the {{ ydb-short-name }} CLI. {% include %} To back up data to a file, run the `ydb tools dump` command. To learn more about this command, follow the to the {{ ydb-short-name }} CLI reference. To restore data from a backup, run the `ydb tools restore` command. To learn more about this command, follow the to the {{ ydb-short-name }} CLI reference. To back up data to S3-compatible storage (such as ), run the `ydb export s3` command. To learn more about this command, follow the to the {{ ydb-short-name }} CLI reference. To restore data from a backup created in S3-compatible storage, run the `ydb import s3` command. To learn more about this command, follow the to the {{ ydb-short-name }} CLI reference. {% include %} {% include %}
Project: Apache Storm (App Definition and Development / Streaming & Messaging)
File: Defining-a-non-jvm-language-dsl-for-storm.md

title: Defining a Non-JVM DSL for Storm layout: documentation documentation: true The right place to start to learn how to make a non-JVM DSL for Storm is . Since Storm topologies are just Thrift structures, and Nimbus is a Thrift daemon, you can create and submit topologies in any language. When you create the Thrift structs for spouts and bolts, the code for the spout or bolt is specified in the ComponentObject struct: ``` union ComponentObject { 1: binary serialized_java; 2: ShellComponent shell; 3: JavaObject java_object; } ``` For a Python DSL, you would want to make use of "2" and "3". ShellComponent lets you specify a script to run that component (e.g., your python code). And JavaObject lets you specify native java spouts and bolts for the component (and Storm will use reflection to create that spout or bolt). There's a "storm shell" command that will help with submitting a topology. Its usage is like this: ``` storm shell resources/ python3 topology.py arg1 arg2 ``` storm shell will then package resources/ into a jar, upload the jar to Nimbus, and call your topology.py script like this: ``` python3 topology.py arg1 arg2 {nimbus-host} {nimbus-port} {uploaded-jar-location} ``` Then you can connect to Nimbus using the Thrift API and submit the topology, passing {uploaded-jar-location} into the submitTopology method. For reference, here's the submitTopology definition: ```java void submitTopology( 1: string name, 2: string uploadedJarLocation, 3: string jsonConf, 4: StormTopology topology) throws ( 1: AlreadyAliveException e, 2: InvalidTopologyException ite); ``` Finally, one of the key things to do in a non-JVM DSL is make it easy to define the entire topology in one file (the bolts, spouts, and the definition of the topology).
Project: Kapitan (Provisioning / Automation & Configuration)
File: code.md

comments: true tags: community Many of our features come from contributions from external collaborators. Please help us improve Kapitan by extending it with your ideas, or help us squash bugs you discover. It's simple, just send us a PR with your improvements! We would like ask you to Kapitan project and create a targeting master branch. All submissions, including submissions by project members, require review. We build kapitan using `poetry`. Install poetry ```bash pip install poetry ``` Install dependencies ```bash poetry install --with test ``` Poetry creates a virtual environment with the required dependencies installed. Run kapitan with the own compiled code ```bash poetry run kapitan <your command> ``` Because we are using a pinned version of reclass which is added as a submodule into Kapitan's repository, you need to pull it separately by executing the command below: ```shell git submodule update --init ``` Check if gcc is installed: ```shell brew install gcc@5 ``` Run `make test` to run all tests. If you modify anything in the `examples/` folder make sure you replicate the compiled result of that in `tests/testkubernetescompiled`. If you add new features, run `make testcoverage && make testformatting` to make sure the test coverage remains at current or better levels and that code formatting is applied. If you would like to evaluate your changes by running your version of Kapitan, you can do that by running `bin/kapitan` from this repository or even setting an alias to it. ```shell python3 -m unittest tests/testvaulttransit.py ``` To make sure you adhere to the Python Black is used to apply the formatting so make sure you have it installed with `pip3 install black`. Run `pip3 install pre-commit` to install precommit framework. In the Kapitan root directory, run `pre-commit install` Git add/commit any changed files you want. Run `make format_codestyle` before submitting. Create a branch named `release-v<NUMBER>`. Use `v0..-rc.` if you want pre-release versions to be uploaded. Update CHANGELOG.md with the release changes. Once reviewed and merged, Github Actions will auto-release. The merge has to happen with a merge commit not with squash/rebase so that the commit message still mentions `kapicorp/release-v` inside. To package any extra resources/files in the pip package, make sure you modify both `MANIFEST.in`.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: kio.md

title: "Kio" icon: /images/logos/powered-by/kio.png hasNav: true cardDescription: "Kio is a set of Kotlin extensions for Apache Beam to implement fluent-like API for Java SDK." <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="case-study-post"> ``` // Create Kio context val kio = Kio.fromArguments(args) // Configure a pipeline kio.read().text("~/input.txt") .map { it.toLowerCase() } .flatMap { it.split("\\W+".toRegex()) } .filter { it.isNotEmpty() } .countByValue() .forEach { println(it) } // And execute it kio.execute().waitUntilDone() ``` For more information about Kio, please see the documentation here: . </div> <div class="clear-nav"></div>