cwielenb/CNCF-Dataset huggingface dataset:
Project: KICS (Provisioning / Security & Compliance)
File: cf4a5f45-a27b-49df-843a-9911dbfe71d4.md

title: Invalid Media Type Value (v3) hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: cf4a5f45-a27b-49df-843a-9911dbfe71d4 Query name: Invalid Media Type Value (v3) Platform: OpenAPI Severity: <span style="color:#5bc0de">Info</span> Category: Best Practices URL: The Media Type value should match the following format: <type>/<subtype><br> ```json title="Positive test num. 1 - json file" hl_lines="28" { "openapi": "3.0.0", "info": { "title": "Simple API Overview", "version": "1.0.0" }, "paths": { "/": { "get": { "operationId": "listVersionsv2", "summary": "List API versions", "responses": { "200": { "description": "200 response", "content": { "application/json": { "encoding": { "code": { "contentType": "image/png, image/jpeg" } } } } } }, "requestBody": { "content": { "multipart/form- data": { "encoding": { "code": { "contentType": "image/png, image/jpeg" } } } } } } } } } ``` ```yaml title="Positive test num. 2 - yaml file" hl_lines="20" openapi: 3.0.0 info: title: Simple API Overview version: 1.0.0 paths: "/": get: operationId: listVersionsv2 summary: List API versions responses: "200": description: 200 response content: application/json: encoding: code: contentType: image/png, image/jpeg requestBody: content: multipart/form- data: encoding: code: contentType: image/png, image/jpeg ``` ```json title="Negative test num. 1 - json file" { "openapi": "3.0.0", "info": { "title": "Simple API Overview", "version": "1.0.0" }, "paths": { "/": { "get": { "operationId": "listVersionsv2", "summary": "List API versions", "responses": { "200": { "description": "200 response", "content": { "application/json": { "encoding": { "code": { "contentType": "image/png, image/jpeg" } }, "schema": { "properties": { "code": { "type": "string", "format": "binary" }, "message": { "type": "string" } }, "type": "object", "discriminator": { "propertyName": "petType" } } } } } }, "requestBody": { "content": { "multipart/form-data": { "schema": { "type": "string", "format": "binary", "properties": { "code": { "type": "string", "format": "binary" } } }, "encoding": { "code": { "contentType": "image/png, image/jpeg" } } } } } } } } } ``` ```yaml title="Negative test num. 2 - yaml file" openapi: 3.0.0 info: title: Simple API Overview version: 1.0.0 paths: "/": get: operationId: listVersionsv2 summary: List API versions responses: "200": description: 200 response content: application/json: schema: type: object discriminator: propertyName: petType properties: code: type: string format: binary message: type: string encoding: code: contentType: image/png, image/jpeg requestBody: content: multipart/form-data: schema: type: string format: binary properties: code: type: string format: binary encoding: code: contentType: image/png, image/jpeg ```
Project: Kapitan (Provisioning / Automation & Configuration)
File: kap_10_azure_key_vault.md

This feature will enable users to encrypt secrets using keys stored in Azure's Key Vault. The `azkms` keyword will be used to access the azure key management backend. `key_id` uniquely identifies an Azure key object and it's version stored in Key Vault. It is of the form `https://{keyvault-name}.vault.azure.net/{object-type}/{object-name}/{object-version}`. It needs to be made accessible to kapitan in one of the following ways: As a part of target ```yaml parameters: kapitan: secrets: azkms: key: key_id #eg https://kapitanbackend.vault.azure.net/keys/myKey/deadbeef ``` As a flag ```shell kapitan refs --key=<keyid> --write azkms:/path/to/secret -f filewithsecretdata.txt ``` The following command will be used to encrypt a secret (using the specified key from Key Vault) and save it in the `refs-path` along with it's metadata ```shell echo "mytreasuredsecret" | kapitan refs --write azkms:path/to/secretinsidekapitan -t <target_name> -f - ``` The `-t <targetname>` is used to get the information about keyid. Once the secret is Base64 encoded and encrypted using the key, it will be stored in `path/to/secretinsidekapitan` as ```yaml data: bXlfdHJlYXN1cmVkX3NlY3JldAo= encoding: original key: https://kapitanbackend.vault.azure.net/keys/myKey/deadbeef type: azkms ``` note Cryptographic algorithm used for encryption would be rsa-oaep-256. Optimal Asymmetric Encryption Padding (OAEP) is a padding scheme often used together with RSA encryption. Secrets can be refered using `?{azkms:path/to/secret_id}` e.g. ```yaml parameter: mysql: storage: 10G storage_class: standard image: mysql:latest users: root: password: ?{azkms:path/to/secret} ``` After compilation, the secret reference will be postfixed with 8 characters from the sha256 hash of the retrieved password/secret ```yaml apiVersion: v1 data: MYSQLROOTPASSWORD: ?{azkms:path/to/secret:deadbeef} kind: Secret metadata: labels: name: example-mysql name: example-mysql namespace: minikube-mysql type: Opaque ``` To reveal the secret, the following command will be used `$ kapitan ref --reveal -f compiled/file/containing/secret` note Kapitan will not be responsible for authentication or access management to Azure
Project: Meshery (Provisioning / Automation & Configuration)
File: aws-mq-controller.md

layout: integration title: AWS MQ subtitle: Collaborative and visual infrastructure as code for AWS MQ image: /assets/img/integrations/aws-mq-controller/icons/color/aws-mq-controller-color.svg permalink: extensibility/integrations/aws-mq-controller docURL: https://docs.meshery.io/extensibility/integrations/aws-mq-controller description: integrations-category: App Definition and Development integrations-subcategory: Streaming & Messaging registrant: GitHub components: name: field-export colorIcon: assets/img/integrations/aws-mq-controller/components/field-export/icons/color/field-export-color.svg whiteIcon: assets/img/integrations/aws-mq-controller/components/field-export/icons/white/field-export-white.svg description: name: adopted-resource colorIcon: assets/img/integrations/aws-mq-controller/components/adopted-resource/icons/color/adopted-resource-color.svg whiteIcon: assets/img/integrations/aws-mq-controller/components/adopted-resource/icons/white/adopted-resource-white.svg description: name: broker colorIcon: assets/img/integrations/aws-mq-controller/components/broker/icons/color/broker-color.svg whiteIcon: assets/img/integrations/aws-mq-controller/components/broker/icons/white/broker-white.svg description: featureList: [ "Drag-n-drop cloud native infrastructure designer to configure, model, and deploy your workloads.", "Invite anyone to review and make changes to your private designs.", "Ongoing synchronization of Kubernetes configuration and changes across any number of clusters." ] howItWorks: "Collaborative Infrastructure as Code" howItWorksDetails: "Collaboratively manage infrastructure with your coworkers synchronously sharing the same designs." language: en list: include type: extensibility category: integrations
Project: YugabyteDB (App Definition and Development / Database)
File: hbase.md

title: Compare Apache HBase with YugabyteDB headerTitle: Apache HBase linkTitle: Apache HBase description: Compare Apache HBase database with YugabyteDB. aliases: /comparisons/hbase/ menu: preview_faq: parent: comparisons identifier: comparisons-hbase weight: 1130 type: docs Following are the key areas of differences between YugabyteDB and . Apache HBase relies on HDFS (another complex piece of infrastructure for data replication) and on Apache Zookeeper for leader election, failure detection, and so on. Running HBase smoothly requires a lot of on-going operational overheads. In HBase, on a region-server death, the unavailability window of shards/regions on the server can be in the order of 60 seconds or more. This is because the HBase master first needs to wait for the server's ephemeral node in Zookeeper to expire, followed by time take to split the transaction logs into per-shard recovery logs, and the time taken to replay the edits from the transaction log by a new server before the shard is available to take IO. In contrast, in Yugabyte, the tablet-peers are hot standbys, and within a matter of few heartbeats (a few seconds) detect failure of the leader, and initiate leader election. Avoids GC tuning; can run better on large memory machines. Richer data model: YugabyteDB offers a multi-model/multi-API using YSQL, YCQL. Rather than deal with just byte keys and values, YugabyteDB offers a rich set of scalar (int, text, decimal, binary, timestamp, etc.) and composite types (such as collections, UDTs, etc.). Flexible deployment choices across multiple data centers or availability zones. HBase provides strong consistency only within a single data center and offers only an asynchronous replication alternative for cross-DC deployments.
Project: KubeArmor (Provisioning / Security & Compliance)
File: conf-data.md

With a shell different than the user owning the file: ```sh $ cat /etc/ca-certificates.conf cat: /etc/ca-certificates.conf: Permission denied $ ``` ``` { "Action": "Block", "ClusterName": "d3mo", "ContainerID": "548176888fca6bb6d66633794f3d5f9d54930a9d9f43d4f05c11de821c758c0f", "ContainerImage": "docker.io/library/wordpress:4.8-apache@sha256:6216f64ab88fc51d311e38c7f69ca3f9aaba621492b4f1fa93ddf63093768845", "ContainerName": "wordpress", "Data": "syscall=SYSOPEN flags=ORDONLY", "Enforcer": "AppArmor", "HostName": "master-node", "HostPID": 39039, "HostPPID": 38787, "Labels": "app=wordpress", "NamespaceName": "wordpress-mysql", "Operation": "File", "Owner": { "Name": "wordpress", "Namespace": "wordpress-mysql", "Ref": "Deployment" }, "PID": 220, "PPID": 219, "ParentProcessName": "/bin/dash", "PodName": "wordpress-fb448db97-wj7n7", "PolicyName": "DefaultPosture", "ProcessName": "/bin/cat", "Resource": "/etc/ca-certificates.conf", "Result": "Permission denied", "Source": "/bin/cat /etc/ca-certificates.conf", "Timestamp": 1696485467, "Type": "MatchedPolicy", "UID": 1000, "UpdatedTime": "2023-10-05T05:57:47.935622Z", "cluster_id": "2302", "component_name": "kubearmor", "instanceGroup": "0", "instanceID": "0", "tenant_id": "167", "workload": "1" } ```
Project: Hazelcast Jet (App Definition and Development / Streaming & Messaging)
File: custom-batch-source.md

title: Add Batching to Custom Source description: Improve throughput by adding batching to a custom data source id: version-4.1.1-custom-batch-source original_id: custom-batch-source In the [Custom Sources and Sinks](../api/sources-sinks.md#custom-sources-and-sinks) section of our programming guide we have seen some basic examples of user-defined sources and sinks. Here we extend this topic by showing you how to add batching Here is how you create a source that reads lines of text from a file: ```java public class Sources { public static BatchSource<String> buildLineSource() { return SourceBuilder .batch("line-source", x -> new BufferedReader(new FileReader("lines.txt"))) .<String>fillBufferFn((in, buf) -> { String line = in.readLine(); if (line != null) { buf.add(line); } else { buf.close(); } }) .destroyFn(buf -> buf.close()) .build(); } } ``` Now it is ready to be used from the pipeline, just like a built-in source: ```java Pipeline p = Pipeline.create(); p.readFrom(Sources.buildLineSource()) .writeTo(Sinks.logger()); ``` While this simple source functions correctly, it's not optimal because it reads just one line and returns. There are some overheads to repeatedly calling `fillBufferFn` and you can add more than one item to the buffer: ```java SourceBuilder .batch("line-source", x -> new BufferedReader(new FileReader("lines.txt"))) .<String>fillBufferFn((in, buf) -> { for (int i = 0; i < 128; i++) { String line = in.readLine(); if (line == null) { buf.close(); return; } buf.add(line); } }) .destroyFn(buf -> buf.close()) .build(); ``` This code adds 128 lines at a time. In your specific case you should use the rule of thumb to target about 1 millisecond spent in a single invocation of `fillBufferFn`.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: enrichment-bigtable.md

title: "Enrichment with Bigtable" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< localstorage language language-py >}} <table> <tr> <td> <a> {{< button-pydoc path="apachebeam.transforms.enrichmenthandlers.bigtable" class="BigTableEnrichmentHandler" >}} </a> </td> </tr> </table> In Apache Beam 2.54.0 and later versions, the enrichment transform includes a built-in enrichment handler for . The following example demonstrates how to create a pipeline that use the enrichment transform with the handler. The data stored in the Bigtable cluster uses the following format: {{< table >}} | Row key | product:productid | product:productname | product:product_stock | |:--:|:--:|:-:|:--:| | 1 | 1 | pixel 5 | 2 | | 2 | 2 | pixel 6 | 4 | | 3 | 3 | pixel 7 | 20 | | 4 | 4 | pixel 8 | 10 | {{< /table >}} {{< highlight language="py" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/enrichment.py" enrichmentwithbigtable >}} {{</ highlight >}} {{< paragraph class="notebook-skip" >}} Output: {{< /paragraph >}} {{< highlight class="notebook-skip" >}} {{< codesample "sdks/python/apachebeam/examples/snippets/transforms/elementwise/enrichmenttest.py" enrichmentwith_bigtable >}} {{< /highlight >}} Not applicable. {{< button-pydoc path="apachebeam.transforms.enrichmenthandlers.bigtable" class="BigTableEnrichmentHandler" >}}
Project: KICS (Provisioning / Security & Compliance)
File: df746b39-6564-4fed-bf85-e9c44382303c.md

title: Apt Get Install Lists Were Not Deleted hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: df746b39-6564-4fed-bf85-e9c44382303c Query name: Apt Get Install Lists Were Not Deleted Platform: Dockerfile Severity: <span style="color:#5bc0de">Info</span> Category: Supply-Chain URL: After using apt-get install, it is needed to delete apt-get lists<br> ```dockerfile title="Positive test num. 1 - dockerfile file" hl_lines="8 2 12 5" FROM busybox1 RUN apt-get update && apt-get install --no-install-recommends -y python FROM busybox2 RUN apt-get install python FROM busybox3 RUN apt-get update && apt-get install --no-install-recommends -y python RUN rm -rf /var/lib/apt/lists/* FROM busybox4 RUN apt-get update && apt-get install --no-install-recommends -y python RUN rm -rf /var/lib/apt/lists/* RUN apt-get clean ``` ```dockerfile title="Positive test num. 2 - dockerfile file" hl_lines="2" FROM busybox5 RUN set -eux; \ apt-get update; \ apt-get install -y --no-install-recommends package=0.0.0 ``` ```dockerfile title="Negative test num. 1 - dockerfile file" FROM busyboxneg1 RUN apt-get update && apt-get install --no-install-recommends -y python \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* FROM busyboxneg2 RUN apt-get update && apt-get install --no-install-recommends -y python && apt-get clean FROM busyboxneg3 RUN apt-get update && apt-get install --no-install-recommends -y python \ && apt-get clean FROM busyboxneg4 RUN apt-get update && apt-get install --no-install-recommends -y python \ && rm -rf /var/lib/apt/lists/* ``` ```dockerfile title="Negative test num. 2 - dockerfile file" FROM busyboxneg5 RUN apt-get update; \ apt-get install --no-install-recommends -y python; \ apt-get clean; \ rm -rf /var/lib/apt/lists/* FROM busyboxneg6 RUN apt-get update; \ apt-get install --no-install-recommends -y python; \ apt-get clean FROM busyboxneg7 RUN set -eux; \ apt-get update; \ apt-get install -y --no-install-recommends package=0.0.0; \ rm -rf /var/lib/apt/lists/* ```
Project: Druid (App Definition and Development / Database)
File: versioning.md

id: versioning title: "Versioning" <!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, ~ software distributed under the License is distributed on an ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY ~ KIND, either express or implied. See the License for the ~ specific language governing permissions and limitations ~ under the License. --> This page discusses how we do versioning and provides information on our stable releases. Versioning Strategy We generally follow . The general idea is "Major" version (leftmost): backwards incompatible, no guarantees exist about APIs between the versions "Minor" version (middle number): you can move forward from a smaller number to a larger number, but moving backwards might* be incompatible. "bug-fix" version ("patch" or the rightmost): Interchangeable. The higher the number, the more things are fixed (hopefully), but the programming interfaces are completely compatible and you should be able to just drop in a new jar and have it work. Note that this is defined in terms of programming API, not in terms of functionality. It is possible that a brand new awesome way of doing something is introduced in a "bug-fix" release version if it doesnt add to the public API or change it. One exception for right now, while we are still in major version 0, we are considering the APIs to be in beta and are conflating "major" and "minor" so a minor version increase could be backwards incompatible for as long as we are at major version 0. These will be communicated via email on the group. For external deployments, we recommend running the stable release tag. Releases are considered stable after we have deployed them into our production environment and they have operated bug-free for some time. Tagging strategy Tags of the codebase are equivalent to release candidates. We tag the code every time we want to take it through our release process, which includes some QA cycles and deployments. So, it is not safe to assume that a tag is a stable release, it is a solidification of the code as it goes through our production QA cycle and deployment. Tags will never change, but we often go through a number of iterations of tags before actually getting a stable release onto production. So, it is recommended that if you are not aware of what is on a tag, to stick to the stable releases listed on the page.
Project: TiDB (App Definition and Development / Database)
File: 2022-03-03-rc-read-tso-optimization.md

Author(s): cfzjywxk Last updated: March. 3, 2022 Discussion at: For the `read-committed` isolation level, each read request in a single transaction will need to fetch a new `ts` to read the latest committed data. If the workload is a read-heavy one whose read QPS is significantly higher than writes or there're few read-write conflicts, fetching ts each time will increase the query lantecy. The new ts itself is used to ensure the most recent data will be returned, if the data version does not change frequently then it's unnecessary to fetch a new timestamp every time. The ts fetching could be processed in an optimistic way that ts fetching happens only when a more recent data version is encoutered, this saves the cost of many unncessary ts fetching. Introduce another system variable `tidbrcreadcheckts` to enable the "lazy ts fetch" read mode. It will take effect for all the in-transaction select statements when the transaction mode is `pessimistic` and the isolation level is `read-committed`. The execution flow is like this: If the query is executed first time, do not fetch a new `forupdatets` and just use the last valid one(`start_ts` for the first time). Do build the plan and executor like before. If the execution is successful, a `resultSet` will be returned. The connection layer will drive the response procesure and call `Next` function using the returned `resultSet`. The read executor will try to fetch data to fill in a data `chunk`. The bottommost executor could be a `pointGet`, `batchPointGet` or a coprocessor task. What's different is a `RcReadCheckTS` flag will be set on all these read requests. The read executor in the storage layer will check the read results, if a more recent version does exist then a `WriteConflict` error is reported. For data write record, do check if it's newer than the current `read_ts`. For lock record, return `ErrKeyIsLocked` even though the `lock.ts` is greater than the `readts` as the `readts` could be a stale one. If no error is returned then the query is finished. Otherwise if there's no `chunk` responsed to the client yet, retry the whole query fetching a new global ts to do the read like normal. The default behaviours of the `read-committed` isolation level will not change. One thing different is that if the user client uses `COMSTMTFETCH` like utility to read data from `TiDB`, there could be problem if the returned first chunk result is already used by the client but an error is reported processing next result chunk.
Project: ThreatMapper (Provisioning / Security & Compliance)
File: console.md

title: Management Console The ThreatMapper Management Console ("Console") is a standalone application, implemented as a fleet of containers. It should be deployed on either a single docker host, or (for larger deployments) a dedicated Kubernetes cluster. The console is self-contained, and exposes an HTTPS interface for administration and API automation. The console allows you to: Manage the users who can access the console. Configure Infrastructure API access and interrogate platform configurations. Visualize and drill down into Kubernetes clusters, virtual machines, containers and images, running processes, and network connections in near real time. Invoke vulnerability scans on running containers and applications and review the results, ranked by risk-of-exploit. Invoke compliance scans on infrastructure configuration ('agentless') and on infrastructure hosts ('agent-based'), manually or automatically when they are added to a cluster. Scan container registries for vulnerabilities, to review workloads before they are deployed. Scan image builds during the CI/CD pipeline, supporting CircleCI, Jenkins, and GitLab. Scan containers and host filesystems for unprotected secrets, including access tokens, keys and passwords. Configure integrations with external notification, SIEM and ticketing systems, including Slack, PagerDuty, Jira, Splunk, ELK, Sumo Logic, and AWS S3. ThreatMapper supports multiple production deployments simultaneously, so that you can visualize and scan workloads across a large production estate. ThreatMapper performs compliance posture scanning for cloud platforms by querying the infrastructure APIs for these platforms. This is achieved using a cloud scanner task that is deployed within each cloud instance using a terraform module. The cloud scanner is granted appropriate access to the local APIs, and operates under instruction from the remote ThreatMapper console. The ThreatMapper console can scan container images at rest in a wide range of supported registries. This is achieved by providing appropriate credentials to the ThreatMapper console so that it can discover and download assets directly from these registries. ThreatMapper performs vulnerability and secret scanning directly on production and non-production hosts using a sensor agent container. The sensor agent is also used for local compliance scanning (Kubernetes and Linux posture) where it has access to configuration and assets that are not exposed through an API.