cwielenb/CNCF-Dataset huggingface dataset:
Project: StarRocks (App Definition and Development / Database)
File: sharedDataIntro.md

This topic describes how to deploy and use a shared-data StarRocks cluster. This feature is supported from v3.0 for S3 compatible storage and v3.1 for Azure Blob Storage. NOTE StarRocks version 3.1 brings some changes to the shared-data deployment and configuration. Please use this document if you are running version 3.1 or higher. If you are running version 3.0 please use the . Shared-data StarRocks clusters do not support data BACKUP and RESTORE. The shared-data StarRocks cluster is specifically engineered for the cloud on the premise of separation of storage and compute. It allows data to be stored in object storage (for example, AWS S3, Google GCS, Azure Blob Storage, and MinIO). You can achieve not only cheaper storage and better resource isolation, but elastic scalability for your cluster. The query performance of the shared-data StarRocks cluster aligns with that of a shared-nothing StarRocks cluster when the local disk cache is hit. In version 3.1 and higher the StarRocks shared-data cluster is made up of Frontend Engines (FEs) and Compute Nodes (CNs). The CNs replace the classic Backend Engines (BEs) in shared-data clusters. Compared to the classic shared-nothing StarRocks architecture, separation of storage and compute offers a wide range of benefits. By decoupling these components, StarRocks provides: Inexpensive and seamlessly scalable storage. Elastic scalable compute. Because data is not stored in Compute Nodes (CNs), scaling can be done without data migration or shuffling across nodes. Local disk cache for hot data to boost query performance. Asynchronous data ingestion into object storage, allowing a significant improvement in loading performance.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: setup.md

<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> To run a code cell, you can click the Run cell button at the top left of the cell, or select it and press `Shift+Enter`. Try modifying a code cell and re-running it to see what happens. To learn more about Colab, see . {:.language-py} First, let's install the `apache-beam` module. {:.language-py} ```sh pip install --quiet -U apache-beam ```
Project: Open Application Model (App Definition and Development / Application Definition & Image Build)
File: 9.design_principles.md

The Open Application Model follows a set of design principles that ensure the clarity, richness, and extensibility of the model. Separation of Concerns is the design philosophy in which architectural choices are made with reference to the discrete problems being solved. Divisions in artifacts like "component" and "application", or "schematics" and "configuration" are made by dividing along functional or behavioral lines. By identifying roles and responsibilities of various groups of users, the spec is divided into concepts that match the problem space. The Open Application Model is runtime-agnostic. It does not assume any runtime-specific features. Instead, it aims to provide a common vocabulary for application owners and operators to describe desired topologies and behaviors independent from any specific platforms. While ensuring separation of concerns, OAM seeks to avoid unnecessary complexity to users who take on multiple roles in smaller teams. Simple scenarios should be achievable with minimal investment of time and energy, but complex scenarios should be accommodated without requiring re-platforming. OAM offers multiple abstraction layers so that operational concerns can be captured independently of developer concerns. Components in the OAM schematics are designed to be re-usable and shareable. In addition, they remain independent of the code that they describe, making it possible to re-use the code (containers), and prevent "lock-in" conditions. The model as a whole is designed to provide application "distributions" where the same application can be executed on different platforms without alteration. This portability of the application is intended to make the following scenarios not just possible, but easy: Transferring an application from a developer workstation to a production cluster or service Migrating from one implementation to another with no code changes Creating marketplace-like environments where applications can be deployed onto customer platforms There is a clear distinction between an application model and a programming model. An application model describes the composition of an application and the topology of its components. It is not concerned with how each component is implemented (language, design patterns, etc). On the other hand, a programming model describes how an individual piece of software is composed. It is used by a developer to implement application components. The Open Application Model offers an application model that does not have any requirements of a programming model. | Previous | | - | | |
Project: Kui (App Definition and Development / Application Definition & Image Build)
File: components.md

Kui divides theming into three aspects: , such as breadcrumbs and tables. , such as a trash can icon. of the component UI. Kui strives to make these three orthogonal choices. Kui employs a combination of home-grown React components and components from third-party open source component frameworks. A selection of the latter components are currently hosted by an SPI, with support for and (partially) . The remaining third-party components in use are not yet SPI'd. Kui uses a to allow for a choice of component provider. Currently, Carbon Component Library is the default choice. To inspect or modify this default, consult . To use Patternfly 4 Library in your client, either modify the default in your fork, or add `components="patternfly"` to your use of the `<Kui/>` client component. For example, the following code will set Kui default client to use Patternfly 4 Library. ```typescript import { Kui, KuiProps } from '@kui-shell/plugin-client-common' export default function MyKui(props: KuiProps) { return ( <Kui productName={productName} components="patternfly" {...props}> {children} </Kui> ) } ``` In this table `carbon` stands for Carbon Components, and `pf4` stands for PatternFly v4. | Component | Model | Implemenatations | | - | -- | | | Breadcrumb | | \| | | Navigation | | \| | The following is a comprehensive list of the third party components that are currently hard-wired to use Carbon Components: Kui has an SPI for icons. You may add new icons or add support for a new icon library by extending either the interface or the implementations. The SPI definition is . Under the subdirectory, you will find the current implementations. The SPI delegate currently ties the choice of icon provider to the choice of component provider. Coming soon.
Project: TDengine (App Definition and Development / Database)
File: 08-tcollector.md

title: TCollector writing sidebar_label: TCollector description: This document describes how to integrate TDengine with TCollector. import TCollector from "../14-reference/_tcollector.mdx" TCollector is part of openTSDB and collects client computer's logs to send to the database. You can write the data collected by TCollector to TDengine by simply changing the configuration of TCollector to point to the domain name (or IP address) and corresponding port of the server running taosAdapter. It can take full advantage of TDengine's efficient storage query performance and clustering capability for time-series data. To write data to the TDengine via TCollector requires the following preparations. The TDengine cluster has been deployed and is working properly taosAdapter is installed and running properly. Please refer to the for details. TCollector has been installed. Please refer to for TCollector installation <TCollector /> Restart taosAdapter: ``` sudo systemctl restart taosadapter ``` Run `sudo ./tcollector.py`: Wait for a few seconds and then use the TDengine CLI to query whether the corresponding database has been created and data are written in TDengine. ``` taos> show databases; name | ================================= information_schema | performance_schema | tcollector | Query OK, 3 rows in database (0.001647s) taos> use tcollector; Database changed. taos> show stables; name | ================================= proc.meminfo.hugepages_rsvd | proc.meminfo.directmap1g | proc.meminfo.vmallocchunk | proc.meminfo.hugepagesize | tcollector.reader.lines_dro... | proc.meminfo.sunreclaim | proc.stat.ctxt | proc.meminfo.swaptotal | proc.uptime.total | tcollector.collector.lines_... | proc.meminfo.vmallocused | proc.meminfo.memavailable | sys.numa.foreign_allocs | proc.meminfo.committed_as | proc.vmstat.pswpin | proc.meminfo.cmafree | proc.meminfo.mapped | proc.vmstat.pgmajfault | ... ``` :::note TDengine will automatically create unique IDs for sub-table names by the rule. :::
Project: KICS (Provisioning / Security & Compliance)
File: 6c8d51af-218d-4bfb-94a9-94eabaa0703a.md

title: S3 Bucket Without Ignore Public ACL hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 6c8d51af-218d-4bfb-94a9-94eabaa0703a Query name: S3 Bucket Without Ignore Public ACL Platform: CloudFormation Severity: <span style="color:#ff7213">Medium</span> Category: Insecure Configurations URL: S3 bucket without ignore public ACL<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="10 4 21" Resources: Bucket11: Type: AWS::S3::Bucket Properties: Resources: Bucket12: Type: AWS::S3::Bucket Properties: PublicAccessBlockConfiguration: BlockPublicPolicy : true RestrictPublicBuckets : true Resources: Bucket13: Type: AWS::S3::Bucket Properties: PublicAccessBlockConfiguration: BlockPublicAcls: false BlockPublicPolicy : true IgnorePublicAcls : false RestrictPublicBuckets : true ``` ```json title="Positive test num. 2 - json file" hl_lines="9" { "Resources": { "Bucket1": { "Type": "AWS::S3::Bucket", "Properties": { "PublicAccessBlockConfiguration": { "BlockPublicAcls": false, "BlockPublicPolicy": true, "IgnorePublicAcls": false, "RestrictPublicBuckets": true }, "AccessControl": "Private" } } } } ``` ```yaml title="Negative test num. 1 - yaml file" Resources: Bucket1: Type: AWS::S3::Bucket Properties: PublicAccessBlockConfiguration: BlockPublicAcls : true BlockPublicPolicy : true IgnorePublicAcls : true RestrictPublicBuckets : true ``` ```json title="Negative test num. 2 - json file" { "Resources": { "Bucket1": { "Type": "AWS::S3::Bucket", "Properties": { "PublicAccessBlockConfiguration": { "BlockPublicAcls": true, "BlockPublicPolicy": true, "IgnorePublicAcls": true, "RestrictPublicBuckets": true }, "AccessControl": "Private" } } } } ```
Project: EMQ Technologies (App Definition and Development / Streaming & Messaging)
File: fix-12773.en.md

Upgrade HTTP client libraries. The HTTP client library (`gun-1.3`) incorrectly appends a `:portnumber` suffix to the `Host` header for standard ports (`http` on port 80, `https` on port 443). This could cause compatibility issues with servers or gateways performing strict `Host` header checks (e.g., AWS Lambda, Alibaba Cloud HTTP gateways), leading to errors such as `InvalidCustomDomain.NotFound` or "The specified CustomDomain does not exist."
Project: KICS (Provisioning / Security & Compliance)
File: 61d1a2d0-4db8-405a-913d-5d2ce49dff6f.md

title: Instance With No VPC hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 61d1a2d0-4db8-405a-913d-5d2ce49dff6f Query name: Instance With No VPC Platform: Ansible Severity: <span style="color:#edd57e">Low</span> Category: Insecure Configurations URL: EC2 Instances should be configured under a VPC network. AWS VPCs provide the controls to facilitate a formal process for approving and testing all network connections and changes to the firewall and router configurations.<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="2 18" name: Start an instance and have it begin a Tower callback on boot community.aws.ec2_instance: name: "tower-callback-test" key_name: "prod-ssh-key" security_group: default tower_callback: tower_address: 1.2.3.4 jobtemplateid: 876 hostconfigkey: '[secret config key goes here]' network: assignpublicip: true image_id: ami-123456 cpucreditspecification: unlimited tags: SomeThing: "A value" name: Start an instance and have it begin a Tower callback on boot v2 amazon.aws.ec2: key_name: mykey instance_type: t2.micro image: ami-123456 wait: yes group: webserver count: 3 assignpublicip: yes ``` ```yaml title="Negative test num. 1 - yaml file" name: Start an instance and have it begin a Tower callback on boot v3 community.aws.ec2_instance: name: tower-callback-test key_name: prod-ssh-key vpcsubnetid: subnet-5ca1ab1e security_group: default tower_callback: tower_address: 1.2.3.4 jobtemplateid: 876 hostconfigkey: '[secret config key goes here]' network: assignpublicip: true image_id: ami-123456 cpucreditspecification: unlimited tags: SomeThing: A value name: Start an instance and have it begin a Tower callback on boot v4 amazon.aws.ec2: key_name: mykey instance_type: t2.micro image: ami-123456 wait: yes group: webserver count: 3 vpcsubnetid: subnet-29e63245 assignpublicip: yes ```
Project: Apache Spark (App Definition and Development / Streaming & Messaging)
File: ml-decision-tree.md

layout: global title: Decision trees displayTitle: Decision trees license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. > This section has been moved into the .
Project: ArangoDB (App Definition and Development / Database)
File: definitions.md

<!-- Copyright 2018 Paul Fultz II Distributed under the Boost Software License, Version 1.0. (http://www.boost.org/LICENSE10.txt) --> Definitions =========== Function Adaptor A takes a function(or functions) and returns a new function with enhanced capability. Each adaptor has a functional form with a corresponding class with `_adaptor` appended to it: ```cpp template<class... Fs> FunctionAdaptor_adaptor<Fs...> FunctionAdaptor(Fs...); ``` Both the functional form and the class form can be used to construct the adaptor. Static Function Adaptor -- A static function adaptor is a that doesn't have a functional form. It is only a class. It has an additional requirement that the function is : ```cpp template<class... Fs> class StaticFunctionAdaptor; ``` Decorator A decorator is a function that returns a . The may be an unspecified or private type. ```cpp template<class... Ts> FunctionAdaptor Decorator(Ts...); ``` Semantics Some parts of the documentation provides the meaning(or equivalence) of an expression. Here is a guide of those symbols: `f`, `g`, `fs`, `gs`, `p` are functions `x`, `y`, `xs`, `ys` are parameters to a function `T` represents some type `...` are parameter packs and represent varidiac parameters Signatures All the functions are global function objects except where an explicit template parameter is required on older compilers. However, the documentation still shows the traditional signature since it is much clearer. So instead of writing this: ```cpp struct if_f { template<class IntegralConstant> constexpr auto operator()(IntegralConstant) const; }; const constexpr iff if = {}; ``` The direct function signature is written even though it is actually declared like above: ```cpp template<class IntegralConstant> constexpr auto if_(IntegralConstant); ``` Its usage is the same except it has the extra benefit that the function can be directly passed to another function.
Project: KubeVirt (App Definition and Development / Application Definition & Image Build)
File: freeze.md

Kubevirt provides a way to freeze/thaw the guest filesystem. That can be used to ensure the filesystem consistency when making a backup or a VM snapshot. The freeze requires a running qemu-guest-agent in the guest VM. Note: To check if your vm has a qemu-guest-agent check for 'AgentConnected' in the vm status. VirtualMachineInstance API implements a subresource for a freeze and unfreeze commands. When called it sends respective freeze/unfreeze commands to the virt-launcher. The freeze/unfreeze can be accessed using HTTP PUT command on a vmi REST API or by a Kubevirt client-go. API subresource is located under a `virtualmachineinstances` resource (example uri for VM `example-vm` in namespace `demo` is `apis/subresources.kubevirt.io/v1alpha3/namespaces/demo/virtualmachineinstances/example-vm/freeze`). It can also be access through kubevirt/client-go. It is available on VirtualMachineInstanceInterface defined in `kubevirt.io/client-go/kubecli/kubevirt.go`. ``` virtClient.VirtualMachineInstance(namespace).Freeze(vmiName, unfreezeTimeout) virtClient.VirtualMachineInstance(namespace).Unfreeze(vmiName) ``` Freeze command has additional parameter - the unfreezeTimeout - the timeout when an automatic unfreeze command is executed in the virt launcher to prevent the vmi from being stuck on freeze. It can be set to 0 to disable automatic unfreeze, but that is not recommended. The freeze/unfreeze subresources are internally used by the VM VirtualMachineSnapshot API (https://kubevirt.io/user-guide/operations/snapshotrestoreapi/). To integrate with kubevirt agnostic tools that are not using kubevirt REST APIs a special virt-freezer application is available in the `compute` container. Virt-freezer binary is located in `/usr/bin/virt-freezer`. Freeze the `example-vm` in the namespace `demo`: ```bash kubectl exec virt-launcher-example-vm-v8vxz -n demo -- /usr/bin/virt-freezer --freeze --namespace demo --name example-vm ``` The freeze command has an unfreeze timeout set to 5 minutes, that can be changed with the use of the `unfreezeTimeoutSeconds` flag. Example with unfreeze timeout set to one minute: ```bash kubectl exec virt-launcher-example-vm-v8vxz -n demo -- /usr/bin/virt-freezer --freeze --namespace demo --name example-vm --unfreezeTimeoutSeconds 60 ``` Check if the freeze is effective. This can be done by checking the value of the Fs Freeze Status. ```bash kubectl describe vmi example-vm -n demo | grep Freeze Fs Freeze Status: frozen ``` Unfreeze the `example-vm` in the namespace `demo` and verify that the filesystem is no longer frozen (the freeze status is empty or missing). ```bash kubectl exec virt-launcher-example-vm-v8vxz -n demo -- /usr/bin/virt-freezer --unfreeze --namespace demo --name example-vm kubectl describe vmi example-vm -n demo | grep Freeze ```
Project: Meshery (Provisioning / Automation & Configuration)
File: aws-elasticsearchservice-controller.md

layout: integration title: AWS ElasticSearch Service subtitle: Collaborative and visual infrastructure as code for AWS ElasticSearch Service image: /assets/img/integrations/aws-elasticsearchservice-controller/icons/color/aws-elasticsearchservice-controller-color.svg permalink: extensibility/integrations/aws-elasticsearchservice-controller docURL: https://docs.meshery.io/extensibility/integrations/aws-elasticsearchservice-controller description: integrations-category: Observability and Analysis integrations-subcategory: Logging registrant: GitHub components: name: elasticsearch-domain colorIcon: assets/img/integrations/aws-elasticsearchservice-controller/components/elasticsearch-domain/icons/color/elasticsearch-domain-color.svg whiteIcon: assets/img/integrations/aws-elasticsearchservice-controller/components/elasticsearch-domain/icons/white/elasticsearch-domain-white.svg description: name: adopted-resource colorIcon: assets/img/integrations/aws-elasticsearchservice-controller/components/adopted-resource/icons/color/adopted-resource-color.svg whiteIcon: assets/img/integrations/aws-elasticsearchservice-controller/components/adopted-resource/icons/white/adopted-resource-white.svg description: featureList: [ "Drag-n-drop cloud native infrastructure designer to configure, model, and deploy your workloads.", "Invite anyone to review and make changes to your private designs.", "Ongoing synchronization of Kubernetes configuration and changes across any number of clusters." ] howItWorks: "Collaborative Infrastructure as Code" howItWorksDetails: "Collaboratively manage infrastructure with your coworkers synchronously sharing the same designs." language: en list: include type: extensibility category: integrations
Project: Beam (App Definition and Development / Streaming & Messaging)
File: goga.md

title: "Beam and Geocoding" icon: /images/logos/powered-by/goga.png hasNav: true cardDescription: "GOGA Data Analysis and Consulting is a company based in Japan that specializes in analytics of geospatial and mapping data. They use Apache Beam and Cloud Dataflow for a smooth data transformation process for analytical purposes. This use case focuses on handling multiple extractions, geocoding, and insertion process by wrangling and requesting API call of each data based on the location provided." <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div> <header class="case-study-header"> <h2 itemprop="name headline">Beam and Geocoding</h2> </header> GOGA Data Analysis and Consulting is a company based in Japan that specializes in analytics of geospatial and mapping data. They use Apache Beam and Cloud Dataflow for a smooth data transformation process for analytical purposes. This use case focuses on handling multiple extractions, geocoding, and insertion process by wrangling and requesting API call of each data based on the location provided. </div>