cwielenb/CNCF-Dataset huggingface dataset:
Project: Meshery (Provisioning / Automation & Configuration)
File: authenticate-with-meshery-via-cli.md

layout: default title: Authenticating with Meshery via CLI permalink: guides/mesheryctl/authenticate-with-meshery-via-cli language: en type: guides category: mesheryctl list: include abstract: Get your authentication token from Meshery CLI. To authenticate with Meshery through `mesheryctl` you will use the command `mesheryctl system login`. Upon execution of this command, select your Provider of choice, then authenticate to your chosen Provider. You can retrieve your authentication token from either of Meshery's two clients: the CLI or the UI. Get your token through , from the `Get Token` option. Downloading the token <a href="{{ site.baseurl }}/assets/img/token/MesheryTokenUI.png"><img alt="Meshery Dashboard" src="{{ site.baseurl }}/assets/img/token/MesheryTokenUI.png" /></a> <br/> <br/> Get your token through Meshery CLI. <br/> To get the token through `mesheryctl` you would have to use the following command and the path to token for authenticating to Meshery API (default "auth.json"). <br/> <pre class="codeblock-pre"> <div class="codeblock"><div class="clipboardjs"> mesheryctl system login</div></div> </pre> <br /> _The need for authentication to `Meshery` is to save your environment setup while also having persistent/steady sessions and to be able to retrieve performance test results._ <br/> For an exhaustive list of `mesheryctl` commands and syntax: See . Guides to using Meshery's various features and components. {% include related-discussions.html tag="mesheryctl" %}
Project: KICS (Provisioning / Security & Compliance)
File: 7814ddda-e758-4a56-8be3-289a81ded929.md

title: Cloud Storage Bucket Versioning Disabled hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 7814ddda-e758-4a56-8be3-289a81ded929 Query name: Cloud Storage Bucket Versioning Disabled Platform: Ansible Severity: <span style="color:#ff7213">Medium</span> Category: Observability URL: Cloud Storage Bucket should have versioning enabled<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="17 3" name: create a bucket google.cloud.gcpstoragebucket: name: ansible-storage-module project: test_project auth_kind: serviceaccount serviceaccountfile: "/tmp/auth.pem" state: present name: create a second bucket google.cloud.gcpstoragebucket: name: ansible-storage-module project: test_project auth_kind: serviceaccount serviceaccountfile: "/tmp/auth.pem" state: present versioning: enabled: no ``` ```yaml title="Negative test num. 1 - yaml file" name: create a bucket google.cloud.gcpstoragebucket: name: ansible-storage-module project: test_project auth_kind: serviceaccount serviceaccountfile: /tmp/auth.pem state: present versioning: enabled: yes ```
Project: Goldilocks (Provisioning / Security & Compliance)
File: guide.md

meta: name: description content: "Issues are essential for keeping goldilocks great. There are a few guidelines that we need contributors to follow so that we can keep on top of things" Issues, whether bugs, tasks, or feature requests are essential for keeping goldilocks great. We believe it should be as easy as possible to contribute changes that get things working in your environment. There are a few guidelines that we need contributors to follow so that we can keep on top of things. This project adheres to a . Please review this document before contributing to this project. Before you can contribute, you will need to sign the . Goldilocks can run in 3 modes. There is a CLI that allows the manipulation of VPA objects, a dashboard, and a controller that runs in-cluster and manages VPA objects based on namespace labels. The CLI uses the cobra package and the commands are in the `cmd` folder. We label issues with the if we believe they'll be a good starting point for new contributors. If you're interested in working on an issue, please start a conversation on that issue, and we can help answer any questions as they come up. This repo contains a pre-commit file for use with . Just run `pre-commit install` and you will have the hooks. Make sure you have the following installed: or higher or higher or higher git kubectl Go into the directory and run `./setup.sh` This will create a kind cluster, place a demo app, install VPA, and install the latest goldilocks. You can run your local development against this cluster. Prerequisites: A properly configured Golang environment with Go 1.11 or higher If you want to see the local changes you make on a dashboard, you will need access to a Kubernetes cluster defined in `~/.kube/config` or the KUBECONFIG variable. The will need to be installed in the cluster. Install the project with `go get github.com/fairwindsops/goldilocks` Change into the goldilocks directory which is installed at `$GOPATH/src/github.com/fairwindsops/goldilocks` Use `make tidy` or `make build` to ensure all dependencies are downloaded. See the dashboard with `go run main.go dashboard`, then open http://localhost:8080/. This assumes that you have a working KUBECONFIG in place with access to a cluster. The e2e tests run using . You can run them yourself by: installing venom setting up a kind cluster `kind create cluster` running `make e2e-test`. The tests are also run automatically by CI You can add tests in the directory. See the Venom README for more info. If you've encountered an issue that is not already reported, please create an issue that contains the following: Clear description of the issue Steps to reproduce it Appropriate labels Each new pull request should: Reference any related issues Add tests that show the issues have been solved Pass existing tests and linting Contain a clear indication of if they're ready for review or a work in progress Be up to date and/or rebased on the master branch Push a new annotated tag. This tag should contain a changelog of pertinent changes.
Project: KubeArmor (Provisioning / Security & Compliance)
File: ens-tls.md

For k8s, the solution gets deployed as a job that scans the k8s service ports. Clone the GitHub repo link: https://github.com/kubearmor/k8tls ```sh Git clone https://github.com/kubearmor/k8tls.git ``` ```sh kubectl apply -f https://raw.githubusercontent.com/kubearmor/k8tls/main/k8s/job.yaml kubectl logs -n k8tls $(kubectl get pod -n k8tls -l job-name=k8tls -o name) -f ```
Project: ClickHouse (App Definition and Development / Database)
File: wikistat.md

sidebar_label: WikiStat The dataset contains 0.5 trillion records. See the video from FOSDEM 2023: https://www.youtube.com/watch?v=JlcI2Vfz_uk And the presentation: https://presentations.clickhouse.com/fosdem2023/ Data source: https://dumps.wikimedia.org/other/pageviews/ Getting the list of links: ``` shell for i in {2015..2023}; do for j in {01..12}; do echo "${i}-${j}" >&2 curl -sSL "https://dumps.wikimedia.org/other/pageviews/$i/$i-$j/" \ | grep -oE 'pageviews-[0-9]+-[0-9]+\.gz' done done | sort | uniq | tee links.txt ``` Downloading the data: ``` shell sed -r 's!pageviews-([0-9]{4})([0-9]{2})[0-9]{2}-[0-9]+\.gz!https://dumps.wikimedia.org/other/pageviews/\1/\1-\2/\0!' \ links.txt | xargs -P3 wget --continue ``` (it will take about 3 days) Creating a table: ``` sql CREATE TABLE wikistat ( time DateTime CODEC(Delta, ZSTD(3)), project LowCardinality(String), subproject LowCardinality(String), path String CODEC(ZSTD(3)), hits UInt64 CODEC(ZSTD(3)) ) ENGINE = MergeTree ORDER BY (path, time); ``` Loading the data: ``` shell clickhouse-local --query " WITH replaceRegexpOne(_path, '^.+pageviews-(\\d{4})(\\d{2})(\\d{2})-(\\d{2})(\\d{2})(\\d{2}).gz$', '\1-\2-\3 \4-\5-\6')::DateTime AS time, extractGroups(line, '^([^ \\.]+)(\\.[^ ]+)? +([^ ]+) +(\\d+) +(\\d+)$') AS values SELECT time, values[1] AS project, values[2] AS subproject, values[3] AS path, (values[4])::UInt64 AS hits FROM file('pageviews*.gz', LineAsString) WHERE length(values) = 5 FORMAT Native " | clickhouse-client --query "INSERT INTO wikistat FORMAT Native" ``` Or loading the cleaning data: ``` sql INSERT INTO wikistat WITH parseDateTimeBestEffort(extract(_file, '^pageviews-([\\d\\-]+)\\.gz$')) AS time, splitByChar(' ', line) AS values, splitByChar('.', values[1]) AS projects SELECT time, projects[1] AS project, projects[2] AS subproject, decodeURLComponent(values[2]) AS path, CAST(values[3], 'UInt64') AS hits FROM s3( 'https://clickhouse-public-datasets.s3.amazonaws.com/wikistat/original/pageviews*.gz', LineAsString) WHERE length(values) >= 3 ```
Project: KubeVela (App Definition and Development / Application Definition & Image Build)
File: container-ports.eg.md

It's used to define Pod networks directly. hostPort routes the container's port directly to the port on the scheduled node, so that you can access the Pod through the host's IP plus hostPort. Don't specify a hostPort for a Pod unless it is absolutely necessary(run `DaemonSet` service). When you bind a Pod to a hostPort, it limits the number of places the Pod can be scheduled, because each <hostIP, hostPort, protocol> combination must be unique. If you don't specify the hostIP and protocol explicitly, Kubernetes will use 0.0.0.0 as the default hostIP and TCP as the default protocol. If you explicitly need to expose a Pod's port on the node, consider using `expose` or `gateway` trait, or exposeType and ports parameter of `webservice` component before resorting to `container-ports` trait. ```yaml apiVersion: core.oam.dev/v1beta1 kind: Application metadata: name: busybox spec: components: name: busybox type: webservice properties: cpu: "0.5" exposeType: ClusterIP image: busybox memory: 1024Mi ports: expose: false port: 80 protocol: TCP expose: false port: 801 protocol: TCP traits: type: container-ports properties: containers: containerName: busybox ports: containerPort: 80 protocol: TCP hostPort: 8080 ```
Project: Apache Hadoop (App Definition and Development / Database)
File: RELEASENOTES.3.3.6.md

<! --> These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. | Minor | Enhance WritableName to be able to return aliases for classes that use serializers* If you have a SequenceFile with an old key or value class which has been renamed, you can use WritableName.addName to add an alias class. This functionality previously existed, but only worked for classes which extend Writable. It now works for any class, notably key or value classes which use io.serializations.
Project: kube-hunter (Provisioning / Security & Compliance)
File: support_question.md

name: Support Question about: I have a question and require assistance labels: question <!-- If you have some trouble, feel free to ask. Make sure you're not asking duplicate question by searching on the issues lists. --> <!-- Explain the problem you are experiencing. --> <!-- If it is possible, create a minimal example of your work that showcases the problem you are having. -->
Project: Podman (App Definition and Development / Application Definition & Image Build)
File: podman-image-diff.1.md

% podman-image-diff 1 podman-image-diff - Inspect changes on an image's filesystem podman image diff [options] image [image] Displays changes on an image's filesystem. The image is compared to its parent layer or the second argument when given. The output is prefixed with the following symbols: | Symbol | Description | |--|-| | A | A file or directory was added. | | D | A file or directory was deleted. | | C | A file or directory was changed. | Alter the output into a different format. The only valid format for podman image diff is `json`. Display image differences from images parent layer: ``` $ podman image diff redis:old C /usr C /usr/local C /usr/local/bin A /usr/local/bin/docker-entrypoint.sh ``` Display image differences between two different images in JSON format: ``` $ podman image diff --format json redis:old redis:alpine { "changed": [ "/usr", "/usr/local", "/usr/local/bin" ], "added": [ "/usr/local/bin/docker-entrypoint.sh" ] } ``` , August 2017, Originally compiled by Ryan Cole <rycole@redhat.com>
Project: ShardingSphere (App Definition and Development / Database)
File: create-default-sharding-strategy.en.md

+++ title = "CREATE DEFAULT SHARDING STRATEGY" weight = 4 +++ The `CREATE DEFAULT SHARDING STRATEGY` syntax is used to create a default sharding strategy {{< tabs >}} {{% tab name="Grammar" %}} ```sql CreateDefaultShardingStrategy ::= 'CREATE' 'DEFAULT' 'SHARDING' ('DATABASE' | 'TABLE') 'STRATEGY' ifNotExists? '(' shardingStrategy ')' ifNotExists ::= 'IF' 'NOT' 'EXISTS' shardingStrategy ::= 'TYPE' '=' strategyType ',' ('SHARDINGCOLUMN' '=' columnName | 'SHARDINGCOLUMNS' '=' columnNames) ',' 'SHARDING_ALGORITHM' '=' algorithmDefinition strategyType ::= string algorithmDefinition ::= 'TYPE' '(' 'NAME' '=' algorithmType ',' propertiesDefinition ')' columnNames ::= columnName (',' columnName)+ columnName ::= identifier algorithmType ::= string propertiesDefinition ::= 'PROPERTIES' '(' key '=' value (',' key '=' value)* ')' key ::= string value ::= literal ``` {{% /tab %}} {{% tab name="Railroad diagram" %}} <iframe frameborder="0" name="diagram" id="diagram" width="100%" height="100%"></iframe> {{% /tab %}} {{< /tabs >}} When using the complex sharding algorithm, multiple sharding columns need to be specified using `SHARDING_COLUMNS`; `algorithmType` is the sharding algorithm type. For detailed sharding algorithm type information, please refer to ; `ifNotExists` clause is used for avoid `Duplicate default sharding strategy` error. create a default sharding table strategy ```sql -- create a default sharding table strategy CREATE DEFAULT SHARDING TABLE STRATEGY ( TYPE="standard", SHARDINGCOLUMN=userid, SHARDINGALGORITHM(TYPE(NAME="inline", PROPERTIES("algorithm-expression"="torder${userid % 2}"))) ); ``` create a default sharding table strategy with `ifNotExists` clause ```sql CREATE DEFAULT SHARDING TABLE STRATEGY IF NOT EXISTS ( TYPE="standard", SHARDINGCOLUMN=userid, SHARDINGALGORITHM(TYPE(NAME="inline", PROPERTIES("algorithm-expression"="torder${userid % 2}"))) ); ``` `CREATE`, `DEFAULT`, `SHARDING`, `DATABASE`, `TABLE`, `STRATEGY`, `TYPE`, `SHARDINGCOLUMN`, `SHARDINGCOLUMNS`, `SHARDING_ALGORITHM`, `NAME`, `PROPERTIES`
Project: kpt (Provisioning / Automation & Configuration)
File: 02-limiting-package-changes.md

Id like to limit what my package consumers can do with my package and it feels safer to just provide a string replacement in one place so they know not to alter the configuration outside of the few places that I designated as OK places to change. Example deployment: ```yaml apiVersion: apps/v1 kind: Deployment metadata: # kpt-merge: /nginx-deploy name: nginx-deploy spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: name: backend image: nginx:1.16.1 # kpt-set: nginx:${tag} ``` kpt configuration that uses a setter: ```yaml apiVersion: kpt.dev/v1 kind: Kptfile metadata: name: dont-change-much pipeline: mutators: image: gcr.io/kpt-fn/apply-setters:v0.2.0 configMap: tag: 1.21 ``` The limitation by parameters does not guarantee that consumers are in fact going to limit their changes to the parameters. A popular pattern is using kustomize to change output of other tools no matter what parameters had. In this particular case I am able to fork or patch this package and add: ```yaml securityContext: runAsNonRoot: false ``` String replacements rarely describe the intent of the package author. When additional resources are added I need additional places where parameters need to be applied. I can easily add other containers to this deployment and the package author's rules are not clear and not easily validated. General ways to describe policy already exist. kpt has a [gatekeeper] function that allows the author to describe intended limitations for a class of resources or the entire package giving the consumer the freedom to customize and get an error or a warning when the policy is violated. In the sample provided by the function we see how to provide a policy that will clearly describe the intent using rego: ```yaml apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: # kpt-merge: /disallowroot name: disallowroot spec: crd: spec: names: kind: DisallowRoot targets: target: admission.k8s.gatekeeper.sh rego: |- package disallowroot violation[{"msg": msg}] { not input.review.object.spec.template.spec.securityContext.runAsNonRoot msg := "Containers must not run as root" } apiVersion: constraints.gatekeeper.sh/v1beta1 kind: DisallowRoot metadata: # kpt-merge: /disallowroot name: disallowroot spec: match: kinds: apiGroups: 'apps' kinds: Deployment ``` The Kptfile can enforce that resources comply with this policy every time `kpt fn render` is used: ```yaml apiVersion: kpt.dev/v1 kind: Kptfile metadata: name: gatekeeper-disallow-root-user pipeline: validators: image: gcr.io/kpt-fn/gatekeeper:v0.2 ```
Project: Pravega (App Definition and Development / Streaming & Messaging)
File: lts-recovery-steps.md

<!-- Copyright Pravega Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> The following procedure describes offline method of doing the LTS recovery. The offline method of recovering Pravega from the data in LTS requires the admin to manually copy metadata chunk files to some directory. The offline method currently only supports filesystem storage type. The major inspiration for this recovery procedure has been [#7011] (https://github.com/pravega/pravega/issues/7011). The way this recovery procedure works is by trying to restore the metadata state(state of all Segments in Pravega) stored in Tier-2 Segment Chunks, by parsing these raw Chunk files. This recovery procedure assumes full Tier-1 loss, and as such the Tier-1 tail data or the latest updates to the system as lost. The aim of this recovery procedure is to get Pravega functional again after performing the recovery procedure, to allow clients to resume their operations. Some cases where the recovery procedure might be useful are ZooKeeper becoming dysfunctional,Bookkeeper disk corruptions etc. In these cases, we intend to start Pravega off the data that exists in Tier-2. The following section describes how to perform the recovery steps. Having pravega installed with data generated on it. ## Executing the ltsrecovery.sh script ``` ./ltsrecovery --install ``` This script will create a recovery pod and has the LTS directory mounted inside it, and it will uninstall pravega, pravega-operator, bookkeeper, bookkeeper-operator, zookeeper and zookeeper-operator. Before proceeding further, make sure the script executed successfully and uninstalled all the services mentioned above. This recovery script will only work where `helm` is used to install Pravega. ## Copy metadata ``` kubectl exec -it recovery -- sh ``` `/mnt/tier2` is the tier2 location. ``` cd /mnt/tier2/_system/containers ``` ``` cp metadata* storagemetadata_* /<Location where the metadata will be downloaded> ``` ## Install the below services After downloading metadata we need to install zookeeper, zookeeper-operator, bookkeeper and bookkeeper-operator. ## Perform Recovery ``` kubectl exec -it recovery -- sh ``` ``` cd /opt/pravega ``` ``` ./bin/pravega-admin ``` Make sure that the admin cli is configured properly before starting the recovery. To start the recovery, please execute the below command in the admin cli console. ``` data-recovery recover-from-storage /<Location of the downloaded metadata> all ``` ## Install pravega Once recovery is performed we can install pravega-operator and pravega and start using it.
Project: KICS (Provisioning / Security & Compliance)
File: f62aa827-4ade-4dc4-89e4-1433d384a368.md

title: IAM Policy Grants Full Permissions hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: f62aa827-4ade-4dc4-89e4-1433d384a368 Query name: IAM Policy Grants Full Permissions Platform: CloudFormation Severity: <span style="color:#bb2124">High</span> Category: Access Control URL: IAM policy should not grant full permissions to resources from the get-go, instead of granting permissions gradually as necessary.<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="8 21" AWSTemplateFormatVersion: "2010-09-09" Description: A sample template Resources: mypolicy: Type: AWS::IAM::Policy Properties: PolicyName: mygrouppolicy PolicyDocument: Version: '2012-10-17' Statement: Effect: Allow Action: ["*"] Resource: "*" Groups: myexistinggroup1 !Ref mygroup mypolicy2: Type: AWS::IAM::Policy Properties: PolicyName: mygrouppolicy PolicyDocument: Version: '2012-10-17' Statement: Effect: Allow Action: "*" Resource: "*" Groups: myexistinggroup1 !Ref mygroup ``` ```json title="Positive test num. 2 - json file" hl_lines="9 29" { "AWSTemplateFormatVersion": "2010-09-09", "Description": "A sample template", "Resources": { "mypolicy2": { "Type": "AWS::IAM::Policy", "Properties": { "PolicyName": "mygrouppolicy", "PolicyDocument": { "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ], "Version": "2012-10-17" }, "Groups": [ "myexistinggroup1", "mygroup" ] } }, "mypolicy": { "Type": "AWS::IAM::Policy", "Properties": { "PolicyName": "mygrouppolicy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "*" ], "Resource": "*" } ] }, "Groups": [ "myexistinggroup1", "mygroup" ] } } } } ``` ```yaml title="Negative test num. 1 - yaml file" AWSTemplateFormatVersion: "2010-09-09" Description: A sample template Resources: adminPolicy: Type: AWS::IAM::Policy Properties: PolicyName: mygrouppolicy PolicyDocument: Version: '2012-10-17' Statement: Effect: Allow Action: ["*"] Resource: arn:aws:iam::aws:policy/AdministratorAccess Groups: myexistinggroup1 !Ref mygroup ``` ```json title="Negative test num. 2 - json file" { "AWSTemplateFormatVersion": "2010-09-09", "Description": "A sample template", "Resources": { "adminPolicy": { "Type": "AWS::IAM::Policy", "Properties": { "PolicyName": "mygrouppolicy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Resource": "arn:aws:iam::aws:policy/AdministratorAccess", "Effect": "Allow", "Action": [ "*" ] } ] }, "Groups": [ "myexistinggroup1", "mygroup" ] } } } } ``` ```yaml title="Negative test num. 3 - yaml file" AWSTemplateFormatVersion: "2010-09-09" Description: A sample template Resources: adminPolicy: Type: AWS::IAM::Policy Properties: PolicyName: mygrouppolicy PolicyDocument: Version: '2012-10-17' Statement: Effect: Allow Action: 'ec2messages:GetEndpoint' Resource: ['*'] Groups: myexistinggroup1 !Ref mygroup ```
Project: Chef InSpec (Provisioning / Security & Compliance)
File: google_iam_custom_roles.md

+++ title = "googleiamcustom_roles resource" draft = false gh_repo = "inspec" platform = "gcp" [menu] [menu.inspec] title = "googleiamcustom_roles" identifier = "inspec/resources/gcp/googleiamcustomroles.md googleiamcustomroles resource" parent = "inspec/resources/gcp" +++ A `googleiamcustom_roles` is used to test a Google CustomRole resource ```ruby describe googleiamcustom_roles(project: 'chef-gcp-inspec') do its('names') { should include "projects/project-id/roles/role-id" } end ``` Properties that can be accessed from the `googleiamcustom_roles` resource: See the resource for more information. `names` : an array of `googleiamcustom_role` name `titles` : an array of `googleiamcustom_role` title `descriptions` : an array of `googleiamcustom_role` description `included_permissions` : an array of `googleiamcustomrole` includedpermissions `stages` : an array of `googleiamcustom_role` stage `deleteds` : an array of `googleiamcustom_role` deleted This resource supports all of the above properties as filter criteria, which can be used with `where` as a block or a method. Ensure the is enabled for the current project.