cwielenb/CNCF-Dataset huggingface dataset:
Project: StarRocks (App Definition and Development / Database)
File: SHOW_RESOURCE_GROUP.md

displayed_sidebar: "English" Shows the information of resource groups. :::tip This operation does not require privileges. ::: ```SQL SHOW { RESOURCE GROUPS [ALL] | RESOURCE GROUP <resourcegroupname> } ``` | Parameter | Description | | - | | | RESOURCE GROUPS | Show the resource groups that match the current user. | | ALL | Specify this parameter to show all resource groups in the cluster. | | RESOURCE GROUP | Show the specified resource group. | | resourcegroupname | Name of the resource group to be shown. | | Return | Description | | -- | | | name | Name of the resource group. | | id | ID of the resource group. | | cpucorelimit | CPU core limit of the resource group. | | mem_limit | Memory limit of the resource group. | | bigquerycpusecondlimit | Big query upper time limit of the resource group. | | bigqueryscanrowslimit | Big query scan row limit of the resource group. | | bigquerymem_limit | Big query memory limit of the resource group. | | concurrency_limit | Concurrency limit of the resource group. | | type | Type of resource group. | | classifiers | Classifiers that are associated with the resource group. `id` is the ID of the classifier, and `weight` is the degree of matching. | Example 1: Shows all resource groups in the cluster. ```Plain mysql> SHOW RESOURCE GROUPS ALL; +-+--+-+--+-+++-+--++ | name | id | cpucorelimit | memlimit | bigquerycpusecondlimit | bigqueryscanrowslimit | bigquerymemlimit | concurrency_limit | type | classifiers | +-+--+-+--+-+++-+--++ | rg1 | 625126 | 10 | 20.0% | 100 | 100000 | 1073741824 | null | NORMAL | (id=625127, weight=4.459375, user=rg1user1, role=rg1role1, querytype in (SELECT), sourceip=172.26.xxx.xx/24) | | rg1 | 625126 | 10 | 20.0% | 100 | 100000 | 1073741824 | null | NORMAL | (id=625128, weight=3.459375, user=rg1user2, querytype in (SELECT), source_ip=172.26.xxx.xx/24) | | rg1 | 625126 | 10 | 20.0% | 100 | 100000 | 1073741824 | null | NORMAL | (id=625129, weight=2.359375, user=rg1user3, sourceip=172.26.xxx.xx/24) | | rg1 | 625126 | 10 | 20.0% | 100 | 100000 | 1073741824 | null | NORMAL | (id=625130, weight=1.0, user=rg1_user4) | | rg1 | 625126 | 10 | 20.0% | 100 | 100000 | 1073741824 | null | NORMAL | (id=625131, weight=10.0, db='db1') | +-+--+-+--+-+++-+--++ ```
Project: Beam (App Definition and Development / Streaming & Messaging)
File: groupbykey.md

title: "GroupByKey" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< localstorage language language-py >}} {{< button-pydoc path="apache_beam.transforms.core" class="GroupByKey" >}} Takes a keyed collection of elements and produces a collection where each element consists of a key and all values associated with that key. See more information in the . Example 1: In the following example, we create a pipeline with a `PCollection` of produce keyed by season. We use `GroupByKey` to group all the produce for each season. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHON_GroupByKey" show="groupbykeysort" >}} {{< /playground >}} Example 2: {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHON_GroupByKey" show="groupbykey" >}} {{< /playground >}} for grouping by arbitrary properties of the elements. for combining all values associated with a key to a single result. for multiple input collections. {{< button-pydoc path="apache_beam.transforms.core" class="GroupByKey" >}}
Project: ClickHouse (App Definition and Development / Database)
File: url.md

slug: /en/sql-reference/table-functions/url sidebar_position: 200 sidebar_label: url `url` function creates a table from the `URL` with given `format` and `structure`. `url` function may be used in `SELECT` and `INSERT` queries on data in tables. Syntax ``` sql url(URL [,format] [,structure] [,headers]) ``` Parameters `URL` HTTP or HTTPS server address, which can accept `GET` or `POST` requests (for `SELECT` or `INSERT` queries correspondingly). Type: . `format` of the data. Type: . `structure` Table structure in `'UserID UInt64, Name String'` format. Determines column names and types. Type: . `headers` - Headers in `'headers('key1'='value1', 'key2'='value2')'` format. You can set headers for HTTP call. Returned value A table with the specified format and structure and with data from the defined `URL`. Examples Getting the first 3 lines of a table that contains columns of `String` and type from HTTP-server which answers in format. ``` sql SELECT * FROM url('http://127.0.0.1:12345/', CSV, 'column1 String, column2 UInt32', headers('Accept'='text/csv; charset=utf-8')) LIMIT 3; ``` Inserting data from a `URL` into a table: ``` sql CREATE TABLE test_table (column1 String, column2 UInt32) ENGINE=Memory; INSERT INTO FUNCTION url('http://127.0.0.1:8123/?query=INSERT+INTO+test_table+FORMAT+CSV', 'CSV', 'column1 String, column2 UInt32') VALUES ('http interface', 42); SELECT * FROM test_table; ``` Patterns in curly brackets `{ }` are used to generate a set of shards or to specify failover addresses. Supported pattern types and examples see in the description of the function. Character `|` inside patterns is used to specify failover addresses. They are iterated in the same order as listed in the pattern. The number of generated addresses is limited by setting. `_path` Path to the `URL`. Type: `LowCardinalty(String)`. `_file` Resource name of the `URL`. Type: `LowCardinalty(String)`. `_size` Size of the resource in bytes. Type: `Nullable(UInt64)`. If the size is unknown, the value is `NULL`. - allows to skip empty files while reading. Disabled by default. - allows to enable/disable decoding/encoding path in uri. Enabled by default. See Also
Project: YDB (App Definition and Development / Database)
File: auth-metadata.md

{% include %} Below are examples of the code for authentication using environment variables in different {{ ydb-short-name }} SDKs. {% list tabs %} Go (native) ```go package main import ( "context" "os" "github.com/ydb-platform/ydb-go-sdk/v3" yc "github.com/ydb-platform/ydb-go-yc" ) func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() db, err := ydb.Open(ctx, os.Getenv("YDBCONNECTIONSTRING"), yc.WithMetadataCredentials(ctx), yc.WithInternalCA(), // append Yandex Cloud certificates ) if err != nil { panic(err) } defer db.Close(ctx) ... } ``` Go (database/sql) ```go package main import ( "context" "database/sql" "os" "github.com/ydb-platform/ydb-go-sdk/v3" yc "github.com/ydb-platform/ydb-go-yc" ) func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() nativeDriver, err := ydb.Open(ctx, os.Getenv("YDBCONNECTIONSTRING"), yc.WithMetadataCredentials(ctx), yc.WithInternalCA(), // append Yandex Cloud certificates ) if err != nil { panic(err) } defer nativeDriver.Close(ctx) connector, err := ydb.Connector(nativeDriver) if err != nil { panic(err) } db := sql.OpenDB(connector) defer db.Close() ... } ``` Java ```java public void work(String connectionString) { AuthProvider authProvider = CloudAuthHelper.getMetadataAuthProvider(); GrpcTransport transport = GrpcTransport.forConnectionString(connectionString) .withAuthProvider(authProvider) .build()); TableClient tableClient = TableClient.newClient(transport).build(); doWork(tableClient); tableClient.close(); transport.close(); } ``` Node.js {% include %} Python {% include %} Python (asyncio) {% include %} C# (.NET) ```C# using Ydb.Sdk; using Ydb.Sdk.Yc; var metadataProvider = new MetadataProvider(); // Await initial IAM token. await metadataProvider.Initialize(); var config = new DriverConfig( endpoint: endpoint, // Database endpoint, "grpcs://host:port" database: database, // Full database path credentials: metadataProvider ); await using var driver = await Driver.CreateInitialized(config); ``` PHP ```php <?php use YdbPlatform\Ydb\Ydb; use YdbPlatform\Ydb\Auth\Implement\MetadataAuthentication; $config = [ // Database path 'database' => '/local', // Database endpoint 'endpoint' => 'localhost:2136', // Auto discovery (dedicated server only) 'discovery' => false, // IAM config 'iam_config' => [ 'insecure' => true, // 'rootcertfile' => './CA.pem', // Root CA file (uncomment for dedicated server) ], 'credentials' => new MetadataAuthentication() ]; $ydb = new Ydb($config); ``` {% endlist %}
Project: KICS (Provisioning / Security & Compliance)
File: f45ea400-6bbe-4501-9fc7-1c3d75c32067.md

title: Image Version Using 'latest' hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: f45ea400-6bbe-4501-9fc7-1c3d75c32067 Query name: Image Version Using 'latest' Platform: Dockerfile Severity: <span style="color:#ff7213">Medium</span> Category: Best Practices URL: When building images, always tag them with useful tags which codify version information, intended destination (prod or test, for instance), stability, or other information that is useful when deploying the application in different environments. Do not rely on the automatically-created latest tag<br> ```dockerfile title="Positive test num. 1 - dockerfile file" hl_lines="1" FROM alpine:latest RUN apk add --update py2-pip RUN pip install --upgrade pip COPY requirements.txt /usr/src/app/ RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt COPY app.py /usr/src/app/ COPY templates/index.html /usr/src/app/templates/ EXPOSE 5000 CMD ["python", "/usr/src/app/app.py"] ``` ```dockerfile title="Negative test num. 1 - dockerfile file" FROM alpine:3.5 RUN apk add --update py2-pip RUN pip install --upgrade pip COPY requirements.txt /usr/src/app/ RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt COPY app.py /usr/src/app/ COPY templates/index.html /usr/src/app/templates/ EXPOSE 5000 CMD ["python", "/usr/src/app/app.py"] ```
Project: external-secrets (Provisioning / Security & Compliance)
File: doppler.md

Sync secrets from the to Kubernetes using the External Secrets Operator. Doppler are recommended as they restrict access to a single config. NOTE: Doppler Personal Tokens are also supported but require `project` and `config` to be set on the `SecretStore` or `ClusterSecretStore`. Create the Doppler Token secret by opening the Doppler dashboard and navigating to the desired Project and Config, then create a new Service Token from the Access tab: Create the Doppler Token Kubernetes secret with your Service Token value: ```sh HISTIGNORE='kubectl' kubectl create secret generic \ doppler-token-auth-api \ --from-literal dopplerToken="dp.st.xxxx" ``` Then to create a generic `SecretStore`: ```yaml {% include 'doppler-generic-secret-store.yaml' %} ``` NOTE: In case of a `ClusterSecretStore`, be sure to set `namespace` in `secretRef.dopplerToken`. The Doppler provider allows for a wide range of use cases: Let's explore each use case using a fictional `auth-api` Doppler project. To sync one or more individual secrets: ``` yaml {% include 'doppler-fetch-secret.yaml' %} ``` To sync every secret from a config: ``` yaml {% include 'doppler-fetch-all-secrets.yaml' %} ``` To filter secrets by `path` (path prefix), `name` (regular expression) or a combination of both: ``` yaml {% include 'doppler-filtered-secrets.yaml' %} ``` To parse a JSON secret to its key-value pairs: ``` yaml {% include 'doppler-parse-json-secret.yaml' %} ``` Name transformers format keys from Doppler's UPPERSNAKECASE to one of the following alternatives: upper-camel camel lower-snake tf-var dotnet-env lower-kebab Name transformers require a specifically configured `SecretStore`: ```yaml {% include 'doppler-name-transformer-secret-store.yaml' %} ``` Then an `ExternalSecret` referencing the `SecretStore`: ```yaml {% include 'doppler-name-transformer-external-secret.yaml' %} ``` A single `DOPPLERSECRETSFILE` key is set where the value is the secrets downloaded in one of the following formats: json dotnet-json env env-no-quotes yaml Downloading secrets requires a specifically configured `SecretStore`: ```yaml {% include 'doppler-secrets-download-secret-store.yaml' %} ``` Then an `ExternalSecret` referencing the `SecretStore`: ```yaml {% include 'doppler-secrets-download-external-secret.yaml' %} ```
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: testexpr.md

The `TestExpr` defines an extractor with an optional microformat body. A test expression has a predicate component. The `Ident` defines the expected microformat the value being tested in a structural pattern match should conform to. If this validates, then an optional microformat expression that is specific to the extractor named by the `Ident` is employed to extract content from the value into a value that tremor can process.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 11_basic_state.md

Prompt: What is a state in Apache Beam? Response: in Apache Beam is an API that allows for managing per-key state in element-wise operations like or `Map`. It enables fine-grained control over aggregations, particularly in situations where and are insufficient. State is to the current processing key within a window, meaning each key has its own state. State is particularly useful for complex aggregations across multiple elements. Apache Beam provides several that can be used for different use cases, including scalar and collection values: `ValueState` `BagState` `SetState` `MapState` `OrderedListState` `CombiningState` The per-key state needs to be garbage-collected because the increasing size of the state can negatively impact performance. For more information about common garbage collection strategies, see . For more information, read the community blog post about . For additional examples of using state in Apache Beam, see .
Project: Chef InSpec (Provisioning / Security & Compliance)
File: google_compute_backend_buckets.md

+++ title = "googlecomputebackend_buckets resource" draft = false gh_repo = "inspec" platform = "gcp" [menu] [menu.inspec] title = "googlecomputebackend_buckets" identifier = "inspec/resources/gcp/googlecomputebackendbuckets.md googlecomputebackendbuckets resource" parent = "inspec/resources/gcp" +++ A `googlecomputebackend_buckets` is used to test a Google BackendBucket resource This resource has beta fields available. To retrieve these fields, include `beta: true` in the constructor for the resource ```ruby describe googlecomputebackend_buckets(project: 'chef-gcp-inspec', name: 'inspec-gcp-backend-bucket') do its('descriptions') { should include 'Backend bucket example' } its('bucket_names') { should include 'gcp-inspec-storage-bucket' } end ``` Properties that can be accessed from the `googlecomputebackend_buckets` resource: See the resource for more information. `bucket_names` : an array of `googlecomputebackendbucket` bucketname `cdn_policies` : an array of `googlecomputebackendbucket` cdnpolicy `creation_timestamps` : an array of `googlecomputebackendbucket` creationtimestamp `descriptions` : an array of `googlecomputebackend_bucket` description `enable_cdns` : an array of `googlecomputebackendbucket` enablecdn `ids` : an array of `googlecomputebackend_bucket` id `names` : an array of `googlecomputebackend_bucket` name This resource supports all of the above properties as filter criteria, which can be used with `where` as a block or a method. Ensure the is enabled for the current project.
Project: Puppet (Provisioning / Automation & Configuration)
File: http_environments.md

Environments ============ The `environments` endpoint allows for enumeration of the environments known to the master. Each environment contains information about itself like its modulepath, manifest directory, environment timeout, and the config version. This endpoint is by default accessible to any client with a valid certificate, though this may be changed in Puppet Server's `auth.conf`. Get Get the list of known environments. GET /puppet/v3/environments `application/json` None GET /puppet/v3/environments HTTP 200 OK Content-Type: application/json { "search_paths": ["/etc/puppetlabs/code/environments"] "environments": { "production": { "settings": { "modulepath": ["/etc/puppetlabs/code/environments/production/modules", "/etc/puppetlabs/code/environments/development/modules"], "manifest": ["/etc/puppetlabs/code/environments/production/manifests"] "environment_timeout": 180, "config_version": "/version/of/config" } } } } The `environment_timeout` attribute could also be the string "unlimited". Schema An environments response body conforms to
Project: ClickHouse (App Definition and Development / Database)
File: overview.md

slug: /en/interfaces/overview sidebar_label: Overview sidebar_position: 1 keywords: [clickhouse, network, interfaces, http, tcp, grpc, command-line, client, jdbc, odbc, driver] description: ClickHouse provides three network interfaces ClickHouse provides three network interfaces (they can be optionally wrapped in TLS for additional security): , which is documented and easy to use directly. , which has less overhead. . In most cases it is recommended to use an appropriate tool or library instead of interacting with those directly. The following are officially supported by ClickHouse: ClickHouse server provides embedded visual interfaces for power users: Play UI: open `/play` in the browser; Advanced Dashboard: open `/dashboard` in the browser; Binary symbols viewer for ClickHouse engineers: open `/binary` in the browser; There are also a wide range of third-party libraries for working with ClickHouse:
Project: Tetragon (Provisioning / Security & Compliance)
File: file-events.md

title: "File Access Monitoring" weight: 4 description: "File access traces with Tetragon" Tracing Policies can be added to Tetragon through YAML configuration files that extend Tetragon's base execution tracing capabilities. These policies do filtering in kernel to ensure only interesting events are published to userspace from the BPF programs running in kernel. This ensures overhead remains low even on busy systems. The following extends the example from Execution Tracing with a policy to monitor sensitive files in Linux. The policy used is the it can be reviewed and extended as needed. Files monitored here serve as a good base set of files. To apply the policy Kubernetes uses a CRD that can be applied with kubectl. Uses the same YAML configuration as Kuberenetes, but loaded through a file on disk. {{< tabpane lang=shell >}} {{< tab Kubernetes >}} kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml {{< /tab >}} {{< tab Docker >}} wget https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml docker stop tetragon-container docker run --name tetragon-container --rm --pull always \ --pid=host --cgroupns=host --privileged \ -v ${PWD}/filemonitoring.yaml:/etc/tetragon/tetragon.tp.d/filemonitoring.yaml \ -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \ quay.io/cilium/tetragon-ci:latest {{< /tab >}} {{< /tabpane >}} With the file applied we can attach tetra to observe events again: {{< tabpane lang=shell >}} {{< tab Kubernetes >}} kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods xwing {{< /tab >}} {{< tab Docker >}} docker exec tetragon-container tetra getevents -o compact {{< /tab >}} {{< /tabpane >}} Then reading a sensitive file: {{< tabpane lang=shell >}} {{< tab Kubernetes >}} kubectl exec -ti xwing -- bash -c 'cat /etc/shadow' {{< /tab >}} {{< tab Docker >}} cat /etc/shadow {{< /tab >}} {{< /tabpane >}} This will generate a read event (Docker events will omit Kubernetes metadata), ``` process default/xwing /bin/bash -c "cat /etc/shadow" process default/xwing /bin/cat /etc/shadow read default/xwing /bin/cat /etc/shadow exit default/xwing /bin/cat /etc/shadow 0 ``` Attempts to write in sensitive directories will similarly create write events. For example, attempting to write in `/etc`. {{< tabpane lang=shell >}} {{< tab Kubernetes >}} kubectl exec -ti xwing -- bash -c 'echo foo >> /etc/bar' {{< /tab >}} {{< tab Docker >}} cat /etc/shadow {{< /tab >}} {{< /tabpane >}} Will result in the following output in the tetra CLI. ``` process default/xwing /bin/bash -c "echo foo >> /etc/bar" write default/xwing /bin/bash /etc/bar write default/xwing /bin/bash /etc/bar exit default/xwing /bin/bash -c "echo foo >> /etc/bar ``` To explore tracing policies for networking try the quickstart. To dive into the details of policies and events please see section.
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: deploy.md

The `Deploy` rule defines the logical entry point into Tremor's command oriented deployment syntax. The deployment grammar defines units of deployment that the runtime manages on behalf of users. The grammar embeds the statement oriented query syntax and expression oriented scripting syntax where appropriate. A legal deployment is composed of: An optional set of module comments A sequence of top level expressions. There must be at least one defined. An optional end of stream token. At least one deploy command.
Project: Distribution (Provisioning / Container Registry)
File: architecture.md

draft: true TODO(stevvooe): Discuss the architecture of the registry, internally and externally, in a few different deployment scenarios. NOTE: This section belongs somewhere, perhaps in a design document. We are leaving this here so the information is not lost. Running the registry on eventually consistent backends has been part of the design from the beginning. This section covers some of the approaches to dealing with this reality. There are a few classes of issues that we need to worry about when implementing something on top of the storage drivers: Read-After-Write consistency (see this [article on s3](http://shlomoswidler.com/2009/12/read-after-write-consistency-in-amazon.html)). . In reality, the registry must worry about these kinds of errors when doing the following: Accepting data into a temporary upload file may not have latest data block yet (read-after-write). Moving uploaded data into its blob location (write-write race). Modifying the "current" manifest for given tag (write-write race). A whole slew of operations around deletes (read-after-write, delete-write races, garbage collection, etc.). The backend path layout employs a few techniques to avoid these problems: Large writes are done to private upload directories. This alleviates most of the corruption potential under multiple writers by avoiding multiple writers. Constraints in storage driver implementations, such as support for writing after the end of a file to extend it. Digest verification to avoid data corruption. Manifest files are stored by digest and cannot change. All other non-content files (links, hashes, etc.) are written as an atomic unit. Anything that requires additions and deletions is broken out into separate "files". Last writer still wins. Unfortunately, one must play this game when trying to build something like this on top of eventually consistent storage systems. If we run into serious problems, we can wrap the storagedrivers in a shared consistency layer but that would increase complexity and hinder registry cluster performance.