cwielenb/CNCF-Dataset huggingface dataset:
Project: Hazelcast Jet (App Definition and Development / Streaming & Messaging)
File: 014-batchstage-sort.md

title: 014 - BatchStage.sort() description: Distributed sorting for batch workloads id: version-4.4-014-batchstage-sort original_id: 014-batchstage-sort Since: 4.3 Currently, Jet can sort the data only indirectly, through a special aggregate operation, resulting in a single output item that is the whole sorted list. This design implements a transform that affects just the order and not the type of the stream items (`T` vs. `List<T>`), but only for a batch workload. We gain the following methods: ```java BatchStage<T> BatchStage.sort() BatchStage<T> BatchStage.sort(Comparator<? super T>) ``` Example: ```java Pipeline p = Pipeline.create(); BatchStage<String> strings = p.readFrom(stringSource); BatchStage<String> sortedStrings = p.sort(); ``` Distributed sorting happens in two phases: Sort the data on every cluster node locally, using a single `SortP` processor per node Receive the partially sorted data on a single node, using a new type of edge, `ordered(Comparator)` `SortP` is a very simple processor, it just puts all the received data into a `PriorityQueue` with a user-supplied `Comparator`. In the completion phase it drains the priority queue into the output. This is the entire implementation: ```java public class SortP<T> extends AbstractProcessor { private final PriorityQueue<T> priorityQueue; private final Traverser<T> resultTraverser; public SortP(@Nullable Comparator<T> comparator) { this.priorityQueue = new PriorityQueue<>(comparator); this.resultTraverser = priorityQueue::poll; } @Override protected boolean tryProcess0(@Nonnull Object item) { priorityQueue.add((T) item); return true; } @Override public boolean complete() { return emitFromTraverser(resultTraverser); } } ``` The `ordered(Comparator)` edge ensures that it always receives the least item available from all the incoming data streams (one from each node). Even though this is trivially simple high-level logic, implementing the monotonic reception with cooperative multithreading in mind is the most complex part of this work. It is implemented inside `ConcurrentInboundEdgeStream.OrderedDrain`. Apart from the mentioned limitation that this is only for batch stages, a big caveat is that Jet may reorder the data between stages through parallel processing, removing the sort order. There is work in progress that will prevent Jet from reordering the stream items where it would break correctness.
Project: KubeBlocks by ApeCloud (App Definition and Development / Database)
File: kbcli_fault_network.md

title: kbcli fault network Network chaos. ``` -h, --help help for network ``` ``` --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --as-uid string UID to impersonate for the operation. --cache-dir string Default cache directory (default "$HOME/.kube/cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use --disable-compression If true, opt-out of response compression for all requests to the server --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. --match-server-version Require server version to match client version -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") -s, --server string The address and port of the Kubernetes API server --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use ``` - Inject faults to pod. - Limit the bandwidth that pods use to communicate with other objects. - Distorts the messages a pod communicates with other objects. - Make pods communicate with other objects lazily. - Inject faults into DNS server. - Make pods communicate with other objects to pick up duplicate packets. - Intercept HTTP requests and responses. - Cause pods to communicate with other objects to drop packets. - Make a pod network partitioned from other objects.
Project: KICS (Provisioning / Security & Compliance)
File: 93d88cf7-f078-46a8-8ddc-178e03aeacf1.md

title: Missing Version Specification In dnf install hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 93d88cf7-f078-46a8-8ddc-178e03aeacf1 Query name: Missing Version Specification In dnf install Platform: Dockerfile Severity: <span style="color:#ff7213">Medium</span> Category: Supply-Chain URL: Specifying a package version allows to reduce failures due to unanticipated changes in required packages.<br> ```dockerfile title="Positive test num. 1 - dockerfile file" hl_lines="2 3" FROM fedora:latest RUN dnf -y update && dnf -y install httpd && dnf clean all RUN ["dnf", "install", "httpd"] COPY index.html /var/www/html/index.html EXPOSE 80 ENTRYPOINT /usr/sbin/httpd -DFOREGROUND ``` ```dockerfile title="Negative test num. 1 - dockerfile file" FROM fedora:latest RUN dnf -y update && dnf -y install httpd-2.24.2 && dnf clean all RUN ["dnf", "install", "httpd-2.24.2"] COPY index.html /var/www/html/index.html EXPOSE 80 ENTRYPOINT /usr/sbin/httpd -DFOREGROUND ```
Project: Meshery (Provisioning / Automation & Configuration)
File: v0.6.144.md

layout: release date: 2023-09-22 tag: v0.6.144 add docs for component generation and updater scripts @MUzairS15 (#8860) Fixed Meshary Integration Updater @ptdatta (#8774) [Designs] Fixes And Enhancements @aabidsofi19 (#8861) Fix Notification center bell icon toggle btn @aabidsofi19 (#8875) [UI] Enhancements :Notification center @aabidsofi19 (#8872) [UI] Fix cluster resources dashboard not getting populated with data @MUzairS15 (#8873) [chore] Improve events for Server @MUzairS15 (#8864) [UI] Notification inverse read indicator @aabidsofi19 (#8865) [UI] Remove old store for notification center @aabidsofi19 (#8858) Fix typo in component-svg/_.txt @eltociear (#8830) [UI] Enhance Error Handling @aabidsofi19 (#8851) [bug] Fix API docs build @theBeginner86 (#8852) [bug] Fix API docs build @theBeginner86 (#8852) Thank you to our contributors for making this release possible: @MUzairS15, @aabidsofi19, @eltociear, @l5io, @leecalcote, @ptdatta and @theBeginner86
Project: Apache Storm (App Definition and Development / Streaming & Messaging)
File: Message-passing-implementation.md

title: Message Passing Implementation layout: documentation documentation: true (Note: this walkthrough is out of date as of 0.8.0. 0.8.0 revamped the message passing infrastructure to be based on the Disruptor) This page walks through how emitting and transferring tuples works in Storm. Worker is responsible for message transfer `refresh-connections` is called every "task.refresh.poll.secs" or whenever assignment in ZK changes. It manages connections to other workers and maintains a mapping from task -> worker Provides a "transfer function" that is used by tasks to send tuples to other tasks. The transfer function takes in a task id and a tuple, and it serializes the tuple and puts it onto a "transfer queue". There is a single transfer queue for each worker. The serializer is thread-safe The worker has a single thread which drains the transfer queue and sends the messages to other workers Message sending happens through this protocol: The implementation for distributed mode uses ZeroMQ The implementation for local mode uses in memory Java queues (so that it's easy to use Storm locally without needing to get ZeroMQ installed) Receiving messages in tasks works differently in local mode and distributed mode In local mode, the tuple is sent directly to an in-memory queue for the receiving task In distributed mode, each worker listens on a single TCP port for incoming messages and then routes those messages in-memory to tasks. The TCP port is called a "virtual port", because it receives The virtual port implementation is here: Tasks listen on an in-memory ZeroMQ port for messages from the virtual port Bolts listen here: Spouts listen here: Tasks are responsible for message routing. A tuple is emitted either to a direct stream (where the task id is specified) or a regular stream. In direct streams, the message is only sent if that bolt subscribes to that direct stream. In regular streams, the stream grouping functions are used to determine the task ids to send the tuple to. Tasks have a routing map from {stream id} -> {component id} -> {stream grouping function} The "tasks-fn" returns the task ids to send the tuples to for either regular stream emit or direct stream emit After getting the output task ids, bolts and spouts use the transfer-fn provided by the worker to actually transfer the tuples Bolt transfer code here: Spout transfer code here:
Project: YugabyteDB (App Definition and Development / Database)
File: do-demo.md

title: do_demo.sql linkTitle: do_demo.sql headerTitle: do_demo.sql description: do_demo.sql - Part of the code kit for the "Analyzing a normal distribution" section within the YSQL window functions documentation. menu: v2.18: identifier: do-demo parent: analyzing-a-normal-distribution weight: 170 type: docs Save this script as `do_demo.sql`. ```plpgsql -- Uses table t4. -- Once you've created it, you can run this script time and again using -- for example, a different number of histogran buckets or a -- different number of buckts for the analyses done by percent_rank(), -- cum_dist(), and ntile(). -- -- STEP ZERO -- Get a clean start. \i docleanstart.sql -- -- STEP ONE -- Create a function to report some useful overall measures of t4 -- and run it, spooling the output. \i crshowt4.sql \o reports/show_t4.txt select t as "Some useful overall measures of t4." from show_t4(); \o \i crdpviews.sql \i crprcdequalityreport.sql \o reports/dpprcdequalityreport.txt select * from prcdequality_report(0.50); \t on select * from prcdequality_report(0.10); select * from prcdequality_report(0.05); select * from prcdequality_report(0.01); \t off \o \i crintviews.sql \o reports/intprcdequalityreport.txt select * from prcdequality_report(0.50); \t on select * from prcdequality_report(0.10); select * from prcdequality_report(0.05); select * from prcdequality_report(0.01); \t off \o -- -- STEP TWO -- Create a function to visualize the data as a histogram. -- This relies on the function bucket(). -- Run it, spooling the output. -- Choose one, then the other, of these two methods to -- demonstrate that they produce identical results. -- \i crbucketusingwidthbucket.sql \i crbucketdedicated_code.sql \i doassertbucket_ok.sql \i cr_histogram.sql \o reports/dp_histogram.txt \t on select * from histogram(50, 100); \t off \o -- -- STEP THREE -- Compare the bucket allocation produced by ntile(), percent_rank(), -- and cumedist() acting on the double precision column dpscore. \i crdontile.sql \i crdopercent_rank.sql \i crdocume_dist.sql \i crdpviews.sql \i dopopulateresults.sql \o reports/dp_results.txt \i doreportresults.sql \o \o reports/comparedpresults.txt \i docomparedp_results.sql \o -- STEP FOUR -- Compare the bucket allocation produced by ntile(), percent_rank(), -- and cumedist() acting on the int column intscore. \i crintviews.sql \i dopopulateresults.sql \o reports/int_results.txt \i doreportresults.sql \o ```
Project: ClickHouse (App Definition and Development / Database)
File: tcp.md

slug: /en/interfaces/tcp sidebar_position: 18 sidebar_label: Native Interface (TCP) The native protocol is used in the , for inter-server communication during distributed query processing, and also in other C++ programs. Unfortunately, native ClickHouse protocol does not have formal specification yet, but it can be reverse-engineered from ClickHouse source code (starting ) and/or by intercepting and analyzing TCP traffic.
Project: YugabyteDB (App Definition and Development / Database)
File: recover-disk.md

title: Disk failure linkTitle: Disk failure headerTitle: Recover failing disk description: Learn how to recover failing disk aliases: /troubleshoot/nodes/disk-failure/ menu: preview: parent: troubleshoot-nodes weight: 40 type: docs YugabyteDB can be configured to use multiple storage disks by setting the flag. This introduces the possibility of disk failure and recovery issues. The YB-TServer service automatically detects disk failures and attempts to spread the data from the failed disk to other healthy nodes in the cluster. In a single-zone setup with a replication factor (RF) of `3`: if you started with four nodes or more, then there would be at least three nodes left after one failed. In this case, re-replication is automatically started if a YB-TServer or disk is down for 10 minutes. In a multi-zone setup with a replication factor (RF) of `3`: YugabyteDB tries to keep one copy of data per zone. In this case, for automatic re-replication of data, a zone needs to have at least two YB-TServers so that if one fails, its data can be re-replicated to the other. Thus, this would mean at least a six-node cluster. To replace a failed disk, perform the following steps: Stop the YB-TServer node. Replace the disks that have failed. Restart the `yb-tserver` service. On restart, the YB-TServer will see the new empty disk and start replicating tablets from other nodes.
Project: Apache StreamPipes (App Definition and Development / Streaming & Messaging)
File: quickstart.md

<!-- ~ Licensed to the Apache Software Foundation (ASF) under one or more ~ contributor license agreements. See the NOTICE file distributed with ~ this work for additional information regarding copyright ownership. ~ The ASF licenses this file to You under the Apache License, Version 2.0 ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. ~ --> As a quick example, we demonstrate how to set up and configure a StreamPipes client. In addition, we will get the available data lake measures out of StreamPipes. ```python from streampipes.client import StreamPipesClient from streampipes.client.config import StreamPipesClientConfig from streampipes.client.credential_provider import StreamPipesApiKeyCredentials config = StreamPipesClientConfig( credential_provider = StreamPipesApiKeyCredentials( username = "test@streampipes.apache.org", api_key = "DEMO-KEY", ), host_address = "localhost", https_disabled = True, port = 80 ) client = StreamPipesClient(client_config=config) measures = client.dataLakeMeasureApi.all() len(measures) ``` Output: ``` 1 ``` <br> ``` measures.to_pandas() ``` Output: ``` measurename timestampfield ... pipelineisrunning numeventproperties 0 test s0::timestamp ... False 2 [1 rows x 6 columns] ``` <br> Alternatively, you can provide your credentials via environment variables. Simply define your credential provider as follows: ```python from streampipes.client.credential_provider import StreamPipesApiKeyCredentials StreamPipesApiKeyCredentials() ``` This requires to set the following environment variables: `SPAPIKEY` and `SP_USERNAME` <br> `username` is always the username that is used to log in into StreamPipes. <br> The `api_key` can be generated within the UI as demonstrated below:
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: stringliteral.md

The `StringLiteral` rule defines a string literal in tremor. Strings are `"` single-quote or `"""` triple-quote delimited blocks of UTF-8 text. A single-quote string is a single line string, supporting sting interpolation. A triple-quote string is a multi-line string, supporting sting interpolation.
Project: Flink (App Definition and Development / Streaming & Messaging)
File: deduplication.md

title: "Deduplication" weight: 16 type: docs <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< label Batch >}} {{< label Streaming >}} Deduplication removes rows that duplicate over a set of columns, keeping only the first one or the last one. In some cases, the upstream ETL jobs are not end-to-end exactly-once; this may result in duplicate records in the sink in case of failover. However, the duplicate records will affect the correctness of downstream analytical jobs - e.g. `SUM`, `COUNT` - so deduplication is needed before further analysis. Flink uses `ROW_NUMBER()` to remove duplicates, just like the way of Top-N query. In theory, deduplication is a special case of Top-N in which the N is one and order by the processing time or event time. The following shows the syntax of the Deduplication statement: ```sql SELECT [column_list] FROM ( SELECT [column_list], ROW_NUMBER() OVER ([PARTITION BY col1[, col2...]] ORDER BY time_attr [asc|desc]) AS rownum FROM table_name) WHERE rownum = 1 ``` Parameter Specification: `ROW_NUMBER()`: Assigns an unique, sequential number to each row, starting with one. `PARTITION BY col1[, col2...]`: Specifies the partition columns, i.e. the deduplicate key. `ORDER BY time_attr . Currently Flink supports and . Ordering by ASC means keeping the first row, ordering by DESC means keeping the last row. `WHERE rownum = 1`: The `rownum = 1` is required for Flink to recognize this query is deduplication. {{< hint info >}} Note: the above pattern must be followed exactly, otherwise the optimizer wont be able to translate the query. {{< /hint >}} The following examples show how to specify SQL queries with Deduplication on streaming tables. ```sql CREATE TABLE Orders ( order_id STRING, user STRING, product STRING, num BIGINT, proctime AS PROCTIME() ) WITH (...); -- remove duplicate rows on order_id and keep the first occurrence row, -- because there shouldn't be two orders with the same order_id. SELECT order_id, user, product, num FROM ( SELECT *, ROWNUMBER() OVER (PARTITION BY orderid ORDER BY proctime ASC) AS row_num FROM Orders) WHERE row_num = 1 ``` {{< top >}}
Project: Podman (App Definition and Development / Application Definition & Image Build)
File: podman-farm-remove.1.md

% podman-farm-remove 1 podman\-farm\-remove - Delete one or more farms podman farm remove [options] name podman farm rm [options] name Delete one or more farms. Remove all farms. Remove specified farm: ``` $ podman farm remove farm1 ``` Remove all farms: ``` $ podman farm rm --all ``` , July 2023, Originally compiled by Urvashi Mohnani (umohnani at redhat dot com)s
Project: KubeDB by AppsCode (App Definition and Development / Database)
File: CHANGELOG-v2021.09.30.md

title: Changelog | KubeDB description: Changelog menu: docs_{{.version}}: identifier: changelog-kubedb-v2021.09.30 name: Changelog-v2021.09.30 parent: welcome weight: 20210930 product_name: kubedb menuname: docs{{.version}} sectionmenuid: welcome url: /docs/{{.version}}/welcome/changelog-v2021.09.30/ aliases: /docs/{{.version}}/CHANGELOG-v2021.09.30/ Prepare for release v0.7.0 (#37) Log warning if Community License is used with non-demo namespace (#36) Prepare for release v0.9.0 (#227) Add MongoDB Offline Volume Expansion (#224) Fix MariaDB Reconfigure TLS OpsReq early Successful status (#225) Add TLS support for Redis Sentinel (#223) Log warning if Community License is used with non-demo namespace (#226) Add `VolumeExpansionMode` in mongodb ops request (#800) Add reconfigure TLS constants (#801) Add RedisConfiguration struct for redis AppBinding (#799) Add Redis Sentinel CRD resources (#794) Prepare for release v0.22.0 (#629) Log warning if Community License is used with non-demo namespace (#628) Prepare for release v0.22.0 (#520) Log warning if Community License is used with non-demo namespace (#519) Prepare for release v2021.09.30 (#360) Use kubedb/mariadb-init:0.4.0 (#359) Add support for Redis Sentinel (#352) Log warning if Community License is used with non-demo namespace (#358) MySQL 5.7.35 and 8.0.26 (#357) Prepare for release v0.6.0 (#94) Fixed security context and governing service patch issue. (#92) Log warning if Community License is used with non-demo namespace (#93) Fix coordinator run (#91) Prepare for release v0.2.0 (#12) Restart mysqld Process if missing (#10) Log warning if Community License is used with non-demo namespace (#11) Prepare for release v0.15.0 (#312) Log warning if Community License is used with non-demo namespace (#311) Prepare for release v0.15.0 (#417) Log warning if Community License is used with non-demo namespace (#416) Prepare for release v0.15.0 (#408) Log warning if Community License is used with non-demo namespace (#407) Prepare for release v0.22.0 (#423) Log warning if Community License is used with non-demo namespace (#422) Prepare for release v0.9.0 (#214) Log warning if Community License is used with non-demo namespace (#213) Prepare for release v0.6.0 (#39) Log warning if Community License is used with non-demo namespace (#38) Prepare for release v0.9.0 (#174) Log warning if Community License is used with non-demo namespace (#173) Prepare for release v0.22.0 (#523) Log warning if Community License is used with non-demo namespace (#522) Prepare for release v0.9.0 (#193) Log warning if Community License is used with non-demo namespace (#192) Prepare for release v0.15.0 (#341) Add support for Redis Sentinel mode (#335) Log warning if Community License is used with non-demo namespace (#340) Prepare for release v0.1.0 (#2) Update Sentinel client (#1) Prepare for release v0.9.0 (#161) Log warning if Community License is used with non-demo namespace (#160) Prepare for release v0.7.0 (#142) Log warning if Community License is used with non-demo namespace (#141)
Project: Meshery (Provisioning / Automation & Configuration)
File: mesheryctl-components-search.md

layout: default title: mesheryctl-components-search permalink: reference/mesheryctl/components/search redirect_from: reference/mesheryctl/components/search/ type: reference display-title: "false" language: en command: components subcommand: search search registered components search components registered in Meshery Server <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl components search [flags] </div> </pre> Search for components using a query <pre class='codeblock-pre'> <div class='codeblock'> mesheryctl exp components search [query-text] </div> </pre> <pre class='codeblock-pre'> <div class='codeblock'> -h, --help help for search </div> </pre> <pre class='codeblock-pre'> <div class='codeblock'> --config string path to config file (default "/home/runner/.meshery/config.yaml") -v, --verbose verbose output </div> </pre> Go back to , if you want to add content manually to the CLI documentation, please refer to the for guidance.