cwielenb/CNCF-Dataset huggingface dataset:
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 01_io_kafka.md

Prompt: Write a sample Java code snippet that writes data to a Apache Kafka topic using Apache Beam. Response: Your Apache Beam pipeline can write data to a Kafka topic using the Apache Beam PubSubIO connector. Here is an example of how to use the KafkaIO connector to accomplish this: ```java package kafka; import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions; import org.apache.beam.sdk.Pipeline; import org.apache.beam.sdk.io.kafka.KafkaIO; import org.apache.beam.sdk.options.Description; import org.apache.beam.sdk.options.PipelineOptionsFactory; import org.apache.beam.sdk.transforms.Create; import org.apache.beam.sdk.values.KV; import org.apache.kafka.common.serialization.IntegerSerializer; import org.apache.kafka.common.serialization.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.Arrays; import java.util.List; // Pipeline to write data to a Kafka topic using Apache Beam KafkaIO connector public class WriteKafkaTopic { private static final Logger LOG = LoggerFactory.getLogger(WriteKafkaTopic.class); // Pipeline options for writing data to Kafka topic public interface WriteKafkaTopicOptions extends WriteKafkaTopicOptions { @Description("Kafka Bootstrap Server") String getBootstrapServer(); void setBootstrapServer(String value); @Description("Kafka Topic Name") String getTopicName(); void setTopicName(String value); } // Main method to run the pipeline public static void main(String[] args) { // Parse the pipeline options from the command line WriteKafkaTopicOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(WriteKafkaTopicOptions.class); // Create the pipeline Pipeline p = Pipeline.create(options); // Create sample messages to write to the Kafka topic final List<KV<Integer, String>> messages = Arrays.asList( KV.of(1, "Sample message 1"), KV.of(2, "Sample message 2"), KV.of(3, "Sample message 3") ); // Write the messages to the Kafka topic p // Create PCollection from the list of sample messages .apply(Create.of(messages)) // Write the messages to the Kafka topic using the KafkaIO connector .apply(KafkaIO.<Integer, String>write() .withBootstrapServers(options.getBootstrapServer()) .withTopic(options.getTopicName()) // Specify the key and value serializers for the Kafka topic .withValueSerializer(StringSerializer.class) .withKeySerializer(IntegerSerializer.class) ); // Run the pipeline p.run(); } } ``` This code snippet utilizes the pipeline options pattern to parse command-line arguments.
Project: KubeBlocks by ApeCloud (App Definition and Development / Database)
File: kbcli_cluster_backup.md

title: kbcli cluster backup Create a backup for the cluster. ``` kbcli cluster backup NAME [flags] ``` ``` kbcli cluster backup mycluster kbcli cluster backup mycluster --method volume-snapshot kbcli cluster backup mycluster --method volume-snapshot --policy <backup-policy-name> kbcli cluster backup mycluster --parent-backup parent-backup-name ``` ``` --deletion-policy string Deletion policy for backup, determine whether the backup content in backup repo will be deleted after the backup is deleted, supported values: [Delete, Retain] (default "Delete") -h, --help help for backup --method string Backup methods are defined in backup policy (required), if only one backup method in backup policy, use it as default backup method, if multiple backup methods in backup policy, use method which volume snapshot is true as default backup method --name string Backup name --parent-backup string Parent backup name, used for incremental backup --policy string Backup policy name, if not specified, use the cluster default backup policy --retention-period string Retention period for backup, supported values: [1y, 1mo, 1d, 1h, 1m] or combine them [1y1mo1d1h1m], if not specified, the backup will not be automatically deleted, you need to manually delete it. ``` ``` --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --as-uid string UID to impersonate for the operation. --cache-dir string Default cache directory (default "$HOME/.kube/cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use --disable-compression If true, opt-out of response compression for all requests to the server --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. --match-server-version Require server version to match client version -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") -s, --server string The address and port of the Kubernetes API server --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use ``` - Cluster command.
Project: Flink (App Definition and Development / Streaming & Messaging)
File: jar.md

title: "JAR Statements" weight: 16 type: docs aliases: /dev/table/sql/jar.html <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> JAR statements are used to add user jars into the classpath or remove user jars from the classpath or show added jars in the classpath in the runtime. Flink SQL supports the following JAR statements for now: ADD JAR SHOW JARS REMOVE JAR {{< tabs "add jar statement" >}} {{< tab "SQL CLI" >}} The following examples show how to run `JAR` statements in . {{< /tab >}} {{< /tabs >}} {{< tabs "add jar" >}} {{< tab "SQL CLI" >}} ```sql Flink SQL> ADD JAR '/path/hello.jar'; [INFO] Execute statement succeeded. Flink SQL> ADD JAR 'hdfs:///udf/common-udf.jar'; [INFO] Execute statement succeeded. Flink SQL> SHOW JARS; +-+ | jars | +-+ | /path/hello.jar | | hdfs:///udf/common-udf.jar | +-+ Flink SQL> REMOVE JAR '/path/hello.jar'; [INFO] The specified jar is removed from session classloader. ``` {{< /tab >}} {{< /tabs >}} ```sql ADD JAR '<pathtofilename>.jar' ``` Add a JAR file to the list of resources, it supports adding the jar locates in a local or remote . The added JAR file can be listed using statements. Please don't use `ADD JAR` statements to load Hive source/sink/function/catalog. This is a known limitation of Hive connector and will be fixed in the future version. Currently, it's recommended to follow this to setup Hive integration. ```sql SHOW JARS ``` Show all added jars which are added by statements. ```sql REMOVE JAR '<pathtofilename>.jar' ``` Remove the specified jar that is added by the statements. <span class="label label-danger">Attention</span> REMOVE JAR statements only work in . {{< top >}}
Project: NATS (App Definition and Development / Streaming & Messaging)
File: CONTRIBUTING.md

Thanks for your interest in contributing! This document contains `nats-io/nats-server` specific contributing details. If you are a first-time contributor, please refer to the general to get a comprehensive overview of contributing to the NATS project. There are there general ways you can contribute to this repo: Proposing an enhancement or new feature Reporting a bug or regression Contributing changes to the source code For the first two, refer to the which guides you through the available options along with the needed information to collect. Prior to opening a pull request, it is recommended to open an issue first to ensure the maintainers can review intended changes. Exceptions to this rule include fixing non-functional source such as code comments, documentation or other supporting files. Proposing source code changes is done through GitHub's standard pull request workflow. If your branch is a work-in-progress then please start by creating your pull requests as draft, by clicking the down-arrow next to the `Create pull request` button and instead selecting `Create draft pull request`. This will defer the automatic process of requesting a review from the NATS team and significantly reduces noise until you are ready. Once you are happy, you can click the `Ready for review` button. A good pull request includes: A high-level description of the changes, including links to any issues that are related by adding comments like `Resolves #NNN` to your description. See for more information. An up-to-date parent commit. Please make sure you are pulling in the latest `main` branch and rebasing your work on top of it, i.e. `git rebase main`. Unit tests where appropriate. Bug fixes will benefit from the addition of regression tests. New features will not be accepted without suitable test coverage! No more commits than necessary. Sometimes having multiple commits is useful for telling a story or isolating changes from one another, but please squash down any unnecessary commits that may just be for clean-up, comments or small changes. No additional external dependencies that aren't absolutely essential. Please do everything you can to avoid pulling in additional libraries/dependencies into `go.mod` as we will be very critical of these. In order to accept a contribution, you will first need to certify that the contribution is your original work and that you license the work to the project under the . This is done by using `Signed-off-by` statements, which should appear in both your commit messages and your PR description. Please note that we can only accept sign-offs under a legal name. Nicknames and aliases are not permitted. To perform a sign-off with `git`, use `git commit -s` (or `--signoff`). If you have questions about the contribution process, please start a , join the , or send your question to the .
Project: KubeBlocks by ApeCloud (App Definition and Development / Database)
File: restart-a-kafka-cluster.md

title: Restart Kafka cluster description: How to restart a Kafka cluster keywords: [kafka, restart] sidebar_position: 4 sidebar_label: Restart You can restart all pods of the cluster. When an exception occurs in a database, you can try to restart it. :::note Restarting a Kafka cluster triggers a concurrent restart and the leader may change after the cluster restarts. ::: Restart a cluster. You can use `kbcli` or create an OpsRequest to restart a cluster. Option 1. (Recommended) Use kbcli Configure the values of `components` and `ttlSecondsAfterSucceed` and run the command below to restart a specified cluster. ```bash kbcli cluster restart NAME --components="kafka" \ --ttlSecondsAfterSucceed=30 ``` `components` describes the component name that needs to be restarted. `ttlSecondsAfterSucceed` describes the time to live of an OpsRequest job after the restarting succeeds. Option 2. Create an OpsRequest Run the command below to apply the restarting to a cluster. ```bash kubectl apply -f - <<EOF apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: name: ops-restart spec: clusterRef: kafka type: Restart restart: componentName: kafka EOF ``` Validate the restarting. Run the command below to check the cluster status to check the restarting status. ```bash kbcli cluster list <name> ``` *Example* ```bash kbcli cluster list kafka > NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE kafka kafka kafka-3.3.2 Delete Running 19m ``` STATUS=Restarting: it means the cluster restart is in progress. STATUS=Running: it means the cluster has been restarted.
Project: Apache Hadoop (App Definition and Development / Database)
File: CHANGELOG.0.14.4.md

<! --> | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | Deadlock in jobtracker | Blocker | . | Michael Bieniosek | Arun C Murthy | | | libhdfs makefile wrongly sets up DT\_SONAME field of libhdfs.so | Critical | . | Arun C Murthy | Arun C Murthy | | JIRA | Summary | Priority | Component | Reporter | Contributor | |:- |:- | : |:- |:- |:- | | | C and C++ files are missing Apache license header | Blocker | . | Owen O'Malley | Owen O'Malley |
Project: Notary (Provisioning / Security & Compliance)
File: login.md

Use `notation login` to log in to an OCI registry. Users can execute `notation login` multiple times to log in multiple registries. ```text Log in to an OCI registry Usage: notation login [flags] <server> Flags: -d, --debug debug mode -h, --help help for login --insecure-registry use HTTP protocol while connecting to registries. Should be used only for testing -p, --password string password for registry operations (default to $NOTATION_PASSWORD if not specified) --password-stdin take the password from stdin -u, --username string username for registry operations (default to $NOTATION_USERNAME if not specified) -v, --verbose verbose mode ``` ```shell notation login -u <username> -p <password> registry.example.com ``` ```shell notation login registry.example.com ```
Project: YDB (App Definition and Development / Database)
File: top_bottom.md

Return a list of the maximum/minimum values of an expression. The first argument is an expression, the second argument limits the number of items. Examples ```yql SELECT TOP(key, 3), BOTTOM(value, 3) FROM my_table; ``` ```yql $top_factory = AggregationFactory("TOP", 3); $bottom_factory = AggregationFactory("BOTTOM", 3); SELECT AGGREGATEBY(key, $topfactory), AGGREGATEBY(value, $bottomfactory) FROM my_table; ``` Return a list of values of the first argument for the rows containing the maximum/minimum values of the second argument. The third argument limits the number of items in the list. When you use , a `Tuple` containing a value and a key is passed as the first argument. In this case, the limit for the number of items is passed by the second argument at factory creation. Examples ```yql SELECT TOP_BY(value, LENGTH(value), 3), BOTTOM_BY(value, key, 3) FROM my_table; ``` ```yql $topbyfactory = AggregationFactory("TOP_BY", 3); $bottombyfactory = AggregationFactory("BOTTOM_BY", 3); SELECT AGGREGATEBY(AsTuple(value, LENGTH(value)), $topby_factory), AGGREGATEBY(AsTuple(value, key), $bottomby_factory) FROM my_table; ```
Project: Apache Heron (App Definition and Development / Streaming & Messaging)
File: observability-prometheus.md

id: version-0.20.0-incubating-observability-prometheus title: Observability with Prometheus sidebar_label: Prometheus original_id: observability-prometheus <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> You can integrate Heron with to monitor and gather runtime metrics exported by Heron topologies. Heron supports custom metric exporters from the Metrics Manager. You can either build your own Prometheus metrics sink or use the . To set up your Heron cluster to export to Prometheus, you need to make two changes to the `metrics_sinks.yaml` configuration file: Add `prometheus-sink` to the `sinks` list Add a `prometheus-sink` map to the file that sets values for the listed below. You can uncomment the existing `prometheus-sink` map to get the default configuration. Parameter | Description | Default :|:|:- `class` | The Java class used to control Prometheus sink behavior | `port` | The port on which Prometheus will scrape for metrics. Note: You must supply a `port` or `port-file` config. | 8080 `port-file` | The path to a text file that contains the port number to be used by Prometheus for metrics scraping. Note: You must supply a `port` or `port-file` config. | `metrics.port` `path` | The Prometheus path on which to publish metrics | `/metrics` `include-topology-name` | Whether the name of the Heron topology will be included in names for metrics | `true` `metrics-cache-max-size` | The maximum number of metrics cached and published | 1000000 `metrics-cache-ttl-sec` | The time to live (TTL) for metrics, i.e. the time, in seconds after which a metric that was collected will stop being published | 600 (10 minutes)
Project: Apicurio Registry (App Definition and Development / Streaming & Messaging)
File: feature_request.md

name: "Feature request" about: Propose a feature request that you think will improve this project title: '' labels: Enhancement assignees: '' <!-- A clear and concise description of what the feature or problem is. --> <!-- How will it benefit the Apicurio Registry and its users? --> <!-- Add any other context like screenshots or mockups, if applicable. -->
Project: Apache Spark (App Definition and Development / Streaming & Messaging)
File: sql-ref-syntax-ddl-create-table-like.md

layout: global title: CREATE TABLE LIKE displayTitle: CREATE TABLE LIKE license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. The `CREATE TABLE` statement defines a new table using the definition/metadata of an existing table or view. ```sql CREATE TABLE [IF NOT EXISTS] tableidentifier LIKE sourcetable_identifier USING data_source [ ROW FORMAT row_format ] [ STORED AS file_format ] [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ] [ LOCATION path ] ``` table_identifier* Specifies a table name, which may be optionally qualified with a database name. Syntax: `[ databasename. ] tablename` USING data_source* Data Source is the input format used to create the table. Data source can be CSV, TXT, ORC, JDBC, PARQUET, etc. ROW FORMAT* SERDE is used to specify a custom SerDe or the DELIMITED clause in order to use the native SerDe. STORED AS* File format for table storage, could be TEXTFILE, ORC, PARQUET, etc. TBLPROPERTIES* Table properties that have to be set are specified, such as `created.by.user`, `owner`, etc. LOCATION* Path to the directory where table data is stored, which could be a path on distributed storage like HDFS, etc. Location to create an external table. ```sql -- Create table using an existing table CREATE TABLE Student_Dupli like Student; -- Create table like using a data source CREATE TABLE Student_Dupli like Student USING CSV; -- Table is created as external table at the location specified CREATE TABLE Student_Dupli like Student location '/root1/home'; -- Create table like using a rowformat CREATE TABLE Student_Dupli like Student ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE TBLPROPERTIES ('owner'='xxxx'); ```
Project: Beam (App Definition and Development / Streaming & Messaging)
File: pipeline-options.md

title: "Pipeline option patterns" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> The samples on this page show you common pipeline configurations. For more information about pipeline configuration options, see and . {{< language-switcher java py >}} Use the `ValueProvider` interface to access runtime parameters after completing a pipeline job. You can use the `ValueProvider` interface to pass runtime parameters to your pipeline, but you can only log the parameters from within the Beam DAG. A solution is to add a pipeline with a `DoFn` that processes a placeholder value and then logs the runtime parameters: {{< highlight java >}} {{< code_sample "examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java" AccessingValueProviderInfoAfterRunSnip1 >}} {{< /highlight >}} {{< highlight py >}} {{< codesample "sdks/python/apachebeam/examples/snippets/snippets.py" AccessingValueProviderInfoAfterRunSnip1 >}} {{< /highlight >}}
Project: Chef InSpec (Provisioning / Security & Compliance)
File: cron.md

+++ title = "cron resource" draft = false gh_repo = "inspec" platform = "linux" [menu] [menu.inspec] title = "cron" identifier = "inspec/resources/os/cron.md cron resource" parent = "inspec/resources/os" +++ Use the `cron` Chef InSpec audit resource to test the Crontab entries of a particular user on the system. You can also use this resource as an alias to the `crontab` resource. This resource is distributed with Chef InSpec. (optional) This parameter tests the cron entries of a particular user. By default, it refers to the current user. The `table` property checks whether a particular cron entry matches the specific regex. its(:table) { should match /regex/ } A `cron` resource block declares a user (which defaults to the current user). describe cron do its(:table) { should match /regex/ } end describe cron(user: "USER") do it { should have_entry "5 /some/scheduled/task.sh" } end The following examples show how to use this audit resource. describe cron do it { should have_entry "5 /some/scheduled/task.sh" } end describe cron('MY_USER') do it { should have_entry "5 /some/scheduled/task.sh" } end describe cron do its(:table) { should match /^5/ } end For a full list of the available matchers, please visit our . it { should have_entry("5 /some/scheduled/task.sh") }