cwielenb/CNCF-Dataset huggingface dataset:
Project: Beam (App Definition and Development / Streaming & Messaging)
File: runinference.md

title: "RunInference" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< localstorage language language-py >}} <table> <tr> <td> <a> {{< button-pydoc path="apache_beam.ml.inference.base" class="RunInference" >}} </a> </td> </tr> </table> Uses models to do local and remote inference. A `RunInference` transform performs inference on a `PCollection` of examples using a machine learning (ML) model. The transform outputs a `PCollection` that contains the input examples and output predictions. Avaliable in Apache Beam 2.40.0 and later versions. For more information about Beam RunInference APIs, see the page and the examples. The following examples show how to create pipelines that use the Beam RunInference API to make predictions based on models. {{< table >}} | Framework | Example | | -- | -- | | PyTorch | | | PyTorch | | | Sklearn| | | Sklearn | |: {{< /table >}} Not applicable. {{< button-pydoc path="apache_beam.ml.inference" class="RunInference" >}}
Project: Beam (App Definition and Development / Streaming & Messaging)
File: inline_python.md

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> The contents of this file have been moved to the main Apache Beam site at https://beam.apache.org/documentation/sdks/yaml-inline-python/
Project: KICS (Provisioning / Security & Compliance)
File: b176e927-bbe2-44a6-a9c3-041417137e5f.md

title: AD Admin Not Configured For SQL Server hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: b176e927-bbe2-44a6-a9c3-041417137e5f Query name: AD Admin Not Configured For SQL Server Platform: Ansible Severity: <span style="color:#ff7213">Medium</span> Category: Insecure Configurations URL: The Active Directory Administrator is not configured for a SQL server<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="3" name: Create (or update) SQL Server azurermsqlserver: resource_group: myResourceGroup name: server_name location: westus admin_username: mylogin admin_password: Testpasswordxyz12! ``` ```yaml title="Negative test num. 1 - yaml file" name: Create (or update) SQL Server azurermsqlserver: resource_group: myResourceGroup name: server_name location: westus admin_username: mylogin admin_password: Testpasswordxyz12! ad_user: sqladmin ```
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: sep.md

The `Sep` rule is a convenience that allows defining a template for a common sub rule sequence. The `Sep` macro rule definition in tremor DSLs allows lists or sequences of expressions to be separated by a specified delimiter. The delimiter is optional for the final item in a list or sequence. |Argument|Description| ||| |T|The term rule - specifies what is to be separated| |D|The delimiter rule - specifies how elements are separated| |L|A list of accumulated terms|
Project: KubeVela (App Definition and Development / Application Definition & Image Build)
File: build-push-image.eg.md

```yaml apiVersion: core.oam.dev/v1beta1 kind: Application metadata: name: build-push-image namespace: default spec: components: name: my-web type: webservice properties: image: fogdong/simple-web-demo:v1 ports: port: 80 expose: true workflow: steps: name: create-git-secret type: export2secret properties: secretName: git-secret data: token: <git token> name: create-image-secret type: export2secret properties: secretName: image-secret kind: docker-registry dockerRegistry: username: <docker username> password: <docker password> name: build-push type: build-push-image properties: context: git: github.com/FogDong/simple-web-demo branch: main image: fogdong/simple-web-demo:v1 credentials: image: name: image-secret name: apply-comp type: apply-component properties: component: my-web ```
Project: KICS (Provisioning / Security & Compliance)
File: 0fd7d920-4711-46bd-aff2-d307d82cd8b7.md

title: User With Privilege Escalation By Actions 'iam:CreateLoginProfile' hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 0fd7d920-4711-46bd-aff2-d307d82cd8b7 Query name: User With Privilege Escalation By Actions 'iam:CreateLoginProfile' Platform: Terraform Severity: <span style="color:#ff7213">Medium</span> Category: Access Control URL: User with privilege escalation by actions 'iam:CreateLoginProfile' and Resource set to '*'. For more information see https://rhinosecuritylabs.com/aws/aws-privilege-escalation-methods-mitigation/.<br> ```tf title="Positive test num. 1 - tf file" hl_lines="1" resource "awsiamuser" "cosmic" { name = "cosmic" } resource "awsiamuserpolicy" "testinline_policy" { name = "testinlinepolicy" user = awsiamuser.cosmic.name policy = jsonencode({ Version = "2012-10-17" Statement = [ { Action = [ "iam:CreateLoginProfile", ] Effect = "Allow" Resource = "*" }, ] }) } resource "awsiampolicy_attachment" "test-attach" { name = "test-attachment" users = [awsiamuser.cosmic.name] roles = [awsiamrole.role.name] groups = [awsiamgroup.group.name] } ``` ```tf title="Negative test num. 1 - tf file" resource "awsiamuser" "cosmic2" { name = "cosmic2" } resource "awsiamuserpolicy" "inlinepolicyruninstances2" { name = "inlinepolicyrun_instances" user = awsiamuser.cosmic2.name policy = jsonencode({ Version = "2012-10-17" Statement = [ { Action = [ "ec2:Describe*", ] Effect = "Allow" Resource = "*" }, ] }) } ```
Project: YugabyteDB (App Definition and Development / Database)
File: ddl_drop_cast.md

title: DROP CAST statement [YSQL] headerTitle: DROP CAST linkTitle: DROP CAST description: Use the DROP CAST statement to remove a cast. menu: v2.18: identifier: ddldropcast parent: statements type: docs Use the `DROP CAST` statement to remove a cast. {{%ebnf%}} drop_cast {{%/ebnf%}} See the semantics of each option in the . Basic example. ```plpgsql yugabyte=# CREATE FUNCTION sqltodate(integer) RETURNS date AS $$ SELECT $1::text::date $$ LANGUAGE SQL IMMUTABLE STRICT; yugabyte=# CREATE CAST (integer AS date) WITH FUNCTION sqltodate(integer) AS ASSIGNMENT; yugabyte=# DROP CAST (integer AS date); ```
Project: Apache Hadoop (App Definition and Development / Database)
File: RELEASENOTES.3.3.5.md

<! --> These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. | Major | Replace all default Charset usage with UTF-8* All of the default charset usages have been replaced to UTF-8. If the default charset of your environment is not UTF-8, the behavior can be different. | Major | Use jersey-json that is built to use jackson2* Use modified jersey-json 1.20 in https://github.com/pjfanning/jersey-1.x/tree/v1.20 that uses Jackson 2.x. By this change, Jackson 1.x dependency has been removed from Hadoop. downstream applications which explicitly exclude jersey from transitive dependencies must now exclude com.github.pjfanning:jersey-json | Major | Slow peer metrics - add median, mad and upper latency limits* Namenode metrics that represent Slownode Json now include three important factors (median, median absolute deviation, upper latency limit) that can help user determine how urgently a given slownode requires manual intervention. | Minor | Improve Magic Committer Performance* S3A filesytem's createFile() operation supports an option to disable all safety checks when creating a file. Consult the documentation and use with care | Minor | Upgrade AWS SDK to V2 - Prerequisites* In preparation for an (incompatible but necessary) move to the AWS SDK v2, some uses of internal/deprecated uses of AWS classes/interfaces are logged as warnings, though only once during the life of a JVM. Set the log "org.apache.hadoop.fs.s3a.SDKV2Upgrade" to only log at INFO to hide these. | Major | Remove the hadoop-openstack module* The swift:// connector for openstack support has been removed. It had fundamental problems (swift's handling of files \> 4GB). A subset of the S3 protocol is now exported by almost all object store services -please use that through the s3a connector instead. The hadoop-openstack jar remains, only now it is empty of code. This is to ensure that projects which declare the JAR a dependency will still have successful builds. | Major | Update Bouncy Castle to 1.68 or later* bouncy castle 1.68+ is a multirelease JAR containing java classes compiled for different target JREs. older versions of asm.jar and maven shade plugin may have problems with these. fix: upgrade the dependencies | Major | Disable abfs prefetching by default* ABFS block prefetching has been disabled to avoid HADOOP-18521 and buffer sharing on multithreaded processes (Hive, Spark etc). This will have little/no performance impact on queries against Parquet or ORC data, but can slow down sequential stream processing, including CSV files -however, the read data will be correct. It may slow down distcp downloads, where the race condition does not arise. For maximum distcp performance re-enable the readahead by setting fs.abfs.enable.readahead to true. | Critical | CryptoOutputStream::close leak when encrypted zones + quota exceptions* WARNING: No release note provided for this change.
Project: JReleaser (App Definition and Development / Application Definition & Image Build)
File: translation.md

name: Translation about: Suggest a translation title: '' labels: 'translation' assignees: '' <!-- Would you like to add a new translation or fix/update an existing translation? --> Please check the translation files you want to add/fix/update: [ ] CLI banner: (plugins/jreleaser/src/main/resources/org/jreleaser/cli/Banner*.properties) [ ] CLI bundle: (plugins/jreleaser/src/main/resources/org/jreleaser/cli/Messages*.properties) [ ] core bundle: (core/jreleaser-utils/src/main/resources/org/jreleaser/bundle/Messages*.properties) A list of language codes can be found at https://sda.berkeley.edu/man41h/localelist.htm
Project: StarRocks (App Definition and Development / Database)
File: any_match.md

displayed_sidebar: "English" Returns whether any elements of an array match the given predicate. Returns `true` (1) if one or more elements match the predicate. Returns `false` (0) if none of the elements matches (a special case is when the array is empty). Returns NULL if the predicate returns NULL for one or more elements and `false` for all other elements. This function is supported from v3.0.6 onwards. ```Haskell anymatch(lambdafunction, arr1, arr2...) ``` Returns whether any elements of `arr1` match the given predicate in the lambda function. `arr1`: the array to match. `arrN`: optional arrays used in the lambda function. `lambda_function`: the lambda function used to match values. Returns a BOOLEAN value. The lambda function follows the usage notes in . If the input array is null or the lambda function results in null, null is returned. If `arr1` is empty, `false` is returned. To apply this function to MAP, rewrite `anymatch((k,v)->k>v,map)` to `anymatch(mapvalues(transformvalues((k,v)->k>v, map)))`. For example, `select anymatch(mapvalues(transform_values((k,v)->k>v, map{2:1})));` returns 1. Check whether any element in `x` is less than the elements in `y`. ```Plain select any_match((x,y) -> x < y, [1,2,8], [4,5,6]); +--+ | any_match((x, y) -> x < y, [1, 2, 8], [4, 5, 6]) | +--+ | 1 | +--+ select any_match((x,y) -> x < y, [11,12,8], [4,5,6]); +-+ | any_match((x, y) -> x < y, [11, 12, 8], [4, 5, 6]) | +-+ | 0 | +-+ select any_match((x,y) -> x < y, [11,12,null], [4,5,6]); +-+ | any_match((x, y) -> x < y, [11, 12, NULL], [4, 5, 6]) | +-+ | NULL | +-+ select any_match((x,y) -> x < y, [], []); ++ | any_match((x, y) -> x < y, [], []) | ++ | 0 | ++ select any_match((x,y) -> x < y, null, [4,5,6]); ++ | any_match((x, y) -> x < y, NULL, [4, 5, 6]) | ++ | NULL | ++ ```
Project: KusionStack (Provisioning / Automation & Configuration)
File: kusion_compile.md

Deprecated: Use 'kusion build' to generate the Intent instead ``` kusion compile [flags] ``` ``` Deprecated ``` ``` -h, --help help for compile ``` ``` --profile string Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex) (default "none") --profile-output string Name of the file to write the profile to (default "profile.pprof") ``` - Kusion is the Platform Orchestrator of Internal Developer Platform
Project: YugabyteDB (App Definition and Development / Database)
File: cmd_show.md

title: SHOW statement [YSQL] headerTitle: SHOW linkTitle: SHOW description: Use the SHOW statement to display the value of a run-time parameter. menu: v2.18: identifier: cmd_show parent: statements type: docs Use the `SHOW` statement to display the value of a run-time parameter. {{%ebnf%}} show_stmt {{%/ebnf%}} The parameter values in YSQL can be set and typically take effect the same way as in PostgreSQL. However, because YugabyteDB uses a different storage engine (), many configurations related to the storage layer will not have the same effect in YugabyteDB as in PostgreSQL. For example, configurations related to connection and authentication, query planning, error reporting and logging, run-time statistics, client connection defaults, and so on, should work as in PostgreSQL. However, configurations related to write ahead log, vacuuming, or replication, may not apply to Yugabyte. Instead related configuration can be set using yb-tserver (or yb-master) . Specify the name of the parameter to be displayed. Show the values of all configuration parameters, with descriptions.
Project: VoltDB (App Definition and Development / Database)
File: StreamToProcedure.md

How to Export Stream data to a VoltDB Procedure =============================================== VoltDB includes an experimental export connector known informally as the VoltDB "Loopback" Export Connector. It processes rows from a STREAM by making a call a procedure within the same VoltDB cluster. The columns from the stream have to match the input parameters of the procedure. One reason this may be useful is chain together procedures that are partitioned on different keys. For example, suppose you have event data that has a device ID, but you need to look up the customer ID in order to update a record for the customer. You could write a client application that receives the event data, looks up the customer_id first, and then updates a record. But if the data is coming from a source system like Kafka, you may be using the Kafka importer so there isn't a custom applicaiton that can call a lookup query. But the importer can call a procedure to do the lookup, and the procedure can insert an enriched record into a stream. Using the loopback export connector, this enriched record can then be passed on to another procedure that is partitioned on the customer id to complete the processing. Note that this is not the same as calling a procedure from a procedure (which VoltDB does not currently support). Each procedure is a separate transaction. The first procedure is committed as soon as it finishes inserting into the stream. The second procedure may not run immediately, and may commit or roll back separately, depending on if it successfully completes. Setting it up is straightforward. In your deployment.xml file, configure an export configuration for each procedure you want to call. The configuration is similar to how custom exporters are configured, except in this case the class (org.voltdb.exportclient.loopback.LoopbackExportClient) is already included in VoltDB's classpath. Example Configuration: <configuration target="callMyProcedure" type="custom" exportconnectorclass="org.voltdb.exportclient.loopback.LoopbackExportClient"> <property name="procedure">MyProcedure</property> <property name="failurelogfile">/var/log/voltdb/MyProcedureFailures.log</property> </configuration> Properties: "procedure" (required) : set to the short procedure name (e.g. "Vote" not "org.voltdb.procedure.Vote"). If you just want to insert into a table, you can specify the table's built-in insert procedure (e.g. TABLENAME.insert) "failurelogfile" (optional) : set this to the path of the filename for a log file into which failed records will be appended. If you specify a relative path, it will be relative to the voltdbroot directory. If you use an absolute path, it will be used exactly.
Project: Apache Hadoop (App Definition and Development / Database)
File: RELEASENOTES.0.21.1.md

<! --> These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. | Major | TestJobOutputCommitter fails in ant test run* Clears a problem that {{TestJobCleanup}} leaves behind files that cause {{TestJobOutputCommitter}} to error out. | Major | Clover build doesn't generate per-test coverage* This fix requires that test coverage is running under Clover 3.0+ | Major | Clover build doesn't generate per-test coverage* This fix requires that test coverage is running under Clover 3.0+ | Major | Clover build doesn't generate per-test coverage* This fix requires that test coverage is running under Clover 3.0+ | Minor | Forrest Documentation for Dynamic Priority Scheduler* Forrest Documentation for Dynamic Priority Scheduler | Major | [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.* I have just committed this to 0.21 and trunk. Thanks Vinay. | Blocker | Context.setStatus() and progress() api are ignored* Moved the api public Counter getCounter(Enum\<?\> counterName), public Counter getCounter(String groupName, String counterName) from org.apache.hadoop.mapreduce.TaskInputOutputContext to org.apache.hadoop.mapreduce.TaskAttemptContext | Major | Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml* Removed references to the older fs.checkpoint.\* properties that resided in core-site.xml | Major | Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml* Removed references to the older fs.checkpoint.\* properties that resided in core-site.xml | Minor | Help message is wrong for touchz command.* Updated the help for the touchz command.
Project: YugabyteDB (App Definition and Development / Database)
File: read-path.md

title: Read I/O path headerTitle: Read I/O path linkTitle: Read I/O path description: Learn how YugabyteDB manages single-row read operations. menu: v2.18: identifier: read-path parent: core-functions weight: 1188 type: docs The read I/O path can be illustrated by an example of a single key read that involves indentifying a tablet leader which then performs a read operation. The user-issued read request interacts with the YQL query layer via a port with the appropriate API (either YSQL or YCQL). This user request is translated by the YQL layer into an internal key, allowing the YQL layer to find the tablet and the YB-TServers hosting it. The YQL layer performs this by making an RPC call to the YB-Master. The response is cached for future uses. Next, the YQL layer issues the read to the YB-TServer that hosts the leader tablet peer. The read is handled by the leader of the Raft group of the tablet owning the internal key. The leader of the tablet Raft group which handles the read request reads from its DocDB and returns the result to the user. As described in , the YugabyteDB smart client can route the application requests directly to the correct YB-TServer, avoiding any extra network hops or master lookups. Suppose there is a requirement to read the value where the primary key column `K` has a value `k` from table `T1`. The table `T1` has a key column `K` and a value column `V`. The following diagram depicts the read flow: The default is strongly-consistent read. The read queries can be quite complex. The YQL query layer has a fully-optimized query engine to handle queries which contain expressions, built-in function calls, and arithmetic operations.
Project: Beam (App Definition and Development / Streaming & Messaging)
File: splitAtFraction-method.md

title: "Dynamic work rebalancing for Beam" date: 2016-05-18 11:00:00 -0700 categories: blog aliases: /blog/2016/05/18/splitAtFraction-method.html authors: dhalperi <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> This morning, Eugene and Malo from the Google Cloud Dataflow team posted . This article discusses Cloud Dataflows solution to the well-known straggler problem. <!--more--> In a large batch processing job with many tasks executing in parallel, some of the tasks -- the stragglers -- can take a much longer time to complete than others, perhaps due to imperfect splitting of the work into parallel chunks when issuing the job. Typically, waiting for stragglers means that the overall job completes later than it should, and may also reserve too many machines that may be underutilized at the end. Cloud Dataflows dynamic work rebalancing can mitigate stragglers in most cases. What Id like to highlight for the Apache Beam (incubating) community is that Cloud Dataflows dynamic work rebalancing is implemented using runner-specific control logic on top of Beams runner-independent . Specifically, to steal work from a straggler, a runner need only call the readers . This will generate a new source containing leftover work, and then the runner can pass that source off to another idle worker. As Beam matures, I hope that other runners are interested in figuring out whether these APIs can help them improve performance, implementing dynamic work rebalancing, and collaborating on API changes that will help solve other pain points.