cwielenb/CNCF-Dataset huggingface dataset:
Project: Flink (App Definition and Development / Streaming & Messaging)
File: conversion_of_pandas.md

title: "Conversions between PyFlink Table and Pandas DataFrame" weight: 41 type: docs aliases: /dev/python/table-api-users-guide/conversionofpandas.html <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> PyFlink Table API supports conversion between PyFlink Table and Pandas DataFrame. Pandas DataFrames can be converted into a PyFlink Table. Internally, PyFlink will serialize the Pandas DataFrame using Arrow columnar format on the client. The serialized data will be processed and deserialized in Arrow source during execution. The Arrow source can also be used in streaming jobs, and is integrated with checkpointing to provide exactly-once guarantees. The following example shows how to create a PyFlink Table from a Pandas DataFrame: ```python from pyflink.table import DataTypes import pandas as pd import numpy as np pdf = pd.DataFrame(np.random.rand(1000, 2)) table = tenv.frompandas(pdf) table = tenv.frompandas(pdf, ['f0', 'f1']) table = tenv.frompandas(pdf, [DataTypes.DOUBLE(), DataTypes.DOUBLE()]) table = tenv.frompandas(pdf, DataTypes.ROW([DataTypes.FIELD("f0", DataTypes.DOUBLE()), DataTypes.FIELD("f1", DataTypes.DOUBLE())])) ``` PyFlink Tables can additionally be converted into a Pandas DataFrame. The resulting rows will be serialized as multiple Arrow batches of Arrow columnar format on the client. The maximum Arrow batch size is configured via the option . The serialized data will then be converted to a Pandas DataFrame. Because the contents of the table will be collected on the client, please ensure that the results of the table can fit in memory before calling this method. You can limit the number of rows collected to client side via {{< pythondoc file="pyflink.table.html#pyflink.table.Table.limit" name="Table.limit">}} The following example shows how to convert a PyFlink Table to a Pandas DataFrame: ```python from pyflink.table.expressions import col import pandas as pd import numpy as np pdf = pd.DataFrame(np.random.rand(1000, 2)) table = tenv.frompandas(pdf, ["a", "b"]).filter(col('a') > 0.5) pdf = table.limit(100).to_pandas() ```
Project: Podman (App Definition and Development / Application Definition & Image Build)
File: podman-untag.1.md

% podman-untag 1 podman\-untag - Remove one or more names from a locally-stored image podman untag image [name[:tag]...] podman image untag image [name[:tag]...] Remove one or more names from an image in the local storage. The image can be referred to by ID or reference. If no name is specified, all names are removed from the image. If a specified name is a short name and does not include a registry, `localhost/` is prefixed (e.g., `fedora` -> `localhost/fedora`). If a specified name does not include a tag, `:latest` is appended (e.g., `localhost/fedora` -> `localhost/fedora:latest`). Print usage statement Remove all tags from the specified image. ``` $ podman untag 0e3bbc2 ``` Remove tag from specified image. ``` $ podman untag imageName:latest otherImageName:latest ``` Remove multiple tags from the specified image. ``` $ podman untag httpd myhttpd myregistryhost:5000/fedora/httpd:v2 ``` December 2019, Originally compiled by Sascha Grunert <sgrunert@suse.com>
Project: Beam (App Definition and Development / Streaming & Messaging)
File: 13_advanced_splittable_dofn.md

Prompt: What is a splittable `DoFn` in Apache Beam? Response: A splittable `DoFn` (SDF) is a generalization of that lets you process elements in a non-monolithic way. A splittable `DoFn` makes it easier to create complex, modular I/O connectors in Apache Beam. When you apply a splittable `DoFn` to an element, the runner can split the elements processing into smaller tasks. You can checkpoint the processing of an element, and you can split the remaining work to yield additional parallelism. At a high level, an SDF is responsible for processing element and restriction pairs. A restriction represents a subset of work that would have been necessary to have been done when processing the element. Executing a uses the following steps: Each element is paired with a restriction. For example, filename is paired with offset range representing the whole file. Each element and restriction pair is split. For example, offset ranges are broken up into smaller pieces. The runner redistributes the element and restriction pairs to several workers. Element and restriction pairs are processed in parallel. For example, the file is read. Within this last step, the element and restriction pair can pause its own processing or be split into further element and restriction pairs. For an example, see the in the Tour of Beam. For more information, see the .
Project: deepstream (App Definition and Development / Streaming & Messaging)
File: README.md

deepstream is an open source server inspired by concepts behind financial trading technology. It allows clients and backend services to sync data, send messages and make rpcs at very high speed and scale. ](https://badge.fury.io/js/%40deepstream%2Fserver)](https://hub.docker.com/r/deepstreamio/deepstream.io/) deepstream has three core concepts for enabling realtime application development records () records are schema-less, persistent documents that can be manipulated and observed. Any change is synchronized with all connected clients and backend processes in milliseconds. Records can reference each other and be arranged in lists to allow modelling of relational data events () events allow for high performance, many-to-many messaging. deepstream provides topic based routing from sender to subscriber, data serialisation and subscription listening. rpcs () remote procedure calls allow for secure and highly available request response communication. deepstream handles load-balancing, failover, data-transport and message routing. security ( and ) deepstream offers a combination of different authentication mechanisms with a powerful permission-language called Valve that allows you to specify which user can perform which action with which data. deepstream development is a great way to get into building performant Node.js applications, and contributions are always welcome with lots of . Contributing to deepstream is as simple as having Node.js (10+) and TypeScript (3+) installed, cloning the repo and making some changes. ``` ~ git clone git@github.com:deepstreamIO/deepstream.io.git ~ cd deepstream.io ~/deepstream.io git submodule update --init ~/deepstream.io npm i ~/deepstream.io npm start | | _ _| |_ _ _ / ` |/ \/ \ ' \/ | | '/ \/ ` | ' ` \ | (| | / / |) \ \ |_| | | / (_| | | | | | | \,_|\|\| ./|_/\|| \|\,|| || || |_| ===================== starting ===================== ``` From here you can make your changes, and check the unit tests pass: ``` ~/deepstream.io npm t ``` If your changes are substantial you can also run our extensive end-to-end testing framework: ``` ~/deepstream.io npm run e2e ``` For power users who want to make sure the binary works, you can run `sh scripts/package.sh true`. You'll need to download the usual build environment for this to work and we only support the latest LTS version to compile. This step is usually not needed though unless your modifying resource files or changing dependencies.
Project: KICS (Provisioning / Security & Compliance)
File: ad5b4e97-2850-4adf-be17-1d293e0b85ee.md

title: Glue Security Configuration Encryption Disabled hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: ad5b4e97-2850-4adf-be17-1d293e0b85ee Query name: Glue Security Configuration Encryption Disabled Platform: Terraform Severity: <span style="color:#bb2124">High</span> Category: Encryption URL: Glue Security Configuration Encryption should have 'cloudwatchencryption', 'jobbookmarksencryption' and 's3encryption' enabled<br> ```tf title="Positive test num. 1 - tf file" hl_lines="5" resource "awsgluesecurity_configuration" "positive1" { name = "example" encryption_configuration { cloudwatch_encryption { cloudwatchencryptionmode = "SSE-KMS" } jobbookmarksencryption { jobbookmarksencryption_mode = "CSE-KMS" kmskeyarn = data.awskmskey.example.arn } s3_encryption { kmskeyarn = data.awskmskey.example.arn s3encryptionmode = "SSE-KMS" } } } ``` ```tf title="Positive test num. 2 - tf file" hl_lines="11" resource "awsgluesecurity_configuration" "positive2" { name = "example" encryption_configuration { cloudwatch_encryption { cloudwatchencryptionmode = "SSE-KMS" kmskeyarn = data.awskmskey.example.arn } jobbookmarksencryption { jobbookmarksencryption_mode = "DISABLED" kmskeyarn = data.awskmskey.example.arn } s3_encryption { kmskeyarn = data.awskmskey.example.arn s3encryptionmode = "SSE-KMS" } } } ``` ```tf title="Positive test num. 3 - tf file" hl_lines="10" resource "awsgluesecurity_configuration" "positive2" { name = "example" encryption_configuration { cloudwatch_encryption { cloudwatchencryptionmode = "SSE-KMS" kmskeyarn = data.awskmskey.example.arn } jobbookmarksencryption { kmskeyarn = data.awskmskey.example.arn } s3_encryption { kmskeyarn = data.awskmskey.example.arn s3encryptionmode = "SSE-KMS" } } } ``` ```tf title="Negative test num. 1 - tf file" resource "awsgluesecurity_configuration" "negative1" { name = "example" encryption_configuration { cloudwatch_encryption { cloudwatchencryptionmode = "SSE-KMS" kmskeyarn = data.awskmskey.example.arn } jobbookmarksencryption { jobbookmarksencryption_mode = "CSE-KMS" kmskeyarn = data.awskmskey.example.arn } s3_encryption { kmskeyarn = data.awskmskey.example.arn s3encryptionmode = "SSE-KMS" } } } ```
Project: Meshery (Provisioning / Automation & Configuration)
File: oras.md

layout: integration title: Oras subtitle: Collaborative and visual infrastructure as code for Operator image: /assets/img/integrations/oras/icons/color/oras-color.svg permalink: extensibility/integrations/oras docURL: https://docs.meshery.io/extensibility/integrations/oras description: integrations-category: Cloud Native Storage integrations-subcategory: Cloud Native Storage registrant: Artifact Hub components: featureList: [ "Drag-n-drop cloud native infrastructure designer to configure, model, and deploy your workloads.", "Invite anyone to review and make changes to your private designs.", "Ongoing synchronization of Kubernetes configuration and changes across any number of clusters." ] howItWorks: "Collaborative Infrastructure as Code" howItWorksDetails: "Collaboratively manage infrastructure with your coworkers synchronously sharing the same designs." language: en list: include type: extensibility category: integrations
Project: Glasskube (App Definition and Development / Application Definition & Image Build)
File: roadmap.md

title: Roadmap Our next and previous are managed on GitHub and are the single source of truth. Our pre releases are technical proof of concepts that aim to inspire technical folks and will give you a way to try our latest development snapshot where we ship features fast. | | Features | Timeline | Status | |-|:--:|::|:--:| | v0.0.1 (Alpha) | - first working PackageOperator<br/>- first working client (UI & CLI)<br/>- install first packages (cert-manager, ) <br/>- `bootstrap` command | Released on Jan 31 | Released | | v0.0.2 | - `open` command<br/>- real-time-updates<br/>- add more supported packages | Released on Feb 09 | Released | | v0.0.3 | - add package updates and outdated information<br/>- support version pinning | Released on Feb 27 | Released | | v0.1.0 | - support packages with dependencies<br/>- dark mode | Released on Mar 21 | Released | | v0.2.0 | - package configuration | Released on Apr 18 | Released | | v0.3.0 (Beta) | - markdown support in package long description | Released on Apr 25 | Released | Our v1.0.0 milestone is something we will combine with a very special launch on multiple platforms. Until then multiple features will be needed. Inclusive, but not limited to: GitOps integration including a GitHub bot that is able to provide valuable insights Package backups and restore functionality Custom private package repositories Package comments and ratings Database support and multi instance support Support for the Glasskube Apps Operator (see below) This should have contained all features we think are necessary for `glasskube` to become the best package manager for Kubernetes and the community can publish their packages via Glasskube. The supports the following Apps and will be part in one of our releases until we launch: Gitea GitLab GlitchTip Keycloak Matomo Metabase Nextcloud Odoo Plane Vault Becoming a stable software from a point where interfaces, manifests, and functionality remain unchanged, typically requires significant time, a dedicated community and a big user base. As we are working towards this goal we will need to iterate fast and things will break. However, already from the beginning we want to provide stable automated upgrade paths so your packages dont break! All planned and already supported packages can be found on our . Can't find a package or want your app included in the list? We are always adding new supported packages & apps, so just join us on or open up a new issue and let us know what is missing!
Project: KICS (Provisioning / Security & Compliance)
File: c4dcdcdf-10dd-4bf4-b4a0-8f6239e6aaa0.md

title: SSH Access Is Not Restricted hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: c4dcdcdf-10dd-4bf4-b4a0-8f6239e6aaa0 Query name: SSH Access Is Not Restricted Platform: Terraform Severity: <span style="color:#ff7213">Medium</span> Category: Networking and Firewall URL: Google Firewall should not allow SSH access (port 22) from the Internet (public CIDR block) to ensure the principle of least privileges<br> ```tf title="Positive test num. 1 - tf file" hl_lines="43 13 31" resource "googlecomputefirewall" "positive1" { name = "test-firewall" network = googlecomputenetwork.default.name direction = "INGRESS" source_ranges = ["0.0.0.0/0"] allow { protocol = "icmp" } allow { protocol = "tcp" ports = ["22", "80", "3389", "8080", "1000-2000"] } source_tags = ["web"] } resource "googlecomputefirewall" "positive2" { name = "test-firewall" network = googlecomputenetwork.default.name source_ranges = ["0.0.0.0/0"] allow { protocol = "icmp" } allow { protocol = "tcp" ports = ["80", "8080", "1000-2000","21-3390"] } source_tags = ["web"] } resource "googlecomputefirewall" "positive3" { name = "test-firewall" network = googlecomputenetwork.default.name source_ranges = ["0.0.0.0/0"] allow { protocol = "icmp" } allow { protocol = "all" } source_tags = ["web"] } ``` ```tf title="Negative test num. 1 - tf file" resource "googlecomputefirewall" "negative1" { name = "test-firewall" network = googlecomputenetwork.default.name allow { protocol = "icmp" } allow { protocol = "tcp" ports = ["80", "8080", "1000-2000"] } source_tags = ["web"] } ```
Project: YDB (App Definition and Development / Database)
File: order_by.md

Sorting the `SELECT` result using a comma-separated list of sorting criteria. As a criteria, you can use a column value or an expression on columns. Ordering by column sequence number is not supported (`ORDER BY N`where `N` is a number). Each criteria can be followed by the sorting direction: `ASC`: Sorting in the ascending order. Applied by default. `DESC`: Sorting in the descending order. Multiple sorting criteria will be applied left-to-right. Example ```yql SELECT key, string_column FROM my_table ORDER BY key DESC, LENGTH(string_column) ASC; ``` {% if featurewindowfunctions %} You can also use `ORDER BY` for . {% endif %}
Project: Apache Hadoop (App Definition and Development / Database)
File: RELEASENOTES.0.19.1.md

<! --> These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. | Major | Large number of decommission freezes the Namenode* Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval. | Blocker | Memory leak ?* fix memory leak of user/group information in fuse-dfs | Blocker | RPC Server can leave a lot of direct buffers* Improve how RPC server reads and writes large buffers. Avoids soft-leak of direct buffers and excess copies in NIO layer. | Major | fair share scheduler does not utilize all slots if the task trackers are configured heterogeneously* HADOOP-4943: Fixed fair share scheduler to utilize all slots when the task trackers are configured heterogeneously. | Blocker | TaskTracker running out of memory after running several tasks* Fix the tasktracker for OOM exception by sharing the jobconf properties across tasks of the same job. Earlier a new instance was held for each task. With this fix, the job level configuration properties are shared across tasks of the same job. | Blocker | 2 core tests TestFileOutputFormat and TestHarFileSystem are failing in branch 19* This patch solves the null pointer exception issue in the 2 core tests TestFileOutputFormat and TestHarFileSystem in branch 19. | Blocker | A spurious IOException log on DataNode is not completely removed* Minor : HADOOP-3678 did not remove all the cases of spurious IOExceptions logged by DataNode. | Major | NameNode should send both replication and deletion requests to DataNode in one reply to a heartbeat* This patch changes the DatanodeProtocoal version number from 18 to 19. The patch allows NameNode to send both block replication and deletion request to a DataNode in response to a heartbeat. | Major | libhdfs does not call FileSystem.append when O\_APPEND passed to hdfsOpenFile* libhdfs supports O\_APPEND flag | Blocker | workaround for tmp file handling on DataNodes in 0.19.1 (HADOOP-4663)* Work around for tmp file handling. sync() does not work as a result. | Blocker | Disable append* HDFS append() is disabled. It throws UnsupportedOperationException.
Project: KICS (Provisioning / Security & Compliance)
File: da9f3aa8-fbfb-472f-b5a1-576127944218.md

title: Audit Log Maxage Not Properly Set hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: da9f3aa8-fbfb-472f-b5a1-576127944218 Query name: Audit Log Maxage Not Properly Set Platform: Kubernetes Severity: <span style="color:#edd57e">Low</span> Category: Observability URL: When using kube-apiserver command, the '--audit-log-maxage' flag should be defined and set to 30 or more days<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="11" apiVersion: v1 kind: Pod metadata: name: command-demo labels: purpose: demonstrate-command spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: ["--audit-log-maxage=26"] restartPolicy: OnFailure ``` ```yaml title="Positive test num. 2 - yaml file" hl_lines="11" apiVersion: v1 kind: Pod metadata: name: command-demo labels: purpose: demonstrate-command spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: [] restartPolicy: OnFailure ``` ```yaml title="Positive test num. 3 - yaml file" hl_lines="40 27 12 55" apiVersion: serving.knative.dev/v1 kind: Service metadata: name: dummy namespace: knative-sequence spec: template: spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: [] restartPolicy: OnFailure apiVersion: serving.knative.dev/v1 kind: Configuration metadata: name: dummy-config namespace: knative-sequence spec: template: spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: [] restartPolicy: OnFailure apiVersion: serving.knative.dev/v1 kind: Revision metadata: name: dummy-rev namespace: knative-sequence spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: [] restartPolicy: OnFailure apiVersion: sources.knative.dev/v1 kind: ContainerSource metadata: name: dummy-cs namespace: knative-sequence spec: template: spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: [] restartPolicy: OnFailure ``` ```yaml title="Negative test num. 1 - yaml file" apiVersion: v1 kind: Pod metadata: name: command-demo labels: purpose: demonstrate-command spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver"] args: ["--audit-log-maxage=30"] restartPolicy: OnFailure ``` ```yaml title="Negative test num. 2 - yaml file" apiVersion: v1 kind: Pod metadata: name: command-demo labels: purpose: demonstrate-command spec: containers: name: command-demo-container image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 command: ["kube-apiserver","--audit-log-maxage=35"] args: [] restartPolicy: OnFailure ```
Project: KusionStack (Provisioning / Automation & Configuration)
File: kusion_workspace_delete.md

Delete a workspace This command deletes a specified workspace. ``` kusion workspace delete ``` ``` kusion workspace delete dev ``` ``` -h, --help help for delete ``` ``` --profile string Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex) (default "none") --profile-output string Name of the file to write the profile to (default "profile.pprof") ``` - Workspace is a logical concept representing a target that stacks will be deployed to