cwielenb/CNCF-Dataset huggingface dataset:
Project: Numaflow (App Definition and Development / Streaming & Messaging)
File: user-defined-sources.md

A `Pipeline` may have multiple Sources, those sources could either be a pre-defined source such as `kafka`, `http`, etc., or a `user-defined source`. With no source data transformer, A pre-defined source vertex runs single-container pods; a user-defined source runs two-container pods. You can build your own user-defined sources in multiple languages. Check the links below to see the examples for different languages. After building a docker image for the written user-defined source, specify the image as below in the vertex spec. ```yaml spec: vertices: name: input source: udsource: container: image: my-source:latest ``` Some environment variables are available in the user-defined source container: `NUMAFLOW_NAMESPACE` - Namespace. `NUMAFLOW_POD` - Pod name. `NUMAFLOW_REPLICA` - Replica index. `NUMAFLOWPIPELINENAME` - Name of the pipeline. `NUMAFLOWVERTEXNAME` - Name of the vertex. If you're looking for examples and usages contributed by the open source community, head over to . These user-defined sources like AWS SQS, GCP Pub/Sub, provide valuable insights and guidance on how to use and write a user-defined source.
Project: Hazelcast Jet (App Definition and Development / Streaming & Messaging)
File: off-heap.md

title: Off-Heap Data Structures description: Utilize off-heap memory in Jet id: version-4.5-off-heap original_id: off-heap Jet runs inside a Java Virtual Machine which uses garbage collected heap memory. Depending on heap size, version of the JVM and the specific garbage collector used garbage collection can suddenly create large pauses in the application, leading to unpredictable spikes in latency and drops in throughput. To avoid this, Jet supports storing data in `IMap` in what's called native or off-heap memory using the feature of Hazelcast. This allows JVM to operate with smaller heaps and do garbage collection more efficiently, while allowing Jet to store large amount of data. Currently off-heap memory support in Jet is limited to data stored in `IMap`, the state for running jobs are still in on-heap data structures. You can configure an `IMap` to use off-heap memory by configuring `hazelcast.yaml`. First, you need to dedicate a specific portion of the physical memory available to off-heap and then enable off-heap memory for the specific maps. The node needs to have enough free physical memory for this feature to work. For example, to dedicate `128GB` of memory for off-heap and to enable it for the `IMap` with name `huge-map`, you can make the following configuration changes in `hazelcast.yaml`: ```yaml hazelcast: native-memory: enabled: true size: unit: GIGABYTES value: 128 map: huge-map: in-memory-format: NATIVE ``` For additional configuration options, see section of the Hazelcast manual. Off-heap memory can also make use of persistent memory technologies such as . Depending on the value size, this offers almost similar performance to memory at a lower cost, making it attractive for storing large amounts of data efficiently. Please refer to the section of the Hazelcast manual for details on how to enable persistent memory.
Project: KICS (Provisioning / Security & Compliance)
File: b5681959-6c09-4f55-b42b-c40fa12d03ec.md

title: IAM User Policy Without MFA hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: b5681959-6c09-4f55-b42b-c40fa12d03ec Query name: IAM User Policy Without MFA Platform: Terraform Severity: <span style="color:#edd57e">Low</span> Category: Insecure Configurations URL: Check if the root user is authenticated with MFA<br> ```tf title="Positive test num. 1 - tf file" hl_lines="18" resource "awsiamuser" "positive1" { name = "root" path = "/system/" tags = { tag-key = "tag-value" } } resource "awsiamaccess_key" "positive2" { user = awsiamuser.lb.name } resource "awsiamuser_policy" "positive3" { name = "test" user = awsiamuser.lb.name policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:root" }, "Action": "sts:AssumeRole" } ] } EOF } ``` ```tf title="Negative test num. 1 - tf file" resource "awsiamuser" "negative1" { name = "root" path = "/system/" tags = { tag-key = "tag-value" } } resource "awsiamaccess_key" "negative2" { user = awsiamuser.lb.name } resource "awsiamuser_policy" "negative3" { name = "test" user = awsiamuser.lb.name policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:root" }, "Action": "sts:AssumeRole", "Condition": { "BoolIfExists": { "aws:MultiFactorAuthPresent" : "true" } } } ] } EOF } resource "awsiamuser_policy" "negative4" { name = "test" user = awsiamuser.lb.name policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::mfa/111122223333:root" }, "Action": "sts:AssumeRole" } ] } EOF } ```
Project: KICS (Provisioning / Security & Compliance)
File: 75418eb9-39ec-465f-913c-6f2b6a80dc77.md

title: RDP Access Is Not Restricted hide: toc: true navigation: true <style> .highlight .hll { background-color: #ff171742; } .md-content { max-width: 1100px; margin: 0 auto; } </style> Query id: 75418eb9-39ec-465f-913c-6f2b6a80dc77 Query name: RDP Access Is Not Restricted Platform: Ansible Severity: <span style="color:#bb2124">High</span> Category: Networking and Firewall URL: Check if the Google compute firewall allows unrestricted RDP access. Allowed ports should not contain RDP port 3389<br> ```yaml title="Positive test num. 1 - yaml file" hl_lines="8 29" name: rdpinrange google.cloud.gcpcomputefirewall: name: test_object source_ranges: "0.0.0.0/0" allowed: ip_protocol: tcp ports: "22" "80" "8080" "2000-4000" target_tags: test-ssh-server staging-ssh-server source_tags: test-ssh-clients project: test_project auth_kind: serviceaccount serviceaccountfile: "/tmp/auth.pem" state: present name: rdpinport google.cloud.gcpcomputefirewall: name: test_object source_ranges: "0.0.0.0/0" allowed: ip_protocol: tcp ports: "22" "80" "3389" target_tags: test-ssh-server staging-ssh-server source_tags: test-ssh-clients project: test_project auth_kind: serviceaccount serviceaccountfile: "/tmp/auth.pem" state: present ``` ```yaml title="Negative test num. 1 - yaml file" name: create a firewall google.cloud.gcpcomputefirewall: name: test_object allowed: ip_protocol: tcp ports: '80' target_tags: test-ssh-server staging-ssh-server source_tags: test-ssh-clients project: test_project auth_kind: serviceaccount serviceaccountfile: /tmp/auth.pem state: present ```
Project: StarRocks (App Definition and Development / Database)
File: ADMIN_SHOW_REPLICA_DISTRIBUTION.md

displayed_sidebar: "English" This statement is used to show the distribution status of a table or a partition replica. :::tip This operation requires the SYSTEM-level OPERATE privilege. You can follow the instructions in to grant this privilege. ::: ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM [dbname.]tblname [PARTITION (p1, ...)] ``` Note: The Graph column in the result shows the distribution ratio of replicas graphically. View the replica distribution of tables ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM tbl1; ``` View the replica distribution of partitions in the table ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM db1.tbl1 PARTITION(p1, p2); ```
Project: Meshery (Provisioning / Automation & Configuration)
File: v0.4.15.md

layout: release date: 2020-10-23 tag: v0.4.15 General Trying to get the signoff check this time @alonsolopezr (#1724) Added SMP Logo in ReadMe @Aarush-Goyal (#1648) [mesheryctl]Attempt to start docker daemon during mesheryctl installation @hexxdump (#1685) Changed lifecycle method for props update @dhruv0000 (#1722) fix(UI): Persist Grafana charts after a Reloading @dhruv0000 (#1721) [Adapter] Remove NGINX SM Adapter from Stable release channel pending adapter beta release @leecalcote (#1714) Add experimental subcommand to mesheryctl @anirudhjain75 (#1715) nginx adapter changes without the svgs @kumarabd (#1705) [mesheryctl] #1708 version command add indicator to user @cardinalion (#1710) [mesheryctl] Fix command provides invalid upgrade prompt #1707 @cardinalion (#1709) Features [Docs] feature update @chandrashritii (#1719) Documentation [Docs] feature update @chandrashritii (#1719)
Project: ClickHouse (App Definition and Development / Database)
File: v23.9.2.56-stable.md

sidebar_position: 1 sidebar_label: 2023 Backported in : Resource with source code including submodules is built in Darwin special build task. It may be used to build ClickHouse without checkouting submodules. (). Backported in : Solve issue with launching standalone clickhouse-keeper from clickhouse-server package. (). Backported in : Fix integration check python script to use gh api url - Add Readme for CI tests. (). Fix deadlock in LDAP assigned role update (). Fix for background download in fs cache (). Fix functions execution over sparse columns (). Fix incorrect merging of Nested for SELECT FINAL FROM SummingMergeTree (). Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy (). Fix SIGSEGV in MergeSortingPartialResultTransform (due to zero chunks after remerge()) (). Fix data-race in CreatingSetsTransform (on errors) due to throwing shared exception (). Fix trash optimization (up to a certain extent) (). Fix parsing of arrays in cast operator (). Fix filtering by virtual columns with OR filter in query (). Fix MongoDB connection issues (). Destroy fiber in case of exception in cancelBefore in AsyncTaskExecutor (). Fix crash in QueryNormalizer with cyclic aliases (). Fix filtering by virtual columns with OR filter in query (resubmit) (). Backported in :. (). Backported in :. (). Add setting allowexperimentalpartial_result (). Fix CI skip build and skip tests checks (). check if block is empty after async insert retries (). MaterializedPostgreSQL: remove back check (). Review and partially revert it (). Remove existing moving/ dir if allowremovestalemovingparts is off (). Bump curl to 8.4 ().
Project: Dgraph (App Definition and Development / Database)
File: readme.md

In Dgraph, we have the abiltity to specify an @index directive for the type vfloat. Here, we want to expand that capability to allow us to specify something like: myAttribute @index("hnsw-euclidian","maxNeighbors":"5","maxLevels":"5","exponent":"6") Roughly, this should be read as: I want to specify an HNSW-type vector index using euclidian distance as a metric with maxNeighbors, maxLevels, and exponent being declared as written, and with all other parameters taking on default values. (In the case of specifying an exponent, this should affect the defaults for the other options, but that is something that can be decided by the factory for the hnsw index)! But this leads to some natural questions: How do I know what option types are allowed? And how do I interpret their types? For example, how do I know that I should interpret "5" as an integer rather than a string? The solution will be to allow each factory to specify a set of "allowed option types". Hence, we get the AllowedOptions class, which specifies a set of named options and corresponding parsers for the given name. Now, if we want to populate an Options instance based on the content of myAttribute @index(hnsw(metric:"euclidian",exponent:"6")), we collect the key-value pairs: "metric":"euclidian" "exponent":"6" And we can now invoke: allowedOpts := hnswFactory.AllowedOpts() myAttributeIndexOpts := NewOptions() val, err := allowedOpts.GetParsedOption("exponent", "6") if err != nil { return ErrBadOptionNoBiscuit } myAttributeIndexOpts.SetOpt("exponent", val) The final resolution of the "puzzle" in fact is to invoke allowedOpts.PopulateOptions(pairs, myAttributeIndexOpts) based on pairs being built from the collection of key-value pairs.
Project: Pilosa (App Definition and Development / Database)
File: README.md

This module provisions a VPC, subnets, instances, keys, and security groups needed for a basic featurebase cluster running in AWS. It is meant to be used as a module. For example: ```hcl module "featurebase" { source "/path/to/module/" cluster_prefix = "sprockets" azs = ["us-east-1a", "us-east-1b", "us-east-1c"] } ``` The path to the module is wherever the `featurebase-cloud` directory is. So if you have put it in `/var/opt/terraform/modules/featurebase-cloud` then calling the module would look like: ```hcl module "featurebase" { source "/var/opt/terraform/modules/featurebase-cloud" cluster_prefix = "sprockets" } ``` Much more is configurable; for a complete list, look in `variables.tf`. Reasonable defaults have been set. Please make sure you have set up your AWS access in either environment variables, or in the credentials file. Some useful links for this are: AWS Environment Variables <https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html> State is currently kept locally, for as this is intended for PoCs. It can be stored in a remote s3 or GCS bucket if desired.
Project: ClickHouse (App Definition and Development / Database)
File: sampling-query-profiler.md

slug: /ru/operations/optimizing-performance/sampling-query-profiler sidebar_position: 54 sidebar_label: Query Profiling ClickHouse runs sampling profiler that allows analyzing query execution. Using profiler you can find source code routines that used the most frequently during query execution. You can trace CPU time and wall-clock time spent including idle time. To use profiler: Setup the section of the server configuration. This section configures the system table containing the results of the profiler functioning. It is configured by default. Remember that data in this table is valid only for a running server. After the server restart, ClickHouse doesnt clean up the table and all the stored virtual memory address may become invalid. Setup the or settings. Both settings can be used simultaneously. These settings allow you to configure profiler timers. As these are the session settings, you can get different sampling frequency for the whole server, individual users or user profiles, for your interactive session, and for each individual query. The default sampling frequency is one sample per second and both CPU and real timers are enabled. This frequency allows collecting enough information about ClickHouse cluster. At the same time, working with this frequency, profiler doesnt affect ClickHouse servers performance. If you need to profile each individual query try to use higher sampling frequency. To analyze the `trace_log` system table: Install the `clickhouse-common-static-dbg` package. See . Allow introspection functions by the setting. For security reasons, introspection functions are disabled by default. Use the `addressToLine`, `addressToSymbol` and `demangle` to get function names and their positions in ClickHouse code. To get a profile for some query, you need to aggregate data from the `trace_log` table. You can aggregate data by individual functions or by the whole stack traces. If you need to visualize `trace_log` info, try and . In this example we: Filtering `trace_log` data by a query identifier and the current date. Aggregating by stack trace. Using introspection functions, we will get a report of: Names of symbols and corresponding source code functions. Source code locations of these functions. <!-- --> ``` sql SELECT count(), arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n ', addressToLine(x)), trace), '\n') AS sym FROM system.trace_log WHERE (queryid = 'ebca3574-ad0a-400a-9cbc-dca382f5998c') AND (eventdate = today()) GROUP BY trace ORDER BY count() DESC LIMIT 10 ``` ``` text {% include "examples/samplingqueryprofiler_result.txt" %} ```
Project: Buildpacks (App Definition and Development / Application Definition & Image Build)
File: bug.md

name: Bug about: Bug report title: '' labels: status/triage, type/bug assignees: '' <! Please provide a general summary of the issue. --> <! What steps should be taken to reproduce the issue? --> <! What actually happened? --> <! What did you expect to happen? --> <! Run `pack report` and copy output here. --> <! Run `docker info` and copy output here. -->
Project: Podman (App Definition and Development / Application Definition & Image Build)
File: build_osx.md

The following describes the process for building the Podman client on macOS. Podman requires brew -- a package manager for macOS. This will allow additional packages to be installed that are needed by Podman. See the for installation instructions. Podman requires some software from brew to be able to build. This can be done using brew from a macOS terminal: ``` $ brew install go go-md2man ``` You can obtain the latest source code for Podman from its github repository. ``` $ git clone https://github.com/containers/podman go/src/github.com/containers/podman ``` After completing the preparatory steps of obtaining the Podman source code and installing its dependencies, the client can now be built. ``` $ cd go/src/github.com/containers/podman $ make podman-remote $ mv bin/darwin/podman bin/podman ``` The binary will be located in bin/ ``` $ ls -l bin/ ``` Recent podman builds depend on a `gvproxy` binary which comes from . A common development scenario may be using the podman desktop app as a baseline, with a development binary of `podman` you build from git. To ensure that the podman you build here can find the gvproxy installed from podman desktop, use: `make podman-remote HELPERBINARIESDIR=/opt/podman/bin` (Also note that because the `Makefile` rules do not correctly invalidate the binary when this variable changes, so if you already have a build you'll need to `rm bin/darwin/podman` first if you have an existing build). Alternatively, you can set `helperbinariesdir=` in `~/.config/containers/containers.conf`. If you would like to build the docs associated with Podman on macOS: ``` $ make podman-remote-darwin-docs $ ls docs/build/remote/darwin ``` To install and view these manpages: ``` $ cp -a docs/build/remote/darwin/* /usr/share/man/man1 $ man podman ``` To learn how to use the Podman client, refer to its .
Project: ArangoDB (App Definition and Development / Database)
File: has_exception.md

+++ title = "`bool has_exception() const noexcept`" description = "Always returns false for `basic_result`. Constexpr, never throws." categories = ["observers"] weight = 593 +++ Always returns false for `basic_result`. Constexpr where possible. Requires: Always available. Complexity: Constant time. Guarantees: Never throws an exception.
Project: YugabyteDB (App Definition and Development / Database)
File: yba_universe_node_reprovision.md

Reprovision a node instance in YugabyteDB Anywhere universe Reprovision a node instance in YugabyteDB Anywhere universe. Re-provision node with already stopped processes. ``` yba universe node reprovision [flags] ``` ``` -h, --help help for reprovision ``` ``` -a, --apiToken string YugabyteDB Anywhere api token. --config string Config file, defaults to $HOME/.yba-cli.yaml --debug Use debug mode, same as --logLevel debug. --disable-color Disable colors in output. (default false) -H, --host string YugabyteDB Anywhere Host (default "http://localhost:9000") -l, --logLevel string Select the desired log level format. Allowed values: debug, info, warn, error, fatal. (default "info") -n, --name string [Required] The name of the universe for the corresponding node operations. --node-name string [Optional] The name of the universe node for the corresponding node operations. Required for add, reboot, release, remove, reprovision, start, stop. -o, --output string Select the desired output format. Allowed values: table, json, pretty. (default "table") --timeout duration Wait command timeout, example: 5m, 1h. (default 168h0m0s) --wait Wait until the task is completed, otherwise it will exit immediately. (default true) ``` - Manage YugabyteDB Anywhere universe nodes
Project: Chef InSpec (Provisioning / Security & Compliance)
File: google_compute_backend_bucket.md

+++ title = "googlecomputebackend_bucket resource" draft = false gh_repo = "inspec" platform = "gcp" [menu] [menu.inspec] title = "googlecomputebackend_bucket" identifier = "inspec/resources/gcp/googlecomputebackendbucket.md googlecomputebackendbucket resource" parent = "inspec/resources/gcp" +++ A `googlecomputebackend_bucket` is used to test a Google BackendBucket resource This resource has beta fields available. To retrieve these fields, include `beta: true` in the constructor for the resource ```ruby describe googlecomputebackend_bucket(project: 'chef-gcp-inspec', name: 'inspec-gcp-backend-bucket') do it { should exist } its('description') { should eq 'Backend bucket example' } its('enable_cdn') { should be 'true' } its('bucket_name') { should eq 'gcp-inspec-storage-bucket' } end describe googlecomputebackend_bucket(project: 'chef-gcp-inspec', name: 'nonexistent') do it { should_not exist } end ``` Properties that can be accessed from the `googlecomputebackend_bucket` resource: `bucket_name` : Cloud Storage bucket name. `cdn_policy` : Cloud CDN configuration for this Backend Bucket. `signedurlcachemaxage_sec` : Maximum number of seconds the response to a signed URL request will be considered fresh. After this time period, the response will be revalidated before being served. When serving responses to signed URL requests, Cloud CDN will internally behave as though all responses from this backend had a "Cache-Control: public, max-age=[TTL]" header, regardless of any existing Cache-Control header. The actual headers served in responses will not be altered. `creation_timestamp` : Creation timestamp in RFC3339 text format. `description` : An optional textual description of the resource; provided by the client when the resource is created. `enable_cdn` : If true, enable Cloud CDN for this BackendBucket. `id` : Unique identifier for the resource. `name` : Name of the resource. Provided by the client when the resource is created. The name must be 1-63 characters long, and comply with RFC1035. Specifically, the name must be 1-63 characters long and match the regular expression `?` which means the first character must be a lowercase letter, and all following characters must be a dash, lowercase letter, or digit, except the last character, which cannot be a dash. Ensure the is enabled for the current project.
Project: KICS (Provisioning / Security & Compliance)
File: integrations_teamcity.md

You can integrate KICS into TeamCity pipelines easily by and KICS docker image. This provide you the ability to run KICS scans as a stage of your pipeline. Just make sure that the has Docker configured to run containers. Checkout and modify you `.teamcity/settings.kts` as the example: ```kotlin import jetbrains.buildServer.configs.kotlin.v2019_2.* import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.script version = "2021.1" project { description = "KICS TeamCity integration test" buildType(Build) } object Build: BuildType({ name = "KICS TeamCity integration test" vcs { root(DslContext.settingsRoot) } steps { script { scriptContent = """ docker run -v ${'$'}PWD:/path checkmarx/kics:latest scan -p /path -o /path --no-progress --ignore-on-exit results """.trimIndent() } } }) ``` With enabled, after pushing the changes to the repository you'll be able to see the build progress, logs and success status. <img src="https://raw.githubusercontent.com/Checkmarx/kics/master/docs/img/teamcity-scan-logs.png" width="850"> <img src="https://raw.githubusercontent.com/Checkmarx/kics/master/docs/img/teamcity-success-status.png" width="850">
Project: Hexa (Provisioning / Security & Compliance)
File: ROADMAP.md

Hexa software has two main components, the policy admin app and the orchestrator. Future enhancements can be associated with one or both of these components. The following list captures the initial roadmap items that will be prioritized and this list will, of course, evolve over time as new use cases and requirements are explored. Google Cloud's Identity Aware Proxy: version 1 is complete Microsoft Azure: version 1 is complete Amazon Web Services Snowflake Database Versa SASE F5 NGINX