cwielenb/CNCF-Dataset huggingface dataset:
Project: CockroachDB (App Definition and Development / Database)
File: README.md

This directory contains unit tests for the script `release-notes.py`. The provided `Makefile` runs all tests. To run a single test invoke with: ``` bash ./testN.sh $PWD/../release-notes.py ``` Optionally, set the `PYTHON` env var to the path of a python 3 interpreter (useful e.g. if the base `python` command is not v3). If the script changes and the reference output is not accurate any more, you can re-generate a reference output as follows: ``` bash ./testN.sh $PWD/../release-notes.py rewrite ``` (i.e. provide one more positional argument)
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: testexpr.md

```tremor json|| ``` The rule identifies an extractor by name and delimits any micro-format arguments using the '|' pipe symbol. In the above example the extractor is a JSON recognizer that can detect well formed JSON embedded inside a string literal. Such values will match the test expression and be parsed so that the content can be used in logic expressions.
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: fndefn.md

Functions defined with an `of` keyword in their signature use pattern matching against arguments ```tremor fn fib_(a, b, n) of case (a, b, n) when n > 0 => recur(b, a + b, n - 1) case _ => a end; ``` Functions defined with a `with` keyword in their signature use ordinary arity based matching. ``` fn fib(n) with fib_(0, 1, n) end; ``` In modular functions, it is customary to provide user level documentation for the intended users of a function. Here is an example from the tremor standard library ```tremor use std::type; use std::binary; use std::array; use std::string; fn isvalid(traceid) of case(traceid) when type::isstring(trace_id) => { "kind": "string", "valid": traceid != "00000000000000000000000000000000" and string::bytes(traceid) == 32, "value": trace_id } case(traceid) when type::isbinary(trace_id) => let arr = binary::intobytes(traceid); { "kind": "binary", "valid": binary::len(arr) == 16 and traceid != << 0:64, 0:64 >>, "value": traceid } case(traceid) when type::isarray(trace_id) => { "kind": "array", "valid": array::len(arr) == 16 and traceid != [ 0, 0, 0, 0, 0, 0, 0, 0], "value": traceid } case _ => false end ```
Project: MongoDB (App Definition and Development / Database)
File: README.md

Meson build system for zstandard ================================ Meson is a build system designed to optimize programmer productivity. It aims to do this by providing simple, out-of-the-box support for modern software development tools and practices, such as unit tests, coverage reports, Valgrind, CCache and the like. This Meson build system is provided with no guarantee and maintained by Dima Krasner \<dima@dimakrasner.com\>. It outputs one `libzstd`, either shared or static, depending on `default_library` option. `cd` to this meson directory (`build/meson`) ```sh meson setup -Dbinprograms=true -Dbincontrib=true builddir cd builddir ninja # to build ninja install # to install ``` You might want to install it in staging directory: ```sh DESTDIR=./staging ninja install ``` To configure build options, use: ```sh meson configure ``` See .
Project: Pachyderm (App Definition and Development / Streaming & Messaging)
File: README.md

The consistent hashing library implements a basic `Ring` that allows users to manage locks in ETCD across multiple processes modelled while handling a dynamic number of Nodes and lock rebalancing. Users should call `WithRing()` to create a scope where a ring can be used within a provided callback function. Once instantiated, a user can `lock` keys in the callback function by calling `Lock()` and release them with `Unlock()`. If a node is added or removed from a ring, each ring instance determines whether it needs to rebalance its own locks using a hash function `HashFn()`. Consistent hashing rings are used in cases where multiple nodes should not attempt to use a given resource, but the number of resources and nodes is dynamic. Because nodes can be removed or added at any time, using consistent hashing enables a user to minimize the number of resources that need to be reassigned when the number of nodes changes. It also provides an elegant way to model resources without having to track all resources across all nodes. In a normal use case, a ring instance is created by each process. In our case that is a container running on Kubernetes. Ring instances that share a `Prefix` watch keys in ETCD that share that prefix for changes by launching an ETCD watcher in a separate goroutine. A Ring has 1 or more `members`. Each ring instance creates a local member called a `node`. Once a new member is detected, rings that share its prefix add its metadata to their `members`. This includes the member's `Id` and its `Hash`, which is used by the consistent hashing algorithm to determine how to distribute keys and to rebalance them if a member is added or removed. Each ring instance also tracks the `locks` that are associated to its node via the ring's hash function. Nodes run a goroutine that refreshes their lease in ETCD which acts as a liveness check. If a lease expires, a delete event is generated, triggering each ring instance to remove that member from their list of members. Deleting a ring instance shuts down its watch and refresh goroutines. Attempting to lock a key is a blocking operation. When attempting to lock a key, the ring hashes the key to determine whether its node associates to the key. If so, the ring calls lock on a mutex for that key. Otherwise, it re-polls the output of the hash function until the call to Lock() is cancelled or until its node associates to key. This can happen when nodes are removed from a ring. When new nodes are added, each ring instance rebalances its node's locks by rehashing the lock keys and checking whether those locks still associate to the same nodes. Unlock also determines whether a lock associates to the ring instance's node. If so, the mutex is unlocked and removed from the set of locks.
Project: ClickHouse (App Definition and Development / Database)
File: materialized-postgresql.md

slug: /en/engines/table-engines/integrations/materialized-postgresql sidebar_position: 130 sidebar_label: MaterializedPostgreSQL Creates ClickHouse table with an initial data dump of PostgreSQL table and starts replication process, i.e. executes background job to apply new changes as they happen on PostgreSQL table in the remote PostgreSQL database. :::note This table engine is experimental. To use it, set `allowexperimentalmaterializedpostgresqltable` to 1 in your configuration files or by using the `SET` command: ```sql SET allowexperimentalmaterializedpostgresqltable=1 ``` ::: If more than one table is required, it is highly recommended to use the database engine instead of the table engine and use the `materializedpostgresqltables_list` setting, which specifies the tables to be replicated (will also be possible to add database `schema`). It will be much better in terms of CPU, fewer connections and fewer replication slots inside the remote PostgreSQL database. ``` sql CREATE TABLE postgresqldb.postgresqlreplica (key UInt64, value UInt64) ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresqltable', 'postgresuser', 'postgrespassword') PRIMARY KEY key; ``` Engine Parameters `host:port` PostgreSQL server address. `database` Remote database name. `table` Remote table name. `user` PostgreSQL user. `password` User password. The setting must have a value `logical` and `maxreplicationslots` parameter must have a value at least `2` in the PostgreSQL config file. A table with `MaterializedPostgreSQL` engine must have a primary key the same as a replica identity index (by default: primary key) of a PostgreSQL table (see ). Only database ) is allowed. The `MaterializedPostgreSQL` table engine only works for PostgreSQL versions >= 11 as the implementation requires the PostgreSQL function. `_version` Transaction counter. Type: . `_sign` Deletion mark. Type: . Possible values: `1` Row is not deleted, `-1` Row is deleted. These columns do not need to be added when a table is created. They are always accessible in `SELECT` query. `_version` column equals `LSN` position in `WAL`, so it might be used to check how up-to-date replication is. ``` sql CREATE TABLE postgresqldb.postgresqlreplica (key UInt64, value UInt64) ENGINE = MaterializedPostgreSQL('postgres1:5432', 'postgresdatabase', 'postgresqlreplica', 'postgresuser', 'postgrespassword') PRIMARY KEY key; SELECT key, value, version FROM postgresqldb.postgresql_replica; ``` :::note Replication of values is not supported. The default value for the data type will be used. :::
Project: EMQ Technologies (App Definition and Development / Streaming & Messaging)
File: README.md

This script starts two EMQX nodes and a influxdb server in docker container. The bootstraping rule engine and data bridge config is provided in influx-bridge.conf which got included in the bootstraping config bundle emqx.conf. ./start.sh The rule and bridge are configured to pipe data from MQTT topic `t/#` to the 'myvalues' measurement in the 'mqtt' bucket. Start the cluster Send mqtt messages to topic `/t/a` with a JSON object as MQTT paylaod like `{"value": 1}` Observe data in influxdb `curl -k -H 'Authorization: Token abcdefg' -G 'https://localhost:8086/query?pretty=true' --data-urlencode "db=mqtt" --data-urlencode "q=SELECT from myvalues"` Example output the curl query against influxdb: ``` {"results":[{"statement_id":0,"series":[{"name":"myvalues","columns":["time","clientid","value"],"values":[["2023-02-28T11:13:29.039Z","a1",123]]}]}] ```
Project: YugabyteDB (App Definition and Development / Database)
File: session-audit-logging-ysql.md

title: Session-level audit logging in YSQL headerTitle: Session-level audit logging in YSQL linkTitle: Session-level audit logging description: Session-level audit logging in YSQL. menu: v2.18: identifier: session-audit-logging-1-ysql parent: audit-logging weight: 760 type: docs <ul class="nav nav-tabs-alt nav-tabs-yb"> <li > <a href="../session-audit-logging-ysql/" class="nav-link active"> <i class="icon-postgres" aria-hidden="true"></i> YSQL </a> </li> </ul> Session logging is enabled for per user session basis. Enable session logging for all DML and DDL statements and log all relations in DML statements. ```sql set pgaudit.log = 'write, ddl'; set pgaudit.log_relation = on; ``` Enable session logging for all commands except MISC and raise audit log messages as NOTICE. In this example session audit logging is used for logging DDL and SELECT statements. Note that the insert statement is not logged because the WRITE class is not enabled. SQL statements are shown below. Open the YSQL shell (ysqlsh), specifying the `yugabyte` user and prompting for the password. ```sh $ ./ysqlsh -U yugabyte -W ``` When prompted for the password, enter the yugabyte password. You should be able to log in and see a response similar to the following: ```output ysqlsh (11.2-YB-2.5.0.0-b0) Type "help" for help. yugabyte=# ``` Enable `pgaudit` extension on the YugabyteDB cluster. ```sql \c yugabyte yugabyte; CREATE EXTENSION IF NOT EXISTS pgaudit; ``` Enable session audit logging in YugabyteDB cluster. ```sql set pgaudit.log = 'read, ddl'; ``` ```sql create table account ( id int, name text, password text, description text ); insert into account (id, name, password, description) values (1, 'user1', 'HASH1', 'blah, blah'); select * from account; ``` You should output similar to the following in the logs: ```output 2020-11-09 19:19:09.262 UTC [3710] LOG: AUDIT: SESSION,1,1,DDL,CREATE TABLE,TABLE,public.account,"create table account ( id int, name text, password text, description text );",<not logged> 2020-11-09 19:19:19.619 UTC [3710] LOG: AUDIT: SESSION,2,1,READ,SELECT,,,"select * from account;",<not logged> ```
Project: Seata (App Definition and Development / Database)
File: Readme.md

rm client -> server ``` RegisterRMRequest MergedWarpMessage BranchRegisterRequest BranchReportRequest GlobalLockQueryRequest ``` tm client -> server ``` RegisterTMRequest MergedWarpMessage GlobalBeginRequest GlobalCommitRequest GlobalRollbackRequest GlobalStatusRequest GlobalReportRequest ``` server -> rm client ``` BranchCommitRequest BranchRollbackRequest UndoLogDeleteRequest ``` server -> tm client ``` // null ``` Server -> rm client ``` RegisterRMResponse MergeResultMessage BranchRegisterResponse BranchReportResponse GlobalLockQueryResponse ``` Server -> tm client ``` RegisterTMResponse MergeResultMessage GlobalBeginResponse GlobalCommitResponse GlobalReportResponse GlobalRollbackResponse ``` rm client -> server ``` BranchCommitResponse BranchRollbackResponse ``` tm client -> server ``` // null ```
Project: TDengine (App Definition and Development / Database)
File: README.md

Error-bounded Lossy Data Compressor For Float/Double TSZ algorithm is come from SZ algorithm, Github url is https://github.com/szcompressor. Bellow is aspect of improvement : 1) Better speed and size SZ head size about 24 bytes, we are reduced to 2 bytes. we delete some no use code and some unnecessary function could be droped. 2) Support multi-threads, interface is thread-safety. 3) Remove 2D 3D 4D 5D function, only 1D be remained. 4) Remove int8 int16 int32 and other datatype, only float double be remained. 5) Optimize code speed 6) Other optimize... After modify, TSZ become fastersmaller and independent. TSZ more suitable for small block data compression. ADT-FSE is an optimization algorithm of TSZ based on timeseries database scenario, which can provide higher compression rate and faster compression and decompression speed for small files. Usage: By default, ADT-FSE algorithm is enabled. Control it by `ifAdtFse` configuration. Citation: ``` @inproceedings{ adtfse_sc2023, author = {Tao Lu, Yu Zhong, Zibin Sun, Xiang Chen, You Zhou, Fei Wu, Ying Yang, Yunxin Huang, and Yafei Yang}, title = {ADT-FSE: A New Encoder for SZ}, year = {2023}, publisher = {IEEE Press}, booktitle = {Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis}, location = {Denver, Colorado}, series = {SC'23} } ```
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: path.md

Path expressions are how events and in memory state in tremor are referenced in scripts. The tremor runtime can provide and consume metadata with the events being processed by the runtime. Metadata is distinguished by a `$` dollar sign ```tremor let foo = $request # The local path `foo` is a copy of the metadata `$request` ``` Connectors such as `kafka` and `http` can generate metadata that scripts, queries and pipelines can manipulate and process to tune tremor's runtime behaviour. The current event streaming through tremor in the current pipeline will be available to queries, logics and scripts via the `event` keyword. The `event` keyword can be further dereferenced via path statements ```tremor let foo = event.snot; # The local path `foo` is a copy of the `snot` field from the current event ``` ```tremor let foo = event[10]; The local path `foo` is a copy of the 10th element of the current event. ``` Scripts in tremor can store state that is available for the lifetime of a pipeline via the `state` keyword. The `state` keyword can be further dereferenced via path statements ```tremor let foo = state.snot; # The local path `foo` is a copy of the `snot` field from the state record ``` ```tremor let foo = state[10]; The local path `foo` is a copy of the 10th element of the state array For operators and structures that support arguments the `args` keyword can be used to dereference values via path statements. ```tremor let foo = args.snot; # The local path `foo` is a copy of the `snot` field from the args record ``` Args are nominal and always record values in tremor. Operations supporting windows and groups can dereference the cached state via the `window` and `group` keywords which both support path operations.
Project: EDB (App Definition and Development / Database)
File: license.md

The MIT License (MIT) Copyright (c) 2014 Mitchell Hashimoto Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. EALINGS IN THE SOFTWARE.
Project: Redpanda (App Definition and Development / Streaming & Messaging)
File: README.md

The `Dockerfile` available in this folder defines the contents of the image published to <https://hub.docker.com/r/vectorized/redpanda-toolchain>. To build Redpanda using this image: ```bash docker pull vectorized/redpanda-toolchain cd redpanda/ docker run --rm -ti -v $PWD:$PWD:Z -w $PWD vectorized/redpanda-toolchain ./build.sh ``` You can add the `--ipc=host` flag to share the host's `/dev/shm` folder, so that the `CCACHE_DIR` folder (`/dev/shm/redpanda` by default) can be available across `docker run` invocations. In systems with SELinux enabled, the `:Z` bind-mount option might cause the container to take too long to setup (due to SELinux labels being applied to individual files). To avoid this, you can add the `--privileged` flag, and that will instruct Docker (or Podman) to skip applying SELinux profiles. ```bash docker run \ --rm -ti --privileged \ --ipc=host \ -v $PWD:$PWD:Z \ -w $PWD vectorized/redpanda-toolchain \ ./build.sh ``` If you need rebuild the toolchain image locally: ```bash DOCKER_BUILDKIT=1 docker build \ -t vectorized/redpanda-toolchain \ -f tools/docker/Dockerfile \ . ``` If errors occur, when tests run during build, try the following: ```bash sudo sysctl -w fs.aio-max-nr=10485760 ```
Project: YugabyteDB (App Definition and Development / Database)
File: type_uuid.md

title: UUID data type [YSQL] headerTitle: UUID data type linkTitle: UUID description: Represents Universally Unique Identifiers (UUIDs). menu: v2.18: identifier: api-ysql-datatypes-uuid parent: api-ysql-datatypes type: docs The `UUID` data type represents Universally Unique Identifiers (UUIDs). A UUID is a sequence of 32 hexadecimal digits separated by hyphens (8 digits - 4 digits - 4 digits - 4 digits - 12 digits) representing the 128 bits. ```ebnf type_specification ::= UUID ``` ```output ffffffff-ffff-ffff-ffff-ffffffffffff {aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa} 12341234-1234-1234-1234-123412341234 ```
Project: Apache StreamPipes (App Definition and Development / Streaming & Messaging)
File: documentation.md

<!-- ~ Licensed to the Apache Software Foundation (ASF) under one or more ~ contributor license agreements. See the NOTICE file distributed with ~ this work for additional information regarding copyright ownership. ~ The ASF licenses this file to You under the Apache License, Version 2.0 ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. ~ --> <p align="center"> <img src="icon.png" width="150px;" class="pe-image-documentation"/> </p> Image Enrichment: Crops an + image based on + given bounding box coordinates An image and an array with bounding boxes. A box consists of the x and y coordinates in the image as well as the height and width A new event for each box containing the cropped image
Project: ShardingSphere (App Definition and Development / Database)
File: hint.en.md

+++ title = "Hint" weight = 1 +++ Apache ShardingSphere uses `ThreadLocal` to manage primary database routing marks for mandatory routing. A primary database routing mark can be added to `HintManager` through programming, and this value is valid only in the current thread. `Hint` is mainly used to perform mandatory data operations in the primary database for read/write splitting scenarios. Call `HintManager.getInstance()` to obtain HintManager instance. Call `HintManager.setWriteRouteOnly()` method to set the primary database routing marks. Execute SQL statements to complete routing and execution. Call `HintManager.close()` to clear the content of ThreadLocal. The same as sharding based on hint. Use `hintManager.setWriteRouteOnly` to configure primary database route. The same as data sharding based on hint. ```java String sql = "SELECT * FROM t_order"; try (HintManager hintManager = HintManager.getInstance(); Connection conn = dataSource.getConnection(); PreparedStatement preparedStatement = conn.prepareStatement(sql)) { hintManager.setWriteRouteOnly(); try (ResultSet rs = preparedStatement.executeQuery()) { while (rs.next()) { // ... } } } ``` The same as sharding based on hint. Use `hintManager.setDataSourceName` to configure database route. ```java String sql = "SELECT * FROM t_order"; try (HintManager hintManager = HintManager.getInstance(); Connection conn = dataSource.getConnection(); PreparedStatement preparedStatement = conn.prepareStatement(sql)) { hintManager.setDataSourceName("ds_0"); try (ResultSet rs = preparedStatement.executeQuery()) { while (rs.next()) { // ... } } } ``` h `try with resource`. ```java // Sharding database and table with HintManager String sql = "SELECT * FROM t_order"; try (HintManager hintManager = HintManager.getInstance(); Connection conn = dataSource.getConnection(); PreparedStatement preparedStatement = conn.prepareStatement(sql)) { hintManager.addDatabaseShardingValue("t_order", 1); hintManager.addTableShardingValue("t_order", 2); try (ResultSet rs = preparedStatement.executeQuery()) { while (rs.next()) { // ... } } } // Sharding database and one database route with HintManager String sql = "SELECT * FROM t_order"; try (HintManager hintManager = HintManager.getInstance(); Connection conn = dataSource.getConnection(); PreparedStatement preparedStatement = conn.prepareStatement(sql)) { hintManager.setDatabaseShardingValue(3); try (ResultSet rs = preparedStatement.executeQuery()) { while (rs.next()) { // ... } } } ```