cwielenb/CNCF-Dataset huggingface dataset:
Project: Pravega (App Definition and Development / Streaming & Messaging)
File: README.md

<!-- Copyright Pravega Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Instructions to generate Server REST API stubs ``` rm -Rf controller/src/main/java/io/pravega/controller/server/rest/generated ``` All REST API modifications should be done by updating the swagger/Controller.yaml specification file. This can be done manually or by using the online editor at http://editor.swagger.io. Download swagger-codegen-cli from maven - https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/2.2.3/swagger-codegen-cli-2.2.3.jar ``` java -jar swagger-codegen-cli.jar generate -i <pravega root>/shared/controller-api/src/main/swagger/Controller.yaml -l jaxrs -c <pravega root>/shared/controller-api/src/main/swagger/server.config.json -o <pravega root>/controller/ ``` All files that get generated outside of the controller/src/main/java/io/pravega/controller/server/rest/generated folder should be deleted and not committed to git. The JAXRS API stubs decorated with swagger annotations are generated in .../server/rest/generated/api/ScopesApi.java class. Copy these API descriptions into interfaces in .../server/rest/v1/ApiV1.java. Also ensure that the APIs in ApiV1.java are modified to use only jersey async interfaces. Swagger codegen truncates common enum prefixes. So until https://github.com/swagger-api/swagger-codegen/issues/4261 is fixed we need to perform the following manual step. In file RetentionConfig.java replace DAYS to LIMITEDDAYS and SIZEMB to LIMITEDSIZEMB. https://jcenter.bintray.com/io/github/swagger2markup/swagger2markup-cli/1.3.3/swagger2markup-cli-1.3.3.jar ``` java -Dswagger2markup.markupLanguage=MARKDOWN -Dswagger2markup.generatedExamplesEnabled=true -jar swagger2markup-cli-1.3.3.jar convert -i <pravega root>/shared/controller-api/src/main/swagger/Controller.yaml -f <pravega root>/documentation/src/docs/rest/restapis ```
Project: StarRocks (App Definition and Development / Database)
File: README.md

stream load import tools ( multi-threading parallel ) , expect memory use in 500M no matter file size First , you need to install jdk , and run the script ```shell ./stream-load-import.sh --url=http://{feip}:{fehttpport}/api/{databasename}/{tablename}/stream_load \ --source_file=/file/path/name.csv \ --H=column_separator:, \ --u=srusername:srpassword ``` Necessary: --source_file: the absolute path of import file --url: the `fe` url , it should contain protocol and so on, not `be` redirect url --u: the starrocks database user and password ,not server Optional: --enabledebug: default false ,`--enabledebug=true` --maxthreads: parallel thread number , default min(servercorenumber,32) ,`--maxthreads=16` --timeout: http protocol connect timeout and process timeout , default 60*1000ms, for example 5s `--timeout=5000` --queue_size: memory queue limit size , default 256 , not reset unless you have a lot of memory and you need reset `-Xmx` --H: http request header , for example `--H=columnseparator:,`,columnseparator as key,`,`as value Other: When appear error a certain thread, other normal thread transaction will not rollback Currently, you need ensure your file not contain error data,and can clear table We are realizing union transaction , it can resolve this problem
Project: GraphScope (App Definition and Development / Database)
File: README.md

GraphScope provides a robust graph loading module to load a graph from a set of files. It also ships with a dataset download module, with pre-configured schemas for known datasets as sample data. Users can always use a function named like `loadXXX` to load a builtin dataset, where the `XXX` represent the name of graph. These functions can be imported by `from graphscope.dataset import loadXXX` in Python. All available datasets are listed in the next section. The signature of the function is `load_XXX(sess=None, prefix=None)`, where `sess` represents the session which the graph is loaded. The default value `None` means the default session(local session). `prefix` assigns the existing files location. By default, users do not need to provide this and `graphscope` will download it from the Internet. For example, we can load the `cora` dataset as a `Graph` in `graphscope` with two lines of code: ```python from graphscope.dataset import load_cora g = load_cora() ``` Note: There are some datasets violate this convention (ogbnmagsmall, ldbc-snb) Currently, the supported graph datasets are listed as below. The list is growing. Hopefully, it would support all datasets from and . You are welcomed to contribute a dataset to GraphScope by submitting a Pull Request. You may following these to add a dataset. Find a popular and appropriate graph data. Reformat and organize it as vertex/edge files. If it is a property graph, make one file for each label of vertices/edges. The columns in vertex files start with the ID, followed by properties. While the columns of edge files start with source vertex ID, destination vertex ID, and followed by properties. e.g., ```shell id,name,score 0,James,95 1,Helen,85 ``` ```shell sender,receiver,msgcount,lastsent 0,1,3,2021-11-23 ``` Put all files into a folder and name it as the graph name. Compress the folder, then upload the compressed file together with the original folder to the dataset folder of the OSS bucket. e.g., assume you have a folder named `foo`, and two files `foo/nodes.csv` and `foo/edge.csv`, the files in the OSS bucket should be organized as (or see ), ```bash dataset |-- foo.tar.gz |-- foo |-- nodes.csv |-- edge.csv ``` Write the loading function load_foo in a new file named `foo.py` inside folder `python/graphscope/dataset/`. (see ) A corresponding unit test is appreciated! See .
Project: Weaviate (App Definition and Development / Database)
File: Readme.md

we use shell to run our (unit/ acceptance / integration) tests ```shell ./test/run.sh ${command} ``` run that command with any of the following available commands, make sure you're in the project root folder. `--unit-only` | `-u` `--unit-and-integration-only` | `-ui` `--integration-only` | `-i` `--acceptance-only` | `-a` `--acceptance-only-fast` | `-aof` `--acceptance-only-graphql` | `-aog` `--acceptance-only-replication` | `-aor` `--acceptance-module-tests-only` | `--modules-only` | `-m` `--acceptance-module-tests-only-backup` | `--modules-backup-only` | `-mob` `--acceptance-module-tests-except-backup` | `--modules-except-backup` | `-meb` `--only-module-{moduleName}`
Project: OrientDB (App Definition and Development / Database)
File: README.md

To run ITs on Kubernetes you need to have kubectl installed, and you must provide a kubeconfig file. Make sure the installed kubectl is with the Kubernetes cluster. Only few ITs can be run on Kubernetes due to development effort, but mostly due to a requied running `OServer` instances. To run the integration tests on Kubernetes you must use the Maven `it-k8s` build profile. Run a single test (e.g. on OrientDB 3.1.3): ``` mvn failsafe:integration-test -Dit.test=BasicSyncIT -P it-k8s -DkubeConfig=/path/to/kubeconfig -DorientdbDockerImage=orientdb:3.1.3 ``` To run all integration tests that can be run on Kubernetes (defined in the build profile): ``` mvn failsafe:integration-test -P it-k8s -DkubeConfig=/path/to/kubeconfig -DorientdbDockerImage=orientdb:3.1.3 ``` To run tests on a specific build, you must build a Docker image and pass it via the `-DorientdbDockerImage=repository/image:tag` flag. The Kubernetes namespace used for the tests, is defined in the build profile and must exist before running the tests. You can use a different namespace by adding `-DkubernetesNamespace=name` to the above commands. The namespace and RBACs created by the tests, are not removed after the tests finish executing. OrientDB Docker images before 3.1.3 do not support node discovery on Kubernetes and therefore do not support this setup. Volume size and storage classes used for the PVs, can also be configured via the `it-k8s` build profile properties. Please note that due to issues that the official Kuberbenetes Java client has with some versions of JDK8 (broken pipe error), you should use JDK9 or later to run the tests. Next steps Use the Kubernetes setup for more sophisticated tests of `distributed`, e.g., add delays.
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: defineconnector.md

```tremor use std::time::nanos; define connector metronome from metronome with config = { "interval": nanos::from_millis(1) } end; ``` Define user defind connector `metronome` from the builtin `metronome` connector using a 1 second periodicity interval. ```tremor define connector exit from exit; ``` Define user dfeind connector `exit` from the builtin `exit` connector with no arguments specified.
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: operatorselect.md

The builtin `select` operator in queries. ```tremor select event from in into out; ``` ```tremor select event from in into out group by event.key; ``` ```tremor select event from in[one_sec] by event.key into out; ``` Multiple windows can be configured in lower resolutions for multi-resolution windowed expressions where lower resolutions are merged into from higher resolution windows ```tremor select aggr::stats::hdr(event.count) form in[onesec, fifteensec, onemin, onehour] into out; ```
Project: YugabyteDB (App Definition and Development / Database)
File: type_int.md

title: Integer and counter data types [YCQL] headerTitle: Integer and counter data types linkTitle: Integer and counter description: There are several different data types for integers of different value ranges. Integers can be set, inserted, incremented, and decremented. menu: v2.18: parent: api-cassandra weight: 1420 type: docs There are several different data types for integers of different value ranges. Integers can be set, inserted, incremented, and decremented while `COUNTER` can only be incremented or decremented. We've extend Apache Cassandra to support increment and decrement operators for integer data types. Datatype | Min | Max | |--|--| `TINYINT` | -128 | 127 | `SMALLINT` | -32,768 | 32,767 | `INT` or `INTEGER` | -2,147,483,648 | 2,147,483,647 | `BIGINT` | 9,223,372,036,854,775,808 | 9,223,372,036,854,775,807 | `COUNTER` | 9,223,372,036,854,775,808 | 9,223,372,036,854,775,807 | `VARINT` | unbounded | unbounded | The following keywords are used to specify a column of type integer for different constraints including its value ranges. ``` type_specification ::= TINYINT | SMALLINT | INT | INTEGER | BIGINT | VARINT | COUNTER integer_literal ::= [ + | - ] digit [ { digit | , } ... ] ``` Columns of type `TINYINT`, `SMALLINT`, `INT`, `INTEGER`, `BIGINT` or `VARINT` can be part of the `PRIMARY KEY`. Values of different integer data types are comparable and convertible to one another. Values of integer data types are convertible but not comparable to floating point number. Values of floating point data types are not convertible to integers. `COUNTER` is an alias of `BIGINT` but has additional constraints. Columns of type `COUNTER` cannot be part of the`PRIMARY KEY`. If a column is of type `COUNTER`, all non-primary-key columns must also be of type `COUNTER`. Column of type `COUNTER` cannot be set or inserted. They must be incremented or decremented. If a column of type `COUNTER` is NULL, its value is replaced with zero when incrementing or decrementing. ```sql example> CREATE TABLE items(id INT PRIMARY KEY, item_count BIGINT); ``` ```sql example> INSERT INTO items(id, item_count) VALUES(1, 1); ``` ```sql example> INSERT INTO items(id, item_count) VALUES(2, 2); ``` ```sql example> UPDATE items SET item_count = 5 WHERE id = 1; ``` ```sql example> UPDATE items SET itemcount = itemcount + 1 WHERE id = 2; ``` ```sql example> SELECT * FROM items; ``` ``` id | item_count -+ 2 | 3 1 | 5 ``` ```sql example> CREATE TABLE itemcounters(id INT PRIMARY KEY, itemcounter COUNTER); ``` For counter type, null values are treated as 0. ```sql example> UPDATE itemcounters SET itemcounter = item_counter + 1 WHERE id = 1; ``` ```sql example> SELECT * FROM item_counters; ``` ``` id | item_counter -+-- 1 | 1 ```
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: matchimut.md

An illustration of the rule is a set of case statements, followed by an optional `default` case ```tremor match event of case %{} => "I am a possibly non-empty record", case %[] => "I am a possibly non-empty array", case %( "snot" ) => "I am a list with 1 element which is the string \"snot\" case _ => "I'm something else" end ```
Project: Timescale (App Definition and Development / Database)
File: README.md

This module implements first level support for gap fill queries, including support for LOCF (last observation carried forward) and interpolation, without requiring to join against `generate_series`. This makes it easier to join timeseries with different or irregular sampling intervals. This introduces a new gapfill customscan node that is inserted above the aggregation node of a query. The node will inject tuples for time intervals without data. The node requires data to be sorted by time, but it will inject sort nodes in the plan to ensure data is sorted correctly if the query order does not match the required order. The timebucketgapfill functions only serves to trigger injecting the gapfill customscan node in the planner all the tuple injecting happens in the gapfill node and timebucketgapfill just calls plain time_bucket. The locf and interpolate function calls serve as markers in the plan to trigger locf or interpolate behaviour. In the targetlist of the gapfill node those functions will be toplevel function calls. The gapfill state transitions are described in gapfill_internal.h Gapfill query ``` SELECT timebucketgapfill(1,time,0,6) AS time, min(value) AS value FROM (values (0,1),(5,6)) v(time,value) GROUP BY 1 ORDER BY 1; ``` Gapfill query with LOCF ``` SELECT timebucketgapfill(1,time,0,6) AS time, locf(min(value)) AS value FROM (values (0,1),(5,6)) v(time,value) GROUP BY 1 ORDER BY 1; ``` Gapfill query with interpolation ``` SELECT timebucketgapfill(1,time,0,6) AS time, interpolate(min(value)) AS value FROM (values (0,1),(5,6)) v(time,value) GROUP BY 1 ORDER BY 1; ```
Project: Tremor (App Definition and Development / Streaming & Messaging)
File: connect.md

Given ```tremor create connector ingest; create connector egress; create pipeline logic; connect /connector/ingress/out to /pipeline/logic/in; connect /pipeline/logic/out to /connector/egress/in; ``` Defines how the `ingress`, `egress` and `logic` runtime instances are interconnected for data to flow through them in a specified order.
Project: OceanBase (App Definition and Development / Database)
File: README.md

If you want to deploy OceanBase on a Linux RPM platform, you can use yum/apt for single-node installation and simple management with systemd. WARNING The installation method is just used for study or test; You should not deploy it with important data as it is not used in production environment. Config yum repo then install OceanBase, it will automatically install the required dependencies. ```bash yum install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo yum install -y oceanbase-ce ``` Config apt repo then install OceanBase, it will automatically install the required dependencies. ```bash apt update apt install -y lsb-release wget gnupg2 mysql-client curl wget http://mirrors.oceanbase.com/oceanbase/oceanbasedeb.pub && apt-key add oceanbasedeb.pub echo "deb http://mirrors.oceanbase.com/oceanbase/community/stable/$(lsbrelease -is | awk '{print tolower($0)}')/$(lsbrelease -cs)/$(dpkg --print-architecture)/ ./" | tee -a /etc/apt/sources.list.d/oceanbase.list apt update apt install -y oceanbase-ce ``` | dependency | version | |-|-| | oceanbase-ce-libs | same version with oceanbase-ce | | jq | / | | oniguruma | / | | curl | / | You can install and run OceanBase service using the following command: ```bash systemctl start oceanbase ``` You can set the OceanBase service to start automatically on boot using the following command: ```bash systemctl enable oceanbase ``` Systemd provides automatic OceanBase startup and shutdown. It also enables manual server management using the systemctl command. For example: ```bash systemctl {start|stop|restart|status} oceanbase ``` Systemd provide `/etc/oceanbase.cnf` to config OceanBase before startup.
Project: Strimzi (App Definition and Development / Streaming & Messaging)
File: README.md

Strimzi Drain Cleaner is a utility which helps with moving the Kafka pods deployed by from Kubernetes nodes which are being drained. It is useful if you want the Strimzi operator to move the pods instead of Kubernetes itself. The advantage of this approach is that the Strimzi operator makes sure that no Kafka topics become under-replicated during the node draining. For more information and installation guide, see .
Project: YugabyteDB (App Definition and Development / Database)
File: test-yedis.md

<!-- +++ private = true +++ --> Initialize YEDIS API in the YugabyteDB universe you just set up by running the following `yb-admin` command. ```sh $ kubectl exec -it yb-master-0 -- /home/yugabyte/bin/yb-admin --masteraddresses yb-master-0.yb-masters.default.svc.cluster.local:7100 setupredis_table ``` ```output ... I0127 19:38:10.358551 115 client.cc:1292] Created table systemredis.redis of type REDISTABLE_TYPE I0127 19:38:10.358872 115 yb-adminclient.cc:400] Table 'systemredis.redis' created. ``` Run `redis-cli` to connect to the service. You can do this as shown below. ```sh $ kubectl exec -it yb-tserver-0 -- /home/yugabyte/bin/redis-cli ``` ```output 127.0.0.1:6379> ``` Run a YEDIS command to verify it is working. ```sql 127.0.0.1:6379> PING ``` ```output "PONG" ```
Project: Qdrant (App Definition and Development / Database)
File: README.md

Hi! This document is our plan for Qdrant development in 2024. Previous year roadmap is available here: Goals of the release: Maintain easy upgrades* - we plan to keep backward compatibility for at least one minor version back (this stays the same in 2024). That means that you can upgrade Qdrant without any downtime and without any changes in your client code within one minor version. Storage should be compatible between any two consequent versions, so you can upgrade Qdrant with automatic data migration between consecutive versions. Make serving easy on multi-billion scale* - Qdrant already can serve billions of vectors cheaply, using techniques as quantization. In the 2024 year, we plan to make it even easier to scale it. Faster and more reliable replications Out-of-the-box read-write segregation Specialized nodes and multi-region deployments Better ecosystem* - in 2023 we introduced to simplify embedding generation but keep it out of the core. In 2024 we plan to continue this trend: implement more advanced and specialized tools while keeping the core focused on the main use-case. Advanced support for sparse vectors - we plan to make sparse vectors inference as fast and easy as the dense one. Hybrid search out of the box with no overhead - something you can build with Qdrant today, but in a more convenient way. Practical RAG - battle-tested RAG practices with production-grade implementation. Various similarity search scenarios* - develop vector similarity beyond just kNN search. If you are a Qdrant user - Data Scientist, ML Engineer, or MLOps, the best contribution would be the feedback on your experience with Qdrant. Let us know whenever you have a problem, face an unexpected behavior, or see a lack of documentation. You can do it in any convenient way - create an , start a , or drop up a . If you use Qdrant or Metric Learning in your projects, we'd love to hear your story! Feel free to share articles and demos in our community. For those familiar with Rust - check out our . If you have problems with code or architecture understanding - reach us at any time. Feeling confident and want to contribute more? - Come to ! Hybrid Search and Sparse Vectors [ ] Make Sparse Vectors serving as cheap and fast as Dense Vectors [ ] Introduce Hybrid Search into Qdrant Client [ ] Dense + Sparse + Fusion in one request [ ] Customizable Re-Ranking Scalability [ ] Faster shard synchronization [ ] Non-blocking snapshotting [ ] Incremental replication [ ] Specialized nodes [ ] Read-only nodes [ ] Indexing nodes [ ] Multi-region deployments [ ] Automatic replication over availability zones Performance [ ] Specialized vector indexing for edge cases HNSW is not good at [ ] Text-index performance and resource consumption improvements [ ] IO optimizations for disk-bound workloads New Data Exploration techniques [ ] Improvements in Discovery API to support more use-cases [ ] Diversity Sampling [ ] Better Aggregations [ ] Advanced text filtering [ ] Phrase queries [ ] Logical operators
Project: YugabyteDB (App Definition and Development / Database)
File: expr_subscript.md

title: Subscripted expressions [YCQL] headerTitle: Subscripted expressions linkTitle: Subscripted expressions description: Use subscripted expressions to access elements in a multi-element value, such as a map collection by using the [] operator. menu: preview: parent: api-cassandra weight: 1340 aliases: /preview/api/cassandra/expr_subscript /preview/api/ycql/expr_subscript type: docs Use subscripted expressions to access elements in a multi-element value, such as a map collection by using operator `. For example, if `ids` refers to a column of type `LIST`, `ids statement. Subscripted expression can only be applied to columns of type `LIST`, `MAP`, or user-defined data types. Subscripting a `LIST` value with a non-positive index will yield NULL. Subscripting a `MAP` value with a non-existing key will yield NULL. Otherwise, it returns the element value that is associated with the given key. Apache Cassandra does not allow subscripted expression in the select list of the SELECT statement. ```sql ycqlsh:yugaspace> CREATE TABLE t(id INT PRIMARY KEY,yugamap MAP<TEXT, TEXT>); ``` ```sql ycqlsh:yugaspace> UPDATE yugatab SET mapvalue['keyvalue'] = 'yuga_string' WHERE id = 7; ```
Project: Doris (App Definition and Development / Database)
File: README.md

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> These scripts are used to make tpc-h test. follow the steps below: ./bin/build-tpch-dbgen.sh If the build failed in dbgen tools' compilation, update your GCC version or change download link used by wget in build-tpch-dbgen.sh from ".../TPC-HToolsv3.0.0new.zip" to ".../TPC-HToolsv3.0.0.zip" ./bin/gen-tpch-data.sh -s 1 ./bin/create-tpch-tables.sh -s 1 ./bin/load-tpch-data.sh ./bin/run-tpch-queries.sh -s 1 NOTICE: At present, Doris's query optimizer and statistical information functions are not complete, so we rewrite some queries in TPC-H to adapt to Doris' execution framework, but it does not affect the correctness of the results. The rewritten SQL is marked with "Modified" in the corresponding .sql file. A new query optimizer will be released in subsequent releases. Currently, differnt scales use the same suite of query sqls.