cwielenb/CNCF-Dataset huggingface dataset:
Project: Ceph (Runtime / Cloud Native Storage)
File: storage_class.md

This directory contains an example `storage_class.lua` on how to use to read and write the Storage Class field of a put request. Create Zonegroup placement info for a Storage Class (QLCCLASS in this example) and point class to a data pool (qlcpool in this example) NOTE: RGW will need restarted due to the Zonegroup placement info change. See: https://docs.ceph.com/en/latest/radosgw/placement/#zonegroup-zone-configuration for more information. ```bash ./bin/radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class QLC_CLASS ./bin/radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class QLCCLASS --data-pool qlcpool ``` Restart radosgw for Zone/ZoneGroup placement changes to take effect. Upload the script: ```bash ./bin/radosgw-admin script put --infile=storage_class.lua --context=preRequest ``` Create a bucket and put and object with a Storage Class header (no modification will occur): ```bash aws --profile=ceph --endpoint=http://localhost:8000 s3api create-bucket --bucket test-bucket aws --profile=ceph --endpoint=http://localhost:8000 s3api put-object --bucket test-bucket --key truv-0 --body ./64KiB_object.bin --storage-class STANDARD ``` Send a request without a Storage Class header (Storage Class will be changed to QLC_CLASS by Lua script): ```bash aws --profile=ceph --endpoint=http://localhost:8000 s3api put-object --bucket test-bucket --key truv-0 --body ./64KiB_object.bin ``` NOTE: If you use s3cmd instead of aws command-line, s3cmd adds "STANDARD" StorageClass to any put request so the example Lua script will not modify it. Verify S3 object had its StorageClass header added ```bash grep Lua ceph/build/out/radosgw.8000.log 2021-11-01T17:10:14.048-0400 7f9c7f697640 20 Lua INFO: Put_Obj with StorageClass: 2021-11-01T17:10:14.048-0400 7f9c7f697640 20 Lua INFO: No StorageClass for Object and size >= threshold: truv-0 adding QLC StorageClass ``` Lua 5.3
Project: HwameiStor (Runtime / Cloud Native Storage)
File: RELEASING.md

Update go.mod for submodules to depend on the new release which will happen in the next step. Run the pre-release script. It creates a branch `prerelease<new tag>` that will contain all release changes. ``` ./pre_release.sh -t <new tag> ``` Verify the changes. ``` git diff main ``` This should have changed the version for all modules to be `<new tag>`. Update the . Make sure all relevant changes for this release are included and are in language that non-contributors to the project can understand. To verify this, you can look directly at the commits since the `<last tag>`. ``` git --no-pager log --pretty=oneline "<last tag>..HEAD" ``` Move all the `Unreleased` changes into a new section following the title scheme (`[<new tag>] - <date of release>`). Update all the appropriate links at the bottom. Push the changes to upstream and create a Pull Request on GitHub. Be sure to include the curated changes from the in the description. Once the Pull Request with all the version changes has been approved and merged it is time to tag the merged commit. *IMPORTANT*: It is critical you use the same tag that you used in the Pre-Release step! Failure to do so will leave things in a broken state. *IMPORTANT*: . It is critical you make sure the version you push upstream is correct. . Run the tag.sh script using the `<commit-hash>` of the commit on the main branch for the merged Pull Request. ``` ./tag.sh <new tag> <commit-hash> ``` Push tags to the upstream remote (not your fork: `github.com/open-telemetry/opentelemetry-go.git`). Make sure you push all sub-modules as well. ``` git push upstream <new tag> git push upstream <submodules-path/new tag> ... ``` Finally create a Release for the new `<new tag>` on GitHub. The release body should include all the release notes from the Changelog for this release. Additionally, the `tag.sh` script generates commit logs since last release which can be used to supplement the release notes. After releasing verify that examples build outside of the repository. ``` ./verify_examples.sh ``` The script copies examples into a different directory removes any `replace` declarations in `go.mod` and builds them. This ensures they build with the published release, not the local copy. Once verified be sure to that uses this release.
Project: WasmEdge Runtime (Runtime / Container Runtime)
File: server_side_render.md

Traditional web applications follows the client-server model. In the past era of application servers, the entire UI is dynamically generated from the server. The browser is simply a thin client that displays the rendered web pages at real time. However, as the browser becomes more capable and sophisticated, the client can now take on more workload to improve application UX, performance, and security. That gives rise to the era of Jamstack. There is now a clear separation between frontend and backend services. The frontend is a static web site (HTML + JavaScript + WebAssembly) generated from UI frameworks such as React.js, Vue.js, Yew or Percy, and the backend consists of microservices. Yet, as Jamstack gains popularity, the diversity of clients (both browsers and apps) makes it very difficult to achieve great performance across all use cases. The solution is server-side rendering (SSR). That is to have edge servers run the "client side" UI code (ie the React generated JavaScript OR Percy generated WebAssembly), and send back the rendered HTML DOM objects to the browser. In this case, the edge server must execute the exact same code (i.e. and WebAssembly) as the browser to render the UI. That is called isomorphic Jamstack applications. The WasmEdge runtime provides a lightweight, high performance, OCI complaint, and polyglot container to run all kinds of SSR functions on edge servers. Vue JS SSR function (coming soon) Yew Rust compiled to WebAssembly SSR function (coming soon) We also exploring ways to render more complex UI and interactions on WasmEdge-based edge servers, and then stream the rendered results to the client application. Potential examples include Render Unity3D animations on the edge server (based on ) Render interactive video (generated from AI) on the edge server Of course, the edge cloud could grow well beyond SSR for UI components. It could also host high-performance microservices for business logic and serverless functions. Read on to the next chapter.
Project: rkt (Runtime / Container Runtime)
File: metadata-service.md

The metadata service is designed to help running apps introspect their execution environment and assert their pod identity. In particular, the metadata service exposes the contents of the pod and image manifests as well as a convenient method of looking up annotations. Finally, the metadata service provides a pod with cryptographically verifiable identity. The metadata service is implemented by the `rkt metadata-service` command. When started, it will listen for registration events over Unix socket on `/run/rkt/metadata-svc.sock`. For systemd-based distributions, it also supports . If using socket activation, ensure the socket is named `/run/rkt/metadata-svc.sock`, as `rkt run` uses this name during registration. Please note that when started under socket activation, the metadata service will not remove the socket on exit. Use the `RemoveOnStop` directive in the relevant `.socket` file to clean up. Example systemd unit files for running the metadata service are available in . In addition to listening on a Unix socket, the metadata service will also listen on a TCP port 2375. When contacting the metadata service, the apps utilize this port. The IP and port of the metadata service are passed by rkt to pods via the `ACMETADATAURL` environment variable. See for more information about the metadata service including a list of supported endpoints and their usage. | Flag | Default | Options | Description | | | | | | | `--listen-port` | `18112` | A port number | Listen port | See the table with .
Project: Alluxio (Runtime / Cloud Native Storage)
File: Advanced-Tuning.md

layout: global title: FUSE SDK Advanced Tuning Alluxio now supports both libfuse2 and libfuse3. Alluxio FUSE on libfuse2 is more stable and has been tested in production. Alluxio FUSE on libfuse3 is currently experimental but under active development. Alluxio will focus more on libfuse3 and utilize new features provided. libfuse3 is used by default. Set to use libfuse2 via: ```shell $ sudo yum install fuse $ alluxio-fuse mount understoragedataset mount_point -o fuse=2 ``` See `logs/fuse.out` for which version is used. ``` INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2(or 3). ``` You can use `alluxio-fuse mount -o mountoptiona -o mountoptionb=value` to set mount options when launching the standalone Fuse process. Different versions of `libfuse` and `osxfuse` may support different mount options. The available Linux mount options are listed . The mount options of MacOS with osxfuse are listed . Some mount options (e.g. `allowother` and `allowroot`) need additional set-up and the set-up process may be different depending on the platform. ```shell $ alluxio-fuse mount <understoragedataset> <mountpoint> -o mountoption ``` By default, Alluxio-FUSE mount point can only be accessed by the user mounting the Alluxio namespace to the local filesystem. For Linux, add the following line to file `/etc/fuse.conf` to allow other users or allow root to access the mounted directory: ``` userallowother ``` Only after this step that non-root users have the permission to specify the `allowother` or `allowroot` mount options. For MacOS, follow the to allow other users to use the `allowother` and `allowroot` mount options. After setting up, pass the `allowother` or `allowroot` mount options when mounting Alluxio-FUSE: ```shell $ alluxio-fuse mount <understoragedataset> <mountpoint> -o allowother ``` ```shell $ alluxio-fuse mount <understoragedataset> <mountpoint> -o allowroot ``` Note that only one of the `allowother` or `allowroot` could be set. This section talks about how to troubleshoot issues related to Alluxio POSIX API. When encountering the out of direct memory issue, add the following JVM opts to `${ALLUXIO_HOME}/conf/alluxio-env.sh` to increase the max amount of direct memory. ```sh ALLUXIOFUSEJAVA_OPTS+=" -XX:MaxDirectMemorySize=8G" ```
Project: Gluster (Runtime / Cloud Native Storage)
File: posix.md

storage/posix translator ======================== Notes -- This is so that all filesystem checks are done with the user's uid/gid and not GlusterFS's uid/gid. This macro concatenates the base directory of the posix volume ('option directory') with the given path. If this flag is passed, lookup returns a xattr dictionary that contains the file's create time, the file's contents, and the version number of the file. This is a hack to increase small file performance. If an application wants to read a small file, it can finish its job with just a lookup call instead of a lookup followed by read. These are used by unify to set and get directory entries. Macro to align an address to a page boundary (4K). In some cases, two exported volumes may reside on the same partition on the server. Sending statvfs info for both the volumes will lead to erroneous df output at the client, since free space on the partition will be counted twice. In such cases, user can disable exporting statvfs info on one of the volumes by setting this option. This fop is used by replicate to set version numbers on files. A key, `GLUSTERFSFILECONTENT_STRING`, is handled in a special way by `getxattr`/`setxattr`. A getxattr with the key will return the entire content of the file as the value. A `setxattr` with the key will write the value as the entire content of the file. This calculates a simple XOR checksum on all entry names in a directory that is used by unify to compare directory contents.
Project: DANM (Runtime / Cloud Native Network)
File: deployment-installer-job.md

``` ${EDITOR} integration/install/danm-installer-config.yaml kubectl apply -f integration/install ``` This (currently experimental) method installs DANM using a Kubernetes Job. Just like the manual deployment, it assumes that a previous CNI (also referred to as "bootstrap CNI") is already installed. In the setup deployed by this installer, the bootstrap CNI will both be used by DANM components themselves (ie. netwatcher and svcwatcher will utilize that bootstrap CNI for their own network connectivity), as well as being configured as a DanmNet or ClusterNetwork with the name "default", that will be used by any Kubernetes applications without a `danm.io` annotation. Please be aware that the existing (bootstrap) CNI configuration must be a single CNI, not a list of CNIs. This means that in your CNI configuration directory, `/etc/cni/net.d`, there should be an existing file with a `.conf` extension, often named something like `10-flannel.conf` or `10-calico.conf`. If there is a file with a `.conflist` extension (such as `10-calico.conflist`), then that is a chained list of multiple CNIs. DANM does not currently support using such a `.conflist` chain as a bootstrap network. Depending on your setup, you may be able to to extract only the first CNI from the list using a command such as the following: ``` jq -M '{ name: .name, cniVersion: .cniVersioni} + .plugins[0]' \ /etc/cni/net.d/${EXISTINGCONFLISTFILE}.conflist \ > /etc/cni/net.d/${FIRSTPLUGINFROMLISTCONFIG_FILE}.conf ``` Either way, please be sure that you have a functional `/etc/cni/net.d/*.conf` CNI configuration before proceeding, and know the name of that `.conf` file. This file will need modification to match your setup. Please review/edit `integration/install/danm-installer-config.yaml`. This file will need modification only if the installation container needs to be pulled from an external registry. If this is the case, then please review/edit `integration/install/danm-installer.yaml`. If you have built DANM locally and do not need to pull images, this file does not need updating. ``` kubectl apply -f integration/install ``` After applying the installer CRD, in `kubectl get pods -n kube-system` you should first see a `danm-installer-*` pod starting, and shortly after, the `danm-cni` and `netwatcher` daemonsets, `svcwatcher`, and `danm-webhook-deployment` pods. The `danm-installer-*` pod should end up in "Completed" status - if not, please check the pod logs for any errors. After the installer pod ran to completion, you can remove the installer itself: ``` kubectl delete -f integration/install ```
Project: containerd (Runtime / Container Runtime)
File: snapshotter_bench_readme.md

Due to its dependency on `dmsetup`, executing the snapshotter process in an environment where a udev daemon is not accessible (such as a container) may result in unexpected behavior. In this case, try executing the snapshotter with the `DMDISABLEUDEV=1` environment variable, which tells `dmsetup` to ignore udev and manage devices itself. See and for more information. `containerd` project contains AWS CloudFormation template to run an EC2 instance suitable for benchmarking. It installs dependencies, prepares EBS volumes with same performance characteristics, and creates thin-pool device. You can make stack with the following command (note: there is a charge for using AWS resources): ```bash aws cloudformation create-stack \ --stack-name benchmark-instance \ --template-body file://benchmark_aws.yml \ --parameters \ ParameterKey=Key,ParameterValue=SSH_KEY \ ParameterKey=SecurityGroups,ParameterValue=sg-XXXXXXXX \ ParameterKey=VolumesSize,ParameterValue=20 \ ParameterKey=VolumesIOPS,ParameterValue=1000 ``` You can find an IP address of newly created EC2 instance in AWS Console or via AWS CLI: ```bash $ aws ec2 describe-instances \ --instance-ids $(aws cloudformation describe-stack-resources --stack-name benchmark-instance --query 'StackResources[*].PhysicalResourceId' --output text) \ --query 'Reservations[].Instances[].PublicIpAddress' \ --output text ``` SSH to an instance and prepare `containerd`: ```bash ssh -i SSH_KEY ec2-user@IP mkdir /mnt/disk1/data /mnt/disk2/data /mnt/disk3/data go get github.com/containerd/containerd cd $(go env GOPATH)/src/github.com/containerd/containerd make ``` Now you're ready to run the benchmark: ```bash sudo su - cd snapshots/benchsuite/ go test -bench . \ -dm.thinPoolDev=bench-docker--pool \ -dm.rootPath=/mnt/disk1/data \ -overlay.rootPath=/mnt/disk2/data \ -native.rootPath=/mnt/disk3/data ``` The output will look like: ```bash goos: linux goarch: amd64 pkg: github.com/containerd/containerd/snapshots/testsuite BenchmarkOverlay/run-4 1 1019730210 ns/op 164.53 MB/s BenchmarkOverlay/prepare 1 26799447 ns/op BenchmarkOverlay/write 1 968200363 ns/op BenchmarkOverlay/commit 1 24582560 ns/op BenchmarkDeviceMapper/run-4 1 3139232730 ns/op 53.44 MB/s BenchmarkDeviceMapper/prepare 1 1758640440 ns/op BenchmarkDeviceMapper/write 1 1356705388 ns/op BenchmarkDeviceMapper/commit 1 23720367 ns/op PASS ok github.com/containerd/containerd/snapshots/testsuite 185.204s ``` Don't forget to tear down the stack so it does not continue to incur charges: ```bash aws cloudformation delete-stack --stack-name benchmark-instance ```
Project: Kata Containers (Runtime / Container Runtime)
File: how-to-hotplug-memory-arm64.md

Memory hotplug is a key feature for containers to allocate memory dynamically in deployment. As Kata Container bases on VM, this feature needs support both from VMM and guest kernel. Luckily, it has been fully supported for the current default version of QEMU and guest kernel used by Kata on arm64. For other VMMs, e.g, Cloud Hypervisor, the enablement work is on the road. Apart from VMM and guest kernel, memory hotplug also depends on ACPI which depends on firmware either. On x86, you can boot a VM using QEMU with ACPI enabled directly, because it boots up with firmware implicitly. For arm64, however, you need specify firmware explicitly. That is to say, if you are ready to run a normal Kata Container on arm64, what you need extra to do is to install the UEFI ROM before use the memory hotplug feature. We have offered a helper script for you to install the UEFI ROM. If you have installed Kata normally on your host, you just need to run the script as fellows: ```bash $ pushd $GOPATH/src/github.com/kata-containers/tests $ sudo .ci/aarch64/installromaarch64.sh $ popd ``` After executing the above script, two files will be generated under the directory `/usr/share/kata-containers/` by default, namely `kata-flash0.img` and `kata-flash1.img`. Next we need to change the configuration file of `kata qemu`, which is in `/opt/kata/share/defaults/kata-containers/configuration-qemu.toml` by default, specify in the configuration file to use the UEFI ROM installed above. The above is an example of `kata deploy` installation. For package management installation, please use `kata-runtime env` to find the location of the configuration file. Please refer to the following configuration. ``` [hypervisor.qemu] pflashes = ["/usr/share/kata-containers/kata-flash0.img", "/usr/share/kata-containers/kata-flash1.img"] ``` Let's test if the memory hotplug is ready for Kata after install the UEFI ROM. Make sure containerd is ready to run Kata before test. ```bash $ sudo ctr image pull docker.io/library/ubuntu:latest $ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/ubuntu:latest hello sh -c "free -h" $ sudo ctr run --runtime io.containerd.run.kata.v2 -t --memory-limit 536870912 --rm docker.io/library/ubuntu:latest hello sh -c "free -h" ``` Compare the results between the two tests. If the latter is 0.5G larger than the former, you have done what you want, and congratulation!
Project: Velero (Runtime / Cloud Native Storage)
File: ark_get_schedules.md

title: "ark get schedules" layout: docs Get schedules Get schedules ``` ark get schedules [flags] ``` ``` -h, --help help for schedules --label-columns stringArray a comma-separated list of labels to be displayed as columns -o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. (default "table") -l, --selector string only show items matching this label selector --show-labels show labels in the last column ``` ``` --alsologtostderr log to standard error as well as files --kubeconfig string Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration --kubecontext string The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context) --logbacktraceat traceLocation when logging hits line file:N, emit a stack trace (default :0) --log_dir string If non-empty, write log files in this directory --logtostderr log to standard error instead of files -n, --namespace string The namespace in which Ark should operate (default "heptio-ark") --stderrthreshold severity logs at or above this threshold go to stderr (default 2) -v, --v Level log level for V logs --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging ``` - Get ark resources
Project: gVisor (Runtime / Container Runtime)
File: networking.md

[TOC] gVisor implements its own network stack called netstack. All aspects of the network stack are handled inside the Sentry including TCP connection state, control messages, and packet assembly keeping it isolated from the host network stack. Data link layer packets are written directly to the virtual device inside the network namespace setup by Docker or Kubernetes. Configuring the network stack may provide performance benefits, but isn't the only step to optimizing gVisor performance. See the [Production guide] for more. The IP address and routes configured for the device are transferred inside the sandbox. The loopback device runs exclusively inside the sandbox and does not use the host. You can inspect them by running: ```bash docker run --rm --runtime=runsc alpine ip addr ``` For high-performance networking applications, you may choose to disable the user space network stack and instead use the host network stack, including the loopback. Note that this mode decreases the isolation to the host. Add the following `runtimeArgs` to your Docker configuration (`/etc/docker/daemon.json`) and restart the Docker daemon: ```json { "runtimes": { "runsc": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--network=host" ] } } } ``` To completely isolate the host and network from the sandbox, external networking can be disabled. The sandbox will still contain a loopback provided by netstack. Add the following `runtimeArgs` to your Docker configuration (`/etc/docker/daemon.json`) and restart the Docker daemon: ```json { "runtimes": { "runsc": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--network=none" ] } } } ``` If your Linux is older than 4.14.77, you can disable Generic Segmentation Offload (GSO) to run with a kernel that is newer than 3.17. Add the `--gso=false` flag to your Docker runtime configuration (`/etc/docker/daemon.json`) and restart the Docker daemon: Note: Network performance, especially for large payloads, will be greatly reduced. ```json { "runtimes": { "runsc": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--gso=false" ] } } } ```
Project: Carina (Runtime / Cloud Native Storage)
File: ADOPTERS.md

This page contains a list of organizations who are using Carina in production or at stages of testing. If you'd like to be included here, please send a pull request which put the picture in the static directory,modifies this file and add information in README.md known users. : BoCloud Cloud Container Platform was awarded by Gratner as the Representative Vendor in China's Cloud Container CaaS segment and was ranked in the top 5 market share in China Mainland (IDC Container Software Market Report). Many of the existing customers have been using carina in production or testing environments for years, typically for cloudnative middlewares. |Logo| Organization | Status | |-| | | || | |