cwielenb/CNCF-Dataset huggingface dataset:
Project: CubeFS (Runtime / Cloud Native Storage)
File: architecture.md

CubeFS consists of a metadata subsystem, a data subsystem, a resource management node (Master), and an object gateway (Object Subsystem), which can access stored data through the POSIX/HDFS/S3 interface. Resource Management Node: Composed of multiple Master nodes, it is responsible for asynchronously processing different types of tasks, such as managing data shards and metadata shards (including creation, deletion, updating, and consistency checks), checking the health status of data or metadata nodes, and maintaining volume information. ::: tip Note There can be multiple Master nodes, and the consistency of metadata is ensured through the Raft algorithm and persisted to RocksDB. ::: Metadata Subsystem: Composed of multiple Meta Node nodes, multiple metadata shards (Meta Partition), and Raft instances (based on the Multi-Raft replication protocol), each metadata shard represents an Inode range metadata, which contains two in-memory B-Tree structures: inode B-Tree and dentry B-Tree. ::: tip Note At least 3 metadata instances are required, and horizontal scaling is supported. ::: Data Subsystem: Divided into Replica Subsystem and Erasure Code Subsystem, both subsystems can coexist or exist independently: The Replica Subsystem consists of DataNodes, with each node managing a set of data shards. Multiple nodes' data shards form a replica group. The Erasure Code Subsystem (Blobstore) is mainly composed of BlobNode modules, with each node managing a set of data blocks. Multiple nodes' data blocks form an erasure-coded stripe. ::: tip Note DataNode support horizontal scaling. ::: Object Subsystem: Composed of object nodes, it provides an access protocol compatible with standard S3 semantics and can be accessed through tools such as Amazon S3 SDK or s3cmd. Volume: A logical concept composed of multiple metadata and data shards. From the client's perspective, a volume can be seen as a file system instance that can be accessed by containers. From the perspective of object storage, a volume corresponds to a bucket. A volume can be mounted in multiple containers, allowing files to be accessed by different clients simultaneously.
Project: gVisor (Runtime / Container Runtime)
File: checkpoint_restore.md

[TOC] gVisor has the ability to checkpoint a process, save its current state in a state file, and restore into a new container using the state file. Checkpoint/restore functionality is currently available via raw `runsc` commands. To use the checkpoint command, first run a container. ```bash runsc run <container id> ``` To checkpoint the container, the `--image-path` flag must be provided. This is the directory path within which the checkpoint related files will be created. All necessary directories will be created if they do not yet exist. Note: Two checkpoints cannot be saved to the same directory; every image-path provided must be unique. ```bash runsc checkpoint --image-path=<path> <container id> ``` There is also an optional `--leave-running` flag that allows the container to continue to run after the checkpoint has been made. (By default, containers stop their processes after committing a checkpoint.) Note: All top-level runsc flags needed when calling run must be provided to checkpoint if `--leave-running` is used. Note: `--leave-running` functions by causing an immediate restore so the container, although will maintain its given container id, may have a different process id. ```bash runsc checkpoint --image-path=<path> --leave-running <container id> ``` To restore, provide the image path to the directory containing all the files created during the checkpoint. Because containers stop by default after checkpointing, restore needs to happen in a new container (restore is a command which parallels start). ```bash runsc create <container id> runsc restore --image-path=<path> <container id> ``` Note: All top-level runsc flags needed when calling run must be provided to `restore`. Run a container: ```bash docker run [options] --runtime=runsc --name=<container-name> <image> ``` Checkpoint the container: ```bash docker checkpoint create <container-name> <checkpoint-name> ``` Restore into the same container: ```bash docker start --checkpoint <checkpoint-name> <container-name> ``` : Docker version 18.03.0-ce and earlier hangs when checkpointing and does not create the checkpoint. To successfully use this feature, install a custom version of docker-ce from the moby repository. This issue is caused by an improper implementation of the `--leave-running` flag. This issue is fixed in newer releases. Docker does not support restoration into new containers: Docker currently expects the container which created the checkpoint to be the same container used to restore. This is needed to support container migration. : Docker does not currently support the `--checkpoint-dir` flag but this will be required when restoring from a checkpoint made in another container.
Project: Cilium (Runtime / Cloud Native Network)
File: cilium-dbg_ip.md

<!-- This file was autogenerated via cilium cmdref, do not edit manually--> Manage IP addresses and associated information ``` -h, --help help for ip ``` ``` --config string Config file (default is $HOME/.cilium.yaml) -D, --debug Enable debug messages -H, --host string URI to server-side API ``` - CLI - List IP addresses in the userspace IPcache
Project: Carina (Runtime / Cloud Native Storage)
File: disk-speed-limit.md

Users can add annotation to pod to limit bandwidth or IOPS. Example: `kubectl apply -f deployment.yaml` ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: carina-deployment-speed-limit namespace: carina labels: app: web-server-speed-limit spec: replicas: 1 selector: matchLabels: app: web-server-speed-limit template: metadata: annotations: carina.storage.io/blkio.throttle.readbpsdevice: "10485760" carina.storage.io/blkio.throttle.readiopsdevice: "10000" carina.storage.io/blkio.throttle.writebpsdevice: "10485760" carina.storage.io/blkio.throttle.writeiopsdevice: "100000" labels: app: web-server-speed-limit spec: containers: name: web-server image: nginx:latest imagePullPolicy: "IfNotPresent" volumeMounts: name: mypvc mountPath: /var/lib/www/html volumes: name: mypvc persistentVolumeClaim: claimName: csi-carina-pvc-speed-limit readOnly: false apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-carina-pvc-speed-limit namespace: carina spec: accessModes: ReadWriteOnce resources: requests: storage: 17Gi storageClassName: csi-carina-sc volumeMode: Filesystem ``` Carina will convert those anntoations into pod's cgroupfs hierarchy. ```shell cgroup v1 /sys/fs/cgroup/blkio/kubepods/burstable/pod0b0e005c-39ec-4719-bbfe-78aadbc3e4ad/blkio.throttle.readbpsdevice /sys/fs/cgroup/blkio/kubepods/burstable/pod0b0e005c-39ec-4719-bbfe-78aadbc3e4ad/blkio.throttle.readiopsdevice /sys/fs/cgroup/blkio/kubepods/burstable/pod0b0e005c-39ec-4719-bbfe-78aadbc3e4ad/blkio.throttle.writebpsdevice /sys/fs/cgroup/blkio/kubepods/burstable/pod0b0e005c-39ec-4719-bbfe-78aadbc3e4ad/blkio.throttle.writeiopsdevice ``` ```shell cgroup v2 /sys/fs/cgroup/kubepods/burstable/pod0b0e005c-39ec-4719-bbfe-78aadbc3e4ad/io.max ``` Users can add one or more annotations. Adding or removing annotations will be synced to cgroupfs in about 60s. Currently, only block device disk speed limit is supported. User can test io throttling with command `dd if=/dev/zero of=out.file bs=1M count=512 oflag=dsync`. Carina can automatically decide whether to use cgroup v1 or cgroup v2 according to the system environment. If the system uses cgroup v2, it supports buffer io speed limit (you need to enable io and memory controllers at the same time), otherwise only direct io speed limit is supported. If user can set io throttling too low, it may cause the procedure of formating filesystem hangs there and then the pod will be in pending state forever.
Project: Cilium (Runtime / Cloud Native Network)
File: cilium-dbg_endpoint_list.md

<!-- This file was autogenerated via cilium cmdref, do not edit manually--> List all endpoints ``` cilium-dbg endpoint list [flags] ``` ``` -h, --help help for list --no-headers Do not print headers -o, --output string json| yaml| jsonpath='{}' ``` ``` --config string Config file (default is $HOME/.cilium.yaml) -D, --debug Enable debug messages -H, --host string URI to server-side API ``` - Manage endpoints
Project: Rook (Runtime / Cloud Native Storage)
File: ceph-object-store-user-crd.md

title: CephObjectStoreUser CRD Rook allows creation and customization of object store users through the custom resource definitions (CRDs). The following settings are available for Ceph object store users. ```yaml apiVersion: ceph.rook.io/v1 kind: CephObjectStoreUser metadata: name: my-user namespace: rook-ceph spec: store: my-store displayName: my-display-name quotas: maxBuckets: 100 maxSize: 10G maxObjects: 10000 capabilities: user: "*" bucket: "*" ``` `name`: The name of the object store user to create, which will be reflected in the secret and other resource names. `namespace`: The namespace of the Rook cluster where the object store user is created. `store`: The object store in which the user will be created. This matches the name of the objectstore CRD. `displayName`: The display name which will be passed to the `radosgw-admin user create` command. `clusterNamespace`: The namespace where the parent CephCluster and CephObjectStore are found. If not specified, the user must be in the same namespace as the cluster and object store. To enable this feature, the CephObjectStore allowUsersInNamespaces must include the namespace of this user. `quotas`: This represents quota limitation can be set on the user. Please refer for details. `maxBuckets`: The maximum bucket limit for the user. `maxSize`: Maximum size limit of all objects across all the user's buckets. `maxObjects`: Maximum number of objects across all the user's buckets. `capabilities`: Ceph allows users to be given additional permissions. Due to missing APIs in go-ceph for updating the user capabilities, this setting can currently only be used during the creation of the object store user. If a user's capabilities need modified, the user must be deleted and re-created. See the for more info. Rook supports adding `read`, `write`, `read, write`, or `*` permissions for the following resources: `user` `buckets` `usage` `metadata` `zone` `roles` `info` `amz-cache` `bilog` `mdlog` `datalog` `user-policy` `odic-provider` `ratelimit`
Project: CubeFS (Runtime / Cloud Native Storage)
File: powershell_completions.md

Cobra can generate PowerShell completion scripts. Users need PowerShell version 5.0 or above, which comes with Windows 10 and can be downloaded separately for Windows 7 or 8.1. They can then write the completions to a file and source this file from their PowerShell profile, which is referenced by the `$Profile` environment variable. See `Get-Help about_Profiles` for more info about PowerShell profiles. Completion for subcommands using their `.Short` description Completion for non-hidden flags using their `.Name` and `.Shorthand` Command aliases Required, filename or custom flags (they will work like normal flags) Custom completion scripts
Project: lxd (Runtime / Container Runtime)
File: cluster_config_networks.md

(cluster-config-networks)= All members of a cluster must have identical networks defined. The only configuration keys that may differ between networks on different members are , , and . See {ref}`clustering-member-config` for more information. Creating additional networks is a two-step process: Define and configure the new network across all cluster members. For example, for a cluster that has three members: incus network create --target server1 my-network incus network create --target server2 my-network incus network create --target server3 my-network ```{note} You can pass only the member-specific configuration keys `bridge.external_interfaces`, `parent`, `bgp.ipv4.nexthop` and `bgp.ipv6.nexthop`. Passing other configuration keys results in an error. ``` These commands define the network, but they don't create it. If you run , you can see that the network is marked as "pending". Run the following command to instantiate the network on all cluster members: incus network create my-network ```{note} You can add configuration keys that are not member-specific to this command. ``` If you missed a cluster member when defining the network, or if a cluster member is down, you get an error. Also see {ref}`network-create-cluster`. (cluster-https-address)= You can configure different networks for the REST API endpoint of your clients and for internal traffic between the members of your cluster. This separation can be useful, for example, to use a virtual address for your REST API, with DNS round robin. To do so, you must specify different addresses for {config:option}`server-cluster:cluster.httpsaddress` (the address for internal cluster traffic) and {config:option}`server-core:core.httpsaddress` (the address for the REST API): Create your cluster as usual, and make sure to use the address that you want to use for internal cluster traffic as the cluster address. This address is set as the `cluster.https_address` configuration. After joining your members, set the `core.https_address` configuration to the address for the REST API. For example: incus config set core.https_address 0.0.0.0:8443 ```{note} `core.https_address` is specific to the cluster member, so you can use different addresses on different members. You can also use a wildcard address to make the member listen on multiple interfaces. ```
Project: Weave Net (Runtime / Cloud Native Network)
File: autoscaling.md

title: Autoscaling menu_order: 40 search_type: Documentation An autoscaling configuration begins with a fixed cluster: Configured as per the scenario. Hosted on reserved or protected instances to ensure long-term stability. Ideally sized at a minimum of three or five nodes (you can make your fixed cluster bigger to accommodate base load as required; a minimum is specified here in the interests of resilience only.) Building on this foundation, arbitrary numbers of dynamic peers can be added or removed concurrently as desired, without requiring any changes to the configuration of the fixed cluster. As with the fixed cluster, dynamically added nodes recover automatically from reboots and partitions. On the additional dynamic peer, at boot, via or equivalent: weave launch --no-restart --ipalloc-init=observer $PEERS Where, `$PEERS` means all peers in the fixed cluster, initial and subsequently added, which have not been explicitly removed. It should include fixed peers which are temporarily offline or stopped. You do not have to keep track of and specify the addresses of other dynamic peers in `$PEERS` - they will discover and connect to each other via the fixed cluster. Note: The use of `--ipalloc-init=observer` prevents dynamic peers from coming to a consensus on their own - this is important to stop a clique forming amongst a group of dynamically added peers if they become partitioned from the fixed cluster after having learned about each other via discovery. On the dynamic peer to be removed: weave reset If for any reason you cannot arrange for `weave reset` to be run on the peer before the underlying host is destroyed (for example when using spot instances that can be destroyed without notice), you will need an asynchronous process to [reclaim lost IP address space](/site/operational-guide/tasks.md#detect-reclaim-ipam).
Project: Submariner (Runtime / Cloud Native Network)
File: GOVERNANCE.md

The opengovernance.dev project [provides the following checklist](https://github.com/opengovernance/opengovernance.dev/blob/master/README.md#checklist) for defining open governance. Submariner's answers are provided for each point. This is not meant to define Submariner's governance, but instead to document it clearly. Authoritative sources are cited inline. Who owns the copyright on contributed code? All Submariner code is to verify the copyright notices specify "Contributors to the Submariner project", per the . Who owns the domain(s) for the project? Submariner is a CNCF Sandbox project, so all Submariner domains are owned by the CNCF. Who owns the trademark for the project, is it neutrally owned and governed? Are there open trademark guidelines? Submariner is a CNCF Sandbox project, so all Submariner trademarks are owned by the CNCF. How can users license the projects branding? Are there open branding guidelines? Submariner is a CNCF Sandbox project, so please see the CNCF/artwork project's . If the project raises funds, who owns it? Submariner is a CNCF Sandbox project, so funds would be owned by the CNCF. Who makes decisions on how the project performs releases? Project Owners are responsible for defining milestones and releases, per the Submariner . How can the project contributors become committers? In short, by reviewing pull requests and receiving Committer/Owner approval. See the for details. How are project committers removed? Committers can be removed by stepping down or by two thirds vote of Project Owners, per the Submariner [Committer Responsibilities and Privileges](https://submariner.io/community/contributor-roles/#committer-responsibilities-and-privileges). Project Owner removals are currently frozen except for stepping down or for Code of Conduct violations. See the [Owner Removal and Future Elected Governance](https://submariner.io/community/contributor-roles/#owner-removal-and-future-elected-governance) documentation for details. If the project raises funds, who decides how this money is spent? Project Owners are responsible for deciding how funds are spent, per the Submariner . Who decides the project roadmap? The is public and open to everyone. Who can participate in security disclosure issues? Project Owners are responsible for receiving and ensuring an adequate response, per the Submariner . How transparent are the decision-making processes? Decisions are made in weekly public community meetings or on public GitHub Issues/PRs. All meetings are documented in Submariner's . Who enforces the code of conduct? The Submariner Project Owners enforce the , per the Submariner .
Project: Weave Net (Runtime / Cloud Native Network)
File: weave-docker-api.md

title: Integrating Docker via the API Proxy menu_order: 5 search_type: Documentation The Docker API proxy automatically attaches containers to the Weave network when they are started using the ordinary Docker or the . There are three ways to attach containers to a Weave network (which method to use is entirely up to you): The Weave Net Docker API Proxy. See . The Docker Network Plugin framework. The Docker Network Plugin is used when Docker containers are started with the --net flag, for example: `docker run --net <docker-run-options>` Where, `<docker-run-options>` are the you give to your container on start Note that if a Docker container is started with the --net flag, then the Weave Docker API Proxy is automatically disabled and is not used to attach containers. See and . Containers can also be attached to the Weave network with `weave attach` commands. This method also does not use the Weave Docker API Proxy. See . The proxy sits between the Docker client (command line or API) and the Docker daemon, and intercepts the communication between the two. It is started along with the router and weaveDNS when you run: host1$ weave launch N.B.: Prior to version 2.0, the `launch-proxy` command allowed to pass configuration options and to start the proxy independently. This command has been removed in 2.0 and `launch` now also accepts configuration options for the proxy. By default, the proxy decides where to listen based on how the launching client connects to Docker. If the launching client connected over a UNIX socket, the proxy listens on `/var/run/weave/weave.sock`. If the launching client connects over TCP, the proxy listens on port 12375, on all network interfaces. This can be adjusted using the `-H` argument, for example: host1$ weave launch -H tcp://127.0.0.1:9999 If no TLS or listening interfaces are set, TLS is auto-configured based on the Docker daemon's settings, and the listening interfaces are auto-configured based on your Docker client's settings. Multiple `-H` arguments can be specified. If you are working with a remote docker daemon, then any firewalls in between need to be configured to permit access to the proxy port. All docker commands can be run via the proxy, so it is safe to adjust your `DOCKER_HOST` to point at the proxy. Weave Net provides a convenient command for this: host1$ eval $(weave env) host1$ docker ps The prior settings can be restored with host1$ eval $(weave env --restore) Alternatively, the proxy host can be set on a per-command basis with host1$ docker $(weave config) ps The proxy can be stopped, along with the router and weaveDNS, with host1$ weave stop If you set your `DOCKER_HOST` to point at the proxy, you should revert to the original settings prior to running `stop`. See Also *
Project: lxd (Runtime / Container Runtime)
File: instances_routed_nic_vm.md

(instances-routed-nic-vm)= When adding a {ref}`routed NIC device <nic-routed>` to an instance, you must configure the instance to use the link-local gateway IPs as default routes. For containers, this is configured for you automatically. For virtual machines, the gateways must be configured manually or via a mechanism like `cloud-init`. To configure the gateways with `cloud-init`, firstly initialize an instance: incus init images:ubuntu/22.04 jammy --vm Then add the routed NIC device: incus config device add jammy eth0 nic nictype=routed parent=my-parent-network ipv4.address=192.0.2.2 ipv6.address=2001:db8::2 In this command, `my-parent-network` is your parent network, and the IPv4 and IPv6 addresses are within the subnet of the parent. Next we will add some `netplan` configuration to the instance using the `cloud-init.network-config` configuration key: cat <<EOF | incus config set jammy cloud-init.network-config - network: version: 2 ethernets: enp5s0: routes: to: default via: 169.254.0.1 on-link: true to: default via: fe80::1 on-link: true addresses: 192.0.2.2/32 2001:db8::2/128 EOF This `netplan` configuration adds the {ref}`static link-local next-hop addresses <nic-routed>` (`169.254.0.1` and `fe80::1`) that are required. For each of these routes we set `on-link` to `true`, which specifies that the route is directly connected to the interface. We also add the addresses that we configured in our routed NIC device. For more information on `netplan`, see . ```{note} This `netplan` configuration does not include a name server. To enable DNS within the instance, you must set a valid DNS IP address. If there is a `incusbr0` network on the host, the name server can be set to that IP instead. ``` You can then start your instance with: incus start jammy ```{note} Before you start your instance, make sure that you have {ref}`configured the parent network <nic-routed>` to enable proxy ARP/NDP. ```