cwielenb/CNCF-Dataset huggingface dataset:
Project: JuiceFS (Runtime / Cloud Native Storage)
File: fio.md

title: Benchmark with fio sidebar_position: 7 slug: /fio :::tip Trash is enabled in JuiceFS v1.0+ by default. As a result, temporary files are created and deleted in the file system during the benchmark, and these files will be eventually dumped into a directory named `.trash`. To avoid storage space being occupied by `.trash`, you can run command `juicefs config META-URL --trash-days 0` to disable Trash before benchmark. See for details. ::: Perform a sequential read/write benchmark on JuiceFS, and with . The following tests are performed with `fio` 3.1. Sequential read test (numjobs: 1): ``` fio --name=sequential-read --directory=/s3fs --rw=read --refill_buffers --bs=4M --size=4G fio --name=sequential-read --directory=/efs --rw=read --refill_buffers --bs=4M --size=4G fio --name=sequential-read --directory=/jfs --rw=read --refill_buffers --bs=4M --size=4G ``` Sequential write test (numjobs: 1): ``` fio --name=sequential-write --directory=/s3fs --rw=write --refillbuffers --bs=4M --size=4G --endfsync=1 fio --name=sequential-write --directory=/efs --rw=write --refillbuffers --bs=4M --size=4G --endfsync=1 fio --name=sequential-write --directory=/jfs --rw=write --refillbuffers --bs=4M --size=4G --endfsync=1 ``` Sequential read test (numjobs: 16): ``` fio --name=big-file-multi-read --directory=/s3fs --rw=read --refill_buffers --bs=4M --size=4G --numjobs=16 fio --name=big-file-multi-read --directory=/efs --rw=read --refill_buffers --bs=4M --size=4G --numjobs=16 fio --name=big-file-multi-read --directory=/jfs --rw=read --refill_buffers --bs=4M --size=4G --numjobs=16 ``` Sequential write test (numjobs: 16): ``` fio --name=big-file-multi-write --directory=/s3fs --rw=write --refillbuffers --bs=4M --size=4G --numjobs=16 --endfsync=1 fio --name=big-file-multi-write --directory=/efs --rw=write --refillbuffers --bs=4M --size=4G --numjobs=16 --endfsync=1 fio --name=big-file-multi-write --directory=/jfs --rw=write --refillbuffers --bs=4M --size=4G --numjobs=16 --endfsync=1 ``` All the following tests are all performed using `fio` on a c5d.18xlarge EC2 instance (72 CPU, 144G RAM) with Ubuntu 18.04 LTS (Kernel 5.4.0) operating system. JuiceFS uses a local Redis instance (version 4.0.9) to store metadata. JuiceFS mount command: ``` ./juicefs format --storage=s3 --bucket=https://<BUCKET>.s3.<REGION>.amazonaws.com localhost benchmark ./juicefs mount --max-uploads=150 --io-retries=20 localhost /jfs ``` EFS mount command (the same as the configuration page): ``` mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport, <EFS-ID>.efs.<REGION>.amazonaws.com:/ /efs ``` S3FS (version 1.82) mount command: ``` s3fs <BUCKET>:/s3fs /s3fs -o host=https://s3.<REGION>.amazonaws.com,endpoint=<REGION>,passwd_file=${HOME}/.passwd-s3fs ```
Project: Weave Net (Runtime / Cloud Native Network)
File: plugin.md

title: Integrating Docker via the Network Plugin (Legacy) menu_order: 30 search_type: Documentation * Docker versions 1.9 and later have a plugin mechanism for adding different network providers. Weave Net installs itself as a network plugin when you start it with `weave launch`. The Weave Docker Networking Plugin is fast and easy to use, and, unlike other networking plugins, does not require an external cluster store. To create a network which can span multiple Docker hosts, Weave Net peers must be connected to each other, by specifying the other hosts during `weave launch` or via . See for a discussion on peer connections. After you've launched Weave Net and peered your hosts, you can start containers using the following, for example: $ docker run --net=weave -ti weaveworks/ubuntu on any of the hosts, and they can all communicate with each other using any protocol, even multicast. In order to use Weave Net's you must pass the additional arguments `--dns` and `-dns-search`, for which a helper is provided in the Weave script: $ docker run --net=weave -h foo.weave.local $(weave dns-args) -tdi weaveworks/ubuntu $ docker run --net=weave -h bar.weave.local $(weave dns-args) -ti weaveworks/ubuntu Just launch the Weave Net router onto each host and make a peer connection with the other hosts: host1$ weave launch host2 host2$ weave launch host1 then run your containers using the Docker command-line: host1$ docker run --net=weave -ti weaveworks/ubuntu root@1458e848cd90:/# hostname -i 10.32.0.2 host2$ docker run --net=weave -ti weaveworks/ubuntu root@8cc4b5dc5722:/# ping 10.32.0.2 PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data. 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.116 ms 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.052 ms Docker enables you to create multiple independent networks and attach different sets of containers to each network. However, coordinating this between hosts requires that you run Docker in or configure a . To operate in swarm mode, you are required to use the plugin v2 of Weave Net. See for more details. If your Docker installation has a key-value store, create a network based on Weave Net as follows: $ docker network create --driver=weave mynetwork then use it to connect a container: $ docker run --net=mynetwork ... or $ docker network connect mynetwork somecontainer Containers attached to different Docker Networks are . The plugin, like all Weave Net components, is started with a policy of `--restart=always`, so that it is always there after a restart or reboot. If you remove this container (for example, when using `weave reset`) before removing all endpoints created using `--net=weave`, Docker may hang for a long time when it subsequently tries to re-establish communications to the plugin. Unfortunately, . If you are using `systemd` with Docker 1.9, it is advised that you modify the Docker unit to remove the timeout on startup. This gives Docker enough time to abandon its attempts. For example, in the file `/lib/systemd/system/docker.service`, add the following under `[Service]`: TimeoutStartSec=0 See Also
Project: Inclavare Containers (Runtime / Container Runtime)
File: pal_programming_guide.md

The enclave runtime currently supported by runE are occlum and WAMR (WebAssembly Micro Runtime). In order to facilitate other libos programs to run in runE, a set of enclave runtime API interfaces is defined. Libos only needs to support this set of API interfaces to run as an enclave runtime in runE. runE enclave runtime is bounded by the enclave runtime pal API layer, below the API layer is runE, above the API layer is the enclave runtime, and the operating mode is libos. ```c struct palattrt { const char* args; const char* log_level; }; struct stdio_fds { int stdin, stdout, stderr; }; struct palcreateprocess_args { char *path; char *argv[]; char *env[]; struct stdio_fds *stdio; int *pid; }attribute((packed)); struct palexecargs { int pid; int *exit_value; }attribute((packed)); struct palkillargs { int pid; int sig; }attribute((packed)); struct pal_opt { int pal_version(); int palinit(struct palattr_t *attr); int palcreateprocess(struct palcreateprocess_args *args); int palexec(struct palexec_args *args); int palkill(struct palkill_args *args); int pal_destroy(); }; ``` The enclave runtime is generated as a so dynamic library, which is dynamically loaded by rune using dlopen; the enclave runtime needs to export symbols according to the function named in the previous chapter.<br /> The value of this global variable is the version of pal_api, refer to the implementation: ```c int pal_version() { return 2; } ``` The main task of this interface should be to create an enclave space and complete the memory layout of the enclave space; libos also needs to complete the initialization of components such as VM, FS, and NET. Reference implementation: ```c int palinit(const struct palattr_t *attr) { ... sgxlaunchtoken_t token; get_token(&token); sgxcreateenclave(..., token, ...); ... } ``` The main job of this interface is to create a new process, reference implementation: ```c int palcreateprocess(struct palcreateprocess_args *args) { ... args->pid = liboscreateprocess(...); ... } ``` The main job of this interface is to run a program created by palcreateprocess, refer to the implementation: ```c int palexec(struct palexec_args *args) { ... libos_exec(...); ... } ``` The main job of this interface is to send a signal to the specified pid, refer to the implementation: ```c int pal_kill(int pid, int sig) { ... libos_kill(...) ... } ``` The main job of this interface is to destroy the entire enclave space. If it is libos, you need to do component de-initialization before destroying the enclave. Reference implementation: ```c int pal_destroy(void) { ... libos_uninitialize(); sgxdestroyenclave(global_eid); ... } ```
Project: Velero (Runtime / Cloud Native Storage)
File: debugging-install.md

title: "Debugging Installation Issues" layout: docs This typically means that no `kubeconfig` file can be found for the Velero client to use. Velero looks for a kubeconfig in the following locations: the path specified by the `--kubeconfig` flag, if any the path specified by the `$KUBECONFIG` environment variable, if any `~/.kube/config` This means that the Velero controllers are not processing the backups/restores, which usually happens because the Velero server is not running. Check the pod description and logs for errors: ``` kubectl -n velero describe pods kubectl -n velero logs deployment/velero ``` This means that the secret containing the AWS IAM user credentials for Velero has not been created/mounted properly into the Velero server pod. Ensure the following: The `cloud-credentials` secret exists in the Velero server's namespace The `cloud-credentials` secret has a single key, `cloud`, whose value is the contents of the `credentials-velero` file The `credentials-velero` file is formatted properly and has the correct values: ``` [default] awsaccesskey_id=<your AWS access key ID> awssecretaccess_key=<your AWS secret access key> ``` The `cloud-credentials` secret is defined as a volume for the Velero deployment The `cloud-credentials` secret is being mounted into the Velero server pod at `/credentials` This means that Velero can't read the content of the S3 bucket. Ensure the following: A Trust Policy document exists that allows the role used by kube2iam to assume Velero's role, as stated in the AWS config documentation. The new Velero role has all the permissions listed in the documentation regarding S3. This means that the secrets containing the Azure service principal credentials for Velero has not been created/mounted properly into the Velero server pod. Ensure the following: The `cloud-credentials` secret exists in the Velero server's namespace The `cloud-credentials` secret has all of the expected keys and each one has the correct value (see ) The `cloud-credentials` secret is defined as a volume for the Velero deployment The `cloud-credentials` secret is being mounted into the Velero server pod at `/credentials` This means that the secret containing the GCE service account credentials for Velero has not been created/mounted properly into the Velero server pod. Ensure the following: The `cloud-credentials` secret exists in the Velero server's namespace The `cloud-credentials` secret has a single key, `cloud`, whose value is the contents of the `credentials-velero` file The `cloud-credentials` secret is defined as a volume for the Velero deployment The `cloud-credentials` secret is being mounted into the Velero server pod at `/credentials`
Project: runc (Runtime / Container Runtime)
File: runc-exec.8.md

% runc-exec "8" runc-exec - execute new process inside the container runc exec [option ...] container-id [--] command [arg ...] runc exec [option ...] -p process.json container-id --console-socket path : Path to an AF_UNIX socket which will receive a file descriptor referencing the master end of the console's pseudoterminal. See . --cwd path : Change to path in the container before executing the command. --env|-e name=value : Set an environment variable name to value. Can be specified multiple times. --tty|-t : Allocate a pseudo-TTY. --user|-u uid[:gid] : Run the command as a user (and, optionally, group) specified by uid (and gid). --additional-gids|-g gid : Add additional group IDs. Can be specified multiple times. --process|-p process.json : Instead of specifying all the exec parameters directly on the command line, get them from a process.json, a JSON file containing the process specification as defined by the . --detach|-d : Detach from the container's process. --pid-file path : Specify the file to write the container process' PID to. --process-label label : Set the asm process label for the process commonly used with selinux(7). --apparmor profile : Set the apparmor(7) profile for the process. --no-new-privs : Set the "no new privileges" value for the process. --cap cap : Add a capability to the bounding set for the process. Can be specified multiple times. --preserve-fds N : Pass N additional file descriptors to the container (stdio + $LISTEN_FDS + N in total). Default is 0. --ignore-paused : Allow exec in a paused container. By default, if a container is paused, runc exec errors out; this option can be used to override it. A paused container needs to be resumed for the exec to complete. --cgroup path | controller[,controller...]:path : Execute a process in a sub-cgroup. If the specified cgroup does not exist, an error is returned. Default is empty path, which means to use container's top level cgroup. : For cgroup v1 only, a particular controller (or multiple comma-separated controllers) can be specified, and the option can be used multiple times to set different paths for different controllers. : Note for cgroup v2, in case the process can't join the top level cgroup, runc exec fallback is to try joining the cgroup of container's init. This fallback can be disabled by using --cgroup /. Exits with a status of command (unless -d is used), or 255 if an error occurred. If the container can run ps(1) command, the following will output a list of processes running in the container: runc(8).
Project: CubeFS (Runtime / Cloud Native Storage)
File: failureDomain.md

Enabling fault domains requires adding cluster-level configuration, otherwise it cannot be distinguished. Adding a new zone is a fault domain zone or belongs to the original cross_zone. ```bash FaultDomain bool // Default is false ``` Reserved: ```bash crossZone bool # Cross-zone ``` New: ```bash default_priority bool # True means prefer the original zone rather than allocating from the fault domain ``` For existing clusters, whether self-built or community-built, whether single zone or cross-zone, if fault domains need to be enabled, the cluster needs to support it, the master needs to be restarted, the configuration needs to be updated, and the policy for updating existing volumes needs to be managed. Otherwise, continue to use the original policy. If the cluster supports it but the volume does not choose to use it, continue to use the original volume policy and allocate resources according to the original policy in the original zone. Use new zone resources when existing resources are exhausted. If the cluster does not support it, the volume cannot enable its own fault domain policy. | Cluster:faultDomain | Vol:crossZone | Vol:normalZonesFirst | Rules for volume to use domain | |||-|-| | N | N/A | N/A | Do not support domain | | Y | N | N/A | Write origin resources first before fault domain until origin reach threshold | | Y | Y | N | Write fault domain only | | Y | Y | Y | Write origin resources first before fault domain until origin reach threshold | Fault domains solve the problem of copyset distribution without planning in multi-zone scenarios, which affects the durability of data, but existing data cannot be automatically migrated. After enabling fault domains, all devices in the new region will be added to the fault domain. The volume created will prefer the resources of the original zone. When creating a new volume, use the domain resources according to the configuration items in the table above. By default, if available, the original zone resources are used first. Create a volume using fault domains ```bash curl "http://192.168.0.11:17010/admin/createVol?name=volDomain&capacity=1000&owner=cfs&crossZone=true&normalZonesFirst=false" ``` Parameter List | Parameter | Type | Description | ||--|| | crossZone | string | Whether to cross zones | | normalZonesFirst | bool | Non-fault domain priority | ```bash curl "http://192.168.0.11:17010/admin/getIsDomainOn" ``` ```bash curl -v "http://192.168.0.11:17010/admin/getDomainInfo" ``` Check the usage of fault domain copyset groups ```bash curl "http://192.168.0.11:17010/admin/getDomainNodeSetGrpInfo?id=37" ``` Update the upper limit of non-fault domain data usage ```bash curl "http://192.168.0.11:17010/admin/updateZoneExcludeRatio?ratio=0.7" ```
Project: Cilium (Runtime / Cloud Native Network)
File: cilium-dbg_policy_get.md

<!-- This file was autogenerated via cilium cmdref, do not edit manually--> Display policy node information ``` cilium-dbg policy get [<labels>] [flags] ``` ``` -h, --help help for get -o, --output string json| yaml| jsonpath='{}' ``` ``` --config string Config file (default is $HOME/.cilium.yaml) -D, --debug Enable debug messages -H, --host string URI to server-side API ``` - Manage security policies
Project: Cilium (Runtime / Cloud Native Network)
File: cilium-dbg_nodeid_list.md

<!-- This file was autogenerated via cilium cmdref, do not edit manually--> List node IDs and the associated IP addresses ``` cilium-dbg nodeid list [flags] ``` ``` -h, --help help for list -o, --output string json| yaml| jsonpath='{}' ``` ``` --config string Config file (default is $HOME/.cilium.yaml) -D, --debug Enable debug messages -H, --host string URI to server-side API ``` - List node IDs and associated information
Project: JuiceFS (Runtime / Cloud Native Storage)
File: destroy.md

title: How to destroy a file system sidebar_position: 8 JuiceFS client provides the `destroy` command to completely destroy a file system, which will result in Deletion of all metadata entries of this file system Deletion of all data blocks of this file system Use this command in the following format. ```shell juicefs destroy <METADATA URL> <UUID> ``` `<METADATA URL>`: The URL address of the metadata engine `<UUID>`: The UUID of the file system JuiceFS client provides a `status` command to view detailed information about a file system by simply specifying the file system's metadata engine URL, e.g. ```shell {8} $ juicefs status redis://127.0.0.1:6379/1 2022/01/26 21:41:37.577645 juicefs[31181] <INFO>: Meta address: redis://127.0.0.1:6379/1 2022/01/26 21:41:37.578238 juicefs[31181] <INFO>: Ping redis: 55.041s { "Setting": { "Name": "macjfs", "UUID": "eabb96d5-7228-461e-9240-fddbf2b576d8", "Storage": "file", "Bucket": "jfs/", "AccessKey": "", "BlockSize": 4096, "Compression": "none", "Shards": 0, "Partitions": 0, "Capacity": 0, "Inodes": 0, "TrashDays": 1 }, ... } ``` :::danger The destroy operation will cause all the data in the database and the object storage associated with the file system to be deleted. Please make sure to back up the important data before operating! ::: ```shell {1} $ juicefs destroy redis://127.0.0.1:6379/1 eabb96d5-7228-461e-9240-fddbf2b576d8 2022/01/26 21:52:17.488987 juicefs[31518] <INFO>: Meta address: redis://127.0.0.1:6379/1 2022/01/26 21:52:17.489668 juicefs[31518] <INFO>: Ping redis: 55.542s volume name: macjfs volume UUID: eabb96d5-7228-461e-9240-fddbf2b576d8 data storage: file://jfs/ used bytes: 18620416 used inodes: 23 WARNING: The target volume will be destroyed permanently, including: WARNING: 1. objects in the data storage WARNING: 2. entries in the metadata engine Proceed anyway? [y/N]: y deleting objects: 68 The volume has been destroyed! You may need to delete cache directory manually. ``` When destroying a file system, the client will issue a confirmation prompt. Please make sure to check the file system information carefully and enter `y` after confirming it is correct. ```shell 2022/01/26 21:47:30.949149 juicefs[31483] <FATAL>: 1 sessions are active, please disconnect them first ``` If you receive an error like the one above, which indicates that the file system has not been properly unmounted, please check and confirm that all mount points are unmounted before proceeding.
Project: Project Calico (Runtime / Cloud Native Network)
File: template-resources.md

Template resources are written in TOML and define a single template resource. Template resources are stored under the `/etc/confd/conf.d` directory by default. `dest` (string) - The target file. `keys` (array of strings) - An array of keys. `src` (string) - The relative path of a . `gid` (int) - The gid that should own the file. Defaults to the effective gid. `mode` (string) - The permission mode of the file. `uid` (int) - The uid that should own the file. Defaults to the effective uid. `reload_cmd` (string) - The command to reload config. `check_cmd` (string) - The command to check config. Use `{{.src}}` to reference the rendered source template. `prefix` (string) - The string to prefix to keys. When using the `reload_cmd` feature it's important that the command exits on its own. The reload command is not managed by confd, and will block the configuration run until it exits. ```TOML [template] src = "nginx.conf.tmpl" dest = "/etc/nginx/nginx.conf" uid = 0 gid = 0 mode = "0644" keys = [ "/nginx", ] check_cmd = "/usr/sbin/nginx -t -c {{.src}}" reload_cmd = "/usr/sbin/service nginx restart" ```
Project: iSulad (Runtime / Container Runtime)
File: include_what_you_use_guide.md

| Author | | | | -- | | Date | 2022-08-11 | | Email | knowledgehao@163.com | ```bash $ git clone https://github.com/include-what-you-use/include-what-you-use.git $ cd include-what-you-use $ git checkout clang_14 $ mkdir build && cd build $ cmake -G "Unix Makefiles" -DCMAKEPREFIXPATH=/usr/lib/llvm-7 .. $ make -j8 $ cp bin/include-what-you-use /usr/bin/include-what-you-use ``` For more details, please refer to . Take for example: ```bash $ mkdir build && cd build $ CC="clang" CXX="clang++" cmake -DCMAKECINCLUDEWHATYOU_USE=include-what-you-use .. $ make > iwyu.log 2>&1 ``` You can view the analysis report through the log file `iwyu.log`. Also, you can use the keyword `should remove` to quickly view unnecessary header files and remove them. Note: After deletion, you need to compile and verify. The effect after optimization: . This tool can be used later to optimize unnecessary header files in `iSulad`. https://github.com/include-what-you-use/include-what-you-use
Project: Soda Foundation (Runtime / Cloud Native Storage)
File: HACKING.md

Clustering : etcd For easy installation, the download link of etcd binary file is at: https://github.com/coreos/etcd/releases You can just use the command followed to set up etcd service: ```shell curl -L https://github.com/coreos/etcd/releases/download/v3.2.0/etcd-v3.2.0-linux-amd64.tar.gz -o $HOME/etcd-v3.2.0-linux-amd64.tar.gz cd $HOME && tar xzvf etcd-v3.2.0-linux-amd64.tar.gz cd etcd-v3.2.0-linux-amd64 && nohup ./etcd &>>etcd.log & ``` Infrastructre : OpenStack, Bare-metal(LVM, Ceph), OceanStor DJ, CoprHD or ... As a Software-Defined-Storage controller, OpenSDS must be able to connect to backend storage environment. You can just deploy OpenSDS in any of these environments directly to avoid troublesomes. Since OpenSDS is writen in Go, the user needs to utilize the golang-client for interacting with OpenStack infrastructures. If you have infrastructures other than OpenStack, you could develop the corresponding plugin for your infrastructure. Language : Go environment To run a Go project, configuring Gopath is indispensable. After downloading the project, you should add "pathtoOpenSDS" in GOPATH environ variable.
Project: Cilium (Runtime / Cloud Native Network)
File: cilium-dbg_prefilter_update.md

<!-- This file was autogenerated via cilium cmdref, do not edit manually--> Update CIDR filters ``` cilium-dbg prefilter update [flags] ``` ``` --cidr strings List of CIDR prefixes to block -h, --help help for update --revision uint Update revision ``` ``` --config string Config file (default is $HOME/.cilium.yaml) -D, --debug Enable debug messages -H, --host string URI to server-side API ``` - Manage XDP CIDR filters