Project: CubeFS (Runtime / Cloud Native Storage)
File: scheduler.md

`Scheduler` is a background task management module, mainly responsible for balancing, disk repair, node offline, deletion, repair and other background tasks. The configuration of the Scheduler is based on the , and the following configuration instructions mainly focus on the private configuration of the Scheduler. | Configuration Item | Description | Required | |:-|:--|:--| | Public configuration | Such as service port, running logs, audit logs, etc., refer to the section | Yes | | cluster_id | Cluster ID, unified ID within the cluster | Yes | | services | List of all nodes of the Scheduler | Yes, refer to the example | | service_register | Service registration information | Yes, refer to the example | | clustermgr | Clustermgr client initialization configuration | Yes, clustermgr service address needs to be configured | | proxy | Proxy client initialization configuration | No, refer to the rpc configuration example | | blobnode | BlobNode client initialization configuration | No, refer to the rpc configuration example | | kafka | Kafka related configuration | Yes | | balance | Load balancing task parameter configuration | No | | disk_drop | Disk offline task parameter configuration | No | | disk_repair | Disk repair task parameter configuration | No | | volume_inspect | Volume inspection task parameter configuration (this volume refers to the volume in the erasure code subsystem) | No | | shard_repair | Repair task parameter configuration | Yes, the directory for storing orphan data logs needs to be configured | | blob_delete | Deletion task parameter configuration | Yes, the directory for storing deletion logs needs to be configured | | topologyupdateinterval_min | Configure the time interval for updating the cluster topology | No, default is 1 minute | | volumecacheupdateintervals | Volume cache update frequency to avoid frequent updates of volumes in a short period of time | No, default is 10s | | freechunkcounter_buckets | Bucket access for freechunk indicators | No, default is \[1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000\] | | task_log | Record information of completed background tasks for backup | Yes, directory needs to be configured, chunkbits default is 29 | leader, ID of the main node node_id, ID of the current node members, all Scheduler nodes ```json { "leader": 1, "node_id": 1, "members": { "1": "127.0.0.1:9800", "2": "127.0.0.2:9800" } } ``` host, service address of the current node idc, IDC information of the current node ```json { "host": "http://127.0.0.1:9800", "idc": "z0" } ``` hosts, clustermgr service list Other RPC parameters, refer to the general module description ```json { "hosts": [ "http://127.0.0.1:9998", "http://127.0.0.2:9998",

] } ``` ::: tip Note Starting from v3.3.0, consumer groups are supported. For previous versions, please refer to the corresponding configuration file. ::: broker_list, Kafka node list failmsgsendertimeoutms, timeout for resending messages to the failed topic after message consumption fails, default is 1000ms version, kafka version, default is 2.1.0 topicsconsume topics shardrepair, normal topic, default are `shardrepair` and `shardrepairprior` shardrepairfailed, failed topic, default is `shardrepairfailed` blobdelete, normal topic, default is `blobdelete` blobdeletefailed, failed topic, default is `blobdeletefailed` ```json { "broker_list": ["127.0.0.1:9095","127.0.0.1:9095","127.0.0.1:9095"], "failmsgsendertimeoutms": 1000, "version": "0.10.2.0", "topics": { "shard_repair": [ "shard_repair", "shardrepairprior" ], "shardrepairfailed": "shardrepairfailed", "blobdelete": "blobdelete", "blobdeletefailed": "blobdeletefailed" } } ``` diskconcurrency, the maximum number of disks allowed to be balanced simultaneously, default is 1 (before v3.3.0, this value was balancediskcntlimit, default is 100) maxdiskfreechunkcnt, when balancing, it will be judged whether there are disks with freechunk greater than or equal to this value in the current IDC. If not, no balance will be initiated. The default is 1024. mindiskfreechunkcnt, disks with freechunk less than this value will be balanced, default is 20 preparequeueretrydelays, retry interval for the preparation queue when a task in the preparation queue fails to execute, default is 10 finishqueueretrydelays, retry interval for the completion queue, default is 10 cancelpunishduration_s, retry interval after task cancellation, default is 20 workqueuesize, size of the queue for executing tasks, default is 20 collecttaskinterval_s, time interval for collecting tasks, default is 5 checktaskinterval_s, time interval for task verification, default is 5 ```json { "disk_concurrency": 700, "maxdiskfreechunkcnt": 500, "mindiskfreechunkcnt": 105, "preparequeueretrydelays": 60, "finishqueueretrydelays": 60, "cancelpunishduration_s": 60, "workqueuesize": 600, "collecttaskinterval_s": 10, "checktaskinterval_s": 1 } ``` ::: tip Note Starting from version v3.3.0, concurrent disk offline is supported. ::: preparequeueretrydelays, retry interval for the preparation queue when a task in the preparation queue fails to execute, default is 10 finishqueueretrydelays, retry interval for the completion queue, default is 10 cancelpunishduration_s, retry interval after task cancellation, default is 20 workqueuesize, size of the queue for executing tasks, default is 20 collecttaskinterval_s, time interval for collecting tasks, default is 5 checktaskinterval_s, time interval for task verification, default is 5 disk_concurrency, the number of disks to be offline concurrently, default is 1 ```json { "preparequeueretrydelays": 60, "finishqueueretrydelays": 60, "cancelpunishduration_s": 60, "workqueuesize": 600, "collecttaskinterval_s": 10, "checktaskinterval_s": 1, "disk_concurrency": 1 } ``` ::: tip Note Starting from version v3.3.0, concurrent disk repair is

::: preparequeueretrydelays, retry interval for the preparation queue when a task in the preparation queue fails to execute, default is 10 finishqueueretrydelays, retry interval for the completion queue, default is 10 cancelpunishduration_s, retry interval after task cancellation, default is 20 workqueuesize, size of the queue for executing tasks, default is 20 collecttaskinterval_s, time interval for collecting tasks, default is 5 checktaskinterval_s, time interval for task verification, default is 5 disk_concurrency, the number of disks to be repaired concurrently, default is 1 ```json { "preparequeueretrydelays": 60, "finishqueueretrydelays": 60, "cancelpunishduration_s": 60, "workqueuesize": 600, "collecttaskinterval_s": 10, "checktaskinterval_s": 1, "disk_concurrency": 1 } ``` inspectintervals, inspection time interval, default is 1s inspect_batch, batch inspection volume size, default is 1000 listvolstep, the size of requesting clustermgr to list volumes, which can control the QPS of requesting clustermgr, default is 100 listvolinterval_ms, time interval for requesting clustermgr to list volumes, default is 10ms timeout_ms, time interval for checking whether a batch of inspection tasks is completed, default is 10000ms ```json { "inspectintervals": 100, "inspect_batch": 10, "listvolstep": 20, "listvolinterval_ms": 10, "timeout_ms": 10000 } ``` taskpoolsize, concurrency of repair tasks, default is 10 messagepunishthreshold, Punishment threshold, if the corresponding number of failed attempts to consume a message exceeds this value, a punishment will be imposed for a period of time to avoid excessive retries within a short period. The default value is 3. messagepunishtime_m, punishment time, default 10 minutes orphanshardlog, record information of orphan data repair failures, directory needs to be configured, chunkbits is the log file rotation size, default is 29 (2^29 bytes) ```json { "taskpoolsize": 10, "messagepunishthreshold": 3, "messagepunishtime_m": 10, "orphanshardlog": { "dir": "/home/service/scheduler/package/orphanshard_log", "chunkbits": 29 } } ``` ::: tip Note Starting from version v3.3.0, it is supported to configure the data deletion time period. ::: taskpoolsize, concurrency of deletion tasks, default is 10 safedelaytime_h, deletion protection period, default is 72h. If a negative value is configured, the data will be deleted directly. messagepunishthreshold, Punishment threshold, if the corresponding number of failed attempts to consume a message exceeds this value, a punishment will be imposed for a period of time to avoid excessive retries within a short period. The default value is 3. messagepunishtime_m, punishment time, default 10 minutes messageslowdowntimes, slow down when it overload, default 3 second delete_log, directory for storing deletion logs, needs to be configured, chunkbits default is 29 deletehourrange, supports configuring the deletion time period in 24-hour format. For example, the following configuration indicates that deletion requests will only be initiated during the time period between 1:00 a.m. and 3:00 a.m. If not configured, deletion will be performed all day. maxbatchsize, batch consumption size of kafka messages, default is 10. If the batch is full or the time interval is reached, consume the Kafka messages accumulated during this period batchintervals, time interval for consuming kafka messages, default is 2s ```json { "taskpoolsize": 400, "messagepunishthreshold": 3, "messagepunishtime_m": 10, "messageslowdowntimes": 3, "safedelaytime_h": 12, "deletehourrange": { "from": 1, "to": 3 }, "maxbatchsize": 10, "batchintervals": 2, "delete_log": { "dir": "/home/service/scheduler/package/deletelog", "chunkbits": 29 } } ```