Patent Description:
The present disclosure relates to the technical field of computer, in particular to a data-set-and-node-cache-based scheduling method and device.

In an artificial intelligence (AI) scenario, training tasks have a strong dependence on data sets. For example, the quality of the data sets will affect the accuracy of a model, and the speed of loading data sets by training scripts will affect the training speed of the model. The data sets used for AI training are usually an open-source data set, such as image network (ImageNet) data set, handwritten digit recognition (MNIST) data set, etc. or an industry-related data set, such as medical treatment, transportation, etc.; for an AI resource management platform, it is usually impossible to evaluate the quality of the data set but are required to be ensured by algorithm personnel. Generally, when initiating a training task, the algorithm personnel usually need to manually download these data to a node to start the training task; however, with regard to the AI resource management platform, a manual download data set is usually optimized as an automatic download data set; and when starting a training task, the AI resource management platform will automatically download the required data set for the training task. As the AI resource management platform, a variety of data sets will be provided for the algorithm personnel, and these data sets will be cached to a computing node according to the requirements of training tasks; however, due to the limited storage resources of nodes, there will be the following problems:.

In view of this, it is an object of embodiments of the present disclosure to provide a data-set-and-node-cache-based scheduling strategy, which may achieve the effect of load balancing of storage resources of cluster nodes on the premise of using local data set cache to satisfy training tasks.

The present disclosure has at least the following advantageous technical effects: the present disclosure is a scheduling strategy for selecting a node based on the sizes of a node storage and a data set required by training task in a cluster environment. According to the present disclosure, an AI training task may be operated on a host node with a required data set or a host node with sufficient node storage space, and at the same time, when the remaining space of all nodes in the cluster is insufficient, a node data set cache deletion strategy is defined, and the training task may be operated on a host node with temporarily insufficient storage space. Based on this node selection strategy, it may effectively reduce the time to download data sets and the time to wait for available nodes, thereby improving the competitiveness of the AI management platform.

In order to explain the embodiments of the present disclosure or the technical solutions in the prior art more clearly, a brief description will be given below with reference to the accompanying drawings which are used in the description of the embodiments or the prior art, and it is obvious that the drawings in the description below are merely some embodiments of the present disclosure, and it would have been obvious for a person skilled in the art to obtain other drawings according to these drawings without involving any inventive effort.

In order that the objects, aspects, and advantages of the present disclosure will become more fully apparent, embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings.

It should be noted that all the expressions using "first" and "second" in the embodiments of the present disclosure are intended to distinguish two entities with the same name but different from the same name or different parameters. "First" and "second" are merely for the convenience of expressions and should not be construed as limiting the embodiments of the present disclosure, and the subsequent embodiments will not be described one by one.

Based on the above object, in a first aspect of an embodiment of the present disclosure, an embodiment of a data-set-and-node-cache-based scheduling method is proposed. <FIG> shows a schematic block diagram of an embodiment of a data-set-and-node-cache-based scheduling method according to the present disclosure. In the embodiment shown in <FIG>, the method includes at least the following steps:.

In some embodiments of the present disclosure, <FIG> shows a flow chart of an embodiment of a data set and host node (which may be simply referred to as a node) cache-based scheduling method according to the present disclosure; as shown in <FIG>, the present disclosure relates to a scheduler extension mechanism based on Kubemetes, which uses a self-developed node agent to report the state of all the data sets of a node to a scheduler, and at the same time, the scheduler queries the operation condition of a training task of an AI resource management platform; according to the remaining storage of the node, the size of the data set cache and the number of times that the data set cache is used, data set cache cleaning strategy and other factors, nodes of the cluster are scored, and the score is combined with other scheduling strategies of Kubemetes to select the optimal node for operating the training tasks.

In some embodiments of the present disclosure, according to step S100, the storage resource information of the node is collected, including a storage space nodeiDiskTotalSize, a storage free space node;DiskFreeSize, and an information list of the data-set node cache (a unique identifier of a data set dataSetjId, a size of the data set dataSetjSize, and a number of times that the data set is used in the last one month dataSetjUseNumber). According to step S200, the user submits a training task on the resource management platform, operation information of the training task includes the used data set information including a name of the data set, a unique identifier of the data set dataSettaskId used by the task, a size of the data set dataSettaskSize used by the task, and other basic resource information (CPU, memory, graphics processing unit (GPU), etc.) for operating the training task; and after receiving the resource request of the training task, the scheduler firstly uses a kubemetes default algorithm to screen nodes with sufficient CPU, memory and GPU cards. According to step S300, when the space nodeiDiskFreeSize of all the nodes in the cluster does not satisfy the space dataSetjSize required by the data set, with regard to the node node;, when the size of the data-set node cache which is no longer used by the node is greater than or equal to the size of the data set cache used by the training task, i.e., <MAT>, the node is taken as an alternate node; for the node nodei, the data-set node caches needing to be deleted is selected, and a model for the data-set node caches is built, and each of the host nodes is scored according to the model. According to step S400, selecting, from among all the host nodes, a task-executing host node that is used to execute the training task according to the scoring results. According to step S500, when the resource scheduling module selects an appropriate node, a list of the data set cache needing to be deleted is notified to an agent, and the agent deletes the data set cache. Since a deletion operation is to delete a node file, and downloading a data set is to download from a remote end using a Hyper Text Transfer Protocol (http) service, the speed of deleting the data set cache must be much greater than the speed of downloading the data set cache. At this point, the download of the data set may begin immediately after the training task is scheduled to the node.

In some embodiments of the data-set-and-node-cache-based scheduling method according to the present disclosure, the method further includes:.

In some embodiments of the present disclosure, for the host node in the cluster, when the host node already has the data set required for the training task (i.e., the host node has the data set cache), the host node is used to operate the training task firstly, thereby avoiding downloading the data set again. When all the nodes in the cluster do not have the node cache of the data set, the node with the largest node residual space nodeiDiskFreeSize is selected to download the data set, and the training task is operated.

According to some embodiments of the data-set-and-node-cache-based scheduling method of the present disclosure, obtaining operation information of the training task in response to receiving the training task, and screening the host nodes to identify any host nodes that satisfy a requirement with respect to the space required by the training task according to the operation information and the storage resource information further includes:
selecting a host node that executes the training task from a plurality of pending host nodes based on a scheduling strategy in response to screening the plurality of pending host nodes that satisfy the requirement with respect to the space required for the training task from the host nodes.

In some embodiments of the present disclosure, after receiving the resource request of the training task, the scheduler firstly uses a kubemetes default algorithm to screen nodes with sufficient CPU, memory, and GPU cards. Based on the scheduling strategy, the node with the largest residual space dataSetjSize is selected to download the data set, and the training tasks are operated.

According to some embodiments of the data-set-and-node-cache-based scheduling method of the present disclosure, the obtaining storage resource information of each of host nodes further includes:
deploying a Kubemetes cluster within the cluster, and obtaining the storage resource information of the host node based on the Kubemetes cluster.

In some embodiments of the present disclosure, a Kubemetes cluster is deployed within the cluster, and a self-developed agent is deployed at each of the host nodes for collecting the storage resource information of the node, including a storage space nodeiDiskTotalSize, a storage free space node;DiskFreeSize, and an information list of the data-set node cache (the unique identifier of a data set dataSetjId, the size of the data set dataSetjSize, and the number of times that the data set is used in the last one month dataSetjUseNumber).

In some embodiments of the present disclosure, when the storage resource information changes, the storage resource information needs to be reported to a resource scheduling module in real time, and the resource scheduling module performs node selection and data-set node cache deletion strategy based on these.

According to some embodiments of the data-set-and-node-cache-based scheduling method of the present disclosure, the giving a score to each of the host nodes according to storage resource information in response to no host node satisfying the requirement with respect to the space required by a training task further includes:
obtaining a usage frequency of all data set caches in each of the host nodes, obtaining the obsolete data set cache in all data set caches according to the usage frequency, and giving a score to the host node according to the obsolete data set cache.

In some embodiments of the present disclosure, when the space nodeiDiskFreeSize of all the nodes in the cluster does not satisfy the space dataSetjSize required by the data set, with regard to the node node;, when the size of the data-set node cache which is no longer used by the node is greater than or equal to the size of the data set cache used by the training task, i.e., <MAT>, the node is taken as an alternate node; for the node node;, the data-set node caches needing to be deleted is selected, and a model for the node data set caches is built.

When the data set cache datasetj of the node is used more times in the last one month, it means that the data set is more likely to be used by other training tasks in a future period of time; in order to avoid downloading the data set again when a new training task needs to use the data set, in this scheduling rule, we do not choose to delete the data set cache as much as possible, and we define the weight value of selecting the data set to be deleted as follows:
<MAT>.

According to some embodiments of the data-set-and-node-cache-based scheduling method of the present disclosure, the giving a score to each of the host nodes according to the storage resource information in response to no host node satisfying the requirement with respect to the space required by a training task further includes:
determining the size of each data set cache in each of the host nodes, taking the data set cache with the size less than a preset size threshold value as the obsolete data set cache, and giving a score to the host node according to the obsolete data set cache.

In some embodiments of the present disclosure, when the size of the data-set node cache is larger, in order to reduce the time for deleting the data-set node cache, we do not select a large data set cache as much as possible to be deleted, i.e., the possibility of deleting the large data set cache is low, and we define the weight value for selecting the data set to be deleted as follows:
<MAT>.

In some embodiments of the present disclosure, for a node data set, we compute and sort according to the following formula:
<MAT>.

According to some embodiments of the data-set-and-node-cache-based scheduling device of the present disclosure, the device <NUM> further includes:
a cache determination module (not shown) configured to determine whether each of the host nodes includes a data set cache required for the training task, and select a host node that executes the training task from the host nodes including the data set cache in response to determining that there is the host node including the data set cache.

According to some embodiments of the data-set-and-node-cache-based scheduling device of the present disclosure, the host node screening module <NUM> is further configured to:
select a host node that executes the training task from a plurality of pending host nodes based on a scheduling strategy in response to screening the plurality of pending host nodes in the host nodes that satisfy the space required for the training task.

As such, a person skilled in the art will appreciate that all embodiments, features, and advantages set forth above with respect to the data-set-and-node-cache-based scheduling method according to the present disclosure apply equally to the device according to the present disclosure. For the sake of brevity of the present disclosure, this description is not repeated here.

It should be noted that a person skilled in the art would understand that the implementation of all or part of the flows in the methods of the above-mentioned embodiments may be performed by a computer program instructing relevant hardware, and a program of a data-set-and-node-cache-based scheduling method may be stored in a computer-readable storage medium, and when executed, the program may include the flows of the embodiments of the methods as described above. The storage medium of the program may be a magnetic disk, an optical disk, a ROM, read-Only Memory or RAM, random Access Memory, etc. Embodiments of the computer program described above may achieve the same or similar effects as any of the method embodiments described above corresponding thereto.

A person skilled in the art would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. Whether such functionality is implemented as software or hardware depends upon the application and design constraints imposed on the overall system. A person skilled in the art may implement the described functionality in varying ways for each application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments.

It will be understood that, as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include all possible combinations of one or more of the associated listed items.

The above-described embodiments of the present disclosure have been disclosed for the purpose of illustration only, and are not intended to represent the advantages and disadvantages of the embodiments.

Claim 1:
A data-set-and-node-cache-based scheduling method, comprising:
obtaining storage resource information of each of host nodes (S100);
obtaining operation information of a training task in response to receiving the training task, and screening the host nodes to identify any host nodes that satisfy a requirement with respect to a space required by the training task according to the operation information and the storage resource information (S200);
giving a score to each of the host nodes according to the storage resource information in response to no host node satisfying the requirement with respect to the space required by the training task (S300);
selecting, from among all the host nodes, a task-executing host node that is used to execute the training task according to scoring results (S400); and
obtaining and deleting an obsolete data set cache in the task-executing host node, and executing the training task in the task-executing host node (S500);
wherein giving the score to each of the host nodes according to the storage resource information in response to no host node satisfying the requirement with respect to the space required by the training task (S300) further comprises:
determining a size of each data set cache in each of the host nodes, taking the data set cache with the size less than a preset size threshold value as the obsolete data set cache, and giving a score to the host node according to the obsolete data set cache.