METHOD AND SYSTEM TO PERFORM COMPLIANCE AND AVAILABILITY CHECK FOR INTERNET SMALL COMPUTER SYSTEM INTERFACE (ISCSI) SERVICE IN DISTRIBUTED STORAGE SYSTEM

One example method for a host in a virtual storage area network (vSAN) cluster to support vSAN Internet small computer system interface (iSCSI) target services in a distributed storage system of a virtualization system is disclosed. The method includes obtaining ownership information of a target and determining, from the ownership information, whether the host is an owner of the target. In response to determining that the host is the owner of the target, the method further includes determining whether the host commits to a policy provided by the vSAN to support the vSAN iSCSI target services. In response to determining that the host fails to commit to the policy, the method includes reporting a warning message.

BACKGROUND

A virtualization software suite for implementing and managing virtual infrastructures in a virtualized computing environment may include (1) a hypervisor that implements virtual machines (VMs) on one or more physical hosts, (2) a virtual storage area network (e.g., vSAN) software that aggregates local storage to form a shared datastore for a vSAN cluster of hosts, and (3) a management server software that centrally provisions and manages virtual datacenters, VMs, hosts, clusters, datastores, and virtual networks. For illustration purposes only, one example of the vSAN may be VMware vSAN™. The vSAN software may be implemented as part of the hypervisor software.

The vSAN software uses the concept of a disk group as a container for solid-state drives (SSDs) and non-SSDs, such as hard disk drives (HDDs). On each host (node) in a vSAN cluster, local drives are organized into one or more disk groups. Each disk group includes one SSD that serves as a read cache and write buffer (e.g., a cache tier), and one or more SSDs or non-SSDs that serve as permanent storage (e.g., a capacity tier). The disk groups from all nodes in the vSAN cluster may be aggregated to form a vSAN datastore distributed and shared across the nodes in the vSAN cluster.

The vSAN software stores and manages data in the form of data containers called objects. An object is a logical volume that has its data and metadata distributed across the vSAN cluster. For example, every virtual machine disk (VMDK) is an object, as is every snapshot. For namespace objects, the vSAN software leverages virtual machine file system (VMFS) as the file system to store files within the namespace objects. A virtual machine (VM) is provisioned on a vSAN datastore as a VM home namespace object, which stores metadata files of the VM including descriptor files for the VM's VMDKs.

vSAN introduces a converged storage-compute platform where VMs are running on hosts as usual while a small percentage of CPU and memory resources is used to serve the storage needs of the same VMs. vSAN enables administrators to specify storage attributes, such as capacity, performance, and availability, in the form of one or more policies on a per-VM basis. vSAN offers many advantages over traditional storage, including scalability, simplicity, and lower total cost of ownership based on the policies.

Internet small computer system interface (iSCSI) is a transport layer protocol that describes how small computer system interface (SCSI) packets are transported over a transmission control protocol/Internet protocol (TCP/IP) network. vSAN iSCSI target (VIT) service allows hosts and physical workloads that reside outside a vSAN cluster to access a vSAN datastore of the vSAN cluster. VIT service enables an iSCSI initiator on a remote host to access block-level data of an iSCSI target in the vSAN cluster. After enabling and configuring VIT service on the vSAN cluster, a user can discover the iSCSI target from the remote host.

VIT service may also provide high availability (HA) and failures to tolerate (FTT) features based on policies provided by vSAN. However, hosts supporting the VIT service may be suffered from some issues which cause difficulties to provide these features.

DETAILED DESCRIPTION

To support virtual storage area network (vSAN) Internet small computer system interface (ISCSI) target services, the following components are generally involved: (1) target, (2) distributed storage device, (3) discovery node (DN), and (4) storage node (SN).

A target can be a container for one or more distributed storage devices, which are typically identified using one or more logical unit numbers (LUNs). In some instances and throughout the following paragraphs, the term “LUN” can also refer to the distributed storage device itself. An initiator connects to a target via an owner of the target and then accesses the one or more LUNs in the target.

A DN is a host that can act as a discovery portal for vSAN iSCSI services that an initiator may access to discover available targets.

A SN is a host that can process iSCSI input/outputs (I/Os) to one or more LUNs within a target. Typically, a SN is also the owner of the target that the SN can access.

FIG.1illustrates an example of virtualization system100that supports vSAN iSCSI target service, in accordance with some embodiments of the present disclosure. Virtualization system100includes underlying hardware such as hosts101,102and103, a management entity108and communication network150(e.g., LAN, WAN, not shown) to interconnect host101,102,103and management entity108. Hosts101,102and103form vSAN cluster120. AlthoughFIG.1illustrates three hosts101,102and103in cluster120, it will be appreciated that cluster120may include additional (or fewer) hosts. Throughout this disclosure, the terms “host(s)” and “node(s)” are used interchangeably.

Host101may include one or more hard disk drives (HDDs)121connected to host101. Similarly, hosts102and103also include HDDs122and123connected to hosts102and103, respectively. In some embodiments, HDDs121,122and123may be configured according to the SCSI (Small Computer System Interface) protocol, and hosts101,102and103may communicate with HDDs121,122and123using the SCSI protocol, respectively.

Hosts101,102and103may also include solid state drive or disk (SSD)124,125and126, respectively. Any of hosts101,102and103may be configured with a hypervisor (not shown for simplicity). The hypervisor may be a combination of computer software, firmware, and/or hardware that supports the execution of virtual machines (VMs) on hosts101,102and103.

Virtualization system100may include virtualized storage system104that provides virtual distributed datastore142. Distributed datastore142may include an aggregation of HDDs121,122,123and SSDs124,125,126. In some embodiments, HDDs121,122and123may be used to provide persistent storage in distributed datastore142, while SSDs124,125and126may serve as read and write caches for data I/O operations. The VMs deployed on hosts101,102and103may access distributed datastore142via a virtual storage interface (VS I/F) comprising commands and protocols defined by virtual storage system104.

Virtualized storage system104may allocate storage from distributed datastore142to define distributed storage devices144and145(also referred to as virtual disks). Distributed storage devices144and145may include all or part of HDDs121,122and123in cluster120. In addition, HDDs121,122and123may include SCSI-based storage devices that provide block-based storage of data. To illustrate, target1includes distributed storage device144corresponding to LUN-1and target2includes distributed storage device145corresponding to LUN-2. LUN-1and LUN-2are shown to be supported by at all or part of HDDs121,122and123.

As an illustration, host101inFIG.1is both a DN and a SN. In some embodiments, virtualized storage system104is configured to provide HA and FTT features based on one or more policies. For illustration, assuming a policy is set to be FTT=1 (referring to that virtualized storage system104is configured to tolerate one failure for an initiator to access a target), virtualized storage system104may be configured to provide two owners for a target to commit to the policy FTT=1. For example, virtualized storage system104is configured to nominate host101as an active owner of target1and host103as a candidate owner of target1.

More specifically, in some embodiments, initiator106, which may be a computer system that is separate from the hosts in cluster120, obtains the Internet Protocol (IP) address of host101and performs a login/authentication/target discovery sequence with host101. After successfully completing the sequence and ensuring that host101is indeed the active owner of target1, initiator106may then perform iSCSI-based Input/Output (I/O) operations to access LUN-1144in target1via host101as the active owner of target1.

In scenarios that initiator106cannot successfully complete the login/authentication/target discovery sequence with host101, initiator106may then obtains the IP address of host103(i.e., candidate owner of target1) and performs the login/authentication/target discovery sequence with host103to try to perform iSCSI-based I/O operations to access LUN-1144in target1via host103.

Virtualization system100may include additional infrastructure to support the vSAN iSCSI target service. In some embodiments, the infrastructure may include cluster monitoring membership and directory service (CMMDS)134which can distribute configurations or notifications in cluster120. Each host may subscribe to CMMDS134. In some embodiments, CMMDS134may have access to a datastore to maintain a list of subscribed hosts in cluster120and also owners of vSAN iSCSI targets. Any host in cluster120(e.g., host101,102or103) may announce changes of its configurations to cluster120. CMMDS134may notify subscribed hosts of the changes. Configurations may include information relating to an iSCSI target, such as, without limitation, its LUNs, the size of the LUNs, the status of the LUNs (e.g., online and offline), its universally unique identifier (UUID), etc. In addition, an ownership of a target is also published in CMMDS134. The published ownership includes both the active ownership to a specific target and the candidate ownership to the specific target. Therefore, the subscribed hosts (e.g., hosts101,102and103) in cluster120will learn which hosts in cluster120are the active owner and the candidate owner(s) of a specific target.

Virtualization system100may also include management entity108which provides management functionalities to various managed objects, such as cluster120, hosts101,102,103and VMs on hosts101,102,103. In addition, information may be transmitted among hosts101,102and103and management entity108through communication network150.

FIG.2is a simplified representation of a virtualization system200that supports vSAN iSCSI target service, in accordance with some embodiments of the present disclosure. In conjunction withFIG.1, in some embodiments, virtualization system200corresponds to virtualization system100; cluster220corresponds to cluster120; and host222, host224and host225correspond to hosts101,103and102, respectively. Although cluster220inFIG.2includes host222, host224, host225, host226and host228, cluster220may include more or fewer hosts. Virtualization system200may also include management entity242which provides management functionalities to various managed objects, such as cluster220, hosts222,224,225,226and228and VMs on these hosts. Virtualization system200also includes LUN-1232in target1and LUN-2234in target2.

In some embodiments, to commit to a set policy of FTT=1, cluster220is configured to provide two owner hosts for a target. For example, host222is configured to be an active owner of target1and host224is configured to be a candidate owner of target1. Similarly, host226is configured to be an active owner of target2and host228is configured to be a candidate owner of target2.

In some embodiments, initiator-1212is configured to perform iSCSI-based I/O operations to access LUN-1232in target1. Initiator-1212is configured to perform the operations via active owner host222via connection251. Active owner host222is then configured to perform iSCSI-based I/O operations on LUN-1232.

In some embodiments, host224is a candidate owner of target1. As a candidate owner of target1, to commit to the policy of FTT=1, host224is configured to perform iSCSI-based I/O operations on LUN-1232when the iSCSI-based I/O operations cannot be performed on LUN-1232via active owner host222.

Similarly, initiator-2214is configured to perform iSCSI-based I/O operations to access LUN-2234in target2. Initiator-2214is configured to perform the operations via active owner host226via connection261. Active owner host226is then configured to perform iSCSI-based I/O operations on LUN-2234. To commit to the policy of FTT=1, candidate owner host228is configured to perform iSCSI-based I/O operations on LUN-2234when the iSCSI-based I/O operations cannot be performed on LUN-2234via active owner host226to.

However, in some scenarios, candidate owner hosts224and228may fail to commit to the policy of FTT=1. For example, connections252and262may have network connection issues so that initiators212and214cannot connect to hosts224and228, respectively. Therefore, initiators212and214cannot access LUN-1232and LUN-2234via hosts224and228, respectively. In other scenarios, candidate owner hosts224and228may experience resource shortages (e.g., lacking computing resources) and cannot process iSCSI-based I/O operations on LUN-1232and LUN-2234, respectively. Such network connection issues and resource shortages will cause failures to commit to the policy of FTT=1 in virtualized system200which results in errors or downgrades of the vSAN iSCSI target service.

FIG.3illustrates a flowchart of example process300for a host in a vSAN cluster to support vSAN iSCSI target service, in accordance with some embodiments of the present disclosure. Example process300may include one or more operations, functions, or actions illustrated by one or more blocks, such as310to350. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated depending on the desired implementation.

Process300may begin with block310“obtain ownership information”. In some embodiments, in conjunction withFIG.1, at block310, hosts101,102and103may subscribe CMMDS134for an event of an ownership change of a target. The ownership may include an active ownership of the target and one or more candidate ownership of the target. For illustration, host101is configured to obtain ownership information from CMMDS134indicating it is the active owner of target1. Similarly, host103is configured to obtain ownership information from CMMDS134indicating it is the candidate owner of target1. Host102is configured to obtain ownership information from CMMDS134indicating it is not an owner of a target. Process300may be followed by block320“owner of target?”.

In some embodiments, at block320, a host is configured to determine whether the host itself is an active owner of a target or a candidate owner of a target based on ownership information obtained at block310. If the host (e.g., host102inFIG.1) determines the host itself not an active owner or a candidate owner of any target, process300ends. If the host (e.g., host101or103inFIG.1) determines the host itself being an active owner or a candidate owner of any target, process300may be followed by block330“check connection with initiator.”

In some embodiments, at block330, a host having an ownership to a target is configured to periodically check a network connection with an initiator to access the target. In conjunction withFIG.2, host222is configured to periodically check status of connection251to initiator-1212and host224is also configured to periodically check status of connection252to initiator-1212. Similarly, host226is configured to periodically check status of connection261to initiator-2214and host228is also configured to periodically check status of connection262to initiator-2214. Process300may be followed by block340“connection issue with initiator?”

In some embodiments, as an active owner, at block340, in conjunction withFIG.2, in response to host222determining there is no connection issues with initiator-1212, process300may be looped back to block330. On the other hand, in response to host222determining these is a connection issue (e.g., network speed lower than a threshold or loss connection) with initiator-1212, process300may be followed by block350“report warning message.” In some embodiments, the warning message includes, but not limited to, UUID of the LUN at issue (i.e., LUN-1), the IP address of the host at issue (i.e., host222), the IP address of the initiator at issue (i.e., initiator-1212) and network connection parameters of connection at issue (i.e., connection251). At block350, host222is configured to report a warning message associated with the connection issues to management entity242through connection255. In some embodiments, management entity242is configured to manage cluster220to commit to policy of FTT=1 so that initiator-1212can access LUN-1232in target1via the candidate owner host224of target1. In some embodiments, host226as an active owner host of target2will also perform similar operations of host222.

In addition, as a candidate owner, at block340, in conjunction withFIG.2, in response to host224determining there is no connection issues with initiator-1212, process300may be looped back to block330. On the other hand, in response to host224determining these is a connection issue (e.g., network speed lower than a threshold or loss connection) with initiator-1212, process300may be followed by block350“report warning message.” At block350, host224is configured to report a warning message associated with the connection issues to management entity242through connection256. In some embodiments, the warning message includes, but not limited to, UUID of the LUN at issue (i.e., LUN-1), the IP address of the host at issue (i.e., host224), the IP address of the initiator at issue (i.e., initiator-1212) and network connection parameters of connection at issue (i.e., connection252). In some embodiments, management entity242is configured to manage cluster220to commit to the policy of FTT=1. Given host224fails to commit to the policy of FTT=1, host224is configured to update an ownership of target1through CMMDS to indicate that host224is no longer a candidate owner of target1. Cluster220is then configured to nominate another host (e.g., host225) in cluster220to be the candidate owner of target1and publish an event of ownership change of target1through CMMDS indicating host225is now a candidate owner of target1. Host225is then configured to perform operations in process300. In some embodiments, host228as a candidate owner host of target2will also perform similar operations of host224. Cluster220is configured to monitor a connection between the newly candidate owner host225and initiator (i.e., initiator-1212). In response to an issue of the monitored connection being detected, cluster220is configured to nominate another host to be the newly candidate owner host.

FIG.4illustrates a flowchart of example process400for a host in a vSAN cluster to support vSAN iSCSI target service, in accordance with some embodiments of the present disclosure. Example process400may include one or more operations, functions, or actions illustrated by one or more blocks, such as410to460. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated depending on the desired implementation. In conjunction withFIG.1andFIG.2, in some embodiments, process400may also be performed by management entity108or242.

Process400may begin with block410“obtain ownership information”. In some embodiments, in conjunction withFIG.1, at block410, hosts101,102and103may subscribe CMMDS134for an event of an ownership change of a target. The ownership may include an active ownership of the target and one or more candidate ownership of the target. For illustration, host101is configured to obtain ownership information from CMMDS134indicating it is the active owner of target1. Similarly, host103is configured to obtain ownership information from CMMDS134indicating it is the candidate owner of target1. Host102is configured to obtain ownership information from CMMDS134indicating it is not an owner of a target. Process400may be followed by block420“owner of target?”.

In some embodiments, at block420, a host is configured to determine whether the host itself is an active owner of a target or a candidate owner of a target based on ownership information obtained at block410. If the host (e.g., host102inFIG.1) determines the host itself not an active owner or a candidate owner of any target, process400ends. If the host (e.g., host101or103inFIG.1) determines the host itself being an active owner or a candidate owner of any target, process400may be followed by block430“obtain historical resource usage and iSCSI-based I/O workload.”

At block430, a host is configured to obtain historical resource usages and a iSCSI-based I/O workload in a previous time period. In some embodiments, a host having an ownership to a target is configured to obtain historical computation, memory and storage usages of the host in a previous time period (e.g., an hour preceding to the present point in time). In addition, the host is also configured to obtain its historical iSCSI-based I/O workload in the previous time period. For illustration only, assuming that historical CPU usage, the historical memory usage and the historical storage usage are 45%, 56% and 43%, respectively. In addition, the historical iSCSI-based I/O workload in the previous time period is 64%. Block430may be followed by block440“predict resource usage and iSCSI-based I/O workload at specific time.”

At block440, the host is configured to predict resource usages of the host and a iSCSI-based I/O workload of the host in an upcoming time period. In some embodiments, the host is configured to use a model fitting approach to predict resource usages of the host and a iSCSI-based I/O workload of the host in an upcoming time period based on information obtained at block430. In some embodiments, the model fitting approach includes, but not limited to, non-linear least-squares minimization and curve-fitting, Prophet modelling and long short-term memory approaches. In some embodiments, inputs of the model fitting approach includes, but not limited to, information obtained at block430, a specified upcoming time period and a default iSCSI-based I/O workload threshold. In some embodiments, the capability of the host to process isCSI-based I/O operations will downgrade when the iSCSI-based I/O workload is more than the default iSCSI-based I/O workload threshold.

In some embodiments, outputs of the model fitting approach includes, but not limited to, a predicted CPU usage of the host in the upcoming time period, a predicted memory usage of the host in the upcoming time period, a predicted storage usage of the host in the upcoming time period and a predicted iSCSI-based I/O workload in the upcoming time period. Block440may be followed by block450“predicted iSCSI-based workload exceeds threshold?.”

At block450, the host is configured to determine whether predicted iSCSI-based I/O workload at block440exceeds the default iSCSI-based I/O workload threshold. If the predicted iSCSI-based I/O workload does not exceed the default iSCSI-based I/O workload threshold, process400may calculate a first difference between the predicted usages at block440and actual CPU, memory and storage usages in the upcoming time period and a second difference between the predicted iSCSI-based I/O workload at block440and an actual iSCSI-based I/O workload in the upcoming time period based on the r-square score or the root mean square error score and update the default iSCSI-based I/O workload based on the differences. Process400may then loop back to block430.

In some embodiments, if the predicted iSCSI-based I/O workload exceed the default iSCSI-based I/O workload threshold, process400may be followed by block460. In some embodiments, in response to the predicted iSCSI-based I/O workload exceeding the default iSCSI-based I/O workload threshold, the host is configured to predict its capability to process isCSI-based I/O workloads will downgrade in the upcoming time period.

At block460, in conjunction withFIG.2, an active owner host222is configured to report a warning message associated with the iSCSI-based I/O workload issues to management entity242through connection255. In some embodiments, management entity242is configured to manage cluster220to commit to policy of FTT=1 so that initiator-1212can access LUN-1232in target1via the candidate owner host224of target1. In some embodiments, host226as an active owner host of target2will also perform similar operations of host222.

In addition, as a candidate owner, at block460, in conjunction withFIG.2, in response to host224determining there are iSCSI-based I/O workload issues in the upcoming time period, host224is configured to report a warning message associated with the iSCSI-based I/O workload issues to management entity242through connection256. In some embodiments, the warning message includes, but not limited to, UUID of the LUN at issue (i.e., LUN-1), the IP address of the host at issue (i.e., host224), the IP address of the initiator at issue (i.e., initiator-1212) and the predicted iSCSI-based I/O workload issues in the upcoming time period. In some embodiments, management entity242is configured to manage cluster220to commit to the policy of FTT=1. Given host224fails to commit to the policy of FTT=1, host224is configured to update an ownership of target1through CMMDS to indicate that host224is no longer a candidate owner of target1. Cluster220is then configured to nominate another host (e.g., host225) in cluster220to be the candidate owner of target1and publish an event of ownership change of target1through CMMDS indicating host225is now a candidate owner of target1. Host225is then configured to perform operations in process400. In some embodiments, host228as a candidate owner host of target2will also perform similar operations of host224. Cluster220is configured to monitor resource usages and iSCSI-based I/O workload of the newly candidate owner host225. In response to an issue associated with the resource usages and/or the iSCSI-based I/O workload being detected, cluster220is configured to nominate another host to be the newly candidate owner host.

The above examples can be implemented by hardware (including hardware logic circuitry), software or firmware or a combination thereof. The above examples may be implemented by any suitable computing device, computer system, etc. The computer system may include processor(s), memory unit(s) and physical NIC(s) that may communicate with each other via a communication bus, etc. The computer system may include a non-transitory computer-readable medium having stored thereon instructions or program code that, when executed by the processor, cause the processor to perform process(es) described herein with reference toFIG.1toFIG.4.

FIG.5is a block diagram of an illustrative embodiment of a computer program product500for implementing process300ofFIG.3and process400ofFIG.4, in accordance with some embodiments of the present disclosure. Computer program product500may include a signal bearing medium504. Signal bearing medium504may include one or more sets of executable instructions502that, in response to execution by, for example, one or more processors of hosts101/102/103ofFIG.1and hosts222/224/225/226/228ofFIG.2, may provide at least the functionality described above with respect toFIG.3andFIG.4.

In some implementations, signal bearing medium504may encompass a non-transitory computer readable medium508, such as, but not limited to, a solid-state drive, a hard disk drive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape, memory, etc. In some implementations, signal bearing medium504may encompass a recordable medium510, such as, but not limited to, memory, read/write (R/W) CDs, R/W DVDs, etc. In some implementations, signal bearing medium504may encompass a communications medium506, such as, but not limited to, a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.). Computer program product500may be recorded on non-transitory computer readable medium508or another similar recordable medium510.

Those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computing systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure.