Patent ID: 12223170

DETAILED DESCRIPTION

A business like a financial or technology corporation can produce large amounts of data and require sharing access to that data among several employees. Such a business often uses storage arrays to store and manage the data. Because a storage array can include multiple storage devices (e.g., hard-disk drives (HDDs) or solid-state drives (SSDs)), the business can scale (e.g., increase or decrease) and manage an array's storage capacity more efficiently than a server. In addition, the business can use a storage array to read/write data required by one or more business applications.

Generally, a host (e.g., a client machine or application) can issue one or more IO messages to a storage array to read/write data on the storage array. In addition, a service level agreement (SLA) can define a performance corresponding to the storage array's processing of the IO messages. For example, the performance can include metrics like the storage array's response times. The response times can include a latency, defining the time to process an IO message. For example, response times can correspond to when a host issues an IO message to when the host receives an acknowledgment from the storage array.

Current naïve approaches to measuring latency cannot determine if a performance bottleneck is internal or external to/from a storage array. Embodiments of the present disclosure identify external and internal sources of latency corresponding to processing IO messages by a storage array. For example, the embodiments can identify external transfer times and internal storage array processing times corresponding to each IO message a storage array receives. The embodiments can identify entities contributing to the latency based on the external transfer times and internal storage array processing times. Further, the embodiments can identify the entities contributing to a significant portion of the latency and, thus, perform a remediation action.

RegardingFIG.1, a distributed network environment100can include a storage array102, a remote system104, and hosts106. In embodiments, the storage array102can include components108that perform one or more distributed file storage services. In addition, the storage array102can include one or more internal communication channels110like Fibre channels, busses, and communication modules that communicatively couple the components108. Further, the distributed network environment100can define an array cluster112, including the storage array102and one or more other storage arrays.

In embodiments, the storage array102, components108, and remote system104can include a variety of proprietary or commercially available single or multi-processor systems (e.g., parallel processor systems). Single or multi-processor systems can include central processing units (CPUs), graphical processing units (GPUs), and the like. Additionally, the storage array102, remote system104, and hosts106can virtualize one or more of their respective physical computing resources (e.g., processors (not shown), memory114, and persistent storage116).

In embodiments, the storage array102and, e.g., one or more hosts106(e.g., networked devices) can establish a network118. Similarly, the storage array102and a remote system104can establish a remote network120. Further, the network118or the remote network120can have a network architecture that enables networked devices to send/receive electronic communications using a communications protocol. For example, the network architecture can define a storage area network (SAN), local area network (LAN), wide area network (WAN) (e.g., the Internet), an Explicit Congestion Notification (ECN), Enabled Ethernet network, and the like. Additionally, the communications protocol can include a Remote Direct Memory Access (RDMA), TCP, IP, TCP/IP protocol, SCSI, Fibre Channel, Remote Direct Memory Access (RDMA) over Converged Ethernet (ROCE) protocol, Internet Small Computer Systems Interface (iSCSI) protocol, NVMe-over-fabrics protocol (e.g., NVMe-over-ROCEv2 and NVMe-over-TCP), and the like.

Further, the storage array102can connect to the network118or remote network120using one or more network interfaces. The network interface can include a wired/wireless connection interface, bus, data link, and the like. For example, a host adapter (HA122), e.g., a Fibre Channel Adapter (FA) and the like, can connect the storage array102to the network118(e.g., SAN). Further, the HA122can receive and direct IOs to one or more of the storage array's components108, as described in greater detail herein.

Likewise, a remote adapter (RA124) can connect the storage array102to the remote network120. Further, the network118and remote network120can include communication mediums and nodes that link the networked devices. For example, communication mediums can include cables, telephone lines, radio waves, satellites, infrared light beams, etc. The communication nodes can also include switching equipment, phone lines, repeaters, multiplexers, and satellites. Further, the network118or remote network120can include a network bridge that enables cross-network communications between, e.g., the network118and remote network120.

In embodiments, hosts106connected to the network118can include client machines126a-n, running one or more applications. The applications can require one or more of the storage array's services. Accordingly, each application can send one or more input/output (IO) messages (e.g., a read/write request or other storage service-related request) to the storage array102over the network118. Further, the IO messages can include metadata defining performance requirements according to a service level agreement (SLA) between hosts106and the storage array provider.

In embodiments, the storage array102can include a memory114, such as volatile or nonvolatile memory. Further, volatile and nonvolatile memory can include random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), and the like. Moreover, each memory type can have distinct performance characteristics (e.g., speed corresponding to reading/writing data). For instance, the types of memory can include register, shared, constant, user-defined, and the like. Furthermore, in embodiments, the memory114can include global memory (GM128) that can cache IO messages and their respective data payloads. Additionally, the memory114can include local memory (LM130) that stores instructions that the storage array's processors144can execute to perform one or more storage-related services. For example, the storage array102can have a multi-processor architecture that includes one or more CPUs (central processing units) and GPUs (graphical processing units).

In addition, the storage array102can deliver its distributed storage services using persistent storage116. For example, the persistent storage116can include multiple thin-data devices (TDATs) such as persistent storage drives132a-n. Further, each TDAT can have distinct performance capabilities (e.g., read/write speeds) like hard disk drives (HDDs) and solid-state drives (SSDs).

Further, the HA122can direct one or more IOs to an array component108based on their respective request types and metadata. In embodiments, the storage array102can include a device interface (DI134) that manages access to the array's persistent storage116. For example, the DI134can include a disk adapter (DA136) (e.g., storage device controller), flash drive interface138, and the like that control access to the array's persistent storage116(e.g., storage devices132a-n).

Likewise, the storage array102can include an Enginuity Data Services processor (EDS140) that can manage access to the array's memory114. Further, the EDS140can perform one or more memory and storage self-optimizing operations (e.g., one or more machine learning techniques) that enable fast data access. Specifically, the operations can implement techniques that deliver performance, resource availability, data integrity services, and the like based on the SLA and the performance characteristics (e.g., read/write times) of the array's memory114and persistent storage116. For example, the EDS140can deliver hosts106(e.g., client machines126a-n) remote/distributed storage services by virtualizing the storage array's memory/storage resources (memory114and persistent storage116, respectively).

In embodiments, the storage array102can also include a controller142(e.g., management system controller) that can reside externally from or within the storage array102and one or more of its components108. When external from the storage array102, the controller142can communicate with the storage array102using any known communication connections. For example, the communications connections can include a serial port, parallel port, network interface card (e.g., Ethernet), etc. Further, the controller142can include logic/circuitry that performs one or more storage-related services. For example, the controller142can have an architecture designed to manage the storage array's computing, processing, storage, and memory resources as described in greater detail herein.

RegardingFIG.2, a storage array102can include a controller142with logic, hardware, and circuitry200that perform one or more data storage services. In embodiments, the controller142can include an input/output (IO) processor202that analyzes an IO workload201and its corresponding IO messages. The IO workload201can correspond to IO messages received by the storage array102during a given interval. Additionally, the IO processor202can identify characteristics corresponding to the IO workload201and its corresponding IO messages. The IO characteristics can include type, size, access rate, thread count, and read/write ratios.

Additionally, each IO message can include metadata (e.g., incremental counters) identifying IO processing times of entities external to the storage array102. Further, the IO processor202can generate IO workload models predicting IO characteristics of future IO workloads and their corresponding IO messages. For example, the IO processor202can process current and historical IO characteristic data using a self-learning technique to generate and maintain the IO workload models in local memory210. Further, the IO processor202can parse temporal metadata (e.g., start and receipt timestamps) from the IO messages to determine their travel times from corresponding hosts to the storage array102.

In embodiments, the IO processor202can also return IO completion messages to the sources (e.g., hosts106ofFIG.1) of corresponding IO messages. For example, the IO completion messages inform the hosts that the storage array102has processed their corresponding IO messages. Further, the IO processor202can include a request for performance metrics (e.g., return response time) in an instruction field of the IO completion messages. Thus, the hosts can transmit the IO completion messages' temporal information (e.g., receipt time) to the storage array102. Using the temporal information from the IO messages and the IO completion messages, the IO processor202can determine the response time for each IO message. Thus, the IO processor202can calculate an average response time for the IO workload.

Further, the controller142can also include an IO path analyzer204that monitors the array's components (e.g., the components108ofFIG.1) involved in processing each IO message of the IO workload201. In embodiments, an IO path can define each component108involved with processing an IO message. For example, one or more daemons212can monitor the activity levels of the components108while processing IO messages. The daemons212can maintain activity logs that map each IO message (e.g., using a unique IO message identifier) to the activity of each component in each IO message's corresponding IO path in local memory210. For instance, the activity logs can record each component's processing times. Thus, the IO path analyzer204can determine the storage array's internal IO processing time using the activity logs for an IO path's corresponding components108.

In embodiments, the IO path analyzer204can generate IO path processing models predicting processing times of IO paths used to process IO messages of anticipated IO workloads. For example, using the IO workload models, the IO path analyzer204can identify IO paths and their components108for IO messages of a future IO workload.

In embodiments, the controller142can include an IO latency manager208with logic, hardware, and circuitry configured to perform one or more latency remediation operations. For example, the IO latency manager208can calculate an external response time based on the external transfer times determined by the IO processor202. Additionally, the IO latency manager208can calculate internal processing times based on the IO processing times of each component and an IO message's corresponding IO path as identified by the IO path analyzer204. Further, the IO latency manager can process current and historical data corresponding to external response and internal processing times using seasonal trend decomposition (e.g., using LOESS) and time series linear projection techniques to identify anomalies in either the external response or internal processing times.

In response to identifying an anomaly, the IO latency manager208can identify a source of the anomaly using the IO workload models and the IO path processing models. For example, the IO workload models can identify the external entities and internal storage array components contributing to each IO message's latency (e.g., total response time). Further, the IO latency manager208can perform a latency remediation action based on the identified source. For example, the local memory210can include a look-up table identifying remediation actions per identified source.

The following text includes details of a method(s) or a flow diagram(s) per embodiments of this disclosure. For simplicity of explanation, each method is depicted and described as a set of alterable operations. Additionally, one or more operations can be performed in parallel, concurrently, or in a different sequence. Further, not all the illustrated operations are required to implement each method described by this disclosure.

RegardingFIG.3, a method300relates to analyzing and mitigating storage array latency. In embodiments, the controller142ofFIG.1can perform all or a subset of operations corresponding to the method300.

For example, the method300, at302, can include receiving an input/output (IO) workload at a storage array. Additionally, at304, the method300can include determining a latency corresponding to processing one or more IO requests of the IO workload. Further, the method300, at306, can include identifying a cause of a significant portion of the latency. At308, the method300can also include performing a remediation action based on the factor identified.

Further, each operation can include any combination of techniques implemented by the embodiments described herein. Additionally, one or more of the storage array's components108can implement one or more of the operations of each method described above.

Using the teachings disclosed herein, a skilled artisan can implement the above-described systems and methods in digital electronic circuitry, computer hardware, firmware, or software. The implementation can be a computer program product. Additionally, the implementation can include a machine-readable storage device for execution by or to control the operation of a data processing apparatus. The implementation can, for example, be a programmable processor, a computer, or multiple computers.

A computer program can be in any programming language, including compiled or interpreted languages. The computer program can have any deployed form, including a stand-alone program, subroutine, element, or other units suitable for a computing environment. One or more computers can execute a deployed computer program.

One or more programmable processors can perform the method steps by executing a computer program to perform the concepts described herein by operating on input data and generating output. An apparatus can also perform the method steps. The apparatus can be a special-purpose logic circuitry. For example, the circuitry is an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Subroutines and software agents can refer to portions of the computer program, the processor, the special circuitry, software, or hardware that implements that functionality.

Processors suitable for executing a computer program include, by way of example, both general and special purpose microprocessors and any one or more processors of any digital computer. A processor can receive instructions and data from a read-only memory, a random-access memory, or both. Thus, for example, a computer's essential elements are a processor for executing instructions and one or more memory devices for storing instructions and data. Additionally, a computer can receive data from or transfer data to one or more mass storage device(s) for storing data (e.g., magnetic, magneto-optical disks, solid-state drives (SSDs, or optical disks).

Data transmission and instructions can also occur over a communications network. Information carriers that embody computer program instructions and data include all nonvolatile memory forms, including semiconductor memory devices. The information carriers can, for example, be EPROM, EEPROM, flash memory devices, magnetic disks, internal hard disks, removable disks, magneto-optical disks, CD-ROM, or DVD-ROM disks. In addition, the processor and the memory can be supplemented by or incorporated into special-purpose logic circuitry.

A computer with a display device enabling user interaction can implement the above-described techniques, such as a display, keyboard, mouse, or any other input/output peripheral. The display device can, for example, be a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor. The user can provide input to the computer (e.g., interact with a user interface element). In addition, other kinds of devices can enable user interaction. Other devices can, for example, be feedback provided to the user in any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). For example, input from the user can be in any form, including acoustic, speech, or tactile input.

A distributed computing system with a back-end component can also implement the above-described techniques. The back-end component can, for example, be a data server, a middleware component, or an application server. Further, a distributing computing system with a front-end component can implement the above-described techniques. The front-end component can, for example, be a client computer having a graphical user interface, a Web browser through which a user can interact with an example implementation or other graphical user interfaces for a transmitting device. Finally, the system's components can interconnect using any form or medium of digital data communication (e.g., a communication network). Examples of communication network(s) include a local area network (LAN), a wide area network (WAN), the Internet, a wired network(s), or a wireless network(s).

The system can include a client(s) and server(s). The client and server (e.g., a remote server) can interact through a communication network. For example, a client-and-server relationship can arise by computer programs running on the respective computers and having a client-server relationship. Further, the system can include a storage array(s) that delivers distributed storage services to the client(s) or server(s).

Packet-based network(s) can include, for example, the Internet, a carrier internet protocol (IP) network (e.g., local area network (LAN), wide area network (WAN), campus area network (CAN), metropolitan area network (MAN), home area network (HAN)), a private IP network, an IP private branch exchange (IPBX), a wireless network (e.g., radio access network (RAN), 802.11 network(s), 802.16 network(s), general packet radio service (GPRS) network, HiperLAN), or other packet-based networks. Circuit-based network(s) can include, for example, a public switched telephone network (PSTN), a private branch exchange (PBX), a wireless network, or other circuit-based networks. Finally, wireless network(s) can include RAN, Bluetooth, code-division multiple access (CDMA) networks, time division multiple access (TDMA) networks, and global systems for mobile communications (GSM) networks.

The transmitting device can include, for example, a computer, a computer with a browser device, a telephone, an IP phone, a mobile device (e.g., cellular phone, personal digital assistant (PDA) device, laptop computer, electronic mail device), or other communication devices. The browser device includes, for example, a computer (e.g., desktop computer, laptop computer) with a world wide web browser (e.g., Microsoft® Internet Explorer® and Mozilla®). The mobile computing device includes, for example, a Blackberry®.

Comprise, include, or plural forms of each are open-ended, include the listed parts, and contain additional unlisted elements. Unless explicitly disclaimed, the term ‘or’ is open-ended and includes one or more of the listed parts, items, elements, and combinations thereof.