Patent Description:
Embodiments described herein generally relate to a sensor storage system, and in an embodiment, but not by way of limitation, a sensor storage system that is made up of removable storage elements of non-volatile memory storage modules interconnected to a network fabric.

Modern sensor systems (e.g., radar systems) are challenged to provide cluster fabric-enabled storage having write rate capabilities and capacities that are sufficient to record and store sensor data in real time for extended periods. The aggregate data rates that must be supported by such storage systems are a function of the number of data sources, the streaming rate of the digitized data, and the additional processed data that must also be collected.

Prior methods of meeting high speed recording and storage needs have typically used hard drives, or solid-state drives in RAID (redundant array of independent discs) configurations on conventional storage interfaces (SATA (serial advanced technology attachment) or SAS (serial attached SCSI (small computer system interface)))) to produce the aggregate data rates and capacities required. These systems manifest themselves as direct-attached storage or network-attached storage driven by a single board computer (SBC) or server system. Software running on the SBC or server provides a means of receiving the sensor data and writing that data to the storage array. Rugged solutions exist, but they have limitations in their connectivity, capacity, supported data rates, and ability to support removable media. These prior solutions are challenged to provide both high performance computing (HPC) fabric and Ethernet fabric integration, both of which can support remote direct memory access (RDMA) methods, and the evolving size, weight, and power needs at extended temperatures and in ground mobile environments.

Simply put, there are no available rugged data recording and boot solutions that can connect directly to RDMA-enabled cluster and network fabrics, support removable solid-state non-volatile memory (NVMe) media, and meet challenging data rate and environmental requirements.

<CIT> discloses a system and method for a unified memory and network controller for an all-flash array (AFA) storage blade in a distributed flash storage clusters over a fabric network. The unified memory and network controller has <NUM>-way control functions including unified memory buses to cache memories and DDR4-AFA controllers, a dual-port PCIE interconnection to two host processors of gateway clusters, and four switch fabric ports for interconnections with peer controllers (e.g., AFA blades and/or chassis) in the distributed flash storage network. The AFA storage blade includes dynamic random-access memory (DRAM) and magnetoresistive random-access memory (MRAM) configured as data read/write cache buffers, and flash memory DIMM devices as primary storage. Remote data memory access (RDMA) for clients via the data caching buffers is enabled and controlled by the host processor interconnection(s), the switch fabric ports, and a unified memory bus from the unified controller to the data buffer and the flash SSDs.

<NPL> discloses a database system (DBS) that allows to perform analysis efficiently. Off-the-shelf DBMSs are often considered too heavy and slow for such usage because of their complex transaction management properties that are crucial for the usage that they were originally designed for. The paper describes the design choices for a generic DBS for packet-level traffic analysis that enable good performance and describe how they are implemented in the case of the InTraBase. Furthermore, the paper demonstrates their importance through performance measurements on the InTraBase. results provide valuable insights for researchers who intend to utilize a DBMS for packet-level traffic analysis.

<CIT> discloses an apparatus and method for supporting Quality of Service (QoS) in middleware for Data Distribution Service (DDS). The apparatus includes a QoS policy analysis module, a QoS policy management module, and a QoS policy process module. The QoS policy analysis module extracts a set of QoS policies set up by Data Centric Publish Subscribe (DCPS) and associated with publication/subscription, and analyzes the QoS policies. The QoS policy management module determines the consistency of QoS policies of the DDS, and negotiates QoS policies for DDS communication. The QoS policy process module handles the QoS policies of the DDS.

<NPL> discloses that: Sent the correct data to the right place in the right time" is the basic requirement of battlefield data distribution system. The increase of widely distributed combat nodes leads to an increased information exchanges between them which can have very different requirements for data exchange; these nodes are developed with different architecture, programming language or operating systems, which leads the heterogeneity. The existing data distributed system can't meet the combat requirements. DDS is released by Object Management Organization (OMG) for datacentric publish-subscribe systems, it's a platform independent standard. DDS decoupled the applications to information transmission; it can transfer data timely and effectively. This paper presents a comprehensive overview of the DDS and discusses how DDS to solve the typical problems existing in battlefield data distribution. The results show that DDS has the potential to be the next battlefield data distribution standard.

<CIT> discloses a processing system that includes a plurality of virtual machines which have shared access to a non-volatile solid-state memory (NVSSM) subsystem, by using remote direct memory access (RDMA). The NVSSM subsystem can include flash memory and other types of non-volatile solid-state memory. The processing system uses scatter-gather lists to specify the RDMA read and write operations. Multiple reads or writes can be combined into a single RDMA read or write, respectively, which can then be decomposed and executed as multiple reads or writes, respectively, in the NVSSM subsystem. Memory accesses generated by a single RDMA read or write may be directed to different memory devices in the NVSSM subsystem, which may include different forms of non-volatile solid-state memory.

<CIT> discloses a storage controller coupled to a storage array comprising one or more storage devices that performs at least one data reduction operation on decrypted data, encrypts the reduced data using a second encryption key to generate a second encrypted data, and stores the second encrypted data on the storage array.

<CIT> discloses that A RAID device stripes a data block across N disk drives. The RAID device receives a storage request from a host computer for the data block, and creates N virtual interface ("VI") queue pairs. The queue pairs form N virtual channels to the host computer. Further, the RAID device posts a descriptor to each of the queue pairs, with each descriptor referring to <NUM>/Nth of the data block. Further, the RAID device receives <NUM>/Nth of the data block over each of the virtual channels and writes each received <NUM>/Nth data block to a different one of the N disk drives.

In an aspect, the present disclosure provides a storage system for radar sensor data comprising: a plurality of storage modules coupled together via a network fabric, each storage module comprising a plurality of form factor non-volatile memory express, NVMe, storage units; and an integrated processor coupled to the network fabric and storage modules, the integrated processor configured for sensing streamed radar data and radar data processing; wherein the integrated processor configuration comprises instructions that, when executed, cause the plurality of storage devices to receive the radar data via a data centric publish subscribe, DCPS, notification followed by a remote direct memory access, RDMA, transfer, to acquire the radar data, and allocate capacity and write rate; wherein the integrated processor comprises instructions to distribute data across the plurality of storage modules to maximize a number of the storage modules to which the radar data are written; wherein the storage modules comprise an NVMe M. <NUM> or U. <NUM> module, and the network fabric comprises a peripheral component interconnect express, PCIe, fabric; and wherein the plurality of storage modules are configured for coupling to external storage modules via the peripheral component interconnect express, PCIe, fabric, thereby expanding capacity or data rate of the storage system.

In another aspect, the present disclosure provides a process comprising: receiving radar sensor data into a sensor storage system via a data centric publish subscribe, DCPS, notification; and after receiving the radar data via the DCPS notification, transferring the radar data via a remote direct memory access, RDMA, to an integrated processor in the sensor storage system; wherein the sensor storage system comprises a plurality of storage modules coupled together via a network fabric, each storage module comprising a plurality of form factor non-volatile memory express, NVMe, storage units; wherein the integrated processor is coupled to the network fabric and storage modules and is configured for sensing streamed radar data and radar data processing; and the process comprising maximizing a number of the storage modules to which the radar data are written; wherein the storage modules comprise an NVMe M. <NUM> or U. <NUM> module, and the network fabric comprises a peripheral component interconnect express, PCIe, fabric; and coupling external storage modules via the peripheral component interconnect express, PCIe, fabric, thereby expanding capacity or data rate of the storage system.

Some embodiments are illustrated by way of example, and not limitation, in the figures of the accompanying drawings.

One or more embodiments relate to a sensor storage system, and in particular, a sensor storage system that is made up of removable storage elements of non-volatile memory storage modules interconnected to a network fabric. In an embodiment, a Sensor Storage Subsystem (SSS) provides a cartridge, module, or unit that includes a removable or fixed (e.g., soldered-down integrated circuits) set of storage elements composed of non-volatile memory express NVMe M. <NUM> and/or U. <NUM> form factor modules (and future form factor modules) and/or ruler form factor modules, interconnected on a PCle (peripheral component interconnect express) fabric, sourced by an integrated processing system providing direct connection to a cluster fabric performing sensor (e.g., radar) signal and data processing. An SSS embodiment operates at extended temperatures making it suited for demanding ground mobile radar deployments. An SSS embodiment also supports the encryption of source data and system initialization (boot). Data writing capacity and rates can be arbitrarily scaled to meet requirements by interconnecting one or more SSS's and by utilizing data centric publish subscribe (DCPS) mechanisms and remote direct memory access (RDMA) transfers to allocate capacity and write rate. <NUM> and/or U. <NUM> modules and NVMe infrastructure use a PCIe interconnect structure that provides high write data rates and capacity in a compact package.

An embodiment has the capability to provide an integrated (multiple type) cluster fabric attached into an extended-temperature, rugged, and modular storage system. The system uses a cartridge containing removable NVMe solid state devices as a scalable solution for sensor data storage. The system provides extendable, software-controlled capabilities that are integrated into a unit support boot, system initialization, and encryption of data.

An embodiment provides a configurable and scalable high-speed data recording and system initialization mechanism that can operate directly on a cluster fabric such as InfiniBand or RoCE (RDMA over Converged Ethernet), and provides Network Attached Storage (NAS) via Ethernet or RoCE. It can be provisioned for various forms of solid-state storage in removable media cartridges or modules. It is a liquid and/or air-cooled rugged solution for demanding environments. The modular design implementation uses standard single board computer designs running a Linux operating system. The solid-state storage options that are available include M. <NUM>, and ruler form factor (e.g., Enterprise and Data Center SSD Form Factor (EDSFF)) NVMe devices. Additionally, encryption is supported. Capacity can be increased by attaching external electrical connectors across units. In addition, this feature also allows for different storage form factors to be used simultaneously.

Data are transferred from one or more hosts to the SSS using mechanisms based on Data Centric Publish Subscribe (DCPS) notification messages followed by RDMA transfers made by the SSS to acquire the data. The software contained within the SSS moves these data to the storage devices of the SSS using operations to maximize the number of devices written to in order to provide high write-rate capabilities necessary for the collection of sensor data, for example, by utilizing multiple PCIe data busses. The configuration of the SSS allows, in the alternative, traditional NAS operation for lower demand collection needs. Ethernet, RoCE, and InfiniBand are supported network and fabric options. Software configuration over a network allows for dynamic and static partitioning and volume configuration of the storage. Offload of the data is provided using the same means in read modes as provided for in collection modes. Specifically, the SSS can be configured to generate notification messages to external hosts and source the previously stored data via RDMA to these hosts over the fabric or operate as Network Attached Storage (NAS) using conventional means such Network File System (NFS) to enable offloading of data.

In summary, an embodiment of the sensor storage subsystem (SSS) permits multiple storage modules to be attached to a cluster fabric. The SSS supports multiple types of fabric interfaces, and supports multiple form factors of NVMe storage using different tray configurations. The SSS can include one or more of an OpenVPX or other standard form factor replaceable/upgradable controllers. The SSS is scalable (both upwards and downwards) in capacity and in performance. The SSS uses DCPS mechanisms followed by remote direct memory access (RDMA) data transfers over a cluster fabric interconnect. Network attached storage operation (e.g., NFS) is also supported simultaneously. The SSS offers an SBC solution for controllers that makes the controllers upgradeable to future processing technologies and fabric interconnects as well as being able to support other storage technologies. The SSS is scalable in capacity and in performance by using multiple units and connecting those units together using an external PCIe fabric. The trays or units in the SSS are replaceable, which allows alternate solid-state drive (SSD) form factors to be used for each scalable recorder element.

An example embodiment of a sensor storage subsystem (SSS) is illustrated in <FIG>, <FIG>, <FIG>, and <FIG>. <FIG> illustrates a side view of an SSS including a controller module <NUM> that includes a processor <NUM>, a storage module <NUM>, and a backplane <NUM>. <FIG> illustrates an example of a front view of a single storage module <NUM>, which supports multiple types of fabric interfaces such as InfiniBand <NUM> and Ethernet <NUM>. A PCIe expansion port <NUM> is also provided. A power port/input <NUM> is present. In an embodiment, the unit is liquid-cooled via a coolant-in port 119A and a coolant-out port 119B. The cooling system can be an air-based cooling system, a liquid-based cooling system, or a combination of an air-based and liquid-based cooling system. <FIG> illustrates several storage modules <NUM> in a stacked arrangement.

<FIG> illustrates a further embodiment of the SSS of <FIG>, <FIG>. In <FIG>, a plurality of storage modules <NUM> is coupled together and coupled to a network fabric <NUM>. In an embodiment, the storage modules <NUM> and the control module <NUM> are independently removable. Also, the storage modules <NUM> are more often removed than the control module <NUM>. Therefore, since it is less complex to disconnect air-cooled connections than it is to disconnect liquid-cooled connections, the storage modules <NUM> are typically air-cooled rather that liquid-cooled. Access to network fabrics is typically done using a network switch, however it is possible to use point-to-point connections for some use cases. The network fabric includes a plurality of hosts from which the SSS collects and processes sensor data. Hosts can include such devices as radar units, computing complexes that use a network fabric (such as computing complexes that are used in High Performance Computing (HPC) systems), and/or in systems that utilize network-connected computing clusters, such as computing systems that are used in weather forecasting, and science and research applications. Each storage module <NUM> includes a plurality of form factor non-volatile memory (NVMe) storage units or soldered-down integrated circuits. The storage modules <NUM> can include one or more NVMe M. <NUM> or U. <NUM> modules. The network fabric <NUM> can include a peripheral component interconnect express (PCIe) fabric. Additionally, the network fabric <NUM> can include Ethernet <NUM>, remote direct memory access (RDMA) over Converged Ethernet (RoCE), or InfiniBand <NUM>.

<FIG> further illustrates a manner in which a processor <NUM> is integrated into a control module <NUM>, and further how the processor <NUM> can be coupled to the network fabric <NUM> and storage modules <NUM>. In an embodiment, the integrated processor <NUM> is configured for control functions and data processing. Specifically, the integrated processor configuration includes instructions such that the plurality of storage devices receive data via a data centric publish subscribe (DCPS) notification, and the DCPS notification is followed by a remote direct memory access (RDMA) transfer to the SSS processors. The SSS processors then optionally encrypt and/or directly store the captured data to storage devices in the storage module. In another embodiment, the integrated processor can also be configured for booting the system, initializing the system, and encrypting the sensed data.

As noted above, the SSS can be useful in collecting sensed data from a radar system. In other embodiments, the SSS can be used as a storage system in any computing complex that uses a network fabric, such as computing complexes that are used in High Performance Computing (HPC) systems, and/or in systems that utilize network-connected computing clusters, such as computing systems that are used in weather forecasting, and science and research applications.

In another embodiment, the integrated processor includes instructions for dynamic partitioning, static partitioning, and/or volume configuration. For example, software executing on the integrated processor(s) can determine how to allocate storage capacity. The software can execute this in several ways. It can be executed statically at system initialization from predetermined configuration files. It can also be executed dynamically while the SSS is actively processing data to be stored. In both cases, the software may designate and use portions of the total amount of storage available and collect these portions into volumes, and these volumes are used as the storage units for allocation. Remote control and configuration of the SSS is accomplished using one or more of the available network connections (e.g., Ethernet or InfiniBand) and a protocol determined to be acceptable to the SSS and the host(s) (e.g., a radar system or other computing complex).

In another embodiment, the integrated processor includes instructions for maximizing the number of the storage modules to which data are written. In one instance, several PCIe data busses can be used to maximize storage. An example is illustrated in <FIG>. Another feature of the SSS is its ability to maximize the data write rate to available storage. This maximizing can be accomplished using software executing on the integrated processors, which distribute data across several storage devices to effectively improve write rate performance by parallelizing write operations. This is executed in a manner that does not require specific (storage system) formatting of the storage devices as might be the case if traditional RAID mechanisms were used. This enables data recovery to be accomplished using ordinary file reading mechanisms provided by the host (e.g., radar) operating system. The number of devices used can be determined by the SSS software or preconfigured by configuration files.

In another embodiment, the SSS is configured to permit network-attached storage for lower demand collection needs. That is, the storage modules can be partitioned according to each particular use case. For example, the SSS can be configured to behave as traditional network-attached storage using IP protocols (e.g., NFS). The SSS software incorporates this feature, and it is configurable to be used in conjunction with other operations that are executed by the SSS (e.g., DCPS/RDMA) simultaneously. The SSS is then able to allocate storage amounts to be used for different types of storage access methods on more than one network connection at a time.

Examples, as described herein, may include, or may operate on, logic or several components, circuits, or engines, which for the sake of consistency are termed engines, although it will be understood that these terms may be used interchangeably. Engines may be hardware, software, or firmware communicatively coupled to one or more processors in order to carry out the operations described herein. Engines may be hardware engines, and as such engines may be considered tangible entities capable of performing specified operations and may be configured or arranged in a certain manner. In an example, circuits may be arranged (e.g., internally or with respect to external entities such as other circuits) in a specified manner as an engine. In an example, the whole or part of one or more computing platforms (e.g., a standalone, client or server computing platform) or one or more hardware processors may be configured by firmware or software (e.g., instructions, an application portion, or an application) as an engine that operates to perform specified operations. In an example, the software may reside on a machine-readable medium. In an example, the software, when executed by the underlying hardware of the engine, causes the hardware to perform the specified operations. Accordingly, the term hardware engine is understood to encompass a tangible entity, be that an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., transitorily) configured (e.g., programmed) to operate in a specified manner or to perform part of or all operations described herein.

Considering examples in which engines are temporarily configured, each of the engines need not be instantiated at any one moment in time. For example, where the engines comprise a general-purpose hardware processor configured using software; the general-purpose hardware processor may be configured as respective different engines at different times. Software may accordingly configure a hardware processor, for example, to constitute a particular engine at one instance of time and to constitute a different engine at a different instance of time.

The drawings show, by way of illustration, specific embodiments that may be practiced. " Such examples may include elements in addition to those shown or described. However, also contemplated are examples that include the elements shown or described. Moreover, also contemplated are examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

Claim 1:
A storage system for radar sensor data comprising:
a plurality of storage modules (<NUM>) coupled together via a network fabric (<NUM>), each storage module comprising a plurality of form factor non-volatile memory express, NVMe, storage units; and
an integrated processor (<NUM>) coupled to the network fabric and storage modules, the integrated processor configured for sensing streamed radar data and radar data processing;
wherein the integrated processor configuration comprises instructions that, when executed, cause the plurality of storage devices to receive the radar data via a data centric publish subscribe, DCPS, notification followed by a remote direct memory access, RDMA, transfer to acquire the radar data, and allocate capacity and write rate;
wherein the integrated processor comprises instructions to distribute data across the plurality of storage modules to maximize a number of the storage modules to which the radar data are written;
wherein the storage modules (<NUM>) comprise an NVMe M.<NUM> or U.<NUM> module, and the network fabric (<NUM>) comprises a peripheral component interconnect express, PCIe, fabric; and
wherein the plurality of storage modules (<NUM>) are configured for coupling to external storage modules via the peripheral component interconnect express, PCIe, fabric, thereby expanding capacity or data rate of the storage system.