page_title: In-situ processing
section_title: Efficiency and utilization
hierarchical_section_title: In-situ processing / Efficiency and utilization
caption_attribution_description: English: This image shows the diagram of a cluster that is equipped with computational storage devices (CSDs) based on the in-storage processing technology that is configured to run distributed applications based on MPI.
context_page_description: In-situ processing also known as in-storage processing is a computer science term that refers to processing data where it resides. In-situ means "situated in the original, natural, or existing place or position." An in-situ process processes data where it is stored such as in solid-state drives or memory devices like NVDIMM, rather than sending the data to a computer's central processing unit.
  The technology utilizes embedded processing engines inside the storage devices to make them capable of running user applications in-place, so data does not need to leave the device to be processed. The technology is not new, but modern SSD architecture, as well as the availability of powerful embedded processors, make it more appealing to run user applications in-place. SSDs deliver higher data throughput in comparison to hard disk drives. Additionally, in contrast to the HDDs, the SSDs can handle multiple I/O commands at the same time.
  The SSDs contain a considerable amount of processing horsepower for managing flash memory array and providing a high-speed interface to host machines. These processing capabilities can provide an environment to run user applications in-place.
context_section_description: The computational storage technology minimizes the data movements in a cluster and also increases the processing horsepower of the cluster by augmenting power-efficient processing engines to the whole system. This technology can potentially be applied to both HDDs and SSDs; however, modern SSD architecture provides better tools for developing such technologies. The SSDs which can run user application in-place are called computational storage devices (CSDs). These storage units are augmentable processing resources, which means they are not designed to replace the high-end processors of modern servers. Instead, they can collaborate with the host’s CPU and augment their efficient processing horsepower to the system. The scientific article “Computational storage: an efficient and scalable platform for big data and HPC applications” which is published by Springer Publishing under open access policy (free for the public to access) shows the benefits of CSD utilization in the clusters.
  Examples of in-storage processing can be seen in fields like visualization efforts, biology and chemistry. This showcases how this technology allows for actions and results to be seen more efficiently than through data movement, regardless of the data being moved. The following figures (from ) show how CSDs can be utilized in an Apache Hadoop cluster and on a Message Passing Interface-based distributed environment.
caption: CSD-equipped MPI-based cluster
exif: {}