Data processing system and data processing method

First type metadata is associated with unstructured data included in an unstructured data source. A data processing system performs an extraction process. This extraction process includes: (a) creating, for each of a plurality of selected pieces of unstructured data in the unstructured data source, second type metadata, which is metadata including content information representing one or more content attributes of the piece of unstructured data; and (b) associating the created second type metadata with the first type metadata of the piece of unstructured data.

TECHNICAL FIELD

The present invention generally relates to data processing.

BACKGROUND ART

Data managed by a storage system may be used in various applications such as search and analysis.

For example, in big-data analysis, analysis of unstructured data in which a storage structure of files and the like is yet to be determined is anticipated as a potentially useful method of obtaining new findings and realizations in business. In big-data analysis, in order to prevent situations where searches take time due to analyzing a large amount of data and completing the analysis becomes a much time-consuming process, a set made up of only data necessary for analysis may be created from the large amount of data. A set made up of only necessary data is referred to as a “data mart” (hereinafter, DM) and creating the data set is referred to as a “DM creation process”.

Generally, in big-data analysis, a DM creation process takes a large amount of time. This is because a process of duplicating (extracting) data necessary for data analysis from a large amount of data and storing the duplicated (extracted) data in the DM is time consuming. PTL 1 discloses a technique that enables data to be duplicated in a short period of time by presenting source data to a host computer as virtually-duplicated snapshot data.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

However, with the technique disclosed in PTL 1, in order to present snapshot data to the host computer, an address of a data source such as a source volume must be designated. It is difficult to apply a snapshot creation technique that requires such addressing to a process of creating a DM from an unstructured data source (for example, a source storing a large amount of unstructured data).

Such a problem is not limited to a process of creating a DM from an unstructured data source for analysis purposes and may also occur in a process of creating a data set (a subset) from an unstructured data source for applications other than analyses.

Solution to Problem

First type metadata is associated with unstructured data included in an unstructured data source. A data processing system executes an extraction process. The extraction process includes, for each piece of corresponding unstructured data in the unstructured data source: (a) creating second type metadata being metadata including content information representing one or more content attributes of the piece of unstructured data; and (b) associating the created second type metadata with the first type metadata of the piece of unstructured data.

Advantageous Effects of Invention

A virtual data set can be created which is constituted by unstructured data referred to by first type metadata associated with second type metadata complying with a condition among a plurality of pieces of second type metadata associated with a plurality of pieces of first type metadata among a plurality of pieces of unstructured data. Therefore, it is expected that a data set can be created in a short period of time.

DESCRIPTION OF EMBODIMENTS

Hereinafter, several embodiments will be described with reference to the drawings.

Moreover, in the following description, an “interface unit” includes one or more interfaces. The one or more interfaces may be one or more interface devices of a same type (for example, one or more NICs (Network Interface Cards)) or two or more interface devices of different types (for example, an NIC and an HBA (Host Bus Adapter)).

In addition, in the following description, a “storage unit” includes one or more memories. At least one memory may be a volatile memory or a non-volatile memory. The storage unit may include one or more PDEVs in addition to the one or more memories. A “PDEV” refers to a physical storage device and may typically be a non-volatile storage device (for example, an auxiliary storage device). For example, a PDEV may be an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

Furthermore, in the following description, a “processor unit” includes one or more processors. Typically, at least one processor is a CPU (Central Processing Unit). A processor may include a hardware circuit which performs a part of or all of processes.

In addition, while a “program” is sometimes used as a subject when describing a process in the following description, since a program causes a prescribed process to be performed while using at least one of a storage unit and an interface unit as appropriate by being executed by a processor unit, the processor unit (or a computer or a computer system including the processor unit) may be used as a subject of a process. The program may be installed to a computer from a program source. The program source may be, for example, a program distribution server or a recording medium that can be read by a computer. In addition, in the following description, two or more programs may be realized as one program or one program may be realized as two or more programs.

Furthermore, although information will be described below using expressions such as an “xxx table”, information may be expressed using any kind of data structure. In other words, an “xxx table” can also be referred to as “xxx information” in order to demonstrate that information is not dependent on data structure. In addition, in the following description, a configuration of each table represents an example and one table may be divided into two or more tables and all of or a part of two or more tables may constitute one table.

In addition, in the following description, when describing elements of a same type without distinguishing the elements from one another, reference signs (or a common portion of reference signs) may be used, but when describing elements of a same type by distinguishing the elements from one another, IDs of the elements (or reference signs of the elements) may be used.

Furthermore, in the following description, a “host system” may be one or more physical host computers (for example, a cluster of host computers) and may include at least one virtual host computer (for example, a VM (Virtual Machine)).

In addition, in the following description, a “management system” may be constituted by one or more computers. Specifically, for example, when a management computer includes a display device and displays information on its own display device, the management computer may constitute a management system. In addition, for example, when a management computer (for example, a server) transmits information to be displayed to a remote display computer (for example, a client) and the display computer displays the information (when a management computer displays information on a display computer), a system at least including the management computer of the management computer and the display computer may constitute a management system.

Furthermore, in the following description, a “storage system” may be one or more physical storage apparatuses and may include at least one virtual storage apparatus (for example, an LPAR (Logical Partition) or an SDS (Software Defined Storage)).

In addition, in the following description, “RAID” stands for a Redundant Array of Independent (or Inexpensive) Disks. A RAID group is constituted by a plurality of PDEVs (typically, PDEVs of a same type) and stores data in accordance with a RAID level associated with the RAID group. A RAID group may also be referred to as a parity group. A parity group may be, for example, a RAID group storing parity.

Furthermore, in the following description, “VOL” is an abbreviation of a logical volume and may be a logical storage device. A VOL may be a real VOL (RVOL) or a virtual VOL (VVOL). An “RVOL” may be a VOL based on a physical storage resource (for example, one or more RAID groups) included in a storage system that provides the RVOL. A “VVOL” may be any one of an externally coupled VOL (EVOL), a capacity expanded VOL (TPVOL), and a snapshot VOL. An EVOL may be a VOL which is based on a storage space (such as a VOL) of an external storage system and which is in accordance with storage virtualization technology. A TPVOL may be a VOL which is constituted by a plurality of virtual areas (virtual storage areas) and which is in accordance with capacity virtualization technology (typically, Thin Provisioning). A snapshot VOL may be a VOL provided as a snapshot of an original VOL. A snapshot VOL may be an RVOL. A “pool” may be a logical storage area (for example, a set of a plurality of pool VOLs). For example, pools may include at least one type of a TP pool and a snapshot pool. A TP pool may be a storage area constituted by a plurality of real areas (real storage areas). When a real area is not allocated to a virtual area (a virtual area of a TPVOL) to which belongs an address designated by a write request received by a storage system (for example, a storage controller to be described later) from a host system, a real area may be allocated to the virtual area (a write destination virtual area) from a TP pool (a real area may be newly allocated to a write destination virtual area even when another real area is already allocated to the write destination virtual area). The storage system may write write target data accompanying the write request to the allocated real area. A snapshot pool may be a storage area in which data saved from an original VOL is stored. One pool may be used as a TP pool and a snapshot pool. A “pool VOL” may be a VOL that is a component of a pool. A pool VOL may be an RVOL or an EVOL.

FIG. 1shows an outline of an embodiment 1.

A computer system according to the embodiment 1 includes one or more host computers200, a management computer100, and a storage apparatus300. The host computer200is coupled to the storage apparatus300via a network500. The management computer100is coupled to the storage apparatus300via a network550.

The host computer200executes an application program (hereinafter, an application)211. For example, a host computer200P executes a business application211B and a host computer200A executes an analytical application211A. The management computer100executes a management program112.

The storage apparatus300is an object storage apparatus and includes a storage controller329. The storage controller329includes a local memory1200and provides a VOL26. Types of the VOL26at least include a data VOL26D. The data VOL26D is an example of a data source (typically, an unstructured data source) such as a name space or a DWH (Data Ware House). A data chunk81is stored in the data VOL26D. In the present embodiment, a “data chunk” refers to a significant unit of data (for example, a still image, a moving image, or an E mail). For example, a data chunk may be a portion of time-series data including data from a sensor (for example, data of each certain time). One or more data chunks81of which prescribed data attributes are common, are included in a same object. In the present embodiment, an “object” is a data set including one or more data chunks81and one piece of S-metadata82corresponding to the one or more data chunks81. For example, when the data chunk81is data from a data issuer (for example, a sensor of a camera or the like), each piece of data from a same data issuer is a “data chunk” and a plurality of data chunks from the same data issuer (a plurality of data chunks sharing a data attribute “issuer”) are included in a same “object”. In the present embodiment, “unstructured data” may be data including at least one data chunk in an object. Moreover, “unstructured data” may be a concept including so-called semi-structured data. Hereinafter, one or more data chunks included in an object may be referred to as a “data chunk unit” or “object data”. “Unstructured data” may be each data chunk in an object, a part of the data chunks in an object, or a data chunk unit (object data) in an object.

In the present embodiment, there exist two types of metadata. At least a part of the two types of metadata is stored in the local memory1200. In the present embodiment, the two types of metadata will be referred to as “S-metadata” and “C-metadata”. The S-metadata82(or S-metadata attribute information1220to be described later corresponding to one data chunk) is an example of first type metadata and the C-metadata83is an example of second type metadata. In the present embodiment, the S-metadata82and an object correspond to each other one to one. Therefore, the S-metadata82and the data chunk81correspond to each other one to one or one to many. On the other hand, the C-metadata83and the data chunk81correspond to each other one to one or many to one. This is because an extraction program to be described later may exist for each user and, in such a case, the C-metadata83to be created may differ depending on the extraction program even when the data chunk81is the same. Therefore, the S-metadata82and the C-metadata83correspond to each other one to one or one to many. The S-metadata82is metadata associated with a data chunk unit (all data chunks81) included in an object and includes, for example, an S-metadata ID (an object ID) and information representing a storage location of each data chunk81included in a corresponding object. On the other hand, C-metadata83is metadata including content information representing one or more content attributes specified from the data chunk81(data content) extracted from the data VOL26D. A “content attribute” is an attribute related to a content of data and examples thereof include a data type (for example, an image or an E mail) and a time point (for example, an acquisition time point or an update time point). While content information is information expressed by a text (for example, a character string), the content information may include other types of information (for example, a numerical value representing a feature amount or the like) instead of or in addition to a text. The S-metadata82and the C-metadata83also mutually hold information indicating a relationship between the S-metadata82and the C-metadata83. Specifically, the C-metadata83refers to the S-metadata82that refers to the data chunk81corresponding to the C-metadata83, and the S-metadata82that is referred to by the C-metadata83refers to the C-metadata83. In other words, the C-metadata83and the S-metadata82corresponding to the same data chunk81refer to each other. Moreover, instead of such a two-way reference (linking), a one-way reference from the C-metadata83to the S-metadata82may be adopted. Since the C-metadata83is a type of metadata of the data chunk81, a data amount of the C-metadata83is smaller than that of the data chunk81. In addition, correspondence between the S-metadata82and an object is not limited to one to one (for example, the correspondence may be many to many or one to many).

The host computer200issues an I/O (Input/Output) request to the storage apparatus300. The I/O request is a write request or a read request. When the I/O request is a read request, an object ID corresponding to the data chunk81that is a read target is designated. For example, when the storage controller329receives a read request from the host computer200A, the storage controller329specifies the S-metadata82in which the object ID designated by the read request is described, reads the data chunk81indicated by the specified S-metadata82from the data VOL26D, and sends back the data chunk81to the host computer200A.

The storage controller329executes a DM creation process. The DM creation process is started in response to a user request that is a request of a specific type from a user. The user request may be an explicit request for DM creation or may be a request defined as one type of a DM creation request such as a search request. In the present embodiment, the storage controller329accepts a search request from a user (for example, an analyst) of the host computer200and accepts a DM creation request from a user (for example, an administrator) of the management computer100. In the user request, a search condition (a condition related to data to be included in a DM) in accordance with an analytical viewpoint or the like is designated. As the search condition, for example, a condition may be adopted with respect to at least one of a data type (for example, a photograph or an Email), a data issuer (for example, a sensor model number), a position (for example, a data acquisition position such as a photography position), a time slot (for example, a time slot such as a photography time point), and a data value range (for example, an upper limit and a lower limit of metric values included in data).

As the search condition, normally, an address of an area (for example, a VOL area) in which the data chunk81is actually stored is not designated. This is because the user is normally unaware of such addresses.

However, the DM creation process according to the present embodiment is expected to be completed in a short period of time due at least (reason 3) among (reason 1) to (reason 3) described below.(Reason 1) In the DM creation process, the C-metadata83is referred to but the data chunk81in the data VOL26D is not referred to.(Reason 2) The C-metadata83referred to in the DM creation process is the C-metadata83created asynchronously from the DM creation process (for example, the C-metadata83created before starting the DM creation process). In other words, the C-metadata83is created at a different timing from a user request that is a timing at which the DM creation process is started. For example, when the data chunk81is stored in the data VOL26D, the C-metadata83of the data chunk81is created.(Reason 3) The data chunk81need not be duplicated to create a DM. In other words, the DM to be created is not a real DM to store a duplicate of the data chunk81in the data VOL26D but a virtual DM (hereinafter, a VDM) which refers to the data chunk81in the data VOL26D. In the present embodiment, a VDM is an SSVOL (a snapshot VOL)26S. Creating the SSVOL26S only requires that first S-metadata82S be duplicated and the data chunk81itself need not be duplicated. Moreover, since the data chunk81included in a VDM is not necessarily a reference destination data chunk81of all S-metadata82, second S-metadata82T that is metadata based on a duplicate of the first S-metadata82S may not completely match the first S-metadata82. The first S-metadata82S is original metadata included in an object and, as described above, the second S-metadata82T is metadata based on a duplicate of the first S-metadata82S. The first S-metadata82S is an example of first first type metadata and the second S-metadata82T is an example of second second type metadata. In other words, in the present embodiment, the first S-metadata82S and the second S-metadata82T are provided as the S-metadata82. Moreover, since the second S-metadata82T is data holding information related to a snapshot data chunk (of which an entity is a data chunk in the data VOL26D) which is a data chunk that can be referred to through the SSVOL26S, an expedient name such as metadata need not necessarily be used and, for example, the second S-metadata82T may be given a different name such as snapshot management data (in this case, since any confusion may be avoided, first S-metadata may be simply referred to as “S-metadata” or “metadata”).

Based on the reasons given above, hereinafter, DM creation according to the present embodiment will be referred to as “C-snap” and a DM creation process will be referred to as a “C-snap process”. Moreover, a DM is an example of a data set and a VDM is an example of a virtual data set.

According to the example shown inFIG. 1, for example, asynchronously from a search request42from the analytical application211A (the host computer200A) (for example, before a C-snap is started in response to the search request42), the storage controller329creates pieces of C-metadata #1and #2respectively corresponding to data chunks #1and #2in the data VOL26D and stores the created pieces of C-metadata #1and #2in the local memory1200. The C-metadata #1refers to first S-metadata #1that refers to the data chunk #1, and the C-metadata #2refers to first S-metadata #2that refers to the data chunk #2.

According to the example shown inFIG. 1, the storage controller329starts a C-snap in response to the search request42. A C-snap process is roughly divided into two: “C-snap (selection)” and “C-snap (snapshot acquisition)”. In C-snap (selection), the storage controller329searches for C-metadata83complying with the search condition designated in the search request42from the existing pieces of C-metadata #1and #2. In other words, the C-metadata83instead of the data chunk81is a search range. When at least one piece of C-metadata83complying with the search condition is found, C-snap (snapshot acquisition) is executed. It is assumed that the C-metadata #1is found. In C-snap (snapshot acquisition), the storage controller329creates second S-metadata #1′ based on a duplicate of first S-metadata #1that is referred to by the C-metadata #1, and creates an SSVOL26S (VDM) to which the second S-metadata #1′ belongs. The storage controller392provides at least the host computer200A (a transmission source of the search request42) among the one or more host computers200with the SSVOL26S. The analytical application211A (the host computer200A) can execute an analysis using the data chunk81referred to by the second S-metadata #1′ belonging to the SSVOL26S. Moreover, as an access state (an access restriction) of the data chunk81referred to by the SSVOL26S, for example, any of “R/W enabled” (both read and write are enabled), “RO” (read only (only read is enabled)), and “R/W disabled” (both read and write are disabled) may be adopted. For example, at least one of the following may be adopted.(V1) When a destination to be provided with the SSVOL26S is a plurality of host computers200, the access state of the SSVOL26S may be set to “RO”. Accordingly, consistency of data can be maintained among the plurality of host computers200.(V2) When the host computer200A is the only destination to be provided with the SSVOL26S, the access state of the SSVOL26S may be set to “R/W”. Accordingly, the host computer200A can customize the SSVOL26S. For example, when the storage controller392receives a write request designating the SSVOL26S, a data chunk accompanying the write request may be stored in a pool.

As described above, since a C-snap process does not require a duplicate of the data chunk81, it is expected that the C-snap process can be finished in a short period of time.

Hereinafter, the present embodiment will be described in detail.

FIG. 2shows an outline of an example of a series of processes including a C-snap process and processes before and after the C-snap process.

According to the example shown inFIG. 2, “(0) normal state” and “(1) extraction process” precede the C-snap process. The “(0) normal state” is a state before the C-metadata83is created. In the “(1) extraction process”, the C-metadata83is created. The C-metadata83refers to the first S-metadata82S.

As described earlier, “(3) analysis” is performed after the C-snap process.

A detailed description ofFIG. 2will be given later.

FIG. 3is a block diagram of a computer system according to the embodiment 1.

As described earlier, the computer system includes the management computer100, the host computer200, and the storage apparatus300. One or more of any of the management computer100, the host computer200, and the storage apparatus300may be provided. The management computer100is an example of a management system. The host computer200is an example of a host system. The storage apparatus300is an example of a storage system.

The management computer100, the host computer200, and the storage apparatus300are coupled to one another via a network (for example, a LAN (Local Area Network))500. In addition, the management computer100is coupled to the host computer200and the storage apparatus300via a network (for example, a SAN (Storage Area Network))550. The networks500and550may be integrated.

The management computer100includes an I/F (interface)131, an I/F130, a memory110, and a processor120coupled to these components. The I/F131and the I/F130are examples of the interface unit. The I/F131is coupled to the network550. The I/F130is coupled to the network500. The memory110stores the management program112. By executing the management program112, the processor120can issue a request to the storage apparatus300. Moreover, the request may be a write request, a read request, a copy control request, and the like.

The host computer200includes an I/F231, an I/F230, a memory210, and a processor220coupled to these components. The I/F231and the I/F230are examples of the interface unit. The I/F231is coupled to the network550. The I/F230is coupled to the network500. The memory210stores programs such as an OS (Operating System)212, an application211, and an agent program213. The processor220executes programs in the memory210. For example, by executing a program, the processor220transmits an I/O request to the storage apparatus300. Accordingly, the VOL26provided by the storage apparatus300can be accessed.

The application211is, for example, an analytical application. For example, the analytical application performs an analytical process such as correlation analysis. The OS212provides overall control of processes performed by the host computer200. The agent program213transmits an instruction to the management computer100, and the management computer100can transfer the instruction to the storage apparatus300. When the analytical application211desires to use a storage function, storage control in conjunction with an analytical process can be provided via the management program112by using the agent program213. For example, when the analytical application includes a DM creation function, in response to a DM creation operation by a user, the agent program213transmits a content of the operation to the management program112, and the management program112converts the content of the operation into a copy control request and transmits the copy control request to the storage apparatus300.

The storage apparatus300includes one or more PDEVs1500and the storage controller329coupled thereto.

The one or more PDEVs1500may constitute one or more RAID groups. The PDEV1500is, for example, an HDD or an SSD. The data chunk81stored in the data VOL26D or the like is stored in the one or more PDEVs1500. At least a part of the plurality of pieces of C-metadata83and the plurality of pieces of S-metadata82may be stored in the one or more PDEVs1500.

The storage controller329includes an I/F1321, an I/F1320, an I/F1400, a cache memory1100, the local memory1200, and a processor1310coupled to these components. The local memory1200stores information and programs. By executing programs in the local memory1200, the processor1310refers to or updates information in the local memory1200, performs an I/O on a VOL, creates the C-metadata83, executes a C-snap.

The I/F1321, the I/F1320, and the I/F1400are examples of the interface unit. The I/F1321is coupled to the network550. The I/F1320is coupled to the network500. The I/F1400is coupled to the one or more PDEVs1500.

The cache memory1100and the local memory1200are examples of the storage unit. The cache memory1100and the local memory1200may be one memory and a cache area as a cache memory and a local memory area as a local memory may be provided in the memory.

The cache memory1100is a memory for temporarily storing data input to or output from the one or more PDEVs1500(for example, data (write target data or read target data) in accordance with an I/O request from the host computer200).

The local memory1200stores information and programs. Specifically, for example, the local memory1200stores S-metadata management information1210, S-metadata attribute information1220, C-metadata management information1230, a storage management table1250, and a copy pair management table1260. In addition, for example, the local memory1200stores an I/O program61, an object program62, a data processing program63, a snapshot program64, an extraction program1290, and a C-snap program1291.

The S-metadata management information1210and the S-metadata attribute information1220exist for each piece of S-metadata82. The S-metadata management information1210is information for managing an object. The S-metadata attribute information1220is information for managing the data chunk81.

The C-metadata management information1230exists for each piece of C-metadata83. The C-metadata83includes content information representing one or more content attributes specified from the data chunk81. The C-metadata management information1230is at least a part of the C-metadata83.

The storage management table1250is a table that stores information related to the VOL26provided by the storage apparatus300. The copy pair management table1260is a table that stores information related to a copy configuration to which the SSVOL26S belongs.

The I/O program61is a program for processing an I/O request. The object program62is a program for processing an object. The data processing program63is a program for accessing the VOL26. The snapshot program64is a program for creating the SSVOL26S.

The extraction program1290is a program for extracting a data chunk81and creating C-metadata83based on the extracted data chunk81. The C-snap program1291is a program for executing a C-snap process. At least one of the extraction program1290and the C-snap program1291may be a user program that is a program created by the user. In other words, at least one of the extraction program1290and the C-snap program1291may exist for each user, and at least one of the extraction program1290and the C-snap program1291corresponding to the user of the host computer200may be executed. By adopting a user program as at least one of the extraction program1290and the C-snap program1291, at least one of the C-metadata83and the SSVOL26S (VDM) which enables an analysis result desirable to the user (for example, an analyst) to be obtained can be expected.

FIG. 4shows an example of a snapshot process.

The snapshot process is a process during write with respect to the SSVOL26S. The storage controller329manages a pool91constituted by one or more pool VOLs26P (pool VOLs #1to #4).

The storage controller329receives a write request designating the SSVOL26S from the host computer200. The write request is, for example, a write request designating an object ID of an object including a data chunk that is a reference destination of S-metadata (an S-metadata duplicate) belonging to the SSVOL26. The storage controller329stores the data chunk81(for example, #1) in accordance with the write request in the pool91instead of the reference destination of the SSVOL26(S-metadata). In other words, a write target data chunk81is stored in the pool VOL26P which is an example of a VOL that differs from the reference destination VOL of the SSVOL26(S-metadata). The storage controller329manages an association between a virtual address of a data chunk (an address of an area of the SSVOL26S) and a real address of the data chunk81(an address of an area of the pool VOL26P). As described above, a process using a Ridirect-on-write system may be adopted as the snapshot process. In other words, when a write occurs with respect to a data chunk in the SSVOL26S (or the data VOL26D), the write is performed on a new area and an area (address) indicated by the first S-metadata82S and the second S-metadata82T is rewritten. In this manner, while a snapshot process using the Ridirect-on-write system may be adopted, a snapshot process using another system such as a Copy-on-write system may also be adopted.

FIG. 5shows a configuration of the storage management table1250.

The storage management table1250includes a storage ID1252. One or more route IDs1251are included for each storage ID1252.

The storage ID1252is information representing an identifier (a storage ID) of the storage apparatus300.

The route ID1251is information representing an identifier (a route ID) of a route. The route ID1251of a route included in the storage apparatus300is associated with the storage ID1252of the storage apparatus300. In the present embodiment, a “route” refers to a group of one or more pieces of S-metadata82. The VOL26exists for each route. Therefore, for example, a route ID can also be described an identifier of a VOL (a VOL_ID). An S-metadata pointer1254of the S-metadata82belonging to a route is associated with the route ID1251of the route. The S-metadata pointer1254is information (a pointer) indicating the whereabouts of the S-metadata82in the local memory1200.

FIG. 6shows configurations of S-metadata management information1210and S-metadata attribute information1220included in one piece of S-metadata82.

The S-metadata82is constituted by the S-metadata management information1210and the S-metadata attribute information1220. As described above, the S-metadata management information1210manages an object and the S-metadata attribute information1220manages the data chunk81. With the S-metadata management information1210, the S-metadata attribute information1220is associated with each data chunk81in an object corresponding to the S-metadata management information1210.

The S-metadata management information1210includes an S-metadata ID121001. The S-metadata ID121001is information representing an identifier (an S-metadata ID) of a piece of S-metadata. An S-metadata ID is, in other words, an object ID.

In addition, the S-metadata management information1210includes, for each data chunk81in a corresponding object, an S-metadata attribute ID121002and an S attribute pointer12103. The S-metadata attribute ID121002is information representing an identifier (an S-metadata attribute ID) of the S-metadata attribute information1220. The S attribute pointer121003is information (a pointer) indicating the whereabouts of the S-metadata attribute information1220in the local memory1200. Accordingly, the C-metadata83as a reference destination of the S-metadata82can be specified.

Furthermore, the S-metadata management information1210includes, for each piece of C-metadata83that refers to the S-metadata82including the S-metadata management information1210, a user ID12011and a user pointer121012. The user ID121011is information representing an identifier (a C-metadata ID) of the C-metadata83and specifically, for example, the user ID121011is information which is used when managing additional information (in other words, the C-metadata83) added to the S-metadata management information1210by a user program (for example, the extraction program1290) and which is an identifier of the additional information. The user pointer121012is information (a pointer) indicating the whereabouts of the C-metadata management information1230included in the C-metadata83in the local memory1200.

The S-metadata attribute information1220includes an S-metadata attribute ID122001, an access state122002, a copy state122003, a storage ID122004, a start address122005, an end address122006, and data validity122007.

The S-metadata attribute ID122001is information representing an S-metadata attribute ID. The S-metadata attribute ID may be an identifier of a data chunk (a data chunk ID). In an I/O request, any one of an object ID and a data chunk ID may be designated.

The access state122002is information representing an access method and an access restriction to the data chunk81. Examples of the access method include object access (“Object”) which is access in object units, block access which is access in block units, and file access which is access in file units. Examples of the access restriction include “R/W enabled”, “RO”, and “R/W disabled”. The access state122002may further include information indicating which user is access-enabled.

The copy state122003is information representing a copy state with respect to a data chunk. Examples of the copy state122003include “SVOL” (indicating that the data chunk is referred to by the SSVOL26S) and “NULL” (indicating that the data chunk81is not a copy target).

The storage ID122004is information representing an identifier (a storage ID) of the storage apparatus300in which the data chunk81is stored. As in another embodiment to be described later, there may be cases where the data chunk81referred to by the S-metadata82is arranged in a storage apparatus300that differs from the storage apparatus300in which the S-metadata82exists. By referring to the storage ID122004, the processor1310can specify the storage apparatus300storing the corresponding data chunk81.

The start address122005is information representing a start address of an area in which the data chunk81exists. The end address122006is information representing an end address of the area in which the data chunk81exists. The data validity122007is information (for example, a flag) representing whether or not the data chunk81itself is valid. “YES” means valid and “NO” means invalid. For example, when there is S-metadata # X that refers to data chunks # A and # B in the data VOL26D and S-metadata # X′ (a duplicate of the S-metadata # X) only refers to the data chunk # A among the data chunks # A and # B, in the S-metadata # X′, the data validity12007corresponding to the data chunk # A is “YES” but the data validity12007corresponding to the data chunk # B is “NO”.

FIG. 7shows a configuration of the C-metadata management information1230included in one piece of C-metadata83.

The C-metadata management information1230is at least a part of the C-metadata83. The C-metadata management information1230includes a C-metadata ID123001, a type123002, a start address123003, an end address123004, an S-metadata attribute ID123005, and a user extension123006.

The C-metadata ID123001is information representing an identifier (a C-metadata ID) of the C-metadata83. The S-metadata82that is a reference destination of the C-metadata83(the S-metadata82including a same C-metadata ID as the user ID121011) can be discerned from the C-metadata ID123001.

The type123002is information representing a type of the C-metadata83. For example, the type123002is referred to when the C-snap program1291searches from a viewpoint of metadata type.

The start address123003is information representing a start address of an area (for example, an area of the VOL26) storing information associated with the C-metadata management information1230(for example, a part of content information (a part of the C-metadata83)). The end address123004is information representing an end address of an area storing information associated with the C-metadata management information1230. When all C-metadata83exist in the local memory1200, the start address123003and the end address123004are respectively “NULL”.

The S-metadata attribute ID123005is information representing an S-metadata attribute ID of the S-metadata attribute information1220indicating a data chunk corresponding to the C-metadata83. The S-metadata attribute information1220indicating the data chunk81corresponding to the C-metadata83can be identified from the S-metadata attribute ID123005.

The user extension123006is extension information added by a user program and is at least apart of content information. For example, when the extracted data chunk81is a photographed image, information on a photography position of the image is included in the C-metadata management information1230as the user extension123006.

FIG. 8shows a configuration of the copy pair management table1260.

The copy pair management table1260is a table that stores information related to a configuration of a copy pair. The copy pair management table1260stores a route ID12601, a copy state12602, a copy target storage ID12603, a copy target route ID12604, and a group ID12605.

The route ID12601is information representing an identifier (a route ID) of a route. The copy state12602is information representing a current state of a copy related to a route (for example, a VOL) identified by the route ID12601. The copy target route ID12604is information representing an identifier of a copy target route that is a route constituting a pair with a route represented by the route ID12601. The copy target route may be any of a copy source and a copy destination. At least one of the route ID12601and the copy target route ID12604may include information (for example, a symbol) representing whether a route corresponding to the information is a copy source or a copy destination. The group ID12605is information representing an identifier (a group ID) of a copy group including the copy pair.

Hereinafter, several processes performed in the embodiment 1 will be described.

FIG. 10is a flow chart of a data read process.

When the storage apparatus300receives an I/O request from the host computer200, the I/O program61determines whether or not the I/O request is a read request (S5010). When a result of the determination of S5010is false (S5010: No), the I/O program61advances to S5510inFIG. 11.

When a result of the determination of S5010is true (S5010: Yes), the I/O program61converts the read request into a common read request and passes the converted read request to process of the object program62(S5020). An I/O request such as a read request is converted into a common I/O request in order to enable various protocols (access methods) to be used as a protocol of the I/O request. For example, given that there are protocols such as blocks, files, and objects, a conversion into a common I/O request enables processing subsequent to the conversion to be commonly performed regardless of the protocol. For example, an object access protocol is an input/output protocol in which data access is performed with an object as a basic unit and which can be operated using a Web interface such as the REST (Representational State Transfer) protocol as an operating format. Specifically, for example, operations can be performed in a format such as PUT <object ID> <write|read|copy control> [<option>] and due to S5020, an I/O request can be converted into a common request in the following common format:
WRITE|READ|COPY <object ID> [<option>].

Next, S5050is performed. In other words, the object program62converts a read source address in accordance with a common read request into an address of a VOL. The S-metadata management information1210and the S-metadata attribute information1220are used in the conversion. Specifically, the object program62refers to the S-metadata management information1210including the S-metadata ID121001matching the object ID in the common request, and refers to the S-metadata attribute information1220from the S attribute pointer121003in the S-metadata management information1210. Next, the object program62acquires the start address122005and the end address122006included in the S-metadata attribute information1220. The object program62converts the object ID in the common request into a start address and an end address represented by the acquired addresses122005and122006, and passes the converted common request to the data processing program63.

The data processing program63determines whether or not data specified from the common request exists in the cache memory1100(S5090). When a result of the determination of S5090is false (S5090: No), the data processing program63writes the data into the cache memory1100and passes the processing to the object program (S5100).

When a result of the determination of5090is true (S5090: Yes) or after S5100, the object program62reads the data from the cache memory1100(S5060). The I/O program61returns the data to the host computer200that is a transmission source of the read request (S5030).

As described above, in a data access process in the storage apparatus300, due to the three programs61to63running in parallel and cooperating with each other as necessary, data in accordance with a read request can be read from the VOL26and returned to the host computer200. The read source VOL may be the data VOL26D or the SSVOL26S. In the data read process, a determination may be made regarding whether or not readout is permitted based on the access state122002corresponding to the data chunk81that is a read target.

FIG. 11is a flow chart of a data write process.

The I/O program61determines whether or not an I/O request is a write request (S5510). When a result of the determination in S5510is false (S5510: No), a process in accordance with the request is performed.

When the result of the determination in S5510is true (S5510: Yes), the I/O program61converts the write request into a common request of the storage apparatus300(S5520).

Next, the object program62determines whether or not the copy state122003of data (an object) that is a write target in accordance with the common request is “SVOL” (S5540). Specifically, the object program62specifies the S-metadata management information1210with the same S-metadata ID121001as the object ID in the common request, further specifies the S-metadata attribute information1220from the S attribute pointer121003in the S-metadata management information1210, and refers to the copy state122003of the specified S-metadata attribute information1220.

When the copy state122003is “SVOL” (S5540: Yes), the snapshot program64changes a write destination VOL to another VOL (pool VOL) (S5550). Specifically, the snapshot program64refers to the S-metadata management information1210including the S-metadata ID121001matching the object ID in the common request, and refers to the S-metadata attribute information1220from the S attribute pointer121003in the S-metadata management information1210. Next, the snapshot program64acquires the start address122005and the end address122006of the S-metadata attribute information1220, and changes a VOL_ID represented by the addresses122005and122006to an ID of the pool VOL. Accordingly, a situation where the data chunk81referred to by the SSVOL26S is updated by a write to the SSVOL26S can be avoided.

When the copy state122003is not “SVOL” (S5540: No), S5560is performed. In other words, the object program62converts the object ID in the common request into an address of a VOL. Specifically, the object program62refers to the S-metadata management information1210including the S-metadata ID121001matching the object ID in the common request, and refers to the S-metadata attribute information1220from the S attribute pointer121003in the S-metadata management information1210. Next, the object program62acquires the start address122005and the end address122006included in the S-metadata attribute information1220, and replaces the object ID in the common request with the acquired addresses122005and122006.

After S5550or S5560, the object program62reserves an area in a cache memory1110(S5570). In addition, the object program62writes data in accordance with the common request to the reserved area (S5530). Once S5530is completed, the I/O program61may return a write completion to the host computer200that is a transmission source of the write request. The data written in the cache memory1110is written to the PDEV1500corresponding to an area indicated by a write destination address of the data by the data processing program63.

As described above, in a data access process in the storage apparatus300, due to the three programs61to63running in parallel and cooperating with each other as necessary, data that is a write request can be written to the cache memory1100and the host computer200may be notified of a completion. In the data write process, a determination may be made regarding whether or not write is permitted based on the access state122002corresponding to the data chunk81that is a write target.

Hereinafter, a series of processes including a C-snap process will be described with reference toFIG. 2andFIGS. 12 to 14.

The data chunk81is stored in the storage apparatus300and first S-metadata82S is associated with an object including the data chunk81. For example, the data chunk81may be image data generated from a monitoring camera or log information output from a piece of manufacturing equipment in a factory.

According toFIG. 2, data chunks #1and #2are stored and there are pieces of first S-metadata #1and #2that respectively refer to the data chunks #1and #2.

An extraction program1290R runs on the processor1310at a time point where at least one data chunk81is stored in the data VOL26D of the storage apparatus300, at a certain time interval, at a time point where a state of low processing load on the processor1310has continued for a certain period of time or more, or the like.

FIG. 12is a flow chart of the extraction process.

The extraction process is performed by the extraction program1290and the object program62. The extraction process may target a route ID designated by the user. The route ID (for example, a VOL_ID) may be designated in advance. The extraction program1290is a program which acquires content information that may constitute an analytical viewpoint from data (an object) stored in the storage apparatus300, associates C-metadata83including the content information with S-metadata82of the data, and stores the associated C-metadata83in the storage apparatus300. While the extraction program1290runs in the storage apparatus300in the present embodiment, the extraction program1290may run on any of the host computer200and the management computer100instead.

By comparing a time point where the data chunk81is stored in a designated route (VOL) with a time point of an immediately-previous extraction process, the extraction program1290determines whether or not there is a data chunk (hereinafter, an updated data chunk)81of which a storage time point is more recent than the time point of the immediately-previous extraction process (S5610). When a result of the determination in S5610is false (S5610: No), the process is ended. Moreover, “the time point of the immediately-previous extraction process” is a time point where the data chunk81has been stored in the local memory1200by the extraction program1290in the immediately-previous extraction process.

When a result of the determination in S5610is true (S5610: Yes), the extraction program1290extracts the updated data chunk81and determines whether or not the extracted updated data chunk81is a data chunk that complies with a set extraction rule (S5620). For example, the extraction rule is designated a data condition of a data chunk to be extracted (a search condition for extraction). For example, the data condition may be a data type (for example, a photograph or an E-mail). An extraction rule may be prepared for each user in place of, or in addition to, the extraction program1290being prepared for each user.

When a result of the determination in S5620is false (S5620: No), the extraction program1290advances to S5670(the process may be ended).

When a result of the determination in S5620is true (S5620: Yes), based on a data format of the updated data chunk81, the extraction program1290extracts, from the updated data chunk81, content information representing one or more content attributes represented by the updated data chunk81(S5630). In order to acquire content information from the updated data chunk81, a different approach must be employed according to the data type. For example, when acquiring positional information from an image, at least a part of content information can be acquired by referring to attribute information of an image file and reading positional information included in the attribute information.

Next, the extraction program1290creates C-metadata83based on the extracted content information (S5640). The content information may be stored in at least one of the local memory1200and the VOL26. When a capacity of the content information is sufficiently smaller than a free capacity of the local memory1200, the entire content information may be stored in the local memory1200. The extraction program1290creates C-metadata management information1230based on a storage location of the content information. The C-metadata ID1230may have an arbitrary value. The start address123003and the end address123004may be “NULL” when the content information is stored in the local memory1200. The S-metadata attribute ID123005may be an identifier of an updated data chunk. The user extension123006may be at least a part of the content information. In this manner, since at least a part of the content information may be registered in the C-metadata management information1230, as a result, the entire content information may be stored in the local memory1200. On the other hand, at least a part of the content information may be stored in the VOL26. In this case, for example, an address of a storage location of the content information can be obtained by making an inquiry to the object program62. In addition, when the entire content information is registered in a VOL, the user extension123006may be “NULL”.

Next, the extraction program1290makes a request to the object program62to register the C-metadata83including the C-metadata management information1230created in S5640(S5650). In response to the request, the object program62associates the C-metadata83with the S-metadata82that refers to the extracted updated data chunk81(S5660). Specifically, the object program62adds a same value as the C-metadata ID1230as the user ID121011and adds a pointer to the C-metadata management information1230as the user pointer121012to the S-metadata management information1210in the S-metadata82that refers to the extracted updated data chunk81.

The extraction program1290makes a similar determination as S5610(S5670). When a result of the determination of S5670is true (S5670: Yes), the extraction program1290advances to S5620with respect to a different updated data chunk. When the result of the determination in S5670is false (S5670: No), the process is ended.

According toFIG. 2, due to the extraction process, the pieces of C-metadata #1and #2respectively corresponding to the data chunks #1and #2are created. The C-metadata #1refers to the first S-metadata #1and the C-metadata #2refers to the first S-metadata #2. Moreover, both pieces of C-metadata #1and #2may include, as content attributes, a designated search condition (a data condition (for example, a time slot)) and a search result (for example, a search hit or a search not hit) of a search performed using the search condition as a key in place of, or in addition to, the data type or the like described earlier.

The C-snap (selection) is a process of referring to C-metadata83associated with S-metadata82in the extraction process and selecting data complying with the search condition designated for the designated route (VOL). While the C-snap program1291runs in the storage apparatus300in the present embodiment, the C-snap program1291may run on any of the management computer100and the host computer200instead.

A start of a C-snap process is instructed by the user. The instruction is received by the C-snap program1291. For example, an instruction format is as follows.
CSNAP <search key> <target route ID> <copy destination route ID> <option>

In the case of the instruction format described above, the data chunks81in a route designated in <target route ID> are narrowed down to data chunks81complying with a search key (a search condition) designated in <search key>. One or more pieces of S-metadata82that refer to the one or more narrowed-down data chunks81are to be duplicated to below the route designated in <copy destination route ID>.

FIG. 13is a flow chart of the C-snap (selection).

S5710is performed. Specifically, the C-snap program1291specifies, from the storage management table1250, the S-metadata pointer1254corresponding to a route ID designated in the instruction from the user. Next, the C-snap program1291refers to the S-metadata management information1210from the specified S-metadata pointer1254, and further specifies C-metadata83associated with the S-metadata from the user ID121011and the user pointer121011in the S-metadata management information1210.

Next, the C-snap program1291determines whether or not the C-metadata83(the content information included in the C-metadata83) complies with the search key designated by the user (S5720).

When a result of the determination in S5720is true (S5720: Yes), the C-snap program1291makes a request to the object program62to duplicate the first S-metadata82S (the S-metadata management information1210and the S-metadata attribute information1220) associated with the C-metadata83(S5730). In response to the request, the object program62duplicates the designated first S-metadata82S (S5740). Moreover, in the duplicate, an S-metadata ID that differs from the S-metadata ID of the original first S-metadata82S may be added as an S-metadata ID of the second S-metadata82T based on a duplicate of the first S-metadata82S. In addition, in the duplicate, any of the C-snap program1291and the object program62may execute a duplicate narrow-down process that is any of (a) and (b) described below.(a) Skip duplicating S-metadata attribute information1220that refers to a data chunk not required for analysis (S-metadata attribute information1220of a reference destination of C-metadata83not complying with the search key).(b) Change the data validity122007of the S-metadata attribute information1220thereof to “NO”.

Whether or not such a duplicate narrow-down process is to be executed may be described in the instruction (the start instruction with respect to the C-snap program1291) from the user. The duplicate narrow-down process enables data chunks81included in the SSVOL26S (VDM) to be narrowed down.

Next, the C-snap program1291determines whether or not S5710has been performed on all pieces of S-metadata82corresponding to the route ID designated by the user (S5750). When a result of the determination in S5750is false (S5750: No), S5710is performed on pieces of S-metadata82yet to be processed. When the result of the determination in S5750is true (S5750: Yes), the process is ended. When S5740is performed on at least one piece of S-metadata82, C-snap (snapshot acquisition) is performed.

Based on the duplicate of the S-metadata82obtained in C-snap (selection), the SSVOL26S is created. The SSVOL26S becomes usable as a DM from the host computer200when the host computer200is provided with the SSVOL26S.

FIG. 14is a flow chart of the C-snap (snapshot acquisition).

The C-snap program1291makes a request to the snapshot program64to create a snapshot (S5770). When creating a snapshot, the C-snap program1291passes the S-metadata ID in the second S-metadata82T created in C-snap (selection) to the snapshot program64.

In response to the request, the snapshot program64specifies the S-metadata management information1210matching the S-metadata ID passed from the C-snap program1291and sets the copy state122003of the S-metadata attribute information1220associated with the S-metadata management information1210to “SVOL” (S5680). Setting the copy state122003to “SVOL” causes a determination of snapshot target data to be made during write to the object and a necessary snapshot process (refer toFIG. 4) is performed.

Next, the snapshot program64adds a copy destination route ID (an ID of the SSVOL26S) designated by the user as the route ID1251to the storage management table1250and associates the pointer1254to a duplicate of the S-metadata82with the route ID1251(S5690). The snapshot program64may provide the host computer200of the user having issued the C-snap start instruction (the user who is a search request source) with the copy destination route ID (the SSVOL26S).

As described above, in the C-snap process in the storage apparatus300, a data chunk that is a snapshot target (a data chunk included in a VDM) is selected based on a search key provided by the user in C-snap (selection), and the SSVOL26S (VDM) including the selected data chunk is created in C-snap (snapshot acquisition).

In addition, in principle, a plurality of copy destination route IDs (SSVOLs26S) can be created with respect to one route ID (one data VOL26D). Specifically, for example, as exemplified inFIG. 15, a plurality of SSVOLs #1-1to #1-3can be created with respect to one data VOL26D.

Furthermore, when accessing a copy destination route ID designated when creating a C-snap from the host computer200after the C-snap process, a DM (the SSVOL26S) is to appear to exist from the host computer200. When the SSVOL26S is created in plurality, for example, DMs (data marts) with different viewpoints are to appear to be created.

An embodiment 2 will now be described. In doing so, differences from the embodiment 1 will be mainly described and descriptions of points in common with the embodiment 1 will be either omitted or simplified. This will similarly apply to other embodiments.

FIG. 16shows an outline of the embodiment 2.

A storage system according to the embodiment 2 is constituted by a plurality of storage apparatuses300X and300A to300C. For example, one virtual storage system is constituted by the storage apparatuses300X and300A to300C. In other words, the storage system according to the present embodiment may be a virtual storage system. The number of storage apparatuses300is not limited to the example shown inFIG. 16.

The storage apparatus300X includes the data VOL26D as a data source. SSVOLs26S (#1-1to #1-3) which refer to at least one data chunk in the data VOL26D are respectively generated in the storage apparatuses300A to300C. In other words, a C-snap process is performed across a plurality of storage apparatuses300. Specifically, for example, when the storage apparatus300A receives a search request designated a route of the data VOL26D, the storage apparatus300A searches, from the storage apparatus300X, C-metadata83complying with the search key designated in the search request, and duplicates first S-metadata1#1associated with the found C-metadata83to the storage apparatus300A. The storage apparatus300A provides the host computer200A with the SSVOL #1-1corresponding to a route to which belongs the second S-metadata #1-1based on the duplicate of the first S-metadata #1.

A scale-out process is required to perform a C-snap process between storage apparatuses300.FIG. 17shows an outline of the scale-out process. The storage apparatuses300X and300A are exemplified inFIG. 17. Scale-out programs74X and74A have been respectively added to the storage apparatuses300X and300A. For example, the scale-out program74X (74A) may relay cooperation between an I/O program61X (61A) and an object program62X (62A). Cache memories1100X and100A respectively exist in the storage apparatuses300X and300A.

In this case, when the storage apparatus300A receives a read request from the host computer200A, the scale-out program74A of the storage apparatus300A determines whether or not the read request is addressed to an own storage apparatus300A. When a result of the determination is false, the scale-out program74A transfers the read request to the storage apparatus300X that is the destination of the read request. The storage apparatus300X having received the transferred read request reads a data chunk81to the cache memory1100X based on the read request.

For example, the flow chart shown inFIG. 10differs from the embodiment 1 in processing after S5020. Specifically, for example, the scale-out program74A acquires a common request and determines whether or not an access destination of the common request is an own storage apparatus300A. When a result of the determination is false, the scale-out program74A transfers the common request to the scale-out program74X of the storage apparatus300X that is the access destination of the common request. The scale-out program74X passes the common request to the object program62X. On the other hand, when the access destination of the common request is the own storage apparatus300A, the scale-out program74A passes the common request to the object program62A of the own storage apparatus300A.

In addition, for example, the flow chart shown inFIG. 11differs in processing after S5520. Specifically, for example, the scale-out program74A acquires a common request and determines whether or not an access destination of the common request is the own storage apparatus300A. When a result of the determination is false, the scale-out program74A transfers the common request to the scale-out program74X of the storage apparatus300X that is the access destination of the common request. The scale-out program74X passes the common request to the object program62X. On the other hand, when the access destination of the common request is the own storage apparatus300A, the scale-out program74A passes the common request to the object program62A of the storage apparatus300A including the scale-out program74A.

As described above, according to the embodiment 2, a C-snap process is performed across a plurality of storage apparatuses300. Accordingly, the storage apparatuses300X and300A can be used for different applications by having the storage apparatus300X store only the data VOL26D (a data chunk) and having the storage apparatus300A store only the SSVOL26S (snapshot data). An analysis performed by the storage apparatus300B using a DM (the SSVOL #1-2) can be prevented from affecting performance of another storage apparatus300C.

FIG. 18shows an outline of an embodiment 3.

A storage system according to the embodiment 3 is constituted by the storage apparatuses300A and300B respectively existing at a plurality of sites A and B and the storage apparatus300X existing at a central data center. The number of sites and the number of storage apparatuses300are not limited to the example shown inFIG. 18. In the present embodiment, each of the storage apparatuses300A and300B executes an extraction process and the storage apparatus300X executes a C-snap process.

One or more IoT devices86A and82B respectively exist at the sites A and B (IoT: Internet of Things). In this case, an “IoT device” refers to a device capable of exchanging, via a network, a state or control of a sensor included in the device or a state or control of the device itself.

Host computers200X and Y are capable of accessing the storage apparatus300X at the central data center.

In the present embodiment, for example, at the site A, the storage apparatus300A receives sensor information, device state information, and the like of the IoT device86A via a network. The storage apparatus300A having received the information stores one or more data chunks including the information in a data VOL (an example of a first unstructured data source) (not shown). The storage apparatus300A executes an extraction process including a process of creating one or more pieces of C-metadata from the one or more data chunks. In the extraction process, the storage apparatus300A may delete data not required during an analysis (for example, information which has not been successfully acquired, sensor information including values small enough to be sufficiently negligible, and a stable device state) and may exclude the data from targets of remote copying to be described later.

After the extraction process, the storage apparatus300A transfers the one or more data chunks and the one or more pieces of C-metadata corresponding to the one or more data chunks to the storage apparatus300X at the central data center (remote copying).

The storage apparatus300X having received the data chunks and the pieces of C-metadata stores the data chunks in the data VOL26D (an example of a second unstructured data source) and stores the C-metadata in the local memory. When the storage apparatus300X receives a search request designating the data VOL26D from the host computers200X and200Y, the storage apparatus300X executes a C-snap process. In other words, the storage apparatus300X generates pieces of second S-metadata #1-1and #1-2by duplicating the first S-metadata #1associated with the C-metadata that complies with the search key, and respectively provides the host computers200X and200Y with SSVOLs #1-1and #1-2to which the pieces of second S-metadata #1-1and #1-2respectively belong.

As described above, according to the embodiment 3, for example, at site A, the storage apparatus300A receives information of the sensor of the IoT device86A and the like, the storage apparatus300A creates C-metadata by extracting content information, and transfers the C-metadata together with a data chunk to the storage apparatus300X at the central data center. Accordingly, the storage apparatus300X located at the central data center can avoid performing an extraction process. In other words, in the storage apparatus300X, a load of an extraction process targeting a large amount of data chunks including information generated from the IoT device86can be reduced. In addition, the storage apparatus300A at a site culls data not considered a target of analysis and subsequently transfers data chunks and C-metadata to the storage apparatus300X at the central data center. Accordingly, a load of a C-snap process on the storage apparatus300X and a total amount of data stored by the storage apparatus300X can be reduced.

FIG. 9shows an outline of an embodiment 4.

An SCM (Storage Class Memory)943is adopted. Each of the host computer200and the storage apparatus300are capable of using the SCM943, to which a memory area of the host computer200and a memory area of the storage apparatus300are allocated, as though the SCM943is its own memory. Therefore, for example, the C-snap program1291can be executed by the host computer200and the C-snap program1291can find C-metadata complying with a search key from the C-metadata in the SCM943.

While some embodiments have been described above, it should be obvious that the present invention is not limited to the described embodiments and that various modifications can be made without departing from the spirit and scope of the invention.

For example, any two or more embodiments among the embodiments 1 to 4 can be combined with one another.

In addition, while the embodiments 1 to 4 describe a storage system as an example of a data processing system, a data processing system may correspond to at least one of a storage system, a host system, and a management system. For example, when a host system corresponds to the data processing system, a request source that transmits a search request designating a search key to the host system may be a client system (one or more client computers).

Furthermore, while not only S-metadata82but C-metadata83also exists in a storage system in the embodiments 1 to 4, the C-metadata83may exist in a host system or a management system in place of, or in addition to, the storage system. Specifically, for example, the C-metadata83may be created for each user (for example, for each host system or each management system) with respect to a same object (a same data chunk81), and the C-metadata83may be provided to a host system or a management system of a user corresponding to the C-metadata83. When a host system or a management system receives designation of a search condition from a user, a processor unit in the host system or the management system may search for a piece of C-metadata83complying with the search condition among pieces of C-metadata83corresponding to the user from the host system or the management system. When the C-metadata83is found, the host system or the management system may make a request to the storage system to create an SSVOL to which belongs the S-metadata82referred to by the C-metadata83. The storage system may execute a C-snap process in response to the request.

In addition, the C-metadata83may exist for each user. For example, with respect to the same data chunk81, C-metadata83created by the extraction program1290of a user A may be stored as C-metadata83for the user A and C-metadata83created by the extraction program1290of a user B may be stored as C-metadata83for the user B. When a search request is received from the user A, the storage controller329(the C-snap program1291) may search for C-metadata83complying with a search key designated in the search request and with the user A who is a request source. In addition, when the C-snap program1291of the user A exists as the C-snap program1291, the C-snap program1291of the user A may search for C-metadata83complying with a search key designated in a search request from the user A and with the user A.

Furthermore, a C-snap process may be started upon detection of a C-snap event that is a prescribed event which is defined to start a C-snap process. The C-snap event may be any of a reception of a user request (for example, an explicit request for a C-snap process or a request defined to execute a C-snap process), an arrival of a time point defined in advance (for example, an execution of a C-snap process is started regularly), and a prescribed performance state (a state regarding performance) such as when a load on a processor executing the C-snap program1291falls below a prescribed value. For example, the storage controller329may receive a user request from at least one of the management computer100and the host computer200and execute a C-snap process in response to the user request.

In addition, a user program (for example, at least one of the extraction program1290and the C-snap program1291) may be executed by any of the management computer100, the host computer200, and the storage controller329.

Furthermore, the SSVOL26S (VDM) may be updated on a regular or irregular basis. For example, the C-snap program1291may specify C-metadata83representing a same content attribute as a content attribute represented by the C-metadata83associated with the second S-metadata82T belonging to an existing SSVOL26S, newly create the second S-metadata821by duplicating the first S-metadata82S that is referred to by the C-metadata83, and newly associate the new second S-metadata82T with the existing SSVOL26S.

In addition, a file may be adopted as an example of an object. Data of the file may be an example of a data chunk in the object and metadata of the file may be an example of S-metadata of the object.

Furthermore, a data VOL may be an example of a data area and an SSVOL may be an example of a snapshot that refers to a part of unstructured data in the data area.

In addition, in an extraction process, determination whether or not the first S-metadata82S complies with a search condition by referring the first S-metadata82S, may be executed in place of, or in addition to, data extraction from an unstructured data source. When a result of the determination is true, the C-metadata83described above may be created based on the first S-metadata82S and the C-metadata83may be associated with the first S-metadata82S complying with the search condition. Furthermore, in this case, one or more data chunks81that are referred to by the first S-metadata82S complying with the search condition may be an example of corresponding unstructured data.

REFERENCE SIGNS LIST