Providing automated targetless snapshots for storage volumes includes estimating storage space needed for a new set of the snapshots based on an amount of expected change of data on the storage volumes, deleting current snapshots to free up an amount of space corresponding to estimated storage space for the new set of the snapshots, and creating the new set of the snapshots. The current snapshots may be selected for deletion based on creation date of the snapshots. Snapshots with an oldest creation data may be deleted. The storage volumes may be provided on a storage system. The storage system may show storage space used by the snapshots as unallocated. Storage space used by the snapshots may be managed by a storage resource pool. The storage resource pool may convert storage space shown as unallocated into allocated storage space in response to snapshots provided therein being converted into accessible snapshots.

TECHNICAL FIELD

This application relates to the field of computer systems and storage systems therefor and, more particularly, to the field of maintaining backup data for storage systems.

BACKGROUND OF THE INVENTION

Host processor systems may store and retrieve data using a storage system containing a plurality of host interface units (I/O modules), disk drives, and disk interface units (disk adapters). The host systems access the storage systems through a plurality of channels provided therewith. Host systems provide data and access control information through the channels to the storage system and the storage system provides data to the host systems also through the channels. The host systems do not address the disk drives of the storage system directly, but rather, access what appears to the host systems as a plurality of logical disk units or logical devices. The logical devices may or may not correspond to any one of the actual disk drives. Allowing multiple host systems to access the single storage system allows the host systems to share data stored therein among different host processor systems.

It is important to provide backups of data on storage systems to protect against data loss and data corruption. A product offered by Dell EMC of Hopkinton, Mass., called zDP, provides continuous or near continuous versions of the data at different points in time by creating a series of targetless snapshots of different logical volumes of the storage system. The snapshots are deemed “targetless” because it is not necessary to define a target volume for a snapshot unless it is desirable to access the snapshot data. In most cases, when the data is not accessed, no target volume is defined and the snapshot data is eventually discarded as the snapshot data ages. However, in instances where it is desirable to access snapshot data (e.g., data on the storage device has been corrupted or deleted), it is possible to define a volume for a prior snapshot to access the snapshot data.

A difficulty with zDP and similar solutions is that there are a number of parameters that need to be set manually by a storage system operator, including the frequency of snapshots, the amount of storage space to allocate, the snapshot retention policy, etc. Moreover, since obtaining snapshots uses system resources (storage for the snapshot data and overhead and processing resources to obtain the snapshots), then managing and manually setting allocation of snapshot frequency and retention becomes a complex tradeoff. In many cases, the result is that storage system operators either don't use snapshots or do so but do not set the operational parameters properly or in a way that maximizes the usefulness of snapshots.

Accordingly, it is desirable to provide a mechanism for obtaining snapshots without requiring manually setting snapshot parameters.

SUMMARY OF THE INVENTION

According to the system described herein, providing automated targetless snapshots for storage volumes includes estimating storage space needed for a new set of the snapshots based on an amount of expected change of data on the storage volumes, deleting current snapshots to free up an amount of space corresponding to estimated storage space for the new set of the snapshots, and creating the new set of the snapshots. The current snapshots may be selected for deletion based on creation date of the snapshots. Snapshots with an oldest creation data may be deleted. The storage volumes may be provided on a storage system. The storage system may show storage space used by the snapshots as unallocated. Storage space used by the snapshots may be managed by a storage resource pool. The storage resource pool may convert storage space shown as unallocated into allocated storage space in response to snapshots provided therein being converted into accessible snapshots. The storage resource pool may delete snapshots in response to additional storage space being requested by a process that is independent of the snapshots. Snapshots may be selected for deletion based on creation date of the snapshots. Snapshots with an oldest creation data may be deleted.

According further to the system described herein, a non-transitory computer readable medium contains software that provides automated targetless snapshots for storage volumes. The software includes executable code that estimates storage space needed for a new set of the snapshots based on an amount of expected change of data on the storage volumes, executable code that deletes current snapshots to free up an amount of space corresponding to estimated storage space for the new set of the snapshots, and executable code that creates the new set of the snapshots. The current snapshots may be selected for deletion based on creation date of the snapshots. Snapshots with an oldest creation data may be deleted. The storage volumes may be provided on a storage system. The storage system may show storage space used by the snapshots as unallocated. Storage space used by the snapshots may be managed by a storage resource pool. The storage resource pool may convert storage space shown as unallocated into allocated storage space in response to snapshots provided therein being converted into accessible snapshots. The storage resource pool may delete snapshots in response to additional storage space being requested by a process that is independent of the snapshots. Snapshots may be selected for deletion based on creation date of the snapshots. Snapshots with an oldest creation data may be deleted.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

The system described herein provides for automatically obtaining and managing snapshots for a storage system without requiring any user settings or input. In some cases, the snapshots may be hidden from the user until accessed so that the storage space used for the snapshots is shown as unallocated space.

FIG. 1is a diagram20showing a relationship between a host22and a storage system24that may be used in connection with an embodiment of the system described herein. In an embodiment, the storage system24may be a PowerMax, Symmetrix, or VMAX storage system produced by Dell EMC of Hopkinton, Mass.; however, the system described herein may operate with other appropriate types of storage systems. Also illustrated is another (remote) storage system26that may be similar to, or different from, the storage system24and may, in various embodiments, be coupled to the storage system24, using, for example, a network. The host22reads and writes data from and to the storage system24via an HA28(host adapter), which facilitates an interface between the host22and the storage system24. Although the diagram20shows the host22and the HA28, it will be appreciated by one of ordinary skill in the art that multiple host adaptors (possibly of different configurations) may be used and that one or more HAs may have one or more hosts coupled thereto.

In an embodiment of the system described herein, in various operations and scenarios, data from the storage system24may be copied to the remote storage system26via a link29. For example, transferring data may be part of a data mirroring or replication process that causes data on the remote storage system26to be identical to the data on the storage system24. Although only the one link29is shown, it is possible to have additional links between the storage systems24,26and to have links between one or both of the storage systems24,26and other storage systems (not shown). The storage system24may include a first plurality of remote adapter units (RA's)30a,30b,30c. The RA's30a-30cmay be coupled to the link29and be similar to the HA28, but are used to transfer data between the storage systems24,26.

The storage system24may include one or more physical storage units (including disks, solid state storage devices, etc.), each containing a different portion of data stored on the storage system24.FIG. 1shows the storage system24having a plurality of physical storage units33a-33c. The storage system24(and/or remote storage system26) may be provided as a stand-alone device coupled to the host22as shown inFIG. 1or, alternatively, the storage system24(and/or remote storage system26) may be part of a storage area network (SAN) that includes a plurality of other storage systems as well as routers, network connections, etc. (not shown inFIG. 1). The storage systems may be coupled to a SAN fabric and/or be part of a SAN fabric. The system described herein may be implemented using software, hardware, and/or a combination of software and hardware where software may be stored in a computer readable medium and executed by one or more processors.

Each of the physical storage units33a-33cmay be coupled to a corresponding disk adapter unit (DA)35a-35cthat provides data to a corresponding one of the physical storage units33a-33cand receives data from a corresponding one of the physical storage units33a-33c. An internal data path exists between the DA's35a-35c, the HA28and the RA's30a-30cof the storage system24. Note that, in other embodiments, it is possible for more than one physical storage unit to be serviced by a DA and that it is possible for more than one DA to service a physical storage unit. The storage system24may also include a global memory37that may be used to facilitate data transferred between the DA's35a-35c, the HA28and the RA's30a-30cas well as facilitate other operations. The memory37may contain task indicators that indicate tasks to be performed by one or more of the DA's35a-35c, the HA28and/or the RA's30a-30c, and may contain a cache for data fetched from one or more of the physical storage units33a-33c.

The storage space in the storage system24that corresponds to the physical storage units33a-33cmay be subdivided into a plurality of volumes or logical devices. The logical devices may or may not correspond to the storage space of the physical storage units33a-33c. Thus, for example, the physical storage unit33amay contain a plurality of logical devices or, alternatively, a single logical device could span both of the physical storage units33a,33b. Similarly, the storage space for the remote storage system26may be subdivided into a plurality of volumes or logical devices, where each of the logical devices may or may not correspond to one or more physical storage units of the remote storage system26.

In some embodiments, another host22′ may be provided. The other host22′ is coupled to the remote storage system26and may be used for disaster recovery so that, upon failure at a site containing the host22and the storage system24, operation may resume at a remote site containing the remote storage system26and the other host22′. In some cases, the host22may be directly coupled to the remote storage system26, thus protecting from failure of the storage system24without necessarily protecting from failure of the host22.

FIG. 2is a schematic diagram40illustrating an embodiment of the storage system24where each of a plurality of directors42a-42nare coupled to the memory37. Each of the directors42a-42nrepresents at least one of the HA28, RAs30a-30c, or DAs35a-35c. The diagram40also shows an optional communication module (CM)44that provides an alternative communication path between the directors42a-42n. Each of the directors42a-42nmay be coupled to the CM44so that any one of the directors42a-42nmay send a message and/or data to any other one of the directors42a-42nwithout needing to go through the memory37. The CM44may be implemented using conventional MUX/router technology where one of the directors42a-42nthat is sending data provides an appropriate address to cause a message and/or data to be received by an intended one of the directors42a-42nthat is receiving the data. Some or all of the functionality of the CM44may be implemented using one or more of the directors42a-42nso that, for example, the directors42a-42nmay be interconnected directly with the interconnection functionality being provided on each of the directors42a-42n. In addition, one or more of the directors42a-42nmay be able to broadcast a message to all or at least some plurality of the other directors42a-42nat the same time.

In some embodiments, one or more of the directors42a-42nmay have multiple processor systems thereon and thus may be able to perform functions for multiple discrete directors. In some embodiments, at least one of the directors42a-42nhaving multiple processor systems thereon may simultaneously perform the functions of at least two different types of directors (e.g., an HA and a DA). Furthermore, in some embodiments, at least one of the directors42a-42nhaving multiple processor systems thereon may simultaneously perform the functions of at least one type of director and perform other processing with the other processing system. In addition, all or at least part of the global memory37may be provided on one or more of the directors42a-42nand shared with other ones of the directors42a-42n. In an embodiment, the features discussed in connection with the storage system24may be provided as one or more director boards having CPUs, memory (e.g., DRAM, etc.) and interfaces with Input/Output (I/O) modules.

Note that, although specific storage system configurations are disclosed in connection withFIGS. 1 and 2, it should be understood that the system described herein may be implemented on any appropriate platform. Thus, the system described herein may be implemented using a platform like that described in connection withFIGS. 1 and 2or may be implemented using a platform that is somewhat or even completely different from any particular platform described herein.

A storage area network (SAN) may be used to couple one or more host systems with one or more storage systems in a manner that allows reconfiguring connections without having to physically disconnect and reconnect cables from and to ports of the devices. A storage area network may be implemented using one or more switches to which the storage systems and the host systems are coupled. The switches may be programmed to allow connections between specific ports of devices coupled to the switches. A port that can initiate a data-path connection may be called an “initiator” port while the other port may be deemed a “target” port.

FIG. 3is a schematic illustration70showing a storage area network (SAN)60providing a SAN fabric coupling a plurality of host systems (H1-HN)22a-cto a plurality of storage systems (SD1-SDN)24a-cthat may be used in connection with an embodiment of the system described herein. Each of the devices22a-c,24a-cmay have a corresponding port that is physically coupled to switches of the SAN fabric used to implement the storage area network60. The switches may be separately programmed by one of the devices22a-c,24a-c or by a different device (not shown). Programming the switches may include setting up specific zones that describe allowable data-path connections (which ports may form a data-path connection) and possible allowable initiator ports of those configurations. For example, there may be a zone for connecting the port of the host22awith the port of the storage system24a. Upon becoming activated (e.g., powering up), the host22aand the storage system24amay send appropriate signals to the switch(es) of the storage area network60, and each other, which then allows the host22ato initiate a data-path connection between the port of the host22aand the port of the storage system24a. Zones may be defined in terms of a unique identifier associated with each of the ports, such as such as a world-wide port name (WWPN).

In various embodiments, the system described herein may be used in connection with performance data collection for data migration and/or data mirroring techniques using a SAN. Data transfer among storage systems, including transfers for data migration and/or mirroring functions, may involve various data synchronization processing and techniques to provide reliable protection copies of data among a source site and a destination site. In synchronous transfers, data may be transmitted to a remote site and an acknowledgement of a successful write is transmitted synchronously with the completion thereof. In asynchronous transfers, a data transfer process may be initiated and a data write may be acknowledged before the data is actually transferred to directors at the remote site. Asynchronous transfers may occur in connection with sites located geographically distant from each other. Asynchronous distances may be distances in which asynchronous transfers are used because synchronous transfers would take more time than is preferable or desired. Examples of data migration and mirroring products includes Symmetrix Remote Data Facility (SRDF) products from Dell EMC.

FIG. 4is a schematic diagram80showing a standard logical device82, a point-in-time image device84, such as a snapshot image device and/or other appropriate point-in-time image device, and a journal (or log) device86that may be used in connection with an embodiment of the system described herein. The standard logical device82may be implemented using any appropriate storage logical device mechanism, such as logical storage devices used on a Symmetrix and/or VPLEX product provided by Dell EMC, and used to access corresponding physical storage disks, like disks36a-c(seeFIG. 1). Similarly, the point-in-time image device84may be any logical or virtual device that can provide point-in-time image (or version) functionality for the logical device82. As discussed herein, the point-in-time image device84may represent a point-in-time image of all or a portion of the standard logical device82. A host coupled to a storage system that accesses the point-in-time image device84may access the point-in-time image device84in the same way that the host would access the standard logical device82. However, the point-in-time image device84does not contain any track data from the standard logical device82. Instead, the point-in-time image device84includes a plurality of table entries that point to tracks on either the standard logical device82or the journal device86.

When the point-in-time image device84is established (e.g., when a point-in-time image is made of the standard logical device82), the point-in-time image device84is created and provided with appropriate table entries that, at the time of establishment, point to tracks of the standard logical device82. A host accessing the point-in-time image device84to read a track would read the appropriate track from the standard logical device82based on the table entry of the point-in-time image device84pointing to the track of the standard logical device82.

After the point-in-time image device84has been established, it is possible for a host to write data to the standard logical device82. In that case, the previous data that was stored on the standard logical device82may be copied to the journal device86and the table entries of the point-in-time image device84that previously pointed to tracks of the standard logical device82would be modified to point to the new tracks of the journal device86to which the data had been copied. Thus, a host accessing the point-in-time image device84may read either tracks from the standard logical device82that have not changed since the point-in-time image device84was established or, alternatively, may read corresponding tracks from the journal device86that contain data copied from the standard logical device82after the point-in-time image device84was established. Adjusting data and pointers in connection with reads and writes to and from the standard logical device82and journal device84is discussed in more detail elsewhere herein.

In an embodiment described herein, hosts may not have direct access to the journal device86. That is, the journal device86would be used exclusively in connection with the point-in-time image device84(and possibly other point-in-time image devices as described in more detail elsewhere herein). In addition, for an embodiment described herein, the standard logical device82, the point-in-time image device84, and the journal device86may be provided on the single storage system24. However, it is also possible to have portions of one or more of the standard logical device82, the point-in-time image device84, and/or the journal device86provided on separate storage systems that are appropriately interconnected.

It is noted that the system described herein may be used with data structures and copy mechanisms other than tables and/or pointers to tracks discussed, for example, in connection with snapshots and/or other point-in-time images. For example, the system described herein may also operate in connection with use of clones and/or deep copy backups automatically synchronized between data and metadata. Accordingly, the system described herein may be applied to any appropriate point-in-time image processing systems and techniques, and it should be understood that the discussions herein with respect to the creation and use of snapshots, and the devices thereof, may be equally applied to the use of any appropriate point-in-time image used for point-in-time image processes in connection with protection of data and configuration metadata that enable the rolling back/forward of a storage system using the point-in-time images of the data and configuration metadata according to the system described herein.

FIG. 5is a schematic diagram90showing another example of the use of virtual devices including a standard logical device92, a plurality of point-in-time images94-97that may be generated by one or more point-in-time devices and a journal device98that may be used in connection with an embodiment of the system described herein. In the illustrated example, a point-in-time image94represents a point-in-time version of the standard logical device92taken at time A. Similarly, a point-in-time image of point-in-time image95represents a point-in-time version of the standard logical device92taken at time B, a point-in-time image96represents a point-in-time version of the standard logical device92taken at time C, and a point-in-time image97represents a point-in-time version of the standard logical device92taken at time D. Note that all of the point-in-time image94-97may share use of the journal device98. In addition, it is possible for table entries of more than one of the point-in-time images94-97, or, a subset of the table entries of the point-in-time image94-97, to point to the same tracks of the journal device98. For example, the point-in-time image95and the point-in-time image96are shown in connection with table entries that point to the same tracks of the journal device98.

In an embodiment discussed herein, the journal device98, and/or other journal devices discussed herein, may be provided by a pool of journal devices that are managed by the storage system24and/or other controller coupled to the SAN. In that case, as a point-in-time image device requires additional tracks of a journal device, the point-in-time image device would cause more journal device storage to be created (in the form of more tracks for an existing journal device or a new journal device) using the journal device pool mechanism. Pooling storage system resources in this manner is known in the art. Other techniques that do not use pooling may be used to provide journal device storage.

FIG. 6is a schematic diagram100that illustrates a system including a logical device102, a point-in-time image device104, a journal device106, and a full copy device108that may be used in connection with an embodiment of the system described herein. As noted elsewhere herein, the logical device102may be implemented using any appropriate storage logical device mechanism. Similarly, the point-in-time image device104may be any logical point-in-time image device that can provide snapshot functionality, and/or other appropriate point-in-time image functionality, for the logical device102. The journal device106provides storage for sections of data (e.g., tracks) of the logical device102that are overwritten after the point-in-time image device104has been initiated. The journal device106may be provided on the same physical device as the logical device102or may be provided on a different physical device.

In an embodiment, the system described herein may also be used in connection with full copies of data generated and stored according operation of the full copy device108. The full copy device108may be a logical storage device like the logical device102. As discussed in more detail elsewhere herein, the full copy device108may be configured to contain data copied from the logical device102and corresponding to one or more point-in-time images. As described below, the point-in-time image device104may create a point-in-time image and then, subsequently, data from the logical device102, and possibly the journal device106, may be copied and/or refreshed to the full copy device108in a background process that does not interfere with access to the logical device102. Once the copy is complete, then the point-in-time image is protected from physical corruption of the data of the logical device102, as discussed in more detail elsewhere herein. Note that, as shown in the figure, it is possible to have multiple copy devices108′,108″ etc. so that all of the copy devices108,108′,108″ protect the point-in-time image from physical corruption. Accordingly, for the discussion herein, it should be understood that references to the copy device108may include, where appropriate, references to multiple copy devices. Note that, for some embodiments, the copy devices108,108′,108″ may be copies provided at different times. Similarly, the system described herein may be applicable to multiple point-in-time copies provided at the same time or different times, like that shown inFIG. 5.

It is noted that the system described herein may be used in connection with use of consistency groups and with features for maintaining proper ordering of writes between storage systems. A consistency group represents a grouping of storage volumes (virtual or not) which together offer an application consistent image of the data. Reference is made to U.S. Pat. No. 7,475,207 to Bromling et al., entitled “Maintaining Write Order Fidelity on a Multi-Writer System,” that discloses a system for maintaining write order fidelity (WOF) for totally active storage system implementations using WOF groups and including application to features such as point-in-time snapshots and continuous data protection, and to U.S. Pat. No. 7,054,883 to Meiri et al., entitled “Virtual Ordered Writes for Multiple Storage Devices,” that discloses features for ordering data writes among groups of storage systems. The above-noted references are incorporated herein by reference.

In an embodiment of the system described herein, it is further noted that content protected by point-in-time images, such as snapshots, e.g. in connection with CS/CDP, may be extended to include not only user data but further include configuration metadata, and/or other appropriate configuration information, of the storage management system. Configuration metadata of the storage management system may be information used for configuration volumes, storage devices, consistency groups and/or other appropriate storage management system elements, as further discussed elsewhere herein. A user may want to rollback a storage management system to a past point due to performance or stability issues attributed to configuration changes. The system described herein enables rollback to prior states based on storage configuration metadata in addition to rollback of user data and provides for synchronization of the data and configuration metadata in connection with a rollback, as further discussed elsewhere herein. For further discussion of systems using point-in-time image technologies involving both user data and configuration metadata, reference is made to U.S. Pat. No. 9,128,901 to Nickurak et al., issued on Sep. 8, 2015, entitled, “Continuous Protection of Data and Storage Management Configuration,” which is incorporated herein by reference.

FIG. 7is a schematic diagram200that illustrates a continuous protection device202that facilitates continuous or near continuous backup of data using snapshots, and/or other appropriate point-in-time images, and that may be used according to an embodiment of the system described herein. The continuous protection device202may contain pointers to a standard logical device204for a plurality of tracks such that, for any particular track, if the continuous protection device202points to a corresponding track of the standard logical device204, then the corresponding track has not changed since creation of the continuous protection device202. Note that any subsections, besides track, may be used to implement the system described herein. Accordingly, it should be understood in connection with the discussion that follows that although tracks are mentioned, other units of data having another size, including variable sizes, may be used. The continuous protection device202also contains pointers to a journal device206for a plurality of corresponding tracks. The journal device206contains data for tracks that have changed since creation of the continuous protection device202.

The diagram200also shows an I/O module208that handles input and output processing to and from other modules, such as input and output requests made by the DA's38a-38cand HA's28a-28c. The I/O module208may be provided with information from a cycle counter210and/or a timer212, among other possible information sources, that may be used to synchronize storage for a plurality of storage systems (i.e., a consistency group). The I/O module208may further include, and/or be coupled to, a user interface220that enables a user to tag data streams, among other functions as further discussed elsewhere herein. The user interface may be implemented using appropriate software and processors and may include a display and/or otherwise include operation using a display.

The system described herein allows for the ability to roll back/forward on multiple levels, including: per-volume basis, for configuration metadata and/or data; per-consistency group basis, for configuration metadata and/or data; per-system basis (all consistency groups, and system-wide configuration), for configuration metadata and/or data; and/or per-multi-system basis with the ability to control multiple systems with one user interface, for rolling management configuration and/or data. Other features and advantages of the system described herein include: elimination of manual storage configuration backups, which means reducing error-prone/inconvenient steps; elimination of manual storage configuration restores, which provides for reducing another set of error-prone/inconvenient steps; automatic write order fidelity across rollback in the presence of configuration changes; ability to control the roll back/forward points for management configuration/data independently. This allows choosing whether to roll management configuration back/forward only in those circumstances that warrant it; and/or ability to control the roll back/forward for configuration/data stream on a per volume and/or consistency-group and/or system-wide basis.

The system described herein allows for choosing the granularity of the roll back/forward of some of the system's volumes/consistency groups without requiring the whole system to roll back. Furthermore, the multi-system control aspect of the system described herein allows for restoring an organization's whole infrastructure (management configuration and data, independently) to a point in the past (or future) with the convenience of a single user interface.

According to the system described herein, techniques are provided for data protection as a process to secure frequent, and space efficient, versions of consistent point-in-time images of a group of volumes using snapshot technology. In an embodiment, the group of volumes may be defined and organized as Versioned Data Group (VDGs). This system described herein may include tools and procedures to plan and operate a VDG and to use the member versions of the VDG to create and terminate target volume sets, particularly in connection with managing and/or optimizing use of log space on a journal or log device, as further discussed in detail elsewhere herein.

The system described herein provides for automation to create and manage frequent snapshots of defined groups of volumes. The incremental approach of the system described herein provides a convenient way to roll back to prior point-in-time versions to investigate data damage due to processing errors or other forms of corruption. The intervals between versions may be controlled. With sufficient resources the version increments may be controlled to be small, such as in minutes or smaller. The system beneficially provides for identifying, monitoring, and reclaiming use of log space in log devices in connection with managing recovery and roll back capabilities of the system to desired data versions for purposes of data protection. The system described herein may be implemented using any appropriate computing architecture and operating system, including, for example, using components of IBM Corporation's System z environment including use of z/OS and z/Architecture computing systems. For further discussion of the use of z/OS and z/Architecture components in simulated I/O environments, including techniques for the emulation of z/OS and z/Architecture components, reference is made to U.S. Pat. No. 9,170,904 to LeCrone et al, issued on Oct. 27, 2015, entitled “I/O Fault Injection Using Simulated Computing Environments,” which is incorporated herein by reference.

The system described herein further provides for that by using target volume sets created from VDG version, repair strategies may be developed and tested without requiring the isolation of production systems or recreations to diagnose problems. Repairs may be possible on the source systems or the creation of a repaired replacement. Diagnostic target sets may not necessarily require full source image capacity. Techniques for implementation may include determining the storage capacity required for the associate snapshot log pool. Advantageously, the log capacity required according to the system described herein may be significantly less than the total duplication of source volumes capacity.

A point-in-time image (or snapshot) system architecture according to an embodiment of the system described herein may be storage efficient in that only first write track pre-write images are logged. The total number of unique tracks written while a snapshot version is active determines the log pool capacity consumed. If multiple versions are created the persistence of the track pre-write image in the pool is dependent on the number of previously activated versions that share that log entry. Reduction of log capacity consumption requires that a track pre-write image is no longer shared by versions. This is achieved by the termination of all snapshot versions sharing that image.

Multiple snapshot versions of a VDG set of volumes are created at regular intervals. Differential data tracking information, such as SDDF tracking information, may be used to analyze the write frequency and density of the source members of a VDG over a representative period of versioning intervals. Based on the analysis, the versioning intervals may be controlled to optimize the storage of the versions and the use of log capacity.

Pre-write images for tracks are created in the log pool or device when the first new write to a track occurs after a new snapshot version is activated. All subsequent writes to that track until the next interval are not logged since they are not needed to recreate a target image of the snapshot version. All prior versions containing the first write track share the same logged pre-write image. According to the system described herein, using the current source volumes and logged track pre-write images a selected version can be recreated on a target volume set.

SDDF provides a local function that marks modified (written) tracks and does not require any remote partner device. The differential update for local and remote devices uses the local and remote SDDF data to determine which tracks need to move to synchronize the pair. According to the system described herein, a first write analysis, as described elsewhere herein, may use local SDDF information that marks which tracks have been modified in a given interval. At the end of a current interval the SDDF information may be collected for future analysis and then cleared from the devices of interest. The SDDF mark, collect, and clear processes may repeat for each subsequent interval. The resulting collection of interval SDDF information provides maps of first writes that may be analyzed. VDG interval addition or reduction in log track space consumption may be determined. The collected SDDF maps may also contain information about persistence of shared first write tracks between VDG intervals.

For small interval SDDF first write maps collected, various VDG characteristics may be analyzed. For example, if the collected map intervals are 2 minutes VDG intervals of 2, 4, 6, 8 etc. . . . minutes may be analyzed for log space impact. The VDG interval duration and the number VDG intervals in a rotation set allows an analysis of rollback resolution (the time between snapshots) and log space consumption and management. The determination of log space versus how granular a CDP period and how far in the past is recovery possible may be assessed, as further discussed elsewhere herein.

FIGS. 8-11are schematic illustrations showing representations of storage devices(s) in connection with a data protection system using a log device according to an embodiment of the system described herein.

FIG. 8shows a representation300according to an embodiment of the data protection system described herein with a five track storage device for which each track one-five may contain source volume data D1-D5, respectively. A journal or log device302is shown, like that discussed elsewhere herein, that may be used in connection with data protection for purposes of roll back or other recovery processing. As discussed elsewhere herein, the log device302is not necessarily a single device and may include log capacity storage of a log pool comprised of one or more devices.

FIG. 9shows a representation300′ according to an embodiment of the data protection system described herein showing a point-in-time image or version (V1) of data D3made. There has been no write yet performed to the source data and thus there are no log entries in the log device302. It is noted that the point-in-time version V1of data D3is illustrated in connection with Track three where the source volume of data D3is stored. However, it is noted that the version V1(and/or any other of the point-in-time versions discussed herein) may be stored in any appropriate storage location, including any suitable one or more of the devices discussed herein, and is not necessarily stored on Track three or any other of the tracks shown in connection with the five track storage system.

FIG. 10shows a representation300″ according to an embodiment of the data protection system described herein showing additional point-in-time versions being made according to the system described herein. There are no writes to the devices over the intervals in which versions V2and V3are made, thereby versions V2and V3may be the same as version V1, and there are no required log entries for any versions V1-V3in the log device302. The figure shows that there are no writes to the device until the time of version V4for a write (W1) to Track three (causing data D3′ on the source volume) which causes a pre-write log entry302ain the log device302to be logged according to the system described herein. The log entry302aat the time of version V4is a log entry corresponding to data D3.

FIG. 11shows a representation300′″ according to an embodiment of the data protection system described herein showing point-in-time version creation continuing until the time of version V8when another write (W2) to Track three (resulting in data D3″ stored on the source volume) creates a pre-write log entry302bin the log device302corresponding to the write W1(for data D3′). The log entry302bat the time of version V8is a log entry corresponding to the write W1. Versions V1, V2, and V3may share the log entry302aholding D3. Versions V4, V5, V6, and V7may share the log entry302bholding W1. V8(reflecting write W2) does not need log capacity until a subsequent write occurs.

The system described herein may be used to recover log space based on desired criteria. For example, the criteria may be to recover 50% of the log space, and a query may be as to which point-in-time version could be terminated to accomplish this such that log space for corresponding log entries may be reclaimed/recovered. Control and management of queries, criteria and/or result output may be performed using control modules and user interfaces like that discussed elsewhere herein (see, e.g.,FIG. 7). Log persistence is where some number of versions share the same pre-write image. This could be representative of data that is periodic and only updated infrequently. In this case, the number of point-in-time versions necessary to terminate could be large in order to reclaim log space. Log entries for more active same track writes may be shared by a smaller number of versions, thereby requiring fewer version terminations to reclaim log space and recover desired log capacity.

FIGS. 12-14show scenario representations according to an embodiment of the system described herein for reclamation processing of a subject device to reclaim 50% of log capacity according to the scenario, discussed above, where Track three (storing data D3) is the subject of data writes. The example of reclaiming 50% log capacity as a criteria is discussed; however, it is noted the system described herein may be appropriately used in connection with reclaiming any desired amount or percentage of log capacity.

FIG. 12is a schematic representation301showing that terminating point-in-time versions V1, V2, and V3would allow the log entry302acorresponding to data D3to be reclaimed in the log device302(shown by dashed lines around log entry302a). In this case, versions V4through V8persist with the W1log pre-write image required to reconstitute V4through V7. V8has no pre-write image required yet.

FIG. 13is a schematic representation301′ showing that, alternatively and/or additionally, terminating versions V4, V5, V6, and V7allow the log entry302bholding W1to be reclaimed in the log device302(shown by dashed lines around log entry302b). In this case, versions V1, V2, V3, and V8persist with the log entry302afor the D3pre-write image required to reconstitute V1through V3. V8has no subsequent pre-write image required yet.

FIG. 14is a schematic representation301″ showing that, alternatively and/or additionally, terminating V5through V8allows the log entry302bholding W1to be reclaimed in the log device302(shown by dashed lines around log entry302b). In this case, versions V1, V2, V3share the log entry302afor the D3pre-write image to reconstitute V1through V3. V4has no subsequent pre-write image required.

FIGS. 15 and 16show scenario representations according to an embodiment of the system described herein for reclamation of a subject device when multiple tracks are involved to reclaim 50% of the log capacity.

FIG. 15is a schematic representation400according to an embodiment of the system described herein showing an ending state of a scenario in which a write W1was made to D3(now data D3′ on source volume) on Track3at a time of the version V4and a write W2was made to data D2(now data D2′ on source volume) on Track2at a time of version V8. Accordingly, in log device402, log entry402acorresponds to the D3pre-write image created at the time of version V4and log entry402bcorresponds to the D2pre-write image created at the time of version V8.

FIG. 16is a schematic representation400′ according to an embodiment of the system described herein showing reclaiming of 50% log capacity based on the scenario ofFIG. 15. In this case, the D3pre-write image is required by versions V1through V3, and the D2pre-write image is required by versions V1through V7. Accordingly, only terminating V1through V3reclaims 50% of the log capacity, namely, the D3pre-write image log space of entry402ain the log device402(shown by dashed lines around the entry402a). The D2pre-write image of log entry402bis the most persistent being shared by all versions except V8. The example of reclaiming 50% log capacity as a criteria has been discussed; however, it is noted the system described herein may be appropriately used in connection with reclaiming any desired amount or percentage of log capacity.

According to the system described herein, using data collected for the first writes to tracks in a volume group during a planning interval allows estimating the potential maximum capacity for the log pool that is needed for various frequency of version creation.

The system described herein provides that information on pre-write image log persistence or the number of consecutive versions sharing a log entry may also be analyzed. This provides information concerning how removing versions from the VDG effects log pool capacity reclamation. This information may be used for understanding the number of versions that may be removed to achieve a target log pool capacity. Accordingly, oldest versions and versions other than the oldest in a rotation set may be considered for removal.

Additionally, rotation of a set number of versions (the VDG) may be analyzed. First writes in an interval give the net add to log pool capacity consumption. In this case, termination of the oldest version member in the rotation set may give the potential maximum reduction in log consumption. The actual reduction is dependent on the number of versions sharing a particular track pre-write image. When a target log pool size is desired the number of versions to terminate can be analyzed.

In a VDG rotation cycle the oldest member version would be removed prior to adding a new version. The log capacity may need to be the maximum expected concurrent log pre-write image capacity plus a margin for safety. It is noted that demand reclaim from oldest to newest may require the least active analysis. For example, using differential data write monitoring, such as SDDF write monitoring, for each version allows for a log capacity by version metric. However, reclaiming pre-write image log capacity may involve termination of some number of versions to achieve a desired log capacity reduction. As seen, for example, in the scenarios discussed herein, three versions (V1, V2, and V3) may need to be terminated before the single pre-write image log capacity associated with the data D3can be reclaimed. A worst case would be where many versions with low or no writes are created and during the most recent version having most or all tracks written. An example might be where a DB2 table create and format occurs in generation100and the prior 99 versions share the pre-write images of the involved tracks. The 99 prior versions would need to be terminated before the pre-write image log capacity could be reclaimed.

Exempting particular versions from rotation termination makes this problem even more evident. While capacity consuming (equal to the source capacity of the VDG) creating a full copy target and unlinking it after being fully populated would be an operational tradeoff to diminishing impact on log reclamation by holding one or more versions exempt from termination.

In another embodiment, the system described herein may be used in connection with a continuous review of which versions contribute the least to log capacity but share the most images with other versions. Referring, for example, back toFIG. 15, in this case it is noted that versions V1, V2, V5, V6and V7could all be terminated without losing any unique version of the source volume data. V3, V4, and V8are unique versions for this source volume.

FIG. 17is a schematic representation500according to the embodiment of the system described herein shown inFIG. 15in which versions V1, V2, V5, V6and V7have been terminated, but all unique first write pre-write images in each version interval are preserved. Tracks with data D1, D2, D3, D4, D5, W1, and W2and the versions that consistently relate them in time are available to create useable target sets based on use of the log entries502a,502bof the log device502. This can be determined by tracking the first write differential (SDDF) data for each version interval.

According further to the system described herein, it is noted that with a VDG creating short interval snapshot members it is possible that some VDG members will have no first write activity and can be terminated after the next interval VDG is activated. If there is first write activity within the VDG there may be subgroupings in that VDG interval that do not have any first writes for the interval. If a subgroup is identified by the user as logically-related volumes (a particular application, for example) only the snapshots of the volumes in that subgroup may be terminated if there are no first write to that subgroup. This could also apply to single volumes within the VDG that do not have interdependent data with other volumes in the VDG. These determinations may be specified by the user of the VDG control mechanism.

Accordingly,FIGS. 18 and 19show scenario representations according to an embodiment of the system described herein for reclamation of a subject device when multiple volumes are involved to reclaim log capacity. Specifically, in an embodiment, the system described herein may also be used in connection with application to volumes instead of tracks and may provide for continuously collapsing volume log images.

FIG. 18is a schematic representation600according to an embodiment of the system described herein showing an ending state of a scenario for storage of 5 volumes (Volumes one-five) and for which eight point-in-time versions (V1-V8) thereof have been made. The representation600shows a state in which a write W1was made to D3(now data D3′) of Volume three at a time of the version V4and a write W2was made to data D2(now data D2′) of Volume two at a time of version V8. Accordingly, in log device602, log entry602acorresponds to the D3pre-write image created at the time of version V4and log entry602bcorresponds to the D2pre-write image created at the time of version V8.

FIG. 19is a schematic representation600′ according to the embodiment of the system described herein shown inFIG. 18in which versions V1, V2, V5, V6and V7have been terminated, but all unique first write pre-write images of the volumes in each version interval are preserved. The capability for reconstruction of a VDG point-in-time when constituent member volumes may have their snapshot terminated is illustrated in the figure. Point in time V1, V2and V3can independently be reconstructed using the original data images D1through D5of the Volumes one-five and the log entries602a,602bof the log device602. V5, V6, and V7only need the W1first write from V4. Reconstruction of version V8needs the Volume three version V4for W1and itself for the Volume two W2first write pre-write image. This figure depicts the minimum (3 versions) needed to reconstruct eight distinct points in time for the illustrated volumes. A first write to any single track on a volume requires the volume snapshot to be preserved.

FIG. 20is a schematic diagram showing a system700implementing data protection according to an embodiment of the system described herein. A point-in-time image device702may facilitate continuous or near continuous backup of data using snapshots, and/or other appropriate point-in-time images, as further discussed in detail elsewhere herein. The point-in-time image device702may contain pointers to a standard logical device704for a plurality of tracks storing data. The point-in-time image device702may also contain pointers to a log device706logging data changes to corresponding tracks, as further discussed in connection with the scenarios discussed elsewhere herein.

The system700may also include a I/O module708that handles input and output processing in connection with receiving and responding to requests and criteria concerning the providing of efficient data protection operations in accordance with the system described herein. The I/O module708may be provided with information from a cycle counter710and/or a timer712, among other possible information sources, that may be used in connection with storage of data among a plurality of storage systems (i.e., for a consistency group and/or VDG). The I/O module708may further include, and/or be coupled to, an interface720that enables interaction with users and/or hosts in connection with operation of the system described herein.

A point-in-time data analytic analyzer730is shown that may be used to automatically/programmatically determine which point-in-image to roll back for one or more data recovery operations according to an embodiment of the system described herein. For example, information, such as host meta structures, may be available to the analyzer730to facilitate the scanning and/or identification of logical data corruption or errors. Such host meta structures may include structures of IBM's System z environment, as discussed elsewhere herein, such as logical structures of a volume table of contents (VTOC), VTOC index (VTOCIX), virtual storage access method (VSAM) volume data sets (VVDS), catalogs and/or related structures that are logical in nature and which may be used in connection with the scanning for logical failures rather than physical failures, and may indicate what a user or customer may be looking for in a roll back or recovery scenario. For example, in an IBM mainframe storage architecture, a VTOC provides a data structure that enables the locating of the data sets that reside on a particular disk volume, and the z/OS may use a catalog and the VTOC on each storage system to manage the storage and placement of data sets. In an embodiment, the system described herein may then use these structures to efficiently provide desired roll-back and data protection operations according to the features discussed herein.

It is noted that the I/O module708, interface720and/or analyzer730may be separate components functioning like that as discussed elsewhere herein and/or may be part of one control unit732, which embodiment is shown schematically by dashed lines. Accordingly, the components of the control unit732may be used separately and/or collectively for operation of the data protection system described herein in connection with the creation, maintenance, identification and termination of point-in-time image versions to respond to requests and criteria, like that discussed elsewhere herein, including criteria concerning identification of necessary point-in-time versions to fulfil desired roll back scenarios and criteria involving the efficient use of log capacity to maintain the desired data protection capability.

For operation and management functions, the system described herein may provide for components like that discussed herein that may be used to create a VDG volume group and support sets of selection options, such as Group Name Services (GNS) in connection with data protection operations. The system described herein may further be used to define version interval frequencies and to define the maximum number of member versions in a VDG. Options for when the maximum is reached may include rotation when the oldest version is terminated before the next version is created, stopping with notification, and terminating n number of oldest versions before proceeding, etc. The system may further define target volume set(s) and validate that the type, geometry, and number match the related VDG.

The system described herein provides for automation to manage one or more VDGs. Point-in-time versions may be created based on defined interval criteria on a continuing cycle. VDG version rotation may be provided to remove the versions prior to next VDG version creation. The number of VDG version terminations necessary to achieve a log pool capacity target may be tracked. Host accessible images of selected VDG versions may be created and metadata of the target set may be managed to allow successful host access. Metadata management may include: validation of type and number of target volumes; online/offline volume verification; structure checking of a target volume set; optional volume conditioning; catalog management and dataset renaming; and providing alternate logical partition (LPAR) access.

A target volume set may be created from a selected VDG version and a user may be provided with selected copy and access options. A selected target volume set may be removed and which may include validating a target volume set system status, providing secure data erase of target volume set volumes and/or returning target volume sets to available pools. Specific versions may also be removed and the system supports explicit version termination, as discussed in detail elsewhere herein.

The system described herein may provide for monitoring and reporting functions using components like that discussed elsewhere herein. The status of created versions in a VDG may be monitored. Log pool capacity may be monitored and the system may provide for alerts and actions for log pool capacity targets, log capacity reclaim reports may be generated when versions are removed (i.e. during cycle rotation), and active target volume sets needed to be removed to allow the removal of a version may be identified. The status of an active target volume set, and related VDG versions may be monitored. The status of target volumes sets created outside (unmanaged) of the VDG environment may be monitored. Versions needed to be removed to reclaim some target amount of log pool capacity may be identified, as discussed in detail elsewhere herein.

Referring toFIG. 21, a flow diagram800illustrates processing performed in connection with automatically obtaining and managing snapshots for storage volumes on a storage device. In an embodiment herein, snapshots are automatically obtained without any user settings or input, thus eliminating the need for user involvement in the snapshot process. In addition, in some cases, the storage used for snapshots, including snapshot data storage and metadata, is shown to the user as unallocated space that can be used, for example, for data write operations by a host. Thus, the automatic snapshot mechanism may be completely transparent to a user.

Processing for the flow diagram800begins at a first step802where the system estimates an amount of storage needed for a next round of snapshots. Each round of snapshots uses storage space for snapshot data as well as for metadata used in connection with maintaining the snapshots. The system described herein estimates the amount of data and metadata space expected for a set of snapshots for all of the volumes of a storage system. The estimate is based on an amount of expected change in an upcoming iteration (i.e., the number of writes expected), which can be estimated based on a number of previous writes for each iteration. Note that each write during an iteration requires corresponding data to be copied to a log volume, as described in more detail elsewhere herein. Thus, for example, if twenty GB of a volume is modified during an iteration, the logfile space for the data needs to be twenty GB. Of course, the associated metadata also requires storage space.

Following the step802is a step804where the estimated amount of space is made available by deleting current snapshots, if necessary. Processing at the step804is described in more detail elsewhere herein. Following the step804is a test step805where it is determined if the system has enough otherwise unused memory space (free space) available to support the snapshot mechanism described herein. In an embodiment herein, the system has a minimum threshold of otherwise free memory space which must be available for use as hidden snapshots. The minimum may be set to any appropriate value, such as five percent of the total memory used by the storage system for data storage. Determining if there is enough available free space is discussed in detail elsewhere herein. If it is determined at the step805that enough free space is available, then control transfers from the test step805to a step806where the snapshots are obtained using, for example, the snapshot mechanism described herein. Following the step806is a step808where the system waits for a next iteration. Note that the step808is reached directly from the test step805if there is not enough free memory space (memory space that is otherwise unused) to support the snapshot mechanism described herein. The amount of delay (wait time) at the step808corresponds to the cycle time of the system, which is how often snapshots are obtained. In an embodiment herein, the cycle time is five minutes, although any appropriate cycle time may be used. Following the step808, control transfers back to the step802, discussed above, for another iteration.

Referring toFIG. 22, a flow diagram850illustrates in more detail processing performed at the step804, described above, where older current snapshots are deleted to make available storage space needed for new snapshots. Processing begins at a test step852where it is determined if more space is needed for the snapshots described herein. In some instances, the system may already have enough available space to accommodate a new round of snapshots. This may occur, for example, in instances where other data on the storage system (unrelated to snapshots) had been previously deleted. In an embodiment herein, the system maintains at least 15% of the otherwise free storage space as unused storage space, although that amount may be adjusted. Maintaining unused storage space protects the system from running out of storage space resources. Note also that the test at the step852determines if otherwise unused memory is available for the snapshot mechanism described herein. Thus, the test at the step852may determine if memory is both needed and available.

If it is determined at the step852that no new storage space is needed, then processing is complete. Otherwise, control transfers from the test step852to a step854to select the least recently created snapshots. As discussed elsewhere herein, the expected amount of space the new snapshots are expected to use is based on the activity level (an amount of expected change of data) of the system where more writes to more portions of the storage system result in more space needed for new snapshots. Following the step854is a step856where the snapshots are deleted (terminated). Following the step856is a test step858where it is determined if still more space is needed. The test at the step858is similar to the test at the step852, described above, where it is also determined if memory is available for the snapshot mechanism described herein. Note that snapshots are deleted at the step856without regard to how much room is used by the deleted snapshots; the snapshots deleted at the step856may take up very little space. Thus, it's possible that the snapshots being deleted at the step856do not necessarily result in enough space being made available for the next iteration of snapshots (new snapshots). If it is determined at the step858that no additional storage space is needed for new snapshots, then processing is complete. Otherwise, control transfers from the test step858back to the step854, discussed above, to select additional older snapshots to be deleted.

Referring toFIG. 23, a flow diagram870illustrates processing performed in connection with a write operation being performed to the storage system. As discussed elsewhere herein, the automatic snapshots may be hidden and the space taken up by the automatic snapshots may be shown as unallocated storage space to a user. Showing space used for automatic snapshots as unallocated is performed by modifying the storage resource pool (SRP) functionality of the storage system to keep track of space used by the hidden snapshots while reporting the space as unallocated. When a user desires to access a version of prior data corresponding to an automatic snapshot, the system provides appropriate access to data from the hidden snapshot (snapshot pointers, log files, etc.). The system may allow the user to review the automatic snapshots in order to restore a specific snapshot corresponding to a specific point in time. The system may also allow a user to convert one or more hidden snapshots into permanent, visible, snapshots by using the modified storage resource pool to transform the associated pointers and log data into visible, conventional, snapshot data stored on allocated memory that is accessible using the system described herein.

In addition, it is possible for data to be written by a process that is independent from the automatic snapshots described herein. In such a case, the automatic snapshot mechanism may need to delete older snapshots in order to make space available for the separate process. This is illustrated in the flow diagram870. Processing for the flow diagram870begins at a step872where the system (storage resource pool) determines how much space will be needed. The processing at the step872is fairly straightforward. For example, if a conventional data write operation that writes data to the storage system is being performed, then the processing at the step872determines how much space will be needed based on the amount of data being written. Following the step872is a step874where the amount of additional memory space that is needed is freed up by the storage resource pool and the automatic snapshot mechanism. The processing at the step874is similar to the processing at the step804, discussed above, which is shown in more detail by the flow diagram850ofFIG. 22. Following the step874is a step876where the write operation is performed to write data that is separate (independent) from the automatic snapshot mechanism. Following the step876, processing is complete.

Note that, when data is written outside the snapshot mechanism described herein, there are three general possibilities with respect to memory state. The first possibility is that the data being written is smaller than the buffer that is used with the system (fifteen percent in the embodiment described herein). In that case, the written data is written in the buffer space and the system deletes older snapshots on a next iteration to restore the buffer to the desired size. The second possibility is that the write is larger than the buffer but small enough so that the snapshot mechanism described herein can continue operation. In that case, older snapshots are deleted at the step874and the buffer size may be subsequently adjusted at a next iteration of snapshots. The third possibility is that the write is so large that it exceeds the buffer and does not leave enough otherwise unused memory to allow the snapshot mechanism to continue operation. In that case, older snapshots are deleted at the step874and, at a next iteration, the test at the step805(described above) causes the system to halt making additional snapshots.

The system described herein may be used with any type of storage volumes on any computer system capable of supporting appropriate functionality, including laptop and desktop computers running Windows OS, Macintosh OS, Linux, Unix, etc. as well as array storage systems that provide snapshot functionality, such as the system described in U.S. Pat. No. 9,965,216, which is incorporated by reference herein. Various embodiments discussed herein may be combined with each other in appropriate combinations in connection with the system described herein. Additionally, in some instances, the order of steps in the flow diagrams, flowcharts and/or described flow processing may be modified, where appropriate. Further, various aspects of the system described herein may be implemented using software, hardware, a combination of software and hardware and/or other computer-implemented modules or devices having the described features and performing the described functions. The system may further include a display and/or other computer components for providing a suitable interface with a user and/or with other computers.

Software implementations of the system described herein may include executable code that is stored in a non-transitory computer-readable medium and executed by one or more processors. The computer-readable medium may include volatile memory and/or non-volatile memory, and may include, for example, a computer hard drive, ROM, RAM, flash memory, portable computer storage media such as a CD-ROM, a DVD-ROM, an SD card, a flash drive or other drive with, for example, a universal serial bus (USB) interface, and/or any other appropriate tangible or non-transitory computer-readable medium or computer memory on which executable code may be stored and executed by a processor. The system described herein may be used in connection with any appropriate operating system.