RECOVERY BY MULTI-CLOUD HANDOFF

One example method includes receiving, at a remote site from a production site, copies of production site assets, storing, at the remote site, the copies of the production site assets, using, at the remote site, the copies of the production site assets to restore a temporary production site, running the temporary production site at the remote site, and restoring, from the remote site to the production site, the copies of the production site assets.

FIELD OF THE INVENTION

Embodiments of the present invention generally relate to failover, and recovery, of a production site. More particularly, at least some embodiments of the invention relate to systems, hardware, software, computer-readable media, and methods, for implementing and using a temporary failover site while an associated production site is offline.

BACKGROUND

Contrary to common belief, data recovery time after a ransomware attack may not be constrained by backup/restore time. Rather, the predominant constraint is that insurance companies, law enforcement, and sometime even government agencies, take over the production site in order to conduct damage assessments and forensic investigation. This process usually takes a few days to conclude, after which the environment is cleansed, to ensure no ransomware is left, and then the backups are restored. This restoration process may take a few hours, or more, depending on the size of the site.

A problem with this approach is that while the production site is being evaluated, the production operations are shut down. Thus, the enterprise may be at a standstill until the investigation is concluded. While, as noted, the production site may only be unavailable for a few days, even this seemingly short timespan may be quite significant to the enterprise in terms of lost business, and/or other considerations.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

Embodiments of the present invention generally relate to failover, and recovery, of a production site. More particularly, at least some embodiments of the invention relate to systems, hardware, software, computer-readable media, and methods, for implementing and using a temporary failover site while an associated production site is offline.

In general, example embodiments of the invention may include creation and use of a failover site, which may comprise a public cloud site, that may be used to conduct production operations while the production site is down. Production site assets which may comprise a combination of virtual, and/or physical, elements may be replicated to the failover site. Similarly, production site data may be replicated to the failover site. Then, when the production site is brought back up, part or all of the failover site may be synched back to the production site, so as to enable resumption of normal production operations at the production site. The production site and/or the failover site may be virtual, physical, or may comprise a combination of virtual and physical elements.

In one example embodiment, data and virtual entities of a production site may be replicated to a vault of an off-premises site. When the production site becomes unavailable for some reason, the data and virtual entities can be restored to the off-premises site from the vault at the off-premises site. The restored system and data may then be used as a temporary production site. When the production site is back up and ready for use, the virtual entities, and data differentials that have accumulated at the remote site while the production site was down, may then be restored from the remote site to the production site.

In particular, one advantageous aspect of an embodiment of the invention is that normal production site operations may continue while the production site itself is down. As another example, a failback process from a remote site to a production site may proceed relatively quickly since only data differentials may need to be restored. Various other advantages aspects of some example embodiments will be apparent from this disclosure.

A. Aspects of an Example Architecture and Environment

In general, embodiments of the invention may be implemented in connection with systems, software, and components, that individually and/or collectively implement, and/or cause the implementation of, data protection operations which may include, but are not limited to, data replication operations, IO replication operations, data read/write/delete operations, data deduplication operations, data backup operations, data restore operations, data cloning operations, data archiving operations, and disaster recovery operations. More generally, the scope of the invention embraces any operating environment in which the disclosed concepts may be useful.

At least some embodiments of the invention provide for the implementation of the disclosed functionality in existing backup platforms, and storage environments such as the Dell PowerProtect Cyber Recovery system (CR). In general however, the scope of the invention is not limited to any particular data backup platform or data storage environment.

New and/or modified data collected and/or generated in connection with some embodiments, may be stored in a data protection environment that may take the form of a public or private cloud storage environment, an on-premises storage environment, and hybrid storage environments that include public and private elements. Any of these example storage environments, may be partly, or completely, virtualized. The storage environment may comprise, or consist of, a datacenter which is operable to service read, write, delete, backup, restore, and/or cloning, operations initiated by one or more clients or other elements of the operating environment. Where a backup comprises groups of data with different respective characteristics, that data may be allocated, and stored, to different respective targets in the storage environment, where the targets each correspond to a data group having one or more particular characteristics.

In addition to the cloud environment, the operating environment may also include one or more clients that are capable of collecting, modifying, and creating, data. As such, a particular client may employ, or otherwise be associated with, one or more instances of each of one or more applications that perform such operations with respect to data. Such clients may comprise physical machines, or virtual machines (VM), containerized computing solutions, mobile devices, IoT (Internet of Things) systems and devices, edge devices and systems, and any other systems and devices, which may comprise hardware and/or software, that are capable of generating new and/or modified data.

As used herein, the term ‘backup’ is intended to be broad in scope. As such, example backups in connection with which embodiments of the invention may be employed include, but are not limited to, full backups, partial backups, clones, snapshots, and incremental or differential backups.

A. Aspects of an Example Architecture and Operating Environment

With attention now toFIG.1, details are provided concerning an example architecture, generally denoted at100, in connection with which some embodiments may be implemented. The configuration presented inFIG.1is by way of illustration and is not intended to limit the scope of the invention in any way.

In the example ofFIG.1, a production site102, which may also be referred to as a ‘local site’ may be provided at which the normal operations of an enterprise are performed. The production site102may comprise various entities104, such as VMs or containers for example, that operate to perform production site functions. As such, the entities104may include various applications, among other things. Further, the entities104may operate to generate data106. Copies of the entities104and data106may be stored locally in a vault108, which may be air-gapped. One example of such a vault is a Dell PowerProtect Cyber Recovery system (CR) vault.

Note that as used herein, an ‘air gap’ embraces a physical isolation, or separation, of a storage vault from any exterior system or device that may be susceptible to attack. That is, when the air gap associated with a vault is open, the air gapped vault is physically detached, and unconnected, from such exterior devices. As such, when the air gap is open, the air gapped vault is not accessible by any form of wireless, or hardwire, including optical, communication system or device. Conversely, when the air gap is closed, the vault, and some or all of its contents, may be accessible by external systems and devices, such as an application host for example, whether by wireless and/or hardwired communication channels. In the example ofFIG.1, the data106and entities104may be communicated to the vault108only when an air gap protecting the vault108is closed.

Thus, with reference to the example ofFIG.1, the vaults108and112and part, or all, of their respective contents, may be isolated from other elements of the production site102and the remote site110, and from external entities, when the respective air gaps protecting those vaults108and112are open. When the air gaps are closed, the vaults108and112may be able to communicate with the production site102and the remote site110to transfer data, information, and metadata, for example, in either or both directions, between the production site102and the remote site110.

With continued reference toFIG.1, a remote site110may also be provided with which the production, or local, site102may communicate. The copies of the data106and entities104that are stored in the vault108may be communicated109by the production site102to the remote site110, and particularly, to the vault112. In an embodiment, the vault112may be a Dell PowerProtect Cyber Recovery system (CR) vault that can be isolated by way of another air gap.

When the local, or production, site102is down for some reason, such as due to inspection and evaluation by a law enforcement agency or insurance company for example, the entities104and data106in the vault112may be restored into a temporary production site116running on the remote site110. In this way, production operations may continue/resume even after the production site102is shut down. Because the temporary production site116has a relatively recent copy of the data106and entities104, the temporary production site116may only need to store data differentials resulting from operations that have taken place at the temporary production site116after the production site102has been shut down. As such, when the production site102becomes available again, the production site102may be restored relatively quickly using either local or remote copies of the entities104, and by restoring111the data differentials to the production site102from the remote site110.

B. Further Aspects of Some Example Embodiments

With continued reference to the illustrative example ofFIG.1, further details are provided now concerning aspects of some example embodiments. These details are provided by way of illustration and are not intended to limit the scope of the invention in any way.

Initially, it is noted that where the production site102is predominantly virtual, such as by virtue of the use of VMs, containers, or Kubernetes-based elements, one embodiment may employ cloud or colocation site infrastructure, such as the temporary production site116for example, as a temporary location, to which to restore working copies of virtual devices, and data, that may be used for failover use cases. The temporary production site116may also be used, on connection with a migration platform such as vMotion to handoff from the temporary production site116to the production site102once the production site102is ready to be restored.

In one embodiment, respective EMC DataDomain/protection appliances may be employed in either standard form factor or (CR)/Vault form factor on the local site102and the remote site110, which may also be referred to as a ‘target’ site. In an embodiment, the vault108may be omitted, and restore operations performed from the vault112. Where both vaults108and112are employed, a data replication link may be created between the vault108and vault112.

Once the data is at the remote site110, such as in the vault112or a data protection appliance hosted at the remote site110, when a recovery event occurs, virtual entities can be restored to the remote site112from the vault112or data protection appliance. One embodiment may assume that the virtualization technology at the local site102and in the remote site110is the same, but even if not, conversion functionality may be provided by either cloud or other vendors, such as DellEMC for example. The restored entities104and data106can be now used as the temporary production site116, which may also be referred to as a failover site, and run production workloads while the production site102is down, or otherwise inaccessible. As noted earlier, the production site102may be shut down to prevent the spread of any malware, and in order to be examined by an auditor, forensic analyst, insurance company, or law enforcement, for example.

When the production site102has been cleansed and is ready to be restored, the data106and/or entities104at the remote site110may be synchronized back to the local site102using a process that may be referred to synchronization, restore, or failback. This process may be performed in various ways.

For example, a migration platform such as vMotion may be used to migrate the entities104, which may be virtual, to the local site102from the remote site110, if the virtualization stacks at the production site102and the remote site110match. This move may be seamless, but it may take significant time as the data may be in its raw/open form and there is no benefit of dedup or compression, although data compression over the wire may be possible as the data is transferred from the remote site110to the production site102.

As another example of a failback process for synchronizing the data106and/or entities104at the remote site110back to the local site102, the entities104and data106in the remote site110may be backed up to the vault112. As entities104and data106originated from the vault112, it may be adequate to capture just the diffs, that is, any changes or ‘differentials’ occurring to the data106and/or entities104, that accumulated over the failover period when the temporary production site116was running.

In an embodiment, the differentials may be deduplicated before being sent from the remote site110to the production site102, so the restored production site102will now include all data changes that were incurred during the failover period. This approach provides an efficient way to transfer data changes to the production site102but may require performance of a data restore process at the production site102. Note that the provisioning of entities104, such as virtual entities, may be performed before those entities104are restored to the production site102from the remote site110. Similarly, a basic baseline restore may be performed at the production site102before the differentials are transferred. In this way, an embodiment may, as a final operation, apply the transferred differentials in order to reduce the overall time needed to restore the production site102, that is, to minimize the time between the time when the temporary production site116is brought up and the time when the production site102is active again.

C. Further Discussion

As will be apparent from this disclosure, example embodiments may provide various useful aspects and features. For example, an embodiment may use a public cloud comprising a time-limited vault, that is, a vault that may be used only for a limited amount of time, to free up infrastructure to set up a temporary production site while the main production site is down. Advantageously, this approach may shorten the downtime of the production site both during production site analysis, and failback to the production site from the remote site.

D. Example Methods

Directing attention now toFIG.2, an example method according to an embodiment is denoted generally at200. In an embodiment, the method200may be performed cooperatively by a local, or production, site, and a remote site.

The example method200may begin when assets, such as entities and data, of a production site are backed up202to a remote site. In an embodiment, the assets may be transmitted from a vault or appliance at the production site to a vault or appliance at the remote site. This vault-to-vault communication, possibly effected by way of a dedicated communication line between the vaults, which may each be air gapped, may provide a relatively high level of security. Alternatively, the production site may retrieve the assets from a local vault, and then transmit the assets to the remote site, which may then receive and store the assets in the remote vault.

Once copies of the assets are received204and stored in the remote vault, those assets may then be used to restore204, at the remote site, a temporary production site. That is, the assets may be restored204from the vault at the remote site into a temporary production site at the remote site. At some point after the asset backup202has been transmitted to the remote site, the production site may be shut down203, or otherwise made inaccessible. As disclosed herein, the shutdown203may enable, for example, forensic analysis of systems and events at the production site.

After the assets have been restored206, the temporary production site may run208at the remote site. The running of the temporary production site208may comprise performing any, or all, operations that otherwise would have been performed at the production site. Such operations may include any of the operations disclosed herein that involve data in some way.

At some point after the temporary production site has come online, the assets of the temporary production site, which may include data differentials, may be restored209to the production site. In an embodiment, those assets may first be backed up by the temporary production site to the vault at the remote site, and then restored209from the vault at the remote site to the production site, and in some particular embodiments, to the vault at the production site. The restored assets may comprise VMs, containers, data, and any other information or entities needed to make the production site fully operational.

In an embodiment, only part of the assets, such as VMs for example, that may be needed to restore the functionality of the production site are restored from the remote site. For example, if only data differentials are restored to the production site, the base data to which those data differentials relate may be restored to the production site from a vault at the production site.

After some, or all, of the assets have been restored209to the production site, the production site may be brought back up210so that it can be restored. In general, the production site may be made available for restoration whenever there is no longer a need for it to be isolated and/or disabled, such as after the production site has been cleansed of malware, and/or an auditing body has completed an audit or forensic analysis of the production site.

With some, or all, of the assets restored209to the production site, the production site may then be brought back up210. As noted above, a local restore212may be performed at the production site to restore assets, such as data for example, that may be needed to restore the functionality of the production site. The data may be restored, for example, from a vault at the production site. In some embodiments, the local restore212may be omitted.

After the production site has been brought back up210and restored212, the production site may then be in an online214status and ready to resume normal operations. Note that at some point after the assets have been restored209from the remote site to the production site, the temporary production site may be shut down215.

E. Further Example Embodiments

Embodiment 1. A method, comprising: receiving, at a remote site from a production site, copies of production site assets; storing, at the remote site, the copies of the production site assets; using, at the remote site, the copies of the production site assets to restore a temporary production site; running the temporary production site at the remote site; and restoring, from the remote site to the production site, the copies of the production site assets.

Embodiment 2. The method as recited in embodiment 1, wherein the copies of the production site assets received by the remote site are stored in a vault at the remote site.

Embodiment 3. The method as recited in any of embodiments 1-2, wherein the copies of the production site assets are received by the remote site from a vault at the production site.

Embodiment 4. The method as recited in any of any of embodiments 1-3, wherein the copies of the production site assets comprise any one or more of: data; virtual machines; and, containers.

Embodiment 5. The method as recited in any of embodiments 1-4, wherein restoring the temporary production site comprises restoring the copies of the production site assets from a vault at the remote site.

Embodiment 6. The method as recited in any of embodiments 1-5, wherein when the temporary production site is running, the temporary production site receives new and/or modified data from an entity.

Embodiment 7. The method as recited in any of embodiments 1-6, wherein the temporary production site runs while the production site is down.

Embodiment 8. The method as recited in any of embodiments 1-7, wherein the restoring, from the remote site to the production site, the copies of the production site assets, is performed after a time period during which the production site was down.

Embodiment 9. The method as recited in any of embodiments 1-8, wherein after the restoring, from the remote site to the production site, the copies of the production site assets, the temporary production site is shut down.

Embodiment 10. The method as recited in any of embodiments 1-9, wherein as part of the restoring of the copies of the production site assets, the remote site transmits data differentials to the production site.

Embodiment 11. A system, comprising hardware and/or software, operable to perform any of the operations, methods, or processes, or any portion of any of these, disclosed herein.

F. Example Computing Devices and Associated Media

In the example ofFIG.3, the physical computing device300includes a memory302which may include one, some, or all, of random access memory (RAM), non-volatile memory (NVM)304such as NVRAM for example, read-only memory (ROM), and persistent memory, one or more hardware processors306, non-transitory storage media308, UI (user interface) device310, and data storage312. One or more of the memory components302of the physical computing device300may take the form of solid state device (SSD) storage. As well, one or more applications314may be provided that comprise instructions executable by one or more hardware processors306to perform any of the operations, or portions thereof, disclosed herein.