External storage of unstructured database objects

Systems and methods to migrate unstructured objects such as, but not limited to, attachments and platform change documents from a database to a cloud-provided external object storage. The unstructured objects may be designated for migration based on their creation date and/or other characteristics. Migration of an object may include conversion of the object to a serialized file in an object notation format. The database may maintain a header of the migrated object indicating the location to which the object was migrated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Indian Provisional Application No. 202011018394, filed Apr. 29, 2020, the contents of which are incorporated by reference herein for all purposes.

BACKGROUND

Modern database systems store vast amounts of data for their respective enterprises. The data may include transactional data, master data, unstructured data and other data. Some database systems are capable of storing data of an entire database in volatile memory, thereby facilitating fast read and write access to such “in-memory” data.

A significant portion of data stored in a database may include unstructured data. Unstructured data may comprise large attachments (images, scans, CAD drawings, movies, etc.) which cannot be queried at the attribute level and therefore do not benefit from in-memory storage. Unstructured data may also comprise change documents which record the changes made to object instances during create, modify, and delete operations. These change documents may be used to satisfy compliance requirements, and typically do not contribute to the primary functions for which the database system is employed.

Storage of unstructured data increases the Total Cost of Ownership (TCO) of a database system, particularly in the case of in-memory database systems which use expensive volatile memory, without providing commensurate benefits. Systems are needed to efficiently reduce the amount of operational memory used to store unstructured data, while still providing efficient access to the unstructured data.

DETAILED DESCRIPTION

The following description is provided to enable any person in the art to make and use the described embodiments and sets forth the best mode contemplated for carrying out some embodiments. Various modifications, however, will be readily-apparent to those in the art.

Some embodiments operate to migrate unstructured objects such as, but not limited to, attachments and platform change documents from a database to a cloud-provided external object storage. The unstructured objects may be designated for migration based on their creation date and/or other characteristics. Migration of an object may include conversion of the object to a serialized file in an object notation format. The database may maintain a header of the migrated object indicating the location to which the object was migrated.

Embodiments may therefore reduce a TCO of a database system by reducing an amount of memory required for data storage. Reducing the amount of occupied memory may in turn reduce downtime required for tenant lifecycle operations (e.g., move, copy, refresh, restore).

FIG. 1is a block diagram of system100according to some embodiments. The illustrated elements of system100may be implemented using any suitable combination of computing hardware and/or software that is or becomes known. In some embodiments, two or more elements of system100are implemented by a single computing device. One or more elements of system100may be implemented as a cloud service (e.g., Software-as-a-Service, Platform-as-a-Service).

Generally, platform110receives queries from client applications120and returns results thereto based on data stored within database server111. Database server111may comprise any combination of volatile memory (e.g., Random Access Memory (RAM)) and non-volatile memory (e.g., fixed disk, Flash memory, non-volatile (RAM)).

Platform110executes program code to provide application server112. Application server112provides services for executing server applications such as application113. For example, a Web application executing on application server112may receive HyperText Transfer Protocol (HTTP) requests from client applications120and return data114from storage111in response thereto.

Data114of database server111may comprise one or more of conventional tabular data, row-based data, column-based data, and object-based data. Metadata115describes the structure and characteristics of data114as is known in the art. Moreover, data114may be indexed and/or selectively replicated in an index to allow fast searching and retrieval thereof. Database server111may support multi-tenancy to separately support multiple unrelated clients by providing multiple logical database systems which are programmatically isolated from one another.

As mentioned above, database server111may implement an “in-memory” database, in which a full database stored in volatile (e.g., non-disk-based) memory. The full database may be persisted in and/or backed up to fixed disks (not shown). Embodiments are not limited to an in-memory implementation. For example, data may be stored in RAM (e.g., cache memory for storing recently-used data) and one or more fixed disks (e.g., persistent memory for storing their respective portions of the full database).

According to some embodiments, external migration agent116may identify objects of data114to be migrated to an external storage (e.g., based on a predetermined retention period), retrieve and convert the item data of the identified objects into JSON (JavaScript Object Notation) format, and push the converted objects to consumer proxy117. As will be described below, the header data of the identified objects may be maintained in data114, and may be updated to indicate that the identified objects are stored in the external storage. External migration agent116may also be responsible for controlling the reading of externally-stored objects, and the movement, replication and deletion of externally-stored objects based on corresponding changes to the objects in database server111.

Consumer proxy117may provide authentication and establish an HTTP connection with bridge application130. The functionalities provided by consumer proxy117may be reused by other agents, services or applications executing within platform110.

Bridge application130receives converted object data (e.g., JSON files) from platform110and pushes the data to external object storage140. External object storage140may comprise a scalable, flexible, and cost-effective cloud storage solution that allows storage of large amounts of unstructured data as ‘Objects’. Bridge application130may, based on information received from external migration agent116via consumer proxy117, read externally-stored objects and move/replicate/delete externally-stored objects based on corresponding changes to the objects in database server111.

FIG. 2comprises a flow diagram of process200according to some embodiments. Process200will be described with respect to the elements of system100, but embodiments are not limited thereto.

Process200and all other processes mentioned herein may be embodied in computer-executable program code read from one or more of non-transitory computer-readable media, such as a hard disk drive, a volatile or non-volatile random access memory, a DVD-ROM, a Flash drive, and a magnetic tape, and then stored in a compressed, uncompiled and/or encrypted format. In some embodiments, hard-wired circuitry may be used in place of, or in combination with, program code for implementation of processes according to some embodiments. Embodiments are therefore not limited to any specific combination of hardware and software.

Initially, at S210, an unstructured object is identified for migration from database storage to external storage. Such identification may be performed by agent116, which runs as a background job to determine objects of one or more specified types (e.g., change documents, video attachments) having a creation date earlier than a threshold creation date. For example, S210may comprise identifying any unstructured data objects stored in data114which were created over one month from the present time. S210may therefore comprise identifying more than one unstructured object for migration.

Next, at S220, the unstructured object is converted to an object notation format. As mentioned above, external migration agent116may convert the item data of the identified object into JSON format at S220. The converted object is transmitted to a cloud application for storage in an external storage at S230.

According to some embodiments of S230, external migration agent116pushes the converted object to consumer proxy117, which in turn authenticates with the cloud application (i.e., bridge application130) and provides converted object to bridge application130via a multipart HTTP request. In response, bridge application130stores the converted object in external object storage140.

At S240, it is determined whether the storage was successful. According to some embodiments, and prior to S230, consumer proxy117calculates a digest (e.g., crypto-code) based on the content of the converted object. The digest is provided along with the object content in the HTTP request at S230. Upon receipt, bridge application130calculates the digest based on the object content and compares the received digest to the calculated digest. If the digests match, the storage is considered successful.

If the storage is not successful, an error is returned at S250. If so, the item data of the object is deleted from data114at S260. Next, at S270, the header data of the unstructured object is updated in data114to indicate that the item data of the unstructured object is stored in external storage140. Such an indication may be used as described below for reading the item data from external storage140.

FIG. 3is a block diagram of system300to provide migration of unstructured data objects according to some embodiments. System300may comprise a specific implementation of system100ofFIG. 1, but embodiments are not limited thereto. The elements of system300may operate as described with respect to similarly-named elements of system100, but with the additional or alternative features described below.

Database server310includes authentication token store318accessed by consumer proxy317. During the authentication with bridge application333as described above, consumer proxy317requests an authentication token from authentication application331using corresponding client credentials. Authentication application331performs the authentication and returns a valid token back to consumer proxy317. This authentication token is stored in token store318and may be used by consumer proxy317as a header parameter for authentication in subsequent requests to bridge application333. Authentication application331also stores the token in cache332, which may store the token for a fixed time before the token becomes invalid.

Authentication application331and bridge application333execute within sub-account330of cloud foundry340running on cloud platform350. According to some embodiments, cloud foundry340is an open source software bundle for providing a polyglot Cloud Computing Platform as a Service (PaaS), which may run on different IaaS offerings (e.g., Openstack, Amazon Elastic Cloud 2 (EC2), SAP Monsoon), enable application development on different runtimes (e.g., Node.js, Java, Ruby, .NET), and allow integration of arbitrary platform services (e.g., mongoDB, RabbitMQ) and applications services (e.g., mail, document). According to some embodiments, authentication application331and bridge application333are Node JS applications.

Bridge application333receives a JSON file from consumer proxy317within a multipart HTTP request as described above. The multipart HTTP request includes the authentication token received from authentication application331. Bridge application333compares this token with a token stored in cache332by authentication application331. Upon successful verification, bridge application333sends the JSON file to object store335via the object store library334, after which the JSON file is stored in a selected one of external object storages360-364.

Process400may be executed to migrate an unstructured object to external storage. Initially, at S405, an unstructured object is identified for migration from database storage to external storage. Such identification may be performed by agent316, and may be based on creation date as described above, on a blacklist of objects which are not to be stored externally, on data indicating historical object usage, and/or on any other suitable parameter(s).

Next, at S410, external migration agent316may convert the item data of the identified object into JSON format.FIG. 5illustrates serialization pattern500which may be used in some embodiments to perform the conversion at S410.

Consumer proxy317requests an authentication token at S415from authentication application331executing on cloud platform350using corresponding client credentials. Authentication application331performs the authentication and returns a valid token back to consumer proxy317. This authentication token is stored in token store318and is used within a multipart HTTP request to transmit the JSON file from consumer proxy317to bridge application333at S420.

Upon successful verification of the token against a token stored in cache332, bridge application333sends the JSON file to object store335via the object store library334, after which the JSON file is stored in a selected one of external object storages360-364.

If it is determined at S430that the storage was not successful, an error is returned at S435. If the storage was successful, the item data of the object is deleted from data314at S440. Next, at S445, the header data of the unstructured object is updated in data314to indicate that the item data of the unstructured object is stored in the corresponding one of external storages360-364.

FIG. 6illustrates process600to read an object from external storage according to some embodiments. Flow begins at S610, at which an object stored in an external storage is determined. S610may comprise reception of a command from application313to read an object, such as a change document or an unstructured attachment, and determination, based on a stored header of the object, that the item data of the object is stored in an external cloud storage.

Next, at S620, authentication with a cloud platform is requested via a proxy class. In some embodiments, consumer proxy317requests an authentication token at S620from authentication application331executing on cloud platform350using corresponding client credentials. Authentication application331performs the authentication and returns a valid token back to consumer proxy317.

The object is requested from a cloud application executing on the cloud platform at S630. The request may comprise a multipart HTTP GET request from consumer proxy317to bridge application333which includes the token received from authentication application331. In response to the request, bridge application333fetches the corresponding JSON object from the external storage360-364in which it is stored. Bridge application333then returns the JSON object to consumer proxy317at S640.

Agent316receives the JSON object from consumer proxy317and converts (i.e., de-serializes) the object into its original unstructured database object at S650. The unstructured database object is then served to application313at S660, where it can be used in display screens, APIs or in any other suitable manner.

FIG. 7is a block diagram of database server700according to some embodiments. Database server700may comprise a general-purpose computing apparatus and may execute program code to perform any of the functions described herein. Database server700may comprise an implementation of database server100or300in some embodiments. Database server700may include other unshown elements according to some embodiments.

Database server700includes processing unit(s)710operatively coupled to communication device720, data storage device730, one or more input devices740, one or more output devices750and memory760. Communication device720may facilitate communication with external devices, such as an external network, the cloud, or a data storage device. Input device(s)740may comprise, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, knob or a switch, an infra-red (IR) port, a docking station, and/or a touch screen. Input device(s)740may be used, for example, to enter information into apparatus700. Output device(s)750may comprise, for example, a display (e.g., a display screen) a speaker, and/or a printer.

Data storage device730may comprise any appropriate persistent storage device, including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives and flash memory), optical storage devices, Read Only Memory (ROM) devices, and RAM devices, while memory760may comprise a RAM device.

Application server731, application732, agent733and consumer proxy734may each comprise program code executed by processing unit(s)710to cause server700to perform any one or more of the processes described herein. Embodiments are not limited to execution of these processes by a single computing device.

Database Management System (DBMS)735may comprise a DBMS as is known in the art, and metadata736and data737may comprise any suitable metadata and data as described herein. Data storage device730may also store data and other program code for providing additional functionality and/or which are necessary for operation of database server700, such as device drivers, operating system files, etc.