Data auditing for object storage public clouds

A method for data auditing for object storage public clouds includes a service broker receiving a request to store data in public object storage, where the request includes user information or a container image. The service broker, based on either the user information or the container image, determines that data auditing is necessary. The service broker creates a storage unit, in public object storage, and a storage proxy. The method further includes the storage proxy storing data, and a data auditor retrieving data from the storage proxy. The data auditor determines a data qualification for the data, and notifies the storage proxy of the data qualification.

BACKGROUND

Cloud computing defines a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Characteristics of the cloud computing model include on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service. The cloud computing model includes several service models, including Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). The cloud computing model may be implemented according to one of the several deployment models, including private cloud, community cloud, public cloud, and hybrid cloud.

Cloud infrastructure is a collection of hardware and software that implements the cloud computing model. Cloud infrastructure may be viewed as including a physical layer and an abstraction layer. The physical layer includes hardware resources designed to support the cloud services being provided, and typically includes a variety of server, storage, and network components. The abstraction layer may include the software deployed across the physical layer, which manifests the cloud characteristics. Conceptually, the abstraction layer resides above the physical layer. Cloud storage is a data storage service that provides storage to users in the form of a virtualized storage device over a network. Using cloud storage service, users may store, retrieve, maintain, and back up data remotely.

SUMMARY

The present disclosure provides new and innovative methods and systems for data auditing for object storage public clouds. An example method includes a service broker receiving a request to store data in public object storage, where the request includes user information or a container image. The service broker, based on either the user information or the container image, determines that data auditing is necessary. The service broker creates a storage unit, in public object storage, and a storage proxy. The method further includes the storage proxy storing data, and a data auditor retrieving data from the storage proxy. The data auditor determines a data qualification for the data, and notifies the storage proxy of the data qualification.

An example system includes one or more processors, a storage unit, a storage proxy, a data auditor, and a service broker. The service broker executes on the one or more processors, and is configured to receive a request to store data in public object storage, where the request includes user information or a container image. The service broker determines that data auditing is necessary based on the user information or the container image and creates a storage unit, in public object storage, and a storage proxy. The storage proxy stores the data. The data auditor retrieves the data from the storage proxy, determines a data qualification for the data, and notifies the storage proxy of the data qualification.

Additional features and advantages of the disclosed methods and system are described in, and will be apparent from, the following Detailed Description and the Figures.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Cloud storage can be either public or private. Private cloud storage is more secure, as it is usually located on site, but more expensive as it requires the enterprise to provide the data center space, network connectivity, power, and cooling capabilities. Maintaining a private infrastructure can be costly and impractical for growing companies. Alternatively, public cloud storage is designed for multi-tenancy and enables enterprises to store data in the cloud and back up their data without having to maintain a storage infrastructure.

Data can be stored onto the cloud in various formats. Block storage is a type of data storage where files are split into evenly sized blocks of data, each block being stored at its own address. However, block storage generally does not include additional information (e.g., metadata) to provide more context for what that block of data is. Generally, block storage is not ideal for data growth. For example, as more data is generated, storage systems have to grow at about the same pace. When an enterprise attempts to expand block-based storage systems by a significant amount, an enterprise may run into durability issues, limitations due to infrastructure, or an increase in management overhead.

Alternatively, object storage is generally a type of cloud based data storage where data is stored in objects, and from the user's perspective, the files are not split up into different blocks of data. Generally, objects are stored in a flat structure, where objects are not organized in a hierarchy and one object is not stored in another object. Typically, each object includes the data to be stored in the object, various metadata, and a globally unique identifier within the object storage system. In some examples, object storage is cheaper than other types of cloud storage, and allows for the retention of massive amounts of unstructured data. Generally, object storage also provides greater possibilities for data analytics, and the ability to store an object anywhere within a distributed data pool. In an example, object storage's flat address space allows it to be easily scaled by adding more storage to the pool.

Generally, object storage is slower than other types of cloud storage because object storage may be rate limited, meaning, for example, it may only be able to process a certain quantity of data at a time. For example, object storage may receive 10,000 requests per second, but may only be able to process 100 requests per second.

Generally, a service broker executes as part of a cloud computing system to send and receive messages, and queue those messages, which helps developers build applications in which independent components work together to accomplish a task. A service broker monitors the completion of tasks, between two different applications, and is responsible for the safe delivery of messages. A service broker may store messages until they are ready for delivery. Generally, a proxy server is a server that acts as an intermediary between an endpoint device and another server from which a user or client is requesting a service. An advantage of a proxy server is that its cache can serve all users. For example, a proxy server may receive a request for a webpage, the proxy server looks in its local cache of previous pages. If the proxy server find the page, it returns it to the user without needing to forward the request to the internet. If the page is not in the cache, the proxy server, uses one of its own IP addresses to request the page from the server out on the internet. When the page is returned, the proxy server relates it to the original request and forwards the page on to the user.

Recently the security of data stored in public cloud storage has drawn public attention. In an example, recent stories have discussed a number of instances where private information placed in public object storage has been wiped, lost, corrupted, or stolen. Typically data stored in public object storage follows the eventual consistency model. Eventual consistency is a consistency model used by public object storage providers to provide high availability of data, where replicas are guaranteed to converge only when the system has been quiesced for a period of time. In eventual consistency, eventually all access to that item will return to the last updated value. However, generally, the eventual consistency model is not reliable for monitoring for data breach. This is because when one server is updated in the eventual consistency model, the updates are not visible on the other servers right away. For example, if data is stored in public object storage in the US by a customer utilizing a US server in a cloud network, a coworker of the US customer based in Tokyo using a Tokyo server in the cloud network monitoring for data breaches will not see the data update on the US server for an extended amount of time until finally the Tokyo server is updated to match the US server. The eventual consistency model makes monitoring data stored on the cloud an unmanageable solution as servers are not updated in a timely manner. Therefore, in an example, if private or sensitive information was put onto the public cloud utilizing the US server, the coworker in Tokyo would find it nearly impossible to notice and catch this data security issue. Due to the rate limits of public object storage, it would also be highly difficult to sort and filter to find private or improper data once it is already stored in public object storage. Further, if the information is caught and deleted from one server, the remaining servers will not be updated right away due to the eventual consistency model, leaving sensitive information exposed even after deletion. Accordingly, a public storage system that prevents private information from being placed onto public object storage is necessary, as it is nearly impossible to discover and remove the data once already loaded onto public object storage. Aspects of the present disclosure address the above noted deficiency by employing data auditing for public object storage clouds.

A service broker reads a request for data to be stored in a storage unit in public object storage and determine whether the data should be audited before being stored in the storage unit. If the service broker determines data should be audited, the service broker provisions, or creates, a storage proxy for temporarily storing the data, as well as a storage unit in public object storage. The service broker directs the data to be sent to the storage proxy. The data is retrieved by a data auditor, which audits the data and determines whether to provide a compliance indication if the data can be stored in the storage unit, or instead to flag the data if the data is not safe to be stored in the storage unit because it contains some type of private information. If the data receives a flag, the data is not stored in the storage unit and the appropriate parties will be notified. If the data is safe to store in the storage unit, the data will be stored in the storage unit located in public object storage. For example, a service broker may determine data containing credit card information should be audited and sends that data to a storage proxy. After an indication, the data is retrieved from the storage proxy by the data auditor, and the data auditor determines whether that information should be stored on the public cloud. If the information contains pin numbers and credit card numbers, the data auditor may flag the information and not store the information in the storage unit.

The systems and methods disclosed herein are inextricably tied to and provide an improvement to computer technology. For example, by adding an additional layer of constraint during the process of placing data onto the public cloud (e.g., determining by a service broker whether data should be audited, and auditing the data by the data auditor to determine if the data is safe to be stored in public object storage), the security of private information is increased before data is stored onto public object storage, reducing the need to continuously monitor rate limited public object storage, that is also unreliable due to the eventual consistency model. The solution described herein is necessarily rooted in computer technology to overcome a problem specifically arising in the realm of computer networks, and more specifically to public object storage rate limits in the field of cloud computing, by addressing the challenge of preventing and filtering the upload of data containing sensitive information in a cloud environment. For example, as described in greater detail below, the service broker filters requests requiring auditing, and dictates to the storage proxy which data should be temporarily stored for auditing by the data auditor. The data auditor designates the data with a data qualification and dictates the storage proxy to respond according to the data qualification. The disclosed system and methods improve computer hardware on which cloud environments exist to allow for the disclosed performance that did not previously exist.

The system100may include public object storage114. The public object storage may include a storage unit116. Further, the system100may include a container cluster102, a container104, and a DNS server106. The container104may include a data auditor134. The container cluster102may include containers120,122,124, and126. The container122may include an application128. The container124may include a storage proxy130. The container126may include a service broker132. In an example, the DNS server106, the container cluster102, and the container104are all located and running on different nodes.

In an example, the service broker132may receive a request from an application128. The service broker132, based on the request received, will determine whether the data associated with that request should be audited or reviewed. If the service broker132determines the data should be audited, the service broker132provisions or creates the storage unit116and the storage proxy130for the data. The data is then stored in the storage proxy130. In an example, each time the service broker132receives a request from the application128, the service broker132will provision or create a new storage proxy and storage unit, such as storage proxy130and storage unit116. Then, the data is retrieved from the storage proxy130by the data auditor134, and given a data qualification. If the data qualification is a compliance indication or an indication that the data is safe to store in the storage unit116, the data is stored in the storage unit116, located in public object storage114. If the data qualification is a flag, the data is not stored in storage unit116.

As discussed herein, a memory device refers to a volatile or non-volatile memory device, such as RAM, ROM, EEPROM, or any other device capable of storing data. As used herein, physical processor or processor refers to a device capable of executing instructions encoding arithmetic, logical, and/or I/O operations. In one illustrative example, a processor may follow Von Neumann architectural model and may include an arithmetic logic unit (ALU), a control unit, and a plurality of registers. In a further aspect, a processor may be a single core processor which is typically capable of executing one instruction at a time (or process a single pipeline of instructions), or a multi-core processor which may simultaneously execute multiple instructions. In another aspect, a processor may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module (e.g., in which individual microprocessor dies are included in a single integrated circuit package and hence share a single socket). A processor may also be referred to as a central processing unit (CPU). Processors may be interconnected using a variety of techniques, ranging from a point-to-point processor interconnect, to a system area network, such as an Ethernet-based network. In an example, the one or more physical processors may be in the system100. In an example, all of the disclosed methods and procedures described herein can be implemented by the one or more processors. Further, the system100may be distributed over multiple processors, memories, and networks.

Further, system100may also include an input/output devices (e.g., a network device, a network interface controller (NIC), a network adapter, any other component that connects a computer to a computer network, a peripheral component interconnect (PCI) device, storage devices, sound or video adaptors, photo/video cameras, printer devices, keyboards, displays, etc.), for example the I/O142,152, and162, may be coupled to a processor.

FIG. 2illustrates a block diagram of an example public storage system. The system200may include public object storage208. Public object storage208includes multiple storage units. For example, public object storage208may include a storage unit222and a storage unit224. A storage unit can be, for example, a unit of storage similar to an Amazon Web Services S3™ bucket, a Google Cloud™ bucket, or a blob in Azure Blob™ storage by Microsoft®. A storage unit, for example, may be used to group data into one unit.

The system200further includes an application202that sends a request204to a service broker206for data to be stored in public object storage208by provisioning a storage unit224for the data. The request204can be sent by a user of the application202, or can be automatically generated by the application202. The request204may include information about the user or administrator of the application202, or it may include a container image of the application202, as an application may typically run in a container. The request is received by the service broker206. The service broker206will read the request204including either the user information or the container image and may determine that data auditing is necessary, as there may be sensitive, confidential, or private information located in the data. In this circumstance, the service broker206will provision or create a storage unit224in public object storage208for this data for a later time. Further, service broker206will create a storage proxy for this data, such as storage proxies212or214, for the data to be stored in temporarily. In this example, the service broker206will provision storage proxy214to store the data.

In this example, once the service broker206identifies which storage proxy should be provisioned for the request204, the service broker206dictates to a DNS server210to change the endpoint for the data to be storage proxy214, and not storage unit224in public object storage208. In an example, the DNS server210may already have the IP addresses corresponding to the endpoints to be changed. However, in an alternative example, the DNS server210may not have the IP addresses corresponding to the endpoints to be changed, and therefore, the DNS server210will request from application202the correct IP addresses corresponding to the endpoints to be changed. Once the DNS server210updates the endpoints for the data to be storage proxy214, the service broker206will direct the application202to send the data to the storage proxy214, and will also direct the storage proxy214to store the data.

In this example, once the data is stored in the storage proxy214, a data auditor, such as data auditors216,218, or220will retrieve the data. The storage proxy214will identify which data auditor the data should be sent to because the data will be labeled by an administrator with at least one workload type. A workload type corresponds to a category of information within the data/container image. Based on the workload type identified by the storage proxy214, the data will be directed to be sent to either data auditor216,218, or220. For example, if the data contained patient information from a doctor's office, the data would be pre-tagged with a workload type indicating medical information. The storage proxy214would read this information, and determine the information should be handled by data auditor218since data auditor218was created to deal with healthcare or medical files, while data auditor220is used for financial information, and data auditor216is used for personal home addresses.

In example system200, the data auditor218will then audit the data and provide the data with a data qualification, such as a compliance indication or a flag. In this example, the data audited receives a compliance indication, and therefore the data auditor218indicates to the storage proxy214that the data has received a compliance indication and may be stored in the provisioned/created storage unit224. The storage proxy214will store the data into the storage unit224in public object storage208.

In an alternative example, the service broker206may read the workload types of the data or the request204, and direct the appropriate storage proxy to provide the data to the appropriate data auditor216,218,220. In an alternative example, the service broker206may directly instruct the appropriate data auditor to retrieve the data from the appropriate storage proxy212or214.

In an alternative example, the storage proxy212or214may direct more than one data auditors212or214to retrieve the data in order to have more than one data auditors212or214audit the data. For example, the workload type could indicate to the storage proxy214that the data to be audited is medical billing data. Therefore, the storage proxy214may dictate that the data must be audited in both data auditor218, relating to medical information, as well as data auditor220, relating to financial information.

FIG. 3is a flowchart illustrating an example method for data auditing for object storage public clouds. System300depicts a flowchart illustrating an example method for data auditing for object storage public clouds. Although the example method300is described with reference to the flowchart illustrated inFIG. 3, it will be appreciated that many other methods of performing the acts associated with the method may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional.

The method300begins by a service broker receiving a request to store data (block302). For example, an application128may send a request to a service broker132. In an example, the request may be a request to provision or create a storage unit116for data. In an alternative example, the request may be a request to store data in the storage unit116. In an example, the request may be sent automatically by the application128, or the request could be sent to the service broker132by a user of application128or an administrator. The request may include a container image of container122, or user/administrator information of the user/administrator making the request.

Next, the service broker determines that data auditing is necessary (block304). For example, after receiving the request from application128, the service broker132reads the container image or the user/administrator information and determines whether or not the data to be stored is likely to include sensitive information. For example, if the request includes information about the user of application128, and the user is a hospital employee, the service broker132determines that the request may include sensitive information, such as medical records or patient information. Therefore, the service broker132determines data auditing is necessary.

Next, the service broker creates a storage unit and a storage proxy (block306). For example, the service broker132provisions or creates a location for the data to be stored temporarily (storage proxy130), and provisions some space and creates a storage unit116in public object storage114. Then, the storage proxy stores the data (block308). For example, the data to be audited is sent from application128to storage proxy130, and storage proxy130stores the data.

Next, the data auditor notifies the storage proxy of the qualification for the data (block314). For example, the data auditor134communicates to the storage proxy130the data qualification (flag or compliance indication) that was assigned to the data. The storage proxy130handles the data based on the data qualification provided by the data auditor134. For example, if the data qualification is a flag, the storage proxy130handles the data appropriately to ensure that the data does not get stored in the storage unit116.

FIGS. 4A to 4Billustrate a flowchart of an example method400for data auditing for object storage public clouds. Although the example method400is described with reference to the flowchart illustrated inFIGS. 4A to 4B, it will be appreciated that many other methods of performing the acts associated with the method may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The method400may be performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both. For example, the method400may be performed by a system including an application402, a service broker404, a storage proxy406, a data auditor408, a storage unit410, and a DNS server412.

In the illustrated example, an application402sends a request, as well as a container image, to the service broker404(block420). The service broker404receives the request and the container image (block422), reads the request and container image (block424), and determines that data auditing is necessary (block426). In an alternative example, the service broker404could determine that data auditing is necessary based on an administrator labeling the request or the container image with a type of data, or by reading the type of data in the container image and determining that auditing is necessary.

The service broker404then provisions or creates both a storage unit410and a storage proxy406for the data (block428). At this time, the service broker404informs the DNS server412to retrieve the IP address from the application402associated with the data in order to update the endpoint to be the storage proxy406rather than the storage unit410(block430). As an alternative example, the DNS server could already have the IP addresses associated with the data stored, and therefore can automatically update the endpoint to the storage proxy406without asking the application402for the IP address.

Next, the service broker404directs the application402to send data to the storage proxy (block432), and the application402sends the data to the storage proxy406(block436). Either simultaneously or sequentially, the service broker404directs the storage proxy406to store the data (block434), and the storage proxy406receives and stores the data (block438).

Once the storage proxy406has received the data, the storage proxy406sends an event indication to the data auditor408indicating that new data has been stored in the storage proxy406(block440), and data auditor408receives that event indication (block442), looks into the storage proxy406for the new data (block444), and retrieves the data (block446). Alternatively, in another example, the data auditor408may periodically conduct a full scale scan of storage proxy406in order to determine if any new data has been stored in the storage proxy406, and if new data has been stored in the storage proxy406, then the data auditor408retrieves the new data.

Next, the data auditor408converts the file type (block448). For example, if the data retrieved by the data auditor408is in the form of a PDF, and the data auditor408is only able to read the data in txt format, the data auditor408converts the data file type to txt format before it reads the data.

Next, the data auditor408determines the qualification for the data as a compliance indication based on rules (block450). In an example, these rules may be preconfigured by an administrator. Next, the data auditor sends the compliance indication as the data qualification to the storage proxy406(block452), and the storage proxy406reads this compliance indication from the data auditor (block454), and sends the data to be stored in the storage unit410(block456). The data is then stored in public object storage in storage unit410(block458).

Alternatively, for example, if data auditor408determines that the data qualification is a flag, the data auditor sends the flag indication to the storage proxy406, and storage proxy406sends an alert notification to either the user of the application402, the application402, or an administrator. Further, the storage proxy406will not send the data to be stored in storage unit410. The above method is advantageous in that sensitive information is filtered out from being transmitted into or stored in public object storage. This includes an additional safeguard to customers that sensitive information will never be placed into the cloud, where it might be lost or stolen, and allows enterprises the ability to comply with strict privacy requirements regarding sensitive information.

FIG. 5is a block diagram of an example public storage system according to an example of the present disclosure. As illustrated inFIG. 5, an example system500may include a service broker516, a storage proxy508, a data auditor510, and processor518.

The example system further includes request502. The request502may include a container image512and user information514. The example system further includes public object storage504. Public object storage504may include a storage unit520.

In an example, the request502, including container image512and user information514, is sent to the service broker516. The request502may be a request to store data in public object storage504. In an example, the service broker516determines whether the data, based on the request502, needs data auditing. In an example, if the service broker516determines data auditing is necessary, the service broker516creates storage proxy508and storage unit520. The service broker516may direct the data to be stored in storage proxy508.

In an example, the data is retrieved by the data auditor510from the storage proxy508. The data auditor510determines a data qualification522for the data, and notifies the storage proxy508of the data qualification522to ensure that only the correct data is sent into the public object storage504, and data that should not be sent is prevented from being placed on the public object storage504where it would be at a higher security risk than is acceptable.

The examples may be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. An example may also be embodied in the form of a computer program code containing instructions embodied in tangible media, such as floppy diskettes, CD-ROMs, DVD-ROMs, hard drives, or any other computer readable non-transitory storage medium, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for carrying out the method. An example may also be embodied in the form of computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, where when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for carrying out the method. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits.