Distributed storage quota enforcement

Embodiments are directed to client-side enforcement of storage quotas in a cloud service or distributed computing environment. A storage driver is configured to monitor access to an external storage system by a cloud process and to enforce storage limits for the external storage system. The storage driver intercepts reads and writes from the process to the external storage system and updates a local state to track the process's usage of the external storage system. A storage quota database is periodically checked for an updated storage usage and, if the updated storage usage is not present, then the external storage is scanned to obtain the updated storage usage. The updated storage usage is provided to the storage driver, which overwriting the local state. The storage driver continues to update the local state based upon the intercepted reads and writes.

BACKGROUND

Applications may run in multiple servers in a cloud service and may requires access to a storage system external to the cloud service. Whenever the applications read, write, or delete content, the external storage system itself enforces quotas, such as the storage space available to the applications. This quota enforcement is provided at the server side and, therefore, the applications are unaware of the current status of the storage or the available space. This makes it difficult to scale out applications on the cloud service when the available storage in a remote system cannot be tracked by the cloud service. Specific interfaces to the external storage or knowledge of the external storage's operation is required to allow the cloud service to track available storage and to keep cloud applications within quotas.

Problems arise when cloud service tenants want to use existing external storage systems that do not provide functionality required by the cloud service for tracking quotas. For example, the external storage systems may provide limited quota enforcement, such as only at the root level and not per folder or per tenant.

SUMMARY

A distributed computing system for controlling access to an external storage comprises a quota database for storing usage information for the external storage, an activation service for configuring an application to run on a machine in the distributed computing system, the activation service configured to retrieve the usage information related to the application's access to the external storage, and a remote storage driver on the machine, the remote storage driver configured to provide access to the external storage and to receive the usage information from the activation service, wherein the remote storage driver enforces external storage quotas by restricting access if the application's usage exceeds a preset quota. The remote storage driver monitors a current state of the external storage by tracking use of the external storage by the application.

The activation service initiates a timer after retrieving the usage information and checks for changes to the usage information in the database when the timer expires. If the usage information in the database has changed, then the activation service provides updated usage information from the database to the remote storage driver. If the usage information in the database has not changed, then the activation service scans the external service for current usage information and provides updated usage information to the database and to the remote storage driver.

The remote storage driver updates a current state of the external storage using information from the activation service.

A computer-implemented method comprises reading a storage limit and a storage usage from a database in a cloud service, wherein the storage limit and storage usage are associated with a storage system external to the cloud service; creating a process on the cloud service, wherein the process is configured to access the external storage system; configuring a storage driver on the cloud service to monitor access to the external storage system by the process and to enforce storage limits for the external storage system; and checking the database for an updated storage usage, wherein if the updated storage usage is present, then providing the updated storage usage to the storage driver, and wherein if the updated storage usage is not present, then scanning the external storage to obtain the updated storage usage.

The method further comprises saving the updated storage usage obtained from the external storage scanning to the database.

The method further comprises initiating a timer to determine how often to check the database for the updated storage usage.

The method further comprises creating two or more additional processes on the cloud service, wherein the two or more additional processes are also configured to access the external storage system; configuring a separate storage driver to monitor access to the external storage system and to enforce storage limits on the external storage system for each of the two or more additional processes; and independently checking the database on behalf of each separate storage driver to detect the updated storage usage, wherein if the updated storage usage is present, then providing the updated storage usage to the storage driver for which the database check was performed.

The method further comprises initiating timers associated with each separate storage driver to determine how often to check the database for the updated storage usage.

The method further comprises intercepting, by the storage driver, reads and writes from the process to the external storage system; and updating a local state in the storage driver, based upon the intercepted reads and writes, to track the process's usage of the external storage system.

The method further comprises blocking, by the storage driver, the reads and writes from the process to the external storage system when the local state exceeds the storage limit.

The method further comprises overwriting the local state in the storage driver when the updated storage usage is present in the database.

The method further comprises saving the updated storage usage as determined by scanning the external storage to the database; and overwriting the local state in the storage driver using the updated storage usage as determined by scanning the external storage.

DETAILED DESCRIPTION

FIG. 1is a simplified block diagram of a distributed computing service or cloud service101that supports a plurality of tenant applications102for cloud service tenants103. Cloud service101may be a public cloud, such as the Azure® cloud service from Microsoft Corporation, that offers distributed computing services to any user. Alternatively, cloud service101may be a private or on-premises cloud service that supports a specific enterprise. The tenant applications102run on worker entities104, which may be, for example, virtual machines (VM), servers, or other processor-based devices. Cloud service101provides the capability to scale-up or scale-down the number of workers104supporting a cloud tenant's application based upon user demand. Cloud service101also provides failover support to keep tenant applications102running when a server or VM fails.

Cloud service tenants103access the cloud service through a tenant portal105, which allows the tenants103to designate the type and number of tenant applications102that should be running. The tenant applications may be, for example, a web process that supports a tenant website. Users may access the website through a designated service endpoint106. A tenant may run multiple tenant applications102across multiple workers104to support the same website, such during times of high user demand.

The tenant applications may use a cloud-based file storage entity107, such as to store tenant and user data. Alternatively, or in addition to storage107, tenant applications102may use external file storage entities108,109. The external storage108or109may be hosted by the tenant or by a third-party service other than the service provider that offers cloud service101.

Cloud service101also enforces quotas and limits for each tenant. Each tenant103enrolls in a selected cloud service subscription, which limits the availability of cloud service resources, such as the number worker processes the tenant may access to run tenant applications102and the amount of storage space available to the tenant's applications. When the tenant applications102use cloud service storage107, it is relatively easy to monitor and enforce a tenant's storage quotas and limits since all the read/write/delete operations take place within the cloud service101. However, when a tenant application102uses external storage108or109, cloud service101does not know how much space is being used on the storage device. Cloud service101may not even be aware of the quotas or limits assigned to a tenant for the external storage device. In existing systems, the external storage devices manage user quotas and limits (i.e., server-side quota management). It would be useful if cloud service101could manage user quotas and limits for external storage devices from the client side.

FIG. 2is a block diagram of a system for providing client-side distributed storage quota enforcement. A cloud service201comprises a plurality of workers202. Each worker202supports a plurality of worker processes203, such as website hosting applications. An activation service204on each worker202is responsible for initiating and managing the website hosting applications203. A database205functions as a quota and limits repository for remote storage206and a storage usage cache for website hosting applications203. The website hosting applications203may access remote storage, such as file server206, which is external to cloud service201. File server206may be accessed using a file sharing protocol, such as the Server Message Block (SMB) protocol. File server206may be designated by an SMB endpoint, and website hosting applications203can read and write data to that endpoint. Website hosting applications203access the remote file server206via remote storage services driver207. Read/write commands from website hosting applications203are sent to remote storage services driver207, which passes215the commands to the appropriate SMB endpoint for file server206.

At deployment time, cloud service201provisions a single shared folder that is the SMB endpoint for the external storage206. From then on, every time a new instance of website hosting applications203is created, then cloud service201points the new instance to the SMB endpoint. The application may use the existing folder for that tenant or a new folder may be created to hold content for other customers on the file server.

Cloud service201is highly dynamic and allows tenant processes to be deployed to any machine. When a tenant requests activation of a website, for example, the cloud service201designates a specific work202afor that website. The activation service204aon that worker creates208a worker process203afor the website hosting application. Activation service204aalso adds the website hosting application process203ainto a work sandbox by notifying209remote storage services driver207athat the process needs to be sandboxed. The tenant may scale up the website by requesting the cloud service201to host additional instances of the website hosting application, which would be loaded on other worker processes203bon other workers202b, for example.

Driver207has its own internal state called sandbox context per worker process203. The sandbox context tracks the quotas and limits for each worker process203and the usage for each worker process203. Driver207then tracks all writes and deletes against the limits and quotas per worker process to maintain an updated local usage value.

Driver207aintercepts all calls210from process203aand functions as the sandbox by blocking the specific operations. One type of operations that may be blocked is a write to the remote storage206at the SMB endpoint whenever the tenant has reached their quota. The driver207ablock the website hosting application203afrom writing more that allowed to protect the external file server206. Driver207aobtains quota and current usage information from database205indirectly, such as by updates209from activation service204a.

Activation service204amay be in charge of interacting with database205for worker202a. Activation service204aobtains the quotas and limits and the last known usage information from database205. This information provided to driver207awhen a new website hosting application is started on worker process203a. While the website is running, activation service204astarts a timer per website or per worker process for a scanning process. When the timer expires, the activation service204agoes back to the database205or directly queries or scans211external storage206to determine whether if the actual content usage has changed. This compensates for the situation where other users are storing more content to the external storage206and causing out-of-band changes to external storage206without the knowledge of worker202aor cloud storage201. The scanning maintains consistency over time. If the website hosting application instances203are running in multiple workers202, there is the possibility that the overall usage is going to exceed the storage limit or quotas for external storage206. The scanning process essentially provides a soft quota wherein out-of-band use will not be caught immediately, but will be detected during scanning211. The scanning timer may be set to any appropriate value based upon, for example, the quotas and limits, current or past usage, known number of users, and the like to reduce the chance of compromising the external storage206.

When multiple workers202are running instances of the same website hosting application203, the activation service204may check the current usage level saved to database205before scanning211the external storage206when the scanning timer expires. After checking, if the external storage usage value was recently changed in the database (e.g., since the last scan211by the activation service), then the activation service204will update using that the value and will not perform another scan. This prevents the activation services204from scanning too frequently. Each activation service204trusts the value that is in database205, which was updated by another activation service204. This provides a means for the activation services204to communicate usage level updates to each other and prevents multiple workers from scanning the external storage within a brief time.

For example, a tenant may have a file storage quota for external storage206, such a limit to write up to only 100 megabytes. Activation service204awill read212the quota information from database205when it starts process203aand then pushes the quota information to driver207a. In addition to providing the quota limit to driver207a, activation service204amay also provide the last usage recorded in database205. This informs driver207ahow much storage has already been used and the total storage limit. For example, if 40 MB of data have been written to external storage206and the limit is 100 MB, then driver207ais not going to allow process203ato write more than 60 MB of additional data to the storage206.

A problem may arise when there is not a single worker process203a. When multiple worker processes203a,203bare running the website hosting application in multiple machines at the same time, when the application will be receiving multiple requests from multiple customers. This may cause a problem when each process203ais writing to the same external storage SMB endpoint206. For example, each worker process203a,203bmay attempt to write 60 megabytes through the respective drivers207a,207b. Each driver believes that the quota has 60 megabytes available, but if both drivers207a,207bwrite 60 MB, then the total allowed quota will be exceeded.

Cloud service201minimizes the risk of exceeding the external store quota by applying scanner logic in the activation service204a,207b. The scanner logic detects if there are out-of-band writes to the external storage206. Activation service204a,204breads current quotas and limits from database205and updates drivers207a,207bwith the current usage on external store206. Drivers207a,207bmay also continuously or periodically update database205as each driver writes additional data to external storage206. This provides notification of any increased storage usage to other drivers207on remote workers202.

In existing systems, a third-party file server manages its own user quotas and limits. Because the file server is an external device, existing cloud services do not know the current status of the file server's quota and limits and, therefore, may continue to write data and information to the external file server well beyond a cloud service tenant's authorized storage limit. This may happen, for example, if the tenant has multiple instances203a,203bof the website hosting application that both use file server206. The total amount of data written by all tenant's instances would need to be tracked. Additionally, a tenant may have one or more other services running on an external system213. These external services may also use storage206, and this usage would also count against the tenant's storage quota. Communications between external system213and storage206are out-of-band to cloud server201. Thus, cloud service201does not know how much of a tenant's storage quota is used by external system213. Similarly, communications between storage206and a website hosting application203bon worker202bare out-of-band to a different website hosting application203aon a different worker202a. As a result, different workers202a,202bin cloud service201are not aware of how much of the tenant's storage quota is used by processes on the other workers.

In the system ofFIG. 2, remote storage services deriver207uses information from database205to manage the quotas and limits for file server206on a per-tenant basis. The tenants may already have existing storage technology or devices when they subscribe to use cloud service201. Instead of requiring tenants to use storage214in cloud service201, the tenants may continue to use their own storage solution206, such as network-attached storage (NAS) devices or storage hosted by other cloud services.

When a tenant configures services, such as a website hosting application203, on cloud service201, they can use cloud based storage214and/or the tenant can configure an external storage206by providing an SMB endpoint for the external device. Remote storage services driver207allows for sandboxing of the processes203on workers202from the client side and enforces storage quotas. Because there may be multiple workers202that access the same remote storage206, cloud service201provides for coordination across different remote storage services drivers207on multiple workers202.

During operation, depending on how the website has been scaled out, multiple workers202may run the same website203on multiple machines. When website hosting application203aattempts to write data to external store206, the remote storage services driver207asees the write request210from website hosting application203a. Remote storage services driver207aautomatically obtains the current content amount and last usage for the tenant's subscription on external store206. Driver207aobtains this information from database205directly through query211or indirectly, such as through updates209from activation service204a. If the tenant has space available on external storage206, then driver207aallows the write request215to pass to the SMB endpoint for storage206.

Driver207amay keep a running count storage206usage so that it does not have to query to database205for every new write request. Instead, after getting a current content amount and last usage from database205, driver207amay update the value locally over a period, such as for five minutes, before requesting an updated content amount and last usage. It will be understood that the driver207amay keep track of both writes and deletes and other changes to data on remote store206. Periodically, such as after a set time or after a set number of operations, driver207amay notify database205of any writes, deletes, or other operations on storage206so that database205will be updated when another driver207bon another worker202bneeds to write to external storage206.

It will be understood that the servers, machines, or processes running in cloud service201may access external storage206using any appropriate communication link. Input/output (I/O) interface216on cloud service201provides access to external storage206through network217, which may be any public or private data network, such as the Internet or an intranet or enterprise network.

FIG. 3is a flowchart illustrating a computer-implemented method for quota enforcement in a distributed storage system. In step301, a storage limit and a storage usage are read from a database in a cloud service. The storage limit and storage usage are associated with a storage system external to the cloud service. In step302, a process is created on the cloud service, wherein the process is configured to access the external storage system. In step303, a storage driver is configured on the cloud service to monitor access to the external storage system by the process and to enforce storage limits for the external storage system. The process and the storage driver may be running on the same processor, virtual machine, or server in the cloud service.

In step304, the cloud storage database is checked for an updated storage usage. In step305, if an updated storage usage is present in the database, then the updated storage usage is provided to the storage driver. In step306, if an updated storage usage is not present in the database, then the external storage is scanned to obtain the updated storage usage. In step307, the updated storage usage obtained from scanning the external storage is saved to the database.

In other embodiments, the method further comprises initiating a timer to determine how often to check the database for the updated storage usage.

In other embodiments, the method further comprises creating two or more additional processes on the cloud service, wherein the two or more additional processes are also configured to access the external storage system; configuring a separate storage driver to monitor access to the external storage system and to enforce storage limits on the external storage system for each of the two or more additional processes; and independently checking the database on behalf of each separate storage driver to detect the updated storage usage, wherein if the updated storage usage is present, then providing the updated storage usage to the storage driver for which the database check was performed. The method may further comprise initiating timers associated with each separate storage driver to determine how often to check the database for the updated storage usage.

In other embodiments, the method further comprises intercepting, by the storage driver, reads and writes from the process to the external storage system; and updating a local state in the storage driver, based upon the intercepted reads and writes, to track the process's usage of the external storage system. The method may further comprise blocking, by the storage driver, the reads and writes from the process to the external storage system when the local state exceeds the storage limit. The method may further comprise overwriting the local state in the storage driver when the updated storage usage is present in the database. The method may further comprise saving the updated storage usage as determined by scanning the external storage to the database; and overwriting the local state in the storage driver using the updated storage usage as determined by scanning the external storage.

FIG. 4is a flowchart illustrating an alternative computer-implemented method for quota enforcement in a distributed storage system. In step401, a storage driver is configured on a cloud service to monitor access to an external storage system by a process and to enforce storage limits for the external storage system. In step402, the storage driver intercepts reads and writes from the process to the external storage system. In the403, a local state is updated in the storage driver, based upon the intercepted reads and writes, to track the process's usage of the external storage system.

In step404, a database on the cloud service is periodically checked for an updated storage usage. In step405, the cloud storage database is checked for an updated storage usage. In step406, if an updated storage usage is present in the database, then the updated storage usage is provided to the storage driver. In step407, if an updated storage usage is not present in the database, then the external storage is scanned to obtain the updated storage usage. In step406, the local state in the storage driver is overwritten with the updated storage usage.

The method may further comprise blocking, by the storage driver, the reads and writes from the process to the external storage system when the local state exceeds the storage limit.

The method may further comprise continuing to update the local state in the storage driver, based upon the intercepted reads and writes, after overwriting the local state with the updated storage usage.