Dynamic data storage management

A storage management subsystem monitors usage of the data stored by a data storage system. Based at least in part on the monitored usage of the stored data, a storage profile is determined for the stored data. The storage profile indicates a first time period during which a first portion of the stored data is anticipated to be accessed and a second portion of the stored data is not anticipated to be accessed and a second time period during which the first portion of the stored data is not anticipated to be accessed. During at least the first time period, the first portion of the data is stored in a decompressed format and the second portion of the data is stored in a compressed format. During at least the second time period, the first portion of the data is stored in the compressed format.

TECHNICAL FIELD

The present disclosure relates generally to data storage systems, more specifically to dynamic data storage management.

BACKGROUND

Data may be generated when users interact with applications and when electronic systems are operated. This data may be stored in a database or data store. For example, a database may receive data from data sources and store the data for use at a later time.

SUMMARY

In one embodiment, a system includes a data storage system that stores data from one or more data sources. A storage management subsystem is communicatively coupled to the data storage system and monitors usage of the data stored by the data storage system. The usage of the stored data includes events associated with receipt of additional data by the data storage system and/or access to portions of the stored data. Based at least in part on the monitored usage of the stored data, a storage profile is determined for the stored data. The storage profile indicates a first time period during which a first portion of the stored data is anticipated to be accessed and a second portion of the stored data is not anticipated to be accessed and a second time period during which the first portion of the stored data is not anticipated to be accessed. During at least the first time period, the first portion of the data is stored in a decompressed format and the second portion of the data is stored in a compressed format. During at least the second time period, the first portion of the data is stored in the compressed format.

Previous technology used to manage electronic data storage can be inefficient and unreliable. For example, the characteristics (file size, data content, etc.) of data provided to a data storage system, such as a database, can change over time, and the storage system may not always be appropriately configured to efficiently and reliably store incoming data. For example, data from a given source may be associated with an initial file size when it is received in a first batch. However, the database or datastore may lack sufficient storage space (e.g., memory) if the file size increases in a second batch of data received from this source at a later time. Previous technology also fails to accommodate the storage of data from new data sources (e.g., data associated with newly identified systems, users, applications, and the like) without extensive development and testing for the manual configuration of the data storage system to receive the new data.

Certain embodiments of this disclosure provide unique solutions to technical problems of previous data storage technology, including those problems identified above by providing intelligent tools and approaches for operating a data storage system based on anticipated use of stored data and associated storage capacity needs. For example, the disclosed systems provide several technical advantages over previous technology, which include: (1) more efficient and reliable operation of a data storage system with fewer downtimes for reconfiguration to handle changing data sources; (2) the automatic configuration of the data storage system to receive data from new data sources; and (3) the automatic compression and decompression of stored data based on data-specific storage profiles based on a history of the use of the data. As such, this disclosure may improve the function of computer systems used to store data from one or more data sources. For example, in some embodiments, data compression and decompression may be automatically implemented based on the anticipated usage of the data storage system. These features may ensure data is reliably available in an efficiently usable form (e.g., in a decompressed form if the data is anticipated to be accessed soon) based on anticipated usage of the data. Similarly, an anticipated usage may indicate that a large amount of data will be received by the data storage system, and system infrastructure may be automatically scaled (e.g., by increasing computing resources available to the system) in order to accommodate the anticipated incoming data. This disclosure may particularly be integrated into a practical application of a data management subsystem, which automatically implements management tasks for an associated data storage system by detecting new data sources, allocating storage to data from these sources, building storage profiles for this data, and intermittently updating, based on the storage profiles, how the data is handled by the data storage system (e.g., how the data is compressed/decompressed and/or how storage resources are allocated for storing the data) and how computing resources are provisioned to the data storage system.

DETAILED DESCRIPTION

As described above, previous technology lacks tools for efficient and reliable management of data storage systems such as databases, datastores, and the like. For example, previous approaches relying on trial and error and resource-intensive testing result in slow development and inefficient use of the computing resources used for testing, developing, and operating data storage systems. This disclosure provides a new approach to data storage management by dynamically adjusting how information is stored in a data storage system and/or the hardware infrastructure allocated to a data storage system. For example, usage of the data storage system may be monitored to determine a storage profile for data from a given data source, and this storage profile may be used to dynamically adjust when stored data is compressed/decompressed, the amount of storage resources allocated for storing data from the data source, and/or the amount of computing resources provisioned to the data storage system.

Data Storage System

FIG. 1is a schematic diagram of an example system100for data storage. The system100generally facilitates the efficient management of data128,130stored in a data storage system126and/or for reconfiguration of the data storage system126for improved efficiency and usability. The system100includes one or more data sources102a,b,a dynamic storage management subsystem (DSMS)112, at least one data storage system126, a user front end132, an administrator monitor136, and a network140. As described in greater detail below with respect toFIGS. 2-3, the system100generally facilitates automatic management of data storage system(s)126, such that data128,130can be reliably stored in the data storage system126and data128,130can be reliably retrieved (e.g., using a user front end132). For example, the DSMS112may implement various processes, including: (1) the allocation120of data storage resources (e.g., memory) of the data storage system126to appropriately store data106a,b;(2) the scaling122of computing infrastructure available to the data storage system126(e.g., the automatic provisioning of additional storage capacity, compute nodes, or the like to the data storage system126); and/or (3) intelligent compression/decompression based on anticipated usage of the data128,130stored in the data storage system126. Further examples of functions of the DSMS112are described in greater detail below.

Each of the data sources102a,bmay be any computing device or collection of computing devices (e.g., a collection of devices implemented as a server, a virtual server, or the like) from which data106a,bmay be provided for storage in the data storage system126(e.g., as data128,130). An example of a device for implementing a data source102a,bis shown inFIG. 4and described in greater detail below. A data source102a,bmay be associated with a system104a,b,particular data106a,b,one or more users108a,b,and/or one or more applications110a,b.A system104a,bmay include hardware and/or software components (e.g., a system may include a processor, memory, and network interface as described below with respect toFIG. 4). As an example, a data source102a,bmay be associated with a system104a,bthat collects and/or generates data106a,band provides the data106a,bfor storage in the data storage system126. The data106a,bmay include any type of electronically stored information (e.g., stored in any format). Users108a,bgenerally correspond to users of the systems104a,band/or applications110a,b.As another example, a user108a,bmay operate an application110a,band/or system104a,b.For instance, a user108a,bmay access an application110a,bwhich provides data106a,bfor storage in the data storage system126. The applications110a,binclude any software applications that execute code and/or instructions to perform one or more tasks. As an example, an application110a,bmay facilitate user interaction with one or more accounts, facilitate data analysis and/or presentation, may facilitate transactions, and the like. Data106a,bcollected and/or generated by the application110a,bmay be provided for storage in the data storage system126.

The DSMS112may be any computing device or collection of computing devices (e.g., a collection of devices implemented as a server, a virtual server, or the like). An example of a device for implementing the DSMS112is shown inFIG. 4and described in greater detail below. While shown as a separate device or subsystem in the example ofFIG. 1, all or a portion of the functions of the DSMS112may be implemented using hardware (e.g., the processor, memory, and network interface—seeFIG. 4) of the data storage system126(described below). The DSMS112generally automates a variety of tasks associated with the management of the data storage system(s)126. The DSMS112is compatible with any computing infrastructure (e.g., central processing units (CPUs) and/or graphical processing units (GPUs)) and storage system type. For example, the DSMS112is compatible with a data storage system126that is a distributed data storage system (e.g., implemented using multiple distributed GPUs).

As described in greater detail below with respect toFIGS. 2-3, the DSMS112is configured to manage the storage of data106a,bfrom known data sources102a,b(e.g., sources102a,bfrom which data106a,bhas previously been provided for storage in the data storage system126) and new data sources102a,b(e.g., sources102a,bfrom which data106a,bhas not previously been provided for storage in the data storage system126). For example, the DSMS112may include instructions for new data source handling114, which facilitate the automatic configuration of the storage system126to efficiently and reliably receive and store data106a,bfrom new data sources102a,b.For example, the DSMS112may use the instructions for new data source handling114to detect a new data source102a,b(e.g., a system104a,b,data106a,b,a user108a,b,and/or an application110a,bthat has not previously provided data106a,bfor storage in the data storage system126(e.g., for which a storage profile118is not yet established). The instructions for new data source handling114may perform automatic storage allocation120in order to automatically allocate a portion of the data storage resources of the data storage system126(e.g., a portion of the memory404ofFIG. 4) for storage of data106a,bprovided by the newly detected data source102a,b.

The DSMS112may include instructions for input and usage monitoring116. Input and usage monitoring116generally involves monitoring characteristics of the data106a,bprovided to the data storage system126(e.g., the input of data106a,b) and how the data128,130that is stored in the data storage system126is subsequently used (e.g., how often the data128,130is accessed, at what times the data128,130tends to be accessed, and the like).

Information determined by input and usage monitoring116may be used to establish and/or update storage profiles118for the data sources102a,b(e.g., for the data106a,bprovided by the data sources102a,bfor storage in the data system126). The storage profiles118generally include properties for configurating the data storage system126for storing the data106a,b.For example, the data storage profiles118may include an amount of storage resources that should be allocated to the data106a,b,an amount of storage resources needed to be provisioned to the data storage system126in order to reliably store the data106a,b,and/or compression properties for the data106a,b.The DSMS112may execute instructions for storage allocation120in order to allocate, based on the storage profiles118, storage resources (e.g., memory404ofFIG. 4) of the data storage system126for storage of data106a,b.The DSMS112may execute instructions for infrastructure scaling122in order to provision, based on the storage profiles118, appropriate computing resources (e.g., processing, memory, and/or network resources) to the data storage system126for handling data106a,bprovided from a data source102a,b.

The DSMS112executes instructions for the automatic compression/decompression124of the data128,130stored in the data storage system126. The storage profiles118may include instructions (e.g., a schedule) for decompressing and compressing the data128,130stored in the data storage system126, such that the data128,130is efficiently and reliably available to users. For example, past usage of the data128,130may be monitored via input and usage monitoring116, in order to determine a schedule for decompressing and/or compressing portions of the data128,130at certain times (e.g., of the day, week, month, year, etc.). For example, traffic trends associated with the access of data128,130may be monitored, and times during which greater than a threshold number of access events are typically detected may be considered high traffic times. The portion of the data128,130that is commonly accessed during high traffic times may be decompressed (e.g., stored as decompressed data130). Data128,130that is not determined to be accessed during a given time period may be compressed during that time period (e.g., stored as compressed data128). A similar schedule-based approach may be used for infrastructure scaling122(e.g., be increasing available computing resources during high traffic times determined by input and usage and monitoring116).

The data storage system126is device or collection of devices (e.g., a collection of devices implemented as a server, a virtual server, or the like) configured to store data128,130. The data storage system may store compressed data128and decompressed data130. The compressed data128may be data of any type that has been compressed using a compression algorithm (e.g., to decrease the storage resources needed to store the data128). Compression may be “lossy” or “lossless.” In lossy compression, a portion of the original data cannot be recovered when the compressed data128is decompressed (e.g., using the appropriate decompression algorithm). In lossless compression, the original data is fully recoverable when the compressed data128is decompressed. The decompressed data130is generally data of any type that is stored in a decompressed format. A given portion of data generally requires more storage resources (e.g., memory) when stored as decompressed data130than when stored as compressed data128. In some cases, decompressed data130may be more efficiently and rapidly accessible to users of the data storage system126(e.g., because the decompressed data130is in a format that is immediately usable by the users). As described briefly above and in greater detail below with respect toFIGS. 2 and 3, the DSMS112may improve performance of the data storage system126by performing automatic compression and decompression124of the data128,130such that the data128,130is decompressed and efficiently accessible to users when needed and is compressed when not needed, such that computing resources of the data storage system126are used efficiently.

The user front end132is any software and/or hardware configured to facilitate user interaction with the data128,130stored in the data storage system. For example, the user front end132may include a processor, memory, and network interface as described below with respect to the device ofFIG. 4. The user front end134may access decompressed data130stored in the data storage system126and generate a visualization134based on the accessed data130. The visualization134may be any report, table, chart, or the like based on the accessed portion of the data130. It may be more efficient for the data accessed by the front end132to be decompressed data130rather than compressed data128.

The administrator monitor136is any software and/or hardware configured to facilitate monitoring of the function and/or performance of the data storage system126by an administrator of the system126. For example, the administrator monitor136may include a processor, memory, and network interface as described below with respect to the device ofFIG. 4. The administrator monitor136may monitor the utilization of storage resources by the data storage system126and/or changes to storage allocation120, infrastructure scaling122, and/or compression/decompression124performed by the DSMS112and generate a visualization138based on the monitored information. The visualization138may be any report, table, chart, or the like based on the monitored information about the data storage system126.

Network140facilitates communication between and amongst the various components of the system100. This disclosure contemplates network140being any suitable network operable to facilitate communication between the components of the system100. Network140may include any interconnecting system capable of transmitting audio, video, signals, data, messages, or any combination of the preceding. Network140may include all or a portion of a public switched telephone network (PSTN), a public or private data network, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a local, regional, or global communication or computer network, such as the Internet, a wireline or wireless network, an enterprise intranet, or any other suitable communication link, including combinations thereof, operable to facilitate communication between the components.

FIG. 2shows a flow diagram200illustrating an example operation of the system100ofFIG. 1. New data source handling114may facilitate the detection202of a new data source102a,b.The detection202of a new data source102a,bmay involve detecting a call (e.g., from the data source102a,b) to provide data106a,bto the data storage system126and/or a call (e.g., from the data storage system126or an associated administrative device) to receive or collect data106a,bfrom a data source102a,b.As an example, a new data source102a,bmay be detected when a new user108a,baccesses an application110a,b(e.g., when a user108a,baccesses an application110a,band no data128,130has previously been stored for this user108a,bin the data storage system126).

Since the data storage system126may not initially be configured to receive data106a,bfrom a new data source102a,b,data source profiling204may be performed to determine how the data106a,bfrom the new data source102a,bshould be stored and/or if further infrastructure should be allocated to the data storage system126to handle data106a,bfrom the new data source102a,b(e.g., to determine a storage profile118for the new data106a,b). Initially, a default storage profile118may be used to accommodate reliable storage of data106a,bfrom the new data source102a,b,and this profile118may be updated following input and usage monitoring116. In some cases, an initial storage profile118may be determined based on characteristics of the new data source102a,b.For example, the initial storage profile118may include properties of the data source102a,b(e.g., characteristics of contents of data106a,bprovided by the new data source102a,band a target, such as an intended end user to which data106a,bfrom the new data source102a,bis to be provided). If a new system104a,band/or application110a,bis detected, profiling204may include determination of characteristics of the infrastructure associated with the system104a,band/or application110a,b(e.g., the processing, memory, and network infrastructure included in the system104a,bor allocated to the application110a,b). In some cases, the initial storage profile118may be determined based on an anticipated file size, data transfer rate, and/or downstream use of the data106a,breceived from the new data source102a,b.For example, the storage profile118may include an amount of storage space anticipated to be needed to store the data106a,bprovided by the new data source102a,b.

The DSMS112may then perform tagging206and cataloging208of the data106a,bprovided by the new data source102a,b.For example, the data106a,bmay be tagged, based at least in part on the initial profile118. Tags may link, or associate, the data106a,bto its anticipated downstream use (e.g., by users of the data storage system126). For example, a first tag may indicate a first portion of the data106a,bis associated with an account of a user108a,b,while a second tag indicates another portion of the data106a,bis associated with a transaction performed using an application110a,b.Cataloging208may be performed to identify how the data106a,bshould be stored in the data storage system126(e.g., at a byte level) and what format the data106a,bshould be stored in (e.g., as compressed data128or decompressed data130). The resulting catalog may be included in the storage profile118in order to indicate how the data106a,bis stored in the infrastructure (e.g., at physical and/or virtual address spaces) of the data storage system126. The storage profile118may be used to allocate data storage resources (e.g., memory) of the data storage system126for storage of the new data106a,bprovided by the data source102a,b.The storage profile118may also or alternatively be used to perform automatic infrastructure scaling122, which involves automatically adjusting the computing resources (e.g., memory) provisioned to the data storage system126, such that the data106a,bcan be stored reliability. As such, the DSMS112facilitates the automatic allocation120of storage resources of the data storage system126for receipt of the new data106a,b.This allows new data sources102a,bto provide data106a,bfor storage in the data storage system126while avoiding the delays and inefficiencies of previous technology.

Input and usage monitoring116may be performed to update the storage profile118for the data106a,band/or the data source102a,bbased on the history of usage of the data storage system126, and the updated storage profile118may be used to dynamically adjust the storage allocation120, infrastructure scaling122, and/or compression/decompression124. For example, over a period of time following the detection of a new data source102a,b,usage of the data106a,bfrom the data source102a,b(e.g., as received as data106a,bor as stored as data128,130) may be monitored. For example, events associated with receipt of additional data106a,bfrom the data source102a,bmay be monitored (e.g., an amount of the data106a,bthat has been received at different times of the day, month, year, or the like). As another example, events associated with access of portions of the stored data128,130may be detected (e.g., a user accessing data128,130to generate a visualization134of the data128,130).

The storage profile118may be updated based on the monitored input of data106a,bto the data storage system126and usage of the stored data128,130. The storage parameters and/or properties included in the storage profile118(e.g., schedules of resource allocation, resource provisioning, and/or compression/decompression) may be determined using a method of machine learning or artificial intelligence. For example, the storage profile118may be determined or updated based at least in part on the monitored usage of the stored data128,130. The storage profile118may include a first time period during which a first portion of the stored data128,130is anticipated to be accessed and a second portion of the stored data128,130is not anticipated to be accessed and a second time period during which the first portion of the data128,130is not anticipated to be accessed. These first and second time periods may be stored (e.g., as a compression/decompression schedule) in the storage profile118for dynamic compression/decompression124of the data128,130, such that the appropriate portions of the data128,130are made readily available as decompressed data130if likely to be accessed or stored as compressed data128if not likely to be accessed. For instance, during at least the first time period, the DSMS112may cause the first portion of the data128,130to be in a decompressed format and the second portion of the data128,130to be in a compressed format. During at least the second time period, the DSMS112cause the first portion of the data128,130to be in a compressed format.

The different time periods during which compression and decompression of the data128,130are appropriate may be determined by monitoring traffic trends at the data storage system126. For example, the DSMS112may monitor user traffic trends over time at the data storage system126, data types stored in the data storage system126, and compression types (e.g., lossy or lossless compression) available to implement on the stored data types. In some cases, certain decompressed data130may not be compressed if a lossless compression algorithm is not available for that data type. Any appropriate method may be used to determine trends of the usage of the data128,130. For example, an appropriately trained machine learning algorithm may be used to identify trends in the usage of the data storage system126over time and determine a schedule for storage allocation120, infrastructure scaling122, and/or compression and decompression124. Based on the traffic trends, a time period (e.g., a continuous interval of times) is determined during which a first data type is accessed at least a threshold number of days per month. During this time period, the first type of data may be stored as decompressed data130. Similarly, a second time period, may be determined during which the same data type is accessed less than the threshold number of days per month. This data type may be stored as compressed data128during the second time period.

The DSMS112may dynamically adjust infrastructure scaling122by adjusting the computing resources provisioned to the data storage system126based on the expected amount of compressed data128and decompressed data130at any given time. For example, during times when a majority of the data128,130is stored as compressed data128, fewer storage resources (e.g., memory) may be needed by the data storage system126. Similarly, when a majority of the data128,130is stored as decompressed data130, additional data storage resources may be needed by the data storage system126. For example, the DSMS112may determine a period of time during which a threshold portion of the data128,130is anticipated to be accessed (e.g., and stored as decompressed data130). If an anticipated file size associated with decompressed data130during a the identified period of time is greater than a threshold file size (e.g., a percentage of the current storage capacity of the data storage system126), the DSMS112may automatically cause additional data storage resources to be provisioned to the data storage system126.

Similar approaches to those described above may be used to dynamically adjust and/or schedule the allocation120of storage resources for storing data106a,bfrom certain data sources102a,band/or for infrastructure scaling122(e.g., to adjust the computing resources allocated to the data storage system126). For example, the DSMS112may determine, based at least in part on the monitored usage of the stored data128,130, a time period during which incoming data106a,bis anticipated to have a file size that is greater than a threshold file size (e.g., a percentage of the current storage capacity of the data storage system126). During at least this time period, the DSMS112may cause additional data storage resources to be provisioned to the data storage system126. Similarly, if data106a,bfrom a given data source102a,bis anticipated to be received during a given period of time, additional storage may be allocated to this data source102a,b.

Example Methods of Operation

FIG. 3illustrates a method300for operating the system100ofFIG. 1. The method300may begin at step302where the DSMS112detects a new data source102a,b.As described above, the detection of a new data source102a,bmay involve detecting a call (e.g., from the data source102a,b) to provide data106a,bto the data storage system126and/or a call (e.g., from the data storage system126) to receive or collect data106a,bfrom a data source102a,b.In some cases, a new data source102a,bmay be detected when a new user108a,baccesses an application110a,b(e.g., when a user108a,baccesses an application110a,band no data128,130has previously been stored for this user108a,bin the data storage system126).

At step304, the DSMS112determines an initial storage profile118for the data106a,bfrom the data source102a,b.As described above, the storage profile118generally includes storage parameters and/or properties for configuring the data storage system126to store the data106a,bfrom the newly detected data source102a,b.For instance, parameters in the storage profile118may indicate how the data106a,bshould be stored and/or an amount of storage resources need to reliably store the data106a,b.The initial storage profile118may be a default storage profile (e.g., with predefined storage parameters for the new data source102a,b). Alternatively, the initial storage profile118may be determined based on characteristics of the new data source102a,b.For example, the initial storage profile may include, or may be based on, properties of the data source102a,b(e.g., characteristics of contents of data106a,bprovided by the new data source102a,band/or an intended target or downstream user of the data106a,b). If a new system104a,band/or application110a,bis detected at step302, the DSMS112may determine characteristics of the infrastructure associated with the system104a,band/or application110a,b(e.g., the processing, memory, and network infrastructure included in the system104a,bor allocated to the application110a,b) and use these characteristics to determine the initial storage profile118. For example, the initial storage profile118may be determined based on an anticipated file size, data transfer rate, and/or downstream use of the data106a,breceived from the new data source102a,b.For example, the storage profile118may include an amount of storage space anticipated to be needed to store the data106a,bprovided by the new data source102a,b.

At step306, the DSMS112may tag the data106a,bbased at least in part on the characteristics determined at step304. For example, the data106a,bmay be tagged, based at least in part on the initial profile118determined at step304. Tags may link, or associate, the data106a,bto its anticipated downstream use (e.g., by users of the data storage system126). At step308, the DSMS112may catalog the data106a,b,based at least in part on the tags from step308, in order to indicate how the data106a,bshould be stored in the data storage system126(e.g., at a byte level) and what format the data106a,bshould be stored in (e.g., as compressed data128or decompressed data130). For instance, portions of memory if the data storage system126may be assigned as indicated in the catalog based on the anticipated storage capacity needed to store data106a,bfrom the data source102a,b.The resulting catalog may be included in the storage profile118in order to indicate how the data106a,bis stored in the infrastructure (e.g., at a physical and/or virtual address) of the data storage system126.

At step310, the DSMS112may determine if the amount, or size, of data106a,bbeing provided to the data storage system126exceeds the current storage capacity of the system126(e.g., or exceeds the amount of storage capacity allocated to the data106a,b). If the data106a,bbeing provided to the data storage system126exceeds the current storage capacity of the system126, the DSMS112may proceed to step310where storage allocation120is adjusted and/or infrastructure scaling122is performed. For example, the DSMS112may allocate more storage resources (e.g., memory) of the data storage system126for storage of data106a,bfrom the data source102a,band/or provision additional storage resources (e.g., memory) to the data storage system126.

If the amount or size of data106a,bbeing provided to the data storage system126does not exceed the storage capacity allocated to the data106a,b,the DSMS112may proceed to step314, where the data106a,bis stored in the data storage system126. In some embodiments, the data106a,bmay initially be stored as decompressed data130(e.g., and older, previously received data106a,bmay be compressed to store as compressed data128after a predefined period of time). For example, the storage profile118determined at step304(e.g., and as updated to include tags and/or catalog information from steps306and/or308) may indicate an initial schedule for compression/decompression of the data128,130,b stored in the data storage system126. As described below with respect to the subsequent steps of method300, the storage profile118may be updated over time to dynamically adjust how the data106a,bis stored as compressed data128and/or decompressed data130and/or how resources are allocated for storage of the stored data128,130.

At step316, the DSMS112monitors the input of additional data106a,band usage of the stored data128,130. For example, the DSMS112may use the instructions for input and usage monitoring116to monitor usage of the data128,130stored by the data storage system126. For example, the DSMS112may detect and monitor events associated with receipt of data (e.g., all or a portion of the data106a,bfrom the data sources102a,b) by the data storage system126. The DSMS112may detect and monitor events associated with accessing the stored data128,130. For example, the DSMS112may determine when (e.g., during which hours of the day, which days of the week, and the like) and in what amount the data128,130is commonly accessed by users of the data storage system126(e.g., requested for presentation as a visualization134in a user front end132) and/or when and in what amount data106a,bfrom a given data source102a,bis typically received by the data storage system126. For example, over a period of time following the detection of a data source102a,b,usage of the data106a,bfrom the data source102a,b(e.g., as received as data106a,bor as stored as data128,130) may be monitored. For example, events associated with receipt of additional data106a,bfrom the data source102a,bmay be monitored (e.g., an amount of the data106a,bthat has been received at different times of the day, month, year, or the like). As another example, events associated with access of portions of the data128,130may be detected (e.g., a user accessing data128,130to generate a visualization134of the data128,130).

At step318, the DSMS112updates the storage profile118for the data106a,bfrom data source102a,band/or related stored data128,130from the same data source102a,b,based on information obtained at step316(e.g., based on the history of usage of the data storage system126). The properties and/or parameters included in the storage profile118(e.g., schedules of resource allocation, resource provisioning, and/or compression/decompression) may be determined using a method of machine learning or artificial intelligence. For example, the storage profile118may be determined or updated based at least in part on the monitored input of data106a,band/or the usage of the stored data128,130(from step316). For instance, the storage profile118may include a first time period during which a first portion of the data128,130is anticipated to be accessed and a second portion of the stored data128,130is not anticipated to be accessed and a second time period during which the first portion of the data128,130is not anticipated to be accessed. These first and second time periods may be stored (e.g., as part of a compression/decompression schedule) in the storage profile118for dynamic compression/decompression124of the data128,130, such that the appropriate portions of the data128,130are stored as decompressed data130if likely to be accessed or compressed data128if not likely to be accessed (see step320).

At step320, the DSMS112uses the storage profile118to compress and/or decompress at least a portion of the stored data128,130. For example, during at least the first time period identified at step318, the DSMS112may cause the first portion of the data128,130to be stored as decompressed data130and the second portion of the data128,130to be stored as compressed data128. During at least the second time period identified at step318, the DSMS112may cause the first portion of the data128,130to be stored as compressed data128(e.g., because this portion of the data128,130is not likely to be accessed during the second time period).

At step322, the DSMS112uses the storage profile118to allocate data storage resources (e.g., memory) of the data storage system126for storage of data106a,bprovided by the data source102a,b.The storage profile118may also or alternatively be used to perform automatic infrastructure scaling122, which involves automatically adjusting the computing resources (e.g., memory) provisioned to the data storage system such that the data106a,bcan be stored reliability. As such, the DSMS112facilitates the automatic allocation120of storage resources of the data storage system126for receipt of the new data106a,b.This allows new data sources102a,bto provide data106a,bfor storage in the data storage system126while avoiding instances where inadequate storage resources are available to store provided data106a,b.

Example Device for API Integration

FIG. 4illustrates an embodiment of a device400configured to implement various components of the system100. One or more devices400may be used to implement the data sources102a,b,DSMS112, data storage system126, user front end132, and administrator monitor136ofFIG. 1. The device400includes a processor402, a memory404, and a network interface406. The device400may be configured as shown or in any other suitable configuration.

The processor402comprises one or more processors operably coupled to the memory404. The processor402is any electronic circuitry including, but not limited to, state machines, one or more central processing unit (CPU) chips, logic units, cores (e.g. a multi-core processor), field-programmable gate array (FPGAs), application specific integrated circuits (ASICs), or digital signal processors (DSPs). The processor402may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The processor402is communicatively coupled to and in signal communication with the memory404and the network interface406. The one or more processors are configured to process data and may be implemented in hardware or software. For example, the processor402may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor402may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The one or more processors are configured to implement various instructions. For example, the one or more processors are configured to execute instructions to implement the function disclosed herein, such as some or all of those described with respect to the flow diagram200ofFIG. 2and the method300ofFIG. 3. In some embodiments, the function described herein is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry.

The memory404is operable to store any of the information described above with respect toFIGS. 1-3along with any other data, instructions, logic, rules, or code operable to execute the function described herein. For example, the memory404may store the data106a,b,128,130and storage profiles118ofFIG. 1described above with respect toFIGS. 1-3. The memory404may also store detection instructions408, which include any logic, code, and/or rules for implementing new data source handling114functions of the DSMS112(e.g., for detection202ofFIG. 2), described above with respect toFIGS. 1-3. The memory404may also store profiling instructions410, which include any logic, code, and/or rules for implementing new data source handling114functions of the DSMS112(e.g., for profiling204ofFIG. 2), described above with respect toFIGS. 1-3. The memory404may also store tagging instructions412, which include any logic, code, and/or rules for implementing new data source handling114functions of the DSMS112(e.g., for tagging206ofFIG. 2), described above with respect toFIGS. 1-3. The memory404may also store cataloging instructions414, which include any logic, code, and/or rules for implementing new data source handling114functions of the DSMS112(e.g., for cataloging208ofFIG. 2), described above with respect toFIGS. 1-3. The memory404may also store monitoring instructions416, which include any logic, code, and/or rules for implementing the input and usage monitoring116functions of the DSMS112, described above with respect toFIGS. 1-3. The memory404may also store compression/decompression instructions418, which include any logic, code, and/or rules for implementing compression/decompression124functions of the DSMS112and/or data storage system126, described above with respect toFIGS. 1-3. The memory404may also store storage allocation instructions420, which include any logic, code, and/or rules for implementing storage allocation120functions of the DSMS112, described above with respect toFIGS. 1-3. The memory404may also store infrastructure scaling instructions422, which include any logic, code, and/or rules for implementing infrastructure scaling122functions of the DSMS112, described above with respect toFIGS. 1-3. The memory404may be volatile or non-volatile and may comprise read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), dynamic random-access memory (DRAM), and static random-access memory (SRAM).

The network interface406is configured to enable wired and/or wireless communications. The network interface406is configured to communicate data between the device400and other network devices, systems, or domain(s). For example, the network interface406may comprise a WIFI interface, a local area network (LAN) interface, a wide area network (WAN) interface, a modem, a switch, or a router. The processor402is configured to send and receive data using the network interface406. The network interface406may be configured to use any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.