Managing software asset environment using cognitive distributed cloud infrastructure

A method and system are provided for performing an extract-transform-load (ETL) process. The method includes collecting load information about a volume and a complexity of raw data to be processed during the ETL process. The method further includes receiving an expected completion time of the ETL process and execution information about (i) hardware resources and (ii) an influence of the hardware resource on an execution time of the ETL process. The method also includes calculating resources for a distributed processing software infrastructure to be used to perform the ETL process, by applying a statistical method to the load information, expected completion time, and execution information. The method additionally includes dynamically assigning cloud resources corresponding to and based on the calculated resources, in accordance with the expected completion time. The method further includes performing the ETL process on the raw data using the assigned cloud resources and storing ETL process results.

BACKGROUND

Technical Field

The present invention relates generally to information processing and, in particular, to managing a software asset environment using a cognitive distributed cloud infrastructure.

Description of the Related Art

In the majority of currently available software asset management tools, the centralized server is treated as the most important part of the software asset environment. Products such as Bigfix® Inventory rely on data gathered from all endpoints and stored in one place. To find relevant information about usage, Processor Value Units (PVUs) consumption and others, an Extract, Transform and Load (ETL) process is performed. Depending on the amount of data from the endpoints, this process can consume a lot of memory resources as well as take a lot of time. Also, all data is imported (from a specific point in time). The end user (software license administrator) has no influence on what data is imported and may not be able to accelerate the ETL process without upgrading system hardware.

SUMMARY

According to an aspect of the present invention, a method is provided for performing an extract-transform-load (ETL) process. The method includes collecting load information about a volume and a complexity of raw data to be processed during the ETL process. The method further includes receiving an expected completion time of the ETL process and execution information about (i) hardware resources and (ii) an influence of the hardware resource on an execution time of the ETL process. The method also includes calculating, by a processor, resources for a distributed processing software infrastructure to be used to perform the ETL process, by applying a statistical method to the load information, the expected completion time, and the execution information. The method additionally includes dynamically assigning cloud resources corresponding to and based on the calculated resources, in accordance with the expected completion time. The method further includes performing the ETL process on the raw data using the assigned cloud resources and storing results of the ETL process in a database.

According to another aspect of the present invention, a computer program product is provided for performing an extract-transform-load (ETL) process. The computer program product comprising a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes collecting load information about a volume and a complexity of raw data to be processed during the ETL process. The method further includes receiving an expected completion time of the ETL process and execution information about (i) hardware resources and (ii) an influence of the hardware resource on an execution time of the ETL process. The method also includes calculating, by a processor, resources for a distributed processing software infrastructure to be used to perform the ETL process, by applying a statistical method to the load information, the expected completion time, and the execution information. The method additionally includes dynamically assigning cloud resources corresponding to and based on the calculated resources, in accordance with the expected completion time. The method further includes performing the ETL process on the raw data using the assigned cloud resources and storing results of the ETL process in a database.

According to yet another aspect of the present invention, a system is provided for performing an extract-transform-load (ETL) process. The system includes a processor, configured to collect load information about a volume and a complexity of raw data to be processed during the ETL process. The processor is further configured to receive an expected completion time of the ETL process and execution information about (i) hardware resources and (ii) an influence of the hardware resource on an execution time of the ETL process. The processor is also configured to calculate resources for a distributed processing software infrastructure to be used to perform the ETL process, by applying a statistical method to the load information, the expected completion time, and the execution information. The processor is additionally configured to dynamically assign cloud resources corresponding to and based on the calculated resources, in accordance with the expected completion time. The system also includes a database configured to store results of the ETL process performed on the raw data using the assigned cloud resources.

DETAILED DESCRIPTION

The present invention is directed to managing a software asset environment using a cognitive distributed cloud infrastructure.

The Extract, Transform and Load (ETL) process is crucial for Software Asset Management. The ETL process aggregates raw data such as information about hardware and software components, and creates correlated data picture that helps customers better understand what they have installed in their IT infrastructure. However, an ETL import conventionally takes a lot of time to complete and consumes a lot of hardware resources.

In an embodiment, a process is provided to facilitate the concept of a dynamically allocated distributed processing software such as Hadoop® to leverage the benefits of the Software as a Service (SaaS) ETL import.

In the first stage of the process, the asset management infrastructure collects information about the volume of raw data that needs to be processed during the ETL. Such information allows for allocating a certain amount of distributed infrastructure resources (nodes, HDFS storage) which will perform the ETL.

In the next step, data is already inserted into the BigSql database from which we can extract additional information regarding aggregated data. This leverages the calculation that needs to be done during every new ETL import.

Moreover, it is to be appreciated that environment200described below with respect toFIG. 2is an environment for implementing respective embodiments of the present invention. Part or all of processing system100may be implemented in one or more of the elements of environment200.

Further, it is to be appreciated that processing system100may perform at least part of the method described herein including, for example, at least part of method300ofFIG. 3. Similarly, part or all of system200may be used to perform at least part of method300ofFIG. 3.

FIG. 2shows an exemplary environment200to which the present invention can be applied, in accordance with an embodiment of the present invention.

The environment200includes a set of clients210, a database220, distributed infrastructure resources230, a software asset management (SAM) tool (also interchangeably referred to herein as “software asset manager”)240, a resource calculator250, and an inventory monitoring system260.

The set of clients210are part of a network211. Other elements of the network211are not shown for the sake of brevity.

The inventory monitoring system260monitors the network211including the set of clients210. Of course, other entities can also be monitored, while maintaining the spirit of the present invention. The inventory monitoring system260can be an endpoint monitoring system (such as, e.g., but not limited to, BigFix® Inventory). The items that can be monitored by the inventory monitoring system260include, but are not limited to, the number of endpoints, the number of components installed at each of the endpoints, the hardware infrastructure complexity, the software catalog complexity, the scan frequency, the numbers of directories and files, number of installed products, type of bundlings used (confirmed/unconfirmed), the number of unconfirmed bundlings to total (unconfirmed and confirmed) bundlings, type of scoping (computer groups), type of licensing, and so forth. The preceding items are merely illustrative and can vary depending upon the implementation.

The inventory monitoring system260can include agents260A in each of the clients210in order to facilitate monitoring of the clients210.

The SAM tool240collects load information about a volume and a complexity of raw data to be processed during a Software as a Service (SaaS) ETL process (hereinafter “ETL process” in short). The SAM tool240receives an expected completion time of the ETL process and execution information about (i) hardware resources and (ii) an influence of the hardware resource on an execution time of the ETL process. The SAM tool240can receive the preceding information from the inventory monitoring system260or determine the preceding information from information received from the inventory monitoring system260.

The distributed infrastructure resources230includes a set of distributed processing nodes230A implemented in the cloud. The distributed infrastructure resources230performs the SaaS ETL on the raw data, in accordance with the present invention.

The database220stores the results of the ETL process. The database220also stores aggregated data, which can be used to leverage the calculations performed during every new ETL import. In an embodiment, the database220is a BigSql database. The database220can thus store the raw data that has been subjected to the ETL process, having been extracted from a different source(s), transformed into a form suitable for the database220, and thereafter saved in database220.

The resource calculator250receives user inputs such as, for example, hardware mapping information, an expected completion time of the ETL, and the time of a last ETL import. The resource calculator250calculates resources for the distributed processing software infrastructure230to use to perform the ETL process, by applying a statistical method to the load information, the expected completion time, and the execution information. The statistical method can also be applied to any other user inputs. The resource calculator250dynamically assigns cloud resources corresponding to and based on the calculated resources, in accordance with the expected completion time. Exemplary statistical methods that can be used include, but are not limited to, mean, simple moving average, standard deviation, self-learning neural networks, and so forth. It is to be appreciated that the present invention are not limited to any particular statistical methods and, thus, any type of statistical method can be used in accordance with the teachings of the present invention, while maintaining the spirit of the present invention.

While one or more elements may be shown as separate elements, in other embodiments, these elements can be combined as one element. For example, in another embodiment, the resource calculator250can be combined into a single entity with the SAM tool240. The converse is also applicable, where while one or more elements may be part of another element, in other embodiments, the one or more elements may be implemented as standalone elements. Additionally, one or more elements inFIG. 2may be implemented by a variety of devices, which include but are not limited to, Digital Signal Processing (DSP) circuits, programmable processors, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), Complex Programmable Logic Devices (CPLDs), and so forth. These and other variations of the elements of environment200are readily determined by one of ordinary skill in the art, given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

FIG. 3shows an exemplary method300for managing a software asset environment using a cognitive distributed cloud infrastructure, in accordance with an embodiment of the present invention.

In an embodiment, method300can be used to dynamically allocate resources used by a software framework for distributed storage and processing of very large data sets on computer clusters (such as, for example, Hadoop®), in order to leverage the benefits of a Software as a Service (SaaS) ETL process.

Since the process of optimizing allocation of resources is done in iterations, the following steps represent a selected nthiteration of a method in accordance with an embodiment of the present invention. The method is based on the neural network approach. In each iteration, the attributes/weights are automatically adjusted based on the results obtained from the previous iteration.

At step310, collect information about the volume of raw data to be processed during an ETL process. In an embodiment, the information is collected from a BigFix® infrastructure. In an embodiment, the information is downloaded from a server or other repository.

In an embodiment, the information about the volume of raw data can include, but is not limited to, one or more of the following parameters: (a) the number of endpoints; (b) the number of components installed on each of the endpoints; (c) the hardware infrastructure complexity, including/involving CPU, RAM, sockets information, virtualization, disk space, and so forth; (d) the software catalog complexity; (e) the scan frequency; and (f) the numbers of directories and files.

In an embodiment, a first run of step310can involve using default entries to fulfill user supplied or expected parameters. In an embodiment, subsequent runs of step310can involve using an aggregate of one or more of the preceding parameters.

At step310A, process the user input. The user input can provide the user with direct influence on the (desirable) time for performing the ETL process.

At step310B, map the hardware. In an embodiment, step310B can involve mapping the amount of time required by the hardware to process data of this type.

In an embodiment, step310B can involve, for example, an eXtensible Markup Language (XML) hashmap with a list of hardware specifications and their influences on ETL time execution. Thus, the user can define the so-called hash map of hardware information that introduces an additional level of customization. In the hash map, the user can define pairs of keys and values such as processor models and their expected ETL time execution in the overall ETL process, respectively. Taking into account that the ETL process is performed in an asynchronous manner and some of the aggregated data may be expected earlier to fulfill the third party business requirements, the customer can define the expected level for some groups of hardware that needs to be calculated in the given timeframe.

At step310C, calculate an expected time of the ETL accomplishment (i.e., expected ETL completion time).

At step320, determine the time of the last executed ETL import.

At step330, calculate the aggregate data. In an embodiment, step330can involve, but is not limited to, one or more of the following: (a) number of installed products (bundled with adequate components); (b) type of bundling (confirmed/unconfirmed); (c) period of calculated data; (d) catalog customization volume; (e) type of scoping (computer groups); (f) type of licensing; (g) number and type of contracts; and (h) time of the previous import.

At step340, calculate and assign cloud resources needed by distributed processing software infrastructure using statistical methods (based on steps310and320) to ensure the expected time of the ETL import. The cloud resources can relate to performing the ETL process as a service in the cloud (e.g., Software as a Service (Saas)).

In an embodiment, the input given by the user is treated as the main condition that needs to be fulfilled regarding step340. Presuming that the software framework (for example, Hadoop®) has unlimited resources to perform the ETL, by setting a lower ETL time execution, the customer automatically forces an increase in the number of resources that need to be used to perform the ETL. Thus, in the mathematical notation, when the expected ETL time tends to 0, the nR, which represents the number of resources in the software framework, tends to infinity.

Further regarding step340, having the expectations set, the method300assigns default attributes/weights that are specified in scalability guides specific to a given product to the collected raw data and maps them in a ratio (e.g., 1:1) to the number of units allocated in the software framework. For example, the number of endpoints is by default translated into the number of virtual units in the cloud infrastructure with the proportion that for1000endpoints we allocate one additional virtual unit in the cloud infrastructure. Similarly, the greater the number of software components that are discovered on the agent, then the greater the number of virtual software framework resources that need to be allocated. In the method300, it can be presumed that50software components are usually discovered on a single endpoint in the asset management IT infrastructure. Of course, other numbers can be presumed. Another raw factor that directly influences the number of virtual software framework units can be the frequency of the hardware/software data collection. The number of scans performed on the endpoints increases the overall total number of raw data that needs to be calculated during the ETL process, so it may have a direct volume relation.

Additionally regarding step340, the attributes and weights used for a current iteration can be adjusted based on the attributes and weights used for a prior iteration. Moreover, convergence or some other criteria (e.g., meeting a time threshold or resource threshold) can be used to determine which resource assignments are actually implemented. For example, iterating can be ceased upon the elapsing of a predetermined time period (time threshold) or upon consuming a predetermined amount of resources (resource threshold).

Also regarding step340, the cloud resources that can be assigned include, but are not limited to, computer applications, processing nodes, distributed file systems, and so forth. Different nodes can have different computer applications, processors, distributed file systems, and/or so forth.

At step350, perform an ETL import on the allocated resources.

At step360, gather the ETL results and update the data in the Software Asset Management (SAM) tool. In an embodiment, the ETL results are gathered into a BigSql database.

Further regarding step360, the same gathers the inputs with their corresponding weights and correlates the information in the summary block. In that step, some of the information is also correlated between the inputs themselves such as, for example, the number of endpoints, the scan frequency and the component numbers are strictly related to each other. This is done by the resource allocator summary block. In the end, the resource allocator summary block returns the number of virtual units that needs to be allocated in the Hadoop® infrastructure. If the difference between the ETL complete time and the time of a previous ETL is negative, then the additional number of Hadoop® units will be increased. The increment is of course done in conjunction with other inputs in order to calculate the increment requirement more precisely. Additionally in this step, an update is performed of the consecutive weights of the given inputs. The updating of the weights is done individually for each input and it is done based on the difference between the times of two previous ETL imports expressed in seconds. Presuming that the ETL import (per step350) is done asynchronously, each group of inputs will include information about the direct influence of the ETL execution time. This will simply imply that if some input volume was not changed in comparison to the two last ETL runs and still has a negative effect on the ETL execution time, its associated weight will be adequately adjusted. As an example, if we have2000endpoints in ETL run number1and ETL run number2but the individual ETL times for this part of the calculation took longer, then this will directly influence that the weight w1will be increased from1000to1500(The update of each individual weight may be done based on the pre-defined parameter, in this example by 50%).

Hence, in sum, method300can be considered to operate on two dimensions. One dimension relates to the input parameters which are usually expressed by volumes. The second dimension relates to the individual weights that, when correlated with a previous dimension, will have a direct influence on the overall ETL time. With each iteration, the weights can be adjusted while the input parameters are gathered from the asset management system.

FIG. 4shows some exemplary parameters400to which the present invention can be applied, in accordance with an embodiment of the present invention. InFIG. 4, raw data410is represented by circles, user inputs420are represented by rectangles, and aggregated data430is represented by octagons. The parameters400are provided to a resource allocator summary block450. The resource allocator summary block450can be implemented by the resource calculator250ofFIG. 2. The output from the resource allocator summary block450is a number of units451represented by a diamond-shaped block.

The raw data410includes: number of endpoints401; component numbers402; frequency scan403; catalog size volume404, number of files and directories405, and number of processor types (complexity of hardware infrastructure)406.

The user inputs420include: ETL time expectation given in seconds421; and custom ETL time expectation hash map with values expressed in seconds422.

The aggregated data430includes: the number of installed products431; the number of unconfirmed bundlings to total (unconfirmed and confirmed) bundlings432; a period of calculated data (e.g., expressed in number of days)433; the number of signatures in a custom catalog434; number of computer groups435; number of license types436, number of contract types437, time of previous ETL import (e.g., expressed in seconds)438.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG. 5, illustrative cloud computing environment550is depicted. As shown, cloud computing environment550includes one or more cloud computing nodes510with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone554A, desktop computer554B, laptop computer554C, and/or automobile computer system554N may communicate. Nodes510may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment550to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices554A-N shown inFIG. 5are intended to be illustrative only and that computing nodes510and cloud computing environment550can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Hardware and software layer660includes hardware and software components. Examples of hardware components include: mainframes661; RISC (Reduced Instruction Set Computer) architecture based servers662; servers663; blade servers664; storage devices665; and networks and networking components666. In some embodiments, software components include network application server software667and database software668.

Virtualization layer670provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers671; virtual storage672; virtual networks673, including virtual private networks; virtual applications and operating systems674; and virtual clients675.

Workloads layer690provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation691; software development and lifecycle management692; virtual classroom education delivery693; data analytics processing694; transaction processing695; and managing a software asset environment using a cognitive distributed cloud infrastructure696.