Methods and systems for integrating business intelligence data into continuous integration environments

A method and system for integrating business intelligence data into a continuous integration system are provided. In one embodiment, a method is provided comprising receiving business intelligence data regarding utilized product features and job data regarding tested product features. The method may further include converting the business intelligence data and the job data into unified business intelligence data and unified job data in a unified format. The method may also include generating a feature map based on the unified business intelligence data and unified job data, determining an insufficient mapping based on the feature map, and identifying a job flag based on the insufficient mapping.

BACKGROUND

Continuous integration development systems are designed to regularly integrate and deploy programs for users. Such continuous integration systems reduce the time required to develop and deploy new features and increase access to newly-developed features for users. In order to continuously deploy newly-developed features or updates, code written by developers is regularly tested before deployment. These developers also design and create tests for program features to run before deployment.

SUMMARY

The present disclosure presents new and innovative methods and system for integrating business intelligence data into continuous integration environments. In one embodiment, a method is provided comprising receiving business intelligence data comprising data regarding a plurality of utilized product features, receiving job data comprising one or more jobs, wherein each job relates to at least one of a plurality of tested product features, converting the business intelligence data into unified business intelligence data in a unified format, and converting the job data into unified job data in the unified format. The method may further comprise generating a feature map between the utilized product features and the tested product features based on the unified business intelligence data and the unified job data, determining an insufficient mapping of the feature map between a tested product feature and at least one of the plurality of utilized product features, and identifying at least one job flag based on the insufficient mapping.

In another embodiment, the at least one job flag comprises an unnecessary job related to a tested product feature with no corresponding utilized product feature in the feature map.

In yet another embodiment, the business intelligence data further comprises data regarding a plurality of developed product features and the at least one job flag comprises a stale job related to a tested product feature with no corresponding developed product feature in the feature map.

In a further embodiment, the at least one job flag comprises a wasteful job containing configured to execute at a frequency that exceeds a threshold set based on the number of utilized product features in the feature map corresponding to the tested product feature.

In a still further embodiment, the at least one job flag comprises a missing job corresponding to a utilized product feature with no corresponding tested product feature in the feature map.

In another embodiment, the method further comprises removing the at least one job flag from a continuous integration environment corresponding to the job data.

In yet another embodiment, the method further comprises generating a job recommendation based on the at least one job flag.

In a further embodiment, the job recommendation includes one or more recommendations selected from the group consisting of: (i) removing a flagged job associated with at least one job flag, (ii) adjusting an execution frequency of the flagged job, and (iii) creating a new job to test a utilized product feature.

In a still further embodiment, converting the job definition data into the unified job definition data comprises analyzing one or both of (i) a job name of a job and (ii) a job parameter of a job to identify a tested product feature.

In another embodiment, the business intelligence data comprises one or more of the data types consisting of: customer support data, customer relationship management (CRM) data, code submission data, and customer sales orders.

In yet another embodiment, the business intelligence data comprises a plurality of data types, and converting the business intelligence data into the unified business intelligence data comprises converting data of each of the plurality of data types into converted data in the unified format, and removing duplicate entries within the converted data.

In a further embodiment, the business intelligence data represents one or both of developer behavior regarding the product feature and customer behavior regarding the product feature.

In a still further embodiment, a system is provided comprising a processor and a memory. The memory may contain instructions which, when executed by the processor, cause the processor to receive business intelligence data comprising data regarding a plurality of utilized product features, receive job data comprising one or more jobs, wherein each job relates to at least one of a plurality of tested product features, convert the business intelligence data into unified business intelligence data in a unified format, and convert the job data into unified job data in the unified format. The memory may contain further instructions which, when executed by the processor, cause the processor to generate a feature map between the utilized product features and the tested product features based on the unified business intelligence data and the unified job data, determine an insufficient mapping of the feature map between a tested product feature and at least one of the plurality of utilized product features, and identify at least one job flag based on the insufficient mapping.

In another embodiment, the at least one job flag comprises an unnecessary job related to a tested product feature with no corresponding utilized product feature in the feature map.

In yet another embodiment, the business intelligence data further comprises data regarding a plurality of developed product features and the at least one job flag comprises a stale job related to a tested product feature with no corresponding developed product feature in the feature map.

In a further embodiment, the at least one job flag comprises a wasteful job configured to execute at a frequency that exceeds a threshold set based on the number of utilized product features in the feature map corresponding to the tested product feature.

In a still further embodiment, the at least one job flag comprises a missing job corresponding to a utilized product feature with no corresponding tested product feature in the feature map.

In another embodiment, the memory contains further instructions which, when executed by the processor, cause the processor to remove the at least one job flag from a continuous integration environment corresponding to the job data.

In yet another embodiment, the memory contains further instructions which, when executed by the processor, cause the processor to generate a job recommendation based on the at least one job flag.

In a further embodiment, a non-transitory, computer-readable medium is provided storing instructions which, when executed by a processor, cause the processor to receive business intelligence data comprising data regarding a plurality of utilized product features, receive job data comprising one or more jobs, wherein each job relates to at least one of a plurality of tested product features, convert the business intelligence data into unified business intelligence data in a unified format, and convert the job data into unified job data in the unified format. The non-transitory, computer-readable medium may store further instructions which, when executed by the processor, cause the processor to generate a feature map between the utilized product features and the tested product features based on the unified business intelligence data and the unified job data, determine an insufficient mapping of the feature map between a tested product feature and at least one of the plurality of utilized product features, and identify at least one job flag based on the insufficient mapping.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Because developers are typically responsible for creating and adding jobs for testing program features to the continuous integration environment, it is typically easy for such developers to add jobs to the continuous integration environment. However, after the development efforts have ceased for certain features, the jobs associated with testing those features may remain in the continuous integration environment for several reasons. For example, a developer that has created and added a job to a continuous integration environment may forget to remove the job once development efforts for that feature have completed. Over time, the number of forgotten or otherwise unused jobs in a continuous integration environment may rise to a large number (e.g., hundreds or thousands of unnecessary jobs) that waste system resources. Such a large number of unnecessary jobs can cause significant delays (e.g., up to several hours or more) in testing new features before deployment. Such delays may seriously delay deployment of new features and may undermine the “continuous” nature of continuous integration systems.

Additionally, sometimes developers may not recognize that certain product features are proving popular with users and may not engage in actively developing or testing such features. This lack of realization on the part of developers may lead to inefficiently-allocated development efforts or future bugs. For example, without identifying which product features are most important to customers, developing efforts may be misallocated to unimportant or unused product features. In another example, if robust testing jobs are not created for popular product features, certain implementations may go untested, leading to bugs in future releases of such product features.

One way of solving the above-identified problems is to access and analyze both business intelligence (BI) data and job data to determine which product features are currently utilized frequently by users and which product features are currently being tested regularly in the continuous integration environment. The business intelligence data may include one or more of customer relationship management (CRM) data, customer support data, code submission data, and customer sales order data. Both the business intelligence data and the job data may be converted to a unified format and used to generate a feature map. The feature map may include mappings between utilized product features identified from the business intelligence data and test product features identified from the job data. For example, the feature map may include mappings between utilized product features and corresponding tested product features. The feature map may then be analyzed in order to determine an insufficient mapping. The insufficient mapping may include, e.g., a missing utilized product feature missing from the business intelligence data suggesting a potentially unnecessary job that may be wasting continuous integration environment resources or a missing tested product feature missing from the job data suggesting a potentially untested product feature that may create risks for future developments. If an insufficient mapping is determined, a job flag may then be identified based on the insufficient mapping for future processing. This future processing may include generating a recommendation for a user or automatically removing or adjusting unnecessary or wasteful jobs. Such an implementation may help identify and remove wasteful or inefficient jobs within the continuous integration environment, thereby improving hardware utilization by reducing processor and memory use and requirements, while also improving the testing frequency of product features that are regularly utilized by users.

FIG. 1depicts a system100according to an exemplary embodiment of the present disclosure. The system100may be used to integrate business intelligence data into a continuous integration workflow to identify inefficient for missing jobs within the continuous integration system.

The system100includes a continuous integration environment102, a business intelligence system118, and a business intelligence integrator134. The continuous integration environment102includes a job storage storing jobs106,108. The job108includes validation tasks110,112. The continuous integration environment102also includes a CPU114and a memory116. The business intelligence system118includes a business intelligence data storage120, a CPU130, and a memory132. The business intelligence data storage120stores customer support data122, CRM data124, code submission data126, and sales order data128. The business intelligence integrator134includes a business intelligence data converter136storing a unified business intelligence data138, a job data converter140storing a unified job data142, a feature mapper144storing a feature map146, and a job flag identifier148storing an insufficient mapping158and job flags150,152. The business intelligence integrator134also includes a CPU154and a memory156.

The continuous integration environment102may be configured to perform jobs106,108to test product features being developed. For example, the jobs106,108may be created by developers to test product features before deployment for utilization by users. To create the jobs106,108, the developers may create one or more validation tasks110,112to verify one or more aspects of the tested product features before deployment. The tested product features may include new product features, old product features, and/or updated versions of previous product features. The jobs may test the product features in certain implementation environments, certain operational environments, and/or interactions with other product features.

The job storage104stores the jobs106,108for use by the continuous integration environment102in testing the tested product features. The continuous integration environment102may be configured to execute the jobs106,108based on one or more triggering actions. For example, the triggering actions may include an action-based trigger triggered by one or more developer actions (e.g., the developer checking in a code base). In another example, the triggering actions may include a time-based trigger activated at a regular time frequency (e.g., every day, every week, or every month). In a further example, the triggering actions may include a manual trigger (e.g., a user request to perform a job106,108).

The validation tasks110,112may include, e.g., copying a code base, compiling a code base, performing one or more unit tests with the compiled code base. The validation tasks110,112and/or the jobs106,108may correspond to one or more product features (i.e., tested product features). For example, the job106may correspond to a single tested product feature. In another example, the job108may correspond to more than one tested product feature (e.g., validation task110may correspond to a first tested product feature and validation task112may correspond to a second tested product feature).

During execution in the continuous integration environment102, the jobs106,108may consume resources and may execute in a queue according to a priority and/or a triggering time of the job106,108. During development, code bases associated with pending jobs106,108may be prevented from deployment for utilization by a user until all associated jobs106,108have been completed successfully within the continuous integration environment102. Therefore, unnecessary and numerous jobs106,108in the execution queue may prevent code bases from being tested in a timely manner for quick deployment, as discussed above.

The business intelligence system118may be configured to store one or more types of business intelligence data. The business intelligence data storage120may be configured to store business intelligence data of multiple types, including customer support data122, CRM data124, code submission data126, and sales order data128(collectively, business intelligence data122,124,126,128). The customer support data122may include customer support tickets, customer support incidents, or other customer support reports (e.g., SOS reports from customers). The sales order data128may include information such as the name and quantity of software products, including one or more specific product features purchased or accessed by users and customers. The CRM data124may include one or more pieces of customer information, including products or features accessed, purchased, or utilized by users and/or customers. The CRM data124may also include user customer interaction and communication records. In certain implementations, the CRM data124may include the customer support data122and the sales order data128. The code submission data126may include data regarding development efforts for software products and product features. For example, the code submission data126may include data regarding code commits for code bases, features and functions associated with code commits, and a number or indication of developers accessing and committing code bases associated with certain product features. In certain implementations, one or more of the customer support data122, the CRM data124, the code submission data126, and the sales order data128may be stored in separate systems based on data type. For example, the CRM data124may be stored in a CRM system, such as a CRM system provided by Salesforce®. Similarly, the code submission data126may be stored in a code management system, such as a code management system provided by GitHub®. In such implementations, the business intelligence data122,124,126,128may be accessed, e.g., via an application programming interface (API) of the associated system.

In certain implementations, data of the same type may also be stored in multiple databases (e.g., SQL databases). For example, data of the same type may be stored on separate databases organized by the product features or software product associated with the business intelligence data122,124,126,128.

The business intelligence integrator134may be configured to combine job data regarding the jobs106,108stored within the job storage with business intelligence data122,124,126,128stored within the business intelligence data storage120. The business intelligence integrator134may then analyze the job data and business intelligence data122,124,126,128to identify job flags associated with stale jobs, wasteful jobs, unnecessary jobs, or missing jobs.

The business intelligence data converter136may convert business intelligence data122,124,126,128into unified business intelligence data138in a unified format. To do so, the business intelligence data converter136may identify one or more utilized or developed product features associated with the business intelligence data122,124,126,128. For example, the business intelligence data converter136may analyze customer support orders stored within the customer support data122to identify utilized product features as the product features with a number of associated customer support orders that exceed a certain threshold. As another example, the business intelligence data converter136may analyze sales order data128to identify utilized product features as the product features with a number of associated sales orders that exceed a certain threshold. As a further example, the business intelligence data converter136may analyze code submission data126to identify developed product features as the product features with code commits or associated developers above a certain threshold. In generating the unified business intelligence data138, the business intelligence data converter136may utilize one or more heuristics or machine learning algorithms, as detailed further below.

The job data converter140may be configured to convert job data concerning jobs106,108from the job storage104into unified job data142. The unified job data142may identify one or more tested product features associated with the jobs106,108. The job data converter140may use one or more heuristics or machine learning algorithms in generating the unified job data142. For example, the heuristics of machine learning algorithms may analyze one or more of the job name, job parameters, and validation tasks110,112of the jobs106,108in generating the unified job data142. In certain implementations, the unified job data142may be in the same unified format as the unified business intelligence data138.

The feature mapper144may be configured to map between utilized and developed product features identified in the unified business intelligence data138and tested product features identified in the unified job data142in order to generate a feature map146. In generating the feature map, the feature mapper144may identify corresponding features between the utilized and developed product features and the tested product features. For example, the feature mapper144may identify a product feature of a software product common between a developed or utilized product feature and a tested product feature and may generate a mapping between the two features indicating that they correspond to the same product feature.

The job identifier148may be configured to determine insufficient mapping158based on the feature map146and may generate one or more job flags150,152based on the insufficient mapping. For example, the job flag identifier148may identify an insufficient mapping158of the feature map146as a missing mapping between a developed or utilized product feature and a tested product feature. For example, the job flag identifier148may analyze each developed product feature and/or each utilized product feature of the feature map146to identify the developed or utilized product features that lack a corresponding tested product feature. Similarly, the job flag identifier148may analyze each tested product feature to identify the tested product features that lack a corresponding developed or utilized product feature. In such cases where a tested product feature lacks a developer utilized product feature, or vice versa, the job flag identifier148may identify an insufficient mapping158.

After identifying the insufficient mapping158, the job flag identify148may identify one or more job flags150,152associated with the insufficient mapping158. For example, the job flag150,152identified may depend on the type of insufficient mapping158. The job flag identifier148may identify a job flag150,152associated with an unnecessary job for jobs testing a tested product feature with no corresponding utilized product feature. The job flag identifier140may also identify a job flag associated with the stale job for jobs106,108testing a tested product feature with no corresponding developed product feature, or may identify a job flag150,152associated with a wasteful job for jobs106,108testing a tested product feature with an execution frequency greater than a predefined threshold, as explained further below. As a further example, the job flag identifier148may also identify a job flag150,152associated with the missing job for utilized or developed product features with no corresponding tested product feature. In implementations where more than one insufficient mapping158is identified in the feature map146, the job flag identifier148may not identify a job flag150,152for each insufficient mapping158identified within the feature map146. For example, the job flag identifier148may identify job flag150,152for the insufficient mappings158corresponding to tested, developed, or utilized product features with a certain priority, which may be determined based on business intelligence data122,124,126,128or execution frequency. For example, more popular product features utilized by users may receive a higher priority. As another example, jobs106,108with a higher execution frequency may be granted a higher priority. As will be explained further below, the job flag150,152identified by the job flag identifier148may be subsequently used to generate a recommendation for a user, to remove jobs106,108from the job storage104, or to adjust associated jobs106,108(e.g., adjusting the execution frequency).

One or more of the components of the system100may be connected over a network such as the Internet or a local area network. For example, as depicted, the continuous integration environment102may be connected to the business intelligence integrator134over a network, and the business intelligence system118and the business intelligence integrator134may be connected over a network. In certain implementations, the continuous integration environment102and the business intelligence system118may be connected to the business intelligence integrator134via the same network. The connections may be implemented by one or more physical or wireless interfaces, e.g., Ethernet, Wi-Fi, cellular data, and Bluetooth connections.

One or more of the continuous integration environment102, the business intelligence integrator134, and the business intelligence system118may be implemented by a computer system. In particular, the CPU114and the memory116may implement one or more features of the continuous integration environment102, the CPU130and the memory132may implement one or more features of the business intelligence system118, and the CPU154and memory156may implement one or more features of the business intelligence integrator134. For example, the memories116,132,156may store instructions which, when executed by the CPUs114,130,154, cause the CPUs114,130,154to perform one or more features of the continuous integration environment102, the business intelligence system118, or the business intelligence integrator134.

In addition, each of the continuous integration environment102, the business intelligence system118, and the business intelligence integrator134may be implemented by one or more computer systems. For example, the business intelligence system118may comprise a plurality of separate computer systems responsible for creating and storing business intelligence data122,124,126,128of different types, as explained above. Similarly, the continuous integration environment102may comprise a plurality of computing systems responsible for implementing, e.g., the job storage104. In such implementations, the CPUs114,130,154and the memories116,132,156may also be implemented by a plurality of CPUs and memories.

FIG. 2depicts a feature map200according to an exemplary embodiment of the present disclosure. The feature map200may be an example of the feature map146generated by the feature mapper144. The feature map200may be generated based off of job data stored in the job storage104and business intelligence data122,124,126,128stored in the business intelligence data storage120. The feature map200includes a plurality of jobs202,218associated with tested product features204,220. The feature map200also includes business intelligence data206,214associated with utilized product features208,216. As an example, the business intelligence data206,214may include one or more of customer support data122, CRM data124, and sales order data120. The feature map200further includes code submission data (e.g., code submission data126) associated with a developed product feature212.

The feature map200further depicts a mapping between the tested product feature204and the utilized product feature208. The mapping may indicate that the tested product feature204and the utilized product feature208concern the same product feature of a software product. For example, both the tested product feature204and the utilized product feature208may concern a networking feature of the software product (e.g., a count of the nodes in which a software program can be implemented, an IP address protocol used by the software product).

However, the developed product feature212, the utilized product feature216, and the tested product feature220as depicted do not have an associated product feature and may therefore lack a mapping. When analyzing the feature map200, the job flag identifier148may therefore determine an insufficient mapping158for these features212,216,220. For example, because the developed product feature212lacks a corresponding tested product feature, the job flag identifier148may determine an insufficient mapping158for the developed product feature212and may identify a job flag150,152for the insufficient mapping158indicative of a missing job that should be included within the continuous integration environment102. Similarly, because the utilized product feature216lacks an associated tested product feature, the job flag identifier148may determine insufficient mapping158for the utilized product feature216and may identify a job flag150,152for the insufficient mapping158indicative of missing job that should be included within the continuous integration environment102. As another example, because the tested product feature220lacks a corresponding developed or utilized product feature, the job flag identifier148may identify insufficient mapping158for the tested product feature220and may identify a job flag150,152for the insufficient mapping158indicative of an unnecessary or stale job that should be altered or removed from the continuous integration environment102. In certain instances, the job flag identifier148may also identify a job flag150,152associated with the tested product feature204and the utilized product feature208, even though a mapping exists between the features. For example, if an execution frequency of the job202exceeds a certain threshold (e.g., a threshold based on the importance of the tested product feature204), the job flag identifier140may identify a job flag150,152indicative of a wasteful job that executes too frequently within the continuous integration environment102, and may recommend that the execution frequency needs to be adjusted.

FIG. 3depicts a method300according to an exemplary embodiment of the present disclosure. The method300may be performed to access and analyze job data and business intelligence data122,124,126,128in order to identify insufficient mappings158and/or job flags150,152within a continuous integration environment102. The method300may be performed by one or more components of the system100. For example, one or more of the continuous integration environment102, the business intelligence system118, and the business intelligence integrator134may perform one or more aspects of the method300.

The method300may be implemented on a computer system, such as the system100. For example, method300may be implemented by the continuous integration environment102, the business intelligence system118, and/or the business intelligence integrator134. The method300may also be implemented by a set of instructions stored on a computer readable medium that, when executed by a processor, cause the computer system to perform the method. For example, all or part of the method300may be implemented by the CPUs114,130,154and the memories116,132,156. Although the examples below are described with reference to the flowchart illustrated inFIG. 3, many other methods of performing the acts associated withFIG. 3may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, one or more of the blocks may be repeated, and some of the blocks described may be optional.

The method300begins with receiving business intelligence data (block302). For example, the business intelligence integrator134may receive business intelligence data including customer support data122, CRM data124, code submission data126, and sales order data128from a business intelligence system118. In particular, the business intelligence integrator134may receive the business intelligence data from a business intelligence data storage120of the business intelligence system118, e.g., over a network connection to the business intelligence system118. The business intelligence data122,124,126,128,206,214may come in one or more formats including JSON and YAML formats. For example, in implementations where the customer support data122, CRM data124, code submission data126, and sales order data128are stored in separate systems, each of the customer support data122, the CRM data124, the code submission data126, and the sales order data128may come in different formats. As another example, business intelligence data122,124,126,128,206,214regarding different program features or software products may come in different formats.

The business intelligence integrator may then receive job data (block304). The job data may include data regarding one or more jobs106,108stored on a job storage104of a continuous integration environment102. The job data may be received from the job storage104, e.g., over a network connection with the continuous integration environment102. Similar to the business intelligence data122,124,126,128,206,214, the job data may arrive in one or more formats, e.g., JSON and YAML formats.

The business intelligence integrator may then convert the business intelligence data into unified business intelligence data (block306). For example, a business intelligence data converter136of the business intelligence integrator134may convert the business intelligence data122,124,126,128,206,214into unified business intelligence data138. As mentioned above, the business intelligence data converter136may rely on one or more heuristics to identify utilized product features208,216and/or developed product features212associated with the business intelligence data122,124,126,128,206,214in order to convert the business intelligence data122,124,126,128,206,214into the unified business intelligence data130a. To generate these heuristics, the business intelligence data converter136may receive feedback from developers regarding which aspects of the business intelligence data122,124,126,128,206,214are associated with developed product features212or utilized product features208,216. For instance, the developers or users may identify which columns of the database correspond to features or parameters relevance to determining and associated developed product feature212or utilized product feature208,216. As an example, the columns may identify a corresponding software product or implementation details regarding the software products (e.g., an operating system, network nodes required, hardware architecture, processed data types, enabled or disabled product features). Additionally, the business intelligence data converter136may rely on one or more machine learning models. The machine learning models may be trained on business intelligence data122,124,126,128,206,214relating to previously-identified developed product features212or utilized product features208,216. In certain implementations, the machine learning model may be preferentially used process free text portions of the business intelligence data122,124,126,128,206,214, e.g., one or more fields of a user request or customer support entry.

The business intelligence integrator may then convert to the job data into unified job data (block308). For example, the job data converter140of the business intelligence integrator134may convert the job data into unified job data142. As mentioned above, the job data converter140may use one or more heuristics to identify one or more tested product features204,220associated with the jobs106,108,202,218indicated in the job data in order to convert to the job data into unified job data142. In generating these heuristics, the job data converter140may rely on user input regarding job names and/or job parameters. The job data converter140may receive user input identifying one or more naming conventions used to name jobs106,108,202,218within the continuous integration environment102. For example, in certain implementations testing networking features, job names may indicate to the version of the Internet protocol (IP) being tested. In such examples, e.g., a job name entitled “testing-ipv6-HA” may indicate that the job106,108,202,218is testing a tested product feature204,220with IP version 6. As another example, jobs106,108,202,218may test software product's deployment across a plurality of nodes. In such examples, e.g., a job name entitled “05.job_name” may indicate that the job106,108,202,218is testing a tested product feature204,220relating to the software product's deployment across 5 nodes. In such examples, the user may provide the above naming conventions. For example, the user may provide a rule that any job106,108,202,218containing “ipv #” is testing a tested product feature204,220relating to an IP protocol, with the “#” character identifying the specific IP version being tested. As another example, the user may provide a rule such as, any job name preceded by two numerical characters (e.g., “##.job_name”) is testing a tested product feature204,220for distributed deployment across a plurality of nodes, with the two numerical characters identifying the number of nodes. After receiving the user input, the job data converter140may store the user input and/or the user-provided rules in a database for use in converting future job data into unified job data142. In implementations with a plurality of continuous integration environments102, or for a plurality of software products are developed, the job data converter140may use a separate set of rules for data from each continuous integration environment102, or for each software product. Additionally, the job data converter140may include one or more machine learning models trained on previously-converted job data and previously-identified tested product features204,220. The machine learning models may perform best when analyzing one or both of the job name and a job data of jobs106,108,202,218stored in the job storage104. For example, rather than relying on user input, the machine learning model may determine based on previous provided examples that jobs106,108,202,218named according to the convention “_.ipv #._” are testing the IP protocol, with the version identified by the “#” character. The machine learning model may determine this by identifying that this naming convention is strongly correlated with IP testing in the provided previous examples. Similarly, the machine learning model may determine that numerical characters proceeding a job name are strongly correlated with jobs106,108,202,218testing tested product features relating to the number of nodes in a distributed deployment of the software product.

The business intelligence integrator may then generate a feature map (block310). For example, the feature mapper140for the business intelligence integrator134may generate a feature map146,200. The feature map146may be generated based on the unified business intelligence data138and the unified job data142. For example, the feature map146,200may be generated based on the developed product features212or utilized product features208,216identified in the unified business intelligence data138and the tested product features204,220identified in the unified job data142. As explained above, in generating the feature map146,200, the feature mapper144may analyze each tested product feature204,220for a corresponding developed product feature212or utilized product feature208,216. Similarly, the feature mapper144may analyze each developed product feature212and utilized product feature208,216for a corresponding tested product feature204,220. In the preceding analysis, where the feature mapper144detects a corresponding product feature, the feature mapper144may generate a mapping between the tested product feature204,220and the corresponding utilized product feature208,216or develop product feature212. In certain implementations, the feature mapper144may count the number of corresponding product features and may generate a stronger mapping for larger numbers of corresponding product features, or may only generate a mapping if the number of corresponding product features exceed a threshold.

The business intelligence integrator may then determine insufficient mapping (block312). For example, the job flag identifier148of the business intelligence integrator134may identify an insufficient mapping158based on the feature map146,200. An insufficient mapping158may be determined as any developed product feature212or utilized product feature208,216that lacks a tested product feature204,220, or vice versa. An insufficient mapping158may also be determined as a mapping that is not strong enough given the number of product features in the database. For example, a given feature of the software product may have multiple utilized product features208,216within the unified business intelligence data138, signifying large user interest in the product feature. However, the utilized product features208,216may not have sufficient corresponding tested product features204,220within the unified job data142. In such implementations, although there may be a mapping corresponding to the utilized product features208,216, that mapping may be determined to be too weak based on the number of corresponding utilized product features208,216(e.g., based on a threshold calculated based on the number of corresponding utilized product features208,216). Similarly, an insufficient mapping150may be determined where the mapping is too strong between a tested product feature204,220and utilized product features208,216. For example, there may be multiple, similar jobs106,108testing a tested product feature204,220that has a corresponding utilized product feature208,216with minimal entries within the unified business intelligence data138. Because of the relative lack of importance of the corresponding utilized product feature208,216, the mapping between the tested product feature204,220and the utilized product feature208,216may be determined to be too strong (e.g., in excess of a threshold calculated based on the number of corresponding utilized product features208,216).

The business intelligence integrator may then identify a job flag (block314). For example, job flag identifier148of the business intelligence integrator134may identify one or more job flags150,152based on the insufficient mapping158. As explained above, the insufficient mapping150may relate to one or more issues with a job106,108,202,218. For example, job flag150,152may be generated to identify one or more of an unnecessary job, a stale job, wasteful job, or a missing job.

The job flag150,152may then be used for further processing. For example, the job flag identifier148and/or the business intelligence integrator134may generate a recommendation and presented to the user (e.g., present a user alert or email the recommendation to the user). The recommendation may suggest removal for unnecessary jobs or stale jobs. As another example, the recommendation may suggest adjusting the execution frequency for wasteful jobs (e.g., reducing the execution frequency). Further, the recommendation may suggest creating a new job106,108,202,218for missing jobs (e.g., recommend creating a job to test the utilized or developed product feature). For example, if the business intelligence data122,124,126,128,206,214indicates that a lot of users are using 6 node deployments of a software product, but there are no corresponding tested product features204,220testing 6 node deployments, the job flag identifier may identify a job flag150,152indicating a missing job and the recommendation may suggest creating a new job106,108,202,218to test the software product using 6 nodes. As another example, if the business intelligence data122,124,126,128,206,214indicates that there are no users using a software product with more than 10 nodes, but there is a tested product feature204,220testing the software product with 11 nodes, the job flag identifier may identify the corresponding job as an unnecessary job and the recommendation may suggest removing the unnecessary job from the continuous integration environment102.

After receiving the recommendation, a user may be able to select an option to implement the recommendation (e.g., by creating and removing jobs106,108,202,218within the continuous integration environment102). In certain implementations, the recommendation may also include a hold button that allows the user to delay acting on the recommendation for certain period of time and to be reminded again of the recommendation in the future. In certain implementations, the business intelligence integrator134may automatically enact the recommendation, e.g., by automatically removing jobs106,108,202,218from the continuous integration environment102or automatically adjusting the execution frequency of wasteful jobs.

All or some of the blocks of the method300may be optional. Similarly, although depicted as occurring in sequence, one or more of the blocks may be performed in parallel. For example, blocks302and304may be performed in parallel (e.g., on multiple threads of a multi-threaded CPU114,130,154).

FIG. 4Adepicts a method400according to an exemplary embodiment of the present disclosure. The method400may be performed to determine insufficient mapping158and/or to identify a job play150,152within a feature map146,200. For example, the method400may be performed of blocks312,314of the method300. For example, method400may be performed after generating the feature map146,200at block310.

The method400may be implemented on a computer system, such as the system100. For example, method400may be implemented by the continuous integration environment102, the business intelligence system118, and/or the business intelligence integrator134. The method400may also be implemented by a set of instructions stored on a computer readable medium that, when executed by a processor, cause the computer system to perform the method. For example, all or part of the method400may be implemented by the CPUs114,130,154and the memories116,132,156. Although the examples below are described with reference to the flowchart illustrated inFIG. 4A, many other methods of performing the acts associated withFIG. 4Amay be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, one or more of the blocks may be repeated, and some of the blocks described may be optional.

The method400may begin by selecting a tested product feature for analysis (block402). For example, the job flag identifier148may select a tested product feature204,220from the feature map146,200for analysis. The job flag identifier148may select a first tested product feature204,220of the feature map146,200for analysis at an initial iteration of the method400, or may select a subsequent tested product feature204,220for analysis if the method400is being repeated, as explained below.

The job flag identifier may then determine whether there is a corresponding utilized product feature for the selected tested product feature (block404). For example, the job flag identifier148may determine whether there is a corresponding utilized product feature208,216for the selected tested product feature204,220. In making this determination, the job flag identifier148may determine whether there is a mapping in the feature map146,200between the selected tested product feature204,220and a utilized product feature208,216. If there is no corresponding utilized product feature, the job flag identifier may identify the selected tested product feature as corresponding to an unnecessary job (block406). For example, if there is no corresponding utilized product feature208,216, the job flag identifier148may identify a job flag150,152for the job106,108associated with the selected tested product feature204,220indicating that the associated job106,108is an unnecessary job. If there is a corresponding utilized product feature208,216, processing may continue.

The job flag identifier may then determine whether there is a corresponding developed product feature for the selected tested product feature (block408). For example, the job flag identifier148may determine whether there is a corresponding developed product feature212for the selected tested product feature204,220. In making this determination, the job flag identifier148may determine whether there is a mapping in the feature map146,200between the selected tested product feature204,220and a developed product feature212. For example, the job flag identifier148may determine whether there is code submission data126,210identifying a sufficient number of recent code check-ins or recent developer activity, where recent is identified as a certain time period (e.g., the preceding 30 days, 3 months, one year). As another example, the job flag identifier148may determine whether the mapping indicates a sufficient number of developers associated with recent code check-ins, which may be indicative of a level of importance of the underlying product feature. If there is no corresponding developed product feature212, the job flag identifier148may identify the selected tested product feature as corresponding to a stale job (block410). For example, if there is no corresponding developed product feature212, the job flag identifier148may identify a job flag150,152for the job106,108,202,218associated with the selected tested product feature204,220indicating that the job106,108,202,218is a stale job. If there is a corresponding developed product feature212, processing may continue.

The job flag identifier may then determine whether an execution frequency of a job corresponding to be selected tested product feature exceeds a threshold (block412). For example, the job flag identifier148may determine whether an execution frequency of a job106,108,202,218corresponding to the selected tested product feature204,220exceeds a maximum threshold. The maximum threshold may be set based on, e.g., a priority of the selected test product feature204,220, a utilization measure such as a count of the number of corresponding sales orders, customer support tickets, price, and development efforts associated with the selected tested product feature204,220. For example, the selected tested product feature204,220may correspond to a developed product feature212associated with code submission data126. The maximum threshold may be determined based on a number of code check-ins or associated developers indicated by the code submission data126. If the number of either the code check-ins or the associated developers is high, the selected tested product feature204,220may be considered high priority and the maximum threshold may be set higher to allow for more frequent testing of the selected testing product feature204,220(e.g., product features that are currently in active development). Conversely, if the number of both is low, the maximum threshold may be set lower to restrict the execution frequency of product features that are not currently in development in order to leave more computing resources for quickly testing actively-developed features. If the execution frequency is greater than the maximum threshold, the job flag identifier148may identify the selected tested product feature is corresponding to a wasteful job (block414). For example, if the execution frequency is greater than the maximum threshold, the job flag identifier148may identify a job flag150,152for the job106,108,202,218associated with the selected tested product feature204,220indicating that the job106,108,202,218is a wasteful job. If the execution frequency does not exceed the maximum threshold, processing may continue.

In certain implementations, a minimum threshold may be also or alternatively set indicating a desired minimum execution frequency for jobs associated with popular product features. For example, popular product features may be identified based on high utilization as indicated in business intelligence data122,124,126,128,206,214. The job flag identifier148may compare the execution frequency of a job106,108,202,218associated with the selected tested product feature204,220to ensure that the execution frequency is greater than the minimum threshold. If not, the job flag identifier148may identify a job flag150,152indicating that the corresponding job106,108,202,218is not being executed frequently enough. Checking against this minimum threshold may help ensure that popular product features are tested sufficiently to protect against unnecessary bugs and/or deployment issues during future development.

The job flag identifier may then determine whether there are additional tested product features within the feature map (block416). For example, the job flag identifier148may identify whether there are additional tested product features204,220within the feature map146,200for analysis. The method400may be repeated once for every tested product feature204,220within the feature map146,200.

All or some of the blocks of the method400may be optional. Similarly, although depicted as occurring in sequence, one or more of the blocks may be performed in parallel. For example, one or more of blocks404,408,412may be performed in parallel. Also, the order of the blocks may be changed. For example, the job flag identifier148may be configured to check for corresponding developed features212(block408) before checking for corresponding utilized features208,216(block404). Additionally, after each of blocks406,410,414, processing may continue with a subsequent decision block404,408,412. For example, after determining that a job106,108,202,218is an unnecessary job (block406), the job flag identifier148may continue with determining whether the selected tested product feature204,220has any corresponding developed product feature212(block408). In this way, a single job106,108,202,218may, e.g., be identified as both an unnecessary job and a stale job. Jobs that are identified by more than one job flag150,152may be more likely to be deleted or recommended for deletion by the business intelligence integrator134.

FIG. 420depicts a method420according to an exemplary embodiment of the present disclosure. The method420may be performed to analyze utilized product features of the feature map146,200to identify job planks150,152. The method420may be performed by the job flag identifier148at blocks312,314of the method300. For example, the method420may be performed after generating the feature map146,200.

The method420may be implemented on a computer system, such as the system100. For example, method420may be implemented by the continuous integration environment102, the business intelligence system118, and/or the business intelligence integrator134. The method420may also be implemented by a set of instructions stored on a computer readable medium that, when executed by a processor, cause the computer system to perform the method. For example, all or part of the method420may be implemented by the CPUs114,130,154and the memories116,132,156. Although the examples below are described with reference to the flowchart illustrated inFIG. 4B, many other methods of performing the acts associated withFIG. 4Bmay be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, one or more of the blocks may be repeated, and some of the blocks described may be optional.

The method420may begin with the job flag identifier selecting a utilized product feature or developed product feature for analysis (block422). For example, the job flag identifier148may select a utilized product feature208,216or developed product feature212from the feature map146,200for analysis. The job flag identifier148may select a first utilized product feature208,216or developed product feature212of the feature map146,200for analysis at an initial iteration of the method420, or may select a subsequent utilized product feature208,216or developed product feature212for analysis if the method420is being repeated, as explained below.

The job flag identifier may then determine whether there is a corresponding tested product feature for the selected utilized product feature or developed product feature (block404). The job flag identifier148may determine whether there is a corresponding tested product feature204,220for the selected utilized product feature208,216or developed product feature212. In making this determination, the job flag identifier148may determine whether there is a mapping in the feature map146,200between the selected utilized product feature208,216or developed product feature212and a tested product feature204,220. If there is no corresponding tested product feature, the job flag identifier may identify the selected utilized product feature or developed product features corresponding to a missing job (block426). For example, if there is no corresponding tested product feature204,220, the job flag identifier140may identify a job flag150,152indicating that there is a missing job that should be associated with the selected utilized product feature208,216or developed product feature212. If there is a corresponding tested product feature204,220, processing may continue.

The job flag identifier may then determine whether there are additional utilized product features or developed product features within the feature map (block428). For example, the job flag identifier148may identify whether there are additional utilized product features208,216or developed product features212for analysis. The method420may be repeated once for every utilized product feature208,216or developed product feature212within the feature map146,200.

The methods400,420may be performed separately or in combination. For example, the method400may be performed in conjunction with the method420in order to implement blocks312,314of the method300. In such implementations, the method400may be performed before, after, or in parallel with the method420.

Additionally, in both or either of the methods400,420, when determining whether there is a corresponding product feature (e.g., a corresponding utilized product feature208,216or a corresponding tested product feature214,220), the job flag identifier148may also be configured to analyze the strength of a mapping. For example, although a mapping may exist between, e.g., a tested product feature204,220and a utilized product feature208,216, the job flag identifier148may, in certain instances, determine that there is no corresponding utilized product feature208,216for the selected tested product feature204,220if the mapping between the two is not strong enough. For example, although mapping may exist, the job flag identifier148may determine that there is no corresponding utilized product feature208,216if the mapping between the utilized product feature208,216and a selected tested product feature204,220does not exceed a threshold. The threshold may be set based on, e.g., an importance of the selected product feature, resources used by a job106,108,202,218corresponding to the selected product feature, or another priority measure of the selected product feature. The threshold determination may vary across, e.g., software products and may depend on user input.

FIG. 5depicts a method500according to an exemplary embodiment of the present disclosure. The method500, when executed, may collect and analyze job data and business intelligence data122,124,126,128,206,214to identify one or more job flags150,152.

The method500may be implemented on a computer system, such as the system100. For example, method500may be implemented by the continuous integration environment102, the business intelligence system118, and/or the business intelligence integrator134. The method500may also be implemented by a set of instructions stored on a computer readable medium that, when executed by a processor, cause the computer system to perform the method. For example, all or part of the method500may be implemented by the CPUs114,130,154and the memories116,132,156. Although the examples below are described with reference to the flowchart illustrated inFIG. 5, many other methods of performing the acts associated withFIG. 5may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, one or more of the blocks may be repeated, and some of the blocks described may be optional.

The method500includes a job storage502, a business intelligence data storage504, a business intelligence data converter506, a job data converter508, a feature mapper510, and a job flag identifier524. The method500may be performed by the system100. For example, the job storage502may be an example implementation of the job storage104, the business intelligence data storage504may be an example implementation of the business intelligence data storage120, the business intelligence data converter506may be an example implementation of the business intelligence data converter136, the job data converter508may be an example implementation of the job data converter140, the feature mapper510may be an example implementation of the feature mapper144, and the job flag identifier524may be an example implementation of the job flag identifier148.

The method500may begin with the job storage502providing job data to the job data converter508(block512). Job data may include information, such as job names and job parameters regarding jobs106,108,202,218stored in the job storage502. The job data converter508may receive the job data (block514). As explained above, the job data provided by the job storage502may come in a first format. In addition, job storage502may store the job data in a plurality of formats. The job data converter508may therefore convert the job data from the first format into unified job data in a unified format (block516). For example, as expended above, the job data converter508may identify one or more tested product features204,220associated with jobs106,108,202,218stored within the job storage502. The job data converter508may analyze the job data using one or more heuristics and/or the machine learning models to identify the tested product features204,220. The feature mapper510may then receive the unified job data142from the job data converter508(block518).

The business intelligence data storage504may then provide business intelligence data122,124,126,128,206,214to the business intelligence data converter506(block520). The business intelligence data206,214may include one or more of customer support data122, CRM data124, code submission data126, and sales order data128. Depending on the implementation, the business intelligence data122,124,126,128,206,214may arrive in a plurality of formats. Therefore, after receiving the business intelligence data122,124,126,128,206,214(block522), the business intelligence data converter506may convert the business intelligence data122,124,126,128,206,214into unified business intelligence data138in the unified format (block526). In converting the business intelligence data122,124,126,128,206,214, the business intelligence data converter506may use one or more heuristics or machine learning models. As expended above, the heuristics may rely on input provided by users, such as business intelligence system118administrators. The feature mapper may then receive the unified business intelligence data138(block528).

In certain implementations, the blocks512-518may be performed after blocks520-528as depicted inFIG. 5. In other examples, blocks512-518and520-528may be performed in parallel.

After receiving the unified job data142in the unified business intelligence data138, the feature mapper510and then generate a feature map146,200(block530). The feature map146,200may identify one or more mappings between tested product features204,220and utilize product features208,216or developed product features212. In generating the feature map146,200, the feature mapper510may analyze each of the tested product features204,220for corresponding utilize product features208,216or corresponding developed product features212. Similarly, the feature mapper510may analyze each of the utilize product features208,216in the developed product features2124corresponding tested product features204,220.

The job flag identifier524may then receive the feature map146,200from the feature mapper510(block532). The job flag identifier524may analyze the feature map146,200to determine one or more insufficient mappings158(block534). For example, as explained above, the job flag identifier may identify insufficient mappings158where the feature map146,200lacks a mapping between tested product features204,220and a utilized product features208,216or developed product features212. Similarly, the job flag identifier524may identify insufficient mappings158where a mapping between the tested product feature and a utilized product feature208,216or developed product feature212is not strong enough. Based on the determined one or more insufficient mappings158, the job flag identifier524may identify one or more job flags150,152(block536). For example, as explained above, the job flag identifier524may identify job flags for wasteful jobs, unnecessary jobs, steel jobs, or missing jobs. The job flags150,152may be subsequently used to generate recommendations or to automatically manipulate, delete, or adjust jobs106,108,202,218within the job storage502.

FIG. 6depicts a system600according to an exemplary embodiment of the present disclosure. The system600includes a processor614and a memory626. The memory626contains instructions which, when executed by the processor614, cause the processor to receive business intelligence data610comprising data regarding a plurality of utilized product features612and receive job data604comprising one or more jobs606,608, where each job606,608relates to at least one of a plurality of tested product features602. The memory626also contains instructions which, when executed by the processor614, cause the processor614to convert the business intelligence data610into unified business intelligence data618in a unified format617, convert the job data604into unified job data616in the unified format617, and generate a feature map620between the utilized product features612and the tested product features602based on the unified business intelligence data618and the unified job data616. The memory626also contains instructions which, when executed by the processor614, cause the processor614to determine an insufficient mapping622of the feature map620between a tested product feature602and at least one of the plurality of utilized product features612and identify at least one job flag624based on the insufficient mapping622.

All of the disclosed methods and procedures described in this disclosure can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer readable medium or machine readable medium, including volatile and non-volatile memory, such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be provided as software or firmware, and may be implemented in whole or in part in hardware components such as ASICs, FPGAs, DSPs, or any other similar devices. The instructions may be configured to be executed by one or more processors, which when executing the series of computer instructions, performs or facilitates the performance of all or part of the disclosed methods and procedures.

It should be understood that various changes and modifications to the examples described here will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims.