Mining code expressions for data analysis

Techniques for computer software code analysis are disclosed. One or more data flows are generated, based on analyzing software code using static analysis. A data object is identified in the software code using the one or more data flows, the data object relating to a structured dataset. A correspondence between a code expression in the software code and a characteristic of the structured dataset is identified, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows. The code expression for the structured dataset is analyzed, based on the correspondence, including at least one of: (i) generating a software code recommendation engine based on the code expression and the structured dataset, or (ii) generating one or more lambda expressions for application to the structured dataset, based on the code expression.

BACKGROUND

The present invention relates to computer software code analysis, and more specifically, to analyzing code expressions for data analysis.

SUMMARY

Embodiments include a computer-implemented method. The method includes generating one or more data flows based on analyzing software code using static analysis. The method further includes identifying a data object in the software code using the one or more data flows, the data object relating to a structured dataset. The method further includes determining a correspondence between a code expression in the software code and a characteristic of the structured dataset, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows. The method further includes analyzing the code expression for the structured dataset, based on the correspondence, including at least one of: (i) generating a software code recommendation engine based on the code expression and the structured dataset, or (ii) generating one or more lambda expressions for application to the structured dataset, based on the code expression.

Embodiments further include a system, including a processor and a memory having instructions stored thereon which, when executed on the processor, performs operations. The operations include generating one or more data flows based on analyzing software code using static analysis. The operations further include identifying a data object in the software code using the one or more data flows, the data object relating to a structured dataset. The operations further include determining a correspondence between a code expression in the software code and a characteristic of the structured dataset, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows. The operations further include analyzing the code expression for the structured dataset, based on the correspondence, including at least one of: (i) generating a software code recommendation engine based on the code expression and the structured dataset, or (ii) generating one or more lambda expressions for application to the structured dataset, based on the code expression.

Embodiments further include a computer program product, including a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform operations. The operations include generating one or more data flows based on analyzing software code using static analysis. The operations further include identifying a data object in the software code using the one or more data flows, the data object relating to a structured dataset. The operations further include determining a correspondence between a code expression in the software code and a characteristic of the structured dataset, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows. The operations further include analyzing the code expression for the structured dataset, based on the correspondence, including at least one of: (i) generating a software code recommendation engine based on the code expression and the structured dataset, or (ii) generating one or more lambda expressions for application to the structured dataset, based on the code expression.

DETAILED DESCRIPTION

Structured data often includes labels for various aspects of the data. For example, comma-separated-value (CSV) files and javascript object notation (JSON) data structures frequently include names for attributes and columns. But to accessors of the data that are not subject matter experts, it can be unclear what can be done with the structured data and how others typically analyze or use the data. For example, data scientists building machine learning (ML) models to analyze a given dataset may not be subject matter experts in the data itself, and may not have domain knowledge in how the data can (or should) be manipulated to improve ML results.

In an embodiment, a corpus of code relating to the same domain as the structured data can be mined to identify the expressions, and types of expressions, commonly employed to use and analyze the data. This can be done using static analysis, without actually executing the code expressions. In an embodiment, these mined code expressions can be used to build a recommendation engine. The recommendation engine can provide recommendations to accessors of the data (e.g., non subject matter experts) about what the accessor might want to do with the data. Further, in an embodiment, the mined code expressions can be applied to the data (e.g., automatically applied) and the expression can be evaluated for their value in analyzing the data.

One or more techniques described below can be used to mine expressions. For example, data flows between programs can be generated using knowledge graphs and other suitable techniques. This is discussed further in U.S. Patent Pub. No. 2021/0173641 (hereinafter the “Dolby Patent Publication”), which is hereby incorporated by reference for its discussion of generating data flows.

In an embodiment, reads from and writes to objects in the programs can then be tracked. For example, invocations in programs that are involved in reading structured data can return data objects (e.g., containers). Reads from and writes to these data objects can be tracked, and used to identify key insights to data manipulation. These reads and writes can further be summarized (e.g., by column and code operation) and used to generate the recommendation engine or to generate abstract lambda expressions to apply to the data.

In an embodiment, mined code expressions can then be correlated with characteristics of the data. For example, a given column in a dataset can be correlated with code expressions relating to that column, and a user can be provided with recommendations on potential expressions to use with that column. As discussed further below, column names (and other aspects of the structured data) can be normalized to improve the correlation.

In an embodiment, one or more of these techniques have significant advantages over prior solutions. For example, code expressions can be mined based on analyzing data flows of the code, without actually running the code. This is extremely valuable for numerous reasons. As one example, mining code expressions based on data flows, without running the code, saves significant computational resources. Many code expression repositories are very large, and actually running the code would be very computationally expensive, and effectively impossible in some circumstances. Mining the code expressions without running the code saves these resources and opens up larger code repositories as candidates for mining. Further, running the code could open up security concerns, either from inadvertent security flaws in the existing code expressions or from malicious code included in the existing code expressions. Mining the code expressions without running the code avoids these security concerns.

As another example, code expressions can be mined for a given dataset based on the structure of the dataset (e.g., characteristics of the dataset, including column and attribute names), without requiring the actual dataset. This also has numerous advantages. For example, the structure of many datasets is readily available, while the underlying data is not available (e.g., the owner of the data makes the structure available but not the data itself). One or more techniques described herein allow for mining of expression that access this data, without requiring access to the actual data. As another example, data may include personally identifying information (PII) or other sensitive information. Mining code expressions without accessing the underlying data avoids any privacy concerns or problems, because the actual sensitive data is not used or accessed.

FIG.1illustrates a computing environment100for mining code expressions for data analysis, according to one embodiment. In an embodiment, a data repository110includes structured data (e.g., data with labels for various aspects of the data). For example, the data repository110can include CSV files, JSON objects, or any other suitable structured data. In an embodiment, the data repository110includes dataset metadata112. The dataset metadata112can describe the structure of the data, without including the actual underlying dataset values. For example, the dataset metadata112can include names for characteristics of the dataset, including names for attributes, names for columns or rows, or any other suitable label or other structure, without including the underlying data.

In an embodiment, one or more users102seek to access the data repository110using a communication network120. The communication network120can be any suitable communication network, including the Internet, a wide area network, a local area network, or a cellular network. The users102can access the communication network120through any suitable electronic or computing device, including a smartphone, a tablet, a laptop computer, a desktop computer, or any other suitable device. Further, the users102can access the communication network120using any suitable wired or wireless communication technique (e.g., an Ethernet connection, a WiFi connection, a cellular connection, or any other suitable network connection).

In an embodiment, the users102are not subject matter experts relating to the data stored in the repository110. A controller140can include an expression mining service142. The expression mining service142can facilitate generating one or more mined code expressions150(e.g., using a number of existing code expressions130), and providing the mined code expressions150to the users102. The controller140is discussed further, below, with regard toFIG.2.

For example, the existing code expressions130can be a corpus of code expressions previously used with the data stored in the data repository110. The existing code expressions130can be generated by other users (e.g., other than the users102) and can include data access methods132. The data access methods132can access the data in the data repository110, and generate corresponding data objects134(e.g., containers). As one example, the existing code expression can be stored in a publicly available repository (e.g., a GitHub® open source repository, a Kaggle® ML code repository, or any other suitable repository). As another example, a large entity (e.g., a corporation, a university, a research cooperative, or any other suitable entity) may maintain existing code expressions130. Users that are associated with the entity may be able to access the existing code expressions. These are merely examples.

In an embodiment, the expression mining service142can be a software service configured to identify the mined code expressions150from the existing code expressions130. For example, the expression mining service142can compute data flows in the existing code expressions130, identify data objects in the existing code expressions130(e.g., the data objects134), find read and write nodes in the existing code expressions130, and identify the mined code expressions150. This is discussed further, below, with regard toFIGS.3-6. The users102can then use the mined code expressions150to identify suitable code expressions to use when accessing the data repository110.

FIG.2is a block diagram of a controller140for mining code expressions for data analysis, according to one embodiment. The controller140includes a processor202, a memory210, and network components220. The memory210may take the form of any non-transitory computer-readable medium. The processor202generally retrieves and executes programming instructions stored in the memory210. The processor202is representative of a single central processing unit (CPU), multiple CPUs, a single CPU having multiple processing cores, graphics processing units (GPUs) having multiple execution paths, and the like.

The network components220include the components necessary for the controller140to interface with a suitable communication network (e.g., the communication network120interconnecting various components of the computing environment100illustrated inFIG.1, or interconnecting the computing environment100with other computing systems). For example, the network components220can include wired, WiFi, or cellular network interface components and associated software. Although the memory210is shown as a single entity, the memory210may include one or more memory devices having blocks of memory associated with physical addresses, such as random access memory (RAM), read only memory (ROM), flash memory, or other types of volatile and/or non-volatile memory.

The memory210generally includes program code for performing various functions related to use of the controller140. The program code is generally described as various functional “applications” or “modules” within the memory210, although alternate implementations may have different functions and/or combinations of functions. Within the memory210, the expression mining service142facilitates identifying mined code expressions (e.g., the mined code expressions150illustrated inFIG.1) from existing code expressions (e.g., the existing code expressions130illustrated inFIG.1). This is discussed further, below, with regard toFIGS.3-6.

While the controller140is illustrated as a single entity, in an embodiment, the various components can be implemented using any suitable combination of physical compute systems, cloud compute nodes and storage locations, or any other suitable implementation. For example, the controller140could be implemented using a server or cluster of servers. As another example, the controller140can be implemented using a combination of compute nodes and storage locations in a suitable cloud environment (e.g., as discussed further below). For example, one or more of the components of the controller140can be implemented using a public cloud, a private cloud, a hybrid cloud, or any other suitable implementation.

AlthoughFIG.2depicts the expression mining service142as being located in the memory210, that representation is also merely provided as an illustration for clarity. More generally, the controller140may include one or more computing platforms, such as computer servers for example, which may be co-located, or may form an interactively linked but distributed system, such as a cloud-based system, for instance. As a result, the processor202, and the memory210, may correspond to distributed processor and memory resources within the computing environment100. Thus, it is to be understood that the expression mining service142may be stored at any suitable location within the distributed memory resources of the computing environment100.

FIG.3is a flowchart300illustrating mining code expressions for data analysis, according to one embodiment. At block302an expression mining service (e.g., the expression mining service142illustrated inFIGS.1-2) generates data flows using static analysis. For example, the expression mining service can access a repository of existing code expressions (e.g., the existing code expressions130illustrated inFIG.1), and can generate data flows in the existing code expressions relating to a data repository (e.g., the data repository110illustrated inFIG.1) using static analysis. The expression mining service analyzes the code expressions, but does not actually execute the code expressions.

For example, as noted above the Dolby Patent Publication is incorporated by reference for its discussion of generating data flows. The Dolby Patent Publication describes “building a knowledge graph using a lightweight, generic code abstraction of computer code form multiple applications.” Dolby Patent Publication at ¶ [0001]. In an embodiment, the expression mining service uses one or more techniques described in the Dolby Patent Publication to generate a knowledge graph describing the data flows. For example, the knowledge graph can identify code methods being called, what those code methods are calling, and what arguments are being passed in the calls. An example of a knowledge graph is further described, below, with reference toFIG.6.

At block304, the expression mining service identifies a next data object (e.g., a next container). In an embodiment, method calls in code expressions that are involved in reading structured data return data objects. These data objects can be referred to as containers. For example, as illustrated inFIG.1, the existing code expressions130include data access methods132. These data access methods132read structured data (e.g., from the data repository110) and return one or more data objects. These objects are the data objects134. In an embodiment, any data object returned from a read of structured data can be a container. This is merely an example (e.g., using terminology from the Python® programming language), and any suitable terminology or technique can be used.

In an embodiment, the expression mining service identifies any object returned by a method accessing the structured data as a container, and selects the next container in this set of containers. Alternatively, the expression mining service identifies a subset of objects returned by methods accessing the structured data as containers, and selects the next container in this subset.

At block306, the expression mining service finds read and write nodes. For example, the expression mining service can track reads from, and writes to, the identified data object (e.g., the container). The expression mining service can first gather all reads from the identified data object. The expression mining service can then find descendants of the read nodes that lead to write nodes. The expression mining service can further gather expressions for these write nodes. This is discussed further, below, with regard toFIG.4.

At block308, the expression mining service identifies expressions. For example, the expression mining service can normalize expressions by call path. The expression mining service can further normalize the expressions using type inference. The expression mining service then groups common expressions by operator. This is discussed further, below, with regard toFIG.5.

At block310the expression mining service determines whether more data objects remain. If so, the flow returns to block304and the expression mining service identifies the next data object. If not, the flow proceeds to block312.

At block312the expression mining service acts on the expressions. In an embodiment, the expression mining service can create lambda expressions. For example, a lambda expression can be a block of software code that takes a parameter (e.g., relating to the data being analyzed) and returns a value (e.g., relating to analyzing the data). In an embodiment, the expression mining service can generate one or more lambda expressions using the expressions mined using the techniques described above in relation to blocks302-310.

Further, in an embodiment, the expression mining service can automatically apply the lambda expressions to the data. For example, as discussed above the expression mining service can generate one or more expressions that receive data as an input and generate an output by analyzing the data. The expression mining service can automatically apply these expressions to the data, to generate the outputs.

The expression mining service can also, in an embodiment, evaluate the value of an expression for a dataset (e.g., for building an ML model using the dataset). For example, one or more of the expressions identified at block308may have value in training (and using) an ML model. The expression mining service can use automated machine learning (AutoML) to identify the value of the expression to building an ML model. For example, the expression mining service can apply a given expression, and include the output from the expression in a dataset used for AutoML. The expression mining service can further evaluate the success of the AutoML with, and without, the expression, and can use that to evaluate the value of the relevant expression to building an ML model.

Alternatively, or in addition, the expression mining service can act as a recommendation engine for code expressions for users. The expression mining service can use a suitable user interface (e.g., a graphical or textual user interface) to present the expressions identified at block308to the user. The user can then select expressions to use or include (e.g., in additional projects).

In an embodiment, the expression mining service correlates identified expression with attributes of the dat. For example, the expression mining service can correlate a given characteristic (e.g., a column) in a dataset can with code expressions relating to that characteristic, and a user can be provided with recommendations on potential expressions to use with that characteristic (e.g., with that column). In this example, characteristic names (e.g., column names) can be normalized using a variety of techniques. For example, one or more techniques described in Semantic Concept Annotation for Tabular Data by Udayan Khurana and Sainyam Galhotra, Proceedings of the 30th ACM International Conference on Information & Knowledge Management Association for Computing Machinery 844-853 (2021) or Exploring Big Data with Helix: Finding Needles in a Big Haystack by Jason Ellis, Achille Fokoue, Oktie Hassanzadeh, Anastasios Kementsietsidis, Kavitha Srinivas, and Michael J. Ward, SIGMOD Rec. vol. 43, issue 4, at 43-54 (December 2014) can be used. These papers are hereby incorporated by reference for their discussion of normalizing structured data.

FIG.4is a flowchart illustrating finding read and write nodes for mining code expressions for data analysis, according to one embodiment. In an embodiment,FIG.4corresponds with block306illustrated inFIG.3. At block402, an expression mining service (e.g., the expression mining service142illustrated inFIGS.1-2) gathers read nodes from a data object (e.g., a container).

As discussed above, a container is an example of a data object returned by an expression call (e.g., a read call). In an embodiment, the expression mining service uses a data flow describing a code expression to gather read nodes from a given container. For example, as discussed above, one or more techniques described in the Dolby Patent Publication can be used to generate a knowledge graph describing data flows for a code expression. This knowledge graph can be used to gather read nodes for a container.FIG.6, described further below, provides an example of gathering read nodes (e.g., the read nodes642,644, and646) from a container (e.g., the container634).

At block404, the expression mining service finds descendants from read nodes to write nodes. For example, a knowledge graph generated from a code expression can describe a descendent from a read node to a write node. The expression mining service identifies these descendants from read nodes to write nodes. This is also described further, below, with regard to the example ofFIG.6(e.g., as illustrated inFIG.6the read nodes642,644, and646each have descendants to the write node626). In an embodiment, the expression mining service identifies all descendants from any read node to any write node. Alternatively, the expression mining service identifies a subset of descendants from a read node to a write node.

At block406, the expression mining service gathers expressions from write nodes. For example, the write nodes identified at block404are each associated with one or more expressions (e.g., write expressions). The expression mining service gathers these write expressions from the write nodes. This is also described further, below, with regard to the example ofFIG.6(e.g., the write expression associated with the write node626).

FIG.5is a flowchart illustrating identifying expressions for mining code expressions for data analysis, according to one embodiment. In an embodiment,FIG.5corresponds with block308illustrated inFIG.3. At block502, an expression mining service (e.g., the expression mining service142illustrated inFIGS.1-2) normalizes expression by call path.

As discussed above, in an embodiment the expression mining service identifies expressions based on calls to the dataset (e.g., based on which column the expression accesses). In an embodiment, expressions can use a variety of variable names to describe accessing the same data characteristic. This is because the programmer writing the code expression can use his or her own preferred naming scheme. In an embodiment, the expression mining service normalizes expressions by call path (e.g., to alleviate discrepancies in naming conventions).

At block504, the expression mining service normalizes using type inference. In addition to the call path, described above in relation to block502, the expression mining service can use type inference (e.g., variable types used in the code expression) to further normalize expressions. This can further alleviate discrepancies in naming conventions.

At block506, the expression mining service groups common expressions by operator. In an embodiment, expressions that use a common operator on a given dataset (e.g., a common mathematical operator) can be grouped. These expressions grouped by operator can be presented to users as recommendations, and the users can select desired expressions based on the recommendations.

FIG.6illustrates an example600of mining code expressions for data analysis, according to one embodiment. In an embodiment,FIG.6provides an example of mining code expressions using one or more of the techniques described above in relation toFIGS.3-5. This is merely one example, and other data and techniques can be used.

In an embodiment, a dataset602is being analyzed. For example, the dataset602can reflect medical data (e.g., disease data) which is available for analysis by a user (e.g., the user102illustrated inFIG.1). This data can be stored in a suitable data repository (e.g., the data repository110illustrated inFIG.1).

In an embodiment, an expression mining service (e.g., the expression mining service142illustrated inFIGS.1-2) identifies a number of existing code expressions610. For example, these existing code expressions610may have been created by others when analyzing the dataset602. The existing code expressions610include a number of expressions used to analyze a given data set, and can also be stored in a suitable repository (e.g., the existing code expressions130illustrated inFIG.1).

In an embodiment, the expression mining service generates a knowledge graph620describing data flows in the existing code expressions610. For example, as described above, the expression mining service can use one or more techniques described in the Dolby Patent Publication to generate a knowledge graph with invocation nodes622and624(e.g., nodes corresponding to method calls), write node626, and read nodes642,644, and646. In an embodiment, the node622describes a “read_csv” expression on data located at the path “../data”. The return value from this node622is a container632. The node624describes a “read_csv.drop” expression on the columns that match “Locale” or “Data” (e.g., an expression to drop columns with the labels “Locale” or “Data” from the DataFrame variable “df”). The return value from this node624is a container634. The node626describes an expression in which df[‘Active’] (e.g., the DataFrame variable “df” at the index “Active”) is set to the value of df[‘Confirmed’]−df[‘Recovered’]−df[‘Deaths’]. The return value from this node626is a container636.

In an embodiment, the expression mining service uses the knowledge graph620to mine the expression. For example, the expression mining service identifies one or more containers in the existing code expressions610. As discussed, the expression mining service identifies three containers: a container632corresponding to the return value from the node622, a container634corresponding to the return value from the node624, and a container636corresponding to the return value from the node626.

In an embodiment, the expression mining service then gathers reads from each container. Using the container634as an example, the expression mining service identifies three reads at the container, and identifies three read nodes642,644, and646. The read node642corresponds to a read of the field ‘Confirmed’ from the container634. The read node644corresponds to a read of the field ‘Recovered’ from the container634. The read node646corresponds to a read of the field ‘Deaths from the container634.

As illustrated, the expression mining service then finds all descendants from the read nodes642,644, and646to any write node. For example, the expression mining service identifies descendants from the read nodes642,644, and646to the write node626. The expression mining service further gathers expressions from the write nodes. As illustrated, the expression mining service gathers the expression df[‘Active’]=[df[‘Confirmed’]-df[‘Recovered’]−df[‘Deaths’] from the node626. In an embodiment, this expression writes to the field ‘Active’ in the container632based on the values of ‘Confirmed,’ ‘Recovered,’ and ‘Deaths’ read from the container634.

In an embodiment, the expression mining service then normalizes expressions by call path. As discussed above, in an embodiment an expression can include variables named by the drafter of the expression. The expression mining service can use the knowledge graph information to normalize variable names in the expression to account for differences among programmer name conventions. For example, the expression mining service can normalize the expression df[‘Confirmed] to pandas.read_csv_drop.Confirmed (e.g., using the knowledge graph620illustrated inFIG.6). The expression mining service can further normalize the expression using type inference. For example, the variable “df” can correspond to variable of the Pandas DataFrame type in Python® (e.g., a two dimensional data structure). The expression mining service can normalize by this type. Further, the expression mining service can group common expressions across different programs by operator (e.g., the minus operator in the expression illustrated in the node626).

For convenience, the Detailed Description includes the following definitions which have been derived from the “Draft NIST Working Definition of Cloud Computing” by Peter Mell and Tim Grance, dated Oct. 7, 2009.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG.9, a set of functional abstraction layers provided by cloud computing environment50(FIG.8) is shown. It should be understood in advance that the components, layers, and functions shown inFIG.9are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

Workloads layer66provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation; software development and lifecycle management; virtual classroom education delivery; data analytics processing; transaction processing; and expression mining. For example, the workloads layer66can implement some, or all, of the functionality of the expression mining service142illustrated inFIGS.1-2, above.