Recommended application evaluation system

A code evaluation tool greatly reduces time, cost, and other resource expenditures needed to validate that an application implements desired functionality. The tool is a search, navigation and visualization tool that accepts high-level processing concepts as inputs to identify, rank, and return the code of a recommended application. A software developer may use the tool to validate that functional requirements are met by the recommended application. The tool provides an efficient way to improve the evaluation of application logic to validate that the application meets specified functional requirement and implements the desired high-level processing concepts.

BACKGROUND OF THE INVENTION

1. Technical Field

This disclosure concerns evaluating applications identified as implementing desired functionality. In particular, this disclosure relates to a search, navigation and visualization tool that accepts high-level processing concepts as inputs that drive a multi-layered search of an application to validate that the logic of the application actually does implement desired functionality.

2. Background Information

Software professionals widely recognize that applications often fail to implement functionality as described by the application descriptions and project summaries of the applications. Software professionals use various inadequate techniques to reduce the time, money, and other costs for validating that an application implements particular functionality. Software professionals recognize API calls as forms of abstraction for high-level processing concepts, and merely search for the existence of particular API calls to validate that the application implements a desired functionality. For example, an API call may be identified and/or described as implementing pull-down menu functionality, although the underlying logic may not actually include the logic necessary to deliver the functionality of a pull-down menu. Current logic evaluation techniques and tools require significant resources and user expertise to accurately validate that an application implements functional requirements in support of high-level processing concepts. Modern search engines do not ensure that the logic of applications identified by the search engines actually implement the functionality as described by the project summaries and documentation of the application.

Software professionals consider the mismatch between the high-level processing concepts (e.g., the intent reflected in the descriptions of applications) and low-level implementation details (e.g., API calls and actual run-time behaviour) found in application logic a fundamental technical challenge to validating that an application implements particular functionality. Software professionals intend to author meaningful descriptions of applications, in the course of depositing applications into software repositories. The mismatch between the description of an application and the actual behaviour of the application represents one example of the “vocabulary problem”, which states that no single word or phrase best describes a programming concept.

In the spiral model of software development, stakeholders describe high-level processing concepts to development teams, and together the stakeholders and development teams identify requirements in support of the high-level processing concepts. In addition, a development team builds a prototype based on the requirements, and the development team demonstrates the prototype to the stakeholders to receive feedback. Prototypes attempt to approximate the desired high-level processing concepts (e.g., features and capabilities) of the new application stakeholders desire development teams to build. The feedback from stakeholders often leads to changes to the prototype and the original requirements, as stakeholders iteratively refine their vision. In the event the stakeholders make a substantial number of changes to the requirements, the development team often discards the prototype and builds a new prototype, and another iteration of refinements repeats. Building prototypes repeatedly without reusing existing application logic costs organizations a great deal in the form of wasted project resources and time. Deploying an application without adequately evaluating and validating that the logic of the application implements desired functionality further complicates software development.

Development teams find the task of evaluating and validating that the logic of an application approximates high-level processing concepts and requirements of a software project resource intensive. In the context of application deployment, software development professionals consider such application logic as highly relevant application logic (HRAL). Many application repositories (e.g., open source repositories and source control management systems maintained by stakeholders internally) contain hundreds of thousands of different existing applications (e.g., potential highly relevant applications (HRAs)). Unfortunately, developers find it difficult to validate the logic of applications, because of the time and expense involved in searching the application, evaluating and validating that the logic of the application implements particular functionality.

The amount of intellectual effort that a developer must expend to move a software system from one stage of development to another may be considered the “cognitive distance”. For example, using current search tools, developers expend significant intellectual effort to validate that an application implements desired functionality. Many developers employ search engines that identify exact matches between keywords that are entered as part of the search query and the words found in an application. The application may include descriptions, application logic comments, program variables names, and variable types of the application. Such search engines actually increase the difficulty of validating the application, because of the poor quality of information contained in application, and the inability to reduce the cognitive distance required to validate that the logic of the application implements the desire functionality, as well as other factors. Additionally, many applications include incomplete, misleading and inaccurate descriptions of the application. Consequently, even matching keywords with words found in the application description does not guarantee that the search engine will identify application logic that actually implements desired functionality (e.g., highly relevant application logic (HRAL)).

Effective software evaluation techniques reduce the cognitive distance between the initial concept of a system (e.g., high-level processing concepts that expressly and implicitly describe the features and capabilities of an application), validating discrete requirements, and the production implementation of the application. Unfortunately, current application evaluation tools lack the ability to reduce the cognitive distance related to validating application logic to identify HRAL.

For example, an application description may indicate that an application includes an encryption feature when in fact the application uses compression as a crude form of encryption. A developer entering “encryption” (e.g., as a high-level processing concept and specific requirement) as a keyword may waste precious time to review a search engine result containing the incorrectly described logic of an application, and ultimately discard the result, because the application fails to meet the encryption requirement. The developer must locate and examine fragments of the application logic that allegedly implements encryption before determining that the application fails to meet the requirement. The developer may spend scarce project development budget resources and significant amount of time to analyze the application before determining that the application is not relevant. The developer may even observe the runtime behavior of the application to ensure that the behavior matches the high-level processing concepts desired by the stakeholders, and meets the requirements in support of the high-level processing concepts before establishing that the logic of the application qualifies as HRAL. Current evaluation tools also lack the ability to assist developers to rapidly identify requirements in support of high-level processing concepts described by stakeholders.

Some evaluation tools return code snippets (e.g., segments of application logic), however, code snippets do not give enough background or context to assist developers to rapidly evaluate and validate the logic, and such evaluation tools require developers to invest significant intellectual effort (e.g., cognitive distance) to understand the broader scope of the code snippets. Other existing approaches and tools retrieve snippets of code based on the context of the application logic that developers work on, but while these approaches and tools improve the productivity of developers, they do not return highly relevant applications logic from high-level processing concepts as inputs.

A need has long existed for a system and method that efficiently evaluates and validates application logic to identify HRAL and deployable application.

SUMMARY

The recommended application evaluation system (RAE) provides a way to rapidly and efficiently evaluate the logic of recommended applications to validate that the recommended applications actually do implement highly relevant application logic (HRAL). One implementation of RAE includes a machine readable medium and logic stored on the machine readable medium that implements RAE area logic. The RAE area logic is operable to obtain recommended applications (e.g., highly relevant applications (HRAs)) for evaluation. The recommended applications may be supplied to the RAE in the form of source code for the recommended application, or in other forms.

The RAE area logic presents a concept query display region that displays topics, concepts and words of a query used to find the recommended applications. The RAE area logic presents a concept distribution display region that displays a recommended application representation of one of the recommended applications. The recommended application representation includes concept distribution locators that identify potential implementation locations of one of the topics, concepts and words. The RAE area logic presents a source code display region that is responsive to a selection of one of the concept distribution locators. The source code display region displays a portion of the source code that includes the potential implementation location for the selection of one of the concept distribution locators. The RAE area logic presents a metadata display region that displays metadata generated upon execution of the query.

In one implementation, the recommended applications (e.g., HRAs) are identified using the EXEcutable exaMPLes ARchive system (Exemplar). Exemplar identifies the location (e.g., concept distribution locators) in application logic where an API call implements a high-level processing concept. Using Exemplar, a developer enters high-level processing concepts (e.g., toolbar, download, smart card) as input (e.g., initial query keywords), and Exemplar uses information retrieval and program analysis techniques to retrieve HRAs that implement the high-level processing concepts in the application. Exemplar may also accept various types of inputs that describe high-level processing concepts (e.g. concept text identifiers, concept visual identifiers, concept audio identifiers, and any other sensory identifier usable to identify high-level processing concepts). Exemplar uses the help pages and help documentation of third-party libraries, software development kits, and other middleware to produce a list of names of API calls that Exemplar in turn uses to expand an initial query (“query expansion”) to identify the highly relevant application logic of an application. Exemplar determines the behavior of the application logic and API call logic and ranks the API calls.

Exemplar uses help documentation or other trusted sources that describe API calls to expand queries. An application provider typically provides the help pages and help documentation for their applications, which developers consider reliable and a trusted source. In particular, developers consider application providers trusted sources for help pages and help documentation of popular and widely used applications written by large development teams, produced under rigorous testing and development best practices, and used by other developers who provide feedback regarding documentation using different forums (e.g., user groups). Developers trust help documentation over the descriptions of applications included in application repositories, because application providers generally produce more verbose and accurate help documentation than the descriptions of applications included in application repositories. Developers also trust help documentation because many different people and review procedures are typically used to produce help documentation.

Exemplar query expansion increases the probability of identifying logic matches that validate that the logic is highly relevant application logic, and addresses the vocabulary problem mentioned above by expanding an initial query to include new keywords, metadata, and semantics information found in help pages and other help documentation determined to have similar meanings to the keywords originally used by a developer in the initial query. Exemplar expands an initial query to include the names of API calls with semantics that reflect (in many cases unequivocally) specific behaviour of the application. Exemplar locates application logic containing the API calls that exhibit desired semantics by identifying API calls through help pages and help documentation. Exemplar provides a user interface that developers can use to navigate directly to the various locations to determine how an HRA implements high-level processing concepts.

Exemplar may rank HRAs according to the number of high-level processing concepts implemented by each API call found in the HRAs, or based on other ranking metrics. In other words, since API calls implement high-level processing concepts, the more high-level processing concepts implemented by an HRA the more relevant the HRA and the higher the rank assigned to the HRA. Exemplar considers keywords included in queries to represent logically connected concepts. Often a question structured as a sentence forms the basis for a query, from which a developer extracts keywords to form the query. For example, consider the query “send receive secure XML.” Where a query presents a relation between multiple concepts (e.g., send secure XML), then a relation should exists between API calls that implement the concepts in the corresponding application logic (e.g., API calls that encrypt, process or handle XML formatted content, and transmit content). Application logic often preserves the relations between concepts (e.g., control flow and data flow links), an instance of the software reflection model concept and known as connectivity heuristics. Exemplar calculates HRAs rankings based on analyzing the connectivity heuristics of API calls that implement the concepts included in the queries. Exemplar uses program analysis algorithms, and computes control flow graphs (CFG), and data flow graphs (DFG) to analyze the connectivity heuristics of API calls.

RAE provides a way to rapidly and efficiently evaluate the logic of recommended applications to validate that the recommended applications implement highly relevant application logic (HRAL).

DETAILED DESCRIPTION

The recommended application evaluation system (RAE) solves the technical problem of providing a tool to easily and quickly evaluate a recommended application to validate that the recommended application implements a particular functionality. In one implementation, the RAE may evaluate recommended applications obtained from the EXEcutable exaMPLes ARchive system (Exemplar).

Exemplar provides a tool that accepts high-level processing concepts as queries to identify, determine the behavior, rank and return the application logic of HRAs. Exemplar solves an instance of the difficult vocabulary problem that exists when users and developers describe processing concept with different words. Exemplar is not limited to basic keyword matching used in queries against application descriptions and comments included with application logic. Accordingly, when an application is highly relevant, and where a query contains keywords different from the words used by the developer to describe application logic and API call logic, Exemplar nevertheless returns the application as a highly relevant application.

Exemplar matches high-level processing concepts (e.g., expressed using keywords) with the descriptions of various API calls found in help documents or other trusted descriptive sources. Because a typical application invokes API calls from several different libraries, several different people who use different vocabularies often author help documents associated with API calls. The richness of different vocabularies increases the probability of finding matches and producing a long list of potentially relevant applications and API calls. Searching help documents or other trusted descriptive sources produces additional benefits. For example, help documents including an API call often indicate where the application logic implements the API call. Consequently, Exemplar may direct a developer to the location in application logic where an API call implements a high-level processing concept. The developer may then determine the relevance of the application logic and API call logic. In other words, the developer may determine whether the application logic and API call logic actually support the high-level processing concept.

Although specific components of Exemplar and RAE will be described, methods, systems, and articles of manufacture consistent with Exemplar and/or the RAE may include additional or different components. For example, a processor may be implemented as a microprocessor, microcontroller, application specific integrated circuit (ASIC), discrete logic, or a combination of other type of circuits or logic. Similarly, memories may be DRAM, SRAM, Flash or any other type of memory. Logic that implements the processing and programs described below may be stored (e.g., as computer executable instructions) on a computer readable medium such as an optical or magnetic disk or other memory. Alternatively or additionally, the logic may be realized in an electromagnetic or optical signal that may be transmitted between entities. An example of such a signal is a physical layer Ethernet signal bearing TCP/IP packets that include program source code or executable programs. Flags, data, databases, tables, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be distributed, or may be logically and physically organized in many different ways. Programs may be parts of a single program, separate programs, or distributed across several memories and processors. Furthermore, the programs, or any portion of the programs, may instead be implemented in hardware.

FIG. 1illustrates the Exemplar system environment100(“Exemplar environment”100) in communication with a recommended application evaluation system (RAE)128. The Exemplar environment100may include an Exempla r prototyping and application development (EPAD) user interface102, a help content processor104, and help content106. The Exemplar environment100also includes an API calls dictionary108, expanded search engine110, logic repository112, heuristic relevance ranking engine114, and a logic analyzer116. Any or all of the elements shown inFIG. 1may be co-located or distributed and in communication over one or more networks118(e.g., the Internet).

In one implementation, the EPAD user interface102, expanded search engine110, heuristic relevance ranking engine114and logic analyzer116form an Exemplar system124within the Exemplar environment100. The Exemplar system124may include additional or different components. The Exemplar system124may communicate with the help content processor104, help content106, API calls dictionary108, and logic repository112, as well as other systems, through the networks118(e.g., Internet) as external systems.

The logic repository112may include application logic120and API call logic122. The Exemplar system124accepts high-level processing concepts (e.g., “send secure XML”) as input and produces output identifying which application logic120and API call logic122developers may use to prototype and develop new applications implementing the high-level processing concepts. In one implementation, the Exemplar environment100implements the help content106and the logic repository112with multiple storage devices (e.g., multiple databases on different disk drives), and interfaces to help content106, application logic120and API call logic122from various available source (e.g., local or remote help databases, websites, knowledge exchanges, document repositories, or other sources).

In one implementation, the help content processor104may be implemented as a web crawler that traverses available application repositories, and downloads help content106(e.g., application descriptions), and logic repository112content (e.g., application logic120, and API logic122). The help content processor104may also perform full text indexing on the help content106and the logic repository112content. The help content processor104may further produce an API calls dictionary108that includes sets of tuples (a form of ordered list) that link selected words from the descriptions of the API calls to the names of the API calls.

The description above used the examples of application logic120and API call logic122. These types of logic may be program source code (e.g., C or C++ code), for example. However, the Exemplar environment100may search, analyze, and determine relevance for many other types of logic. As examples, the logic repository112may include programs or program components expressed in a visual programming language using graphical program elements and spatial arrangements of text and graphic symbols. The visual programming logic may include icon-based logic, form-based logic, diagram-based logic or other types of visual expression. The visual expression may be consistent with dataflow languages, flow-based programming, domain-specific modelling, or other programming paradigms.

In one implementation, the Exemplar environment100and the RAE128are present in a RAE environment126. The RAE128includes a RAE user interface generation logic130that comprises RAE area logic132. The RAE area logic132comprises concept query display logic134, concept distribution display logic136, source code display logic138and metadata display logic140. Aspects of the RAE128are discussed in further detail below.

FIG. 2shows an Exemplar data flow diagram200. Exemplar system124accepts a high-level processing concept as input to create an original query202that Exemplar system124may forward to the help content processor104. The help content processor104may produce a basis API call list204from the API calls dictionary108by matching the words in the high-level processing concepts (e.g., “send secure XML”) found in the original query202executed to search the help content106.

The expanded search engine110may combine the original query202and the basis API call list204to form an expanded query206. The expanded search engine110may execute an expanded search using the expanded query206to search through the logic repository112to obtain an expanded search result208. In one implementation, the logic repository112may return the expanded search results208to the heuristic relevance ranking engine114. The expanded search result208may contain a list of potentially relevant applications210and potentially relevant API calls212that the heuristic relevance ranking engine114analyzes using the logic analyzer116. In one implementation, the heuristic relevance ranking engine114may include the logic analyzer116. The logic analyzer116may include a parser generator such as ANTLR (“ANother Tool for Language Recognition”) available from www.antlr.org that provides support for generating data flow graphs and control flow graphs.

The logic analyzer116may return connectivity rankings214, discussed in detail below, to further determine an application heuristic relevance ranking216and an API call heuristic relevance ranking218. The heuristic relevance ranking engine114may return the application heuristic relevance ranking216and an API call heuristic relevance ranking218to the EPAD user interface102. The expanded search engine110may also return a relevant applications list220and a relevant API calls list222to the EPAD user interface102. The Exemplar system124may assign an application heuristic relevance ranking216to one or more relevant applications found in the relevant applications list220to indicate how closely each relevant application supports the high-level processing concept represented by the original query202. Similarly, Exemplar system124may assign an API call heuristic relevance ranking218to one or more relevant API calls found in the relevant API call list222to indicate how closely each relevant API call supports the high-level processing concept represented by the original query202.

In one implementation, the RAE area logic132obtains the connectivity rankings214, the application heuristic relevance rankings216, the API call heuristic relevance rankings218, the relevant applications list220, and the relevant API call list222. The RAE area logic132may use the connectivity rankings214, the application heuristic relevance rankings216, the API call heuristic relevance rankings218, the relevant applications list220, and the relevant API call list222to present a recommended application evaluation area, discussed in detail below, to evaluate recommended applications obtained from Exemplar124.

FIG. 3illustrates an Exemplar query interface300that may be used to input an original query202. The original query202may represent a high-level processing concept such as “compress uncompress ZIP file,” as shown in the text entry field302. Several developers may have implemented the example high-level processing concept “compress uncompress ZIP file”302in different ways with various API calls described in the help content106, API calls dictionary108and logic repository112. A user may specify search refinement criteria304using interface elements such as a drop down box, menu or user input field. The search refinement criteria304may dictate the types of logic of interest (e.g., C, C++, JAVA, or other types of logic), may dictate the information sources searched (e.g., search only compiled Help files, or *.doc files), or may specify other search criteria. The Exemplar query interface300may include graphical user interface elements (e.g., the search button306) used to execute the original query202.

Table 1 shows an example of an original query202in the form of a structured query language statement (SQL) that represents the high-level processing concept “compress uncompress ZIP file”302. Table 1 shows that the original query202will search the help content106(e.g., Java Help Documents) to identify a basis API calls list204.

TABLE 1original query 202SELECT basis API CallsFROM Java Help DocumentsWHERE Words in these Documents =compress or uncompress or ZIP or file.

Table 2 shows one example of the help content106represented by a fragment of Java Help Documentation released by Sun Microsystems, Inc. that describes the functionality of classes exported from the Java.util package. The Java.util package defines a number of classes, primarily collections classes that a developer may use when working with groups of objects. Referring to Table 2, the help content processor104may identify partial matches for the class ZipEntry to the original query202. The help content processor104may search the help content106and identify a fragment of the help documentation for the ZipEntry class shown in Table 3.

TABLE 2help content 106 (e.g. a fragment of Java Help Document)ClassSummaryAdler32A class that can be used to compute theAdler-32 checksum of a data stream.CheckedInputStreamAn input stream that also maintains achecksum of the data being read.CheckedOutputStreamAn output stream that also maintains achecksum of the data being written.CRC32A class that can be used to compute theCRC-32 of a data stream.DeflaterThis class provides support for generalpurpose compression using the popular ZLIBcompression library.DeflaterInputStreamImplements an input stream filter forcompressing data in the “deflate”compression format.DeflaterOutputStreamThis class implements an output streamfilter for compressing data in the“deflate” compression format.GZIPInputStreamThis class implements a stream filter forreading compressed data in the GZIP fileformat.GZIPOutputStreamThis class implements a stream filter forwriting compressed data in the GZIP fileformat.InflaterThis class provides support for generalpurpose decompression using the popularZLIB compression library.InflaterInputStreamThis class implements a stream filter foruncompressing data in the “deflate”compression format.InflaterOutputStreamImplements an output stream filter foruncompressing data stored in the“deflate” compression format.ZipEntryThis class is used to represent a ZIP fileentry.ZipFileThis class is used to read entries from a zipfile.ZipInputStreamThis class implements an input stream filterfor reading files in the ZIP file format.ZipOutputStreamThis class implements an output stream filterfor writing files in the ZIP file format.

Table 3 shows the descriptions of two different methods (e.g., getCompressedSize, and setMethod) for the ZipEntry class that include the terms compress and uncompress found in the high-level processing concept “compress uncompress ZIP file”302. The basis API call list204may include the getCompressedSize and setMethod methods.

TABLE 3help content 106 (e.g., fragment of helpdocumentation for ZipEntry class)MethodSummaryObjectclone( )Returns a copy of this entry.StringgetComment( )Returns the comment string for the entry, or null if none.longgetCompressedSize( )Returns the size of the compressed entry data, or −1if not known.longgetCrc( )Returns the CRC-32 checksum of the uncompressed entrydata, or −1 if not known.byte[ ]getExtra( )Returns the extra field data for the entry, or null if none.intgetMethod( )Returns the compression method of the entry, or −1if not specified.StringgetName( )Returns the name of the entry.longgetSize( )Returns the uncompressed size of the entry data, or −1if not known.longgetTime( )Returns the modification time of the entry, or −1if not specified.inthashCode( )Returns the hash code value for this entry.booleanisDirectory( )Returns true if this is a directory entry.voidsetComment(String comment)Sets the optional comment string for the entry.voidsetCompressedSize(long csize)Sets the size of the compressed entry data.voidsetCrc(long crc)Sets the CRC-32 checksum of the uncompressed entry data.voidsetExtra(byte[ ] extra)Sets the optional extra field data for the entry.voidsetMethod(int method)Sets the compression method for the entry.voidsetSize(long size)Sets the uncompressed size of the entry data.voidsetTime(long time)Sets the modification time of the entry.StringtoString( )Returns a string representation of the ZIP entry.

Table 4 shows an example of two equivalent forms of an expanded query206that expand the original search from the help content106(e.g., Java Help Documents) to the logic repository112using the basis API call list204from the original query202. Table 4 statement A shows the getCompressedSize and setMethod that may be included in the basis API call list204. Table 4 statement B shows the expanded query206as a nested query, where the original query202and the basis API call list204(e.g., getCompressedSize and setMethod) drive the outer query that searches the logic repository112for potentially relevant applications210to obtain the expanded query result208including potentially relevant applications210and potentially relevant API calls212. The expanded query206may improve upon the original query202by targeting the search performed against the logic repository112to obtain application logic120with a high probability of including potentially relevant applications210and potentially relevant API calls212.

TABLE 4expanded query 206A.SELECT Potentially Relevant ApplicationsFROM Logic RepositoryWHERE API Calls inSource Code Files of these Application= getCompressedSize or setMethod./************* The SQL statement above also expressed below. ****************/B.SELECT Potentially Relevant ApplicationsFROM Logic RepositoryWHERE API Calls inSource Code Files of these Application= {SELECT basis API CallsFROM Java Help DocumentsWHERE Words in these Documents =compress or uncompress or ZIP or file}.

Table 5 shows another example of two equivalent forms of an expanded query206that expand the original search from the help content106(e.g., Java Help Documents) to the logic repository112by combining the original query202and the basis API call list204to form the expanded query206. Table 5 statement A shows the getCompressedSize and setMethod (e.g., the basis API call list204) combined with the original query202. Table 5 statement B shows the expanded query206as a nested query, where the original query202and the basis API call list204(e.g., getCompressedSize and setMethod) drive the outer query that searches the logic repository112for potentially relevant applications210to obtain the expanded query result208including potentially relevant applications210and potentially relevant API calls212. The expanded query206may improve upon the original query202by targeting the search performed against the logic repository112to obtain application logic120with a high probability of including potentially relevant applications210and potentially relevant API calls212.

Table 6 shows an example of a fragment of logic extracted from the logic repository112(e.g., potentially relevant application210) that includes a potentially relevant API call212(e.g., getCompressedSize).

FIG. 4shows an Exemplar system124prototyping and application development (EPAD) project area400. The EPAD project area400may include a relevant application list display area402, relevant API list display area404, heuristic relevance ranking results display area406, logic display area408, and application creation display area410. The relevant application list display area402may include the relevant application list220produced by the Exemplar system124based on the expanded query206search results. The relevant applications list220may include a relevant application identifier-1412(e.g., a program name, repository identifier, file name, or other program specifier) of relevant application logic414. A user may select any identifier, as indicated by the arrow416, to display the relevant application logic414(e.g., source code for the program) in the logic display area408. The EPAD user interface102may include a keyboard, mouse, a microphone (or other sensors), joystick, game pad, or the like for the user to interact with the EPAD project area400.

The relevant API list display area404may include the relevant API call list222returned by Exemplar system124based on the original query202. The relevant API call list222may include a relevant API call identifier-1418(e.g., a function call name) and a relevant API call identifier-2420of the relevant API call logic-1422(e.g., source code for the function call) and relevant API call logic-2424, respectively. The EPAD project area400may present the relevant API call identifier-1418and the relevant API call identifier-2420as user selectable, indicated by the arrow426, to display and highlight the relevant API call logic-1422and the relevant API call logic-2424in the logic display area408. In one implementation, the logic display area408may highlight the relevant application logic414, and relevant API call logic-1422and relevant API call logic-2424so that the user can further determine the relevance of the logic to the high-level processing concept represented in the original query202.

The heuristic relevance ranking results display area406, shown inFIG. 4, may include an application relevance threshold428, an API call relevance threshold430, data flow graph432, control flow graph433, and an API call graph434. The heuristic relevance ranking results display area406may display heuristic relevance ranking engine114information to assist the user to determine the relevance of user selected logic. As will be described in more detail below, the heuristic relevance ranking engine114may determine the application heuristic relevance ranking216for the relevant application logic414based on the number of relevant API calls (e.g., relevant API call logic-1422and relevant API call logic-2424) found in the relevant application logic414in comparison to other relevant application logic436identified by Exemplar system124. For example, the high-level processing concept example “compress uncompress ZIP file”302may be entirely implemented in relevant application logic414, but only partially implemented in the other relevant application logic436. As a result, the heuristic relevance ranking engine114may assign the relevant application logic414a higher application heuristic relevance ranking216than the other relevant application logic436. In another implementation, the heuristic relevance ranking engine114may determine the API call heuristic relevance rankings218of the relevant API call logic-1422and the relevant API call logic-2424, based on analyzing semantics derived from the expanded query206and the expanded search result208, which establish the behaviour of the relevant API call logic-1422, the relevant API call logic-2424, and the relevant applications logic-1414.

The application relevance threshold428and API call relevance threshold430, shown inFIG. 4, may be user selectable and/or pre-configured with system default values. In another implementation, Exemplar system124may determine the application relevance threshold428and the relevance threshold430based a number of factors (e.g., the complexity of the high-level processing concept represented by the original query202, and the number of potentially relevant applications210and potentially relevant API calls212identified by the expanded search result208). Exemplar system124may use the application relevance threshold428and the relevance threshold430to further refine the relevant applications list220and the relevant API calls list222, respectively. In one implementation, the application relevance threshold428and the relevance threshold428may determine an application heuristic relevance ranking216value that the potentially relevant applications210must meet to be included on the relevant applications list220. The API call relevance threshold430may also determine the API call heuristic relevance ranking218value that the potentially relevant API calls212must meet to be included on the relevant API calls list222. For example, an application relevance threshold428of 1 may indicate a low relevance requirement (e.g., requiring loosely relevant applications, and low application heuristic relevance rankings216) and allow a large number of potentially relevant applications210to qualify as relevant applications (e.g., relevant application logic-1414). In another example, an application relevance threshold428of 10 may indicate a high relevance requirement (e.g., requiring highly relevant applications, and high application heuristic relevance rankings216) and allow only a fewer number of potentially relevant applications210to qualify as relevant applications. The heuristic relevance ranking engine114may also use the data flow graph432and control flow graph433to determine the application heuristic relevance ranking216and API call heuristic relevance ranking218, and visually describe the relationships between the relevant application logic414, the relevant API call logic-1422, and the relevant API call logic-2424, discussed in further detail below.

The relevant API call logic-1422and the relevant API call logic-2424may be user selectable (indicated by the arrow442), and provide the user the ability to generate a new application440with the selected logic. To that end, the EPAD project area400may implement point-and-click, drag-and-drop functionality for a user to select relevant API call logic-1422and relevant API call logic-2424to generate the new application440. The EPAD project area400may also build the new application440by combining user selectable other relevant application logic436, relevant API call logic-1, and relevant API call logic-2. The application creation display area410may also identify requirements444for the high-level processing concept represented by the original query202. For example, a developer may desire to identify and confirm the requirements444for implementing a high-level processing concept (e.g., “send secure XML”). In one implementation, Exemplar may generate requirements documentation and end user documentation based on the help content106related to the other relevant application logic436, the relevant API call logic-1, and the relevant API call logic-2 used to build the new application440, and identify the requirements444in support of the new application440.

FIG. 5shows a more detailed view of the Exemplar system124. The Exemplar system124may include a communications interface504used to communicate with various resources internal and external to Exemplar system124, memory506, and a processor508. The processor508may execute any of the logic described below. The memory506may include the EPAD user interface102that employs the interface logic510to generate the Exemplar query interface300, and the EPAD project area400. The interface logic510may include graphics libraries, window rendering calls, and other user interface logic operable to display interface elements, receive input, and pass the input to any particular program logic in the Exemplar system124.

The memory506may also include expanded search logic514. Table 5, above, shows an expanded query206where the search logic514forms the expanded query by combining the original query202and the basis API call list204to form the expanded query206. More generally, the expanded search logic514combines the original query202and the basis logic results516to form the expanded query206, and executes an expanded search using the expanded query206. The basis logic results516may include the basis API call list204, including zero or more basis API call identifiers (e.g., the basis API call identifier-1518), and a basis application list520, including zero or more basis application identifiers (e.g., the basis application identifier-1521). The expanded search logic514thereby obtains the expanded search results208. The expanded search result208may include potentially relevant applications210, and potentially relevant API calls212that include zero or more potentially relevant application identifiers-1522and zero or more potentially relevant API call identifiers (e.g., potentially relevant API call identifier-1524and potentially relevant API call identifier-2526).

FIG. 6shows other features of the Exemplar system124. The memory506may also include the heuristic relevance ranking engine114with the heuristic relevance ranking logic602that generates the application heuristic relevance ranking216and API call heuristic relevance ranking218. The threshold logic604may apply the application relevance threshold428and API call relevance threshold430to the application heuristic relevance rankings216and API call heuristic relevance rankings218to determine whether potentially relevant applications210and potentially relevant API calls212qualify for inclusion in the relevant applications list220and the relevant API calls list222. In other words, the threshold logic604may implement comparison logic to determine when potentially relevant logic qualifies as relevant logic.

The memory506may also include analyzer logic606that the processor508executes to identify application metadata608and API metadata610of the potentially relevant applications210, and the potentially relevant API calls212, respectively. Examples of application metadata608include application descriptions, application logic comments, application parameter names, and application parameter types of existing applications. Similarly, examples of API metadata610include API descriptions, API logic comments, API parameter names, and API parameter types.

The analyzer logic606may generate the data flow graph432and control flow graph433to obtain the API call graph434. The API call graph434may include nodes (e.g., node-1612and node-2614) that represent potentially relevant API calls212and data flow edges (e.g., data flow edge616) between the potentially relevant API calls212to indicate data flow.FIG. 7provides additional examples. The analyzer logic606may determine the data flow edge count618corresponding to the number of connections between potentially relevant API calls212within the potentially relevant application210. A graph with ‘n’ nodes has as many as n(n−1) edges between nodes. The data flow edge count618provides insight into the degree of connectedness for the data flow graph432. The analyzer logic606may also assign link values620to the edges between nodes, discussed in detail below. In one implementation, the analyzer logic606may determine the connectivity rankings214(e.g., strong connectivity ranking622and weak connectivity ranking624) for each connection between the potentially relevant API calls212based on common API parameters626, discussed in detail below.

FIG. 6further illustrates that memory506may include selection logic628and application creation logic630. The processor508may execute the selection logic628to allow a user to select relevant application logic414, and relevant API call logic (e.g., the relevant API call logic-1422and the relevant API call logic-2424) to develop the new application440. In another implementation, selection logic628may provide a user drag-and-drop point-and-click functionality to select other relevant application logic436to combine with the relevant API call logic-1422, and the relevant API call logic-2424to build the new application440. The processor508may execute the application creation logic630to identify requirements444for the high-level processing concept represented by the original query202by identifying the help content106used to obtain the basis logic results516corresponding to the user selected other relevant application logic436, relevant application logic414, relevant API call logic-1422, and relevant API call logic-2424. In one implementation, the application creation logic may form a query using the other relevant application logic436, relevant application logic414, relevant API call logic-1422, and relevant API call logic-2424to obtain the help content106that describes the requirements444. The application creation logic630may generate customized requirements documents from the help content106corresponding to the user selected other relevant application logic436, relevant application logic414, relevant API call logic-1422, and relevant API call logic-2424.

FIG. 7shows API call graphs434for two different potentially relevant applications (e.g., a potentially relevant application A and potentially relevant application B). The heuristic relevance ranking engine114may assign a higher heuristic relevance ranking216to the potentially relevant application A than the potentially relevant application B based on the number of potentially relevant API calls212, the connectivity rankings214and link values620assigned to each connection between potentially relevant API calls212included in the potentially relevant application A and potentially relevant application B, respectively.

In one implementation, the logic analyzer116produces the API call graphs434. The logic analyzer116may identify the application metadata608and API metadata610of the potentially relevant applications210, and the potentially relevant API calls212, respectively, to analyze the data flow paths and connectivity between the potentially relevant API calls212. The logic analyzer116may provide the application metadata608and API metadata610to the heuristic relevance ranking engine114. In an alternative implementation, the heuristic relevance ranking engine114may identify application metadata608and API metadata610, and produce the data flow graph432and control flow graph433using logic analysis formulas, rules and equations to obtain the API call graphs434. The data flow graphs432, control flow graphs433and API call graphs434may be represented as mathematical structures. The logic analyzer116may obtain the API call graphs434as a result of comparing data flow and control flow between potentially relevant API calls212.

In one implementation, the logic analyzer116may perform control flow analysis on the potentially relevant application210to obtain control flow graphs433, and perform data flow analysis on the control flow graphs433to obtain data flow graphs. The data flow graphs432, control flow graphs433, and API call graphs may similarly include nodes and edges. The logic analyzer116may obtain a control flow graph433by logically partitioning a potentially relevant application210as a result of parsing the logic of the potentially relevant application210into nodes that represent logic that includes API calls. The logic analyzer116may assign parsed logic of the potentially relevant application210to an assigned node until the logic analyzer116identifies a potentially relevant API call or branching logic (e.g., if-then, switch-case, and do-while), and add the assigned node to the control flow graph433. Where a program includes multiple potentially relevant applications210, the logic analyzer116may merge the control flow graphs433produced for each potentially relevant application into a single control flow graph433. The logic analyzer116may obtain the API call graph434by comparing the edges in the control flow graphs433with the edges in the data flow graph432. For example, where a control flow graph433includes an edge that a data flow graph432does not include, the logic analyzer116may not include the edge in the corresponding API call graph434. However, where a control flow graph433includes an edge that the data flow graph432also includes, the logic analyzer116may include the edge in the API call graph434.

In one implementation, the logic analyzer116may receive user input to determine particular dependencies between API calls. For example, where a potentially relevant application210uses a function pointer (e.g., a type of pointer used in C, and C++ languages) to reference a potentially relevant API call212and a hash table (e.g., a data structure that associates keys with values) to store an object that represents a data element passed between API calls, the logic analyzer116may receive user input to determine dependencies between API calls because the logic analyzer116may otherwise interpret multiple possible dependencies between API calls when in fact only one or a finite set of valid dependencies exists.

In another implementation, the logic analyzer116may analyze the data flow paths (e.g., edges between nodes discussed below) (e.g., link heuristics) of the potentially relevant applications210, and potentially relevant API call logic212to determine the connectivity rankings214of each connection between potentially relevant API calls212. In one implementation, the heuristic relevance ranking engine114may determine the application heuristic relevance ranking216for the potentially relevant application210, shown inFIG. 7as potentially relevant application A, based on the total number of API calls ‘n’ represented by nodes712-720that represent different potentially relevant API calls212found in the potentially relevant application210, the total number of connections between the potentially relevant API calls212(e.g., edges712-720) equal to n(n−1) (e.g., data flow edge count614), the quality of the connections (e.g., strong connectivity or weak connectivity), and the type of link (e.g., loop link, single link, or no link) between the potentially relevant API calls212.

The applications metadata608and API metadata610may describe the data flow paths between the different potentially relevant API calls212(e.g., nodes702-710) within the potentially relevant application210. For example, the logic analyzer116may determine common API parameters626and logic branches (e.g., if-then-else) found within the potentially relevant application210and potentially relevant API calls212to generate the data flow graphs432, control flow graphs433and API call graphs434. The logic analyzer116may, asFIG. 7also illustrates, identify the function (e.g., K(x), J(x), S(y), P(y), F(x), and G(z)) of each potentially relevant API call212(e.g.,702-710, and722) to determine the connectivity rankings214.

In one implementation, the logic analyzer116may assign a weight Wi(e.g., connectivity ranking214) to each connection between the potentially relevant API calls212(e.g., nodes712-720). The logic analyzer116may assign weak connections a weight of 0.5 and strong connections a weight of 1.0 depending on multiple factors. For example, edge712, edge716and edge720may represent weak connections between potentially relevant API calls212represented by node pairs702and710,702and704, and706and708(e.g., function pairs K(x) and F(x), K(x) and J(x), and S(y) and P(y), respectively). Following the above example, where functions K(x) and F(x) share a common API parameter626, but neither function generates the value of the common API parameter626then the logic analyzer116may assign the connectivity ranking214between node pair702and710, represented by edge720, a weak connection weight of 0.5. A weak connection assigned to a node pair (e.g.,702and710) may indicate a low relative probability (e.g., in comparison to the connectivity rankings of other node pairs) that the node pair implements the high-level processing concept represented by the original query202. The logic analyzer116may use other heuristic analysis methods and tools to determine whether to assign a weak connection to a connectivity ranking214.

Alternatively, edge714, and edge718may represent strong connections between potentially relevant API calls212, represented by node pairs702and708, and704and706(e.g., function pairs K(x) and P(y), and J(x) and S(y), respectively). The logic analyzer116may determine that where function J(x) produces variable y, which both J(x) and S(y) share then the node pair704and706, represented by edge714, may be assigned a strong connectivity ranking622. A strong connection assigned to a node pair (e.g.,704and706) may indicate a high relative probability (e.g., in comparison to the connectivity rankings of other node pairs) that the node pair implements the high-level processing concept represented by the original query202. The logic analyzer116may use other heuristic analysis methods and tools to determine whether to assign a strong connection to a connectivity ranking214.

The logic analyzer116may also assign a link value L (e.g. link value620) to each connection between potentially relevant API calls212. For example, Exemplar system124may assign a link value L equal to 1 where a loop link (e.g., edges712-718form a loop) exists between potentially relevant API calls212(e.g., nodes702-708). Exemplar system124may assign a link value L equal to 0.5 where a single link (e.g., edge720) exists between potentially relevant API calls212(e.g., nodes702-708). In another implementation, Exemplar system124may assign a link value L equal to 0 where no link exists between potentially relevant API calls212(e.g., node722represents a potentially relevant API call212that does not have a connection with other potentially relevant API calls in a potentially relevant application210). Additional, different, or fewer weights may be used. The heuristic relevance ranking engine114may use the connectivity rankings214and link values620assigned to each connection between potentially relevant API calls212to determine the application heuristic relevance ranking216according to:

The logic analyzer116may determine an API call heuristic relevance ranking218for a potentially relevant API call212based on the connectivity ranking214and link value620assigned to each edge that includes the potentially relevant API call212. For example, where m represents the number of node pair including a particular node (e.g.,702and704,702and708, and702and710) and the number of edges (e.g.,712,718and720) that include the node equals m(m−1), and the assigned value for each connectivity ranking214and link value620for each edge that includes the node represent W and L, respectively, the API call heuristic relevance ranking218for the node may be determined according to Equation 1 above where m substitutes for n:

FIG. 8shows the acts that the Exemplar system124may take to obtain a relevant application list210. The Exemplar system124may use the interface logic510to receive an original query202representing a high-level processing concept (e.g., “send secure XML”) (802). The help content processor104may execute an original search using the original query202(804) to obtain an original search result that includes basis logic results516(e.g., basis API call list204or basis application list520). The basis logic results516may include a basis logic (e.g., API call) identifier (806). The Exemplar system124may combine the original query202with any part of the basis logic list to form an expanded query (808). The expanded search logic514may execute the expanded query206to obtain an expanded search result208that includes a potentially relevant applications210list and potentially relevant API calls212list (810). The potentially relevant logic list may identify potentially relevant logic. The analyzer logic606may analyze the potentially relevant logic identified by the potentially relevant application identifiers with respect to the logic repository112(812). The heuristic relevance ranking logic602may use the connectivity rankings214, and link values620to determine the application heuristic relevance rankings216and API call heuristic relevance rankings218for the potentially relevant applications210and potentially relevant API calls212using Equation 1 or another formulation (814). The heuristic relevance ranking logic602may apply the application relevance threshold428and the API call relevance threshold430using the threshold logic604to determine whether the potentially relevant applications210and the potentially relevant API calls meet the application relevance threshold428, and API call relevance threshold430, respectively (816). The heuristic relevance ranking logic602may add the potentially relevant application210to the relevant application list220where the potentially relevant application210meets the application relevance threshold428(818). The heuristic relevance ranking logic602may determine the application heuristic relevance ranking216and API call heuristic relevance ranking218for each potentially relevant application210and potentially relevant API call212included the expanded search result208(820).

FIG. 9shows the acts that the Exemplar system124heuristic ranking engine114may take to assign an application heuristic relevance ranking216to a potentially relevant application210. The analyzer logic602may analyze the potentially relevant application210and potentially relevant API calls212found in the logic repository (904). The analyzer logic602may generate and analyze a data flow graph432and control flow graph433(906) used to generate and analyze an API call graph43(908). The analyzer logic602may use the data flow graph432, control flow graph433, and API call graph to determine the link values650for the potentially relevant API calls included in a potentially relevant application210and assign a connectivity ranking214to each connection between potentially relevant API calls212(910). The heuristic relevance ranking logic602may determine an application heuristic relevance ranking216for each of the potentially relevant applications210(912) (e.g., the application heuristic relevance ranking216and an API call heuristic relevance ranking218may be determined according to Equation 1, as discussed above). The heuristic relevance ranking logic602may determine the application heuristic relevance ranking216and API call heuristic relevance ranking218for each potentially relevant application210and potentially relevant API call212included the expanded search result208(914).

FIG. 10shows the processing that the selection logic and application creation logic may take to generate a new application440. EPAD project area400may use the selection logic624to detect selection of a relevant API call Identifier (e.g., as indicated by arrows426drawn from the relevant API call identifier-1418and the relevant API call identifier-2420to the relevant API call logic-1422and the relevant API call logic-2424) from the relevant API call list222(1002). The EPAD project area400may present the relevant API call logic (e.g., the relevant API call logic-1422and the relevant API call logic-2424) that implements the relevant API calls (1004). The EPAD project area may use the selection logic624to detect selection of the relevant API call logic (e.g., the relevant API call logic-1422and the relevant API call logic-2424) and the other relevant application logic436to generate a new application440(1006). The EPAD project area may provide an option to generate the new application440, using the application creation logic626, following selection of the relevant API call logic (e.g., the relevant API call logic-1422and the relevant API call logic-2424) and the other relevant application logic436to generate a new application440(1008). Exemplar system124may also identify requirements444for the high-level processing concept represented by the original query202. In one implementation, Exemplar system124may generate requirements documentation and end user documentation based on the help content106related to the other relevant application logic436, the relevant API call logic-1422, and the relevant API call logic-2424combined to generate the new application440and identify the requirements444.

FIG. 11shows a recommended application evaluation system (RAE)128. In one implementation, the RAE128includes a communications interface1102in communication with the network118and used to communicate with various resources internal and external to the RAE128, a memory1104, and a processor1106. The processor1106may execute any of the logic of the RAE128and/or Exemplar124. The memory1106may include the RAE user interface generation logic130that generates a RAE user interface1202, as shown inFIG. 12, and employs the RAE area logic132.

The RAE area logic132may include graphics libraries, window rendering calls, and other user interface logic operable to display interface elements, receive input, and pass the input to any particular program logic in the RAE128. The RAE area logic132may further include the concept query display logic134, the concept distribution display logic136, the source code display logic138and the metadata display logic140. The concept query display logic134may include topics1108, words1110, and concepts1112of the original query202and/or the expanded query206. The concept query display logic134accepts the topics1108, words1110, concepts1112, the original query202and the expanded query206from Exemplar, and generates a window in the RAE user interface1202(as described in more detail in connection withFIG. 12) that presents the topics as selectable focus items for verifying recommended application functionality.

The concept distribution display logic136may include recommended application representations1116of the recommended applications1118(e.g., HRAs). In one implementation, the recommended applications1118include the relevant application logic414and relevant API call logic (e.g.,422and424) obtained from Exemplar based on the original query202and/or the expanded query206. The recommended application representations1116include concept distribution locators1120that identify potential implementation locations of one of the topics, concepts and words. Exemplar identifies the location (e.g., concept distribution locators) in application logic where an API call implements a high-level processing concept. The concept distribution display logic136generates a visualization of a recommended application. The concept distribution display logic136may, for example, generate a concept distribution display region1206that shows one or more of the modules, source code files, header files, library files, or other recommended application source files. The concept distribution display logic136further locates within the recommended application source files the API calls that implement the topics1108, words1110, concepts1112, the original query202and the expanded query206, selected by an operator. The concept distribution display logic136may denote each location with a concept distribution locator1120, such as a solid horizontal line with hyperlink functionality through the recommended application source file at the location where the API call exists in the source file. Other concept distribution locators1120may be used such as geometric shapes (e.g., circles, squares) and (e.g., fonts, graphics (e.g., lines or pointers), color and/or sound).

The concept distribution display logic136is operable to receive concept distribution locator selections1122. The concept distribution display logic136may convey the concept distribution locator selection1122to the source code display logic138. As described in more detail below, the source code display logic138generates a source code display region on the user interface which displays the actual code at the selection point. As a result, an operator may evaluate the code to determine whether the code actually implements the search terms selected from the concept query display region.

The source code display logic138may include source code portions1124of a recommended application1118. The source code portions1124may include the potential implementation location of source code (e.g., highlighted concepts1126) for a selected concept distribution locator1122. The source code display logic138accepts a concept distribution location selection1122from the concept distribution display logic136. In response, the source code display logic138retrieves the recommended application code at the selection point and presents the source code for review in the source code display region. Furthermore, the concept distribution display logic136highlights source code components in the display such as API calls in the source code that match the concepts selected from the concept query display region. Accordingly, an operator may review the source code to determine whether the source code actually does implement the concepts selected from the concept query display region. The source code display logic138may include read-only and edit modes that allow the source code display logic138to be responsive to source code edits1128entered by a developer in the course of evaluating the recommended application1118.

The metadata display logic140may include application metadata608and API metadata610for the recommended applications1118based on the topics1108, words1110, concepts1112, the original query202and the expanded query206. The metadata display logic140may also include concept query statistics1130and semantics information1132based on the topics1108, words1110, concepts1112, the original query202, the expanded query206and/or recommended applications1118. In one implementation, concept query statistics1130and semantics information1132are determined by the heuristic relevance ranking engine114as a product of determining the API call heuristic relevance rankings218of the relevant API call logic-1422and the relevant API call logic-2424, based on analyzing semantics derived from the expanded query206and the expanded search result208.

FIG. 12shows a recommended application evaluation area1202that the RAE area logic132may be operable to display. The RAE area1202may include a concept query display region1204, a concept distribution display region1206, source code display region1214and metadata display region1210. The concept query display region1204displays the topics1108, words1110, concepts1112of the original query202and/or the expanded query206alone or in combination. The concept query display region1204may further include check boxes and/or other selection GUI elements so that an operator may select the topics1108, words1110, and concepts1112from the concept query display region1204.

The concept distribution display region1206displays the recommended application representations (RAR)1212and1214of a recommended application1118. For example, the RAR1212may represent the source code file that includes the main( ) function, while the RAR1214may represent the source code file with an alleged JPEG to BITMAP conversion routine. The recommended application representations1212and1214include concept distribution locators1216,1218,1220,1222,1224, and1226inserted by the concept distribution display logic136that identify potential implementation locations in the source code files of the topics1108, words1110, concepts1112, the original query202and the expanded query206alone or in combination. The concept distribution locators1216,1218,1220,1222,1224, and1226may employ different identifiers (e.g., fonts, graphics (e.g., lines or pointers), color and/or sound) to identify a relationship between the concept distribution locators and the topics1108, words1110, concepts1112of the original query202and the expanded query206. The source code display region1208includes highlighted concepts1126(e.g.,1228and1230). In one implementation, the highlighted concepts1126(e.g.,1228and1230) include identifiers (e.g., fonts, color and/or sound) to indentify a relationship between the highlighted concepts1126and the concept distribution locators (e.g.,1216,1218,1220,1222,1224, and1226). The RAE area logic132may use fonts, color and/or sound (e.g., audio queues) to indicate relationships between the topics1108, words1110, concepts1112, concept distribution locators (e.g.,1216,1218,1220,1222,1224, and1226) and highlighted concepts1126, as indicated by the arrows1232and1234.

In one implementation, the recommended application representations (1116,1212and1214) and/or the concept distribution locators (e.g.,1216,1218,1220,1222,1224, and1226) are displayed in response to the selection of a particular topic1108, word1110, concept1112, and/or element of the original query202and/or the expanded query206. In other words, the recommended application representations (1116,1212and1214) and/or the concept distribution locators (e.g.,1216,1218,1220,1222,1224, and1226) displayed in the concept distribution display region1206may be responsive to selections made in the concept query display region1204.

FIG. 13shows an example of the recommended application evaluation area1202as shown on a display. The recommended application evaluation area1202includes the concept query display region1204, concept distribution display region1206, a recommended application representation1214, and a metadata display region1210. The concept query display region1204provides term selection elements1302that an operator may select to chose the concepts of interest for verification in the recommended applications. The concept distribution display region1206shows the location of API calls and other source code components that implement the selected terms. In the example shown inFIG. 13, the locations are indicated using concept distribution locators: dashed lines (e.g.,1304) proportionally positioned within graphical representations of the source code files that make up a recommended application. These source code files are labelled Recommended Application Code (RAC) file1(RAC1), RAC2, RAC3, RAC4, and RAC5inFIG. 13.

The recommended application representation1214shows the source code at a specific concept location in a specific source code file, as selected by an operator. Thus, when an operator selects, for example, RAC1, and the concept distribution locator1304, the concept distribution display logic136responds by retrieving the RAC1source code at and around the concept location. The concept distribution display logic136displays the retrieved code in the recommended application representation1214area. Furthermore, the concept distribution display logic highlights the sections of code that Exemplar's search techniques determined relate to the search terms shown in the concept query display region1204. As shown inFIG. 13, for example, the concept distribution display logic136has highlighted (using underlining) the source code components1306“new” and “.set” as relevant to the selected search terms “length” and “create” in the concept query display region1204.

FIG. 14shows the processing that the RAE area logic132may take to evaluate a recommended application. The RAE area logic132obtains recommended applications for evaluation (1402). In one implementation, the RAE area logic132obtains the recommended applications for evaluation from Exemplar124, where the recommended applications1118are HRAs. The concept query display logic134displays the topics, concepts and words of the original query202and/or the expanded query206executed to identify the recommended applications (1404). The concept query display logic134displays the topics, concepts and words in a concept query display area. The concept distribution display logic136displays representations of the recommended applications1118, including concept distribution locators1120in the concept distribution display region1206(1406). The concept distribution locators1120identify potential implementation locations of the topics, concepts and words of the query executed to identify the recommended applications1118. When a concept distribution locator1120is selected from the concept distribution display region1206(1408) the source code display logic138displays a portion of the source code1124of the recommended application identified by the concept distribution locator1120(1410) in the source code display region. The metadata display logic140displays the metadata generated upon the execution of the query executed to identify the recommended applications (1412). A developer may use the RAE128to evaluate the recommended application and portion of source code to validate that the recommended application is a highly relevant application (1414). The developer may select another recommended application to evaluate (1416) by selecting another concept distribution locator from the concept distribution display area.

The RAE128greatly reduces the time, cost, and other resource expenditures associated with evaluating a recommended application. The RAE128produces relevant results starting with high-level processing concepts. A software developer may deploy the highly relevant application into production upon successful evaluation.

Furthermore, it is noted that the system carries out electronic transformation of data that may represent underlying physical objects. For example, the RAE area logic visually transforms source code by adding search term locators and highlighting of relevant code that matches search terms. In addition, the RAE may be implemented as a particular machine. For example, the particular machine may include a CPU, memory, and a software library for carrying out the RAE area logic noted above.