Patent Description:
The foregoing is merely an introduction. For instance, previously translated software in the form of libraries, operating systems, or hypervisors may also be invoked or otherwise relied upon by a given program. But even when previously translated software is available, developers are often called upon to create new source or to modify existing source code. Accordingly, technologies that help developers edit source code to achieve their desired semantics within a given syntax, and to meet other goals, may be worth investigating. <CIT> describes that methods, systems, and computer program products are provided for inferring the programming intent of code developers to suggest code solutions. Program code is retrieved from a code repository that includes program code generated by a plurality of code developers. The program code is analyzed to determine one or more program code design patterns. A knowledge set is generated that includes the determined program code design pattern(s), and that is network-accessible by software development applications to provide program code suggestions for developing software programs. <CIT> describes that lines of code entered in a source code editor graphical user interface (GUI) in a particular coding project are identified and similarities are autonomously determined between the particular coding project and a subset of coding project templates in a plurality of coding project templates based on the lines of code. The subset of coding project templates is based on a set of other coding projects, and the similarities are determined based at least in part on a correlation between the lines of code of the particular coding project and lines of code in the set of other coding projects. Code suggestions are determined, which are defined according to the subset of code templates, for lines of code following the particular lines of code. A particular GUI window is presented for display with the source code editor GUI, where the suggestions are presented within the particular GUI window.

Code synthesizers can automatically create source code editing suggestions, sometimes doing so in large quantities which may or may not have much diversity in form or effect. A code editing suggestion created by a code synthesizer or by machine intelligence may be referred to as an "autocreated suggestion", an "autocreated code editing suggestion", an "autocreated source code editing suggestion", an "automatically created software development suggestion", or an "ACSDS", for example. Such autocreated suggestions may be presented to developers in autocompletion interfaces or quick action (e.g., "lightbulb") interfaces of software development tools, but presenting each and every created suggestion could rapidly overwhelm and frustrate developers as they try to decide which suggestions are trustworthy.

Some embodiments described in this document provide improved technology for software development, which can increase developer productivity and code consistency, by determining automatically which autocreated coding suggestions are presented to developers. Some embodiments automatically manage autocreated suggestions by identifying suggestions that have been endorsed by a developer's team and filtering out ones that have not been endorsed, based for instance on a trust score. Some ACSDS filters can operate within and enhance existing development team practices of code review, and enhance existing editors or code review tools. Some ACSDS filter embodiments decrease or eliminate developer configuration burdens while increasing or maximizing consistent implementation of endorsed changes inside a codebase.

Some embodiments use or provide a software development hardware and software combination which includes a digital memory, and a processor which is in operable communication with the memory. The processor is configured, e.g., by tailored software, to perform suggestion filtering steps. The steps may include automatically (a) obtaining provenance metadata of an automatically created software development suggestion (ACSDS), the provenance metadata indicating implicit or explicit software developer endorsement of the ACSDS, (b) assigning a provenance-derived trust score to the ACSDS based at least in part on the provenance metadata, (c) comparing the provenance-derived trust score to a criterion for supplying the ACSDS to a suggestion presentation interface of a software development tool, and (d) supplying the ACSDS to the suggestion presentation interface or withholding the ACSDS from the suggestion presentation interface, in response to a result of the comparing. This example system tends to present automatically created suggestions to software developers more often or less often depending on whether other software developers have or have not at least implicitly endorsed the automatically created suggestions.

Some embodiments use or provide steps of a software development tool method. The steps may include obtaining provenance metadata of an ACSDS, ascertaining at least one provenance value at least in part by using the metadata, computing a provenance-derived trust score for the ACSDS based at least in part on a result of the ascertaining, assigning the provenance-derived trust score to the ACSDS, and employing the provenance-derived trust score within a software development tool.

For these embodiments, some examples of ascertained provenance values may include: a codebase utilization level which indicates an extent to which the ACSDS has been utilized within a codebase; a codebase utilizer identity which identifies at least one developer who submitted one or more suggestion-based changes to a codebase, each suggestion-based change including a software implementation of the ACSDS; a developer review level which indicates an extent to which the ACSDS has been reviewed by at least one developer; a suggestion saver identity which identifies at least one developer who chose to save the ACSDS instead of discarding it; a library affiliation which indicates that the ACSDS is affiliated with a library by an authorized author of the library or an authorized distributor of the library; or a repository affiliation which indicates that the ACSDS is affiliated with a code repository by an authorized user of the repository.

Some embodiments use or provide a computer-readable storage medium configured with data and instructions, or use other computing items, which upon execution by a processor cause a computing system to perform a suggestion filtering method. In particular, some embodiments obtain provenance metadata of an ACSDS. The provenance metadata includes a codebase utilization level which indicates an extent to which the ACSDS has been utilized within at least one codebase. These embodiments assign a provenance-derived trust score to the ACSDS based at least in part on the provenance metadata, and compare the provenance-derived trust score to a criterion for supplying the ACSDS to a suggestion presentation interface of a software development tool. Then the embodiments either supply the ACSDS to the suggestion presentation interface of a software development tool or withhold the ACSDS from the suggestion presentation interface, in response to a result of the comparing.

Other technical activities and characteristics pertinent to teachings herein will also become apparent to those of skill in the art. The examples given are merely illustrative. Rather, this Summary is provided to introduce - in a simplified form - some technical concepts that are further described below in the Detailed Description. The innovation is defined with claims as properly understood, and to the extent this Summary conflicts with the claims, the claims should prevail.

A more particular description will be given with reference to the attached drawings. These drawings only illustrate selected aspects and thus do not fully determine coverage or scope.

Innovations may expand beyond their origins, but understanding an innovation's origins can help one more fully appreciate the innovation. In the present case, some teachings described herein were motivated by technical challenges faced by Microsoft innovators who were working to improve the feature set and usability of Microsoft development tool offerings, including some versions of Microsoft Visual Studio® or VSCode™ integrated development environments (marks of Microsoft Corporation).

Initially, the innovators recognized that program synthesis technology, such as Microsoft PROSE™ technology and other machine intelligence technologies, can be used to automatically create suggestions for software edits. These autocreated suggestions can be displayed to a software developer, e.g., in the form of an IntelliCode™ menu item or another autocompletion or quick action suggestion in an integrated development environment tool or a text editor, for example (marks of Microsoft Corporation).

In some situations, the autocreated suggestions are synthesized or otherwise derived from edits made by a particular developer and are then displayed to the same developer. For example, after determining that a developer has changed, in a particular way, several instances of source code that match a certain pattern, when the next instance of that pattern is encountered during an editing session the editing tool may recognize the instance and then proactively offer the developer an option to change the instance in the same way the earlier instances were changed. Recognition of the pattern may be more general than a simple string comparison; regular expressions may be matched, for example.

On further consideration, the innovators recognized that the potential presentation or actual presentation or use of autocreated suggestions could be challenging when more than one developer is working on a given application program or another shared codebase. For example, one technical challenge faced by the innovators was how to determine whether an autocreated suggestion that was generated to assist one developer would be presented later to another developer. Emergent associated technical challenges were how to configure control over autocreated suggestion presentation, how to integrate autocreated suggestion presentation into existing developer tools and workflows, what origination or other information would be maintained about autocreated suggestions, and whether to present some or all of that information along with the suggestion when the suggestion is presented to a developer in a tool.

One goal of some resulting embodiments is to avoid flooding a developer with unwanted autocreated suggestions. Another goal of some resulting embodiments is to avoid burdening a developer with responsibility for configuring the management of autocreated suggestions; freed of configuration burdens, the developer can concentrate instead on code development. One of skill will recognize these and other technical challenges of managing autocreated suggestions in development team environments, as challenges are addressed at various points within the present disclosure.

To help address such technical challenges, some embodiments compute a trust score for autocreated suggestions. The trust score is based on metadata that indicate the extent (at least) and possibly also the nature of any endorsement, by other developers, of an autocreated suggestion. As used here, "developer" means a software developer who is a human person, as opposed to being a software process or computing device, for example. Endorsement of an autocreated suggestion may take various forms, such as developer review of the autocreated suggestion, submission to a repository of changes that were made when adopting the autocreated suggestion, or an affiliation of the autocreated suggestion with a library release as a suggested way to update library-using code that would otherwise be broken by changes made in the library release. The trust score may be viewed as a provenance-derived trust score, because it is based on the provenance (i.e., the origin or history or both) of the autocreated suggestion and its usage.

Some embodiments compare the trust score to a cutoff. Then they either present the autocreated suggestion to the developer as an option, if it is sufficiently trustworthy, or else withhold it from presentation, if it is not. Filtering suggestions based on endorsement metadata helps prevent a development tool from overwhelming the developer with autocreated suggestions, and helps the developer understand the provenance of such suggestions. Even though some of the withheld autocreated suggestions may be harmless, or would even be helpful, if they have not been sufficiently endorsed they are automatically withheld from view. Thus, the developer is not compelled to either sort manually through many autocreated suggestions to find the most useful ones, or to individually define and tune an ad hoc filter mechanism to categorize the autocreated suggestions. Presenting endorsed suggestions also promotes consistency of code within a given codebase; in effect, embodiments can help developers see and comply with a "this is how we do it" team approach to coding.

Other aspects of these embodiments, and other enhanced software development tool embodiments, are also described herein.

With reference to <FIG>, an operating environment <NUM> for an embodiment includes at least one computer system <NUM>. The computer system <NUM> may be a multiprocessor computer system, or not. An operating environment may include one or more machines in a given computer system, which may be clustered, client-server networked, and/or peer-to-peer networked within a cloud. An individual machine is a computer system, and a network or other group of cooperating machines is also a computer system. A given computer system <NUM> may be configured for end-users, e.g., with applications, for administrators, as a server, as a distributed processing node, and/or in other ways.

Human users <NUM> may interact with the computer system <NUM> by using displays, keyboards, and other peripherals <NUM>, via typed text, touch, voice, movement, computer vision, gestures, and/or other forms of I/O. A screen <NUM> may be a removable peripheral <NUM> or may be an integral part of the system <NUM>. A user interface may support interaction between an embodiment and one or more human users. A user interface may include a command line interface, a graphical user interface (GUI), natural user interface (NUI), voice command interface, and/or other user interface (UI) presentations, which may be presented as distinct options or may be integrated.

System administrators, network administrators, cloud administrators, security analysts and other security personnel, operations personnel, developers, testers, engineers, auditors, and end-users are each a particular type of user <NUM>. Automated agents, scripts, playback software, devices, and the like acting on behalf of one or more people may also be users <NUM>, e.g., to facilitate testing a system <NUM>. Storage devices and/or networking devices may be considered peripheral equipment in some embodiments and part of a system <NUM> in other embodiments, depending on their detachability from the processor <NUM>. Other computer systems not shown in <FIG> may interact in technological ways with the computer system <NUM> or with another system embodiment using one or more connections to a network <NUM> via network interface equipment, for example.

Each computer system <NUM> includes at least one processor <NUM>. The computer system <NUM>, like other suitable systems, also includes one or more computer-readable storage media <NUM>. Storage media <NUM> may be of different physical types. The storage media <NUM> may be volatile memory, non-volatile memory, fixed in place media, removable media, magnetic media, optical media, solid-state media, and/or of other types of physical durable storage media (as opposed to merely a propagated signal or mere energy). In particular, a configured storage medium <NUM> such as a portable (i.e., external) hard drive, CD, DVD, memory stick, or other removable non-volatile memory medium may become functionally a technological part of the computer system when inserted or otherwise installed, making its content accessible for interaction with and use by processor <NUM>. The removable configured storage medium <NUM> is an example of a computer-readable storage medium <NUM>. Some other examples of computer-readable storage media <NUM> include built-in RAM, ROM, hard disks, and other memory storage devices which are not readily removable by users <NUM>. For compliance with current United States patent requirements, neither a computer-readable medium nor a computer-readable storage medium nor a computer-readable memory is a signal per se or mere energy under any claim pending or granted in the United States.

The storage medium <NUM> is configured with binary instructions <NUM> that are executable by a processor <NUM>; "executable" (and "execution") are used in a broad sense herein to include (running) machine code, interpretable code, bytecode, and/or code that runs on a virtual machine, for example. The storage medium <NUM> is also configured with data <NUM> which is created, modified, referenced, and/or otherwise used for technical effect by execution of the instructions <NUM>. The instructions <NUM> and the data <NUM> configure the memory or other storage medium <NUM> in which they reside; when that memory or other computer readable storage medium is a functional part of a given computer system, the instructions <NUM> and data <NUM> also configure that computer system. In some embodiments, a portion of the data <NUM> is representative of real-world items such as product characteristics, inventories, physical measurements, settings, images, readings, targets, volumes, and so forth. Such data is also transformed by backup, restore, commits, aborts, reformatting, and/or other technical operations.

Although an embodiment may be described as being implemented as software instructions executed by one or more processors in a computing device (e.g., general purpose computer, server, or cluster), such description is not meant to exhaust all possible embodiments. One of skill will understand that the same or similar functionality can also often be implemented, in whole or in part, directly in hardware logic, to provide the same or similar technical effects. Alternatively, or in addition to software implementation, the technical functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without excluding other implementations, an embodiment may include hardware logic components <NUM>, <NUM> such as Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip components (SOCs), Complex Programmable Logic Devices (CPLDs), and similar components. Components of an embodiment may be grouped into interacting functional modules based on their inputs, outputs, and/or their technical effects, for example.

In addition to processors <NUM> (e.g., CPUs, ALUs, FPUs, TPUs and/or GPUs), memory / storage media <NUM>, and displays <NUM>, an operating environment may also include other hardware <NUM>, such as batteries, buses, power supplies, wired and wireless network interface cards, for instance. The nouns "screen" and "display" are used interchangeably herein. A display <NUM> may include one or more touch screens, screens responsive to input from a pen or tablet, or screens which operate solely for output. In some embodiments peripherals <NUM> such as human user I/O devices (screen, keyboard, mouse, tablet, microphone, speaker, motion sensor, etc.) will be present in operable communication with one or more processors <NUM> and memory.

In some embodiments, the system includes multiple computers connected by a wired and/or wireless network <NUM>. Networking interface equipment <NUM> can provide access to networks <NUM>, using network components such as a packet-switched network interface card, a wireless transceiver, or a telephone network interface, for example, which may be present in a given computer system. Virtualizations of networking interface equipment and other network components such as switches or routers or firewalls may also be present, e.g., in a software-defined network or a sandboxed or other secure cloud computing environment. In some embodiments, one or more computers are partially or fully "air gapped" by reason of being disconnected or only intermittently connected to another networked device or remote cloud or enterprise network. In particular, autocreated suggestion filtering functionality could be installed on an air gapped network and then be updated periodically or on occasion using removable media. A given embodiment may also communicate technical data and/or technical instructions through direct memory access, removable nonvolatile storage media, or other information storage-retrieval and/or transmission approaches.

One of skill will appreciate that the foregoing aspects and other aspects presented herein under "Operating Environments" may form part of a given embodiment. This document's headings are not intended to provide a strict classification of features into embodiment and non-embodiment feature sets.

One or more items are shown in outline form in the Figures, or listed inside parentheses, to emphasize that they are not necessarily part of the illustrated operating environment or all embodiments, but may interoperate with items in the operating environment or some embodiments as discussed herein. It does not follow that items not in outline or parenthetical form are necessarily required, in any Figure or any embodiment. In particular, <FIG> is provided for convenience; inclusion of an item in <FIG> does not imply that the item, or the described use of the item, was known prior to the current innovations.

<FIG> illustrate aspects of an environment having an enhanced system <NUM>, <NUM> with a software development tool <NUM> that includes filtering functionality <NUM>. Specifically, as shown in <FIG> the tool <NUM> includes functionality <NUM> for filtering autocreated suggestions <NUM> to control which suggestions <NUM> are presented to a developer <NUM> in conjunction with source code <NUM> displayed in a user interface <NUM> of the development tool <NUM>.

The suggestions <NUM> are created by a code synthesizer, by a machine intelligence-driven code generator, or by another mechanism <NUM>, as opposed to being conceived by a human. For example, machine intelligence may be used to find patterns in a set of code changes. In general, some of the autocreated suggestions <NUM> will be adopted to a large extent by developers, while other suggestions will be adopted less or not at all. A suggestion is "adopted" when a developer (or a tool performing a command from the developer) applies the suggestion to modify the code <NUM>. One benefit of some embodiments is increased adoption of suggestions <NUM> that have been endorsed by prior review or adoption elsewhere. Another benefit of some embodiments is reduction or avoidance of time and effort spent by developers on considering non-endorsed autocreated suggestions <NUM>.

The developer <NUM> who is editing or reviewing the source code <NUM> may be the only developer responsible for working on that code <NUM>, but it is also contemplated that in many of the scenarios that can benefit from teachings herein the developer will be a member of a multi-person development team <NUM>. In a team <NUM> development scenario, one or more developers may be responsible for development of a particular source code file, e.g., responsible for writing, modifying, documenting, reviewing, analyzing, correcting, testing, or otherwise performing software development actions with the code in the file. Even if a particular developer is the only person currently responsible for a particular file of source code <NUM> which is being displayed, that source code is often part of a larger codebase <NUM> that is collectively edited, reviewed, or otherwise developed by multiple members of the team. One benefit of some embodiments is greater consistency in the adoption of particular autocreated suggestions <NUM> by different developers within a given codebase.

<FIG> illustrates embodiments in which the filtering functionality <NUM> includes suggestion presentation software <NUM>. From a user interface design perspective, the suggestion presentation software <NUM> may look and operate largely like a non-enhanced tool, but differences will become apparent. One difference in many embodiments is the presence and operation of filter code <NUM> that tests suggestions' <NUM> trust scores <NUM> against one or more filter criteria <NUM>, rather than simply allowing all suggestions <NUM> to be displayed to users. Another difference in some embodiments is the display of suggestion metadata <NUM>. Still other differences will be apparent to one of skill in view of the present disclosure.

The teachings provided herein can be applied to enhance many kinds of development tools <NUM>. These teachings can be applied to help developers as they work with source code <NUM> in various programming languages, such as C#, C++, Objective-C, Fortran, COBOL, Java, Python, Perl, or another general-purpose programming language, other imperative general purpose languages, declarative languages, or even special-purpose languages such as those used for string formatting, document object model component selection, or other purposes.

In some embodiments, an enhanced tool includes a source code editor <NUM>. In some, the enhanced tool <NUM> includes a code analyzer <NUM>, e.g., a static analysis tool <NUM> such as a Microsoft Roslyn™ analyzer (mark of Microsoft Corporation) enhanced with functionality <NUM>, or a code review tool <NUM>, e.g., a pull request <NUM> review tool <NUM> enhanced with functionality <NUM>. In some embodiments the enhanced tool <NUM> includes data <NUM> which was generated through interaction with a code repository <NUM>, such as one using GitHub® (mark of GitHub, Inc. ), BitBucket® (mark of Atlassian Pty Ltd), SourceForge® (mark of SourceForge Media, LLC), or other repository software <NUM>. Pull requests <NUM> are one example of repository data <NUM>; source code <NUM> sent to or retrieved from a repository are another example of repository data <NUM>, since any data <NUM> sent or received through communication with a repository <NUM> is deemed repository data <NUM> herein. In some of these cases, and in others, the enhanced tool <NUM> may be, or be within, or include, an integrated development environment <NUM>.

The editors <NUM>, analyzers <NUM>, repository tools <NUM>, IDEs <NUM>, and other tools <NUM> which are enhanced according to teachings presented herein include or communicate with a user interface <NUM> that configures a visible display <NUM>. The filtering functionality <NUM> code <NUM> executes to influence or otherwise control which autocreated suggestions <NUM> will be shown on that display <NUM> by the user interface <NUM>, and to influence or otherwise control what metadata <NUM> about the provenance <NUM> of the suggestions <NUM> is used in the filtering. The filtering functionality <NUM> also controls what provenance metadata <NUM> is shown on the display in order to help developers understand and assess autocreated suggestions <NUM>.

In some embodiments the suggestion presentation software <NUM> is designed and configured to comfortably enhance an existing development workflow. Such "comfortable enhancement" may be accomplished, e.g., by virtue of an embodiment having one or more of the following characteristics: (a) not requiring a developer to consult guidance outside the development tool <NUM> user interface <NUM> to decide whether to adopt an autocreated suggestion <NUM>, (b) enhancing one or more familiar development functionalities such as source code editing, autocompletion, quick actions (a. "lightbulb" actions, "code actions", or "quick fixes"), or source code analysis, (c) utilizing familiar user interface mechanisms (albeit in new ways) such as dialog boxes, tooltips, cursors, visual highlighting, and flowing of text in response to text insertion, or (d) presenting endorsed autocreated suggestions at the point in a tool <NUM> where they are most useful.

The system <NUM> may be networked generally or communicate via network packets, shared memory, shared files, or otherwise with a cloud, a repository <NUM> or other source code control system <NUM>, other network(s) <NUM>, and other computing devices through one or more interfaces <NUM>. An interface <NUM> may include hardware such as network interface cards, software such as network stacks, APIs, or sockets, combination items such as network connections, or a combination thereof.

<FIG> illustrates some aspects of provenance metadata <NUM>. <FIG> illustrates some examples of configuration burdens <NUM>. <FIG> illustrates some aspects of some autocreated suggestions <NUM>. These illustrated items are discussed at various points herein, and additional details regarding them are provided in the discussion of a List of Reference Numerals later in this disclosure document.

With reference now to all of the drawing figures, some embodiments use or provide a functionality-enhanced system, such as system <NUM> or another system <NUM> that is enhanced with suggestion filtering as taught herein. In some embodiments, a system which is configured for software development includes a digital memory <NUM> and a processor <NUM> in operable communication with the memory. The processor <NUM> is configured to perform suggestion filtering steps. The steps may include automatically (a) obtaining provenance metadata <NUM> of an automatically created software development suggestion (ACSDS) <NUM>, the provenance metadata indicating implicit software developer endorsement <NUM> of the ACSDS or explicit software developer endorsement <NUM> of the ACSDS, (b) assigning a provenance-derived trust score <NUM> to the ACSDS based at least in part on the provenance metadata, (c) comparing the provenance-derived trust score to a criterion <NUM> for supplying the ACSDS to a suggestion presentation interface <NUM> of a software development tool <NUM>, and (d) supplying the ACSDS to the suggestion presentation interface or withholding the ACSDS from the suggestion presentation interface, in response to a result of the comparing. As it gains access to autocreated suggestions over time, this enhanced system tends to present automatically created suggestions <NUM> to software developers more often or less often depending on whether other software developers have or have not at least implicitly endorsed the automatically created suggestions. Endorsed suggestions are more likely to be deemed by a team the kind of suggestion whose change belongs in a codebase the team is working on.

Some embodiments will comfortably enhance an existing development team workflow by favoring endorsed autocreated suggestions <NUM> over non-endorsed suggestions. Endorsements <NUM>, <NUM> may be detected by automatically scanning logs or other output data of tools <NUM> whose use is already established by a development team. For instance, enhanced tools <NUM> may operate to support code reviews or other workflow performed by a development team or by other co-workers developing a codebase <NUM>. Changes by team members to their established workflow behaviors (e.g., doing code reviews and submitting pull requests) are not required in order for the developed software to benefit from tool enhancements that apply suggestion filtering as taught herein.

In some embodiments, sufficiently trusted suggestions are presented in a code editor, e.g., by IntelliCode™ technology. In some, such suggestions are presented by a code analysis tool, e.g., a Roslyn™ analyzer (marks of Microsoft Corporation). Thus, in some embodiments the system <NUM> is characterized in at least one of the following ways: the software development tool <NUM> includes a source code editor <NUM> and the suggestion presentation interface <NUM> operates within the source code editor, the software development tool <NUM> includes a source code review tool <NUM> and the suggestion presentation interface <NUM> operates within the source code review tool, or the software development tool <NUM> includes a source code analysis tool <NUM> and the suggestion presentation interface <NUM> operates within the source code analysis tool.

In particular, in some embodiments the enhanced source code analysis tool <NUM> is configured to perform a repository pull request <NUM>. Repository pull requests <NUM> are also referred to herein as "repo pulls" or "repo pull lists". Repo pull lists can be employed by an embodiment as evidence of codebase utilization <NUM> of a suggestion <NUM> endorsement <NUM> in the form of an implementation <NUM> of a change <NUM> to the codebase <NUM>. The endorsement is implicit, in that changes placed in a repo by a pull request will not necessarily be included in a build, and will not necessarily be executed event if a build makes them part of an executable. Repo pull lists can also or instead be employed by an embodiment as evidence of developer review <NUM> of a suggestion <NUM>, although in practice a particular change placed in a repo <NUM> by a pull request will not necessarily have been actually considered by the developer making the pull request <NUM>, especially if multiple changes <NUM> are placed in a single pull request.

In some embodiments, the provenance metadata <NUM> includes at least one of the following: a codebase utilization level <NUM> which indicates an extent to which the ACSDS <NUM> has been utilized within a codebase <NUM>; a codebase utilizer identity <NUM> which identifies at least one developer who submitted one or more suggestion-based changes <NUM> to a codebase, each suggestion-based change including a software implementation <NUM> of the ACSDS <NUM>; a developer review level <NUM> which indicates an extent to which the ACSDS has been reviewed by at least one developer; a suggestion saver identity <NUM> which identifies at least one developer who chose to save the ACSDS instead of discarding it; a library affiliation <NUM> which indicates that the ACSDS is affiliated with a library <NUM> by an authorized author of the library or an authorized distributor of the library; or a repository affiliation <NUM> which indicates that the ACSDS is affiliated with a code repository <NUM> by an authorized user of the repository.

In some scenarios utilization in a codebase is indicative of human review, but an embodiment may nonetheless measure these aspects separately.

Some embodiments use evidence of utilization <NUM> to extract metadata <NUM> about developer identity, e.g., using a source code control <NUM> API <NUM>. For instance, an embodiment may determine that a particular change <NUM> was added by Pat Coder on <NUM> June <NUM> according to a source code control query result.

In some embodiments, the codebase utilization level <NUM> may be Boolean, e.g., Yes: the suggestion has apparently been utilized, or No: nothing was found to indicate that the suggestion has been utilized. Alternately, the codebase utilization level <NUM> may be non-Boolean, e.g., it may be a count of the number of times the suggestion has apparently (based on repo pull list, log, etc. scanning) been utilized. Similarly, the developer review level <NUM> may be Boolean in some embodiments and non-Boolean in others. Thus, in some embodiments the system <NUM> is characterized in at least one of the following ways: the codebase utilization level <NUM> is selected from a set consisting of two values (it is Boolean), the codebase utilization level <NUM> is selected from a set which includes more than two values (it is not Boolean), the developer review level <NUM> is selected from a set consisting of two values (it is Boolean), or the developer review level is selected from a set which includes more than two values (it is not Boolean).

Some embodiments reduce or even minimize configuration burdens <NUM> on developers. As a result, teams of developers can receive the benefit of trustworthy autocreated suggestions without themselves curating rules that define trustworthiness. Over time, the enhanced system surfaces suggestions <NUM> that tend to matter to the team. Some embodiments may, however, include a settings mechanism, e.g., to let a manager indicate to the enhanced system that a rule is not being applied but should be. In some scenarios, an embodiment detects lack of use of a suggestion <NUM> and prompts a repo owning dev lead or other admin, to the effect that a particular rule <NUM> is not being used and offers an option of retiring that rule <NUM>.

In some embodiments, the enhanced system <NUM> is characterized in at least one of the following ways: the suggestion presentation interface <NUM> operates subject to one or more user-defined settings <NUM>, and the suggestion filtering steps operate without any additional user-defined settings; or the suggestion filtering steps operate without any user-defined settings <NUM> which are unique to the suggestion filtering steps. For instance, in a particular embodiment a user-defined setting may specify the maximum number of suggestions that will be shown together in a tool tip of an autocompletion feature, or a setting may determine whether a tool will show the user autocompletion suggestions that are based on repeated edits. This setting is not unique to suggestion <NUM> filtering because it does not bear on suggestion <NUM> trustworthiness. But in the same or another embodiment, a different user-defined setting may specify the number of lines of source code to display as context <NUM> illustrating the provenance of a given suggestion <NUM>; this setting is unique to suggestion <NUM> filtering because suggestion <NUM> trustworthiness may depend on where the suggestion came from.

Although the teachings herein may be applied in various development environments, it is contemplated that many embodiments will be used in some scenario where the source code <NUM> is being edited in an IDE <NUM>. Thus, in some embodiments the software development tool <NUM> includes an IDE <NUM>. Some IDEs which are suitable for enhancement as taught herein include a Microsoft® Visual Studied integrated development environment (Microsoft and Visual Studio are marks of Microsoft Corporation), a NetBeans® integrated development environment (NetBeans is a mark of Oracle America, Inc. ), an Eclipse™ integrated development environment (Eclipse is a mark of Eclipse. org Foundation, Inc. ) and other integrated development environments.

Other system embodiments are also described herein, either directly or derivable as system versions of described processes or configured media, duly informed by the extensive discussion herein of computing hardware.

Although specific architectural examples are shown in the Figures, an embodiment may depart from those examples. For instance, items shown in different Figures may be included together in an embodiment, items shown in a Figure may be omitted, functionality shown in different items may be combined into fewer items or into a single item, items may be renamed, or items may be connected differently to one another.

Examples are provided in this disclosure to help illustrate aspects of the technology, but the examples given within this document do not describe all of the possible embodiments. A given embodiment may include additional or different technical features, operation sequences, data structures, or GUI presentation functionalities, for instance, and may otherwise depart from the examples provided herein.

<FIG> illustrates a family of software development tool methods <NUM> that may be performed or assisted by an enhanced system, such as system <NUM> or another functionality <NUM> enhanced system as taught herein. <FIG> further illustrates software development tool methods <NUM> (which may also be referred to as "processes" in the legal sense of that word) that are suitable for use during operation of a system which has innovative functionality taught herein. <FIG> includes some refinements, supplements, or contextual actions for steps shown in <FIG>, as well as methods which do not necessarily involve steps of <FIG>. <FIG> as a whole does incorporate all steps shown in <FIG>, as steps that may be part of a particular embodiment.

Technical processes shown in the Figures or otherwise disclosed will be performed automatically, e.g., by an enhanced system <NUM> or software component thereof, unless otherwise indicated. Processes may also be performed in part automatically and in part manually to the extent activity by a human person is implicated. For instance, a user <NUM> may set the value of a user-defined setting <NUM>, or may enter commands which cause an enhanced tool <NUM> to follow <NUM> a navigational item <NUM> in order to view context <NUM> of an autocreated suggestion's endorsements. However, no process contemplated as innovative herein is entirely manual.

In a given embodiment zero or more illustrated steps of a process may be repeated, perhaps with different parameters or data to operate on. Steps in an embodiment may also be done in a different order than the top-to-bottom order that is laid out in <FIG> and <FIG>. Steps may be performed serially, in a partially overlapping manner, or fully in parallel. In particular, the order in which flowchart <NUM> or flowchart <NUM> operation items are traversed to indicate the steps performed during a process may vary from one performance of the process to another performance of the process. The flowchart traversal order may also vary from one process embodiment to another process embodiment. Steps may also be omitted, combined, renamed, regrouped, be performed on one or more machines, or otherwise depart from the illustrated flow, provided that the process performed is operable and conforms to at least one claim.

Some embodiments use or provide a software development tool method <NUM> that is suitable for performance by a software development tool <NUM> which has a user interface <NUM>, including at least the following metadata obtaining <NUM>, value ascertaining <NUM>, score computing <NUM>, score assigning <NUM>, and score employing <NUM> steps.

During a metadata obtaining <NUM> step, the embodiment obtains provenance metadata <NUM> of an automatically created software development suggestion (ACSDS) <NUM>. In some embodiments, metadata <NUM> may be obtained by scanning pull requests <NUM> or other source code control system <NUM> logs, or program synthesizer <NUM> or other suggestion creator mechanism <NUM> logs, or logs of autocompletion activity in enhanced development tools <NUM>, for example. Scanning involves computational activity such as reading files and copying data from one digital data structure to another, as opposed to viewing by a human; logs for scanning are digital data <NUM>, not markings on paper.

During a value ascertaining <NUM> step, the embodiment ascertains at least one of the following provenance values <NUM>, at least in part by using the metadata: a codebase utilization level <NUM> which indicates an extent to which the ACSDS has been utilized within a codebase <NUM>; a codebase utilizer identity <NUM> which identifies at least one developer who submitted one or more suggestion-based changes <NUM> to a codebase, each suggestion-based change including a software implementation <NUM> of the ACSDS; a developer review level <NUM> which indicates an extent to which the ACSDS has been reviewed by at least one developer; a suggestion saver identity <NUM> which identifies at least one developer who chose to save the ACSDS instead of discarding it; a library affiliation <NUM> which indicates that the ACSDS is affiliated with a library <NUM> by an authorized author of the library or an authorized distributor of the library; or a repository affiliation <NUM> which indicates that the ACSDS is affiliated with a code repository <NUM> by an authorized user of the repository.

In some embodiments, provenance values may be ascertained <NUM> by computational analysis or other processing of metadata <NUM>. For example, suppose a suggestion <NUM> includes adding a new parameter to invocations of a routine foo(). Then a codebase utilization level <NUM> could be calculated by counting the number of pull requests that include an edit to "foo". This could overestimate the actual utilization, because other changes unrelated to the particular suggestion might also edit foo() for other reasons. But codebase utilization level <NUM> and other provenance values can be beneficially used as taught herein even when they are not necessarily entirely accurate.

During a score computing <NUM> step, the embodiment computes a provenance-derived trust score <NUM> for the ACSDS, based at least in part on a result of the ascertaining <NUM>. Although a normalized provenance value <NUM> may serve as a trust score in some embodiments, other embodiments may compute <NUM> a trust score as a (potentially weighted) combination <NUM> of normalized (or raw) provenance values <NUM>. That is, in some embodiments computing <NUM> the provenance-derived trust score <NUM> includes calculating <NUM> a weighted combination <NUM> of at least two of the provenance values <NUM>. In others, a single provenance value is used as a basis for the trust score. Some embodiments give greater weight to particular metadata when computing <NUM> a trust score, e.g., by giving more weight to suggestions <NUM> that have been adopted many times or in a wide variety of contexts or by a large number of developers or in a particularly often-edited codebase.

Many different trust score <NUM> computations <NUM> may be used, in various embodiments, but each computation <NUM> is based at least partially on provenance metadata <NUM>. For example, in some cases the presence of code implementing a suggestion is treated as evidence that the suggestion is trusted. In some embodiments the provenance-derived trust score for an ACSDS is computed <NUM> based at least in part on the codebase utilization level or the repository affiliation or both.

As another example, denoted here as Scoring Example A, an embodiment may compute <NUM> a trust score <NUM> of an autocreated suggestion <NUM> as the number <NUM> of times that suggestion has been reviewed <NUM> by other developers (e.g., how many times it has been offered via autocompletion) plus twice the number <NUM> of times the suggestion has been utilized <NUM> by other developers (e.g., per pull request logs), divided by six. In practice, this would yield trust scores such as the following:.

During a score assigning <NUM> step, the embodiment assigns the provenance-derived trust score to the ACSDS. This may be accomplished by computationally copying the computed trust score into a data structure in memory <NUM> that contains or points to the ACSDS <NUM>, for example.

During a score employing <NUM> step, the embodiment employs the provenance-derived trust score within a software development tool <NUM>. For example, an embodiment may employ <NUM> a trust score by determining <NUM> that the trust score meets <NUM> a suggestion supplying criterion <NUM>, or by determining <NUM> that the trust score meets <NUM> a suggestion withholding criterion <NUM>. When a supplying criterion <NUM> is met <NUM>, the embodiment supplies <NUM> the suggestion <NUM> to the interface <NUM> for display; a developer is thus given an opportunity to adopt <NUM> the suggestion <NUM>. When a withholding criterion <NUM> is met <NUM>, the embodiment withholds <NUM> the suggestion <NUM> from the interface <NUM>, by not supplying <NUM> the suggestion for display. The developer is thus not given an opportunity to adopt <NUM> the suggestion <NUM>, but is also not going to spend time and effort deciding whether to adopt the suggestion.

As a specific example, an embodiment may employ <NUM> trust scores <NUM> and criteria <NUM> such that the embodiment supplies <NUM> the suggestion when the trust score is less than a cutoff <NUM>. Even more specifically, an embodiment which computes <NUM> trust scores per Scoring Example A above, could have a cutoff <NUM> set to <NUM>, in which case the withholding criterion <NUM> could be "trust score less than <NUM>" and the supplying criterion <NUM> could be "trust score <NUM> or greater". In this hypothetical scenario, which is denoted here as Supplying Example A, the results of employing <NUM> the trust score to determine whether the particular suggestion <NUM> is supplied to the interface <NUM> for the interface <NUM> to display to a developer would be as follows:.

In some embodiments, another example of score employing <NUM> includes adjusting <NUM> a filtering criterion <NUM> in response to results (supply <NUM> or withhold <NUM>) of one or more scores <NUM>. Similar results can be accomplished in some embodiments by adjusting <NUM> the trust score of the suggestion. In particular, some embodiments support untrusting a suggestion, that is, treating a suggestion as less trustworthy than before. In one scenario, additional developer review in the form of ratings leads to a lower rating for a suggestion and hence the suggestion's trust score goes down. In another scenario, further change in the codebase leads to less codebase utilization of the suggestion and hence the suggestion's trust score goes down. In a third scenario, the suggestion is rarely taken <NUM> and hence the suggestion's trust score goes down.

Accordingly, in some embodiments at a first point in time the provenance-derived trust score <NUM> meets a criterion for supplying <NUM> the ACSDS to a suggestion presentation interface <NUM> of the software development tool, and a method that is performed by or as the embodiment includes obtaining <NUM> additional provenance metadata at a second point in time after the first point in time, adjusting <NUM> the assigned provenance-derived trust score or the criterion based at least in part on the additional provenance metadata, and then determining that the (modified) provenance-derived trust score does not meet the (modified) criterion for supplying the ACSDS to the suggestion presentation interface. This may occur, in particular, during software development by a team <NUM>.

In some embodiments, employing <NUM> the provenance-derived trust score includes at least one of the following: computationally determining <NUM> that the provenance-derived trust score meets a criterion for supplying the ACSDS to a suggestion presentation interface of the software development tool, and supplying <NUM> the ACSDS to the suggestion presentation interface; computationally determining <NUM> that the provenance-derived trust score meets a criterion for withholding the ACSDS from a suggestion presentation interface of the software development tool, and withholding <NUM> the ACSDS from the suggestion presentation interface; or computationally adjusting <NUM> a criterion for supplying the ACSDS to a suggestion presentation interface of the software development tool, based at least in part on the provenance-derived trust score and on data indicating whether the ACSDS was adopted after it was supplied. "Computationally" herein means using computing hardware, as opposed to being performed in a human mind.

Some embodiments provide users with the option of showing some provenance info <NUM> when displaying a suggestion <NUM>, or at least showing a link <NUM> to some provenance info. In some, a method includes displaying <NUM> the ACSDS <NUM> in a screen view in the software development tool <NUM>, and doing at least one of the following in the same screen view: displaying <NUM> one or more of the provenance values <NUM>, or showing <NUM> a navigational item which upon being followed <NUM> displays <NUM> one or more of the provenance values. Some display <NUM> source code <NUM> for which the suggestion was previously endorsed.

One of skill informed by the teachings disclosed herein will acknowledge that an embodiment can provide technical benefits in several situations, e.g., by promoting consistent and prompt adoption of suggestions <NUM> which have been endorsed by other development team members. Some potential benefits in a given situation may include fixing security bugs, decreasing API antipattern usage, updating code in view of a breaking change in a library, making the same fix in many places, or making the same change across multiple codebases or multiple repositories, for example, while avoiding the introduction of widespread inconsistencies through use of various non-endorsed suggestions in response to a repeated code <NUM> pattern.

Accordingly, some embodiments are characterized in at least one of the following ways: an ACSDS <NUM>, <NUM> targets <NUM> a security vulnerability <NUM> in a source code <NUM>; an ACSDS <NUM>, <NUM> targets <NUM> a use <NUM> of an application programming interface <NUM> antipattern <NUM>; an ACSDS <NUM>, <NUM> targets <NUM> a source code <NUM> that was made noncompilable <NUM> by a breaking change <NUM> in a library <NUM> the source code relies on; an ACSDS <NUM>, <NUM> targets <NUM> a source code <NUM> that was made noninterpretable <NUM> by a breaking change <NUM> in a <NUM> library the source code relies on; an ACSDS <NUM>, <NUM> matches a source code pattern <NUM> which appears <NUM> at least ten times in at least one codebase <NUM>; an ACSDS <NUM>, <NUM> has been adopted <NUM> within multiple codebases <NUM>; or an ACSDS <NUM>, <NUM> has been adopted <NUM> within multiple repositories <NUM>.

One of skill informed by the present disclosure will acknowledge that repo pull lists <NUM> can be treated in an embodiment as evidence of codebase utilization <NUM> of a suggestion being implemented by a change to the codebase. Repo pull lists <NUM> can also or instead be treated in an embodiment as evidence of developer review <NUM> of a suggestion that is implemented by a change to the codebase.

In some embodiments, ascertaining <NUM> the codebase utilization level <NUM> includes searching a list of accepted pull requests of a repository, and in some ascertaining <NUM> the codebase utilization level <NUM> includes searching a list of all pull requests of a repository.

In some embodiments, ascertaining <NUM> the developer review level <NUM> includes searching a list of accepted pull requests of a repository, and in some ascertaining <NUM> the developer review level <NUM> includes searching a list of all pull requests of a repository.

A repository is merely one example of a source code control system, and a pull request is one example of a communication with a source code control system. Accordingly, in some embodiments other events in communications with a source code control system <NUM> are treated as evidence while ascertaining <NUM> one or more provenance values <NUM>.

Some embodiments track how often an autocreated suggestion <NUM> is implemented, and use that adoption data <NUM> as provenance metadata. Some embodiments ascertain <NUM> a suggestion adoption level <NUM> which indicates an extent to which the ACSDS <NUM> has been adopted <NUM> after being presented within the software development tool.

Some embodiments use a source code control system <NUM> to get <NUM> identity metadata <NUM>, such as suggestion utilizer identity <NUM> or suggestion saver identity <NUM>. It is understood that appropriate privacy controls will be enforced in implementations. However, additional privacy controls beyond those already in place will not necessarily be called for in team development scenarios in which team members already have proper access to identity information about other team members, e.g., the ability to see who submitted a given pull request.

Accordingly, some embodiments are characterized in at least one of the following ways: ascertaining <NUM> the codebase utilizer identity <NUM> includes querying <NUM> a source code control <NUM> interface, ascertaining <NUM> the suggestion saver identity <NUM> includes querying <NUM> a source code control <NUM> interface, ascertaining <NUM> the codebase utilizer identity <NUM> includes querying <NUM> a code repository <NUM> interface, or ascertaining <NUM> the suggestion saver identity <NUM> includes querying <NUM> a code repository <NUM> interface.

Some embodiments include a configured computer-readable storage medium <NUM>. Storage medium <NUM> may include disks (magnetic, optical, or otherwise), RAM, EEPROMS or other ROMs, and/or other configurable memory, including in particular computer-readable storage media (which are not mere propagated signals). The storage medium which is configured may be in particular a removable storage medium <NUM> such as a CD, DVD, or flash memory. A general-purpose memory, which may be removable or not, and may be volatile or not, can be configured into an embodiment using items such as suggestion presentation software <NUM>, provenance metadata <NUM>, trust scores <NUM>, filter criteria <NUM>, enhanced tools <NUM>, user-defined settings <NUM>, and autocreated suggestions <NUM>, in the form of data <NUM> and instructions <NUM>, read from a removable storage medium <NUM> and/or another source such as a network connection, to form a configured storage medium. The configured storage medium <NUM> is capable of causing a computer system <NUM> to perform technical process steps for filtering autocreated suggestions <NUM> based on development team member endorsements <NUM>, <NUM>, as disclosed herein. The Figures thus help illustrate configured storage media embodiments and process (a. method) embodiments, as well as system and process embodiments. In particular, any of the process steps illustrated in <FIG> or <FIG> or otherwise taught herein, may be used to help configure a storage medium to form a configured storage medium embodiment.

Some embodiments use or provide a computer-readable storage medium <NUM>, <NUM> configured with data <NUM> and instructions <NUM> which upon execution by at least one processor <NUM> cause a computing system to perform a suggestion filtering method. This method includes: obtaining <NUM> provenance metadata <NUM> of an automatically created software development suggestion (ACSDS) <NUM>, the provenance metadata including a codebase utilization level <NUM> which indicates an extent to which the ACSDS has been utilized within at least one codebase <NUM>; assigning <NUM> a provenance-derived trust score <NUM> to the ACSDS based at least in part on the provenance metadata; comparing <NUM> the provenance-derived trust score to a criterion <NUM> for supplying the ACSDS to a suggestion presentation interface <NUM> of a software development tool <NUM>; and supplying <NUM> the ACSDS to the suggestion presentation interface of a software development tool or withholding <NUM> the ACSDS from the suggestion presentation interface, in response to a result of the comparing.

Some embodiments display a context <NUM> in which the suggestion has been applied <NUM> before. This could be the original context in which the suggestion was first automatically created, or it could be a later application of the suggestion. In particular, in some embodiments the method includes displaying <NUM> a source code <NUM> context <NUM> in which the ACSDS <NUM> was previously created <NUM> or previously adopted <NUM> or both. Some embodiments display context <NUM> that includes a detected edit, together with a source (developer ID) of a suggestion adopted by a team member, and the filename of code that shows the adopted suggestion and the location of the adoption within that file.

In some scenarios, a suggestion <NUM> comes from edits made by one or more other developers than the develop to whom the suggestion is offered after filtering <NUM>. In particular, in some embodiments the ACSDS <NUM> was created <NUM> based on source code edits made by a set of one or more developers, and wherein the method supplies <NUM> the ACSDS to the suggestion presentation interface <NUM> while the suggestion presentation interface is displayed <NUM> to another developer who is not in the first set.

Some embodiments have enhanced usability due to the trust score interpretation not being user-defined. That is, a low configurability burden <NUM> on users is a benefit of some embodiments. In particular, in some the criterion <NUM> for supplying the ACSDS to a suggestion presentation interface is not user-defined (i.e., it is not a user-defined setting <NUM>).

Some embodiments use machine learning technology to learn where to set the trust score cutoff by tracking how often suggestions are implemented. In some, the method includes ascertaining <NUM> a suggestion adoption level <NUM> which indicates an extent to which the ACSDS has been adopted after being presented <NUM> within the software development tool, and adjusting <NUM> the criterion <NUM> for supplying the ACSDS to the suggestion presentation interface based at least in part on the suggestion adoption level.

One of skill will recognize that not every part of this disclosure, or any particular details therein, are necessarily required to satisfy legal criteria such as enablement, written description, or best mode. Any apparent conflict with any other patent disclosure, even from the owner of the present innovations, has no role in interpreting the claims presented in this patent disclosure. With this understanding, which pertains to all parts of the present disclosure, some additional examples and observations are offered.

Some embodiments provide technical advantages through enhancement of software development tools to automatically and proactively determine provenance of a code suggestion <NUM> based, for exmple, on proof-of-use of the suggestion in a commit tracked by a repository or other source code control system.

One of skill will acknowledge that several software development tools generate suggestions <NUM> based on learnings from a development team's codebases and developer interactions with those codebases. The Visual Studied IntelliCode™ feature set is producing many automatic insights for developers, for example (marks of Microsoft Corporation). However, this activity presents the problem of determining which automatically generated suggestions <NUM> the user can trust as being reliable. In some development scenarios there are established review processes, such as repository pull requests, which allow developers to peer review contributions to a codebase. However, such processes do not adequately review suggestions created by an automated system, at least not without adding more burden to the review process or adding user interface capabilities (and underlying logic) to allow explicit suggestion <NUM> review.

Some embodiments leverage review mechanisms (e.g., pull requests, continuous integration tools, static analysis tools) with additional functionality <NUM> to determine whether a suggestion <NUM> has been applied in at least one place in a codebase, before making that suggestion a verified rule for that repo or codebase. In this manner developers can know that any suggestion proffered by the system is trustworthy in that it has been through at least one review and has been accepted into the body of code. Alternatives could involve additional review processes, which could be burdensome for the developer or team. Alternatives could also prevent suggestions from being verified, which leads to fewer suggestions or less trust or both. By contrast, some embodiments presented herein fit comfortably into established team processes of review and development.

Some embodiments define a chain of trust for a team suggestion <NUM> (also referred to here as a "rule"). Users benefit from automatic verification that a suggestion <NUM> came from a source they can trust, in order avoid random or even malicious suggestions.

Some embodiments implement a verified individual trust approach to suggestion provenance, whereby a user can see who saved the suggestion and may use that identity to determine whether to also trust the suggestion. In some, this approach uses authenticated credentials and accordingly presumes proper login of the user submitting the saved suggestion. An embodiment may capture a record of the credentials associated to the team member X who did the editing that led to a suggestion being initially attached to the repository. When another user Y gets a suggestion <NUM> based on the edits made by X, that origin is explicitly called out to Y and the enhanced tool shows the recipient of the suggestion Y the identity of the suggestion saver X. For instance, the user interface <NUM> may display an attribution message along with the suggestion, e.g., "Based on edits done by X". The attribution message may include an email or other contact info for X. This approach provides a chain of trust: user Y can adopt this suggestion, treating it as trustworthy because it is derived from the work of a person X that Y trusts.

Some embodiments implement a verified codebase change trust approach to suggestion provenance, whereby a user can confirm that the suggestion is trustworthy because the changes it suggests are verifiably from code changes in a repo the user trusts and the user can see the repo's record of who made those changes. In this approach, instead of using the authenticated identity of the user who saved the suggestion to establish a verifiable chain of trust, an embodiment leverages the fact that users <NUM> generally already trust code that originates in the repo they are working on, and that users trust that repo to tell them who the code and associated rule <NUM> were from and when they were introduced. This is often a richer context than simply knowing who committed a code change per the rule <NUM>.

It follows that if an embodiment can determine that a change suggested by a rule <NUM> has already been made in a repo <NUM>, users can safely trust the rule <NUM>, since they trusted that change when it came through from the repo. In broad terms, a user Y trusts a suggestion origin <NUM>, therefore Y trusts commits in the origin, therefore Y trusts this particular rule <NUM> because it reflects a change introduced in a commit to the trusted origin.

Accordingly, in some embodiments, a rule <NUM> is associated with a given commit on origin <NUM>, which gives same verifiability as a new or changed code file. For verifiability, the rule <NUM> will be associated with the commit that had the changes that caused the rule to be used <NUM> for the first time. When an embodiment saves rules, e.g., to perform an APPLY & COMMIT, the commit will have the rule's ACTION incorporated in it in order to become verified (i.e., trusted per functionality <NUM>). That is, the action the rule recommends would have been taken in the repo for the rule to become a verified rule. After a rule is verified, it may be supplied <NUM> for additional potential usage and be displayed with, e.g., the name of the person(s) who made the commit(s) that used the rule (per metadata <NUM> from the repo <NUM> or source code control system <NUM>) and the date and time that code edits were made (also from the repo <NUM> or SCCS <NUM>).

Some embodiments implement a verified binary change trust approach to suggestion provenance, especially in scenarios where a user acquires a binary package (e.g., a library or a NuGet package), whereby the user can verify that the suggestion is part of that shipped binary version distributed by the library author. Suggestions may be provided to help the library's consumers adapt to breaking changes when moving from one library version to another.

The technical character of embodiments described herein will be apparent to one of ordinary skill in the art, and will also be apparent in several ways to a wide range of attentive readers. Some embodiments address technical activities such as computationally determining the origins and utilizations of automatically created rules <NUM>, computing trust scores <NUM> based on rule provenance <NUM>, and determining <NUM> within special-purpose software <NUM> whether to supply <NUM> particular data associated with a suggestion <NUM> to a development tool user interface <NUM> or instead withhold <NUM> that data from being displayed, each of which is an activity deeply rooted in computing technology. Some of the technical mechanisms discussed include, e.g., suggestion filtering code <NUM>, provenance-derived trust scoring <NUM>, suggestion synthesizers <NUM>, source code control systems <NUM>, navigational items <NUM>, and development tool user interfaces <NUM>. Some of the technical effects discussed include, e.g., supplying <NUM> for display autocreated suggestions <NUM> or withholding <NUM> them from display according to whether the suggestions <NUM> have been previously endorsed by software development actions of other members of a development team <NUM>, affiliation <NUM> of one or more suggestions <NUM> with a library <NUM> release, and reduced or avoided configuration burdens <NUM> on developers in connection with autocreated suggestions <NUM>. Thus, purely mental processes are clearly excluded. Other advantages based on the technical characteristics of the teachings will also be apparent to one of skill from the description provided.

Some embodiments described herein may be viewed by some people in a broader context. For instance, concepts such as confidence, convenience, history, identity, origin, usage, or trust may be deemed relevant to a particular embodiment. However, it does not follow from the availability of a broad context that exclusive rights are being sought herein for abstract ideas; they are not. Rather, the present disclosure is focused on providing appropriately specific embodiments whose technical effects fully or partially solve particular technical problems, such as how to facilitate consistent utilization <NUM> of autocreated suggestions <NUM> while reducing exposure to malicious efforts to modify source code <NUM> and also reducing or avoiding rule <NUM> configuration burdens <NUM> on developers. Other configured storage media, systems, and processes involving confidence, convenience, history, identity, origin, usage, or trust are outside the present scope. Accordingly, vagueness, mere abstractness, lack of technical character, and accompanying proof problems are also avoided under a proper understanding of the present disclosure.

Any of these combinations of code, data structures, logic, components, communications, and/or their functional equivalents may also be combined with any of the systems and their variations described above. A process may include any steps described herein in any subset or combination or sequence which is operable. Each variant may occur alone, or in combination with any one or more of the other variants. Each variant may occur with any of the processes and each process may be combined with any one or more of the other processes. Each process or combination of processes, including variants, may be combined with any of the configured storage medium combinations and variants described above.

More generally, one of skill will recognize that not every part of this disclosure, or any particular details therein, are necessarily required to satisfy legal criteria such as enablement, written description, or best mode. Also, embodiments are not limited to the particular motivating examples and scenarios, operating system environments, suggestion creation mechanisms, software processes, development tools, identifiers, machine learning technologies, source code control systems, programming languages, data structures, data formats, notations, control flows, naming conventions, or other implementation choices described herein. Any apparent conflict with any other patent disclosure, even from the owner of the present innovations, has no role in interpreting the claims presented in this patent disclosure.

Some acronyms, abbreviations, names, and symbols are defined below. Others are defined elsewhere herein, or do not require definition here in order to be understood by one of skill.

Reference is made herein to exemplary embodiments such as those illustrated in the drawings, and specific language is used herein to describe the same. But alterations and further modifications of the features illustrated herein, and additional technical applications of the abstract principles illustrated by particular embodiments herein, which would occur to one skilled in the relevant art(s) and having possession of this disclosure, should be considered within the scope of the claims.

The meaning of terms is clarified in this disclosure, so the claims should be read with careful attention to these clarifications. Specific examples are given, but those of skill in the relevant art(s) will understand that other examples may also fall within the meaning of the terms used, and within the scope of one or more claims. Terms do not necessarily have the same meaning here that they have in general usage (particularly in non-technical usage), or in the usage of a particular industry, or in a particular dictionary or set of dictionaries. Reference numerals may be used with various phrasings, to help show the breadth of a term. Omission of a reference numeral from a given piece of text does not necessarily mean that the content of a Figure is not being discussed by the text. The inventors assert and exercise the right to specific and chosen lexicography. Quoted terms are being defined explicitly, but a term may also be defined implicitly without using quotation marks. Terms may be defined, either explicitly or implicitly, here in the Detailed Description and/or elsewhere in the application file.

As used herein, a "computer system" (a. "computing system") may include, for example, one or more servers, motherboards, processing nodes, laptops, tablets, personal computers (portable or not), personal digital assistants, smartphones, smartwatches, smartbands, cell or mobile phones, other mobile devices having at least a processor and a memory, video game systems, augmented reality systems, holographic projection systems, televisions, wearable computing systems, and/or other device(s) providing one or more processors controlled at least in part by instructions. The instructions may be in the form of firmware or other software in memory and/or specialized circuitry.

A "multithreaded" computer system is a computer system which supports multiple execution threads. The term "thread" should be understood to include code capable of or subject to scheduling, and possibly to synchronization. A thread may also be known outside this disclosure by another name, such as "task," "process," or "coroutine," for example. However, a distinction is made herein between threads and processes, in that a thread defines an execution path inside a process. Also, threads of a process share a given address space, whereas different processes have different respective address spaces. The threads of a process may run in parallel, in sequence, or in a combination of parallel execution and sequential execution (e.g., time-sliced).

A "processor" is a thread-processing unit, such as a core in a simultaneous multithreading implementation. A processor includes hardware. A given chip may hold one or more processors. Processors may be general purpose, or they may be tailored for specific uses such as vector processing, graphics processing, signal processing, floating-point arithmetic processing, encryption, I/O processing, machine learning, and so on.

"Kernels" include operating systems, hypervisors, virtual machines, BIOS or UEFI code, and similar hardware interface software.

"Code" means processor instructions, data (which includes constants, variables, and data structures), or both instructions and data. "Code" and "software" are used interchangeably herein. Executable code, interpreted code, and firmware are some examples of code.

A "codebase" is code pertaining to a single project or a single web program or a single product. A "repository" is a computational mechanism for version-controlled storage of software including source code. A codebase may reside in a repository or may reside elsewhere, e.g., in a storage that is not currently version controlled. A repository may contain one or more codebases.

"Program" is used broadly herein, to include applications, kernels, drivers, interrupt handlers, firmware, state machines, libraries, and other code written by programmers (who are also referred to as developers), or automatically generated or automatically synthesized.

A "routine" is a callable piece of code which normally returns control to an instruction just after the point in a program execution at which the routine was called. Depending on the terminology used, a distinction is sometimes made elsewhere between a "function" and a "procedure": a function normally returns a value, while a procedure does not. As used herein, "routine" includes both functions and procedures. A routine may have code that returns a value (e.g., sin(x)) or it may simply return without also providing a value (e.g., void functions).

"Service" means a consumable program offering, in a cloud computing environment or other network or computing system environment, which provides resources to multiple programs or provides resource access to multiple programs, or does both.

"Autocompletion" refers to a code editing activity driven by suggestions offered by an editing tool, or to a result of such activity, and to a feature or other portion of software which performs autocompletion activity. Autocompletion may include, e.g., an editing functionality which uses part or all of a typed identifier to compute and display a list of suggestions for completing the identifier, or which uses a code fragment having a detected pattern to compute and display a list of suggestions for editing that code fragment.

"Cloud" means pooled resources for computing, storage, and networking which are elastically available for measured on-demand service. A cloud may be private, public, community, or a hybrid, and cloud services may be offered in the form of infrastructure as a service (IaaS), platform as a service (PaaS), software as a service (SaaS), or another service. Unless stated otherwise, any discussion of reading from a file or writing to a file includes reading/writing a local file or reading/writing over a network, which may be a cloud network or other network, or doing both (local and networked read/write).

"IoT" or "Internet of Things" means any networked collection of addressable embedded computing or data generation or actuator nodes. Such nodes may be examples of computer systems as defined herein, and may include or be referred to as a "smart" device, "endpoint", "chip", "label", or "tag", for example, and IoT may be referred to as a "cyber-physical system". IoT nodes and systems typically have at least two of the following characteristics: (a) no local human-readable display; (b) no local keyboard; (c) a primary source of input is sensors that track sources of non-linguistic data to be uploaded from the IoT device; (d) no local rotational disk storage - RAM chips or ROM chips provide the only local memory; (e) no CD or DVD drive; (f) embedment in a household appliance or household fixture; (g) embedment in an implanted or wearable medical device; (h) embedment in a vehicle; (i) embedment in a process automation control system; or (j) a design focused on one of the following: environmental monitoring, civic infrastructure monitoring, agriculture, industrial equipment monitoring, energy usage monitoring, human or animal health or fitness monitoring, physical security, physical transportation system monitoring, object tracking, inventory control, supply chain control, fleet management, or manufacturing. IoT communications may use protocols such as TCP/IP, Constrained Application Protocol (CoAP), Message Queuing Telemetry Transport (MQTT), Advanced Message Queuing Protocol (AMQP), HTTP, HTTPS, Transport Layer Security (TLS), UDP, or Simple Object Access Protocol (SOAP), for example, for wired or wireless (cellular or otherwise) communication. IoT storage or actuators or data output or control may be a target of unauthorized access, either via a cloud, via another network, or via direct local access attempts.

"Access" to a computational resource includes use of a permission or other capability to read, modify, write, execute, or otherwise utilize the resource. Attempted access may be explicitly distinguished from actual access, but "access" without the "attempted" qualifier includes both attempted access and access actually performed or provided.

"Software development workflow" means steps or actions taken by or on behalf of a software developer during software code creation or modification or testing or debugging or profiling or deployment or monitoring.

A "source code" is code in human-readable text form in a programming language, with the understanding that code includes more than merely data values - code includes statements, expressions, identifiers, variable declarations, functions, etc..

The "execution" of a code implies compiling the code and running the compiled code, or interpreting the code; the distinction between compilers and interpreters is acknowledged herein but does not prevent beneficial application of the teachings herein with either category of code.

As used herein, "include" allows additional elements (i.e., includes means comprises) unless otherwise stated.

"Optimize" means to improve, not necessarily to perfect. For example, it may be possible to make further improvements in a program or an algorithm which has been optimized.

"Process" is sometimes used herein as a term of the computing science arts, and in that technical sense encompasses computational resource users, which may also include or be referred to as coroutines, threads, tasks, interrupt handlers, application processes, kernel processes, procedures, or object methods, for example. As a practical matter, a "process" is the computational entity identified by system utilities such as Windows® Task Manager, Linux® ps, or similar utilities in other operating system environments (marks of Microsoft Corporation, Linus Torvalds, respectively). "Process" is also used herein as a patent law term of art, e.g., in describing a process claim as opposed to a system claim or an article of manufacture (configured storage medium) claim. Similarly, "method" is used herein at times as a technical term in the computing science arts (a kind of "routine") and also as a patent law term of art (a "process"). "Process" and "method" in the patent law sense are used interchangeably herein. Those of skill will understand which meaning is intended in a particular instance, and will also understand that a given claimed process or method (in the patent law sense) may sometimes be implemented using one or more processes or methods (in the computing science sense).

"Automatically" means by use of automation (e.g., general purpose computing hardware configured by software for specific operations and technical effects discussed herein), as opposed to without automation. In particular, steps performed "automatically" are not performed by hand on paper or in a person's mind, although they may be initiated by a human person or guided interactively by a human person. Automatic steps are performed with a machine in order to obtain one or more technical effects that would not be realized without the technical interactions thus provided. Steps performed automatically are presumed to include at least one operation performed proactively.

One of skill understands that technical effects are the presumptive purpose of a technical embodiment. The mere fact that calculation is involved in an embodiment, for example, and that some calculations can also be performed without technical components (e.g., by paper and pencil, or even as mental steps) does not remove the presence of the technical effects or alter the concrete and technical nature of the embodiment. Autocreated suggestion filtering <NUM> operations such as obtaining <NUM> provenance metadata <NUM> from a repository <NUM> or other source code control system <NUM> API <NUM> or log <NUM>, ascertaining <NUM> a codebase utilization level <NUM>, ascertaining <NUM> a rule <NUM> adoption level <NUM>, computing <NUM> a provenance-derived trust score <NUM>, determining <NUM> or <NUM> which filter criterion <NUM> is met, and many other operations discussed herein, are understood to be inherently digital. A human mind cannot interface directly with a CPU or other processor, or with RAM or other digital storage, or with a display <NUM>, to read and write the necessary data to perform the suggestion filtering functionality <NUM> steps that are taught herein. This would all be well understood by persons of skill in the art in view of the present disclosure.

"Computationally" likewise means a computing device (processor plus memory, at least) is being used, and excludes obtaining a result by mere human thought or mere human action alone. For example, doing arithmetic with a paper and pencil is not doing arithmetic computationally as understood herein. Computational results are faster, broader, deeper, more accurate, more consistent, more comprehensive, and/or otherwise provide technical effects that are beyond the scope of human performance alone. "Computational steps" are steps performed computationally. Neither "automatically" nor "computationally" necessarily means "immediately". "Computationally" and "automatically" are used interchangeably herein.

"Proactively" means without a direct request from a user. Indeed, a user may not even realize that a proactive step by an embodiment was possible until a result of the step has been presented to the user. Except as otherwise stated, any computational and/or automatic step described herein may also be done proactively.

Throughout this document, use of the optional plural "(s)", "(es)", or "(ies)" means that one or more of the indicated features is present. For example, "processor(s)" means "one or more processors" or equivalently "at least one processor".

For the purposes of United States law and practice, use of the word "step" herein, in the claims or elsewhere, is not intended to invoke means-plus-function, step-plus-function, or <NUM> United State Code Section <NUM> Sixth Paragraph / Section <NUM>(f) claim interpretation. Any presumption to that effect is hereby explicitly rebutted.

For the purposes of United States law and practice, the claims are not intended to invoke means-plus-function interpretation unless they use the phrase "means for". Claim language intended to be interpreted as means-plus-function language, if any, will expressly recite that intention by using the phrase "means for". When means-plus-function interpretation applies, whether by use of "means for" and/or by a court's legal construction of claim language, the means recited in the specification for a given noun or a given verb should be understood to be linked to the claim language and linked together herein by virtue of any of the following: appearance within the same block in a block diagram of the figures, denotation by the same or a similar name, denotation by the same reference numeral, a functional relationship depicted in any of the figures, a functional relationship noted in the present disclosure's text. For example, if a claim limitation recited a "zac widget" and that claim limitation became subject to means-plus-function interpretation, then at a minimum all structures identified anywhere in the specification in any figure block, paragraph, or example mentioning "zac widget", or tied together by any reference numeral assigned to a zac widget, or disclosed as having a functional relationship with the structure or operation of a zac widget, would be deemed part of the structures identified in the application for zac widgets and would help define the set of equivalents for zac widget structures.

One of skill will recognize that this innovation disclosure discusses various data values and data structures, and recognize that such items reside in a memory (RAM, disk, etc.), thereby configuring the memory. One of skill will also recognize that this innovation disclosure discusses various algorithmic steps which are to be embodied in executable code in a given implementation, and that such code also resides in memory, and that it effectively configures any general purpose processor which executes it, thereby transforming it from a general purpose processor to a special-purpose processor which is functionally special-purpose hardware.

Accordingly, one of skill would not make the mistake of treating as nonoverlapping items (a) a memory recited in a claim, and (b) a data structure or data value or code recited in the claim. Data structures and data values and code are understood to reside in memory, even when a claim does not explicitly recite that residency for each and every data structure or data value or piece of code mentioned. Accordingly, explicit recitals of such residency are not required. However, they are also not prohibited, and one or two select recitals may be present for emphasis, without thereby excluding all the other data values and data structures and code from residency. Likewise, code functionality recited in a claim is understood to configure a processor, regardless of whether that configuring quality is explicitly recited in the claim.

Throughout this document, unless expressly stated otherwise any reference to a step in a process presumes that the step may be performed directly by a party of interest and/or performed indirectly by the party through intervening mechanisms and/or intervening entities, and still lie within the scope of the step. That is, direct performance of the step by the party of interest is not required unless direct performance is an expressly stated requirement. For example, a step involving action by a party of interest such as adjusting, affiliating, ascertaining, assigning, calculating, comparing, computing, conforming, creating, determining, discarding, displaying, employing, following, identifying, indicating, obtaining, operating, performing, presenting, saving, searching, selecting, supplying, withholding (and adjusts, adjusted, affiliates, affiliated, etc.) with regard to a destination or other subject may involve intervening action such as the foregoing or forwarding, copying, uploading, downloading, encoding, decoding, compressing, decompressing, encrypting, decrypting, authenticating, invoking, and so on by some other party, including any action recited in this document, yet still be understood as being performed directly by the party of interest.

Whenever reference is made to data or instructions, it is understood that these items configure a computer-readable memory and/or computer-readable storage medium, thereby transforming it to a particular article, as opposed to simply existing on paper, in a person's mind, or as a mere signal being propagated on a wire, for example. For the purposes of patent protection in the United States, a memory or other computer-readable storage medium is not a propagating signal or a carrier wave or mere energy outside the scope of patentable subject matter under United States Patent and Trademark Office (USPTO) interpretation of the In re Nuijten case. No claim covers a signal per se or mere energy in the United States, and any claim interpretation that asserts otherwise in view of the present disclosure is unreasonable on its face. Unless expressly stated otherwise in a claim granted outside the United States, a claim does not cover a signal per se or mere energy.

Moreover, notwithstanding anything apparently to the contrary elsewhere herein, a clear distinction is to be understood between (a) computer readable storage media and computer readable memory, on the one hand, and (b) transmission media, also referred to as signal media, on the other hand. A transmission medium is a propagating signal or a carrier wave computer readable medium. By contrast, computer readable storage media and computer readable memory are not propagating signal or carrier wave computer readable media. Unless expressly stated otherwise in the claim, "computer readable medium" means a computer readable storage medium, not a propagating signal per se and not mere energy.

An "embodiment" herein is an example. The term "embodiment" is not interchangeable with "the invention". Embodiments may freely share or borrow aspects to create other embodiments (provided the result is operable), even if a resulting combination of aspects is not explicitly described per se herein. Requiring each and every permitted combination to be explicitly and individually described is unnecessary for one of skill in the art, and would be contrary to policies which recognize that patent specifications are written for readers who are skilled in the art. Formal combinatorial calculations and informal common intuition regarding the number of possible combinations arising from even a small number of combinable features will also indicate that a large number of aspect combinations exist for the aspects described herein. Accordingly, requiring an explicit recitation of each and every combination would be contrary to policies calling for patent specifications to be concise and for readers to be knowledgeable in the technical fields concerned.

The following list is provided for convenience and in support of the drawing figures and as part of the text of the specification, which describe innovations by reference to multiple items. Items not listed here may nonetheless be part of a given embodiment. For better legibility of the text, a given reference number is recited near some, but not all, recitations of the referenced item in the text. The same reference number may be used with reference to different examples or different instances of a given item. The list of reference numerals is:.

In short, the teachings herein provide a variety of autocreated suggestion filtering functionalities <NUM> which operate in enhanced systems <NUM>. Accuracy and productivity of software development is enhanced, with particular attention to situations such as editing source code <NUM> as a development team <NUM> member using tools <NUM> which provide autocompletion or quick actions. However, the teachings are not limited to those situations.

Some embodiments determine automatically which synthesized or otherwise autocreated suggestions <NUM> for source code <NUM> editing are presented <NUM> to developers. Some embodiments filter out <NUM> autocreated coding suggestions <NUM> that have not been sufficiently endorsed <NUM> or <NUM> or both by a developer's team <NUM>, based on a suggestion trust score <NUM> and filter criterion <NUM>. The trust score <NUM> may reflect the suggestion's adoption <NUM> in a particular repository <NUM> or codebase <NUM>, or affiliation <NUM> of the suggestion <NUM> with a library <NUM> release, or an actual <NUM> or implied <NUM> review <NUM> of the suggestion <NUM> by team <NUM> members <NUM>. Some suggestion filters <NUM> enhance existing development team code review practices, by offering endorsed suggestions <NUM> in autocompletion or analysis interfaces <NUM> of tools <NUM> and by withholding <NUM> non-endorsed suggestions from display. Context <NUM> illustrating the autocreated suggestion's provenance <NUM> may be displayed <NUM> to help developers decide whether to adopt the suggestion <NUM> themselves while editing code <NUM>. Some tools <NUM> that are enhanced with suggestion <NUM> filtering functionality <NUM> avoid developer configuration burdens <NUM> while increasing consistent adoption <NUM> of endorsed suggestions <NUM> inside a codebase <NUM>.

Embodiments are understood to also themselves include or benefit from tested and appropriate security controls and privacy controls such as the General Data Protection Regulation (GDPR). Use of the tools and techniques taught herein is compatible with use of such controls.

Although Microsoft technology is used in some motivating examples, the teachings herein are not limited to use in technology supplied or administered by Microsoft. Under a suitable license, for example, the present teachings could be embodied in software or services provided by other cloud service providers.

Although particular embodiments are expressly illustrated and described herein as processes, as configured storage media, or as systems, it will be appreciated that discussion of one type of embodiment also generally extends to other embodiment types. For instance, the descriptions of processes in connection with <FIG> and <FIG> also help describe configured storage media, and help describe the technical effects and operation of systems and manufactures like those discussed in connection with other Figures. It does not follow that limitations from one embodiment are necessarily read into another. In particular, processes are not necessarily limited to the data structures and arrangements presented while discussing systems or manufactures such as configured memories.

Those of skill will understand that implementation details may pertain to specific code, such as specific thresholds or ranges, specific architectures, specific attributes, and specific computing environments, and thus need not appear in every embodiment. Those of skill will also understand that program identifiers and some other terminology used in discussing details are implementation-specific and thus need not pertain to every embodiment. Nonetheless, although they are not necessarily required to be present here, such details may help some readers by providing context and/or may illustrate a few of the many possible implementations of the technology discussed herein.

With due attention to the items provided herein, including technical processes, technical effects, technical mechanisms, and technical details which are illustrative but not comprehensive of all claimed or claimable embodiments, one of skill will understand that the present disclosure and the embodiments described herein are not directed to subject matter outside the technical arts, or to any idea of itself such as a principal or original cause or motive, or to a mere result per se, or to a mental process or mental steps, or to a business method or prevalent economic practice, or to a mere method of organizing human activities, or to a law of nature per se, or to a naturally occurring thing or process, or to a living thing or part of a living thing, or to a mathematical formula per se, or to isolated software per se, or to a merely conventional computer, or to anything wholly imperceptible or any abstract idea per se, or to insignificant post-solution activities, or to any method implemented entirely on an unspecified apparatus, or to any method that fails to produce results that are useful and concrete, or to any preemption of all fields of usage, or to any other subject matter which is ineligible for patent protection under the laws of the jurisdiction in which such protection is sought or is being licensed or enforced.

Reference herein to an embodiment having some feature X and reference elsewhere herein to an embodiment having some feature Y does not exclude from this disclosure embodiments which have both feature X and feature Y, unless such exclusion is expressly stated herein. All possible negative claim limitations are within the scope of this disclosure, in the sense that any feature which is stated to be part of an embodiment may also be expressly removed from inclusion in another embodiment, even if that specific exclusion is not given in any example herein. The term "embodiment" is merely used herein as a more convenient form of "process, system, article of manufacture, configured computer readable storage medium, and/or other example of the teachings herein as applied in a manner consistent with applicable law. " Accordingly, a given "embodiment" may include any combination of features disclosed herein, provided the embodiment is consistent with at least one claim.

Not every item shown in the Figures need be present in every embodiment. Conversely, an embodiment may contain item(s) not shown expressly in the Figures. Although some possibilities are illustrated here in text and drawings by specific examples, embodiments may depart from these examples. For instance, specific technical effects or technical features of an example may be omitted, renamed, grouped differently, repeated, instantiated in hardware and/or software differently, or be a mix of effects or features appearing in two or more of the examples. Functionality shown at one location may also be provided at a different location in some embodiments; one of skill recognizes that functionality modules can be defined in various ways in a given implementation without necessarily omitting desired technical effects from the collection of interacting modules viewed as a whole. Distinct steps may be shown together in a single box in the Figures, due to space limitations or for convenience, but nonetheless be separately performable, e.g., one may be performed without the other in a given performance of a method.

Reference has been made to the figures throughout by reference numerals. Any apparent inconsistencies in the phrasing associated with a given reference numeral, in the figures or in the text, should be understood as simply broadening the scope of what is referenced by that numeral. Different instances of a given reference numeral may refer to different embodiments, even though the same reference numeral is used. Similarly, a given reference numeral may be used to refer to a verb, a noun, and/or to corresponding instances of each, e.g., a processor <NUM> may process <NUM> instructions by executing them.

As used herein, terms such as "a", "an", and "the" are inclusive of one or more of the indicated item or step. In particular, in the claims a reference to an item generally means at least one such item is present and a reference to a step means at least one instance of the step is performed. Similarly, "is" and other singular verb forms should be understood to encompass the possibility of "are" and other plural forms, when context permits, to avoid grammatical errors or misunderstandings.

Headings are for convenience only; information on a given topic may be found outside the section whose heading indicates that topic.

All claims and the abstract, as filed, are part of the specification.

To the extent any term used herein implicates or otherwise refers to an industry standard, and to the extent that applicable law requires identification of a particular version of such as standard, this disclosure shall be understood to refer to the most recent version of that standard which has been published in at least draft form (final form takes precedence if more recent) as of the earliest priority date of the present disclosure under applicable patent law.

Claim 1:
A software development system (<NUM>), comprising:
a digital memory (<NUM>); and
a processor (<NUM>) in operable communication with the digital memory, the processor configured to perform suggestion filtering steps which include automatically (a) obtaining (<NUM>) provenance metadata (<NUM>) of an automatically created software development suggestion (ACSDS) (<NUM>), the provenance metadata indicating implicit or explicit software developer endorsement of the ACSDS, (b) assigning (<NUM>) a provenance-derived trust score (<NUM>) to the ACSDS based at least in part on the provenance metadata, (c) comparing (<NUM>) the provenance-derived trust score to a threshold for supplying the ACSDS to a suggestion presentation interface (<NUM>) of a software development tool (<NUM>), and (d) supplying (<NUM>) the ACSDS to the suggestion presentation interface when the provenance-derived trust score is equal to or greater than the threshold or withholding (<NUM>) the ACSDS from the suggestion presentation interface if the provenance-derived trust score is less than the threshold;
whereby the system is configured to present (<NUM>) automatically created suggestions to software developers more often or less often depending on whether other software developers have or have not at least implicitly endorsed (<NUM>) the automatically created suggestions.