Patent Description:
This disclosure relates generally to code recommendation and, more particularly, to automatically evolve a code recommendation engine.

Machine programming is concerned with the automation of software. In recent years, there has been a growing body of work in the space of machine programming. One of the open research areas in machine programming is code semantics similarity. Code semantics similarity is the process of determining whether two or more code snippets have some degree of semantic similarity (or equivalence). A particular topic of interest in code semantics similarity is code recommendation. Some of the benefits of an accurate code recommendation engine may include improved programmer productivity, boosted code performance on specific hardware, and education for novice coders. <CIT> relates to a system for facilitating reusability of a code snippet during development of a software application. Initially, a plurality of tokens is extracted, by using an Artificial Intelligence, Al, based syntactic analysis, from a sequence of lines of code entered by a developer. Further, each token of the plurality of tokens is converted into a vector by using a neural word embedding technique. Subsequently, a context of the plurality of tokens is determined by using a deep autoencoder neural network technique. Furthermore, at least one code snippet is recommended from a plurality of code snippets corresponding to the context. <CIT> relates to a method, system and computer program product for obtaining vector representations of code snippets capturing semantic similarity. A first and second training set of code snippets are collected, where the first training set of code snippets implements the same function representing semantic similarity and the second training set of code snippets implements a different function representing semantic dissimilarity. <CIT>relates to a system, program product, and method for automatically determining which activation data points in a neural model have been poisoned to erroneously indicate association with a particular label or labels. A neural network is trained using potentially poisoned training data. Each of the training data points is classified using the network to retain the activations of the last hidden layer, and segment those activations by the label of corresponding training data. Clustering is applied to the retained activations of each segment, and a cluster assessment is conducted for each cluster associated with each label to distinguish clusters with potentially poisoned activations from clusters populated with legitimate activations.

The figures are not to scale. As used herein, connection references (e.g., attached, coupled, connected, and joined) may include intermediate members between the elements referenced by the connection reference and/or relative movement between those elements unless otherwise indicated. As such, connection references do not necessarily infer that two elements are directly connected and/or in fixed relation to each other. As used herein, stating that any part is in "contact" with another part is defined to mean that there is no intermediate part between the two parts.

Unless specifically stated otherwise, descriptors such as "first," "second," "third," etc., are used herein without imputing or otherwise indicating any meaning of priority, physical order, arrangement in a list, and/or ordering in any way, but are merely used as labels and/or arbitrary names to distinguish elements for ease of understanding the disclosed examples. " In such instances, it should be understood that such descriptors are used merely for identifying those elements distinctly that might, for example, otherwise share a same name. As used herein, "approximately" and "about" refer to dimensions that may not be exact due to manufacturing tolerances and/or other real world imperfections. As used herein "substantially real time" refers to occurrence in a near instantaneous manner recognizing there may be real world delays for computing time, transmission, etc. Thus, unless otherwise specified, "substantially real time" refers to real time +/- <NUM> second. As used herein, the phrase "in communication," including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events. As used herein, "processor circuitry" is defined to include (i) one or more special purpose electrical circuits structured to perform specific operation(s) and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors), and/or (ii) one or more general purpose semiconductor-based electrical circuits programmed with instructions to perform specific operations and including one or more semiconductor-based logic devices (e.g., electrical hardware implemented by one or more transistors). Examples of processor circuitry include programmed microprocessors, Field Programmable Gate Arrays (FPGAs) that may instantiate instructions, Central Processor Units (CPUs), Graphics Processor Units (GPUs), Digital Signal Processors (DSPs), XPUs, or microcontrollers and integrated circuits such as Application Specific Integrated Circuits (ASICs). For example, an XPU may be implemented by a heterogeneous computing system including multiple types of processor circuitry (e.g., one or more FPGAs, one or more CPUs, one or more GPUs, one or more DSPs, etc., and/or a combination thereof) and application programming interface(s) (API(s)) that may assign computing task(s) to whichever one(s) of the multiple types of the processing circuitry is/are best suited to execute the computing task(s).

There are a few existing techniques for code recommendation. For example, code-to-code search tools retrieve relevant code snippets from a corpus using a partial code snippet as a query. Such tools can be beneficial in areas like code transplantation recommendation and patch recommendation. These tools use manual rules to extract code features for retrieving recommendations. Manual rules may not adapt to changes in the recommended code corpus. Alternatively, supervised machine-learning based methods can make up such shortcomings. However, these supervised approaches may require millions of manually labeled training data to be effective and obtaining the training data can be expensive.

<FIG> is a schematic illustration of example circuitry to implement an automatically evolving code recommendation engine.

In the illustrated example in <FIG>, a user <NUM> interacts with a code recommendation circuitry <NUM> through an integrated development environment (IDE) circuitry <NUM>. In some examples, the code recommendation circuitry <NUM> is a plug-in for the IDE circuitry <NUM>. In some examples, the user writes code in a user interface in the IDE circuitry <NUM> and receives recommendations of suggested code snippets from the code recommendation circuitry <NUM>. The term IDE can also be referred to as an "interface" or "interface circuitry. " The term "snippet", when referring to code, defines any arbitrary amount of code operated upon within the system shown in <FIG> (e.g., as little as one character, word, or command up to one or more pages of code).

The apparatus includes means for collecting a user code snippet. The means for collecting is implemented by IDE circuitry <NUM>. The IDE circuitry <NUM> is implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which is implemented by the processor circuitry <NUM> of <FIG>, the processor circuitry <NUM> of <FIG>, and/or the Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. Alternatively, the IDE circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. The IDE circuitry <NUM> is implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In some examples, the code recommendation circuitry <NUM> learns over time how to better recommend code snippets to user <NUM> through the IDE circuitry <NUM>. In some examples, the code recommendation circuitry <NUM> utilizes a neural network model to recommend code snippets that are similar to user input code.

In the illustrated example in <FIG>, a set of functional blocks are shown within code recommendation circuitry <NUM>. The example code recommendation circuitry <NUM> includes a code similarity engine circuitry <NUM>. In some examples, the code similarity engine circuitry <NUM> processes input code (e.g., C++, Python functions) and produces a machine understandable representation of the code. In some examples, the machine understandable representation of the code is a real-valued vector that incorporates the code's syntactic and semantic information. In some examples, this representation is used by the code similarity engine circuitry <NUM> to retrieve similar code snippets from a database using distance metrics (e.g., L2-norm).

The example code similarity engine circuitry <NUM> includes a recommended code snippet generator circuitry <NUM> to generate a recommended code snippet using the structured representation of the user code snippet. In some examples, the recommended code snippet generator circuitry <NUM> receives a snippet of code from the user <NUM> through the IDE circuitry <NUM>. In some examples, the recommended code snippet generator circuitry <NUM> includes a code vector creator circuitry <NUM>. The example code vector creator circuitry <NUM> includes a structured representation determiner circuitry <NUM>. The structured representation determiner circuitry <NUM> takes a snippet of source code and translates into a structured representation. In some examples, the structured representation can include an abstract syntax tree and context-aware semantics structure.

In other examples, the functionality of the recommended code snippet generator circuitry <NUM> is instantiated in instructions. The example recommended code snippet generator instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to generate the recommended code snippet.

In other examples, the functionality of the code vector creator circuitry <NUM> is instantiated in instructions. The example code vector creator instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to create a code vector from a code snippet.

In other examples, the functionality of the structured representation determiner <NUM> is instantiated in instructions. The example structured representation determiner instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to take a snippet of source code and translate it into a structured representation.

The code similarity engine circuitry <NUM> includes means for generating a recommended code snippet using the structured representation of the user code snippet. The means for generating is implemented by the recommended code snippet generator circuitry <NUM>. The recommended code snippet generator circuitry <NUM> is implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which is implemented by the processor circuitry <NUM> of <FIG>, the processor circuitry <NUM> of <FIG>, and/or the Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. Alternatively, the recommended code snippet generator circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. The recommended code snippet generator circuitry <NUM> is implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

The code vector creator circuitry <NUM> includes means for determining a structured representation of a user code snippet. The means for determining is implemented by structured representation determiner circuitry <NUM>. The structured representation determiner circuitry <NUM> is implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which is implemented by the processor circuitry <NUM> of <FIG>, the processor circuitry <NUM> of <FIG>, and/or the Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. Alternatively, the structured representation determiner circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. The structured representation determiner circuitry <NUM> is implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In the illustrated example in <FIG>, the code vector creator circuitry <NUM> includes a neural network pre-task model circuitry <NUM>. The example neural network pre-task model circuitry <NUM> processes the structured representation. The completion of the structured representation processing by the example neural network pre-task model circuitry <NUM> produces a real-valued vector that represents the original input code snippet. A more detailed description of the processing performed by the code vector creator circuitry <NUM> is described below in the discussion related to <FIG>.

In some examples, the neural network pre-task model circuitry <NUM> is a low-confidence code semantic similarity engine. The example neural network pre-task model circuitry <NUM> utilizes a neural network downstream task model circuitry <NUM> to solve specific code recommendation problems (e.g., function name prediction, high-performance code recommendation). The example neural network downstream task model circuitry <NUM> boosts the performance of the neural network pre-task model circuitry <NUM> because downstream tasks are specialized to single areas (e.g., high performance code). However, in some examples, the neural network downstream task model circuitry <NUM> benefits from an amount of manually labeled data, which may not exist at the instantiation of the system shown in <FIG>. Therefore, in some examples, the neural network pre task model circuitry <NUM> can be deployed to instantiate the system. In some examples, the accuracy of the code similarity engine circuitry <NUM> will increase over time as a data set of labeled data is collected to train the neural network downstream task model circuitry <NUM>.

In the illustrated example in <FIG>, a recommendation database <NUM> stores recommended code (e.g., recommended code snippets). In some examples, developers may decide initially what code should be used for recommendation. For example, developers may identify an area of code that is being utilized (e.g., BLAS library code, Android developer code, etc.) and then work with a related group (e.g., a library team) to populate the code base at instantiation. In some examples, the code recommendation circuitry <NUM> is not constrained to one specific area of code recommendation, but rather can work in general for all code databases with different characteristics.

In the illustrated example in <FIG>, the recommended code snippet generator circuitry <NUM> includes a recommended code snippet retriever circuitry <NUM>. As mentioned above the structured representation is used by the code similarity engine circuitry <NUM> to retrieve similar code snippets from a database using distance metrics (e.g., L2-norm). More specifically, the example code vector creator circuitry <NUM> provides the code vector to the recommended code snippet retriever circuitry <NUM>. The example recommended code snippet retriever circuitry <NUM> uses the provided code vector to look up a recommended code snippet in a recommendation database <NUM>.

In other examples, the functionality of the recommended code snippet retriever circuitry <NUM> is instantiated in instructions. The example recommended code snippet retriever instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to use a code vector to look up a recommended code snippet in the recommendation database <NUM>.

In some examples, the recommended code snippet generator circuitry <NUM> includes means for retrieving the recommended code snippet from the recommendation database. For example, the means for retrieving may be implemented by recommended code snippet retriever circuitry <NUM>. In some examples, the recommended code snippet retriever circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the recommended code snippet retriever circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the recommended code snippet retriever circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In some examples, the recommendation database <NUM> includes a number of code snippets. The code snippets can be stored in the recommendation database <NUM> in one or more of several formats. For example, each code snippet stored in the recommendation database <NUM> may be associated with a representative code vector. In some examples, similar code vectors (and their associated code snippets) are clustered together in the recommendation database <NUM>.

In the illustrated example in <FIG>, the recommended code snippet retriever circuitry <NUM> includes a similarity score calculator circuitry <NUM>. In some examples, the similarity score calculator circuitry <NUM> calculates a similarity score of the real-valued code vector to the closest recommended code snippet database cluster using a distance metric. The distance is calculated from the real-valued code vector created by the code vector creator circuitry <NUM> to a centroid of a closest cluster of code vectors and, their associated code snippets, stored in the recommendation database <NUM>. The similarity score calculated by the similarity score calculator circuitry <NUM> represents a distance between the created code vector and a code vector in the recommendation database <NUM> (or to a centroid of a cluster of code vectors in the recommendation database <NUM>).

In other examples, the functionality of the similarity score calculator circuitry <NUM> is instantiated in instructions. The example similarity score calculator instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to calculate a similarity score of the real-valued code vector to the closest recommended code snippet database cluster using a distance metric.

In some examples, the recommended code snippet retriever circuitry <NUM> includes means for calculating a similarity score of the real-valued code vector to a closest recommended code snippet database cluster including the recommended code snippet. For example, the means for calculating may be implemented by similarity score calculator circuitry <NUM>. In some examples, the similarity score calculator circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the similarity score calculator circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the similarity score calculator circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In the illustrated example in <FIG>, the recommended code snippet retriever circuitry <NUM> includes a list populator circuitry <NUM>. The example list populator circuitry <NUM> populates a list of code snippets that have the smallest calculated distances to the created code vector. In some examples, the list populator circuitry <NUM> populates the list of code snippets with a predetermined number of code snippets. In other examples, the list populator circuitry <NUM> populates the list of the code snippets with any number of code snippets under a calculated specific distance to the created code vector.

In other examples, the functionality of the list populator circuitry <NUM> is instantiated in instructions. The example list populator instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to populate a list of code snippets that have the smallest calculated distances to the created code vector.

In some examples, the recommended code snippet receiver circuitry <NUM> includes means for populating a list of recommended code snippets at least partially with the recommended code snippet. For example, the means for populating may be implemented by list populator circuitry <NUM>. In some examples, the list populator circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the list populator circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the list populator circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In the illustrated example in <FIG>, the recommended code snippet retriever circuitry <NUM> includes a list sorter circuitry <NUM>. the example list sorter circuitry <NUM> sorts the list created by list populator circuitry <NUM> starting with a least distant code vector (and associated code snippet) and ending with a most distant code vector (and associated code snippet). In different examples, the populated and sorted list of code snippets maybe from a single code snippet up to any number of code snippets greater than one. The list of code snippets is referred to as a list of "recommended" code snippets because the distances between the code vectors of the code snippets in the list to the created code vector are small (i.e., the code snippets are similar and therefore recommended).

In other examples, the functionality of the list sorter circuitry <NUM> is instantiated in instructions. The example list sorter instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by a processor circuitry (e.g., <NUM> in <FIG>) to sort the list created by list populator circuitry <NUM> starting with a least distant code vector (and associated code snippet) and ending with a most distant code vector (and associated code snippet).

In some examples, the recommended code snippet receiver circuitry <NUM> includes means for sorting a list of recommended code snippets with a most confident recommended code snippet with a closest similarity score first to a least confident recommended code snippet with a furthest similarity score last. For example, the means for sorting may be implemented by list sorter circuitry <NUM>. In some examples, the list sorter circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the list sorter circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the list sorter circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In some examples, the recommended code snippet generator circuitry <NUM> includes means for creating a real-valued code vector from the structured representation of the user code snippet. For example, the means for creating may be implemented by code vector creator circuitry <NUM>. In some examples, the code vector creator circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the code vector creator circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the code vector creator circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In some examples, the recommended code snippet generator circuitry <NUM> sends the list of recommended code snippets to the user <NUM> through a user interface in the IDE circuitry <NUM>. In different examples, the visual format of the list of recommended code snippets presented to the user <NUM> can be any format that presents the list of recommended code snippets in an understandable way. In some examples, the IDE circuitry <NUM> prompts the user for feedback. In some examples, user feedback includes a user input comparing each recommended code snippet in the list to the user code snippet (e.g., the user code snippet received from the user <NUM> through the IDE circuitry <NUM> and represented by the code vector created by the code vector creator circuitry <NUM>). In some examples, the user input is one of the following:.

In the illustrated example in <FIG>, the code similarity engine circuitry <NUM> includes a feedback obtainer circuitry <NUM> and a code snippet pair labeler circuitry <NUM>. The example feedback obtainer circuitry <NUM> receives the user input (e.g., match, no match, or uncertain) for a given recommended code snippet from the IDE circuitry <NUM>. The user input is utilized as a label for a code snippet training pair. In some examples, each code snippet training pair includes a user code snippet (UCS), a recommended code snippet (RCS), and a label corresponding to the user input (as described above) for the recommended code snippet. The example code snippet pair labeler circuitry <NUM> labels the code snippet pair with a value corresponding to a label (e.g., match, no match, or uncertain).

In other examples, the functionality of the feedback obtainer circuitry <NUM> is instantiated in instructions. The example feedback obtainer instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to receive the user input for a given recommended code snippet from the IDE circuitry <NUM>.

In other examples, the functionality of the code snippet pair labeler circuitry <NUM> is instantiated in instructions. The example code snippet pair labeler instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to label the code snippet pair with a value corresponding to a label.

In some examples, the code similarity engine circuitry <NUM> includes means for labeling a code snippet pair. For example, the means for labeling may be implemented by code snippet pair labeler circuitry <NUM>. In some examples, the code snippet pair labeler circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the code snippet pair labeler circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the code snippet pair labeler circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

The code similarity engine circuitry <NUM> includes means for obtaining a user-determined code snippet feedback. The means for obtaining is implemented by feedback obtainer circuitry <NUM>. The feedback obtainer circuitry <NUM> is implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which is implemented by the processor circuitry <NUM> of <FIG>, the processor circuitry <NUM> of <FIG>, and/or the Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. Alternatively, the feedback obtainer circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. The feedback obtainer circuitry <NUM> is implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In the illustrated example in <FIG>, the code recommendation circuitry <NUM> includes a downstream task model training circuitry <NUM>. The example code snippet pair labeler circuitry <NUM> provides the code snippet pair (with the label) to the downstream task model training circuitry <NUM> to use for training the neural network downstream task-model circuitry <NUM>.

In some examples, prior to utilizing any code snippet pair for training, The downstream task model training circuitry <NUM> checks for malicious user input. In the illustrated example in <FIG>, the downstream task model training circuitry <NUM> includes a malicious user control circuitry <NUM> to provide a safety mechanism to prevent malicious user feedback on code snippet pairs from being used to train the neural network downstream task model circuitry <NUM>.

The example malicious user control circuitry <NUM> includes a malicious user feedback detector circuitry <NUM>. The example malicious user feedback detector circuitry <NUM> detects if there is any data that is maliciously created by the user to fool the system. One example of malicious data would be code snippet pairs where the user intentionally chooses inappropriate feedback. If the system does not verify the user feedback against the code snippet pair, malicious feedback could mislead the learning of the neural network downstream task model circuitry <NUM>, which would lead to false results from the similarity score calculator circuitry <NUM>.

In other examples, the functionality of the malicious user feedback detector circuitry <NUM> is instantiated in instructions. The example malicious user feedback detector instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to detect if there is any data that is maliciously created by the user to fool the system.

In some examples, the apparatus includes malicious user control circuitry <NUM> for detecting malicious user-determined code snippet feedback from a user. For example, the means for detecting may be implemented by malicious user feedback detector circuitry <NUM>. In some examples, the malicious user feedback detector circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least blocks <NUM> of <FIG> and <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the malicious user feedback detector circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the malicious user feedback detector circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In some examples, malicious data (e.g., user feedback) detection includes two phases: <NUM>) a manual detection phase, and <NUM>) a learned detection phase. During the manual detection phase, which happens for a limited time period, one or more users/developers inspect code snippet pairs and their corresponding user feedback labels to determine if user feedback was entered maliciously. If so, the code snippet pair, the label, and information regarding the user that entered the malicious feedback is stored. Once a sufficient malicious data set has been generated in phase one, then the example malicious user control circuitry <NUM> enters the learned detection phase.

In the illustrated example in <FIG>, the malicious user control circuitry <NUM> includes a malicious user feedback handler circuitry <NUM>. During the learned detection phase, when the example malicious user feedback detector circuitry <NUM> detects malicious user feedback, the malicious user feedback handler circuitry <NUM> disallows the affected code snippet pair from being saved to the neural network downstream task model circuitry <NUM> training data set. In some examples, the code snippet pair and the user feedback label are ignored. Additionally, in some examples the malicious user control circuitry <NUM> includes a malicious user data storing circuitry <NUM>. In some examples, the malicious user data storing circuitry <NUM> stores information regarding the code snippet pair, the malicious user feedback, and information regarding the user in a malicious code snippet pair database <NUM>.

In some examples, the malicious code snippet pair database <NUM> stores all known code snippet pairs that have been linked to malicious user feedback. Thus, in the learned detection phase, the malicious user feedback detector circuitry <NUM> can search the malicious code snippet pair database <NUM> for known code snippet pairs with malicious user feedback. In some examples, the malicious code snippet pair database <NUM> keeps a running tally of malicious feedback attempts by user ID. In some examples, if a given user ID has provided enough malicious user feedback to pass a threshold value, the malicious user feedback handler circuitry <NUM> will not allow any user feedback from that user ID. In other examples, each user ID is tracked in the malicious code snippet pair database <NUM> with a user rating value. Each time a user provides valid feedback, that user's rating value increases. Conversely, each time the user <NUM> provides malicious feedback, that user's rating value decreases. In these examples, a user's rating value can be applied to his/her user feedback as a weight component, which can increase or decrease the importance of the user feedback.

In other examples, the functionality of the malicious user feedback handler circuitry <NUM> is instantiated in instructions. The example malicious user feedback handler instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to disallow a maliciously labeled code snippet pair from being saved to the neural network downstream task model circuitry <NUM> training data set.

In other examples, the functionality of the malicious user data storing circuitry <NUM> is instantiated in instructions. The example malicious user data storing instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to store information regarding the code snippet pair, the malicious user feedback, and information regarding the user in a malicious code snippet pair database <NUM>.

In some examples, the malicious user control circuitry <NUM> includes means for disallowing the storing of the code snippet training pair in the training database in response to malicious user-determined code snippet feedback from a user being detected and means for ignoring user-determined code snippet feedback from the user in response to the count of malicious user-determined code snippet feedback attempts exceeding a threshold. For example, the means for disallowing and the means for ignoring may be implemented by malicious user feedback handler circuitry <NUM>. In some examples, the malicious user feedback handler circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least blocks <NUM> of <FIG> and <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the malicious user feedback handler circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the malicious user feedback handler circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In some examples, the malicious user control circuitry <NUM> includes means for storing the code snippet training pair with the detected malicious user-determined code snippet feedback in a malicious code snippet pair database, means for storing identification information about the user, and means for storing a count of malicious user-determined code snippet feedback attempts by the user. For example, the means for storing may be implemented by malicious user data storing circuitry <NUM>. In some examples, the malicious user data storing circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least blocks <NUM>, <NUM>, and <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the malicious user data storing circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the malicious user data storing circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In the illustrated example in <FIG>, the downstream task model training circuitry <NUM> includes a training data set storing circuitry <NUM>. Assuming the user feedback is not malicious, the example training data sets storing circuitry <NUM> stores the code snippet pair (and label) as a code snippet training pair in a training database <NUM>. In some examples, the labeled code snippet training pair is added to the training data set for the neural network downstream task models circuitry <NUM>. The training data set, stored in the training database <NUM>, increases in size as the user <NUM> (or users) code in the IDE circuitry <NUM> over time.

In other examples, the functionality of the training dataset storing circuitry <NUM> is instantiated in instructions. The example training dataset storing instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to store the code snippet pair (and label) as a code snippet training pair in a training database <NUM>.

In some examples, the downstream task model training circuitry <NUM> includes means for storing a code snippet training pair in a training database. For example, the means for storing may be implemented by training dataset storing circuitry <NUM>. In some examples, the training dataset storing circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the training dataset storing circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the training dataset storing circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

In the illustrated example in <FIG>, the downstream task model training circuitry <NUM> includes a training data set sending circuitry <NUM>. During training of the example neural network downstream task model circuitry <NUM>, the training data set sending circuitry <NUM> sends the training data set from the training database <NUM> to the NN downstream task model circuitry <NUM> to train the downstream task model. The example training data set sending circuitry <NUM> includes a timer. In these examples, the training data set is built up over a period of time (e.g., <NUM> months) and when the timer expires, the training data set sending circuitry <NUM> enters into training mode it sends the training data set to the neural network downstream task model circuitry <NUM>. In other examples, the training is dynamic and happens continuously as new code snippet pairs are received. In some examples, the downstream task model can be trained as a binary classification problem (i.e., predict if a code pair is semantically equivalent) on the accumulated dataset collected from users. In some examples, once the downstream task model is trained, a continuous integration and continuous deployment pipeline may be implemented to automatically deploy and update the model to the framework. In other examples, the NN downstream task model circuitry <NUM> is a part of the downstream task model training circuitry <NUM>.

In other examples, the functionality of the training dataset sending circuitry <NUM> is instantiated in instructions. The example training dataset sending instructions (e.g., instructions <NUM> in <FIG>) are loaded into a memory (e.g., local memory <NUM> in <FIG>) and executed by processor circuitry (e.g., <NUM> in <FIG>) to sends the training data set from the training database <NUM> to the NN downstream task model circuitry <NUM> to train the downstream task model.

In some examples, the downstream task training model circuitry <NUM> includes means for training a downstream task model to calculate the similarity score by feeding the downstream task model a training dataset and means for training the downstream task model when a timer expires. For example, the means for training may be implemented by training dataset sending circuitry <NUM>. In some examples, the training dataset sending circuitry <NUM> may be implemented by machine executable instructions such as that implemented by at least block <NUM> of <FIG> executed by processor circuitry, which may be implemented by the example processor circuitry <NUM> of <FIG>, the example processor circuitry <NUM> of <FIG>, and/or the example Field Programmable Gate Array (FPGA) circuitry <NUM> of <FIG>. In other examples, the training dataset sending circuitry <NUM> is implemented by other hardware logic circuitry, hardware implemented state machines, and/or any other combination of hardware, software, and/or firmware. For example, the training dataset sending circuitry <NUM> may be implemented by at least one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an Application Specific Integrated Circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware, but other structures are likewise appropriate.

While an example manner of implementing the code recommendation circuitry <NUM> is illustrated in <FIG>, one or more of the elements, processes, and/or devices illustrated in <FIG> may be combined, divided, re-arranged, omitted, eliminated, and/or implemented in any other way. Further, the example integrated development environment circuitry <NUM>, the example code similarity engine circuitry <NUM>, the example recommended code snippet generator circuitry <NUM>, the example code vector creator circuitry <NUM>, the example structured representation determiner circuitry <NUM>, the example neural network pre-task model circuitry <NUM>, the example neural network downstream task model circuitry <NUM>, the example recommendation database <NUM>, the example recommended code snippet retriever circuitry <NUM>, the example similarity score calculator circuitry <NUM>, the example list populator circuitry <NUM>, the example list sorter circuitry <NUM>, the example feedback obtainer circuitry <NUM>, the example code snippet pair labeler circuitry <NUM>, the example downstream task model training circuitry <NUM>, the example malicious user control circuitry <NUM>, the example malicious user feedback detector circuitry <NUM>, the example malicious user feedback handler circuitry <NUM>, the example malicious user data storing circuitry <NUM>, the example malicious code snippet pair database <NUM>, the example training dataset storing circuitry <NUM>, the example training database <NUM>, the example training dataset sending circuitry <NUM>, and/or, more generally, the example code recommendation circuitry <NUM> of <FIG>, may be implemented by hardware, software, firmware, and/or any combination of hardware, software, and/or firmware. Thus, for example, any of the example integrated development environment circuitry <NUM>, the example code similarity engine circuitry <NUM>, the example recommended code snippet generator circuitry <NUM>, the example code vector creator circuitry <NUM>, the example structured representation determiner circuitry <NUM>, the example neural network pre-task model circuitry <NUM>, the example neural network downstream task model circuitry <NUM>, the example recommendation database <NUM>, the example recommended code snippet retriever circuitry <NUM>, the example similarity score calculator circuitry <NUM>, the example list populator circuitry <NUM>, the example list sorter circuitry <NUM>, the example feedback obtainer circuitry <NUM>, the example code snippet pair labeler circuitry <NUM>, the example downstream task model training circuitry <NUM>, the example malicious user control circuitry <NUM>, the example malicious user feedback detector circuitry <NUM>, the example malicious user feedback handler circuitry <NUM>, the example malicious user data storing circuitry <NUM>, the example malicious code snippet pair database <NUM>, the example training dataset storing circuitry <NUM>, the example training database <NUM>, the example training dataset sending circuitry <NUM>, and/or, more generally, the example code recommendation circuitry <NUM>, could be implemented by processor circuitry, analog circuit(s), digital circuit(s), logic circuit(s), programmable processor(s), programmable microcontroller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), and/or field programmable logic device(s) (FPLD(s)) such as Field Programmable Gate Arrays (FPGAs). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example integrated development environment circuitry <NUM>, the example code similarity engine circuitry <NUM>, the example recommended code snippet generator circuitry <NUM>, the example code vector creator circuitry <NUM>, the example structured representation determiner circuitry <NUM>, the example neural network pre-task model circuitry <NUM>, the example neural network downstream task model circuitry <NUM>, the example recommendation database <NUM>, the example recommended code snippet retriever circuitry <NUM>, the example similarity score calculator circuitry <NUM>, the example list populator circuitry <NUM>, the example list sorter circuitry <NUM>, the example feedback obtainer circuitry <NUM>, the example code snippet pair labeler circuitry <NUM>, the example downstream task model training circuitry <NUM>, the example malicious user control circuitry <NUM>, the example malicious user feedback detector circuitry <NUM>, the example malicious user feedback handler circuitry <NUM>, the example malicious user data storing circuitry <NUM>, the example malicious code snippet pair database <NUM>, the example training dataset storing circuitry <NUM>, the example training database <NUM>, the example training dataset sending circuitry <NUM>, and/or, more generally, the example code recommendation circuitry of <FIG> is/are hereby expressly defined to include a non-transitory computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including the software and/or firmware. Further still, the example code recommendation circuitry <NUM> of <FIG> may include one or more elements, processes, and/or devices in addition to, or instead of, those illustrated in <FIG>, and/or may include more than one of any or all of the illustrated elements, processes and devices.

Flowcharts representative of example hardware logic circuitry, machine readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the code recommendation circuitry <NUM> of <FIG> is shown in <FIG>. The machine readable instructions may be one or more executable programs or portion(s) of an executable program for execution by processor circuitry, such as the processor circuitry <NUM> shown in the example processor platform <NUM> discussed below in connection with <FIG> and/or the example processor circuitry discussed below in connection with <FIG> and/or <NUM>. The program may be embodied in software stored on one or more non-transitory computer readable storage media such as a CD, a floppy disk, a hard disk drive (HDD), a DVD, a Blu-ray disk, a volatile memory (e.g., Random Access Memory (RAM) of any type, etc.), or a non-volatile memory (e.g., FLASH memory, an HDD, etc.) associated with processor circuitry located in one or more hardware devices, but the entire program and/or parts thereof could alternatively be executed by one or more hardware devices other than the processor circuitry and/or embodied in firmware or dedicated hardware. The machine readable instructions may be distributed across multiple hardware devices and/or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, the client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that may facilitate communication between a server and an endpoint client hardware device). Similarly, the non-transitory computer readable storage media may include one or more mediums located in one or more hardware devices. Further, although the example program is described with reference to the flowchart illustrated in <FIG>, many other methods of implementing the example code recommendation circuitry <NUM> of <FIG> may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., processor circuitry, discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware. The processor circuitry may be distributed in different network locations and/or local to one or more hardware devices (e.g., a single-core processor (e.g., a single core central processor unit (CPU)), a multi-core processor (e.g., a multi-core CPU), etc.) in a single machine, multiple processors distributed across multiple servers of a server rack, multiple processors distributed across one or more server racks, a CPU and/or a FPGA located in the same package (e.g., the same integrated circuit (IC) package or in two or more separate housings, etc).

The machine readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packaged format, etc. Machine readable instructions as described herein may be stored as data or a data structure (e.g., as portions of instructions, code, representations of code, etc.) that may be utilized to create, manufacture, and/or produce machine executable instructions. For example, the machine readable instructions may be fragmented and stored on one or more storage devices and/or computing devices (e.g., servers) located at the same or different locations of a network or collection of networks (e.g., in the cloud, in edge devices, etc.). The machine readable instructions may require one or more of installation, modification, adaptation, updating, combining, supplementing, configuring, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., in order to make them directly readable, interpretable, and/or executable by a computing device and/or other machine. For example, the machine readable instructions may be stored in multiple parts, which are individually compressed, encrypted, and/or stored on separate computing devices, wherein the parts when decrypted, decompressed, and/or combined form a set of machine executable instructions that implement one or more operations that may together form a program such as that described herein.

In another example, the machine readable instructions may be stored in a state in which they may be read by processor circuitry, but require addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the machine readable instructions on a particular computing device or other device. In another example, the machine readable instructions may need to be configured (e.g., settings stored, data input, network addresses recorded, etc.) before the machine readable instructions and/or the corresponding program(s) can be executed in whole or in part. Thus, machine readable media, as used herein, may include machine readable instructions and/or program(s) regardless of the particular format or state of the machine readable instructions and/or program(s) when stored or otherwise at rest or in transit.

As mentioned above, the example operations of <FIG> may be implemented using executable instructions (e.g., computer and/or machine readable instructions) stored on one or more non-transitory computer and/or machine readable media such as optical storage devices, magnetic storage devices, an HDD, a flash memory, a read-only memory (ROM), a CD, a DVD, a cache, a RAM of any type, a register, and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the terms non-transitory computer readable medium and non-transitory computer readable storage medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media.

Thus, whenever a claim employs any form of "include" or "comprise" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within a claim recitation of any kind, it is to be understood that additional elements, terms, etc., may be present without falling outside the scope of the corresponding claim or recitation. The term "and/or" when used, for example, in a form such as A, B, and/or C refers to any combination or subset of A, B, C such as (<NUM>) A alone, (<NUM>) B alone, (<NUM>) C alone, (<NUM>) A with B, (<NUM>) A with C, (<NUM>) B with C, or (<NUM>) A with B and with C. As used herein in the context of describing structures, components, items, objects and/or things, the phrase "at least one of A and B" is intended to refer to implementations including any of (<NUM>) at least one A, (<NUM>) at least one B, or (<NUM>) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects and/or things, the phrase "at least one of A or B" is intended to refer to implementations including any of (<NUM>) at least one A, (<NUM>) at least one B, or (<NUM>) at least one A and at least one B. As used herein in the context of describing the performance or execution of processes, instructions, actions, activities and/or steps, the phrase "at least one of A and B" is intended to refer to implementations including any of (<NUM>) at least one A, (<NUM>) at least one B, or (<NUM>) at least one A and at least one B. Similarly, as used herein in the context of describing the performance or execution of processes, instructions, actions, activities and/or steps, the phrase "at least one of A or B" is intended to refer to implementations including any of (<NUM>) at least one A, (<NUM>) at least one B, or (<NUM>) at least one A and at least one B.

The term "a" or "an" object, as used herein, refers to one or more of that object. The terms "a" (or "an"), "one or more", and "at least one" are used interchangeably herein. Furthermore, although individually listed, a plurality of means, elements or method actions may be implemented by, e.g., the same entity or object.

<FIG> is a flowchart representative of machine readable instructions and/or operations that may be executed and/or instantiated by processor circuitry to implement an automatically evolving code recommendation engine. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the integrated development environment circuitry <NUM> collects a user code snippet.

At block <NUM>, the structured representation determiner circuitry <NUM> determines a structured representation of the collected user code snippet.

At block <NUM>, the recommended code snippet generator circuitry <NUM> generates a recommended code snippet using the structured representation of the user code snippet.

At block <NUM>, the feedback obtainer circuitry <NUM> obtains a user-determined code snippet feedback from the user, comparing the recommended code snippet (RCS) to the user code snippet (UCS).

At block <NUM>, the training dataset storing circuitry <NUM> stores the code snippet training pair (e.g., the RCS and the UCS) in a training database. At this point the process ends.

<FIG> is a flowchart representative of example machine readable instructions and/or example operations that may be executed and/or instantiated by processor circuitry to implement a process to generate a recommended code snippet using a structured representation. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the example code vector creator circuitry <NUM> creates a real-valued code vector from the structured representation of the user code snippet by processing the structured representation through a neural network.

At block <NUM>, the example recommended code snippet retriever circuitry <NUM> retrieves a recommended code snippet from a recommendation database using the real-valued code vector. At this point the process ends.

<FIG> is a flowchart representative of example machine readable instructions and/or example operations that may be executed and/or instantiated by processor circuitry to implement a process to retrieve a recommended code snippet from a recommendation database using a real-valued code vector. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the example similarity score calculator circuitry <NUM> calculates a similarity score of the real-valued code vector to a closest recommended code snippet database cluster. In some examples, the closest recommended code snippet database cluster includes the recommended code snippet.

At block <NUM>, the example list populator circuitry <NUM> populates a list of recommended code snippets at least partially with the recommended code snippet.

At block <NUM>, the example list sorter circuitry <NUM> sorts the list of recommended code snippets from a code snippet with a closest similarity score to a code snippet with a furthest similarity score.

At block <NUM>, the example feedback obtainer circuitry <NUM> sends the sorted list of recommended code snippets to the user. In some examples, the sorted list of recommended code snippets may be sent to the IDE circuitry <NUM> and presented to the user through an IDE user interface. At this point the process ends.

In some examples, an unsupervised sub-linear search algorithm is used to retrieve the list of recommended code snippets from the recommendation database <NUM>. In some examples, the recommendation database <NUM> has machine-understandable representations (e.g., real-valued vectors) of code that can be classified into sub-groups using clustering algorithms such as k-means. In some examples, representations that are closer to each other, using a distance measurement (e.g. L2-norm), belong to a unique cluster. In some examples, each cluster has a centroid. The example centroid is used to calculate the distance of the cluster to the user code snippet's structured representation. The example recommended code snippet generator circuitry <NUM> then recommends recommended code snippets (e.g., programs) stored in the recommendation database <NUM> from clusters that have a small distance from the user code snippet's structured representation.

<FIG> is a flowchart representative of example machine readable instructions and/or example operations that may be executed and/or instantiated by processor circuitry to implement a process to label a code snippet pair with user-selected feedback. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the example feedback obtainer circuitry <NUM> sets a default user-selected feedback to UNCERTAIN. In some examples, an UNCERTAIN feedback is equivalent to the user not selecting any feedback (e.g., NO ANSWER).

At block <NUM>, the example feedback obtainer circuitry <NUM> checks if the user selects a MATCH feedback between the user code snippet and the recommended code snippet. If the user has selected a MATCH feedback, then, at block <NUM> the example code snippet pair labeler circuitry <NUM> labels the code snippet pair as a MATCH.

If the user has not selected a MATCH at block <NUM>, then, at block <NUM>, the example feedback obtainer circuitry <NUM> checks if the user selects a NO MATCH feedback between the user code snippet and the recommended code snippet. If the user has selected a NO MATCH feedback, then, at block <NUM> the example code snippet pair labeler circuitry <NUM> labels the code snippet pair as a NO MATCH.

If the user has not selected a NO MATCH at block <NUM>, then, at block <NUM>, the example feedback obtainer circuitry <NUM> checks if the user selects an UNCERTAIN feedback between the user code snippet and the recommended code snippet. If the user has selected an UNCERTAIN feedback, then, at block <NUM> the example code snippet pair labeler circuitry <NUM> labels the code snippet pair as UNCERTAIN.

If the user has not selected any feedback at any of blocks <NUM>, <NUM>, and <NUM>, then, at block <NUM> then the example code snippet pair labeler circuitry <NUM> labels the code snippet pair as UNCERTAIN. At this point the process ends.

In some examples, if the code snippet pair can be referred to as a code snippet "training" pair because the code snippet pair may be stored in a training dataset in a training database for downstream model training.

In some examples, the data collected to be stored in the training database is the pair of code snippets including the user code snippet (UCS) and the recommended code snippet (RCS). In some examples, each {RCS, UCS} pair corresponds to one of three categories: <NUM>) a positive pair, <NUM>) a negative pair, and <NUM>) an uncertain pair. A positive pair is defined as a semantically similar pair, a negative pair is defined as a semantically dissimilar pair, and an uncertain pair is defined as showing the user ignores or is unsure about the RCS.

In some examples, given a UCS, the recommended code snippet retriever circuitry <NUM>, in concert with the feedback obtainer circuitry <NUM>, provides a sorted list of recommended code snippets. When paired with the user code snippet, an example list may be written as {RCS1, UCS}, {RCS2, UCS},. {RCSn, UCS}, where n is the number of RCS in the list. In some examples, the smaller numbers in the list indicate more confident recommendations. For example, the recommended code snippet in the recommendation database with the smallest distance metric from the user code snippet's code vector would be listed as RCS1, otherwise considered and referred to as the RCS the recommended code snippet retriever circuitry <NUM> is most confident about.

In some examples, for each RCS in a recommended list, a user can mark one of the states from the list [MATCH, NO MATCH, UNCERTAIN]. In some examples, the code snippet pair labeler circuitry <NUM> may simplify the labeling of a given code snippet pair by labeling each code snippet pair with number in the range of <NUM> to <NUM>. In some examples, when the user selects a MATCH, the code snippet pair labeler circuitry <NUM> labels (e.g., assigns) the code snippet pair (e.g., {RCSn, UCS]) with a value of <NUM>. In some examples, when the user selects a NO MATCH, the code snippet pair labeler circuitry <NUM> labels the code snippet pair with a value of <NUM>. If a user selects RCSm as the answer from the list of recommended code snippets, where m is the position of the selected code snippet in the list, in some examples, the code snippet pair labeler circuitry <NUM> assigns a value of less than the similarity score between {RCSm, UCS} to the uncertain pairs before m (e.g., values between but not including <NUM> and <NUM>). In these examples, the remaining uncertain pairs with an RCS greater than m will be ignored. In some examples, if the user did not select a solution from the list (i.e., the user feedback is UNCERTAIN), all uncertain pairs will be ignored.

<FIG> is a flowchart representative of machine readable instructions and/or operations that may be executed and/or instantiated by processor circuitry to implement a process to detect and handle malicious user feedback for a code snippet pair. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the malicious user feedback detector circuitry <NUM> determines if malicious user feedback for a given code snippet pair has been detected.

If malicious user feedback has been detected, then at block <NUM>, the malicious user feedback handler circuitry <NUM> disallows storing the code snippet training pair in the training database. At this point the process ends.

In some examples, when a user (e.g., user <NUM>) takes an action (e.g., selects or ignores a recommended code snippet), the recommended code snippet generator circuitry <NUM> creates a list of labeled data pairs. The example malicious user control circuitry <NUM> and, more specifically, the example malicious user feedback detector circuitry <NUM>, then detects if there are any data that is maliciously created by the user <NUM> to fool the system. For example, code snippet pairs generated by the user <NUM> intentionally choosing an inappropriate recommendation is a malicious user-recommended feedback.

As described above, the malicious user feedback detection includes two phases: <NUM>) manual detection, and <NUM>) learned detection. In some examples, the manual detection phase utilizes humans to inspect code pairs produced by users' actions to create an initial dataset. In some examples, the manual phase only lasts for a certain amount of time (e.g., <NUM> months). At the end of the manual phase, in some examples, a database of human-labeled benign and malicious code pairs are present. The example malicious user feedback detector circuitry <NUM> then utilizes the dataset during the learned (automated) detection phase to learn a classifier to detect malicious code pairs. Specifically, in some examples, the example malicious user feedback detector circuitry <NUM> can use machine learning models (e.g., MISIM, ControlFlag, etc.) that learn patterns of malicious code pairs. The example malicious user feedback detector circuitry <NUM> utilizes one or more of the machine-learning models to take a code snippet pair as an input and produces a binary-valued output that indicates if the code snippet pair is malicious.

<FIG> is a flowchart representative of example machine readable instructions and/or example operations that may be executed and/or instantiated by processor circuitry to implement another process to detect and handle malicious user feedback for a code snippet pair. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the example malicious user feedback detector circuitry <NUM> determines if malicious user feedback for a given code snippet pair has been detected.

If malicious user feedback has been detected, then at block <NUM>, the example malicious user data storing circuitry <NUM> stores the code snippet training pair with the detected malicious user-determined feedback in a malicious code snippet pair database.

At block <NUM>, the example malicious user data storing circuitry <NUM> stores identification information about the user that provided the malicious user-determined feedback in the malicious code snippet pair database.

At block <NUM>, the example malicious user data storing circuitry <NUM> stores a count of malicious user-determined feedback attempts by the user in the malicious code snippet pair database. At this point the process ends.

<FIG> is a flowchart representative of example machine readable instructions and/or example operations that may be executed and/or instantiated by processor circuitry to implement a process to handle a user that has a number of attempts of malicious user feedback that exceed a threshold. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the example malicious user feedback handler circuitry <NUM> determines if malicious user-determined feedback attempts from a user has exceeded a threshold number of attempts.

If the number of malicious user-determined feedback attempts by the user has exceeded the threshold, then, at block <NUM>, the example malicious user feedback handler circuitry <NUM> ignores the user-determined code snippet feedback from the user. At this point the process ends.

In some examples, a user credibility ranking system is implemented by the example malicious user feedback handler circuitry <NUM>. In some examples, the example malicious user feedback handler circuitry <NUM> may use the malicious user detection process described above in the discussion regarding <FIG> to dynamically update a user's credibility. In some examples, the example malicious user feedback handler circuitry <NUM> additionally utilizes environmental meta-data help determine user credibility.

In some examples, if a code snippet pair selected by a user (e.g., user <NUM>) is detected and marked/labeled as malicious, then the example malicious user feedback handler circuitry <NUM> decreases the credibility of that user. On the other hand, if the user-selected code snippet pair is benign, then the example malicious user feedback handler circuitry <NUM> increases the credibility of the user.

In some examples, the credibility of a user is determined by a count of malicious user-determined feedback events by that user. In some examples, the example malicious user feedback handler circuitry <NUM> keeps a dynamic tally of the count of malicious feedback attempts and if the count exceeds a threshold value, the example malicious user feedback handler circuitry <NUM> ignores further feedback from the user. In some examples, malicious feedback increases the count and benign feedback reduces the count, thus the current dynamic count can be monitored by the example malicious user feedback handler circuitry <NUM> over time and the feedback from the user is utilized whenever the count is at or below the threshold or ignored when the count exceeds the threshold.

In other examples, the example malicious user feedback handler circuitry <NUM> may utilize the count of malicious user feedback attempts for a given user as a weight when considering the user's current feedback. For example, when the malicious user feedback count for a given user is high, the significance of feedback from that user is low, and vice versa, when the malicious user feedback count for a given user is low, the significance of feedback from that user is high.

The user credibility process implemented by the example malicious user feedback handler circuitry <NUM> learns over time to put less trust in users who select malicious code pairs and to put more trust in users who continue to select benign code pairs. In some examples, the malicious user feedback handler circuitry <NUM> stores a trustworthiness score for each known user in the malicious code snippet pair database <NUM>. The example malicious user feedback handler circuitry <NUM> begins with a fixed initial trustworthiness score for all users and the trustworthiness score is update periodically (e.g., each time the code similarity engine circuitry <NUM> is trained/updated). In some examples, if a user has several malicious actions in during this period, the user's score decreases by some configurable number. Once a user's score is below a pre-determined threshold, the example malicious user feedback handler circuitry <NUM> no longer collects data from such users (i.e., ignores the user's feedback and/or disallows storing a code snippet pair with a label from the user). In other examples, the example malicious user feedback handler circuitry <NUM> updates the user trustworthiness score in real-time each time a code snippet pair is processed.

<FIG> is a flowchart representative of example machine readable instructions and/or example operations that may be executed and/or instantiated by processor circuitry to implement a process to train a downstream task model. The machine readable instructions and/or operations of <FIG> begin at block <NUM>, at which the example training dataset sending circuitry <NUM> determines if a training timer has expired.

If the training timer has expired, then, at block <NUM>, the example training dataset sending circuitry <NUM> trains the downstream task model (e.g., the neural network downstream task model circuitry) to calculate similarity scores by feeding the training dataset of labeled code snippet training pairs, stored in the training database, to the code similarity engine circuitry <NUM>. At this point the process ends.

In some examples, the neural network downstream task model circuitry can be trained at different training periods to improve code similarity determination accuracy (e.g., once a week, once a month, once every three months, etc.). In some examples, the neural network downstream task model circuitry <NUM> uses weights from the neural network pre-task model circuitry <NUM>. In some examples, every training period (e.g., <NUM> months), the neural network downstream task model circuitry <NUM> can be trained as a binary classification problem (e.g., to predict if a code snippet pair is semantically equivalent) on the accumulated dataset (stored in the training database <NUM>) collected from users. Once the example neural network downstream task model circuitry <NUM> is trained, the example downstream task model training circuitry <NUM> employs a continuous integration and continuous deployment pipeline implemented to automatically deploy the example neural network downstream task model circuitry <NUM>. In some examples, the deployment of the neural network downstream task model circuitry <NUM> takes place once each training (block <NUM>) completes.

<FIG> is a block diagram of an example processor platform <NUM> structured to execute and/or instantiate the machine readable instructions and/or operations of <FIG> to implement the apparatus of <FIG>. The processor platform <NUM> can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, a headset (e.g., an augmented reality (AR) headset, a virtual reality (VR) headset, etc.) or other wearable device, or any other type of computing device.

The processor platform <NUM> of the illustrated example includes processor circuitry <NUM>. The processor circuitry <NUM> of the illustrated example is hardware. For example, the processor circuitry <NUM> can be implemented by one or more integrated circuits, logic circuits, FPGAs microprocessors, CPUs, GPUs, DSPs, and/or microcontrollers from any desired family or manufacturer. The processor circuitry <NUM> may be implemented by one or more semiconductor based (e.g., silicon based) devices. In this example, the processor circuitry <NUM> implements the example integrated development environment circuitry <NUM>, the example code similarity engine circuitry <NUM>, the example recommended code snippet generator circuitry <NUM>, the example code vector creator circuitry <NUM>, the example structured representation determiner circuitry <NUM>, the example neural network pre-task model circuitry <NUM>, the example neural network downstream task model circuitry <NUM>, the example recommended code snippet retriever circuitry <NUM>, the example similarity score calculator circuitry <NUM>, the example list populator circuitry <NUM>, the example list sorter circuitry <NUM>, the example feedback obtainer circuitry <NUM>, the example code snippet pair labeler circuitry <NUM>, the example downstream task model training circuitry <NUM>, the example malicious user control circuitry <NUM>, the example malicious user feedback detector circuitry <NUM>, the example malicious user feedback handler circuitry <NUM>, the example malicious user data storing circuitry <NUM>, the example training dataset storing circuitry <NUM>, the example training dataset sending circuitry <NUM>, and/or, more generally, the example code recommendation circuitry of <FIG>.

The processor circuitry <NUM> of the illustrated example includes a local memory <NUM> (e.g., a cache, registers, etc.). The processor circuitry <NUM> of the illustrated example is in communication with a main memory including a volatile memory <NUM> and a non-volatile memory <NUM> by a bus <NUM>. The volatile memory <NUM> may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and/or any other type of RAM device. Access to the main memory <NUM>, <NUM> of the illustrated example is controlled by a memory controller <NUM>.

The processor platform <NUM> of the illustrated example also includes interface circuitry <NUM>. The interface circuitry <NUM> may be implemented by hardware in accordance with any type of interface standard, such as an Ethernet interface, a universal serial bus (USB) interface, a Bluetooth® interface, a near field communication (NFC) interface, a PCI interface, and/or a PCIe interface.

In the illustrated example, one or more input devices <NUM> are connected to the interface circuitry <NUM>. The input device(s) <NUM> permit(s) a user to enter data and/or commands into the processor circuitry <NUM>. The input device(s) <NUM> can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, an isopoint device, and/or a voice recognition system.

One or more output devices <NUM> are also connected to the interface circuitry <NUM> of the illustrated example. The output devices <NUM> can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube (CRT) display, an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer, and/or speaker. The interface circuitry <NUM> of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip, and/or graphics processor circuitry such as a GPU.

The interface circuitry <NUM> of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and/or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) by a network <NUM>. The communication can be by, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, an optical connection, etc..

The processor platform <NUM> of the illustrated example also includes one or more mass storage devices <NUM> to store software and/or data. Examples of such mass storage devices <NUM> include magnetic storage devices, optical storage devices, floppy disk drives, HDDs, CDs, Blu-ray disk drives, redundant array of independent disks (RAID) systems, solid state storage devices such as flash memory devices, and DVD drives.

The machine executable instructions <NUM>, which may be implemented by the machine readable instructions of <FIG>, may be stored in the mass storage device <NUM>, in the volatile memory <NUM>, in the non-volatile memory <NUM>, and/or on a removable non-transitory computer readable storage medium such as a CD or DVD.

<FIG> is a block diagram of an example implementation of the processor circuitry <NUM> of <FIG>. In this example, the processor circuitry <NUM> of <FIG> is implemented by a microprocessor <NUM>. For example, the microprocessor <NUM> may implement multi-core hardware circuitry such as a CPU, a DSP, a GPU, an XPU, etc. Although it may include any number of example cores <NUM> (e.g., <NUM> core), the microprocessor <NUM> of this example is a multi-core semiconductor device including N cores. The cores <NUM> of the microprocessor <NUM> may operate independently or may cooperate to execute machine readable instructions. For example, machine code corresponding to a firmware program, an embedded software program, or a software program may be executed by one of the cores <NUM> or may be executed by multiple ones of the cores <NUM> at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is split into threads and executed in parallel by two or more of the cores <NUM>. The software program may correspond to a portion or all of the machine readable instructions and/or operations represented by the flowcharts of <FIG>.

The cores <NUM> may communicate by an example bus <NUM>. In some examples, the bus <NUM> may implement a communication bus to effectuate communication associated with one(s) of the cores <NUM>. For example, the bus <NUM> may implement at least one of an Inter-Integrated Circuit (I2C) bus, a Serial Peripheral Interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the bus <NUM> may implement any other type of computing or electrical bus. The cores <NUM> may obtain data, instructions, and/or signals from one or more external devices by example interface circuitry <NUM>. The cores <NUM> may output data, instructions, and/or signals to the one or more external devices by the interface circuitry <NUM>. Although the cores <NUM> of this example include example local memory <NUM> (e.g., Level <NUM> (L1) cache that may be split into an L1 data cache and an L1 instruction cache), the microprocessor <NUM> also includes example shared memory <NUM> that may be shared by the cores (e.g., Level <NUM> (L2_ cache)) for high-speed access to data and/or instructions. Data and/or instructions may be transferred (e.g., shared) by writing to and/or reading from the shared memory <NUM>. The local memory <NUM> of each of the cores <NUM> and the shared memory <NUM> may be part of a hierarchy of storage devices including multiple levels of cache memory and the main memory (e.g., the main memory <NUM>, <NUM> of <FIG>). Typically, higher levels of memory in the hierarchy exhibit lower access time and have smaller storage capacity than lower levels of memory. Changes in the various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherency policy.

Each core <NUM> may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuitry. Each core <NUM> includes control unit circuitry <NUM>, arithmetic and logic (AL) circuitry (sometimes referred to as an ALU) <NUM>, a plurality of registers <NUM>, the L1 cache <NUM>, and an example bus <NUM>. Other structures may be present. For example, each core <NUM> may include vector unit circuitry, single instruction multiple data (SIMD) unit circuitry, load/store unit (LSU) circuitry, branch/jump unit circuitry, floating-point unit (FPU) circuitry, etc. The control unit circuitry <NUM> includes semiconductor-based circuits structured to control (e.g., coordinate) data movement within the corresponding core <NUM>. The AL circuitry <NUM> includes semiconductor-based circuits structured to perform one or more mathematic and/or logic operations on the data within the corresponding core <NUM>. The AL circuitry <NUM> of some examples performs integer based operations. In other examples, the AL circuitry <NUM> also performs floating point operations. In yet other examples, the AL circuitry <NUM> may include first AL circuitry that performs integer based operations and second AL circuitry that performs floating point operations. In some examples, the AL circuitry <NUM> may be referred to as an Arithmetic Logic Unit (ALU). The registers <NUM> are semiconductor-based structures to store data and/or instructions such as results of one or more of the operations performed by the AL circuitry <NUM> of the corresponding core <NUM>. For example, the registers <NUM> may include vector register(s), SIMD register(s), general purpose register(s), flag register(s), segment register(s), machine specific register(s), instruction pointer register(s), control register(s), debug register(s), memory management register(s), machine check register(s), etc. The registers <NUM> may be arranged in a bank as shown in <FIG>. Alternatively, the registers <NUM> may be organized in any other arrangement, format, or structure including distributed throughout the core <NUM> to shorten access time. The bus <NUM> may implement at least one of an I2C bus, a SPI bus, a PCI bus, or a PCIe bus.

Each core <NUM> and/or, more generally, the microprocessor <NUM> may include additional and/or alternate structures to those shown and described above. For example, one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHAs), one or more converged/common mesh stops (CMSs), one or more shifters (e.g., barrel shifter(s)) and/or other circuitry may be present. The microprocessor <NUM> is a semiconductor device fabricated to include many transistors interconnected to implement the structures described above in one or more integrated circuits (ICs) contained in one or more packages. The processor circuitry may include and/or cooperate with one or more accelerators. In some examples, accelerators are implemented by logic circuitry to perform certain tasks more quickly and/or efficiently than can be done by a general puspose processor. Examples of accelerators include ASICs and FPGAs such as those discussed herein. A GPU or other programmable device can also be an accelerator. Accelerators may be on-board the processor circuitry, in the same chip package as the processor circuitry and/or in one or more separate packages from the processor circuitry.

<FIG> is a block diagram of another example implementation of the processor circuitry <NUM> of <FIG>. In this example, the processor circuitry <NUM> is implemented by FPGA circuitry <NUM>. The FPGA circuitry <NUM> can be used, for example, to perform operations that could otherwise be performed by the example microprocessor <NUM> of <FIG> executing corresponding machine readable instructions. However, once configured, the FPGA circuitry <NUM> instantiates the machine readable instructions in hardware and, thus, can often execute the operations faster than they could be performed by a general purpose microprocessor executing the corresponding software.

More specifically, in contrast to the microprocessor <NUM> of <FIG> described above (which is a general purpose device that may be programmed to execute some or all of the machine readable instructions represented by the flowcharts of <FIG> but whose interconnections and logic circuitry are fixed once fabricated), the FPGA circuitry <NUM> of the example of <FIG> includes interconnections and logic circuitry that may be configured and/or interconnected in different ways after fabrication to instantiate, for example, some or all of the machine readable instructions represented by the flowcharts of <FIG>. In particular, the FPGA <NUM> may be thought of as an array of logic gates, interconnections, and switches. The switches can be programmed to change how the logic gates are interconnected by the interconnections, effectively forming one or more dedicated logic circuits (unless and until the FPGA circuitry <NUM> is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on data received by input circuitry. Those operations may correspond to some or all of the software represented by the flowcharts of <FIG>. As such, the FPGA circuitry <NUM> may be structured to effectively instantiate some or all of the machine readable instructions of the flowcharts of <FIG> as dedicated logic circuits to perform the operations corresponding to those software instructions in a dedicated manner analogous to an ASIC. Therefore, the FPGA circuitry <NUM> may perform the operations corresponding to the some or all of the machine readable instructions of <FIG> faster than the general purpose microprocessor can execute the same.

In the example of <FIG>, the FPGA circuitry <NUM> is structured to be programmed (and/or reprogrammed one or more times) by an end user by a hardware description language (HDL) such as Verilog. The FPGA circuitry <NUM> of <FIG>, includes example input/output (I/O) circuitry <NUM> to obtain and/or output data to/from example configuration circuitry <NUM> and/or external hardware (e.g., external hardware circuitry) <NUM>. For example, the configuration circuitry <NUM> may implement interface circuitry that may obtain machine readable instructions to configure the FPGA circuitry <NUM>, or portion(s) thereof. In some such examples, the configuration circuitry <NUM> may obtain the machine readable instructions from a user, a machine (e.g., hardware circuitry (e.g., programmed or dedicated circuitry) that may implement an Artificial Intelligence/Machine Learning (AI/ML) model to generate the instructions), etc. In some examples, the external hardware <NUM> may implement the microprocessor <NUM> of <FIG>. The FPGA circuitry <NUM> also includes an array of example logic gate circuitry <NUM>, a plurality of example configurable interconnections <NUM>, and example storage circuitry <NUM>. The logic gate circuitry <NUM> and interconnections <NUM> are configurable to instantiate one or more operations that may correspond to at least some of the machine readable instructions of <FIG> and/or other desired operations. The logic gate circuitry <NUM> shown in <FIG> is fabricated in groups or blocks. Each block includes semiconductor-based electrical structures that may be configured into logic circuits. In some examples, the electrical structures include logic gates (e.g., And gates, Or gates, Nor gates, etc.) that provide basic building blocks for logic circuits. Electrically controllable switches (e.g., transistors) are present within each of the logic gate circuitry <NUM> to enable configuration of the electrical structures and/or the logic gates to form circuits to perform desired operations. The logic gate circuitry <NUM> may include other electrical structures such as look-up tables (LUTs), registers (e.g., flip-flops or latches), multiplexers, etc..

The interconnections <NUM> of the illustrated example are conductive pathways, traces, vias, or the like that may include electrically controllable switches (e.g., transistors) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more of the logic gate circuitry <NUM> to program desired logic circuits.

The storage circuitry <NUM> of the illustrated example is structured to store result(s) of the one or more of the operations performed by corresponding logic gates. The storage circuitry <NUM> may be implemented by registers or the like. In the illustrated example, the storage circuitry <NUM> is distributed amongst the logic gate circuitry <NUM> to facilitate access and increase execution speed.

The example FPGA circuitry <NUM> of <FIG> also includes example Dedicated Operations Circuitry <NUM>. In this example, the Dedicated Operations Circuitry <NUM> includes special purpose circuitry <NUM> that may be invoked to implement commonly used functions to avoid the need to program those functions in the field. Examples of such special purpose circuitry <NUM> include memory (e.g., DRAM) controller circuitry, PCIe controller circuitry, clock circuitry, transceiver circuitry, memory, and multiplier-accumulator circuitry. Other types of special purpose circuitry may be present. In some examples, the FPGA circuitry <NUM> may also include example general purpose programmable circuitry <NUM> such as an example CPU <NUM> and/or an example DSP <NUM>. Other general purpose programmable circuitry <NUM> may additionally or alternatively be present such as a GPU, an XPU, etc., that can be programmed to perform other operations.

Although <FIG> and <FIG> illustrate two example implementations of the processor circuitry <NUM> of <FIG>, many other approaches are contemplated. For example, as mentioned above, modern FPGA circuitry may include an on-board CPU, such as one or more of the example CPU <NUM> of <FIG>. Therefore, the processor circuitry <NUM> of <FIG> may additionally be implemented by combining the example microprocessor <NUM> of <FIG> and the example FPGA circuitry <NUM> of <FIG>. In some such hybrid examples, a first portion of the machine readable instructions represented by the flowcharts of <FIG> may be executed by one or more of the cores <NUM> of <FIG> and a second portion of the machine readable instructions represented by the flowcharts of <FIG> may be executed by the FPGA circuitry <NUM> of <FIG>.

In some examples, the processor circuitry <NUM> of <FIG> may be in one or more packages. For example, the processor circuitry <NUM> of <FIG> and/or the FPGA circuitry <NUM> of <FIG> may be in one or more packages. In some examples, an XPU may be implemented by the processor circuitry <NUM> of <FIG>, which may be in one or more packages. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in still yet another package.

A block diagram illustrating an example software distribution platform <NUM> to distribute software such as the example machine readable instructions <NUM> of <FIG> to hardware devices owned and/or operated by third parties is illustrated in <FIG>. The example software distribution platform <NUM> may be implemented by any computer server, data facility, cloud service, etc., capable of storing and transmitting software to other computing devices. The third parties may be customers of the entity owning and/or operating the software distribution platform <NUM>. For example, the entity that owns and/or operates the software distribution platform <NUM> may be a developer, a seller, and/or a licensor of software such as the example machine readable instructions <NUM> of <FIG>. The third parties may be consumers, users, retailers, OEMs, etc., who purchase and/or license the software for use and/or re-sale and/or sub-licensing. In the illustrated example, the software distribution platform <NUM> includes one or more servers and one or more storage devices. The storage devices store the machine readable instructions <NUM>, which may correspond to the example machine readable instructions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>, as described above. The one or more servers of the example software distribution platform <NUM> are in communication with a network <NUM>, which may correspond to any one or more of the Internet and/or any of the example networks <NUM> described above. In some examples, the one or more servers are responsive to requests to transmit the software to a requesting party as part of a commercial transaction. Payment for the delivery, sale, and/or license of the software may be handled by the one or more servers of the software distribution platform and/or by a third party payment entity. The servers enable purchasers and/or licensors to download the machine readable instructions <NUM> from the software distribution platform <NUM>. For example, the software, which may correspond to the example machine readable instructions <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>, may be downloaded to the example processor platform <NUM>, which is to execute the machine readable instructions <NUM> to implement the code recommendation circuitry <NUM>. In some example, one or more servers of the software distribution platform <NUM> periodically offer, transmit, and/or force updates to the software (e.g., the example machine readable instructions <NUM> of <FIG>) to ensure improvements, patches, updates, etc., are distributed and applied to the software at the end user devices.

From the foregoing, it will be appreciated that example systems, methods, apparatus, and articles of manufacture have been disclosed that implement an automatically evolving code recommendation engine (e.g., circuitry, logic). The disclosed systems, methods, apparatus, and articles of manufacture improve the efficiency of using a computing device by implementing an automatically evolving code recommendation circuitry. The code recommendation circuitry utilizes user input to help classify recommended code snippets as to whether they match against a user-inputted code snippet. The process removes a large amount of overhead involved in classifying code snippets by hand in advance of deployment of the system. The disclosed systems, methods, apparatus, and articles of manufacture are accordingly directed to one or more improvement(s) in the operation of a machine such as a computer or other electronic and/or mechanical device.

Claim 1:
A computer-implemented method, comprising:
collecting (<NUM>) a user code snippet;
determining (<NUM>) a structured representation of the user code snippet;
generating (<NUM>) a recommended code snippet using the structured representation of the user code snippet;
obtaining (<NUM>) user-determined code snippet feedback comparing the user code snippet to the recommended code snippet, the user-determined code snippet feedback indicating one of a match, no match, or uncertain; and
causing (<NUM>) storage of a code snippet training pair in a training database, the code snippet training pair including the user code snippet and the recommended code snippet,
characterized in that the method further comprises:
detecting malicious user-determined code snippet feedback from a user; and
disallowing the storing of the code snippet training pair in the training database in response to detecting malicious user-determined code snippet feedback.