SIMPLIFYING CONVOLUTIONAL NEURAL NETWORKS USING AGGREGATED REPRESENTATIONS OF IMAGES

One embodiment of the present invention sets forth a technique for simplifying a trained machine learning model. The technique includes determining a first set of images associated with a first output class predicted by the trained machine learning model. The technique also includes generating a first aggregated representation of the first set of images, wherein the first aggregated representation includes a first plurality of representative pixel values for a plurality of pixel locations included in the first set of images. The technique further includes generating a simplified representation of the trained machine learning model that includes a first mapping of the first aggregated representation to the first output class, wherein the first mapping indicates that the trained machine learning model predicts the first output class for one or more input images.

BACKGROUND

Field of the Various Embodiments

The various embodiments relate generally to computer science and machine learning and, more specifically, to techniques for simplifying convolutional neural networks using aggregated representations of images.

Description of the Related Art

Non-quantized machine learning models are commonly trained to generate or predict classes, numeric values, images, audio, text, and/or various types of attributes. For example, non-quantized neural networks could use floating point numbers to represent inputs, weights, or activations to achieve a high-level of accuracy in the resulting computations. As non-quantized machine learning models grow in size and complexity, these models require increasing amounts of power, computational resources (e.g., storage, working memory, cache, and processor speed), network bandwidth (e.g., for transferring a machine learning model to a device or updating a machine learning model), and/or latency to execute. These requirements limit the ability to use the machine learning models in devices or environments with limited memory, power, network bandwidth, and/or computational capabilities.

To address the above limitations, various compression techniques have been developed to enable machine learning models to be used with a wider range of devices and hardware platforms. For example, a neural network can be modified or quantized to use lower precision numbers (e.g., integers) when performing various computations. As a general matter, a quantized neural network is less resource intensive and incurs less latency than a corresponding non-quantized neural network. Accordingly, the quantized neural network typically requires less memory, power, network bandwidth, and computational resources than the corresponding non-quantized neural network.

One drawback of conventional techniques for compressing machine learning models is the tradeoff between the amount a given machine learning model is compressed and the accuracy of the resulting compressed model. In this regard, compressing a machine learning model a certain amount can reduce the resource overhead and latency associated with the machine learning model without materially decreasing the accuracy of the machine learning model. However, when a machine learning model is compressed too much, the accuracy of the machine learning model can become adversely impacted, thereby limiting the usefulness of the compressed model.

Another drawback of conventional techniques for compressing machine learning models is that the size and complexity of a compressed machine learning model is a function of the size and complexity of the corresponding uncompressed machine learning model. For example, the level of compression applied to a given neural network could be expressed as a compression ratio between the size of the uncompressed version of that neural network and the size of the compressed version of the neural network. Accordingly, a compressed machine learning model can still be too large or complex to run on a device or environment with limited memory, power, network bandwidth, and/or computational capabilities.

As the foregoing illustrates, what is needed in the art are more effective techniques for compressing machine learning models.

SUMMARY

One embodiment of the present invention sets forth a technique for simplifying a trained machine learning model. The technique includes determining a first set of images associated with a first output class predicted by the trained machine learning model. The technique also includes generating a first aggregated representation of the first set of images, where the first aggregated representation includes a first plurality of representative pixel values for a plurality of pixel locations included in the first set of images. The technique further includes generating a simplified representation of the trained machine learning model that includes a first mapping of the first aggregated representation to the first output class, wherein the first mapping indicates that the trained machine learning model predicts the first output class for one or more input images.

One technical advantage of the disclosed techniques relative to the prior art is that, with the disclosed techniques, a simplified representation of a trained machine learning model is generated that includes a mapping between a compact representation of a set of images and an output class. During operation, the mapping is used instead of the trained machine learning model to predict the output class for certain types of input images. With this approach, the size and complexity of the simplified representation of the trained machine learning model is independent of the size and complexity of the actual trained machine learning model, which allows the trained machine learning model to be compressed beyond the point at which conventional compression techniques cause the accuracy of the compressed model to be adversely impacted. Another technical advantage of the disclosed techniques is the ability to perform machine learning inference operations via efficient “lookup” operations using the mapping based on pixel values in an input image. Consequently, machine learning inference operations performed based on the disclosed techniques can be faster and incur less resource overhead relative to conventional approaches that require inference operations to be performed by compressed machine learning models that are proportional in size and complexity to their corresponding uncompressed machine learning models. These technical advantages provide one or more technological improvements over prior art approaches.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one of skill in the art that the inventive concepts may be practiced without one or more of these specific details.

System Overview

FIG.1illustrates a computing device100configured to implement one or more aspects of the present invention. Computing device100includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), tablet computer, server computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments of the present invention. Computing device100is configured to run a processing engine122and an inference engine126that reside in a memory116.

It is noted that computing device100described herein is illustrative and that any other technically feasible configurations fall within the scope of the present invention. For example, multiple instances of processing engine122and inference engine126could execute on a set of nodes in a data center, cluster, or cloud computing environment to implement the functionality of computing device100. In another example, processing engine122and inference engine126could be implemented together and/or separately using one or more hardware and/or software components or layers.

In one embodiment, computing device100includes, without limitation, an interconnect (bus)112that connects one or more processors102, an input/output (I/O) device interface104coupled to one or more input/output (I/O) devices108, memory116, a storage114, and a network interface106. Processor(s)102may be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (Al) accelerator, any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in conjunction with a GPU. In general, processor(s)102may be any technically feasible hardware unit capable of processing data and/or executing software applications. Further, in the context of this disclosure, the computing elements shown in computing device100may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executing within a computing cloud.

In one embodiment, I/O devices108include devices capable of receiving input, such as a keyboard, a mouse, a touchpad, and/or a microphone, as well as devices capable of providing output, such as a display device and/or speaker. Additionally, I/O devices108may include devices capable of both receiving input and providing output, such as a touchscreen, a universal serial bus (USB) port, and so forth. I/O devices108may be configured to receive various types of input from an end-user (e.g., a designer) of computing device100, and to also provide various types of output to the end-user of computing device100, such as displayed digital images or digital videos or text. In some embodiments, one or more of I/O devices108are configured to couple computing device100to a network110.

In one embodiment, network110is any technically feasible type of communications network that allows data to be exchanged between computing device100and external entities or devices, such as a web server or another networked computing device. For example, network110could include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and/or the Internet, among others.

In one embodiment, storage114includes non-volatile storage for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-Ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. Processing engine122and inference engine126may be stored in storage114and loaded into memory116when executed.

In one embodiment, memory116includes a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or combination thereof. Processor(s)102, I/O device interface104, and network interface106are configured to read data from and write data to memory116. Memory116includes various software programs that can be executed by processor(s)102and application data associated with said software programs, including processing engine122and inference engine126.

Processing engine122and inference engine126include functionality to simplify a convolutional neural network (CNN) and/or another type of trained machine learning model that predicts classes associated with images. As described in further detail below, processing engine122generates one or more simplified representations of the machine learning model based on predictions of classes outputted by the trained machine learning model from various images. Each simplified representation includes one or more mappings corresponding to one or more output classes predicted by the machine learning model. Each mapping includes a compact representation of a set of images and a common class predicted by the machine learning model from the set of images.

Inference engine126uses the simplified representation(s) of the trained machine learning model to generate predictions for additional images. More specifically, inference engine126compares each image to the compact representations mapped to different classes within a given simplified representation of the machine learning model. When the comparison indicates that pixel values in the image are highly similar to or “fall within” the pixel values in a compact representation, inference engine126determines that the image is to be assigned the class to which the compact representation is mapped. As a result, the simplified representation(s) of the machine learning model can be used to perform inference related to the machine learning model without executing the machine learning model, thereby reducing resource overhead associated with performing inference using the machine learning model without significantly impacting the accuracy of the machine learning model.

Simplifying Convolutional Neural Networks

FIG.2is a more detailed illustration of processing engine122and inference engine126ofFIG.1, according to various embodiments. As mentioned above, processing engine122is configured to generate a simplified representation204of a trained machine learning model208, and inference engine126is configured to use the simplified representation to perform inference related to machine learning model208. Each of these components is described in further detail below.

Machine learning model208includes a number of learnable parameters and an architecture that specifies an arrangement, a set of relationships, and/or a set of computations related to the parameters. For example, machine learning model208could include one or more recurrent neural networks (RNNs), convolutional neural networks (CNNs), deep neural networks (DNNs), deep convolutional networks (DCNs), and/or other types of artificial neural networks or components of artificial neural networks. Machine learning model208could also, or instead, include a logistic regression model, support vector machine, decision tree, random forest, gradient boosted tree, naïve Bayes classifier, Bayesian network, hierarchical model, ensemble model, and/or another type of machine learning model that does not include artificial neural network components.

In one or more embodiments, machine learning model208is trained to generate predictions206of labels212assigned to images210in a training dataset202. For example, training dataset202could include images210of 10 handwritten digits ranging from 0 to 9, as well as labels212that identify one of the 10 digits to which each of the corresponding images210belongs. During training of machine learning model208, a training technique such as stochastic gradient descent and backpropagation could be used to update weights of a CNN corresponding to machine learning model208in a way that reduces errors between predictions206generated by the CNN from inputted images210and the corresponding labels212.

After training of machine learning model208is complete, the trained machine learning model208can be used to generate additional predictions206of classes represented by labels212for images that are not in training dataset202. Continuing with the above example, the trained machine learning model208could be applied to an input image to generate a set of 10 confidence scores for 10 classes representing 10 different handwritten digits. Each confidence score could range from 0 to 1 and represent a probability or another measure of certainty that the input image belongs to a certain class (i.e., that the input image is of a certain handwritten digit), and all confidence scores could sum to 1. When a confidence score outputted by machine learning model208for the input image exceeds a threshold, the input image could be determined to be from the corresponding class.

As shown inFIG.2, processing engine122generates a simplified representation204of the trained machine learning model208based on predictions206generated by machine learning model from images210in training dataset202. During the generation of simplified representation204, processing engine122identifies a set of representative images214in training dataset202for each class predicted by machine learning model208.

In one or more embodiments, representative images214include images210in training dataset202that are “typical” or unambiguous examples of classes or categories represented by the corresponding labels212. For example, representative images214assigned to a label representing a specific handwritten digit could include images210in training dataset202that are associated with high confidence scores outputted by machine learning model208for that handwritten digit. Processing engine122could identify these representative images214by applying one or more thresholds to confidence scores generated machine learning model208for images210assigned to the label. The thresholds could include (but are not limited to) a minimum threshold (e.g., 0.8, 0.9, 0.95, etc.) for a confidence score associated with the handwritten digit and/or a maximum threshold (e.g., 0.1, 0.05, etc.) for confidence scores for all other handwritten digits. Processing engine122could also use these thresholds to identify additional sets of representative images214for other labels212in training dataset202. As a result, processing engine122could generate 10 sets of representative images214for 10 different handwritten digits ranging from 0 to 9.

In some embodiments, representative images214include images that are not found in training dataset202. Continuing with the above example, representative images214for a given class could include additional images for which the trained machine learning model208generates confidence scores that meet the minimum and/or maximum thresholds. These additional images could also, or instead, be validated by one or more humans as belonging to the class before the additional images are added to the set of representative images214for the class.

Processing engine122also generates compact representations220(1)-220(N) of representative images214for different classes222(1)-222(N) represented by labels212in training dataset202. Each of compact representations220(1)-220(N) is referred to individually as compact representation220, and each of classes222(1)-222(N) is referred to individually as class222. A given compact representation220indicates a set of valid pixel values for a corresponding set of representative images214. For example, a given compact representation220could include a statistical aggregation of pixel values in representative images214for a corresponding class, as described in further detail below with respect toFIG.3. A given compact representation220could also, or instead, include a logical representation of pixel values in representative images214for a corresponding class, as described in further detail below with respect toFIG.4.

Processing engine122can also generate multiple compact representations220of representative images214for each class222. For example, processing engine122could divide a set of representative images214for a given class222into multiple subsets of representative images214for the same class222. This division could be performed by clustering representative images214by visual similarity and/or other visual attributes. Processing engine122could then generate a separate aggregated representation and/or a logical representation of each subset of representative images214.

Processing engine122populates simplified representation204with mappings of compact representations220to the corresponding classes222. Each mapping indicates that machine learning model208predicts a certain class222for a set of images from which a corresponding compact representation220was generated. For example, processing engine122could store a mapping of each compact representation220to a corresponding class222in a lookup table, database, file, key-value store, and/or another type of data store or structure corresponding to simplified representation204.

Inference engine126uses simplified representation204to perform inference related to machine learning model208for a new image240. For example, inference engine126could execute within an online, offline, nearline, streaming, search-based, and/or another type of environment to generate a prediction246of a class to which image240belongs based on simplified representation204.

More specifically, inference engine126performs comparisons and/or evaluations involving pixel values in image240and compact representations220of pixel values in simplified representation204. Inference engine126uses the results of these comparisons and/or evaluations to generate a compact representation match242for image240. Compact representation match242includes one or more compact representations220that are “closest” to the pixel values in image240and/or one or more compact representations220under which the pixel values in image240fall.

Inference engine126then generates prediction246based on compact representation match242. For example, inference engine126could retrieve one or more classes222to which one or more compact representations220in compact representation match242are mapped within simplified representation204. If compact representations220in compact representation match242are all mapped to the same class222, inference engine126could generate prediction246of that class222for image240. If compact representations220in compact representation match242are mapped to more than one class222, inference engine126could generate prediction246to include a single class222to which the majority of compact representations220in compact representation match242are mapped. Inference engine126could also, or instead, use a set of rules, a weighted combination associated with the output classes222to which compact representations220in compact representation match242are mapped, a formula, and/or another technique to select a single class222that is “closest” or “most relevant” to image240. When compact representations220in compact representation match242are mapped to multiple classes222and/or when image240cannot be matched to any compact representations220in simplified representation204, inference engine126could use machine learning model208and/or human input to generate prediction246for image240. Inference engine126could also, or instead, generate prediction246to indicate that image240is not a member of any output classes222associated with labels212.

Processing engine122can also update machine learning model208and/or simplified representation204based on additional labels212and/or human input related to image240and/or prediction246. For example, processing engine122and/or inference engine126could receive input from one or more users confirming the accuracy of prediction246and/or specifying a different class for image240. Processing engine122could add image240and the corresponding class to a record in training dataset202and retrain machine learning model208using the record. Processing engine122could also, or instead, update compact representations220and/or classes222to which compact representations220are mapped based on predictions206outputted by the retrained machine learning model208and/or images210and labels212that have been added to training dataset202. Consequently, the accuracy of machine learning model208and/or simplified representation204improve as predictions of classes222for images (e.g., image240) are generated using simplified representation204and validated.

FIG.3illustrates the use of an aggregated representation304of a set of images302(1)-302(X) to predict a class322associated with an input image240, according to various embodiments. Each of images302(1)-302(X) is referred to individually as image302. Image302(1) includes a set of pixel values310(1)-310(Y), each of which is referred to individually as pixel value310. Image302(X) includes a different set of pixel values312(1)-312(Y), each of which is referred to individually as pixel value312. Pixel values310-312include RGB values, intensity values, HSV values, and/or other representations of color, brightness, and/or other types of visual attributes for various pixel locations in images302.

In some embodiments, images302include the same number of pixel values310and312(i.e., Y pixel values, where Y is an integer greater than or equal to 0). For example, images302could have the same image size (e.g., height and width). If images302differ in image size, one or more images302could be cropped, scaled, or otherwise resized to have the same size as other images302. One or more images302could also, or instead, be recentered, rotated, and/or otherwise transformed to standardize the locations, positions, and/or orientations of objects (e.g., handwritten digits, faces, etc.) within images302. This standardization of image sizes and objects across images302allows pixel values310-312representing the same pixel locations to be compared or processed across images302. For example, pixel values310or312in each image302could be stored in an array or matrix, with each element in the array or matrix corresponding to a pixel location in the image. Because pixel values310-312from the same array or matrix element in images302are from the same pixel locations in images302, pixel values310-312from the same array or matrix indexes can be directly compared or processed across images302to characterize similarities or differences in images302.

In one or more embodiments, images302are included in a set of representative images214for a given label in training dataset202. For example, images302could be associated with predictions206of the same class322by machine learning model208. These predictions206could additionally be associated with high confidence scores outputted by machine learning model208for class322, labels212of class322for images302, human validation of class322for images302, and/or other indicators of high likelihood that images302belong to class322.

As mentioned above, processing engine122generates compact representations220of a set of representative images214and uses compact representations220to produce simplified representation204of machine learning model208. In some embodiments, these compact representations220include aggregated representation304of a set of images302from the same class322. For example, processing engine122could generate a single aggregated representation304from multiple images302for which machine learning model208generates high confidence scores for a given class322.

To generate aggregated representation304, processing engine122combines pixel values310-312from the same pixel locations in images302into representative pixel values306(1)-306(Y) (each of which is referred to individually as representative pixel value306) for these pixel locations. For example, processing engine122could compute each representative pixel value306as a mean, median, set of percentiles, standard deviation, mode, minimum, maximum, histogram, and/or another set of summary statistics for multiple pixel values310-312at the same pixel location within images302. Representative pixel values306in aggregated representation304could thus characterize the ranges or distributions of pixel values310-312in the corresponding pixel locations within images302.

As described above, processing engine122populates simplified representation204with a mapping of aggregated representation304to class322to which images302belong. Within simplified representation204, the mapping indicates that machine learning model208generates a prediction of class322for images that can be matched to aggregated representation304. Consequently, the mapping serves as a proxy for the operation of machine learning model208in predicting class322for certain images.

Inference engine126uses the mapping of aggregated representation304to class322in simplified representation204to generate a prediction of class322for a new image240that is not in training dataset202. More specifically, inference engine126computes a similarity320between image240and aggregated representation304. For example, inference engine126could compute similarity320as a cosine similarity, Euclidean distance, edit distance, dot product, and/or another measure of vector similarity or distance between pixel values308(1)-308(Y) (each of which is referred to individually as pixel value308) in image240and the corresponding representative pixel values306(1)-306(Y) in aggregated representation304. In another example, inference engine126could use a function to convert pixel values308in image240into a first embedding, hash value, or other lower-dimensional representation and use the same function to convert representative pixel values306in aggregated representation304into a second lower-dimensional representation. Inference engine126could then compute similarity320as a cosine similarity, Euclidean distance, edit distance, dot product, and/or another measure of vector similarity or distance between the two lower-dimensional representations. In a third example, inference engine126could use a distribution of pixel values denoted by one or more summary statistics in each representative pixel value306in aggregated representation304to estimate a z-score representing the number of standard deviations between the mean pixel value in the distribution and a corresponding pixel value308in image240. Inference engine126could then calculate similarity320as an average and/or another aggregation of z-scores for all pixel values308in image240.

Inference engine126compares similarity320to other measures of similarity (not shown) between image240and other aggregated representations (not shown) in simplified representation204. For example, inference engine126could calculate and/or aggregate one or more vector similarities and/or z-scores between image240and each aggregated representation in simplified representation204. Inference engine126could also compare the vector similarities and/or z-scores across the aggregated representations in simplified representation204. When similarity320between image240and aggregated representation304is greater than the other measures of similarity between image240and other aggregated representations in simplified representation204(e.g., when the distance between pixel values308in image240and representative pixel values306in aggregated representation304is lower than the distances between pixel values308and other representative pixel values in the other aggregated representations) and/or exceeds a threshold for minimum similarity with aggregated representations in simplified representation204, inference engine126determines that image240belongs to class322to which aggregated representation304is mapped. If no measures of similarity between image240and aggregated representations in simplified representation204meet the threshold for minimum similarity, inference engine126can compare image240to logical representations of representative images214in simplified representation204to predict a class for image240, as described in further detail below with respect toFIG.4. Inference engine126can also, or instead, use machine learning model208to predict a class for image240.

FIG.4illustrates the use of a logical representation404of a set of images402(1)-402(Z) to predict a class422associated with an input image, according to various embodiments. Each of images402(1)-402(Z) is referred to individually as image402. Image402(1) includes a set of pixel values410(1)-410(A), each of which is referred to individually as pixel value410. Image402(Z) includes a different set of pixel values412(1)-412(A), each of which is referred individually to as pixel value412. As with pixel values310-312ofFIG.3, pixel values410-412include RGB values, intensity values, HSV values, and/or other representations of color, brightness, and/or other types of visual attributes for various pixel locations in images402.

In one or more embodiments, images402ofFIG.4include the same number of pixel values410and412(i.e., A pixel values, where A is an integer greater than or equal to 0). For example, images402could have the same image size (e.g., height and width). If images402differ in image size, one or more images402could be cropped, scaled, or otherwise resized to have the same size as other images402. One or more images402could also, or instead, be recentered, rotated, and/or otherwise transformed to standardize the locations, positions, and/or orientations of objects (e.g., handwritten digits, faces, etc.) within images402. This standardization of image sizes and objects across images402allows pixel values410-412representing the same pixel locations to be compared or processed across images402. For example, pixel values410or412in each image402could be stored in an array or matrix, with each element in the array or matrix corresponding to a pixel location in the image. Because pixel values410-412from the same array or matrix element in images402are from the same pixel locations in images402, pixel values410-412from the same array or matrix indexes can be directly compared or processed across images402to characterize similarities or differences in images402.

As with images302ofFIG.3, images402can be included in a set of representative images214for a given label in training dataset202. For example, images402could be associated with predictions206of the same class322by machine learning model208. These predictions206could additionally be associated with high confidence scores outputted by machine learning model208for class422, labels212of class422for images402, human validation of class422for images402, and/or other indicators of high likelihood that images402belong to class422.

As mentioned above, compact representations220of a set of representative images214can include logical representation404of images402from the same class422. For example, processing engine122could generate a single logical representation404from multiple representative images402for a given class422.

As shown inFIG.4, logical representation404includes representations of a set of pixel values406(1)-406(B) (where B is an integer greater than or equal to 0), each of which is referred to individually as pixel value406. Pixel values406can include one or more pixel values for each image402and each of A pixel locations in images402. For example, pixel values406could include a numeric pixel value ranging from 0 to 256 for a given pixel location and a red, green, blue, brightness, and/or another channel in each image402. In another example, pixel values406could include a binary pixel value of 0 or 1 for each pixel location in images402. Pixel values406can also, or instead, reflect pixel value ranges, minimum or maximum pixel value thresholds, and/or other representations of “valid” pixel values406for each pixel location in images402. For example, logical representation404could include an upper and/or lower limit on pixel values406within a given image402. In another example, logical representation404could include a complex expression that returns true or false, given a pixel value for a certain pixel location.

Logical representation404also includes a set of logical operators414(1)-414(C), each of which is referred to individually as logical operator414. Each logical operator414is applied to one or more pixel values406to characterize the set of valid pixel values410-412in images402.

In some embodiments, processing engine122initially represents pixel values410-412in images402as a disjunctive normal form (DNF) that includes an “OR of ANDs” within logical representation404. Within logical representation 404, each set of pixel values410or412within a particular image402is represented as a set of pixel values406connected by logical operators414corresponding to logical conjunctions (i.e., ANDs). Multiple images402are represented within logical representation404by connecting the corresponding sets of pixel values406by additional logical operators414corresponding to logical disjunctions (i.e., ORs). An example logical representation404for images402that are three pixels high by three pixels wide includes the following:{P(0,0)=0 AND P(0,1)=1 AND P(0,2)=0 AND P(1,0)=0 AND P(1,1)=1 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0} OR{P(0,0)=1 AND P(0,1)=0 AND P(0,2)=0 AND P(1,0)=1 AND P(1,1)=0 AND P(1,2)=0 AND P(2,0)=1 AND P(2,1)=0 AND P(2,2)=0} OR

In the example logical representation404above, each pixel is denoted by P(row, col), where “row” is an index into the row in which the pixel is located and “col” is an index into the column in which the pixel is located. Each pixel is also assigned a binary pixel value of 0 or 1. This binary pixel value can be determined by assigning a threshold (e.g., 128, 250, etc.) to original pixel values410-412(e.g., eight-bit pixel values410-412ranging from 0 to 256) in images402. A single image402is represented by a set of nine logical expressions that specify valid pixel values for nine different pixel locations within the image. These nine logical expressions are connected by a set of eight AND operators. Multiple images402are represented by OR operators between sets of nine logical representations denoting valid pixel values from different images402. The number of OR operators is one less than the number of images402.

In one or more embodiments, processing engine122compresses the initial DNF in logical representation404to reduce the complexity and/or overhead associated with storing or evaluating logical representation404. For example, processing engine122could use a set of Boolean algebra rules, a Karnaugh map, a truth graph, a logic optimization technique, a truth table reduction technique, and/or another Boolean expression reduction technique to convert pixel values406and/or logical operators414in logical representation404into a simpler form.

Continuing with the above example of 3×3 binary pixel images, processing engine122can generate the following initial logical representation404for images402that belong to a class representing vertical lines:{P(0,0)=1 AND P(0,1)=0 AND P(0,2)=0 AND P(1,0)=1 AND P(1,1)=0 AND P(1,2)=0 AND P(2,0)=1 AND P(2,1)=0 AND P(2,2)=0} OR{P(0,0)=1 AND P(0,1)=0 AND P(0,2)=0 AND P(1,0)=1 AND P(1,1)=0 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0} OR{P(0,0)=0 AND P(0,1)=1 AND P(0,2)=0 AND P(1,0)=1 AND P(1,1)=0 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0} OR{P(0,0)=0 AND P(0,1)=1 AND P(0,2)=0 AND P(1,0)=0 AND P(1,1)=1 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0} OR{P(0,0)=1 AND P(0,1)=0 AND P(0,2)=0 AND P(1,0)=0 AND P(1,1)=1 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0} OR{P(0,0)=0 AND P(0,1)=0 AND P(0,2)=1 AND P(1,0)=0 AND P(1,1)=1 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0} OR{P(0,0)=0 AND P(0,1)=1 AND P(0,2)=0 AND P(1,0)=0 AND P(1,1)=1 AND P(1,2)=0 AND P(2,0)=1 AND P(2,1)=0 AND P(2,2)=0} OR{P(0,0)=0 AND P(0,1)=1 AND P(0,2)=0 AND P(1,0)=0 AND P(1,1)=1 AND P(1,2)=0 AND P(2,0)=0 AND P(2,1)=0 AND P(2,2)=1} OR{P(0,0)=0 AND P(0,1)=0 AND P(0,2)=1 AND P(1,0)=0 AND P(1,1)=0 AND P(1,2)=1 AND P(2,0)=0 AND P(2,1)=0 AND P(2,2)=1} OR{P(0,0)=0 AND P(0,1)=1 AND P(0,2)=0 AND P(1,0)=0 AND P(1,1)=0 AND P(1,2)=1 AND P(2,0)=0 AND P(2,1)=0 AND P(2,2)=1} OR{P(0,0)=0 AND P(0,1)=0 AND P(0,2)=1 AND P(1,0)=0 AND P(1,1)=0 AND P(1,2)=1 AND P(2,0)=0 AND P(2,1)=1 AND P(2,2)=0}

The example logical representation404above includes 11 sets of nine logical expressions, where each logical expression specifies a valid pixel value406for a corresponding pixel location within an image. The nine logical expressions within each set are connected by logical conjunctions, and different sets of nine logical expressions are connected by logical disjunctions. Each set of nine logical expressions represents a different image of a vertical line. The first three sets of nine logical expressions represent three images402in which a vertical line is located substantially on the left side of each image402. The next five sets of nine logical expressions represent five images402in which a vertical line is located substantially in the middle of each image402. The last three sets of nine logical expressions represent three images402in which a vertical line is located substantially on the right side of each image402. As a result, the example logical representation404represents all possible combinations of pixel values406that correspond to images402of vertical lines.

The example simplified logical representation404above includes three logical expressions in brackets that are separated by logical disjunctions. The first logical expression represents the three images402in which a vertical line is located substantially on the left side of the image. The second expression represents the five images402in which a vertical line is located substantially in the center of the image. The third expression represents the three images402in which a vertical line is located substantially on the right side of the image.

After logical representation404is created and simplified, processing engine122stores a mapping of logical representation404to the corresponding class422in simplified representation204. For example, processing engine122could store a mapping of the simplified logical representation404of images402of vertical lines above to a given class422representing images402of vertical lines within a key-value store, database, file, and/or another data structure or data store corresponding to simplified representation204.

Inference engine126uses the mapping of logical representation404to class422in simplified representation204to generate a prediction of class422for a new image240that is not in training dataset202. More specifically, inference engine126performs an evaluation420of logical representation404using pixel values408(1)-408(A) (each of which is referred to individually as pixel value408) in image240. For example, inference engine126could evaluate logical expressions in logical representation404using pixel values408in image240. When a given pixel value408for a pixel location corresponds to a valid pixel value406for the same pixel location in logical representation404, inference engine126could determine that the logical expression representing the valid pixel value406in logical representation404evaluates to true. Conversely, when a given pixel value408does not correspond to a valid pixel value406for the same pixel location in logical representation404, inference engine126could determine that the logical expression evaluates to false. Inference engine126could then use logical operators414in logical representation404to combine true and/or false values resulting from evaluation of logical expressions for valid pixel values406to determine an overall value of true or false for the entire logical representation404.

When logical representation404evaluates to true given pixel values408in image240, inference engine126determines that image240belongs to class422to which logical representation404is mapped. When a different logical representation (not shown) mapped to another class (not shown) evaluates to true given pixel values408in image240, inference engine126determines that image240belongs to the other class. If no logical representations404within simplified representation204evaluate to true given pixel values408in image240, inference engine126can compare image240to aggregated representations of representative images214in simplified representation204to predict a class for image240, as discussed above with respect toFIG.4. Inference engine126can also, or instead, use machine learning model208to predict a class for image240and/or indicate that image240does not belong to any class associated with an aggregated representation and/or a logical representation.

FIG.5sets forth a flow diagram of method steps for simplifying a machine learning model, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-2, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, processing engine122determines502one or more sets of images associated with an output class predicted by a trained machine learning model. For example, the trained machine learning model could include a CNN that predicts classes representing handwritten digits, animals, landmarks, faces, machines, and/or other types of objects in images. Processing engine122could identify an initial set of images that are labeled with the output class in a training dataset and/or that are associated with predictions of the output class by the CNN. For each of these images, processing engine122could filter the initial set of images by a minimum threshold for a first confidence score outputted by the trained machine learning model for the output class and/or a maximum threshold for other confidence scores outputted by the trained machine learning model for other output classes. If an image meets all the thresholds and/or other criteria indicating a high likelihood that the image belongs to a certain output class, processing engine122could include the image in a set of “typical” images for the output class. After a set of “typical” images is populated with images that meet the threshold(s) and/or criteria for the output class, processing engine122could optionally cluster the images in the set by visual similarity and/or other criteria to divide the images into multiple sets of visually similar and/or visually related images.

Next, processing engine122generates504an aggregated representation and/or a logical representation of each set of images. For example, processing engine122could aggregate pixel values for the same pixel location in each set of images into a representative pixel value for that pixel location. The representative pixel value could include a mean, median, mode, minimum, maximum, percentile, standard deviation, histogram, and/or another set of summary statistics related to the distribution of pixel values in the set of images for the pixel location. Processing engine122could store the representative pixel values for a set of pixel locations shared by the set of images in an aggregated representation of the set of images. In another example, processing engine122could generate a logical representation of pixel values across the set of images. The logical representation could include conjunctions between pixel values in the same image and disjunctions between sets of pixels from different images. In other words, the logical representation would represent all combinations of pixel values that are present in the set of images. After the logical representation is generated, processing engine122could use a Boolean expression reduction technique to simplify the logical representation.

Processing engine122then adds506one or more mappings of the aggregated representation and/or the logical representation to the output class to a simplified representation of the trained machine learning model. For example, processing engine122could store a mapping of each aggregated representation and/or logical representation to the output class within a lookup table, database, and/or another data structure or data store corresponding to the simplified representation.

Processing engine122also determines508whether or not any output classes remain. For example, processing engine122could determine that output classes are remaining if mappings for these output classes have not been added to the simplified representation of the trained machine learning model. For each remaining output class, processing engine122repeats operations502-506to determine one or more sets of “typical” images for that output class, generate aggregated and/or logical representations of each set of images, and add mappings between the aggregated and/or logical representations to the output class to the simplified representation of the trained machine learning model. Processing engine122finishes creating the simplified representation after mappings between aggregated and/or logical representations of images to all output classes have been added to the simplified representation.

FIG.6sets forth a flow diagram of method steps for predicting a class associated with an image, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS.1-2, persons skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present disclosure.

As shown, inference engine126searches602a simplified representation of a trained machine learning model for a match between an input image and a compact representation of a set of images. In some embodiments, the compact representation includes an aggregated representation of a set of images. Inference engine126can thus perform operation602by attempting to match the input image to the aggregated representation based on measures of similarities between the image and multiple aggregated representations mapped to different output classes in the simplified representation.

Next, inference engine126determines604whether or not a match is found between the input image and a compact representation. For example, if the highest similarity between the input image and an aggregated representation in the simplified representation of the machine learning model exceeds a threshold for minimum similarity, inference engine could determine that the input image matches the aggregated representation. Conversely, inference engine126could determine that no match is found if the highest similarity between the input image and an aggregated representation in the simplified representation of the machine learning model does not meet the threshold and/or the image “matches” to multiple aggregated representations that are mapped to different output classes within the simplified representation of the machine learning model.

When inference engine126finds a match between the input image and a compact representation, inference engine126determines606an output class mapped to the compact representation within the simplified representation of the machine learning model. For example, inference engine126could retrieve the output class from a record in which the compact representation is stored and/or via a link from the compact representation within the simplified representation of the machine learning model. Inference engine126also generates608a prediction of the output class for the image. For example, inference engine126could store a mapping between the image and the output class, generate output indicating that the class belongs to the output class, and/or otherwise associate the input image with the output class.

When inference engine126cannot find a match between the input image and a compact representation, inference engine126searches610the simplified representation for a match between the input image and an alternative compact representation of a set of images. In some embodiments, the alternative compact representation corresponds to a logical representation of a set of images associated with a given output class. Inference engine126can thus perform operation610by evaluating each logical representation in the simplified representation using a set of pixel values in the input image.

Inference engine126determines604whether or not a match is found between the input image and an alternative compact representation. For example, if a logical representation evaluates to true given the pixel values in the input image, inference engine126could determine that the input image matches the logical representation. When no logical representations evaluate to true given the pixel values in the input image, inference engine126could determine that the input image does not match any alternative compact representations included in the simplified representation of the machine learning model.

When inference engine126finds a match between the input image and an alternative compact representation, inference engine126determines606an output class mapped to the alternative compact representation within the simplified representation of the machine learning model. For example, inference engine126could retrieve the output class from a record in which the alternative compact representation is stored and/or via a link from the alternative compact representation within the simplified representation of the machine learning model. Inference engine126also generates608a prediction of the output class for the image. For example, inference engine126could store a mapping between the image and the output class, generate output indicating that the class belongs to the output class, and/or otherwise associate the input image with the output class.

When inference engine126is unable to find a match between the input image and any type of compact representation (e.g., aggregated representation, logical representation, etc.) included in the simplified representation of the machine learning model, inference engine126generates614a prediction indicating that the input image is not a member of any output classes included in the simplified representation. For example, inference engine126could output a prediction that the input image falls into an “unknown” or “other” class that is not associated with a label predicted by the machine learning model.

Inference engine126can also, or instead, execute the trained machine learning model to generate a prediction of an output class for the input image. For example, inference engine126could apply a trained CNN for which the simplified representation was generated to the input image to generate a set of confidence scores for a set of output classes. Inference engine126could also apply one or more thresholds to the confidence scores. If the highest confidence score meets a minimum threshold and/or other confidence scores meet a maximum threshold, inference engine could determine that the CNN has predicted the output class associated with the highest confidence score for the input image.

In sum, the disclosed techniques generate simplified representations of CNNs and/or other trained machine learning models that predict classes to which images belong. A set of images that is “typical” of a given output class predicted by a machine learning model is determined by applying thresholds to confidence scores generated by the machine learning model from a set of images. An image is added to the set of “typical” images if a first confidence score outputted by the trained machine learning model for the output class meets a minimum threshold and/or other confidence scores outputted by the trained machine learning model for other output classes meet a maximum threshold.

A compact representation of the set of “typical” images is generated from pixel values in the images. The compact representation can include an aggregated representation of the pixel values. The aggregated representation includes one or more summary statistics for a distribution of pixel values for each pixel location within the set of images. The compact representation can also, or instead, include a logical representation of pixel values in the images. The logical representation includes conjunctions between pixel values in the same image and disjunctions between sets of pixel values in different images. The logical representation can also be simplified using a Boolean expression reduction technique. After the compact representation is generated, the compact representation is mapped to the output class within a simplified representation of the machine learning model. The process can be repeated for other output classes predicted by the machine learning model. As a result, the simplified representation of the machine learning model is populated with multiple mappings of compact representations of images to the corresponding output classes.

The simplified representation of the machine learning model can then be used to generate predictions of output classes for additional images. In particular, pixel values in the input image are used to match the input image to a compact representation in the simplified representation of the machine learning model. For example, measures of similarity between the input image and aggregated representations of images in the simplified representation could be computed, and the input image could be matched to the aggregated representation with the highest similarity to the pixel values in the input image. When the input image does not match any of the aggregated representations (e.g., when the highest similarity between the input image and an aggregated representation does not meet a minimum threshold), logical representations of images in the simplified representation of the machine learning model could be evaluated using the pixel values in the input image. When a logical representation evaluates to true, the input image could be matched to the logical representation.

After the input image is matched to a compact representation, the output class to which the compact representation is mapped is retrieved, and a prediction of the output class for the input image is generated. If the input image does not match any aggregated representations or logical representations, a prediction of an “unknown” or “other” class can be generated for the image.

One technical advantage of the disclosed techniques relative to the prior art is that, with the disclosed techniques, a simplified representation of a trained machine learning model is generated that includes a mapping between a compact representation of a set of images and an output class. During operation, the mapping is used instead of the trained machine learning model to predict the output class for certain types of input images. With this approach, the size and complexity of the simplified representation of the trained machine learning model is independent of the size and complexity of the actual trained machine learning model, which allows the trained machine learning model to be compressed beyond the point at which conventional compression techniques cause the accuracy of the compressed model to be adversely impacted. Another technical advantage of the disclosed techniques is the ability to perform machine learning inference operations via efficient “lookup” operations using the mapping based on pixel values in an input image. Consequently, machine learning inference operations performed based on the disclosed techniques can be faster and incur less resource overhead relative to conventional approaches that require inference operations to be performed by compressed machine learning models that are proportional in size and complexity to their corresponding uncompressed machine learning models. These technical advantages provide one or more technological improvements over prior art approaches.

1. In some embodiments, a computer-implemented method for simplifying a trained machine learning model comprises determining a first set of images associated with a first output class predicted by the trained machine learning model; generating a first aggregated representation of the first set of images, wherein the first aggregated representation comprises a first plurality of representative pixel values for a plurality of pixel locations included in the first set of images; and generating a simplified representation of the trained machine learning model that includes a first mapping of the first aggregated representation to the first output class, wherein the first mapping indicates that the trained machine learning model predicts the first output class for one or more input images.

2. The computer-implemented method of clause 1, further comprising storing, within the simplified representation of the trained machine learning model, a second mapping of a second aggregated representation of a second set of images to a second output class predicted by the trained machine learning model.

3. The computer-implemented method of any of clauses 1-2, further comprising matching an input image to the first mapping based on a plurality of pixel values included in the input image and the first plurality of representative pixel values in the first aggregated representation; and generating a prediction of the first output class for the input image based on the first mapping.

4. The computer-implemented method of any of clauses 1-3, wherein matching the input image to the first mapping comprises determining that a first similarity between the plurality of pixel values and the first plurality of representative pixel values is higher than a second similarity between the plurality of pixel values and a second plurality of representative pixel values associated with a second output class predicted by the trained machine learning model.

5. The computer-implemented method of any of clauses 1-4, wherein matching the input image to the first mapping comprises computing a vector similarity based on the plurality of pixel values included in the input image and the first plurality of representative pixel values included in the first aggregated representation.

6. The computer-implemented method of any of clauses 1-5, wherein matching the input image to the first mapping comprises computing a deviation of each pixel value included in the plurality of pixel values from a corresponding representative pixel value included in the first plurality of representative pixel values.

7. The computer-implemented method of any of clauses 1-6, wherein generating the first aggregated representation of the first set of images comprises populating a representative pixel value for each pixel location included in the plurality of pixel locations with one or more summary statistics associated with a set of pixel values for the pixel location from the first set of images.

8. The computer-implemented method of any of clauses 1-7, wherein determining the first set of images comprises filtering a second set of images included in a training dataset used to generate the trained machine learning model based on one or more thresholds for a set of confidence scores generated by the trained machine learning model from the second set of images.

9. The computer-implemented method of any of clauses 1-8, wherein the one or more thresholds comprise a minimum threshold for a first confidence score associated with the first output class and a maximum threshold for a second confidence score associated with a second output class.

10. The computer-implemented method of any of clauses 1-9, wherein the trained machine learning model comprises a trained convolutional neural network.

11. In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of determining a first set of images associated with a first output class predicted by a trained machine learning model; generating a first aggregated representation of the first set of images, wherein the first aggregated representation comprises a first plurality of representative pixel values for a plurality of pixel locations included in the first set of images; and generating a simplified representation of the trained machine learning model that includes a first mapping of the first aggregated representation to the first output class, wherein the first mapping indicates that the trained machine learning model predicts the first output class for one or more input images.

12. The one or more non-transitory computer-readable media of clause 11, wherein the instructions further cause the one or more processors to perform the step of storing, within the simplified representation of the trained machine learning model, a second mapping of a logical representation of a second set of images to a second output class predicted by the trained machine learning model, wherein the logical representation comprises one or more conjunctions of a first set of pixel values included in a first image and a disjunction of the first set of pixel values and a second set of pixel values included in a second image.

13. The one or more non-transitory computer-readable media of any of clauses 11-12, wherein the instructions further cause the one or more processors to perform the steps of matching a first input image to the first mapping based on a plurality of pixel values included in the first input image and the first plurality of representative pixel values included in the first aggregated representation; and generating a first prediction of the first output class for the first input image based on the first mapping.

14. The one or more non-transitory computer-readable media of any of clauses 11-13, wherein matching the first input image to the first mapping comprises determining that a first similarity between the plurality of pixel values and the first plurality of representative pixel values is higher than a second similarity between the plurality of pixel values and a second plurality of representative pixel values associated with a second output class predicted by the trained machine learning model.

15. The one or more non-transitory computer-readable media of any of clauses 11-14, wherein matching the first input image to the first mapping comprises determining that a first similarity between the plurality of pixel values and the first plurality of representative pixel values meets or exceeds a threshold.

16. The one or more non-transitory computer-readable media of any of clauses 11-15, wherein the instructions further cause the one or more processors to perform the steps of determining a lack of match between a second input image and the first mapping; and executing the trained machine learning model to generate a second prediction of a second output class for the second input image.

17. The one or more non-transitory computer-readable media of any of clauses 11-16, wherein determining the first set of images comprises filtering a second set of images included in a training dataset used to generate the trained machine learning model based on one or more thresholds for a set of confidence scores generated by the trained machine learning model from the second set of images.

18. The one or more non-transitory computer-readable media of any of clauses 11-17, wherein the first plurality of representative pixel values comprises a range of pixel values for each pixel location included in the first set of images.

19. The one or more non-transitory computer readable media of any of clauses 11-18, wherein the first plurality of representative pixel values comprises a plurality of summary statistics associated with a plurality of pixel values included in the first set of images.

20. In some embodiments, a system comprises one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to determine a first set of images associated with a first output class predicted by a trained machine learning model; generate a first aggregated representation of the first set of images, wherein the first aggregated representation comprises a first plurality of representative pixel values for a plurality of pixel locations in the first set of images; and generate a simplified representation of the trained machine learning model that includes a first mapping of the first aggregated representation to the first output class, wherein the first mapping indicates that the trained machine learning model predicts the first output class for one or more input images.