Filtering in trainable networks

Some embodiments perform, in a multi-layer neural network in a computing device, a convolution operation on input feature maps with multiple convolutional filters. The convolutional filters have multiple filter precisions. In other embodiments, electronic design automation (EDA) systems, methods, and computer-readable media are presented for adding such a multi-layer neural network into an integrated circuit (IC) design.

TECHNICAL FIELD

Embodiments described herein relate to trainable networks such as neural networks.

BACKGROUND

Trainable networks are used for many different applications such as image classification. However, existing trainable networks that have sufficient detection or recognition performance have impractical data and filter storage requirements, memory access bandwidth requirements, and computational complexity.

Alternative approaches to trainable networks that have sufficient detection or recognition performance include pruning of network connections, quantization of coefficients, and optimization of bit precisions of data and filters. However, such approaches have failed to reduce resource requirements such as computational complexity sufficiently.

DETAILED DESCRIPTION

Example embodiments described herein relate to methods, computer-readable media, cloud server systems, and devices used for neural networks that perform convolution. While certain example embodiments are discussed, it will be apparent that other embodiments not specifically described herein are attainable within the scope of the innovations presented herein. The following description and drawings illustrate specific embodiments to enable those skilled in the art to practice the example embodiments. Other embodiments incorporate specific structural, logical, electrical, process, and/or other changes. In further example embodiments, portions and/or features of some embodiments are included in, or substituted for, portions and/or features of other example embodiments. Elements of the example embodiments described herein cover all available equivalents of the described elements.

Some example embodiments perform, for example in a multi-layer neural network in a computing device, a convolution operation on input feature maps with multiple convolutional filters. The convolutional filters have a plurality of filter precisions including lower and higher filter precisions. In some example embodiments, along with at least one of the filter precisions being lower, the associated precisions of multiply operations and/or addition operations are also lower than if all filter precisions were high. Example circuitry that performs multiply operations and/or addition operations accordingly may occupy less chip area and consumes less power.

FIG. 1is a block diagram of an example of a multi-layer neural network110. The multi-layer neural network110receives input data, for example an image input102, and provides a neural network output104. Example multi-layer neural networks include feed-forward neural networks such as convolutional neural networks, recurrent neural networks that can include components such as convolutional neural networks, spiking neural networks, and support vector machines.

An example application of the multi-layer neural network110is in an embedded device that is deployed in, for example, a handheld device, a wearable device, a virtual reality device, or an environment with real-time requirements such as a car, or the like. In other example embodiments, the multi-layer neural network110receives as input a depth image formed by ranging signals such as radar or lidar. It should, however, be noted the methods and systems described herein may be deployed in a variety of different applications and is not limited to processing images.

The multi-layer neural network110is shown to include a plurality of neural network layers. More particularly, the multi-layer neural network is shown to include a neural network layer1122to a layer N134. In an example embodiment, the layers122to134are arranged in groups including one or more layers. In the multi-layer neural network110, the layer1122to a layer I−1124are shown in a layer group120, a layer1125is shown in a layer group130, and a layer I+1 to a layer N are shown in a layer group131. The neural network layer I125has a plurality of filter precisions to perform filtering operations. In an example embodiment, each neural network layer performs a neural network function such as convolution130, subsampling, or the like. The layer1122receives the example image input102and the layer N134is shown to provide the example neural network output104. The layer1125receives the example input feature maps126from the layer I−1124, and provides the example output feature maps127to the layer I+1132.

FIG. 2is an example of a method to perform convolution operations with multiple precisions in a layer of a neural network, for example, the layer1125of the multi-layer neural network110. In the example embodiment, the method is shown to occur in the neural network layer1125where filters within the layer1125have a plurality of different precisions to perform convolution130on the input data. The input is input feature maps210, shown as M feature maps from X1to XM. The input feature maps210undergo convolution with convolutional filters220. The convolutional filters220include filters with multiple precisions. In the shown example, an array of convolutional filters220from F11to FMNare used in the convolution operation. The convolutional filters220from F11to FMNare sums of convolutional basis filters G11to GMNand convolutional residual filtersto. The output is output feature maps230, shown as N feature maps from Y1to YN. To form output feature map Y1, respective convolutional filters F11to FM1are used in convolutions with respective input feature maps from X1to XM. In a convolution of a convolutional filter and an input feature map, the convolutional filter is a sliding window matrix that overlies part of the input feature map. The values of the sliding window matrix and the underlying input feature map are termwise multiplied and added to result in one element of the output feature map. The sliding window matrix is slid over the underlying input feature map and the process repeated until one convolution is complete. The resulting M convolutions are summed. Similarly, to form output feature map YJ, respective convolutional filters FIJto FMJare used in convolutions with respective input feature maps from X1to XM, and the resulting M convolutions are summed.

In one example, the convolutions are performed with the multiply and accumulate circuitry of a processor222. The processor may be a digital signal processor, general processor, or other processor.

FIG. 3is an example of a more detailed method of convolution with multiple precisions in a layer of the multi-layer neural network110.FIG. 3is similar toFIG. 2exceptFIG. 3makes more explicit the separation of convolutional filters into different sets of convolutional filters having different filter precisions. In particular, the convolutional filters220ofFIG. 2from F11to FMNare separated into a set of convolutional basis filters320G11to GMNhaving higher precision and a set of convolutional residual filters322tohaving lower precision. Theconvolutional filters220from F11to FMNequal the sum of the set of convolutional basis filters320G11to GMNhaving higher precision and the set of convolutional residual filters322to, on a filter-by-filter basis such that FIJ≈=GIJ+. Each sum of filters is a sum of matrices.

Convolutions are performed with input feature maps210and a set of convolutional basis filters320G11to GMNhaving higher precision, and with input feature maps210and a set of convolutional residual filters322tohaving lower precision. Then the output feature maps of the convolutions with a plurality of different filter precisions undergo a sum330, resulting in output feature maps340.

To form output feature map Y1, a sum330is performed by adding the results of first and second convolutions. The first convolution uses the set of convolutional basis filters320G11to GM1having higher precision with respective input feature maps from X1to XM. The convolutional basis filter320is a sliding window matrix, as in in the previously described convolution process. The resulting M convolution outputs are summed.

The second convolution uses the set of convolutional residual filters322tohaving lower precision with respective input feature maps from X1to XM. The convolutional residual filter is a sliding window matrix that overlies part of the input feature map. The convolutional basis filter is a sliding window matrix, as in the previously described convolution process. The resulting M convolutions are summed. Results of the first convolution with the set of convolutional basis filters and the second convolution with the set of convolutional residual filters are summed in the convolution sum330to form output feature map Y1.

Forming output feature map YJis similar to forming output feature map Y1, except that respective convolutional basis filters G1Jto GMJare used in convolutions with respective input feature maps from X1to XM, and respective convolutional residual filterstoare used in convolutions with respective input feature maps from X1to XM. The basis filters G are formed by clustering the N columns of F into L clusters. By mapping each of the N columns of G to one ofL<=N centroids, only L unique column operations are needed to compute the convolution between G and X. As a result, only M×L operations are performed at a higher precision to generate the contribution from G to the output feature maps Y. Additionally, M×N operations are performed at a lower precision to generate the contribution from the residual filters {tilde over (F)} to the output feature maps Y. With appropriate selection of the number of clusters L and the precisions applied to the operations from G and {tilde over (F)}, quantization error and complexity are traded off, thereby achieving significant savings in complexity at limited degradation in performance.

In one embodiment, the first convolution uses 8-bit data and 8-bit coefficients in the convolutional filters. Given that the computational complexity of an 8-bit-by-8-bit multiply accumulate (MAC) is twice the complexity of an 8-bit-by-4-bit MAC, the reduction in complexity is:

where K is the spatial dimension of the K×K convolutional filters being quantized, P2is the number of pixels in each output P×P feature map, M is the number of input feature maps, L is the number of cluster filters, and N is the number of output feature maps. As can be seen in this example, a reduction in computational complexity is possible if the number of cluster filters L is sufficiently low (e.g., L<N/2) and if the residual filters {tilde over (F)} can be represented in lower precision without a significant impact to accuracy.

FIG. 4is an example of a method of making convolutional filters with multiple precisions. The input is convolutional filters410from F11to FMNwhich may have a single precision or multiple precisions. A convolutional filter component synthesis engine420separates the convolutional filters410into a set of convolutional basis filters320G11to GMNhaving higher precision and a set of convolutional residual filters322tohaving lower precision. The convolutional filters410from F11to FMNare well approximated by the sum of the set of convolutional basis filters320G11to GMNhaving higher precision and the set of convolutional residual filters322tohaving lower precision, on a filter-by-filter basis such that FIJ≈=GIJ+. Each sum of filters is a sum of matrices. The convolutional filter component synthesis engine420has one or more of the following engines: an averaging engine422, a matrix factorization engine424which performs algorithms such as singular-value decomposition, and another clustering engine426, which performs algorithms such as K-means clustering.

Some embodiments were used in experiments with a German Traffic Sign Recognition Benchmark (GTSRB), comparing performance of a homogenous quantization approach with 8-bit data and 8-bit coefficients against hybrid quantization where convolutional basis filters were selected as the centroids of a K-means clustering applied to three-dimensional convolutional filters of each layer. K-means cluster convolutional basis filters were implemented with 8-bit coefficients, and residual convolutional filters with 4 bits. The cluster convolutional basis filters were applied to 8-bit data. The residual convolutional filters were applied to 8-bit data in one experiment and 4-bit data in a second experiment. The results are summarized in the table below, which shows an improvement of up to 41% in computational complexity with no degradation in recognition performance for 8-bit input data. For 4-bit input data, the complexity is reduced by 62% with a small decrease in recognition performance.

FIG. 5is an example of a method of optimizing a multi-layer neural network by making a convolutional change with a plurality of convolutional filters. More particularly, the method500is shown to include a convolutional change with convolutional filters in a neural network layer510and a balance540between a complexity520and a correct classification rate by the neural network530. In an example embodiment, the convolutional change with convolutional filters in a neural network layer510has one or more examples such as a number of convolutional filters511, and a change in filter parameters such as a number of input feature maps512, a number of output feature maps513, a height dimension of convolutional filters514, and a width dimension of convolutional filters515. Such convolutional changes with convolutional filters in a neural network layer occur inFIG. 3, for example. In an example embodiment, the complexity520has one or more examples such as a number of multiply-accumulate operations521and a number of processor cycles522.

In the example method500, one or more convolutional changes with convolutional filters are performed in a neural network layer510. In view of the one or more changes, the balance540is determined between the complexity520and the correct classification rate by the neural network530. In an example of a convolutional change with convolutional filters in a neural network layer510that increases complexity, the correct classification rate by the neural network530increases. In another example of a convolutional change with convolutional filters in a neural network layer510that decreases complexity, the correct classification rate by the neural network530decreases.

In the example method500, the complexity520is increased until the correct classification rate by the neural network530is adequate. However, the complexity520may be decreased until the correct classification rate by the neural network530, which was already adequate, is decreased and yet remains adequate. Further, the complexity520may be both increased and decreased with a resulting net increase or net decrease in the correct classification rate by the neural network530. At operation550, optimization stops, responsive to adequate complexity and correct classification rate of the neural network.

FIG. 6is an example of a method600of optimizing a multi-layer neural network by making a connection change of a plurality of convolutional filters. More particularly, the method600is shown to include a connection change with convolutional filters in a neural network610and a balance540between the complexity520and a correct classification rate by the neural network530. In an example embodiment, the connection change of convolutional filters in a neural network layer610has one or more examples, such as a disconnection of convolutional filters611, a connection of convolutional filters612, and a replacement of convolutional filters613.

In the method600, one or more connection changes of convolutional filters are performed in the neural network610. In view of the one or more changes, the balance540is determined between the complexity520and the correct classification rate by the neural network530. In other respects, the method600may be similar to the method500ofFIG. 5.

FIG. 7is an example method of optimizing a multi-layer neural network by performing a convolutional change with a plurality of convolutional filters, and performing a connection change of a plurality of convolutional filters. More particularly, the method700is shown to include a convolutional change with convolutional filters in the neural network layer510, a connection change of convolutional filters in a neural network610, and the balance540between a complexity520and the correct classification rate by the neural network530.

In certain circumstances, one or more convolutional changes of convolutional filters are performed in the neural network layer510, and one or more connection changes of convolutional filters are performed in the neural network610. In view of the changes, the balance540is determined between the complexity520and the correct classification rate by the neural network530. In other respects, the method700is similar to the method500ofFIG. 5and the method600ofFIG. 6.

FIG. 8is a block diagram of an example of a multi-layer neural network with examples of connection changes of convolutional filters. More particularly, the multi-layer neural network includes a neural network layer I−1802, a neural network layer I805, and a neural network layer I+1808. An input to layer I804is communicated from neural network layer I−1802to a neural network layer I805. An output from layer I806is communicated from the neural network layer I805to a neural network layer I+1808.

Various example embodiments are implemented by connection changes in between, for example, the input to layer I804and a concatenator850. One example embodiment connects, in series, a network path with a convolution with 1×1 filters812and a convolution with 5×5 filters814. Another example embodiment connects a network path with, in series, a convolution with 1×1 filters822and a convolution with 3×3 filters824. Another example embodiment connects a network path with a convolution with 1×1 filters832. Another example embodiment connects a network path with a bypass842. Another example embodiment includes parallel network paths of connections of convolutions. It is to be appreciated that various combinations of filters of varying dimensions may be used in the example embodiments. Further, although the layer I805is shown to include three parallel paths and a single bypass842, other example embodiments may include more, or less, parallel paths. Different parallel path may include filters of varying dimensions.

In an example embodiment, a connection change to process data in the multi-layer neural network includes a connection of a convolution. In another example embodiment, a connection change to not process data in the multi-layer neural network includes a disconnection of a convolution. For example, in the case that a convolution does not connect the input to layer I804and a concatenator850, then in one example embodiment a bypass842connects the input to layer I804and the concatenator850. In another example embodiment, the connection change is a replacement of convolutional filters that includes disconnection of a first convolution and connection of a second convolution.

In the example embodiment shown inFIG. 8, the concatenator850sums two or more of the parallel connections, such as convolutions or the bypass842. The output feature maps of the concatenator850are provided to other layer operations860. Some example embodiments of the other layer operations860include one or more of pooling, rectified linear units, subsampling, and a fully connected layer.

InFIGS. 5-7, one example embodiment measures complexity as a number of multiply-and-accumulate operations (MACs). In an example embodiment, a convolutional neural network (CNN) has neural network layers 1, 2, 3, etc., with respective convolutional filter output dimensions, or equivalently a number of output feature maps such as N1, N2, N3, etc., that are generated by each neural network layer. In an example embodiment, the total number of MACs is expressed as a function of a vector formed by the number of outputs from each layer, such as x=[N1N2N3. . . ]. In one example embodiment the complexity, expressed as a number of MACs, is defined by a quadratic form:
MAC(x)=xtQx+ftx+C

Q, f and C are constants derived from calculating the number of MACs over an entire convolutional network. If x=[N1N2N3. . . ]. has L dimensions then Q is an L×L matrix. The elements of Q count the complexity of connected layers. For example, if layer J's output goes into layer K, and layer J has NJoutput feature maps while layer K has NKoutput feature maps, then QJK=QKJ=q/2 for some constant q indicates that layer K performs q NJNKMAC calculations. Vector f is an L×1 vector that counts the MACs related to the bias terms in a convolutional network and C are constant calculations that do not depend on the parameters in x being optimized.

InFIGS. 5-7, one example embodiment measures a correct classification rate (CCR) of the neural network. In one example embodiment, CCR data are collected with a matrix factorization approach. An orthonormal basis is computed for a matrix constructed from the coefficients of the convolutional filters in a layer of a CNN. Variations in the number of basis vectors utilized to reconstruct this matrix correspond to variations in the output dimensions for that layer via a linear transformation. With this approach, changes in the CCR are measured for a variety of dimensions of each neural network layer. A nonlinear multi-dimensional function of these output dimension perturbations is fitted to the CCR data-points. In an example embodiment, the CCR curve is modeled as an exponential of an n-th order multivariate polynomial:

CCR⁡(x)=ρ-exp(-∑ei·1<n[ei]k≥0⁢∝ei⁢xei),
where x=[N1N2N3]t, e1=[0 2 1]t, and xe1=N10N22N31], such that [ei]k≥0 indicates that all elements of the exponential are non-negative. The coefficients αeiare experimentally obtained during the fitting process.

FIG. 9is a diagram illustrating one possible design method for hybrid precision filtering in a machine learning system, according to some example embodiments. As illustrated, the overall design flow900includes a design phase910, a device fabrication phase920, a design simulation phase930, and a device verification phase940. The design phase910involves an initial design inputs operation901where the basic elements and functionality of a device are determined, as well as revisions based on various analyses and optimization of a circuit design. This design input operation901is where instances of an electronic design automation (EDA) circuit design file are used in the design and any additional circuitry is selected. The initial strategy, tactics, and context for the device to be created are also generated in the design input operation901, depending on the particular design algorithm to be used.

In some embodiments, the design phase910includes creation and/or access of the circuit design with hybrid precision filtering in a machine learning system.

After design inputs are used in the design input operation901to generate a circuit layout, and any optimization operations911are performed, a layout is generated in a layout instance912. The layout describes the physical layout dimensions of the device that match the design inputs. This layout may then be used in a device fabrication operation922to generate a device, or additional testing and design updates may be performed using designer inputs or automated updates based on design simulation932operations, and extraction, three-dimensional (3D) modeling, and analysis944operations.

Once the device is generated, the device can be tested as part of device test942operations and layout modifications generated based on actual device performance.

Design updates936from the design simulation932, design updates946from the device test942and extraction, 3D modeling, and analysis944operations, or the design inputs operation901may occur after the initial layout instance912is generated. In various embodiments, whenever design inputs are used to update or change an aspect of a circuit design, a timing analysis and optimization operation911may be performed.

FIG. 10is a block diagram1000illustrating an example of a software architecture1002operating on an EDA computer and used to add logic to implement any of the embodiments described herein, such as convolution with multiple precisions in a layer, optimization to include connection changes of convolutional filters in a neural network, and optimization to include convolutional changes.

FIG. 10is merely a non-limiting example of a software architecture1002, and it will be appreciated that many other architectures can be implemented to facilitate the functionality described herein. In various embodiments, the software architecture1002is implemented by hardware such as a machine1100that includes processors1110, memory1130, and input/output (I/O) components1150. In this example, the software architecture1002can be conceptualized as a stack of layers where each layer may provide a particular functionality. For example, the software architecture1002includes layers such as an operating system1004, libraries1006, software frameworks1008, and applications1010. Operationally, the applications1010invoke application programming interface (API) calls1012through the software stack and receive messages1014in response to the API calls1012, consistent with some embodiments. In various embodiments, any client device, server computer of a server system, or any other device described herein may operate using elements of the software architecture1002. An EDA computing device described herein may additionally be implemented using aspects of the software architecture1002, with the software architecture1002adapted for adding hybrid precision filtering in a machine learning system.

In one embodiment, an EDA application of the applications1010adds hybrid precision filtering in a machine learning system according to embodiments described herein using various modules such as design phase modules1036within the software architecture1002. For example, in one embodiment, an EDA computing device similar to the machine1100includes memory1130and one or more processors1110. The processors1110implement the design phase910to add convolution with multiple precisions in a layer, optimization to include connection changes of convolutional filters in a neural network, and optimization to include convolutional changes in a machine learning system.

In various other embodiments, rather than being implemented as modules of one or more applications1010, some or all of the design phase modules1037may be implemented using elements of the libraries1006or the operating system1004.

In various implementations, the operating system1004manages hardware resources and provides common services. The operating system1004includes, for example, a kernel1020, services1022, and drivers1024. The kernel1020acts as an abstraction layer between the hardware and the other software layers, consistent with some embodiments. For example, the kernel1020provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services1022can provide other common services for the other software layers. The drivers1024are responsible for controlling or interfacing with the underlying hardware, according to some embodiments. For instance, the drivers1024can include display drivers, signal-processing drivers to optimize modeling computation, memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), WI-FI® drivers, audio drivers, power management drivers, and so forth.

In some embodiments, the libraries1006provide a low-level common infrastructure utilized by the applications1010. The libraries1006can include system libraries1030such as libraries of analog, digital, and power-management blocks for use in an EDA environment or other libraries that can provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries1006can include API libraries1032such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render two-dimensional (2D) and 3D graphic content on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries1006may also include other libraries1034.

The software frameworks1008provide a high-level common infrastructure that can be utilized by the applications1010, according to some embodiments. For example, the software frameworks1008provide various graphic user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The software frameworks1008can provide a broad spectrum of other APIs that can be utilized by the applications1010, some of which may be specific to a particular operating system or platform. In various embodiments, the systems, methods, devices, and instructions described herein may use various files, macros, libraries, and other elements of an EDA design environment to implement analysis described herein. This includes analysis of input design files for an integrated circuit design, along with any element of hierarchical analysis that may be used as part of or along with the embodiments described herein. While netlist files, library files, Synopsys Design Constraint (SDC) files, and view definition files are examples that may operate within the software architecture1002, it will be apparent that other files and structures may provide a similar function, in various embodiments.

In some embodiments, a hardware module is implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware module can include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, a hardware module can be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware module can include software encompassed within a general-purpose processor or other programmable processor. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) can be driven by cost and time considerations.

Similarly, the methods described herein can be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors or processor-implemented modules. Moreover, the one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines1100including processors1110), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). In certain embodiments, for example, a client device may relay or operate in communication with cloud computing systems, and may store media content such as images or videos generated by devices described herein in a cloud environment.

The performance of certain of the operations may be distributed among the processors, not only residing within a single machine1100, but deployed across a number of machines1100. In some example embodiments, the processors1110or processor-implemented modules are located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors1110or processor-implemented modules are distributed across a number of geographic locations.

FIG. 11is a diagrammatic representation of a machine1100in the form of a computer system within which a set of instructions1116are executable, causing the machine1100to add convolution with multiple precisions in a layer, optimization to include connection changes of convolutional filters in a neural network, and optimization to include convolutional changes in a machine learning system, according to some example embodiments discussed herein.FIG. 11shows components of the machine1100, which is, according to some embodiments, able to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically,FIG. 11shows a diagrammatic representation of the machine1100in the example form of a computer system, within which instructions1116(e.g., software, a program, an application, an applet, an app, or other executable code) causing the machine1100to perform any one or more of the methodologies discussed herein are executable. In alternative embodiments, the machine1100operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine1100operates in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Examples of the machine1100are a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), a media system, a cellular telephone, a smart phone, a mobile device, or any machine capable of executing the instructions1116, sequentially or otherwise, that specify actions to be taken by the machine1100. Further, while only a single machine1100is illustrated, the term “machine” also includes a collection of machines1100that individually or jointly execute the instructions1116to perform any one or more of the methodologies discussed herein.

In various embodiments, the machine1100comprises processors1110, memory1130, and I/O components1150, which are configurable to communicate with each other via a bus1102. In an example embodiment, the processors1110(e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) include, for example, a processor1112and a processor1114that are able to execute the instructions1116. In one embodiment, the term “processor” includes multiple processors1110that comprise two or more independent processors1112,1114(also referred to as “cores”) that are able to execute the instructions1116contemporaneously. AlthoughFIG. 11shows multiple processors1110, in another embodiment the machine1100includes a single processor1112with a single core, a single processor1112with multiple cores (e.g., a multi-core processor1112), multiple processors1110with a single core, multiple processors1110with multiples cores, or any combination thereof.

The memory1130comprises a main memory1132, a static memory1134, and a storage unit1136accessible to the processors1110via the bus1102, according to some embodiments. The storage unit1136can include a machine-readable medium1138on which are stored the instructions1116embodying any one or more of the methodologies or functions described herein. The instructions1116can also reside, completely or at least partially, within the main memory1132, within the static memory1134, within at least one of the processors1110(e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine1100. Accordingly, in various embodiments, the main memory1132, the static memory1134, and the processors1110are examples of machine-readable media1138.

As used herein, the term “memory” refers to a machine-readable medium1138able to store data volatilely or non-volatilely and may be taken to include, but not be limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. While the machine-readable medium1138is shown, in an example embodiment, to be a single medium, the term “machine-readable medium” includes a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) storing the instructions1116. The term “machine-readable medium” also includes any medium, or combination of multiple media, that is capable of storing instructions (e.g., instructions1116) for execution by a machine (e.g., machine1100), such that the instructions, when executed by one or more processors of the machine (e.g., processors1110), cause the machine to perform any one or more of the methodologies described herein. Accordingly, a “machine-readable medium” refers to a single storage apparatus or device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” includes, but is not limited to, one or more data repositories in the form of a solid-state memory (e.g., flash memory), an optical medium, a magnetic medium, other non-volatile memory (e.g., erasable programmable read-only memory (EPROM)), or any suitable combination thereof. The term “machine-readable medium” specifically excludes non-statutory signals per se.

In some embodiments, outputs from an EDA computing device may include design documents, files for additional steps in a design flow, or outputs for circuit fabrication. In various embodiments, EDA outputs are used to generate updates and changes to a circuit design, and once a final closure of timing with all associated timing thresholds and design requirements are met, circuit design output files are used to generate masks and other physical outputs for generation of a circuit. As described herein, “requirements,” “design elements,” and other aspects of a circuit design refer to selectable values that are set as part of the design of a circuit. Such design requirements or elements may be adjusted by a system operator or circuit designer to suit the particular goals of a project or circuit that results from the operations described herein.

Communication is implementable using a wide variety of technologies. The I/O components1150may include communication components1164operable to couple the machine1100to a network1180or devices1170via a coupling1182and a coupling1172, respectively. For example, the communication components1164include a network interface component or another suitable device to interface with the network1180. One example embodiment causes a networking device such as the machine1100to transmit instructions that, when executed by one or more processors, cause the one or more processors to perform operations.

For example, one operation performs, in the multi-layer neural network, a convolutional change with a first plurality of convolutional filters connected in a first neural network layer of the multi-layer neural network. Another operation balances a complexity and a correct classification rate of the multi-layer neural network thereby to optimize the neural network.

In further examples, the communication components1164include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, BLUETOOTH® components (e.g., BLUETOOTH® Low Energy), WI-FI® components, and other communication components to provide communication via other modalities. The devices1170may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

Transmission Medium

Furthermore, the machine-readable medium1138is non-transitory (in other words, not having any transitory signals) in that it does not embody a propagating signal. However, labeling the machine-readable medium1138“non-transitory” should not be construed to mean that the machine-readable medium1138is incapable of movement; the machine-readable medium1138should be considered as being transportable from one physical location to another. Additionally, since the machine-readable medium1138is tangible, the machine-readable medium1138is a machine-readable device.

Example Operations

FIG. 12is an example method1200of transmitting instructions that cause a processor to perform operations. Operation1210causes a networking device to transmit instructions that, when executed by one or more processors, cause the one or more processors to perform operations. An example is shown inFIG. 11with the machine1100transmitting instructions to the network1180.

FIG. 13is an example1300of operations performed by transmitted instructions. Operation1310performs, in the multi-layer neural network, a convolutional change with a first plurality of convolutional filters connected in a first neural network layer of the multi-layer neural network. Operation1320balances a complexity and a correct classification rate of the multi-layer neural network thereby to optimize the neural network. Examples are shown inFIGS. 5 and 7.

FIG. 14is an example1400of operations performed by transmitted instructions. Operation1410performs, in the multi-layer neural network, a connection change with a first plurality of convolutional filters connected in a first neural network layer of the multi-layer neural network. Operation1420balances a complexity and a correct classification rate of the multi-layer neural network thereby to optimize the neural network. Examples are shown inFIGS. 6 and 7.

FIG. 15is an example1500of operations performed by transmitted instructions. Operation1510performs, in the multi-layer neural network in a computing device, a first convolution operation on a first plurality of input feature maps with a first plurality of convolutional filters, the first plurality of convolutional filters including a first set of one or more convolutional filters and a second set of one or more convolutional filters, the first plurality of convolutional filters having a plurality of filter precisions including a first filter precision and a second filter precision, the second filter precision being less than the first filter precision. An example is shown inFIG. 3.

Language

As used herein, the term “or” may be construed in either an inclusive or exclusive sense. Moreover, plural instances may be provided for resources, operations, or structures described herein as a single instance. Additionally, boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and particular operations are illustrated in a context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within a scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within a scope of embodiments of the present disclosure as represented by the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The description above includes systems, methods, techniques, instruction sequences, and computing machine program products that embody illustrative embodiments of the disclosure. In the description, for the purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that embodiments of the inventive subject matter may be practiced without these specific details. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.