Patent Description:
Machine learning is generally the process of producing a trained model (e.g., an artificial neural network), which represents a generalized fit to a set of training data. Applying the trained model to new data enables production of inferences, which may be used to gain insights into the new data.

As the use of machine learning has proliferated for enabling various machine learning (or artificial intelligence) tasks, the need for more efficient processing of machine learning model data has arisen. In some cases, dedicated hardware, such as machine learning (or artificial intelligence) accelerators or processors or similar circuits, may be used to enhance a processing system's capacity to process machine learning model data. For example, processing data with a nonlinear activation function may be distributed to a processor other than the primary matrix multiplication processor. However, distributing various aspects of processing a machine learning model across different processing devices may incur latency, memory use, power use, and other processing penalties.

Accordingly, there is a need for improved techniques for processing machine learning model data with nonlinear activation functions. <NPL>, discusses the hardware implementation of Tanh Exponential Activation Function using FPGA. Document <NPL>; patent document <CIT>; and document <NPL>; are also referenced.

Aspects of the present disclosure provide improved techniques for processing nonlinear activation functions associated with machine learning models.

Nonlinear activations are key components of various types of machine learning models, including neural network models. While some nonlinear activation functions are implemented as piecewise linear functions (e.g., rectified linear unit (ReLU), leaky ReLU, and others), other nonlinear activations functions require complex mathematical functions (e.g., sigmoid, hyperbolic tangent (tanh), and others). In some cases, the complex mathematical functions may be implemented using interpolation, such as cubic spline interpolation. For example, an interpolated output value may be determined in some aspects using a look-up table (LUT) to match output values with input values. When a target input value is not mapped in the LUT, LUT values associated with input values adjacent to the target input value may be used to interpolate an output value for the target input value.

Conventionally, nonlinear activation functions may be implemented in software rather than hardware owing to the wide range of possible activation functions usable in machine learning models. However, such implementations typically require moving model data between processing devices (e.g., between a neural processing unit (NPU) performing matrix multiplication and accumulation and a digital signal processor (DSP) processing the nonlinear activation function), thus incurring power and latency penalties. Where nonlinear activation functions have been implemented in hardware, they have generally been limited to supporting only a small number of nonlinear activation functions and thus cannot be configured to support evolving machine learning model architectures without falling back to outsourcing the nonlinear activation function processing to a distributed processing unit.

For example, the rectified linear unit (ReLU) is a commonly used activation function in deep learning models. The function returns <NUM> if it receives a negative input, and returns the input, x, other. Thus it can be written as f(x) = max(<NUM>, x). ReLU functions are generally not implemented by the primary matrix multiplication and accumulation processing unit, such as a compute-in-memory (CIM) array in some examples. Thus, the need to distribute the ReLU function, or another nonlinear activation function, is costly from a processing standpoint. Moreover, as the activation function gets more complex, the processing cost likewise gets more significant (e.g., for performing relatively higher power exponential and division operations that are part of certain nonlinear activation functions, as described further below).

To overcome the shortcomings of conventional solutions, aspects described herein relate to a configurable nonlinear activation (CNLA) function circuit that may be implemented in hardware for efficient processing. In particular, because it can be implemented in hardware, the CNLA function may be co-located with other processing circuits optimized for other machine learning model processing tasks, such as CIM arrays and digital multiply-and-accumulate (DMAC) circuits that are optimized for performing vector and matrix multiplication and accumulation functions.

In order to improve processing efficiency, aspects described herein may use polynomial approximations to approximate complex functions, such as may be used within nonlinear activation functions. In some cases, aspects described herein may use series expansions, such as a Taylor series. Generally, a Taylor series of a function (e.g., f(x)) is an infinite sum of terms that are expressed in terms of the function's derivatives at a single point. For many functions, the function and the sum of its Taylor series are equal near this point. The partial sum formed by the first n + <NUM> terms of a Taylor series is a polynomial of degree n that is referred to as the nth Taylor polynomial of the function. Thus, Taylor polynomials allow for processing efficient approximations of a function, which generally become better as n increases.

The CNLA function circuits described herein may implement one or more polynomial approximation blocks, such as cubic approximation blocks, which generally enhance cubic spline interpolation to make it more efficient and more generalized to cover a wider variety of nonlinear activation functions. Moreover, the CNLA function circuits may be implemented as a pipelined digital block that can use nonlinearly segmented look-up tables (LUTs) and mixed orders of approximations (e.g., pipelined linear, quadratic, and cubic approximations). Thus, the CNLA function circuits described herein can be configured to meet many different performance goals, unlike conventional nonlinear activation function circuits.

Accordingly, the CNLA function circuits described herein provide a technical solution to the technical problem of implementing a wide range of nonlinear activation functions in machine learning model processing systems. Further, the CNLA function circuits described herein provide a technical improvement by way of increased model processing performance compared to existing solutions, including lower latency, lower power use, improved memory efficiency, and others as described herein.

<FIG> depicts an example configurable nonlinear activation (CNLA) function circuit <NUM>.

Generally, CNLA function circuit <NUM> may be configured to receive input data <NUM> (e.g., an output value from a layer of a machine learning model) and to perform various nonlinear activation functions to generate output data <NUM> (e.g., "activations"). CNLA function circuit <NUM> may be co-located and pipelined with other machine learning model processing circuits, such as a CIM array, DMAC, and others, and may be configured to perform activation functions based on the output of the other machine learning model processing circuits.

In some examples, input data <NUM> may be received from a buffer or other memory. In other examples, input data <NUM> may be received directly from the output of another processing block, such as the output of a CIM array or another vector and matrix multiplication and accumulation block, or the like.

CNLA function circuit <NUM> includes a first approximator block <NUM>, which may generally be configured to perform a hardware-based mathematical function, such as on input data <NUM>. An example approximator is described in detail with respect to <FIG>.

In some cases, first approximator is one of a linear approximator (e.g., configured to perform a function, such as ax + b), a quadratic approximator (e.g., configured to perform a function, such as ax<NUM> + bx + c), or a cubic approximator (e.g., configured to perform a function, such as ax<NUM> + bx<NUM> + cx + d), where x is the input data and a, b, c, and d are configurable parameters. Generally, a linear, quadratic, or cubic approximator may be used to approximate some given function, which may or may not be a polynomial function. First approximator <NUM> may be configured with parameters, retrieved from, for example, a memory, a register, a look-up table, or the like. As described in further detail below with respect to Table <NUM>, these different forms of approximation and associated configurable parameters can be used to approximate many types of nonlinear activation functions.

CNLA function circuit <NUM> further includes a second approximator block <NUM>, which, like first approximator block <NUM>, may generally be configured to perform a hardware-based mathematical function, such as a linear, quadratic, or cubic function. As described in more detail below, CNLA function circuit <NUM> may be configured to use first approximator block <NUM> and second approximator block <NUM> in series for more complex functions, such that the output of first approximator block <NUM> becomes an input to second approximator block <NUM>. CNLA function circuit <NUM> may be further configured to use only one of first approximator block <NUM> or second approximator block <NUM> when a simpler nonlinear function is being processed, thereby saving power.

In some implementations, first approximator <NUM> and second approximator <NUM> may comprise the same circuit block (e.g., two instances of the same circuit elements within circuit <NUM>). For example, first approximator <NUM> and second approximator <NUM> may comprise cubic approximators in some aspects. In other implementations, first approximator <NUM> and second approximator <NUM> may comprise different circuit elements, and in such cases, generally second approximator <NUM> will comprise a cubic approximator and first approximator <NUM> will comprise a lower order approximator, such as a quadratic or linear approximator. However, in other embodiments, the order of the higher and lower order approximators may be reversed.

CNLA function circuit <NUM> includes a configurable bypass <NUM>, which allows first approximator <NUM> to be bypassed in various scenarios, such as if a function only requires a lower order approximator than first approximator <NUM> and second approximator <NUM> is such a lower order approximator. When, for example, first approximator <NUM> is bypassed via configurable bypass <NUM>, then input data <NUM> is provided directly to second approximator <NUM> instead and not processed by first approximator <NUM>. In various aspects, first approximator <NUM> may be a higher order approximator compared to second approximator <NUM>, or vice versa, or they may be of the same order (e.g., both linear, quadratic, or cubic). The configurable bypass <NUM> allows for saving processing time and energy when only one approximator is necessary.

CNLA function circuit <NUM> further includes another configurable bypass <NUM>, which allows second approximator <NUM> to be bypassed in various scenarios, such as if a function only requires a first approximation, which first approximator <NUM> is capable of performing without second approximator <NUM>. When, for example, second approximator <NUM> is bypassed via configurable bypass <NUM>, the output of first approximator <NUM> is provided directly to multiplier <NUM>.

Generally, configurable bypasses <NUM> and <NUM> allow CNLA function circuit <NUM> to be configured for maximum versatility, while saving power and avoiding unnecessary circuit block processing in various scenarios. Further, configurable bypasses allow for non-symmetric and anti-symmetric nonlinear activation functions to be configured for processing by CNLA function circuit <NUM>. <FIG> depicts example circuit aspects for implementing configurable bypasses <NUM> and <NUM> (e.g., bypasses 205A and 205B).

CNLA function circuit <NUM> further includes a gain block <NUM> configured to provide a gain value to multiplier <NUM>. In some aspects, gain block <NUM> is configured to generate a gain value <NUM> based on a gain function implemented by gain block <NUM>. In one example, the gain function may be in the form g = ax + b, where g is the gain value, x is the input data <NUM> value, and a and b are configurable parameters. More generally, the gain block <NUM> may modify the input data multiplicatively (a) and/or additively (b) to generate the gain value.

The gain value <NUM> generated by gain block <NUM> is multiplied with the output of first and/or second approximators <NUM> and <NUM> via multiplier <NUM>. In other aspects, gain block <NUM> may be configured to generate a gain value that is not based on a function of input data <NUM> (e.g., by setting a to zero in the above expression for g). Generally, the parameters (e.g., a and b in the example above) or value for gain block <NUM> may be retrieved from, for example, a memory, a register, a look-up table, or the like.

CNLA function circuit <NUM> further includes a constant block <NUM> configured to store a configurable (e.g., programmable) constant value <NUM> and adder <NUM> configured to add the constant value <NUM> to the output of multiplier <NUM> (e.g., a gain multiplier). The constant value <NUM> stored in constant block <NUM> may be retrieved from, for example, a memory, a register, a look-up table, or the like.

The inclusion and arrangement of first approximator block <NUM>, second approximator block <NUM>, configurable bypasses <NUM> and <NUM>, gain block <NUM>, multiplier <NUM>, constant block <NUM>, and adder <NUM> allows for CNLA function circuit <NUM> to be configured to perform a wide variety of known and later developed nonlinear activation functions. Moreover, CNLA function circuit <NUM> may be efficiently configured to process a wide variety of nonlinear activation functions by merely updating parameters for the first approximator <NUM>, second approximator <NUM>, gain block <NUM>, and constant block <NUM>. When both approximator blocks <NUM> and <NUM> are used to simulate a nonlinear function, each may be referred to as performing an individual function (e.g., a first function for the first approximator block <NUM> and a second function for the second approximator <NUM>). This design beneficially supports arbitrary non-symmetric nonlinear curves for complex functions.

Table <NUM>, below, provides example parameters for various nonlinear activation functions that CNLA function circuit <NUM> of <FIG> can be configured to perform, including parameters for approximator blocks 206A and 206B of <FIG>. In Table <NUM>, the gain is considered to have the form ax + b, as is in the example of gain block <NUM> in <FIG>, but note that in other embodiments, the gain may be a scalar value, or a different functional form. Similarly, a quadratic approximator is considered to have the form ax<NUM> + bx + c and a cubic approximator is considered to have the form ax<NUM> + bx<NUM> + cx + d. In the following table, subscripts are used to indicate parameter assignments, e.g., G for gain parameters, <NUM> for first approximator, and <NUM> for second approximator parameters.

Notably, in some implementations, parameters for an approximator may be given in a form (e.g., cubic with a, b, c, and d parameters or quadratic with a, b, and c parameters), even where the approximator is performing a lower order function (e.g., linear). This is because setting, for example, the cubic parameter a to zero effectively collapses the approximation equation to a lower order quadratic function, and likewise setting the quadratic parameter a to zero effectively collapses the approximation equation to a linear equation. Thus, an approximator may be configured, for example, for a "quadratic function" when it is configured with quadratic parameters, but the result of the parameters may reduce the function to a linear function, as in the example of ReLU above in Table <NUM>. This may allow standardization of the parameter set despite the order of the underlying function to be configured by the parameters, thereby simplifying the implementation.

<FIG> depicts example circuit blocks <NUM> and <NUM> for implementing bypassable approximator blocks 206A and 206B. Bypassable approximator blocks 206A and 206B may correspond to first approximator block <NUM> and second approximator block <NUM> of <FIG> in one example.

In <FIG>, circuit block <NUM> is configured to control use of function block 214A, which includes first approximator 206A and minimum and maximum function block 208A in this example. Similarly, circuit block <NUM> controls use of function block 214B, which includes minimum and maximum function block 208B and second approximator 206B, in this example. The first and second approximator blocks 206A and 206B may be configured to implement nonlinear activation functions, such as those described above with respect to Table <NUM>.

Note that the first approximator <NUM> in <FIG> requires only one input, but circuit block <NUM> includes two input ports, 201A and 201B, which allows for multiple inputs. The depicted configuration of circuit block <NUM> may be adopted in order to present the same external interface for both circuit blocks <NUM> and <NUM>, which may simplify configuration and integration. In some aspects, the two input ports 201A and 201B of circuit block <NUM> may be tied together in an implementation where circuit block <NUM> receives a single input (such as input data <NUM> in <FIG>) via input port 201A. In an alternative implementation, circuit block <NUM> can be simplified by removing input port 201B and removing input mux 203A such that 201A would be provided directly to 214A and 207A.

Generally, input ports 201A and 201B may receive various types of input data for processing, including signed multibit integer data. In one example, the input data is <NUM>-bit <NUM> complement input data.

Input selector muxes 203A and 203B are configured to control which input data port is used for circuit blocks <NUM> and <NUM>, respectively. For example, input selector mux 203B may select between input data port 201A (e.g., when circuit block <NUM> is being bypassed) or 212B (e.g., when circuit blocks <NUM> and <NUM> are being processed in series).

Bypass selector muxes 211A and 211B are configured to control bypassing function blocks 214A and 214B of circuit blocks <NUM> and <NUM>, respectively. For example, when circuit block <NUM> is to be bypassed, bypass selector mux 211A selects bypass line 205A to provide an output to output port 212A. Similarly, when circuit block <NUM> is to be bypassed, bypass selector mux 211B selects bypass line 205B to provide an output to output port <NUM>. Thus, processing with circuit block <NUM> and/or <NUM>, as controlled by the configurable bypasses 205A and 205B, results in an output at output port <NUM>.

As discussed in more detail with respect to <FIG>, approximator blocks 206A and 206B may be configured with configuration parameters (e.g., function specific coefficients as in Table <NUM>, above) stored in registers 219A and 219B, respectively. Similarly, as in Table <NUM>, above, where approximator blocks 206A or 206B are configured to perform a look-up table-based function, the table values may be stored in registers 219A and 219B, respectively.

Each circuit block (<NUM> and <NUM>) further includes a minimum and maximum function block (208A for circuit block <NUM> and 208B for circuit block <NUM>) for providing minimum and maximum functions. Generally, a minimum (or "min") function will return the minimum value of the provided inputs. Similarly, a maximum (or "max") function will return the maximum value of the provided inputs. In one example, minimum and maximum function blocks 208A and 208B may comprise multibit digital comparators that run in either a single cycle or multi-cycle mode.

The configuration of function blocks 214A and 214B may include a setting for function selector muxes 209A and 209B, respectively. In other words, whether or not function blocks 214A and 214B output a min/max output from mix/max blocks 208A and 208B or a value from approximators 206A and 206B is based on the configuration of function selector muxes 209A and 209B. Note that in other examples, function blocks 214A and 214B may include additional function blocks that may be selected by a mux.

As depicted in <FIG> where approximator blocks can be processed in series, in <FIG> the output 212A of circuit block 202A, which includes a first approximator block 206A, is provided as an input 212B to circuit block <NUM>, which includes a second approximator block 206B. As in <FIG> where bypasses <NUM> and <NUM> control use of the first and second approximator blocks <NUM> and <NUM>, here the selectable bypasses 205A and 205B control use of approximator blocks 206A and 206B.

An asymmetric signal line <NUM> controls a configuration of the circuit blocks <NUM> and <NUM>. In one example, circuit blocks <NUM> and <NUM> are configured based on values on asymmetric signal line <NUM> and output values from sign blocks 207A and 207B based on the input data received via input data port 201A. For example, the binary value received via the asymmetric signal line <NUM> and the binary value output from sign block 207A interact at AND gate <NUM> to control the selection of output by mux 211A. As another example, the binary value received via the asymmetric signal line <NUM> and the binary value output from sign block 207B interact at AND gate <NUM> to control the selection of an input data port (as between 201A and 212B) via mux 203B. As a further example, the binary value received via the asymmetric signal line <NUM> and the inverted binary value output from sign block 207B interact at and gate <NUM> to control the selection of output mux 211B.

Table <NUM>, below, provides a summary of configurations for circuit blocks <NUM> and <NUM>:.

<FIG> depicts an example approximator <NUM>, which may be an example of one or both of first approximator <NUM> and second approximator <NUM> of <FIG> and/or approximators 206A and 206B of <FIG>.

Approximator <NUM> receives input data <NUM> (e.g., pre-activation data) for processing. In some examples, input data <NUM> may be received from a buffer or other memory. In other examples, input data may be received directly from the output of another processing block, such as the output of a CIM array or another vector and matrix multiplication and accumulation block. Further, input data may be received from another approximator, such as if approximator <NUM> is the second approximator <NUM> in <FIG> or the second approximator 206B in <FIG>.

In some implementations, an approximator (such as <NUM>) may include alternative processing paths. In such cases, path logic <NUM> may be configured to route input data <NUM> to the appropriate processing path based on, for example, a configuration parameter for approximator <NUM>.

In this example, processing path 306A provides a cubic approximation path for input data <NUM>.

In processing path 306A, input data <NUM> is provided to cubic calculator <NUM>, which performs a cubic operation (e.g., x<NUM>, where x is the input data) and then the output is multiplied with cubic parameter <NUM> at multiplier <NUM>. The output of multiplier <NUM> is then provided to accumulator <NUM>.

Input data <NUM> is also provided to quadratic calculator <NUM>, which performs a quadratic operation (e.g., x<NUM>, where x is the input data) and then the output is multiplied by quadratic parameter <NUM> at multiplier <NUM>. The output of multiplier <NUM> is then provided to accumulator <NUM>.

Input data <NUM> is also provided to multiplier <NUM> where it is multiplied by linear parameter <NUM>. The output of multiplier <NUM> is then provided to accumulator <NUM>.

Accumulator (adder) <NUM> accumulates the outputs of multipliers <NUM>, <NUM>, and <NUM> as well as intercept parameter <NUM> to generate output data <NUM>.

Cubic parameter <NUM>, quadratic parameter <NUM>, linear parameter <NUM> and intercept parameter <NUM> may all be stored in a memory or the like (e.g., in registers) accessible to accumulator <NUM>. In some cases, a control unit, such as a memory control unit or finite state machine, may configure approximator <NUM> with parameters stored in the memory. In various examples, cubic parameter <NUM>, quadratic parameter <NUM>, linear parameter <NUM> and intercept parameter <NUM> may be set according to values described above with respect to Table <NUM>.

As above, the order of the approximation can be configured by configuring the aforementioned parameter values. For example, for approximator <NUM> to perform a quadratic approximation, cubic parameter <NUM> can be set to zero. Similarly, for approximator <NUM> to perform a linear approximation, cubic parameter <NUM> and quadratic parameter <NUM> can be set to zero.

Certain nonlinear activation functions require alternative functions, such as minimum and maximum functions. Accordingly, processing path 306B provides a minimum and/or maximum calculator that may be used, for example, with the ReLU and ReLU6 functions described above in Table <NUM>. Processing path 306B may be selected by path logic <NUM> based on configuration data for approximator <NUM>.

Further, certain nonlinear activation functions may be implemented using look-up tables, which provide a more power and time efficient mechanism for generating values for certain nonlinear activation functions. Accordingly, processing path 306C provides a look-up table-based processing path that may be used, for example, wherever a sigmoid, tanh, or similar function is used by a nonlinear activation function. Note that sigmoid and tanh may be calculated from each other, so in some cases, only a single look-up table (e.g., sigmoid or tanh, but not both) is stored and used to implement both functions. One or more look-up tables may be stored in a memory and accessible to approximator <NUM>, including a memory tightly coupled to approximator <NUM>.

<FIG> depicts an example machine learning model data flow <NUM> that implements a configurable nonlinear activation function circuit, such as described above with respect to <FIG>.

In flow <NUM>, input data is stored in an input data buffer <NUM> (e.g., machine learning model layer input data) and then provided to a multiply and accumulate (MAC) circuit <NUM>. MAC circuit <NUM> may generally be configured to perform vector, array, and matrix multiplication and accumulation operations, such as those used frequently in convolutional neural networks. In some examples, MAC circuit <NUM> may include one or more compute-in-memory (CIM) arrays. Alternatively, or additionally, MAC circuit <NUM> may include a digital multiply and accumulate (DMAC). In yet further examples, multiply and accumulate circuit <NUM> may be a portion of a machine learning accelerator, such as a neural processing unit (NPU), or another type of processing unit optimized for performing machine learning processing. In another implementation, MAC circuit <NUM> may be replaced by a vector/matrix or matrix/matrix processing engine.

MAC circuit <NUM> processes the input data with weight data (e.g., neural network weight data) to generate pre-activation data. For example, MAC circuit <NUM> may process input data to a layer of a neural network model and generate pre-activation data as an output.

The pre-activation data is provided to configurable nonlinear activation (CNLA) function circuit <NUM>, which is configured to generate output data (e.g., activations) based on a configured nonlinear activation function. The output data may then be stored in output data buffer <NUM> for subsequent use, such as for processing another layer in a machine learning model, or as output from the machine learning model, and the like.

CNLA function circuit <NUM> may be configured with configuration parameters, such as described with respect to CNLA function circuit <NUM> in <FIG> and those described in Tables <NUM> and <NUM>. Further, CNLA function circuit <NUM> may be configured to access look-up tables depending on the configured activation function.

In some cases, configuration parameters may include identification of a nonlinear activation function to be applied to the input data. Based on the determined nonlinear activation function, appropriate parameters (such as those in Table <NUM>) may be retrieved from a memory (e.g., registers) and applied to CNLA function circuit <NUM> thereby configuring it for processing the input data. In some examples, a finite state machine, a memory control unit, or another controller, may perform the configuration of CNLA function circuit <NUM>.

Notably, CNLA circuit <NUM> may be configured to process multiple batches of input data using the same configuration, or may update its configuration for every new batch of input data. Thus, CNLA circuit <NUM> provides a very flexible and efficient means for performing configurable nonlinear activations for machine learning tasks, such as training and inferencing.

<FIG> depicts an example method <NUM> for performing processing using a configurable nonlinear activation function circuit.

Method <NUM> begins at step <NUM> with selecting a nonlinear activation function for application to input data. For example, the nonlinear activation function may be one of the functions listed in Table <NUM>, or another nonlinear activation function.

Method <NUM> then proceeds to step <NUM> with determining, based on the selected nonlinear activation function, a set of parameters for a configurable nonlinear activation function circuit. For example, the parameters for the determined nonlinear activation function may be as above in Tables <NUM> and <NUM>.

Method <NUM> then proceeds to step <NUM> with processing input data with the configurable nonlinear activation function circuit based on the set of parameters to generate output data. For example, the output data may be activation data for a layer of a neural network model.

In some examples, the set of parameters includes a combination of one or more gain parameters, a constant parameter, and one or more approximation functions to apply to the input data via the configurable nonlinear activation function circuit. For example, the set of parameters may be as discussed above with respect to <FIG> and <FIG> and in Table <NUM>.

In some examples, method <NUM> further includes retrieving the set of parameters from a memory based on the determined nonlinear activation function. In some examples, the memory may be one or more registers storing the parameter values.

In some examples, the configurable nonlinear activation function circuit includes a first approximator configured to approximate a first function of the one or more approximation functions; a second approximator configured to approximate a second function of the one or more approximation functions; a first gain multiplier configured to multiply a first gain value based on one or more gain parameters; and a constant adder configured to add a constant value, such as depicted and described with respect to <FIG>.

In some examples, the configurable nonlinear activation function circuit includes a first bypass configured to bypass the first approximator. In some examples, the configurable nonlinear activation function circuit includes a second bypass configured to bypass the second approximator. In some examples, the configurable nonlinear activation function circuit includes an input data bypass configured to bypass the first approximator and to provide input data to the second approximator.

In some examples, at least one of the first approximator and the second approximator is a cubic approximator. In some examples, an other one of the first approximator and the second approximator is one of a quadratic approximator or a linear approximator. In some examples, an other one of the first approximator and the second approximator is configured to perform a min or max function, such as depicted with respect to path 306B in <FIG>. In some examples, an other one of the first approximator and the second approximator is configured to access a look-up table for an approximated value, such as depicted with respect to path 306C in <FIG>.

In some examples, both the first approximator and the second approximator are cubic approximators.

Note that <FIG> is just one example, and in other examples, methods such as those described herein, may be implemented with more, fewer, and/or different steps.

<FIG> depicts an example of a pipelined successive-linear-approximation architecture <NUM> that may be used to implement nonlinear functions used in machine learning model acceleration. For example, architecture <NUM> may be used to implement the approximator <NUM> and <NUM> described with respect to <FIG> and 206A and 206B described with respect to <FIG>.

Generally, <FIG> depicts input x flowing in a pipelined fashion through the linear approximators 604A-C. Architecture <NUM> is a power and space efficient way to implement cubic approximation using successive linear approximation blocks 604A-C. Specifically, an input x is processed in linear approximator block 604A to determine an output y<NUM> = Ax + B , where A and B are coefficients 605A used by linear approximator block 604A.

Next, y<NUM> is provided as an input to linear approximator block 604B to determine an output y<NUM> = x(y<NUM>) + C, where C is a coefficient 605B used by linear approximator block 604B. Note that x(y<NUM>) indicates y<NUM> multiplied by x.

Next, y<NUM> is provided as an input to linear approximator block 604C to determine an output y<NUM> = x(y<NUM>) + D, where D is a coefficients 605C used by linear approximator block 604C. Expanding y<NUM> based on the successive linear approximations gives: <MAT>.

which is a cubic approximation based on the input x. Notably, the successive linear approximation approach implemented in architecture <NUM> reduces the number of multipliers needed, which reduces complexity and power used compared to a conventional cubic approximator architecture. Further, architecture <NUM> allows a pipelined implementation, which in steady-state produces a new output every cycle and therefore improves throughput.

To further improve processing efficiency, the coefficients A, B, C, and D (605A-C) may be stored in registers rather than, for example, an SRAM or DRAM, which improves power and latency performance of architecture <NUM>. For example, the use of read power from a host system SRAM can be avoided by using local registers storing coefficient values. The register approach also supports the pipelined architecture by avoiding additional cycles for reading and transporting coefficient values from a remote host system memory (e.g., DRAM). In various aspects, pipeline register may be internal or external to linear approximator 604A-C.

In some implementations, as depicted with respect to architecture <NUM>, the coefficients may be determined based on approximations of non-uniformly segmented non-linear functions, as is described in one example with respect to <FIG>. In such embodiments, region finder <NUM> may determine the appropriate coefficients based on the region (or segment) of a non-uniformly segmented approximation according to the input x. <FIG> thus depicts the region finder's output "REGION_SEL" as passed in a pipelined fashion through the various stages of linear approximation.

Note that the "regions" being found by region finder <NUM> are segments of the "domain" of an original nonlinear function. That is, a nonlinear function may be defined over a domain from negative infinity to positive infinity, and that domain can be partitioned into segments. In various aspects described herein, the segments are of non-uniform lengths. For each such segment, coefficients are chosen to best approximate the function within that segment. <FIG> depicts an example circuit implementation with a region finder element <NUM>, where "cutoffs" are used to describe segment endpoints.

In various aspects, region finder <NUM> may be configured to choose the entire set of coefficients used for the entire approximation. In the depicted example, because the approximation occurs serially in a pipelined manner, the determined region would get registered running alongside each approximator datapath so the determined region can be used for subsequent coefficient lookups at each stage (e.g., for linear approximators 604A-C).

Region finder <NUM> generally allows for a variety of offline algorithms to be used to search for and set the region choices, so the implementation need not be limited by how the regions are found. For example, a cubic spline algorithm may be used to ensure a continuous derivative across region boundaries, which is beneficial for model training. In another example, the mean-square error may be minimized on a per-point basis (based upon a subsequent search for best cubic coefficients given the region being considered). This second approach may achieve better accuracy in the calculation at the expensive of the continuous derivative (achieving better accuracy for inferences). Yet another approach is to set regions based upon where <NUM>nd or <NUM>rd derivatives are <NUM>, though this approach might not use a full set of possible regions because more regions results in less error.

Finally, architecture <NUM> is configured to exploit odd and even symmetry properties of common neural network nonlinearity functions, such as sigmoid and hyperbolic tangent (tanh) to beneficially reduce circuit area and power during processing. Beneficially, exploiting such symmetries allows for reducing the number of coefficients that need to be stored to approximate a nonlinear function because, for example, a simple sign flip can emulate both sides (e.g., positive and negative) of the function from the origin without needing to store coefficients for both sides. In some implementations, approximation results are only calculated for positive inputs, and then the same result can be used for a negative input with the same magnitude (even symmetry) or the sign of the result may be flipped in the case of a function with odd symmetry.

In the depicted example, sign and offset corrector component <NUM> is configured to take as inputs the cubic approximation value y<NUM>, the original input x (e.g., to determine its sign), and an offset value <NUM> and to apply a sign and/or offset correction to generate a final output <NUM>. For example, the final output <NUM> may be calculated as offset (<NUM>) +/- y<NUM>. As with coefficients 605A-C, offset <NUM> may be stored in local registers for efficiency and speed.

Generally, offset <NUM> may be used for functions that are symmetric, except not centered on x = <NUM>. An example is sigmoid, which is symmetric about the line y = <NUM>. In such cases, a "shifted" sigmoid is first approximated that is odd symmetric about the x-axis, and then an offset of <NUM> is added to the result.

Note that <FIG> depicts an example architecture <NUM> including three linear approximators (604A-C) that can be used to approximate a cubic nonlinear function. In other aspects, two linear approximators may be used where a quadratic nonlinear function is intended to be approximated. Though not depicted in <FIG>, in some aspects, control logic may switch the output of a linear approximator, such as linear approximator 604B to bypass a following linear approximator (such as linear approximator 604C), when only a quadratic function is needing to be approximated. In such cases, the bypass may take the output of linear approximator 604B, y<NUM>, and route it to sign and offset corrector component <NUM>.

<FIG> depicts an example <NUM> of a hyperbolic tangent (tanh) nonlinear activation function with non-uniform segments for defining approximated function outputs.

In the depicted example, a first segment 702A is defined from the origin (x = <NUM>) to a first point along curve <NUM>. As depicted, segments like 702B are longer in length along curve <NUM> where the slope is more constant across input values in the segment, and segments like 702C are shorter in length along curve <NUM> where the curve is changing slope more quickly along input values in the segment. Using non-uniform segments beneficially allows for defining fewer segments across the output value range for a nonlinear function while still maintaining similar output resolution as compared to the actual function. Conventionally, to maintain resolution, a curve such as <NUM> was uniformly segmented, which meant that many segments had very similar output values and thus created redundant values stored in memory.

Note that each segment may generally be defined by a start point and an end point along curve <NUM>. For example, a starting input value xstart and an ending input value, xend , may define a segment. Each of these segments may be associated with a set of coefficients for use in approximating the hyperbolic tangent function based input values falling within the defined range, such as xstart ≤ x < xend. In order to further save memory, the first segment 702A need only store the end point of the segment as the starting point (the origin) may be assumed to be x = <NUM>. Similarly, the last segment is defined only by the endpoint of the second-to-last segment. That is, all inputs greater than or equal to the second-last segment's endpoint are considered part of the last segment.

For example, the following table gives examples of non-uniform segments that may be stored in registers for a tanh function that is going to be approximated. When the input value x falls within a particular segment (or region), then coefficients for that segment may be retrieved and used to generate an approximated function value.

Note that the hyperbolic tangent function curve <NUM> is an example of a function with odd symmetry. That is, the tanh output value for a positive input (e.g., <NUM> for point 704A) is equivalent to the sign-flipped output of a negative input (e.g., -<NUM> for 704B) of the same magnitude. Thus, the non-uniform segments need only be defined with respect to the positive input value ranges for the hyperbolic tangent function curve <NUM> and a sign corrector component, such as <NUM> in <FIG>, can appropriately flip the sign of the ultimate output (e.g., <NUM> in <FIG>) based on the sign of the input value (x).

<FIG> depicts an example of a circuit <NUM> for implementing a pipelined successive-linear-approximation architecture, such as described above with respect to <FIG>. In particular, <FIG> depicts the pipelined nature of the design, including how the region finder <NUM>'s output and a sign bit is passed through the pipeline for efficient processing.

The depicted example includes four stages 806A-806D, of which three stages include linear approximators (804A-804C, respectively) and one sign and offset correction stage 806D.

In the depicted example, various operational blocks are depicted. "SAT" represents a "saturation" block. Saturation is useful when reducing the number of integer bits used in representing a value. If a saturation module is used to reduce an input having M integer bits to an output having N integer bits (where N < M), then input values that are larger than what can be represented with N integer bits are "saturated" to either the largest positive N-bit number (for positive inputs) or the smallest negative N-bit number (for negative inputs). For example, if [x > <NUM>. <NUM>], then output y(<NUM>. <NUM>), or if [x <= <NUM>. <NUM>], then output y(<NUM>.

"ABS" represents a block that computes the absolute value (or magnitude) of an input.

"COMP" represents a comparator block that serves as a region finder using programmable cutoffs. The cutoffs define the endpoints of the various segments (and, indirectly, the start point for the very last segment). The output of the COMP block <NUM> is an identifier that indicates to which region (segment) the present input belongs. This identifier is used to select the appropriate set of coefficients to approximate the non-linear function within that segment.

In one example implementation, <NUM> cutoff values are used to define a total of <NUM> segments. By exploiting the symmetry properties of the non-linearity function, the COMP module only operates on the absolute value of the input. The following algorithm describes its operation:
IF <NUM> <= ABS(x) < CUTOFF1 : Select Segment <NUM> coefficients
ELSE IF (CUTOFF1 <= ABS(x) < CUTOFF2) : Select Segment <NUM> coefficients
ELSE IF (CUTOFF2 <= ABS(x) < CUTOFF3) : Select Segment <NUM> coefficients. ELSE IF (CUTOFF10<= ABS(x) < CUTOFF11): Select Segment <NUM> coefficients
ELSE IF (CUTOFF11<= ABS(x)): Select Segment <NUM> coefficients.

"TRUNC" represents a truncate block, which is configured to reduce the precision of a fractional number represented in two's complement fashion by omitting the least significant bits. For example, to reduce an input from having M fractional bits to having N fractional bits, the least significant (M - N) bits are omitted (truncated).

"SYMMSAT" represents a "symmetric saturation" block. Conventional saturation (SAT) modules having N integer bits saturate to either to the largest positive value (<NUM>N-<NUM> - <NUM>) or the smallest negative value (-<NUM>N-<NUM>). In a symmetric saturation module, negative inputs instead saturate to one larger than the smallest negative value (e.g., (-<NUM>N-<NUM> + <NUM>). This way the number of distinct negative and positive outputs is identical. For example, if [x > <NUM>. <NUM>], then output y(<NUM>. <NUM>), or if [x <= <NUM>. <NUM>], then output y(<NUM>.

"SE" represents a sign-extension block, which is useful when increasing the number of integer bits used to represent a value. For example, when a value represented with N integer bits is sought to be represented with M integer bits (M > N), the sign bit is replicated (M - N) times.

"TC" represents a block used to convert a positive number to a negative number by taking its two's complement.

Finally "ZF" represents a "zero filling" block that is used when a number with N fractional bits is sought to be represented as a number with M fractional bits (M > N), by appending (M - N) zeros in the least significant positions.

<FIG> depicts an example of a linear approximator circuit <NUM>, which may be an example of the linear approximator blocks 804A-804C in circuit <NUM> of <FIG>. As depicted, the linear approximator circuit includes a multiplication element <NUM> that takes as input the stage input and a coefficient <NUM> as well as an adder element <NUM> that takes as input the output of the multiplication element after being processed by a truncation and saturation block and a second coefficient <NUM>.

<FIG> depicts an example processing system <NUM> that may be configured to perform the methods described herein, such as with respect to <FIG>.

Processing system <NUM> includes a central processing unit (CPU) <NUM>, which in some examples may be a multi-core CPU. Instructions executed at the CPU <NUM> may be loaded, for example, from a program memory associated with the CPU <NUM> or may be loaded from memory partition <NUM>.

Processing system <NUM> also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) <NUM>, a digital signal processor (DSP) <NUM>, a neural processing unit (NPU) <NUM>, a multimedia processing unit <NUM>, and a wireless connectivity component <NUM>.

An NPU, such as <NUM>, is generally a specialized circuit configured for implementing all the necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, and the like. An NPU may sometimes alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

NPUs, such as <NUM>, may be configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other tasks. In some examples, a plurality of NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples they may be part of a dedicated machine learning accelerator device.

NPUs may be optimized for training or inference, or in some cases configured to balance performance between both. For NPUs that are capable of performing both training and inference, the two tasks may still generally be performed independently.

NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly compute-intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, in order to improve model performance. Generally, optimizing based on a wrong prediction involves propagating back through the layers of the model and determining gradients to reduce the prediction error.

NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs may thus be configured to input a new piece of data and rapidly process it through an already trained model to generate a model output (e.g., an inference).

In some embodiments, NPU <NUM> may be implemented as a part of one or more of CPU <NUM>, GPU <NUM>, and/or DSP <NUM>.

In some embodiments, wireless connectivity component <NUM> may include subcomponents, for example, for third generation (<NUM>) connectivity, fourth generation (<NUM>) connectivity (e.g., <NUM> LTE), fifth generation connectivity (e.g., <NUM> or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Wireless connectivity processing component <NUM> is further connected to one or more antennas <NUM>.

Processing system <NUM> may also include one or more sensor processing units <NUM> associated with any manner of sensor, one or more image signal processors (ISPs) <NUM> associated with any manner of image sensor, and/or a navigation processor <NUM>, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

Processing system <NUM> may also include one or more input and/or output devices <NUM>, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, and the like.

In some examples, one or more of the processors of processing system <NUM> may be based on an ARM or RISC-V instruction set.

Processing system <NUM> also includes various circuits in accordance with the various embodiments described herein.

In this example, processing system <NUM> includes compute-in-memory (CIM) circuit <NUM>, which may be configured to perform efficient multiply-and-accumulate (MAC) functions for processing machine learning model data. Processing system <NUM> further includes configurable nonlinear activation (CNLA) function circuit <NUM>. In some cases, CNLA function circuit <NUM> may be like CNLA function circuit <NUM> described with respect to <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>. CNLA function circuit <NUM>, as well as others not depicted, may be configured to perform various aspects of the methods described herein, such as methods <NUM> and <NUM> with respect to <FIG> and <FIG>, respectively.

In some examples, CNLA function circuit <NUM> may be implemented as a part of another processing unit, such as CPU <NUM>, GPU <NUM>, DSP <NUM>, or NPU <NUM>.

Processing system <NUM> also includes memory <NUM>, which is representative of one or more static and/or dynamic memories, such as a dynamic random access memory, a flash-based static memory, and the like. In this example, memory <NUM> includes computer-executable components, which may be executed by one or more of the aforementioned components of processing system <NUM>.

In particular, in this example, memory <NUM> includes determining component 1024A, configuring component 1024B, processing component 1024C, retrieving component 1024D, nonlinear activation function parameters 1024E, look-up table(s) 1024F, and model parameters <NUM> (e.g., weights, biases, and other machine learning model parameters). One or more of the depicted components, as well as others not depicted, may be configured to perform various aspects of the methods described herein.

Generally, processing system <NUM> and/or components thereof may be configured to perform the methods described herein.

Notably, in other embodiments, aspects of processing system <NUM> may be omitted, such as where processing system <NUM> is a server computer or the like. For example, multimedia component <NUM>, wireless connectivity <NUM>, sensors <NUM>, ISPs <NUM>, and/or navigation component <NUM> may be omitted in other embodiments. Further, aspects of processing system <NUM> maybe distributed.

Note that <FIG> is just one example, and in other examples, alternative processing system with more, fewer, and/or different components may be used.

The preceding description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not limiting of the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. In addition, the scope of the disclosure is intended to cover such an apparatus or method that is practiced using other structure, functionality, or structure and functionality in addition to, or other than, the various aspects of the disclosure set forth herein.

As used herein, the term "connected to", in the context of sharing electronic signals and data between the elements described herein, may generally mean in data communication between the respective elements that are connected to each other. In some cases, elements may be directly connected to each other, such as via one or more conductive traces, lines, or other conductive carriers capable of carrying signals and/or data between the respective elements that are directly connected to each other. In other cases, elements may be indirectly connected to each other, such as via one or more data busses or similar shared circuitry and/or integrated circuit elements for communicating signals and data between the respective elements that are indirectly connected to each other.

Further, the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions.

Claim 1:
A processor, comprising:
a configurable nonlinear activation function circuit configured to:
determine (<NUM>), based on a selected nonlinear activation function, a set of parameters for the selected nonlinear activation function; and
generate (<NUM>) output data based on application of the set of parameters for the selected nonlinear activation function,
the processor further characterized in that
the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two successive linear approximators, and
each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the set of parameters.