Patent Description:
Camera technology continues to advance. Cameras (e.g., digital cameras) are integrated in mobile electronic devices (smartphones, tablet computers, etc.) and can be implemented in recreation devices, such as action cameras, as well as many other applications, such as automotive systems, security systems, and so forth. In such digital cameras, an image signal processor (ISP) can be implemented, which can operate using one or more look up tables (LUTs) to process (e.g., adjust, tune, etc.) images (image frames, etc.) captured by an image sensor of the camera, such as by a CMOS sensor or a charge-coupled sensor, for example. Such LUTs can be stored in configuration registers (configuration memory) of the ISP and can each include a finite number of configuration values used by an ISP for processing captured image frames. Such configuration information can be related to, or correspond with, different aspects of captured image frames. These attributes can include lighting, exposure time, cumulative color temperature, chrominance, luminance, contrast, gamma, etc. of captured image frames.

Such LUTs are typically large data structures, and the attributes they correspond with can be expressed as quantitative values, which may be referred to as image factors. However, because such image factors are continuous values, generating respective predetermined LUTs for possible values of each of a plurality of image factors is not practical due, at least, to an amount of memory that would be required to store those predetermined LUTs. Therefore, interpolation (e.g., bilinear interpolation, trilinear interpolation, etc.) and other processing can be performed using predetermined lookup tables that are tuned for achieving high quality captured image frames at key (index, etc.) values of various image factors to produce interpolated lookup tables (including interpolated LUT values).

In current approaches, software drivers running on a general purpose processor are used to perform such lookup table processing, which can include interpolation operations to calculate interpolated values included in an interpolated LUT, and operations for formatting LUTs for use in specific ISP hardware. Also, because more than one image factor often is involved in interpolating each LUT used by an ISP, generating LUTs that are used by an ISP can include performing more than one interpolation operation per LUT.

Further, LUT interpolation and processing is done on a per-frame basis. Therefore, for cameras that capture image frames (e.g., video) at <NUM> frames per second (FPS), an associated SW driver must perform interpolation and other processing of the LUTs used by an associated ISP run <NUM> times per second. Using current approaches limits (or negatively impacts) image processing performance of a camera, and can also adversely impact power consumption of a corresponding general purpose processor (main processor, central processor, etc.) of an electronic devices including a digital camera.

<CIT> and <CIT> disclose to generate an interpolated LUT corresponding to an estimated quantitative image value by using a first LUT with a first image value and a second LUT with a second image value.

In a general aspect, a camera can include a dynamic memory, and a software driver configured to store, in the dynamic memory, a plurality of predetermined lookup tables (LUTs), a first LUT of the predetermined LUTs including first image signal processor (ISP) configuration information corresponding with a first value of a quantitative image factor, and a second LUT of the predetermined LUTs including second ISP configuration information corresponding with a second value of the quantitative image factor. The second value can be greater than the first value. The software driver can be further configured to issue an interpolation command indicating (including, specifying, etc.) a third value of the quantitative image factor corresponding with an image frame received by the ISP. The third value can be greater than the first value and less than the second value. The camera can also include a LUT processing circuit configured to receive the interpolation command, and in response to receiving the interpolation command: read the first LUT and the second LUT; and perform at least one interpolation operation to generate an interpolated LUT, the interpolated LUT can be generated based, at least, on the third value, the first LUT and the second LUT. The camera can further include an image signal processor (ISP) including a configuration register. The LUT processing circuit can be further configured to write the interpolated LUT to the configuration register.

The proposed solution thus, in particular, relates to a camera as specified in claim <NUM> and which can include an image sensor configured to capture image frames; a dynamic memory; an image signal processor, ISP, including a configuration register and a software driver. The software driver may be configured to store, in the dynamic memory, a plurality of predetermined lookup tables, LUTs, for processing image frames depending on aspects of a captured image frame and/or on a use case. A first LUT of the predetermined LUTs may include first ISP configuration information corresponding with a first value of a quantitative image factor, and a second LUT of the predetermined LUTs may include second ISP configuration information corresponding with a second value of the quantitative image factor, the second value being greater than the first value. The software drive may be further configured to issue an interpolation command including a third value of the quantitative image factor corresponding with an image frame received by the ISP, the third value being greater than the first value and less than the second value. A LUT processing circuit of the camera may be configured to receive the interpolation command, and in response to receiving the interpolation command may read the first LUT and the second LUT; and perform at least one interpolation operation to generate an interpolated LUT, the interpolated LUT being generated based, at least, on the third value, the first LUT and the second LUT. The LUT processing circuit may then be further configured to write the interpolated LUT to the configuration register of the ISP. The predetermined LUTs can be associated with specific values (key configuration values) of the qualitative image factors relating to captured and/or computed image data for an image frame, such as sensor gain, exposure time, cumulative color temperature (CCT), white balance, luminance, etc. The specific factors used can depend, at least, on the particular camera implementation. The predetermined LUTs can include configuration information for an image processing stack (e.g., of the ISP) based on, or corresponding with conditions of a scene depicted in a captured image frame and/or based on predetermined use cases, where different conditions and different use cases correspond to different ISP configuration information, such as different key configuration values of the LUTs, each LUT being associated with specific ISP configuration information so that an LUT is selectable (e.g., for use in an interpolation operation) depending on a condition of a scene and/or a use case.

In another general aspect, a method is defined in appended claim <NUM> and can include storing, in memory of a computing device, a plurality of predetermined lookup tables (LUTs), a first LUT of the predetermined LUTs including first image signal processor (ISP) configuration information corresponding with a first value of a quantitative image factor, and a second LUT of the predetermined LUTs including second ISP configuration information corresponding with a second value of the quantitative image factor. The second value can be greater than the first value. The method can further include receiving, at a LUT processing circuit, an interpolation command indicating (including, specifying, etc.) a third value of the quantitative image factor corresponding with an image frame received by the ISP. The third value can be greater than the first value and less than the second value. The method can still further include, in response to receiving the interpolation command: reading, by the LUT processing circuit, the first LUT and the second LUT; performing an interpolation operation using the third value, the first LUT and the second LUT to generate an interpolated LUT; and writing the interpolated LUT to at least one of the memory or a configuration register of the ISP.

To address the drawbacks discussed above, such as limited image processing performance and power consumption, the approaches described herein use hardware apparatuses, such as lookup table (LUT) processing circuits and/or LUT hardware engines (LUT engines) to perform LUT processing operations, such as interpolation operations (e.g., bilinear interpolation operations, trilinear interpolation operations, etc.), operations to perform packing (transformation, translation, etc.) of LUTs (e.g., to configure them for specific image signal processor (ISP) hardware configuration registers), etc..

Such approaches can achieve a significant reduction in an amount of general purpose processor cycles (e.g., of a mobile electronic device including a camera implementation such as those described herein) used in image processing (by a camera software (SW) driver). Further, these approaches can reduce power consumption by the associated general purpose processor (and overall power consumption of an electronic device implementing such approaches) by performing computationally intensive operations (e.g., interpolation operations) using a special purpose LUT processing circuit or LUT engine, such as the example implementations described herein. For instance, executing arithmetic operations on a general-purpose CPU (such as for LUT interpolation, etc.) can consume more power than the same operations performed on fixed-function hardware (e.g., an LUT processing circuit) specifically designed to perform (execute, etc.) such operations. For instance, general-purpose CPUs can consume more power because they utilize a complex, generic execution pipeline for instruction fetch-decode-execute sequences. Therefore, an LUT interpolation process (operation, etc.), when performed using a general purpose CPU, will be performed as a series of general CPU instructions that are processed this execution pipeline.

In contrast, specialized hardware (e.g., the LUT processing circuits and/or LUT engines described herein), may not include a complex pipeline and, instead, can be configured to perform a set of fixed-functions in hardware that does not require complex programming and, therefore, can be implemented using less complicated circuitry than a general purpose CPU. Such less complicated circuit can include significantly less hardware gates than are used in general purpose CPU, which, as a result, uses less power (e.g., less power per operation. ) Further, the implementations described herein can also significantly improve image signal processing performance, allowing for improvements in image quality (e.g., as a result of possible additional image processing), increases in frame rates for captured image frames, and so forth.

<FIG> is a block diagram illustrating a camera <NUM> (e.g., a digital camera) that can be included, for example, in a mobile computing device, such as the smartphone <NUM> shown in <FIG> below. The camera <NUM> can, of course, be implemented in other electronic devices, such as tablet computers, notebook computers, laptop computers, action cameras, body cameras, etc. The elements of the camera <NUM> (as well as the elements of other camera implementations described herein) are given by way of example and for purposes of illustration. It will be appreciated that, a camera implementing the approaches described herein can, include other elements, can have elements replaced with different elements, can have multiple elements combined into a single integrated element, can have elements omitted, and so forth.

As shown in <FIG>, the camera <NUM> includes a general purpose processor <NUM> (e.g., a general purpose CPU), a memory <NUM>, such as dynamic random access memory (DRAM), a LUT processing circuit <NUM>, an ISP <NUM>, and an image sensor <NUM> (e.g., a CMOS sensor, a charge-coupled device (CCD) sensor, etc.). In <FIG>, communication paths (operational couplings) are shown between various elements of the camera <NUM>. These communication paths are illustrative and, in some implementations, other communication paths or operational relationships can be included or defined in the camera <NUM>.

The camera <NUM> of <FIG> includes a camera SW driver <NUM> running on the general purpose processor <NUM>. In this example, the camera SW driver <NUM> can be configured, e.g., as part of an initialization process, to store, in the memory <NUM>, a plurality of predetermined lookup tables (LUTs) as part of the LUTs <NUM>. For instance, the camera SW driver <NUM> can be configured to store lookup tables that have been previously generated (e.g., by image quality engineers) and that include ISP configuration information for various aspects of image capturing and processing by the ISP <NUM> and/or the image sensor <NUM>. For instance, these predetermined LUTs can be associated with specific values (key configuration values) of qualitative image factors relating to captured and/or computed image data for an image frame, such as sensor gain, exposure time, cumulative color temperature (CCT), white balance, luminance, etc. The specific factors used will depend, at least, on the particular camera implementation. In some implementations, the LUTs <NUM> can also include intermediate interpolated LUTs, which are written by the LUT processing circuit <NUM> and used in interpolations where more than one factor (and multiple interpolation operations) is (are) used to generate an interpolated LUT (e.g., ISP configuration data) that is written to the configuration registers <NUM> for use by the ISP <NUM>. Depending on the particular implementation, such LUTs (predetermined LUTs and/or interpolated LUTs) can be one-dimensional, two-dimensional, and/or three-dimensional, and can include integer values, floating point values, etc..

Such predetermined LUTs can include configuration information for the image processing stack (e.g., of the ISP <NUM>) that can correspond with specific conditions of a scene depicted in a captured image frame and/or with specific use cases, where different conditions and different use cases correspond to different key configuration values of the LUTs, each LUT being associated with a specific key configuration value so that an LUT is selectable (e.g., for use in an interpolation operation) depending on a condition of a scene and/or a use case. In particular, the predetermined LUTs can include configuration information for the image processing stack (e.g., of the ISP <NUM>) based on various (e.g., potential) scene conditions and/or use cases (e.g. cloudy, sunny, video, preview, etc.) corresponding, for each predetermined LUT, with a respective key configuration (factor) value of the set of key configuration values for each different factor that is used by a given implementation. However, in some implementations, because such scene factors are continuous variables, interpolation between the predetermined tables may need to be performed when a computed factor for a captured image frame is between key values (e.g., greater than one key factor value and less than a second key factor value) for corresponding predetermined lookup tables. In the camera <NUM> (and similarly in the camera <NUM> of <FIG> and the camera <NUM> of <FIG>), such interpolation operations can be performed by the LUT processing circuit <NUM> (the LUT processing circuit <NUM>, or the LUT engine <NUM>), e.g., using one or more special purpose LUT arithmetic and logic units (ALUs), such as are further described below.

As shown in <FIG>, the memory <NUM> can also include a command buffer <NUM>, which can receive LUT processing commands (e.g., interpolation commands, bitwise operation commands, shift commands, etc.) from the camera SW driver <NUM>. The LUT processing circuit <NUM> can then read commands for execution from the command buffer <NUM> and perform associated LUT operations (e.g., using special purpose LUT ALUs included in the LUT processing circuit <NUM>).

The ISP <NUM> of the camera <NUM>, as shown in <FIG>, includes configuration registers <NUM>, which can be used to store LUTs (e.g., interpolated LUTs) that include configuration information (e.g., corresponding with qualitative images factors that were determined based on a previously captured image frame) to capture and process a next image frame. In some implementations, the LUTs stored in the configuration registers <NUM> can be updated after each image frame is captured. That is, LUT processing, such as described herein, can be performed for every frame in a sequence of captured image frames and configuration information in the configuration registers <NUM> for the ISP <NUM> can be updated after each image frame capture. For instance, if the camera <NUM> captures image frames at <NUM> frames per second (fps), interpolation for all configuration LUTs (using one or more factors) can be performed <NUM> times per second (after each image frame is captured) and the configuration registers <NUM> can be updated with the newly interpolated LUTs. For instance, LUTs used in image processing can each include hundreds of values, or more, and multiple LUTs can be used for processing each image (e.g., at a frame capture rate). As an example, for an LUT that includes <NUM> entries, interpolation of that one LUT would involve <NUM>,<NUM> interpolation calculations per second (e.g., <NUM> x <NUM>, for an image frame capture rate of <NUM> fps). The number of calculations can be further multiplied in implementations where LUT interpolation is cascaded. For instance, if tri-linear interpolation is performed, seven interpolation operations can be performed to produce a final interpolated LUT (e.g., with each interpolation operation including sixty-thousand calculations using the above example).

<FIG> is a block/state flow diagram <NUM> illustrating operation of a camera that includes hardware lookup table (LUT) processing, such as described herein. In some implementations, the state flow diagram <NUM> can be implemented by the cameras disclosed herein (e.g., the cameras <NUM>, <NUM> and <NUM>). However, for purposes of discussion of <FIG>, the state flow diagram <NUM> will be described generally, without specific reference to any particular camera implementation.

As shown in <FIG>, the state flow diagram <NUM>, which implements a feedback loop, can include an ISP state <NUM> that operates based on configuration information (e.g., LUTs) stored in configuration registers <NUM>. In the state flow diagram <NUM>, at the ISP state <NUM>, an image frame (e.g., pixel data for the image frame) can be captured and statistics on the captured image frame can be determined, which can then be communicated as image and statistics <NUM> for processing and analysis at state <NUM>.

At state <NUM>, auto-tuning algorithms (e.g., auto-focus, auto-exposure and auto-white-balance), which can be referred to as 3A algorithms <NUM>, can be performed on the image and statistics <NUM> (e.g., by a camera SW driver). The 3A algorithms <NUM> can be configured to calculate qualitative image factors (for each captured image frame), such as the image factors described herein. The calculated image factors can then be used, as part of LUT hardware processing <NUM>, to generate (and in some implementations, pack or transform) LUTs that adjust the configuration information included in the configuration registers <NUM> based a current scene (as represented by the image and statistics <NUM>). This updated (adjusted) configuration information can be included in register updates <NUM>. The register updates <NUM> can then be communicated to the configuration registers <NUM> via a feedback path <NUM>, and the process of the state flow diagram <NUM> can be repeated for a new (next) captured image frame.

As described herein, the LUT hardware processing <NUM> can include performing LUT interpolation operations on various LUTs (e.g., predetermined LUTs and/or intermediate interpolated LUTs). Depending on the particular implementation, each LUT that is used at the ISP state <NUM> (e.g., that is stored in the configuration registers <NUM>) can be interpolated using one or more qualitative image factors, such as using the approaches described herein.

In addition to LUT interpolation operations, the LUT hardware processing <NUM> can include performing packing operations on one or more LUTs (e.g., interpolated LUTs). For instance, in some implementations, camera ISP hardware can require LUTs including configuration information to be in a specific format and/or arrangement. For example, in some implementation, specific ISP hardware can expect different bit widths LUT entries, or that more than one LUT be packed (merged) into a single (combined) LUT. In some implementations, each LUT entry of such a packed or merged LUT could include different values from the LUTs that are packed (merged) together. Packing operations on LUTs can be performed using arithmetic operations, bitwise (logical) operations, and/or shift operation. Accordingly, special purpose LUT ALUs that are used for performing interpolation operations can also be configured to perform LUT packing operations.

<FIG> are diagrams, <NUM> and <NUM> respectively, that illustrate an example process of LUT interpolation that can be implemented by an LUT processing circuit. The LUT interpolation process shown in <FIG> illustrates LUT interpolation for a single value of one qualitative image factor X (e.g., an interpolation factor X <NUM>), which can be a value of a qualitative image factor, such as those factors described herein. It will be appreciated, however, that the process of <FIG> (or a similar process) can be implemented to perform a second interpolation operation based on a second qualitative image factor to produce a final interpolated LUT that is used as configuration information for an ISP. Such multifactor LUT interpolations can be performed in a number of manners, such as using a hierarchical flow to generate intermediate LUTs from predetermined LUTs, and then using intermediate LUTs and/or other predetermined LUTs to generate a final interpolated LUT that is written to configuration registers of an associated ISP. In some implementations, values of qualitative image factors (e.g., key values and interpolation factors) can be normalized to have values between zero and one, though other approaches can be used.

As indicated above, <FIG> illustrate an LUT interpolation process using a value of an image factor X (e.g., the interpolation factor X <NUM>). In this example, the interpolation factor X <NUM> can be used, depending on the computed value of the interpolation factor X <NUM>, as a ratio for interpolating (e.g., bilinear interpolation) between configuration information included in predetermined LUTs <NUM>, <NUM>, <NUM> and <NUM>. In the example, the predetermined LUTs <NUM>, <NUM>, <NUM> and <NUM> are respectively associated with key values of A, B, C and D of the factor X, as is shown in <FIG>. As is also shown in <FIG>, the key values A, B, C and D, for this example, define interpolation factor ranges <NUM> illustrated by the horizontal, doubled-ended arrows in <FIG>. For instance, a first interpolation factor range is between A and B, a second interpolation factor range is between B and C, and a third interpolation factor range is between C and D. In this example A<B<C<D and X is between B and C. That is B<X<C.

Accordingly, as shown by the diagram <NUM> of <FIG>, for this example, because the image factor X falls within the interpolation factor range between key values B and C, interpolation between the predetermined LUT <NUM> associated with the key value B and the predetermined LUT <NUM> associated with the key value C is performed using the interpolation factor X <NUM>. That is, in this example, the interpolation factor X <NUM> can being used as a ratio for performing (bilinear) hardware interpolation <NUM> on the predetermined LUTs <NUM> and <NUM> to produce an interpolated LUT <NUM> (which can be an intermediate interpolated LUT or a final interpolated LUT that is used to update ISP configuration registers. In some implementations, such bilinear interpolation can be performed by special purpose LUT ALUs that can be configured to perform bilinear interpolation computations, such as discussed below with respect to, at least, <FIG>.

<FIG> is a block diagram illustrating an implementation of a camera <NUM> including an LUT processing circuit <NUM> (LUT processing hardware). As with the camera <NUM>, the camera <NUM> includes a camera SW driver <NUM> (e.g., running on a general purpose processor), memory <NUM> (DRAM) and camera/ISP hardware <NUM>. Also, as discussed with respect to the camera SW driver <NUM> of the camera <NUM>, the camera SW driver <NUM> can be configured to write a plurality of predetermined LUTs, such as the LUTs <NUM> and <NUM> of the example of <FIG>, e.g., as part of an initialization process of the camera <NUM>. In some implementations, the camera SW driver <NUM> can also be configured to issue an interpolation command (Interp_command) or other LUT processing commands, which can be directly communicated to the LUT processing circuit <NUM> or, as discussed herein, could be placed in a command buffer in the memory <NUM>, and then read from the command buffer by the LUT processing circuit <NUM> prior to execution. Further referring to the LUT interpolation example of <FIG>, the interpolation command issued by the camera SW driver <NUM> for performing such an interpolation operation could take the form of: <MAT> where x is the interpolation factor X <NUM>, LUT1_addr is the address of the predetermined LUT <NUM>, LUT2_addr is the address of the predetermined LUT <NUM>, and dst_LUT is the address of the intermediate LUT <NUM> and/or the address in the configuration LUTs <NUM> (configuration registers) of the camera / ISP hardware <NUM> of the camera <NUM> for writing the interpolated LUT.

In response to receiving the interpolation command (e.g., from the camera SW driver <NUM> or a command buffer in the memory <NUM>), the LUT processing circuit <NUM> can be configured to read the LUT <NUM> and the LUT <NUM> from the memory <NUM>, e.g., using a programming direct memory access (DMA) controller (programming DMA <NUM>). In some implementations, the LUT processing circuit <NUM> can be implemented in a programming DMA (the programming DMA <NUM>), or a limited programming DMA can be implemented as part of the LUT processing circuit <NUM>. In some implementations, a programming DMA can be excluded, and the LUT processing circuit <NUM> can include a read DMA and a write DMA for, respectively accessing (reading) LUTs and writing LUTs.

In the example implementation of <FIG>, the programming DMA <NUM> can be a specialized DMA that can read and execute register programming commands that are stored in memory. For instance a register command can be a register write, poll, etc. command (e.g. write_register(register_address, value)), which can direct the DMA to perform an operation on the register space of the ISP hardware (e.g., writing/polling of ISP hardware registers without direct intervention of a main CPU). For instance, in some implementations, one supported command of the programming DMA <NUM> can be to accept an address of an LUT the memory <NUM> (e.g., an address of the LUT <NUM>) that is to be written to a hardware LUT (e.g., an ISP configuration LUT <NUM>) and coordinate writing of the LUT, taking into account any dependencies (e.g., waiting from the LUT to be generate before initiating writing the LUT).

As shown in <FIG>, one or more special purpose LUT ALUs <NUM> can be implemented in the LUT processing circuit <NUM>. In this example, after reading the LUTs <NUM> and <NUM>, the LUT ALUs <NUM> can perform interpolation operations (computations, operations, etc.) consistent with the interpolation command issued by the camera SW driver <NUM> to generate an interpolated LUT. These operations, depending the number of LUT ALUs <NUM>, can be performed sequentially and/or in parallel. In this example, each entry of the interpolated LUT can be computed from corresponding entries in the LUTs <NUM> and <NUM>, and the interpolation factor X <NUM> using the following equation: <MAT> In the above equation, LUT_out_val represents values of the interpolated LUT being generated, LUT1_val represents corresponding values in the LUT <NUM>, x represents the interpolation factor X <NUM>, and LUT2_val represents corresponding values in the LUT <NUM>. As discussed above, in this example, the values of the qualitative image factor (key values corresponding with the LUTs, and the interpolation factor X) can be normalized to be between <NUM> and <NUM>, though specific values (e.g., ISP configuration values) included in the LUTs may not be normalized. In some implementations, additional interpolation commands, or other LUT processing commands (e.g., packing commands) can be received by the LUT processing circuit <NUM> (e.g., directly from the camera SW driver <NUM> or read from a command buffer in the memory <NUM>). Once the commands for generating a final interpolated LUT are completed, the final interpolated LUT can be written as a configuration LUT to the LUT storage are <NUM> of the camera / ISP hardware <NUM>.

For instance, in some implementation, multiple interpolation commands for generating an interpolated LUT using multiple interpolation operations and interpolation factors (computed values of qualitative images factors based on a captured image frame) can be expressed as a chained command. For instance, a command for generating an interpolated LUT based on multiple image factor (interpolation) values, e.g., one interpolation operation based on sensor gain (e.g., taking sensor gain into account) and another interpolation operation based on luminance (e.g., taking luminance into account) can be issued by the camera SW driver <NUM> as a chained command in the form of: <MAT> Where, in this example, lut1 and lut2 are interpolated using the interpolation factor x to generate an intermediate interpolated LUT lut3_out. After lut3_out is generated, the LUT processing circuit <NUM> can, in accordance with the above chained command, interpolate between lut3_out and lut4 using the interpolation factor y to generate the final interpolated LUT lut_out, and then lut_out can be written to lut_register_addr (e.g., in the LUT registers <NUM> of the camera <NUM>).

Using such command chaining can allow for the camera SW driver <NUM> to express all requested interpolation operations for a captured image frame as a sequence of interpolation commands that can be interwoven with other register programming commands expressed by the camera SW driver <NUM>. In some implementations, such as in the camera <NUM>, dependencies between the LUT commands (e.g., interpolation and packing commands) and ISP configuration register write commands can be enforced by the LUT processing circuit <NUM>, e.g., such as by using a wait for interrupt request feature (e.g., wait for irq) of the programming DMA <NUM>.

<FIG> is a block diagram illustrating another implementation of a camera <NUM> including an LUT processing circuit (LUT engine <NUM>). As with the cameras <NUM> and <NUM>, the camera <NUM> includes a camera SW driver <NUM> (e.g., running on a general purpose processor), memory <NUM> (DRAM) and camera/ISP hardware <NUM> (including configuration LUTs <NUM>). Also, as discussed with respect to the camera SW driver <NUM> of the camera <NUM> and the camera SW driver <NUM> of the camera <NUM>, the camera SW driver <NUM> can be configured to write a plurality of predetermined LUTs, such as the LUTs <NUM> and <NUM> of the example of <FIG>, e.g., as part of an initialization process of the camera <NUM>. In some implementations, the camera SW driver <NUM> can also be configured to issue an interpolation command (Interp_command) or other LUT processing commands, which can be directly communicated to the LUT engine <NUM> or, as discussed herein, could be placed in a command buffer in the memory <NUM>, and then read from the command buffer by the LUT engine <NUM> prior to execution.

In some implementations, the camera SW driver <NUM>, the memory <NUM> and the camera / ISP hardware <NUM> can operate similar to the corresponding elements of the camera <NUM>. Accordingly, for purposes of brevity and clarity, those elements of the camera <NUM> will not be described in further detail again with respect to <FIG>.

As can be seen from a comparison of <FIG> with <FIG>, the camera <NUM> can differ from the camera <NUM> in at least the following ways. For instance, the LUT engine <NUM> is implemented as a standalone LUT processing circuit from a programming DMA <NUM> of the camera <NUM>. Further, the LUT engine <NUM>, as compared to the LUT processing circuit <NUM>, includes, in addition to the LUT ALUs <NUM>, DMAs <NUM> and an operation control circuit <NUM>. In some implementations, the DMAs <NUM> can include at least one basic read DMA for accessing LUTs in the memory <NUM>, and a basic write DMA for writing LUTs to the/ <NUM>, or to the LUT configuration register <NUM> of the camera / ISP hardware <NUM>.

The LUT engine <NUM> of the camera <NUM> can be configured to work in conjunction with the programming DMA <NUM>, e.g., to coordinate writing of LUTs to the configuration LUTs <NUM>. For instance, the LUT engine <NUM> can be configured to send an interrupt request IRQ to the programming DMA <NUM> when an interpolated (and packed) LUT is ready to be written to the configuration LUTs <NUM> of the camera / ISP hardware <NUM>.

In some implementations, the operational control circuit <NUM> can be configured to decode more complicated command structures than the LUT processing circuit <NUM>. For instance, the operation control circuit <NUM> can be configured to decode and schedule execution of operations of the compute kernel <NUM> illustrated in <FIG>. As a result, supported LUT processing operations of the LUT engine <NUM> can be more complicated, flexible and computationally sophisticated than supported LUT processing operations of the LUT processing circuit <NUM>. Further, the operation control circuit <NUM> can also be configured to initiate operations of the programming DMA <NUM> (e.g., via interrupt requests), as well as schedule compute kernel execution (computations) on available ALUs of the LUT ALUs <NUM>. As discussed herein, the LUT ALUs <NUM> can include a plurality of parallel implemented special purpose LUT ALUs that are configured to execute (perform) LUT processing operation (e.g., interpolation operations, bitwise operations, shift operations, etc.) for processing LUTs for use in image processing (e.g., to generate configuration LUTs for use by the camera / ISP hardware <NUM> of the camera <NUM>).

<FIG> is a diagram of a compute kernel <NUM> that can be implemented in, for example, the camera of <FIG>. In other implementations, a compute kernel that is executed by an LUT processing circuit, such as the LUT engine <NUM> of <FIG>, can have other forms. For instance, for purposes of example and illustration, the compute kernel <NUM> incudes only a single instruction (e.g., an interpolation instruction using a single interpolation factor). In other instances, the compute kernel <NUM> could include multiple (e.g., chained) interpolation instructions based on multiple interpolation factors, bitwise (logical) instructions, shift instructions, etc. The specific instructions and fields included in such a compute kernel will depend on the specific implementation and/or the instructions an LUT processing circuit (e.g., the LUT engine <NUM>) is being instructed to execute.

For the compute kernel <NUM>, with further reference to the camera <NUM> of <FIG>, in this example, the camera SW driver <NUM> can generate the compute kernel <NUM> (as well as other compute kernels and instructions that are sent to the programming DMA <NUM>, such as ISP configuration register write instructions). The camera SW driver <NUM> can then communicate the compute kernel <NUM>, as well as other compute kernels, to the LUT engine <NUM>. The operation control circuit <NUM> can schedule (control, direct, etc.) execution of any compute kernels received from the camera SW driver <NUM> (e.g., using the special purpose LUT ALUs <NUM> and the DMAs <NUM>, which can include one or more read DMAs and one or more write DMAs).

As illustrated by the compute kernel <NUM> of <FIG>, such compute kernels can have a simple format. However, as noted above, such compute kernels (e.g., the compute kernel <NUM>) can be extended to include additional operations, which can be used in conjunction with adding and/or changing functionality and/or capabilities of an LUT engine (e.g., the LUT ALUs <NUM> of the LUT engine <NUM>).

In some implementations, LUT compute kernels can be executed using element-wise computations. That is, a compute kernel, such as the compute kernel <NUM>, can be executed by performing computations (e.g., interpolation computes) one at a time (e.g., using single corresponding entries from multiple LUTs, such as predetermined LUTs, and/or intermediate interpolated LUTs). Accordingly, using the approaches described herein, parallelization of LUT processing operations can be implemented by scheduling, e.g., by the operation control circuit <NUM>, such element-wise computations across multiple, parallel implemented LUT ALUs <NUM> for concurrent processing of interpolation (or other) computations. In an example implementation, the LUT engine <NUM> could be used to interpolate between predetermined LUTs <NUM> and <NUM> using a computed interpolation factor (ratio) X. In this example, each of the predetermined LUTs can include one-thousand entries. If the LUT engine <NUM> of the camera <NUM> includes one-thousand special purpose LUT ALUs <NUM>, interpolation (e.g., bilinear interpolation) between the LUTs <NUM> and <NUM> could then be completed in a single operation cycle, with each of the one-thousand LUT ALUs <NUM> concurrently completing a single element-wise interpolation computation. Accordingly, parallelization using multiple ALUs can be used to further improve image processing capabilities (power, bandwidth, speed, etc.).

Referring again to <FIG>, the elements of the example compute kernel <NUM> are briefly described below. As shown in <FIG>, the compute kernel <NUM> is illustrated using a binary layout of a linear interpolation compute kernel that the LUT engine <NUM> can execute (e.g., as administered by the operation control circuit <NUM>). As noted above, the compute kernel <NUM> is given by of example, and for purposes of illustrating a compute kernel for a single command. In some implementations, additional fields related to additional commands (e.g., for additional interpolation operations, LUT packing operations, etc.) can be included in such a compute kernel.

In the example of <FIG>, the compute kernel <NUM> includes an Op field <NUM>, a size field <NUM>, a numargs field <NUM>, a len field <NUM>, a first (lut1) address field <NUM>, a second (lut2) address field <NUM>, a third (lut3) address field <NUM>, a ratio (factor) <NUM> and an operation (Ierp) field <NUM>, that lists the fields <NUM>, <NUM>, <NUM> and <NUM> as arguments. In this example, the op field <NUM> can identify the type of command(s) included in the compute kernel <NUM>. The size field <NUM> can specify the total size (e.g., in bytes, etc.) of the compute kernel <NUM>. The numargs field <NUM> can specify a number of argument entries for the compute kernel <NUM>. The len field <NUM> can specify a number of the kernel computation entries for the compute kernel <NUM>.

For this example, which can be for a bilinear interpolation operation, the first address field <NUM> can be a memory address of a first input LUT (e.g., LUT <NUM>). The second address field <NUM> can be a memory address of a second input LUT (e.g., LUT <NUM>). The third address field <NUM> can be an address of an output LUT (e.g., an intermediate interpolated LUT <NUM>) and/or an address of an ISP configuration (final interpolated) LUT <NUM>, e.g., in configuration registers of the ISP <NUM>. The ratio field <NUM> can be an interpolation factor (e.g., X), such as those discussed herein. And the operation field <NUM> can be, in this example, a bilinear interpolation opcode (Ierp) that is executable (e.g., directly executable) by the special purpose LUT ALUs <NUM>.

<FIG> is a flowchart illustrating a method <NUM> for image processing that can be implemented, for example, by the cameras of <FIG>, <FIG> and <FIG>. It will be appreciated, however, that the method <NUM> can be implemented by cameras having other configurations. For purposes of discussion and illustration, the method <NUM> will be described with further reference to, at least, <FIG> and <FIG>.

The method <NUM> includes, at block <NUM>, performing an initial camera setup. For instance, in the camera <NUM>, at block <NUM>, the camera SW driver <NUM> can write a plurality of predetermined LUTs (e.g., LUTs <NUM>) to the memory <NUM>, such as the predetermined LUTs as described herein (e.g., predetermined LUTs for adjusting chrominance, luminance, gamma, brightness, etc.). Further, the camera SW driver <NUM> can instruct the LUT processing circuit <NUM> (or other circuit, such as a programming DMA) to write (copy, store, etc.) initial LUTs in the configuration registers <NUM> of the ISP <NUM>, as an initial image frame capture configuration. In other implementations, the initial setup at block <NUM> can be performed in other ways.

At block <NUM>, the method <NUM> includes capturing an image frame including the image and statistics <NUM> (e.g., pixel data and quantitative values corresponding with the captured image frame, such as brightness, video, preview, etc.). The ISP <NUM> can provide the statistics <NUM> to the camera SW driver <NUM>, which can execute the 3A algorithms <NUM> on the image and statistics <NUM> and determine one or more value of qualitative image factors that can by used to adjust the configuration of the ISP <NUM> based on the captured scene of the image of the image and statistics <NUM>.

At block <NUM>, the camera SW driver <NUM> can, based on the calculated factors, send one or more interpolation commands and/or other LUT processing commands (e.g., packing operations) to the LUT processing circuit <NUM>, such as in the form of a single interpolation operation, a chained interpolation operation, a compute kernel (such as the compute kernel <NUM>), and so forth. As noted above, the factors computed can depend on the specific camera implementation and/or on the image and statistics <NUM> of the captured frame. For instance, the factors can include values (e.g., to be used to adjust the ISP <NUM> configuration) corresponding with sensor gain, exposure time, cumulative color temperature, white balance, etc..

At block <NUM>, the LUT processing circuit <NUM> can then execute the interpolation commands (e.g., by executing interpolation operations) and/or the other LUT processing commands (e.g., packing operations), such as using the approaches described herein, or using other approaches as appropriate for a specific implementation. For instance, LUTs that will be used in the interpolation operations can be read from the memory <NUM> using an identity operation and one or more LUT ALUs can perform LUT processing in correspondence with the commands (and factors) received from the camera SW driver <NUM>. As noted herein, multiple LUT interpolation operations can performed in generating a single interpolated LUT that is ultimately written to the ISP <NUM>'s configuration registers <NUM>.

After completion of LUT processing by the LUT processing circuit <NUM> at block <NUM>, the method <NUM> includes, at block <NUM>, writing interpolated and processed LUTs (e.g., intermediate or final LUTs) to the memory <NUM> and/or the configuration registers <NUM> to update the configuration information for the ISP <NUM>. As shown, in <FIG>, after completion of the operations of block <NUM>, the method <NUM> can return to block <NUM>, and another image frame capture and LUT processing sequence (blocks <NUM> to <NUM>) can be performed (e.g., where this sequence is repeated at a frame rate of the associated camera, such as <NUM> FPS, <NUM> FPS, etc.).

<FIG> shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

The processor <NUM> can be a semiconductor-based processor. The memory <NUM> can be a semiconductor-based memory. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

Thus, for example, expansion memory <NUM> may be provided as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention which is defined by the appended claims.

Claim 1:
A camera comprising:
a dynamic memory;
an image signal processor, ISP, including a configuration register;
a software driver configured to:
store, in the dynamic memory, a plurality of predetermined lookup tables, LUTs, a first LUT of the predetermined LUTs including first ISP configuration information corresponding with a first value of a quantitative image factor, and a second LUT of the predetermined LUTs including second ISP configuration information corresponding with a second value of the quantitative image factor, the second value being greater than the first value; and
issue an interpolation command indicating a third value of the quantitative image factor corresponding with an image frame received by the ISP, the third value being greater than the first value and less than the second value;
a LUT processing circuit configured to receive the interpolation command, and in response to receiving the interpolation command:
read the first LUT and the second LUT; and
perform at least one interpolation operation to generate an interpolated LUT, the interpolated LUT being generated based, at least, on the third value, the first LUT and the second LUT,
wherein the LUT processing circuit is further configured to write the interpolated LUT to the configuration register of the ISP, wherein:
the interpolation command is included in a compute kernel that is executed by the LUT processing circuit; and
the LUT processing circuit includes an operation control circuit and at least one LUT arithmetic and logic unit (ALU)
the operation control circuit being configured:
to decode the compute kernel; and
schedule LUT processing operations included in the compute kernel for execution by the at least one LUT ALU; and
the at least one LUT ALU is configured to execute the scheduled LUT processing operations, including the interpolation operation.