Patent Description:
With the increase in the use of cameras in electronic devices, such as mobile communication devices, there is also an increase in memory and processing requirements. Various factors may not only affect the quality of an image, but the processing requirements to process the image. For example, warping of an image recorded by the camera may often occur, such as for example by a distortion created by a lens of the camera. Warping may result in an image being rotated for example.

Image processing may be used to correct undesirable image warping, or introduce image warping as desired. That is, warping is a remapping of data associated with pixels of a captured image that may cause distortion, or may cause anti-distortion to correct unintentional distortion, such as distortion created by a lens. Therefore, where a lens of a mobile communication device such as a smart phone or tablet causes distortion, warping of the received image associated with input data can be applied to generate output data that is corrected. Because of the warping, a high bandwidth is typically required at the input of an image processing function for the camera to process the warped image.

Typically input and output tiles are described as rectangular tiles which includes a region of interest (ROI) in an input or output frame. For large warping effects (e.g., large rotations), the input bandwidth (BW) of a warped image is much larger than output BW of a corrected image. In a case of limited on-chip memory, smaller tile sizes exacerbates the problem because each tile has an associated overhead.

Accordingly, there is a need to reduce the input BW for an image captured by a camera to address warping of an image.

<CIT> relates to a method and apparatus for transforming anon-linear lens-distorted image.

<CIT> relates to a method and system for correcting a distorted input image.

<CIT> relates to an image processing apparatus.

<CIT> relates to a method and device for image manipulation.

A method of implementing memory transfers for image warping in an electronic device according to claim <NUM> is disclosed.

According to another embodiment, an electronic device according to claim <NUM> is disclosed.

A non-transitory computer-readable storage medium having data stored therein representing instructions executable by a processor according to claim <NUM> is also disclosed.

Other features will be recognized from consideration of the Detailed Description and the Claims, which follow.

While the specification includes claims defining the features of one or more implementations of the invention that are regarded as novel, it is believed that the circuits and methods will be better understood from a consideration of the description in conjunction with the drawings. While various circuits and methods are disclosed, it is to be understood that the circuits and methods are merely exemplary of the inventive arrangements, which can be embodied in various forms. Therefore, specific structural and functional details disclosed within this specification are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the inventive arrangements in virtually any appropriately detailed structure. Further, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of the circuits and methods.

The circuits and methods set forth below reduce the bandwidth required to address image warping for imaging applications of mobile electronic devices. According to some embodiments, all of the required pixels are enclosed in a polygon rather than a vertical tile to reduce the bandwidth. Enclosing the pixels in a polygon may reduce the bandwidth requirement significantly, for example, the bandwidth requirement may be reduced by up to <NUM> times for Ultra High Definition (UHD) images processed at 60fps and having warping at <NUM> degrees. Reducing the bandwidth makes real-time processing for image warping for high resolution and high frame rates and multi-frame use cases feasible. Methods of transferring pixels inside a polygon are also disclosed.

According to one implementation, a convex hull is identified for given set of pixels, where the pixels are transferred inside the polygon formed by convex hull. According to another implementation that does not form part of the claimed invention, a minimum bounding box, such as a rectangle, is determined, where the pixels are transferred inside a bounding box. A closed form expression may be used to reduce the complexity of convex hull calculations for rotations. Other embodiments may use mesh mode transfers, where a look up table is used to map output pixel indices to implement pixel indices. The various circuits and methods reduce memory bandwidth and power consumption in image processing and allow for real-time image processing to address image warping.

Turning first to <FIG>, an exemplary block diagram of an electronic device for implementing memory transfers for image warping is shown. In particular, a device <NUM> comprises a processor <NUM> coupled to an image sensor <NUM>. The device <NUM> could be any type of electronic device, or a component of an electronic device such as an integrated circuit of a device or system, adapted to capture a digital image. The image sensor <NUM> could be a digital camera, or any portion of a digital camera, such as a lens or other image sensing element. While a single image sensor is shown, it should be understood that a plurality of image sensors could be employed. The processor could be coupled to a display <NUM> for displaying a captured image, and more particularly, displaying a digital image having enhanced image quality. The processor <NUM> is a hardware component (i.e., integrated electronic circuit) that executes a program. The processor <NUM> could be an ARM processor, an X86 processor, a MIPS processor, a graphics processing unit (GPU), a general purpose GPU, or any other processor configured to execute instructions stored in a memory. The processor <NUM> could be implemented in one or more processing devices, where the processors may be different. For example, the electronic device could include a central processing unit (CPU) as well as a GPU for example. The processor may implement an operating system (OS) that accesses software on the memory and receives various inputs, such as a camera for example. As will be described in more detail below in reference to <FIG>, the processor, alone or in combination with other elements, can introduce or compensate for image warping.

The processor <NUM> may also be coupled to a memory <NUM> that enables storing information related to various frames of an image, or resulting digital images after introducing or compensating for warping. While a single memory is shown, it should be understood memory <NUM> could be implemented as a plurality of different memories, as will be described in more detail below. The memory <NUM> could include any type of memory, such as a solid-state drive (SSD) or Flash memory for example, or any other memory element that provides long term memory, where the memory could be any type of internal memory of the electronic drive or external memory accessible by the electronic device. That is, while single memory <NUM> is shown as a part of the electronic device, it should be understood that the memory or a portion of the memory could be implemented external to the electronic device. A user interface <NUM>, which may be separate from the display, or also may be a part of, or responsive to, the display, is also shown. The processor <NUM> may also be coupled to other elements that receive inputs or enable the capturing of a digital image. For example, an inertial measurement unit (IMU) <NUM> can provide various information related to the motion or orientation of the device <NUM>. The processor <NUM> may also receive input by way of an input/output (I/O) port <NUM> or a transceiver <NUM> coupled to an antenna <NUM>. A battery <NUM> may be implemented to provide power to the processor and other elements of the device <NUM>.

Turning now to <FIG>, an exemplary circuit <NUM> implemented in an electronic device for implementing memory transfers to introduce or compensate for image warping is shown. The circuit <NUM> may be implemented using elements of the device of <FIG>, such as the processor <NUM> and the memory <NUM> for example. The circuit <NUM> comprises a controller <NUM> coupled to receive transformation information from a transformation circuit <NUM>. The transformation information enables the transfer of pixels of an input image to an output image to introduce warping or compensate for a warping of an image for example. As will be described in more detail below, the transformation information enables the remapping of pixels of an input image to an output image. The controller <NUM> is also coupled to a direct memory access (DMA) circuit <NUM> which receives data which may be an input image from a memory, such as a dynamic random access memory (DRAM) <NUM>. The controller also controls an input on-chip memory <NUM> and a warp compute circuit <NUM> that is coupled to a memory <NUM>. The input on-chip memory may receive input image data stored in DRAM <NUM> by way of the DMA circuit <NUM>. The warp compute circuit <NUM> computes the warping of an input image, as shown for example in <FIG> and <FIG>, and generate an output image that is stored in a memory <NUM>. The output image may be a corrected image of a warped image generated at an input or an image having warping that is intentionally introduced.

Turning now to <FIG>, a rotation of an input tile is shown. According to the example of <FIG>, assuming a <NUM> degree rotation of the image defined by a top left corner A0, a top right corner A1, a bottom right A2 and a bottom left corner A3, where a width (W) is defined between A0 and A1 and a height (H) is defined between A0 and A3, the received image which may be warped to have a <NUM> degree rotation as shown. As a result of the rotation, the width W' and the height H' of an area <NUM>, such as a matrix for storing pixels associated with the image, is increased. For example, for an output tile with no rotation having <NUM> corners, A0=[<NUM>,<NUM>], A1=[W-<NUM>, <NUM>], A2=[W-<NUM>, H-<NUM>], and A3=[<NUM>, H-<NUM>], the corners after a rotation of Θ (theta) would be defined as Bk=T*Ak, where:
<MAT> <MAT> <MAT> <MAT> and <MAT>.

As can be seen in <FIG>, a larger matrix as defined by W' and H' would be required to store the pixel data associated with the image. By compensating for the unintentional warping, the data associated with the image can be stored more efficiently.

According to various implementations, the circuit of <FIG> enables the transferring of data associated with an input image to points within a boundary, such as a convex hull or a bounding box. A convex hull comprises a set of outer points on a perimeter of a region of interest associated with input data for an image. The convex hull may be defined by a polygon to reduce the number of unnecessary pixels of the input data that are processed. A bounding box comprises a rectangle that includes the region of interest, and may include additional pixels of the input data that may not be in a region of interest. Although a bounding box may generally cover an area that is greater than necessary to process input data of the region of interest, the processing of data may be simplified by having a uniform area defining the bounding box, such as a rectangle, to more easily process the data which is a simple uniform area.

In order to compensate for an image that experiences warping, a Kalman filter can be used to estimate the translation, rotation, scaling or a change in perspective of the pixels of the image. That is, the translation, rotation, scaling or change in perspective of the pixels of the image can be described with equations, and therefore the processing of the image can be accomplished using lookup table (LUT) mapping. By way of example, the LUT mapping could be represented by
<MAT>
representing a transformation of the input image. According to one implementation, rotation of the image can be considered. When considering rotation, where only a height (h), width (w) and a rotation angle Θ are known, the pixels associated with the image can be represented by:
<MAT>.

It should be noted that compensation for rotation is generally more difficult to accomplish than translation and scaling of an image, which generally involve a shifting.

Rotation, translation and perspective can be taken into account according to:
<MAT>
where a00, a01, a10 and all relate to rotation, a02 and a12 relate to translation, and a20 and a21 relate to perspective. Input coordinates for image are obtained as
<MAT>.

Because of the warping of the image, a transformation of the image could be achieved using the matrix transformations in equations (<NUM>), (<NUM>) and (<NUM>), which could be stored in the transformation circuit <NUM> for example. For the case of rotation only, a02, a12, and a20 and a21 are set to <NUM>.

In implementing a convex hull, the input image can be scanned starting with a line at y=<NUM> to determine a left-most point of the convex hull and a right most point of the convex hull associated with pixels in a region of interest for a row of the image, and continue to determine the intersection points from y=<NUM> to y=H'-<NUM>. For <NUM> byte granularity in DMA transfers, the intersection points for each row can be quantized to 64B boundaries, and determine the left-most data point and the right-most data point to identify the convex hull for the image.

According to one embodiment, the distortion by a lens of the camera is corrected after the image is detected. When large distortions need to be corrected, a mesh mode correction of the image can be achieved using a LUT. According to one embodiment, multiple images taken at different times or from a different angle can be aligned to look that same before they are combined to create a single image. If the images are taken from a different angle, it is possible to compensate for rotation to enable adding the images to create a composite image.

As shown in <FIG>, a rotated input tile having bi-cubic filters is shown. When implementing bi-cubic filtering, a square region having a length of a side of L at each of the <NUM> corners provides filtering samples near corner pixels to provide a better estimate of the pixels in those edge regions. A square portion for each of the corners Bn, the square is defined by Cn0, Cn1, Cn2 and Cn3, where
<MAT>
<MAT>
<MAT> and
<MAT>.

As shown in <FIG>, a convex hull associated with a rotated input tile is defined to include a ROI encompassing the squares. While a polygon associated with a bi-cubic filter squares at four corners of a rectangle, it should be understood that a region of interest could be a shape other than a rectangle, and squares of the bi-cubic filter could define a polygon other than a rectangle and could be at more than <NUM> locations.

Turning now to <FIG>, an exemplary flow diagram of a method of performing warping using an image having a convex hull is shown. All pixel locations for an input tile are found at a block <NUM>. A convex hull for the pixel locations is found at a block <NUM>. The pixels inside the convex hull are transferred at a clock <NUM>. Warping is performed at a block <NUM>. By way of example, the pixels in the convex hull maybe transferred to a rectangular tile, where the warped output is a rectangular tile having width W and a height H, which are less than the width W' and the height H' as shown in <FIG>.

In order to transfer of points inside a polygon, for row=<NUM>:<NUM>:nrow, it is first necessary to find intersection with each of the n sides for convex polygon as col(row,<NUM>:npts), npts=<NUM>, <NUM>, <NUM> or <NUM>. It is then necessary to find mincol(row) and maxcol(row) of the intersection points, where mincol(row)=min(col(row,:)) and maxcol(row)=max(col(row,:)). The pixels at locations [mincol(row), maxcol(row)] are then transferred. If DMA transfers are based on bursts (e.g., 64B bursts), convert transfer to burst addresses as follows: <MAT> and <MAT>.

Turning now to <FIG>, a bounding box associated with a rotated input tile is shown. A minimum bounding box for rotations can be defined by
<MAT> <MAT> <MAT> and <MAT>.

For a <NUM> x <NUM> window and the image being rotated by an angle Θ as shown in <FIG>, h0 = L*(<NUM>+sin(<NUM>* Θ) and h1=Lcos(<NUM>* e)), where
<MAT> and <MAT>
where the area of the bounding box is greater that the area of the convex hull.

Turning now to <FIG>, an exemplary flow diagram illustrating a method of performing warping using an image having a bounding box is shown. All pixel locations for an input tile are found at a block <NUM>. A bounding box for the pixel locations is found at a block <NUM>. The bounding box may be implemented as shown for example in <FIG>. The pixels are transferred inside the bounding box at a block <NUM>. Warping is performed at a block <NUM>. The warping can be performed to either introduce warping or compensate for warping of an image, such as warping caused by a lens of an electronic device for example.

Turning now to <FIG>, is an exemplary flow diagram illustrating a method of implementing memory transfers for image warping in an electronic device is shown. An input tile associated with an image is received at a block <NUM>. A geometric boundary around pixels of the input tile is generated at a block <NUM>. The geometric boundary could be a convex hull or a bounding box as described above. The pixels in the geometric boundary are remapped to an output tile at a block <NUM>. An output is generated using the output tile at a block <NUM>. As described above in reference to <FIG>, reducing the warping can result in an output tile having pixels associated with an image that is smaller than the input tile.

Claim 1:
A method of implementing memory transfers for image warping in an electronic device, the method comprising:
receiving (<NUM>), from a first memory of the electronic device, an input tile associated with an image;
defining plurality of square regions for corners of the input tile, each square region surrounding a corresponding corner (Bn) of the input tile
wherein the square regions provide filtering samples for implementing bi-cubic filtering;
generating (<NUM>) a convex hull (<NUM>) associated with the input tile, wherein the convex hull includes a region of interest, ROI, encompassing each of the plurality of square regions;
remapping (<NUM>) pixels in the convex hull to an output tile;
generating (<NUM>) an output image using the output tile; and
storing the output image in a second memory of the electronic device.