Patent Description:
Various types of image data processing techniques are been actively used, including techniques for changing an image represented by data to suit a purpose, and is used for image manipulation, image analysis, image recognition, and image communication. Among the various image data processing techniques, convolution processing is widely used for a blurring effect or a sharpening effect. In an example, the convolution processing is a processing method that multiplies pixel values of a central pixel and its adjacent pixels in image data by a weight (or coefficient) assigned to a filter (or mask), and assigns a result value corresponding to the sum thereof as the pixel value of the central pixel.

<CIT>, here acknowledged as the closest prior art, discloses a demosaicing device and method relative to which at least the characterizing features in the appended independent claims are novel.

<CIT> and <CIT> are acknowledged as further, more distant, prior art.

In one general aspect, there is provided a sensor device exhibiting features of the appended independent device claim.

In another general aspect, there is provided an image data processing method as defined in the appended independent method claim.

In another general aspect, there is provided a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image data processing method of the invention.

Hereinafter, examples will be further described with reference to the accompanying drawings. The following structural or functional descriptions of examples disclosed in the present disclosure are merely intended for the purpose of describing the examples and the examples may be implemented in various forms. The examples are not meant to be limited, but it is intended that various modifications, equivalents, and alternatives are also covered within the scope of the claims.

Terms, such as first, second, and the like, may be used herein to describe components. It should be noted that if it is described that one component is "connected", "coupled", or "joined" to another component, a third component may be "connected", "coupled", and "joined" between the first and second components, although the first component may be directly connected, coupled, or joined to the second component.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like constituent elements and a repeated description related thereto will be omitted.

<FIG> illustrates an example of an overall configuration of a recognition system.

Referring to <FIG>, a recognition system <NUM> includes a sensor device <NUM> and a recognition apparatus <NUM>. The recognition system <NUM> acquires image data through the sensor device <NUM> and recognizes an object shown in the image data through the recognition apparatus <NUM>. The recognition system <NUM> is used for object recognition such as face recognition or recognition of things.

The sensor device <NUM> acquires the image data using an image sensor such as a camera and generates image-processed result data by performing image processing on the acquired image data. The sensor device <NUM> generates convolution-processed result data by performing convolution processing on the acquired image data. The sensor device <NUM> performs convolution processing by applying a filter for performing a convolution operation to the image data. If the image data acquired by the image sensor is image data of a Bayer pattern, the sensor device <NUM> generates convolution-processed result data by performing convolution processing even with respect to the image data of the Bayer pattern. As described above, convolution processing of the image data is performed in the sensor device <NUM>. Herein, the term "filter" is used interchangeably with the term mask, kernel, or template.

In another example, the sensor device <NUM> compresses the convolution-processed image data. For example, the sensor device <NUM> generates quantized image data as the result data by quantizing the convolution-processed image data. The quantized image data is image data having fewer bits than the original image data acquired by the image sensor, for example, binary image data in which each of all the pixels of the image data has one of a first pixel value (for example, "<NUM>") and a second pixel value (for example, "<NUM>").

In an example, the sensor device <NUM> autonomously performs image processing such as convolution processing and/or quantization using an internal processing unit, for example, an image processor or a micro controller unit. The image-processed result data is transmitted to the recognition apparatus <NUM>. Herein, the image processor refers to a processor that performs a function to process image data in the sensor device <NUM>, and should not be interpreted as limiting to a processor that performs only the function to process images. In an example, the image processor performs a function other than image processing, such as, for example, controlling the sensor device <NUM>.

As described above, image processing is first performed with respect to the image data acquired by the sensor device <NUM> in the sensor device <NUM> rather than the recognition apparatus <NUM>. For example, the sensor device <NUM> performs convolution processing and/or quantization with respect to the image data. In this example, the image-processed result data is transmitted from the sensor device <NUM> to the outside of the sensor device <NUM>, and thus a bandwidth is required for data transmission as well as a data transmission quantity may be reduced.

The sensor device <NUM> autonomously performs a processing operation requiring a relatively low computational complexity. For example, the sensor device <NUM> performs a processing operation such as object detection (for example, face detection) after performing convolution processing and/or quantization on the acquired image data. The sensor device <NUM> transmits convolution-processed and/or quantized image data to the outside of the sensor device <NUM> if an object is detected in the image data, and does not transmit the image data to the outside if an object is not detected in the image data. In the present example, only when an object is detected in the image data, the convolution-processed and/or quantized image data is transmitted from the sensor device <NUM>.

The recognition apparatus <NUM> receives the result data image-processed by the sensor device <NUM> from the sensor device <NUM> and performs object recognition based on the received result data. In an example, the recognition apparatus <NUM> performs object recognition using a trained object recognition model. The object recognition model is, for example, a neural network model, and provides a score (for example, an expected value or a probability value) indicating an object corresponding to an object shown in image data input into the object recognition model based on the image data. The object recognition model is a neural network model having a bit width corresponding to a bit width of the result data received from the sensor device <NUM>. If the result data received from the sensor device <NUM> is quantized image data having a low bit width, a neural network model having a low bit width corresponding thereto is used, such that the amount of required resources is reduced, and the processing rate increases. By using the object recognition model having a low bit width as described above, the recognition system <NUM> performs an object recognition process at low power, uses less memory capacity, and high speed in a limited embedded system such as a smart sensor or a smart phone.

<FIG> illustrates an example of an object recognition process. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. The blocks of the object recognition process of <FIG>, and combinations of the blocks, are performed by an object recognition apparatus. In an example, the object recognition apparatus is implemented by special purpose hardware-based computer, and devices such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions included in the object recognition apparatus. In addition to the description of <FIG> below, the descriptions of <FIG> is also applicable to <FIG> and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, an object recognition process includes operation <NUM> in which a sensor device generates image-processed result data by performing image processing on image data, and operation <NUM> in which a recognition apparatus performs object recognition based on the image-processed result data.

In operation <NUM>, the sensor device acquires image data using an image sensor (in operation <NUM>), and stores the acquired image data in an image buffer in the sensor device (in operation <NUM>). In an example, the image buffer included in the sensor device has a limited capacity and thus, sequentially stores a portion of the image data acquired by the image sensor, rather than storing the entire image data. For example, the image buffer stores only a number of (for example, <NUM>) lines (for example, rows) of image data, from among a plurality of lines constituting the image data, at a point in time. Here, a line refers to a group of pixels arranged in the same horizontal direction or in the same vertical direction, from among pixels included in the image data. In an example, the image buffer stores the image data in the unit of lines, and stores the image data in a manner of sequentially overwriting the image data due to the limited buffer space. For example, assuming that the image buffer stores only five lines of the image data due to a limited buffer area, the most recently received new line of image data is stored in a row where the oldest image data is stored in the buffer area of the image buffer, each time a new line of image data is received by the image buffer. Accordingly, the oldest recorded image data is removed from the buffer area from among the image data stored in the buffer area of the image buffer.

The image data acquired by the image sensor has a pattern (for example, a Bayer pattern) in which pixels of a plurality of color components are arranged, and the image data of the pattern is stored in the image buffer. A general Bayer pattern includes pixel values of different color components, for example, a pixel value of a green component, a pixel value of a blue component, and pixel value of a red component, wherein the pixel value of the blue component and the pixel value of the green component are alternately arranged or the pixel value of the red component and the pixel value of the green component are alternately arranged, depending on a line. However, a Bayer pattern of image data acquirable by the image sensor is not limited to a specific Bayer pattern, and there may be various arrangement patterns of pixel values of color components constituting the image data. For example, the image data has a tetracell pattern in which pixel values of same-color components are set as a single group, or a britecell pattern in which a pixel value of a green component is replaced with a pixel value of a white component in the Bayer pattern. Hereinafter, the description will be provided based on a general Bayer pattern for ease of description. However, a pattern of image data acquirable by the image sensor is not limited to the general Bayer pattern, and there may be unlimited types of patterns.

In operation <NUM>, the sensor device performs image processing with respect to the image data stored in the image buffer, and in operation <NUM>, the sensor device generates the image-processed image data as result data. The sensor device performs image processing such as convolution processing using a pattern corresponding to a storage pattern of the image buffer depending on a point in time at which the image processing is performed. As described above, the storage pattern of the image data stored in the image buffer changes over time due to sequential storage, and the sensor device performs convolution processing using a filter of a pattern corresponding to the storage pattern of the image data stored in the image buffer, from among a plurality of filters. Thus, the pattern of the filter applied to the image data in the image buffer changes over time as well, and the change in the pattern is repeated with periodicity.

In some examples, quantization may further be performed after convolution processing is performed with respect to the image data. In an example, during the quantization, the sensor device quantizes the image data using the halftone technique. The halftone technique is a technique that converts image data having a high bit width to image data (for example, binary image data) having a low bit width, wherein pixel values of an image are expressed as a density on a two-dimensional space. For example, the sensor device quantizes the image data by applying a quantization filter such as a dither matrix to the image data, or quantizes the image data using error diffusion.

The image-processed result data, generated by the sensor device, is transmitted to the recognition apparatus through an output interface of the sensor device. For example, the sensor device transmits the result data to the recognition apparatus through an output interface such as a mobile industry processor interface (MIPI). The number of lines of the result data transmitted from the sensor device to the recognition apparatus is determined based on the MIPI standards. The recognition apparatus includes, for example, an application processor (AP) connected to the MIPI.

In operation <NUM>, the recognition apparatus performs object recognition using an object recognition model <NUM>. The object recognition model <NUM> is, for example, a trained neural network model, and provides a recognition result with respect to the object recognition based on the image-processed result data received from the sensor device. The object recognition process is performed under the control of a processor, which may be a hardware-implemented data processing device having a circuit that is physically structured to execute desired operations, such as a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU), included in the recognition apparatus. If the image data is input into the object recognition model <NUM>, the object recognition model <NUM> outputs a score indicating a probability or possibility that an object shown in the image data corresponds to each class (or label). The recognition apparatus determines an object recognition result based on information about the score.

<FIG> illustrates an example of operations of an image data processing method performed by a sensor device. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. The blocks of the image data processing method of <FIG>, and combinations of the blocks, are performed by special purpose hardware-based computer, and devices such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions included in the sensor device. In addition to the description of <FIG> below, the descriptions of <FIG> is also applicable to <FIG> and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, in operation <NUM>, a sensor device acquires image data using an image sensor such as a camera. In operation <NUM>, the sensor device stores the acquired image data in an image buffer. The sensor device sequentially stores a portion of the image data in the unit of lines due to a limited buffer area of the image buffer. In another example, the image sensor acquires image data of a Bayer pattern, and the image data of the Bayer pattern is stored in the image buffer.

In operation <NUM>, the sensor device generates image-processed result data by performing image processing on the image data stored in the image buffer. The sensor device generates the image-processed result data by applying a filter corresponding to a current storage pattern of the image buffer to the image data stored in the image buffer. For example, the sensor device generates, as the result data, convolution-processed image data by applying a convolution filter for convolution processing to the image data stored in the image buffer. When a line of the image data acquired by the image sensor is input and stored in the image buffer, the sensor device outputs a line of a result image by performing convolution processing by applying the convolution filter to the image data stored in the image buffer. The convolution filter includes coefficients or weights to be applied to the image data, and pixel values of the result image are determined based on pixel values of the image data stored in the image buffer by the convolution processing using the convolution filter. As described above, the convolution processing with respect to the image data stored in the image buffer is performed in the sensor device.

The sensor device selects the filter corresponding to the current storage pattern of the image buffer from among filters of different patterns and applies the selected filter to the image data stored in the image buffer. In an example, the number of filters of different patterns is determined based on the number of lines of image data that can be stored in the image buffer and the size of the filters. For example, if the image buffer is able to store five rows of the image data in relation to the buffer size of the image buffer and the size of the filter is <NUM> × <NUM>, there may be five filters of different patterns.

The sensor device selects a filter of a pattern to be applied to image data currently stored in the image buffer from among filters of a plurality of patterns, based on a buffer area in which the image data has been most recently stored in the image buffer. A filter to be used for image processing is determined from among the filters of several patterns, based on the most recently changed line from among the lines included in the buffer area of the image buffer.

The image processing process using the filter starts when the image data is stored in the entire buffer area of the image buffer or when the image data is stored in a portion of the entire buffer area. If the image data is stored in a portion of the entire buffer area, zero-padding that performs image processing by filling a portion required for image processing with a value of "<NUM>" or wrap-around that processes the image data as if the start portion and the end portion of the image data are connected to each other is used for image processing.

If the image data stored in the image buffer is image data of a pattern in which pixels of a plurality of color components are arranged, for example, a Bayer pattern, the sensor device generates the image-processed result data by applying a filter corresponding to an arrangement pattern of the color components in the Bayer pattern of the image data stored in the image buffer to the image data stored in the image buffer. For example, at a first point in time with respect to the image data stored in the image buffer, the sensor device applies a first filter corresponding to an arrangement pattern of first color components (for example, blue components) in the Bayer pattern to the image data stored in the image buffer, and applies a second filter corresponding to an arrangement pattern of second color components (for example, green components) that are different from the first color components to the image data stored in the image buffer. The first filter and the second filter are alternately applied to pixels of the image data stored in the image buffer. For example, at a second point in time with respect to the image data stored in the image buffer, the sensor device applies the second filter corresponding to the arrangement pattern of the second color components (for example, green components) in the Bayer pattern to the image data stored in the image buffer, and applies a third filter corresponding to an arrangement pattern of third color components (for example, red components) that are different from the second color components to the image data stored in the image buffer. The second filter and the third filter are also alternately applied to pixels of the image data stored in the image buffer.

In some examples, the sensor device generates result data having a low bit width by performing quantization after the convolution processing. The sensor device generates quantized image data by applying a quantization filter to the image data on which the convolution processing is performed. The image data acquired by the image sensor has a great number of bits. The sensor device reduces the number of bits of the image data corresponding to the result data by autonomously performing quantization through a processing unit, such as an image processor or a micro controller unit, included in the sensor device.

In operation <NUM>, the sensor device transmits the image-processed result data to the outside of the sensor device. The sensor device transmits the result data to the outside of the sensor device through an output interface such as, for example, MIPI. As described above, the image data acquired through the image sensor is output from the sensor device after image-processed in the sensor device.

<FIG> illustrates an example of a convolution processing process. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. The blocks of the convolution processing process of <FIG>, and combinations of the blocks, are performed by special purpose hardware-based computer, and devices such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions included in a convolution processing device. In addition to the description of <FIG> below, the descriptions of <FIG> is also applicable to <FIG> and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, in operation <NUM>, a current line of image data is received by an image buffer. In operation <NUM>, the sensor device stores the received current line of the image data in the image buffer. For example, the sensor device records the current line of the image data in a buffer area in which the oldest image data is recorded, in the buffer area of the image buffer. Here, the buffer area in which the image data is recorded is defined in the unit of lines.

In operation <NUM>, the sensor device performs convolution processing using a filter corresponding to a storage pattern of the image data in the image buffer. For example, the sensor device performs image processing using a convolution filter corresponding to a storage pattern of image data currently stored in the image buffer, from among convolution filters of predefined patterns. Here, the storage pattern of the image data changes, for example, depending on a line in which the image data is most recently stored, among the lines included in the buffer area of the image buffer.

In operation <NUM>, the sensor device outputs the convolution-processed result data. The sensor device outputs a line of the result image as the result data through convolution processing by applying the convolution filter to the image data stored in the image buffer.

In operation <NUM>, the sensor device determines whether image processing with respect to the entire image data is completed. The sensor device determines whether the last line of the image data acquired by the image sensor is transmitted and stored in the image buffer and convolution processing with respect to the last line of the image data is all completed. If the image processing with respect to the entire image data is all completed by completing image processing with respect to the last line of the image data, the convolution processing process is terminated.

If image processing with respect to the entire image data is yet to be completed, in operation <NUM>, a next line of the image data is received by the image buffer. In an example, the next line of the image data is a row immediately after the current line of the image data received in operation <NUM>, in the image data acquired by the image sensor. In operation <NUM>, the sensor device stores the received next line of the image data in the image buffer. The sensor device records the next line of the image data in a buffer area in which the oldest image data is recorded, in the buffer area of the image buffer. The sensor device performs the image processing process again, starting from operation <NUM>.

As described above, the sensor device sequentially stores, in the image buffer, the image data acquired by the image sensor in the unit of lines, and performs convolution processing using a filter corresponding to a storage pattern of the image data stored in the image buffer, such that the convolution processing is effectively performed in the sensor device, irrespective of the limited buffer area of the image buffer.

<FIG> and <FIG> illustrate an example of performing convolution processing with respect to image data.

<FIG> illustrates that image data <NUM> is acquired by an image sensor of a sensor device. The image data <NUM> includes multiple lines <NUM>, <NUM>, <NUM>,. each being a set of pixels. For example, a line "<NUM>" of the image data <NUM> is a set of pixels corresponding to a first row, and a line "<NUM>" of the image data <NUM> is a set of pixels corresponding to a second row. The image data <NUM> is stored in an image buffer included in the sensor device. In the present example, it is assumed that a buffer area of the image buffer only stores five rows of image data. However, the size of the buffer area of the image buffer is not limited thereto.

Similar to rolling shutter, the image buffer sequentially stores the image data <NUM> line by line. For example, the image buffer sequentially stores lines "<NUM>" to "<NUM>" of the image data <NUM>, as shown in a storage pattern <NUM>. Due to a limited buffer area, the image buffer stores the received image data in a manner of overwriting. For example, if a line "<NUM>" of the image data <NUM> is received in a storage state of the image buffer shown in the storage pattern <NUM>, the image buffer stores the line "<NUM>" of the image data by overwriting the line "<NUM>" of the image data to the oldest recorded buffer area (the buffer area in which the line "<NUM>" of the image data is recorded), as shown in a storage pattern <NUM>. After that, if a line "<NUM>" of the image data <NUM> is received, the image buffer records the line "<NUM>" of the image data to the oldest recorded buffer area (the buffer area in which the line "<NUM>" of the image data is recorded), as shown in a storage pattern <NUM>. In this manner, the storage pattern of the image data stored in the image buffer is represented sequentially in an order of the storage pattern <NUM>, the storage pattern <NUM>, the storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>. As described above, the image data <NUM> is transmitted to the image buffer line by line, and the image buffer stores a received line of the image data in the buffer area in a manner of overwriting.

Referring to <FIG>, an example of performing convolution processing with respect to each storage pattern of an image buffer is illustrated. As a storage pattern of the image buffer changes, a pattern of a filter for convolution processing also changes according to the storage pattern of the image buffer. A sensor device selects a filter corresponding to a current storage pattern of the image buffer from among filters of different patterns <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, and applies the selected filter to the image data stored in the image buffer.

For example, a filter of a pattern <NUM> is applied to the image data in correspondence to the storage pattern <NUM> of the image data in the image buffer, and a filter of a storage pattern <NUM> is applied in correspondence to the storage pattern <NUM> of the image data in the image buffer. In the case of the storage pattern <NUM>, the filter of the pattern <NUM> which is a basic pattern is directly applied. However, in the case of the storage pattern <NUM>, the filter of the pattern <NUM> which is a basic pattern should not be directly applied since an order of rows of the image data stored in the image buffer is not in sequence. Accordingly, the sensor device rotates the components of the filter to corresponds to the order of the storage pattern <NUM>, as shown in the pattern <NUM>, and uses the filter of the pattern <NUM> in which the components are arranged in an order of "<NUM>, <NUM>, <NUM>, <NUM>, <NUM>". Data (convolution-processed pixels values) corresponding to a line "a" of result data <NUM> is determined by applying the filter of the pattern <NUM> to the image data of the storage pattern <NUM>, and data corresponding to a line "b" of the result data <NUM> is determined by applying the filter of the pattern <NUM> to the image data of the storage pattern <NUM>.

Further, the filter of the pattern <NUM>, the filter of the pattern <NUM>, and the filter of the pattern <NUM> are applied respectively in correspondence to the storage pattern <NUM>, the storage pattern <NUM>, and the storage pattern <NUM>. If a line "<NUM>" of the image data is stored in the image buffer as shown in the storage pattern <NUM>, the order of the image data stored in the image buffer is "<NUM>, <NUM>, <NUM>, <NUM>, <NUM>". In this example, the sensor device uses the filter of the pattern <NUM> with a component order of "<NUM>, <NUM>, <NUM>, <NUM>, <NUM>" corresponding to the order of the storage pattern <NUM>.

In the case of the storage pattern <NUM>, data corresponding to a line "f" of the result data <NUM> is determined by applying the filter of the storage pattern <NUM> again. The pattern of the filter applied to the image buffer is changed repeatedly from the pattern <NUM> to the pattern <NUM> as one cycle. A filter with a pattern changed repeatedly with periodicity as described above will be referred to as a "rotational filter". Each time image data is input into the image buffer, the pattern of the filter used for convolution processing is rotated. The number of different patterns of filters corresponds to the number of lines of the image data that may be stored in the image buffer. For example, assuming that the number of lines of image data that can be stored in the image buffer is "<NUM>", there are five filters of different patterns. The sensor device obtains the convolution-processed result data <NUM> by sequentially applying a filter corresponding to a storage pattern of the image buffer each time a new line of image data is input into the image buffer.

<FIG> illustrate an example of performing convolution processing with respect to image data of a Bayer pattern.

Referring to <FIG>, an example of image data <NUM> of a Bayer pattern acquired by an image sensor is illustrated. The image data <NUM> of the Bayer pattern is expressed in the formed in which a pixel value of a Blue component <NUM>, pixel values of green components <NUM> and <NUM>, and pixel values of red component <NUM> are arranged. A color component corresponding to each pixel of the image data <NUM> is predefined. For example, odd-numbered lines of the image data <NUM> include the pixel value of the blue component <NUM> and the pixel value of the green component <NUM> that are alternately arranged, and even-numbered lines of the image data <NUM> include the pixel value of the green component <NUM> and the pixel value of the red component <NUM> that are alternately arranged. However, the Bayer pattern is not limited to the present example, and may have various forms.

<FIG> illustrates an example of storing the image data <NUM> of the Bayer pattern of <FIG>. Referring to <FIG>, the image data <NUM> is stored sequentially in a buffer area of the image buffer in the unit of lines, and thus a storage pattern of the image data stored in the image buffer changes over time.

For example, the image buffer sequentially stores lines "<NUM>" to "<NUM>" of the image data <NUM> of the Bayer pattern, as shown in a storage pattern <NUM>. Due to the limited buffer area, the image buffer stores the received image data in a manner of overwriting. For example, if a line "<NUM>" of the image data <NUM> is received in a storage state of the image buffer shown in the storage pattern <NUM>, the image buffer stores the line "<NUM>" of the image data by overwriting the line "<NUM>" of the image data to the oldest recorded buffer area (the buffer area in which the line " <NUM>" of the image data is recorded), as shown in a storage pattern <NUM>. If the image data <NUM> is stored in the image buffer in this manner, the image data is stored in the image buffer in an order of the storage pattern <NUM>, the storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>, a storage pattern <NUM>,. , as shown in <FIG>. The storage pattern <NUM> shows the same pattern as the storage pattern <NUM>, and the storage pattern <NUM> to the storage pattern <NUM> are repeated as from the storage pattern <NUM>.

<FIG> illustrates an example of performing convolution processing with respect to image data of a Bayer pattern stored with different patterns in an image buffer. When the image data of the Bayer pattern is stored in the image buffer, the storage pattern of the image buffer changes in an order from the storage pattern <NUM> to the storage pattern <NUM>. The sensor device generates image-processed result data <NUM> by applying a filter corresponding to an arrangement pattern of color components in the Bayer pattern of the image data stored in the image buffer to the image data stored in the image buffer. The result data <NUM> also has the Bayer pattern. The sensor device selects the filter corresponding to the current storage pattern of the image buffer from among filters of predefined patterns and applies the selected filter to the image data stored in the image buffer.

The sensor device applies, to the image data, a filter corresponding to an arrangement pattern of color components in the Bayer pattern of the image data stored in the image buffer. For example, a convolution operation is performed on the image data of the storage pattern <NUM> by alternately applying a filter <NUM> corresponding to a blue component and a filter <NUM> corresponding to a green component (or applying the filter <NUM> and <NUM> for every two spaces) thereto, and data (convolution-processed pixel values) corresponding to a line "a" of the result data <NUM> is determined. Here, a filter <NUM> corresponding to a red component is not applied. A convolution operation is performed on the image data of the storage pattern <NUM> by alternately applying a filter <NUM> corresponding to a green component and a filter <NUM> corresponding to a red component thereto, and data corresponding to a line "b" of the result data <NUM> is determined. Here, a filter <NUM> corresponding to a blue component is not applied. A convolution operation is performed on the image data of the storage pattern <NUM> by alternately applying a filter <NUM> corresponding to a blue component and a filter <NUM> corresponding to a green component thereto, and data corresponding to a line "c" of the result data <NUM> is determined. Here, a filter <NUM> corresponding to a red component is not applied. A convolution operation is performed on the image data of the storage pattern <NUM> by alternately applying a filter <NUM> corresponding to a green component and a filter <NUM> corresponding to a red component thereto, and data corresponding to a line "d" of the result data <NUM> is determined. Here, a filter <NUM> corresponding to a blue component is not applied. A convolution operation is performed on the image data of the storage pattern <NUM> by alternately applying a filter <NUM> corresponding to a blue component and a filter <NUM> corresponding to a green component thereto, and data corresponding to a line "e" of the result data <NUM> is determined. Here, a filter <NUM> corresponding to a red component is not applied.

As described above, the sensor device obtains the result data <NUM> by sequentially applying filters of patterns corresponding to the Bayer pattern of the image data stored in the image buffer each time a new line of the image data is input and stored in the image buffer. In this example, the sensor device performs a convolution operation for each color component, but does not apply a filter for a convolution operation to a color component not included in a row of the output data <NUM>.

<FIG> illustrates an example of a configuration of a sensor device.

Referring to <FIG>, a sensor device <NUM> includes an image sensor <NUM>, an image processor <NUM>, a memory <NUM>, an image buffer <NUM>, and an output interface <NUM>. The sensor device <NUM> corresponds to the sensor device described herein.

The image sensor <NUM> acquires image data. For example, the image sensor <NUM> acquires image data such as a color image, a gray-scale image, or an infrared image. The image buffer <NUM> stores the image data acquired by the image sensor <NUM>. The image buffer <NUM> sequentially stores a portion of the image data in the unit of lines.

The image processor <NUM> controls an operation of the sensor device <NUM> and includes, for example, any one or any combination of a digital signal processor (DSP), an image signal processor (ISP), and a micro controller unit (MCU).

The image processor <NUM> performs image processing such as convolution processing and/or quantization with respect to the image data stored in the image buffer <NUM>. The image processor <NUM> performs one or more operations related to the image processing described above with reference to <FIG>.

For example, the image processor <NUM> generates convolution-processed result data by applying a convolution filter corresponding to a current storage pattern of the image buffer <NUM> to the image data stored in the image buffer. If the image data stored in the image buffer is image data of a Bayer pattern, the image processor <NUM> generates image-processed result data by applying a filter corresponding to an arrangement pattern of color components in the Bayer pattern of the image data stored in the image buffer to the image data stored in the image buffer. In another example, the image processor <NUM> generates quantized image data as result data by performing quantization with respect to the convolution-processed image data.

The memory <NUM> stores instructions to be executed by the processor <NUM> and information to be used to perform image processing. The memory <NUM> stores the image data acquired by the image sensor <NUM> and the image-processed result data. The memory <NUM> includes, for example, a high-speed random-access memory and/or a non-volatile computer-readable storage medium. Further details regarding the memory <NUM> is provided below.

The output interface <NUM> transmits the result data image-processed by the image processor <NUM> to the outside of the sensor device <NUM>. The output interface <NUM> transmits the result data to a recognition apparatus through wired or wireless communication. For example, the output interface <NUM> is an MIPI. In this example, the sensor device <NUM> transmits the result data with a bandwidth determined by the MIPI standard.

<FIG> illustrates an example of a configuration of a recognition apparatus.

Referring to <FIG>, a recognition apparatus <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a communication device <NUM>. The elements of the recognition apparatus <NUM> communicate with each other through a communication bus. The recognition apparatus <NUM> corresponds to the recognition apparatus described herein.

The processor <NUM> controls an operation of the recognition apparatus <NUM> and executes instructions and functions to perform object recognition. For example, the processor <NUM> executes instructions stored in the memory <NUM> or the storage device <NUM>. The processor <NUM> includes, for example, any one or any combination of a CPU, a GPU, and an NPU, and performs one or more of the operations related to object recognition described with reference to <FIG>. For example, the processor <NUM> performs object recognition based on image-processed result data received from the sensor device <NUM>.

The memory <NUM> stores instructions to be executed by the processor <NUM> and information to be used to perform object recognition. The memory <NUM> includes, for example, a high-speed random-access memory and/or a non-volatile computer-readable storage medium. Further details regarding the memory <NUM> is provided below.

The storage device <NUM> includes a computer-readable storage medium. The storage device <NUM> stores a greater quantity of information than the memory <NUM> and for a longer time. For example, the storage device <NUM> includes storage mediums such as, for example, a hard disk, an optical disk, and a solid-state drive.

The input device <NUM> receives an input from a user through a tactile, video, audio, or touch input. For example, the input device <NUM> includes a keyboard, a mouse, a touch screen, a microphone, or any other device that detects the input from the user and transmits the detected input to the recognition apparatus <NUM>.

The output device <NUM> provides an output of the recognition apparatus <NUM> to the user through a visual, auditory, or tactile channel. The output device <NUM> includes, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user.

The communication device <NUM> communicates with an external device through a wired or wireless network. For example, the communication device <NUM> receives image-processed result data from the sensor device <NUM> through an MIPI.

The apparatuses, units, modules, devices, and other components described herein are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing. One or more processors may include hardware, such as, for example, a processor, a controller and an arithmetic logic unit (ALU), a DSP, a microcomputer, an FPGA, a programmable logic unit (PLU), a microprocessor or any other device capable of responding to and executing instructions in a defined manner.

The methods that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods.

Instructions or software to control a processor or computer to implement the hardware components and perform the methods as described above are written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and the methods as described above. In an example, the instructions or software includes at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, an application program storing the image data processing method. In one example, the instructions or software include machine code that is directly executed by the processor or computer, such as machine code produced by a compiler. In another example, the instructions or software include higher-level code that is executed by the processor or computer using an interpreter. Programmers of ordinary skill in the art can readily write the instructions or software based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and the methods as described above.

Claim 1:
A sensor device (<NUM>) comprising:
an image sensor (<NUM>) configured to acquire image data (<NUM>; <NUM>);
an image buffer (<NUM>) configured to store the image data (<NUM>; <NUM>) by sequentially overwriting portions of the image data (<NUM>; <NUM>) according to a storage pattern (<NUM>-<NUM>; <NUM>-<NUM>); and
an image processor (<NUM>) configured to generate image-processed data (<NUM>; <NUM>) by applying a filter to the image data (<NUM>; <NUM>) stored in the image buffer (<NUM>),
characterized in that the image processor (<NUM>) is further configured to sequentially change an order of components of the filter corresponding to the portions of the image data (<NUM>; <NUM>) in correspondence with the sequence of overwriting of the portions of the image data (<NUM>; <NUM>) in the image buffer (<NUM>) wherein the order of the components of the filter corresponds to the storage pattern (<NUM>-<NUM>; <NUM>-<NUM>) of the image data (<NUM>; <NUM>) stored in the image buffer (<NUM>).