Patent Description:
In the field of three-dimensional (3D) image technology, research has been actively conducted to develop apparatuses for realizing a high-definition hologram in real time by using a complex spatial light modulator (SLM) capable of simultaneously controlling the amplitude and phase of light.

To reproduce a hologram moving picture, a computer-generated hologram (CGH) has been used. Image processing apparatuses perform a very large number of calculations to calculate a hologram value for each location in a hologram plane. In this aspect, in order to express a point in space, image processing apparatuses need to perform a Fourier transform operation one time. To express an image of a space, image processing apparatuses need to perform as many Fourier transform operations as the number of corresponding pixels of the image.

Image processing apparatuses, such as televisions (TVs) and mobile devices, can process image data in order to reproduce a hologram image. In this case, the image processing apparatuses can perform Fourier transformation operations on the image data and reproduce an image by using the transformed data.

When the image processing apparatuses perform Fourier transformation operations, a large number of calculations are performed, which is time consuming. In particular, portable devices such as mobile devices are limited with respect to both size and available power. Thus, there is a demand for methods of reducing the number of calculations and the calculation time when image processing apparatuses perform Fourier transformation operations.

United States Patent Application Publication Number <CIT> discloses a variable length FFT system.

Provided are methods and apparatuses for performing Fourier transformation operations on image data.

Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented exemplary embodiments.

According to an aspect of the invention, there is provided an image processing apparatus according to claim <NUM>.

According to another aspect of the invention, there is provided an image processing method according to claim <NUM>.

These and/or other aspects will become apparent and more readily appreciated from the following description of exemplary embodiments, taken in conjunction with the accompanying drawings in which:.

Exemplary embodiments now will be described more fully hereinafter with reference to the accompanying drawings.

<FIG> is a schematic view illustrating a process of processing image data, according to an exemplary embodiment. Referring to <FIG>, an image processing apparatus may receive image data and output an image on which image processing has been performed.

In operation <NUM>, the image processing apparatus receives image data. For example, in computer-generated holography (CGH), when a layer-based algorithm is applied to image data, the image data may include color data (or a color image), depth data (or a depth image), or the like. The color data may include data that represents a plurality of colors for each plane of a plurality of planes. For example, the color data may include a red image, a blue image, and a green image. The layer-based algorithm is used to process data of each of the plurality of planes into which a reproduction area of a hologram is split based on depths. The image processing apparatus may generate a hologram image by performing a Fourier transform operation or an inverse Fourier transform operation on the data of each of the planes.

In operation <NUM>, the image processing apparatus performs an image quality correction operation and a field calculation operation. The image processing apparatus may correct the image data in order to improve an image quality.

In operation <NUM>, the image processing apparatus performs a Fourier transform operation or a fast Fourier transform (FFT) operation. For example, the image processing apparatus may perform a Fourier transform operation on a two-dimensional (2D) matrix type of image data. The image processing apparatus may perform a one-dimensional (1D) Fourier transform operation twice to accomplish a 2D Fourier transform. The image processing apparatus may perform a first 1D Fourier transform operation on the image data in a row direction and perform a second 1D Fourier transform operation on a result of the first 1D Fourier transform operation in a column direction. The image processing apparatus generates a hologram image via the Fourier transform operation.

The image processing apparatus may include a plurality of cores. The plurality of cores may be configured to perform a Fourier transform operation on the image data in parallel. For example, the image processing apparatus may allocate the image data of each plane to a respective one from among the plurality of cores, and each of the plurality of cores may perform a Fourier transform operation on the allocated image data. A process in which the image processing apparatus performs a Fourier transform operation on the image data according to exemplary embodiments will be described below in detail with reference to <FIG>, <FIG>, <FIG>, and <FIG>.

In operation <NUM>, the image processing apparatus performs a pixel encoding operation. The image processing apparatus generates data that is to be input to a screen, via the pixel encoding operation.

In operation <NUM>, the image processing apparatus outputs an image to an image display.

<FIG> illustrates a process of transforming data, according to an exemplary embodiment. Referring to <FIG>, the image processing apparatus or a Fourier transform apparatus generates final data <NUM> by performing a 1D FFT operation twice on image data <NUM> (i.e., a primary 2D FFT operation). For example, the image processing apparatus performs a 1D FFT operation once on the image data <NUM> in the column direction to generate intermediate data <NUM>, and then performs a 1D FFT operation once on the intermediate data <NUM> in the row direction to generate the final data <NUM>. A secondary 2D FFT operation may also be achieved by performing a 1D FFT operation twice. The primary 2D FFT operation is an FFT operation from the pupil of a user to the retina of the user, and the secondary 2D FFT operation may be an FFT operation from a panel to the pupil.

The order of execution, in terms of column and row directions, of 1D FFT operations for the primary 2D FFT operation may be opposite to that of execution of 1D FFT operations for the secondary 2D FFT operation. For example, if 1D FFT operations are performed in the column direction and then in the row direction when a primary 2D FFT operation is performed, 1D FFT operations may be performed in the row direction and then in the column direction when a secondary 2D FFT operation is performed.

In <FIG>, for example, a case in which the image processing apparatus performs a 1D FFT operation first in the column direction is described. In <FIG>, for example, a case in which the image processing apparatus performs a 1D FFT operation first in the row direction is described.

Although only a case where a primary 2D FFT operation is performed is illustrated in <FIG> and <FIG>, a secondary 2D FFT operation may also be performed in the same manner as the primary 2D FFT operation or by changing the order of a row and a column.

The image processing apparatus performs a 1D FFT operation on the image data <NUM> in the column direction. The intermediate data <NUM> is data obtained by performing a 1D FFT operation on the image data <NUM> in the column direction. Arrows marked on the image data <NUM> indicate directions in which the image processing apparatus performs a 1D FFT operation. Straight lines marked on the intermediate data <NUM> indicate directions in which the image data <NUM> is transformed.

The image processing apparatus reads stored intermediate data <NUM> from a memory and performs a 1D FFT operation on the read intermediate data <NUM> in the row direction. When reading out the intermediate data <NUM> from the memory, the image processing apparatus may read out the intermediate data <NUM> in the row direction and output the read-out intermediate data <NUM> to each 1D FFT processor.

The image processing apparatus generates the final data <NUM> by performing a 1D FFT operation on the intermediate data <NUM> in the row direction. The final data <NUM> is data obtained as the image data <NUM> is 1D FFT-transformed respectively in the column direction and the row direction.

<FIG> illustrates a process of transforming data, according to another exemplary embodiment. Referring to <FIG>, the image processing apparatus or a Fourier transform apparatus generates final data <NUM> by performing a 1D FFT operation twice on image data <NUM>. For example, the image processing apparatus performs a 1D FFT operation once on the image data <NUM> in the row direction to generate intermediate data <NUM> and then performs a 1D FFT operation once on the intermediate data <NUM> in the column direction to generate the final data <NUM>. In <FIG> and <FIG>, the order of a column and a row is switched and the description presented in <FIG> may be identically applied to the description of <FIG>.

<FIG> is a block diagram of an image processing apparatus <NUM>, according to an exemplary embodiment. Referring to <FIG>, the image processing apparatus <NUM> may include a camera <NUM>, a processor <NUM>, and a memory <NUM>. The image processing apparatus <NUM> may be any of an electronic apparatus (e.g., a computer, a mobile device, a display device, a wearable device, or a digital camera), a central processing unit (CPU), a graphic processing unit (GPU), or the like.

The camera <NUM> may capture an image and acquire a color image and a depth image from the captured image. The color image and the depth image are acquired in units of frames. The color image may be a composite image that includes a red image, a green image, and a blue image. Each of the red image, the green image, and the blue image is a single frame. The depth image is acquired for each color. In this aspect, the camera <NUM> acquires a depth image for the red image, a depth image for the green image, and a depth image for the blue image. The depth image for each of the red, green, and blue images is also a single frame.

The memory <NUM> stores the color image and the depth image. The memory <NUM> stores the frame generated by the processor <NUM>.

<FIG> is a block diagram of an image processing apparatus <NUM>, according to another exemplary embodiment. The image processing apparatus <NUM> may be any of a mobile device, a display device, a wearable device, a CPU, a GPU, or the like.

The image processing apparatus <NUM> includes a controller (also referred to herein as a "processor") <NUM>, a core <NUM>, and a memory <NUM>. The memory <NUM> may include dynamic random-access memory (DRAM) or static random-access memory (SRAM).

The controller <NUM> controls the core <NUM> and the memory <NUM>. The controller <NUM> may determine data that is input to the core <NUM>. The controller <NUM> may designate a calculation that is to be performed by the core <NUM>. For example, the controller <NUM> may control the core <NUM> to perform a 1D FFT operation on data in the row direction, and may also control the core <NUM> to perform a 1D FFT operation on data in the column direction. The controller <NUM> may store data generated during a Fourier transformation operation in the memory <NUM>.

The controller <NUM> controls the core <NUM> to perform a primary 2D FFT operation and a secondary 2D FFT operation on image data. The primary 2D FFT operation includes two 1D FFT operations, and the secondary 2D FFT operation includes two 1D FFT operations. The controller <NUM> may control the data that is input to the core <NUM>, in order to perform a 2D FFT operation twice (i.e., to perform a primary 2D FFT operation and a secondary 2D FFT operation). The controller <NUM> controls a flow of data so that the core <NUM> performs a primary 2D FFT operation and then performs a secondary 2D FFT operation. Accordingly, the image processing apparatus <NUM> may perform a primary 2D FFT operation and a secondary 2D FFT operation (i.e., a total of four 1D FFT operations) by using the single core <NUM>.

The controller <NUM> may reset the core <NUM>. Resetting the core <NUM> may refer to changing the amount of data that is processible by the core <NUM>. Resetting the core <NUM> may also refer to determining whether an FFT processor included in the core <NUM> is to operate. For example, the controller <NUM> may reset the core <NUM> so that the core <NUM> performs a <NUM>-POINT FFT operation or a <NUM>-POINT FFT operation. The controller <NUM> determines the amount of the data that is input to the core <NUM>, based on the flow rate of the data, and determines whether FFT processors included in the core <NUM> are to operate, based on the determined amount of data.

The core <NUM> may Fourier-transform data included in each line of the frame. For example, the core <NUM> may perform a 1D FFT operation on the frame in a row direction. A single row or a single column may be referred to as a single line. The core <NUM> performing a 1D FFT operation on the frame in the row direction indicates performing a 1D FFT operation on pixel values included in the row of the frame.

The core <NUM> may output the data to the memory <NUM>. Every time a result value obtained from performing a 1D FFT operation is generated, the core <NUM> may output the result value to the memory <NUM>.

The core <NUM> may include a plurality of 1D FFT processors. The 1D FFT processors may perform a respective 1D FFT operation on each line of the frame.

The memory <NUM> may store and output the data. The memory <NUM> may include SDRAM or DRAM.

<FIG> is a block diagram for explaining a flow of processing data, according to an exemplary embodiment.

An input interface <NUM> receives image data. The image data may be transmitted to a memory <NUM> or a core <NUM> via a bus.

The memory <NUM> may store and output data. The memory <NUM> may include SDRAM or DRAM.

An output interface <NUM> outputs image data. The output interface <NUM> may be implemented as a display.

The core <NUM> may be reset based on the amount of the image data in order to perform an FFT operation on the image data.

The controller <NUM> controls a flow of the image data that is processed by an image processing apparatus <NUM>, and resets the core <NUM> based on the amount of the image data that is input to the core <NUM>.

Although not shown in <FIG>, the image processing apparatus <NUM> may further include a plurality of operators. The plurality of operators may perform a focus term operation <NUM>, a depth summation operation <NUM>, a prism phase operation <NUM>, a left-right (L/R) summation operation <NUM>, and an encoding operation <NUM>.

<FIG> is a block diagram of a core <NUM>, according to an exemplary embodiment. Referring to <FIG>, the core <NUM> may perform an FFT operation on image data of two amounts.

The amount of the image data may vary based on the size of a panel. For example, when the size of the panel is <NUM> X <NUM>, the amount of the image data may be <NUM> X <NUM>. In this case, the core <NUM> needs to perform a <NUM> FFT operation and a <NUM> FFT operation. When the size of a replacement panel is <NUM> X <NUM>, the core <NUM> performs a <NUM> FFT operation and a <NUM> FFT operation.

In the core <NUM>, according to a mode signal of the controller <NUM>, only a basic module <NUM> may operate, or both a first additional module <NUM> and the basic module <NUM> may operate. Accordingly, the core <NUM> performs an FFT operation on image data of a first amount when only the basic module <NUM> operates, and performs an FFT operation on image data of a second amount when both the first additional module <NUM> and the basic module <NUM> operate.

The core <NUM> includes the first additional module <NUM> and the basic module <NUM>. The basic module <NUM> includes a plurality of FFT processors. For example, one FFT processor may be a <NUM>-POINT FFT processor. The first additional module <NUM> includes a single <NUM>-POINT FFT processor. Whether the first additional module <NUM> is to operate may be determined under the control of the controller <NUM>. Accordingly, when the first additional module <NUM> operates, the image data input to the first additional module <NUM> is transformed and then output to the basic module <NUM>. Conversely, when the first additional module <NUM> does not operate, the image data input to the first additional module <NUM> is output to the basic module <NUM> without being transformed.

<FIG> is a block diagram of a core <NUM>, according to an exemplary embodiment. Referring to <FIG>, the core <NUM> may perform an FFT operation on image data of three amounts. Only the basic module <NUM> may operate, the first additional module <NUM> and the basic module <NUM> may operate, or a second additional module <NUM>, the first additional module <NUM>, and the basic module <NUM> may operate. Accordingly, the core <NUM> performs an FFT operation on image data of a first amount when only the basic module <NUM> operates; performs an FFT operation on image data of a second amount when the first additional module <NUM> and the basic module <NUM> operate; and performs an FFT operation on image data of a third amount when the second additional module <NUM>, the first additional module <NUM>, and the basic module <NUM> operate.

Although the core <NUM> that includes two additional modules, namely, the first and second additional modules <NUM> and <NUM>, is illustrated in <FIG>, the core <NUM> may include three or more additional modules, and may perform an FFT operation on image data of four or more amounts.

<FIG> is a block diagram of a core <NUM>, according to an exemplary embodiment. Referring to <FIG>, the core <NUM> is an example of a <NUM>-FFT module. The core <NUM> may control an additional module <NUM> to perform a <NUM>-FFT operation or a <NUM>-FFT operation.

For example, the additional module <NUM> performs a function of a <NUM>-POINT processor. A <NUM>-FFT module <NUM> includes a plurality of <NUM>-POINT processors or a plurality of <NUM>-POINT FFT processors. For example, the <NUM>-FFT module <NUM> may include ten <NUM>-POINT FFT processors.

Image data is input to the additional module <NUM>. The image data is input to a ButterFly (BF) unit (also referred to herein as a ButterFly (BF) component) <NUM> and a multiplexer (MUX) <NUM> included in the additional module <NUM>.

A mode signal is input to the MUX <NUM>. A signal that is output by the MUX <NUM> varies based on the mode signal. For example, when the mode signal is one (<NUM>), the MUX <NUM> outputs the input image data to the additional module <NUM>. When the mode signal is zero (<NUM>), the MUX <NUM> outputs image data obtained from a transformation operation performed by the additional module <NUM>. In particular, when the mode signal is <NUM>, the MUX <NUM> outputs data received from a complex multiplier <NUM>.

A twiddle factor (TF) read-only memory (ROM) <NUM> outputs a TF value. The TF ROM <NUM> may include any of a shift register, a cache, a memory, or the like.

The BF unit <NUM> performs an FFT operation or an inverse FFT (IFFT) operation on the received image data. The BF unit <NUM> is controlled by a specific bit of an up-counter and constructs a single data path by using a Simple Dual-Port Block RAM (SDP-BRAM)-based delay feedback logic.

The complex multiplier <NUM> performs a complex multiplication operation on the TF value output by the TF ROM <NUM> and data output by the BF unit <NUM>.

Whether the additional module <NUM> is to operate is determined based on the mode signal. According to whether the additional module <NUM> operates, a determination is made as to whether the core <NUM> is to operate as a <NUM>-FFT module or as a <NUM>-FFT module.

<FIG> is a block diagram of the core <NUM> of <FIG>. <FIG> illustrates the core <NUM> of <FIG> in greater detail.

The <NUM>-FFT module <NUM> includes five <NUM>-POINT FFT processors. Each <NUM>-POINT FFT processor includes two BF units, two TR ROMs, and one complex multiplier. Each BF unit may be a BF2I component or a BF2II component. A clock signal is applied to each BF unit.

The additional module <NUM> includes the same components as those described above with reference to <FIG>. The additional module <NUM> may be connected to a front end of the <NUM>-FFT module <NUM> and configured to determine the image data that is input to the <NUM>-FFT module <NUM>. The additional module <NUM> may include the FFT processors included in the <NUM>-FFT MODULE <NUM> and a MUX, and may output image data and/or transformed image data.

<FIG> is a block diagram of a core <NUM>, according to an exemplary embodiment. Referring to <FIG>, the core <NUM> is a detailed illustration of the core <NUM> of <FIG>.

The core <NUM> may process image data of three amounts. For example, the core <NUM> may perform a <NUM>-POINT FFT operation, a <NUM>-POINT FFT operation, or a <NUM>-POINT FFT operation, wherein <NUM>, <NUM>, and <NUM> indicate amounts of image data.

Since the core <NUM> includes two additional modules, whether the additional modules are to operate is determined based on respective amounts of data. When the amount of data is equal to <NUM>, a <NUM> is input as a first mode signal to a MUX of a <NUM>-FFT module <NUM>. When the amount of data is equal to <NUM>, a <NUM> is input as the first mode signal to the MUX of the <NUM>-FFT module <NUM>, and a <NUM> is input as a second mode signal to a MUX of an additional module <NUM>. When the amount of data is equal to <NUM>, a <NUM> is input as each of the first and second mode signals to the MUXes of the <NUM>-FFT module <NUM> and the additional module <NUM>.

<FIG> is a flowchart of an image processing method, according to an exemplary embodiment.

In operation <NUM>, an image processing apparatus determines the amount of image data. The amount of the image data may vary based on the size of a panel. The amount of the image data includes the sizes of rows and columns of the image data.

In operation <NUM>, the image processing apparatus sets a core based on the amount of the image data. The image processing apparatus sets the core based on the sizes of columns when performing an FFT operation on the image data in the row direction, and sets the core based on the sizes of rows when performing an FFT operation on the image data in the row direction. In particular, when rows X columns of the image data is equivalent to <NUM> X <NUM>, the image processing apparatus needs to perform a <NUM>-POINT FFT operation in order to perform a FFT operation in the row direction (because the number of rows is equal to <NUM>, whereas one row includes data of <NUM>-POINT).

The image processing apparatus determines whether an additional module included in the core is to operate, and outputs a control signal (or mode signal) to the additional module based on the determination. A controller of the image processing apparatus may output a <NUM> or a <NUM> to a MUX included in the additional module and thus may determine output data of the MUX. For example, when a <NUM> is input to the MUX, the image data input to the additional module is output by the MUX. When a <NUM> is input to the MUX, the image data obtained from transformation performed by the additional module is output by the MUX.

The core of the image processing apparatus may include a basic module, a first additional module, and a second additional module. The image processing apparatus outputs the control signal to a MUX of the first additional module and a MUX of the second additional module based on the amount of the image data.

In operation <NUM>, the core of the image processing apparatus performs a 1D FFT operation on the image data.

According to an exemplary embodiment, the image processing apparatus may perform an FFT operation on image data even when the amount of the image data is changed by controlling an operation of the additional module.

According to an exemplary embodiment, the image processing apparatus may process various amounts of image data by using the additional module connected to the basic module.

The apparatuses described herein may comprise a processor, a memory configured for storing program data and executing a program that relates to the stored program data, a permanent storage unit such as a disk drive, a communications port configured for handling communications with external devices, and user interface devices, including a touch panel, keys, buttons, etc. When software modules or algorithms are involved, these software modules may be stored as program instructions or computer readable codes executable on a processor on a transitory or non-transitory computer-readable recording medium. Examples of the non-transitory computer-readable recording medium include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.), and optical recording media (e.g., compact disc - ROMs (CD-ROMs), or Digital Versatile Discs (DVDs)). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributive manner. This media can be read by the computer, stored in the memory, and executed by the processor.

Exemplary embodiments may be described in terms of functional block components and various processing steps. Such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, exemplary embodiments may employ various integrated circuit (IC) components, e.g., memory elements, processing elements, logic elements, look-up tables, and the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. Similarly, where the elements are implemented by using software programming or software elements, the exemplary embodiments described herein may be implemented with any programming or scripting language such as C, C++, Java, assembler language, or the like, with the various algorithms being implemented with any combination of data structures, objects, processes, routines or other programming elements. Functional aspects may be implemented by using algorithms that are executed on one or more processors. Furthermore, the exemplary embodiments described herein could employ any number of conventional techniques for electronics configuration, signal processing and/or control, data processing and the like. The words "mechanism," "element," "means," and "configuration" are used broadly and are not limited to mechanical or physical embodiments, but can include software routines in conjunction with processors, etc..

The particular implementations shown and described herein are illustrative examples and are not intended to otherwise limit the scope of the present inventive concept in any way. For the sake of brevity, conventional electronics, control systems, software development and other functional aspects of the systems may not be described in detail. Furthermore, the connecting lines, or connectors shown in the various figures presented are intended to represent exemplary functional relationships and/or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships, physical connections or logical connections may be present in a practical apparatus.

Claim 1:
An image processing apparatus (<NUM>), comprising
a core (<NUM>) configured to perform a fast Fourier transformation, FFT, operation on image data;
a memory (<NUM>) configured to store data that is output by the core; and
a controller (<NUM>) configured to control the core to perform a primary 2D FFT operation and a secondary 2D FFT operation by executing a 1D FFT operation twice on the image data in terms of column and rows directions for each of 2D FFT operations,
wherein the core is resettable based on an amount of the image data,
wherein the order of execution, in terms of column and row directions, of 1D FFT operations for the primary 2D FFT operation is opposite to that of execution of 1D FFT operations for the secondary 2D FFT operation,
wherein 1D FFT operations are performed in the column direction and then in the row direction when a primary 2D FFT operation is performed, 1D FFT operations are performed in the row direction and then in the column direction when a secondary 2D FFT operation is performed, and
wherein the controller (<NUM>) is further configured to set the core based on a size of the column of the image data, when performing 1D FFT operation on the image data in the row direction, and set the core based on a size of the row of the image data, when performing 1D FFT operation on the image data in the column direction.