Abstract:
An image interpolation apparatus which generates a single frame of image data interpolated using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, comprises a direction decision section configured to detect a direction in which there is a strong correlation in a position in the field corresponding to the pixel position of the single frame of image data, on the basis of an index value for a specific direction calculated from a plurality of groups of pixel data arranged in the specific direction in the vicinity of the position, and an interpolation section configured to determine pixel data in the pixel position, using operation data selected and switched on the basis of which a field to be interpolated comes under and of the direction in which there is a strong correlation.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]     This is a Continuation Application of PCT Application No. PCT/JP2004/018296, Dec. 8, 2004, which was published under PCT Article 21(2) in Japanese. 
     
    
       [0002]     This application is based upon and claims the benefit of priority from prior Japanese Patent Application No. 2003-408885, filed Dec. 8, 2003, the entire contents of which are incorporated herein by reference.  
       BACKGROUND OF THE INVENTION  
       [0003]     1. Field of the Invention  
         [0004]     This invention relates to an image interpolation apparatus and an image interpolation method which generate high-quality frame images from a plurality of frames of image data consecutively input in time-series order.  
         [0005]     2. Description of the Related Art  
         [0006]     The technique for interpolating image data is required in various cases, including a case where an image is enlarged, a case where the resolution of an image is increased, and a case where an image obtained by interlaced scanning is converted into a progressive-scanned image.  
         [0007]     A conventional interpolating method has employed an approach of, for example, when the number of lines is increased in the vertical direction, using the value of the line immediately above or below the line to be interpolated without change or the average of the value of the upper line and that of the lower line. Such a method has a problem: when there is an oblique line on the photographic subject, the interpolated image has a ragged outline or gets blurred.  
         [0008]     To overcome this problem, U.S. Pat. No. 5,825,429 has proposed the following method. Of the field data obtained by interlaced scanning, when the data of three consecutive fields, a first field, a second field, and a third field, are given, the image of the second field is converted into a frame image. To do this, it is determined whether there is a movement between images in a window set in each of the first field image and third field image, the position of the window corresponding to a pixel to be interpolated in the second field. Then, according to the result of the determination, interpolation is performed.  
         [0009]     Whether there is a movement is determined by adding the absolute values in pixel level of difference between the window of the first field and that of the third field corresponding to the pixel position to be interpolated and determining whether the obtained additional value exceeds a predetermined value. If it has been determined that there is a movement between the windows, interpolated image data is created on the basis of the pixels vertically adjacent to each other since there is a stronger correlation in the window of the second field image than the correlation between the window of the first field and that of the third field. If it has been determined that there is no movement between the windows, a check is made in the direction in which there is a correlation in the windows. Specifically, data on the pixels above and below the pixels horizontally adjacent to the position to be interpolated are extracted, the additional value of these data is found, and it is determined whether the obtained additional value has been strongly increased or decreased than the additional value of the pixels present in the horizontal direction. If the obtained additional value has not been increased or decreased that way, an interpolated value is generated using the image data above and below the position to be interpolated. If the obtained additional value has been increased or decreased that way, it is determined that the image in that part is a step-like area. Then, interpolated data is generated from pixel data with strong correlation in three directions, that is, not only above and below the pixel to be interpolated but also obliquely above and below the pixel to be interpolated.  
       BRIEF SUMMARY OF THE INVENTION  
       [0010]     According to a first aspect of the present invention, there is provided an image interpolation apparatus which generates a single frame of image data interpolated using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, the image interpolation apparatus comprising:  
         [0011]     an index value computing section configured to calculate an index value for a specific direction from a plurality of groups of pixel data arranged in the specific direction in the vicinity of a position in the field corresponding to the pixel position of the single frame of image data;  
         [0012]     a direction decision section configured to detect a direction in which there is a strong correlation in the pixel position on the basis of the index value calculated at the index value computing section;  
         [0013]     a switching section configured to select operation data used in operation for interpolation from the field on the basis of which of the fields where the line reading method has been switched periodically a field to be interpolated comes under and of the direction in which there is a strong correlation detected by the direction decision section and switch operation data; and  
         [0014]     an interpolation section configured to determine pixel data in the pixel position using the operation data switched by the switching section.  
         [0015]     According to a second aspect of the present invention, there is provided an image interpolation apparatus which generates a single frame of color image data whose size is changed in magnification at least one of the horizontal direction and the vertical direction by using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, the image interpolation apparatus comprising:  
         [0016]     an index value computing section configured to extract discrete pixel data from the fields, select a plurality of groups of pixel data arranged in a specific direction from the pixel data in the vicinity of a position corresponding to the pixel position of the single frame of color image data in the extracted pixel data, and calculate an index value for the specific direction;  
         [0017]     a direction decision section configured to detect a direction in which there is a strong correlation in the pixel position on the basis of the index value calculated at the index value computing section;  
         [0018]     a switching section configured to select operation data used in operation for interpolation for each of a plurality of primary colors constituting the color image data from the field on the basis of which of the fields where the line reading method has been switched periodically a field to be interpolated at the change of magnification comes under and of the direction in which there is a strong correlation detected by the direction decision section and switch operation data; and  
         [0019]     an interpolation section configured to determine pixel data in the pixel position using the operation data switched by the switching section.  
         [0020]     According to a third aspect of the present invention, there is provided an image interpolation method which generates a single frame of image data interpolated using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, the image interpolation method comprising:  
         [0021]     calculating an index value for a specific direction from a plurality of groups of pixel data arranged in the specific direction in the vicinity of a position in the field corresponding to the pixel position of the single frame of image data;  
         [0022]     detecting a direction in which there is a strong correlation in the pixel position on the basis of the index value;  
         [0023]     selecting operation data used in operation for interpolation from the field on the basis of which of the fields where the line reading method has been switched periodically a field to be interpolated comes under and of the direction in which there is a strong correlation and switching operation data; and  
         [0024]     determining pixel data in the pixel position using the operation data.  
         [0025]     According to a fourth aspect of the present invention, there is provided an image interpolation apparatus which generates a single frame of image data interpolated using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, the image interpolation apparatus comprising:  
         [0026]     index value computing means for calculating an index value for a specific direction from a plurality of groups of pixel data arranged in the specific direction in the vicinity of a position in the field corresponding to the pixel position of the single frame of image data;  
         [0027]     direction decision means for detecting a direction in which there is a strong correlation in the pixel position on the basis of the index value calculated at the index value computing means;  
         [0028]     switching means for selecting operation data used in operation for interpolation from the field on the basis of which of the fields where the line reading method has been switched periodically a field to be interpolated comes under and of the direction in which there is a strong correlation detected by the direction decision means and switching operation data; and  
         [0029]     interpolation means for determining pixel data in the pixel position using the operation data switched by the switching means.  
         [0030]     According to a fifth aspect of the present invention, there is provided an image interpolation apparatus which generates a single frame of color image data whose size is changed in magnification at least one of the horizontal direction and the vertical direction by using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, the image interpolation apparatus comprising:  
         [0031]     index value computing means for extracting discrete pixel data from the fields, selecting a plurality of groups of pixel data arranged in a specific direction from the pixel data in the vicinity of a position corresponding to the pixel position of the single frame of color image data in the extracted pixel data, and calculating an index value for the specific direction;  
         [0032]     direction decision means for detecting a direction in which there is a strong correlation in the pixel position on the basis of the index value calculated at the index value computing means;  
         [0033]     switching means for selecting operation data used in operation for interpolation for each of a plurality of primary colors constituting the color image data from the field on the basis of which of the fields where the line reading method has been switched periodically a field to be interpolated at the change of magnification comes under and of the direction in which there is a strong correlation detected by the direction decision means and switching operation data; and  
         [0034]     interpolation means for determining pixel data in the pixel position using the operation data switched by the switching means.  
         [0035]     According to a sixth aspect of the present invention, there is provided an image interpolation method which generates a single frame of image data interpolated using pixel data in a plurality of fields input in time-series order while switching a line reading method periodically, the image interpolation method comprising:  
         [0036]     an index value computing step of calculating an index value for a specific direction from a plurality of groups of pixel data arranged in the specific direction in the vicinity of a position in the field corresponding to the pixel position of the single frame of image data;  
         [0037]     a direction decision step of detecting a direction in which there is a strong correlation in the pixel position on the basis of the index value;  
         [0038]     a switching step of selecting operation data used in operation for interpolation from the field on the basis of which of the fields where the line reading method has been switched periodically a field to be interpolated comes under and of the direction in which there is a strong correlation and switching operation data; and  
         [0039]     an interpolation step of determining pixel data in the pixel position using the operation data.  
         [0040]     Advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter. 
     
    
     BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING  
       [0041]     The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention, and together with the general description given above and the detailed description of the embodiments given below, serve to explain the principles of the invention.  
         [0042]      FIG. 1  shows the configuration of an image interpolation apparatus according to a first embodiment of the present invention;  
         [0043]      FIG. 2A  shows an example of the data stored in a first memory when thinning-out sampling is done;  
         [0044]      FIG. 2B  shows an example of the data stored in a second memory when thinning-out sampling is done;  
         [0045]      FIG. 2C  shows an example of the data obtained from the first and second memories;  
         [0046]      FIG. 2D  shows an example of the data stored in the first memory when the arithmetic average of consecutive pixel data is calculated and the result is sampled;  
         [0047]      FIG. 2E  shows an example of the data stored in the second memory when the arithmetic average of consecutive pixel data is calculated and the result is sampled;  
         [0048]      FIG. 3  shows a detailed configuration of the index value computing section of  FIG. 1 ;  
         [0049]      FIG. 4  shows an example of the image data taken in by the data taking-in section during interpolation;  
         [0050]      FIG. 5  shows a combination of pixels used in detecting a direction in which there is a strong correlation;  
         [0051]      FIG. 6  shows an example of pixels used according to a direction in which there is a correlation;  
         [0052]      FIG. 7  shows the configuration of an image interpolation apparatus according to a second embodiment of the present invention;  
         [0053]      FIG. 8  shows an example of sampling done in the horizontal direction;  
         [0054]      FIG. 9A  shows an example of the data stored in a first memory when the size of an image is reduced by half;  
         [0055]      FIG. 9B  shows an example of the data stored in a second memory when the size of an image is reduced by half;  
         [0056]      FIG. 9C  shows an example of the data obtained from the first and second memories;  
         [0057]      FIG. 10  shows an example of the image data taken in by the data taking-in section during interpolation;  
         [0058]      FIG. 11  shows a combination of pixels used in detecting a direction in which there is a strong correlation;  
         [0059]      FIG. 12  shows an example of pixels (only G) used according to a direction in which there is a correlation;  
         [0060]      FIG. 13A  is a diagram to help explain a method of interpolating the top field when there is a strong correlation in the vertical direction;  
         [0061]      FIG. 13B  is a diagram to help explain a method of interpolating the bottom field when there is a strong correlation in the vertical direction; and  
         [0062]      FIG. 14  shows a pixel data array causing erroneous decision in a direction in which there is a correlation in the prior art. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0063]     Hereinafter, referring to the accompanying drawings, the best mode for carrying out the invention will be explained.  
       First Embodiment  
       [0064]     (Configuration)  
         [0065]     As shown in  FIG. 1 , in an image interpolation apparatus according to a first embodiment of the present invention, the image data obtained in capturing by an image sensor  10  is converted into digital image data by an A/D converter  11 . Under the control of a data control section  12 , the image data output from the A/D converter  11  is transferred to a first memory  13  or a second memory  14  connected to the data control section  12  and stored there.  
         [0066]     An index value computing section  15  is connected to the first memory  13  and second memory  14 . A direction decision section  16  is connected to the index value computing section  15 . The direction decision section  16  receives the data output from the index value computing section  15  and operates on the basis of the data. The data control section  12 , first memory  13 , second memory  14 , and direction decision section  16  are connected to a switching section  17 . The switching section  17  operates on the basis of the control of the data control section  12  and the data output from the direction decision section  16 . An interpolation section  18  is connected to the switching section  17 .  
         [0067]     The image data output from the interpolation section  18  is transferred to an output section  19 , such as a monitor or a memory card, which displays the data or records and stores the data.  
         [0068]     In  FIG. 1 , the broken line indicates the flow of the control signal.  
         [0069]     (Operation)  
         [0070]     The signal flow in the above configuration will be explained.  
         [0071]     The image sensor  10  outputs analog image data corresponding to the incoming subject image passed through a lens system and a low-pass filter (which are not shown). The analog image data is converted into digital image data by the A/D converter  11 . In the first embodiment, the image sensor  10  is assumed to be a monochromatic one with no color filter.  
         [0072]     Under the control of the data control section  12 , the image data output from the A/D converter  11  is selectively stored in the first memory  13  and second memory  14  each functioning as a frame memory. Specifically, the data control section  12  switches connections between the first memory  13  and the second memory  14  and transfers data to the first memory  13  and second memory  14  alternately in such a manner that consecutive image data are stored in the two memories  13 ,  14 . For example, in the case of interlaced scanning, the image data composed of the pixel data of odd-numbered lines are stored in one memory, and the image data composed of the pixel data of even-numbered lines are stored in the other memory.  
         [0073]     The index value computing section  15  connected to the two memories  13 ,  14  receives the pixel data around the position to be interpolated from the memories  13 ,  14  and finds an index value for a specific direction. The index value for the specific direction found by the index value computing section  15  is sent to the direction decision section  16 . On the basis of the index value, the direction decision section  16  determines in which direction there is a strong correlation in the position to be interpolated.  
         [0074]     The switching section  17  switches the operation according to the control of the data control section  12  and the direction with a strong correlation output from the direction decision section  16  and sends the image data necessary for interpolation from the first memory  13  or second memory  14  to the interpolation section  18 .  
         [0075]     The interpolation section  18  finds an interpolated value using the pixel data received from the switching section  17  and creates interpolated image data. Then, the image data created by the interpolation section  18  is displayed on a display medium, such as a monitor, or is recorded on a recording medium, such as a memory card, at the output section  19 .  
         [0076]     Hereinafter, the operation of each section will be explained in more detail with reference to  FIGS. 2A  to  2 E.  FIGS. 2A  to  2 E show an example of the data format stored in the first memory  13  and second memory  14  when the size of an image is reduced by half in the first embodiment. In  FIGS. 2A  to  2 E, of the input data to the data control section  12 , the white pixels represent the pixels from which data are read and stored in the memory  13  or  14 . The hatched pixels represent the pixels from which no data is read, that is, the pixels whose data are not stored.  
         [0077]     Specifically, when thinning-out sampling is done, the data ( FIG. 2A ) transferred from the data control section  12  to the first memory  13  and stored there and the data ( FIG. 2B ) transferred from the data control section  12  to the second memory  14  and stored there are designed to be data at intervals of one pixel in the horizontal direction and data at intervals of one line in the vertical direction. At this time, the data control section  12  shifts the unread lines in frames and performs control in such manner that read pixel positions do not overlap with one another in consecutive image data. Then, under the control of the data control section  12 , the switching section  17  combines the data stored in the memories  13  and  14  with each other, thereby producing the data as shown in  FIG. 2C .  
         [0078]     When the arithmetic average of consecutive pixel data is calculated and the result is sampled, the data stored in the first memory  13  is as shown in, for example,  FIG. 2D  and the data stored in the second memory  14  is as shown in, for example,  FIG. 2E . In this case, the data control section  12  stores in the memories  13 ,  14  the values obtained by adding the pixel data connected with the white circle in the figures, that is, the data control section  12  calculates the arithmetic average of the data on two consecutive pixels in the horizontal direction and stores the result in the memories  13 ,  14 . As a result, under the control of the data control section  12 , the switching section  17  combines the data stored in the two memories  13 ,  14  with one another, thereby producing the data corresponding to the array shown in  FIG. 2C  as in the thinning-out sampling.  
         [0079]     Then, the interpolation section  18  finds the interpolated values in the positions shown by the white circles in  FIG. 2C , thereby creating an interpolated image reduced by half.  
         [0080]     While in the example, the image size has been reduced by half, the reduction ratio is, of course, not limited to half in the present invention.  
         [0081]     For example, the same holds true even when the image size is reduced by one third. In this case, however, it is necessary to provide three memories, or a first memory, a second memory, and a third memory. The data control section  12  transfers data on every three pixels in the horizontal direction and data on every three lines in the vertical direction to the three memories in frames sequentially. In this case, too, the data control section  12  shifts the unread lines image by image and performs control in such manner that read pixel positions do not overlap with one another in consecutive image data. At this time, one frame of image data is created from the thinned-out data corresponding to three consecutive screens.  
         [0082]     As shown in  FIG. 3 , the index value computing section  15  used in the first embodiment is composed of a data taking-in section  21 , an absolute difference value detecting section  22 , and an adding section  23 . The first memory  13  and second memory  14  are connected to the data taking-in section  21 . The data taking-in section  21  is connected to the absolute difference value detecting section  22 . The absolute difference value detecting section  22  is connected to the adding section  23 . The adding section  23  is connected to the direction decision section  16 .  
         [0083]     In the index value computing section  15  configured as described above, to interpolate an arbitrary point, the data taking-in section  21  takes in the image data in a block of a specific size centered on the pixel position to be interpolated from the first memory  13  and second memory  14 . Of the pixel data in the block taken in by the data taking-in section  21 , the absolute difference value detecting section  22  extracts a set of pixels adjacent to one another in a specific direction in a plurality of places, and calculates the absolute difference value between the individual sets. The adding section  23  calculates the sum total of the absolute difference values calculated at the absolute difference value detecting section  22  in the specific direction and outputs an index value for each direction.  
         [0084]     To find the image data interpolated value for the position of a white circle in the center of a block of a specific size (in this case, 4×4 pixels), the data taken in by the data taking-in section  21  is, for example, as shown in  FIG. 4 . In  FIG. 4 , the data of the odd-numbered lines represent the data transferred from the first memory  13 . The data of the even-numbered lines represent the data transferred from the second memory  14 . That is, in the first embodiment, the image data in a block of 4×4 pixels obtained by combining the data stored in the two memories  13 ,  14 , centering on the position to be interpolated, are taken in. In  FIG. 4 , the white pixels represent the image data read.  
         [0085]     In the absolute difference value detecting section  22 , from the taken-in pixel data, the sets of pixel data used in sensing the absolute difference value in a specific direction are, for example, as shown in  FIG. 5 . In the example, sensing is done in the following four directions as the specific directions: a horizontal direction, a vertical direction, a diagonally-right-up direction at 45 degrees, and a diagonally-right-down direction at 45 degrees.  
         [0086]     The adding section  23  calculates the sum total of the absolute difference values between the sets of pixels in each direction and finds an index value. If the index value for the horizontal direction is V hor , the index value for the vertical direction is V ver , the index value for the diagonally-right-up direction is V sla , and the index value for the diagonally-right-down direction is V bac , the index values in the respective directions are determined using the following equation (1):  
             [     Formula   ⁢           ⁢   1     ]                           {             Vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   ver       =              I   22     -     I   42            +            I   13     -     I   33                            Horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   hor       =              I   22     -     I   24            +            I   31     -     I   33                            Diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   sla       =              I   13     -     I   31            +            I   24     -     I   42                            Diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   bac       =              I   11     -     I   33            +            I   22     -     I   44                              (   1   )             
 
         [0087]     While in the first embodiment, the index value is determined from the average value of the absolute difference values between two pixels, the present invention is not limited to this. For instance, the accumulated value of the absolute difference values or the ratio of the pixel values at two points may be used. As long as a set of pixels used to find the absolute difference values is a set of pixels arranged in each direction, it is not limited to the set as shown in the Formula 1. Moreover, the number of sets of pixels used to find the absolute difference values is not limited to the case as shown in the Formula 1.  
         [0088]     The direction decision section  16  receives the index value data about the specific direction output from the index value computing section  15  and determines the direction in which there is the strongest correlation even in the position to be interpolated. To determine the correlation, the ratio of each index value to another index value is used. That is, if the index value meets the conditions shown in expressions (2) below, it is determined that there is a strong correlation in each direction.  
             [     Formula   ⁢           ⁢   2     ]                           {             Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢       V   ver       V   hor         &lt;         V   sla       V   bac       ⁢   and   ⁢       V   ver       V   hor         &lt;       V   bac       V   sla                     Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢       V   hor       V   ver         &lt;         V   sla       V   bac       ⁢   and   ⁢       V   hor       V   ver         &lt;       V   bac       V   sla                     Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢           ⁢   at   ⁢           ⁢   45   ⁢           ⁢   degrees   ⁢     :     ⁢           ⁢       V   sla       V   bac         &lt;         V   ver       V   hor       ⁢   and   ⁢       V   sla       V   bac         &lt;       V   hor       V   ver                     Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢           ⁢   at   ⁢           ⁢   45   ⁢           ⁢   degrees   ⁢     :     ⁢           ⁢       V   bac       V   sla         &lt;         V   ver       V   hor       ⁢   and   ⁢       V   bac       V   sla         &lt;       V   hor       V   ver                       (   2   )             
 
         [0089]     If the index value meets none of the above conditions and there is no correlation in each direction, or each index value is smaller than a predetermined specific value and the correlation is almost the same in all of the directions, it is determined that there is no correlation in a specific direction.  
         [0090]     Then, the direction decision section  16  outputs an identifying signal caused to correspond to each direction. For example, the following values are output: horizontal direction=1, vertical direction=2, diagonally-right-up direction at 45 degrees=3, diagonally-right-down direction at 45 degrees=4, and no correlation=0.  
         [0091]     While in the first embodiment, the direction has been determined from the ratio of index values, the present invention is not limited to this. The invention may use a method of determining a direction in which the minimum value of the index value is obtained to be a direction in which there is a strong correlation.  
         [0092]     The data control section  12  connected to the switching section  17  switches connections between the two memories  13  and  14  frame by frame and stores consecutive image data in the two memories  13 ,  14  alternately as described above. Then, the data control section  12  transmits to the switching section  17  a control signal indicating in which memory the latest input data has been recorded. For example, in the case of interlaced scanning, image data made up of the pixel data of the even-numbered lines and image data made up of the pixel data of the odd-numbered lines are stored in the memories  13 ,  14  in frames sequentially. When the latest input data is the pixel data in the even-numbered lines, “0” is sent as the control signal to the switching section  17 . When the latest input data is the pixel data of the odd-numbered lines, “1” is sent as the control signal to the switching section  17 .  
         [0093]     While numerals have been used as the values of the control signal, the control signal is, of course, not limited to these, as long as it can be determined in which memory the latest input data has been recorded.  
         [0094]     According to the identifying signal output from the direction decision section  16  and the control signal from the data control section  12 , the switching section  17  switches between the image data stored in the first memory  13  and that in the second memory  14  to select pixel data to be used for interpolation and transfers the selected pixel data to the interpolation section  18 .  
         [0095]     In the first embodiment, the selected pixel data to be used which is determined on the basis of the identifying signal input from the direction decision section  16  and the control signal from the data control section  12  is, for example, as shown in  FIG. 6 . In the table of  FIG. 6 , the direction in which there is a strong correlation is represented horizontally as the horizontal direction, the vertical direction, the diagonally-right-up direction at 45 degrees, and the diagonally-right-down direction at 45 degrees. In addition, the latest input data is represented vertically as even-numbered line data and odd-numbered line data.  
         [0096]     Using the pixel data input from the switching section  17 , the interpolation section  18  calculates an interpolated value by multiplying the pixel data by the weighting factor according to the position of each pixel data and the distance from the position in which an interpolated value is to be determined and averaging the result. That is, in the first embodiment, according to the latest input data and the direction in which there is a strong correlation, an interpolated value is determined using the following expressions (3) and (4).  
         [0097]     &lt;&lt;When the Latest Input Data is Odd-Numbered Line Data&gt;&gt; 
             [     Formula   ⁢           ⁢   3     ]                           {           Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   4     ⁢     (           I   11     +     3   ⁢     I   31         4     +     3   ⁢         I   13     +     3   ⁢     I   33         4         )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   4     ⁢     (           I   11     +     3   ⁢     I   13         4     +     3   ⁢         I   31     +     3   ⁢     I   33         4         )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   2     ⁢     (           I   31     +     I   13       2     +     I   33       )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢         I   11     +     3   ⁢     I   33         4                 No   ⁢           ⁢   correlation   ⁢     :     ⁢           ⁢         I   22     +     I   33       2                     (   3   )             
 
         [0098]     &lt;&lt;When the Latest Input Data is Even-Numbered Line Data&gt;&gt; 
             [     Formula   ⁢           ⁢   4     ]                           {           Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   4     ⁢     (       3   ⁢         3   ⁢     I   22       +     I   42       4       +         3   ⁢     I   24       +     I   44       4       )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   4     ⁢     (       3   ⁢         3   ⁢     I   22       +     I   24       4       +         3   ⁢     I   42       +     I   44       4       )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   2     ⁢     (       I   22     +         I   24     +     I   42       2       )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢         3   ⁢     I   22       +     I   44       4                 No   ⁢           ⁢   correlation   ⁢     :     ⁢           ⁢         I   22     +     I   33       2                     (   4   )             
 
         [0099]     While in the first embodiment, weighted average has been used in determining the interpolated value, this invention is not limited to this. Another method, such as the median value, mode value, or arithmetic average of adjacent pixels, may be used.  
         [0100]     The image data interpolated this way is transferred from the interpolation section  18  to the output section  19  and is stored in a recording medium, such as a memory card or a hard disk or displayed on a display medium, such as a monitor.  
         [0101]     As described above, according to the first embodiment, it is possible to detect with high accuracy a direction in which there is a strong correlation and create a high-accuracy interpolated image.  
         [0102]     Furthermore, in the index value computing section  15 , for example, by using the difference between the absolute values of pixel sets in the individual directions, such as the difference between the absolute value of I 22 -I 42  and the absolute value of I 13 -I 33  in the vertical direction, as the index value, it is possible to determine the direction in which the subject or camera moves between frames at the direction decision section  16 . Therefore, according to the state of the subject or camera, the switching section  17  can switch the pixel data used in interpolation at the interpolation section  18 .  
       Second Embodiment  
       [0103]     (Configuration)  
         [0104]     In an image interpolation apparatus according to a second embodiment of the present invention, a color filter  30  is provided in front of an image sensor  10 . The image of a subject entering the image sensor  10  via a lens system and a low-pass filter (which are not shown) and the color filter  30  is picked up by the image sensor  10 . The color video data obtained at the image sensor  10  is converted by an A/D converter  11  into digital color image data.  
         [0105]     Under the control of the data control section  12 , the color image data output from the A/D converter  11  is transferred to a first memory  13  or a second memory  14  connected to the data control section  12  and stored there.  
         [0106]     An index value computing section  15  is connected to the first memory  13  and second memory  14 . A direction decision section  16  is connected to the index value computing section  15 . The direction decision section  16  receives the data output from the index value computing section  15  and operates on the basis of the data. The direction decision section  16 , data control section  12 , first memory  13  and second memory  14  are connected to a switching section  17 . The switching section  17  operates on the basis of the control of data control section  12  and the data output from the direction decision section  16 . An interpolation section  18  is connected to the switching section  17 .  
         [0107]     The color image data output from the interpolation section  18  is transferred to an output section  19 , such as a monitor or a memory card, which displays the data or records and stores the data.  
         [0108]     (Operation)  
         [0109]     Next, the signal flow in the configuration of  FIG. 7  will be explained. The basic operation of the image interpolation apparatus of the second embodiment is almost the same as that of the first embodiment. Therefore, what differs from the first embodiment will be mainly explained.  
         [0110]     In the second embodiment, the color filter  30  is provided in front of the image sensor  10 . Analog color video data is obtained from the image sensor  10  and is converted by the A/D converter  11  into digital color image data. In the second embodiment, a CMOS (Complementary Metal-Oxide Semiconductor) is used as the image sensor  10 . A primary-color Bayer filter composed of the three primary colors, red (R), green (G), and blue (B), is used as the color filter  30 . In the Bayer filter, the frequency of appearance of a G pixel is twice that of an R pixel and that of a B pixel.  
         [0111]     Under the control of the data control section  12 , the color image data read from the image sensor  10  via the A/D converter  11  is stored in the first memory  13  and second memory  14  which are frame memories. That is, the data control section  12  transfers the data to the first memory  13  and the second memory  14  in frames alternately by switching connections between the memories  13  and  14 .  
         [0112]     For example, when the size of an image is reduced by half, the data stored in the memories  13 ,  14  are as follows. Since the image sensor  10  is provided with the primary-color Bayer color filter  30 , the pixel data appears repeatedly at intervals of two pixels in both the horizontal direction and the vertical direction. When the image sensor  10  is a CMOS, the pixel data of the same color horizontally adjacent to each other can be averaged and read. This is shown in  FIG. 8 . In  FIG. 8 , the pixel group in the upper part represents the input data and the pixel group in the lower part represents the data read out. The pixel data in the line direction is read only once and the pixel data of the same color adjacent to each other in the line direction are added, thereby producing readout data. Therefore, in the horizontal direction, the data is obtained which is equivalent to the result of extracting, in units of two pixels at intervals of two pixels, the pixel data obtained by calculating the arithmetic average of the pixels of the same color adjacent to each other. In the vertical direction, read-out pixel lines and unread pixel lines are repeated at intervals of two lines. As a result, the data formats stored in the two memories  13 ,  14  are as shown in  FIGS. 9A and 9B , respectively. At this time, under the control of the data control section  12 , the line read from the image sensor  10  is shifted by changing the position according to each item of image data, with the result that the read line position changes so that the consecutive image data may not overlap with one another.  
         [0113]     The data stored in the two memories  13 ,  14  this way are combined at the switching section  17 , thereby producing the data as shown in  FIG. 9C . Then, finding an interpolated value in the positions marked with white circles in  FIG. 9C  enables an interpolated image reduced by half to be created.  
         [0114]     While in the second embodiment, the image size has been reduced by half, the percentage at which the image size is enlarged or reduced is not limited to half. For instance, the same holds true when the image size is reduced by one third. In this case, three memories, or a first memory, a second memory, and a third memory, are provided. The data control section  12  transfers the data obtained by calculating the arithmetic average of three pixels of the same color adjacent to one another in the horizontal direction and the data in units of two lines at intervals of six lines in the vertical direction to the three memories sequentially. In this case, too, the data control section  12  performs control in such a manner that the thinning-out position is shifted image by image and the read pixel positions do not overlap with one another in consecutive image data. At this time, one frame of image data is created from the thinned-out data corresponding to the three consecutive screens stored in the memories.  
         [0115]     The configuration of the index value computing section  15  of the second embodiment is the same as that of  FIG. 3  in the first embodiment. To interpolate an arbitrary point, the data taking-in section  21  takes in the pixel data in a block enclosing the position to be interpolated. Of the pixel data taken in by the data taking-in section  21 , the absolute difference value detecting section  22  extracts a set of pixels adjacent to one another in a specific direction in a plurality of places, and calculates the absolute difference value between the pixel values of the individual pixel sets. The adding section  23  calculates the sum total of the obtained absolute difference values in the specific direction and outputs the index value in each direction.  
         [0116]     When the image size is reduced by half, the data taken in out of the image memories  13 ,  14  by the data taking-in section  21  used in the second embodiment is, for example, as shown in  FIG. 10 . In this case, an interpolated value in the position marked with a white circle in  FIG. 10  is calculated. In  FIG. 10 , the hatched pixels represent the read-out pixel data. In  FIG. 10 , the line data in the first row (R 11 , G 12 , R 13 , G 14 ), the second row (G 21 , B 22 , G 23 , B 24 ), the fifth row (R 51 , G 52 , R 53 , G 54 ), and the sixth row (R 61 , B 62 , G 63 , B 64 ) represent the data transferred from the first memory  13 . In  FIG. 10 , the line data in the third row (R 31 , G 32 , R 33 , G 34 ), the fourth row (G 41 , B 42 , G 43 , B 44 ), the seventh row (R 71 , G 72 , R 73 , G 74 ), and the eighth row (R 81 , B 82 , G 83 , B 84 ) represent the data transferred from the second memory  14 . In the second embodiment, the pixel data in a block of 8×8 pixels including the position to be interpolated by combining the data from the two memories  13  and  14  are taken in.  
         [0117]     In the absolute difference value detection section  22 , when an interpolated value in the position marked with a white circle in  FIG. 10  is estimated, the pixel data set used in detecting the index value for a specific direction from the taken-in pixel data are, for example, as shown in  FIG. 11 . In the second embodiment, the direction is detected for four patterns, that is, a vertical direction, a horizontal direction, a diagonally-right-up direction at 45 degrees, and a diagonally-right-down direction at 45 degrees. The pixel data of the same color lying in each direction is used. As shown in the table of  FIG. 11 , the pixels to be used are switched according to a desired direction.  
         [0118]     The adding section  23  calculates the sum total of the absolute difference values between the pixel values in each direction and finds an index value. In the second embodiment, if the index value for the vertical direction is V ver , the index value for the horizontal direction is V hor , the index value for the diagonally-right-up direction at 45 degrees is V sla , and the index value for the diagonally-right-down direction at 45 degrees is V bac , the index values are determined using the following equations (5):  
             [     Formula   ⁢           ⁢   5     ]                           {             Vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   ver       =       (              R   33     -     R   73            +            G   43     -     G   83              )     ×   2                   Horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   hor       =              R   51     -     R   53            +            G   52     -     G   54            +            G   61     -     G   63            +            B   62     -     B   64                            Diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   sla       =              R   33     -     R   71            +            G   34     -     G   72            +            G   43     -     G   81            +            B   44     -     B   82                            Diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     V   bac       =              R   31     -     R   73            +            G   32     -     G   74            +            G   41     -     G   83            +            B   42     -     B   84                              (   5   )             
 
         [0119]     The direction decision section  16  receives the index value data about the specific direction output from the index value computing section  15  and outputs the direction in which there is the strongest correlation in the position to be interpolated. As in the first embodiment, to determine the correlation, the ratio of each index value to another index value is used. That is, it is determined in which of the horizontal direction, vertical direction, diagonally-right-up direction at 45 degrees, and diagonally-right-up direction at 45 degrees there is a strong correlation or whether there is no specific correlation. Identifying signals caused to correspond to the respective directions are output.  
         [0120]     While in the second embodiment, the index values have been determined from the sum of the absolute difference values between the pixel values, the present invention is not limited to this. For instance, a method of using the ratio of the pixel values at two points may be used. In addition, a group of pixels arranged in a specific direction is not restricted to the group meeting the above equations. Moreover, while the correlation has been determined by the ratio of the index values, a method of finding the minimum value of the index values may be used.  
         [0121]     The data control section  12  connected to the switching section  17  switches connections between the two memories  13  and  14  image by image and stores consecutive image data in the two memories  13 ,  14  alternately. Then, the data control section  12  transmits to the switching section  17  a control signal indicating in which memory the latest input data has been recorded.  
         [0122]     For example, when lines to be read and lines not to be read are switched at intervals of two lines, image data is composed of a frame where a period in which the first two lines are read and the following two lines are not read is repeated (hereinafter, this frame is referred to as a top field) and a frame where a period in which the first two lines are not read and the following two lines are read is repeated (hereinafter, this frame is referred to as a bottom field).  
         [0123]     In this case, when the image data in the top field is the latest input data, the data control section  12  transmits “0” as the control signal to the switching section  17 . When the image data in the bottom field is the latest input data, the data control section  12  transmits “1” as the control signal to the switching section  17 . As long as the control signal makes it possible to determine which memory data is the latest input data, it is not restricted to “0” and “1”. For instance, letters of the alphabet or characters may be used as the control signal.  
         [0124]     According to the identifying signal output from the direction decision section  16  and the control signal output from the data control section  12 , the switching section  17  switches between the image data stored in the first memory  13  and that in the second memory  14  for interpolation.  
         [0125]     In the second embodiment, when G (green) is used in the pixel data used in each case, the pixel data to be used are, for example, as shown in  FIG. 12  from the input identifying signal and control signal. In the table of  FIG. 12 , the directions in which there is a strong correlation are classified into the horizontal direction, the vertical direction, the diagonally-right-up direction at 45 degrees, the diagonally-right-down direction at 45 degrees, and no correlation. In addition, the latest input data is described in the cases of top-field data and bottom-field data.  
         [0126]     Using the pixel data that is input from the switching section  17  and lies in the direction in which there is a strong direction, the interpolation section  18  calculates an interpolated value by multiplying the pixel data by the weighting factor according to the position of each pixel data and the distance from the position in which an interpolated value is to be determined and averaging the result.  
         [0127]     As a result of determining an interpolated value for each of R, G, and B, the output of the interpolation section  18  becomes color image data.  
         [0128]     For example, when there is a strong correlation in the vertical direction, a method of finding an interpolated value for G (green) is shown in  FIGS. 13A and 13B . In this case, an interpolated value for the position marked with a black circle is determined temporarily from the pixel data at only two points arranged in the vertical direction in which there is a strong correlation in the pixel data at three points used for interpolation. Then, on the basis of the interpolated value and the image data at the remaining point, an interpolated value for the position marked with a white circle in  FIG. 10  is determined.  
         [0129]     In the second embodiment, an interpolated value for G in the position marked with a white circle in  FIG. 10  is determined using the following expressions (6) and (7), depending on the input data and the direction in which there is a correlation.  
         [0130]     &lt;&lt;When the Latest Input Data is Top-Field Data&gt;&gt; 
             [     Formula   ⁢           ⁢   6     ]                           {             Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   6     ⁢     G   52       +       5   6     ⁢     (         G   23     +     9   ⁢           ⁢     G   63         10     )                   Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   2     ⁢     (           G   52     +     G   54       2     +     G   63       )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   2     ⁢     (           G   54     +     G   61       2     +         G   54     +     G   63       2       )                 Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢         G   21     +     7   ⁢           ⁢     G   63         8                 No   ⁢           ⁢   correlation   ⁢     :     ⁢           ⁢         13   ⁢           ⁢     G   43       +     30   ⁢           ⁢     G   63       +     8   ⁢           ⁢     G   52       +     13   ⁢           ⁢     G   54         64                     (   6   )             
 
         [0131]     &lt;&lt;When the Latest Input Data is Bottom-Field Data&gt;&gt; 
             [     Formula   ⁢           ⁢   7     ]                           {             Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   vertical   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   6     ⁢     G   72       +       5   6     ⁢     (         7   ⁢     G   43       +     3   ⁢           ⁢     G   83         10     )                   Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   horizontal   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     1   2     ⁢     (           G   52     +     G   54       2     +     G   63       )                   Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   up   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢     3   4     ⁢     (         2   ⁢     G   43       +     G   72       3     )       +       1   4     ⁢     G   74                   Correlation   ⁢           ⁢   in   ⁢           ⁢   the   ⁢           ⁢   diagonally   ⁢     -     ⁢   right   ⁢     -     ⁢   down   ⁢           ⁢   direction   ⁢     :     ⁢           ⁢         3   ⁢     G   32       +     5   ⁢           ⁢     G   74         8                 No   ⁢           ⁢   correlation   ⁢     :     ⁢           ⁢         13   ⁢           ⁢     G   43       +     30   ⁢           ⁢     G   63       +     8   ⁢           ⁢     G   52       +     13   ⁢           ⁢     G   54         64                     (   7   )             
 
         [0132]     Similarly, for R and B, too, the interpolated values are determined by the weighted average corresponding to the distance between the position of the pixel data used for interpolation and the distance from the position in which an interpolated value is to be found.  
         [0133]     If there is not much pixel data of the same color to be referred to in the vicinity of the position to be interpolated, an interpolated value is found using color correlation with another color.  
         [0134]     As described above, in the second embodiment, too, it is possible to detect with high accuracy a direction in which there is a strong correlation and create a high-accuracy interpolated image as in the first embodiment.  
         [0135]     In addition, as in the first embodiment, it is possible to determine the direction in which the subject or the camera moves and switch the pixel data used for interpolation according to the state of the subject or camera.  
         [0136]     While in the second embodiment, weighted average has been used in determining the interpolated value, this invention is not limited to this. Another method, such as arithmetic average, may be used.  
         [0137]     Furthermore, the color filter  30  is not limited to the primary-color type. For instance, a complementary color filter composed of C (cyan), M (magenta), Y (yellow), and G (green) may be used.  
         [0138]     Moreover, the image sensor  10  is not restricted to a CMOS. For instance, a CCD (Charge Coupled Device) may be used. In this case, however, the data obtained by sampling is such that the horizontal direction switches positions with the vertical direction.  
         [0139]     While the present invention has been described using the first and second embodiment, the invention is not limited to these embodiments and may, of course, be practiced or embodied in still other ways without departing from the spirit or essential character thereof.  
         [0140]     For instance, the invention is not restricted to the interpolation of the image data from the image sensor  10  and may be applied to the interpolation of broadcast image data in television or the like.  
         [0141]     Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details, representative devices, and illustrated examples shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.