Patent Publication Number: US-2016234425-A1

Title: Image processing apparatus

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to an image processing apparatus. 
     2. Description of the Related Art 
     There is known an autofocus (AF) technology for obtaining pupil-divided images by an image pickup element having a plurality of photoelectric converters arranged therein for one microlens and detecting a focus based on a phase difference between the plurality of obtained pupil-divided images. 
     For example, in Japanese Patent Application Laid-Open No. 2001-83407, there is disclosed a configuration for treating a plurality of photoelectric converters as one photoelectric converter by adding all the signals from photoelectric converters sharing a single microlens as well as detecting a focus based on a phase difference between pupil-divided images. With this, it is possible to treat the signal in the same way as in the case of one photoelectric converter, and an image for viewing may be created by a signal processing technology. 
     However, in the configuration of the above-mentioned related art, processing of adding obtained signals for generation is necessary to generate an image signal. For example, when four photoelectric converters share one microlens, the number of signals to be read out from an image pickup element is four times the number of microlenses in order to obtain a desired image signal. In other words, the number of signals to be read out is proportional to the number of photoelectric converters in one microlens, and hence a processing load may be high in a system such as a digital camera or a digital video camera where the image signal is read out every predetermined time period. 
     As a solution to the above-mentioned problem, it is conceivable to reduce the number of signals to be read out by adding signals from the photoelectric converters within the image pickup element for a direction other than the pupil division direction. However, also in this case, compared to a read-out scheme for an image pickup element array where pupils are not divided, the number of signals to be read out is twice in a phase difference focus detection for detecting a focus by calculating a phase difference between two pupil-divided images, and hence the processing load may similarly be high. 
     SUMMARY OF THE INVENTION 
     According to one embodiment of the present invention, there is provided an image processing apparatus, including: an image pickup element including pixel portions, the pixel portions each including a plurality of photoelectric converters and each being configured to output a first added signal obtained by adding signals output from a first group among the plurality of photoelectric converters and output a second added signal obtained by adding signals output from a second group, which is a part of the first group among the plurality of photoelectric converters; an added signal separation unit configured to generate a third added signal by subtracting the second added signal from the first added signal, and output the second added signal and the third added signal; a phase difference measurement unit configured to perform a phase difference measurement based on the second added signal and the third added signal; and an image pickup element drive unit configured to change a combination of photoelectric converters included in the second group so as to change a pupil division direction for the phase difference measurement performed in the phase difference measurement unit. 
     Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram for illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. 
         FIG. 2  is a schematic diagram of a pixel portion of an image pickup element according to the first embodiment. 
         FIG. 3  is a block diagram for illustrating a configuration of the image pickup element according to the first embodiment. 
         FIG. 4  is a diagram for illustrating a circuit configuration of the pixel portion of the image pickup element according to the first embodiment. 
         FIG. 5  is a drive timing chart of the image pickup element according to the first embodiment. 
         FIG. 6A  and  FIG. 6B  are drive timing charts of the image pickup element in a horizontal blanking period according to the first embodiment. 
         FIG. 7  is an arrangement diagram of a focus detection area of the image pickup element according to the first embodiment. 
         FIG. 8A  and  FIG. 8B  are block diagrams for illustrating a method of generating an output signal of an added signal separation unit according to the first embodiment. 
         FIG. 9  is a cross-sectional view of the pixel portion according to the first embodiment. 
         FIG. 10  is a graph for illustrating phase difference measurement processing according to the first embodiment. 
         FIG. 11  is a flowchart for illustrating an added signal separation program according to a second embodiment of the present invention. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings. Like components are denoted by like reference symbols throughout the drawings, and descriptions of overlapping components are sometimes simplified or omitted. 
     First Embodiment 
       FIG. 1  is a block diagram for illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. The image processing apparatus includes an optical system  101 , an optical system drive unit  102 , an image pickup element  103 , an image pickup element drive unit  104 , an added signal separation unit  105 , a camera signal processing unit  106 , a phase difference measurement unit  107 , an AF control unit  108 , and a system control unit  109 . 
     The optical system  101  is a part configured to guide incident light to the image pickup element  103 , and includes at least one of a zoom lens, a diaphragm, and a focus lens. The optical system drive unit  102  is configured to control the optical system  101  based on focus information output from the AF control unit  108  and optical system drive information output from the system control unit  109 . 
     The image pickup element  103  is configured to convert an object image entering the image pickup element  103  through the optical system  101  to an electrical signal by photoelectric conversion, and output two signals, namely, an image signal and a pupil-divided image signal, to the added signal separation unit  105 . The image pickup element drive unit  104  is a drive apparatus configured to control the image pickup element  103  based on image pickup element drive instruction information from the system control unit  109 . Note that, the image pickup element  103  may have an electronic shutter function. In that case, the image pickup element  103  may execute the electronic shutter function to achieve a desired exposure time in accordance with a control signal output from the image pickup element drive unit  104 . 
     The added signal separation unit  105  is configured to subtract one pupil-divided image signal from the image signal output from the image pickup element  103  to generate the other pupil-divided image signal. The added signal separation unit  105  is configured to output the image signal to the camera signal processing unit  106  and output the generated two pupil-divided images to the phase difference measurement unit  107 . 
     The camera signal processing unit  106  is configured to perform image processing on the image signal input from the added signal separation unit  105  and generate a video signal for display/record. The generated video signal is output to a display apparatus, a recording medium, and the like outside the image processing apparatus. 
     The phase difference measurement unit  107  is configured to calculate a phase difference estimated value for performing the phase difference measurement based on the two pupil-divided images obtained from the added signal separation unit  105 , and output the phase difference estimated value to the AF control unit  108 . 
     The AF control unit  108  is configured to calculate focus information for controlling a focus position of the optical system  101  based on the phase difference estimated value input from the phase difference measurement unit  107 , and output the focus information to the optical system drive unit  102 . 
     The system control unit  109  is a control apparatus configured to control the entire image processing apparatus. The system control unit  109  is configured to generate drive information for each unit of the image processing apparatus based on photographing information obtained through a user instruction, a photographing scene detection, an object detection, and the like. The system control unit  109  is configured to transmit drive information for the optical system  101 , such as a zoom lens or diaphragm, to the optical system drive unit  102 . Further, the system control unit  109  is configured to transmit drive information for the image pickup element  103 , such as an instruction to switch the pupil division direction and an exposure time, to the image pickup element drive unit  104 . 
     Next, a configuration of the image pickup element  103  and how the image pickup element  103  is driven according to this embodiment are described. 
       FIG. 2  is a schematic diagram for illustrating a structure of a pixel portion  206  in the image pickup element  103  according to this embodiment. The image pickup element  103  includes a plurality of the pixel portions  206  arranged in a matrix having rows and columns. The pixel portion  206  includes a plurality of photoelectric converters such as photodiodes (PD) configured to generate an electric charge corresponding to the incident light. In this embodiment, the pixel portion  206  having four photoelectric converters is exemplified. 
     The pixel portion  206  includes a photoelectric converter  201  (first photoelectric converter), a photoelectric converter  202  (second photoelectric converter), a photoelectric converter  203  (third photoelectric converter), and a photoelectric converter  204  (fourth photoelectric converter). The pixel portion  206  further includes a microlens  205 , which is shared by the photoelectric converters  201 ,  202 ,  203 , and  204 . That is, light guided by the same microlens  205  enters the photoelectric converters  201 ,  202 ,  203 , and  204 . Other pixel portions arranged in the image pickup element  103  include similar photoelectric converters. 
     Signals obtained from the photoelectric converter  201 , the photoelectric converter  202 , the photoelectric converter  203 , and the photoelectric converter  204  are referred to as A image, B image, C image, and D image, respectively. Adding signals output from the photoelectric converters  201 ,  202 ,  203 , and  204  (first group) produces an image signal that is not pupil-divided. This image signal is referred to as A+B+C+D image (first added signal). 
     Further, adding signals obtained from the photoelectric converters  201  and  202  (second group), which are a part of the photoelectric converters  201 ,  202 ,  203 , and  204  (first group), produces a signal (upper signal) from the photoelectric converters arranged on an upper side of the pixel portion  206 . This is referred to as A+B image (second added signal). Adding signals obtained from the photoelectric converters  203  and  204  (third group) produces a signal (lower signal) from the photoelectric converters arranged on a lower side of the pixel portion  206 . This is referred to as C+D image (third added signal). In this way, signals having vertically (first pupil division direction) divided pupils are obtained. 
     On the other hand, adding signals obtained from the photoelectric converters  201  and  203  (second group), which are a part of the photoelectric converters  201 ,  202 ,  203 , and  204  (first group) chosen differently from the above-mentioned group, produces a signal (left signal) from the photoelectric converters arranged on a left side of the pixel portion  206 . This is referred to as A+C image (second added signal). Adding signals obtained from the photoelectric converters  202  and  204  (third group) produces a signal (right signal) from the photoelectric converters arranged on a right side of the pixel portion  206 . This is referred to as B+D image (third added signal). In this way, signals having horizontally (second pupil division direction) divided pupils are obtained. 
     As described above, it is possible to obtain signals having horizontally divided pupils or signals having vertically divided pupils by changing the addition target signals from respective photoelectric converters, which means that it is possible to change the pupil division direction. Further, it is also possible to obtain the image signal that is not pupil-divided. 
       FIG. 3  is a block diagram for illustrating a configuration of the image pickup element  103  according to this embodiment. In  FIG. 3 , for the simplicity of description, only the structure having four pixel portions is illustrated. However, the number of rows and columns may be different from that illustrated, and an arbitrary number of rows and columns is conceivable. 
     The image pickup element  103  includes, as units for inputs from the image pickup element drive unit  104 , a pupil division direction instruction input terminal  104 - 1 , a horizontal synchronization signal input terminal  104 - 2 , a vertical synchronization signal input terminal  104 - 3 , and an instruction information update signal input terminal  104 - 4 . Further, the image pickup element  103  includes, as units for outputs to the added signal separation unit  105 , a first output terminal  103 - 1  and a second output terminal  103 - 2 . 
     Pupil division direction instruction information is input to the pupil division direction instruction input terminal  104 - 1  from the image pickup element drive unit  104 . A horizontal synchronization signal HD is input to the horizontal synchronization signal input terminal  104 - 2  from the image pickup element drive unit  104 . A vertical synchronization signal VD is input to the vertical synchronization signal input terminal  104 - 3  from the image pickup element drive unit  104 . An instruction information update signal is input to the instruction information update signal input terminal  104 - 4  from the image pickup element drive unit  104 . 
     The image pickup element  103  includes a timing signal generation circuit  301 , a first buffer  302 , a second buffer  303 , a transfer signal correction circuit  305 , and a horizontal read-out circuit  311  as well as the above-mentioned pixel portion  206 . Further, the image pickup element  103  includes a transfer signal common bus  304 , a transfer signal line  306 , a row read-out control signal line  307 , a reset signal line  308 , a column read-out signal line  309 , and a horizontal drive control signal line  310 . The first buffer  302 , the second buffer  303 , and the transfer signal correction circuit  305  are provided for each pixel column of the image pickup element  103 . 
     The first buffer  302  is configured to hold the input pupil division direction instruction information, and output this information to the second buffer  303 . The second buffer  303  is configured to update held information to the input pupil division direction instruction information at the timing that is based on the instruction information update signal, and output this updated information to the transfer signal correction circuit  305 . 
     The horizontal synchronization signal HD and the vertical synchronization signal VD are input to the timing signal generation circuit  301 . The timing signal generation circuit  301  is configured to output a control signal to the transfer signal common bus  304 , the row read-out control signal line  307 , the reset signal line  308 , and the horizontal drive control signal line  310  at the timing that is based on the horizontal synchronization signal HD and the vertical synchronization signal VD. 
     The transfer signal correction circuit  305  is configured to correct a logical value of the transfer signal input from the transfer signal common bus  304  based on the pupil division direction instruction information, and output the corrected logical value to the transfer signal line  306  of each array. The transfer signal line  306  is formed of four transfer signal lines  306 - 1 ,  306 - 2 ,  306 - 3 , and  306 - 4  corresponding to the photoelectric converters  201 ,  202 ,  203 , and  204 , respectively. Those transfer signal lines  306 - 1 ,  306 - 2 ,  306 - 3 , and  306 - 4  are connected to the pixel portion  206 . 
     The timing signal generation circuit  301  is configured to supply a row read-out control signal SEL to each pixel portion  206  via the row read-out control signal line  307 . Further, the timing signal generation circuit  301  is configured to supply a reset signal RES to each pixel portion  206  via the reset signal line  308 . 
     The pixel portion  206  is configured to output a signal from the photoelectric converters  201 ,  202 ,  203 , and  204  to the column read-out signal line  309  based on those control signals. The signal output from the column read-out signal line  309  of each column is input to the horizontal read-out circuit  311 . The horizontal read-out circuit  311  is configured to sequentially output, for each column, the signal from the column read-out signal line  309  to the first output terminal  103 - 1  and the second output terminal  103 - 2  based on a control signal input from the timing signal generation circuit  301  via the horizontal drive control signal line  310 . 
       FIG. 4  is a diagram for illustrating a circuit configuration of the pixel portion of the image pickup element  103  according to the first embodiment of the present invention. The image pickup element  103  includes transfer transistors  401 ,  402 ,  403 , and  404  respectively connected to the photoelectric converters  201 ,  202 ,  203 , and  204 . Electric charges generated in respective photoelectric converters  201 ,  202 ,  203 , and  204  are transferred to a floating diffusion  407  via respective transfer transistors  401 ,  402 ,  403 , and  404 . The transfer transistors  401 ,  402 ,  403 , and  404  are controlled to conduct electricity or not to conduct electricity based on transfer signals TX 1 , TX 2 , TX 3 , and TX 4  input from the transfer signal lines  306 - 1 ,  306 - 2 ,  306 - 3 , and  306 - 4 , respectively. 
     The image pickup element  103  further includes a reset transistor  406 , a row read-out transistor  408 , and a source follower transistor  409 . The reset transistor  406  is connected between a power line  405  and the floating diffusion  407 . The row read-out transistor  408  is connected between the floating diffusion  407  and the gate node of the source follower transistor  409 . The drain of the source follower transistor  409  is connected to the power line  405 . The source of the source follower transistor  409  is connected to the column read-out signal line  309 . The reset transistor  406  is controlled to conduct electricity or not to conduct electricity based on the reset signal RES input from the reset signal line  308 . The row read-out transistor  408  is controlled to conduct electricity or not to conduct electricity based on the row read-out control signal SEL input from the row read-out control signal line  307 . 
     Now, an operation of the image pickup element drive unit  104  instructing the image pickup element  103  to divide the pupil and an operation of reading out a signal are described with reference to  FIG. 3 ,  FIG. 4 , and  FIG. 5 . 
       FIG. 5  is a drive timing chart of the image pickup element  103 . In  FIG. 5 , operation timings of the horizontal synchronization signal HD, the vertical synchronization signal VD, and the instruction information update signal are illustrated. One vertical synchronization period includes a plurality of horizontal synchronization periods. Each horizontal synchronization period is formed of a horizontal blanking period (H-BLK) and a horizontal drive period (H drive). The horizontal blanking period and the horizontal drive period of a horizontal synchronization period are represented by a period t 501  and a period t 502 , respectively, and the horizontal blanking period and the horizontal drive period of the next horizontal synchronization period are represented by a period t 503  and a period t 504 , respectively. 
     In the period t 502 , pupil division direction instruction information corresponding to a pixel row is input to the pupil division direction instruction input terminal  104 - 1 . The pupil division direction instruction information is held in the first buffers  302  corresponding to respective pixel arrays. The pupil division direction instruction information held in the first buffers  302  is output to the second buffers  303 . Note that, the pupil division direction instruction information is a digital signal having a logical value of 0 or 1. A signal having a value of 0 is a signal for instructing a pupil division in a vertical direction, and a signal having a value of 1 is a signal for instructing a pupil division in a horizontal direction. 
     Early in the period t 503 , values held in the second buffers  303  are updated to the values output from the first buffers  302  in response to an instruction information update signal input to the second buffers  303  from the instruction information update signal input terminal  104 - 4 . As a result, the pupil division direction instruction information is input to the transfer signal correction circuits  305  from the second buffers  303 . 
     A transfer signal from the transfer signal common bus  304  is input to the transfer signal correction circuits  305 . The transfer signal correction circuit  305  corrects the transfer signal based on the pupil division direction instruction information such that the corrected transfer signal represents an operation corresponding to an instruction of the pupil division direction. Note that, the pupil division direction instruction information may allow the transfer signal from the transfer signal common bus  304  to pass as it is, and this operation is also included in “correction”. 
       FIG. 6A  and  FIG. 6B  are drive timing charts of the pixel portion  206  in the horizontal blanking period t 503 .  FIG. 6A  is a drive timing chart in the case where the value of the pupil division direction instruction information is 0 while  FIG. 6B  is a drive timing chart in the case where the value of the pupil division direction instruction information is 1. In  FIG. 6A  and  FIG. 6B , the transfer signals TX 1 , TX 2 , TX 3 , and TX 4 , the row read-out control signal SEL, the reset signal RES, a signal accumulated in the floating diffusion (FD)  407 , and a signal output from the column read-out signal line (OUT)  309  are illustrated. 
     A drive timing in the case where the value of the pupil division direction instruction information is 0 is described with reference to  FIG. 6A . In a period t 503 - 1 , the transfer signal TX 1  is set High. Because of this, the transfer transistor  401  conducts electricity, and the electric charge that corresponds to A image and is accumulated in the photoelectric converter  201  is transferred to the floating diffusion  407 . 
     In a period t 503 - 2 , the transfer signal TX 2  is set High. Because of this, the transfer transistor  402  conducts electricity, and the electric charge that corresponds to B image and is accumulated in the photoelectric converter  202  is transferred to the floating diffusion  407 . Those electric charges are added in the floating diffusion  407  and the accumulated electric charge corresponds to A+B image. 
     In a period t 503 - 3 , the row read-out control signal SEL is set High. Because of this, the row read-out transistor  408  conducts electricity, and the voltage of the floating diffusion  407  is input to the gate node of the source follower transistor  409 . A voltage signal corresponding to A+B image is output to the column read-out signal line  309 . 
     In a period t 503 - 5 , the transfer signal TX 3  is set High. Because of this, the transfer transistor  403  conducts electricity, and the electric charge that corresponds to C image and is accumulated in the photoelectric converter  203  is transferred to the floating diffusion  407 . Those electric charges are added in the floating diffusion  407  and the accumulated electric charge corresponds to A+B+C image. 
     In a period t 503 - 6 , the transfer signal TX 4  is set High. Because of this, the transfer transistor  404  conducts electricity, and the electric charge that corresponds to D image and is accumulated in the photoelectric converter  204  is transferred to the floating diffusion  407 . Those electric charges are added in the floating diffusion  407  and the accumulated electric charge corresponds to A+B+C+D image. 
     In a period t 503 - 7 , the row read-out control signal SEL is set High in the same way as in the period t 503 - 3 , to thereby cause the voltage signal corresponding to A+B+C+D image to be output to the column read-out signal line  309 . 
     In a period t 503 - 8 , the reset signal RES and all the transfer signals TX 1 , TX 2 , TX 3 , and TX 4  are set High. Because of this, all the electric charges accumulated in the photoelectric converters  201 ,  202 ,  203 , and  204  and the floating diffusion  407  are all reset to initial conditions. After that, photoelectric charges are accumulated in the photoelectric converters  201 ,  202 ,  203 , and  204  in preparation for the next read-out drive. 
     In this way, when the value of the pupil division direction instruction information is 0, two signals, namely, the image signal that is not pupil-divided (A+B+C+D image) and the upper signal (A+B image), are read out from the image pickup element  103  and input to the horizontal read-out circuit  311 . 
     In a horizontal drive period t 504 , the horizontal read-out circuit  311  outputs the upper signal (A+B image) from the first output terminal  103 - 1  and outputs the image signal that is not pupil-divided (A+B+C+D image) from the second output terminal  103 - 2 . 
     Next, a drive timing in the case where the value of the pupil division direction instruction information is 1 is described with reference to  FIG. 6B . The differences from  FIG. 6A  are operations in the period t 503 - 2  and in the period t 503 - 5 . A description of an operation that is the same with that of  FIG. 6A  is omitted. 
     In the period t 503 - 2 , the transfer signal TX 3  is set High. Because of this, the transfer transistor  403  conducts electricity, and the electric charge that corresponds to C image and is accumulated in the photoelectric converter  203  is transferred to the floating diffusion  407 . Those electric charges are added in the floating diffusion  407  and the accumulated electric charge corresponds to A+C image. Thus, in the period t 503 - 3 , the voltage signal to be output to the column read-out signal line  309  corresponds to A+C image. 
     In the period t 503 - 5 , the transfer signal TX 2  is set High. Because of this, the transfer transistor  402  conducts electricity, and the electric charge that corresponds to B image and is accumulated in the photoelectric converter  202  is transferred to the floating diffusion  407 . Those electric charges are added in the floating diffusion  407  and the accumulated electric charge corresponds to A+B+C image. Thus, in the period t 503 - 7 , the voltage signal to be output to the column read-out signal line  309  corresponds to A+B+C+D image as in the case of the drive timing chart of  FIG. 6A . 
     In this way, when the value of the pupil division direction instruction information is 1, two signals, namely, the image signal that is not pupil-divided (A+B+C+D image) and the left signal (A+C image), are read out from the image pickup element  103  and input to the horizontal read-out circuit  311 . In the horizontal drive period t 504 , the horizontal read-out circuit  311  outputs the left signal (A+C image) from the first output terminal  103 - 1  and outputs the image signal that is not pupil-divided (A+B+C+D image) from the second output terminal  103 - 2 . In this way, a combination of photoelectric converters provided to generate an added signal in order to generate the pupil-divided image is changed based on the value of the pupil division direction instruction information output from the image pickup element drive unit  104 . 
     Note that, while only one pixel row is focused on in the above description of the drive timing, pupil division direction instruction information of the next row may be input in parallel in the same period. Further, the above-mentioned drive is sequentially performed for pixel rows forming the image pickup element  103  in one vertical synchronization period, and the accumulation and reading out of photoelectric charges are completed. 
     The image signals obtained from the image pickup element  103  are input to the added signal separation unit  105 . The added signal separation unit  105  is configured to generate an image signal to be output to the camera signal processing unit  106  and a phase difference detection signal to be output to the phase difference measurement unit  107 . The phase difference detection signal is a signal for detecting a phase difference that is obtained by subtracting the left signal (A+C image) or the upper image (A+B image), which is output from the first output terminal  103 - 1 , from the image signal that is not pupil-divided (A+B+C+D image), which is output from the second output terminal  103 - 2 . 
     The phase difference measurement unit  107  is configured to choose whether to perform a focus detection with use of the vertically pupil-divided image or perform a focus detection with use of the horizontally pupil-divided image based on the pupil division direction instruction information, and execute the phase difference measurement with use of the input phase difference detection signal. 
     The image signal that is not pupil-divided (A+B+C+D image) output from the added signal separation unit  105  is input to the camera signal processing unit  106 . The camera signal processing unit  106  is configured to perform, on A+B+C+D image, image processing such as a color conversion, a white balance, and a gamma correction, resolution conversion processing, image compression processing, and the like to generate a video signal for display/record. 
     Note that, the pupil division direction instruction information may be set such that 0 and 1 switch in turn in units of one vertical synchronization period. With this, the horizontally (left and right) pupil-divided image and the vertically (upper and lower) pupil-divided image can be obtained alternately for each frame, thereby enabling an accurate phase difference measurement. 
       FIG. 7  is an arrangement diagram of a focus detection area of the image pickup element  103  according to the first embodiment. It is not necessary that the pixel portion  206  capable of obtaining the above-mentioned pupil-divided image for focus detection be arranged in all the areas of the image pickup element  103 . A range indicated by an arrow  701  of  FIG. 7  is an area for arranging effective pixels. In other words, the image signal that is not pupil-divided (A+B+C+D image) can be read out from this area of the pixel portion  206 . 
     A range indicated by an arrow  702  of  FIG. 7  is an area for arranging the pixel portions  206  capable of selectively reading out any one of the left signal (A+C image) and the upper signal (A+B image) as well as the image signal that is not pupil-divided (A+B+C+D image). In this way, it is possible to shorten the read-out time by obtaining the signal for focus detection only from a part of the image pickup element  103 . 
       FIG. 8A  and  FIG. 8B  are block diagrams for illustrating a method of generating an output signal of the added signal separation unit  105  according to the first embodiment.  FIG. 8A  is an illustration of processing in the case where the value of the pupil division direction instruction information is 0 while  FIG. 8B  is an illustration of processing in the case where the value of the pupil division direction instruction information is 1. 
     In  FIG. 8A , the signals input to the added signal separation unit  105  are A+B+C+D image and A+B image as described above. The added signal separation unit  105  subtracts A+B image from A+B+C+D image to generate C+D image. Further, in addition to this, the added signal separation unit  105  outputs A+B+C+D image and A+B image as it is. Thus, the added signal separation unit  105  outputs three images, namely, A+B+C+D image, A+B image, and C+D image. In this way, the vertically pupil-divided images and the image signal that is not pupil-divided are output. 
     In  FIG. 8B , the signals input to the added signal separation unit  105  are A+B+C+D image and A+C image as described above. The added signal separation unit  105  subtracts A+C image from A+B+C+D image to generate B+D image. Further, in addition to this, the added signal separation unit  105  outputs A+B+C+D image and A+C image as it is. Thus, the added signal separation unit  105  outputs three images, namely, A+B+C+D image, A+C image, and B+D image. In this way, the horizontally pupil-divided images and the image signal that is not pupil-divided are output. 
     Next, a method of calculating a phase difference estimated value with use of the pupil-divided image, which is performed in the phase difference measurement unit  107 , is described with reference to  FIG. 9  and  FIG. 10 . 
       FIG. 9  is a diagram for schematically illustrating a cross-section of the pixel portion  206  illustrated in  FIG. 2  taken along the line S-S′. In  FIG. 9 , a positional relationship between the pixel portion  206  and an exit pupil is illustrated. The photoelectric converters  201  and  202  are arranged on the left side of the pixel portion  206  while the photoelectric converters  203  and  204  are arranged on the right side of the pixel portion  206 . The microlens  205  shared by those photoelectric converters is arranged above the photoelectric converters  201 ,  202 ,  203 , and  204 . The top portions of the microlenses  205  form an image formation plane  906  of an image pickup lens (not illustrated) at the time of in-focus. 
     Further, in  FIG. 9 , exit pupils  901 ,  904 , and  905  are illustrated. The exit pupil  901  is an exit pupil of an image pickup lens when seen from the pixel portion  206  side. The exit pupil  904  is an exit pupil of the photoelectric converters  203  and  204  that is projected onto an exit pupil position by the microlens  205 . The exit pupil  905  is an exit pupil of the photoelectric converters  201  and  202  that is similarly projected onto an exit pupil position. The distance from the image formation plane  906  of the image pickup lens at the time of in-focus to the exit pupil  901  is referred to as an exit pupil position. The exit pupil position changes depending on the curvature of a lens group located behind (image formation plane side) a lens diaphragm (not illustrated) and a positional relationship with respect to the diaphragm. Further, the size of the exit pupil changes depending on the radius of the diaphragm. A light flux  903  passing through the exit pupil  905  is designed to enter the photoelectric converters  201  and  202 , and a light flux  902  passing through the exit pupil  904  is designed to enter the photoelectric converters  203  and  204 . 
     In this way, an image seen in an area of the exit pupil  905  on the right side of the exit pupil  901  of the image pickup lens is obtained in the photoelectric converters  201  and  202  located on the left side of the pixel portion  206 . Similarly, an image seen in an area of the exit pupil  904  on the left side of the exit pupil  901  of the image pickup lens is obtained in the photoelectric converters  203  and  204 . Note that, other pixel portions  206  forming the image pickup element  103  (not illustrated) have similar optical designs. 
     Assuming that the image obtained on the image pickup element  103  through the light flux  902  is C+D image and the image obtained on the image pickup element  103  through the light flux  903  is A+B image, the difference between A+B image and C+D image corresponds to a parallax between the light flux  902  and the light flux  903 . 
       FIG. 10  is a graph for illustrating the phase difference measurement. In  FIG. 10 , an image signal  1001  (A+B image) and an image signal  1002  (C+D image) are illustrated under a front-focus state, that is, a state in which the focus is in a nearer side than the object. The vertical axis represents a signal strength and the horizontal axis represents a position of a pixel in the S-S′ line direction. The information on a distance to the object based on the phase difference detection method is calculated based on a distance  1003  between images obtained by the image signal  1001  and the image signal  1002  and on a distance from the image forming plane at the focus position to the exit pupil. The phase difference measurement unit  107  outputs, as a phase difference estimated value, the calculated information on the distance to the object to the AF control unit  108 . 
     The AF control unit  108  determines a target focus position based on the phase difference estimated value output from the phase difference measurement unit  107 , and outputs a movement direction and a movement amount from the current focus position to the optical system drive unit  102  as focus information. The optical system drive unit  102  drives the optical system  101  based on the focus information and adjusts the focus position. 
     Note that, in the above description of the phase difference measurement, the phase difference measurement is exemplified by using image signals (upper signal and lower signal) that are vertically pupil-divided and read out. However, the phase difference measurement can similarly be achieved by using image signals (left signal and right signal) that are horizontally pupil-divided and read out. 
     In this embodiment, there is provided an image pickup apparatus capable of measuring a phase difference, which is configured to be capable of suppressing the increase in number of signals to be read out even when there are a large number of photoelectric converters, and allowing the pupil division direction to be switched. 
     Second Embodiment 
     An image processing method according to a second embodiment of the present invention is described with reference to the flowchart of  FIG. 11 . 
     In the second embodiment, a configuration for carrying out generation of the pupil-divided image by a computer is described. In other words, this embodiment is directed to realizing generation of the pupil-divided image in the added signal separation unit  105  in the first embodiment with use of a computer. The same configuration as that of the first embodiment may be employed for other components forming the image processing apparatus. 
     The computer to be used for image processing in this embodiment includes a CPU for performing calculations, a memory for storing an output from the image pickup element  103  and for storing a program, and the like. The computer realizes the generation of the pupil-divided image by executing a program stored in the memory. In this embodiment, an output from the image pickup element  103  (image signal that is not pupil-divided and pupil-divided signal) described in the first embodiment is temporarily stored in the memory, and this output data is used to perform processing. 
       FIG. 11  is a flowchart for illustrating a program for generating C+D image based on A+B+C+D image and A+B image, which are stored in the memory of the computer. 
     In Step S 1101 , the generation of the pupil-divided image is started. In Step S 1102 , the computer sets a pointer to secure a memory area for storing A+B+C+D image, A+B image, and C+D image, and initializes this memory area. 
     In Step S 1103 , the computer reads A+B image and stores this image in the memory area of A+B image. In Step S 1104 , the computer reads A+B+C+D image and stores this image in the memory area of A+B+C+D image. 
     In Step S 1105 , the computer subtracts A+B image from A+B+C+D image to generate C+D image, and stores this image in the memory area of C+D image. 
     In Step S 1106 , the computer determines whether or not all the pixels have been processed. When not all the pixels have been processed, the processing proceeds to Step S 1107 . In Step S 1107 , the computer increases the pointer and the processing returns to Step S 1103 . Thus, the same processing is repeatedly executed for the next pixel data. 
     When the computer determines that all the pixels have been processed in Step S 1106 , the processing proceeds to Step S 1108  and ends. 
     In accordance with the above-mentioned flow, C+D image is generated with use of A+B+C+D image and A+B image in the same way as in the first embodiment. Note that, in this embodiment, the same configuration as that of the first embodiment may be employed for configurations other than the generation of the pupil-divided image. 
     As described above, it is possible to realize the generation of the pupil-divided image with use of the computer easily as well as to obtain the same effect as that of the first embodiment also in this embodiment. 
     Note that, in the flowchart of  FIG. 11 , the upper signal (A+B image) is stored in the memory, and this signal is used to generate the lower signal (C+D image). However, a configuration may be employed in which the left signal (A+C image) is stored in the memory and used to generate the right signal (B+D image). The same processing may be employed also in this case. 
     Other Embodiments 
     Four photoelectric converters  201 ,  202 ,  203 , and  204  share one microlens  205  in the first and second embodiments. However, the number of photoelectric converters may be five or more. 
     The value of the pupil division direction instruction information may be switched to 0 or 1 in units of pixel row or pixel column. In this case, the horizontally (left and right) pupil-divided image and the vertically (upper and lower) pupil-divided image can be obtained alternatively for each row or for each column, and it is possible to achieve an accurate phase difference measurement using both the phase difference in the horizontal direction and the phase difference in the vertical direction. 
     The pupil division direction instruction information may be configured to take the value of 1 as an initial value to indicate that the pupil division is performed horizontally (left and right) in normal conditions, and change the value to 0 only when a predetermined condition is satisfied. For example, the pupil division direction instruction information may be configured to change the value to 0 to indicate that the pupil division is performed vertically (upper and lower) only when, for example, the accuracy of the phase difference measurement is equal to or less than a predetermined value in the horizontal (left and right) pupil division. With this, it is possible to achieve the phase difference measurement appropriate to the photographing situation. Note that, the pupil division direction instruction information may be configured to take the value of 0 as an initial value to indicate that the pupil division is performed vertically (upper and lower) in normal conditions, and change the value to 1 to indicate that the pupil division is performed horizontally (left and right) only when the accuracy of the phase difference measurement is equal to or less than a predetermined value. 
     Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like. 
     While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions. 
     This application claims the benefit of Japanese Patent Application No. 2015-021317, filed Feb. 5, 2015, which is hereby incorporated by reference herein in its entirety.