IMAGING CONTROL DEVICE, IMAGING APPARATUS, IMAGING CONTROL METHOD, AND IMAGING CONTROL PROGRAM

An imaging control device that controls an imaging element including a plurality of pixel rows in each of which a plurality of pixels each including a photoelectric conversion unit and a charge holding unit which holds charges transferred from the photoelectric conversion unit are arranged in one direction includes a processor, and the processor is configured to: perform a first control as defined herein; perform a second control as defined herein; perform a third control as defined herein; and perform a fourth control as defined herein.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. § 119 to Japanese Patent Application No. 2022-104770 filed on Jun. 29, 2022. The above application is hereby expressly incorporated by reference, in its entirety, into the present application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging control device, an imaging apparatus, an imaging control method, and a computer readable medium storing an imaging control program.

2. Description of the Related Art

WO2020/021887 A discloses an image processing apparatus including a readout unit that reads out image data which is captured by an imaging element comprising a plurality of photoelectric conversion elements and transmitted to a memory unit and on which light noise is superimposed, as region image data for each of a plurality of regions divided in the memory unit, and that reads out data of a predetermined region again after the readout for each region image data ends, and an output unit that outputs correction image data obtained by correcting captured image data which is captured by the imaging element and stored in the memory unit, for each of the plurality of regions in accordance with the light noise determined in accordance with the data read out again by the readout unit.

WO2018/088119A discloses an imaging apparatus that exposes a light-receiving surface of an imaging element in a case where an imaging instruction is provided, and that reads out an imaging signal from the imaging element in five separate field periods after the end of the exposure, in which the imaging signal is read out from a field including a phase difference detection pixel in the initial field period, the imaging signal is read out from a field including only an imaging pixel in the subsequent field period, a postview image is displayed based on the imaging signal read out in the initial field period, and the postview image is updated based on the imaging signal read out in the subsequent field period.

SUMMARY OF THE INVENTION

The disclosed technology is as follows.(1) An imaging control device controls an imaging element (imaging element5) including a plurality of pixel rows (pixel rows62) in each of which a plurality of pixels (pixels61) each including a photoelectric conversion unit (photoelectric conversion unit61A) and a charge holding unit (charge holding unit61B) which holds charges transferred from the photoelectric conversion unit are arranged in one direction (row direction X), the imaging control device comprising a processor (system controller11), in which the processor is configured to perform a first control of exposing the plurality of pixel rows (a group G1, a group G2, a group G3, and a group G4) and of transferring charges accumulated in the photoelectric conversion units of the plurality of pixel rows by the exposing to the charge holding units, perform a second control of reading out a signal corresponding to the charges held in the charge holding units of a part (group G1) of the pixel rows among the charge holding units in which the charges are held by the first control, perform a third control of reading out a signal corresponding to the charges held in the charge holding units of the pixel rows (any of the group G2, the group G3, and the group G4) other than the part of the pixel rows among the charge holding units in which the charges are held by the first control, and of exposing the part of the pixel rows from which the signal is read out by the second control, and perform a fourth control of reading out a signal corresponding to the charges held in the charge holding units of the part of the pixel rows from which the signal is read out by the second control.(2) The imaging control device according to (1), in which the processor is configured to perform a display control of a live view image based on a result of one or both of the second control and the fourth control.(3) The imaging control device according to (1) or (2), in which the processor is configured to, in the third control, expose the part of the pixel rows by shifting exposure periods of all or a part of the part of the pixel rows.(4) The imaging control device according to any one of (1) to (3), in which the imaging element includes a phase difference detection pixel row (second pixel row) that is the pixel row including a phase difference detection pixel, the part of the pixel rows includes the phase difference detection pixel row, and the processor is configured to derive an evaluation value for focal point adjustment based on a signal that is read out from the charge holding units of the phase difference detection pixel row by the second control.(5) The imaging control device according to (4), in which the processor is configured to determine whether to execute or not execute the third control based on the evaluation value.(6) The imaging control device according to (5), in which the processor is configured to not execute the third control in a case where the evaluation value is greater than or equal to a threshold value.(7) The imaging control device according to (5) or (6), in which the processor is configured to, in a case of not executing the third control, perform, instead of the third control and of the fourth control, a fifth control of reading out a signal corresponding to the charges held in the charge holding units of the pixel rows other than the part of the pixel rows among the charge holding units in which the charges are held by the first control.(8) The imaging control device according to (5) or (6), in which the processor is configured to, in a case of not executing the third control, perform the first control instead of the third control and of the fourth control.(9) The imaging control device according to (4), in which the processor is configured to control a readout speed of the signal in the fourth control based on the evaluation value.(10) The imaging control device according to (9), in which the processor is configured to, in a case where the evaluation value is greater than or equal to a threshold value, set the readout speed of the signal in the fourth control to be higher than a readout speed of the signal in the third control.(11) The imaging control device according to any one of (1) to (8), in which the processor is configured to set a readout speed of the signal in the fourth control to be higher than a readout speed of the signal in the third control.(12) The imaging control device according to any one of (1) to (11), in which the processor is configured to, after the second control is performed, perform, at least once, processing of performing the third control and the fourth control in this order.(13) The imaging control device according to (12), in which the processor is configured to, in a case where the readout of the signal from the charge holding units of the pixel rows other than the part of the pixel rows is completed by the third control, start the first control by omitting the fourth control that is to be performed subsequently to the third control, or together with the fourth control.(14) The imaging control device according to (12), in which the processor is configured to perform the processing n times with n being a plural number and, after the processing performed for the n-th time, perform the first control and a control (control of executing driving illustrated by straight line ROg4) of reading out a signal corresponding to the charges held in the charge holding units in a pixel row in which signal readout is not completed among the pixel rows other than the part of the pixel rows.(15) An imaging apparatus (digital camera100) comprising the imaging control device according to any one of (1) to (14), and the imaging element.(16) An imaging control method for controlling an imaging element including a plurality of pixel rows in each of which a plurality of pixels each including a photoelectric conversion unit and a charge holding unit which holds charges transferred from the photoelectric conversion unit are arranged in one direction, the imaging control method comprising performing a first control of exposing the plurality of pixel rows and of transferring charges accumulated in the photoelectric conversion units of the plurality of pixel rows by the exposing to the charge holding units, performing a second control of reading out a signal corresponding to the charges held in the charge holding units of a part of the pixel rows among the charge holding units in which the charges are held by the first control, performing a third control of reading out a signal corresponding to the charges held in the charge holding units of the pixel rows other than the part of the pixel rows among the charge holding units in which the charges are held by the first control, and of exposing the part of the pixel rows from which the signal is read out by the second control, and performing a fourth control of reading out a signal corresponding to the charges held in the charge holding units of the part of the pixel rows from which the signal is read out by the second control.(17) An imaging control program, which is stored in a computer readable medium, for controlling an imaging element including a plurality of pixel rows in each of which a plurality of pixels each including a photoelectric conversion unit and a charge holding unit which holds charges transferred from the photoelectric conversion unit are arranged in one direction, the imaging control program causing a processor to execute performing a first control of exposing the plurality of pixel rows and of transferring charges accumulated in the photoelectric conversion units of the plurality of pixel rows by the exposing to the charge holding units, performing a second control of reading out a signal corresponding to the charges held in the charge holding units of a part of the pixel rows among the charge holding units in which the charges are held by the first control, performing a third control of reading out a signal corresponding to the charges held in the charge holding units of the pixel rows other than the part of the pixel rows among the charge holding units in which the charges are held by the first control, and of exposing the part of the pixel rows from which the signal is read out by the second control, and performing a fourth control of reading out a signal corresponding to the charges held in the charge holding units of the part of the pixel rows from which the signal is read out by the second control.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG.1is a diagram illustrating a schematic configuration of a digital camera100that is one embodiment of an imaging apparatus according to the present invention. The digital camera100illustrated inFIG.1comprises a lens device40including an imaging lens1, a stop2, a lens drive unit8that drives the imaging lens1, a stop drive unit9that drives the stop2, and a lens controller4that controls the lens drive unit8and the stop drive unit9; and a body part100A.

The body part100A comprises an imaging element5, a system controller11that controls the entire electric control system of the digital camera100, an operation unit14, a display device22, a memory16including a random access memory (RAM), a read only memory (ROM), and the like, a memory controller15that controls data storage in the memory16and data readout from the memory16, a digital signal processing section17, and an external memory controller that controls data storage in a storage medium21and data readout from the storage medium21.

The lens device40may be attachable to and detachable from the body part100A or may be integrated with the body part100A. The imaging lens1includes a focus lens or the like that can be moved in an optical axis direction. The focus lens is a lens for adjusting a focal point of an imaging optical system including the imaging lens1and the stop2, and is composed of a single lens or of a plurality of lenses. By moving the focus lens in the optical axis direction, a position of a principal point of the focus lens changes along the optical axis direction, and a focal position on a subject side is changed. A liquid lens of which a position of a principal point in the optical axis direction can be changed by electrical control may be used as the focus lens.

The lens controller4of the lens device40changes the position of the principal point of the focus lens included in the imaging lens1by controlling the lens drive unit8based on a lens drive signal transmitted from the system controller11. The lens controller4of the lens device changes an amount of opening (F number) of the stop2by controlling the stop drive unit9based on a driving control signal transmitted from the system controller11.

The imaging element5images a subject through the imaging optical system including the imaging lens1and the stop2.

The imaging element5includes a light-receiving surface60(refer toFIG.2) on which a plurality of pixels are two-dimensionally disposed, converts a subject image formed on the light-receiving surface60by the imaging optical system into a pixel signal group via the plurality of pixels, and outputs the pixel signal group. For example, a complementary metal-oxide semiconductor (CMOS) image sensor is used as the imaging element5. The imaging element5is driven by a driver, not illustrated, and the driver is controlled by the system controller11.

The system controller11controls the entire digital camera100and has a hardware structure corresponding to various processors that perform processing by executing programs including an imaging control program. The programs executed by the system controller11are stored in the ROM of the memory16.

Examples of the various processors include a central processing unit (CPU) that is a general-purpose processor performing various types of processing by executing a program, a programmable logic device (PLD) such as a field programmable gate array (FPGA) that is a processor of which a circuit configuration can be changed after manufacture, or a dedicated electric circuit such as an application specific integrated circuit (ASIC) that is a processor having a circuit configuration dedicatedly designed to execute specific processing. More specifically, the various processors have a structure of an electric circuit in which circuit elements such as semiconductor elements are combined.

The system controller11may be configured with one of the various processors or may be configured with a combination of two or more processors of the same type or of different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and a FPGA).

The system controller11drives the imaging element5and the lens device40and outputs the subject image captured through the imaging optical system of the lens device40as a pixel signal group in accordance with the imaging control program. The system controller11and the memory16constitute an imaging control device. By processing the pixel signal group output from the imaging element5via the digital signal processing section17, captured image data that is data suitable for display on the display device22or is data suitable for storage in the storage medium21is generated.

A command signal from a user is input into the system controller11through the operation unit14. The operation unit14includes a touch panel integrated with a display surface22b, and various buttons and the like.

The display device22comprises the display surface22bconfigured with an organic electroluminescence (EL) panel, a liquid crystal panel, or the like, and a display controller22athat controls display on the display surface22b.

The memory controller15, the digital signal processing section17, the external memory controller20, and the display controller22aare connected to each other through a control bus24and through a data bus25and are controlled in accordance with instructions from the system controller11.

FIG.2is a schematic plan view illustrating a schematic configuration of the imaging element5illustrated inFIG.1.FIG.3is a schematic plan view illustrating a schematic configuration of a pixel61in the imaging element5illustrated inFIG.2.FIG.4is a schematic cross section view of the pixel61illustrated inFIG.3taken along line A-A.

The imaging element5comprises the light-receiving surface60on which a plurality of pixel rows62each consisting of a plurality of the pixels61arranged in a row direction X are arranged in a column direction Y orthogonal to the row direction X, a drive circuit63that drives the pixels61arranged on the light-receiving surface60, and a signal processing circuit64that processes a pixel signal read out into a signal line from each pixel61of the pixel rows62arranged on the light-receiving surface60.

The plurality of pixels61include a phase difference detection pixel that receives one of a pair of luminous fluxes which have passed through two different parts arranged in the row direction X in a pupil region of the imaging optical system, and that detects a signal corresponding to an amount of received light, a phase difference detection pixel that receives the other of the pair of luminous fluxes and that detects a signal corresponding to an amount of received light, and a normal pixel that receives both of the pair of luminous fluxes and that detects a signal corresponding to an amount of received light.

The pixel rows62include a first pixel row including only the normal pixel, and a second pixel row including the phase difference detection pixel and the normal pixel. For example, the second pixel rows are discretely disposed at equal intervals in the column direction Y. In the imaging element5, the phase difference detection pixel is not essential, and all of the pixels61may be configured with the normal pixels.

Hereinafter, inFIG.2, an end part on an upper side of the light-receiving surface60in the column direction Y will be referred to as an upper end, and an end part on a lower side of the light-receiving surface60in the column direction Y will be referred to as a lower end. The upper end constitutes one end of the light-receiving surface60, and the lower end constitutes the other end of the light-receiving surface60.

The pixels61disposed on the light-receiving surface60are classified into a plurality of groups. For example, it is assumed that M is an integer greater than or equal to 0, the number of groups set on the light-receiving surface60is n (here, n=4), and k is 1 to n. In the example inFIG.2, the pixel row62that is the (n×M+k)-th pixel row from the upper end side of the light-receiving surface60among all pixel rows62arranged on the light-receiving surface60is a group Gk. The pixels61disposed on the light-receiving surface60are classified into a group G1, a group G2, a group G3, and a group G4as illustrated inFIG.2. This grouping is an example, and the present invention is not limited thereto.

As illustrated inFIG.3, each pixel61comprises a photoelectric conversion unit61A, a charge holding unit61B, a charge transfer unit61C, a floating diffusion61D, and a readout circuit61E.

The photoelectric conversion unit61A receives light that has passed through the imaging optical system of the lens device40, and generates and accumulates charges corresponding to an amount of received light. The photoelectric conversion unit61A is configured with a photodiode or the like.

The charge transfer unit61C transfers the charges accumulated in the photoelectric conversion unit61A to the charge holding unit61B. The charge transfer unit61C is configured with an impurity region in a semiconductor substrate and with an electrode formed above the impurity region.

Charges are transferred from the photoelectric conversion unit61A to the charge holding unit61B by causing the drive circuit63to control a voltage applied to the electrode constituting the charge transfer unit61C.

The charge holding unit61B holds the charges transferred from the photoelectric conversion unit61A by the charge transfer unit61C. The charge holding unit61B is configured with the impurity region in the semiconductor substrate.

The floating diffusion61D is used for converting charges into signals, to which the charges held in the charge holding unit61B are transferred.

The readout circuit61E is a circuit that reads out a signal corresponding to a potential of the floating diffusion61D into a signal line65as a pixel signal. The readout circuit61E is driven by the drive circuit63. A set of the pixel signals constitutes the image signal.

As illustrated inFIG.4, a P-well layer71is formed on a surface of an N-type substrate70, and the photoelectric conversion unit61A is formed in a surface part of the P-well layer71.

The photoelectric conversion unit61A is configured with an N-type impurity layer73and with a P-type impurity layer74formed on the N-type impurity layer73. The N-type substrate70and the P-well layer71constitute the semiconductor substrate.

The charge holding unit61B consisting of an N-type impurity layer is formed in the surface part of the P-well layer71to be slightly spaced from the photoelectric conversion unit61A.

A transfer electrode76is formed above a region75of the P-well layer71between the charge holding unit61B and the photoelectric conversion unit61A through an oxide film, not illustrated.

The region75and the transfer electrode76constitute the charge transfer unit61C. While the transfer electrode76is also formed above the charge holding unit61B in the example inFIG.3, the transfer electrode76may be formed above at least the region75.

The charges accumulated in the photoelectric conversion unit61A can be transferred to the charge holding unit61B by controlling a potential of the transfer electrode76to form a channel in the region75. The potential of the transfer electrode76is controlled by the drive circuit63.

The floating diffusion61D consisting of an N-type impurity layer is formed in the surface part of the P-well layer71to be slightly spaced from the charge holding unit61B.

A reading electrode72is formed above the P-well layer71between the charge holding unit61B and the floating diffusion61D through an oxide film, not illustrated.

The charges held in the charge holding unit61B can be transferred to the floating diffusion61D by controlling a potential of the reading electrode72to form a channel in a region between the charge holding unit61B and the floating diffusion61D. The potential of the reading electrode72is controlled by the drive circuit63.

In the example illustrated inFIG.4, the readout circuit61E is configured with a reset transistor77for resetting the potential of the floating diffusion61D, an output transistor78that converts the potential of the floating diffusion61D into a pixel signal and that outputs the pixel signal, and a selection transistor79for selectively reading out the pixel signal output from the output transistor78into the signal line65. The configuration of the readout circuit is merely an example, and the present invention is not limited thereto. The readout circuit61E may be shared by the plurality of pixels61.

In each pixel61, a light shielding film, not illustrated, is provided, and a region other than the photoelectric conversion unit61A is shielded from light by the light shielding film.

The structure of each pixel61illustrated inFIG.3and inFIG.4is merely an example, and the present invention is not limited thereto.

The drive circuit63illustrated inFIG.2performs, by independently driving the transfer electrode76, the reading electrode72, and the readout circuit61E of each pixel61for each pixel row62, reset of each photoelectric conversion unit61A included in the pixel row62(discharge of the charges accumulated in the photoelectric conversion unit61A), readout of a pixel signal corresponding to the charges accumulated in each photoelectric conversion unit61A into the signal line65, and the like.

In addition, the drive circuit63transfers the charges from the photoelectric conversion unit61A to the charge holding unit61B of each pixel61at the same time by driving the charge transfer units61C of all of the pixels61at the same time. The drive circuit63is controlled by the system controller11.

The reset of the photoelectric conversion unit61A is performed by setting the charge transfer unit61C to a state of being able to transfer charges and by resetting the floating diffusion61D via the reset transistor77in a state where a channel is formed in the semiconductor substrate below the reading electrode72.

Thus, in a state where the readout of the pixel signal corresponding to the charges held in the charge holding unit61B is completed, the reset of the photoelectric conversion unit61A (in other words, a start of exposure of the photoelectric conversion unit61A) that transfers charges to the charge holding unit61B can be performed.

The signal processing circuit64illustrated inFIG.2performs correlative double sampling processing on the pixel signal read out into the signal line65from each pixel61of the pixel row62, converts the pixel signals after the correlative double sampling processing into a digital signal, and outputs the digital signal to the data bus25(refer toFIG.1). The signal processing circuit64is controlled by the system controller11. The digital signal processing section17generates the captured image data by performing signal processing such as demosaicing and gamma-correction processing on a pixel signal group output to the data bus25from the imaging element5.

The system controller11can drive the imaging element5in each of global reset driving, global shutter driving, rolling reset driving, rolling shutter driving, and rolling readout driving.

The global reset driving is driving of starting exposure of each pixel61at the same time by resetting the photoelectric conversion unit61A of each pixel61to be exposed at the same time.

The global shutter driving is driving of ending the exposure in each pixel61at the same time by transferring the charges accumulated in the photoelectric conversion unit61A of each pixel61because of the exposure started in each pixel61by the global reset driving to the charge holding unit61B at the same time.

The rolling reset driving is driving of sequentially performing, while changing the pixel row62, processing of resetting each photoelectric conversion unit61A of the pixel row62to start the exposure of each photoelectric conversion unit61A with respect to the plurality of pixel rows62to be exposed.

The rolling shutter driving is driving of sequentially performing, while changing the pixel row62, processing of transferring charges from the photoelectric conversion units61A of the pixel row62of which exposure is started by the rolling reset driving, to the charge holding units61B of the pixel row62to end the exposure of the pixel row62.

Performing the rolling reset driving and the rolling shutter driving results in a state where exposure periods of all of the plurality of pixel rows62to be exposed are slightly shifted from each other. However, a state where exposure periods of a part of the plurality of pixel rows to be exposed are shifted from each other is also possible by performing the rolling reset driving and the rolling shutter driving.

For example, a configuration in which the signal processing circuit64is further disposed above the light-receiving surface60is assumed. In this configuration, since two signal processing circuits64are present, pixel signals can be read out from two pixel rows62at the same time. Thus, in a case where every two adjacent pixel rows62among the plurality of pixel rows62to be exposed are set as a pair, it is possible to cause each pair to have the same exposure period and to shift the exposure period for each pair by sequentially performing processing of starting the exposure of the pair while changing the pair and by sequentially performing processing of ending the exposure of the pair while changing the pair.

The rolling readout driving is driving of sequentially reading out, for each pixel row62, the pixel signal corresponding to the charges held in each charge holding unit61B.

In a case where the digital camera100is set to an imaging mode, the system controller11continuously performs imaging for live view image display (hereinafter, referred to as LV imaging) based on a set of, for example, the rolling reset driving, the rolling shutter driving, and the rolling readout driving. The system controller11may perform the LV imaging based on a set of the global reset driving, the global shutter driving, and the rolling readout driving.

In a case where an instruction (hereinafter, referred to as an imaging instruction) to perform imaging for storage (hereinafter, referred to as main imaging) for storing still image data in the storage medium21is received during execution of the set, the system controller11performs the main imaging by performing a control of executing the global reset driving and the global shutter driving. The digital signal processing section17illustrated inFIG.1generates the captured image data by processing the pixel signal group output from the imaging element5by the main imaging and stores the captured image data in the storage medium21.

FIG.5is a timing chart illustrating operation of the digital camera100illustrated inFIG.1in the imaging mode. InFIG.5, a horizontal axis denotes a time point.

Driving timings of the photoelectric conversion units61A and the charge holding units61B of each pixel row62of the imaging element5are illustrated in the upper part and the middle part ofFIG.5. In the upper part and the middle part ofFIG.5, a vertical axis denotes a position of the pixel row62in the column direction Y.

Straight lines illustrated by solid lines and by broken lines in the upper part ofFIG.5illustrate timings at which the global reset driving and the global shutter driving are performed.

Straight lines illustrated by solid lines in the middle part ofFIG.5illustrate a timing at which charges are held in the charge holding units61B by the global shutter driving.

Straight lines illustrated by broken lines in the middle part ofFIG.5illustrate a timing at pixel signals are read out from the charge holding units61B by the rolling readout driving.

A drawing state of the display surface22bis illustrated in the lower part ofFIG.5. In the lower part ofFIG.5, a vertical axis denotes a position of a display pixel row of the display surface22bin the column direction Y. Straight lines of solid lines illustrated in the lower part ofFIG.5illustrate a timing at which drawing is performed in the display pixel row of the display surface22b.

In a case where the imaging mode is set, the system controller11performs a control of continuously performing the LV imaging and of displaying a live view image on the display surface22b. In a case where the imaging instruction is provided while the LV imaging is performed, the system controller11ends the LV imaging that is being executed when the imaging instruction is received.

At time point t1, the system controller11performs a control of executing the global reset driving illustrated by straight line GR and resets the photoelectric conversion units61A at the same time in all of the pixel rows62formed on the light-receiving surface60. Accordingly, exposure is started at the same timing in all of the pixel rows62formed on the light-receiving surface60.

Then, in a case where a predetermined exposure time elapses, the system controller11performs a control of executing the global shutter driving illustrated by straight line GS at time point t2. By this global shutter driving, charges are transferred from the photoelectric conversion units61A to the charge holding units61B at the same time in all of the pixel rows62formed on the light-receiving surface60, and the charges are held in the charge holding units61B as illustrated by straight line ST. Accordingly, exposure ends at the same timing in all of the pixel rows62formed on the light-receiving surface60. InFIG.5, a period surrounded by straight line GR and by straight line GS is illustrated as an exposure period EX of the main imaging.

After performing the control of executing the global shutter driving illustrated by straight line GS, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROg1. By this rolling readout driving, the pixel rows62in any one (here, the group G1is assumed) of the group G1, the group G2, the group G3, and the group G4are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62. A pixel signal group read out from the group G1by the rolling readout driving illustrated by straight line ROg1will be referred to as a pixel signal group IMG1.

The system controller11causes the digital signal processing section17to process the pixel signals that are sequentially read out from the pixel rows62of the group G1by the rolling readout driving illustrated by straight line ROg1. In a case where a line image is generated by this processing, the system controller11performs a control of displaying the line image on the display surface22b. By this control, a live view image (a so-called postview image PV) of the subject imaged by the main imaging (exposure period EX) is displayed on the display surface22bas illustrated by straight line DR1.

In a case where time point t3at which the rolling readout driving illustrated by straight line ROg1ends is reached, the system controller11performs a control of executing the global reset driving illustrated by straight line GRg1with respect to the group G1. By this global reset driving, the photoelectric conversion units61A are reset at the same time in all of the pixel rows62of the group G1, and exposure is started at the same timing in all of the pixel rows62of the group G1. At time point t3, readout of pixel signals from all of the charge holding units61B in the group G1is completed by the rolling readout driving illustrated by straight line ROg1. Thus, exposure can be started at the same time in all of the pixels61of the group G1at time point t3.

In addition, at time point t3, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROg2. By this rolling readout driving, the pixel rows62in any one (here, the group G2is assumed) of the group G2, the group G3, and the group G4are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62. A pixel signal group read out from the group G2by the rolling readout driving illustrated by straight line ROg2will be referred to as a pixel signal group IMG2.

In a case where time point t4at which the rolling readout driving illustrated by straight line ROg2ends is reached, the system controller11performs a control of executing the global shutter driving illustrated by straight line GSg1with respect to the group G1. By this global shutter driving, charges are transferred from the photoelectric conversion units61A to the charge holding units61B at the same time in all of the pixel rows62of the group G1, and the charges are held in the charge holding units61B as illustrated by straight line STg1. Accordingly, exposure ends at the same timing in all of the pixel rows62of the group G1. InFIG.5, a period surrounded by straight line GRg1and by straight line GSg1is illustrated as the exposure period EX (G1).

In addition, after performing the control of executing the global shutter driving illustrated by straight line GSg1at time point t4, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROG1. In this rolling readout driving, the pixel rows62of the group G1are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62.

The system controller11causes the digital signal processing section17to process the pixel signals that are sequentially read out from the pixel rows62of the group G1by the rolling readout driving which is started at time point t4and which is illustrated by straight line ROG1. In a case where a line image is generated by this processing, the system controller11performs a control of displaying the line image on the display surface22b. By this control, a live view image LV1of the subject captured by the group G1in a period F2between time point t3and time point t4is displayed on the display surface22bas illustrated by straight line DR2. InFIG.5, a period between time point t2and time point t3is illustrated as a period F1.

In a case where time point t5at which the rolling readout driving which is started at time point t4and which is illustrated by straight line ROG1ends is reached, the system controller11performs a control of executing the global reset driving illustrated by straight line GRg1with respect to the group G1. By this global reset driving, the photoelectric conversion units61A are reset at the same time in all of the pixel rows62of the group G1, and exposure is started at the same timing in all of the pixel rows62of the group G1.

In addition, at time point t5, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROg3. In this rolling readout driving, the pixel rows62in any one (here, the group G3is assumed) of the group G3and the group G4are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62. A pixel signal group read out from the group G3by the rolling readout driving illustrated by straight line ROg3will be referred to as a pixel signal group IMG3.

In a case where time point t6at which the rolling readout driving illustrated by straight line ROg3ends is reached, the system controller11performs a control of executing the global shutter driving illustrated by straight line GSg1with respect to the group G1. By this global shutter driving, charges are transferred from the photoelectric conversion units61A to the charge holding units61B at the same time in all of the pixel rows62of the group G1, and the charges are held in the charge holding units61B as illustrated by straight line STg1. Accordingly, exposure ends at the same timing in all of the pixel rows62of the group G1.

In addition, after performing the control of executing the global shutter driving illustrated by straight line GSg1at time point t6, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROG1. In this rolling readout driving, the pixel rows62of the group G1are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62.

The system controller11causes the digital signal processing section17to process the pixel signals that are sequentially read out from the pixel rows62of the group G1by the rolling readout driving which is started at time point t6and which is illustrated by straight line ROG1. In a case where a line image is generated by this processing, the system controller11performs a control of displaying the line image on the display surface22b. By this control, a live view image LV2of the subject captured in a period F4between time point t5and time point t6is displayed on the display surface22bas illustrated by straight line DR3. InFIG.5, a period between time point t4and time point t5is illustrated as a period F3.

In a case where time point t7at which the rolling readout driving which is started at time point t6and which is illustrated by straight line ROG1ends is reached, the system controller11performs a control of executing the global reset driving illustrated by straight line GRg1with respect to the group G1. By this global reset driving, the photoelectric conversion units61A are reset at the same time in all of the pixel rows62of the group G1, and exposure is started at the same timing in all of the pixel rows62of the group G1.

In addition, at time point t7, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROg4. In this rolling readout driving, the pixel rows62of the group G4are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62. A pixel signal group read out from the group G4by the rolling readout driving illustrated by straight line ROg4will be referred to as a pixel signal group IMG4.

In a case where time point t8at which the rolling readout driving illustrated by straight line ROg4ends is reached, the system controller11performs a control of executing the global shutter driving illustrated by straight line GSg1with respect to the group G1. By this global shutter driving, charges are transferred from the photoelectric conversion units61A to the charge holding units61B at the same time in all of the pixel rows62of the group G1, and the charges are held in the charge holding units61B as illustrated by straight line STg1. Accordingly, exposure ends at the same timing in all of the pixel rows62of the group G1.

In addition, after performing the control of executing the global shutter driving illustrated by straight line GSg1at time point t8, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROG1. In this rolling readout driving, the pixel rows62of the group G1are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62.

The system controller11causes the digital signal processing section17to process the pixel signals that are sequentially read out from the pixel rows62of the group G1by the rolling readout driving which is started at time point t8and which is illustrated by straight line ROG1. In a case where a line image is generated by this processing, the system controller11performs a control of displaying the line image on the display surface22b. By this control, a live view image LV3of the subject captured in a period F6between time point t7and time point t8is displayed on the display surface22bas illustrated by straight line DR4. InFIG.5, a period between time point t6and time point t7is illustrated as a period F5. In addition, a period between time point t8and time point t9at which the rolling readout driving which is started at time point t8and which is illustrated by straight line ROG1ends is illustrated as a period F7.

At time point t8, the digital signal processing section17generates the captured image data by processing the pixel signal group IMG1, the pixel signal group IMG2, the pixel signal group IMG3, and the pixel signal group IMG4read out between time point t2and time point t8, and stores the captured image data in the storage medium21.

InFIG.5, a control of executing driving illustrated by straight line GR and by straight line GS constitutes a first control. In addition, a control of executing driving illustrated by straight line ROg1in the period F1constitutes a second control. In addition, each of a control of executing driving illustrated by straight line ROg2, straight line GRg1, and straight line GSg1in the period F2, a control of executing driving illustrated by straight line ROg3, straight line GRg1, and straight line GSg1in the period F4, and a control of executing driving illustrated by straight line ROg4, straight line GRg1, and straight line GSg1in the period F6constitutes a third control. In addition, each of a control of executing driving illustrated by straight line ROG1in the period F3, a control of executing driving illustrated by straight line ROG1in the period F5, and a control of executing driving illustrated by straight line ROG1in the period F7constitutes a fourth control.

As described above, according to the digital camera100, even in a case where a total number of pixels61formed on the light-receiving surface60is large, and time is required until the readout of the pixel signals from all of the pixels61is completed, the postview image PV can be displayed before the readout of all pixel signals is completed, that is, specifically, shortly after time point t2illustrated inFIG.5. Thus, a state of the subject imaged in the exposure period EX can be instantly checked, and convenience of use can be improved.

In addition, according to the digital camera100, even after the postview image PV is displayed, imaging (exposure in the period F2, the period F4, and the period F6inFIG.5) for generating the live view image can be performed using the group G1in which the readout of the pixel signals is already completed. By this imaging, it is possible to display the live view image before the readout of all pixel signals obtained by the main imaging is completed, and furthermore, to update the live view image. Thus, the user can continue checking the state of the subject to be imaged based on the postview image PV and on the live view image continuously displayed from the postview image PV, and can perform favorable imaging by having an appropriate chance to press a shutter.

It is preferable that in each of the period F1, the period F3, and the period F5illustrated inFIG.5, the system controller11reads out the pixel signals from a group that includes the second pixel row including the phase difference detection pixel. That is, in the above description, it is preferable that the second pixel row is included in the group G1.

By reading out the pixel signals from the second pixel row in each of the period F1, the period F3, and the period F5, the system controller11can derive an evaluation value (a value for evaluating an amount of shift between a main subject and the focal position; for example, a defocus amount) for focal point adjustment using the pixel signal read out from the phase difference detection pixel. Accordingly, for example, in each of a period from time point t3to time point t5, a period from time point t5to time point t7, and a period from time point t7to time point t9, it is possible to derive the evaluation value and to perform control of the focus lens, prediction of a subject position, and the like based on the evaluation value. Consequently, focal point adjustment can be performed at a high speed with high accuracy before the subsequent main imaging is started. Particularly, in a continuous shooting mode in which the main imaging is continuously performed a plurality of times in accordance with the imaging instruction, the evaluation value can be derived with high frequency. Thus, quality of a captured image can be improved.

The pixel signals from the group including the second pixel row may be read out in at least one of the period F2or the period F4instead of the period F1, the period F3, and the period F5. Even in this case, focal point adjustment in the subsequent main imaging can be performed at a high speed with high accuracy.

In the operation illustrated inFIG.5, the display of the postview image PV may be omitted. Even in this case, the state of the subject can be checked immediately after the main imaging based on the live view image LV1and on the live view image LV2.

In addition, in the operation illustrated inFIG.5, the display of the live view image LV1may be omitted. In this case, the postview image PV can be displayed longer, and the subject imaged in the main imaging can be checked in detail.

In addition, in the operation illustrated inFIG.5, the display of the live view image LV1, the live view image LV2, and the live view image LV3may be omitted. Even in this case, for example, by performing subject detection processing or derivation processing of the evaluation value using the captured image data captured by the group G1in each of the period F2, the period F4, and the period F6, it is possible to improve accuracy of motion prediction of a specific subject or to improve accuracy of a control of continuously focusing on the specific subject.

Even in a case where the postview image PV, the live view image LV1, the live view image LV2, and the live view image LV3are displayed as illustrated inFIG.5, for example, by performing the subject detection processing or the derivation processing of the evaluation value together using the captured image data captured by the group G1in each of the period F2, the period F4, and the period F6, it is possible to improve the accuracy of the motion prediction of the specific subject or to improve the accuracy of the control of continuously focusing on the specific subject.

While the pixels61formed on the light-receiving surface60are classified into four groups in the above description, the pixels61may be classified into two or more groups. For example, in a configuration in which the group G4is not present on the light-receiving surface60, the processing illustrated inFIG.5may be changed by removing the processing of time point t7and later.

In addition, for example, the group G1and the group G3may be combined as a first group, and the group G2and the group G4may be combined as a second group. In this case, the system controller11may perform a control of displaying the postview image by reading out the pixel signals from the first group in the period F1, perform a control of exposing the first group and of reading out the pixel signals from the second group in the period F2, and perform a control of displaying the live view image by reading out the pixel signals from the first group, performing the subject detection processing, or performing the derivation processing of the evaluation value in the period F3.

Hereinafter, modification examples of the operation of the digital camera100in the imaging mode will be described.

FIG.6is a timing chart illustrating a first modification example of the operation of the digital camera100illustrated inFIG.1in the imaging mode. The timing chart illustrated inFIG.6is the same as the timing chart illustrated inFIG.5except that the global reset driving illustrated by straight line GRg1inFIG.5is changed to the rolling reset driving illustrated by straight line RRg1, the global shutter driving illustrated by straight line GSg1inFIG.5is changed to the rolling shutter driving illustrated by straight line RSg1, straight line STg1illustrated inFIG.5is removed, and the rolling readout driving illustrated by straight line ROG1inFIG.5is changed to the rolling readout driving illustrated by straight line rog1.

The system controller11performs a control of executing the rolling reset driving illustrated by straight line RRg1at time point t3. In this rolling reset driving, the pixel rows62of the group G1are sequentially selected from the upper end side, and the photoelectric conversion units61A in the selected pixel row62are reset.

In a case where the rolling reset driving ends at time point t4, the system controller11performs a control of executing the rolling shutter driving illustrated by straight line RSg1. In this rolling shutter driving, the pixel rows62of the group G1are sequentially selected from the upper end side, and the charges accumulated in the photoelectric conversion units61A in the selected pixel row62are transferred to the charge holding units61B. In a case where charges are transferred to the charge holding units61B by this rolling shutter driving, pixel signals corresponding to the charges are read out by the rolling readout driving illustrated by straight line rog1, and the live view image LV1is displayed based on the pixel signals.

The system controller11also performs a control of executing the rolling reset driving with respect to the group G1at time point t5, and in a case where this rolling reset driving ends at time point t6, performs a control of executing the rolling shutter driving with respect to the group G1. In a case where charges are transferred to the charge holding units61B by this rolling shutter driving, pixel signals corresponding to the charges are read out by the rolling readout driving illustrated by straight line rog1, and the live view image LV2is displayed based on the pixel signals.

The system controller11also performs a control of executing the rolling reset driving with respect to the group G1at time point t7, and in a case where this rolling reset driving ends at time point t8, performs a control of executing the rolling shutter driving with respect to the group G1. In a case where charges are transferred to the charge holding units61B by this rolling shutter driving, pixel signals corresponding to the charges are read out by the rolling readout driving illustrated by straight line rog1, and the live view image LV3is displayed based on the pixel signals.

As described above, according to the processing illustrated inFIG.6, a time required from the end of the exposure of each pixel row62of the group G1illustrated by straight line RSg1to the start of the display of the line image corresponding to each pixel row62can be uniform for all of the pixel rows62of the group G1. Thus, quality of the live view image LV1, the live view image LV2, and the live view image LV3can be improved.

In the processing illustrated inFIG.6, each of a control of executing driving illustrated by straight line ROg2, straight line RRg1, and straight line RSg1in the period F2and in the period F3, a control of executing driving illustrated by straight line ROg3, straight line RRg1, and straight line RSg1in the period F4and in the period F5, and a control of executing driving illustrated by straight line ROg4, straight line RRg1, and straight line RSg1in the period F6and in the period F7constitutes the third control. In addition, a control of executing driving illustrated by straight line rog1constitutes the fourth control.

FIG.7is a timing chart illustrating a second modification example of the operation of the digital camera100illustrated inFIG.1in the imaging mode. In the timing chart illustrated inFIG.7, the processing of time point t3and later is different from that inFIG.5. In addition, in the second modification example, it is assumed that the second pixel row is included in a group (group G1) in which pixel signals are initially read out after the main imaging.

The system controller11acquires the pixel signal of the phase difference detection pixel among the pixel signals read out from the group G1by the rolling readout driving started in the period F1, and derives the evaluation value for focal point adjustment based on the pixel signal.

The system controller11determines whether to execute or not execute the control of time point t3and later (that is, the third control and the fourth control) inFIG.5based on the derived evaluation value. Specifically, in a case where the evaluation value is less than a threshold value, the system controller11determines to execute the control of time point t3and later inFIG.5and executes the control. The operation in this case is the same as illustrated in FIG.5.

On the other hand, in a case where the evaluation value is greater than or equal to the threshold value, the system controller11determines not to execute the control of time point t3and later inFIG.5and, as illustrated inFIG.7, performs a control of executing the rolling readout driving illustrated by straight line RO instead of the third control and of the fourth control. In this rolling readout driving, the pixel rows62included in the group G2, the group G3, and the group G4are sequentially selected from the upper end toward the lower end of the light-receiving surface60, and pixel signals are read out from the charge holding units61B of the selected pixel row62.

The control of executing the rolling readout driving illustrated by straight line RO constitutes a fifth control.

A pixel signal group read out from the group G2, the group G3, and the group G4by the rolling readout driving illustrated by straight line RO will be referred to as a pixel signal group IMG5.

In a case where the rolling readout driving illustrated by straight line RO ends, the digital signal processing section17generates the captured image data by processing the pixel signal group IMG1and the pixel signal group IMG5read out from the imaging element5, and stores the captured image data in the storage medium21. In a case where the rolling readout driving illustrated by straight line RO ends, the system controller11starts the subsequent main imaging.

In a case where the evaluation value derived based on the pixel signals read out from the group G1in the period F1is high, it is estimated that a degree of blurriness of the captured image data obtained by the main imaging is high. Accordingly, in such a case, a time until the subsequent main imaging can be performed can be shortened by reading out pixel signals from the group G2, the group G3, and the group G4at once instead of reading out pixel signals from the group G2, the group G3, and the group G4in separate periods and of performing imaging based on the group G1between each readout. Consequently, it is possible to perform favorable imaging by having an appropriate chance to press the shutter.

In the operation example illustrated inFIG.7, in a case where the evaluation value derived based on the pixel signals read out from the group G1is greater than or equal to the threshold value, the rolling readout driving illustrated by straight line RO is performed.

As a modification example, in a case where the evaluation value is greater than or equal to the threshold value, the system controller11may stop reading out pixel signals from the group G2, the group G3, and the group G4and perform a control of performing the subsequent main imaging (that is, the first control).

FIG.8is a timing chart illustrating a third modification example of the operation of the digital camera100illustrated inFIG.1in the imaging mode. The timing chart illustrated inFIG.8is different fromFIG.7in that the system controller11stops reading out pixel signals from the group G2, the group G3, and the group G4at time point t3and then, performs a control of starting the subsequent main imaging.

InFIG.8, after the end of the exposure period EX started after time point t3, for example, the processing of time point t3and later illustrated inFIG.5is performed in a case where the evaluation value derived based on the pixel signals read out from the group G1exposed in the exposure period EX is less than the threshold value, and the subsequent main imaging is performed in a case where the evaluation value is greater than or equal to the threshold value. Even in the case inFIG.8, it is possible to shorten a time until the subsequent main imaging can be performed, and to perform favorable imaging by having an appropriate chance to press the shutter.

FIG.9is a timing chart illustrating a fourth modification example of the operation of the digital camera100illustrated inFIG.1in the imaging mode. The timing chart illustrated inFIG.9is different fromFIG.5in that a length of each of the period F3, the period F5, and the period F7is shorter than a length of each of the period F1, the period F2, the period F4, and the period F6.

That is, in the timing chart illustrated inFIG.9, a readout speed of pixel signals by the rolling readout driving illustrated by straight line ROG1is higher than a readout speed of pixel signals by the rolling readout driving illustrated by each of straight line ROg1, straight line ROg2, straight line ROg3, and straight line ROg4.

Changing of the readout speed of pixel signals from the imaging element5can be performed by, for example, changing the number of conversion bits set in an analog to digital (AD) converter included in the signal processing circuit64or by changing a clock frequency of the AD converter. In addition, the readout speed can also be changed by changing the number of pixels61in which pixel signals are read out from the group G1.

According to the processing illustrated inFIG.9, it is possible to shorten a time until the readout of all pixel signals obtained by the main imaging is completed, and to start the subsequent main imaging quickly. In the continuous shooting mode, a continuous shooting speed can be increased.

A control of increasing the readout speed of pixel signals in the period F3, the period F5, and the period F7as illustrated inFIG.9may be performed at all times or may be performed only in a case where the evaluation value is greater than or equal to the threshold value.

That is, in a case where the evaluation value derived based on the pixel signals read out from the group G1including the phase difference detection pixel in the period F1is greater than or equal to the threshold value, the system controller11may execute the control of increasing the speed. In a case where the evaluation value is less than the threshold value, the system controller11may not execute the control of increasing the speed so that the readout speed of pixel signals in the period F3, the period F5, and the period F7are the same as the readout speed of pixel signals in the period F1, the period F2, the period F4, and the period F6.

By doing so, a time until the subsequent main imaging can be performed can be shortened in a case where significant blurriness occurs in the subject image captured by the main imaging. On the other hand, in a case where significant blurriness does not occur in the subject image captured by the main imaging, it is possible to display the postview image and the live view image under the same condition or to increase image quality of the live view image, and it is possible to favorably observe the subject being imaged.

FIG.10is a timing chart illustrating a fifth modification example of the operation of the digital camera100illustrated inFIG.1in the imaging mode. The timing chart illustrated inFIG.10is the same as the timing chart illustrated inFIG.5except that the subsequent main imaging is started in the period F7.

In the timing chart illustrated inFIG.10, in a case where the rolling readout driving illustrated by straight line ROg4ends at time point t8, the system controller11performs a control of executing the subsequent main imaging (that is, the first control) in parallel with a control of executing the rolling readout driving illustrated by straight line ROG1.

InFIG.10, operation after the end of the exposure period EX started after time point t8is the same as the operation from time point t2to time point t9. According to the processing illustrated inFIG.10, it is possible to shorten a time until the subsequent main imaging, and the continuous shooting speed can be increased in the continuous shooting mode. InFIG.10, driving illustrated by straight line GRg1and by straight line GSg1in the period F6and driving illustrated by straight line ROG1in the period F7are not essential and can be omitted.

FIG.11is a timing chart illustrating a sixth modification example of the operation of the digital camera100illustrated inFIG.1in the imaging mode. The timing chart illustrated inFIG.11is the same as the timing chart illustrated inFIG.5except that the exposure of the group G1in the period F6and the readout of pixel signals obtained by the exposure in the period F7are removed, and that the subsequent main imaging is started at time point t7.

In the timing chart illustrated inFIG.11, in a case where the rolling readout driving illustrated by straight line ROG1ends at time point t7, the system controller11performs a control of executing the rolling readout driving illustrated by straight line ROg4(that is, the fourth control) in parallel with a control of executing the subsequent main imaging (that is, the first control). Operation after the end of the exposure period EX started at time point t7is the same as the operation from time point t2to time point t8. According to the processing illustrated inFIG.11, it is possible to shorten a time until the subsequent main imaging, and the continuous shooting speed can be increased in the continuous shooting mode.

Next, a configuration of a smartphone that is another embodiment of the imaging apparatus according to the present invention will be described.

FIG.12illustrates an exterior of a smartphone200. The smartphone200illustrated inFIG.12includes a casing201having a flat plate shape and comprises a display and input unit204in which a display panel202as a display unit and an operation panel203as an input unit are integrated on one surface of the casing201.

The casing201comprises a speaker205, a microphone206, an operation unit207, and a camera unit208. The configuration of the casing201is not limited thereto and can employ, for example, a configuration in which the display unit and the input unit are independently disposed, or a configuration that has a folded structure or a sliding mechanism.

FIG.13is a block diagram illustrating a configuration of the smartphone200illustrated inFIG.12.

As illustrated inFIG.13, a wireless communication unit210, the display and input unit204, a call unit211, the operation unit207, the camera unit208, a storage unit212, an external input-output unit213, a global navigation satellite system (GNSS) reception unit214, a motion sensor unit215, a power supply unit216, and a main controller220are comprised as main constituents of the smartphone.

In addition, a wireless communication function of performing mobile wireless communication with a base station apparatus BS, not illustrated, through a mobile communication network NW, not illustrated, is provided as a main function of the smartphone200.

The wireless communication unit210performs wireless communication with the base station apparatus BS accommodated in the mobile communication network NW in accordance with an instruction from the main controller220. By using the wireless communication, transmission and reception of various file data such as audio data and image data, electronic mail data, or the like and reception of web data, streaming data, or the like are performed.

The display and input unit204is a so-called touch panel that visually delivers information to the user by displaying images (still images and video images), text information, or the like and that detects a user operation with respect to the displayed information under control of the main controller220. The display and input unit204comprises the display panel202and the operation panel203.

The display panel202uses a liquid crystal display (LCD), an organic electro-luminescence display (OELD), or the like as a display device.

The operation panel203is a device that is placed such that an image displayed on a display surface of the display panel202can be visually recognized, and that detects one or a plurality of coordinates operated with a finger of the user or with a stylus. In a case where the device is operated with the finger of the user or with the stylus, a detection signal generated by the operation is output to the main controller220. Next, the main controller220detects an operation position (coordinates) on the display panel202based on the received detection signal.

As illustrated inFIG.13, while the display panel202and the operation panel203of the smartphone200illustrated as one embodiment of the imaging apparatus according to the present invention are integrated to constitute the display and input unit204, the operation panel203is disposed to completely cover the display panel202.

In a case where such disposition is employed, the operation panel203may have a function of detecting the user operation even in a region outside the display panel202. In other words, the operation panel203may comprise a detection region (hereinafter, referred to as a display region) for an overlapping part overlapping with the display panel202and a detection region (hereinafter, referred to as a non-display region) for an outer edge part, other than the overlapping part, that does not overlap with the display panel202.

A size of the display region and a size of the display panel202may completely match, but both sizes do not need to match. In addition, the operation panel203may comprise two sensitive regions of the outer edge part and an inner part other than the outer edge part. Furthermore, a width of the outer edge part is appropriately designed depending on a size and the like of the casing201.

Furthermore, examples of a position detection method employed in the operation panel203include a matrix switch method, a resistive membrane system, a surface acoustic wave method, an infrared method, an electromagnetic induction method, and a capacitance method. Any of the methods can be employed.

The call unit211comprises the speaker205or the microphone206, and converts voice of the user input through the microphone206into audio data processable in the main controller220and outputs the audio data to the main controller220, or decodes audio data received by the wireless communication unit210or by the external input-output unit213and outputs the decoded audio data from the speaker205.

In addition, as illustrated inFIG.12, for example, the speaker205can be mounted on the same surface as a surface on which the display and input unit204is provided, and the microphone206can be mounted on a side surface of the casing201.

The operation unit207is a hardware key that uses a key switch or the like, and receives an instruction from the user. For example, as illustrated inFIG.12, the operation unit207is a push button-type switch that is mounted on a side surface of the casing201of the smartphone200, and that is set to an ON state in a case where the switch is pressed with the finger or the like and is set to an OFF state by restoring force of a spring or the like in a case where the finger is released.

In the storage unit212, a control program and control data of the main controller220, application software, address data in which a name, a telephone number, or the like of a communication counterpart is associated, transmitted and received electronic mail data, web data downloaded by web browsing, and downloaded contents data are stored, and streaming data or the like is temporarily stored. In addition, the storage unit212is configured with an internal storage unit217incorporated in the smartphone and with an external storage unit218that includes a slot for an attachable and detachable external memory.

Each of the internal storage unit217and the external storage unit218constituting the storage unit212is implemented using a storage medium such as a memory (for example, a MicroSD (registered trademark) memory) of a flash memory type, a hard disk type, a multimedia card micro type, or a card type, a random access memory (RAM), or a read only memory (ROM).

The external input-output unit213functions as an interface with all external apparatuses connected to the smartphone200and is directly or indirectly connected to other external apparatuses by communication or the like (for example, a universal serial bus (USB), IEEE1394, Bluetooth (registered trademark), radio frequency identification (RFID), infrared communication (Infrared Data Association (IrDA) (registered trademark)), Ultra Wideband (UWB) (registered trademark), or ZigBee (registered trademark)) or through a network (for example, Ethernet (registered trademark) or a wireless local area network (LAN)).

For example, the external apparatuses connected to the smartphone200include a wired/wireless headset, a wired/wireless external charger, a wired/wireless data port, a memory card and a subscriber identity module (SIM)/user identity module (UIM) card connected through a card socket, an external audio and video apparatus connected through an audio and video input/output (I/O) terminal, a wirelessly connected external audio and video apparatus, a smartphone connected in a wired/wireless manner, a personal computer connected in a wired/wireless manner, and an earphone connected in a wired/wireless manner.

The external input-output unit213can deliver data transferred from the external apparatuses to each constituent in the smartphone200or transfer data in the smartphone200to the external apparatuses.

The GNSS reception unit214receives GNSS signals transmitted from GNSS satellites ST1to STn, executes positioning computation processing based on the received plurality of GNSS signals, and detects a position consisting of a latitude, a longitude, and an altitude of the smartphone200in accordance with an instruction from the main controller220.

In a case where positional information can be acquired from the wireless communication unit210or from the external input-output unit213(for example, a wireless LAN), the GNSS reception unit214can detect the position using the positional information.

The motion sensor unit215comprises, for example, a three-axis acceleration sensor and detects a physical motion of the smartphone200in accordance with an instruction from the main controller220. By detecting the physical motion of the smartphone200, a movement direction or acceleration of the smartphone200is detected. A detection result is output to the main controller220.

The power supply unit216supplies power stored in a battery (not illustrated) to each part of the smartphone200in accordance with an instruction from the main controller220.

The main controller220comprises a microprocessor, operates in accordance with the control program and with the control data stored in the storage unit212, and generally controls each part of the smartphone200. The microprocessor of the main controller220has the same function as the system controller11. In addition, the main controller220has a mobile communication control function of controlling each part of a communication system and an application processing function in order to perform voice communication or data communication through the wireless communication unit210.

The application processing function is implemented by operating the main controller220in accordance with the application software stored in the storage unit212. For example, the application processing function is an infrared communication function of performing data communication with counter equipment by controlling the external input-output unit213, an electronic mail function of transmitting and receiving electronic mails, or a web browsing function of viewing a web page.

In addition, the main controller220has an image processing function such as displaying an image on the display and input unit204based on image data (data of a still image or of a video image) such as reception data or downloaded streaming data.

The image processing function refers to a function of causing the main controller220to decode the image data, perform image processing on the decoding result, and display the image on the display and input unit204.

Furthermore, the main controller220executes a display control of the display panel202and an operation detection control of detecting the user operation performed through the operation unit207and through the operation panel203.

By executing the display control, the main controller220displays an icon for starting the application software or for a software key such as a scroll bar or displays a window for creating an electronic mail.

The scroll bar refers to a software key for receiving an instruction to move a display part of a large image or the like that does not fit in the display region of the display panel202.

In addition, by executing the operation detection control, the main controller220detects the user operation performed through the operation unit207, receives an operation with respect to the icon and an input of a text string in an input field of the window through the operation panel203, or receives a request for scrolling the display image made through the scroll bar.

Furthermore, by executing the operation detection control, the main controller220is provided with a touch panel control function of determining whether the operation position on the operation panel203is in the overlapping part (display region) overlapping with the display panel202or is in the other outer edge part (non-display region) not overlapping with the display panel202and of controlling the sensitive region of the operation panel203or a display position of the software key.

In addition, the main controller220can detect a gesture operation with respect to the operation panel203and execute a function set in advance in accordance with the detected gesture operation.

The gesture operation is not a simple touch operation in the related art and means an operation of drawing a path with the finger or the like, designating a plurality of positions at the same time, or as a combination thereof, drawing a path for at least one of the plurality of positions.

The camera unit208includes the lens device40, the imaging element5, and the digital signal processing section17illustrated inFIG.1. In the smartphone200, the main controller220and the internal storage unit217constitute the imaging control device.

Captured image data generated by the camera unit208can be stored in the storage unit212or be output through the external input-output unit213or through the wireless communication unit210.

In the smartphone200illustrated inFIG.13, the camera unit208is mounted on the same surface as the display and input unit204. However, a mount position of the camera unit208is not limited thereto. The camera unit208may be mounted on a rear surface of the display and input unit204.

In addition, the camera unit208can be used in various functions of the smartphone200. For example, an image acquired by the camera unit208can be displayed on the display panel202, or the image of the camera unit208can be used as an operation input of the operation panel203.

In addition, in detecting the position via the GNSS reception unit214, the position can be detected by referring to the image from the camera unit208. Furthermore, by referring to the image from the camera unit208, it is possible to determine an optical axis direction of the camera unit208of the smartphone200or to determine the current use environment without using the three-axis acceleration sensor or by using the three-axis acceleration sensor together. The image from the camera unit208can also be used in the application software.

In addition, image data of a still image or of a video image to which the positional information acquired by the GNSS reception unit214, voice information (may be text information acquired by performing voice to text conversion via the main controller or the like) acquired by the microphone206, posture information acquired by the motion sensor unit215, or the like is added can be stored in the storage unit212or be output through the external input-output unit213or through the wireless communication unit210. Even with the smartphone200having the above configuration, the same effect as the digital camera100can be obtained.

EXPLANATION OF REFERENCES