Device and method for processing high-resolution image

A device includes an image sensor configured to generate a first signal corresponding to an image having a first resolution in a first mode, a second signal corresponding to an image having a second resolution higher than the first resolution in a second mode. The image sensor is configured to generate frame information regarding a resolution, the first mode and the second mode respectively determined based on a mode signal. The device further includes a channel allocator configured to allocate the first signal and the second signal to different channels, of a plurality of channels, based on the frame information; and an image signal processor (ISP) comprising the plurality of channels, a first channel configured to process the first signal and a second channel configured to process the second signal. The ISP is configured to post-process image data processed by the plurality of channels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from Korean Patent Application Nos. 10-2019-0161678 and 10-2020-0030381, filed on Dec. 6, 2019 and Mar. 11, 2020, respectively, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entireties by reference.

BACKGROUND

The inventive concept relates to image processing, and more particularly, to a device and method for processing a high-resolution image.

As the number of cameras mounted on devices increases or the number of images that may be captured per second increases, the image processing capability of the device may be required to be increased. Accordingly, it may be required to efficiently solve various problems caused by limitations of image processing capabilities, such as frame drop and shutter lag.

SUMMARY

The inventive concept provides a device capable of determining that a photographing mode for preview or capture is changed in a frame unit and allocating an appropriate channel according to resolution.

According to an aspect of the inventive concept, there is provided a device including an image sensor configured to generate a first signal corresponding to an image having a first resolution in a first mode, a second signal corresponding to an image having a second resolution higher than the first resolution in a second mode, and generate frame information regarding a resolution, the first mode and the second mode respectively determined based on a mode signal; a channel allocator configured to allocate the first signal and the second signal to different channels, of a plurality of channels, based on the frame information; and an image signal processor (ISP) comprising the plurality of channels, a first channel of the plurality of channels configured to process the first signal and a second channel of the plurality of channels configured to process the second signal, and wherein the ISP is configured to post-process image data processed by the plurality of channels.

According to another aspect of the inventive concept, there is provided a device including an image sensor configured to generate a first signal corresponding to an image having a first resolution in a first mode, a second signal corresponding to an image having a second resolution higher than the first resolution in a second mode, and generate frame information regarding a resolution, the first mode and the second mode respectively determined based on a mode signal; a channel allocator configured to allocate the first signal and the second signal to different channels, of a plurality of channels, based on the frame information; an image signal processor (ISP) comprising the plurality of channels, a first channel of the plurality of channels configured to process the first image and a second channel of the plurality of channels configured to process the second image, and configured to generate a third signal as a result of post-processing the first signal, and generate a fourth signal as a result of post-processing the second signal; a memory subsystem including a first memory in which the third signal is temporarily stored and a second memory in which the fourth signal is temporarily stored; a controller configured to generate the mode signal and apply the mode signal corresponding to the second mode to the image sensor in response to a capture command; a display unit configured to load and display at least one of the third signal and the fourth signal for a user to preview the image having the first resolution in advance; and a capture unit configured to load the fourth signal based on the capture command to generate the image having the second resolution.

According to another aspect of the inventive concept, there is provided an image processing method including generating a first signal corresponding to an image having a first resolution in a first mode; generating a second signal corresponding to an image having a second resolution higher than the first resolution in a second mode; generating frame information in the form of a virtual channel ID based on a predetermined channel standard; and determining a resolution of an image frame based on the frame information and allocating the first signal and the second signal to different channels based on the determined resolution.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the inventive concept will be described with reference to the accompanying drawings.

FIG. 1is a block diagram illustrating a device10according to embodiments of the inventive concept.

Referring toFIG. 1, the device10is capable of capturing and/or storing an image of an object using a solid-state image sensor such as a charge-coupled device and a complementary metal oxide semiconductor (CMOS) and may be implemented in a digital camera, a digital camcorder, a mobile phone, or a tablet computer, or any other portable device. The portable device may include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, a wearable device, and the like. In addition, the device10may be mounted on an electronic device such as a drone, an advanced driver assistance system (ADAS), etc., or an electronic device provided as parts for a vehicle, furniture, manufacturing facilities, doors, various measuring devices, etc.

The device10according to the inventive concept may capture an image of a subject (or an object) for each frame. Based on a mode signal MODE generated according to a user's capture command CAPTURE, the device10senses a low-resolution image or a high-resolution image. Frame information FI may be generated to distinguish a low-resolution image frame from a high-resolution image frame among sensed image frames. The generated frame information FI may be included in channel information according to a standardized channel standard or an arbitrary channel standard determined by a product producer group. For example, the channel information may be stored in a header of a transmitted signal, and may be transmitted to another intellectual property (IP) through a virtual channel without using additional data storage space. The arbitrary channel standard determined by the product producer group may be a Mobile Industry Processor Interface (MIPI) Alliance in which mobile device manufacturers have established a common interface specification, but is not limited thereto and various arbitrary channel standards may apply. In addition, the channel information may be transmitted to another IP in the form of embedded data in which the frame information FI is stored in an embedded chip other than the virtual channel.

The device10according to the inventive concept may read the channel information from the frame information FI to determine the resolution of the captured image frame. Based on the frame information FI, when it is determined that the current image frame is a low-resolution image frame, the device10may allocate a signal in regard to the low-resolution image frame to a low-resolution channel with optimized environment configurations to process the low-resolution image frame. When it is determined that a received next image frame is a high-resolution image frame, the device10may allocate a signal in regard to the high-resolution image frame to a high-resolution channel with optimized environment configurations to process the high-resolution image frame. That is, the device10may set a path such that the high-resolution image frame and the low-resolution image frame pass through different channels. That is, the device10may dynamically allocate the high-resolution image frame and the low-resolution image frame according to resolution.

The device10according to the inventive concept may post-process the high-resolution image frame and the low-resolution image frame input through different channels using one processor. That is, although the device10uses one processor, the device10processes image frames input through channels with different environment configurations and thereby does not require the use of multiple processors.

The device10according to embodiments of the inventive concept may temporarily store the post-processed low-resolution image frame, and then display the low-resolution image frame on a preview screen such that the user may preview the captured subject (or object). Because the low-resolution image frame has fewer pixels and less data processing than the high-resolution image frame, the low-resolution image frame may be quickly displayed on the preview screen. After temporarily storing the post-processed high-resolution image frame, the device10may capture the object in high-resolution in response to the user's capture command CAPTURE. The device10uses a channel with an environment configuration to rapidly process the high-resolution image frame, and thus, the device10may quickly capture an object without delay or frame loss even when the resolution changes from the low-resolution image frame to the high-resolution image frame. In addition, the device10may perform scaling on the post-processed high-resolution image frame to generate a low-resolution image, thereby displaying the processed image on the preview screen (of a display).

Referring toFIG. 1, the device10according to an embodiment of the inventive concept may include an image sensor100and an image processor250including a channel allocator200, a controller260, and an image signal processor (ISP)300.

The image sensor100may convert an optical signal of a subject (or an object) incident through an optical lens into an electrical signal, generate image data for each frame based on electrical signals, and output the generated image data to the image processor250. The image sensor100may be mounted on an electronic device having an image or light sensing function. For example, the image sensor100may be mounted on an electronic device such as a camera, a smart phone, a wearable device, an Internet of Things (IoT) device, a tablet personal computer (PC), a PDA, a PMP, a navigation system, a drone, an ADAS, etc. In addition, the image sensor100may be mounted on an electronic device provided as parts in a vehicle, furniture, manufacturing facilities, doors, various measuring devices, and the like. The image sensor100may sense an object or subject (i.e., capture the light (i.e., optical signal) required to form a digital image of an object or subject) through a lens under the control of the ISP300or the image processor250.

The image sensor100may include a plurality of elements that process the received optical signal for each frame. The plurality of elements may include additional components for processing an optical signal or for improving image sensing sensitivity, such as a pixel array, a row driver, a ramp signal generator, a timing generator, an analog-to-digital converter and a readout circuit including an output buffer. Each of a plurality of pixels may include a photo sensing element that senses light and converts the sensed light into a pixel signal that is an electrical signal. For example, the photo sensing element may be a photodiode, a photo transistor, a photo gate, a pinned photodiode (PPD), or a combination thereof. Each of a plurality of photo sensing elements may have a 4-transistor structure including a photodiode, a transfer transistor, a reset transistor, an amplification transistor, and a selection transistor. According to an embodiment, each of the plurality of photo sensing elements may have a 1-transistor structure, a 3-transistor structure, or a 5-transistor structure, or a structure in which the plurality of pixels share some transistors.

In processing an incident subject (or object) for each frame, the image sensor100may sense the incident object by varying the resolution based on a mode signal MODE received from the image processor250. According to an embodiment of the inventive concept, the image sensor100may set a photographing mode to a low-resolution mode as a default mode, thereby generating a low-resolution pixel signal. The mode signal MODE may be generated based on a capture command CAPTURE generated by a user. When the image sensor100receives the mode signal MODE based on the capture command CAPTURE from the user or the mode signal MODE in response to a photographing mode change command (e.g., change to a high-resolution mode), the image sensor100may change the photographing mode from the previous low-resolution mode to the high-resolution mode to sense the image frame of the object with high sensitivity, thereby generating a high-resolution pixel signal. In other words, the image sensor100may process the image of the object for each frame, and each frame may have a different resolution. Here, for convenience of description, resolutions sensed by the image sensor100are two resolutions of low-resolution and high-resolution, but are not limited thereto and various levels of resolution may be sensed. If necessary, the resolution may be sensed with low-resolution, medium resolution, or high-resolution, or may include N levels of resolution (N is an integer greater than 1).

The image sensor100according to the inventive concept may generate the frame information FI in accordance with a standardized channel standard or an arbitrary channel standard determined by a product producer group to distinguish whether the sensed image frame has low-resolution or high-resolution. The generated frame information FI may be transmitted to the image processor250. The channel information including the frame information FI may be included in a header region of a signal transmitted to the image processor250. In addition, the channel information may be transmitted to another IP in the form of embedded data in which the frame information FI is stored in an embedded chip other than the virtual channel.

According to some embodiments, the image sensor100may be formed integrally on the same chip with other digital logic such as the ISP300, a scaler (not shown), the post-processor250, etc. As illustrated inFIG. 1, the image processor250may include the channel allocator200, the controller260, and the ISP300, and the ISP300may further include a first channel310, a second channel320, and a third channel330and the post-processor340. The image processor250may include and/or be a central processing unit (CPU), a microprocessor, or a microcontroller unit (MCU). Post-processing may be performed directly by the image processor250or performed by the ISP300included in the image processor250. Post-processing may be executed through the application of an image enhancement algorithm to image artifacts. Here, post-processing may mean a series of subsequent processing operations to reduce errors and distortions based on the sensed image data. For example, the post-processing may include image processing to change the data format of the image data (for example, change the image data of a Bayer pattern to a YUV or RGB format), image processing for improving image quality such as noise reduction, brightness adjustment, sharpness adjustment, white balancing, denoising, demosaicking, lens shading, gamma correction, etc. on the received image frame.

The channel allocator200may receive image data which is an output signal of the output buffer of the image sensor100, and set a path such that different image frames having different resolutions pass through different channels based on the image data. According to an embodiment of the inventive concept, the channel allocator200may read the frame information FI based on the received signal and determine the resolution of the received current image frame. When the determined image frame has low resolution, the channel allocator200may transmit the image frame to a channel with an optimized environment configuration to process the low-resolution image frame. When the determined image frame has high resolution, the channel allocator200may transmit the image frame to a channel with an optimized environment configuration to process the high-resolution image frame.

The channel allocator200may read channel information and determine which frame has high resolution and which frame has low resolution among the plurality of frames output from the image sensor100. In other words, the channel allocator200may determine from which frame of a series of frames that are continuously processed is the high-resolution image frame, and allocate channels such that high-resolution image frames pass through different channels than the low-resolution frames.

The controller260may generate the mode signal MODE capable of setting the photographing mode of the image sensor100to the high-resolution mode or the low-resolution mode. The controller260may receive the capture command CAPTURE from the user. The controller260may generate the mode signal MODE based on the received capture command CAPTURE, or may generate the mode signal MODE in advance based on information that the capture command CAPTURE will be issued soon. The information that the capture command CAPTURE will be issued soon may be learned in advance or received from a user who has performed an operation such as half-shutter (e.g., half-pressing a shutter actuator). InFIG. 1, the controller260is configured separately from the ISP300and configured inside the image processor250, but is not limited thereto.

Referring toFIG. 1, a first channel310to a third channel330of the ISP300may be configured to process image frames of different resolutions, respectively. That is, the channel allocator200may determine the frame information FI from the received signal, and set the path such that signals corresponding to the high-resolution image frame and the low-resolution image frame pass through different channels. In other words, the channel allocator200may dynamically allocate the high-resolution image frame and the low-resolution image frame to different channels according to resolution. The specific configuration of the channel allocator200will be described with respect toFIG. 2.

The ISP300may process/treat an image such that a person may see the image well, and output the processed/treated image to a memory subsystem400or directly to a display unit500. Alternatively, the ISP300may receive a control signal from an external host through a PC interface (I/F), and provide the processed/treated image to the external host.

The ISP300according to the inventive concept may receive image frames through the first channel310to the third channel330. The first channel310to the third channel330may be set to an environment optimized for processing the image frames according to resolution. For example, the first channel310may have an environment configuration optimized for processing the low-resolution image frame, the third channel330may have an environment configuration optimized for processing the high-resolution image frame, and the second channel320may have an environment configuration optimized for processing the medium resolution image frame having a higher resolution than the low-resolution image frame and a lower resolution than the high-resolution image frame. Three channels, that is, the first channel310to the third channel330, are shown inFIG. 1, but are not limited thereto, and there may be a plurality of channels. In other words, by the need to make the processing speed or power consumption different according to the resolution of the image data, in response to a plurality of resolution levels sensed by the image sensor100, the ISP300may have a plurality of channels each having a preset environment configuration. Also, by the need to make the processing algorithms, speed, or power consumption different due to other reasons than the resolution of the image data, in correspondence to a plurality of operation modes sensed by the image sensor100, the ISP300may have a plurality of channels each having a preset environment configurations.

The ISP300according to the inventive concept may be a multi-image signal processor capable of processing multiple sensor outputs. The multi-image signal processor may include a plurality of channels for simultaneously processing image data output from multiple sensors, or may include a pipeline designed internal processor capable of sequentially processing multiple input signals to efficiently process a plurality of pieces of image data.

The ISP300may include the first to third channels310,320, and330and the post-processor340to configure hardware of the device10. In addition, the ISP300processes a plurality of channels by using a single post-processor, for example, the post-processor340, and thereby does not require the use of a plurality of processors. Because the ISP300has the effect of performing the functionality of a plurality of processors (i.e., cores) with only one processor chip (e.g., a core), the space occupied by hardware may be reduced, and additionally, the cost of mounting the processor chip may also be saved. InFIG. 1, the ISP300is configured separately from the image sensor100, but is not limited thereto.

In an embodiment, the ISP300and the channel allocator200may be located inside the image sensor100and the ISP300may output the processed/treated image to a memory subsystem400or directly to a display unit500. In addition, in an embodiment in which the ISP300and channel allocator200are located inside the image sensor, the image processor250may alternatively perform, using one or more processors, post-processing other than the post-processing performed by the ISP300.

The device10according to an embodiment of the inventive concept may further include the memory subsystem400, the display unit500, and a capture unit600. The memory subsystem400may include a first memory410and a second memory420.

The memory subsystem400may store the post-processed image data received from the ISP300and provide the stored data to other components of the device10. In addition, the memory subsystem400may store various system or user data necessary for operating the device10. For example, the memory subsystem400may include a nonvolatile memory that stores various types of information in a nonvolatile way, and a volatile memory that loads information such as firmware in connection with the operation of the device10.

The memory subsystem400according to an embodiment of the inventive concept may temporarily store the image data corresponding to the low-resolution image frame among the image data received from the ISP300or the low-resolution image data generated based on the high-resolution image frame in a first memory410. The device10may load all the low-resolution image frames temporarily stored in the first memory410to output the low-resolution image frames to the display unit500, thereby seamlessly displaying a captured image. The memory subsystem400may temporarily store image data corresponding to a high-resolution image frame among image data received from the ISP300in the second memory420. The device10may load a high-resolution image frame corresponding to a capture command CAPTURE time from the second memory420according to the user's capture command CAPTURE or a processing command of the image processor250to output the high-resolution image frame to the capture unit600, thereby controlling the capture unit600to generate a high-resolution image.

Because the first memory410stores the low-resolution image frame, the storage capacity of the first memory410may be relatively less than that of the second memory420that stores the high-resolution image frame. However, the first memory410may have a relatively larger storage capacity than the second memory420according to the number of temporarily stored image frames. Alternatively, while the first memory410and the second memory420may have the same storage capacity, the amount of data stored under the control of the device10may be dynamically adjusted.

The memory subsystem400may be implemented as volatile memory or nonvolatile memory. The volatile memory may include Dynamic Random Access Memory DRAM, Static RAM (SRAM), etc. and the nonvolatile memory may include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable and Programmable ROM (EEPROM( ) Flash memory, Phase-change RAM (PRAM), Magnetic RAM (MRAM), Resistive RAM (RRAM), Ferroelectric RAM (FRAM), etc.

The display unit500according to an embodiment of the inventive concept may continuously load the image frame temporarily stored in the memory subsystem400to provide a preview screen for the user to capture an object. For example, the display unit500may load the low-resolution image data from the first memory410in which the post-processed low-resolution image frame is stored, and seamlessly provide the preview screen to the user. The display unit500is not limited to providing the preview screen, and may display the high-resolution image generated by the capture unit600to the user.

The display unit500may include any device capable of outputting an image. For example, the display unit500may include a computer, a mobile phone, and other image output terminals. The display unit500may be an example of an output device. Other examples of the output device include a graphics/display device, a computer screen, an alarm system, a computer aided design/computer aided machining (CAD/CAM) system, a video game station, a smart phone display screen, or any other type of data output device.

The capture unit600may load the high-resolution image data from the second memory420in which the post-processed high-resolution image frame is stored to generate a high-resolution image.

The device10for processing the high-resolution image according to the inventive concept may reduce or eliminate auto-focus delay, auto-exposure delay, and auto-white balance delay that occur when switching the photographing mode from a low-resolution mode to a high-resolution mode. However, the auto focus delay, the auto exposure delay, and the auto white balance delay are only some examples of delays that may occur in the device10for processing the high-resolution image, and there may be various delay factors in addition to the described delays. In addition, the device10for processing the high-resolution image according to the inventive concept may process a frame without any delay even without a frame drop, thereby improving the processing speed of the image frame. As used herein a “frame drop” refers to the dropping of a frame that has been processed while switching the photographing mode. In addition, the device10for processing the high-resolution image according to the inventive concept may not need to store all signals output by an image sensor in a memory, thereby reducing power consumption.

FIG. 2is a circuit diagram illustrating the channel allocator200according to embodiments of the inventive concept. Referring toFIG. 2, the channel allocator200may include a selector210and a plurality of multiplexers (hereinafter referred to as MUXs)220,230, and240. The channel allocator200may determine the resolution of an image frame based on a signal received from the image sensor100and dynamically allocate a path through which the signal is processed according to the resolution. According to an embodiment of the inventive concept, the channel allocator200may select any one of a plurality of image sensors and dynamically allocate image frames input from the selected one image sensor sequentially.

In addition to the image sensor shown inFIG. 1, the device10may further include a plurality of image sensors. For example, the device10may further include a plurality of image sensors to capture images of various angles and image quality, such as a front camera image sensor, a rear camera wide-angle image sensor, a rear camera proximity image sensor, etc.

The selector210may select an image sensor that captures a current object from among the plurality of image sensors. The selector210may receive image data sensed from all of the plurality of image sensors. The selector210may generate a selection signal SEL for selecting the image sensor that captures the current object from among the plurality of image sensors, and may simultaneously apply the selection signal SEL to all of the plurality of MUXs220,230, and240. For example, the channel allocator200may receive image data sensed by an image sensor1, an image sensor2, and an image sensor3, and at this time, the selector210may generate the selection signal SEL for selecting the image sensor (e.g., the image sensor1) that needs to be currently processed and apply the selection signal SEL to the plurality of MUXs220,230, and240. InFIG. 2, the selector210is included in the channel allocator200, but is not limited thereto, and may be included in the ISP300or may be included in the image processor250ofFIG. 1. InFIG. 2, three image sensors are illustrated for convenience of description, but the inventive concept is not limited thereto, and there may be a plurality of image sensors.

The MUXs220,230, and240may be logic circuits that select one of several input signals and transfer the selected input signal to one line.

Each of the plurality of MUXs220,230, and240may output image data of the selected image sensor to the ISP300based on the selection signal SEL applied from the selector210. As described above, the ISP300is a multi-image signal processor and may be an IP capable of efficiently processing image data output from multiple image sensors. Each of the plurality of MUXs220,230, and240may simultaneously apply the image data with respect to the selected image sensor (e.g., the image sensor1) to a plurality of channels of the ISP300. Therefore, the plurality of channels provided in the ISP300may be originally configured to process image data from multiple image sensors, but instead may process an image frame having a different resolution of one image sensor (i.e., the image sensor1) in the same way as if processing multiple image sensors. That is, referring toFIGS. 1 and 2together, the low-resolution image frame, the medium-resolution image frame, and the high-resolution image frame are respectively applied to the first channel310, the second channel320and the third channel330of the ISP300, which may have a similar effect as applying image data of the image sensor1, image sensor2and the image sensor3to ISP300. InFIG. 2, for convenience of description, three MUXs are shown, but the inventive concept is not limited thereto, and the number of MUXs may be increased or decreased in various ways according to the level of resolution required to be processed by the device10or according to the number of channels that the ISP300may simultaneously process.

FIG. 3is a timing diagram of signals processed by the image sensor100and the ISP300based on channels according to embodiments of the inventive concept. Specifically, image frames sensed by the image sensor100are divided according to resolutions, and a virtual channel1and a virtual channel2in which a low-resolution image frame is processed are illustrated. Additionally, the signal processing timing of the first channel310and the second channel320that process the image frames of the image sensor100for each resolution is also illustrated. Here, the horizontal axis indicates time, and the rising/falling of an edge may be configured in units of frames.

Referring toFIG. 3, the image sensor100may sense a low-resolution image Low frame and a high-resolution image High frame. For example, the image sensor100may sense first two frames as the low-resolution image Low and generate the frame information FI corresponding to the low-resolution image Low. In the device10, the low-resolution image Low may have been previously determined to be allocated to the first channel310with an environment configuration to process the low-resolution image Low. The channel allocator200may read the frame information FI from the received signal to determine that the currently received image frame has low-resolution and allocate the image frame to the first channel310of the ISP300. In other words, when the low-resolution image Low is sensed by the image sensor100, the low-resolution image Low may be transmitted to the first channel310of the ISP300, and the ISP300may efficiently post-process low-resolution image data using the first channel310with the environment configuration to process the low-resolution image Low.

The image sensor100may sense a third frame as the high-resolution image High. In the device10, the high-resolution image High may have been previously determined to be allocated to the second channel320with an environment configuration to process the high-resolution image High. The channel allocator200may read the frame information FI from the received signal to determine that the currently received image frame has high-resolution, and allocate the image frame to the second channel320of the ISP300. In other words, when the high-resolution image High is sensed by the image sensor100, the high-resolution image High is transmitted to the second channel320of the ISP300, and the ISP300may efficiently post-process high-resolution image data using the second channel320with the environment configuration to process the high-resolution image High.

The image sensor100may subsequently sense the fourth and fifth frames as the low-resolution image Low. A process of processing a low-resolution image frame is the same as described above, and thus, further detailed description of the process is omitted.

The virtual channel described with reference toFIG. 3may mean a virtual channel according to the MIPI standard.

FIG. 4is a timing diagram of signals processed by the image sensor100and the ISP300based on channels according to an embodiment of the inventive concept. Specifically, image frames sensed by the image sensor100are divided according to resolutions, the signal processing timing of the first channel310and the second channel320that process the image frames of the image sensor100for each resolution is illustrated, the signal processing timing of the display unit500that continuously displays the low-resolution image frame is illustrated, and the signal processing timing of the capture unit600that generates the high-resolution image frame in response to the capture command CAPTURE is illustrated.FIGS. 1 and 4are referred to together.

Initially, the image sensor100may set a photographing mode of the device10to a low-resolution mode P in which a low-resolution image is sensed. At time t1, the image sensor100may sense a first image frame of a captured object as the low-resolution image. Because the speed at which sensed data is transmitted is limited, after a predetermined time has elapsed from time t1, the sensed low-resolution image may be post-processed in the first channel310.

At time t2, the device10may receive the capture command CAPTURE. It may take a predetermined time for the image sensor100to switch the photographing mode. As illustrated inFIG. 4, the image sensor100does not complete the switch of the photographing mode until time t4.

At time t3, because the photographing mode is not completely switched, the image sensor100may still sense the second frame as the low-resolution image. After the predetermined time required for the low-resolution image to be post-processed in the first channel310, the display unit500may display a preview screen with respect to the first frame.

At time t4, in response to the CAPTURE command at time t2, the photographing mode of the image sensor100may be changed from the low-resolution mode P to a high-resolution mode C, and the image sensor100may sense a third image frame as the high-resolution image. Similar to the case in which sensed low-resolution image frame is transmitted to the ISP300, the speed at which the sensed high-resolution image frame is transmitted to the ISP300is also limited. Therefore, after a predetermined time has elapsed at time t4, the sensed high-resolution image may be post-processed by the post-processor340through the second channel320.

The display unit500may continuously provide preview screens with respect to image frames at a constant time interval (e.g., 1/60 second and a display speed of 60 fps (frames per second)). For example, the display unit500may start to provide the preview screen by displaying the first frame Frame1input through the first channel310and processed by the post-processor340between the time t3and time t4. After a certain period of time has elapsed from displaying the first frame Frame1, the display unit500may provide the preview screen with respect to the second frame Frame2following the first frame Frame1.

Moreover, given an illustrative constant time interval of 1/60 second, and the time needed to sense high-resolution Frame3, transmit high-resolution image Frame3to the post-processor340, and post-process high-resolution Frame3, the display unit500may display the already displayed second frame Frame2again when the third frame Frame3needs to be displayed but has not been completely sensed or has not been completely post-processed by the ISP300. The display unit500may display the third frame Frame3in response to the completion of processing of the third frame Frame3by the ISP300. At time t5, the photographing mode may be changed again to the low-resolution mode P after sensing of the third image frame is completed. That is, the image sensor100may sense only one frame (the third frame) as the high-resolution image in response to the capture command CAPTURE. After the predetermined time (from t4to t5, or (t5-t4)) for the image sensor100to change the photographing mode, the image sensor100may sense the fourth frame Frame4again as the low-resolution image, and the ISP300may process the fourth frame Frame4input to the first channel310.

At time t6, the capture unit600may load the image data post-processed by the ISP300and stored in the second memory420, thereby generating a high-resolution image with respect to the third frame Frame3.

At time t7, the image sensor100may sense the low-resolution image with respect to a fifth frame, and the ISP300may post-process the image data received by the first channel310at time t7after a predetermined time has elapsed. In an embodiment, the display unit500may display the fourth frame Frame4regardless of whether the image sensor100or the ISP300is processing the fifth frame Frame5. In other words, the display unit500may provide the preview screens from the first frame Frame1to the fourth frame Frame4consecutively from time t3, and may twice display the second frame Frame2in which the next frame (e.g., the third frame Frame3) among the frames has not been processed. According to an embodiment, the display unit500may display the preview screen, but is not limited thereto, and may display the high-resolution image generated by the capture unit600for a predetermined time instead of preview.

The frame rate (or scan rate) of the display unit500may be the same as or faster than the image frame sensing rate of the image sensor100. In an embodiment, the image frame sensing rate of the image sensor100may be 24 frames per second (fps), and the frame rate (or scan rate) of the display unit500may be 60 fps, but is not limited thereto.

In the device10according to the inventive concept, the number of image frames sensed by the image sensor100as high-resolution images is limited to only as many as is needed in response to the user's capture command CAPTURE. Other image frames are sensed by the image sensor100as low-resolution images. That is, the device10may set the photographing mode to the low-resolution mode P, and then switch the low-resolution mode P to the high-resolution mode C only when there is the capture command CAPTURE, and immediately after sensing the high-resolution image, change back the high-resolution mode C to the low-resolution mode P, thereby reducing power consumption required to sense an object as the high-resolution image.

FIG. 5is a timing diagram of signals processed by the image sensor100and the ISP300based on channels according to another embodiment of the inventive concept. The horizontal and vertical axes and the signals shown inFIG. 5are similar to those inFIG. 4. It is similar toFIG. 4that the frame rate (or scan rate) of the display unit500may be the same as or faster than the image frame sensing rate of the image sensor100.

Initially, the image sensor100may set a photographing mode to the low-resolution mode P for sensing a low-resolution image if no command is received from the device10. At the time t1, the image sensor100may sense a first image frame of a captured object as the low-resolution image. After a predetermined time has elapsed from the time t1, the sensed low-resolution image may be transmitted through the first channel310and post-processed.

At the time t2, the image processor250issues a high-resolution switching command (P→C CMD), and the issued high-resolution switching command (P→C CMD) is input to the image sensor100. It may take a predetermined time to change the photographing mode of the image sensor100. As illustrated inFIG. 5, the image sensor100does not complete the switch of the photographing mode until time t4.

At the time t3, the image sensor100has not completely changed the photographing mode, and thus, the image sensor100senses a low-resolution image of a second frame. After a predetermined time has elapsed from the time t3, the display unit500may load image data processed by the ISP300and stored in the first memory410and provide a preview screen with respect to the first frame Frame1to a user.

At the time t4, in response to the issued high-resolution switching command (P→C CMD), the photographing mode of the image sensor100may be changed to the high-resolution mode C. The image sensor100may sense a third frame as a high-resolution image, and image data received through the second channel320may be post-processed by the post-processor340. While the image data of the third frame Frame3is post-processed, the display unit500may seamlessly provide a preview screen with respect to the second frame Frame2. Because the display unit500displays an image at a constant time interval (e.g., 1/60 second), the display unit500may display the already displayed second frame Frame2again when the third frame Frame3needs to be displayed but has not been completely sensed or has not been completely post-processed by the ISP300.

At the time t5, the user's capture command CAPTURE may be received. Because the image sensor100has already changed the photographing mode to the high-resolution mode C in response to the high-resolution switching command (P→C CMD) of the image processor250, a separate photographing mode change time is not necessary.

At the time t6, the capture unit600may immediately generate a high-resolution image of the third frame Frame3. Because the time t5when the capture command CAPTURE is received is during the sensing of the third frame Frame3or the time when the user is provided with a preview screen with respect to the third frame Frame3through the display unit500, image data of the third frame Frame3may be captured.

At the time t7, the image processor250issues a low-resolution conversion command (C→P CMD), and the issued high-resolution conversion command (P→C CMD) is input to the image sensor100. It may take a predetermined time to change the photographing mode of the image sensor100as in the case of high-resolution switching. According to an embodiment, the display unit500may display the already displayed third frame Frame3again when the fourth frame Frame4needs to be displayed but has not been completely sensed or has not been completely post-processed by the ISP300.

At the time t8, the image sensor100may sense a high-resolution image of a fifth frame Frame5. The capture unit600may generate a high-resolution image of the fourth frame Frame4that has already been post-processed by the ISP300.

At the time t9, the photographing mode of the image sensor100may be changed back to the low-resolution mode P. The image sensor100may sense a sixth frame Frame6as a low-resolution image. The display unit500may display the fourth frame Frame4because the post-processing of the fifth frame Frame5has not been completed.

Continuously from the time t3, the display unit500may load the image frames post-processed by the ISP300through the first channel310or the second channel320from the memory subsystem400to seamlessly provide the preview screen to the user. According to an embodiment, the display unit500may provide an image frame post-processed through at least one of the first channel310and the second channel320of the ISP300to the user. According to an embodiment, the display unit500may display the preview screen, but is not limited thereto, and may display a high-resolution image generated by the capture unit600for a predetermined time instead of preview.

The device10according to the inventive concept may change the photographing mode in advance by the image processor250before the capture command CAPTURE. For example, the image processor250may change the photographing mode in advance, and if there is the user's capture command CAPTURE, immediately may refer to the photographing mode for taking a picture as a capture preparation mode. In the capture preparation mode, auto-focusing may be fixed in order to quickly process an object in high-resolution. Accordingly, the device10may reduce or eliminate the delay time (i.e., shutter-leg time) between the time of the shutter operation and the time when the high-resolution image is completely processed when capturing. The virtual channel described with respect toFIG. 5may mean a virtual channel according to the MIPI standard.

FIG. 6is a flowchart of an image processing method according to embodiments of the inventive concept. Referring toFIGS. 1 and 6, the image sensor100of the device10may receive an image of a captured subject (or object) for each frame, and generate a first signal (e.g., a binary signal containing low-resolution image frame) with respect to a low-resolution image frame and a second signal (e.g., a binary signal containing high-resolution image frame) with respect to a high-resolution image frame (S110).

The image sensor100may generate the frame information FI related to the resolution of the image in the form of a virtual channel ID or in the form of embedded data based on a previously determined channel standard (e.g., the MIPI standard) (S120). The frame information FI may be generated according to the resolution of the image, but is not limited thereto. A signal carrying the generated frame information FI is output to the channel allocator200. The frame information FI may be carried as header information of the signal.

The channel allocator200may read the frame information FI of the received signal, determine the resolution of the received image frame, and set a path such that the first and second signals pass through different paths based on the determined resolution (S130).

The channel allocator200may output the first signal corresponding to the low-resolution image frame to the first channel310with an environment configuration to efficiently process the low-resolution image, and output the second signal corresponding to the high-resolution image frame to the second channel320with an environment configuration to efficiently process the high-resolution image. The ISP300may post-process the received first signal to generate a third signal corresponding to the low-resolution image, and post-process the received second signal to generate a fourth signal corresponding to the high-resolution image (S140).

The memory subsystem400may temporarily store the third signal, and the display unit500may display the captured object for a user to preview the captured object (S150). In addition, the memory subsystem400may temporarily store the fourth signal, and the capture unit600may generate a high-resolution image according to a user's capture command, and/or generate a low-resolution image corresponding to the fourth signal to display the captured object for the user to preview the captured object (S160). That is, the high-resolution image frame may be post-processed and simultaneously the low-resolution image of the corresponding frame may be generated and displayed on a preview screen using a scaler, but the inventive concept is not limited thereto. Here, the meaning that the third or fourth signal is stored may be understood to mean that binary data in which the third or fourth signal is post-processed is stored.

FIGS. 7, 8, and 9are more detailed flowcharts of an image processing method according to an embodiment of the inventive concept.

FIG. 7is a more detailed flowchart of operation S110ofFIG. 6according to an embodiment of the inventive concept. Referring toFIGS. 1, 4, 6, and 7together, at a first time (t2inFIG. 4), the image processor250may issue a command for changing a photographing mode from a low-resolution mode to a high-resolution mode based on the user's capture command (“CAPTURE” inFIG. 4) (S111_1). The image sensor100may sense a low-resolution image to generate the first signal (e.g., a low-resolution image frame binary signal) at a second time (t3inFIG. 4) at which the low-resolution mode has not been changed to the high-resolution mode (S112_1). Thereafter, the image sensor100may change the photographing mode to the high-resolution mode at a third time (t4inFIG. 4) and generate the second signal (e.g., a high-resolution image frame binary signal) corresponding to the high-resolution image (S113_1).

FIG. 8is a more detailed flowchart illustrating operation S130ofFIG. 6according to an embodiment of the inventive concept. Referring toFIGS. 1, 4, 6, and 8together, after operation S120, at the second time (t3inFIG. 4), the channel allocator200may set (i.e., allocate) a path such that the generated first signal (e.g., a low-resolution image frame binary signal) is output to the first channel310of the ISP300(S131_1). Then, at the third time (t4inFIG. 4), the channel allocator200may set (i.e., allocate) a path such that the generated second signal (e.g., a high-resolution image frame binary signal) is output to the second channel320of the ISP300(S132_1).

FIG. 9is a flowchart showing an operation further added to the operations ofFIG. 6according to an embodiment of the inventive concept. Referring toFIGS. 1, 4, 6 and 9together, after operation S160, the image sensor100may change a photographing mode again to the low-resolution mode (P ofFIG. 4) at the fourth time (t5ofFIG. 4) immediately after sensing the high-resolution image (S170_1).

FIGS. 7 to 9are similar toFIG. 4according to an embodiment of the inventive concept, and thus redundant descriptions are omitted.

FIGS. 10, 11, 12, and 13are more detailed flowcharts of an image processing method according to another embodiment of the inventive concept.FIG. 10is a flowchart illustrating operation S110ofFIG. 6in more detail according to another embodiment of the inventive concept. Referring toFIGS. 1, 5, 6 and 10together, the image processor250may issue a high-resolution switching command (P→C CMD) that changes a photographing mode from a low-resolution mode to a high-resolution mode at a first time (t2inFIG. 5), and the image sensor100may receive the high-resolution switching command (P→C CMD) (S111_2). The image sensor100may sense a low-resolution image to generate a first signal at a second time (t3inFIG. 5) before switching to the high-resolution mode (C inFIG. 5) (S112_2). Thereafter, the image sensor100is changed to the high-resolution mode at the third time (t4inFIG. 5), and accordingly, sense a high-resolution image to generate a second signal (S113_2).

FIG. 11is a more detailed flowchart of operation S130ofFIG. 6according to another embodiment of the inventive concept. Referring toFIGS. 1, 5, 6 and 11together, after operation S120, at the second time (t3inFIG. 5), the channel allocator200may set (i.e., allocate) a path such that the generated first signal (e.g., a low-resolution image frame binary signal) is output to the first channel310of the ISP300(S131_2). In addition, at the third time (t4inFIG. 5), the channel allocator200may set (i.e., allocate) a path such that the generated second signal (e.g., a high-resolution image frame binary signal) is output to the second channel320of the ISP300(S132_2).

FIG. 12is a more detailed flowchart of operation S160ofFIG. 6according to another embodiment of the inventive concept. Referring toFIGS. 1, 5, 6 and 12together, after operation S150, the image sensor100may receive the capture command (“CAPTURE” inFIG. 5) at a fourth time (t5inFIG. 5) (S161_2). The capture unit600may generate a high-resolution image by loading a fourth signal obtained by post-processing the second signal corresponding to the high-resolution image from the second memory420(S162_2). However, the inventive concept is not limited thereto, and the capture unit600may simultaneously generate a low-resolution image corresponding to the fourth signal and a high-resolution image and load the low-resolution image and the high-resolution image into the first memory410to display the low-resolution image and the high-resolution image on the display unit500. The image sensor100may continuously sense the high-resolution image until the photographing mode is changed to the low-resolution mode, and the capture unit600may generate the high-resolution image (S163_2).

FIG. 13is a flowchart showing an operation added to the operations ofFIG. 6according to another embodiment of the inventive concept. Referring toFIGS. 1, 5, 6 and 13together, after operation S160, at a fifth time (t7inFIG. 5), the image processor250may issue a command to change the photographing mode from the high-resolution mode (C inFIG. 5) to the low-resolution mode (P inFIG. 5) again (S171_2). The image sensor100may switch the photographing mode back to the low-resolution mode P at a sixth time (t9inFIG. 5) after a predetermined time has elapsed (S172_2).

FIGS. 10 to 13are similar toFIG. 4according to an embodiment of the inventive concept, and thus redundant descriptions are omitted.

FIG. 14is a block diagram illustrating a device10′ according to embodiments of the inventive concept.FIG. 1is also referred to.

Referring toFIG. 14, the device10′ according to an embodiment of the inventive concept may include an image sensor100′, a channel allocator200′, a controller260′, an image processor250′ including an ISP300′, a memory subsystem400′, a display unit500′, a capture unit600′, and a video processing unit700′. The image processor250′ may include the channel allocator200′, the controller260′, and the ISP300′, and the ISP300′ may include a first channel310′, a second channel320′, a third channel330′ and a post-processor340′. Each functional unit ofFIG. 14is similar to each functional unit already described with respect toFIG. 1, and thus redundant descriptions are omitted.

In addition to a first memory410′ and a second memory420′, a memory subsystem400′ may include a third memory430′ for other purposes, such as video recording, and storing both image data corresponding to a low-resolution image frame and image data corresponding to a high-resolution frame among image data received from the ISP300′.

The video processing unit700′ may be included in the device10′ separately from the capture unit600′ to efficiently process a video among images captured by the device10′, and in this case, the capture unit600′ may process a still picture. The video processing unit700′ may load a third signal corresponding to the low-resolution image or a fourth signal corresponding to the high-resolution image to generate a compressed video stream.

Because the still picture and the video have different image features, throughput to be calculated, and compression methods, the still picture and the video may be efficiently processed to suit each characteristic through different IPs.

According to an embodiment of the inventive concept, the capture unit600′ may include a codec and a hardware accelerator for processing the still picture, and the video processing unit700′ may include a codec and a hardware accelerator for processing the video.

The still picture processed by the capture unit600′ may be stored in the first memory410′ and/or the second memory420′, and the video processed by the video processing unit700′ may be stored in the first memory410′ and/or the second memory420′, or may be stored in a separate memory, that is, the third memory430′. Because the resolution of the video and the size of the video are different from the resolution of the still picture and the size of the still picture, the still picture and the video may be efficiently stored and processed in a separate memory. However, the inventive concept is not limited thereto, and the video may be stored using the first memory410′ and the second memory420′.

FIG. 15is a block diagram illustrating a system1including a device according to embodiments of the inventive concept. Referring toFIG. 15, the system1according to an embodiment of the inventive concept may include an image sensor1000, a channel allocator1500, an ISP2000, a display device3000, an application processor (AP)4000, a working memory5000, a storage6000, a user interface7000, and a wireless transceiver8000, and the ISP2000may be implemented as an integrated circuit separate from the AP4000. The image sensor1000ofFIG. 15may be similar to the image sensor100ofFIG. 1, and the channel allocator1500ofFIG. 15may be similar to the channel allocator200ofFIG. 1. The ISP2000ofFIG. 15may be similar to the ISP300ofFIG. 1, the working memory5000ofFIG. 15may be similar to the memory subsystem400ofFIG. 1, and the display device3000ofFIG. 15may be similar to the display unit500ofFIG. 1. In addition, the AP4000ofFIG. 15may include the image processor250ofFIG. 1.

The image sensor1000may generate image data based on a received optical signal and provide binary data to the ISP2000, and may perform image processing similarly to the ISP2000if necessary. The AP4000may control overall operation of the system1and may be provided as a system-on-chip (SoC) that drives an application program, an operating system, etc. The AP4000may control the operation of the ISP2000and may provide the converted image data generated by the ISP2000to the display device3000or store the image data in the storage6000.

The working memory5000may store programs and/or data that the AP4000processes or executes. The storage6000may be implemented as a non-volatile memory device such as a NAND flash or a resistive memory, and, for example, the storage6000may be provided as a memory card (e.g., an MMC memory card, an eMMC memory card, an SD memory card, or a micro SD memory card). The storage6000may store data and/or program with respect to an execution algorithm that controls the image processing operation of the ISP2000, and when the image processing operation is performed, the data and/or program may be loaded into the working memory5000.

The user interface7000may be implemented as various devices capable of receiving user input, such as a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, a microphone, etc. The user interface7000may receive a user input and provide a signal corresponding to the received user input to the AP4000. The wireless transceiver8000may include a modem8100, a transceiver8200, and an antenna8300. For convenience of explanation, the ISP2000is included in the AP4000, but is not limited thereto, and the ISP2000may be mounted as an IP separately from the AP4000.