Image signal processor, operating method thereof, and image processing system including the image signal processor

An image signal processor, an operating method thereof, and an image processing system are provided. The image processing system includes: a control processor configured to generate and output setting information corresponding to N (where N is an integer of 2 or more) image frames; and an image signal processor configured to perform image processing on the N image frames received from an image sensor based on the setting information, and generate an interrupt signal and transmit the interrupt signal to the control processor based on completion of the image processing performed on the N image frame.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No. 10-2019-0068267, filed on Jun. 10, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

The inventive concepts relate to image signal processing, and more particularly, to image signal processors for performing image processing on image data received from an image sensor, operating methods of the image signal processor, and/or image processing systems including the image signal processor.

An image signal processor included in an imaging device such as a camera or a smartphone may perform image processing such as converting a data format of image data provided from an image sensor into a data format, such as RGB or YUV, or removing noise from the image data and adjusting brightness. The image signal processor may process, by units of frames, the image data output from the image sensor. As imaging devices have supported operation modes of providing an image at a high frame rate such as a slow motion mode or a super slow motion mode, the image sensor may generate and output image data at a high frame rate. Therefore, an image signal processor for normally performing image processing on image data having a high frame rate may be beneficial.

SUMMARY

The inventive concepts provide image signal processors capable of fast readout for normally performing image processing on image data received at a high frame rate, operating methods of the image signal processor, and an image processing system including the image signal processor.

According to aspects of the inventive concepts, there is provided an image processing system including a control processor configured to generate and output setting information corresponding to N (where N is an integer equal to or more than two) image frames and an image signal processor configured to perform image processing on the N image frames received from an image sensor based on the setting information, and generate an interrupt signal and transmit the interrupt signal to the control processor based on completion of the image processing performed on the N image frame.

According to other aspects of the inventive concepts, there is provided an image signal processor including an image processing engine configured to perform image processing on image frames sequentially received from an image sensor, a direct memory access (DMA) controller configured to store processing data, generated by the image processing engine, in a memory, and a fast readout circuit configured to receive setting information, including N (where N is an integer equal to or more than two) setting values corresponding to N image frames, from a control processor and, provide a setting value of an image frame, on which image processing is performed, to the image processing engine or the DMA controller, based on the image processing being sequentially performed on the N image frames.

According to other aspects of the inventive concepts, there is provided an operating method of an image signal processor, the operating method including receiving N (where N is an integer equal to or more than two) setting values from a control processor, storing the N setting values in a storage area, receiving image frames from an image sensor, sequentially performing image processing on N image frames of the image frames based on the N setting values, and generating an end interrupt signal based on completion of the image processing performed on the N image frames.

According to other aspects of the inventive concepts, there is provided an application processor including a main processor configured to generate and output setting information including N (where N is an integer equal to or more than two) setting values and an image signal processor configured to receive and store the setting information and sequentially perform image processing on N image frames received from an image sensor based on the setting information, and generate an end interrupt signal and transmit the end interrupt signal to an image signal processor based on completion of image processing performed on the N image frames.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG.1is a block diagram illustrating an image processing device1000according to an example embodiment.

The image processing device1000may be implemented as an electronic device which captures an image and displays the captured image or performs an operation based on the captured image. The image processing device1000may be implemented as, for example, a personal computer (PC), an Internet of things (IoT) device, and/or a portable electronic device. Examples of the portable electronic device may include a laptop computer, a mobile phone, a smartphone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, wearable device, etc. Also, the image processing device1000may be equipped in an electronic device, such as a drone or an advanced drivers assistance system (ADAS), and/or an electronic device provided as a componentry in vehicles, furniture, manufacturing facilities, and various measuring machines.

Referring toFIG.1, the image processing device1000may include an image sensor1100and an image processing system1200. The image processing device1000may further include other elements such as a display and a user interface. The image processing system1200may include an image signal processor100, a control processor200, and a memory300. The image signal processor100, the control processor200, and the memory300may be implemented as a single semiconductor chip or a plurality of semiconductor chips. For example, the image signal processor100and the control processor200may be integrated into one semiconductor chip.

As disclosed herein, the term “memory”, “storage medium”, “computer readable storage medium” or “non-transitory computer readable storage medium,” may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other tangible machine readable mediums for storing information. The term “computer-readable medium” may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and/or data.

The image sensor1100may convert an optical signal, which is input through an optical lens LS and corresponds to an object, into an electrical signal and may generate and output the image data IDT on the basis of electrical signals. The image sensor1100may include, for example, a pixel array including a plurality of pixels arranged two-dimensionally and a readout circuit, and the pixel array may convert received optical signals into the electrical signals. The pixel array may be implemented with, for example, an optical-to-electric conversion device such as a charge coupled device CCD or a complementary metal oxide semiconductor (CMOS), and in addition, may be implemented with other various kinds of optical-to-electric conversion devices. The readout circuit may generate raw data on the basis of an electrical signal provided from the pixel array and may output, as image data IDT, the generated raw data or raw data on which preprocessing such as removing bad pixel has been performed. The image sensor1100may be implemented as a semiconductor chip or package including the pixel array and the readout circuit.

The image signal processor100may perform image processing on the image data IDT provided from the image sensor1100. For example, the image signal processor100may perform image processing, such as image processing of converting a data format of the image data IDT (for example, converting an image data having a Bayer pattern into a YUV or RGB format), removing noise, adjusting brightness, and adjusting sharpness, for enhancing image quality. The image signal processor100may configure hardware of the image processing system1200.

The image signal processor100may include an image signal processing core110(hereinafter referred to as an ISP core) and a fast readout circuit120(hereinafter referred to as an FRO circuit). The ISP core110may perform, by units of frames, image processing on the image data IDT output from the image sensor1100. The ISP core110may be referred to as an image processing engine. Processing data PDT (for example, an image-processed frame (hereinafter referred to as converted image data) and/or result data (statistic data, histogram, etc.) generated through image processing) generated through image processing may be stored in the memory300.

The FRO circuit120may store setting information IF_N including setting values of a plurality of image frames (hereinafter referred to as N (where N is a positive integer equal to or more than two) number of frames) provided from the control processor200, and in a case where image processing is performed on a certain image frame, the FRO circuit120may provide a setting value (a current setting value) of a corresponding image frame (a current image frame). For example, setting information (e.g., setting values of frames) may include register values (for example, register values used by the ISP core110in an image processing process) for adjusting the image quality of an image frame and address register values representing an area, the processing data PDT corresponding to each frame is to be stored, of the memory300. Such setting values may be set (or changed) at every frame.

The FRO circuit120may receive and store the setting information IF_N about the N frames from the control processor200, and in a case where image processing is performed on each of the N frames, the FRO circuit120may provide setting values of a corresponding frame to the ISP core110or another circuit (for example, a direct memory access (DMA) controller130ofFIG.2). Therefore, image processing may be performed on the N frames by units of frames.

When image processing performed on the N frames is completed, the FRO circuit120may generate an interrupt signal INT indicating completion of image processing performed on the N frames, or may issue a request, to the ISP core110, to generate the interrupt signal INT.

For example, the FRO circuit120may store the setting information IF_N about the N frames received from the control processor200though a one-time receiving process. In a case where image processing is performed on each of the N frames, the FRO circuit120may provide setting values of a corresponding frame, and when image processing performed on the N frames is completed, the FRO circuit120may generate the interrupt signal INT.

The control processor200may control the image signal processor100to perform image processing. The control processor200may include processing circuitry such as hardware including logic circuits; a hardware/software combination such as a processor executing software; or a combination thereof, for example to configure software of the image processing system1200. The control processor200may be a central processing unit (CPU), a microprocessor, an ARM processor, an X86 processor, a microprocessor without interlocked pipeline stages (MIPS) processor, a graphics processing unit (GPU), a general-use GPU, and/or another processor configured to execute instructions stored in a memory. The control processor200may process or execute data and an instruction code (or programs) including an execution algorithm of the image signal processor100to generate a control signal CONS for controlling the control processor200. The control signal CONS may include the setting information IF_N about the N frames.

The control processor200may previously generate the setting information IF_N about the N frames, and before image processing starts to be performed on the N frames, the control processor200may transmit the setting information IF_N to the image signal processor100. The control processor200may generate setting information about next N frames while the image signal processor100is performing image processing on the N frames, and before image processing performed on the N frames is completed, namely, before the interrupt signal INT indicating completion of image processing performed on the N frames is received from the image signal processor100, the control processor200may transmit the generated setting information about the next N frames to the image signal processor100. The control processor200may generate and transmit the setting information about the next N frames while the image signal processor100is performing image processing on the N frames.

The memory300may store the processing data PDT received from the image signal processor100and may provide the processing data PDT to the image signal processor100, the control processor200, or the other elements of the image processing device1000.

The memory300may be implemented as a volatile memory or a non-volatile memory. Examples of the volatile memory may include dynamic random access memory (DRAM), static random access memory (SRAM), etc., and examples of the non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc.

FIG.2is a block diagram illustrating an image signal processor100according to an example embodiment.FIG.2illustrated in detail the image signal processor100ofFIG.1.

Referring toFIG.2, the image signal processor100may include an ISP core110, an FRO circuit120, and a DMA controller130. The ISP core110may include a controller111and a plurality of intellectual property (IP) blocks112. InFIG.2, the plurality of IP blocks112are illustrated as including first to third IP blocks11to13(e.g., IP_B1, IP_B2and IP_B3), but are not limited thereto and may include two or more IP blocks.

The controller111may receive a control signal CONS from the control processor200and may control an overall operation of the image signal processor100on the basis of the control signal CONS. The control signal CONS may include setting information IF_N about N frames, and the controller111may provide the FRO circuit120with the setting information IF_N about the N frames. The FRO circuit120may store the received setting information IF_N, and when image processing is performed on the basis of setting values included in the setting information IF_N at every frame, the FRO circuit120may provide the stored setting information IF_N to the plurality of IP blocks112or the DMA controller130.

The plurality of IP blocks112(e.g., the first to third IP blocks11to13) set as an image processing block may perform image processing, and the first to third IP blocks11to13may perform different image processing. In some example embodiments, the first IP block11may convert a data format of image data IDT, the second IP block12may adjust brightness, and the third IP block13may adjust contrast. The first to third IP blocks11to13may sequentially perform image processing on a frame. An image-processing-completed frame and/or result data (for example, converted image data) obtained based on image processing by each of the first to third IP blocks11to13may be stored in the memory300. The plurality of IP blocks112may receive a setting value corresponding to a processed frame from the FRO circuit120and may perform image processing on a corresponding frame on the basis of the setting value.

The DMA controller130may store the processing data PDT, received from at least one of the first to third IP blocks11to13, in the memory300. The processing data PDT may include result data and/or converted image data based on image processing. At this time, the DMA controller130may receive an address ADDR (or an address register value) from the FRO circuit120and may store the processing data PDT in a storage area, corresponding to the address ADDR, of the memory300.

For example, when image processing is performed on a first frame (or when image processing is completed), the FRO circuit120may generate a first address on the basis of an address register value representing an area where converted image data (for example, first converted image data) of the first frame is to be stored in the stored setting information IF_N and may provide the first address (or a first address register value) to the DMA controller130, and the DMA controller130may store the first converted image data in an area corresponding to the first address. Subsequently, when image processing is performed on a second frame, the FRO circuit120may generate a second address on the basis of an address register value representing an area where converted image data (for example, second converted image data) of the second frame is to be stored in the stored setting information IF_N and may provide the second address (or a second address register value) to the DMA controller130, and the DMA controller130may store the second converted image data in an area corresponding to the second address.

When image processing performed on the N frames is completed, the FRO circuit120may generate an interrupt signal INT representing completion of image processing. The FRO circuit120may transmit the interrupt signal INT to the control processor200directly or the controller111. In some example embodiments, the FRO circuit120may issue a request, to the controller111, to generate the interrupt signal, and in response to the request, the controller111may generate and transmit the interrupt signal INT.

FIG.3Ais a timing diagram illustrating a transmitted/received signal and an operation of an image signal processor100according to an example embodiment, andFIG.3Bis a timing diagram illustrating a transmitted/received signal and an operation of an image signal processor according to a comparative example.FIG.3Aillustrates an operation of the image signal processor100ofFIG.1.

Referring toFIG.3A, the image signal processor100may receive image data by units of frames. For example, the image signal processor100may receive first to n+1thframes F1to Fn+1. The image signal processor100may perform image processing on a received frame.

Before image processing is performed, the image signal processor100may receive a control signal for controlling processing of a frame from a control processor (200ofFIG.1). The image signal processor100may receive setting information IF_N1including setting values of N frames (for example, first to Nthframes) F1to Fn. When a frame (for example, the first frame F1) is received after the setting information IF_N1is received, the image signal processor100may start to perform image processing and may transmit a start interrupt signal INT_S1indicating the start of image processing to the control processor200.

The image signal processor100may perform image processing on the N frames F1to Fn. As described above, an FRO circuit (120ofFIG.1) may store the setting information IF_N1, and when image processing starts to be performed on each frame, the FRO circuit120may provide a setting value of a corresponding frame. Moreover, the image signal processor100may receive setting information IF_N2about next N frames (for example, n+1thto 2nthframes) Fn+1 to F2nfrom the control processor200while image processing is being performed on the N frames F1to Fn.

When image processing is performed on the N frame F1to Fn, the image signal processor100may transmit an end interrupt signal INT_E1, indicating completion of image processing performed on the N frames, to the control processor200. Subsequently, when a frame is received again (for example, when the n+1thframe Fn+1 is received), the image signal processor100may start to perform image processing and may transmit a start interrupt signal INT_S2, indicating the start of image processing, to the control processor200.

As described above, the image signal processor100according to some example embodiments may receive the setting information IF_N1and IF_N2by units of N frames and may perform image processing on the N frames on the basis of the received setting information, and when image processing performed on the N frames is completed, the image signal processor100may transmit end interrupt signals INT_E1and INT_E2, indicating completion of image processing, to the control processor200. At this time, the FRO circuit120included in the image signal processor100may store the setting information IF_N1and IF_N2and may provide a corresponding setting value when image processing is performed on each frame. The control processor200may generate and transmit the setting information IF_N2about the next N frames (for example, the n+1thto 2nthframes Fn+1 to F2n) during a first period T1where image processing is performed on the N frames (for example, the first to nthframes F1to Fn). The first period T1may be secured as a setting margin of the control processor200, namely, a time margin for a control setting.

A transmitted/received signal and an operation of the image signal processor according to the comparative example will be described below with reference toFIG.3B. The image signal processor according to the comparative example does not include an FRO circuit.

The image signal processor according to the comparative example may receive a respective one of setting information IF1to IFn+2 from a control processor at every frame and may perform image processing on a corresponding frame on the basis of received setting information, and then, when image processing is completed, the image signal processor may transmit an interrupt signal to the control processor. For example, the image signal processor may receive the setting information IF1including a setting value of a first frame F1from the control processor and may perform image processing on the received first frame F1on the basis of the setting information IF1. When image processing starts to be performed on the first frame F1, the image signal processor may transmit a start interrupt signal INT_S1to the control processor, and when image processing is completed, the image signal processor may transmit an end interrupt signal INT_E1to the control processor.

The control processor may generate and transmit the setting information IF2about a next frame (for example, a second frame F2) when image processing is performed on the first frame F1, namely, until the end interrupt signal INT_E1is received after the start interrupt signal INT_S1is received.

As described above, according to an operation of the image signal processor according to the comparative example, the control processor may generate and transmit setting information about next one frame during a second period T2where image processing is performed on one frame. The second period T2may be secured as a setting margin of the control processor.

When image data is received at a high frame rate (for example, a frame rate equal to or greater than a predetermined, or alternatively, desired, threshold value, for example, 120 fps (frame per second), 240 fps, 960 fps, etc.), an image processing time of one frame may be shortened. In a case where the image signal processor according to the comparative example performs image processing, since the control processor has to generate and transmit setting information about a next frame during an image processing time (e.g., the second period T2) of one frame, a setting margin of the control processor may not be sufficient, and due to this, the image signal processor may not be normally controlled and may abnormally operate.

However, as described above, in a case where the image signal processor100according to some example embodiments performs image processing, the control processor200may secure N number of image processing periods (e.g., the first period T1) as a setting margin, and thus, a setting margin may be sufficiently secured. Accordingly, abnormal control by the control processor200may be prevented, and the image signal processor100may normally perform image processing on image data having a high frame rate.

FIG.4is a block diagram illustrating the FRO circuit120ofFIG.1.

Referring toFIG.4, an FRO circuit120may include a storage area21which stores setting information about N frames and a logic circuit22.

The storage area21may store setting values (for example, first to nthsetting values IF1to IFn) of each of the N frames. For example, the storage area21may include N registers (e.g., first to nthregisters) REG1to REGn, and the first to nthregisters REG1to REGn may respectively store the first to nthsetting values IF1to IFn.

In response to a command CMD for requesting information FIF representing a frame on which image processing is performed or a setting value of a corresponding frame, the logic circuit22may read a setting value IF corresponding to the corresponding frame from the storage area21and may output the setting value IF of the corresponding frame or an address ADDR of the corresponding frame. For example, the logic circuit22may provide an ISP core (110ofFIG.1) with register values for adjusting the image quality of a corresponding frame, or may provide a DMA controller (130ofFIG.2) with an address register value of the corresponding frame or an address ADDR generated based on the address register value.

FIG.5is a diagram exemplarily illustrating setting information received by an image signal processor according to an example embodiment.

As described above, an image signal processor (100ofFIG.1) may receive setting information IF_N about N frames from the control processor200. The setting information IF_N about the N frames may include a size211of received data, address register values212of the N frames (e.g., ADD_F1, ADD_F2, . . . , ADD_Fn), and frame information213about at which frame interrupt occurs (e.g., FINT). The image signal processor100may generate an interrupt signal when image processing performed on a corresponding frame is completed, based on the frame information213about where an interrupt occurs.

In some example embodiments, the setting information IF_N may be received in a packet data format and may further include a header bit representing the start of packet data and tail bits representing an end of the packet data.

FIG.6is a flowchart illustrating an operating method of an image processing system according to an example embodiment.FIG.6illustrates an operating method of each of the image signal processor100and the control processor200ofFIG.1.

Referring toFIG.6, the control processor200may generate setting information IF_N1about N frames in operation S110and may transmit the setting information IF_N1to the image signal processor100in operation S120. As described above with reference toFIG.1, the control processor200may execute an instruction code and data for controlling the image signal processor100to generate the setting information IF_N1.

In detail, the FRO circuit120of the image signal processor100may store received setting information IF_N1in operation S130. Subsequently, image frames may be received from an image sensor in operation S140, and the image signal processor100may transmit a start interrupt signal to the control processor200in operation S150and may start to perform image processing.

In operation S160, the image signal processor100may perform image processing on the N frames on the basis of the setting information IF_N1. When image processing is performed on frames, the FRO circuit120may provide setting values of each of the frames on the basis of the setting information IF_N1, and the image signal processor100may perform image processing by units of frames on the basis of the setting values.

At this time, the control processor200may generate setting information IF_N2about next N frames in operation S170and may transmit the setting information IF_N2to the image signal processor100in operation S180. In detail, the FRO circuit120of the image signal processor100may store received setting information IF_N2in operation S190.

When image processing performed on the N frames is completed, the image signal processor100may transmit an end interrupt signal to the control processor200.

Subsequently, the image signal processor100and the control processor200may repeatedly perform operations S150to S200, and thus, the control processor200may generate and transmit setting information by units of N frames and the image signal processor100may perform image processing by units of frames, namely, may generate an interrupt signal by units of N frames and may transmit the interrupt signal to the control processor200.

FIG.7is a flowchart illustrating an operating method of an image processing system according to an example embodiment. The operating method ofFIG.7may be performed by the image signal processor ofFIG.1.

Referring toFIG.7, the image processing system1200may determine an operation mode in operation S210. The image processing system1200may determine the operation mode on the basis of a setting of a user, or may detect a frame rate of image data IDT received from the image sensor1100and may determine the operation mode on the basis of the detected frame rate. In some example embodiments, when frames are received from the image sensor1100at a high speed (for example, a speed of 240 fps or more), the image processing system1200may determine the operation mode as a high speed operation mode.

The image processing system1200may determine whether the determined operation mode is the high speed operation mode in operation S220, and when the determined operation mode is the high speed operation mode, as described above with reference toFIGS.1to6, the image processing system1200may perform image processing by units of N frames in operation S230. The control processor200may generate setting information about N frames and may transmit the setting information to the image signal processor100in operation S231. The image signal processor100may perform image processing on the N frames in operation S232, and when image processing is completed, the image signal processor100may generate an interrupt signal in operation S233. In operation S231, the image signal processor100may substantially perform image processing by units of frames on the basis of setting values of a corresponding frame provided from the FRO circuit120included therein at every frame.

When the determined operation mode is not the high speed operation mode, namely, when the determined operation mode is a normal operation mode or a low speed operation mode, the image processing system1200may perform image processing by units of one frame in operation S240. The control processor200may generate setting information about one frame and may transmit the setting information to the image signal processor100in operation S241. The image signal processor100may perform image processing on one frame in operation S242, and when image processing is completed, the image signal processor100may generate an interrupt signal in operation S243.

FIG.8is a timing diagram showing an operating method based on an operation mode of an image signal processor according to an example embodiment.FIG.8illustrates an operating method of the image signal processor100ofFIG.1.

Referring toFIGS.1and8, a first operation mode may be a low speed or normal operation mode, and a second operation mode may be a high speed operation mode. When the image processing system1200operates in the second operation mode, a frame rate (e.g., a reception speed of image data) of the image data may be higher than a frame rate of the image data of when the image processing system1200operates in the first operation mode. A period T4where N frames (for example, first to nthframes) are received and processed in the second operation mode may be relatively shorter than a period T3where the N frames are received and processed in the first operation mode.

In the first operation mode, the image signal processor100may receive setting information IF1to IFn+2 from the control processor200by units of frames. The image signal processor100may perform image processing by units of frames on the basis of the received setting information and may transmit interrupt signals (for example, start interrupt signals INT_S1to INT_Sn+1 and end interrupt signals INT_E1to INT_En+1) indicating the start and end of image processing to the control processor200at every frame.

In the second operation mode, the image signal processor100may receive setting information (for example, first to third setting information) IF_N1, IF_N2, and IF_N3from the control processor200by units of N frames. For example, the first setting information IF_N1may include setting values of first to nthframes F1to Fn. The image signal processor100may perform image processing on the N frames and may generate and transmit interrupt signals (for example, start interrupt signals INT_S1and INT_S2and end interrupt signals INT_E1and INT_E2) by units of N frames. In this case, the image signal processor100may substantially perform image processing by units of frames on the basis of setting values of each frame provided from the FRO circuit120which stores the setting information IF_N1, IF_N2, and IF_N3.

When the image signal processor100operates in the first operation mode, an image processing period T5of one frame may be sufficient for the control processor200to generate setting information about a next frame. Therefore, in the first operation mode, the image processing period T5of one frame may be secured as a setting margin of the control processor200, and the control processor200may generate and transmit the setting information IF1to IFn+2 by units of one frame. The image signal processor100may perform image processing by units of frames on the basis of received setting information. At this time, the FRO circuit120may be deactivated. When image processing performed on one frame is started, the image signal processor100may generate and transmit corresponding one of the interrupt signals INT_S1to INT_Sn+1. When image processing performed on one frame is completed, the image signal processor100may generate and transmit corresponding one of the interrupt signals INT_E1to INT_En+1.

When the image signal processor100operates in the second operation mode, namely, when a frame rate of image data is high, an image processing period T7of one frame may be very short, and thus, the control processor200may not be sufficient to generate setting information about a next frame. Therefore, the control processor200may generate and transmit the setting information IF_N1to IF_N3by units of N frames, and thus, may secure a processing period T6of N frames as a setting margin. The image signal processor100may store received setting information in the FRO circuit120, and when image processing is performed on each frame, the image signal processor100may perform image processing on the basis of a setting value of a corresponding frame provided from the FRO circuit120and may store converted image data in the memory300. When image processing performed on the N frames is started, the image signal processor100may generate and transmit the interrupt signals INT_S1and INT_S2. When image processing performed on the N frames is completed, the image signal processor100may generate and transmit the interrupt signals INT_E1, and INT_E2.

As described above, the image signal processor100may change an operation mode on the basis of a frame rate of image data to adaptively change a frame setting method and an operation method. Accordingly, even when a frame rate of image data is changed, a setting margin of the control processor200may be sufficiently secured.

FIG.9is a block diagram illustrating an image signal processor100aaccording to an example embodiment. A configuration and an operation of the image signal processor100aofFIG.9are similar to those of the image signal processor100ofFIG.2, and thus, a difference therebeween will be mainly described.

Referring toFIG.9, the image signal processor100amay include an ISP core110a, an FRO circuit120, and a DMA controller130, and at least one IP block (for example, a first IP block11a) of a plurality of IP blocks112a(for example, first to third IP blocks11ato13a) included in the ISP core110amay include an FRO circuit121. A controller111amay provide the FRO circuit (FRO1)121of the first IP block11awith some setting information about the first IP block11aamong pieces of setting information IF_N about N frames received from a control processor200. In a case which performs image processing on each of the N frames, the first IP block11amay perform image processing on the basis of a setting value of a corresponding frame in setting information stored in the FRO circuit121.

FIG.10is a block diagram illustrating an image signal processor100baccording to an example embodiment.

The image signal processor100bofFIG.10may include an ISP core110band a DMA controller130. The ISP core110bmay include a plurality of IP blocks112b(for example, first to third IP blocks11bto13b), and the first to third IP blocks11bto13bmay respectively include FRO circuits (FRO1to FRO3)121to123. In other words, in the image signal processor100bofFIG.10, the FRO circuits FRO1to FRO3may be respectively included in the plurality of IP blocks112b.

The controller111bmay provide each of the plurality of IP blocks112bwith setting information (e.g., setting values of N frames of each of the plurality of IP blocks112b) about each of the plurality of IP blocks112bamong pieces of setting information IF_N about the N frames received from the control processor200. Each of the plurality of IP blocks112bmay store received setting information in a corresponding FRO circuit included therein, and in a case which performs image processing on each frame, each of the plurality of IP blocks112bmay perform image processing on the basis of a setting value of a corresponding frame in setting information stored in a corresponding FRO circuit. Also, each of the plurality of IP blocks112bmay generate an address of a corresponding frame and may provide the address to the DMA controller130, and thus, may store result data and/or converted image data based on image processing in a memory300.

FIG.11is a block diagram illustrating an image signal processor100caccording to an example embodiment.

The image signal processor100cofFIG.11may include an ISP core110c, an FRO circuit120, a first DMA controller140, and a second DMA controller150. The ISP core110cmay include a controller111c, a plurality of IP blocks112c, and a post-processing block113c.

A configuration and an operation of the image signal processor100cofFIG.11are similar to those of the image signal processor100ofFIG.2. Therefore, a difference will be mainly described.

Referring toFIG.11, a post-processing block113cincluded in an ISP core110cmay perform post-processing on converted image data generated by each of a plurality of IP blocks112c. For example, the post-processing block113cmay include a scaler, a joint photographic coding experts group (JPEG) circuit, etc.

Each of the plurality of IP blocks112cmay directly transmit the converted image data to the post-processing block113c, or may store the converted image data in a memory300through a first DMA controller140. At this time, as described above with reference toFIG.2, the first DMA controller140may store the converted image data in the memory300on the basis of an address ADDR received from an FRO circuit120at every frame.

The post-processing block113cmay receive the converted image data from each of the plurality of IP blocks112c, or may receive, through a second DMA controller150, the converted image data stored in the memory300. The post-processing block113cmay store post-processed image data IDT′ in the memory300through the second DMA controller150, or may output the post-processed image data IDT′ to other elements (for example, a display) included in an image processing device (1000ofFIG.1).

FIG.12is a block diagram illustrating an image processing system20according to an example embodiment.

Referring toFIG.12, the image processing system20may include a main processor210, a ROM220, a RAM230, an image signal processor240, a non-volatile memory interface250, a camera interface260, a memory interface270, and a display interface280. The elements (e.g., the main processor210, the ROM220, the RAM230, the image signal processor240, the non-volatile memory interface250, the camera interface260, the memory interface270, and the display interface280) of the image processing system20may transmit or receive data through a system bus290. In some example embodiments, the image processing system20may be implemented as a system-on chip (SoC). In some example embodiments, the image processing system20may be an application processor.

The main processor210may control an overall operation of the image processing system20. The main processor210may be implemented with, for example, a CPU, a microprocessor, an ARM processor, an X86 processor, or an MIPS processor. According to some embodiments, the main processor210may be implemented with one computing component (e.g., a multi-core processor) including two or more independent processors (or cores). The main processor210may process or execute data and an instruction code (or programs) each stored in the ROM220or the RAM230.

The ROM220may store programs and/or data which are/is used continuously. The ROM220may be implemented as EPROM or EEPROM.

The RAM230may temporarily store programs, data, and/or instructions. According to some embodiments, the RAM230may be implemented as DRAM or SRAM. The RAM230may temporarily store image data which is input/output through the interfaces250to280or is generated through image processing by the image signal processor240.

The non-volatile memory interface250may interface data input from a non-volatile memory device255or data output to a non-volatile memory device255. The non-volatile memory device255may be implemented with, for example, a memory card (for example, multi-media card (MMC), embedded multi-media card (eMMC), secure digital (SD) card, or micro SD card).

The camera interface260may interface image data (for example, raw image data) input from a camera265disposed outside the image processing system20. The camera265may generate data corresponding to an image captured by using a plurality of light sensing devices. Image data received through the camera interface260may be provided to the image signal processor240or may be stored in a memory275through the memory interface270.

The memory interface270may interface data input from the memory275or data output to the memory275. According to some embodiments, the memory275may be implemented as a volatile memory such as DRAM or SRAM or a non-volatile memory such as ReRAM, PRAM, or NAND flash.

The image signal processor240may perform image processing on the image data provided from the camera265to generate converted image data and may store the converted image data in the memory275or may scale the converted image data to provide a scaled image to the display device285.

The control processor and the image signal processor each described above with reference toFIGS.1to11may be respectively applied as the main processor210and the image signal processor240. In a high speed operation mode, the main processor210may generate setting information by units of N frames and may transmit the setting information to the image signal processor240, and when image processing performed on the N frames is completed, the image signal processor240may transmit an interrupt signal to the main processor210. The image signal processor240may include an FRO circuit (120ofFIG.1), and the FRO circuit may store the setting information. When image processing is performed on each frame, the FRO circuit may provide a setting value of a corresponding frame. Accordingly, even in the high speed operation mode, the image signal processor240may normally perform image processing.

FIG.13is a block diagram illustrating an image processing system30according to an example embodiment.

Referring toFIG.13, the image processing system30may include a CPU310, a ROM320, a post-processing block330, a sensor interface340, an ISP core350, an FRO circuit360, and a DMA controller370. The CPU310, the ROM320, the post-processing block330, the sensor interface340, the ISP core350, the FRO circuit360, and the DMA controller370may transmit or receive data through a system bus380.

The CPU310may control an overall operation of the image processing system30and may process or execute programs stored in the ROM320to control an image processing operation.

The ROM320may store data and/or an instruction code (e.g., programs) including an image processing algorithm.

The post-processing block330may perform post-processing (for example, adjusting a size of data, or compressing data) on converted image data generated by the ISP core350. Post-processed image data may be stored in a memory375through the DMA controller370.

In some example embodiments, the image processing system30may further include a display interface, and the post-processed image data may be provided to a display device through the display interface. Alternatively, the image data stored in the memory375may be read through the DMA controller370and may be provided to the display device through the display interface.

The sensor interface340may communicate with an image sensor345and may receive image data (for example, raw image data) from the image sensor345.

The control processor, the ISP core, the FRO circuit, and the DMA controller each described above with reference toFIGS.2,9,10, and11may be respectively applied as the CPU310, the ISP core350, the FRO circuit360, and the DMA controller370. In a high speed operation mode, the CPU310may generate setting information by units of N frames and may transmit the setting information to the ISP core350, and when image processing performed on the N frames is completed by the ISP core350, the FRO circuit360may transmit an interrupt signal to the CPU310. The FRO circuit360may store the setting information, and when image processing is performed on each frame, the FRO circuit360may provide a setting value of a corresponding frame to the ISP core350and/or the DMA controller370. Therefore, image processing may be performed by units of one frame, and processing data may be stored in the memory375. Even in a high speed operation mode, the image processing system30may normally perform image processing and may store image-processed image data (for example, converted image data or post-processed image data) in the memory375.

FIG.14is a block diagram illustrating an image processing device2000according to an example embodiment. The image processing device2000ofFIG.14may be a portable terminal.

Referring toFIG.14, the image processing device2000according to some example embodiments may include an application processor (AP)2100, an image sensor2200, a display device2400, a working memory2500, a storage2600, a user interface2700, and a wireless transceiver2800, and the application processor2100may include an image signal processor (ISP)2300. The image signal processor100ofFIG.1may be applied as the image signal processor2300. In some embodiments, the image signal processor100may be implemented as a separate integrated circuit independently from the application processor2100.

The application processor2100may control an overall operation of the image processing device2000and may be provided as an SoC which drives an application program and an operating system (OS).

The application processor2100may control an operation of the image signal processor2300and may provide or store converted image data, generated by the image signal processor2300, to the display device2400or in the storage2600.

The image sensor2200may generate image data (for example, raw image data) on the basis of a received light signal and may provide the image data to the image signal processor2300.

The image signal processor described above with reference toFIGS.1to11may be applied as the image signal processor2300. The image signal processor2300may receive setting information about an Nthframe from a processor included in the application processor2100and may perform image processing on the Nthframe on the basis of the setting information. When image processing performed on the Nthframe is completed, the image signal processor2300may transmit an interrupt signal to the processor.

The working memory2500may be implemented as a volatile memory such as DRAM or SRAM or a non-volatile resistive memory such as FeRAM, RRAM, or PRAM. The working memory2500may store programs and/or data each processed or executed by the application processor2100.

The storage2600may be implemented as a non-volatile memory device such as NAND flash or a resistive memory, and for example, may be provided as a memory card (for example, MMC, eMMC, SD, or micro SD). The storage2600may store data and/or a program which correspond(s) to an execution algorithm for controlling an image processing operation of the image signal processor2300, and when the image processing operation is performed, the data and/or the program may be loaded into the working memory2500. In some embodiments, the storage2600may store image data (for example, converted image data or post-processed image data) generated by the image signal processor2300.

The user interface2700may be implemented with various devices, such as a keyboard, a curtain key panel, a touch panel, a fingerprinted sensor, and a microphone, for receiving a user input. The user interface2700may receive the user input and may provide the application processor2100with a signal corresponding to the received user input.

The wireless transceiver2800may include a transceiver2810, a modem2820, and an antenna2830.