Patent Description:
Image processing of a general image signal processor (ISP) is performed in a single device in which a plurality of separate ISP image processing modules is connected.

When a plurality of devices performs ISP processing, a first device and a second device may sequentially perform ISP. Although image processing operations of the first device and the second device have the same purpose, each device may have different performance, complexity, and sophistication level. Alternatively, an ISP image processing module present in the first device may not be present in the second device.

<NPL>, discloses an algorithmic strategy to approach vision problems in a wireless camera sensor network. The major aspect of the strategy is to reduce video data locally to both exploit rich visual information and fit the data load into the network requirement.

<NPL>, discloses an approach to task assignment in a cooperative multi-agent design system.

<NPL>, discloses types of parallel processing for parallel computing in digital image processing.

When a plurality of devices performs ISP processing on one image, ISP processing performed by a first device that performs ISP processing first may be different from that performed by a second device that performs ISP processing second, for various reasons related to the first device that performs ISP processing first. As a result, an ISP processing operation performed in the first device may be redundantly performed or an ISP processing operation that is not performed in the first device may be omitted in the second device.

According to various embodiments, when ISP processing is performed in a plurality of devices in a divided manner, information of ISP processing performed in an electronic device may be transmitted to an external electronic device that may identify the ISP processing information and perform ISP processing.

The present invention relates to an electronic device for processing an image and an operation method thereof as defined by the appended set of claims.

According to various embodiments, when ISP processing is performed in a plurality of devices in a divided manner, information of ISP processing performed in an electronic device may be transmitted to an external electronic device, such that redundant execution or omission may be prevented when the external electronic device performs ISP processing.

According to an embodiment, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input device <NUM>, a sound output device <NUM>, a display device <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module (SIM) <NUM>, and an antenna module <NUM>. In some embodiments, at least one (e.g., the display device <NUM> or the camera module <NUM>) of the components may be omitted from the electronic device <NUM>, or other components may be added in the electronic device <NUM>. In some embodiment, some components may be implemented to be integrated together, e.g., as if the sensor module <NUM> (e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) is embedded in the display device <NUM> (e.g., a display).

The processor <NUM> may drive, e.g., software (e.g., a program <NUM>) to control at least one other component (e.g., a hardware or software component) of the electronic device <NUM> connected with the processor <NUM> and may process or compute various data. The processor <NUM> may load and process a command or data received from another component (e.g., the sensor module <NUM> or the communication module <NUM>) on a volatile memory <NUM>, and the processor <NUM> may store resultant data in a non-volatile memory <NUM>. According to an embodiment, the processor <NUM> may include a main processor <NUM> (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor <NUM> that is operable independently from the main processor <NUM>. In addition to, or instead of, the main processor <NUM>, the auxiliary processor <NUM> may include an auxiliary processor <NUM> (e.g., a graphics processing unit (GPU), an image signal processor, a sensor hub processor, or a communication processor) that consumes less power than the main processor <NUM> or is specified for a designated function. Here, the auxiliary processor <NUM> may be operated separately from or embedded in the main processor <NUM>.

In such case, the auxiliary processor <NUM> may control at least some of functions or states related to at least one (e.g., the display device <NUM>, the sensor module <NUM>, or the communication module <NUM>) of the components of the electronic device <NUM>, instead of the main processor <NUM> while the main processor <NUM> is in an inactive (e.g., sleep) state or along with the main processor <NUM> while the main processor <NUM> is an active state (e.g., performing an application). The memory <NUM> may store various data used by at least one component (e.g., the processor <NUM>) of the electronic device <NUM>, e.g., software (e.g., the program <NUM>) and input data or output data for a command related to the software.

The program <NUM>, as software stored in the memory <NUM>, may include, e.g., an operating system (OS) <NUM>, middleware <NUM>, or an application <NUM>.

The input device <NUM> may be a device for receiving a command or data, which is to be used for a component (e.g., the processor <NUM>) of the electronic device <NUM>, from an outside (e.g., a user) of the electronic device <NUM>. The input device <NUM> may include, e.g., a microphone, a mouse, or a keyboard.

The sound output device <NUM> may be a device for outputting sound signals to the outside of the electronic device <NUM>. The sound output device <NUM> may include, e.g., a speaker which is used for general purposes, such as playing multimedia or recording and playing, and a receiver used for call receiving purposes only. According to an embodiment, the receiver may be formed integrally or separately from the speaker.

The display device <NUM> may be a device for visually providing information to a user of the electronic device <NUM>. The display device <NUM> may include, e.g., a display, a hologram device, or a projector and a control circuit for controlling the display, hologram device, or projector. According to an embodiment, the display <NUM> may include a touch circuitry or a pressure sensor capable of measuring the strength of a pressure with respect to a touch.

According to an embodiment, the audio module <NUM> may obtain the sound via the input device <NUM>, or output the sound via the sound output device <NUM> or a headphone of an external electronic device (e.g., an electronic device <NUM>) wiredly or wirelessly coupled with the electronic device <NUM>.

The sensor module <NUM> may generate an electrical signal or data value corresponding to an internal operating state (e.g., power or temperature) or external environmental state of the electronic device <NUM>. The sensor module <NUM> may include, e.g., a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a bio sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface <NUM> may support a designated protocol enabling a wired or wireless connection with an external electronic device (e.g., the electronic device <NUM>). According to an embodiment, the interface <NUM> may include a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal <NUM> may include a connector, e.g., an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector), which is able to physically connect the electronic device <NUM> with an external electronic device (e.g., the electronic device <NUM>).

The haptic module <NUM> may include, e.g., a motor, a piezoelectric element, or an electric stimulator.

According to an embodiment, the camera module <NUM> may include one or more lenses, an image sensor, an image signal processor, or a flash.

The power management module <NUM> may be a module for managing power supplied to the electronic device <NUM>. The power management module <NUM> may be configured as at least part of, e.g., a power management integrated circuit (PMIC).

The battery <NUM> may be a device for supplying power to at least one component of the electronic device <NUM>. The battery <NUM> may include, e.g., a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module <NUM> may support establishing a wired or wireless communication channel between the electronic device <NUM> and an external electronic device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) and performing communication through the established communication channel. The communication module <NUM> may include one or more communication processors that are operated independently from the processor <NUM> (e.g., an application processor) and supports wired or wireless communication. According to an embodiment, the communication module <NUM> may include a wireless communication module <NUM> (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS communication module) or a wired communication module <NUM> (e.g., an LAN communication module or a power-line communication module), and may communicate with the external electronic device via the first network <NUM> (e.g., a short-range communication network, such as BluetoothTM, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network <NUM> (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., an LAN or wide area network (WAN)). The above-enumerated types of communication modules <NUM> may be implemented in a single chip, where at least some of the modules are integrated, or individually in separate chips.

According to an embodiment, the wireless communication module <NUM> may differentiate and authenticate the electronic device <NUM> in the communication network using user information stored in the subscriber identification module <NUM>.

The antenna module <NUM> may include one or more antennas for transmitting or receiving a signal or power to/from an outside. According to an embodiment, the communication module <NUM> (e.g., the wireless communication module <NUM>) may transmit or receive a signal to/from an external electronic device through an antenna appropriate for a communication scheme.

According to an embodiment, some or all of operations performed by the electronic device <NUM> may be performed in another electronic device or a plurality of other electronic devices. According to an embodiment, when the electronic device <NUM> has to perform a function or a service automatically or at a request, the electronic device <NUM> may request an external electronic device to perform at least some functions associated with the function or the service, instead of or in addition to executing the function or the service. The external electronic device having received the request may execute the requested function or additional function and deliver the execution result to the electronic device <NUM>. The electronic device <NUM> may then process or further process the received result to provide the requested function or service.

<FIG> is a block diagram <NUM> of the camera module <NUM> according to various embodiments. Referring to <FIG>, the camera module <NUM> may include a lens assembly <NUM>, a flash <NUM>, an image sensor <NUM>, an image stabilizer <NUM>, memory <NUM> (e.g., buffer memory), or an image signal processor <NUM>. The lens assembly <NUM> may collect light emitted or reflected from an object whose image is to be taken. The lens assembly <NUM> may include one or more lenses. According to an embodiment, the camera module <NUM> may include a plurality of lens assemblies <NUM>. In such a case, the camera module <NUM> may form, for example, a dual camera, a <NUM>-degree camera, or a spherical camera. The plurality of lens assemblies <NUM> may have the same lens attribute (e.g., view angle, focal length, auto-focusing, f number, or optical zoom), or at least one lens assembly may have at least one other lens attribute different from those of another lens assembly. The lens assembly <NUM> may include, for example, a wide-angle lens or a telephoto lens. The flash <NUM> may emit a light source that is used to reinforce light emitted from an object. The flash <NUM> may include one or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp.

The image sensor <NUM> may obtain an image corresponding to an object by converting light transmitted from the object via the lens assembly <NUM> into an electrical signal. According to an embodiment, the image sensor <NUM> may include one selected from image sensors having different attributes, such as an RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor <NUM> may be implemented with, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer <NUM> may move or control (e.g., adjust read-out timing) at least one lens included in the lens assembly <NUM> or the image sensor <NUM> to at least partially compensate for a negative influence (e.g., image blurring) of movement of the camera module <NUM> or the electronic device <NUM> including the same upon a captured image in response to the movement. According to an embodiment, the image stabilizer <NUM> may be implemented with, for example, an optical image stabilizer, and may sense such a movement using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module <NUM>.

The memory <NUM> may store, at least temporarily, at least part of an image obtained via the image sensor <NUM> for a subsequent image processing task. For example, if image capturing is delayed due to shutter lag or multiple images are quickly captured, a raw image obtained (e.g., a high-resolution image) may be stored in the memory <NUM>, and its corresponding copy image (e.g., a low-resolution image) may be previewed via the display device <NUM>. Thereafter, if a specified condition is met (e.g., by a user's input or system command), at least part of the raw image stored in the memory <NUM> may be obtained and processed, for example, by the image signal processor <NUM>. According to an embodiment, the memory <NUM> may be configured as at least part of the memory <NUM> or as a separate memory that is operated independently from the memory <NUM>.

The image signal processor <NUM> may perform image processing (e.g., depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening) on an image obtained by the image sensor <NUM> or an image stored in the memory <NUM>. Additionally or alternatively, the image signal processor <NUM> may perform control (e.g., exposure time control or read-out timing control) with respect to at least one (e.g., the image sensor <NUM>) of the components included in the camera module <NUM>. An image processed by the image signal processor <NUM> may be stored back in the memory <NUM> for further processing, or may be provided to an external component (e.g., the memory <NUM>, the display device <NUM>, the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) outside the camera module <NUM>. According to an embodiment, the image signal processor <NUM> may be configured as at least part of the processor <NUM>, or as a separate processor that is operated independently from the processor <NUM>. When the image signal processor <NUM> is configured as a separate processor, images processed by the image signal processor <NUM> may be displayed, by the processor <NUM>, via the display device <NUM> as it is or after being further processed.

According to an embodiment, the electronic device <NUM> may include two or more camera modules <NUM> having different attributes or functions. In this case, for example, at least one camera module <NUM> may be a wide-angle camera or a front camera, and at least one another camera module may be a telescopic camera or a rear camera.

<FIG> is a diagram for describing operations of an electronic device and an external electronic device, according to various embodiments.

The electronic device <NUM> may include an image sensor <NUM>, an image signal processor (ISP) <NUM>, and a memory <NUM>. The external electronic device <NUM> (e.g., the server <NUM>) may include the recognition module <NUM>, the ISP <NUM>, and a storage <NUM>. The recognition module <NUM> may be a logic module and may be implemented with a processor of the external electronic device <NUM>. The ISP <NUM> may also be implemented with a processor of the external electronic device <NUM>, and for example, a processor of the external electronic device <NUM> may perform both recognition and image processing. Although not shown, the electronic device <NUM> may include a communication module (e.g., the communication interface <NUM> or the communication module <NUM>) capable of transmitting and receiving data to and from an external electronic device <NUM>. The external electronic device <NUM> may include a communication module capable of transmitting and receiving data to and from the electronic device <NUM>.

The image sensor <NUM> (e.g., the image sensor <NUM>) may obtain an image regarding an external object and generate a raw image <NUM> (a native image) corresponding to the obtained image. The image sensor <NUM> may deliver the raw image <NUM> to the ISP <NUM>. According to various embodiments, the image sensor <NUM> may generate the small raw image <NUM> and transmit the same to the external electronic device <NUM> through the communication module. According to another embodiment, the processor <NUM> of the electronic device <NUM> instead of the image sensor <NUM> may generate the small raw image <NUM> and transmit the generated small raw image <NUM> to the external electronic device <NUM> through the communication module. The image sensor <NUM> may transmit the raw image <NUM> in a compressed state to the ISP <NUM> or the external electronic device <NUM>. The image sensor <NUM> may compress the raw image <NUM> for partial processing with respect to the same and store the compressed raw image <NUM> in an internal memory of the image sensor <NUM>. The recognition module <NUM> of the external electronic device <NUM> may obtain the small raw image <NUM> through the communication module, and segment at least one image region from the small raw image <NUM>. The recognition module <NUM> may recognize each of at least one image region divided as a result of segmentation. Correction region information <NUM> may be generated, which includes information related to a plurality of image regions generated from the recognition module <NUM>, e.g., coordinate information or a recognition result of an image region. The correction region information <NUM> may be transmitted to the electronic device <NUM>. The ISP <NUM> may correct the raw image <NUM> by using the correction region information <NUM>, thus generating a corrected image. The corrected image may have, for example, a format of YUV. The corrected image may be stored in the memory <NUM>. The corrected image may be compressed according to, for example, the Joint Photographic Experts Group (JPEG) scheme, and the compressed image may be stored in the memory <NUM>.

In various embodiments of the present disclosure, the raw image <NUM> provided from the image sensor <NUM> may be transmitted to the external electronic device <NUM> separately from the small raw image <NUM>. The raw image <NUM> has a larger volume than the small raw image <NUM>, such that the small raw image <NUM> may be first transmitted to the external electronic device <NUM> and then the raw image <NUM> may be transmitted to the external electronic device <NUM>. For example, when the ISP <NUM> performs correction on the raw image <NUM>, the raw image <NUM> may be transmitted to the external electronic device <NUM>. The raw image <NUM> may be uploaded to the external electronic device <NUM> in the original form generated by the image sensor <NUM> or may be uploaded after being pre-processed by lens distortion compensation or noise cancellation. The pre-processing may be performed in the external electronic device <NUM>. The external electronic device <NUM> may perform pre-processing for demosaic processing or image format change, or pre-processing for improving an image recognition rate. The ISP <NUM> of the external electronic device <NUM> may correct the received raw image <NUM>. The external electronic device <NUM> may correct the raw image <NUM> by using the existing generated correction region information <NUM> or by using scaled correction region information. The raw image <NUM> may have a higher resolution than the small raw image <NUM>, such that the ISP <NUM> of the external electronic device <NUM> may obtain detailed scaled correction region information from the high-resolution image. The ISP <NUM> may generate the scaled correction region information by using the existing generated correction region information and the raw image <NUM>. The ISP <NUM> may obtain a high-quality image <NUM> by enhancing the raw image <NUM> using the scaled correction region information. The high-quality image <NUM> may be stored in the storage <NUM> of the external electronic device <NUM>, and may be downloaded in the electronic device <NUM>.

According to various embodiments, an image partially corrected in the ISP <NUM> may be transmitted to the external electronic device <NUM>. In this case, the electronic device <NUM> may transmit information about a correction execution state, together with the image partially corrected by the ISP <NUM>, to the external electronic device <NUM>. The external electronic device <NUM> may further perform correction by using the image processing module <NUM> based on the received information about the correction execution state (e.g., ISP processing state information), which will be described in more detail.

<FIG> is a diagram for describing operations of an electronic device and an external electronic device, according to various embodiments. <FIG> is a diagram of a configuration in which an electronic device and an external electronic device perform ISP image processing, according to various embodiments.

In operation <NUM>, the electronic device <NUM> (e.g., the processor <NUM>) may identify a raw image. For example, the electronic device <NUM> may obtain a raw image through the camera module <NUM> or obtain a raw image from another electronic device through the communication module <NUM>.

In operation <NUM>, the electronic device <NUM> may start execution of image processing according to a first image processing set. Referring to <FIG>, an electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may identify a raw image (a native image) <NUM> that is subject to image processing. The electronic device <NUM> and an external electronic device <NUM> may have a different image processing set (e.g., an image processing set <NUM> or an image processing set <NUM>) depending on a type of an electronic device. According to an embodiment, the image processing set <NUM> may be implemented with any one of a hardware configuration or a software configuration, and as a part of the processor <NUM> of the electronic device <NUM> or a part of the ISP <NUM> of the external electronic device <NUM> (e.g., the server <NUM> of <FIG> or the external electronic device <NUM>). The image processing set <NUM> may include a plurality of ISP image processing modules (e.g., ISP image processing modules <NUM>-<NUM> through <NUM>-<NUM>) that are sequentially connected, and the electronic device <NUM> may continuously perform image processing by using the plurality of ISP image processing modules. The role and performance of an ISP image processing module included in each image processing set may differ with an electronic device. The execution order of an image processing algorithm in an image processing set may vary with a type of an electronic device.

According to an embodiment, a first electronic device <NUM> may start execution of image processing by using the image processing set <NUM>. Upon start of image processing, the first electronic device <NUM> may start execution of image processing on the raw image <NUM> sequentially by using the plurality of ISP image processing modules included in the image processing set <NUM>. The second electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may identify a raw image <NUM> that is subject to image processing. The second electronic device <NUM> may start execution of image processing by using the image processing set <NUM> included in the second electronic device <NUM>. Upon start of image processing, the second electronic device <NUM> may start execution of image processing on the raw image <NUM> sequentially by using the plurality of ISP image processing modules included in the image processing set <NUM>.

In operation <NUM>, the electronic device <NUM> may transmit a raw image on which at least partial processing of a designated image processing set is performed, to the external electronic device <NUM> by using a communication module during execution of image processing. The electronic device <NUM> may transmit the raw image on which at least partial processing of an image processing set included in the electronic device <NUM> is performed, to the external electronic device <NUM>, taking characteristics of the electronic device <NUM> (e.g., version information of the image processing set <NUM> of the electronic device <NUM>, a power state of the electronic device <NUM>, an environment in capturing of the raw image, user settings, etc.).

Referring to <FIG>, the first electronic device <NUM> may transmit the raw image <NUM> to the external electronic device <NUM> by using a communication module <NUM> in a state where the first electronic device <NUM> does not perform image processing on the raw image <NUM>. The second electronic device <NUM> may transmit the raw image to the external electronic device <NUM> by using a communication module <NUM> in a state where some <NUM> of the image processing set <NUM>, e.g., ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> have performed processing on a raw image <NUM> during image processing start in operation <NUM>. Although not shown, according to another embodiment, the first electronic device <NUM> may transmit a raw image resulting from image processing on the raw image <NUM> by using the entire image processing set <NUM> to the external electronic device <NUM>, by using the communication module <NUM>. When the first electronic device <NUM> performs image processing on the raw image <NUM> by using the entire image processing set <NUM> and transmits the image-processed raw image to the external electronic device <NUM>, the external electronic device <NUM> may skip image processing using an image processing set <NUM>.

According to an embodiment, the first electronic device <NUM> may transmit the raw image <NUM> to the external electronic device <NUM> by using the communication module <NUM>. Alternatively, when the first electronic device <NUM> transmits the raw image <NUM> to the external electronic device <NUM> by using the communication module <NUM>, the first electronic device <NUM> may compress the raw image <NUM> and generate and transmit a small raw image.

In operation <NUM>, the electronic device <NUM> may transmit information corresponding to at least partial processing to an external electronic device <NUM> (e.g., the external electronic device <NUM>). In <FIG>, transmission of the raw image that is subject to at least partial processing of the first image processing set and transmission of information corresponding to at least partial processing are shown as operations of two steps, but the illustration is merely an example. The electronic device <NUM> according to various embodiments may transmit the raw image on which at least partial processing of the first image processing set is performed and then transmit information corresponding to at least partial processing to the external electronic device <NUM>, or may transmit the raw image on which at least partial processing of the first image processing set is performed and the information corresponding to at least partial processing to the external electronic device <NUM> through one communication signal. The electronic device <NUM> may transmit the information corresponding to at least partial processing to the external electronic device <NUM> such that the external electronic device <NUM> skips at least partial processing of an image processing set included in the external electronic device <NUM>. Referring to <FIG>, the first electronic device <NUM> may transmit image data corresponding to the raw image <NUM>, together with information corresponding to at least partial processing. The electronic device <NUM> may combine the image data with information about a processing state of an ISP image processing module corresponding to the information corresponding to at least partial processing to generate first transmission data <NUM>, and transmit the first transmission data <NUM> to the external electronic device <NUM>. The external electronic device <NUM> may combine the raw image <NUM> with information about a processing state of an ISP image processing module, which is information corresponding to at least partial processing on image data corresponding to a raw image image-processed by some <NUM> of the image processing set <NUM>, as header information to generate second transmission data <NUM>, and transmit the second transmission data <NUM> to the external electronic device <NUM>.

<FIG> is an example of processing state information of an ISP image processing module, according to various embodiments. According to various embodiments, processing state information of an ISP image processing module may include at least one of device information or processing state information. The device information may include model information of the electronic device (e.g., the electronic device <NUM> of <FIG>), version information of an image processing set included in the electronic device, version information of each of a plurality of ISP image processing modules included in the image processing set of the electronic device, lens characteristic information included in the electronic device, and sensor model information included in the electronic device. Depending on the above-described information, a way to process a raw image and which ISP image processing module among the plurality of ISP image processing modules included in the image processing set is to be used for image processing by the electronic device may differ. The processing state information may include information about whether each ISP image processing module included in the image processing set of the electronic device performs processing. For example, in <FIG>, when the electronic device transmits a raw image to the external electronic device <NUM> by performing image processing by using ISP image processing modules <NUM>, <NUM>, and <NUM> and performing image processing without using ISP image processing modules <NUM>, <NUM>, and <NUM>, the electronic device may store information indicating whether each ISP image processing module is to perform image processing in processing state information of the ISP image processing module.

In operation <NUM>, the external electronic device <NUM> may skip processing corresponding to at least partial processing as a part of image processing according to a designated image signal processing set included in the external electronic device by using the at least partially processed raw image. The external electronic device <NUM> may determine a process to be skipped, based on the received information corresponding to at least partial processing. In operation <NUM>, the electronic device <NUM> may perform image processing according to an image signal processing set included in the external electronic device <NUM> by excluding processing corresponding to at least partial processing. For example, the external electronic device <NUM> may skip image processing corresponding to at least partial processing performed in the electronic device <NUM> to avoid redundant execution of the image processing.

Referring to <FIG>, the external electronic device <NUM> may divide the first transmission data <NUM> received using the communication module <NUM> into image data and processing state information of an ISP image processing module. When the image data is compressed and transmitted in the first electronic device <NUM>, the processor <NUM> of the external electronic device <NUM> may decode the image data by using a decoder having a compression algorithm used for compression in the first electronic device <NUM>.

According to an embodiment, the processor <NUM> of the external electronic device <NUM> may determine an ISP image processing module that is to perform image processing and an ISP image processing module that is not to perform image processing among the plurality of ISP image processing modules included in the image processing set <NUM>, based on the information corresponding to at least partial processing. For example, the processor <NUM> may determine based on the information corresponding to at least partial processing received from the first electronic device <NUM> that image processing is not performed in the first electronic device <NUM>. The processor <NUM> may input the raw image received from the first electronic device <NUM> to the ISP image processing modules <NUM> through <NUM> and sequentially start execution of image processing, thus performing image processing by using all of the plurality of ISP image processing modules included in the image processing set <NUM> included in the external electronic device <NUM>. The external electronic device <NUM> may transmit an image-processed image to the first electronic device <NUM> by using the communication module <NUM>.

According to an embodiment, the processor <NUM> may determine based on the information corresponding to at least partial processing received from the second electronic device <NUM> that image processing is performed using a part of the image processing set in the second electronic device <NUM>. The processor <NUM> may determine that image processing has been performed on the raw image <NUM> by using the ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> in the second electronic device <NUM>. The processor <NUM> may skip image processing using ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> to avoid redundant image processing because the ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are the same as the ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> included in the external electronic device <NUM>. The processor <NUM> may input the raw image received from second electronic device <NUM> to an ISP image processing module <NUM>-<NUM> and sequentially start execution of image processing. The processor <NUM> may perform image processing by using a plurality of ISP image processing modules included in the image processing set <NUM> included in the external electronic device <NUM>, except for the ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. The external electronic device <NUM> may transmit an image-processed image to the second electronic device <NUM> by using the communication module <NUM>.

In operation <NUM>, the electronic device <NUM> (e.g., the processor <NUM>) may transmit version information of a first image processing set as a part of information corresponding to at least partial processing to an external electronic device <NUM> (e.g., the external electronic device <NUM> of <FIG>, the external electronic device <NUM> of <FIG>, or the ISP <NUM> of <FIG>). For example, the electronic device <NUM> may transmit version information of an image processing set included in the electronic device <NUM> and version information of a plurality of ISP image processing modules included in the image processing set to the external electronic device <NUM> through the information corresponding to the at least partial processing described with reference to <FIG>. Alternatively, the electronic device <NUM> may transmit version information of an ISP image processing corresponding to at least partial processing to the external electronic device <NUM> through the information corresponding to the at least partial processing.

Referring to <FIG>, a first electronic device <NUM> (e.g., the electronic device <NUM> or the processor <NUM> of <FIG> or the electronic device <NUM> of <FIG>) may transmit version information of an image processing set <NUM> included in the first electronic device <NUM> and version information of each of a plurality of ISP image processing modules included in the image processing set <NUM> to an external electronic device <NUM> (e.g., the server <NUM>) through first transmission data <NUM>. A second electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may transmit version information of an image processing set <NUM> included in the second electronic device <NUM> and version information of each of a plurality of ISP image processing modules included in the image processing set <NUM> to the external electronic device <NUM> through second transmission data <NUM>.

In operation <NUM>, the electronic device <NUM> may compare the version information of the first image processing set with the version information of the second image processing set included in the external electronic device <NUM>. For example, the external electronic device <NUM> may compare the version information of the first image processing set included in the electronic device <NUM> with the version information of the second image processing set, and compare version information of a plurality of ISP image processing modules included in the first image processing set with version information of a plurality of ISP image processing modules included in the second image processing set. In operation <NUM>, the external electronic device <NUM> may determine whether to skip processing corresponding to at least partial processing based on a result of the comparison. In operation <NUM>, the external electronic device <NUM> may perform image processing by using the second image processing set, based on a result of the determination. The external electronic device <NUM> may determine whether to skip processing corresponding to at least partial processing performed in the electronic device <NUM> based on the result of the determination to perform image processing based on a result of the determination.

According to an embodiment, referring to <FIG>, an external electronic device <NUM> (e.g., the server <NUM> of <FIG>, the external electronic device <NUM> of <FIG>, the external electronic device <NUM> of <FIG>, or the external electronic device <NUM> of <FIG>) may receive information regarding at least partial processing corresponding to the raw image <NUM> from the first electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG>, or the electronic device <NUM> of <FIG>) through a communication module <NUM>. The processor <NUM> of the external electronic device <NUM> may receive the information corresponding to the at least partial processing from the communication module <NUM>. The processor <NUM> may compare version information of the plurality of ISP image processing modules included in the image processing set <NUM> with version information of a plurality of ISP image processing modules included in an image processing set <NUM>, based on the information corresponding to the at least partial processing. The processor <NUM> may determine based on the information corresponding to at least partial processing that image processing is not performed in the first electronic device <NUM>. The processor <NUM> may perform image processing on the raw image received from the first electronic device <NUM> by using the image processing set <NUM>.

According to an embodiment, an external electronic device <NUM> may receive information regarding at least partial processing corresponding to the raw image <NUM> from the first electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG>, or the electronic device <NUM> of <FIG>) through the communication module <NUM>. The processor <NUM> of the external electronic device <NUM> may receive information <NUM> corresponding to the at least partial processing from the communication module <NUM>. The processor <NUM> may compare version information of the plurality of ISP image processing modules included in the image processing set <NUM> with version information of a plurality of ISP image processing modules included in an image processing set <NUM>, based on the information corresponding to the at least partial processing. The processor <NUM> may determine based on information <NUM> corresponding to at least partial processing that image processing has been performed by using the ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> in the second electronic device <NUM>. The processor <NUM> may compare the version information of the image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> with the version information of the image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>.

According to an embodiment, even when image processing is performed by some ISP image processing modules included in the image processing set <NUM> of the second electronic device <NUM>, in case that some ISP image processing modules included in the image processing set <NUM> have higher versions, the processor <NUM> may perform image processing by using an ISP image processing module having a higher version among the some ISP image processing modules included in in the image processing set <NUM>. For example, the versions of the ISP image processing modules <NUM>-<NUM> and <NUM>-<NUM> are the same as those of the ISP image processing modules <NUM>-<NUM> and <NUM>-<NUM> and the version of the ISP image processing module <NUM>-<NUM> is higher than that of the ISP image processing module <NUM>-<NUM>, the processor <NUM> may not perform image processing using the ISP image processing modules <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. In another example, the versions of the ISP image processing modules <NUM>-<NUM> and <NUM>-<NUM> are the same as those of the ISP image processing modules <NUM>-<NUM> and <NUM>-<NUM> and the version of the ISP image processing module <NUM>-<NUM> is lower than that of the ISP image processing module <NUM>-<NUM>, the processor <NUM> may input a raw image received from the second electronic device to the ISP image processing module <NUM>-<NUM> and perform image processing such that image processing is performed sequentially from the ISP image processing module <NUM>-<NUM>.

<FIG> is a diagram of a configuration in which an electronic device and an external electronic device perform ISP image processing, according to various embodiments.

Referring to <FIG>, an image processing set <NUM> of an external electronic device <NUM> (e.g., the server <NUM> of <FIG>, the external electronic device <NUM> or the ISP <NUM> of <FIG>, the external electronic device <NUM> of <FIG>, or the external electronic device <NUM> of <FIG>) may include an ISP image processing module that is not included in an image processing set <NUM> of the first electronic device (e.g., the electronic device <NUM> of <FIG>, the processor <NUM> of <FIG>, the electronic device <NUM> of <FIG>, or the electronic device <NUM> of <FIG>). For example, the image processing set <NUM> may include the ISP image processing module <NUM>-<NUM> and the ISP image processing module <NUM>-<NUM> that are not included in the image processing set <NUM>. The external electronic device <NUM> may receive a raw image on which at least partial processing of the ISP image processing module <NUM>-<NUM> is performed and information <NUM> corresponding to the at least partial processing, from a first electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>). The external electronic device <NUM> may determine that the raw image has undergone image processing of the ISP image processing module <NUM>-<NUM>, based on the at least partially processed raw image and the information <NUM> corresponding to the at least partial processing. The external electronic device <NUM> may input the raw image to the ISP image processing module <NUM>-<NUM> and perform image processing sequentially, such that the received raw image is image-processed by the ISP image processing module <NUM>-<NUM> and the ISP image processing module <NUM>-<NUM> that are not included in the image processing set <NUM>. The external electronic device <NUM> may perform image processing on the raw image by using the image processing set <NUM> and transmit the image-processed image to the first electronic device <NUM> by using a communication module <NUM>.

An electronic device (e.g., the electronic device <NUM> of <FIG>) according to various embodiments may include a memory (e.g., the memory <NUM> of <FIG>), a communication module (e.g., the communication module <NUM> of <FIG>), and a processor (e.g., the processor <NUM> of <FIG>), in which the processor <NUM> is configured to identify a raw image, to perform image processing according to a designated first image signal processing set, on the raw image, to transmit the raw image on which at least partial processing of the designated first image signal processing set is performed, to an external electronic device (e.g., the external electronic device <NUM> of <FIG>) by using the communication module <NUM> during the performing image processing, and to transmit information corresponding to the at least partial processing to the external electronic device <NUM> such that the external electronic device <NUM> skips processing corresponding to the at least partial processing among a designated second image signal processing set included in the external electronic device <NUM> by using the raw image on which the at least partial processing is performed.

In the electronic device <NUM> according to an embodiment, the processor <NUM> may be configured to transmit version information regarding at least a part of the first image signal processing set corresponding to the at least partial processing to the external electronic device <NUM> as a part of the information corresponding to the at least partial processing such that the external electronic device <NUM> determines whether to skip processing, based on the version information. According to an embodiment, the processor <NUM> may be configured to receive an image for which image processing is skipped by a first image processing module, which is included in at least a part of the first image signal processing set, from the external electronic device, when a version of a first image processing module is higher than a version of a second image processing module included in at least a part of the external electronic device, and to receive an image for which image processing is performed by the first image processing module, which is included in at least a part of the first image signal processing set, from the external electronic device, when the version of the first image processing module is lower than the version of the second image processing module included in at least a part of the external electronic device. According to an embodiment, the processor <NUM> may be configured to receive, using the communication module, a corrected image obtained by the external electronic device <NUM> performing image processing on the raw image by using the second image signal processing set. According to an embodiment, the information corresponding to the at least partial processing may include information regarding whether image processing is performed by at least a part of the designated first image signal processing set and at least one of device model information of the electronic device, lens characteristic information of the electronic device, or sensor model information of the electronic device. According to an embodiment, the processor <NUM> may be configured to receive an image on which image processing of the first image processing module, which is included in the designated second image signal processing set included in the external electronic device <NUM>, is performed, from the external electronic device <NUM>, when the first image processing module is not included in the first image signal processing set.

A method of controlling the electronic device <NUM> according to various embodiments may include identifying a raw image, performing image processing according to a designated first image signal processing set by using the raw image, transmitting the raw image on which at least partial processing of the designated first image signal processing set is performed, to the external electronic device <NUM> by using the communication module <NUM> during execution of image processing, and transmitting information corresponding to the at least partial processing to the external electronic device <NUM> such that the external electronic device <NUM> skips processing corresponding to the at least partial processing as a part of performing image processing according to a designated second image signal processing set included in the external electronic device <NUM> by using the raw image on which the at least partial processing is performed.

According to an embodiment, the method may further include transmitting version information regarding at least a part of the first image signal processing set corresponding to the at least partial processing to the external electronic device as a part of the information corresponding to the at least partial processing to determine whether to skip processing, based on the version information.

According to an embodiment, the method may further include receiving an image for which image processing is skipped by the first image processing module from the external electronic device, when a version of a first image processing module included in at least a part of the first image signal processing set is higher than a version of a second image processing module included in at least a part of the external electronic device, and receiving an image for which image processing is performed by the first image processing module, which is included in at least a part of the first image signal processing set, from the external electronic device, when the version of the first image processing module is lower than the version of the second image processing module included in at least a part of the external electronic device.

According to an embodiment, the method may further include receiving a corrected image obtained by the external electronic device <NUM> performing image processing on the raw image by using the second image signal processing set. According to an embodiment, the information corresponding to the at least partial processing may include information regarding whether image processing is performed by at least a part of the designated first image signal processing set and device model information of the electronic device <NUM>, lens characteristic information of the electronic device, or sensor model information of the electronic device. According to an embodiment, the method may further include receiving an image on which image processing of the first image processing module, which is included in the designated second image signal processing set included in the external electronic device <NUM>, is performed, from the external electronic device <NUM>, when the first image processing module is not included in the first image signal processing set.

The electronic device <NUM> according to various embodiments may include the communication module <NUM> and the processor <NUM>, in which the processor <NUM> is configured to receive a raw image on which at least partial processing of a designated first image signal processing set is performed, from the external electronic device <NUM> by using the communication module <NUM>, to receive information corresponding to the at least partial processing from the external electronic device <NUM>, to skip processing corresponding to the at least partial processing among a designated second image signal processing set by using the raw image on which the at least partial processing is performed, based on the information corresponding to the at least partial processing, and to perform image processing by using the second image signal processing set, based on a result of the skipping.

According to an embodiment, the processor <NUM> may be further configured to receive version information regarding at least a part of the first image signal processing set corresponding to the at least partial processing and to determine based on the version information whether to perform processing corresponding to the at least partial processing, as a part of performing image processing according to the designated second image signal processing set by using the raw image on which the at least partial processing is performed.

According to an embodiment, the processor <NUM> may be further configured to compare version information regarding at least a part of the first image signal processing set corresponding to the at least partial processing with version information regarding the designated second image signal processing set to determine whether to perform processing corresponding to the at least partial processing. According to an embodiment, the processor <NUM> may be further configured to, when a version of at least a part of the first image signal processing set corresponding to the at least partial processing is higher than a version of at least a part of the designated second image signal processing set, skip image processing using at least the part of the second image signal processing set. According to an embodiment, the processor <NUM> may be further configured to, when a version of at least a part of the first image signal processing set corresponding to the at least partial processing is lower than a version of at least a part of the designated second image signal processing set, perform image processing on the raw image by performing image processing using at least the part of the second image signal processing set. According to an embodiment, the processor <NUM> may be configured to transmit to the external electronic device <NUM>, using the communication module <NUM>, a second image corrected by performing image processing on the raw image using the second image signal processing set. According to an embodiment, the information corresponding to the at least partial processing may include information regarding whether image processing is performed by at least a part of the designated first image signal processing set and at least one of device model information of the external electronic device <NUM>, lens characteristic information of the external electronic device <NUM>, or sensor model information of the external electronic device <NUM>. According to an embodiment, the processor <NUM> may be configured to perform image processing by using a first image processing module, when the first image processing module included in the second image signal processing set is not included in the first image signal processing set and transmit an image on which image processing is performed to the external electronic device <NUM>.

A method of controlling an electronic device according to various embodiments may include receiving a raw image on which at least partial processing of a designated first image signal processing set is performed, from an external electronic device, receiving information corresponding to the at least partial processing from the external electronic device, skipping processing corresponding to the at least partial processing among a designated second image signal processing set by using the raw image on which the at least partial processing is performed, based on the information corresponding to the at least partial processing, and performing image processing by using the second image signal processing set, based on a result of the skipping.

According to an embodiment, the method may further include receiving version information regarding at least a part of the first image signal processing set corresponding to the at least partial processing and determining based on the version information whether to perform processing corresponding to the at least partial processing, as a part of performing image processing according to the designated second image signal processing set by using the raw image on which the at least partial processing is performed.

According to an embodiment, the method may further include comparing version information regarding at least a part of the first image signal processing set corresponding to the at least partial processing with version information regarding the designated second image signal processing set to determine whether to perform processing corresponding to the at least partial processing. According to an embodiment, the method may further include, when a version of at least a part of the first image signal processing set corresponding to the at least partial processing is higher than a version of at least a part of the designated second image signal processing set, skipping image processing using at least the part of the second image signal processing set. According to an embodiment, the method may further include, when a version of at least a part of the first image signal processing set corresponding to the at least partial processing is lower than a version of at least a part of the designated second image signal processing set, performing image processing on the raw image by performing image processing using at least the part of the second image signal processing set. According to an embodiment, the method may further include transmitting to the external electronic device, using the communication module, a second image corrected by performing image processing on the raw image using the second image signal processing set. According to an embodiment, the information corresponding to the at least partial processing may include information regarding whether image processing is performed by at least a part of the designated first image signal processing set and at least one of device model information of the external electronic device, lens characteristic information of the external electronic device, or sensor model information of the external electronic device. According to an embodiment, the method may further include performing image processing by using a first image processing module, when the first image processing module included in the second image signal processing set is not included in the first image signal processing set and transmit an image on which image processing is performed to the external electronic device.

An electronic device according to various embodiments may be one of various types of electronic devices, according to various embodiments of the present disclosure.

Expressions such as "first," "second," "primarily," or "secondary," used herein may represent various elements regardless of order and/or importance and do not limit corresponding elements. When it is described that an element (such as a first element) is "operatively or communicatively coupled with/to" or "connected" to another element (such as a second element), the element can be directly connected to the other element or can be connected to the other element through another element (e.g., a third element).

A term "module" used herein may mean, for example, a unit including one of or a combination of two or more of hardware, software, and firmware, and may be used interchangeably with terms such as logic, a logic block, a part, or a circuit. The module may be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, the module may be implemented as an application-specific integrated circuit (ASIC).

The machine may invoke stored instructions from the storage medium and operate according to the invoked instructions, and may include an electronic device (e.g., the electronic device <NUM>) according to the disclosed embodiments. When the instructions are executed by a processor (for example, the processor <NUM>), functions corresponding to the instructions may be performed directly by the processor or using other components under control of the processor.

The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., Play StoreTM), or between two user devices (e.g., smart phones) directly.

Claim 1:
An electronic device (<NUM>) comprising:
a memory (<NUM>);
a communication module (<NUM>); and
a processor (<NUM>),
wherein the processor (<NUM>) is configured to:
identify a raw image;
perform image processing according to a designated first image signal processing set, on the raw image;
transmit the raw image on which at least partial processing of the designated first image signal processing set is performed, to an external electronic device (<NUM>) by using the communication module (<NUM>), during the performing of the image processing; and
transmit information corresponding to the at least partial processing to the external electronic device (<NUM>) such that the external electronic device (<NUM>) skips processing corresponding to the at least partial processing among a designated second image signal processing set included in the external electronic device (<NUM>) by using the raw image on which the at least partial processing is performed,
wherein the designated image processing set comprises image processing modules and the information corresponding to the at least partial processing comprises information concerning at least one of said image processing modules and regarding whether image processing is already performed by said image processing module.