Patent Description:
A conventional camera may be configured to record a video or take a picture, collect brightness information and color information of a scenario, but cannot collect depth information. Along with increase of application requirements, depth cameras have been added to cameras of some electronic devices to form camera arrays. The depth camera may include an array camera module, a structured light module and a time of flight (TOF) module, and depth information may be obtained according to a working principle of each module. However, the camera array requires independent arrangement of the depth camera, which results in occupation of valuable space on the electronic device and is unfavorable for miniaturization and cost reduction of the electronic device. Related technologies are known from <CIT>, <CIT> and <CIT>. <CIT> discloses an apparatus that has a processing device coupled to a projection device and cameras to receive images. The processing device obtains first and second depth information from the images and generates a depth map of a scene from the depth information selectively. The processing device comprises a first depth information unit comparing a visible light image of the first image with a visible light image of the second image to obtain the first depth information. A second depth information unit identifies an invisible light image in the second image to obtain the second depth information.

<CIT> discloses a method that involves capturing two images illuminated by natural light and infrared light respectively.

<CIT> discloses an image processing device that includes an invisible image filter separator that separates an invisible detail component, a visible image filter separator that separates a visible base component and a visible detail component, a base luminance color separator that separates a visible luminance base component and a visible color base component, a detail luminance color separator that separates a visible luminance detail component, a detail synthesizer that generates a synthetic luminance detail component, a synthetic luminance component generator that generates a synthetic luminance component, and a luminance color synthesizer that generates a synthetic image.

In view of this, the present disclosure provides an image processing method and device, a camera component, an electronic device, and a storage medium.

According to a first aspect of the present disclosure, a camera component is provided as defined by claim <NUM>.

According to a second aspect of the present disclosure, an image processing method is provided as defined by claim <NUM>,.

According to another aspect of the present disclosure, a readable storage medium is provided, in which an executable computer program is stored, the computer program is executed to implement the steps of any abovementioned method.

The embodiments set forth in the following exemplary description do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatuses consistent with aspects related to the present disclosure as recited in the appended claims.

A conventional camera may be configured to record a video or take a picture, collect brightness information and color information of a scenario, but cannot collect depth information. Along with increase of application requirements, depth cameras have been added to cameras of some electronic devices to form camera arrays. The depth camera may include an array camera module, a structured light module and a TOF module, and depth information may be obtained according to a working principle of each module. However, the camera array requires independent arrangement of the depth camera, which results in occupation of a valuable space of the electronic device and is unfavorable for miniaturization and cost reduction of the electronic device.

To solve the above technical problem, embodiments of the present disclosure provide an image processing method and device, a camera component, an electronic device, and a storage medium. The inventive concept is that: a first camera module configured to sense first-band light and acquire a first image and a second camera module configured to sense the first-band light and second-band light and acquire a second image are arranged in a camera module array, and a processor may perform image processing, for example, acquisition of a depth image, on at least one of a bayer subimage or an infrared subimage in the second image and the first image.

<FIG> is a block diagram of a camera component, according to an exemplary embodiment. Referring to <FIG>, the camera component may include a first camera module <NUM>, a second camera module <NUM>, an infrared light source <NUM> and a processor <NUM>. The first camera module <NUM> may sense first-band light, and the second camera module <NUM> may sense the first-band light and second-band light. The processor <NUM> is coupled with the first camera module <NUM>, the second camera module <NUM> and the infrared light source <NUM> respectively. The term being coupled with refers to that the processor <NUM> may send a control instruction and acquire images from the camera modules <NUM>, <NUM>, and may specifically be implemented through a communication bus, a cache or a wireless manner. No limits are made herein.

The first camera module <NUM> is configured to generate a first image under the control of the processor <NUM>. The first image may be a red green blue (RGB) image.

The infrared light source <NUM> is configured to emit the second-band light under the control of the processor <NUM>.

The second camera module <NUM> is configured to generate a second image under the control of the processor <NUM>. The second image may include a bayer subimage generated by sensing the first-band light and an infrared subimage generated by sensing the second-band light.

The processor <NUM> is configured to acquire the bayer subimage and the infrared subimage according to the second image, and perform image processing on at least one of the bayer subimage or the infrared subimage and the first image.

Exemplarily, in the embodiments, the first-band light may be light of a visible light band, and the second-band light may be light of an infrared band.

In the embodiments, the first camera module <NUM> may include an image sensor, a camera lens, an infrared filter and other elements responding to the first-band light (for example, the visible light band), and may further include a voice coil motor, a circuit substrate and other elements. The image sensor may respond to the first-band light by use of a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like. For facilitating cooperative use of the first camera module <NUM> and the second camera module <NUM>, a filter in the image sensor of the first camera module <NUM> may be a color filter array only responding to the visible light band such as a bayer template, a cyan yellow yellow magenta (CYYM) template, a cyan yellow green magenta (CYGM) template, and the like. A mounting position and working principle of each element of the first camera module <NUM> may refer to a related art and will not be repeated herein.

In the embodiments, the second camera module <NUM> may include an image sensor, a camera lens, a visible light-near infrared bandpass filter and other elements responding to both the first-band light (for example, light of the visible light band) and the second-band light (for example, light of the infrared band), and may further include a voice coil motor, a circuit substrate and other elements. The image sensor responding to both the first-band light and the second-band light may be implemented by use of the CCD, the CMOS, or the like, and a filter thereof may be a color filter array responding to both the visible light band and the infrared band such as an RGB infrared (RGBIR) template, an RGB white (RGBW) template, and the like. A mounting position and working principle of each element of the second camera module <NUM> may refer to the related art and will not be repeated herein.

In the embodiments, the infrared light source <NUM> may include at least one of an infrared flood light source, a structured light source or a TOF light source. A working principle of the infrared flood light source is to increase infrared illuminating brightness to an object in a framing range. A working principle of the structured light source is to project specific light information to a surface of the object and a background and calculate information of a position, depth and the like of the object according to a change of a light signal caused by the object. A working principle of the TOF light source is to project an infrared pulse into the framing range and calculate a distance of the object according to round-trip time of the infrared pulse.

In the embodiments, the processor <NUM> may be implemented by an independently arranged microprocessor, and may also be implemented by a processor of an electronic device with the camera component. The processor <NUM> has functions of the following two aspects.

In a first aspect, an operation signal of one or combination of a button, a microphone (MIC) and the image sensor may be received to control the first camera module <NUM>, the second camera module <NUM> and the infrared light source <NUM>. For example, when the electronic device is in a normal photographic mode, the processor <NUM> may adjust a parameter of the first camera module <NUM>, such as a focal length, brightness, and the like. When detecting a shutter pressing action of a user, the processor <NUM> may control the first camera module <NUM> to take a picture. When the electronic device is in a panorama, high-dynamic range (HDR), entire focus or other mode or in a low-light scenario, the processor <NUM> may turn on/activate the infrared light source <NUM>, simultaneously adjust parameters of the first camera module <NUM> and the second camera module <NUM>, such as focal lengths, brightness and the like. When detecting the shutter pressing action of the user, the processor <NUM> may control the first camera module <NUM> to take a picture to obtain the first image and control the second camera module <NUM> to take a picture to obtain the second image.

In a second aspect, when a depth image is required, as illustrated in <FIG>, the electronic device may process the first image and the second image to acquire a visible light depth image or a depth fused image.

In an example, the processor may extract the bayer subimage corresponding to the first-band light from the second image, and calculate the visible light depth image according to the first image and the bayer subimage. The calculating process is as follows.

Referring to <FIG>, P is a certain point of an object to be detected (i.e., a shooting target) in a framing range, CR and CL are optical centers of a first camera and a second camera respectively, imaging points of the point P on light sensors of the two cameras are PR and PL respectively (image planes of the cameras are positioned in front of the camera lenses after rotation), f is a focal length of the camera, B is a center distance of the two cameras, and Z is a depth to be detected. If a distance between the point PR and the point PL is set to be D:
<MAT>.

According to a principle of similar triangles:
<MAT>
it may be obtained that:
<MAT>.

The focal length f, the center distance B of the cameras, a coordinate XR of the point P on the right image plane and a coordinate XL of the point P on the left image plane may be obtained by calibration, therefore it is necessary to obtain (XR-XL) to obtain the depth, where f, B, XR and XL may be determined by calibration, correction and matching operations. The calibration, correction and matching operations may refer to the related art and will not be repeated herein.

The processor <NUM> may repeat the abovementioned steps to obtain depths of all pixels in the first image to obtain the visible light depth image. The visible light depth image may be configured for service scenarios including a large aperture, face/iris unlocking, face/iris payment, three-dimensional (3D) retouching, studio lighting, Animoji, and the like.

In an example, fields of view of each camera in the first camera module <NUM> and the second camera module <NUM> may be different, and a magnitude relationship of them is not limited. In such case, the processor <NUM> may crop images in corresponding sizes from the first image and the second image in combination with the fields of view of the two cameras. For example, a frame of image in a relatively large size is cropped from the bayer subimage extracted from the second image, and then a frame of image in a relatively small size is cropped from the first image, that is, the image cropped from the bayer subimage of the second image is larger than the image cropped from the first image. Then, the images are sequentially displayed. Therefore, a zooming effect may be achieved, that is, a shooting effect like optical zooming may be achieved in the embodiments, which is favorable for improving a shooting experience.

In another example, considering that the second image further includes infrared information, the processor <NUM> extracts the infrared subimage generated by sensing the second-band light from the second image. Since high-frequency information in a frequency domain of the infrared subimage is richer than information in a frequency domain of the first image, the infrared subimage and the first image are fused, the high-frequency information of the infrared subimage is extracted and added to the frequency domain of the first image, to achieve an effect of enhancing the first image and ensure that the fused first image has richer details and is higher in resolution and more accurate in color. In addition, the infrared subimage may further be configured for a biometric recognition function of the electronic device, for example, fingerprint unlocking, face recognition and other scenarios.

In another example, considering that the infrared light source may be the structured light source, still referring to <FIG>, under the condition that the infrared light source includes the structured light source, the processor <NUM> may further acquire infrared depth data based on the infrared subimage. For example, the processor <NUM> may control the infrared light source <NUM> to project a light beam of a specific direction to a shooting target such as an object or a background, and acquire a parameter of an echo signal of the light beam, such as strength, a spot size or the like. The processor <NUM> may obtain infrared depth data from the shooting target to the camera based on a preset corresponding relationship between a parameter and a distance, and the infrared depth data is relative to the visible light depth image and may include texture information of the shooting target such as the object or the background. In such case, the processor <NUM> may select to use the visible light depth image or the infrared depth data according to a specific scenario. For example, the visible light depth image may be used in a high-light scenario (that is, an ambient brightness value is greater than a preset brightness value, like a daytime scenario), in a scenario that the shooting target is semitransparent or in a scenario that the shooting target absorbs infrared light. For example, the infrared depth data may be used in a low-light scenario (that is, the ambient brightness value is smaller than the preset brightness value, like a night scenario), in a scenario that the shooting target is a texture-less object or in a scenario that the shooting target is an object that periodically appears. The visible light depth image or the infrared depth data may also be fused to obtain the depth fused image. The depth fused image may compensate various defects of the visible light depth image and the infrared depth data, may be applied to almost all scenarios, particularly to scenarios of a poor illumination condition, the texture-less object, the periodically appearing object or the like, and is favorable for improving the confidence of the depth data.

In another example, considering that the infrared light source may be the TOF light source, still referring to <FIG>, under the condition that the infrared light source includes the TOF light source, the processor <NUM> may further acquire the infrared depth data based on the infrared subimage, and the infrared depth data is relative to the visible light depth image and may include the texture information of the shooting target such as the object or the background. For example, the processor <NUM> may control the TOF light source to project a light beam of a specific direction to the object or the background, acquire a time difference between emission time and return time of an echo signal of the light beam, and calculate the infrared depth data from the object to the camera. In such case, the processor <NUM> may select to use the visible light depth image or the infrared depth data according to a specific scenario. For example, the visible light depth image may be used in a high-light scenario (that is, an ambient brightness value is greater than a preset brightness value, like a daytime scenario), in a scenario that the shooting target is semitransparent or in a scenario that the shooting target absorbs infrared light. For example, the infrared depth data may be used in a low-light scenario (that is, the ambient brightness value is smaller than the preset brightness value, like a night scenario), in a scenario that the shooting target is a texture-less object or in a scenario that the shooting target is an object that periodically appears. The visible light depth image or the infrared depth data may also be fused to obtain the depth fused image. The depth fused image may compensate defects of the visible light depth image and the infrared depth data, may be applied to almost all scenarios, particularly to scenarios that an illumination condition is poor, the shooting target is the texture-less object, the periodically appearing object or the like, and is favorable for improving the confidence of the depth data.

It is to be noted that, in the embodiments, the structured light source or the TOF light source is selected, and improvement or addition of camera modules is not involved, such that difficulties in design may be greatly reduced.

Herein, in the embodiments of the present disclosure, the first camera module in the camera component may collect the first image, the second camera module acquires the second image, the bayer subimage and the infrared subimage may be acquired from the second image, and then image processing, for example, acquisition of the depth image, may be performed on at least one of the bayer subimage or the infrared subimage and the first image, that is, the depth image may be acquired without arranging any depth camera in a camera module array, such that the size of the camera component may be reduced, a space occupied by it in the electronic device may be reduced, and miniaturization and cost reduction of the electronic device are facilitated.

The embodiments of the present disclosure also provide an image processing method. <FIG> is a flow chart showing an image processing method, according to an exemplary embodiment. Referring to <FIG>, the image processing method is applied to the camera component provided in the abovementioned embodiments and may include the following steps.

In step <NUM>, a first image generated by a first camera module and a second image generated by a second camera module are acquired, and the second image includes a bayer subimage generated by the second camera module by sensing first-band light and an infrared subimage generated by sensing second-band light.

In step <NUM>, image processing is performed on at least one of the bayer subimage or the infrared subimage and the first image.

In an embodiment, the operation in step <NUM> that the image processing is performed on at least one of the bayer subimage or the infrared subimage and the first image may include: fusing the infrared subimage and the first image to enhance the first image.

In an embodiment, the operation in step <NUM> that the image processing is performed on at least one of the bayer subimage or the infrared subimage and the first image may include: acquiring a visible light depth image according to the bayer subimage and the first image.

In an embodiment, when an infrared light source includes a structured light source or a TOF light source, the operation in step <NUM> that the image processing is performed on at least one of the bayer subimage or the infrared subimage and the first image may include: fusing the visible light depth image and depth data of the infrared subimage to obtain a depth fused image.

In an embodiment, the method may further include: responsive to a zooming operation of a user, performing image zooming based on the first image and the bayer subimage.

It can be understood that the method provided in each embodiment of the present disclosure is matched with a working process of the camera component and specific contents may refer to the contents of each embodiment of the camera component and will not be repeated herein.

The embodiments of the present disclosure also provide an image processing device, referring to <FIG>, which may include: an image acquisition module <NUM> and an image processing module <NUM>.

The image acquisition module <NUM> is configured to acquire a first image generated by a first camera module and a second image generated by a second camera module, and the second image includes a bayer subimage generated by the second camera module by sensing first-band light and an infrared subimage generated by sensing second-band light.

The image processing module <NUM> is configured to perform image processing on at least one of the bayer subimage or the infrared subimage and the first image.

In an embodiment, the image processing module <NUM> may include: an image enhancement unit, configured to fuse the infrared subimage and the first image to enhance the first image.

In an embodiment, the image processing module <NUM> may include: a depth image acquisition unit, configured to acquire a visible light depth image according to the bayer subimage and the first image.

In an embodiment, when an infrared light source includes a structured light source or a TOF light source, the image processing module includes: a depth fusion unit, configured to fuse the visible light depth image and depth data of the infrared subimage to obtain a depth fused image.

In an embodiment, the device may further include: a zooming module, configured to, responsive to a zooming operation of a user, perform image zooming based on the first image and the bayer subimage.

It can be understood that the device provided in each embodiment of the present disclosure corresponds to the method embodiments and specific contents may refer to the contents of each method embodiment and will not be repeated herein.

<FIG> is a block diagram of an electronic device, according to an exemplary embodiment. For example, the electronic device <NUM> may be a smart phone, a computer, a digital broadcast terminal, a tablet, a medical device, exercise equipment, a personal digital assistant, and the like.

Referring to <FIG>, the electronic device <NUM> may include one or more of the following components: a processing component <NUM>, a memory <NUM>, a power component <NUM>, a multimedia component <NUM>, an audio component <NUM>, an input/output (I/O) interface <NUM>, a sensor component <NUM>, a communication component <NUM> and an image collection component <NUM>.

The processing component <NUM> typically controls overall operations of the electronic device <NUM>, such as the operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component <NUM> may include one or more processors <NUM> to execute computer programs. Moreover, the processing component <NUM> may include one or more modules which facilitate interaction between the processing component <NUM> and other components. For instance, the processing component <NUM> may include a multimedia module to facilitate interaction between the multimedia component <NUM> and the processing component <NUM>.

The memory <NUM> is configured to store various types of data to support the operation of the electronic device <NUM>. Examples of such data include computer programs for any applications or methods operated on the electronic device <NUM>, contact data, phonebook data, messages, pictures, video, etc. The memory <NUM> may be implemented by any type of volatile or non-volatile memory devices, or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, and a magnetic or optical disk.

The power component <NUM> provides power for various components of the electronic device <NUM>. The power component <NUM> may include a power management system, one or more power supplies, and other components associated with generation, management and distribution of power for the electronic device <NUM>. The power component <NUM> may include a power chip, and a controller may communicate with the power chip, thereby controlling the power chip to turn on or turn off a switching device to cause or disable a battery to supply power to a mainboard circuit.

The multimedia component <NUM> includes a screen providing an output interface between the electronic device <NUM> and a target object. If the screen includes the TP, the screen may be implemented as a touch screen to receive an input signal from the target object. The TP includes one or more touch sensors to sense touches, swipes and gestures on the TP. The touch sensors may not only sense a boundary of a touch or swipe action, but also detect a period of time and a pressure associated with the touch or swipe action.

The audio component <NUM> is configured to output and/or input an audio signal. For example, the audio component <NUM> includes a MIC, and the MIC is configured to receive an external audio signal when the electronic device <NUM> is in an operation mode, such as a call mode, a recording mode and a voice recognition mode. The received audio signal may further be stored in the memory <NUM> or sent through the communication component <NUM>. In some embodiments, the audio component <NUM> further includes a speaker configured to output the audio signal.

The sensor component <NUM> includes one or more sensors configured to provide status assessments in various aspects for the electronic device <NUM>. For instance, the sensor component <NUM> may detect an on/off status of the electronic device <NUM> and relative positioning of components, such as a display screen and small keyboard of the electronic device <NUM>, and the sensor component <NUM> may further detect a change in a position of the electronic device <NUM> or a component, presence or absence of contact between the target object and the electronic device <NUM>, orientation or acceleration/deceleration of the electronic device <NUM> and a change in temperature of the electronic device <NUM>.

The communication component <NUM> is configured to facilitate wired or wireless communication between the electronic device <NUM> and other devices. The electronic device <NUM> may access a communication-standard-based wireless network, such as a wireless fidelity (WiFi) network, a 2nd-generation (<NUM>) or 3rd-generation (<NUM>) network or a combination thereof. In an exemplary embodiment, the communication component <NUM> receives a broadcast signal or broadcast associated information from an external broadcast management system through a broadcast channel. In an exemplary embodiment, the communication component <NUM> further includes a near field communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra-wide band (UWB) technology, a Bluetooth (BT) technology, and other technologies.

The image collection component <NUM> is configured to collect images. For example, the image collection component <NUM> may be implemented by the camera component provided in the abovementioned embodiments.

In an exemplary embodiment, the electronic device <NUM> may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components.

In an exemplary embodiment, there is also provided a non-transitory readable storage medium including an executable computer program, such as the memory <NUM> including instructions, and the executable computer program may be executed by the processor. The readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disc, an optical data storage device, and the like.

Other implementation solutions of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. This present disclosure is intended to cover any variations, uses, or adaptations following the general principles thereof and including such departures from the present disclosure as come within known or customary practice in the art. It is intended that the specification and examples be considered as exemplary only, with the scope of the invention being defined by the following claims.

Claim 1:
A camera component, wherein the camera component comprises a first camera module (<NUM>) sensing first-band light, a second camera module (<NUM>) sensing the first-band light and second-band light, an infrared light source (<NUM>) emitting the second-band light, and a processor (<NUM>); wherein the processor (<NUM>) is coupled with the first camera module (<NUM>), the second camera module (<NUM>) and the infrared light source (<NUM>) respectively;
the first camera module (<NUM>) is configured to generate a first image under control of the processor (<NUM>);
the infrared light source (<NUM>) is configured to emit the second-band light under the control of the processor (<NUM>);
the second camera module (<NUM>) is configured to generate a second image under the control of the processor (<NUM>), the second image comprising a bayer subimage generated by sensing the first-band light and an infrared subimage generated by sensing the second-band light; and
the processor (<NUM>) is further configured to perform image processing on at least one of the bayer subimage or the infrared subimage and the first image;
characterized in that
the processor is configured to perform the image processing on the infrared subimage and the first image by fusing the infrared subimage and the first image to enhance the first image, wherein the processor is configured, therefor, to extract high-frequency information of the infrared subimage and add the extracted high-frequency information into a frequency domain of the first image.