Patent Description:
With the advancement of technology, mobile phones, digital cameras, tablet computers, notebook computers and other portable electronic devices have become an indispensable to our daily lives. Since portable electronic devices are mostly for personal uses, they contain a certain level of private information, and hence, the data stored therein, such as the contacts, photos, personal information, etc. are privately owned. Once the electronic device is lost, these data may be used by others, thereby resulting in unnecessary losses and damages. Although one can use the password protection to prevent others from using the electronic device, passwords can be leaked or cracked easily, and hence the security level of password protection is relatively low. Also, users have to remember the password in order to use electronic devices, and if the user forgets your password, it will cause a lot of inconvenience. Therefore, personal fingerprint identification systems have been developed to achieve the purpose of authentication so as to enhance data security.

On the other hand, with the advancement of fingerprint identification technology, invisible fingerprint sensors (IFS) have gained increasing favor from consumers, gradually. In the invisible fingerprint sensor technology, an optical fingerprint sensor may be disposed under the touch screen (i.e., an under-display fingerprint sensing). In other words, the user can press through the touch screen for implementing fingerprint identification.

Further, the prior art has developed a background subtraction technology, which can remove the background image so that the image of the meaningful signals is more significant. However, in the application of optical fingerprint identification, the touch screen not only has the display components arranged in an array (such as an organic light emitting diode (OLED)) but also has components or materials, such as an indium tin oxide (ITO) transparent conductive film, a conductive silver paste, an ITO substrate, an optical clear adhesive (OCA) optical gel, etc., related to the touch control function disposed on the display component. Since the signals between finger ridges and finger valleys of the fingerprint image are quite small, and the light transmittances of the above-mentioned components/materials related to the touch control function at each pixel position are different, plus the light reflectivity of the user's finger and the light reflectivity of the background object used to create the background image are different, interference to the fingerprint image takes place when implementing the optical fingerprint identification. In other words, because of the difference between the reflectivity of the user's finger and the reflectivity of the background object used to create the background image, the application of the existing background subtraction technology alone cannot effectively eliminate the interference, thereby jeopardizing the accuracy of the optical fingerprint identification. <CIT> discloses a fingerprint image processing method, applied in an optical fingerprint identification system of an electronic device, wherein the electronic device comprises a display panel, the display panel comprises a display pixel array, the optical fingerprint identification system comprises an image sensing array, the image sensing array is disposed under the display pixel array, the fingerprint image processing method obtaining a background image and obtaining at least an interfering frequency when the display panel is not pressed by a finger of a user, wherein no fingerprint image is included in the background image; receiving a received image when the display panel is pressed by the finger of the user; performing a subtracting operation on the received image and the background image, to obtain a difference image; and performing a filtering operation on the difference image at the at least an interfering frequency, to obtain an operational result; wherein the optical fingerprint identification system determines a fingerprint within the received image according to the operational result. <CIT> discloses performing background noise calculation based on an original vacant image, collecting an original fingerprint image and removing the background noise in the original fingerprint image according to the calculation result of the background noise. <CIT> discloses operating a fingerprint sensor for reducing or eliminating undesired contributions from the TFT influence and the background light in fingerprint sensing based on obtaining a background light eliminated frame by subtracting the signals of a frame B from the signals of a frame A, wherein the frame A is captured with the OLED display turned on to illuminate the finger touching area and the frame B is captured with the illumination changed or turned off. A signal frame is then obtained by subtracting the background light eliminated frame and a reference frame captured without fingerprint signals.

In view of the foregoing, one goal of the related field aims to provide a background subtraction technology that can effectively eliminate the interference.

In view of the foregoing, the purpose of some embodiments of the present invention is to provide a background subtraction method, image module and optical fingerprint identification system capable of effectively eliminating the interference so as to address the disadvantages in the existing art.

The present invention uses objects having different reflectivity/color to create a first background image and a second background image, and uses a mask image generated according to the first background image and the second background image, so as to use a first background-removed image generated from a first stage background subtraction (the existing background subtraction technology) calculation to further implement a second stage background subtraction calculation. Compared with the existing technology, the present invention may further eliminate the interference to the image.

To further explain the purposes, technical solutions and advantages of the present application, the appended drawings and embodiments are discussed below to give a detailed description of the present invention. It should be noted that the embodiments provided herein are used to explain the present invention, and shall not be used to limit the present application.

In the specification and claims of the present invention, implementing or performing the addition, subtraction, multiplication and division between an image A and an image B means implementing or performing the addition, subtraction, multiplication and division between the elements of the image A and image B. More specifically, multiplying the image A by image B (denoting as A*B) means multiplying the (i,j)th pixel value ai,j of the image A by the (i,j)th pixel value bi,j of the image B; dividing the image A by the image B (denoting as A/B) means dividing the (i,j)th pixel value ai,j of the image A by the (i,j)th pixel value bi,j of the image B; adding the image A and the image B (denoting as A+B) means adding up the (i,j)th pixel value ai,j of the image A and the (i,j)th pixel value bi,j of the image B; subtracting the image B from the image A (denoting as A-B) means subtracting the (i,j)th pixel value bi,j of the image B from the (i,j)th pixel value ai,j of the image A. The averaging value of the image A means the pixel value obtained by averaging all the pixel values of the image A.

Please refer to <FIG> is a schematic diagram of an electronic device <NUM> according to embodiments of the present invention; <FIG> is a schematic diagram of an image module <NUM> according to embodiments of the present invention. The electronic device <NUM> comprises an optical fingerprint identification system <NUM> and a touch screen <NUM>, wherein the optical fingerprint identification system <NUM> comprises the image module <NUM> and a fingerprint identification module <NUM>, and the image module <NUM> comprises an image capturing unit <NUM> and a background subtraction unit <NUM>. The image module <NUM> is disposed under the touch screen <NUM> and coupled to the fingerprint identification module <NUM>. The optical fingerprint identification system <NUM> may implement the under-display fingerprint sensing; that is, the user may implement fingerprint identification by pressing the touch screen <NUM>.

More specifically, the image capturing unit <NUM> of the image module <NUM> is disposed under the touch screen <NUM> directly, and the background subtraction unit <NUM> may be disposed under the image capturing unit <NUM> or at other positions (but still under the touch screen <NUM>). The image capturing unit <NUM> may comprise a plurality of light-sensing devices (e.g., photo diodes (PD)) arranged in an array, and the image capturing unit <NUM> is configured to capture the image imaged on plurality of light-sensing devices by the light reflected by the touch screen <NUM> from the region RGN. The background subtraction unit <NUM> may be implemented using a digital circuit or a digital signal processor (DSP), and the background subtraction unit <NUM> receives the image captured by the image capturing unit <NUM> and implements background subtraction to a target image F captured by the image capturing unit <NUM> so as to generate a background-removed image R, thereby eliminating the interference to the target image F; in this way, the fingerprint identification module <NUM> may implement fingerprint identification to the background-removed image R. In this example, the interference to the target image F may be the interference to the fingerprint image during the optical fingerprint identification, which is caused by the difference in the light transmittance of the touch-related component/material/circuit (such as the indium tin oxide (ITO) transparent conductive film, conductive silver adhesive, ITO substrate, optical clear adhesive (OCP) optical glue and the like) in the touch screen <NUM> at different pixel positions, and the reflectivity of the user's finger to light is different from the reflectivity, in a circumstance where the background image is created, of the background object.

Please refer to <FIG> is a schematic diagram of a background subtraction flow <NUM> according to embodiments of the present invention; the background subtraction flow <NUM> is implemented by the image module <NUM> to generate the background-removed image R, wherein the background subtraction flow <NUM> comprises the following steps:.

In the step <NUM>, the image module <NUM> records a first background image B <NUM> corresponding to a first object and a second background image B2 corresponding to a second object. The person operating the electronic device <NUM> covers a first object (which may be a uniformly all-black object having a first reflectivity) over a fingerprint sensing region (i.e., the region RGN above the image module <NUM>) of the touch screen <NUM>. At this time, the image capturing unit <NUM> captures at least one first image corresponding to the first object/all-black object. Moreover, the operator covers a second object with a uniform color (which may be a uniformly all-white object having a second reflectivity) over the fingerprint sensing region (i.e., the region RGN above the image module <NUM>) of the touch screen <NUM>. At this time, the image capturing unit <NUM> captures at least one second image corresponding to the second object/all-white object. After the image capturing unit <NUM> captures the at least one first the image and the at least one second image, the background subtraction unit <NUM> calculates an average of the at least one first image so as to remove noise in the first image, thereby generating the first background image B1. Moreover, the background subtraction unit <NUM> calculates an average of the at least one second image, so as to remove noise in the second image, thereby generating the second background image B2. The first background image B1 and the second background image B2 correspond respectively to the first object and the second object having different reflectivity (or different object colors). That is, the first reflectivity and the second reflectivity are different, and the first object and the second object are considered as the background object.

In the step <NUM>, the background subtraction unit <NUM> calculates a plurality of relative values of the first background image B1 relative to the second background image B2, so as to obtain a mask image M. The background subtraction unit <NUM> calculates the mask image M by dividing the first background image B1 by the second background image B2; in this way, the (i,j)th mask pixel value mi,j of the mask image M is the relative value of the value of the (i,j)th pixel of background image B1 relative to the value of the (i,j)th pixel of the second background image B2, i.e., mi,j = b1i,j/b2i,j, wherein b1i,j and b2i,j respectively represent the (i,j)th pixel value of the first background image B1 and mean the (i,j)th pixel value of the second background image B2. The plurality of pixel values of the mask image M (that is, the plurality of relative values of the first background image B1 relative to the second background image B2) reflect the combine effect of the difference in the background object reflectivity and the different light transmittance of the touch-related component/material at different pixel positions. In one embodiment, the mask image M may be subjected to a normalization calculation so that each mask pixel value of the mask image M is between <NUM> and <NUM>.

In the step <NUM>, the image capturing unit <NUM> captures a target image F. When the optical fingerprint identification system <NUM> implements optical fingerprint identification, the user presses his/her finger(s) on the fingerprint sensing region (that is, the region RGN above the image module <NUM>) of the touch screen <NUM>, the target image F thus captured by the image capturing unit <NUM> comprises the fingerprint image.

In the step <NUM> (which may be considered as a first stage background subtraction calculation), the background subtraction unit <NUM> subtracts the second background image B2 from the target image F, so as to obtain a first background-removed image X, wherein the first background-removed image X may be expressed as X=F- B2.

In the step <NUM> (which may be considered as a second stage background subtraction calculation), the background subtraction unit <NUM> calculates and outputs a second background-removed image R according to the first background image B1, the second background image B2, the first background-removed image X and the mask image M. More specifically, the background subtraction unit <NUM> multiplies the first background-removed image X by the mask image M, so as to obtain a first mask-multiplied image W (the first mask-multiplied image W may be expressed as W=X*M). Next, the background subtraction unit <NUM> calculates a compensated image C according to the first background-removed image X and the first mask-multiplied image W. Lastly, the background subtraction unit <NUM> adds the first background-removed image X and the compensated image C, thereby obtaining a second background-removed image R, wherein the second background-removed image R may be expressed as R=X+C. The compensated image C is positively proportional to the first mask-multiplied image W. That is, the compensated image C may be expressed as C=g*W, whereas the background subtraction unit <NUM> may calculate a compensation coefficient g according to the first background-removed image X and the first mask-multiplied image W, thereby calculating the compensated image C.

More specifically, in order to calculate the compensated image C, the background subtraction unit <NUM> may generate an inverse mask image N according to the mask image M, wherein the inverse mask image N is an inverse image of the mask image M. In other words, if the mask image M is very bright at the position of the (i,j)th pixel, the inverse mask image N is very dark at the position of the (i,j)th pixel, vice versa. In one embodiment, the background subtraction unit <NUM> may subtract the mask image M from a uniformly all-white the image AWH to generate the inverse mask image N, wherein the inverse mask image N may be expressed as N=AWH-M. In the case that each mask pixel value in the mask image M is between <NUM> and <NUM>, the pixel value of each pixel in the all-white the image AWH is <NUM>. The background subtraction unit <NUM> may multiply the first background-removed image X by the inverse mask image N, so as to obtain a first inverse mask-multiplied image B (the first inverse mask-multiplied image B may be expressed as B=X*N). In one embodiment, the background subtraction unit <NUM> may calculate a compensated image C/ a compensation coefficient g, so that the averaging value of pixels in any region of the image R*M is a constant value, or the averaging value of pixels in any region of the image R*N is another constant value.

To this end, the background subtraction unit <NUM> may again multiply the first inverse mask-multiplied image B by the inverse mask image N, so as to obtain a second inverse mask-multiplied image BN, wherein the second inverse mask-multiplied image BN may be expressed as BN =B*N =X*N*N. On the other hand, the background subtraction unit <NUM> may again multiply the first mask-multiplied image W by the mask image M, so as to obtain a second mask-multiplied image WM, wherein the second mask-multiplied image WM may be expressed as WM = W*M = X*M*M. The background subtraction unit <NUM> may calculate a compensation coefficient g according to Equation <NUM>. That is, the background subtraction unit <NUM> calculates the compensation coefficient g according to the first mask-multiplied image W, the second mask-multiplied image WM, the first inverse mask-multiplied image B and the second inverse mask-multiplied image BN. In this example, mean(W), mean(B), mean(WM), and mean(BN) respectively mean the first mask-averaged value, the first inverse mask-averaged value, the second mask-averaged value, and the second inverse mask-averaged value corresponding to the first mask-multiplied image W, the first inverse mask-multiplied image B, the second mask-multiplied image WM, and the second inverse mask-multiplied image BN.

In this way, after the background subtraction unit <NUM> calculates the compensation coefficient g, the background subtraction unit <NUM> may calculate the compensated image C as C=g*W, and calculate and output a second background-removed image R as R=X+ g*W=X(<NUM>+g*M).

Please refer to <FIG> is a schematic diagram of background images B1, B2, a mask image M, an inverse mask image N, target images F1 and F2, first background-removed images X1 and X2, and second background-removed images R1 and R2 according to embodiments of the present invention. In this example, the target image F1 is the target image F captured when the operator of the electronic device <NUM> covers an object of horizontal strips over the fingerprint sensing region (RGN) of the touch screen <NUM>. The target image F2 is the target image F captured when the operator actually presses his/her finger(s) on the fingerprint sensing region (RGN). The first background-removed images X1 and X2 are results obtained from performing the step <NUM> to the target images F <NUM> and F2, respectively. The second background-removed images R1 and R2 are results obtained from performing the step <NUM> to the first background-removed images X1 and X2, respectively.

As can be seen from <FIG>, the mask image M and the inverse mask image N are in the diamond shape, which reflects the image imaged by the touch-related components/materials/circuits, wherein some pixel positions in the mask image M is darker (having a smaller mask pixel value) whereas some pixel positions are brighter (having a larger mask pixel value). The inverse mask image N has a larger inverse mask pixel value at pixel positions with a smaller mask pixel value, and has a smaller inverse mask pixel value at pixel positions with a larger mask pixel value. The target image F1 has fine horizonal stripe signals, whereas the target image F2 has fine fingerprint signals. The first background-removed image X1 comprises a combined image of the image formed from the diamond-shaped circuit structure and the horizontal strips signals, whereas the first background-removed image X2 comprises a combined image of the image formed from the diamond-shaped circuit structure and the fingerprint signal. As can be ascertain from the second background-removed images R1 and R2, the image module <NUM> may effectively eliminate the image, corresponding to the diamond-shaped circuit structure, in the first background-removed images X1 and X2 by performing the step <NUM>, so that the second background-removed image R1 has only the horizontal stripe signals, and the second background-removed image R2 has only the fingerprint signals.

Moreover, the first background-removed images X1 and X2 generated from the subtraction calculation of the first stage background are, in fact, the execution results of the existing background subtraction technology. In addition to the first stage background subtraction calculation, the present invention further implements a second stage background subtraction calculation (that is, the step <NUM>). As can be seen from <FIG>, the second background-removed images R1 and R2 generated by the second stage background subtraction calculation according to the present invention may further eliminate the interference to the image, as compared with the existing technology.

Moreover, to increase the convenience to users, the image module <NUM> may, during a calibration stage before the product delivery, first calculate the background images B1 and B2, mask image M and inverse mask image N, and store the same in a storage unit (not illustrated in <FIG>). After the product delivery, when the user presses his/her finger(s) on the fingerprint sensing region RGN, the image module <NUM> may capture the target image F in real-time and calculate the first background-removed image X and the second background-removed image R.

The following modifications to the above-mentioned embodiments are possible. For example, the mask image M is not limited to the quotient of the first background image B1 and the second background image B2. In one embodiment, the mask image M may be M = (B1*B1) /(B2*B2) = B1<NUM>/B2<NUM>, or M = B1n/B2n, wherein B1n(B2n) represents the B1(B2) to the nth power, as long as a mask value in the mask image M may show/represent the relative value of the first background image B <NUM> relative to the second background image B2. These variations all belong to the scope of the present invention. Moreover, when recording the background images B1 and B2, the image module <NUM> may first implement a temperature compensation operation, so as to reduce the effect of the temperature on the background image B1 and B2. Moreover, considering the cases where the user does not press his/her finger(s) on the whole fingerprint sensing region RGN, the image module <NUM> may first implement image segmentation to the image captured by the image capturing unit <NUM>, and the segmented images may be used as the target image F. Moreover, the background subtraction unit <NUM> may implement the binarization calculation to the mask image M, and the use of the binarized mask image M to calculate the second background-removed image R also falls within the scope of the present invention. Moreover, the first object and the second object are not limited to those of black or white and may have another color, as long as the first object and the second object have different reflectivity to light. These variations also fall within the scope of the present invention.

Claim 1:
A background subtraction method implemented by an image module, wherein the image module is disposed under a fingerprint sensing region of a touch screen (<NUM>) of an optical fingerprint identification system (<NUM>), and the background subtraction method comprises:
recording at least one first image of a first object, at least one second image of a second object and a target image (F), wherein the first object has a uniform color and has a first reflectivity, the second object has a uniform color and has a second reflectivity, and the first reflectivity and the second reflectivity are different, wherein the first object covers the image module when the at least one first image is captured and the at least one second object covers the image module when the at least one second image is captured;
generating a first background image (B1) by calculating an average of the at least one first image so as to remove noise in the first image;
generating a second background image (B2) by calculating an average of the at least one second image so as to remove noise in the second image;
dividing a plurality of first pixel values of the first background image (B1) to the power of one or more by a plurality of second pixel values of the second background image (B2) to the power of one or more to obtain a plurality of relative values of the first background image relative to the second background image (B2) to obtain a mask image (M);
subtracting the second background image (B2) from the target image (F) to obtain a first background-removed image (X); and
calculating and outputting a second background-removed image (R), wherein a first mask-multiplied image (W) is obtained by multiplying the first background-removed image (X) by the mask image (M), wherein a compensated image (C) is calculated to be positively proportional to the first mask-multiplied image (W), and wherein the second background-removed image (R) is calculated by adding the first background-removed image (X) and the compensated image (C).