Patent Description:
If a camera lens is fully functional and without defects, it will reflect or pass light in relatively uniform fashion. However, if the camera lens is damaged (e.g. scratched), the light passing through it will be distorted or offset in some way and this will affect the quality of the resulting image.

There are also some known optical effects that may make the captured image different from what is photographed. For example, most lenses have at least some vignetting, which means that the corners of a picture are less luminous than the centre of the picture. However, generally this error causes a luminosity difference of less than one Exposure Value (EV) between the corners and the centre and it is therefore not usually noticeable. Other well-known and common lens errors are barrel and pincushion distortions, which also alter the captured image, but again these errors are generally not strongly visible in photographs taken with a camera. Thus, although these types of optical effects can have an impact on a resulting photograph they, generally, do not mask a distortion caused by a lens defect.

Commonly, surface defects or structural defects in a camera lens cannot be reliably detected using existing solutions, as the resulting effect in a photograph is often too slight to notice. Even with a severely damaged lens, the resulting photo might still be above an acceptable level for a naked eye.

Current methods for diagnosing an in-built camera module of a mobile device prompt the user to take any photograph using the camera and then visually inspect that photograph. This method is hard to automate for a diagnostic system and cannot be used to detect surface defects or structural defects in the lens reliably.

Even if the user were to look at the lens itself, any surface defects or structural defects might not be visible to the naked eye.

<CIT> discloses a system for diagnosing camera or display defects in a mobile device wherein the device display is arranged as a light source which is directed to the camera using a series of mirrors. However, this method cannot be used to diagnose the camera component individually, as defects in the display or some of the mirrors may also affect the quality of the photograph taken. Furthermore, this method requires use of a complicated test jig comprising multiple mirrors and requiring accurate alignment of the display and camera.

<CIT> discloses the detection of camera lens contamination caused by dust, dirt, fingerprints and moisture by detecting whether a light source located between the lens cover and the digital image sensor, and directed toward the sensor, is scattered by a contamination.

The publication "<NPL>, discloses an algorithm that determines if a same artifact is present in several images by using pixel-wise correlations between images.

It is therefore an aim of the present invention to provide a system and method for determining whether a camera component is damaged, which aims to address the above problems.

In general terms, the present disclosure proposes a system and computer-implemented method for determining whether a camera component of a camera is damaged (i.e. defective) or undamaged (i.e. intact).

A first aspect of the invention provides a computer-implemented method for determining whether a camera component of a camera is damaged according to claim <NUM>.

Thus, embodiments of the invention provide a method which allows for automated diagnostics of surface defects or structural defects in a camera component such as a lens or other transparent layer between the camera and the light-source. Advantageously, the method may be employed in a diagnostic system for checking camera components on mobile devices. For example, cracks or scratches in the lens or dislocation of the lens of a digital camera may be detected based on artefacts in the image taken, for example, as a result of internal reflections of light from the light source when incident on the camera component. Notably, the image is taken using the camera whose lens (or other camera component) is being diagnosed. Moreover, the present method does not require use of a complicated test jig including multiple mirrors and requiring accurate alignment of the camera.

The light from the light source may be directly incident on the camera component (i.e. without encountering an intermediate optical component such as a mirror).

The information relating to one or more damage indicators comprises a known shape of the light source such that the one or more damage indicators correspond to a lack of a corresponding shape in the image; the image comprises an imaged shape resulting from the light-source; and the step of analysing each area comprises determining whether, based on the known shape of the light source, the imaged shape is as expected for the case when the camera component is undamaged and/or for the case when the camera component is damaged.

Thus, embodiments of the invention provide a method which may detect defects such as cracks or scratches in a lens or dislocation of a lens of a camera based on artefacts in a photograph taken of a light-source with a known shape. In some embodiments, the light-source may be a light-emitting diode (LED) producing an essentially hemi-spherical or circular area of light as observed by a camera.

The step of analysing the image may comprise digitally comparing the imaged shape with the known shape.

The step of analysing the image may comprise using a trained machine learning algorithm to classify the imaged shape as resulting from a damaged or undamaged camera component.

The machine learning algorithm may comprise a neural network.

The machine learning algorithm may comprise a deep learning algorithm.

The method may further comprise training the machine learning algorithm by providing multiple examples of imaged shapes from damaged and undamaged camera components.

During training, the machine learning algorithm may perform the following processes:.

The step of analysing the image may comprise using spatial mathematics to compare the imaged shape with the known shape.

The method may comprise generating an outline of the known shape on top of the imaged shape and calculating a percentage of bright pixels, from the imaged shape, that fit within the outline.

The step of generating an outline of the known shape on top of the imaged shape may comprise detecting a centre of the brightest area in the image, drawing the outline of the known shape around the centre, checking if the brightest area extends beyond the outline or checking if the brightest area does not extend to the outline and adjusting the size of the outline such that the brightest area extends to the outline in at least one direction.

The step of calculating a percentage of bright pixels, from the imaged shape, that fit within the outline may comprise determining a maximum luminosity of the imaged shape, determining the number of bright pixels within the outline having a luminosity within a predetermined threshold of the maximum luminosity, and dividing said number of bright pixels by a total number of pixels within the outline.

In some embodiments, the predetermined threshold may be determined by the user or by the machine learning algorithm. The predetermined threshold may be <NUM>% of the maximum luminosity, for example.

According to an embodiment, the camera component may be determined to be damaged if the percentage of bright pixels from the imaged shape that fit within the outline is less than <NUM>%.

The known shape may be a circle or an essentially round or elliptical area.

The images from damaged camera components may be further classified as resulting from defective or destroyed components.

The defective components may be further classified as scratched, dented, dislocated, distorted or opaque.

The camera component may be a camera lens, window or transparent front element or transparent protective cover.

A second aspect of the invention provides a non-transitory computer-readable medium according to claim <NUM>.

A third aspect of the invention provides a system for determining whether a camera component of a camera is damaged according to claim <NUM>.

The system may further comprise a neutral background such that the light-source is considerably more luminous than the background. According to an embodiment, the light-source is at least <NUM> times more luminous than the background.

The background may comprise a contrasting focal feature thereon for the camera to focus on when taking the image.

The system may further comprise a holder and/or robotic arm configured to position the camera for taking the image of the light-source.

The camera may be provided on a mobile device.

The system may further comprise the diagnostic processor and a communication means for communication with the camera.

The system may comprise multiple light-sources, each having a known shape, provided within a field of view of the camera and a controller configured to turn each individual light-source on and off such that only one of the multiple light-sources is active when an image is taken.

In some embodiments of the first aspect of the invention, the light source is present in a field of view of the camera and in other embodiments the light source is in a vicinity of the field of view, when the image is taken.

The method may further comprise taking the image.

The one or more damage indicators may comprise one or more artifact, pattern, contrast change, saturated region, blurred area, chromatic effect, light streak or other symptom.

The step of analysing each area may comprise using a statistical analysis to determine whether at least one of the one or more damage indicators is present.

The step of analysing each area may comprise calculating an optical parameter for each area and determining whether each optical parameter is indicative of at least one of the one or more damage indicators.

The optical parameter may comprise one or more of: a colour; a wavelength; a luminosity; an intensity; a brightness or a contrast.

The method may comprise calculating an average optical parameter for each area and determining whether each average optical parameter is indicative of at least one of the one or more damage indicators.

The method may comprise determining a percentage of a total number of pixels within each area, for which the optical parameter is within a predetermined range.

The predetermined range may be <NUM>% or more of an expected optical parameter.

The step of analysing each area may comprise using a trained machine learning algorithm to classify each area as comprising none of the one or more damage indicators or at least one of the one or more damage indicators.

The machine learning algorithm may comprise a neural network or a deep learning algorithm.

The method may comprise: extracting information from each area; comparing the extracted information against one or more predetermined probability vectors to establish whether the area should be classified as comprising none of the one or more damage indicators or at least one of the one or more damage indicators; and calculating a probability that the area is correctly classified.

The method may further comprise training the machine learning algorithm by providing multiple examples of images from damaged and undamaged camera components.

During training, the machine learning algorithm may perform the following processes: extracting information from the multiple examples; transforming the extracted information into information matrices; manipulating the information matrices into combined matrices; and using the combined matrices to establish a probability vector for each classification.

The method may further comprise negating a light source in the image by: determining a brightest region corresponding to an area of greatest intensity in the image and all adjacent areas having an intensity in a pre-determined range of the greatest intensity; and excluding the brightest region from the step of analysing each area.

The image may comprise a neutral background.

The method may comprise calculating a percentage of the areas determined as comprising at least one of the one or more damage indicators, compared to all areas of a single image, and classifying the camera component as damaged if the percentage is at least <NUM>%, <NUM>%, <NUM>% or <NUM>%.

The method may further comprise classifying damaged camera components as defective or destroyed.

The method may further comprise classifying defective components as scratched, dented, dislocated, distorted or opaque.

The system may further comprise a light source arranged to provide light incident on the camera component.

The system may comprise a fibre optic cable arranged to direct light from the light source to the camera component.

The light source may be arranged outside of a field of view of the camera.

The light source and/or camera may be movable such that different images can be taken at different angles of illumination.

The light source may be a white light source. According to an embodiment, the colour or the wavelength of the light source may be adjustable such that different images can be taken in different lighting conditions.

The system may comprise a controller configured to activate said light source when an image is taken.

The system may further comprise a focal feature for the camera to focus on when taking the image.

The system may further comprise a holder and/or robotic arm configured to position the camera for taking the image.

Multiple light sources may be arranged to provide light to the camera component and a controller may be configured to turn each individual light source on and off such that one or more of the multiple light sources is active when an image is taken. According to an embodiment, the colour or the wavelength of each or some of the multiple light sources may differ from the rest or some of the multiple light sources.

The claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. For instance, the claimed subject matter may be implemented as a computer-readable medium embedded with a computer executable program, which encompasses a computer program accessible from any computer-readable storage device or storage media. For example, computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips. ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD). ), smart cards, and flash memory devices (e.g., card, stick, key drive.

Non-limiting embodiments of the invention will now be described for the sake of example only, with reference to the following drawings in which:.

Embodiments of the present invention relate generally to automated camera lens scratch diagnostics by geometrical referencing.

Surface defects or structural defects in flat, transparent surfaces, such as that of a camera lens or window, can be diagnosed using embodiments of the invention when a photograph is taken of a known shape of light-source (e.g. a round light emitting diode, LED) such that light travels from the light source to the camera through the camera component being diagnosed. A small light source such as an LED should look like a luminous circular area in the photograph taken, particularly if the LED is the only source of high brightness in the photograph. However, if the camera component is deformed, damaged or broken, the LED light source will not appear in the photograph as such a luminous circle, but as a different/distorted shape. Embodiments of the invention therefore provide a system and method to exploit this phenomenon to determine whether a camera component is damaged or not.

<FIG> shows a system <NUM> for determining whether a camera component <NUM> (e.g. a lens, window or transparent front element) of a camera <NUM> is damaged in accordance with an embodiment of the invention. The system <NUM> comprises a device under test <NUM> (which, in this case is a mobile device although in other embodiments may be a dedicated camera or other device including a camera) which is mounted in a holder <NUM> and is in communication with a diagnostic processor <NUM> configured to carry out a computer-implemented method for determining whether the camera component <NUM> is damaged. The computer-implemented method will be described in more detail below. However, it should be noted that the computer-implemented method may be encoded on a non-transitory computer-readable medium such that either the diagnostic processor <NUM> may carry out the method or a processor within the device under test <NUM> may carry out the method. For example, instructions for carrying out the computer-implemented method may be downloaded to a memory of the device under test <NUM> such that no separate diagnostic processor <NUM> is required. Thus, embodiments of the invention relate to both diagnostic hardware (i.e. in the form a diagnostic processor <NUM>) and diagnostic software configured to carry out the computer-implemented method.

In some embodiments, the system <NUM> may include a robotic arm (not shown) which is configured to either position the device under test <NUM> in the holder <NUM> or directly hold the device under test <NUM> for the taking of an image. In other embodiments, a user may hold the device under test <NUM> without the need for the holder <NUM>.

The system <NUM> also comprises at least one light-source with a known shape. In this embodiment, the light-source is a round LED which is configured to produce a circular area of brightness <NUM> when viewed from the camera <NUM>. The LED is mounted on a board <NUM> as shown in more detail in <FIG>. However, in some embodiments the board <NUM> is not required.

As shown in <FIG>, the board <NUM> comprises four LEDs <NUM>, each one being provided in a space formed by a black marker "X" <NUM> to form a diamond pattern. Although not illustrated, the board <NUM> comprises a neutral dull grey background <NUM> of uniform colour and brightness, thus forming a so-called "grey card". The grey background <NUM> helps to ensure that the LEDs <NUM> (when active) are the only source of brightness in the images taken by the device under test <NUM> so that the imaged shape of an individual LED <NUM> is clearly discernible. In some embodiments the LEDs <NUM> may be at least <NUM> times, at least <NUM> times or at least <NUM> times more luminous than the grey background <NUM>. In some embodiments the luminosity of the LEDs <NUM> may be adjustable. Furthermore, the colour of the background <NUM> need not be grey as long as it is relatively uniform and less luminous than the LEDs <NUM>.

The black marker "X" <NUM> constitutes a contrasting focal feature on the board <NUM> for the camera <NUM> to focus on when taking an image. In other embodiments, a different focal feature may be provided.

In the present embodiment, the four LEDs <NUM> were mounted on a simple breadboard and driven by a microcontroller (not shown) configured to turn each individual LED <NUM> on and off such that only one of the four light-sources is active when an image is taken. For example, one LED <NUM> is turned on for three seconds and off for one second before the next LED <NUM> is turned on for three seconds and off for one second and so on until the power is switched off. In other embodiments a different number of light-sources may be employed and/or a different control sequence may be used.

<FIG> shows steps of a computer-implemented method <NUM> (referred to above) for determining whether the camera component <NUM> of a camera <NUM> is damaged in accordance with an embodiment of the invention. As explained above, the method <NUM> may be performed by the device under test <NUM> or the diagnostic processor <NUM> of <FIG>.

The method <NUM> comprises a first step <NUM> of obtaining, from the camera <NUM>, at least one image of a light-source <NUM> with a known shape <NUM>, the image comprising an imaged shape resulting from the light-source <NUM> and a second step <NUM> of analysing the image. A third step <NUM> comprises determining whether the imaged shape is as expected for the case when the camera component <NUM> is undamaged and/or for the case when the camera component <NUM> is damaged. In a fourth step <NUM> the method <NUM> provides an indication of whether the camera component <NUM> is determined to be damaged or undamaged. Further details of particular embodiments will be described in more detail below.

In operation, defects such as surface defects, structural defects, misalignment or dislocation of a part in the camera component <NUM> of the device under test <NUM> can be identified by the following process. The system <NUM> of <FIG> is arranged by providing a light-source of known shape (i.e. round LED <NUM> which is embedded within or positioned in front of the grey background <NUM>, with black marker "X" <NUM>, on the board <NUM>) and arranging the device under test <NUM> (including the camera component <NUM> of the in-built digital camera <NUM> which is to be checked) such that the light-source <NUM> and board <NUM> are in the camera's field of view. Ideally, the board <NUM> is orientated in a plane substantially parallel to a front face of the camera component <NUM>, and orthogonal to an optical axis of the camera <NUM>.

The device under test <NUM> may be placed in the holder <NUM> by an operator or by a robotic arm and may be fixed in position by the holder <NUM> itself or by one or more mechanical components such as clamps or fasteners.

The device under test <NUM> is then connected to diagnostic software configured to carry out the method <NUM> of <FIG>. This may involve the device under test <NUM> downloading the diagnostic software into its memory or by connecting the device under test <NUM> to the diagnostic processor <NUM> (either through a wired or wireless connection). The step of connecting the device under test <NUM> to the diagnostic processor <NUM> may be performed automatically such that the diagnostic processor <NUM> automatically detects the presence of the device under test <NUM> once it is placed into the holder <NUM>. According to another embodiment, this step may be performed manually such that an operator makes the necessary connections by using a USB cable, for example, and/or initiates the connection from a user interface connected to the diagnostic processor <NUM> or device under test <NUM>.

Similarly, a connection is made between the diagnostic software and the camera <NUM> of the device under test <NUM>. This connection may be performed automatically such that the diagnostics software automatically detects whether the device under test <NUM> includes a camera <NUM> and then connects to the camera <NUM> using interfaces built into an operating system (OS) of the device under test <NUM>.

The diagnostic software may then operate the camera <NUM> to take one or more photographs of the light-source <NUM>. This will involve the camera <NUM> focussing on the black marker "X" <NUM> and capturing an image including the light-source <NUM>. In some embodiments, the required photographs may be taken automatically by the diagnostics software while in other embodiments the diagnostic software may guide an operator to take the required photographs via a user interface to the device under test <NUM> or diagnostic processor <NUM>. Thus, the step <NUM> of obtaining, from the camera <NUM>, at least one image of the light-source <NUM> with a known shape <NUM> may be fulfilled either by the diagnostic software directly taking an image using the camera <NUM> or by the device under test <NUM> transferring an image taken by the camera <NUM> to the diagnostic software or diagnostic processor <NUM>.

Next, the image will be analysed. This may be performed by the diagnostic software within the device under test <NUM> or by transferring the image to the diagnostic processor <NUM> for analysis. In either case, the imaged shape is analysed to determine whether it is as expected for the case when the camera component <NUM> is undamaged and/or for the case when the camera component <NUM> is damaged. In other words, the diagnostic software checks whether the image of the round LED <NUM> is rendered essentially circular or distorted. Of course, in other embodiments a different known shape of light-source may be employed (e.g. elliptical, triangular, square, rectangular etc.) and the analysis will check whether the rendered image is as expected for such a shape. The analysis may be performed, for example, using spatial mathematics or machine learning as explained in more detail below.

Finally, an indication of whether the camera component <NUM> is determined to be damaged or undamaged is provided. This indication or diagnostic verdict may take the form of an audible and/or visible alert, especially in the case where the camera component <NUM> is determined to be damaged.

In some embodiments, the diagnostic verdict may be displayed to the operator and may comprise a simple overall grade, a sparse detailed grade, a very detailed grade, or a full report of parameters affecting the grading. For example, the camera component <NUM> may be classified as intact, scratched, destroyed, dented, dislocated, distorted or opaque. These results may be shown on a screen of the device under test <NUM>, via the user interface of the diagnostic processor <NUM>, or in a separate report provided by the diagnostics software.

The results of the analysis of the images, or the images themselves, may be transferred from the device under test <NUM> to a remote server (e.g. on which the diagnostic processor <NUM> is provided) with assistance or input from the operator, or automatically by the diagnostics software.

For reference, <FIG> shows an image of the board <NUM> and one LED <NUM> light-source of <FIG> captured using an undamaged camera component <NUM> (intact lens) in accordance with an embodiment of the invention. Thus, it can be seen that the imaged shape 400a from the LED <NUM> is essentially round.

<FIG> shows an image of the board <NUM> and same LED <NUM> light-source of <FIG> captured using a damaged camera component <NUM> (scratched lens) in accordance with an embodiment of the invention. Thus, it can be seen that the imaged shape 400b from the LED <NUM> is not round but rather elongate.

The following examples will detail how the above images can be analysed automatically to determine whether the camera component <NUM> is damaged or undamaged in accordance with embodiments of the invention.

A number of images were generated using graphics creation software to illustrate potential captured images of the round LED <NUM> from intact, scratched and destroyed lenses. Thus, Fig.'s 5A, 5B and 5C show respective examples of captured light-source images 500a, 500b and 500c for undamaged camera components <NUM>. Each of these images 500a, 500b and 500c are essentially round. For the purposes of this example, <NUM> images of the type shown in Fig.'s 5A, 5B and 5C were generated, all of which comprised a roundish shape but all were slightly different.

<FIG> show respective examples of captured light-source images 600a, 600b and 600c for scratched camera components <NUM>. Each of these images 600a, 600b and 600c has a round shape which is stretched or deformed in some way. For the purposes of this example, <NUM> images of the type shown in Fig.'s 6A, 6B and 6C were generated, all of which comprised a round shape stretched or deformed in a different way.

<FIG> show examples of captured light-source images 700a, 700b and 700c for destroyed camera components <NUM>. Each of these images 700a, 700b and 700c has a very badly deformed and broken shape. For the purposes of this example, <NUM> images of the type shown in Fig.'s 7A, 7B and 7C were generated, all of which comprised badly deformed and broken shapes.

Use of the above images is described below for different analysis techniques.

The step of analysing the photographs taken through the camera component <NUM> of the device under test <NUM> may be performed, for example, with mathematical calculus, where a circular light source (e.g. LED <NUM>) is approximated with an equation. Solving the equation with parameters extracted from photographs taken through the camera component <NUM> gives a result that may be compared with a known reference value. A match with the reference value indicates an intact camera component <NUM> (e.g. lens) while a deviation from the reference value indicates a faulty camera component <NUM> (e.g. lens).

More specifically, the step of analysing the photographs using a spatial mathematics approach may comprise the following steps:.

<FIG> shows a schematic representation of a geometrical analysis of a captured light-source image 800a for an undamaged camera component <NUM> in accordance with an embodiment of the invention. In <FIG>, the captured image of <FIG> is shown with a solid line outline 800a. This captured image 800a has been surrounded with a dashed line that is perfectly round, indicating the shape <NUM> expected from the LED <NUM>. Comparing the amount of pixels within the captured image 800a to the amount of pixels within the dashed line <NUM> there is only roughly a <NUM>% difference. Thus, almost <NUM>% of the area inside the dashed line <NUM> is occupied by bright pixels in the captured image 800a. This is within determined tolerances and therefore the captured image 800a is determined to be from an intact camera component <NUM>.

<FIG> shows a schematic representation of a geometrical analysis of a captured light-source image 800b for a scratched camera component <NUM> in accordance with an embodiment of the invention. In this case, the captured image of <FIG> is processed in the same way as above and is shown with a solid line outline 800b which is non-circular and results from a scratched lens. This captured image 800b has been surrounded with a dashed line that is perfectly round, indicating the shape <NUM> expected from the LED <NUM>. Comparing the amount of pixels within the captured image 800b to the amount of pixels within the dashed line <NUM> there is approximately a <NUM>% difference. Thus, according to the pre-set rules, the camera component <NUM> in this instance may be determined to be scratched.

In a real world case the pixels within the solid lines 800a and 800b would be bright pixels observed in the captured image that have a luminosity greater than a pre-determined value (for example, at least <NUM>% as bright as the maximum luminosity).

In summary, this approach simply requires a measurement of the fraction of pixels inside the expected shape that are bright. In other words, the following formula is applied: <MAT>.

In other embodiments, the step of analysing the photographs taken through the camera component <NUM> of the device under test <NUM> may be performed, for example, by means of machine learning via computer vision. In this case, an algorithm is taught by a sufficient set of photographs to identify flaws in the images, which is then used in grading defects in the camera component <NUM>. As explained above, this analysis may be performed either by diagnostics software running on the device under test <NUM> or by a remote server or diagnostic processor <NUM>, for example.

The machine learning algorithm may comprise a neural network or deep learning algorithm, for example. In the case of a neural network, the algorithm is first taught using example pictures such as those described above in relation to Fig.'s 5A to 7C, before it is able to classify similar pictures with some efficiency.

In general, use of the machine learning algorithm may comprise:.

An example machine learning approach was taught using a random selection of <NUM>% of the training pictures described above in relation to Fig.'s 5A to 7C as training material and then using the remaining <NUM>% of the pictures as examples of captured images to test the classification efficiency of the method.

<FIG> shows a confusion matrix for the results of the machine learning analysis where the captured images were classified as being obtained using intact, scratched and destroyed camera components <NUM> in the form of camera lenses.

As can be seen, all "intact" lenses were correctly classified. Classification of "scratched" lenses was in this test case somewhat less reliable, as <NUM> out of <NUM> were incorrectly classified as "intact". In this case only one destroyed picture was in the sample, and it was correctly classified.

The classifier used in the present case was a known "Random Forest Classifier". Overall, the classification was <NUM>% accurate and similar numbers can be expected in a real-world use case.

Aspects of the present invention relate, generally, to methods to incite inner reflections in a camera component such as a lens system (objective), to check their uniformity. As such, there is no real interest in the subject of the image but rather, the background of the image should be as featureless and uniform as possible, so as to enhance the image of reflections. By observing such reflections both in intact and damaged lens systems, it is possible to identify the differences in reflections, and determine the damaged lens system from the intact one. When enough samples are obtained, this decision may be automated via a suitable algorithm.

Thus, aspects of the invention provide a novel way to diagnose camera components, such as smartphone lenses, by using a high-quality light source and optionally fiber optics. Any lens defects will cause errors that can be detected using statistical spatial analysis or machine learning.

<FIG> shows steps of a computer-implemented method <NUM> for determining whether the camera component <NUM> of a camera <NUM> is damaged in accordance with embodiments of the invention. As per the method explained above, this method <NUM> may be performed by the device under test <NUM> or the diagnostic processor <NUM>.

The method <NUM> comprises a first step <NUM> of obtaining information relating to one or more damage indicators and a second step <NUM> of obtaining, from the camera, at least one image which has been taken when light from a light source has been incident on the camera component. A third step <NUM> requires dividing the image into one or more areas and a fourth step <NUM> comprises analysing each area to determine whether it comprises at least one of the one or more damage indicators. A fifth step <NUM> comprises providing an indication of whether the camera component is classified as damaged or undamaged, based on said analysing.

In the particular embodiments described below that do not form part of the claimed invention, the camera component is directly exposed to light from the light source, however, the shape of the light source need not be known. Thus, this technique may be used if a known light source not available. However, in some embodiments, the present technique may be employed alongside the above technique, using a known light source, to be able to identify other artefacts even if the imaged light source is of the expected shape.

In the embodiment illustrated in Fig.'s 11A, 11B and 11C, the light source <NUM> is provided in the field of view (FOV) of the camera (i.e. the FOV of a sensor or detector in the camera). However, in other embodiments the light source may be outside of the FOV of the camera (i.e. outside of the FOV of the sensor or detector) but still arranged to provide light directly onto the camera component being tested, as will be explained further below. In some embodiments, an operator may choose whether or not to include the light source in the FOV.

Ideally, the light source <NUM> includes different wavelengths (i.e. is as close to natural/white light as possible to enable chromatic effects to be more easily identified). The light source <NUM> should be brighter than ambient light (e.g. at least <NUM> times more luminous) and could be the sun or other source of bright light such as an LED or fibre optic light cable.

As illustrated in <FIG>, in first step, a picture/image is taken with the camera under test, so that the background of the picture is preferably of substantially uniform color. Additionally, there may be a focusing feature for the camera, for example an "X" printed on the uniform background, as above. In this embodiment, there is also one very bright small light source <NUM> present in the image. If the camera component is damaged (as per <FIG>) the image will also show one or more damage indicators <NUM>, which may be in the form of light streaks, chromatic (i.e. rainbow) effects, blurriness or the like.

In a second step, shown in <FIG>, the image is divided into a plurality of areas, which in this case are small squares <NUM>. In other embodiments, the image may be divided into one or more areas using squares or other shapes. An optical parameter is then determined for each area. In this embodiment, an average RGB colour value is determined for each square <NUM>. In other embodiments, the intensity, of each area may be determined.

In a third step, shown in <FIG>, the largest and/or brightest continual area in the image is determined and disregarded from further analysis to mask out the light source from the image, if present in the FOV. In this case, a circle is used to mask out the light source <NUM> in the image, although any shape could be used to mask the light source.

In a fourth step, not illustrated, a statistical analysis method and/or a machine learning technique is employed to classify each of the areas <NUM> as either "expected" or "unexpected". In other words, each area <NUM> is analysed to determine whether it comprises at least one damage indicator <NUM>. A damage indicator may take the form of a pattern, artefact, chromatic error (i.e. rainbow effect), saturated (i.e. burnt out) area or other symptom (e.g. a sharp contrast change between pixels or "blurriness") which may appear in images taken through a damaged camera component. For example, an optical parameter such as an average intensity, luminosity, colour or wavelength of the area may be determined and compared against a reference or threshold value to determine whether the area comprises at least one damage indicator <NUM>. In some embodiments, more than one optical parameter may be obtained to determine whether the area comprises one or more damage indicators <NUM>. Thus, a calibration or set-up procedure may be employed to obtain information relating to one or more damage indicators <NUM> (i.e. to establish a normal or expected range of values for reference).

In some embodiment, a large data set of potential damage indicators would be provided to teach a machine learning algorithm to identify whether the presence of such damage indicators and thereby to determine whether a camera component is damaged or undamaged.

If all of the areas <NUM> outside the light source are as expected (i.e. do not contain any damage indicators, a signal is communicated to the operator that the camera is fully functional. However, if one or more of the areas <NUM> are not as expected (i.e. do contain damage indicators), a signal is communicated to the operator that the camera is damaged.

If the camera is classified as undamaged (i.e. functional) the process may end without the need for any human intervention.

If the camera is classified as damaged (e.g. broken), the operator may verify that the camera is not functioning as it should.

A benefit of such embodiments of the invention is that, in most cases, (e.g. for <NUM>% to <NUM>% of the cameras tested) no human intervention is needed to evaluate if the camera is intact or broken. In rare cases where the camera is broken, the operator can label it as broken and either send it to be repaired (if this is deemed worthwhile i.e. if the device is a relatively new and expensive model) or lower its price (if it is too cheap for the repairs to make sense).

Compared to the initial embodiments described above, this embodiment can detect wider range of errors. For example, dust in a lens array or dirt in a sensor should create artifacts that can be detected automatically using this method.

Thus far most smartphone camera lens diagnostic systems rely on visual inspection by human beings. This present embodiment can either make that visual inspection easier, or can fully automate the inspection by using an algorithm that detects when image artifacts appear as described above.

In some aspects of the invention, the method can be considered as a probability calculus exercise, since no lens is perfect. That being said, a threshold value may be set for an allowable defect probability. Consequently, the step of providing an indication of whether the camera component is classified as damaged or undamaged may comprise calculating a probability of a camera component defect based on, for example, a number of observed damage indicators when compared with the obtained information about damage indicators. The information may comprise a defect probability distribution based on a number of observed damage indicators. Further, the probability distribution might be specific to a type of damage indicator, in which case more than one probability distribution may be obtained. In some embodiments, this sort of probability calculus may be carried out by employing a machine learning algorithm, such that the estimate becomes more accurate as more information about damage indicators is accumulated.

<FIG> shows a schematic of a test set-up <NUM> for testing cameras <NUM> on a series of mobile devices <NUM> in accordance with embodiments of the invention. The test set-up <NUM> comprises a high quality light source <NUM> which passes through a lens system <NUM> and into a plurality of optical fibres <NUM> which are arranged to direct light into each camera <NUM> of each mobile device <NUM>.

<FIG> shows a side view of a single mobile device <NUM> camera <NUM> being tested using the optical fibre <NUM> delivered light from the light source <NUM>. Although the light may, generally, be directed straight towards a center of a camera component (e.g. lens) being tested, in some embodiments, the light may be come from other angles, or it may be directed towards the camera component from several angles (e.g. by moving the light source or camera) to help catch any image artifacts.

<FIG> shows an enlarged schematic of a scratched lens <NUM> being tested using an optical fibre <NUM> light source. When the light encounters an error, such as scratch <NUM>, in the lens <NUM>, the light generally creates artifacts <NUM> or damage indicators, which may take the form of prismatic effects or bright spots or streaks in a created image, which are clearly outside an area of brightness corresponding to the imaged light source if present in the FOV. Thus, enabling the system to detect the presence of damage indicators in accordance with the method of <FIG> in order to determine whether the camera is damaged or undamaged.

Even a relatively simple setup such as that described above with a high-quality light source may help an operator to spot lens errors that otherwise might be difficult to notice. In some embodiments, such lens error detection may be performed semi-automatically, possibly by using a robotics arm and an artificial intelligence system. This is faster and requires less workforce than fully manual camera optics testing.

Fig.'s 15A, 15B, 15C, 15D, 15E and 15F show images taken using another test set-up in accordance with embodiments of the invention. In <FIG> an image is taken of a box <NUM> with a hole <NUM> in it, in ambient light, by a camera having a very damaged lens. However, no errors are visible.

<FIG> shows an image of the box <NUM> with a bright light source <NUM> in front, taken by a camera having an undamaged lens. This image shows a bright, substantially round area <NUM> of light from the light source <NUM>.

<FIG> shows an image of the box <NUM> with the bright light source <NUM> in front, taken by a camera having a damaged (scratched) lens. This image shows a non-circular bright area <NUM> and light streaks <NUM> which can be identified as damage indicators indicative of the fact that the lens is damaged.

<FIG> shows an image of the box <NUM> with the bright light source <NUM> in front, taken by a camera having a more damaged lens. This image also shows a non-circular bright area <NUM> and stronger light streaks <NUM> (some of which show a chromatic rainbow effect) which can be identified as damage indicators indicative of the fact that the lens is damaged.

<FIG> shows an image of the box <NUM> with a brighter light source <NUM> in front, taken by the same camera used for <FIG> including a very damaged lens. This image also shows a non-circular bright area <NUM> and strong light streaks <NUM> (some of which show a chromatic rainbow effect) which can be identified as damage indicators indicative of the fact that the lens is damaged.

<FIG> shows an image of the box <NUM> with a smaller light source <NUM> in front, taken by the same camera used for <FIG> including a very damaged lens. This image also shows a non-circular bright area <NUM> and strong light streaks <NUM> (some of which show a chromatic rainbow effect) which can be identified as damage indicators indicative of the fact that the lens is damaged.

It is noted that even with a very damaged lens, it is almost impossible to notice the damage by looking at the camera with the naked eye. The methods described in accordance with embodiments of the present invention therefore provide a useful and reliable tool for easily identifying damaged camera components.

Fig.'s 16A and 16B show the results of ray-tracing simulations <NUM> for undamaged and damaged lenses, respectively. The simulation utilized a 3D model of a lens in conjunction with ray tracing software to simulate how the lens works both when intact and when damaged. This example simulated a scene comprising one very small light source, the lens being tested and a wall behind the lens. Photons from the light source illuminated the lens externally and traveled through the lens. For each pixel in the simulated image, <NUM><NUM> samples were taken before the image was considered complete.

In the simulations, the wall behind the lens was burned totally white, except for a circular area that was in the shadow of the lens. In that area the image of the light source is visible, and additionally some stray photons can be seen.

As shown in <FIG>, when the lens is intact, the photons mostly concentrate in one bright spot <NUM> behind the lens.

However, if the lens is damaged (i.e. geometry of the lens closest to the image sensor is altered) both the shape of the bright spot <NUM> alters, and there are a lot more stray photons. In the real world those photons would cause chromatic errors in the image.

While alterations to the shape of the bright spot can be detected with the method described in <FIG>, the method described in relation to <FIG> is better in detecting chromatic errors and other such flaws that may not significantly change the shape of the bright area.

Furthermore, in comparison to the embodiments described above in relation to <FIG>, the present embodiment does not depend on use of a known light source. On the contrary, if the light source is in the captured image, it may be advantageous for it to be excluded or masked out.

Additionally, the scene of which the image is taken does not need to be known, but it only needs to satisfy certain conditions, such as there needs to be a light source in the scene or in its vicinity such that light travels from the light source to the camera component being tested.

Fig.'s 17A, 17B, 17C, 17D and 17E show schematically apparatus configured for determining whether a camera component of a camera is damaged in accordance with embodiments of the invention.

<FIG> shows a neutral background card <NUM> with a small bright light source <NUM> provided in front of the background <NUM>. In some embodiments, the light source <NUM> may be moved in any direction in front of the background <NUM> (e.g. horizontally, vertically or diagonally) whilst a series of images are taken so as to provide a higher chance of light entering the camera component being affected by any errors in the component. This is because defects in the lens may cause artefacts to appear only at a certain angle (i.e. when light reflected via the defective portion of the lens hits the camera sensor). Additionally, or alternatively, the camera component (or device on which it is provided) may be moved relative to the light source <NUM> for the same reason.

Fig.'s 17B and 17C show a test set-up in which the light source <NUM> is provided in front of the background <NUM>, in a field of view (FOV) <NUM> of a camera component <NUM> of a smartphone <NUM>. In this case, the smartphone <NUM> is provided in a smartphone holder <NUM>, which is arranged to move the smartphone <NUM> around three degrees of motion (i.e. vertically, horizontally and around a vertical axis) in order to take images including the light source <NUM> for analysis in accordance with the invention.

<FIG> shows another test set-up in which the light source <NUM> is provided in front of the background <NUM>, but outside of the field of view (FOV) <NUM> of the camera component <NUM> of the smartphone <NUM>. As above, the smartphone <NUM> is provided in the smartphone holder <NUM>, which is arranged to move the smartphone <NUM> around three degrees of motion.

<FIG> shows a close-up view of a damaged lens camera component <NUM> being tested using the set-up of <FIG>. Thus, although the light source <NUM> is outside of the FOV <NUM>, it is clear that light from the light source <NUM> is still incident on the camera component <NUM> of the smartphone <NUM>. When light from the light source <NUM> encounters a defect or error <NUM> in the camera component <NUM>, at least some of the light may be directed towards an image sensor <NUM> in the camera. Accordingly, the image will include artefacts or damage indicators indicative of a damaged camera component.

In some embodiments, the error <NUM> in the lens may lead to a distorted rendering of the original light source <NUM> and/or light streaks or chromatic errors.

Fig.'s 18A, 18B, 18C, 18D, 18E, 18F, <NUM> and <NUM> illustrate various different artefacts that may be identified as damage indicators in embodiments of the invention to determine whether a camera component of a camera is damaged.

<FIG> shows an image comprising a light source <NUM>, light streaks <NUM> and an area <NUM> of unsharp light smearing or blurriness. <FIG> is a black and white line drawing representation of the image of <FIG>, which shows the location of the light source <NUM>, light streaks <NUM> and area <NUM> of unsharp light smearing or blurriness.

<FIG> shows an image taken with the light source outside of a field of view but which still shows light streaks <NUM>. <FIG> is a black and white line drawing representation of the image of <FIG>, which shows the location of the light streaks <NUM>.

<FIG> shows an image taken with the light source outside of a field of view but which shows light streaks <NUM> and a bright (saturated) area <NUM>. <FIG> is a black and white line drawing representation of the image of <FIG>, which shows the location of the light streaks <NUM> and bright (saturated) area <NUM>.

<FIG> shows an image taken with the light source at a top edge of a field of view and which shows a different kind of light streaks <NUM> and a bright (saturated) area <NUM>. <FIG> is a black and white line drawing representation of the image of <FIG>, which shows the location of the light streaks <NUM> and bright (saturated) area <NUM>.

<FIG> shows an example image taken with the sun outside of the field of view of the camera but with artefacts in the form of blurred light streams <NUM> present in the image which indicate damage to the camera component.

It will be understood that the images of the above figures may be analysed in embodiments of the invention to identify damage indicators and thereby determine whether the camera component is damaged or undamaged.

Claim 1:
A computer-implemented method (<NUM>) for determining whether a camera component (<NUM>) of a camera (<NUM>) is damaged comprising:
obtaining (<NUM>) information relating to one or more damage indicators (<NUM>);
obtaining (<NUM>), from the camera (<NUM>), at least one image which has been taken when light from a light source (<NUM>) has been incident on the camera component (<NUM>);
dividing the image into one or more areas (<NUM>);
analysing each area to determine whether it comprises at least one of the one or more damage indicators (<NUM>); and
based on said analysing, providing an indication of whether the camera component (<NUM>) is classified as damaged or undamaged (<NUM>)
characterized in that:
the information relating to one or more damage indicators (<NUM>) comprises a known shape of the light source (<NUM>) such that the one or more damage indicators (<NUM>) correspond to a lack of a corresponding shape in the image;
the image comprises an imaged shape resulting from the light-source (<NUM>); and
the step of analysing each area comprises determining whether, based on the known shape of the light source (<NUM>), the imaged shape is as expected for the case when the camera component (<NUM>) is undamaged and/or for the case when the camera component (<NUM>) is damaged.