Patent Description:
Multi-spectral imaging provides a rich amount of information not readily obtainable from single or triple spectral channel (e.g. Red Green Blue (RGB)) imaging. The advantage is more evident in imaging of complex subjects such as the human skin, for which the detected back-scattered light is modulated by several chromophores in the skin/body, including haemoglobin, melanin, water, lipids, carotenoid and bilirubin.

Reflectance spectroscopy is the study of light as a function of wavelength. It finds many uses in medical applications, material analysis and soil analysis. Reflectance spectroscopy is enabled by the fact that different materials have different absorption and scattering spectra. This also holds for the chromophores in the human skin, such as oxygenated and deoxygenated haemoglobin, carotenoids, melanin, bilirubin, etc. Therefore, reflectance spectroscopy of the human skin allows for a detailed decomposition of the chromophores present. It is also useful to relate detected chromophores to (systemic) health as well as specific skin conditions that cannot be resolved by the human eye.

Reflectance spectroscopy typically observes only the spectrum of a single point or area. When reference is made to multi- or hyper-spectral imaging, a whole grid (or image) is observed at different wavelengths. A specific field of interest is multi-spectral imaging of the human body and face.

Although fast multi-spectral imaging can be implemented using single-shot multi-spectral cameras, these devices are very expensive and not for commercial home use. A cost-effective solution is by using multispectral illumination for which each light wavelength can be sequentially captured or scanned by the imaging device (e.g. camera) from multiple frames. <CIT>, <CIT>, <CIT>, and <CIT> disclose a sequential multispectral illumination resulting in different frames with respective illumination, motion detection in one frame sequence and frame alignment for further image processing.

A multi-spectral image typically consists of a sequence of image frames, where, for each frame, an object/subject to be imaged is illuminated by light having a specific wavelength. In practice, such a multi-spectral image 'scan' can take up to a few seconds to be completed. This has the disadvantage that there will typically be small movements by the subject (e.g. human) during the scan.

Since the spectrum at a particular spatial location on the subject is typically of interest, it is important that all frames are aligned. For example, a certain point of the face may be imaged while illuminated with light at a wavelength of <NUM>, with that certain point residing at coordinates (X, Y) in the image. This exact same point of the face imaged when illuminated with light at, e.g., <NUM>, should reside at exactly the same coordinates (X, Y) in the image. If this is not the case, e.g. due to movement of the face and/or movement of the imaging device with respect to the face, image alignment (also known as image registration) should be implemented such that the image frames are aligned before any spectral analysis is performed using the image frames.

Conventionally, sequential image frames using non-varying illumination can be easily aligned or registered using simple algorithms. However, image alignment for images of a subject illuminated by different (time-varying) light, i.e. multi-spectral light, is hindered by the fact that the reflectance of skin spectrally varies, and thus images or parts of images at different wavelengths have different spatial contrast and detail.

Therefore there is a need for an improved method and apparatus for performing spectral analysis of skin of a subject that includes aligning image frames of the skin of the subject obtained using multi-spectral light.

According to a non-claimed example, there is provided a method for aligning image frames in a sequence of image frames of a subject. The method comprises controlling one or more light sources to emit multi-spectral light to illuminate the subject, wherein the multi-spectral light comprises at least a first light component having a first illumination wavelength, and a second light component having a second illumination wavelength; controlling a multi-channel imaging device to obtain two or more sequences of image frames of the subject, wherein each sequence of image frames is obtained using a respective light detection channel of the multi-channel imaging device, wherein the multi-channel imaging device comprises a first light detection channel that is responsive to light in a first spectral band to generate a first sequence of frames of the subject, and a second light detection channel that is responsive to light in a second spectral band to generate respective a second sequence of frames of the subject, wherein the first illumination wavelength is within the first spectral band, and the second illumination wavelength is within the second spectral band; processing the first sequence of frames to calculate motion vectors for each frame, wherein each motion vector represents a movement of the subject between frames in the first sequence; and aligning the frames in the second sequence of frames according to the corresponding determined motion vectors such that the subject is aligned across the frames in the second sequence of frames.

According to a non-claimed example, there is provided a computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method according to the first aspect or any embodiment thereof.

According to a non-claimed example, there is provided an apparatus configured for aligning image frames in a sequence of image frames of a subject. The apparatus comprises a processing unit configured to control one or more light sources to emit multi-spectral light to illuminate the subject, wherein the multi-spectral light comprises at least a first light component having a first illumination wavelength, and a second light component having a second illumination wavelength; control a multi-channel imaging device to obtain two or more sequences of image frames of the subject, wherein each sequence of image frames and each is obtained using a respective light detection channel of the multi-channel imaging device, wherein the multi-channel imaging device comprises a first light detection channel that is responsive to light in a first spectral band to generate a first sequence of frames of the subject, and a second light detection channel that is responsive to light in a second spectral band to generate a second sequence of frames of the subject, wherein the first illumination wavelength is within the first spectral band, and the second illumination wavelength is within the second spectral band; process the first sequence of frames to calculate motion vectors for each frame, wherein each motion vector represents a movement of the subject between frames in the first sequence; and align the frames in the second sequence of frames according to the corresponding determined motion vectors such that the subject is aligned across the frames in the second sequence of frames.

According to a non-claimed example, there is provided a system configured for aligning image frames in a sequence of image frames of a subject. The system comprises an apparatus according to the third aspect or any embodiment thereof, one or more light sources; and a multi-channel imaging device.

According to a first aspect, there is provided a method according to claim <NUM>, for performing spectral analysis of skin of a subject. The method comprises (i) aligning image frames in a sequence of image frames of skin of a subject by: controlling one or more light sources to emit multi-spectral light to illuminate the skin, wherein the multi-spectral light comprises at least a first light component having a first illumination wavelength, and a second light component having a second illumination wavelength; controlling a RGB imaging device to obtain two or more sequences of image frames of the skin, wherein each sequence of image frames is obtained using a respective light detection channel of the RGB imaging device, wherein the RGB imaging device comprises a first light detection channel that is responsive to light in a first spectral band to generate a first sequence of frames of the skin, and a second light detection channel that is responsive to light in a second spectral band to generate a respective a second sequence of frames of the skin, wherein the first illumination wavelength is within the first spectral band, and the second illumination wavelength is within the second spectral band; processing the first sequence of frames to calculate motion vectors for each frame, wherein each motion vector represents a movement of the skin between frames in the first sequence; aligning the frames in the second sequence of frames according to the corresponding determined motion vectors such that the skin is aligned across the frames in the second sequence of frames; and (ii) performing spectral analysis of the skin using the aligned second sequence of frames. The first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.

According to a second aspect, there is provided a computer program product according to claim <NUM>, comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method according to the fifth aspect or any embodiment thereof.

According to a third aspect, there is provided an apparatus according to claim <NUM>, configured for performing spectral analysis of skin of a subject. The apparatus comprises a processing unit configured to: (i) align image frames in a sequence of image frames of skin of a subject by: controlling one or more light sources to emit multi-spectral light to illuminate the skin, wherein the multi-spectral light comprises at least a first light component having a first illumination wavelength, and a second light component having a second illumination wavelength; controlling a RGB imaging device to obtain two or more sequences of image frames of the skin, wherein each sequence of image frames and each is obtained using a respective light detection channel of the RGB imaging device, wherein the RGB imaging device comprises a first light detection channel that is responsive to light in a first spectral band to generate a first sequence of frames of the skin, and a second light detection channel that is responsive to light in a second spectral band to generate a second sequence of frames of the skin, wherein the first illumination wavelength is within the first spectral band, and the second illumination wavelength is within the second spectral band; processing the first sequence of frames to calculate motion vectors for each frame, wherein each motion vector represents a movement of the skin between frames in the first sequence; aligning the frames in the second sequence of frames according to the corresponding determined motion vectors such that the skin is aligned across the frames in the second sequence of frames; and (ii) perform spectral analysis of the skin using the aligned second sequence of frames. The first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.

According to an fourth aspect, there is provided a system according to claim <NUM> configured for performing spectral analysis of skin of a subject. The system comprises: an apparatus according to the seventh aspect or any embodiment thereof; the one or more light sources; and the RGB imaging device.

<FIG> is a simplified illustration of a system <NUM> for obtaining and aligning image frames of a subject <NUM> according to an embodiment the techniques described herein. The subject <NUM> is typically a human subject, although the system <NUM> and techniques can be used with non-human subjects, and/or objects, where movement of the subject in a sequence of image frames may occur as the image frames are obtained. When used with a human subject <NUM>, the techniques described herein are particularly useful where the obtained image frames are to be spectrally analysed to determine one or more properties of the skin, for example relating to the chromophores present in the skin.

The system <NUM> generally comprises an apparatus <NUM>, a multi-channel imaging device <NUM> and one or more light source(s) <NUM>. In the embodiment shown in <FIG>, the apparatus <NUM> comprises a processing unit <NUM>, and also optionally comprises a memory unit <NUM> and/or one or more user interface components <NUM>.

The processing unit <NUM> is provided to control the operations of the multi-channel imaging device <NUM> and the light source(s) <NUM> to illuminate the subject <NUM> with the required multi-spectral light and to obtain the sequence of image frames of the subject <NUM>. The processing unit <NUM> also operates according to the techniques described herein to align the image frames in the obtained sequence.

The processing unit <NUM> can be implemented in numerous ways, with software and/or hardware, to perform the various functions described herein. The processing unit <NUM> may comprise one or more microprocessors or digital signal processors (DSPs) that may be programmed using software or computer program code to perform the required functions and/or to control components of the processing unit <NUM> to effect the required functions. The processing unit <NUM> may be implemented as a combination of dedicated hardware to perform some functions (e.g. amplifiers, pre-amplifiers, analog-to-digital convertors (ADCs) and/or digital-to-analog convertors (DACs)) and a processor (e.g., one or more programmed microprocessors, controllers, DSPs and associated circuitry) to perform other functions. Examples of components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, DSPs, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), hardware for implementing a neural network and/or so-called artificial intelligence (AI) hardware accelerators (i.e. a processor(s) or other hardware specifically designed for AI applications that can be used alongside a main processor).

The processing unit <NUM> can be connected to a memory unit <NUM> that can store data, information and/or signals for use by the processing unit <NUM> in executing or performing the techniques described herein. For example the memory unit <NUM> can store image frames obtained by the multi-channel imaging device <NUM>, and/or store the image frames after alignment by the processing unit <NUM>. In some implementations the memory unit <NUM> stores computer-readable code that can be executed by the processing unit <NUM> so that the processing unit <NUM> performs one or more functions, including the techniques described herein. The memory unit <NUM> can comprise any type of non-transitory machine-readable medium, such as cache or system memory including volatile and non-volatile computer memory such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM) and electrically erasable PROM (EEPROM), and the memory unit <NUM> can be implemented in the form of a memory chip, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD) or a Blu-Ray disc), a hard disk, a tape storage solution, or a solid state device, including a memory stick, a solid state drive (SSD), a memory card, etc..

The user interface component(s) <NUM> may be provided that includes one or more components that enables a user of the system <NUM> and/or apparatus <NUM> (e.g. the subject <NUM> or another party) to input information, data and/or commands into the system <NUM> and/or apparatus <NUM>, and/or enables the apparatus <NUM> to output information or data to the user. For example, the user interface components <NUM> can include a display screen for displaying one or more image frames of the subject, before and/or after the image frames are aligned according to the techniques described herein. A display screen or other visual display element may also or alternatively display the result of any spectral analysis on the aligned image frames. The user interface components <NUM> can comprise any suitable input component(s), including but not limited to a keyboard, keypad, one or more buttons, switches or dials, a mouse, a track pad, a touchscreen, a stylus, a camera, a microphone, etc., and/or the user interface components <NUM> can comprise any suitable output component(s), including but not limited to a display screen, one or more lights or light elements, one or more loudspeakers, a vibrating element, etc..

The multi-channel imaging device <NUM> is provided to obtain a sequence of image frames of a subject <NUM>. The multi-channel imaging device <NUM> comprises a plurality of light detection channels that are each responsive to light in a respective spectral band. Each detection channel outputs a respective sequence of frames of the subject. The multi-channel imaging device <NUM> in <FIG> comprises a first light detection channel <NUM> that is responsive to light in a first spectral band and a second light detection channel <NUM> that is responsive to light in a second spectral band. Thus, the first light detection channel <NUM> outputs a first sequence of image frames formed from incident light in the first spectral band, and the second light detection channel <NUM> outputs a second sequence of image frames formed from incident light in the second spectral band.

The first and second spectral bands are non-overlapping, i.e. the first and second spectral bands cover separate ranges of wavelengths. Optionally, the multi-channel imaging device <NUM> can comprise at least a third light detection channel that is responsive to light in a third spectral band to output a third sequence of image frames (with the wavelengths covered by the third spectral band being non-overlapping with the wavelengths in the first and second spectral bands).

As described further below, light having a wavelength in the first spectral band is used as 'reference light' in order to identify movement of the subject between image frames. Consequently, the first spectral band is also referred to herein as the "reference band". In preferred embodiments, in particular where the image frames are images of skin, the first spectral band encompasses green light, for example including light having wavelengths of around <NUM> nanometres (nm) and/or <NUM>. Green light is preferred for skin imaging applications, as images of skin under green light are effective for skin tracking, i.e. for identifying features on the skin such as wrinkles, spots, scars, moles, etc. In these embodiments, the second spectral band may encompass red light or blue light. In the case of red light, the second spectral band can include light having wavelengths of around <NUM>, <NUM>, <NUM> and/or <NUM>. In the case of blue light, the second spectral band can include light having wavelengths of around <NUM>, <NUM>, and/or <NUM>. In alternative embodiments, for example where image frames obtained using green light are also used for spectral analysis of the subject <NUM>, the first spectral band can encompass infrared (IR) light instead of green light.

In embodiments where the first light detection channel <NUM> is responsive to green light, and the second light detection channel <NUM> is responsive to red or blue light (and optionally when the multi-channel imaging device <NUM> has a third light detection channel responsive to the other one of red and blue light), the multi-channel imaging device <NUM> may be in the form of a RGB camera or imaging unit.

As noted, each light detection channel <NUM>, <NUM> obtains or generates a respective sequence of image frames. The image frames are generated according to a common frame rate, i.e. the light detection channels each obtain/generate a respective image frame at the same time, and at the same frame rate. For example, the light detection channels <NUM>, <NUM> can obtain/generate <NUM> image frames per second (fps), although other frame rates could be used.

In typical implementations, the multi-channel imaging device <NUM> is a 'rolling-shutter' type of imaging device, i.e. an imaging device where each light detection channel captures the image frame line-by-line rather than by capturing the entire frame at once.

The one or more light source(s) <NUM> are provided in order to emit multi-spectral light to illuminate the subject <NUM> when image frames are to be obtained by the multi-channel imaging device <NUM>. The multi-spectral light comprises at least a first light component having a first illumination wavelength λ<NUM> and a second light component having a second illumination wavelength λ<NUM>. The first illumination wavelength λ<NUM> is a wavelength in the first spectral band and the second illumination wavelength λ<NUM> is a wavelength in the second spectral band. The one or more light source(s) <NUM> are configured to emit the first light component and the second light component at the same time (i.e. simultaneously). According to the preferred embodiment of the multi-channel imaging device <NUM> described above, the first illumination wavelength λ<NUM> can be a wavelength of green light, and the second illumination wavelength λ<NUM> can be a wavelength of red or blue light.

In some embodiments, the multi-spectral light emitted by the one or more light source(s) <NUM> further comprises at least a third light component having a third illumination wavelength λ<NUM> that is in a third spectral band. For example, where the second illumination wavelength λ<NUM> is a wavelength of red light, the third illumination wavelength λ<NUM> can be a wavelength of blue light, or vice versa.

In some embodiments, the one or more light source(s) <NUM> may be configured to selectively emit multiple light components having respective illumination wavelengths within a particular spectral band. In these embodiments, the light source(s) <NUM> emit the light components having different illumination wavelengths within a particular spectral band at different times (e.g. the light source(s) <NUM> alternate between emitting two or more illumination wavelengths within a particular spectral band). While light components within a particular spectral band are alternated, it will be appreciated that the light source(s) <NUM> can continue to emit a light component in another spectral band. For example, the light source(s) <NUM> can emit a first light component within a first spectral band having a wavelength of <NUM> (i.e. green light), and the light source(s) <NUM> can alternately switch between emitting a red light component in a second spectral band having a wavelength of <NUM>, and another red light component having a wavelength of <NUM>.

In some embodiments, the one or more light source(s) <NUM> comprises a respective light source <NUM> for each light component that is to be emitted. Alternatively, the one or more light source(s) <NUM> can comprise a single light source that can be controlled to emit multiple light components having respective illumination wavelengths. In some embodiments, the required illumination wavelengths can be achieved using respective colour filters.

The one or more light sources <NUM> may be light emitting diodes (LEDs) or diode lasers, for example. As noted, each light source <NUM> may be configured to emit light at specific wavelengths or may be controllable to emit light at varying wavelengths.

In the embodiment shown in <FIG>, the apparatus <NUM>, the multi-channel imaging device <NUM> and the one or more light source(s) <NUM> are separate from each other. Therefore, although not shown in <FIG>, suitable communication circuitry or communication components may be provided to enable control signals and other information (e.g. data representing image frames obtained by the multi-channel imaging device <NUM>) to be communicated to/from the apparatus <NUM>. Thus, communication circuitry or communication components can be provided to enable a data connection and/or data exchange between the apparatus <NUM>, the multi-channel imaging device <NUM>/imaging unit and/or the one or more light source(s) <NUM>/illumination unit, depending on the implementation. The connection may be direct or indirect (e.g. via the Internet), and thus the communication circuitry or communication components can enable a connection between the apparatus <NUM> and the multi-channel imaging device <NUM>/imaging unit and/or the one or more light source(s) <NUM>/illumination unit via a network (such as the Internet), or directly between the apparatus <NUM> and the multi-channel imaging device <NUM>/imaging unit and/or one or more light source(s) <NUM>/illumination unit, via any desirable wired or wireless communication protocol. For example, the communication circuitry or communication components can operate using WiFi, Bluetooth, Zigbee, or any cellular communication protocol. In the case of a wireless connection, the communication circuitry or communication components (and thus apparatus <NUM> and multi-channel imaging device <NUM>/imaging unit and one or more light source(s) <NUM>/illumination unit) may include one or more suitable antennas for transmitting/receiving over a transmission medium (e.g. the air). Alternatively, in the case of a wireless connection, the communication circuitry or components may include means (e.g. a connector or plug) to enable the communication circuitry or components to be connected to one or more suitable antennas external to the apparatus <NUM> and multi-channel imaging device <NUM>/imaging unit and one or more light source(s) <NUM>/illumination unit for transmitting/receiving over a transmission medium (e.g. the air).

It will be appreciated that a practical implementation of the system <NUM> may include additional components to those shown in <FIG>. For example, the system <NUM> may include one or more power supplies for the various components of the system <NUM>, such as batteries, or components for enabling the system <NUM>, and/or parts of the system <NUM>, to be connected to a mains power supply.

Although in the embodiment illustrated in <FIG> the multi-channel imaging device <NUM> and the one or more light source(s) <NUM> are shown as being separate from the apparatus <NUM>, it will be appreciated that in alternative embodiments one or both of the multi-channel imaging device <NUM> and the one or more light source(s) <NUM> can be integrated with, or be part of, the apparatus <NUM>. In embodiments where the multi-channel imaging device <NUM> is separate from the apparatus <NUM>, the multi-channel imaging device <NUM> can be considered to be part of an imaging unit that can comprise one or more other components or circuitry for enabling the operation of the multi-channel imaging device <NUM> (e.g. a power source, control circuitry, etc.). Likewise, in embodiments where the one or more light source(s) <NUM> are separate from the apparatus <NUM>, the light source(s) <NUM> can be considered to be part of an illumination unit that can comprise one or more other components or circuitry for enabling the operation of the one or more light source(s) <NUM> (e.g. a power source, control circuitry, etc.).

For example, the apparatus <NUM> can be in the form of, or be part of, an electronic device such as a smartphone, tablet, smart watch, laptop, computer, server, etc. Embodiments where the apparatus <NUM> and the multi-channel imaging device <NUM> are integrated together include where the apparatus <NUM> is a smartphone, tablet or other device that typically includes an imaging unit. In these embodiments, the one or more light source(s) <NUM> can be, or correspond to, a controllable 'camera flash' light source(s). Alternatively, the one or more light source(s) <NUM> can be provided in a separate component or device that can be connected to and controlled by the smart phone/tablet/laptop/etc. to emit the required multi-spectral light.

Briefly, according to the techniques described herein, the processing unit <NUM> receives sequences of image frames from each of the light detection channels <NUM>, <NUM> that were obtained while the one or more light source(s) <NUM> illuminated the subject <NUM> with light of the appropriate illumination wavelengths, processes the first sequence of frames (i.e. provided by the first light detection channel <NUM>) to calculate motion vectors for each frame, and align the frames in the second (and any other) sequence of frames according to the calculated motion vectors. Each motion vector represents a movement of the subject <NUM> between frames in the first sequence. Thus, aligning the frames according to the motion vectors means that the subject is aligned across the frames in the second (and any other) sequences.

The flow chart in <FIG> illustrates a method of aligning image frames according to an embodiment of the techniques described herein. The method starts at step <NUM>. In step <NUM> the subject <NUM> is illuminated with the first light component (i.e. the reference light). At the same time, in step <NUM> the subject <NUM> is illuminated with the second light component (e.g. red light), and any other desired light components in spectral bands other than the first spectral band. As described further below, the first light component and/or second light component (and/or any other light components) may be pulsed, so the specific light components illuminating the subject <NUM> at any particular time instant may vary. In particular embodiments, the rate at which the light is pulsed is higher, or much higher, than the frame rate of the multi-channel imaging device <NUM>.

In step <NUM>, which occurs while light is illuminating the subject <NUM> according to steps <NUM> and <NUM>, the multi-channel imaging device <NUM> is used to capture one or more image frames. That is, each light detection channel <NUM>, <NUM> in the multi-channel imaging device <NUM> generates one or more image frames from the incident light.

Then, in step <NUM>, the one or more image frames generated by the first light detection channel <NUM> in the multi-channel imaging device <NUM> are stored for subsequent processing. The image frame(s) generated by the first light detection channel <NUM> are also referred to herein as "reference frames".

At the same time as step <NUM>, in step <NUM>, the one or more image frames generated by the second light detection channel <NUM> (and any further light detection channels) in the multi-channel imaging device <NUM> are stored for subsequent processing. The image frame(s) generated by the second light detection channel <NUM> (and any other non-first light detection channel) are also referred to herein as "spectral frames", since after alignment these frames can be spectrally analysed to determine one or more properties of the subject <NUM> in the images.

The method then passes to step <NUM> in which the obtained reference frames are processed to calculate a motion vector for each reference frame that represents a movement of the subject <NUM> in the reference frame relative to the previous reference frame. Step <NUM> can comprise using conventional techniques for comparing multiple reference frames to identify landmarks or other significant features in the reference frames (e.g. edges of objects/subjects), and to determine the movement of those landmarks or features between the reference frames. That movement can be represented as a motion vector indicating how the subject <NUM> has moved between the reference frames. Those skilled in the art will be aware of various techniques that can be used to implement step <NUM>, including Farnebäck's optical flow for motion estimation (e.g. as described in "Two-Frame Motion Estimation Based on Polynomial Expansion" by Gunnar Farnebäck), and other optical flow techniques, and details are not provided herein.

Next, in step <NUM>, the spectral frames (i.e. the sequence of image frames output by the second light detection channel <NUM>) are aligned based on the motion vectors determined in step <NUM>. That is, for a particular spectral frame, the motion vector determined for the corresponding reference frame (i.e. the reference frame generated by the first light detection channel <NUM> at the same time as the particular spectral frame generated by the second light detection channel <NUM>) is applied to the particular spectral frame to align that spectral frame with the previous spectral frame. This is repeated for each spectral frame to provide an aligned sequence of spectral frames in which the subject <NUM> is in the same position in each of the frames.

In more detail, motion estimation in step <NUM> results in a motion vector, so for each reference frame the amount of (local) displacement is known. These displacements now need to be superimposed onto the spectral frames. Since the sequence(s) of spectral frames and the reference frames are acquired simultaneously by the same multi-channel imaging device <NUM>, they are perfectly mapped onto each other. The calculated motion vector is directly mapped through the mapping function and applied to the spectral image(s). As a result, the sequence(s) of spectral images will be properly aligned.

In some embodiments, the one or more light source(s) <NUM> are controlled to provide pulses of light that include light components with different illumination wavelengths. For example, the one or more light source(s) <NUM> can be controlled to follow the light pulse pattern below:.

The specific illumination wavelengths of these light components can be as follows:.

In a specific example, each light pulse (light pulse <NUM>, light pulse <NUM>, light pulse <NUM>, etc.) can have a duration of <NUM> milliseconds (ms). Further, each light component can have a colour temperature and luminance set so that the pulsing light is perceived by a user and/or the subject <NUM> as constant white light, without flickering or colour shifting being perceived.

When the multi-channel imaging device <NUM> is a rolling-shutter RGB imaging device with a frame rate of <NUM> fps, the light pulses produce around <NUM> bright and dark bands in each image frame output by the red, green and blue light detection channels. The image frames output by the green light detection channel are used to calculate the motion vectors for each image frame as described above in step <NUM>, and these motion vectors are applied to the image frames output by the red light detection channel and the blue light detection channel to align the subject <NUM> in the image frames.

To enable subsequent spectral analysis of the image frames, the 'banded' image frames from each light detection channel (i.e. the image frames including <NUM> bright and dark bands) can be decomposed into <NUM> separate wavelength-specific spectral image frames (i.e. one wavelength-specific spectral image frame per wavelength of light included in the light pulses).

In a further embodiment, to improve the spectral analysis, the effect of ambient light has to be removed from the image frames. In that case, the above light pulse pattern can be modified to include a 'light pulse' where no light is emitted by the one or more light source(s) <NUM> and the multi-channel imaging device <NUM> generates image frames from incident ambient light. For example, the one or more light source(s) <NUM> can be controlled to follow the light pulse pattern below:.

Again, each light pulse (light pulse <NUM>, light pulse <NUM>, light pulse <NUM>, light pulse <NUM>) can have a duration of <NUM>. Further, each light component can have a colour temperature and luminance set so that the pulsing light is perceived by a user and/or the subject <NUM> as constant white light, without flickering or colour shifting being perceived.

When the multi-channel imaging device <NUM> is a rolling-shutter RGB imaging device with a frame rate of <NUM> fps, the four light pulses produce around <NUM> bright and dark bands in each image frame output by the red, green and blue light detection channels. Alignment of the image frames is performed as before.

To enable subsequent spectral analysis of the image frames, the 'banded' image frames from each light detection channel are decomposed into an ambient light image frame and <NUM> separate wavelength-specific spectral image frames (i.e. one wavelength-specific spectral image frame per wavelength of light included in the light pulses). The effect of the ambient light can be removed from the wavelength-specific spectral image frames by linear subtraction of the ambient light image frame from each of the wavelength-specific spectral image frames before any spectral analysis is performed on the spectral images.

The flow chart in <FIG> illustrates a general method of aligning image frames according to the techniques described herein. In some embodiments, the method can be performed by the processing unit <NUM> in the apparatus <NUM>, for example in response to the processing unit <NUM> executing suitable computer readable code. Such code can be stored in or on the memory unit <NUM>, or on an alternative type of computer readable medium.

In step <NUM>, one or more light sources <NUM> are controlled to emit multi-spectral light to illuminate the skin of the subject <NUM>. The multi-spectral light comprises at least a first light component having a first illumination wavelength, and a second light component having a second illumination wavelength. The first light component is also referred to as the "reference light" or "reference light component".

In step <NUM>, a multi-channel imaging device <NUM> is controlled to obtain a sequence of image frames of the skin of the subject <NUM>. Each sequence of image frames is obtained using a respective light detection channel <NUM>, <NUM> of the multi-channel imaging device <NUM>. Step <NUM> is performed while the one or more light sources <NUM> are emitting the multi-spectral light. The multi-channel imaging device <NUM> comprises a first light detection channel <NUM> that is responsive to light in a first spectral band to generate a first sequence of frames of the skin of the subject <NUM>, and a second light detection channel <NUM> that is responsive to light in a second spectral band to generate respective a second sequence of frames of the skin of the subject <NUM>. The first illumination wavelength is within the first spectral band, and the second illumination wavelength is within the second spectral band.

In step <NUM>, the first sequence of frames is processed to calculate motion vectors for each frame. Each motion vector represents a movement of the skin of the subject <NUM> between frames in the first sequence.

In step <NUM>, the frames in the one or more further sequences of frames according to the corresponding determined motion vectors such that the skin of the subject is aligned across the frames in the second sequence of frames.

In some embodiments, the method further comprises a step of performing a spectral analysis of the skin of the subject <NUM> using the aligned frames.

In some embodiments, step <NUM> comprises alternately activating the one or more light sources <NUM> to emit the multi-spectral light and deactivating the one or more light sources <NUM>. In these embodiments, step <NUM> comprises controlling the multi-channel imaging device <NUM> to obtain the sequences of frames while the one or more light sources are activated and deactivated. These embodiments can further comprise the step of processing the sequences of frames to determine an ambient light contribution to the sequences of frames corresponding to ambient light around the subject <NUM> when the one or more light sources were deactivated. Then, after step <NUM>, the aligned frames in the second sequence of frames can be corrected for the ambient light using the determined ambient light contribution. In some embodiments, the determined ambient light contribution can be subtracted from one or more, or all, of the frames in the second sequence, and optionally also the first sequence of frames. Any spectral analysis to be performed is then performed on the corrected aligned frames.

In some embodiments, the multi-spectral light can include a third light component having a third illumination wavelength that is within a third spectral band, and the multi-channel imaging device <NUM> can include a third light detection channel that is responsive to light in the third spectral band for generating a third sequence of frames of the subject <NUM>. In these embodiments, the frames in the third sequence of frames can also be aligned according to the corresponding determined motion vectors such that the skin is aligned across the frames in the third sequence of frames. In embodiments where spectral analysis of the skin is performed, the spectral analysis can be performed on one or both of the aligned second sequence of frames and third sequence of frames.

In some embodiments, the multi-channel imaging device <NUM> is a RGB imaging device. In these embodiments, the first light detection channel can be a green channel of the RGB imaging device, the second light detection channel is one of the blue and red channels of the RGB imaging device, and the third light detection channel is the other one of the blue and red channels.

In some embodiments, the multi-spectral light can include a third (or further) light component having a third (or further) illumination wavelength, with both the second illumination wavelength and the third illumination wavelength being in the second spectral band. In this case, the step of controlling the one or more light sources <NUM> comprises controlling the one or more light sources <NUM> to alternately emit the second light component and the third light component (i.e. alternate between the two colours of light in the second spectral band). In these embodiments, the multi-channel imaging device <NUM> may be a rolling-shutter imaging device having a frame rate that is less than a frequency with which the second light component and the third light component are alternated. Preferably, the bands of different light components in the frames need to be separated and recombined into frames relating to a single light component to enable spectral analysis of the subject <NUM> to be performed. Therefore, in some embodiments the method further comprises, for the second sequence of aligned frames, decomposing a subset of the aligned frames according to the second illumination wavelength and the third illumination wavelength to provide frame components for the second illumination wavelength and frame components for the third illumination wavelength. Then, for each illumination wavelength in the second spectral band, the frame components for that illumination wavelength are combined to form a composite frame for that illumination wavelength.

In the above embodiment with the multi-spectral light including a third (or further) light component having a third (or further) illumination wavelength in the second spectral band, ambient light correction can also be applied. In this case, step <NUM> can comprise controlling the one or more light sources to alternately emit the second light component, emit the third light component and not emit light. The method can further comprise aligning the frames in the first sequence of subframes according to the corresponding motion vectors determined in step <NUM>; for each sequence of aligned frames, decomposing a subset of the aligned frames to provide ambient light frame components corresponding to periods of time where the one or more light sources were not emitting light; and combining the ambient light frame components for all sequences of aligned frames to form a composite frame representing ambient light around the subject <NUM> when the one or more light sources were not emitting light. Finally, each sequence of aligned frames can be corrected for the ambient light using the determined composite frame representing ambient light.

Therefore there is provided an improved method and apparatus for aligning image frames of a subject obtained using multi-spectral light.

Claim 1:
A method for performing spectral analysis of skin of a subject, the method comprising:
(i) aligning image frames in a sequence of image frames of skin of a subject by:
controlling (<NUM>) one or more light sources to emit multi-spectral light to illuminate the skin, wherein the multi-spectral light comprises at least a first light component having a first illumination wavelength, and a second light component having a second illumination wavelength;
controlling (<NUM>) a RGB imaging device to obtain two or more sequences of image frames of the skin, wherein each sequence of image frames is obtained using a respective light detection channel of the RGB imaging device, wherein the RGB imaging device comprises a first light detection channel that is responsive to light in a first spectral band to generate a first sequence of frames of the skin, and a second light detection channel that is responsive to light in a second spectral band to generate a respective a second sequence of frames of the skin, wherein the first illumination wavelength is within the first spectral band, and the second illumination wavelength is within the second spectral band;
processing (<NUM>) the first sequence of frames to calculate motion vectors for each frame, wherein each motion vector represents a movement of the skin between frames in the first sequence;
aligning (<NUM>) the frames in the second sequence of frames according to the corresponding determined motion vectors such that the skin is aligned across the frames in the second sequence of frames; and
(ii) performing spectral analysis of the skin using the aligned second sequence of frames to determine one or more properties of the skin;
wherein the first light component is a green light component, the first spectral band is a green spectral band, the second light component is one of a red light component and a blue light component, and the second spectral band is a corresponding one of a red spectral band and a blue spectral band.