Patent Description:
A typical Complementary Metal Oxide Semiconductor (CMOS) camera sensor has a certain resolution and operates in a synchronous fashion in which each collection of sensor data is read out and transferred to a host circuit at a certain frame rate. The analysis of the data (e.g., object detection) is then typically handled in the host processing subsystem, either in a central processing unit (CPU), a graphics processing unit (GPU), or in a neural accelerator or neural processing unit (NPU). This leads to high power consumption and is not feasible for a device that needs to have such functionality always on while at the same time being battery-operated. There are different approaches known for how to reduce the power consumption.

One example involves having only some parts of the sensor active, while others remain inactive. Such an approach is described in <CIT>.

In another example, described in International Application Number <CIT>, a first level of image analysis takes place with low power consumption in the sensor itself, and a deeper analysis in the host processor is activated only when a positive identification (but at a lower confidence level) is made at the first level of analysis.

There are also strategies proposed for optimizing activity detection, and these typically trigger on any lighting change and/or activity. Such strategies are described in.

<CIT> shows an image sensor with normal CMOS pixels. The sensor has an ultra-low power mode to detect changes in an image, a low power which operates at reduced resolution and/ or frame rate to detect large/slow gestures and a normal mode to detect the smallest and fastest gestures accurately.

<CIT> shows an image sensor with both event detection pixels and CMOS image pixels. An external CPU analyzes the image or motion data output from the sensor and reconfigures its operation to be either an event or an image sensor.

A Dynamic Vision Sensor (DVS), also known as an event camera, a neuromorphic camera or a silicon retina, is described in <NPL>. Unlike a conventional CMOS sensor, which has a specific shutter speed, a DVS triggers asynchronously based on local changes in the brightness of the sensed light. The temporal framerate can be up to <NUM>,<NUM>,<NUM> frames per second in state-of-the-art dynamic vision sensors. Other benefits are that its dynamic range of 120dB is superior to that of other sensors and it does not suffer from motion blur or over/under exposure.

<FIG> is a schematic diagram of a DVS sensor <NUM> as disclosed in the above-referenced Lichsteiner document. The design comprises a fast-logarithmic photoreceptor circuit <NUM> coupled to a differencing circuit <NUM> that amplifies changes with high precision. An output of the differencing circuit <NUM> is supplied to a two-transistor comparator circuit <NUM>. The indicated inverters are symbols for single-ended inverting amplifiers.

The DC mismatch is removed by balancing the output of the differencing circuit to a reset level after the generation of an event. The gain of the change amplification is determined by the well-matched capacitor ratio C<NUM>/C<NUM>. The effect of comparator mismatch is reduced by the precise gain of the differencing circuit. The principle of operation of the DVS sensor <NUM> is depicted in <FIG>.

The discussion will now focus on technology in which always-on optical sensing devices are employed. To illustrate one example, <FIG> is a block diagram of a low-power object detection/identification system <NUM> for authorizing performance of a device action. As non-limiting examples, such actions include:.

In each instance, the associated device is in some way associated with the system <NUM>. There are many types of such associated devices and a complete list is beyond the scope of this description. Non-limiting examples include vehicle locking mechanisms (e.g., vehicle doors, vehicle engines); home locking systems; workplace locking systems; workplace entry authorization systems; secure area authorization systems; consumer appliance (including electronic device) start and stop functions; consumer device (including electronic device) activation/deactivation functions.

In one class of embodiments, extended reality (XR) glasses with an image sensor can be employed as a sensing device, and the system <NUM> configured to recognize when a particular user and/or object is within a line of sight of the XR glasses. It will further be recognized that opening/activating locked items or devices is but one example out of many possible examples and is for purposes of illustration. As mentioned before, other types of actions can be performed when the system <NUM> recognizes that a particular object or person is present, and all are contemplated to be within the scope of inventive embodiments.

The system <NUM> comprises a camera module <NUM> that can include a standard Red Green Blue (RGB) Camera Image Sensor (CIS). In alternative embodiments, cameras with different setups of color filters can instead be used if the use case is specific and only needs a specific set of colors. In yet other alternative embodiments, the camera module <NUM> can employ a monochrome camera using only one color or no color filter. In that case the threshold in the system (discussed later) would be a gradient of the specific color.

The camera module <NUM> is coupled to a host system <NUM> that includes an Image Signal Processor (ISP) <NUM> that controls the camera module <NUM> down to subsets of its sensing capability. These subsets are herein called "sensing structures" and can be different in different embodiments. As non-limiting examples, a "sensing structure" can be any of the following:.

It will be observed that, since a pixel can itself be associated with one of a number of different color filters, a sensing structure can also be associated with a particular color or set of colors.

The ISP <NUM> that controls the camera module <NUM> is, in some embodiments, custom made to have specific characteristics for a specific use case.

The host system <NUM> further includes an analyzer <NUM> that controls threshold levels that are used when deciding whether a sensed image sufficiently matches a given model. The analyzer <NUM> also decides what power level the system <NUM> should operate in.

The host system <NUM> also includes a central processing unit (CPU) <NUM> for general-purpose calculations.

The exemplary host system <NUM> also includes a Graphics Processing Unit (GPU) / Neural Processing Unit (NPU) <NUM> that is configured to run a neural network for detection and identification of sensed image features and compares these with one or more models of authorized objects/users. In addition, or as an alternative, identification can be accomplished by running state-of-the-art algorithms that are known to those of ordinary skill in the art.

The exemplary host system <NUM> also includes a memory <NUM> for storing things such as, but not limited to, neural network weights, images, settings for hardware, and other needed software features.

There exist applications in which power savings are achieved by performing a device authorization / activation process in stages, with a lowest-power stage being performed by a camera module <NUM> configured such as in the exemplary embodiment illustrated in <FIG>. The camera module <NUM> comprises a sensor array <NUM> comprising a plurality of sensing structures that are configurable as will be discussed further with reference to <FIG>. the camera module <NUM> further comprises a control unit <NUM> for controlling the various components of the camera module <NUM> in a way that conforms with the various actions described herein.

In one respect, the camera module <NUM> is able to make an initial conclusion whether an object or person being presented to the sensor module <NUM> is one of one or more previously authorized objects / persons, and to facilitate this function the camera module <NUM> includes a register <NUM> for users/objects. As will be discussed further below, recognition of an object/person includes detecting whether a sufficient number of activated sensing structures are being triggered by the object/person being presented, and this means comparing the number of triggered sensing structures to a threshold. It is a purpose of the register <NUM> for users/objects to store settings for all of the objects/users that are available in a given model. There may be one or more than one such object/user.

The initial conclusion made by the camera module <NUM> should be understood as being a sufficiently close match (within a defined threshold) between the results of the sensed data and the data stored in the register <NUM> per user/object or version of user/object, and the like. The initial conclusion is not made with <NUM>% certainty. But finding an initial match provides sufficient confidence to warrant engaging a higher power stage of analysis that can be used to further enhance the detection and security of the detection with additional sensors or pixels to give higher resolution and the like. By forming an initial conclusion as presented herein, the system is able to expend only minimal energy in order to avoid having non-matching conditions constantly wake up the higher power system. The ratio of negative detections will likely always be much higher than positive results, thus resulting in significant power savings.

In another respect, the camera module <NUM> is able to detect when no object/user is being presented to the sensor module <NUM> and to remain in a very low power state under such circumstances, and then to be able to revert to a more active state when something (object/person) is then presented to the sensor module <NUM>. As will be described further below, this function involves tracking whether the sensor module <NUM> presently detects an image corresponding only to an image of an environmental background (i.e., an image without an object or person being presented to the sensor module <NUM>). So long as only background is detected, the camera module <NUM> can remain in the very low power state. If a sufficient enough change to the scanned image is detected, the more active state is entered. To facilitate this purpose, the camera module <NUM> further includes a register <NUM> for a background image. It is a purpose of the register <NUM> for background to store settings for a background that is available in a given model.

<FIG> is a schematic / state diagram illustrating the concept of a multi-stage authorization / activation process.

Processing effort begins at a minimal level, depicted as the smallest circle in the left-most position in the diagram, which represents a lowest-power processing state <NUM>. Embodiments consistent with the invention, such as but not limited to the exemplary system <NUM> of <FIG>, perform the processing of at least state <NUM>. A baseline model selects a first number of sensing structures of the camera image sensor <NUM>, with the number being less than a total number of sensing structures of the camera image sensor <NUM>. The baseline model is configured to detect a very small number of key image features that would be expected to be found in an image of an authorized object/user, and the first number of sensing structures is selected to be able to detect this small number of key image features.

If the features are not found in the sensed image ("Fail") then further testing, and its consequent greater expenditure of energy, is avoided.

But if the features are found from this sensing ("Pass"), authorization has passed this first stage of testing, and testing moves on to a next state <NUM> in which a larger number, but depending on embodiment perhaps still not the total number, of sensing structures are activated to detect a match between a sensed image and a model, this time at a higher level of granularity. A failure to match at this level can cause the processing to revert to the initial lowest-power state <NUM>. But a "pass" allows still further testing, and further power to be expended, to ensure that the match is accurate.

The number of stages performed before a final conclusion is reached is implementation dependent. In the non-limiting example of <FIG>, four states <NUM>, <NUM>, <NUM>, and <NUM> are shown. But in other embodiments there could be more or fewer states, each associated with a different amount of processing effort to detect whether a sensed image matches a model of an authorized object/user. An advantage of this multi-stage testing arrangement is that, at the conclusion of processing, the system is able to detect, with as little energy consumption as possible, whether a known object/user has been identified, and this reduction in energy consumption enables the system to employ an always-on camera.

To illustrate the power consumption aspect, consider an example shown in <FIG> in which an <NUM> megapixel (MP) camera having an array <NUM> of 3264x2448 pixel elements consuming about <NUM> mW is used as the camera module <NUM>. For purposes of this example, it is assumed that each pixel consumes the same amount of energy. Assuming one color and one row we get (only half of the pixels in a row have one color) <NUM>/<NUM> * <NUM> → <NUM>. <NUM> mW per row → <NUM> uW per row and color. The sensor usually has <NUM>-<NUM> bit raw sensor bit depth, but this could be quantized to minimize the power consumption. It could also be that the A/D converter in the camera in low power mode could be one bit and it will be the trigger signal itself.

It is advantageous to have the lowest level of power handled in the camera module <NUM> itself. Self-contained hardware embodiments of the camera module <NUM> can be configured for this purpose in order to enable a low-power, always-on embodiment. Based on the baseline signature a few lines of the CIS <NUM> will be active. For example, suppose a user has a green shirt and blue trousers at the time of using the system <NUM>. The system <NUM> adds these important colors to the model during a model updating phase of processing. When entering the lowest power level the system, the camera module <NUM> is activated accordingly. In this case with the green shirt and blue trousers; green pixels in a few rows are activated at the top of the CIS <NUM> (above the center of the vertical field of view for example) and blue pixels are activated in the bottom of the CIS <NUM> (below the center of the vertical field of view for example). The pixel rows that are active indicate a triggering event once a threshold of pixel value is found. The algorithm used to analyze the selected set of pixels is also robust enough to manage a range of values that can identify a certain pattern but with alignment towards the sensor somewhat askew. In the example it can be one of many situations affecting the intensity of certain colors for example slightly lower, to the side or further away. This could be done on sensor level of the camera module <NUM> or in the ISP <NUM> running in low power mode.

It will be noted that the use of the colors green and blue in the above example are for purposes of illustration only, and are non-limiting. Any colors (e.g., purple, gold, silver) could have been used in the example, and the aspects illustrated in the exemplary embodiment would still hold, with pixel rows being selected for activation based on the extent to which they sense color components of the object's color(s).

In the example of <FIG>, a "sensing structure" is considered to be a row of pixels, and four of these are shown in an activated state: a first sensing structure <NUM> for detecting blue; a second sensing structure <NUM>, also for detecting blue; a third sensing structure <NUM> for detecting green; and a fourth sensing structure <NUM>, also for detecting green. Again, these selections are for purposes of illustration only; there is no particular significance to the number of illustrated sensing structures, their locations or orientations within the array <NUM>, the particular colors being sensed, or even the fact that pixels are selected to form a contiguous portion (e.g., a row) of the array <NUM>. To highlight this point, the schematically depicted sensing structure <NUM> is included in the figure to represent sensing structures in the "general case".

Remaining sensing structures illustrated in <FIG> are, in the illustration, inactive.

If an image obtained from the activated sensing structures <NUM>, <NUM>, <NUM>, <NUM> passes a first round of testing against a model, then a next level of identification could, for purposes of example, be that <NUM>/<NUM> of the Vertical Field of View (VFOV) starting at the top looks at green and <NUM>/<NUM> of the VFOV starting from the bottom looks at blue. The question being asked could be, for example, is it possible to identify a green shirt and blue trousers? If yes and the image obtained from the further activated sensing structures sufficiently matches this level's model, then processing moves on to a next power level; otherwise processing reverts back to an adaptive identification sleep mode. The identification processing can be performed by the neural processing unit <NUM> and the analyzer <NUM> can be used to decide whether testing at the next power level should be activated or instead whether the current (higher) power level should be deactivated based on the result of the neural network.

To take the example further, a third level of testing can be that the full FOV is activated with all colors but at a lower resolution, for example 1MP instead of 8MP (if that is the maximum resolution of the camera module <NUM>). The image is again sent to the neural processing unit <NUM> and the analyzer <NUM>, this time with a different set of weights. Testing can, for example, look to see whether a person and a face can be recognized; if so, the analyzer <NUM> will decide to go to the next power level.

To take the example still further, if a fourth power level of testing is reached, an image with full 8MP resolution will be taken. The ISP <NUM> will crop the face and/or other biometrical attributes and send the information to the neural processing unit <NUM> and the analyzer <NUM> for identification. If the imaged person is identified as a user, the item or device will be triggered to open / unlock depending on the use case.

To handle the case in which, at a certain point the user has changed appearance from the baseline (e.g., is no longer wearing a green shirt and blue trousers) and is not recognizable at the lowest level of testing using the most-recent (but now outdated) model, the full system can be further configured to turn on and provide the image to a more capable processor for identification/analysis. The more capable processor can be configured to be nearby the camera module <NUM> (e.g., disposed within the host system <NUM> as illustrated in <FIG>), or alternatively it can be a central computing resource (e.g., server, mobile edge cloud, etc.). The design choice of how to configure any particular embodiment depends on a number of factors, including whether the low-power system is to be a stand-alone system, or whether it is part of a larger system of sensors and gates - in the latter, an updated profile might need to be shared with the other sensor nodes. In the examples presented herein for purposes of illustrating aspects of inventive embodiments, a local application processor is assumed, since it would be the natural system implementation (there should be a CPU with an ISP and some image processing in the form of accelerator/GPU/NPU). But it will be noted that, for example, the final one or two steps <NUM>, <NUM> of processing (see <FIG>) can be performed in a system that is not local to the system <NUM>, so that communication between the sensor system and the more capable device is needed.

The change from one of the power-saving states to the full system being on can be done from any of the power levels. One trigger of this could be that when a person stands sufficiently close to the associated device (e.g., car), at least a certain number of horizontal pixels will be filled and that will trigger the full system. Other triggers are also possible.

Further aspects of embodiments consistent with the invention will now be described with reference to <FIG>, which in one respect is a flowchart of actions performed by the system <NUM> in accordance with a number of embodiments. In other respects, the blocks depicted in <FIG> can also be considered to represent means <NUM> (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

Before the system <NUM> can recognize a particular object/user, it must first perform initializing actions that include taking baseline measurements (step <NUM>) of the object/user and calculating what sensor settings (e.g., which sensing structures <NUM> should be activated) for lowest power mode recognition of a particular object/user (step <NUM>). At the conclusion of the initializing actions, the system <NUM> is ready to recognize the particular object/user when that object/user is detectable by the camera module <NUM>.

During operation, the system <NUM> is in an adaptive identification sleep mode (step <NUM>) that consumes a lowest amount of energy. The system <NUM> can be configured to detect when an object/user is being presented to the camera module <NUM> and in response to the detection, to use a previously generated baseline model of an authorized object/user to select and activate a lowest number of sensing structures <NUM>. Alternatively, the system <NUM> can be configured to be always-on with a lowest number of sensing structures <NUM> always selected and activated so that an authorized object/user can be immediately recognized when presented to the camera module <NUM>. In this alternative, there is no need to detect that an object/user has come within view of the camera module <NUM>.

In a scanning step <NUM>, signals from the active sensing structures <NUM> are compared with the baseline model (e.g., by comparing the strength of the signals to a threshold) (decision block <NUM>). If the comparison fails ("No" path out of decision block <NUM>) then the system <NUM> remains in the lowest power state and processing in some embodiments reverts back to step <NUM>.

Looping between steps <NUM> and <NUM> when the system fails to recognize a particular object/person is consistent with low-power, always-on operation and may be sufficient in some use cases. In some but not necessarily all embodiments, the system <NUM> can be further configured to handle situations in which an object or user that should be recognized is, for some reason, not recognized. To avoid endless looping, the system <NUM> can be further configured to include a mechanism that allows the user to override the low power operation and force the system to transition to a higher-power analysis that would then more accurately decide whether the object/person is recognized. For example, as shown in <FIG>, upon failing to recognize the object/user at decision block <NUM>, the system <NUM> can test to determine whether an override operation has been triggered (decision block <NUM>). An override operation can be triggered by an action such as the user pressing a button, or gripping a doorknob. The particular form of such triggering is not essential to inventive embodiments. If no override is detected ("No" path out of decision block <NUM>), then processing continues to step <NUM> as discussed above. But if an override has been triggered ("Yes" path out of decision block <NUM>), then higher level processing is invoked to provide a more reliable (and more power consuming) analysis. In some embodiments, this involves sending information about the failed scan to a host system <NUM> or other higher level processor for analysis (step <NUM>), the host system (or other) processor performing the higher level analysis (step <NUM>) and performing another test against a threshold (decision block <NUM>). If this test passes ("Yes" path out of decision block <NUM>) then the object/user is considered to be identified (step <NUM>). Further actions, triggered by the recognition, can then be performed. But if this test fails ("No" path out of decision block <NUM>), the object/person was not recognized and the system <NUM> can revert to its adaptive identification sleep mode (step <NUM>).

Referring back now to decision block <NUM>, step <NUM> is reached if the initial comparison of the signals from the active sensing structures <NUM> with the baseline model passed ("Yes" path out of decision block <NUM>). Successfully passing the lowest level testing means that the system is now willing to expend more energy to determine with greater accuracy whether the object/user presented to the camera module <NUM> is actually an authorized object/user. Accordingly, a next (increased) number of sensing structures <NUM> is activated (step <NUM>), a comparison of the activated sensor signals with a baseline measurement is made (step <NUM>) (e.g., by processing elements of the host system <NUM>), and the comparison result is compared with a predetermined threshold value (decision block <NUM>). If the comparison result is greater than or equal to the predetermined threshold value ("Yes" path out of decision block <NUM>) the object/person was again recognized by this next level of processing, so to provide even greater accuracy processing continues to step <NUM> and further actions are taken as described above. As previously discussed, this involves performing an even higher-level analysis by the host system <NUM> or other processing system that is more capable but also more energy demanding than what is done in the camera module <NUM>.

If the comparison result is lower than the predetermined threshold value ("No" path out of decision block <NUM>) then the higher number of activated sensors are deactivated (step <NUM>). At this point, actions may further include adjusting the predetermined threshold value and/or other hysteresis-related parameters), deactivating the host system <NUM> (if it had been activated) and returning to a lowest level of energy consumption operation at step <NUM>.

The exemplary embodiment of <FIG> illustrated three increasingly power consuming levels of analysis. However, other embodiments could differ, for example using only two levels or using more than three levels.

Further aspects of embodiments consistent with the invention will now be described with reference to <FIG> which, in one respect, is a flowchart of actions taken by the system <NUM> in accordance with a number of embodiments. In other respects, the blocks depicted in <FIG> can also be considered to represent means <NUM> (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

<FIG> focuses on actions taken as part of system initialization. As mentioned above, during operation the system <NUM> relies on one or more baseline models which specify key image features that should be looked for during each stage of authorization processing. <FIG> focuses on aspects relating to the creation and updating of a baseline model.

An initial test decides whether the camera module <NUM> is presently imaging an object/user that it has decided is an approved object/user (decision block <NUM>). This decision can be made by, for example, taking actions such as those described above with respect to <FIG>.

If the system <NUM> detects that it is imaging an approved object/user ("Yes" path out of decision block <NUM>) this means that a baseline model for that object/user already exists. The question is whether it should be updated to take into account a changed appearance of the object/user. Accordingly, a picture of the object/user is obtained (e.g., a high resolution image) (step <NUM>) and features from the obtained image are compared with corresponding features of the stored model to produce a set of calculated differences and the model is updated accordingly (step <NUM>). Predefined threshold levels that are used during the authorization process (e.g., refer to <FIG>) at different power levels are adapted based on the updated information (step <NUM>) and these are stored as well.

So long as the object/user remains in front of the camera ("No" path out of decision block <NUM>), the system <NUM> can continue to take more pictures and update the model accordingly, although this is not an essential aspect of this feature.

Once the object/user is no longer in front of the camera module <NUM> ("Yes" path out of decision block <NUM>), the model has been updated and system operation begins at the lowest energy consumption level (step <NUM>). A device associated with the system <NUM> is also put into a locked/deactivated state (step <NUM>).

Returning to decision block <NUM>, if the object/user presently in front of the camera module <NUM> is not recognized as an approved object/user ("No" path out of decision block <NUM>) then the question is whether this object/user is an authorized one for which a baseline model should be created. One way of testing this is detecting whether an authorization token is being presented to the system (decision block <NUM>). If not ("No" path out of decision block <NUM>) then the device associated with the system <NUM> should be maintained in a locked/deactivated state (step <NUM>).

But if a valid authorization token is presented to the system <NUM> ("Yes" path out of decision block <NUM>) then there can be an optional step to further determine whether this is an appropriate time to build a model (decision block <NUM>). If not ("No" path out of decision block <NUM>), then the object/user has been authorized (i.e., by means of an authorization token) and the device associated with the system <NUM> should be activated/opened (step <NUM>).

If a baseline model should be built at this time ("Yes" path out of decision block <NUM>), then some number of pictures (the exact number depends on the particular algorithm being used) are taken of the object/user, important features extracted from those pictures, and the resultant model (e.g., a two- or three-dimensional model) built (step <NUM>). It is advantageous to calculate the model using a higher-level (e.g., highest level) processing capability, since model building is a power- and processing-intensive activity. The model is then stored so that it can be ready for use (step <NUM>) and processing proceeds to step <NUM> in which parameters of the system <NUM> are initialized for use at each of the different power levels. Further processing continues as already described above.

Additional features are now described with reference to <FIG> and <FIG> which, in one respect, are in combination a flowchart of actions taken by the system <NUM> in accordance with a number of embodiments. In other respects, the blocks depicted in <FIG> and <FIG> can also be considered to represent means <NUM> (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions in order to authorize activation of, for example, devices associated with (e.g., by means of proximity) the system <NUM>.

The strategy adopted in the illustrated embodiment involves, loading the one or more latest models of known users (step <NUM>) and for each user, calculating pixel values that will be used in the low power state (step <NUM>). Before the low power mode of the camera module <NUM> is activated, the calculated settings are loaded into the user/object register <NUM> (step <NUM>), and these loaded settings will set the thresholds for all colors and the number of rows that are active for each user. The user/object register <NUM> has room for at least one user.

After the setting(s) for all users are set, the model for the background is put in place. The background model is calculated (step <NUM>) every time just before the low power mode of the sensor is activated. From the background model, pixel values and thresholds for colors and number of sensing structures <NUM> (e.g., rows of pixels) that will be actively scanned are calculated (step <NUM>) and loaded into the background register <NUM> (step <NUM>).

Once all users and background models are set the sensor goes into low power mode (step <NUM>). In low power mode each user setting is scanned one after another. In some but not necessarily all embodiments, the background setting is also scanned as a last step. If the signal from the sensor is over or under a set threshold a trigger is sent to the host.

More particularly, the sensor <NUM>, <NUM> is scanned based on the loaded user setup (step <NUM>). The output from the sensor <NUM> is compared to the loaded threshold value (decision block <NUM>), and if the comparison passes (e.g., is greater than or equal to the threshold) ("Yes" path out of decision block <NUM>), then the host system <NUM> is triggered/activated (step <NUM>) to further process a scanned image to decide, with better accuracy, whether the object/user presented at the sensor <NUM> is recognized. This higher level processing may, in some embodiments, comprise several levels of ever higher level processing, as described above.

If the output from the sensor <NUM>, <NUM> does not pass the threshold test ("No" path out of decision block <NUM>), processing determines whether the last object/user in the list of known objects/users has been checked (decision block <NUM>). If not ("No" path out of decision block <NUM>), settings for the next object/user in the list are loaded into the camera module <NUM> (step <NUM>), processing reverts back to step <NUM> for further testing.

If the last user in the list has been checked without being recognized ("Yes" path out of decision block <NUM>) The sensor is scanned using the loaded settings for the background (step <NUM>). The strategy assumes that, so long as no object or person is presented to the camera module <NUM>, it should be "seeing" only the background, and its output should accordingly be more or less constant over time (within some tolerance level, determined based on application). The strategy allows for some level of background deviation since, for example, people may be walking by the sensor without presenting themselves, lighting conditions can change over time, and the like. Therefore, with the background settings in effect, the sensor output is checked against its corresponding threshold level (decision block <NUM>) to see whether or not it is stable. If the sensor output is within some predetermined margin of the threshold ("Yes" path out of decision block <NUM>) then no significant change has been detected. To avoid the possibility that the system <NUM> might miss the appearance of an object or person being presented to the camera module <NUM>, the effective system settings are changed to be those of the first user in the list (step <NUM>) and processing reverts to step <NUM> and proceeds as discussed above.

It is desired to be able to recognize a known user even if there is some change in their appearance, for example, if the user has changed clothing or looks different in another way (e.g., changed hair color). Therefore, if the output of the camera module <NUM> is not within the margin of the threshold ("No" path out of decision block <NUM>), the host system <NUM> is activated/triggered (step <NUM>) to further process a scanned image to decide, with better accuracy, whether the object/user presented at the sensor <NUM>, <NUM> is recognized. This higher level processing may, in some embodiments, comprise several levels of ever higher level processing, as described above. It is recognized that this may lead to more activations of the higher-power consuming system, but being able to recognize a user even when some changes in appearance have been made justify this action.

Despite the power savings achieved by embodiments such as those just described, there is still room for improvement. One area where problems still exist in current solutions to the challenge of object detection for a positive user ID relates to the amount of power consumed by the sensor and the host subsystem. Although various proposals exist for how to reduce the power consumption, these are still fairly high and not suitable for an always-on battery-driven device. For example, the above-described arrangement that relies on a multiple-staged approach, where the first stage of sensor data analysis is implemented in the sensor with ultra-low power consumption and hence suitable for an always-on system, does not itself address how much power is consumed by the sensor when operating at the lowest level of analysis. Although the arrangement achieves power savings compared to conventional arrangements that require more processing at all times, it is still desirable to further improve performance at a lowest level of processing.

Conventional solutions to the problem of providing optimized sensor structures for performing activity detection typically trigger on any lighting change / activity. In areas where there are frequent activities or changes in the lightning, such sensors lead to unnecessarily high activation levels if there are only rare situations where a relevant person or object becomes visible and should trigger further actions.

There is therefore a need for technology that addresses the above and/or related problems.

It should be emphasized that the terms "comprises" and "comprising", when used in this specification, are taken to specify the presence of stated features, integers, steps or components; but the use of these terms does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

Moreover, reference letters may be provided in some instances (e.g., in the claims and summary) to facilitate identification of various steps and/or elements. However, the use of reference letters is not intended to impute or suggest that the so-referenced steps and/or elements are to be performed or operated in any particular order.

In accordance with one aspect of the present invention, the foregoing and other objects are achieved in technology (e.g., methods, apparatuses, nontransitory computer readable storage media, program means) that provides for low-power always-on image sensing and pattern recognition. In some embodiments consistent with the invention, the technology is an optical sensor module having a controller, a sensor array, present state decision circuitry, and a pattern recognizer. The sensor array comprises a plurality of CMOS sensor pixels, wherein the sensor array is configured to supply one of a plurality of analog sensor signals at each of a sequence of sample times, wherein at each of the sample times, the one of the plurality of analog sensor signals is derived from one or more of the plurality of CMOS sensor pixels. The present state decision circuitry is configured to compare the one of the plurality of the analog sensor signals for each of the sequence of sample times with a respective controller-selected reference voltage and to generate therefrom a respective single state decision signal. The pattern recognizer is configured to assert a pattern recognition signal whenever a plurality of sequentially generated single state decision signals matches a currently active one of a set of one or more reference sequences of single state decision signals. The controller is configured to assert a host system trigger signal when the pattern recognition signal is asserted a predefined number of times in sequence.

In an aspect of some but not necessarily all embodiments, at least one of said one of the plurality of analog sensor signals is an analog sensor signal representing an output from a single one of the plurality of CMOS sensor pixels.

In an aspect of some but not necessarily all embodiments, at least one of said one of the analog sensor signals is a binned analog sensor signal produced by two or more of the plurality of CMOS sensor pixels.

In an aspect of some but not necessarily all embodiments, the set of one or more reference sequences includes at least two reference sequences of single decision signals; a first one of the set of one or more reference sequences differs from a second one of the set of one or more reference sequences; a first assertion of the pattern recognition signal occurs at a first time whenever the plurality of sequentially generated single state decision signals matches the first one of the set of one or more reference sequences; a second assertion of the pattern recognition signal occurs at a sequentially second time whenever the plurality of sequentially generated single state decision signals matches the second one of the set of one or more reference sequences; and the controller is configured to assert the host system trigger signal whenever the pattern recognition signal is asserted at both the first and second times.

In an aspect of some but not necessarily all embodiments, the optical sensor module further comprises a voltage multiplexor that selects the controller-selected reference voltage from a plurality of reference voltages, wherein the controller is configured to select the controller-selected reference voltage based on profile information stored in a register.

In an aspect of some but not necessarily all embodiments, the optical sensor module further comprises one or more Dynamic Vision Sensor, DVS, sensor pixels that collectively assert a DVS trigger signal when luminance detected by at least one of the one or more DVS sensor pixels changes by a threshold amount.

In an aspect of some but not necessarily all embodiments, the controller is configured to:.

wherein at least one of the CMOS sensor pixels is activated when the optical sensor module operates in the low power mode.

In an aspect of some but not necessarily all embodiments, operation in the low power mode comprises activating at least one but fewer than all of the CMOS sensor pixels.

In an aspect of some but not necessarily all embodiments, the controller is configured to increase a number of activated CMOS sensor pixels when the host system trigger signal is asserted.

In an aspect of some but not necessarily all embodiments, the present state decision circuitry further comprises circuitry configured to invert a polarity of the single state decision signal.

In an aspect of some but not necessarily all embodiments, the optical sensor module (<NUM>, <NUM>) is comprised in a mobile communication device.

In an aspect of some but not necessarily all embodiments, the optical sensor module (<NUM>, <NUM>) is comprised in a low-power sensor device.

In aspects of some but not necessarily all embodiments, methods are provided that include actions consistent with at least any of the above described embodiments.

The objects and advantages of the invention will be understood by reading the following detailed description in conjunction with the drawings in which:.

The various features of the invention will now be described with reference to the figures, in which like parts are identified with the same reference characters.

The various aspects of the invention will now be described in greater detail in connection with a number of exemplary embodiments. To facilitate an understanding of the invention, many aspects of the invention are described in terms of sequences of actions to be performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be recognized that in each of the embodiments, the various actions could be performed by specialized circuits (e.g., analog and/or discrete logic gates interconnected to perform a specialized function), by one or more processors programmed with a suitable set of instructions, or by a combination of both. The term "circuitry configured to" perform one or more described actions is used herein to refer to any such embodiment (i.e., one or more specialized circuits alone, one or more programmed processors, or any combination of these). Moreover, the invention can additionally be considered to be embodied entirely within any form of nontransitory computer readable carrier, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein. Thus, the various aspects of the invention may be embodied in many different forms, and all such forms are contemplated to be within the scope of the invention. For each of the various aspects of the invention, any such form of embodiments as described above may be referred to herein as "logic configured to" perform a described action, or alternatively as "logic that" performs a described action.

An aspect of the herein-described technology pertains a highly optimized sensor design and mechanisms that are suitable for the initial stages of object analyses for a system that can be always-on. The system design utilizes a multi-stage approach in which decisions made at the lowest level of analysis or triggering mechanism have the least confidence but are made with minimal power consumption. A positive triggering at that lowest stage leads to a subsequent stage of a (perhaps slightly) more complex analysis - one that is still far below the level of complexity (and power consumption) associated with a host system involving CPUs, GPUs, or NPUs as well as data communication between chips.

In one aspect, the first level of detection is produced by a sensor having a sparse matrix of DVS (Dynamic Vision Sensor) pixels embedded in an otherwise conventional CMOS matrix. Additional circuitry is provided to enable operation in which the CMOS matrix remains inactive while the DVS pixels detect changes. When enough such pixels indicate a change above a certain threshold, a triggering signal is generated indicating that something fundamental has changed in the image. The trigger from the DVS mechanisms causes the mode of operation to change from the lowest level to a second ultra-low power mode in which a subset of the CMOS array is activated. This subset operates in conjunction with analog circuitry as well as simple state machines that, together, enable analysis of the sensed color patterns at a very low level of power consumption. The second ultra-low power mode mechanism is able to test the sensed image against a limited set of user profiles, thereby allowing this stage to trigger (i.e., indicate a match) based on different possible patterns.

Upon the occurrence of a positive triggering from this second stage, the sensor can be put into a more normal mode, or a previously known sub-array mode, for a more in-depth analysis of camera content in, for example, a host processor in the device. This type of mode consumes much more energy but comes with the benefit of higher confidence in the decisions being made. Several different options for implementing the higher-level analysis are generally known, and it is beyond the scope of this invention to describe these in detail.

These and other aspects will now be described in greater detail in connection with the figures. Referring first to <FIG>, it is a block diagram of a system including an always-on sensor module <NUM> coupled to a host system <NUM> such as, but not limited to, the exemplary host system <NUM> described earlier. The sensor module <NUM> comprises a sensor array <NUM> comprising a plurality of sensing structures that are configurable as was discussed earlier with reference to <FIG>. In addition to the configurable sensing structures, in some but not necessarily all embodiments the sensor array <NUM> is further populated with some number of DVS pixels (i.e., in addition to the CMOS pixels), this being for enabling ultra-low always on features as will be further described below.

The sensor module <NUM> further comprises a control unit <NUM> for controlling the various components of the sensor module <NUM> in a way that conforms with the various actions described herein.

In one respect, the sensor module <NUM> is able to make an initial conclusion whether an object or person being presented to the sensor array <NUM> is one of one or more previously authorized objects / persons, and to facilitate this function the sensor module <NUM> includes a register <NUM> for users/objects. As will be discussed further below, recognition of an object/person includes detecting whether a sufficient number of activated sensing structures are being triggered by the object/person being presented, and this means comparing the number of triggered sensing structures to a threshold. It is a purpose of the register <NUM> for users/objects to store settings for all of the objects/users that are available in a given model. There may be one or more than one such object/user.

The initial conclusion made by the sensor module <NUM> should be understood as being a sufficiently close match (within a defined threshold) between the results of the sensed data and the data stored in the register <NUM> per user/object or version of user/object, and the like. The initial conclusion is not made with <NUM>% certainty. But finding an initial match provides sufficient confidence to warrant engaging a higher power stage of analysis that can be used to further enhance the detection and security of the detection with additional sensors or pixels to give higher resolution and the like. By forming an initial conclusion as presented herein, the system is able to expend only minimal energy in order to avoid having non-matching conditions constantly wake up the higher power system. The ratio of negative detections will likely always be much higher than positive results, thus resulting in significant power savings.

In another respect, the sensor module <NUM> is able to detect when no object/user is being presented to the sensor array <NUM> and to remain in a very low (or in some embodiments, an ultra-low) power state under such circumstances, and then to be able to revert to a more active state when something (object/person) is then presented to the sensor array <NUM>. This function involves tracking whether the sensor array <NUM> presently detects an image corresponding only to an image of an environmental background (i.e., an image without an object or person being presented to the sensor array <NUM>). So long as only background is detected, the sensor module <NUM> can remain in the very (or ultra) low power state. If a sufficient enough change to the scanned image is detected, the more active state is entered. To facilitate this purpose, the sensor module <NUM> further includes a register <NUM> for a background image. It is a purpose of the background image register <NUM> to store settings for a background that is available in a given model.

In addition to the above-mentioned elements, the exemplary sensor module <NUM> further includes:.

In an aspect of its operation, the sensor module <NUM> supplies an image to the host system <NUM> for it to undergo a more computationally complex analysis once an object has been detected using the lower-power detection strategies of the sensor module <NUM> itself.

In one aspect of some embodiments, always-on operation is facilitated by utilizing a sensor array <NUM> populated with both CMOS sensor (pixel) elements and DVS sensor (pixel) elements. The CMOS sensor elements are further configured to be dynamically configurable such that only selected ones are activated, based on what level of sensor resolution is desired. Power savings are achieved because only the activated sensor elements expend energy. These aspects are discussed in greater detail in the following.

<FIG> is a diagram of a non-limiting example of a sensor array <NUM>. The sensor array <NUM> comprises a plurality of CMOS sensor pixels <NUM> (illustrated by diagonal shading), and also one or more DVS sensor pixels <NUM> (each one of the sensors denoted by "D"). During ultra-low power operation, none of the CMOS sensor pixels <NUM> are activated. Instead, the DVS sensor pixels <NUM> are relied on exclusively for the purpose of detecting changes in luminescence. Such changes are taken as an indication that some person/object has come into view of the sensor array <NUM>, requiring that a next higher level of processing be activated to determine whether the person/object is recognized. Since the DVS sensor pixels <NUM> serve only to indicate the presence or absence of an object in front of the sensor array <NUM>, there need not be many of them. In the example, there are only a few DVS sensor pixels <NUM> spread out over the sensor array <NUM> to cover the field of view of the camera, thus being able to detect changes in luminescence over a larger area. Since the DVS sensor pixels <NUM> occupy positions that would otherwise hold CMOS sensor pixels, the image data from the "missing" CMOS sensor pixels can be calculated / estimated from data provided by neighboring conventional pixels. This may possibly produce some artifacts in those areas. The calculations involved in providing the missing data are the same as calculations known by those of ordinary skill in the art for compensating for defective pixels on a sensor. There are a number of well-known algorithms for accomplishing this, such as nearest neighbor calculations, bi-linear interpolation, linear interpolation, and the like. A complete description of such known algorithms is beyond the scope of this disclosure.

The CMOS sensor pixels <NUM>, on the other hand, are used to provide image data that is analyzed to detect the presence of a recognized image. As described earlier with reference to <FIG>, the analysis may be performed at any of a number of different levels of image resolution, and to save energy, only those CMOS sensor pixels <NUM> necessary to satisfy the desired level of resolution are activated. A design of a suitably configurable CMOS image sensor is described in US Patent Application No. <CIT>. The use of Backside Illuminated CMOS sensor technology or similar stacking in the manufacturing process allows for a more complex sensor design without affecting the aperture ratio of the sensor, as described in the just-mentioned US Patent Application. It will be understood that the use of such processing technology is not an essential aspect of the herein-described embodiments.

In many embodiments, the number of CMOS sensor pixels <NUM> greatly outnumbers the number of DVS sensor pixels (which, as noted above, can be treated as "defective" pixels when the CMOS sensor pixels <NUM> are operational). Therefore, the CMOS sensor pixels <NUM> comprise an array of approximately M pixels <NUM>, e.g., an approximately M<NUM> × M<NUM> matrix, where M = M<NUM>. M<NUM>, where the total number of pixels <NUM> (approximately M) defines the maximum resolution of the sensor array <NUM>, and thus of the sensor module <NUM>. It will be appreciated that each pixel <NUM> is an electronic circuit, and thus pixels <NUM> may also be referred to herein as pixel circuits <NUM>. <FIG> shows an exemplary pixel <NUM>. As understood by those skilled in the art, a pixel <NUM> must be pre-charged before it can detect light. Once pre-charged, the pixel <NUM> detects any input light. The detected light is output when the pixel <NUM> is driven. Thus, each pixel <NUM> of the sensor array <NUM> is controlled by a RESET signal, which selectively connects the pixel <NUM> to a voltage level, VRST, to pre-charge the pixel to enable the pixel to capture light; and a separate DRIVE signal, which selectively connects the pixel <NUM> to a voltage level, VDD, to drive the pre-charged pixel <NUM> to enable the pixel <NUM> to output the detected light.

<FIG> shows a more detailed example of an exemplary pixel <NUM> comprising multiple transistors and a diode. It will be appreciated that implementations other than the one shown in <FIG> may be used for the pixel <NUM> of <FIG>. Exemplary pixels include, but are not limited to, Front Side Illuminated (FSI) pixels and Backs Side Illuminated (BSI) pixels. It will be appreciated that the structure of BSI pixels enables wiring to be added without impacting the aperture ratio of the pixel, and thus enables implementation of the solution presented herein without reducing the aperture ratio of the pixel.

The dynamically configurable array of CMOS sensor pixels <NUM> provides a mechanism for controlling the analog domain power consumption of the system by controlling how many and which pixels <NUM> are pre-charged. As such, the solution reduces the power consumption of the sensor array <NUM> by an amount proportional to the uncharged pixels <NUM>. For example, if only half of the pixels <NUM> are pre-charged, the analog power consumption is reduced by approximately <NUM>%. It will be appreciated that only those pixels that are pre-charged are driven. As such, in some embodiments, the pre-charging and driving aspects of the pixels <NUM> may be coordinated, e.g., by the control unit <NUM> and/or by any charge control/drive circuits.

<FIG> shows a block diagram of an exemplary sensor array <NUM>. Image sensor array <NUM> comprises a sensor array <NUM> comprising a plurality of CMOS sensor pixels <NUM> (the DVS sensor pixels <NUM> (which are also present as discussed above) are ignored for purposes of this discussion), a charge control circuit <NUM>, and a drive control circuit <NUM>. The sensor array <NUM> comprises two or more sensor segments <NUM>, where each of the two or more sensor segments <NUM> comprises a different set of pixel(s) <NUM>. While sensor array <NUM> is shown as a two-dimensional matrix of pixels <NUM>, it will be appreciated that the sensor array <NUM> may alternatively comprise a vector of pixels <NUM>, where the vector of pixels <NUM> comprises two or more sensor segments <NUM>. The charge control circuit <NUM> is configured to pre-charge pixels <NUM> via RESET control lines, while the drive control circuit <NUM> is configured to drive pre-charged pixels <NUM> via DRIVE control lines. While <FIG> shows separate RESET and DRIVE control lines for each of three sensor segments 1014a-c, it will be appreciated that each segment-specific RESET and/or DRIVE control line may represent individual RESET and DRIVE control lines for each pixel <NUM> in the corresponding segment <NUM>, or may represent a common RESET and/or DRIVE control line specific to the corresponding segment <NUM>. Further, it will be appreciated that the solution presented herein does not require three sensor segments 1014a-c as shown in <FIG>; sensor array <NUM> may comprise fewer or additional sensor segments <NUM> than those shown.

In another aspect of exemplary embodiments, the sensor is able to cope with different environmental illumination levels and shifts by adjusting the white balance and exposure level based on a rolling average of the pixels <NUM> that are scanned during the low power mode. The white balance and exposure levels are fed back to the control unit <NUM> which responds by adjusting the trigger levels by amounts that depend on the measured environment illumination.

In another aspect of some but not necessarily all exemplary embodiments, the sensor readout can be configured to operate in any of a number of different modes, each associated with a correspondingly different pattern. This enables easy and power efficient readout from the pixels <NUM>. In the different modes, the output from pixels <NUM> in the selected area are averaged in the analog domain before being sent to an A/D converter, a comparator and a pattern recognizer (all discussed further below).

<FIG> illustrates a number of exemplary pixel readout modes. It will be appreciated that other modes, not illustrated, are also possible. The illustrated modes are:.

All modes are designed in the sensor and cannot be changed after the design is completed (i.e., at the point that the design will not change anymore). For example, a sensor that is intended to support binning of triangles and rectangles needs to be designed that way.

<FIG> depicts an example of how a binning can be performed and support different shapes. A 4x4 pixel array is shown to illustrate the point, but embodiments consistent with the invention are not limited to this particular arrangement. Three different binning options are supported in this example: a top triangle portion "a" <NUM>; a bottom triangle portion "b" <NUM>; and a square <NUM> formed by combining the top and bottom triangle portions <NUM>, <NUM>. To support more complex shapes more readouts have to be done and matched in the pattern recognizer.

Optical sensors, such as the CMOS sensor pixels <NUM>, produce an analog signal whose magnitude is in proportion to the detected luminosity of the sensed light. One or more analog-to-digital (A/D) converters are used to convert the analog signal from the CMOS sensor pixels <NUM> into digital signals that can be supplied to digital processing circuitry for the purpose of image analysis. As mentioned earlier, higher levels of analysis are performed by a host system <NUM>.

But in another aspect of embodiments consistent with the invention, the sensor module <NUM> also includes analog circuitry capable of making a coarse decision about whether an image is recognized or not while consuming very low amounts of energy. An exemplary embodiment of such circuity is illustrated in <FIG>.

The circuitry <NUM> can be conceptually divided into two parts: an A/D converter that is used in an image sensor system such as is found in the host system <NUM>; and additional, simple circuitry for performing the low power object identification step. Some but not necessarily all embodiments consistent with the invention utilize a single A/D converter for both purposes; some but not necessarily all other embodiments use separate circuitry for these two different functions. All such embodiments are contemplated to be within the scope of the invention.

The example shown in <FIG> employs a full multi-bit A/D converter <NUM> whose multi-bit digital output signal enables image analysis requiring higher levels of image resolution (e.g., some number, X, bits). The digital output of the X-bit A/D converter <NUM> can be provided to the host system <NUM> as shown in <FIG>. The X-bit A/D converter <NUM> receives an analog signal <NUM> from the sensor array <NUM>, and also receives reference voltages <NUM> against which the analog signal <NUM> is compared, as is known in the art of analog-to-digital signal conversion.

The additional, simple circuitry for performing the low power object identification step in this exemplary embodiment includes a voltage multiplexor <NUM>, a present state decision circuit <NUM>, and pattern recognition circuitry (e.g., a state machine) <NUM>.

The present state decision circuit <NUM> can, in its simplest form, be a <NUM>-bit comparator <NUM> that compares the received analog signal <NUM> from the sensor array <NUM> with a reference voltage set to an object-dependent decision threshold <NUM> to produce a <NUM>-bit comparison result signal <NUM> having either a binary <NUM> or <NUM> value, depending on the results of the comparison. Production of a positive comparison result by the comparator <NUM> is taken as an indication that an object in front of the CMOS sensor pixel <NUM> array may have been recognized.

In an aspect of some embodiments, the decision threshold <NUM> is selected from one of the reference voltages <NUM> that are already available for the high power A/D converter <NUM>. This is not an essential aspect of the technology; to the contrary, a decision threshold voltage can, in other embodiments, be generated from any other source. In the exemplary embodiment of <FIG>, selection is performed by the voltage (or power) multiplexor <NUM> having a selection control signal supplied by the control unit <NUM>. The control unit <NUM> bases selection on whichever object profile is active at any given moment, and in particular bases selection on profile-related values stored in the registers for users/objects <NUM> and registers for background <NUM> as shown in <FIG>. In some embodiments, the voltage multiplexor <NUM> is capable of supporting all voltages from the high power A/D converter; in other embodiments, only a subset of the available voltages.

The exemplary comparator <NUM> asserts the comparison result signal <NUM> (e.g., from low to high) when a positive comparison is made between the received analog signal <NUM> and the object-dependent decision threshold <NUM> but in some instances, it may be desirable in the low-power mode to detect when an analog signal <NUM> is not greater than or equal to a reference voltage (i.e., to detect transitions from high to low instead of from low to high), and for this purpose in some but not necessarily all embodiments the circuitry <NUM> is further equipped with logic circuitry, such as an Exclusive OR (XOR) gate <NUM>, that performs a logical Exclusive OR between the <NUM>-bit comparison result signal <NUM> from the comparator <NUM> and a decision polarity signal <NUM> generated by the control unit <NUM>. When the decision polarity signal <NUM> is low, the <NUM>-bit digital comparison result signal <NUM> passes through unchanged; when the decision polarity signal <NUM> is high, the <NUM>-bit digital comparison result signal <NUM> passes through in an inverted form. The output of the XOR gate <NUM> is herein denoted single state decision signal <NUM>.

To save as much energy as possible, the above-described elements need not be activated when operation is in the ultra-low power mode. The control unit <NUM> therefore receives a DVS trigger signal <NUM> that is asserted when the DVS sensor pixels <NUM> detect a change in luminosity. Assertion of the trigger signal <NUM> causes the control unit <NUM> to transition the unit from ultra-low power mode into the (higher level) low power mode.

In yet another aspect of embodiments consistent with the invention, decision accuracy is improved by detecting different sequences of patterns of pixels rather than occurrence of only a single analog signal <NUM>. In order to be able to detect the different sequences of patterns of pixels, the sensor module <NUM> includes a state machine or pattern recognizer <NUM>. The pattern recognizer receives the single state decision signal <NUM>, which represents a comparison decision based on sensor output from any of the above-described sensor operational modes. The pattern recognizer <NUM> expands the low-power image analysis capability by freeing it from having to make decisions based on only a single, instantaneous sensor output value. Instead, since it is often desirable to make a "recognized"/"not recognized" decision based on whether a known sequence of sensor signals have been produced, the pattern recognizer <NUM> is configured to detect whether a sequentially occurring pattern of particular decisions has been made. It is noted that assertion of the single state decision signal <NUM> can denote detection of different patterns at different times, since the control unit <NUM> can dynamically control what pattern the sensor unit <NUM> will be scanning for at any moment, as well as what luminance levels are being looked for (as represented by the selected voltage being fed to the comparator <NUM>), and the decision polarity <NUM>. Therefore, an unbroken string of assertions does not mean that the same pattern has been detected over a stretch of time; one assertion can mean for example that a triangle of a particular luminance has been detected, and a next assertion can mean for example that a blue circle has been detected.

As an example, consider a case in which it is desired to trigger a higher-level analysis (e.g., by the host system <NUM>) when a certain number of consecutive image patterns with positive comparison decisions occurs. These triggers could all be based on the same decision threshold <NUM>, or could alternatively be based on different decision threshold values <NUM> depending on what voltage selection signal the control unit <NUM> supplied to the voltage multiplexor <NUM> at the time of decision making. <FIG> illustrates a state machine <NUM> for this example. As can be seen in the figure, transitions are made from a first state to a second, then to a third, and so on for some number, n, states, so long as each next transition indicates that a sensor output satisfied a particular threshold value. But if, in any one of the states, a next decision indicates that the sensor output transition did not satisfy a particular threshold value, then the pattern has been broken and the state machine <NUM> reverts back to its initial, first, state. It can be seen that, in this way, the sequence of single state decision signals <NUM> is compared against a reference sequence of single state decision signals <NUM>.

A pattern recognition decision signal <NUM> is, in some embodiments, routed back to the controller <NUM>. In another aspect of embodiments consistent with the invention, the controller <NUM> is configured to assert a host system trigger signal <NUM> when the pattern recognition signal <NUM> is asserted. Assertion of the host system trigger signal <NUM> can, for example, cause an affiliated host system <NUM> to become activated and accordingly perform a higher-level of object recognition analysis.

In another aspect of some but not necessarily all embodiments, the controller <NUM> is configured to increase a number of activated CMOS sensor pixels <NUM> when the pattern recognition signal <NUM> is asserted. This serves to provide a higher resolution image capture in support of the host system's higher-level of object recognition analysis.

The pattern recognizer / state machine <NUM> can take on any of a number of different forms depending on what the designer would like to use as a trigger scheme. For example, several different state machines could be provided in a design, with the active one being chosen by the control unit <NUM>; the control unit <NUM> gets state machine selection information from the settings loaded into the register for users/objects <NUM> and register for background <NUM>.

The system is able to detect a wider range of different complex patterns by having several different pattern recognizers <NUM>. It can also be that several complex patterns need to be recognized before the host is triggered. This is controlled by the control unit <NUM> that get this information from the register settings for each profile that is present in the system.

Further aspects of embodiments consistent with the invention will now be described with reference to <FIG> and <FIG>, which in one respect depict a flowchart of actions performed by the system sensor module <NUM> in accordance with a number of embodiments. In other respects, the blocks depicted in <FIG> and <FIG> can also be considered to represent means <NUM> (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

As shown beginning in <FIG>, the process includes loading the different object/user/background profiles into the respective register settings of the sensor module <NUM> (step <NUM>). These are the user register settings, object register settings, and background register settings produced at, for example, steps <NUM> and <NUM> (see <FIG>). The sensor module <NUM> is then placed into an ultra-low power mode, in which only the DVS pixels <NUM> are active (step <NUM>). As explained earlier, the DVS pixels <NUM> operate asynchronously, and include circuitry that issues a trigger when an illumination difference is detected according to the DVS pixel setting (step <NUM>).

The system remains in ultra-low power mode so long as the DVS pixels <NUM> do not issue a trigger signal <NUM> ("No" path out of decision block <NUM>). When the DVS pixels <NUM> do trigger ("Yes" path out of decision block <NUM>), the system transitions to the low power mode in which the CMOS sensor pixels <NUM> are started (step <NUM>).

In the low power mode, the system iterates through all of the object/user profiles to see if CMOS sensor outputs match any of them (step <NUM>). More particularly, register settings are loaded for initially a first and then (if warranted) a next profile (step <NUM>), and the CMOS sensor pixels <NUM> are selectively activated according to the loaded register settings (step <NUM>).

If the threshold specified by the loaded profile settings are not satisfied ("No" path out of decision block <NUM>), a further test is performed to determine whether a timeout condition has been satisfied (decision block <NUM>). If it has ("Yes" path out of decision block <NUM>), processing reverts back to step <NUM>, where the system is switched back into ultra-low power mode. In some but not necessarily all alternative embodiments, instead of using the described timeout mechanism, a strategy is adopted in which all profiles are checked some predetermined number (one or more) of times and if so, processing reverts back to step <NUM>.

But if a timeout has not occurred ("No" path out of decision block <NUM>), processing reverts back to step <NUM> where settings from a next profile are loaded into the registers and iteration continues for another round.

If the threshold settings are satisfied for one of the profiles ("Yes" path out of decision block <NUM>), then processing advances to state machine processing (step <NUM>) in which a pattern of successively generated output signals from the CMOS sensor pixels <NUM> are tested against a predefined pattern for the current profile. Each generated CMOS sensor output is tested against a corresponding next predefined state condition (decision block <NUM>) (see, e.g., <FIG>). If at any point the threshold condition is not satisfied ("No" path out of decision block <NUM>), then a match has not been found and processing reverts to step <NUM> where iteration through the profiles is taken up again.

But if the state machine processing advances through all of the predefined (profile-dependent) states (e.g., states <NUM>. n as shown in <FIG>) with the threshold conditions being satisfied each time ("Yes" path out of decision block <NUM>), then a trigger is generated and sent to the host system <NUM> (step <NUM>) so that a higher-resolution analysis can be performed by that unit as described earlier.

Aspects of an exemplary controller <NUM> that may be included in the system sensor module <NUM> to cause any and/or all of the above-described actions to be performed as discussed in the various embodiments are shown in <FIG>, which illustrates an exemplary controller <NUM> of a sensor module <NUM> in accordance with some but not necessarily all exemplary embodiments consistent with the invention. In particular, the controller <NUM> includes circuitry configured to carry out any one or any combination of the various functions described above. Such circuitry could, for example, be entirely hard-wired circuitry (e.g., one or more Application Specific Integrated Circuits - "ASICs"). Depicted in the exemplary embodiment of <FIG>, however, is programmable circuitry, comprising a processor <NUM> coupled to one or more memory devices <NUM> (e.g., Random Access Memory, Magnetic Disc Drives, Optical Disk Drives, Read Only Memory, etc.) and to an interface <NUM> that enables bidirectional communication with other elements of the sensor module <NUM>. The memory device(s) <NUM> store program means <NUM> (e.g., a set of processor instructions) configured to cause the processor <NUM> to control other system elements so as to carry out any of the aspects described above. The memory device(s) <NUM> may also store data (not shown) representing various constant and variable parameters as may be needed by the processor <NUM> and/or as may be generated when carrying out its functions such as those specified by the program means <NUM>.

Various embodiments consistent with the invention provide an advantage over conventional technology in that a first level of object detection can be done at extremely low power consumption enabling the use in always-on battery-operated devices.

Aspects relating to the use of DVS sensor pixels <NUM> especially enable always-on operation, since the next higher level of processing (at a higher level of power consumption) is invoked only when actually needed.

In another advantage, the use of analog circuitry and selective activation of CMOS sensor pixels <NUM> to perform a low-level of image analysis in turn ensures that higher levels of processing power are not invoked unless they are actually needed.

Embodiments consistent with the invention may be deployed in a number of different kinds of devices. For example, low power, always-on sensor technology can advantageously be used in any number of battery-powered devices, such as mobile communication devices (e.g., User Equipment - "UE"). It is also well-suited for use in low-power devices such as Internet-of-Things (IoT) devices. Such devices may themselves be sensor devices for sensing and reporting other aspects of an environment (e.g., temperature, movement, etc.).

Claim 1:
An optical sensor module (<NUM>) comprising:
a controller (<NUM>);
a sensor array (<NUM>, <NUM>) comprising a plurality of CMOS sensor pixels (<NUM>), wherein the sensor array (<NUM>, <NUM>) is configured to supply one of a plurality of analog sensor signals (<NUM>) at each of a sequence of sample times, wherein at each of the sample times, the one of the plurality of analog sensor signals (<NUM>) is derived from one or more of the plurality of CMOS sensor pixels (<NUM>);
present state decision circuitry (<NUM>) configured to compare the one of the plurality of analog sensor signals (<NUM>) for each of the sequence of sample times with a respective controller-selected reference voltage (<NUM>) and to generate therefrom a respective single state decision signal (<NUM>); and
a pattern recognizer (<NUM>) configured to assert a pattern recognition signal (<NUM>) whenever a plurality of sequentially generated single state decision signals (<NUM>) matches a currently active one of a set of one or more reference sequences (<NUM>) of single state decision signals (<NUM>),
wherein the controller (<NUM>) is configured to assert a host system trigger signal (<NUM>) when the pattern recognition signal (<NUM>) is asserted a predefined number of times in sequence.