Patent Description:
Mobile computing devices, e.g., smart phones, personal digital assistants, electronic tablets, laptops, and the like, typically use power provided by one or more rechargeable batteries. A rechargeable battery provides only a finite amount of power to a device before the battery must be recharged, e.g., by applying an electric current to the battery. Recharging the battery of a mobile computing device generally requires connecting the mobile computing device to an electric grid, which reduces or eliminates its mobility. Consequently, reducing power consumption of mobile computing devices, thereby extending their battery life, is an important objective.

Mobile computing devices can include one or more or peripheral sensors. For example, some mobile computing devices include microphones, cameras, accelerometers, and the like. Mobile computing devices can also include one or more processing components to process data collected by the one or more peripheral sensors. For example, some mobile computing devices include central processing units (CPUs), digital signal processors (DSPs), or other processing components.

Mobile computing devices can perform actions based on data collected by their one or more peripheral sensors and processed by their one or more processing components. For example, some mobile computing devices can perform actions in response to voice commands detected by a microphone and processed by a CPU. However, maintaining processing components in an active state so that they can process sensor data consumes significant amounts of power.

<CIT> describes sensing of scene-based occurrences.

<CIT> describes a terminal control method and a terminal.

<CIT> describes an electronic device which may have a camera module.

<CIT> describes a method of controlling a sleep mode in a portable terminal having a main controller and a sub-controller operating at low power.

This specification describes technologies for implementing low-power vision sensing on computing devices. These techniques allow for a variety of complex applications that rely on continual monitoring of vision sensor data to be run in low-power states.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. An ambient computing system can perform a variety of complex tasks based on vision sensing while consuming less power. This results in power savings and an increase in battery life. These complex vision sensing tasks can be performed without waking up a main CPU cluster of a device. In addition, the architecture of the system can protect user privacy by preventing threads from the main CPU cluster from ever accessing the vision sensor data. This allows the ambient computing system to be more responsive to changes in its environment while also reducing power consumption. Accordingly, an ambient computing system can provide greater functionality with limited impact on the battery life of a computing device, and increase the total battery life of a computing device with complex ambient state.

According to a first unclaimed aspect of the present disclosure, there is provided a computing device comprising: a vision sensor configured to generate vision sensor data; and an ambient computing system. The ambient computing system is configured to repeatedly process the vision sensor data generated by the vision sensor according to a low-power detection process, and, when a detection is indicated by the low-power detection process, to wake one or more other components of the computing device to perform a high-power detection process using the vision sensor data.

According to a second aspect of the present disclosure there is provided a computing device according to claim <NUM>.

The following features may be provided in combination with either of the first or second aspects.

The device may be configured to perform the low-power detection process with a first processing component and to perform the high-power detection process with a second processing element, wherein the first processing component consumes less power than the second processing component.

The second processing component may be disabled until a detection is indicated by the low-power detection process.

The computing device may further comprise a main machine learning engine having a plurality of compute tiles. The device may be configured to perform the high-power detection process using the main machine learning engine. The main machine learning engine may comprise the second processing element.

The device may be configured to perform the low-power detection process using fewer than all of the plurality of compute tiles of the main machine learning engine.

The ambient machine learning engine may comprise the first processing element.

The computing device may further comprise a main CPU cluster. The device may be configured to perform the low-power detection process and the high-power detection process without ever storing the vision sensor data in memory that is accessible by threads executing in the main CPU cluster.

Referring to the first aspect, the computing device may further comprise a camera and a main image signal processor (ISP) that is configured to process data generated by the camera. The device may be configured to use a private channel between a frame buffer storing vision sensor data and the main ISP to generate preprocessed vision sensor data for use during the high-power detection process. Referring to both aspects, the private channel may not be accessible by threads executing in the main CPU cluster.

The vision sensor may be configured to generate a first frame rate for the low-power detection process and a second frame that is higher than the first frame rate for the high-power detection process.

The vision sensor may be configured to generate vision sensor data at a first resolution for the low-power detection process and a second resolution that is higher than the first resolution for the low-power detection process.

When a detection is indicated by the low-power detection process, the ambient computing system may be further configured to wake one or more other components of the computing device to perform a high-power detection process using the vision sensor data. Said means for performing a high-power detection process when a detection is indicated by the low-power detection process may comprise said one or more other components of the computing device.

According to a third aspect of the present disclosure, there is provided a method comprising performing the operations performed by the computing device of either one of first or second aspects. The method may comprise generating, by a vision sensor of the computing device, vision sensor data; repeatedly processing the vision sensor data according to a low-power detection process; and performing a high-power detection process when a detection is indicated by the low-power detection process.

The method may comprise, repeatedly processing by an ambient computing system of the computing device the vision sensor data generated by the vision sensor according to the low-power detection process, and, when a detection is indicated by the low-power detection process, waking wake one or more other components of the computing device to perform the high-power detection process using the vision sensor data.

According to a further aspect of the present disclosure, there is provided one or more computer storage media encoded with computer program instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the operations of the first or second aspects, or the method of the third aspect.

It will be appreciated that optional features described above in the context of the first and second aspects may be used in combination with the third and further aspects also described above.

Like reference numbers and designations in the various drawings indicate like components.

<FIG> is a diagram of an example computing device <NUM> that has low-power vision sensing capabilities. The device <NUM> is an example of a device that can use a low-power vision sensor to react to objects in its environment. The device <NUM> can be a system implemented in an any appropriate computing device, e.g., a smart phone, a smart watch, a fitness tracker, a personal digital assistant, an electronic tablet, or a laptop, to name just a few examples.

The system of computing device <NUM> can be used so that the computing device <NUM> can remain in a low-power state while continually processing inputs from a vision sensor <NUM>. In this context, being in a low-power state means that one or more of the most powerful computing components are not used, which can mean that these devices are powered down partially or fully turned off. In <FIG>, for example, the most powerful components include a main CPU cluster <NUM>, a main machine learning (ML) engine <NUM>, and a main image signal processor (ISP) (<NUM>). For brevity, these components will be referred to as the high-power components of the device because typically these devices consume more power when operational than the ambient computing system <NUM>.

Thus, the device <NUM> uses the ambient computing system <NUM> to repeatedly process vision sensor data without using any of the high-power components. However, the high-power components can be woken if the ambient computing system <NUM> determines that a low-power detection has been made. In this specification, the terms wake and activate will be used to mean supplying an increased amount of power to a particular processing component or other electronics circuitry. The system may or may not have been supplying power to a processing component or other circuitry that is being awoken or activated. In other words, a component being awoken or activated may or may not have been completely powered down previously. Waking or activating a processing component can result in the processing component performing a boot process and causing instructions and data for the processing component to be loaded into random-access memory. Alternatively or in addition, waking or activating a processing component can include resuming from a previously suspended state.

In this specification, vision sensor data refers to a two-dimensional array of data elements generated by one or more light sensors of the vision sensor. Each data element can include one or more light intensity values, e.g., for red, blue, and green light. The vision sensor data can thus be stored as raw intensity values for each of the colors of light. Alternatively or in addition, each data element can store a single intensity value with no color information.

In this specification, a low-power detection means that data captured by the vision sensor <NUM> has been processed according to a low-power detection model and that the low-power detection model has indicated that further processing should be undertaken. The low-power detection can be performed by one or more of the low-power components of the processing subsystem <NUM>, e.g., the low-power CPU <NUM>, the low-power DSP, or an ambient ML engine <NUM>.

If a low-power detection has been made, the device <NUM> can wake one or more of the main CPU cluster <NUM>, the main ML engine <NUM>, or the main ISP in order to further process the vision sensor data <NUM>. These components can process the vision sensor data using more sophisticated and more refined models, which will be referred to as high-power detection models because they also utilize more powerful processing devices and thus consume more power.

One illustrative example of this technology is to use facial recognition in order to unlock a mobile phone. Full facial recognition models that have the sophisticated feature spaces required to reliably distinguish the faces of different users tend to consume a lot of power. It is thus impractical and inefficient to repeatedly run, on a device having finite battery power, a full facial recognition model on all data captured by the vision sensor.

Instead, the ambient computing system <NUM> repeatedly processes the data captured by the vision sensor <NUM> using a low-power detection model. The low-power detection model can be a model with fewer layers, fewer features, or operate on smaller input frames. Thus, the low-power detection model may only be able to distinguish human heads or human faces from other objects while not having the sophistication required to distinguish individual users from each other.

Thus, the device can use a low-power detection model to indicate with high reliability whether or not the vision sensor <NUM> is capturing data corresponding to a human head or face, but not any particular human head or face. If a low-power detection indicates that it is likely that the vision sensor <NUM> is capturing data of a human face, the ambient computing system <NUM> can wake one or more of the high-power components to execute the full facial recognition model in order to determine if the vision sensor <NUM> is capturing data of a particular human face, e.g., the human face of owner of the mobile phone. And, in the case of the phone unlock scenario, the system can use the output of the full facial recognition model in order to determine whether or not to unlock the phone.

In the low-power detection state, the ambient computing system <NUM> can instruct the vision sensor <NUM> to use a lower frame rate, a lower resolution, or both, than when in a high-power detection state. For example, in the low-power detection state, the vision sensor <NUM> can capture data at <NUM> pixel resolution at only <NUM> frames per second. Then, in the high-power detection state, the vision sensor <NUM> can switch to capturing data at <NUM> pixel resolution at <NUM> frames per second.

In addition to preserving power, this arrangement also enhances user privacy while still providing for the capabilities to use advanced facial recognition models. In other words, in the low-power detection state, the vision sensor <NUM> is not capturing data that is of high enough quality that it could be used to identify any particular person. Thus, even if the mobile phone was compromised with malware that could read the data being captured by the vision sensor <NUM>, the captured data would be useless for the purposes of identifying specific people in the data captured by the vision sensor <NUM> because the resolution is not high enough.

The device <NUM> also includes an integrated camera <NUM>, and includes a main ISP <NUM> for processing images captured by the camera <NUM>. Notably, the camera <NUM> can include a far higher resolution capabilities than the vision sensor <NUM>. For example, the camera can capture <NUM>, <NUM>, or <NUM> megapixel images, while the vision sensor <NUM> might capture only a maximum of <NUM> megapixel images even when in the high-power detection mode.

The architecture of the ambient computing system <NUM> can also help to enhance user privacy. In particular, the ambient computing system <NUM> can be designed to prevent vision sensor data from leaking out into other components of the chip where they could be accessed by compromised software processes.

Thus, the ambient computing system <NUM> allocates a dedicated frame buffer <NUM> in its SRAM <NUM> of the ambient computing system <NUM>. In some implementations, the ambient computing system <NUM> allocates the frame buffer in a portion of the SRAM <NUM> that is not accessible by threads running the main CPU cluster <NUM> or the main ML engine <NUM>. Thus, even if a software process running on the main CPU cluster <NUM> is compromised, the compromised process has no mechanism for accessing the frame buffer <NUM> that stores data captured by the vision sensor <NUM>. This is because unlike a general purpose DRAM used by the main CPU cluster <NUM>, the SRAM <NUM> is accessible only by the components of the ambient computing system <NUM>, which might never run arbitrary user code.

In some situations, in order to provide for higher quality detections, the ambient computing system can borrow the processing pipeline of the main ISP <NUM>. The main ISP <NUM> can be a hardware or software-implemented system that includes a pipeline of multiple functional components that process raw image data in sequence. For example, the main ISP <NUM> can have modules that apply linearization, black level correction, lens shading correction, white balance gain, and highlight recovery to generate a final output image.

The image processing pipeline of the main ISP <NUM> can be used to process raw image data captured by the full-resolution camera <NUM>. In addition, in some implementations, the same processing pipeline of the main ISP can be used to enhance the data captured by the vision sensor <NUM>.

In order to prevent data leakage of the data captured by the vision sensor <NUM>, the computing device has a private dedicated channel <NUM> from the frame buffer <NUM> to the main ISP <NUM>. Being a private dedicated channel means that the architecture of the device <NUM> does not provide for any other devices reading from the channel <NUM>. In other words, threads running on the main CPU cluster <NUM> have no mechanism for accessing data that is passed along the private channel <NUM>.

Similarly, the device <NUM> can also implement a private channel <NUM> between the main ISP and the main ML engine <NUM>. This allows the main ML engine <NUM> to execute very sophisticated models on high-quality data run through the main ISP <NUM> on channels that prevent such data from leaking out to other components of the device <NUM> where they could be read by compromised threads running the main CPU cluster <NUM>.

Another use case for the different power detection levels is translating text from one language to another. The ambient computing system <NUM> can continually run a low-power optical character recognition (OCR) model on data received by the vision sensor. The low-power OCR model indicating a low-power detection means that the vision sensor <NUM> is picking up an image having text, e.g., on a sign or on a piece of paper. The ambient computing system <NUM> can thus wake the main ML engine <NUM> to run a full, high-power translation model on the text recognized from the vision sensor. The output can then be provided to the user, e.g., on an ambient display or on an ambient text-to-speech model. In this way, the device can both continually search for text to translate, can detect text, and can automatically perform a full translation of any captured text and output the results all while never waking the main CPU cluster <NUM>.

The one or more components of the computing device <NUM> can be implemented on a system on a chip (SoC) within the computing device. An SoC can be an integrated circuit that includes each component of the system on a single silicon substrate or on multiple interconnected dies, e.g., using silicon interposers, stacked dies, or interconnect bridges. Other components of the computing device, including a main CPU cluster <NUM>, can be implemented on the same or on a separate die. The computing device <NUM> may include components, including the sensors <NUM>, one or more displays, a battery, and other components, that are separate from and independent of the SoC, and may for example be mounted on a common housing.

Briefly, and as described in further detail below, the device <NUM> includes a number of peripheral sensors <NUM> configured to generate sensor signals based on input from the environment of the computing device. The device <NUM> includes a control subsystem <NUM> for controlling the supply of power and sensor signals to components in the system. And the device <NUM> includes a processing subsystem <NUM> for processing sensor signals and generating outputs.

The device <NUM> also includes a main CPU cluster <NUM>. The main CPU cluster <NUM> is a component of the computing device that includes one or more general-purpose processors that are separate from the devices in the processing subsystem <NUM>. The processors of the main CPU cluster <NUM> generally have more computing power than any of the devices in the processing subsystem <NUM>, and therefore, the processors of the main CPU cluster <NUM> may also consume more power than any of the devices in the processing subsystem <NUM>.

The device <NUM> can also optionally include a main machine learning (ML) engine <NUM>. The main ML engine <NUM> is a special-purpose processing device that is configured to perform inference passes through one or more machine learning models. Each inference pass uses inputs and learned parameter values of a machine learning model to generate one or more outputs predicted by the learned model. The main ML engine <NUM> can include one or more compute tiles. In general, a compute tile is a self-contained computational component configured to execute a set of computations independently. Tiles used for accelerating machine learning typically have massively parallel architectures. In some implementations, each compute tile includes a grid of computational arrays, with each element in the computational array being a processing element that can independently execute mathematical operations. Thus, for example, to compute a single 3x3 convolution, a tile can use <NUM> computational arrays in parallel, with each computational array performing <NUM> or <NUM> tensor multiplications in parallel between the inputs and the weights of the model. A suitable machine learning engine having multiple compute tiles is described in <CIT>, which is incorporated herein by reference.

The tiles of the main ML engine <NUM> can be arranged in a network and programmed so that each tile of the main ML engine <NUM> is configured to perform operations of one portion of an inference pass through the machine learning model. For example, if the machine learning model is a neural network, each tile in the main ML engine <NUM> can be configured to compute the computations of one layer of the neural network.

The main ML engine <NUM> provides higher performance computing power than any of the devices in the processing subsystem <NUM> of the ambient computing system <NUM>. Therefore, the main ML engine <NUM> also consumes more power than any of the devices in the processing subsystem <NUM>.

The processing subsystem <NUM> optionally includes an ambient machine learning engine <NUM>. The ambient ML engine <NUM> is also a special-purpose processing device that is arranged within the ambient computing system <NUM> and configured to perform inference passes through one or more machine learning models. When the device <NUM> includes both a main ML engine <NUM> and an ambient ML engine <NUM>, the ambient ML engine <NUM> has fewer compute tiles and therefore has less processing power than the main ML engine <NUM> and consumes less power than the main ML engine <NUM>. For example, the ambient ML engine <NUM> can be implemented as one or two tiles, whereas the main ML engine <NUM> can have <NUM>-<NUM> or more interconnected tiles.

As described above, the processing subsystem <NUM> can save power by performing low-power detections of vision sensor data using an ambient ML engine <NUM>, and can then perform high-power detections using the fully enabled ML engine <NUM>.

Alternatively or in addition, the ambient computing system <NUM> can reconfigure the main ML engine <NUM> to operate in a reduced power mode. In the reduced power mode, fewer than all compute tiles are enabled. Thus, some compute tiles might not be used, while other compute tiles might be used repeatedly for different portions of the inference pass. For example, the system can enable a single tile on the main ML engine <NUM> and can use the single tile to compute all layers of a neural network for low-power detections. Of course, using a single tile makes computing detections slower, but it also consumes power at a lower rate. In addition, as described above, the vision sensor can generate data at a lower frame rate for low-power detections. Thus, the decrease in processing speed due to using one compute tile may still be enough to compute one inference pass per frame. Other configurations are possible, e.g., a reduced power mode that uses <NUM>, <NUM>, <NUM>, <NUM>, or some other proper subset of tiles in the main ML engine <NUM>.

An advantage to using a reduced power mode of the main ML engine <NUM> is reduced chip size and cost due to not needing to include a separate compute tile within the ambient computing system to implement the ambient ML engine <NUM>.

Although not depicted, the computing device <NUM> can also include one or more other components commonly found on such computing devices, e.g., a display, a modem, a graphics processing unit, a display processor, or a special-purpose image processor, to name just a few examples. These components can be powered down during the low-power states described below and activated if the system determines that the sensor signals match an application requiring their activation.

The device <NUM> includes a number of peripheral sensors <NUM>. The peripheral sensors <NUM> include one or more audio sensors <NUM>, one or more radar sensors <NUM>, one or more touch sensors <NUM>, a Global Positioning System (GPS) sensor <NUM>, and an accelerometer <NUM>. The system can include additional, fewer, or alternative peripheral sensors. For example, the system can include a Wi-Fi signal detector, a cellular signal detector, a barometer, a thermometer, a magnetometer, or other types of peripheral sensors.

The peripheral sensors <NUM> can be devices configured to generate sensor signals in response to environmental inputs. The one or more audio sensors <NUM>, e.g., microphones, can generate audio signals based on sounds in the environment. For example, the audio sensors <NUM> can generate audio signals corresponding to human speech. The one or more radar sensors <NUM> can detect radar signals based on reflected radio waves emitted by a transmitter of the computing device. Variations in reflected radio waves can indicate movement in the environment. For example, the radar sensors <NUM> can generate radar signals that are received due to being reflected off of the user, e.g., when the user is making gestures in proximity to the computing device. Similarly, the one or more touch sensors <NUM> can generate signals due to touch gestures made by a user of the computing device on a presence-sensitive or pressure-sensitive interface of the device. The GPS sensor <NUM> can generate signals in response to received location data communications. And the accelerometer <NUM> can generate signals due to accelerations experienced by the computing device. And as described above, the vision sensor <NUM> can generate vision sensor data, which can have a lower resolution and framerate for performing low-power detections. In this specification, whenever sensor signals are described as being inputs to other processing components, the inputs can be analog electrical signals generated by the sensors themselves, digital representations of the sensor signals, or processed digital representations of the sensor signals that represent one or more properties of the original signals. The peripheral sensors of the computing device <NUM> can also include an inertial measurement sensor, a barometer, a specific absorption rate proximity sensors, and WiFi network name sensors, to name just a few other examples.

The ambient computing system <NUM> includes one or more peripheral interfaces <NUM>. The peripheral interfaces <NUM> can be a component of the computing device <NUM> that is powered on even when the device is in its lowest power state. The peripheral interfaces <NUM> can include any appropriate peripheral interface for converting inputs received from the peripheral sensors <NUM> into sensor signals to be used by the ambient computing system <NUM>. For example, the peripheral interfaces <NUM> can include a pulse density modulation (PDM) interface, an inter-IC sound (I2S) interface, an inter-integrated circuit (I2C) interface, an I3C interface, a time division multiplexed (TDM) interface, and a serial peripheral interface (SPI), to name just a few examples.

Each of the peripheral interfaces <NUM> is configured to generate a respective interrupt upon detecting an environmental input. In general, each interrupt can identify a source of the sensor data, e.g., an identifier of a peripheral interface or sensor responsible for the interrupt. The interrupts are received and processed by one or more interrupt controllers <NUM>. For example, upon receiving an interrupt, the interrupt controller <NUM> can wake a power control unit (PCU) <NUM>, which includes a power management unit (PMU) <NUM> and a clock control unit <NUM>. The PMU <NUM> can control which components of the device <NUM> receive power and how much power each component receives. The clock control unit <NUM> can control the frequency at which the components of the device <NUM> operate. In some implementations, each processing component has a different clock frequency that is a multiple or a fraction of a base clock frequency. By having a clock frequency that is a multiple or a fraction of a base clock frequency, each processing component can more efficiently exchange signals with other processing components.

Upon receiving an interrupt, the PCU <NUM> can determine based on the source of the interrupt which other components of the ambient computing system <NUM> should be activated in order to further process the sensor signals causing the interrupt. In order to provide processing support for such components, the PCU <NUM> can wake the static random access memory (SRAM) <NUM> and the system communications fabric. The fabric is a communications subsystem that communicatively couples the internal components of the ambient computing system <NUM>, their communications to external components, or some combination of these. The fabric can include any appropriate combination of communications hardware, e.g., buses or dedicated interconnect circuitry.

The static random access memory (SRAM) <NUM> is a general purpose random-access memory device that can be shared by multiple processing components of the processing subsystem <NUM>. For example, the SRAM <NUM> can store sensor signals, processor instructions and data, system outputs, and other data, e.g., neural network parameters of neural network models that are or will be implemented by the ambient ML engine <NUM>. In general, an SRAM is distinguishable from dynamic random-access memory (DRAM) in that an SRAM need not be periodically refreshed. As described in more detail below, the SRAM <NUM> is accessible to the processing components in the processing subsystem <NUM> directly or through one or more DMA controllers. In some implementations, the SRAM <NUM> includes multiple banks, which can each store substantially similar amounts of data, e.g., <NUM>, <NUM>, or <NUM> MB each. In addition, each individual bank can include multiple blocks that can be individually powered-down when entering the low-power state. By carefully sequencing the order that the blocks are powered-down amongst the four banks, the SRAM address space can remain contiguous.

When the PCU <NUM> wakes the SRAM <NUM>, the PCU <NUM> can wake fewer than all of the blocks or all of the memory banks of the SRAM <NUM>. The PCU <NUM> can instead wake only a number of blocks that is sufficient for the next component of the processing subsystem <NUM> to determine whether to further escalate powering up of components of the device <NUM>.

The PCU <NUM> can also supply different power levels to different blocks of the SRAM <NUM>. For example, in the monitoring power state, the PMU <NUM> can supply a lower, retention voltage to the entire SRAM <NUM> to reduce its power consumption. The PMU <NUM> can also supply the retention voltage to the SRAM <NUM> if no processing components need to access to the SRAM <NUM>. In the processing power state, the PMU <NUM> can provide normal voltage to all or portions of the SRAM <NUM> and lowered or no voltage to other parts of the SRAM <NUM>.

During the process of handling an interrupt, the ambient computing system <NUM> can also wake one or more DMA controllers <NUM>. The DMA controllers <NUM> can manage DMA pathways that allow higher data bandwidth for incoming sensor signals. For example, a DMA controller <NUM> can be used to continuously stream audio data from a microphone into the SRAM <NUM> for access by processing components in the processing subsystem <NUM>. Conversely, a DMA controller can also be used to continuously stream audio data stored in the SRAM <NUM> for output as sound through one or more speakers. As another example, a DMA controller <NUM> can be used to stream vision sensor data from the vision sensor <NUM> into the SRAM <NUM>. The DMA controllers <NUM> can also be used to stream any appropriate sensor data into the SRAM <NUM>, but using programmed IO may be computationally cheaper than activating a DMA controller for small quantities of data. Thus, the ambient computing system <NUM> can activate and use the DMA controllers <NUM> for relatively high-bandwidth sensor data, e.g., audio data and radar data, and can used programmed IO for other types of sensor data.

After preparing the fabric and the SRAM <NUM>, the PCU <NUM> can then use the interrupts to determine which other components of the processing subsystem <NUM> to wake. For example, the PMU <NUM> can control whether power is provided to the low-power CPU <NUM>, the low-power DSP <NUM>, or other components of the processing subsystem <NUM> depending on which of one or more sensors generated an interrupt. In some implementations, the peripheral interfaces <NUM> and the components of the control subsystem <NUM> are the only components of the device <NUM> that are powered on in a monitoring power state, which is a power state in which the system is waiting to receive interrupts due to environmental inputs to the computing device.

The processing components of the processing subsystem <NUM> include a low-power CPU <NUM>, an ambient ML engine <NUM>, a low-power DSP <NUM>, and a high-power DSP <NUM>. In some implementations, the processing subsystem has multiple instances of one or more of these components, e.g., multiple low-power DSPs or multiple high-power DSPs. For example, the processing subsystem <NUM> can have one high-power DSP that is dedicated to processing audio signals and a separate high-power DSP that is dedicated to processing radar signals. Alternatively or in addition, the processing subsystem <NUM> can have a high-power DSP that is dedicated to processing vision sensor data generated by the vision sensor <NUM>.

In the monitoring power state, the processing components in the processing subsystem <NUM> can be maintained in a retention mode. The PCU <NUM> can maintain a component in retention mode by reducing or eliminating power that is provided to the component. For example, in the retention mode, the PCU <NUM> can supply a processing component with just enough power to maintain register states, but not enough power to process data in the registers.

The low-power CPU <NUM> can be a general-purpose programmable processor that includes registers, control circuitry, and an arithmetic logic unit (ALU). In general, the low-power CPU <NUM> consumes less power than the main CPU cluster <NUM> of the computing device, and may contain fewer processing cores. In some implementations, the low-power CPU <NUM> is primarily a scalar processor that operates on single instructions and single data inputs.

The low-power DSP <NUM> and the high-power DSP <NUM> can be special-purpose processors configured for efficient decoding and processing of highly-vectorized signals. The processing subsystem <NUM> can include a variety of DSPs that are designed for different purposes. For example, the processing subsystem <NUM> can include a DSP that is configured to process vision sensor data, a DSP that is configured to process audio signals, a DSP that is configured to perform dataplane algorithms, a DSP that is configured to process wireless communications signals, and a DSP that is configured to process GPS signals, to name just a few examples. In general, high-power DSPs consume higher levels of power than low-power DSPs because they have more active registers, they access and process more data in parallel, because they rely more heavily on memory operations, or some combination of these.

In operation, the low-power CPU <NUM> can receive interrupts and sensor signals when the system enters the processing power state. Based on the type of sensor signals the lower-power CPU <NUM> receives and based on the properties of those sensor signals, the low-power CPU <NUM> can determine that other components of the system should be activated, e.g., the communications fabric, the DMA controllers <NUM>, the SRAM <NUM>, or some combination of these. After activating these components, the low-power CPU <NUM> can optionally return to a non-operational state.

To support low-power detections, the low-power CPU <NUM> can wake other components of the processing subsystem <NUM> when vision sensor data is received. As described above, this can include waking an ambient ML engine <NUM> to perform a low-power detection on vision sensor data stored in the frame buffer <NUM> of the SRAM <NUM>. Alternatively or in addition, the low-power CPU <NUM> can wake the main ML engine <NUM> in a low-power mode in which fewer than all tiles are enabled. The low-power CPU <NUM> can then stream data in the frame buffer <NUM> to the main ML engine <NUM> to perform the low-power detections.

In some implementations, the vision sensor data is first preprocessed by a software image processing pipeline that is executed by one or more components of the processing subsystem. For example, the low-power DSP <NUM> or the high-power DSP <NUM> can perform one or more stages of an image processing pipeline to enhance the vision sensor data before being used for low-power detections. These components can execute the image processing pipeline using instructions stored in a portion of the SRAM <NUM>.

In some other implementations, the vision sensor data is preprocessed by a special-purpose image processing device. For example, the high-power DSP <NUM> can be a two-dimensional DSP that is specifically designed for processing vision sensor data. The high-power DSP <NUM> can be configured to operate on the same resolution and framerate that the vision sensor <NUM> generates for low-power detections.

If the result of the low-power detection is positive, the low-power CPU <NUM> can wake higher powered components to perform a high-power detection process. This can include waking the main CPU cluster <NUM>, the main ML engine <NUM>, the main ISP <NUM>, or some combination of these. Alternatively or in addition, the high-power detection can be performed by another component of the ambient computing system <NUM>, e.g., the high-power DSP <NUM>. As described above, the high-power detections may operate on vision sensor data that is preprocessed by an image processing pipeline implemented by the main ISP <NUM> or another component of the ambient computing system <NUM>. If the image preprocessing is performed by the main ISP <NUM>, the ambient computing system <NUM> can wake the main ISP <NUM> upon receiving an indication that the low-power detection was positive. If the image preprocessing is performed by another component of the ambient computing system <NUM>, the results may be less sophisticated, but with the advantage that the main ISP <NUM> does not need to be powered on. The control subsystem <NUM> can also include a timer <NUM>, which is an electronic timer that can detect system malfunctions and resolve those malfunctions. During normal operation, the system can regularly reset the timer <NUM> to prevent the timer <NUM> from timing out. If, e.g., due to a hardware fault or a program error, the system fails to reset a timer, the timer will elapse and generate a timeout signal. The timeout signal can be used to initiate one or more corrective actions. A corrective action can include placing the system in a safe state and restoring normal system operation.

<FIG> is a flowchart of an example process for performing a two-stage vision detection process. As described above, an ambient computing system can use a low-power processing component to perform a low-power detection process on vision sensor data, and, if a detection is indicated, use a high-power processing component to perform a high-power detection process. The example process will be described as being performed by a system having a vision sensor, a low-power processing component, and high-power processing component, e.g., the computing device <NUM> of <FIG>.

The system generates vision sensor data using a vision sensor (<NUM>).

The system performs a low-power detection process using a low-power processing component (<NUM>). As described above, a low-power processing component can be any of the devices on the ambient computing system or a main ML engine that is partially enabled. In general, the low-power processing component consumes less power than a main CPU cluster or a main ML engine when fully enabled. In addition, during the low-power detection process, the vision sensor can be configured to generate data at a lower framerate, a lower resolution, or both.

If no detection is indicated by the low-power detection process (<NUM>), the system again repeats the low-power detection process using the next captured vision sensor data (branch to <NUM>).

If a detection is indicated by the low-power detection process (<NUM>), the system wakes a high-power processing component (branch to <NUM>). As described above, this can mean supplying power to a high-power processing component when none was supplied before, increasing power to a high-power processing component, or enabling additional capabilities of the high-power processing component. For example, the system can wake additional compute tiles of a machine learning engine.

The system performs a high-power detection process using a high-power processing component (<NUM>). As described above, the high-power processing component can be a main ML engine or a DSP of the ambient computing system. The high-power detection process can either use the already captured vision sensor data, or updated vision sensor data.

If no detection is indicated by the high-power detection process (<NUM>), the system can again power down the high-power processing component to its original state (branch to <NUM>) and repeat the low-power detection process using the next captured vision sensor data (<NUM>). Alternatively or in addition, the system can perform multiple iterations of the high-power detection process before switching back to performing the low-power detection process.

If a detection is indicated by the high-power detection process (<NUM>), the system can invoke a processing component to handle the high-power detection (branch to <NUM>). For example in the phone unlock scenario, the system can invoke the main CPU cluster to unlock the phone.

For some applications, the system invokes a processing component to process the output of the high-power detection process regardless of the results of the process. For example, in the application of automatic text translation, the system can repeatedly output, to a display of the device, results of the high-power text translation process regardless of the results of that process.

Embodiments of the subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, an engine, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, engine, subroutine, or other unit suitable for executing in a computing environment, which environment may include one or more computers interconnected by a data communication network in one or more locations.

A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on, or configured to communicate with, a computer having a display device, e.g., a LCD (liquid crystal display) monitor, for displaying information to the user, and an input device by which the user can provide input to the computer, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball or touchpad. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., a smartphone or electronic tablet.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client device having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what is being or may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim may be directed to a subcombination or variation of a subcombination.

Claim 1:
A computing device (<NUM>) comprising:
a vision sensor (<NUM>) configured to generate vision sensor data;
an ambient computing system (<NUM>) configured to repeatedly process the vision sensor data generated by the vision sensor (<NUM>) according to a low-power detection process, wherein the ambient computing system comprises a general purpose random-access memory, SRAM, device (<NUM>);
means for performing a high-power detection process when a detection is indicated by the low-power detection process;
a camera (<NUM>) and a main image signal processor, ISP, (<NUM>) that is configured to process data generated by the camera (<NUM>), characterised in that the ambient computing system (<NUM>) is configured to allocate a dedicated frame buffer (<NUM>) in the general purpose random-access memory device of the ambient computing system (<NUM>); wherein the computing device (<NUM>) is configured to use a private channel (<NUM>) between the frame buffer (<NUM>) configured for storing the vision sensor data and the main ISP (<NUM>) to generate preprocessed vision sensor data for use during the high-power detection process; and
the computing device (<NUM>) further comprising
the private channel (<NUM>) from the frame buffer (<NUM>) to the main ISP (<NUM>).