Patent Description:
A machine learning ("ML") accelerator is a device or a component on a device, e.g., an integrated circuit having a specialized architecture designed for efficiently training machine learning models, executing machine learning models, or both training machine models and executing machine learning models.

An ML accelerator can be configured to perform inference passes through one or more machine learning models. Each inference pass uses inputs and learned parameter values of a machine learning model to generate one or more outputs predicted by the learned model. The ML accelerator can include one or more compute tiles. In general, a compute tile is a self-contained computational component configured to execute a set of computations independently. The tiles of an ML accelerator can be arranged in a network and programmed so that each tile of the ML accelerator is configured to perform operations of one portion of an inference pass through the machine learning model. For example, if the machine learning model is a neural network, each tile in the main ML engine <NUM> can be configured to compute the computations of one layer of the neural network.

ML accelerators require large amounts of memory to flexibly process different kinds of machine learning models. If an ML accelerator is a component in an ambient computing device, e.g., a cellphone or other computing device, which can be in a low-power state yet monitor and respond to inputs from the environment, this requirement creates at least two problems:.

<CIT> proposes a system to facilitate virtual page translation. An embodiment of the system includes a processing device, a front end unit, and address translation logic. The processing device is configured to process data of a current block of data. The front end unit is coupled to the processing device. The front end unit is configured to access the current block of data in an electronic memory device and to send the current block of data to the processor for processing. The address translation logic is coupled to the front end unit and the electronic memory device. The address translation logic is configured to pre-fetch a virtual address translation for a predicted virtual address based on a virtual address of the current block of data. Embodiments of the system increase address translation performance of computer systems including graphic rendering operations.

<CIT> proposes a compute apparatus to perform machine learning operations. The compute apparatus comprises a decode unit to decode a single instruction into a decoded instruction, the decoded instruction to cause the compute apparatus to perform a complex machine learning compute operation.

An aspect of the present disclosure provides an ambient computing system as defined in claim <NUM>. Other aspects provide a method for virtualizing memory on a ambient computing system as defined in claim <NUM> and one or more computer-readable storage media encoded with instructions to be executed by a machine learning engine of an ambient computing system as defined in claim <NUM>.

This specification describes techniques for virtualizing external memory for use by an ML accelerator. The ML accelerator can include logic, e.g., implemented as an integrated circuit, for translating virtual memory addresses accessed by the ML accelerator while processing or training a machine learning model. The virtual memory addresses are translated to memory locations external to the ML accelerator, such as on RAM or on a system-level cache communicatively connected to the processing subsystem the ML accelerator is implemented in. Machine learning models with corresponding parameters can be streamed from outside the ML accelerator and accessed by the ML accelerator to simulate reading and writing to locations in memory that are local to the ML accelerator.

A small, low-power, ML accelerator can be implemented to process ambient signals received by an ambient computing device. The ML accelerator can be a single compute tile with access to larger shared caches of memory, without restricting access of the larger caches of memory only to the ML accelerator.

Virtual memory addresses can be assigned to stream machine learning models and corresponding parameters economically, because the ML accelerator can expand or shrink a virtual memory address range as dictated by memory requirements for a particular machine learning model. The ML accelerator can access a machine learning model and parameters stored on memory external to the ML accelerator without having to re-stream data, e.g., without having to re-stream often re-used model parameters in a convolutional neural network. Further, the ML accelerator can access the machine learning model and parameters stored externally without any special configuration. Put another way, from the perspective of the ML accelerator, the ML accelerator appears to be accessing memory local to the accelerator.

Similarly, the ML accelerator can also access data stored on a system-level cache, in devices having that memory configuration. The ability to allocate virtual memory addresses obviates the need for large amounts of dedicated memory to the ML accelerator, thereby reducing power consumption and the physical space the ML accelerator has to occupy in the ambient computing device.

Machine learning models compiled for executing on the ML accelerator can use the memory virtualization features disclosed without extensive configuration or customization. A compiler for a machine learning model can compile the model with minimal added instructions indicating that memory virtualization should be used.

This specification describes techniques for implementing memory virtualization as local to a machine learning ("ML") accelerator. Instead of dedicating large amounts of memory as local to the ML accelerator, the ML accelerator can access external memory through virtual memory addresses automatically assigned by virtual address logic local to the ML accelerator. The virtual address logic for the ML accelerator can also include logic to turn this memory virtualization feature on or off, depending on a configuration option added to a machine learning model during compilation of a program that implements inference passes over the model.

This and other configuration options can be included during the compilation of a machine learning model as compiled instructions that are executable by an ML accelerator, such as an ambient ML engine <NUM> or a main ML engine <NUM>, discussed below.

<FIG> is a diagram of an example computing device <NUM> implementing an ML accelerator with virtual address logic. The device <NUM> can include an ambient computing system <NUM> implemented in any appropriate computing device, e.g., a smart phone, a smart watch, a fitness tracker, a personal digital assistant, an electronic tablet, a laptop, to name just a few examples. The ambient computing system <NUM> of the computing device <NUM> can be used so that the computing device <NUM> can remain in a low-power state yet continually monitor and respond to inputs from the environment by sequentially waking appropriate processing components of the system. While the ambient computing system <NUM> is discussed here with respect to <FIG>, a general discussion of implementing low-power ambient computing systems can be found in International App. No. PCT/US2018/<NUM>.

The one or more components of the computing device <NUM> can be implemented on a system on a chip ("SOC") within the computing device. An SOC can be an integrated circuit that includes each component of the system on a single silicon substrate or on multiple interconnected dies, e.g., using silicon interposers, stacked dies, or interconnect bridges. Other components of the computing device, including a main CPU cluster <NUM>, a main ML engine <NUM>, or a processing subsystem <NUM>, can be implemented on the same or on a separate die.

The computing device <NUM> may include components, including the sensors <NUM>, one or more displays, a battery, and other components, that are separate from and independent of the SOC, and may for example be mounted on a common housing. The computing device <NUM> includes a control subsystem <NUM> for controlling the supply of power and sensor signals to components in the system. The device <NUM> includes a processing subsystem <NUM> for processing sensor signals and generating outputs.

The device <NUM> can include a number of peripheral sensors <NUM>. The peripheral sensors <NUM> can include one or more audio sensors <NUM>, one or more radar sensors <NUM>, one or more touch sensors <NUM>, a Global Positioning System ("GPS") sensor <NUM>, and/or an accelerometer <NUM>. The system can include additional, fewer, or alternative peripheral sensors. The peripheral sensors <NUM> can be devices configured to generate sensor signals in response to environmental inputs.

The ambient computing system <NUM> can include one or more peripheral interfaces <NUM>. The peripheral interfaces <NUM> can be a component of the computing device <NUM> that is powered on even when the device is in its lowest power state. The peripheral interfaces <NUM> can include any appropriate peripheral interface for converting inputs received from the peripheral sensors <NUM> into sensor signals to be used by the ambient computing system <NUM>.

Each of the peripheral interfaces <NUM> is configured to generate a respective interrupt upon detecting an environmental input. In general, each interrupt can identify a source of the sensor data, e.g., an identifier of a peripheral interface or sensor responsible for the interrupt. The interrupts are received and processed by one or more interrupt controllers <NUM>. For example, upon receiving an interrupt, the interrupt controller <NUM> can wake a power control unit ("PCU") <NUM>, which includes a power management unit ("PMU") <NUM> and a clock control unit <NUM>. The PMU <NUM> can control which components of the device <NUM> receive power and how much power each component receives. The clock control unit <NUM> can control the frequency at which the components of the device <NUM> operate.

In this specification, whenever sensor signals are described as being inputs to other processing components, the inputs can be analog electrical signals generated by the sensors themselves, digital representations of the sensor signals, or processed digital representations of the sensor signals that represent one or more properties of the original signals.

Upon receiving an interrupt, the PCU <NUM> can determine based on the source of the interrupt which other components of the ambient computing system <NUM> should be activated in order to further process the sensor signals causing the interrupt. In order to provide processing support for such components, the PCU <NUM> can wake the static random access memory ("SRAM") <NUM> and the system communications fabric.

The system communications fabric is a communications subsystem that communicatively couples the internal components of the ambient computing system <NUM>, their communications to external components, or some combination of these. The fabric can include any appropriate combination of communications hardware, e.g., buses or dedicated interconnect circuitry.

Although not depicted, the computing device <NUM> can also include one or more other components commonly found on such computing devices, e.g., a display, a modem, a graphics processing unit, a display processor, or a special-purpose image processor, to name just a few examples. These components can be powered down during the low-power states described below and activated if the system determines that the sensor signals match an application requiring their activation.

The device <NUM> also includes a main CPU cluster <NUM>. The main CPU cluster <NUM> is a component of the computing device <NUM> that can include one or more general-purpose processors that are separate from the components in the processing subsystem <NUM>. The processors of the main CPU cluster <NUM> generally have more computing power than any of the components in the processing subsystem <NUM>, and therefore, the processors of the main CPU cluster <NUM> may also consume more power than any of the components in the processing subsystem <NUM>.

The control subsystem <NUM> can also include a timer <NUM>, which is an electronic timer that can detect system malfunctions and resolve those malfunctions. During normal operation, the control subsystem <NUM> can regularly reset the timer <NUM> to prevent the timer <NUM> from timing out. If, e.g., due to a hardware fault or a program error, the control subsystem <NUM> fails to reset a timer, the timer will elapse and generate a timeout signal. The timeout signal can be used to initiate one or more corrective actions. A corrective action can include placing the ambient computing system <NUM> in a safe state and restoring normal system operation.

The processing subsystem <NUM> includes an ambient machine learning engine <NUM>. The ambient ML engine <NUM> is a special-purpose processing device that is configured to perform inference passes through one or more machine learning models.

The ambient ML engine <NUM> can include one or more multiply accumulate ("MAC") units and one or more sum registers for computing neural network activations or other neural network layer outputs, and a controller for controlling data exchange between sum registers and the MAC units. The ambient ML engine <NUM> can also include instruction memory, direct memory access paths, registers, and other processing components. In some implementations, the ambient ML engine <NUM> is a single machine learning compute tile that is configured to accelerate the computation of machine learning inference passes.

The ambient ML engine includes virtual address logic <NUM>. The virtual address logic <NUM> can be a specialized circuit in the ambient ML engine <NUM> that can translate virtual addresses generated by the ambient ML engine into physical memory addresses in the SRAM <NUM>, which is nonlocal memory for the ambient ML engine <NUM>. In this specification, nonlocal memory for a component of the computing device <NUM>, e.g., the ambient ML engine <NUM>, refers to memory that is used by the component and one or more other components. In other words, nonlocal memory is not used exclusively by the component.

For example, the SRAM <NUM> can be a general purpose static random-access memory device that can be shared by multiple processing components of the processing subsystem <NUM>, e.g., the low-power DSP <NUM>, the high-power DSP <NUM>, the low-power CPU <NUM>, as well as the ambient ML engine <NUM>. Therefore, the SRAM is nonlocal memory for the ambient ML engine <NUM>. In contrast, the ambient ML engine <NUM> can also include local memory that is used exclusively by the ambient ML engine <NUM>, and which may be integrated into the same silicon die as the rest of the ambient ML engine. For example, the ambient ML engine <NUM> can have local memory that includes one or more integrated registers. The integrated registers are local memory for the ambient ML engine <NUM> because data in the registers can only be read from or written to by only the ambient ML engine <NUM>.

The SRAM <NUM> can store sensor signals, processor instructions and data, system outputs, and other data, e.g., neural network parameters of neural network models that are or will be implemented by the ambient ML engine <NUM>.

In general, an SRAM is distinguishable from DRAM in that SRAM need not be periodically refreshed. As described in more detail below, the SRAM <NUM> is accessible to the processing components in the processing subsystem <NUM> directly or through direct memory access ("DMA") controllers <NUM>. In some implementations, the SRAM <NUM> includes multiple banks of memory, each having substantially similar data capacities, e.g., <NUM>, <NUM>, or <NUM> MB each. In addition, each individual bank of memory can include multiple memory blocks that can be individually powered-down when entering a low-power state. By carefully sequencing the order that the blocks are powered-down amongst the multiple banks of memory, the SRAM memory address space can remain contiguous.

The virtual address logic <NUM> can translate virtual memory addresses that the ambient ML engine <NUM> generates. In some implementations, the virtual address logic <NUM> maintains a mapping between virtual pages and physical pages, e.g., using the most significant bits of the virtual addresses generated by the ambient ML engine <NUM>.

The virtual address logic <NUM> can receive a request to read from or write to a virtual address generated by the compiled instructions executed by the ambient ML engine <NUM>. The virtual address logic <NUM> can then map the virtual address to a physical address in the SRAM <NUM>. In some implementations, the virtual address logic <NUM> maps a virtual page number to a physical page number and copies the least significant bits to generate the physical address.

Translating a virtual memory address into a physical memory address means that when an ML accelerator executes the compiled instructions for a machine learning model, every read or write instruction to data in a memory address in the compiled instructions results in data at a corresponding physical memory address location being read or written to, instead. In some implementations, the virtual address logic <NUM> is configured to issue read or write instructions to a corresponding physical memory address location in the SRAM <NUM>, in response to the ambient ML engine <NUM> executing a read or write instruction to data at the virtual memory address location mapped to the corresponding physical memory address.

The virtual address logic <NUM> can map the virtual memory addresses to physical memory addresses for locations in memory that are nonlocal to the ambient ML engine <NUM>. In some implementations, the processing subsystem <NUM> can overwrite existing data at the locations referenced by the mapped physical memory addresses. The virtual address logic <NUM> can be configured to perform this initialization step automatically, or in response to a configuration option indicated by a compiled program performing the inference pass on the machine learning model, e.g., as one or more instructions on the compiled program, to be executed on the ambient ML engine <NUM>.

The ambient ML engine <NUM> can execute a compiled program having one or more instructions that performs an inference pass using a machine learning model, by accessing allocated nonlocal memory. From the point of view of the ambient ML engine <NUM>, the data accessed at the virtual memory address locations is treated as local to the ambient ML engine <NUM>, when in practice, the data is accessed from a shared source of memory, such as from the SRAM <NUM> or a system-level cache.

The device <NUM> can also optionally include a main ML engine <NUM>. The main ML engine <NUM> is a special-purpose processing device that is configured to perform inference passes through one or more machine learning models, i.e., execute the machine learning model on the main ML engine <NUM>. Each inference pass uses inputs and learned parameter values of a machine learning model to generate one or more outputs predicted by the learned model, as with the ambient ML engine <NUM>. The main ML engine <NUM> can include one or more compute tiles, which can be arranged in a network and programmed so that each tile of the main ML engine <NUM> is configured to perform operations of one portion of an inference pass through the machine learning model. A suitable machine learning engine having multiple compute tiles is described in <CIT>.

When the device <NUM> includes both a main ML engine <NUM> and an ambient ML engine <NUM>, the ambient ML engine <NUM> generally has fewer compute tiles and therefore has less processing power than the main ML engine <NUM> and consumes less power than the main ML engine <NUM>. For example, the ambient ML engine <NUM> can be implemented as one or two compute tiles, whereas the main ML engine <NUM> can have <NUM> or more interconnected tiles.

Each compute tile may have a small amount of memory local to the tile. The amount of memory local to the compute tile is often insufficient for processing a machine learning model alone, which is why, as discussed above, the individual tiles can be configured into a network to share resources and to allocate the task of processing a machine learn model for a given input as a series of sub-tasks assigned to each compute tile.

Because the ambient ML engine <NUM> is generally implemented with relatively fewer compute tiles than the main ML engine <NUM>, the one or two compute tiles available may not be sufficient for processing a machine learning model, even after optimizing a network configuration or partition the processing among the tiles. This can be because the compute tiles may lack the computational capacity to process the machine learning model, or because the compute tiles may not have sufficient memory. In some implementations, even the most rudimentary networking or partitioning is unavailable, because the ambient ML engine <NUM> is implemented as a single compute tile. Therefore, the virtual address logic <NUM> can perform memory virtualization as required by the ambient ML engine <NUM> to execute a machine learning model.

Although not shown in <FIG>, the main ML engine <NUM> can also include virtual address logic for virtualizing nonlocal memory for the main ML engine <NUM>. While the main ML engine <NUM> generally has more computational resources than the ambient ML engine <NUM>, the main ML engine <NUM> may also require accessing memory not local to the main ML engine <NUM> to execute certain machine learning models. In those cases, virtual address logic can be implemented for the main ML engine <NUM> using the same techniques described for the virtual address logic <NUM> in the ambient ML engine <NUM>. Instead of translating virtual memory addresses to physical memory addresses in SRAM as described with the ambient ML engine <NUM>, virtual address logic for a main ML engine <NUM> can be configured to translate virtual memory addresses to physical memory addresses of a system-level cache ("SLC").

An SLC can be a device or a component of a device, e.g., the computing device <NUM>, that can cache data retrieved from memory or data to be stored in memory for multiple different hardware devices in a system. In other words, different cache lines of the SLC can store data belonging to different hardware devices. In some implementations and as discussed below with respect to <FIG>, virtual address logic can be implemented on the main ML engine <NUM> to translate virtual addresses corresponding to physical memory locations on the SLC.

Next, a discussion of an example operation of the ambient computing system <NUM> is presented. In this specification, the terms "wake" and "activate" will be used to mean supplying an increased amount of power to a particular processing component or other circuitry for electronics. The ambient computing system <NUM> may or may not have been supplying power to a processing component or other circuitry that is being awoken or activated. In other words, a component being awoken or activated may or may not have been completely powered down previously. Waking or activating a processing component can result in the processing component performing a boot process and causing instructions and data for the processing component to be loaded into random-access memory. Alternatively or in addition, waking or activating a processing component can include resuming from a previously suspended state.

When the PCU <NUM> wakes the SRAM <NUM>, the PCU <NUM> can wake fewer than all of the blocks or all of the memory banks of the SRAM <NUM>. The PCU <NUM> can instead wake only a number of blocks that is sufficient for the next component of the processing subsystem <NUM> to determine whether to further escalate powering up of components of the device <NUM>.

The PCU <NUM> can also supply different power levels to different blocks of the SRAM <NUM>. For example, in the monitoring power state, the PMU <NUM> can supply a lower, retention voltage to the entire SRAM <NUM> to reduce its power consumption. The PMU <NUM> can also supply the retention voltage to the SRAM <NUM> if no processing components need access to the SRAM <NUM>. In the processing power state, the PMU <NUM> can provide normal voltage to all or portions of the SRAM <NUM> and lowered or no voltage to other parts of the SRAM <NUM>.

During the process of handling an interrupt, the ambient computing system <NUM> can also wake the one or more DMA controllers <NUM>. The DMA controllers <NUM> can manage DMA pathways that allow higher data bandwidth for incoming sensor signals. For example, a DMA controller can be used to continuously stream audio data from a microphone into the SRAM <NUM> for access by processing components in the processing subsystem <NUM>. Conversely, a DMA controller can also be used to continuously stream audio data stored in the SRAM <NUM> for output as sound through one or more speakers. The DMA controllers <NUM> can also be used to stream any appropriate sensor data into the SRAM <NUM>, but using programmed IO may be computationally cheaper than activating a DMA controller for small quantities of data. Thus, the ambient computing system <NUM> can activate and use the DMA controllers <NUM> for relatively high-bandwidth sensor data, e.g., audio data and radar data, and can used programmed IO for other types of sensor data.

After preparing the fabric and the SRAM <NUM>, the PCU <NUM> can then use the interrupts to determine which other component of the processing subsystem <NUM> to wake. For example, the PMU <NUM> can control whether power is provided to the low-power CPU <NUM>, the low-power DSP <NUM>, or other components of the processing subsystem <NUM> depending on which of one or more sensors generated an interrupt. In some implementations, the peripheral interfaces <NUM> and the components of the control subsystem <NUM> are the only components of the device <NUM> that are powered on in a monitoring power state, which is a power state in which the ambient computing system <NUM> is waiting to receive interrupts due to environmental inputs to the computing device.

The processing components of the processing subsystem <NUM> can include a low-power CPU <NUM>, the ambient ML engine <NUM>, a low-power DSP <NUM>, and a high-power DSP <NUM>. In some implementations, the processing subsystem has multiple instances of one or more of these components, e.g., multiple low-power DSPs or multiple high-power DSPs. For example, the processing subsystem <NUM> can have one high-power DSP that is dedicated to processing audio signals and a separate high-power DSP that is dedicated to processing radar signals. Alternatively or in addition, the processing subsystem <NUM> can have a high-power DSP that is dedicated to processing image data.

In the monitoring power state, the processing components in the processing subsystem <NUM> can be maintained in a retention mode. The PCU <NUM> can maintain a component in retention mode by reducing or eliminating power that is provided to the component. For example, in the retention mode, the PCU <NUM> can supply a processing component with just enough power to maintain register states, but not enough power to process data in the registers.

The low-power CPU <NUM> can be a general-purpose programmable processor that includes registers, control circuitry, and an arithmetic logic unit ("ALU"). In general, the low-power CPU <NUM> consumes less power than the main CPU cluster <NUM> of the computing device, and may contain fewer processing cores. In some implementations, the low-power CPU <NUM> is primarily a scalar processor that operates on single instructions and single data inputs. Based on the type of sensor signals the low-power CPU <NUM> receives and based on the properties of those sensor signals, the low-power CPU <NUM> can determine that other components of the system should be activated, e.g., the communications fabric, the DMA controllers <NUM>, some or all of the SRAM <NUM>, or some combination of these. After activating these components, the low-power CPU <NUM> can optionally return to a non-operational state.

The low-power CPU <NUM> can provide the sensor signals, or a processed version thereof, to the ambient ML engine <NUM> for further interpretation. For example, if the low-power CPU <NUM> receives sensor signals corresponding to accelerometer input, the low-power CPU <NUM> can determine that the ambient ML engine <NUM> should further process the sensor signals. The ambient ML engine <NUM> can then further process the sensor signals.

One task of the ambient ML engine <NUM> is to use sensor signals to perform an inference pass over a machine learning model to generate an output that may trigger waking other processing components to further process the sensor signals. In other words, the ambient ML engine <NUM> can receive sensor signals, or a processed version thereof generated by the low-power CPU <NUM> or another processing component, and the ambient ML engine <NUM> can generate an output that represents which other processing components should further process the sensor signals.

The ambient ML engine <NUM> can also execute machine learning models for a variety of different tasks, including for: on-chip automatic speech recognition, text-to-speech generation, or gesture recognition by a user of the computing device. The ambient ML engine <NUM> can provide output from executing a machine learning model to the low-power CPU <NUM>, or another processing component, for further action.

As discussed above, the virtual address logic <NUM> can be configured to execute memory virtualization when indicated to, e.g., from an instruction in the compiled instructions for a machine learning model. This option can be set during the compilation of the machine learning model by an appropriately configured compiler, e.g., as a default option or in response to input from a user prompt for a program implementing the compiler. The compiled instructions for executing the machine learning model on the ambient ML engine <NUM> remain the same whether memory virtualization is selected or not, but the compiler can additionally indicate, e.g., by one or more instructions, that the ambient ML engine <NUM> should execute the machine learning model using memory virtualization.

If memory virtualization is enabled, then the ambient ML engine <NUM> can stream in model parameters and other model configuration information from sources of memory not local to the ambient ML engine <NUM>, using the mapped virtual memory addresses generated by the virtual address logic <NUM>. For example, model parameters for the machine learning model can be stored in the SRAM <NUM> and referenced by one or more physical memory addresses.

The compiled instructions for the machine learning model can include instructions for loading, reading, and writing data in memory. The virtual address logic can be configured so that virtual memory addresses referenced in the compiled instructions of a compiled machine learning model are translated into corresponding physical memory addresses without altering the references in memory of the compiled instructions.

As an additional step, the processing subsystem <NUM> can initially load model parameters and other configuration information for the machine learning model into the SRAM <NUM>, from another memory device. For example, the DMA controllers <NUM> can stream model parameters from DRAM into the SRAM <NUM>. The DRAM can be local or external to the ambient computing system <NUM>. Then, the virtual address logic <NUM> can map virtual memory addresses to the physical memory locations in the SRAM <NUM> where the model parameters were loaded. As part of streaming the parameters into the SRAM <NUM>, the DMA controllers <NUM> can be configured to overwrite existing data stored in the SRAM <NUM>, or alternatively stream the model parameters into available space in the SRAM <NUM>. One or more instructions in the compiled instructions of the machine learning model can specify whether the DMA controllers <NUM> should overwrite existing data in the SRAM <NUM>.

As discussed above, because the SRAM <NUM> can include multiple memory banks that may or may not be activated depending on the state of the computing device <NUM>, some blocks or banks of memory in the SRAM <NUM> may not be available because there is already existing data, or because those particular blocks or banks of memory have not been activated.

While the DRAM can be part of the ambient computing system <NUM>, the DRAM can also be external to the ambient computing system <NUM>. In some implementations, the DRAM is external to the ambient computing system <NUM> but still on the same SOC. In some implementations, the DRAM is external to the SOC the ambient computing system is implemented on. The DMA controllers <NUM> can be configured to stream model parameters from the DRAM in either implementation.

If model parameters cannot be streamed into the SRAM <NUM>, e.g., because the SRAM <NUM> is currently being used by other processing components of the ambient computing system <NUM>, then the virtual address logic <NUM> can translate virtual memory addresses to physical memory addresses where the model parameters are stored, directly. In some implementations, the virtual address logic can be configured to translate physical memory addresses on the DRAM where model parameters are stored, instead of first streaming the model parameters into the SRAM <NUM> using the DMA controllers <NUM>.

Configuration information for the machine learning model can also include one or more instructions specified at compile-time, indicating how much local memory the ambient ML engine <NUM> should have access to. For example, if the compiler compiled a machine learning model into a set of instructions under the assumption that the ML engine executing the model would have a certain size of memory allocated to it, the virtual address logic can be configured to provide that size of memory as virtual memory addresses mapped to physical memory addresses in the SRAM <NUM>. Alternatively, at compile-time, a program executing the compiler can set a memory size in response to a user prompt or default condition.

The ambient ML engine <NUM> can reference virtual memory addresses as described above in combination with accessing data stored local to the ambient ML engine <NUM>. In some implementations where the ambient ML engine <NUM> includes one or more registers, the ambient ML engine <NUM> can access data not local to the ambient ML engine <NUM> through a virtual memory address, and then load the accessed data to the registers of the ambient ML engine <NUM>. This way, data most likely to be accessed repeatedly by the ambient ML engine <NUM>, such as parameters or a subset of common parameters for a convolution in a convolution neural network model, can be stored on the fastest memory available to the ambient ML engine <NUM>, which is often local registers.

The output generated by the ambient ML engine <NUM> can explicitly specify a combination of processing component IDs or an identifier of an enumerated power state or the output can be a representation of a power state that is interpreted by a low-power processing component, e.g., the low-power CPU or the low-power DSP, in order to identify other higher-power processing components that should process the sensor signals. As part of this process, the low-power processing component can explicitly or implicitly determine whether any other processing is required. For example, the low-power processing component can determine, based on the output of the ambient ML engine <NUM>, that no further processing is required and that the ambient computing system <NUM> can transition back to the monitoring power state.

In the lowest-level monitoring power state, the PCU <NUM> can keep the ambient ML engine <NUM> in a low-power state or powered down completely. In the processing power state, the PCU <NUM> may or may not provide power to the ambient ML engine <NUM> depending on what sensor signals are available at the peripheral interfaces <NUM> and how the low-power CPU <NUM> or the low-power DSP <NUM> interpret the signals. In some implementations, the low-power DSP <NUM> or the low-power CPU <NUM> can interpret the signals to instruct the PCU <NUM> to provide power for an additional, intermediate power state, in which the ambient ML engine <NUM> is also powered on for the inference pass, but no other high-power processing components are yet powered on.

The low-power DSP <NUM> and the high-power DSP <NUM> are special-purpose processors configured for efficient decoding and processing of highly-vectorized signals. The processing subsystem <NUM> can include a variety of DSPs that are designed for different purposes. For example, the processing subsystem <NUM> can include a DSP that is configured to process radar signals or a DSP that is configured to process audio signals.

As described above, the low-power DSP <NUM> can perform the initial interpretation of sensor signals from the control subsystem <NUM>. The low-power DSP <NUM> can also perform other signal processing tasks as well. In general, high-power DSPs consume higher levels of power than low-power DSPs because they have more active registers, they access and process more data in parallel, because they rely more heavily on memory operations, or some combination of these.

<FIG> is a diagram of an example system on a SOC <NUM> implementing an SLC communicatively connected to the computing device <NUM> of <FIG>. The SOC <NUM>, for example, can be installed on or integrated into the computing device <NUM>, or be a separate device or component of a separate device.

Computing device components <NUM> can communicate with an SLC <NUM> through an SOC fabric <NUM>. The computing device components <NUM> can be any components on the computing device <NUM> that are configured to be able to communicate with the SLC <NUM>, and can include the main ML engine <NUM>, the main CPU cluster <NUM>, and the ambient computing system <NUM>.

The SOC fabric <NUM> is a communications subsystem of the SOC <NUM> and can include communications pathways that allow the computing device components <NUM> to communicate with one another as well as to make requests to read and write data on the SLC <NUM>. The SLC <NUM> has dedicated cache memory, which can be implemented using dedicated registers or high-speed RAM. The SOC fabric <NUM> can include any appropriate combination of communications hardware, e.g., buses or dedicated interconnect circuitry.

The SOC <NUM> also includes communications pathways <NUM> that allow communication between the SLC <NUM> and a memory controller <NUM> as well as inter-chip communications pathways <NUM> that allow communication between the memory controller <NUM> and DRAM <NUM> that is not local to the SOC <NUM>. The memory controller <NUM> can handle requests to read and write memory to and from the SLC <NUM> and the DRAM <NUM>. Although only the DRAM <NUM> is shown in <FIG>, the memory controller <NUM> can communicate with other memory devices not shown, such as any volatile or non-volatile memory device, e.g., a hard drive or a solid state drive.

The SLC <NUM> can cache read requests, write requests, or both from the computing device components <NUM>. The SLC <NUM> can cache read requests from client devices by responding to the request with data stored in the cache data rather than fetching the data from the DRAM <NUM>. Similarly, the SLCs can cache write requests from client devices by writing the new data in the cache rather than writing the new data in the DRAM. The SLC <NUM> can then perform a write-back at a later time to store the updated data in the DRAM <NUM>.

As discussed above, the main ML engine <NUM> can include virtual address logic for translating virtual memory addresses to mapped physical memory addresses to memory not local to the main ML engine <NUM>. Whereas the virtual address logic <NUM> mapped physical memory addresses to the shared SRAM <NUM> of the processing subsystem <NUM>, virtual address logic for the main ML engine <NUM> can, in some implementations, map virtual memory addresses to physical memory addresses located in the SLC <NUM>.

When memory virtualization is enabled for a machine learning model executing on the main ML engine <NUM>, the virtual address logic can be configured to communicate with the SLC <NUM> through the SOC fabric <NUM> to translate virtual memory addresses to physical memory addresses in the DRAM <NUM>. As an initialization step, the memory controller <NUM> can stream in model parameters that are being streamed into the SLC <NUM> for the first time from the DRAM <NUM> or other memory device not local to the SOC <NUM>.

<FIG> is a flowchart for an example process for executing a machine learning model on an ML accelerator using memory virtualization. For convenience, the process in <FIG> will be described as being performed by a system of one or more computers located in one or more locations. For example, an ambient computing system, e.g., the ambient computing system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process in <FIG>. Additional detail for implementing a system that can perform the process in <FIG> can be found in the description of <FIG> and <FIG>, above.

The system streams in model parameters from a memory device not local to the system and into a shared memory device (<NUM>). As discussed above with reference to <FIG> and <FIG>, model parameters for a machine learning model executing on an ML accelerator may be stored in memory not local to the system, such as on DRAM. The system, for example through a DMA controller, can stream in the model parameters and any configuration options for executing the machine learning model. The memory device can be, for example, shared SRAM. In some implementations, as discussed earlier, the system can overwrite existing data in the shared memory device. The ML accelerator can be, as discussed above, the ambient ML engine or the main ML engine of the system. If the ML accelerator is the main ML engine of the system, then the shared memory device can be a system-level cache.

The system through the virtual address logic on the ML accelerator generates virtual memory addresses that are mapped to corresponding physical memory addresses for locations storing the model parameters in the shared memory device (<NUM>). As discussed above with reference to <FIG>, the virtual address logic can generate the virtual memory addresses from the memory addresses referenced in read or write instructions in the compiled instructions for the machine learning model. The virtual memory logic can also be configured to generate mappings between the virtual and physical memory addresses and store the mappings in memory local to the ML accelerator, such as in registers.

The system executes a compiled program to perform an inference pass on the machine learning model by executing the compiled instructions translated by the virtual address logic (<NUM>). As discussed above with reference to <FIG>, the system executes a compiled program that performs an inference pass using the machine learning model, specifically using the model parameters stored in the shared memory device. As also discussed above, translating between virtual and physical memory addresses means issuing appropriate read or write instructions to a physical memory address location matching the read or write instructions in the compiled instructions to a location for a virtual memory address mapped to the physical memory address.

A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

As used in this specification, an "engine," or "software engine," refers to a hardware-implemented or software implemented input/output system that provides an output that is different from the input. An engine can be implemented in dedicated digital circuitry or as computer-readable instructions to be executed by a computing device.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a host device having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g., a mouse, trackball, or a presence sensitive display or other surface by which the user can provide input to the computer. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

Claim 1:
An ambient computing system (<NUM>), the ambient computing system configured to monitor and respond to inputs from its environment while in a low-power state, the ambient computing system comprising:
a machine learning engine (<NUM>);
a low-power CPU (<NUM>); and
an external memory (<NUM>) that is shared among at least the machine learning engine (<NUM>) and the low-power CPU (<NUM>);
wherein the machine learning engine (<NUM>) comprises virtual address logic (<NUM>) to translate from virtual addresses generated by the machine learning engine (<NUM>) to physical addresses within the external memory (<NUM>);
wherein the ambient computing system (<NUM>) is configured to stream, into the external memory (<NUM>), parameters for a machine learning model from a second memory (<NUM>) that is separate from the ambient computing system;
and wherein the machine learning engine (<NUM>) is configured:
to receive one or more sensor signals, or a processed version of one or more sensor signals;
to perform an inference pass over the machine learning model by reading, from the external memory (<NUM>), the parameters of the machine learning model; and
to generate an output that represents which other processing components should further process the one or more sensor signals.