Distributed sensor data processing using multiple classifiers on multiple devices

According to an aspect, a method for distributed sound/image recognition using a wearable device includes receiving, via at least one sensor device, sensor data, and detecting, by a classifier of the wearable device, whether or not the sensor data includes an object of interest. The classifier configured to execute a first machine learning (ML) model. The method includes transmitting, via a wireless connection, the sensor data to a computing device in response to the object of interest being detected within the sensor data, where the sensor data is configured to be used by a second ML model on the computing device or a server computer for further sound/image classification.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a 35 U.S.C. § 371 National Phase Entry Application from PCT/US2020/055374, filed on Oct. 13, 2020, entitled “DISTRIBUTED SENSOR DATA PROCESSING USING MULTIPLE CLASSIFIERS ON MULTIPLE DEVICES”, the disclosure of which is incorporated herein by reference in its entirety.

FIELD

This disclosure relates to a distributed sensor data processing using multiple classifiers on multiple devices.

BACKGROUND

Computing devices (e.g., wearable devices, smartglasses, smart speakers, action cameras, etc.) are often relatively compact devices, and in some examples, may be on or around the body of a person for an extended period of time. However, computer processing requirements for processing sensor data (e.g., image data, audio data) can be relatively high especially for devices that include display and perception capabilities. For example, a device may perform energy-intensive operations (e.g., audio and/or image processing, computer vision, etc.) that requires a number of circuit components, which can cause several challenges. For example, the device may generate a relatively large amount of heat, thereby making the device uncomfortable to be in proximity to the skin for extended periods of time. In addition, the amount of circuit components (including batteries) adds weight to the device, thereby increasing the discomfort of wearing the device over an extended period of time. Further, the energy-intensive operations (in conjunction with the limitations on battery capacity) can cause the battery life to be relatively short. As such, some conventional devices can be used for only short durations throughout the day.

SUMMARY

This disclosure relates to a low-power device (e.g., smartglasses, wearable watches, portable action cameras, security cameras, smart speakers, etc.) that connects to a computing device (e.g., smartphone, laptop, tablet, etc.) over a wireless connection, where energy-intensive operations are offloaded to the computing device (or a server computer connected to the computing device), which can cause improvement to the device's performance (e.g., power, bandwidth, latency, computing capabilities, machine learning precision, etc.) and the user's experience. In some examples, the wireless connection is a short-range wireless connection such as a Bluetooth connection or near field communication (NFC) connection. In some examples, the low-power device includes a head-mounted display device such as smartglasses. However, the techniques discussed herein may be applied to other types of low-power devices such as portable action cameras, security cameras, smart doorbells, smart watches, etc.

According to an aspect, a method for distributed sound recognition using a wearable device includes receiving, via a microphone of the wearable device, audio data, detecting, by a sound classifier of the wearable device, whether or not the audio data includes a sound of interest, where the sound classifier executes a first machine learning (ML) model, and transmitting, via a wireless connection, the audio data to a computing device in response to the sound of interest being detected within the audio data, where the audio data is configured to be used by a second ML model for further sound classification.

According to an aspect, a non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to receive audio data via a microphone of a wearable device, detect, by a sound classifier of the wearable device, whether or not the audio data includes a sound of interest, where the sound classifier is configured to execute a first machine learning (ML) model, and transmit, via a wireless connection, the audio data to a computing device in response to the sound of interest being detected within the audio data, where the audio data is configured to be used by a second ML model on the computing device for further sound classification.

According to an aspect, a wearable device for distributed sound recognition includes a microphone configured to capture audio data, a sound classifier configured to detect whether or not the audio data includes a sound of interest, the sound classifier including a first machine learning (ML) model, and a radio frequency (RF) transceiver configured to transmit the audio data to a computing device via a wireless connection in response to the sound of interest being detected within the audio data, where the audio data is configured to be used by a second ML model to translate the sound of interest to text data.

According to an aspect, a computing device for sound recognition including at least one processor, and a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to receive, via a wireless connection, audio data from a wearable device, the audio data having a sound of interest detected by a sound classifier executing a first machine-learning (ML) model, determine whether to translate the sound of interest to text data using a sound recognition engine on the computing device, translate, by the sound recognition engine, the sound of interest to the text data in response to the determination to use the sound recognition engine on the computing device, the sound recognition engine configured to execute a second ML model, and transmit, via the wireless connection, the text data to the wearable device.

According to an aspect, a method for distributed image recognition using a wearable device includes receiving, via at least one imaging sensor of the wearable device, image data, detecting, by an image classifier of the wearable device, whether or not an object of interest is included within the image data, the image classifier executing a first machine-learning (ML) model, and transmitting, via a wireless connection, the image data to a computing device, the image data configured to be used by a second ML model on the computing device for further image classification.

According to an aspect, a non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to receive image data from one imaging sensor on a wearable device, detect, by an image classifier of the wearable device, whether or not an object of interest is included within the image data, the image classifier configured to execute a first machine-learning (ML) model, and transmit, via a wireless connection, the image data to a computing device, the image data configured to be used by a second ML model on the computing device to compute object location data, the object location data identifying a location of the object of interest in the image data.

According to an aspect, a wearable device for distributed image recognition includes at least one imaging sensor configured to capture image data, an image classifier configured to detect whether or not an object of interest is included within the image data, the image classifier configured to execute a first machine-learning (ML) model, and a radio frequency (RF) transceiver configured to transmit, via a wireless connection, the image data to a computing device, the image data configured to be used by a second ML model on the computing device to compute object location data, the object location data identifying a location of the object of interest in the image data.

According to an aspect, a computing device for distributed image recognition includes at least one processor, and a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to receive, via a wireless connection, image data from a wearable device, the image data having an object of interest detected by an image classifier executing a first machine-learning (ML) model, compute object location data based on the image data using a second ML model, the object location data identifying a location of the object of interest in the image data, and transmit, via the wireless connection, the object location data to the wearable device.

DETAILED DESCRIPTION

For sensor data captured by one or more sensors on the wearable device, the wearable device performs a portion of audio and/or image processing (e.g., the lower energy-intensive operation(s)) and the computing device (and/or the server computer and/or other multiple devices) performs other portion(s) of the audio and/or image processing (e.g., the higher energy-intensive operation(s)). For example, the wearable device can intelligently detect, using a relatively small machine-learning (ML) model, the presence of sensor data (e.g., whether audio data includes a sound of interest such as speech, music, an alarm, a hot-word for a voice command, etc. or whether image data includes an object of interest such as objects, text, bar codes, facial features, etc.), and, if so, can stream the sensor data to the computing device, over the wireless connection, to perform more complex audio and/or image processing using a relatively larger ML model. The results of the more complex audio and/or image processing can be provided back to the wearable device via the wireless connection, which can cause the wearable device to perform an action (including additional image/audio processing) and/or can cause the wearable device to render the results on the wearable device's display.

In some examples, this hybrid architecture may enable a compact form-factor with less circuit components in a wearable device such as a head-mounted display device (e.g., smartglasses). For example, since the system offloads more energy-intensive operations to the connected computing device(s) (and/or server computer), the wearable device may include less powerful/complex circuits. In some examples, the wearable device's architecture may enable a relatively compact printed circuit board within the frame of the eyeglasses, where the printed circuit board includes circuitry that is relatively low in power while still being able to execute wearable applications that are based on image processing and/or computer vision such as object classification, optical character recognition (OCR), and/or barcode decoding. As a result, battery life may be increased so that the user can use the wearable device over extended periods of time.

In some examples, sound recognition operations are distributed between the wearable device and the computing device (and potentially a server computer or other computing devices). For example, the wearable device includes a sound classifier (e.g., a small ML model) configured to detect whether or not a sound of interest (e.g., speech, music, alarm, etc.) is included within the audio data captured by a microphone on the wearable device. If not, the sound classifier continues to monitor the audio data to determine if the sound of interest is detected. If so, the wearable device can stream the audio data (e.g., raw sound, compressed sound, sound snippet, extracted features, and/or audio parameters, etc.) to the computing device over the wireless connection. The sound classifier may save power and latency through its relatively small ML model. The computing device includes a more powerful sound recognition engine (e.g., a more powerful classifier) that executes a larger ML model to translate (or convert) the audio data to text data (or other forms of data), where the computing device transmits the text data back to the wearable device via the wireless connection to be displayed on the wearable device's display and/or auditorily read back to the user. In some examples, the computing device is connected to a server computer over a network (e.g., the Internet), and the computing device transmits the audio data to the server computer, where the server computer executes a larger ML model to translate the audio data to text data (e.g., in cases of translating into a different language). Then, the text data is routed back to the computing device and then to the wearable device for display.

In some examples, image recognition operations are distributed between the wearable device and the computing device. In some examples, the image recognition operations include facial detection and tracking. However, the image recognition operations may include operations to detect (and track) other regions of interest in image data such as objects, barcodes, and/or text. The wearable device includes an image classifier (e.g., a small ML model) configured to detect whether or not an object of interest (e.g., facial features, text, OCR code, etc.) are included within the image data captured by one or more imaging sensors on the wearable device. If so, the wearable device may transmit an image frame (that includes the object of interest) to the computing device over the wireless connection. The computing device includes a more powerful object detector (e.g., a more power classifier) that executes a larger ML model to calculate object location data (e.g., bounding box dataset) that identifies a location of the detected object of interest, where the computing device transmits the object location data back to the wearable device. The wearable device uses one or more low-complexity tracking mechanisms (e.g., inertial measurement unit (IMU)-based warping, blob detection, optical flow, etc.) to propagate the object location data for subsequent image frames captured on the wearable device. The wearable device may compress and send the cropped regions to the computing device, where the object detector on the computing device may perform object detection on the cropped regions and send updated object location data back to the wearable device.

In some examples, perception operations with multi-resolutions are distributed between the wearable device and the computing device. Perception operations may include always-on sensing and sensing a voice-input request (e.g., hot-word detection). For example, the wearable device may include a low-power/low-resolution (LPLR) camera and a high-power/high-resolution (HPHR) camera. In some examples, the wearable device may include the image classifier that executes a small ML model to detect objects of interest (e.g., faces, text, barcodes, buildings, etc.) from image data captured by the LPLR camera. If an object of interest is detected, the HPHR camera may be triggered to capture one or more image frames with a higher quality (e.g., higher resolution, less noise, etc.). Higher quality images may be required for some applications.

Then, the image frame(s) from the HPHR camera may be transmitted to the computing device over the wireless connection, where the computing device executes a larger ML model to perform more complex image recognition operations on the image frame(s) with the higher quality. In some examples, the operations may be similar to the object detection example described above, where object location data (e.g., bounding box dataset) is computed and sent to the wearable device, and the wearable device uses one or more tracking mechanisms to propagate the object location data to subsequent frames, and then the wearable device crops and compresses image regions to be sent back to the computing device for further processing. In some examples, a stream of images of a product can be used to capture label text or barcodes and look up associated product information (e.g., price, shopping suggestions, comparable products, etc.). This information can be shown on a display surface present on the wearable device or read back to the user auditorily.

In terms of sensing a voice-input request, the wearable device may include a voice command detector that executes a small ML model (e.g., a gatekeeping model) to continuously (e.g., periodically) process microphone samples for an initial portion of a hot-word (e.g., “ok G” or “ok D”). If the voice command detector detects that initial portion, the voice command detector may cause a buffer to capture the subsequent audio data. Also, the wearable device may transmit a portion of the buffer (e.g., 1-2 seconds of audio from the head of the buffer) to the computing device over the wireless connection, where the computing device includes a hot-word recognition engine having a larger ML model to perform the full hot-word recognition. If the utterance is a false positive, the computing device may transmit a disarm command to the wearable device, which discards the contents of the buffer. If the utterance is a true positive, the rest of the audio buffer is transmitted to the computing device for automatic speech recognition and user-bound response generation.

The systems and techniques described herein may reduce the wearable device's power consumption, increase the battery life, decrease the amount of heat generated by the wearable device, and/or decrease the amount of circuit components within the wearable device (which can cause the weight to be decreased), which may cause the wearable device to be used for extended periods of time. In some examples, in terms of power, the systems and techniques described herein can extend the wearable device's battery life to an extended period of time (e.g., five to fifteen hours, or more than fifteen hours). In contrast, some conventional smartglasses and other image/audio processing products may have only a few hours of usage.

In some examples, in terms of bandwidth, the systems and techniques described herein may distribute computation operations (e.g., inference operations) across the wireless connection using gatekeeping models (e.g., small classifiers, binary classifiers, etc.) to limit unnecessary transmission, which can reduce the latency and reduce power usage. In some examples, in terms of latency, the systems and techniques described herein may enable the use of inference both near the wearable device's sensors and across the components of the computing device (and potentially the server computer), which can provide flexibility to tune performance to meet the requirements of various applications. The ML decisions can occur dynamically as application use and power (e.g., remaining battery life) or computing requirements change under use. In some examples, in terms of computing capabilities, the systems and techniques described herein may provide a flexible use of computing resources to meet application requirements.

FIG.1illustrates a system100for distributing image and/or audio processing on sensor data128across multiple devices including a device102, a computing device152, and/or a server computer160. In some examples, the sensor data128is real-time sensor data or near real-time sensor data (e.g., data collected from one or more sensors138in real-time or near real-time). In some examples, image and/or audio processing on sensor data128may be distributed among the device102and the computing device152. In some examples, the image and/or audio processing on sensor data128may be distributed among any two or more of the device102, the computing device152, or the server computer160(or any combination thereof). In some examples, the system100includes multiple devices102and/or multiple computing devices152, where each device executes a classifier that renders a decision on if and what data to relay to the next classifier, which may be on the same device or a different device.

The device102is configured to be connected to the computing device152via a wireless connection148. In some examples, the wireless connection148is a short-range communication link such as near-field communication (NFC) connection or Bluetooth connection. The device102and the computing device152may exchange information via the wireless connection148. In some examples, the wireless connection148defines an application-layer protocol that is implemented using protocol buffers with message types for drawing graphic primitives, configuring sensors138and peripherals, and changing device modes. In some examples, the application-layer protocol defines another set of message types that can transmit sensor data128and remote procedure call (RPC) return values back to the computing device152.

The computing device152may be coupled to the server computer160over a network150. The server computer160may be computing devices that take the form of a number of different devices, for example a standard server, a group of such servers, or a rack server system. In some examples, the server computer160is a single system sharing components such as processors and memories. The network150may include the Internet and/or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, satellite network, or other types of data networks. The network150may also include any number of computing devices (e.g., computer, servers, routers, network switches, etc.) that are configured to receive and/or transmit data within network150. In some examples, the device102is also configured to be connected to the server computer160over the network150.

With respect to audio and/or image processing for sensor data128captured by one or more sensors138on the device102in real-time or near real-time, a portion of audio and/or image processing (e.g., the lower energy-intensive operation(s)) is performed at the device102, and other portion(s) of the audio and/or image processing (e.g., the higher energy-intensive operation(s)) are performed at the computing device152(and/or the server computer160). In some examples, another portion of the audio and/or image processing is performed at another device. In some examples, another portion of the audio and/or image processing is performed at yet another device, and so forth. In some examples, the sensor data128includes audio data131. In some examples, the sensor data128includes image data129. In some examples, the sensor data128includes audio data131and image data129.

The device102can intelligently detect the presence of certain types of data within the sensor data128captured by the sensor(s)138. In some examples, the device102can detect whether audio data131captured by a microphone140includes a sound of interest such as speech, music, alarm, or at least a portion of a hot-word for command detection, etc. In some examples, the device102can detect whether image data129includes an object of interest (e.g., objects, text, barcodes, facial features, etc.). If the device102detects the relevant data within the sensor data128, the device102can stream the sensor data128to the computing device152, over the wireless connection148, to perform more complex audio and/or image processing. In some examples, the device102can stream the image data129to the computing device152. In some examples, the device102can stream the audio data131to the computing device152. In some examples, the device102can stream both the audio data131and the image data129to the computing device152.

In some examples, the device102compresses the audio data131and/or the image data129before transmission to the computing device152. In some examples, the device102extracts features from the sensor data128and sends the extracted features to the computing device152. In some examples, the device102extracts features from the sensor data128and sends the extracted features to the computing device152. For example, the extracted features may include sound intensity, computed angle-of-arrival (e.g., what direction the sound came from), and/or the type of the sound (e.g., speech, music, alarm, etc.). In some examples, the extracted features may include compressed encoding which can save transmission bandwidth for a particular type of sound. The results of the more complex audio and/or image processing performed at the computing device152can be provided back to the device102via the wireless connection148to cause the device102to perform an action (including further audio and/or image processing), cause the device102to render the results on a display116of the device102, and/or cause the device102to provide the results auditorily.

In some examples, the device102is a display device capable of being worn on or in proximity to the skin of a person. In some examples, the device102is a wearable device. In some examples, the device102is a head mounted display (HMD) device such as an optical head-mounted display (OHMD) device, a transparent heads-up display (HUD) device, an augmented reality (AR) device, or other devices such as googles or headsets having sensors, display, and computing capabilities. In some examples, the device102is smartglasses. Smartglasses is an optical head-mounted display designed in the shape of a pair of eyeglasses. For example, smartglasses are glasses that add information (e.g., projects a display116) alongside what the wearer views through the glasses. In some examples, superimposing information (e.g., digital images) onto a field of view may be achieved through smart optics. Smartglasses are effectively wearable computers which can run self-contained mobile apps (e.g., the applications112). In some examples, smartglasses may be hands-free and can communicate with the Internet via natural language voice commands, while others use touch buttons. In some examples, the device102may include any type of low-power device. In some examples, the device102includes a security camera. In some examples, the device102includes an action camera. In some examples, the device102includes a smart watch. In some examples, the device102includes a smart doorbell. As indicated above, the system100may include multiple devices102(e.g., a smart watch, smartglasses, etc.), where each device102is configured to execute a classifier that can perform image/audio processing, and then route data to the next classifier in the network of classifiers.

The device102may include one or more processors104, which may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. In some examples, the processor(s)104are included as part of a system on chip (SOC). The processor(s)104can be semiconductor-based—that is, the processors can include semiconductor material that can perform digital logic. The processor(s)104includes a microcontroller106. In some examples, the microcontroller106is a subsystem within the SOC and can include a process, memory, and input/output peripherals. In some examples, the microcontroller106is a dedicated hardware processor that executes a classifier. The device102may include a power management unit (PMU)108. In some examples, the PMU108is integrated with or included within the SOC. The microcontroller106is configured to execute a machine-learning (ML) model126to perform an inference operation124-1related to audio and/or image processing using sensor data128. As further discussed below, the relatively small size of the ML model126can save power and latency. In some examples, the device102includes multiple microcontrollers106and multiple ML models126that perform multiple inference operations124-1, which can communicate with each other and/or other devices (e.g., computing device(s)152and/or server computer160).

The device102includes one or more memory devices110. In some examples, the memory devices110include flash memory. In some examples, the memory devices110may include a main memory that stores information in a format that can be read and/or executed by the processor(s)104including the microcontroller106. The memory devices110may store weights109(e.g., inference weights, or model weights) for the ML model126that is executed by the microcontroller106. In some examples, the memory devices110may store other assets such as fonts and images.

In some examples, the device102includes one or more applications112, which can be stored in the memory devices110, and that, when executed by the processor(s)104, perform certain operations. The applications112may widely vary depending on the use case, but may include browser applications to search web content, sound recognition applications such as speech-to-text applications, image recognition applications (including object and/or facial detection (and tracking) applications, barcode decoding applications, text OCR applications, etc.), and/or other applications that can enable the device102to perform certain functions (e.g., capture an image, record a video, get directions, send a message, etc). In some examples, the applications112include an email application, a calendar application, a storage application, a voice call application, and/or a messaging application.

The device102includes a display116, which is a user interface that displays information. In some examples, the display116is projected onto the field of view of the user. In some examples, the display116is a built-in lens display. The display116may include a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting display (OLED), an electro-phoretic display (EPD), or a micro-projection display adopting an LED light source. In some examples, the display116may provide a transparent or semi-transparent display such that the user wearing the glasses can see images provided by the display116but also information located in the field of view of the smartglasses behind the projected images. In some examples, the device102includes a touch pad117that allows the user to control the device102(e.g., which can allow swiping through an interface displayed on the display116). The device102includes a battery120configured to provide power to the circuit components, one or more radio frequency (RF) transceivers114to enable communication with the computing device152via the wireless connection148and/or the server computer160via the network150, a battery charger122configured to control the charging of the battery120, and one or more display regulators118that controls information displayed by the display116.

The device102includes a plurality of sensors138such as a microphone140configured to capture audio data131, one or more imaging sensors142configured to capture image data, a lighting condition sensor144configured to obtain lighting condition information, and/or a motion sensor146configured to obtain motion information. The microphone140is a transducer device that converts sound into an electrical signal, which is represented by the audio data131. The light condition sensor144may detect the amount of light exposure. In some examples, the lighting condition sensor144includes an ambient light sensor that detects the amount of ambient light that is present, which can be used to ensure that image data129is captured with a desired signal-to-noise ratio (SNR). However, the lighting condition sensor144may include other types of photometric (or colorimeter) sensors. The motion sensor146may obtain motion information, which may include blur estimation information. The motion sensor146can be used for monitoring device movement such as tilt, shake, rotation, and/or swing and/or for determining blur estimation.

The imaging sensors142are sensors (e.g., cameras) that detect and convey information used to make an image, which is represented by the image data129. The imaging sensors142can take pictures and record video. In some examples, the device102includes a single imaging sensor142. In some examples, the device102includes multiple imaging sensors142. In some examples, the imaging sensors142include an imaging sensor142aand an imaging sensor142b. The imaging sensor142amay be considered a low power, low resolution (LPLR) image sensor. The imaging sensor142bmay be considered a high power, high resolution (HPHR) image sensor. An image captured by imaging sensor142bhas a higher quality (e.g., higher resolution, lower noise) than an image captured by imaging sensor142a. In some examples, the device102includes more than two imaging sensors142.

In some examples, the imaging sensor142ais configured to obtain image data129while the device102is activated (e.g., continuously or periodically captures image data129while the device102is activated). In some examples, the imaging sensor142ais configured to operate as an always-on sensor. In some examples, the imaging sensor142bis activated (e.g., for a short duration) in response to the detection of an object of interest, as further discussed below.

The computing device152may be any type of computing device capable of being wirelessly connected to the device102. In some examples, the computing device152is a mobile computing device. In some examples, the computing device152is a smartphone, a tablet, or a laptop computer. In some examples, the computing device152is a wearable device. The computing device152may include one or more processors154formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processors154can be semiconductor-based—that is, the processors can include semiconductor material that can perform digital logic.

The computing device152may include one or more memory devices156. The memory devices156may include a main memory that stores information in a format that can be read and/or executed by the processors154. The operating system155is a system software that manages computer hardware, software resources, and provides common services for computing programs. Although not shown inFIG.1, the computing device152can include a display (e.g., a touchscreen display, an LED display, etc.) that can display a user interface for an application158that is being executed by the computing device152. The applications158may include any type of computer program executable by the operating system155. The applications158may include mobile applications, e.g., software programs that are developed for a mobile platform or mobile device.

In some examples, the audio and/or image processing that is performed on the sensor data128obtained by the sensor(s)138are referred to as inference operations (or ML inference operations). An inference operation (e.g., inference operation124-1or inference operation124-2) may refer to an audio and/or image processing operation, step, or sub-step that involves a ML model that makes (or leads to) one or more predictions. Certain types of audio and/or image processing use ML models to make predictions. For example, machine learning may use statistical algorithms that learn data from existing data, in order to render a decision about new data, which is a process called inference. In other words, inference refers to the process of taking a model that is already trained and using that trained model to make predictions. Some examples of inference may include sound recognition (e.g., speech-to-text recognition), image recognition (e.g., facial recognition and tracking, etc.), and/or perception (e.g., always-on sensing, voice-input request sensing, etc.).

In some examples, a ML model includes one or more neural networks. Neural networks transform an input, received by the input layer, transform it through a series of hidden layers, and produce an output via the output layer. Each layer is made up of a subset of the set of nodes. The nodes in hidden layers are fully connected to all nodes in the previous layer and provide their output to all nodes in the next layer. The nodes in a single layer function independently of each other (i.e., do not share connections). Nodes in the output provide the transformed input to the requesting process. In some examples, the neural network is a convolutional neural network, which is a neural network that is not fully connected. Convolutional neural networks therefore have less complexity than fully connected neural networks. Convolutional neural networks can also make use of pooling or max-pooling to reduce the dimensionality (and hence complexity) of the data that flows through the neural network and thus this can reduce the level of computation required. This makes computation of the output in a convolutional neural network faster than in neural networks.

With respect to a particular inference type, the device102may perform one or more parts of the inference to intelligently detect the presence of sensor data128(e.g., whether audio data131includes a sound of interest such as speech, an alarm, or at least a portion of a hotword, and/or whether image data129includes an object of interest (e.g., facial features, text, objects, bar codes, etc.)) and, if so, then transmits the sensor data128, over the wireless connection148, to the computing device152, where computing device152performs one or more other parts of the ML inference (e.g., the more complex parts of audio and/or image processing) using the sensor data128. In other words, the inference operations may be distributed among the device102and the computing device152(and potentially the server computer160) so that the energy-intensive operations are performed at the more powerful computing device (e.g., the computing device152or the server computer160) as opposed to the relatively small computing device (e.g., the device102).

In some examples, the system100may include other devices (e.g., besides the device102, the computing device152, and the server computer160), where one or more of these other devices may execute one or more classifiers (where each classifier executes a ML model related to object/sound recognition). For example, the system100may have one or more classifiers on the device102, one or more wearable devices (e.g., one or more devices102) and/or one or more classifiers on the computing device152. Further, the data may be sent to the server computer160for server-side processing—which may have additional classification steps. As such, in some examples, the system100may include a network of classifiers that analyze audio/camera streams and make decisions on if and what to relay to the next node (or classifier).

In some examples, the microcontroller106of the device102may execute an inference operation124-1using sensor data128(e.g., audio data131from the microphone140and/or image data129from one or more of the imaging sensors142) and the ML model126stored on the device102. In some examples, the ML model126may receive the sensor data128as an input, and detect whether or not the sensor data128has a classification in which the ML model126is trained to classify (e.g., whether audio data131includes a sound of interest or whether the image data129includes an object of interest). In some examples, the ML model126is a sound classifier that can evaluate incoming sound for specific criteria (e.g., frequency, amplitude, feature detection, etc.). In some examples, the analyzed criteria determines if audio data (e.g., raw sound, compressed sound, sound snippet, audio parameters, etc.) should be sent to other device(s) (including computing device152, server computer160, etc.), which does further classification.

In some examples, the ML model126is a speech classifier (e.g., a binary speech classifier) that detects whether the audio data131includes speech or does not include speech. In some examples, the ML model126is an image object classifier (detector) that detects whether the image data129includes an object of interest or does not include an object of interest. In some examples, the ML model126is an object classifier that detects whether the image data129includes facial features or does not include facial features. In some examples, the ML model126is a classifier that determines whether the audio data131includes at least a portion of a hot-word for a voice command.

If the output of the ML model126indicates that the classification has been detected, the RF transceiver114of the device102may transmit the sensor data128to the computing device152via the wireless connection148. In some examples, the device102may compress the sensor data128, and then transmit the compressed sensor data128to the computing device152. Then, the computing device152is configured to execute an inference operation124-2using the sensor data128(received from the device102) and the ML model127stored on the computing device152. In some examples, in terms of sound recognition (e.g., speech-to-text processing), the ML model127is used to convert audio data131to text, where the results are transmitted back to the device102. In some examples, in terms of hot-word command recognition, the ML model127is used to perform full hot-word command recognition on the audio data131received from the device102. In some examples, in terms of image processing, the ML model127is used to compute object location data (identifying a location of the object of interest in the image data), where the results are transmitted back to the device102for further image processing, which is further described later in the specification.

However, generally, the inference operation124-2may refer to an audio and/or image processing operation that involves a ML model that is different from the inference operation124-1. In some examples, the inference operations include sound recognition operations, where the inference operation124-1refers to a first sound recognition operation that is executed using the ML model126, and the inference operation124-2refers to a second sound recognition that is executed using the ML model127. In some examples, the inference operations include image recognition operations, where the inference operation124-1refers to a first image recognition operation that is executed using the ML model126, and the inference operation124-2refers to a second image recognition operation that is executed using the ML model127. In some examples, the inference operations include perception sensing operations (e.g., always-on sensing, voice command sensing (e.g., hotword recognition), etc.), where the inference operation124-1refers to a first perception sensing operation that is executed using the ML model126, and the inference operation124-2refers to a second perception sensing operation that is executed using the ML model127.

The ML model126may have a size less than (e.g., substantially less than) a size of the ML model127. In some examples, the ML model126may be required to perform less computational operations to make a prediction as compared to the ML model127. In some examples, the size of a particular ML model may be represented by the number of parameters required for that model to make a prediction. A parameter is a configuration variable that is internal to the ML model and whose value can be estimated from the given data. The ML model126may include parameters111. For example, the ML model126may define a number of parameters111that are required for the ML model126to make a prediction. The ML model127includes parameters113. For example, the ML model127may define a number of parameters113that are required for the ML model127to make a prediction. The number of parameters111may be less than (e.g., substantially less than) the number of parameters113. In some examples, the number of parameters113is at least ten times greater than the number of parameters111. In some examples, the number of parameters113is at least one hundred times greater than the number of parameters111. In some examples, the number of parameters113is at least one thousand times greater than the number of parameters111. In some examples, the number of parameters113is at least one million times greater than the number of parameters111. In some examples, the number of parameters111is in a range between 10 k and 100 k. In some examples, the number of parameters111is less than 10 k. In some examples, the number of parameters113is in a range between 1 M and 10 M. In some examples, the number of parameters113is greater than 10 M.

In some examples, sound recognition operations (e.g., speech, alarm, or generally any type of sound) are distributed between the device102and the computing device152. In some examples, the sound recognition operations are distributed between the device102and the computing device152. For example, the microcontroller106is configured to execute an inference operation124-1by invoking the ML model126to detect whether or not a sound of interest is included within audio data131captured by the microphone140on the device102. The ML model126may be a classifier that classifies the audio data131as containing the sound of interest or not containing the sound of interest. For example, the ML model126receives the audio data131from the microphone140and computes a prediction on whether the audio data131includes the sound of interest. If the sound of interest is not detected within the audio data131by the ML model126, the ML model126continues to receive the audio data131from the microphone140as an input to compute a prediction on whether the sound of interest is detected within the audio data131. If the sound of interest is detected within the audio data131by the ML model126, the device102streams the audio data131(e.g., raw sound, compressed sound, sound snippet, and/or audio parameters, etc.) to the computing device152over the wireless connection148. In some examples, the device102compresses the audio data131, and then transmits the compressed audio data131to the computing device152over the wireless connection148.

The computing device152receives the audio data131over the wireless connection148from the device102and executes an inference operation124-2by invoking the ML model127. The ML model127may save power and latency through its relatively small ML model. The computing device152includes a more powerful sound recognition engine (e.g., another type of classifier) that executes a ML model127(e.g., a larger ML model) to convert the audio data131(potentially to text data), where the computing device152transmits the text data back to the device102via the wireless connection148to be displayed on the device's display. In some examples, the computing device152is connected to a server computer160over a network150(e.g., the Internet), and the computing device152transmits the audio data131to the server computer160, where the server computer160executes a larger ML model to convert the audio data131to text data (e.g., in cases of translating into a different language). Then, the text data is routed back to the computing device152and then to the device102for display.

In some examples, image recognition operations are distributed between the device102and the computing device152. In some examples, the image recognition operations include facial detection and tracking. However, the image recognition operations may include operations to detect (and track) other regions of interest in image data such as objects, text, and barcodes. The microcontroller106is configured to execute an inference operation124-1by invoking the ML model126to detect whether or not an object of interest is included within image data129captured by one or more imaging sensors142on the device102. If so, the device102may transmit an image frame (that includes the object of interest) to the computing device152over the wireless connection148. In some examples, device102compresses the image frame, and then transmits the compressed image frame to the computing device152over the wireless connection148.

The computing device152is configured to execute an inference operation124-2by invoking the ML model127to perform a more complex image processing operation using the image data129such as calculating object location data (e.g., a bounding box dataset) identifying a location of the object of interest, where the computing device152transmits the object location data back to the device102. The device102uses one or more low-complexity tracking mechanisms (e.g., IMU-based warping, blob detection, optical flow, etc.) to propagate the object location data for subsequent image frames captured on the device102. The device102may compress and send the cropped regions to the computing device152, where the computing device152may perform image classification on the cropped regions and send updated object location data back to the device102.

In some examples, perception operations with multi-resolutions are distributed between the device102and the computing device152. Perception operations may include always-on sensing and sensing a voice-input request (e.g., hot-word detection). In some examples, the imaging sensor142a(e.g., the LPLR camera) is activated when the user is wearing the device102in order to capture image data129with relatively low resolution to search for regions of interest. For example, the microcontroller106is configured to perform an inference operation124-1by invoking the ML model126(using the image data129as an input to the ML model126) to detect objects of interest (e.g., faces, text, barcodes, buildings, etc.). If an object of interest is detected, the imaging sensor142bmay be activated to capture one or more image frames having a higher resolution.

Then, the image data129with the higher resolution may be transmitted to the computing device152over the wireless connection148. In some examples, the device102compresses the image data129with the higher resolution and transmits the compressed image data129over the wireless connection148. The computing device152is configured to execute an inference operation124-2by invoking the ML model127(inputted with the image data129having the higher resolution) to perform image recognition. In some examples, the operations may be similar to the face detection example described above, where the object location data (e.g., bounding box dataset) is computed by the computing device152and sent to the device102, and the device102uses one or more tracking mechanisms to propagate the object location data to subsequent frames, and then the device102crops and compresses image regions to be sent back to the computing device152for further image classification. In some examples, a stream of images of a product can be used to capture label text or barcodes and look up associated product information (e.g., price, shopping suggestions, comparable products, etc.). This information can be shown on a display116on the device102or read back to the user auditorily.

In terms of sensing a voice-input request, the microcontroller106is configured to execute an inference operation124-1by invoking the ML model126to continuously (e.g., periodically) process microphone samples (e.g., audio data131) for an initial portion of a hot-word (e.g., “ok G” or “ok D”). If the ML model126detects that the initial portion, the microcontroller106may cause a buffer to capture the subsequent audio data131. Also, the device102may transmit a portion of the buffer (e.g., 1-2 seconds of audio from the head of the buffer) to the computing device152over the wireless connection148. In some examples, the portion of the buffer is compressed before transmitting to the computing device152. The computing device152is configured to execute an inference operation124-2by invoking the ML model127to perform the full hot-word recognition using the audio data131. If the utterance is a false positive, the computing device152may transmit a disarm command to the device102, which discards the contents of the buffer. If the utterance is a true positive, the rest of the audio buffer is compressed and transmitted to the computing device152for automatic speech recognition and user-bound response generation.

In some examples, in order to increase the transmission efficiency, the device102may buffer multiple data packets134and transmit the data packets134as a single transmission event132to the computing device152over the wireless connection148. For example, each transmission event132may correlate with a power consumption that causes power to be dissipated from the battery120. In some examples, the device102determines the type of information to be transmitted to the computing device152. In some examples, if the type of information to be transmitted to the computing device152relates to latency-dependent information (e.g., audio streaming), the device102may not buffer the audio data131but rather stream the audio data131without delay. In some examples, if the information to be transmitted is not latency-dependent information, the device102may store the information as one or more data packets134in a buffer130and transmit the information to the computing device152at a later time. The buffer130may be a portion of the memory device(s)110. In some examples, other non-latency-dependent information may be combined with the existing data in the buffer130, and the information contained in the buffer130may be transmitted to the computing device152as a single transmission event132.

For example, the buffer130may include a data packet136aand a data packet136b. The data packet136amay include information obtained at a first time instance, and the data packet136bmay include information obtained at a second time instance, where the second time instance is after the first time instance. However, instead of transmitting the data packet136aand the data packet136bas different transmission events132, the device102may store the data packet136aand the data packet136bin the buffer130and transmit the data packet136aand the data packet136bas a single transmission event132. In this manner, the number of transmission events132may be reduced, which may increase the energy efficiency of communicating information to the computing device152.

FIG.2illustrates a system200for distributing image and/or audio processing across multiple devices including a device202, a computing device252, and a server computer260. The system200may be an example of the system100ofFIG.1and may include any of the details disclosed with reference to those figures. The device202is connected to the computing device252over a wireless connection248. In some examples, the device202is a head-mounted display device such as smartglasses. However, the device202may be other types of low-power devices as discussed herein. The computing device252is connected to the server computer260over a network250. InFIG.2, the device202obtains sensor data228from one or more sensors238on the device202. The sensor data228may include at least one of image data or audio data. The device202(e.g., the microcontroller106ofFIG.1) may execute an inference operation224-1by invoking a ML model226to perform image and/or audio processing on the sensor data228to detect whether the sensor data228includes a type of data in which the ML model226is trained. In some examples, the device202may include multiple classifiers (e.g., multiple microcontrollers106), where each classifier may render a decision to send the sensor data228(or the results of the decision) to another classifier, which may be on the device202or another device such as the computing device252.

If the type of data in which the ML model226is trained is detected, the device202may transmit the sensor data228to the computing device252over the wireless connection248. Then, the computing device252may transmit the sensor data228, over the network250, to the server computer260. In some examples, the computing device252may include one or more classifiers that process audio/image data captured by the sensor(s)238to render decision(s) on whether to invoke another classifier on the computing device252, the device202, or the server computer260. The server computer260includes one or more processors262, which may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s)262can be semiconductor-based—that is, the processors can include semiconductor material that can perform digital logic. The server computer260includes one or more memory devices264. The memory devices264may include a main memory that stores information in a format that can be read and/or executed by the processors262.

The server computer260is configured to execute an inference operation224-2using the sensor data228and a ML model229stored on the server computer260. The inference operation224-1and the inference operation224-2relate to different audio and/or image processing operations. In some examples, the inference operation224-1and the inference operation224-2relate to different audio processing operations. In some examples, the inference operation224-1and the inference operation224-2relate to different image recognition operations. In some examples, the inference operation224-1and the inference operation224-2relate to different perception operations.

The ML model226may have a size less than (e.g., substantially less than) a size of the ML model229. The ML model226may define a number of parameters211that are required for the ML model226to make a prediction. The ML model229may define a number of parameters215that are required for the ML model229to make a prediction. The number of parameters211is less than (e.g., substantially less than) the number of parameters215. In some examples, the number of parameters215is at least one thousand times greater than the number of parameters211. In some examples, the number of parameters215is at least one million times greater than the number of parameters211. In some examples, the number of parameters211is in a range between 10 k and 100 k. In some examples, the number of parameters211is less than 10 k. In some examples, the number of parameters215is in a range between 10 M and 100 M. In some examples, the number of parameters215is greater than 100 M.

FIG.3illustrates a system300for distributing image and/or audio processing across multiple devices including a device302, a computing device352, and a server computer360. The system300may be an example of the system100ofFIG.1and/or the system200ofFIG.2and may include any of the details disclosed with reference to those figures. The device302is connected to the computing device352over a wireless connection348. In some examples, the device302is a head-mounted display device such as smartglasses. However, the device302may be other types of low-power devices as discussed herein. The computing device352is connected to the server computer360over a network350. InFIG.3, the device302obtains sensor data328from one or more sensors338on the device302. The sensor data328may include at least one of image data or audio data. The device302(e.g., the microcontroller106ofFIG.1) may execute an inference operation324-1by invoking a ML model326to perform image and/or audio processing on the sensor data328to detect whether the sensor data328includes a type of data in which the ML model326is trained.

If the type of data in which the ML model326is trained is detected, the device302may transmit the sensor data328to the computing device352over the wireless connection348. The computing device352is configured to execute an inference operation324-2using the sensor data328and a ML model327stored on the computing device352. Then, the computing device352may transmit the results of the inference operation324-2and/or the sensor data328, over the network350, to the server computer360.

The server computer360is configured to execute an inference operation324-3using the results of the inference operation324-2and/or the sensor data328and a ML model329stored on the server computer360. The inference operation324-1, the inference operation324-2, and the inference operation324-3relate to different audio and/or image processing operations. In some examples, the inference operation324-1, the inference operation324-2, the inference operation324-3relate to different audio processing operations. In some examples, the inference operation324-1, the inference operation324-2, the inference operation324-3relate to different image recognition operations. In some examples, the inference operation324-1, the inference operation324-2, and the inference operation324-3relate to different perception operations.

The ML model326may have a size less than (e.g., substantially less than) a size of the ML model327. The ML model327may have a size less than (e.g., substantially less than) a size of the ML model329. The ML model326may define a number of parameters311that are required for the ML model326to make a prediction. The ML model327may define a number of parameters313that are required for the ML model327to make a prediction. The ML model329may define a number of parameters315that are required for the ML model329to make a prediction. The number of parameters311is less than (e.g., substantially less than) the number of parameters313. The number of parameters313is less than (e.g., substantially less than) the number of parameters315. In some examples, the number of parameters311is in a range between 10 k and 100 k. In some examples, the number of parameters311is less than 10 k. In some examples, the number of parameters313is in a range between 100K and 1 M. In some examples, the number of parameters313is greater than 1 M. In some examples, the number of parameters315is in a range between 10 M and 100 M. In some examples, the number of parameters315is greater than 100 M.

FIG.4illustrates an example of a head-mounted display device402according to an aspect. The head-mounted display device402may be an example of the device102ofFIG.1, the device202ofFIG.2, and/or the device302ofFIG.3. The head-mounted display device402includes smartglasses469. Smartglasses469are glasses that add information (e.g., project a display416) alongside what the wearer views through the glasses. In some examples, instead of projecting information, the display416is an in-lens micro display. Smartglasses469(e.g., eyeglasses or spectacles), are vision aids, including lenses472(e.g., glass or hard plastic lenses) mounted in a frame471that holds them in front of a person's eyes, typically utilizing a bridge473over the nose, and legs474(e.g., temples or temple pieces) which rest over the ears. The smartglasses469include an electronics component470that includes circuitry of the smartglasses469. In some examples, the electronics component470includes a housing that encloses the components of the device102ofFIG.1, the device202ofFIG.2, and/or the device302ofFIG.3. In some examples, the electronics component470is included or integrated into one (or both) of the legs474of the smartglasses469.

FIG.5illustrates an example of an electronics component570of a pair of smartglasses according to an example. The electronics component570may be an example of the electronics component470ofFIG.4. The smartglasses' electronics component570may include display regulators518, a display516, a flash memory510, an RF transceiver514, a universal serial bus (USB) interface521, a power management unit (PMU)508, a system on chip (SOC)504, a battery charger522, a battery520, a plurality of user controls581, and a user light emitting diode (LED)585. The display regulators518, the display516, the RF transceiver514, the battery charger522, and the battery520may be an example of the display regulators118, the display116, the RF transceiver114, the battery charger122, and the battery120ofFIG.1. The SOC504may include the processor(s)104(including the microcontroller106) ofFIG.1. The flash memory510may be an example of the memory device110ofFIG.1. The flash memory510may store the weights for any ML models executable by the SOC504.

The SOC504may provide the data and control information to the display516that is projected in the field of view of the user. In some examples, the PMU508is included within or integrated with the SOC504. The display regulators518are connected to the PMU508. The display regulators518may include a first converter576(e.g., a VDDD DC-DC converter), a second converter579(e.g., a VDDA DC-DC converter), and a LED driver580. The first converter576is configured to activate in response to an enable signal, and the second converter579is configured to activate in response to an enable signal. The LED driver580is configured to be driven according to a pulse width modulation (PWM) control signal. The plurality of user controls581may include a reset button582, a power button583, a first user button584-1, and a second user button584-2.

FIG.6illustrates a printed circuit board (PCB) substrate668for smartglasses according to an aspect. The PCB substrate668may be an example of and/or included within the electronics component470ofFIG.4and/or the electronics component570ofFIG.5. The PCB substrate668includes a plurality of circuit components. In some examples, the circuit components are coupled on one side of the PCB substrate668. In some examples, the circuit components are coupled on both sides of the PCB substrate668. The PCB substrate668may include a battery charger622, an SOC604, a display flex669, display regulators618, and a flash memory610. The PCB substrate668may be relatively compact. For example, the PCB substrate668may define a length (L) and a width (W). In some examples, the length (L) is in a range of 40 mm to 80 mm. In some examples, the length (L) is in a range of 50 mm to 70 mm. In some examples, the length (L) is 60 mm. In some examples, the width (W) is in a range of 8 mm to 25 mm. In some examples, the width (W) is in a range of 10 mm to 20 mm. In some examples, the width (W) is 14.5 mm.

FIGS.7A and7Billustrate a system700for distributing sound recognition operations between a device702and a computing device752. The system700may be an example of the system100ofFIG.1, the system200ofFIG.2, and/or the system300ofFIG.3and may include any of the details discussed with reference to those figures. In some examples, the device702may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device702may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6.

As shown inFIG.7A, sound recognition operations are distributed between the device702and the computing device752. The device702is connected to the computing device752via a wireless connection748such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection748is a Bluetooth connection. In some examples, the device702includes a sound recognition application that enables audio data731to be captured by a microphone740on the device702and text data707to be displayed on a display716of the device702.

The device702includes a microcontroller706that executes a sound classifier703to detect whether or not a sound of interest (e.g., speech, alarm, etc.) is included within audio data731captured by a microphone740on the device702. The sound classifier703may include or be defined by a ML model726. The ML model726may define a number of parameters711that are required for the ML model726to make a prediction (e.g., whether or not the sound of interest is included within the audio data731). The ML model726may be relatively small since the actual conversion is offloaded to the computing device752. For example, the number of parameters711may be in a range between 10 k and 100 k. The sound classifier703may save power and latency through its relatively small ML model726.

Referring toFIG.7B, in operation721, the sound classifier703may receive audio data731from the microphone740on the device702. In operation723, the sound classifier703may determine whether or not the sound of interest is detected in the audio data731. If the sound of interest is not detected (No), the sound classifier703continues to monitor the audio data731, received via the microphone740, to determine whether or not the sound of interest is detected. If the sound of interest is detected (Yes), in operation725, the device702streams the audio data731to the computing device752over the wireless connection748. For example, an RF transceiver714on the device702may transmit the audio data731over the wireless connection748. In some examples, the device702compresses the audio data731, and then transmits the compressed audio data731to the computing device752.

Referring toFIG.7A, the computing device752includes a sound recognition engine709(e.g., another classifier) that executes a ML model727(e.g., a larger ML model) to convert the sound of the audio data731to text data707. The ML model727may define a number of parameters713that are required for the ML model727to make a prediction. In some examples, the number of parameters713is at least ten times greater than the number of parameters711. In some examples, the number of parameters713is at least one hundred times greater than the number of parameters711. In some examples, the number of parameters713is at least one thousand times greater than the number of parameters711. In some examples, the number of parameters713is at least one million times greater than the number of parameters711. In some examples, the number of parameters713is in a range between 1 M and 10 M. In some examples, the number of parameters713is greater than 10 M. The computing device752transmits the text data707to the device702via the wireless connection748. The device702displays the text data707on the device's display716.

FIG.8illustrates a system800for distributing sound recognition operations between a device802and a server computer860. The system800may be an example of the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, and/or the system700ofFIGS.7A and7Band may include any of the details discussed with reference to those figures. In some examples, the device802may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device802may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6.

As shown inFIG.8, sound recognition operations are distributed between the device802and the server computer860, where audio data831can be provided to the server computer860via a computing device852. The device802is connected to the computing device852via a wireless connection848such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection848is a Bluetooth connection. The computing device852is connected to the server computer860over a network850(e.g., the Internet such as Wi-Fi or mobile connection). In some examples, the device802includes a sound recognition application that enables audio data831to be captured by a microphone840on the device802and text data807to be displayed on a display816of the device802.

The device802includes a microcontroller806that executes a sound classifier803to detect whether or not a sound of interest is included within audio data831captured by a microphone840on the device802. The sound classifier803may include or be defined by a ML model826. The ML model826may define a number of parameters811that are required for the ML model826to make a prediction (e.g., whether or not the sound of interest is included within the audio data831). The ML model826may be relatively small since the actual conversion is offloaded to the server computer860. For example, the number of parameters811may be in a range between 10 k and 100 k. The sound classifier803may save power and latency through its relatively small ML model826.

If the sound of interest is not detected, the sound classifier803continues to monitor the audio data831, received via the microphone840, to determine whether or not sound of interest is detected. If the sound of interest is detected, the device802streams the audio data831to the computing device852over the wireless connection848. For example, an RF transceiver814on the device802may transmit the audio data831over the wireless connection848. In some examples, the device802compresses the audio data831, and then transmits the compressed audio data831to the computing device852.

In some examples, the computing device852may transmit the audio data831to the server computer860over the network850. In some examples, the computing device852determines whether or not the computing device852has the capabilities of converting the sound to text data807. If not, the computing device852may transmit the audio data831to the server computer860. If so, the computing device852may perform the sound conversion, as discussed with reference to the system700ofFIGS.7A and7B.

In some examples, the computing device852determines whether the sound conversion includes the translation into another language. For example, the audio data831may include speech in the English language, but the parameters of the sound recognition application indicate to provide the text data807in another language such as German. In some examples, if the conversation includes the translation into another language, the computing device852may transmit the audio data831to the server computer860. In some examples, upon receipt of audio data831from the device802, the computing device852may automatically transmit the audio data831to the server computer860. In some examples, the device802transmit the audio data831directly to the server computer860via the network850(e.g., without using the computing device852) and the device802receives the text data807from the server computer860via the network850(e.g., without using the computing device852).

The server computer860includes a sound recognition engine809that executes a ML model829(e.g., a larger ML model) to convert the sound of the audio data831to text data807. In some examples, the conversion of speech to text data807includes the translation into a different language. The ML model829may define a number of parameters815that are required for the ML model829to make a prediction (e.g., the conversation of sound to text data807). In some examples, the number of parameters815is at least one thousand times greater than the number of parameters811. In some examples, the number of parameters815is at least one million times greater than the number of parameters811. In some examples, the number of parameters815is at least one hundred million times greater than the number of parameters811. In some examples, the number of parameters815is in a range between 1 M and 100 M. In some examples, the number of parameters815is greater than 100 M. The server computer860transmits the text data807to the computing device852over the network850. The computing device852transmits the text data807to the device802via the wireless connection848. The device802displays the text data807on the device's display816.

FIG.9illustrates a system900for sound recognition operations using a device902. The system900may be an example of the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the system700ofFIGS.7A and7B, and/or the system800ofFIG.8and may include any of the details discussed with reference to those figures. In some examples, the device902may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device902may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6.

The device902is connected to the computing device952via a wireless connection948such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection948is a Bluetooth connection. The computing device952may include a microphone921configured to capture audio data931, and a sound recognition engine909configured to convert sound of the audio data931to text data907. The sound recognition engine909may include or be defined by a ML model, as discussed with reference to the previous figures. After the conversion of sound to text data907, the computing device952may transmit the text data907to the device902via the wireless connection948, and the device902receives the text data907via an RF transceiver914on the device902. The device902is configured to display the text data907on a display916of the device902.

FIG.10illustrates a system1000for executing sound recognition operations using a device1002. The system1000may be an example of the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the system700ofFIGS.7A and7B, the system800ofFIG.8, and/or the system900ofFIG.9and may include any of the details discussed with reference to those figures. In some examples, the device1002may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device1002may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6.

As shown inFIG.10, sound recognition operations are distributed between the computing device1052and the server computer1060, where text data1007is displayed via the device1002. The device1002is connected to the computing device1052via a wireless connection1048such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection1048is a Bluetooth connection. The computing device1052is connected to the server computer1060over a network1050(e.g., the Internet such as Wi-Fi or mobile connection).

The computing device1052includes a microphone1021configured to capture audio data1031. Also, the computing device1052includes a sound classifier1003(e.g., a ML model) to detect whether or not the sound of interest is included within audio data1031captured by the microphone1021on the computing device1052. If the sound of interest is not detected, the sound classifier1003continues to monitor the audio data1031, received via the microphone1021, to determine whether or not the sound of interest is detected. If sound of interest is detected, the computing device1052streams the audio data1031to the server computer1060over the network1050. In some examples, the computing device1052determines whether or not the computing device1052has the capabilities of converting the sound to text data1007. If not, the computing device1052may transmit the audio data1031to the server computer1060. If so, the computing device1052may perform the sound conversion, as discussed with reference to the system900ofFIG.9. In some examples, the computing device1052compresses the audio data1031, and sends the compressed audio data1031to the server computer1060.

In some examples, the computing device1052determines whether the sound conversion includes the translation into another language. For example, the audio data1031may include speech in the English language, but the parameters of the speech-to-text application indicate to provide the text data1007in a different language. In some examples, if the speech-to-text conversation includes the translation into another language, the computing device1052may transmit the audio data1031to the server computer1060. In some examples, upon the detection of speech within the audio data1031, the computing device1052may automatically transmit the audio data1031to the server computer1060.

The server computer1060includes a sound recognition engine1009that executes a ML model to convert the sound of the audio data1031to text data1007. In some examples, the conversion of sound to text data1007includes the translation into a different language. The server computer1060transmits the text data1007to the computing device1052over the network1050. The computing device1052transmits the text data1007to an RF transceiver1014on the device1002via the wireless connection1048. The device1002displays the text data1007on the device's display1016.

FIG.11is a flowchart1100depicting example operations of the system700ofFIGS.7A and7B. Although the flowchart1100ofFIG.11is explained with respect to the system700ofFIGS.7A and7B, the flowchart1100may be applicable to any of the embodiments discussed herein including the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the head-mounted display device402ofFIG.4, the electronics component570ofFIG.5, the electronics component670ofFIG.6, the system800ofFIG.8, the system900ofFIG.9, and/or the system1000ofFIG.10. Although the flowchart1100ofFIG.11illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations ofFIG.11and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion.

Operation1102includes receiving, via a microphone740of the device702, audio data731. Operation1104includes detecting, by a sound classifier703, whether or not the audio data731includes a sound of interest (e.g., speech), where the sound classifier703executes a first ML model (e.g., ML model726).

Operation1106includes transmitting, via a wireless connection748, the audio data731to a computing device752, where the audio data731is configured to be used by the computing device752to translate the sound of interest to text data707using a second ML model (e.g., ML model727). Operation1108includes receiving, via the wireless connection748, the text data707from the computing device752. Operation1110includes displaying, by the device702, the text data707on a display716of the device702.

FIG.12is a flowchart1200depicting example operations of the system800ofFIG.8. Although the flowchart1200ofFIG.12is explained with respect to the system800ofFIG.8, the flowchart1200may be applicable to any of the embodiments discussed herein including the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the head-mounted display device402ofFIG.4, the electronics component570ofFIG.5, the electronics component670ofFIG.6, the system700ofFIGS.7A and7B, the system900ofFIG.9, and/or the system1000ofFIG.10. Although the flowchart1200ofFIG.12illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations ofFIG.12and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion.

Operation1202includes receiving, via a microphone840of the device802, audio data831. Operation1204includes detecting, by a sound classifier803of the device802, whether or not the audio data831includes a sound of interest (e.g., speech), where the sound classifier803executes a first ML model (e.g., ML model826).

Operation1206includes transmitting, by the device802, the audio data831to a computing device852via a wireless connection848, where the audio data831is further transmitted to a server computer860over a network850to translate the sound to text data807using a second ML model (e.g., ML model829). Operation1208includes receiving, by the device802, the text data807from the computing device852via the wireless connection848. Operation1210includes displaying, by the device802, the text data807on a display816of the device802.

FIGS.13A through13Cillustrate a system1300for distributing image recognition operations between a device1302and a computing device1352. The system1300may be an example of the system100ofFIG.1, the system200ofFIG.2, and/or the system300ofFIG.3and may include any of the details discussed with reference to those figures. In some examples, the device1302may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device1302may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6. In some examples, the system1300also includes the capabilities of distributed sound recognition operations and may include any of the details discussed with reference to the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, and/or the system1000ofFIG.10.

As shown inFIG.13A, image recognition operations are distributed between the device1302and the computing device1352. In some examples, the image recognition operations include facial detection and tracking. However, the image recognition operations may include operations to detect (and track) other regions of interest in image data such as objects, text, and barcodes. The device1302is connected to the computing device1352via a wireless connection1348such as a short-term wireless connection such as Bluetooth or NFC. In some examples, the wireless connection1348is a Bluetooth connection. In some examples, the device1302and/or the computing device1352include an image recognition application that enables objects to be recognized (and tracked) via image data captured by one or more imaging sensors1342.

The device1302includes a microcontroller1306that executes an image classifier1303to detect whether or not an object of interest1333is included within image data1329captured by the imaging sensor(s)1342on the device1302. In some examples, the object of interest1333includes facial features. In some examples, the object of interest1333includes text data. In some examples, the object of interest1333includes OCR code. However, the object of interest1333may be any type of object capable of being detected in image data. The image classifier1303may include or be defined by a ML model1326. The ML model1326may define a number of parameters1311that are required for the ML model1326to make a prediction (e.g., whether or not the object of interest1333is included within the image data1329). The ML model1326may be relatively small since some of the more intensive image recognition operations are offloaded to the computing device1352. For example, the number of parameters1311may be in a range between 10 k and 100 k. The image classifier1303may save power and latency through its relatively small ML model1326.

Referring toFIG.13B, in operation1321, the image classifier1303may receive image data1329from the imaging sensor(s)1342on the device1302. In operation1323, the image classifier1303may be activated. In operation1325, the image classifier1303may determine the object of interest1333is detected in an image frame1329aof the image data1329. If the object of interest1333is not detected (No), in operation1328, the image classifier1303(and/or the imaging sensor(s)1342) may transition to a power-saving state. In some examples, after a period of time has elapsed, the image classifier1303may be re-activated (e.g., process returns to operation1323) to determine whether the object of interest1333is detected in an image frame1329aof the image data1329. If the object of interest1333is detected (Yes), in operation1330, the device1302transmits the image frame1329ato the computing device1352over the wireless connection1348. For example, an RF transceiver1314on the device1302may transmit the image frame1329aover the wireless connection1348. In some examples, the device1302compresses the image frame1329a, and transmits the compressed image frame1329ato the computing device1352.

Referring toFIG.13A, the computing device1352includes an object detector1309that executes a ML model1327(e.g., a larger ML model) to compute a bounding box dataset1341. In some examples, the bounding box dataset1341is an example of object location data. The bounding box dataset1341may be data that defines the location in which the object of interest1333(e.g., the facial features) are located within the image frame1329a. In some examples, referring toFIG.13C, the bounding box dataset1341defines coordinates of a bounding box1381that includes the object of interest1333within the image frame1329a. In some examples, the coordinates include a height coordinate1383, a left coordinate1385, a top coordinate1387, and a width coordinate1389. For example, the height coordinate1383may be the height of the bounding box1381as a ratio of the overall image height. The left coordinate1385may be the left coordinate of the bounding box1381as a ratio of overall image width. The top coordinate1387may be the top coordinate of the bounding box1381as a ratio of overall image height. The width coordinate1389may be the width of the bounding box1381as a ratio of the overall image width.

The ML model1327may define a number of parameters1313that are required for the ML model1327to make a prediction (e.g., computation of the bounding box dataset1341). In some examples, the number of parameters1313is at least ten times greater than the number of parameters1311. In some examples, the number of parameters1313is at least one hundred times greater than the number of parameters1311. In some examples, the number of parameters1313is at least one thousand times greater than the number of parameters1311. In some examples, the number of parameters1313is at least one million times greater than the number of parameters1311. In some examples, the number of parameters1313is in a range between 1 M and 10 M. In some examples, the number of parameters1313is greater than 10 M. The computing device1352transmits the bounding box dataset1341to the device1302via the wireless connection1348.

The device1302includes an object tracker1335configured to use the bounding box dataset1341to track the object of interest1333in one or more subsequent image frames1329b. In some examples, the object tracker1335is configured to execute a low-complexity tracking mechanism such as inertial measurement unit (IMU)-based warping, blob detection, or optical flow. For example, the object tracker1335may propagate the bounding box dataset1341for subsequent image frames1329b. The object tracker1335may include a cropper1343and a compressor1345. The cropper1343may use the bounding box dataset1341to identify an image region1347within the image frame1329b. The compressor1345may compress the image region1347. For example, the image region1347may represent an area within the image frame1329bthat has been cropped and compressed by the object tracker1335.

The device1302may then transmit the image region1347to the computing device1352over the wireless connection1348. For example, as the object tracker1335is tracking the object of interest1333, the computing device1352may receive a stream of image regions1347. At the computing device1352, the object detector1309may perform image recognition on the image regions1347received from the device1302over the wireless connection1348. In some examples, if the object of interest1333is relatively close to the edges of the image regions1347(or not present at all), the computing device1352may transmit a request to send a new full fame (e.g., a new image frame1329a) to compute the bounding box dataset1341again. In some examples, if the image frame1329adoes not contain the object of interest1333, the computing device1352may transmit a request to enter a power-saving state to poll for the object of interest. In some examples, a visual indicator1351(e.g., a visual box) may be provided on a display1316of the device1302, where the visual indicator1351identifies the object of interest1333(e.g., the facial features).

FIG.14is a flowchart1400depicting example operations of the system1300ofFIGS.13A through13C. Although the flowchart1400ofFIG.14is explained with respect to the system1300ofFIGS.13A through13C, the flowchart1400may be applicable to any of the embodiments discussed herein including the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the head-mounted display device402ofFIG.4, the electronics component570ofFIG.5, and/or the electronics component670ofFIG.6, the system700ofFIGS.7A and7B. Although the flowchart1400ofFIG.14illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations ofFIG.14and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion. In some examples, the operations of the flowchart1400ofFIG.14may be combined with the operations of the flowchart1100ofFIG.11and/or the flowchart1200ofFIG.12.

Operation1402includes receiving, via at least one imaging sensor1342on the device1302, image data1329. Operation1404includes detecting, by an image classifier1303of the device1302, whether or not the object of interest1333is included within the image data1329, where the image classifier1303executes a ML model1326.

Operation1406includes transmitting, via the wireless connection1348, the image data1329(e.g., image frame1329a) to a computing device1352, where the image frame1329aincludes the object of interest1333. The image data1329is configured to be used by the computing device1352for image recognition using a ML model1327.

Operation1408includes receiving, via the wireless connection1348, a bounding box dataset1341from the computing device1352. Operation1410includes identifying, by the device1302, an image region1347in subsequent image data (e.g., image frame1329b) using the bounding box dataset1341. Operation1412includes transmitting, via the wireless connection1348, the image region1347to the computing device1352, where the image region1347is configured to be used by the computing device1352for image recognition.

FIG.15illustrates a system1500for distributing image recognition operations between a device1502and a computing device1552. The system1500may be an example of the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, and/or the system1300ofFIGS.13A through13Cand may include any of the details discussed with reference to those figures. In some examples, the device1502may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device1502may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6. In some examples, the system1500also includes the capabilities of distributed sound recognition operations and may include any of the details discussed with reference to the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, and/or the system1000ofFIG.10.

As shown inFIG.15, image recognition operations are distributed between the device1502and the computing device1552. In some examples, the image recognition operations include facial detection and tracking. However, the image recognition operations may include operations to detect (and track) other regions of interest in image data such as objects, text, and barcodes. The device1502is connected to the computing device1552via a wireless connection1548such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection1548is a Bluetooth connection. In some examples, the device1502and/or the computing device1552include an image recognition application that enables objects to be recognized (and tracked) via image data captured by imaging sensor1542aand imaging sensor1542b.

The imaging sensor1542amay be considered a low power, low resolution (LPLR) image sensor. The imaging sensor1542bmay be considered a high power, high resolution (HPHR) image sensor. An image frame1529bcaptured by the imaging sensor1542bhas a resolution1573bthat is higher than a resolution1573aof an image frame1529acaptured by the imaging sensor1542a. In some examples, the imaging sensor1542ais configured to obtain image data (e.g., image frames1529a) while the device1502is activated and coupled to the user (e.g., continuously or periodically captures image frames1529awhile the device1502is activated). In some examples, the imaging sensor1542ais configured to operate as an always-on sensor. In some examples, the imaging sensor1542bis activated (e.g., for a short duration) in response to the detection of an object of interest, as further discussed below.

The device1502includes a lighting condition sensor1544configured to estimate a lighting condition for capturing image data. In some examples, the lighting condition sensor1544includes an ambient light sensor that detects the amount of ambient light that is present, which can be used to ensure that the image frame1529ais captured with a desired signal-to-noise ratio (SNR). However, the lighting condition sensor1544may include other types of photometric (or colorimeter) sensors. The motion sensor1546may be used for monitoring device movement such as tilt, shake, rotation, and/or swing and/or for blur estimation. The sensor trigger1571may receive lighting condition information from the lighting condition sensor1544and motion information from the motion sensor1546, and, if the lighting condition information and the motion information, indicate that the conditions are acceptable to obtain an image frame1529a, the sensor trigger1571may activate the imaging sensor1542ato capture an image frame1529a.

The device1502includes a microcontroller1506configured to execute an image classifier1503that detects whether or not an object of interest is included within the image frame1529acaptured by the imaging sensor1542a. Similar to the other embodiments, the image classifier1503may include or be defined by a ML model. The ML model may define a number of parameters that are required for the ML model to make a prediction (e.g., whether or not the object of interest is included within the image frame1529a). The ML model may be relatively small since some of the more intensive image recognition operations are offloaded to the computing device1552. For example, the number of parameters may be in a range between 10 k and 100 k. The image classifier1503may save power and latency through its relatively small ML model.

If the image classifier1503detects the existence of the object of interest within the image frame1529a, the image classifier1503is configured to trigger the imaging sensor1542bto capture the image frame1529b. As indicated above, the image frame1529bhas a resolution1573bthat is higher than the resolution1573aof the image frame1529a. The device1502transmits the image frame1529bto the computing device1552via the wireless connection1548for further processing. In some examples, the device1502compresses the image frame1529b, and then transmits the compressed image frame1529bto the computing device1552. In some examples, the motion information and/or the lighting condition information is used to determine whether to transmit the image frame1529b. For example, if the motion information indicates motion above a threshold level (e.g., motion is high), the image frame1529bmay not be transmitted, and the microcontroller1506may activate the imaging sensor1542bto capture another image frame. If the lighting condition information indicates that the lighting condition is below a threshold level, the image frame1529bmay not be transmitted, and the microcontroller1506may activate the imaging sensor1542bto capture another image frame.

The computing device1552includes an object detector1509configured to perform image recognition operations (including the computation of a bounding box dataset) using the image frame1529b. Similar to the embodiment of system1300ofFIGS.13A through13C, the object detector1509executes a larger ML model to compute a bounding box dataset using the higher resolution image (e.g., the image frame1529b), which is transmitted back to the device1502via the wireless connection1548. Then, the device1502uses the bounding box dataset to track the object of interest in one or more subsequent image frames. For example, the device1502may use a low-complexity tracking mechanism such as inertial measurement unit (IMU)-based warping, blob detection, or optical flow to propagate the bounding box dataset for subsequent image frames. The device1502may use the bounding box dataset to identify an image region within the image frame1529b, and the device1502may compress the image region, which is then transmitted back to the computing device1552for image recognition.

FIG.16is a flowchart1600depicting example operations of the system1500ofFIG.15. Although the flowchart1600ofFIG.16is explained with respect to the system1500ofFIG.15, the flowchart1600may be applicable to any of the embodiments discussed herein including the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the head-mounted display device402ofFIG.4, the electronics component570ofFIG.5, the electronics component670ofFIG.6, and/or the system1300ofFIG.13. Although the flowchart1600ofFIG.16illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations ofFIG.16and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion. In some examples, the operations of the flowchart1600ofFIG.16may be combined with the operations of the flowchart1100ofFIG.11, the flowchart1200ofFIG.12, and/or the flowchart1400ofFIG.14.

Operation1602includes receiving, by a first imaging sensor (e.g., imaging sensor1542a) of the device1502, a first image frame1529a. Operation1604includes detecting, by an image classifier1503of the device1502, the presence of the object of interest in the first image frame1529a.

Operation1606includes receiving, by a second imaging sensor (e.g., imaging sensor1542b) of the device1502, a second image frame1529b, the second image frame1529bhaving a resolution1573bhigher than a resolution1573aof the first image frame1529a, where the second image frame1529bis transmitted to the computing device1552via a wireless connection1548, and the second image frame1529bis configured to be used by an object detector1509at the computing device1552.

FIG.17illustrates a system1700for distributing image recognition operations between a device1702and a computing device1752. The system1700may be an example of the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the system1300ofFIGS.13A through13C, and/or the system1500ofFIG.15and may include any of the details discussed with reference to those figures. In some examples, the device1702may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device1702may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6. In some examples, the system1700also includes the capabilities of distributed sound recognition operations and may include any of the details discussed with reference to the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, and/or the system1000ofFIG.10.

As shown inFIG.17, image recognition operations are distributed between the device1702and the computing device1752. In some examples, the image recognition operations include facial detection and tracking. However, the image recognition operations may include operations to detect (and track) other regions of interest in image data such as objects, text, and barcodes. The device1702is connected to the computing device1752via a wireless connection1748such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection1748is a Bluetooth connection. In some examples, the device1702and/or the computing device1752include an image recognition application that enables objects to be recognized (and tracked) via image data captured by imaging sensor1742aand imaging sensor1742b.

The imaging sensor1742amay be considered a low power, low resolution (LPLR) image sensor. The imaging sensor1742bmay be considered a high power, high resolution (HPHR) image sensor. An image frame1729bcaptured by the imaging sensor1742bhas a resolution1773bthat is higher than a resolution1773aof an image frame1729acaptured by the imaging sensor1742a. In some examples, the imaging sensor1742ais configured to obtain image data (e.g., image frames1729a) while the device1702is activated and coupled to the user (e.g., continuously or periodically captures image frames1729awhile the device1702is activated). In some examples, the imaging sensor1742ais configured to operate as an always-on sensor. In some examples, the imaging sensor1742bis activated (e.g., for a short duration) in response to the detection of an object of interest, as further discussed below.

The device1702includes a lighting condition sensor1744configured to estimate a lighting condition for capturing image data. In some examples, the lighting condition sensor1744includes an ambient light sensor that detects the amount of ambient light that is present, which can be used to ensure that the image frame1729ais captured with a desired signal-to-noise ratio (SNR). However, the lighting condition sensor1744may include other types of photometric (or colorimeter) sensors. The motion sensor1746may be used for monitoring device movement such as tilt, shake, rotation, and/or swing and/or for blur estimation. The sensor trigger1771may receive lighting condition information from the lighting condition sensor1744and motion information from the motion sensor1746, and, if the lighting condition information and the motion information, indicate that the conditions are acceptable to obtain an image frame1729a, the sensor trigger1771may activate the imaging sensor1742ato capture an image frame1729a.

The device1702includes a microcontroller1706configured to execute a classifier1703that detects whether or not a region of interest (ROI)1789is included within the image frame1729acaptured by the imaging sensor1742a. The ROI1789can also be referred to as an object of interest. The classifier1703may include or be defined by a ML model. The ML model may define a number of parameters that are required for the ML model to make a prediction (e.g., whether or not the ROI1789is included within the image frame1729a). The ML model may be relatively small since some of the more intensive image recognition operations are offloaded to the computing device1752. For example, the number of parameters may be in a range between 10 k and 100 k. The classifier1703may save power and latency through its relatively small ML model.

If the classifier1703detects the existence of the ROI1789within the image frame1729a, the classifier1703is configured to trigger the imaging sensor1742bto capture the image frame1729b. As indicated above, the image frame1729bhas a resolution1773bthat is higher than the resolution1773aof the image frame1729a. The device1702transmits the image frame1729bto the computing device1752via the wireless connection1748for further processing. In some examples, the device1702compresses the image frame1729b, and transmits the compressed image frame1729bto the computing device1752.

The computing device1752includes a ROI classifier1709that executes a ML model (e.g., a larger ML model) to compute a ROI dataset1741. In some examples, the ROI dataset1741is an example of the object location data and/or the bounding box dataset. The ROI dataset1741may be data that defines the location in which the ROI1789are located within the image frame1729b. The computing device1752may transmit the ROI dataset1741to the device1702via the wireless connection1748.

The device1702includes an ROI tracker1735configured to use the ROI dataset1741to track the ROI1789in one or more subsequent image frames. In some examples, the ROI tracker1735is configured to execute a low-complexity tracking mechanism such as inertial measurement unit (IMU)-based warping, blob detection, or optical flow. For example, the ROI classifier1709may propagate the ROI dataset1741for subsequent image frames. The ROI tracker1735may include a cropper1743and a compressor1745. The cropper1743may use the ROI dataset1741to identify an image region1747within the image frame1729b. The compressor1745may compress the image region1747. For example, the image region1747may represent an area within the image frame1729bthat has been cropped and compressed by the ROI tracker1735, where the image region1747includes the ROI1789.

The device1702may then transmit the image region1747to the computing device1752over the wireless connection1748. For example, as the ROI tracker1735is tracking the ROI1789, the computing device1752may receive a stream of image regions1747. At the computing device1752, the ROI classifier1709may perform object detection on the image regions1747received from the device1702over the wireless connection1748. In some examples, if the ROI1789is relatively close to the edges of the image regions1747(or not present at all), the computing device1752may transmit a request to send a new full fame (e.g., a new image frame1729b) to compute the ROI dataset1741again. In some examples, if the image frame1729adoes not contain the ROI1789, the computing device1752may transmit a request to enter a power-saving state to poll for ROIs1789. In some examples, a visual indicator1787is provided on a display1716of the device1702, where the visual indicator1787identifies the ROI1789.

FIG.18illustrates a system1800for distributing image recognition operations between a device1802and a computing device1852. The system1800may be an example of the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the system1300ofFIGS.13A through13C, the system1500ofFIG.15, and the system1700ofFIG.17and may include any of the details discussed with reference to those figures. In some examples, the device1802may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device1802may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6. In some examples, the system1800also includes the capabilities of distributed sound recognition operations and may include any of the details discussed with reference to the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, and/or the system1000ofFIG.10.

As shown inFIG.18, image recognition operations are distributed between the device1802and the computing device1852. In some examples, the image recognition operations include facial detection and tracking. However, the image recognition operations may include operations to detect (and track) other regions of interest in image data such as objects, text, and barcodes. The device1802is connected to the computing device1852via a wireless connection (e.g., radio resources1867) such as a short-term wireless connection such as Bluetooth or NFC connection. In some examples, the wireless connection is a Bluetooth connection. In some examples, the device1802and/or the computing device1852include an image recognition application that enables objects to be recognized (and tracked) via image data captured by camera1842aand camera1842b.

The camera1842amay be considered a low power, low resolution (LPLR) camera. The camera1842bmay be considered a high power, high resolution (HPHR) camera. An image frame captured by the camera1842bhas a resolution that is higher than a resolution of an image frame captured by the camera1842a. In some examples, the camera1842ais configured to obtain image data while the device1802is activated and coupled to the user (e.g., continuously or periodically captures image frames while the device1802is activated). In some examples, the camera1842ais configured to operate as an always-on sensor. In some examples, the camera1842bis activated (e.g., for a short duration) in response to the detection of an object of interest, as further discussed below.

The device1802includes a lighting condition sensor1844configured to estimate a lighting condition for capturing image data. In some examples, the lighting condition sensor1844includes an ambient light sensor that detects the amount of ambient light that is present, which can be used to ensure that the image frame is captured with a desired signal-to-noise ratio (SNR). However, the lighting condition sensor1844may include other types of photometric (or colorimeter) sensors. The motion sensor1846may be used for monitoring device movement such as tilt, shake, rotation, and/or swing and/or for blur estimation. The sensor trigger1871may receive lighting condition information from the lighting condition sensor1844and motion information (e.g., blur estimate) from the motion sensor1846, and, if the lighting condition information and the motion information, indicate that the conditions are acceptable to obtain an image frame, the sensor trigger1871may activate the camera1842ato capture an image frame with a lower resolution. In some examples, the device1802includes a microphone1840that provides audio data to the classifier1803.

The device1802includes a classifier1803that detects whether or not a region of interest is included within the image frame captured by the camera1842a. The classifier1803may include or be defined by a ML model. The ML model may define a number of parameters that are required for the ML model to make a prediction (e.g., whether or not a region of interest is included within the image frame). The ML model may be relatively small since some of the more intensive image recognition operations are offloaded to the computing device1852. For example, the number of parameters may be in a range between 10 k and 100 k. The classifier1803may save power and latency through its relatively small ML model.

If the classifier1803detects the existence of a region of interest within the image frame captured by the camera1842a, the classifier1803is configured to trigger the camera1842bto capture a higher resolution image. In some examples, the device1802transmits the full image frame captured by the camera1842bvia radio resources1867.

The computing device1852includes a classifier1809that executes a ML model (e.g., a larger ML model) to compute a ROI dataset (e.g., object box, x, y). The ROI dataset may be data that defines the location in which the object of interest is located within the image frame. The computing device1852may transmit the ROI dataset to the device1802. The classifier1803may provide the ROI dataset to a copper1843that crops the subsequent image frames to identify an image region. The image region is compressed by a compressor1845and transmitted to the computing device1852via the radio resources1867. In some examples, the device1802includes an action manager1865that receives the ROI detection from the classifier1809and may provide a visual indicator or other action on a display1816of the device1802.

FIG.19is a flowchart1900depicting example operations of the system1700ofFIG.17. Although the flowchart1900ofFIG.19is explained with respect to the system1700ofFIG.17, the flowchart1900may be applicable to any of the embodiments discussed herein including the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the head-mounted display device402ofFIG.4, the electronics component570ofFIG.5, the electronics component670ofFIG.6, the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, the system1000ofFIG.10, the system1300ofFIGS.13A through13C, the system1500ofFIG.15, and/or the system1800ofFIG.18. Although the flowchart1900ofFIG.19illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations ofFIG.19and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion. In some examples, the operations of the flowchart1900ofFIG.19may be combined with the operations of the flowchart1100ofFIG.11, the flowchart1200ofFIG.12, the flowchart1400ofFIG.14, and/or the flowchart1600ofFIG.16.

Operation1902includes activating a first imaging sensor1742aof the device1702to capture first image data (e.g., image frame1729a). Operation1904includes detecting, by a classifier1703of the device1702, whether or not a region of interest (ROI)1789is included within the first image data, where the classifier1703executes a ML model.

Operation1906includes activating a second imaging sensor1742bof the device1702to capture second image data (e.g., image frame1729b) in response to the ROI1789being detected within the first image data. The second image data has a resolution1773bhigher than a resolution1773aof the first image data. Operation1908includes transmitting, via the wireless connection1748, the second imaging data to a computing device1752, where the second image data1729bis used by the computing device1752for image processing using a ML model.

FIG.20illustrates a system2000for distributing image recognition operations between a device2002and a computing device2052. The system2000may be an example of the system100ofFIG.1, the system200ofFIG.2, and/or the system300ofFIG.3and may include any of the details discussed with reference to those figures. In some examples, the device2002may be an example of the head-mounted display device402ofFIG.4and may include any of the details discussed with reference to that figure. In some examples, the components of the device2002may include the electronics component570ofFIG.5and/or the electronics component670ofFIG.6. In some examples, the system2000also includes the capabilities of distributed sound recognition operations and may include any of the details discussed with reference to the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, and/or the system1000ofFIG.10. In some examples, the system2000also includes the capabilities of distributed image recognition operations may include any of the details discussed with reference to the system1300ofFIGS.13A through13C, the system1500ofFIG.15, the system1700ofFIG.17, and the system1800ofFIG.18.

As shown inFIG.20, hotword recognition operations for voice commands are distributed between the device2002and the computing device2052. The device2002may include a voice command detector2093that executes a ML model2026(e.g., a gatekeeping model) to continuously (e.g., periodically) process microphone samples (e.g., audio data2031) from a microphone2040on the device2002for an initial portion of a hot-word (e.g., “ok G” or “ok D”) for a voice command2090. If the voice command detector2093detects that initial portion, the voice command detector2093may cause a buffer2091to capture the subsequent audio data2031. Also, the device2002may transmit an audio portion2092to the computing device2052over the wireless connection2048. In some examples, the device2002compresses the audio portion2092, and then transmits the compressed audio portion2092. The audio portion2092may be a portion of the buffer. For example, the audio portion2092may be 1-2 seconds of audio data2031from the head of the buffer2091.

The computing device2052includes a hot-word recognition engine2094configured to execute a ML model2027(e.g., a larger ML model) to perform the full hot-word recognition using the audio portion2092. For example, the ML model2027receives the audio portion2092as an input, and the ML model2027predicts whether the audio portion2092includes a hot-word (e.g., “ok Google, Ok device”). If the audio portion2092is a false positive2094, the computing device2052may transmit a disarm command2096to the device2002, which discards the contents (e.g., the audio data2031) of the buffer2091. If the audio portion2092is a true positive2095, the remainder2099of the buffer2091is transmitted to the computing device2052. In some examples, the device2002compresses the audio data2031within the buffer2091(or the remainder2099of the buffer2091) and transmits the compressed audio data2031to the computing device2052. The computing device2052includes a command generator2097that uses the audio data2031(e.g., the remainder2099of the buffer2091and the audio portion2092) to determine an action command2098(e.g., compose an email, take a picture, etc.). The computing device2052may transmit the action command2098to the device2002over the wireless connection2048.

FIG.21is a flowchart2100depicting example operations of the system2000ofFIG.20. Although the flowchart2100ofFIG.21is explained with respect to the system2000ofFIG.20, the flowchart2100may be applicable to any of the embodiments discussed herein including the system100ofFIG.1, the system200ofFIG.2, the system300ofFIG.3, the head-mounted display device402ofFIG.4, the electronics component570ofFIG.5, the electronics component670ofFIG.6, the system700ofFIGS.7A and7B, the system800ofFIG.8, the system900ofFIG.9, the system1000ofFIG.10, the system1300ofFIGS.13A through13C, the system1500ofFIG.15, and/or the system1800ofFIG.18. Although the flowchart2100ofFIG.21illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations ofFIG.21and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion. In some examples, the operations of the flowchart2100ofFIG.21may be combined with the operations of the flowchart1100ofFIG.11, the flowchart1200ofFIG.12, the flowchart1400ofFIG.14, the flowchart1600ofFIG.16, and/or the flowchart1900ofFIG.19.

Operation2102includes receiving, via a microphone2040of the device2002, audio data2031. Operation2104includes detecting, by a voice command detector2093, a presence of a portion of a hotword from the audio data2031, where the voice command detector2093executes a ML model.

Operation2106includes storing, in a buffer2091of the device2002, the audio data2031that is received via the microphone2040in response to the portion of the hot-word being detected. Operation2108includes transmitting, via a wireless connection2048, an audio portion2092of the buffer2091to a computing device2052, where the audio portion2092of the buffer2091is configured to be used by the computing device2052to perform hotword recognition.

Although the disclosed inventive concepts include those defined in the attached claims, it should be understood that the inventive concepts can also be defined in accordance with the following embodiments:

Embodiment 1 is a method for distributed sound recognition using a wearable device comprising: receiving, via a microphone of the wearable device, audio data; detecting, by a sound classifier of the wearable device, whether or not the audio data includes a sound of interest; and transmitting, via a wireless connection, the audio data to a computing device in response to the sound of interest being detected within the audio data.

Embodiment 2 is the method of embodiment 1, wherein the sound classifier executes a first machine learning (ML) model.

Embodiment 3 is the method of any of embodiments 1 through 2, wherein the audio data is configured to be used by the computing device or a server computer for further sound recognition using a second ML model.

Embodiment 4 is the method of any of embodiments 1 through 3, wherein the audio data is configured to be used by the computing device for further sound recognition.

Embodiment 5 is the method of any of embodiments 1 through 4, wherein the audio data is configured to be used by the server computer for further sound recognition.

Embodiment 6 is the method of any of embodiments 1 through 5, wherein the server computer is connected to the computing device over a network.

Embodiment 7 is the method of any of embodiments 1 through 6, wherein the sound of interest includes speech.

Embodiment 8 is the method of any of embodiments 1 through 7, wherein the audio data is configured to be used by the computing device or the server computer to translate the speech to text data using the second ML model.

Embodiment 9 is the method of any of embodiments 1 through 8, wherein the method further comprises receiving, via the wireless connection, the text data from the computing device.

Embodiment 10 is the method of any of embodiments 1 through 9, wherein the speech is in a first language, and the text data is in a second language, the second language being different from the first language.

Embodiment 11 is the method of any of embodiments 1 through 10, further comprising displaying the text data on a display of the wearable device.

Embodiment 12 is the method of any of embodiments 1 through 11, further comprising compressing the audio data, wherein the compressed audio data is transmitted to the computing device via the wireless connection.

Embodiment 13 is the method of any of embodiments 1 through 12, further comprising extracting features from the audio data, wherein the extracted features are transmitted to the computing device via the wireless connection.

Embodiment 14 is the method of any of embodiments 1 through 13, wherein the wireless connection is a short-range wireless connection.

Embodiment 15 is the method of any of embodiments 1 through 14, wherein the wearable device includes smartglasses.

Embodiment 16 is a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 through 15.

Embodiment 17 is a wearable device configured to perform any of the embodiments 1 through 15.

Embodiment 18 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any of embodiments 1 through 15.

Embodiment 19 is a non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to receive audio data from a microphone of a wearable device, detect, by a sound classifier of the wearable device, whether or not the audio data includes a sound of interest, and transmit, via a wireless connection, the audio data to a computing device in response to the sound of interest being detected within the audio data.

Embodiment 20 is the non-transitory computer-readable medium of embodiment 19, wherein the sound classifier is configured to execute a first machine learning (ML) model.

Embodiment 21 is the non-transitory computer-readable medium of any of embodiments 19 through 20, wherein the audio data is configured to be used by the computing device for further sound recognition using a second ML model.

Embodiment 22 is the non-transitory computer-readable medium of any of embodiments 19 through 21, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to continue to detect, by the sound classifier, whether or not the audio data includes the sound of interest in response to the sound of interest not being detected within the audio data.

Embodiment 23 is the non-transitory computer-readable medium of any of embodiments 19 through 22, wherein the sound of interest includes speech.

Embodiment 24 is the non-transitory computer-readable medium of any of embodiments 19 through 23, wherein the audio data is configured to be used by the computing device to translate the speech to text data using the second ML model.

Embodiment 25 is the non-transitory computer-readable medium of any of embodiments 19 through 24, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to receive, via the wireless connection, the text data from the computing device.

Embodiment 26 is the non-transitory computer-readable medium of any of embodiments 19 through 25, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to compress the audio data, wherein the compressed audio data is transmitted to the computing device via the wireless connection.

Embodiment 27 is the non-transitory computer-readable medium of any of embodiments 19 through 26, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to extract features from the audio data, wherein the extracted features are transmitted to the computing device via the wireless connection.

Embodiment 28 is the non-transitory computer-readable medium of any of embodiments 19 through 27, wherein the wearable device includes smartglasses.

Embodiment 29 is the non-transitory computer-readable medium of any of embodiments 19 through 28, wherein the computing device includes a smartphone.

Embodiment 30 is a method that includes operations of the non-transitory computer-readable medium of any of embodiments 19 through 29.

Embodiment 31 is a wearable device that includes the features of any of embodiments 19 through 29.

Embodiment 32 is a wearable device for distributed sound recognition, the wearable device comprising a microphone configured to receive audio data, a sound classifier configured to detect whether or not the audio data includes a sound of interest, and a radio frequency (RF) transceiver configured to transmit the audio data to a computing device via a wireless connection in response to the sound of interest being detected within the audio data.

Embodiment 33 is the wearable device of embodiment 32, wherein the sound classifier includes a first machine learning (ML) model.

Embodiment 34 is the wearable device of any of embodiments 30 through 33, wherein the audio data is configured to be used by the computing device or a server computer to translate the sound of interest to text data using a second ML model.

Embodiment 35 is the wearable device of any of embodiments 30 through 34, wherein the RF transceiver is configured to receive the text data from the computing device over the wireless connection.

Embodiment 36 is the wearable device of any of embodiments 30 through 35, wherein the wearable device further comprises a display configured to display the text data.

Embodiment 37 is the wearable device of any of embodiments 30 through 36, wherein the wearable device includes smartglasses.

Embodiment 38 is the wearable device of any of embodiments 30 through 37, wherein the wireless connection is a Bluetooth connection.

Embodiment 39 is a computing device for sound recognition including at least one processor; and a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to receive, via a wireless connection, audio data from a wearable device, the audio data having a sound of interest detected by a sound classifier executing a first machine-learning (ML) model, determine whether to translate the sound of interest to text data using a sound recognition engine on the computing device, translate, by the sound recognition engine, the sound of interest to the text data in response to the determination to use the sound recognition engine on the computing device, the sound recognition engine configured to execute a second ML model, and transmit, via the wireless connection, the text data to the wearable device.

Embodiment 40 is the computing device of embodiment 39, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to transmit, over a network, the audio data to a server computer in response to the determination to not use the sound recognition engine on the computing device, and receive, over the network, the text data from the server computer.

Embodiment 41 is the computing device of any of embodiments 39 through 40, wherein the computing device includes a smartphone.

Embodiment 42 is a method that includes operations of the computing device of any of embodiments 39 through 40.

Embodiment 43 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the operations of the computing device of any of embodiments 39 through 40.

Embodiment 44 is a method for distributed image recognition using a wearable device including receiving, via at least one imaging sensor of the wearable device, image data, detecting, by an image classifier of the wearable device, whether or not an object of interest is included within the image data, and transmitting, via a wireless connection, the image data to a computing device.

Embodiment 45 is the method of embodiment 44, wherein the image classifier executes a first machine-learning (ML) model.

Embodiment 46 is the method of any of embodiments 44 through 45, wherein the image data is configured to be used by the computing device for further image recognition using a second ML model.

Embodiment 47 is the method of any of embodiments 44 through 46, further comprising receiving, via the wireless connection, a bounding box dataset from the computing device.

Embodiment 48 is the method of any of embodiments 44 through 47, further comprising identifying, by an object tracker of the wearable device, an image region in subsequent image data captured by the at least one imaging sensor using the bounding box dataset.

Embodiment 49 is the method of any of embodiments 44 through 48, further comprising transmitting, via the wireless connection, the image region to the computing device, the image region configured to be used by the computing device for further image recognition.

Embodiment 50 is the method of any of embodiments 44 through 49, further comprising cropping, by the object tracker, the image region from the subsequent image data.

Embodiment 51 is the method of any of embodiments 44 through 50, further comprising compressing, by the object tracker, the image region, wherein the compressed image region is transmitted to the computing device over the wireless network.

Embodiment 52 is the method of any of embodiments 44 through 51, wherein the object of interest includes facial features.

Embodiment 53 is the method of any of embodiments 44 through 52, further comprising activating a first imaging sensor of the wearable device to capture first image data.

Embodiment 54 is the method of any of embodiments 44 through 46, further comprising detecting, by the image classifier, whether the first image data includes the object of interest.

Embodiment 55 is the method of any of embodiments 44 through 54, further comprising activating a second imaging sensor to capture second image data.

Embodiment 56 is the method of any of embodiments 44 through 45, wherein the second image data has a quality higher than the quality of the first image data.

Embodiment 57 is the method of any of embodiments 44 through 56, wherein the second image data is transmitted to the computing device via the wireless connection, the second image data configured to be used by the computing device for further image recognition.

Embodiment 58 is the method of any of embodiments 44 through 57, further comprising receiving, via a light condition sensor of the wearable device, light condition information.

Embodiment 59 is the method of any of embodiments 44 through 58, further comprising activating the first imaging sensor based on the light condition information.

Embodiment 60 is the method of any of embodiments 44 through 59, further comprising receiving, via a motion sensor of the wearable device, motion information.

Embodiment 61 is the method of any of embodiments 44 through 60, further comprising activating the first imaging sensor based on the motion information.

Embodiment 62 is the method of any of embodiments 44 through 61, wherein the wireless connection is a short-range wireless connection.

Embodiment 63 is the method of any of embodiments 44 through 62, wherein the wearable device includes smartglasses.

Embodiment 64 is the method of any of embodiments 44 through 63, wherein the computing device includes a smartphone.

Embodiment 65 is a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 44 through 64.

Embodiment 66 is a wearable device configured to perform any of the embodiments 44 through 64.

Embodiment 67 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any of embodiments 44 through 64.

Embodiment 68 is a non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to receive image data from one imaging sensor on a wearable device, detect, by an image classifier of the wearable device, whether or not an object of interest is included within the image data, the image classifier configured to execute a first machine-learning (ML) model, and transmit, via a wireless connection, the image data to a computing device, the image data configured to be used by the computing device to compute a bounding box dataset using a second ML model.

Embodiment 69 is the non-transitory computer-readable medium of embodiment 68, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to receive, via the wireless connection, the bounding box dataset from the computing device, identify, by an object tracker of the wearable device, an image region in subsequent image data captured by the at least one imaging sensor using the bounding box dataset, and/or transmit, via the wireless connection, the image region to the computing device, the image region configured to be used by the computing device for further image recognition.

Embodiment 70 is the non-transitory computer-readable medium of any of embodiments 68 through 69, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to crop, by the object tracker, the image region from the subsequent image data and/or compress, by the object tracker, the image region, wherein the compressed image region is transmitted to the computing device over the wireless network.

Embodiment 71 is the non-transitory computer-readable medium of any of embodiments 68 through 70, wherein the object of interest includes a barcode or text.

Embodiment 72 is the non-transitory computer-readable medium of any of embodiments 68 through 71, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to activate a first imaging sensor of the wearable device to capture first image data, detect, by the image classifier, whether the first image data includes the object of interest, and/or activate a second imaging sensor to capture second image data, the second image data having a quality higher than the quality of the first image data, wherein the second image data is transmitted to the computing device via the wireless connection, the second image data configured to be used by the computing device for further image recognition.

Embodiment 73 is the non-transitory computer-readable medium of any of embodiments 68 through 72, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to compress the second image data, wherein the compressed image data is transmitted to the computing device via the wireless connection.

Embodiment 74 is the non-transitory computer-readable medium of any of embodiments 68 through 73, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to receive light condition information from a light condition sensor of the wearable device and/or determine whether to transmit the second image data based on the light condition information.

Embodiment 75 is the non-transitory computer-readable medium of any of embodiments 68 through 74, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to receive motion information from a motion sensor of the wearable device, and determine whether to transmit the second image data based on the motion information.

Embodiment 76 is a wearable device for distributed image recognition, the wearable device comprising at least one imaging sensor configured to capture image data, an image classifier configured to detect whether or not an object of interest is included within the image data, the image classifier configured to execute a first machine-learning (ML) model, and a radio frequency (RF) transceiver configured to transmit, via a wireless connection, the image data to a computing device, the image data configured to be used by the computing device to compute a bounding box dataset using a second ML model.

Embodiment 77 is the wearable device of embodiment 76, wherein the RF transceiver is configured to receive, via the wireless connection, the bounding box dataset from the computing device, the wearable device further including an object tracker configured to identify an image region in subsequent image data captured by the at least one imaging sensor using the bounding box dataset, wherein the RF transceiver is configured to transmit, via the wireless connection, the image region to the computing device, the image region configured to be used by the computing device for further image recognition.

Embodiment 78 is the wearable device of any of embodiments 76 through 77, wherein the wearable device further comprises a sensor trigger configured to activate a first imaging sensor to capture first image data, the image classifier is configured to detect whether the first image data includes the object of interest, the sensor trigger configured to activate a second imaging sensor to capture second image data in response to the object of interest being detected in the first image data, the second image data having a quality higher than the quality of the first image data, wherein the RF transceiver is configured to transmit the second image data to the computing device over the wireless connection.

Embodiment 79 is a computing device for distributed image recognition, the computing device including at least one processor, and a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to receive, via a wireless connection, image data from a wearable device, the image data having an object of interest detected by an image classifier executing a first machine-learning (ML) model, compute a bounding box dataset based on the image data using a second ML model, and transmit, via the wireless connection, the bounding box dataset to the wearable device.

Embodiment 80 is the computing device of embodiment 79, wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to receive, via the wireless connection, an image region in subsequent image data, and/or execute, by the second ML model, object recognition on the image region.

Embodiment 81 is a method for distributed hot-word recognition using a wearable device including receiving, via a microphone of the wearable device, audio data, detecting, by a voice command detector of the wearable device, a presence of a portion of a hot-word from the audio data, the voice command detector executing a first machine-learning (ML) model, storing, in a buffer of the wearable device, the audio data that is received via the microphone in response to the portion of the hot-word being detected, and transmitting, via a wireless connection, a portion of the audio data included in the buffer to a computing device, the portion of the audio data configured to be used by the computing device to perform hot-word recognition using a second ML model.

Embodiment 82 is the method of embodiment 81, further comprising transmitting, via the wireless connection, a remaining portion of the audio data included in the buffer to the computing device.

Embodiment 83 is the method of any of embodiments 81 through 82, further comprising receiving, via the wireless connection, an action command from the computing device, the action command causing the wearable device to perform an action.

Embodiment 84 is the method of any of embodiments 81 through 83, further comprising receiving, via the wireless connection, a disarm command from the computing device and/or discarding the audio data included in the buffer in response to the disarm command.

Embodiment 85 is a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 81 through 84.

Embodiment 86 is a wearable device configured to perform any of the embodiments 81 through 84.

Embodiment 87 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any of embodiments 81 through 84.

Embodiment 88 is a method for sensing image data with multi-resolution using a wearable device, the method comprising activating a first imaging sensor of the wearable device to capture first image data, detecting, by a classifier of the wearable device, whether or not a region of interest (ROI) is included within the first image data, the classifier executing a first machine-learning (ML) model, activating a second imaging sensor of the wearable device to capture second image data in response to the ROI being detected within the first image data, the second image data having a resolution higher than a resolution of the first image data, and transmitting, via a wireless connection, the second image data to a computing device, the second image data configured to be used by the computing device for image processing using a second ML model.

In this specification and the appended claims, the singular forms “a,” “an” and “the” do not exclude the plural reference unless the context clearly dictates otherwise. Further, conjunctions such as “and,” “or,” and “and/or” are inclusive unless the context clearly dictates otherwise. For example, “A and/or B” includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to represent exemplary functional relationships and/or physical or logical couplings between the various elements. Many alternative or additional functional relationships, physical connections or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the embodiments disclosed herein unless the element is specifically described as “essential” or “critical”.

Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that a precise value or range thereof is not required and need not be specified. As used herein, the terms discussed above will have ready and instant meaning to one of ordinary skill in the art.

Moreover, use of terms such as up, down, top, bottom, side, end, front, back, etc. herein are used with reference to a currently considered or illustrated orientation. If they are considered with respect to another orientation, it should be understood that such terms must be correspondingly modified.

Further, in this specification and the appended claims, the singular forms “a,” “an” and “the” do not exclude the plural reference unless the context clearly dictates otherwise. Moreover, conjunctions such as “and,” “or,” and “and/or” are inclusive unless the context clearly dictates otherwise. For example, “A and/or B” includes A alone, B alone, and A with B.

Although certain example methods, apparatuses and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It is to be understood that terminology employed herein is for the purpose of describing particular aspects and is not intended to be limiting. On the contrary, this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.