Audio based projector control

This disclosure proposes systems and methods to perform audio based projector control by a projector device to detect that a user is close to a projector and disable the projector when certain conditions are detected. The projector device may detect the user by performing voice activity detection (VAD) and/or breathing activity detection (BAD) on reference audio data generated by a reference microphone by the projector. In some examples, the projector device may determine that the user is located near the projector by performing beamforming on input audio data and/or comparing a first signal strength of the reference audio data to a second signal strength of the input audio data. The projector device may also implement two level device peripheral control architecture to provide additional safeguards to ensure that control mechanisms are in place even if a central processing unit (CPU) stops operating normally.

BACKGROUND

With the advancement of technology, the use and popularity of electronic devices has increased considerably. Electronic devices are commonly used to capture and send audio data and/or image data.

DETAILED DESCRIPTION

Electronic devices are commonly used to interact with a user by capturing audio data, image data, and/or other input data. Electronic devices may capture the audio data, image data, and/or other input data during a communication session and may provide additional functionality using a projector. For example, a projector device may project an image to enable a graphical interaction between a local user and a remote user during the communication session. Described herein is, among other things, new technology directed to enabling audio based projector control processing to turn off a projector of the projector device when certain conditions are detected.

To improve a user experience with a projector device, this disclosure includes a system that implements audio based projector control to detect that a user is close to a projector and disable the projector when certain conditions are detected. The system may detect the user by performing voice activity detection (VAD) and/or breathing activity detection (BAD) on reference audio data generated by a reference microphone located near the projector. In some examples, the projector device may determine that the user is located near the projector by performing beamforming on input audio data and/or comparing a first signal strength of the reference audio data to a second signal strength of the input audio data.

In addition, the system may implement two level device peripheral control enforcement architecture. The two level control enforcement architecture offers double fault protection, providing additional safeguards to ensure that control mechanisms (such as turning off or reducing the output of peripheral components) are in place even if the CPU stops operating normally. For example, a device may control peripheral components (e.g., fan components, projector, microphones, loudspeaker, etc.) using primary state machines running on the CPU and secondary state machines running on a signal processing unit. The CPU may control the peripheral components using the primary state machines during normal operation, but if the signal processing unit determines that the CPU is no longer operating normally then the signal processing unit may control the peripheral component using the secondary state machines.

FIG. 1illustrates a conceptual diagram of a system configured with audio based projector control according to embodiments of the present disclosure. Although the figures and discussion of the present disclosure illustrate certain operational steps of the system in a particular order, the steps described may be performed in a different order (as well as certain steps removed or added) without departing from the intent of the disclosure. As shown inFIG. 1, a device110may be local to a user5and may include a projector116configured to interact with the user5via a projected space125. In addition, the device110may be part of a broader system100, which may include a remote device20associated with a remote user25.

As illustrated inFIG. 1, the device110and the remote device20may be connected across one or more network(s)199. In some examples, the device110and the remote device20may communicate using a media transport system102, described in greater detail below with regard toFIGS. 3A-4B. Thus, the device110and the remote device20may send and receive audio data, image data, and/or the like via the network(s)199and/or the media transport system102. For example, the remote device20may generate first image data representing the remote user25and/or first audio data representing utterances generated by the remote user25and may send the first image data and/or the first audio data to the device110via the network(s)199. The device110may output first audio corresponding to the first audio data using at least one loudspeaker120(illustrated inFIG. 2) and/or may display a first image corresponding to the first image data using a display (not illustrated) and/or the projector116. However, the disclosure is not limited thereto and the device110may not display the image and/or output the audio without departing from the disclosure.

Similarly, the device110may generate second image data representing the user5using one or more camera(s)122(illustrated inFIG. 2) and/or generate second audio data representing utterances generated by the user5using microphones118(illustrated inFIG. 2) and may send the second image data and/or the second audio data to the remote device20via the network(s)199. The remote device20may output second audio corresponding to the second audio data and/or may display a second image corresponding to the second image data using a display. However, the disclosure is not limited thereto and the device110may not generate the second audio data and/or the second image data without departing from the disclosure.

FIG. 2illustrates an example of a graphical interaction using a shared projected space according to embodiments of the present disclosure. As illustrated inFIG. 2, the system100may enable a graphical interaction200between the user5and the remote user25within a projected space125.

In some examples, the remote device20may generate a user interface that enables the remote user25to input graphical data to the remote device20. For example, the remote device20may enable the remote user25to draw using a touchscreen of the remote device20and/or other input components, although the disclosure is not limited thereto. The remote device20may send the graphical data to the device110and the device110may generate a third image based on the graphical data. For example, the device110may project at least a portion of the graphical data onto the projected space125using the projector116. This enables the user5to interact with the graphical data within the projected space125. For example, the user5may draw on paper positioned across the projected space125, with the graphical data being represented on the paper by the projector116. Thus, the user5and the remote user25may collaborate on a drawing, play a game, and/or the like within the projected space125without departing from the disclosure.

In some examples, the second image data represents at least a portion of the projected space125, such that the remote device20may display the portion of the projected space125to the remote user25. For example, a first camera122aof the device110may capture the second image data representing the user5and the projected space125. However, the disclosure is not limited thereto, and in some examples the device110may generate third image data representing the projected space125using a second camera122b. For example, the first camera122amay capture the second image data representing the user5and the second camera122bmay capture the third image data representing the projected space125without departing from the disclosure. Thus, the device110may include a first camera122aconfigured to capture the user5, a second camera122bconfigured to capture the projected space125, and/or additional camera(s)122without departing from the disclosure. In some examples, the device110may include a specialized camera122, such as an infrared (IR) image sensor, to monitor the projected space125without departing from the disclosure.

As illustrated inFIG. 2, the device110may include at least one loudspeaker120that is configured to generate output audio for the user5. For example, the loudspeaker120may generate first audio corresponding to the first audio data received from the remote device20using at least one loudspeaker120. However, the disclosure is not limited thereto and the device110may include two or more loudspeakers120without departing from the disclosure. In addition,FIG. 2is intended to conceptually illustrate an example of the device110and is not intended to illustrate a specific location of the loudspeaker120on the device110. Thus, location(s) of one or more loudspeaker(s)120may vary without departing from the disclosure.

WhileFIG. 2does not illustrate an example of a display, the device110may include one or more displays configured to generate an output image for the user5. For example, the device110may include a display near the projector116that is configured to generate an output image representing the remote user25without departing from the disclosure. Thus, the device110may enable the graphical interaction200between the user5and the remote user25within the projected space125while also enabling a video interaction between the user5and the remote user25using the display. The disclosure is not limited thereto, however, and in some examples the device110may generate the output image representing the remote user25within the projected space125without departing from the disclosure.

Using the audio data, image data, graphical data, and/or the like sent and received between the device110and the remote device20, the system100may enable the user5and the remote user25to see and/or hear each other while also interacting with each other within the projected space125. Thus, the system100may be configured to provide greater functionality during a communication session, enabling the user5to interact with the remote user25in new and creative ways.

As illustrated inFIG. 2, in some examples a second user210local to the device110may facilitate the communication session for the user5. For example, if the user5is a child, the user210may be a local caregiver (e.g., parent) that sets up and/or authorizes the communication session, such as by using a smart phone or other mobile device. However, the disclosure is not limited thereto and the device110may be configured to initiate and/or configure the communication session independently from a mobile device.

To protect the user5during the communication session, the device110may implement safeguards to reduce potential concerns associated with the device110and/or components of the device110. For example, the device110may be configured to detect unsafe conditions in which the projector116may shine a bright light in the user's face and may turn off the projector as a precaution, although the disclosure is not limited thereto.

In some examples, the device110may include a signal processing unit114configured to perform audio based projector control processing to disable (e.g., turn off) the projector116when certain conditions are present. To illustrate an example, the signal processing unit114may detect that the user5is in the projected space125and may turn off the projector116when appropriate to avoid the projector116shining a bright light into the user's face. For example, the signal processing unit114may use input audio data to detect when certain conditions are present, which may indicate presence of the user5in the projected space125, and may turn off the projector116in response to detecting these conditions.

As the device110is configured to enable a graphical interaction using the shared projected space125, the user5may move within the projected space125while drawing, gesturing, and/or performing other movements. Thus, while the device110may be capable of detecting the presence of the user5using motion sensors or other techniques known to one of skill in the art, these techniques may negatively affect a user experience as the device110may detect the user5and turn off the projector even when the user5is interacting normally with the projected space125. For example, using motion sensors and/or the like, the device110may detect the presence of an arm or a hand of the user5while the user5is drawing in the projected space125, which is not associated with undesirable conditions involving the projector116.

In order to implement additional control mechanisms associated with the projector116, the signal processing unit114may process audio data to detect the presence of the user's face within and/or in proximity to the projected space125. For example, the signal processing unit114may process first audio data captured by the microphones118on top of the device110and/or second audio data captured by the reference microphone118rpositioned below the projector116to enable audio based projector control processing, as described in greater detail below with regard toFIG. 7. When the signal processing unit114determines that a condition is satisfied (e.g., detects that unsafe conditions are present), the signal processing unit114may turn off the projector116.

As illustrated inFIG. 1, the device110may receive (130) first image data from the remote device20during a communication session and may project (132) a first image using the first image data. For example, the device110may project the first image into the projected space125. The device110may then generate (134) first audio data using at least a first microphone and may send (136) the first audio data to the remote device20as part of the communication session. For example, the device110may include one or more microphones118at a first position on the device110, which are illustrated inFIGS. 1-2on a top face of the device110.

To detect presence of the user5, the device110may generate (138) second audio data using a second microphone. For example, the second microphone may be a reference microphone118rat a second position on the device110, which is illustrated inFIGS. 1-2on a side face of the device110below the projector116. The device110may detect (140) potentially unsafe conditions using the second audio data and may turn (142) off the projector when the unsafe conditions are detected.

To illustrate a simple example, the signal processing unit114may perform voice activity detection (VAD) processing on the second audio data generated by the reference microphone118rand may turn off the projector116when voice activity is detected and a signal strength of the second audio data satisfies a first condition (e.g., above a threshold value). Additionally or alternatively, the signal processing unit114may perform breathing activity detection (BAD) processing on the second audio data and may turn off the project116when breathing activity is detected and the signal strength of the second audio data satisfies the first condition.

In some examples, the signal processing unit114may perform VAD and/or BAD processing on the second audio data and may turn off the projector116when (i) voice activity and/or breathing activity is detected and (ii) a first signal strength of the second audio data satisfies a second condition (e.g., greater than a second signal strength of the first audio data, greater than the second signal strength multiplied by a coefficient value, etc.). Additionally or alternatively, the signal processing unit114may perform audio beamforming on the first audio data and may determine that human presence is detected when an audio source is detected in a first direction corresponding to the projected space125.

When the device110disables the projector116, the device110may send an indication to the remote device20to indicate that the projector116is disabled and/or to notify the remote user25why the projector116is disabled. For example, the remote device20may generate a notification indicating that the projector116is disabled as a mechanism to prevent the projector116from shining in the user's face. Similarly, if the device110implements other mechanisms that affect the graphical interaction200with the remote user25, the device110may send a notification to the remote device20indicating that a mechanism was activated or otherwise notifying the remote user25of what occurred.

In some examples, the device110may be configured to enable the user210and/or the remote device20to disable the projector116without departing from the disclosure. For example, the user and/or210the remote user25may notice that the user5is doing something potentially unsafe (e.g., moving too close to the projector116, looking directly at the projector116, and/or the like) and may manually disable the projector116using an input command to the device110and/or the remote device20. Thus, the device110may enable the user210and/or the remote user25to control components of the device110to further ensure a safety of the user5without departing from the disclosure. The user210may generate the input command as a voice command to the device110, an input to a device associated with the user210, and/or the like without departing from the disclosure.

To further protect the user5during the communication session, the device110may implement additional safeguards to reduce potential operational concerns associated with the device110and/or components of the device110. For example, the device110may control components of the device110using state machines using techniques known to one of skill in the art. The state machines may be configured to detect that certain conditions are present and perform one or more actions in response to the conditions in order to reduce potential operational concerns. To illustrate an example, the state machines may determine that a component is overheating (e.g., temperature exceeds a threshold value) and may increase a fan speed of a cooling fan associated with the component, although the disclosure is not limited thereto. Thus, the state machines may monitor the components of the device110and/or conditions associated with the device110and may perform one or more action tasks based on the conditions.

In some examples, the device110may implement these state machines using a CPU component112of the device110. To implement additional safeguards to ensure that the control mechanisms remain in place, the system100may implement two level device peripheral control architecture to provide a fallback system if the CPU component112fails. For example, the device110may include a second set of control mechanisms implemented by a signal processing unit114included in the device110. Thus, the CPU component112may implement primary state machines and the signal processing unit114may implement secondary state machines that provide redundancy in case of CPU failure.

In addition to implementing the secondary state machines, the signal processing unit114may be used to perform audio signal processing, image signal processing, and/or the like for the device110. In some examples, the signal processing unit114may be configured to implement the audio based projector control processing described above to disable (e.g., turn off) the projector116when certain conditions are present. The signal processing unit114may perform this audio based projector control processing even when the CPU component112is operating normally and the primary state machines are running on the device110(e.g., primary processor is in control of the device110).

To illustrate an example of additional safeguards implemented by the state machines, the device110may monitor a temperature of the device110and/or the components of the device110and may control one or more fan components to ensure that the device110and/or the components do not overheat, both to ensure proper functionality of the device110and to avoid any danger associated with high temperatures. For example, the device110may control a first fan component associated with a central processing unit (CPU) component112of the device110, a second fan component associated with the projector116, and/or any additional fans (e.g., general cooling fans located within the device110).

In some examples, the device110may set a fan speed of an individual fan component based on a temperature measured at a specific location within the device110. For example, the device110may monitor a first temperature of the CPU component112and control a fan speed of the first fan component based on the first temperature, may monitor a second temperature of the projector116and control a fan speed of the second fan component based on the second temperature, and/or the like. However, the disclosure is not limited thereto and the device110may control the fan components using any technique known to one of skill in the art.

To illustrate another example of control mechanisms, the device110may be configured to monitor the projector116and/or the projected space125to ensure that the projector116does not shine in the eyes of the user5. For example, the device110may turn off the projector116if a human face is detected in close range to the projector116, if a human face is detected within the projected space125, if movement of the device110is detected (e.g., using an accelerometer or other sensor configured to detect tilt, lift, or movement), if a voice signal is associated with the bottom of the device110(e.g., using beamforming or a second microphone118positioned at the bottom of the device110), if a reflective surface is detected within the projected space125, and/or the like. WhileFIG. 2illustrates multiple microphones118positioned at a top surface of the device110and a single microphone118positioned near a bottom surface of the device110, the disclosure is not limited thereto. Instead, the device110may include two or more microphones118near the bottom surface, may have zero microphones118near the bottom surface, and/or may include additional microphones118not illustrated inFIG. 2without departing from the disclosure.

To illustrate a further example, the device110may be configured to monitor the microphones118and/or the loudspeaker120to ensure that the input and/or the output is within a safe range. For example, the device110may run auto-calibration audio algorithms to ensure that the input is within the safe range and no unsafe gains are added to the input (e.g., to control a volume level of the second audio data that will be sent to the remote device20). Additionally or alternatively, the device110may monitor an output voltage and output current associated with the loudspeaker120to calculate an output level of the output audio in decibels. Based on the output level of the output audio, the device110may disable (e.g., shutdown) the loudspeaker120, clip the volume of the first audio data being sent to the loudspeaker120, and/or other signal processing in order to improve device and human ear comfort.

While the control mechanisms described above may reduce potential operational concerns, these control mechanisms may be circumvented if the CPU component112fails for any reason. To implement additional safeguards to ensure that the control mechanisms remain in place, the system100may implement two level device peripheral control architecture to provide a fallback system if the CPU component112fails. For example, the device110may include a second set of control mechanisms implemented by a signal processing unit114included in the device110. Thus, the device110may control the peripheral components (e.g., fan components, projector116, microphones118, loudspeaker120, and/or the like) using primary state machines running on the CPU component112and secondary state machines running on the signal processing unit114. If the CPU component112is operating normally, the primary state machines control the peripheral components, but if the CPU component112stops operating normally, the secondary state machines take control of the peripheral components to ensure that the safety/control mechanisms remain in place.

In some examples, the primary state machines may prioritize a user experience, user interface, and/or the like and may implement complex control mechanisms (e.g., advanced state machines). For example, the primary state machines may control the fan components based on a measured temperature, as described in greater detail above, in order to reduce a noise that the user5may find unpleasant or distracting. While this may improve a user experience and reduce distractions to the user5, this complexity may cause issues and/or result in control concerns. To improve the overall operation of the device110, the secondary state machines may be intentionally simple (e.g., simple state machines) in order to increase robustness and ensure operation within desired parameters. For example, instead of dynamically controlling a fan speed of an individual fan component, the secondary state machines may set the fan component to a sufficiently high speed to ensure that overheating does not occur. Additionally or alternatively, after a duration of time has elapsed, the secondary state machines may disable (e.g., shutdown) the peripheral components without departing from the disclosure.

FIGS. 3A-3Billustrate example component diagrams of a media transport system configured to perform media processing according to embodiments of the present disclosure. As illustrated inFIG. 3A, a skill component305(e.g., specific skill configured to support communication sessions on the device110) may interact with a media transport system102to request and utilize resources available within the media transport system102. For example, the skill component305may correspond to an application (e.g., process, skill, and/or the like) running on a local device (e.g., device110) and/or one or more servers, and the skill component305may enable a user5to interact with the media transport system102to initiate and manage a communication session involving media processing, although the disclosure is not limited thereto. To illustrate an example, the user5may input a command to an application programming interface (API) for the skill component305that is running on the device110. The device110may send a request corresponding to the command to the one or more servers associated with the skill component305and the one or more servers may send the request to the media transport system102.

In some examples, the skill component305may be developed (e.g., programmed) by an internal client or other development team (e.g., developer, programmer, and/or the like) to perform specific functionality. Thus, the skill component305may be designed to utilize specific resources available within the media transport system102and a finished product is made available to the public (e.g., end-user such as user5). For example, the skill component305may enable the user5to initiate and/or participate in a communication session (e.g., group conference call, such as videoconferencing), to consume media content (e.g., streaming video data) with unique functionality or processing, and/or perform additional functionality (e.g., perform computer vision processing on image data, speech processing on audio data, machine learning, and/or the like) without departing from the disclosure. In this example, the media transport system102provides a simplified interface that enables the internal client to utilize resources within the skill component305, but the interface and/or resources are not visible to and/or customizable by the end-user that uses the skill component305.

The disclosure is not limited thereto, however, and in other examples the skill component305may be made available for external development to third party clients and/or to individual users. Thus, the media transport system102may provide a simplified interface for unique programming without technical expertise. For example, an individual user5may customize the skill component305using a drag and drop graphical user interface (GUI) to enable unique functionality, enabling the user5to program custom routines, skills, and/or the like. To illustrate an example, the user5may customize the skill component305to receive image data generated by an image sensor, process the image data using computer vision, and then perform specific action(s). For example, the skill component305may be programmed so that when a device (e.g., doorbell camera) detects motion and captures image data, the skill component305processes the image data using facial recognition to detect authorized users (e.g., family members or other invited guests) and either performs a first action (e.g., unlock the front door when an authorized user is detected) or performs a second action (e.g., send a notification to the user5including image data representing an unauthorized user). Thus, the interface and/or resources associated with the media transport system102may be visible to and/or customizable by the end-user that uses the skill component305without departing from the disclosure.

To enable the skill component305to request and utilize resources from within the media transport system102, the media transport system102may include a media session orchestrator (MESO) component310configured to coordinate (e.g., define, establish, manage, etc.) a communication session (e.g., media session).

As illustrated inFIG. 3A, the MESO component310may interface between components that fall within four distinct categories: media processing components320, media routing components330, session signaling components340, and/or gateway components350.

Media processing components320refers to processing media content to enable unique functionality. For example, the media transport system102may provide a hosted back-end that performs media processing on individual streams of data, enabling the skill component305to define and control how media content is processed by the media transport system102. The media processing components320may correspond to real time processing (e.g., data is processed during run-time, such as while streaming video to a user5, during a videoconference, and/or the like) or offline processing (e.g., data is processed and stored in a database for future requests, such as during batch processing) without departing from the disclosure.

The media processing components320may include at least one media control component322and/or at least one media processing unit (MPU)324(e.g., first MPU324a, second MPU324b, etc.). The media control component322may coordinate media processing by sending control data to and/or receiving control data from other components within the media transport system102. For example, the MESO component310may send a request to the media control component322to launch a specific application (e.g., skill, process, etc.) to perform media processing and the media control component322may send an instruction to a corresponding MPU324.

The MPU324may be configured to perform media processing to enable additional functionality. Thus, the MPU324may receive first data and process the first data to generate second data. As part of performing media processing, the MPU324may perform speech processing on audio data and/or image data, perform computer vision processing on image data, modify audio data and/or image data, apply visual effects (e.g., overlay or other graphical element(s)) to image data, and/or the like to enable interesting functionality without departing from the disclosure. For example, the MPU324may generate subtitles (e.g., text data) corresponding to speech represented in image data, may translate the subtitles to a different language, may perform text-to-speech processing to enable additional functionality (e.g., describing visual cues for someone that is visually impaired, replacing dialog with speech in a different language, etc.), may perform voice recognition to identify voices represented in audio data, may perform facial recognition to detect and/or identify faces represented in image data, may perform object recognition to detect and/or identify objects represented in image data, may add a graphical overlay to image data (e.g., censoring portions of the image data, adding symbols or cartoons to the image data, etc.), may perform other processing to media content (e.g., colorize black and white movies), and/or the like without departing from the disclosure.

In some examples, the media transport system102may perform media processing using two or more MPUs324. For example, the media transport system102may perform first media processing using a first MPU324aand perform second media processing using a second MPU324b. To illustrate an example, a communication session may correspond to a video chat implementation that includes image data and audio data and the media transport system102may perform media processing in parallel. For example, the media transport system102may separate the image data and the audio data, performing first media processing on the image data and separately performing second media processing on the audio data, before combining the processed image data and the processed audio data to generate output data. However, the disclosure is not limited thereto, and in other examples the media transport system102may perform media processing in series without departing from the disclosure. For example, the media transport system102may process first image data using the first MPU324a(e.g., first media processing) to generate second image data and may process the second image data using the second MPU324b(e.g., second media processing) to generate output image data. Additionally or alternatively, the media transport system102may perform multiple media processing steps using a single MPU324(e.g., more complex media processing) without departing from the disclosure.

The media transport system102may include media routing components330that are configured to route media (e.g., send data packets) to and from the device(s)110via the network(s)199. For example, the media routing components330may include one or more routing control components332, media relay components334, point of presence selection components336, geographic selection components337, and/or capability selection components338. Examples of media relay components may include a Session Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs) system (e.g., STUN system) and/or a Traversal Using relays around NAT (TURN) system, although the disclosure is not limited thereto. WhileFIG. 3Aillustrates the media routing components330including the point of presence selection components336, geographic selection components337, and/or capability selection components338as separate components, this is for ease of illustration and the disclosure is not limited thereto. Instead, a single component may perform point of presence selection, geographic selection, and/or capability selection without departing from the disclosure.

In some examples, the media transport system102may separate the MPUs324from the network(s)199so that the MPUs324do not have a publicly accessible internet protocol (IP) address (e.g., cannot route outside of a local network). Thus, the system100may use the media relay components334to send the first data from a first device to the MPUs324and/or the second data (e.g., processed data) generated by the MPUs324from the MPUs324to a second device. For example, an individual device110may be associated with a specific TURN server, such that the system100may route data to and from the first device using a first TURN server and route data to and from the second device using a second TURN server.

While the example described above illustrates routing data to and from the media processing components320, the media routing components330may be used to route data separately from the media processing components320without departing from the disclosure. For example, the system100may route data directly between devices110using one or more TURN servers (e.g., TURN system) without departing from the disclosure. Additionally or alternatively, the system100may route data using one or more STUN servers (e.g., STUN system), such as when a device110has a publicly accessible IP address. In some examples, the system may establish communication sessions using a combination of the STUN system and the TURN system without departing from the disclosure. For example, a communication session may be more easily established/configured using the TURN system, but may benefit from latency improvements using the STUN system. Thus, the system100may route data using the STUN system, the TURN system, and/or a combination thereof without departing from the disclosure.

In addition to routing data, the media routing components330also perform topology optimization. For example, the media routing components330may include geographically distributed media relay components (e.g., TURN/STUN servers) to enable the media transport system102to efficiently route the data packets. For example, the media routing components330may include a control plane that coordinates between the media relay components to select an optimum route (e.g., data path) to send the data packets. To illustrate an example, the media routing components330may determine a location of parties in a communication session and determine a data path that bypasses a particular country or chokepoint in the data network. In some examples, the media routing components330may select an enterprise specific route and only use specific connected links associated with the enterprise. Additionally or alternatively, the routing components330may apply machine learning models to further reduce latency by selecting the optimum route using non-geographical parameters (e.g., availability of servers, time of day, previous history, etc.).

While the description of the media relay components334refers to the STUN system and/or the TURN system, the disclosure is not limited thereto. Instead, the media routing components330may use any alternative systems known to one of skill in the art to route the data packets. For example, the media routing components330may use any technique that routes UDP data packets and allows the UDP data packets to traverse the NATs without departing from the disclosure. To illustrate an example, the media routing components330may include UDP packet forwarding and relay devices instead of the TURN system without departing from the disclosure.

The media transport system102may include session signaling components340(e.g., edge signaling, signaling network, etc.) that may be configured to coordinate signal paths (e.g., routing of data packets) and/or a type of data packets sent between the devices110and server(s) within the media transport system102. For example, the session signaling components340may enable the devices110to coordinate with each other to determine how data packets are sent between the devices110. In some examples, a signal path may correspond to a routing table that indicates a particular route or network addresses with which to route data between two devices, although the disclosure is not limited thereto. As illustrated inFIG. 3A, the session signaling components340may support protocols including Session Initiation Protocol (SIP)341, Real-Time Communication (RTC) protocol342(e.g., WebRTC protocol), Alexa Voice Service (AVS) protocol343or other voice user interface protocols, Extensible Messaging and Presence Protocol (XMPP)344, IP Multimedia Core Network Subsystem (IMS)345, H.323 standard 346, and/or the like, although the disclosure is not limited thereto.

The media transport system102may include gateway components350that enable the media transport system102to interface with (e.g., send/receive media content or other data) external networks. As illustrated inFIG. 3A, the gateway components350may include a public switched telephone network (PSTN) gateway352, a mobile carrier gateways354, a social networking gateway356, an IP communication network gateway358, and/or other gateways known to one of skill in the art. WhileFIG. 3Aillustrates the gateway components350including a single gateway for each external network, this is intended for illustrative purposes only and the gateway components350may include multiple gateways for each external network without departing from the disclosure. For example, the gateway components350may include multiple PSTN gateways352having different locations without departing from the disclosure. Additionally or alternatively, a single type of external network may correspond to multiple external networks without departing from the disclosure. For example, the gateway components350may include a first mobile carrier gateway354corresponding to a first mobile carrier network and a second mobile carrier gateway354bcorresponding to a second mobile carrier network without departing from the disclosure. However, the disclosure is not limited thereto and two or more mobile carrier networks may share a mobile carrier gateway354without departing from the disclosure.

To illustrate an example of using the gateway components350, the system100may use the PSTN gateway352to establish a communication session with a PSTN device (e.g., wired/wireless telephone, cellular phone, and/or the like that is associated with a PSTN telephone number) using the PSTN. For example, the system100may use the session signaling components340to send SIP data packets from a device110to a PSTN gateway352. The PSTN gateway352may receive the SIP data packets, convert the SIP data packets to audio data in a different format, and send the audio data to the PSTN device via the PSTN. Thus, the gateway components350may include a plurality of gateways, with each gateway being associated with a specific external network and configured to act as an interface between the media transport system102and the external network.

FIG. 3Billustrates an example of signal paths and data flow between components within the media transport system102. As illustrated inFIG. 3B, the skill component305may send data to a media transport system (MTS) application programming interface (API)360. The MTS API360may include an MTS API gateway component362that receives the data (e.g., request) and sends data to the MESO component310, the media processing components320, the media routing components330, and/or other components. For example,FIG. 3Billustrates the MTS API gateway component362communicating with the MESO component310, the media control component322, and the routing control component332.

As described above with regard toFIG. 3A, the MESO component310may communicate with the media processing components320, the media routing components330, the session signaling components340, and/or the gateway components350. Internal signaling within the media transport system102is represented inFIG. 3Bas dotted lines.

The components within the media transport system102may process the request received from the MTS API gateway362and send data to the MTS API360in response to processing the request. For example, components within the media transport system102may send data to an MTS event bus364of the MTS API360and the MTS event bus364may send data (e.g., event, notification, etc.) to the skill component305. Data sent as part of the MTS interface between the skill component305and the media transport system102is represented inFIG. 3Busing a solid line.

As illustrated inFIG. 3B, the skill component305may communicate with the MPU324. For example, the skill component305may communicate with an MPU pipeline instance326running within the MPU324that includes a skill MPU application328. Thus, the skill component305may communicate directly with the skill MPU application as part of an application interface, which is represented as a dashed line inFIG. 3B. In addition to communicating with the skill component305, the MPU pipeline instance326may send data (e.g., media content) to the devices110, either directly or via the media relay components334.

As used herein, an MPU pipeline instance or any other instance may refer to a specific component that is executing program code; all of the logic associated with the media processing unit is running in memory in a single host, which decreases latency associated with the media processing. For example, conventional techniques for executing asynchronous workflows perform checkpointing to store data in storage components between events. Thus, when a new event occurs, the conventional techniques retrieve the stored session and loads data into the memory, resulting in a large amount of latency. As part of reducing the latency, the media transport system102may use the MESO component310to route triggers and events directly to the MPU pipeline instance that is performing the media processing, enabling the media transport system102to perform media processing in real-time.

Using the MESO component310, the media transport system102allows skills and/or applications to enable unique functionality without requiring the skill/application to independently develop and/or program the functionality. Thus, the media transport system102may offer media processing operations as a service to existing skills/applications. For example, the media transport system102may enable a skill to provide closed captioning or other features without building a closed captioning service. Instead, the media transport system102may route a communication session through an MPU324configured to perform closed captioning. Thus, an MPU324configured to enable a specific feature may be utilized to enable the feature on multiple skills without departing from the disclosure.

As the MESO component310is capable of executing requests and commands with low latency, the media transport system102may utilize multiple components within a single communication session. For example, the media transport system102may combine multiple different components (e.g., MPUs324associated with one or more skills) to piece together a custom implementation enabling a combination of existing features. To illustrate an example, the media transport system102may build back to back SIP user engine that is customizable for a specific implementation. Thus, the MESO component310may mix and match different components and/or features to provide a customized experience.

FIGS. 4A-4Billustrate examples of establishing media connections between devices according to embodiments of the present disclosure. In some examples, an originating device110may have a publicly accessible IP address and may be configured to establish a real-time transport (RTP) protocol communication session directly with a SIP endpoint450. The SIP endpoint450may correspond to a device110, a component within the media transport system102, a gateway component configured to interface with a remote network, and/or a device associated with the remote network itself. To enable the originating device110to establish the RTP communication session, the media transport system102may include Session Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs) system (e.g., STUN system410). The STUN system410may be configured to allow NAT clients (e.g., an originating device110behind a firewall) to setup calls to a Voice over Internet Protocol (VoIP) provider hosted outside of the local network by providing a public IP address, the type of NAT they are behind, and a port identifier associated by the NAT with a particular local port. As illustrated inFIG. 4A, the originating device110may perform (412) IP discovery using the STUN system410and may use this information to set up an RTP communication session414(e.g., UDP communication) between the originating device110and the SIP endpoint450to establish a call.

In some examples, the originating device110may not have a publicly accessible IP address. For example, in some types of NAT the originating device110cannot route outside of the local network. To enable the originating device110to establish an RTP communication session, the media transport system102may include Traversal Using relays around NAT (TURN) system420. The TURN system420may be configured to connect the originating device110to the SIP endpoint450when the originating device110is behind a NAT. As illustrated inFIG. 4B, the originating device110may establish (422) an RTP session with the TURN system420and the TURN system420may establish (424) an RTP session with the SIP endpoint450. Thus, the originating device110may communicate with the SIP endpoint450via the TURN system420. For example, the originating device110may send audio data and/or image data to the media transport system102and the media transport system102may send the audio data and/or the image data to the SIP endpoint450. Similarly, the SIP endpoint450may send audio data and/or image data to the media transport system102and the media transport system102may send the audio data and/or the image data to the originating device110.

In some examples, the system may establish communication sessions using a combination of the STUN system410and the TURN system420without departing from the disclosure. For example, a communication session may be more easily established/configured using the TURN system420, but may benefit from latency improvements using the STUN system410. Thus, the system may use the STUN system410when the communication session may be routed directly between two devices and may use the TURN system420for all other communication sessions. Additionally or alternatively, the system may use the STUN system410and/or the TURN system420selectively based on the communication session being established. For example, the system may use the STUN system410when establishing a communication session between two devices (e.g., point-to-point) within a single network (e.g., corporate LAN and/or WLAN), but may use the TURN system420when establishing a communication session between two devices on separate networks and/or three or more devices regardless of network(s).

When the communication session goes from only two devices to three or more devices, the system may need to transition from the STUN system410to the TURN system420. Thus, if the system anticipates three or more devices being included in the communication session, the communication session may be performed using the TURN system420. Similarly, when the communication session goes from three or more devices to only two devices, the system may need to transition from the TURN system420to the STUN system410.

WhileFIGS. 4A-4Billustrate an RTP communication session being established between the originating device110and the SIP endpoint450, the present disclosure is not limited thereto and the RTP communication session may be established between the originating device110and a gateway component or other device associated with the SIP endpoint450without departing from the present disclosure. Additionally or alternatively, whileFIGS. 4A-4Billustrate examples of enabling communication sessions using the SIP protocol, the disclosure is not limited thereto and the media transport system102may use any protocols known to one of skill in the art.

WhileFIGS. 4A-4Billustrate examples of enabling communication sessions using a data connection (e.g., using Voice over Internet Protocol (VoIP), session initiation protocol (SIP), and/or the like), the disclosure is not limited thereto and the system100may enable communication sessions using any type of network without departing from the disclosure. For example, the media transport system102may enable communication sessions using a cellular connection (e.g., mobile phone network) or other external network without departing from the disclosure. For example, the media transport system102may send instructions (e.g., command data) to endpoints (e.g., caller devices, such as the device110) instructing the endpoint to establish a communication session (e.g., dial a telephone number) in response to the voice command.

FIG. 5illustrates a schematic diagram of an example device implementing audio based projector control architecture according to embodiments of the present disclosure. As illustrated inFIG. 5, the device110may include a main CPU component112, a signal processing unit114, and a plurality of peripheral components that are connected to both the CPU component112and the signal processing unit114via a bus510. For example,FIG. 5illustrates that the CPU component112and the signal processing unit114may be connected to a projector116, a plurality of microphones118(e.g., three or more microphones118along a top surface of the device110and an individual reference microphone118rlocated near a bottom surface of the device110below the projector116), a loudspeaker120, one or more camera(s)122, and/or a fan component515via the bus510. However, the disclosure is not limited thereto, and the number and/or location of the peripheral components may vary without departing from the disclosure. WhileFIG. 5illustrates that the device110includes at least one loudspeaker120, the disclosure is not limited thereto and a number of loudspeakers120may vary without departing from the disclosure.

As illustrated inFIG. 5, the signal processing unit114may be independent from the CPU component112and may be configured to perform signal processing (e.g., audio/video processing capabilities) for the device110. For example, the signal processing unit114may be configured to perform specialized computation tasks such as voice/audio/video processing. Thus, the signal processing unit114is already included in the device110and is already connected to the peripheral components via the bus510.

As described above with regard toFIG. 2, the device110may be configured to enable a graphical interaction using a shared projected space. For example, the device110may receive image data from a remote device20during a communication session and may generate projected space125using the projector116. As illustrated inFIG. 5, the projector116may be configured to project an image onto projected space125, which corresponds to a flat surface positioned below the device110.

During the communication session, the device110may generate image data using one or more camera(s)122, generate audio data using one or more microphones118, and may send the image data and/or the audio data to the remote device20. In some examples, the device110may include a first camera122aconfigured to capture the projected space125, such that the device110may send first image data corresponding to the projected space125to the remote device20. For example, the device110may enable the user5to collaborate on a drawing with the remote user25associated with the remote device20. Additionally or alternatively, the device110may include a second camera122bconfigured to capture the user5, such that the device110may send second image data representing the user5to the remote device20. Thus, the device110may enable the remote user25to see the user5separately from the projected space125. However, the disclosure is not limited thereto and the device110may send the first image data, the second image data, the first image data and the second image data, or no image data without departing from the disclosure.

In some examples, the device110may include one or more microphones118positioned on a first surface of the device110(e.g., top surface, although the disclosure is not limited thereto) that are configured to capture first audio data. In addition, the device110may include a reference microphone118rpositioned on a second surface of the device110(e.g., below the projector116) that is configured to capture second audio data. Thus, the device110may send the first audio data, the second audio data, and/or a combination thereof to the remote device20during the communication session.

During a communication session, the signal processing unit114may perform audio processing and/or image processing (e.g., video processing) for the device110. For example, the signal processing unit114may process the first image data and/or the second image data prior to sending the first image data and/or the second image data to the remote device20. Additionally or alternatively, the signal processing unit114may perform audio processing on the first audio data and/or the second audio data prior to sending the first audio data and/or the second audio data to the remote device20. Examples of audio processing may include audio beamforming, echo cancellation processing (e.g., acoustic echo cancellation (AEC), acoustic interference cancellation (AIC), and/or the like), residual echo suppression (RES) processing, and/or other techniques known to one of skill in the art.

In addition to performing audio processing for the communication session, in some examples the device110may use the signal processing unit114to implement additional control mechanisms. To illustrate an example, the signal processing unit114may perform audio based projector control processing to disable (e.g., turn off) the projector116when certain conditions are present. For example, the signal processing unit114may detect the presence of the user5in the projected space125and may turn off the projector116when appropriate to avoid the projector116shining a bright light into the user's face.

As the device110is configured to enable a graphical interaction using the shared projected space125, the user5may move within the projected space125while drawing, gesturing, and/or performing other movements. Thus, while the device110may be capable of detecting the presence of the user5using motion sensors or other techniques known to one of skill in the art, these techniques may negatively affect a user experience as the device110may detect the user5and turn off the projector even when the user5is interacting normally with the projected space125. For example, using motion sensors and/or the like, the device110may detect the presence of an arm or a hand of the user5while the user5is drawing in the projected space125, which is not associated with an operational concern involving the projector116.

In order to implement additional control mechanisms associated with the projector116, the signal processing unit114may process audio data to detect the presence of the user's face within and/or in proximity to the projected space125. For example, the signal processing unit114may process first audio data captured by the microphones118on top of the device110and/or second audio data captured by the reference microphone118rpositioned below the projector116to enable audio based projector control, as described in greater detail below with regard toFIG. 7. To illustrate a simple example, the signal processing unit114may perform voice activity detection (VAD) processing on the second audio data generated by the reference microphone118rand may turn off the projector116when voice activity is detected and a signal strength of the second audio data satisfies a first condition (e.g., above a threshold value). Additionally or alternatively, the signal processing unit114may perform breathing activity detection (BAD) processing on the second audio data and may turn off the project116when breathing activity is detected and the signal strength of the second audio data satisfies the first condition.

In some examples, the signal processing unit114may perform VAD and/or BAD processing on the second audio data and may turn off the projector116when (i) voice activity and/or breathing activity is detected and (ii) a first signal strength of the second audio data satisfies a second condition (e.g., greater than a second signal strength of the first audio data, greater than the second signal strength multiplied by a coefficient value, etc.). Additionally or alternatively, the signal processing unit114may perform audio beamforming on the first audio data and may determine that human presence is detected when an audio source is detected in a first direction corresponding to the projected space125.

The bus510enables the CPU component112and/or the signal processing unit114to control all of the peripheral components, either directly or indirectly. During normal operation, the CPU component112may be configured to control the peripheral components using advanced state machines, as will be described in greater detail below. However, as an additional safeguard, if the CPU component112stops functioning normally the signal processing unit114may be configured to control the peripheral components using simple state machines. Thus, the device110may configure the signal processing unit114to run simple state machines in order to provide additional safeguards (e.g., double fault protection) for the device110.

FIG. 6illustrates an example of advanced state machines and simple state machines included in a projector device according to embodiments of the present disclosure. As illustrated inFIG. 6, the system100may implement double-fault protection600by including primary state machines610that are running on the CPU component112as well as secondary state machines650that are running on the signal processing unit114.

FIG. 6illustrates examples of the primary state machines610, which include a projector monitoring advanced state machine615, a CPU monitoring advanced state machine620, a microphone monitoring advanced state machine625, and a loudspeaker monitoring advanced state machine630, although the disclosure is not limited thereto. A detailed explanation of the primary state machines610is provided below with regard toFIGS. 19-20.

FIG. 6also illustrates examples of the secondary state machines650, which include a projector monitoring simple state machine655, a CPU monitoring simple state machine660, a microphone monitoring simple state machine665, and a loudspeaker monitoring simple state machine670, although the disclosure is not limited thereto. A detailed explanation of the secondary state machines650is provided below with regard toFIGS. 17-18.

FIG. 7illustrates an example of projector monitoring according to embodiments of the present disclosure. As illustrated inFIG. 7, the device110may use the projector monitoring advanced state machine615to monitor a projector output zone710using a monitoring routine720. For example, the monitoring routine720may monitor (722) the projector output zone710for human presence and may determine (724) whether presence is detected. If presence is not detected, the monitoring routine720may loop to step722and continue monitoring the project output zone710. If presence is detected (e.g., conditions are satisfied), the monitoring routine720may turn (726) off the projector116to avoid shining light in the user's face.

WhileFIG. 7illustrates the projector monitoring advanced state machine615performing the monitoring routine720, these steps may be performed by the signal processing unit114, as described above. For example, the signal processing unit114may perform audio processing to monitor the projector output zone710and determine whether human presence is detected. However, the disclosure is not limited thereto, and the signal processing unit114may perform the monitoring routine720separately from the projector monitoring advanced state machine615without departing from the disclosure.

FIG. 8illustrates an example of performing audio beamforming according to embodiments of the present disclosure. As illustrated inFIG. 8, the device110may generate first audio data using the microphones118positioned on a first surface of the device110(e.g., top surface) and may perform audio beamforming using the first audio data to generate directional audio810.

In audio systems, beamforming refers to techniques that are used to isolate audio from a particular direction in a multi-directional audio capture system. Beamforming may be particularly useful when filtering out noise from non-desired directions. Beamforming may be used for various tasks, including isolating voice commands to be executed by a speech-processing system or identifying a direction associated with an audio source.

One technique for beamforming involves boosting audio received from a desired direction while dampening audio received from a non-desired direction. In one example of a beamformer system, a fixed beamformer unit employs a filter-and-sum structure to boost an audio signal that originates from the desired direction (sometimes referred to as the look-direction) while largely attenuating audio signals that original from other directions. A fixed beamformer unit may effectively eliminate certain diffuse noise (e.g., undesireable audio), which is detectable in similar energies from various directions, but may be less effective in eliminating noise emanating from a single source in a particular non-desired direction. The beamformer unit may also incorporate an adaptive beamformer unit/noise canceller that can adaptively cancel noise from different directions depending on audio conditions.

In the example illustrated inFIG. 8, the device110may distinguish between four different directions (e.g., beams). Thus, the device110may determine whether speech or other audible sounds are associated with a first direction (e.g., Direction 1, from 0-90 degrees using the top ofFIG. 8as a reference point), a second direction (e.g., Direction 2, from 90-180 degrees), a third direction (e.g., Direction 3, from 180-270 degrees), or a fourth direction (e.g., Direction 4, from 270-360 degrees). For purposes of detecting human presence in the projector output zone710, the device110may determine whether an audio source is located in the first direction or the second direction (e.g., 0-180 degrees, in front of the projector116), as described in greater detail below. Thus, the device110may ignore audible sounds associated with the third direction or the fourth direction (e.g., 180-360 degrees, opposite the projector116) as this is outside of the projector output zone710.

WhileFIG. 8illustrates an example of performing beamforming to distinguish between four different directions, the disclosure is not limited thereto and the device110may perform beamforming using any number of directions (e.g., beams) without departing from the disclosure. For example, the device110may perform beamforming using eight different directions without departing from the disclosure. Using this example, the device110may associate the projector output zone710with Directions 1-4 (e.g., 0-180 degrees) and may ignore Directions 5-8 (e.g., 180-360 degrees). However, the disclosure is not limited thereto, and the device110may instead associate the projector output zone710with Directions 2-3 (e.g., 45-135 degrees), while ignoring Directions 1 and 4-8 (e.g., 0-45 degrees and 135-360 degrees), without departing from the disclosure.

FIG. 9illustrates an example of detecting human presence near the projected space according to embodiments of the present disclosure. As illustrated inFIG. 9, the device110may distinguish between four different directions (e.g., beams). For purposes of detecting human presence in the projector output zone710, the device110may determine whether an audio source is located in the first direction or the second direction (e.g., 0-180 degrees, in front of the projector116), represented as directional audio910. Thus, the device110may ignore audible sounds associated with the third direction or the fourth direction (e.g., 180-360 degrees, opposite the projector116) as this is outside of the projector output zone710.

As discussed above, the directional audio910is intended to conceptually illustrate an example of beamforming and the disclosure is not limited thereto. Instead, the number of directions may vary without departing from the disclosure, such that the device110may generate directional audio with eight or more directions without departing from the disclosure. Additionally or alternatively, the device110may associate any number of directions with the projector output zone710without departing from the disclosure. In some examples, the device110may associate half of the directions (e.g., 0-180 degrees) with the projector output zone710and may ignore audible sounds originating from the other directions (e.g., 180-360 degrees). However, the disclosure is not limited thereto, and the device110may associate any range of directions with the projector output zone710without departing from the disclosure.

In addition to determining that the audible sound is associated with the first direction or the second direction (e.g., Direction 1 or Direction 2, corresponding to 0-180 degrees), the device110may also determine whether audio input920is detected by the reference microphone118rand may perform additional processing on the audio input920. For example, the device110may perform voice activity detection (VAD) processing to determine whether voice activity is detected in the audio input920. Additionally or alternatively, the device110may perform breathing activity detection (BAD) processing to determine whether breathing activity is detected in the audio input920. In some examples, the device110may compare a first signal strength of the audio input920to a second signal strength of audio data generated by the microphones118on the top surface of the device110, although the disclosure is not limited thereto.

FIG. 10is a flowchart conceptually illustrating an example method for turning off a projector in response to detecting human presence according to embodiments of the present disclosure. As illustrated inFIG. 10, the device110may receive (1010) first audio at a using a microphone array and receive (1012) second audio data from a reference microphone. For example, the microphone array may correspond to microphones118positioned on a first surface of the device110(e.g., top surface) and the reference microphone may correspond to the reference microphone118rpositioned on a second surface of the device110(e.g., below the projector116).

The device110may perform (1014) beamforming to the first audio data to identify a direction associated with the audio source and may determine (1016) whether the direction corresponds to a first direction or a second direction (e.g., first beam or a second beam). For example, the device110may determine whether the audio source is located in Direction 1 or Direction 2, as represented by directional audio910illustrated inFIG. 9.

While step1016corresponds to an example in which the beamforming separates the first audio data into four different directions, with the first direction and the second direction corresponding to the projector output zone710, the disclosure is not limited thereto. Instead, the device110may perform beamforming using any number of different directions, may associate different and/or additional directions with the projector output zone710, and/or the like without departing from the disclosure. Thus, step1016may correspond to identifying that the direction associated with the audio source corresponds to the projector output zone710even if the direction does not correspond to the first direction or the second direction without departing from the disclosure.

If the audio source is not located in the first direction or the second direction, the device110may determine that human presence is not detected in the projector output zone710and may loop to step1010to continue monitoring while leaving the projector116operating normally. If the device110determines that the audio source is located in the first direction or the second direction in step1016, however, this indicates that the audio source may be in the projector output zone710. Thus, the device110may perform (1018) voice activity detection (VAD) processing on the second audio data and determine (1020) whether voice activity is detected. For example, the device110may use voice activity detection to identify that a human is present near the reference microphone118r. If voice activity is not detected, the device110may determine that human presence is not detected in the projector output zone710and may loop to step1010to continue monitoring while leaving the projector116operating normally.

If the device110detects voice activity in the second audio data in step1020, the device110may determine (1022) a first signal strength value of the first audio data, may determine (1024) a second signal strength value of the second audio data, and may determine (1026) whether the first signal strength value is less than the second signal strength value. The device110may determine the first signal strength value and/or the second signal strength value using any techniques to measure a signal strength or other signal quality metric known to one of skill in the art without departing from the disclosure. For example, the device110may determine the first signal strength value based on an amount of energy associated with the first audio data. Additionally or alternatively, the device110may determine the first signal strength value based on a signal-to-noise ratio (SNR) value or other signal quality metric without departing from the disclosure.

If the device110determines that the first signal strength value is greater than the second signal strength value, this indicates that the audio source is closer to the microphone array than to the reference microphone and is therefore unlikely to be in the projector output zone710. Thus, the device110may determine that human presence is not detected in the projector output zone710and may loop to step1010to continue monitoring while leaving the projector116operating normally.

If the device110determines that the second signal strength value is greater than the first signal strength value, this indicates that the audio source is closer to the reference microphone118rthan to the microphone array and is therefore likely to be in the projector output zone710. Thus, the device110may turn (1028) off the projector116to ensure that the projector116is not shining a bright light at a face of the user within the projector output zone710.

WhileFIG. 10illustrates an example of comparing the first signal strength value directly to the second signal strength value in step1026, the disclosure is not limited thereto. In some examples, the device110may multiply the first signal strength value by a coefficient value to determine a third signal strength value and may compare the second signal strength value to the third signal strength value. For example, the device110may multiply the first signal strength value by a factor that reduces the third signal strength value relative to the first signal strength value. The device110may store a fixed value for the coefficient value and the coefficient value may be selected to improve presence detection.

FIG. 11is a flowchart conceptually illustrating an example method for turning off a projector in response to detecting human presence according to embodiments of the present disclosure. The example method illustrated inFIG. 11expands on the example method described above with regard toFIG. 12by adding breathing activity detection (BAD) processing. Thus, some of the steps illustrated inFIG. 12were described above with regard toFIG. 11and a redundant description may be omitted.

As illustrated inFIG. 12, the device110may receive (1010) first audio data using the microphone array, receive (1012) second audio data from the reference microphone, perform (1014) beamforming to the first audio data to identify a direction associated with the audio source, determine (1016) whether the direction corresponds to the first direction or the second direction, and perform (1018) voice activity detection (VAD) processing on the second audio data, as described in greater detail above with regard toFIG. 11.

In addition, the device110may perform (1110) breathing activity detection (BAD) processing on the second audio data and may determine (1112) whether voice activity or breathing activity is detected. Breathing activity has unique features that may enable breathing activity to be detected. For example, normal respiratory sounds are cyclostationary and exhibit repetition of respiratory cycles which includes inspiratory and expiratory phase with inter-respiratory pause. In addition, a range of frequencies is evenly distributed between 100 Hz and 1000 Hz, except for wheezing sounds which are mainly between 250 Hz and 800 Hz. Typically, breathing phases are about 250 ms long.

In some examples, the device110may perform breathing activity detection by extracting potential breathing activity from the second audio data using a bandpass filter with a range between 100 Hz and 1000 Hz. The device110may then analyze the potential breathing activity extracted by the bandpass filter to identify portions of the second audio data in which an energy level exceeds a threshold value for a duration of time (e.g., 200 ms, although the disclosure is not limited thereto). The frequency range, threshold value, and/or the duration of time may vary without departing from the disclosure and may be selected to improve breathing activity detection. When the device110detects that portions of the second audio data are associated with this frequency range (e.g., 100 Hz to 1000 Hz) and have an energy level exceeding the threshold value for the selected duration of time, the device110may determine that breathing activity is represented in the second audio data (e.g., breathing activity is detected).

If the device110determines that neither voice activity nor breathing activity is detected in step1112, the device110may loop to step1010and continue monitoring while leaving the projector116operating normally. If the device110detects voice activity and/or breathing activity in step1112, however, the device110may select (1114) a higher of the voice activity or the breathing activity for additional processing. For example, the device110may select the stronger signal strength associated with the voice activity or the breathing activity, may select a portion of the second audio data corresponding to the voice activity or the breathing activity, and/or the like without departing from the disclosure. The device110may then determine (1022) the first signal strength of the first audio data, determine (1024) the second signal strength of the second audio data, determine (1026) whether the second signal strength is greater than the first signal strength, and may turn (1028) off the projector, as described in greater detail above with regard toFIG. 10.

FIG. 12is a flowchart conceptually illustrating an example method for turning off a projector in response to detecting human presence according to embodiments of the present disclosure. As illustrated inFIG. 12, the device110may receive (1210) first audio data using the microphone array and may receive (1212) second audio data from the reference microphone. The device110may perform (1214) voice activity detection on the second audio data, may perform (1216) breathing activity detection on the second audio data, and may determine (1216) whether voice activity or breathing activity is detected, as described in greater detail above with regard toFIG. 11.

If the device110does not detect voice activity or breathing activity, the device110may loop to step1210to continue monitoring while leaving the projector116operating normally. However, if the device110detects voice activity or breathing activity, the device110may select (1220) a higher signal strength associated with the voice activity or the breathing activity as a first signal strength value of the second audio data and may determine (1222) whether the first signal strength value exceeds a threshold value. If the first signal strength value does not exceed the threshold value, the device110may loop to step1210to continue monitoring while leaving the projector116operating normally. However, if the device110determines that the first signal strength value exceeds the threshold value, the device110may turn (1224) off the projector to avoid shining a bright light in the user's face.

FIGS. 13A-13Bare flowcharts conceptually illustrating example methods for initializing a central processing unit (CPU) component and a signal processing component according to embodiments of the present disclosure. As illustrated inFIG. 13A, in some examples the device110may power (1310) on the CPU component112and may initialize (1312) the CPU component112. For example, the device110may power on the CPU component112in response to a user pressing a power button or other input. The device110may initialize the CPU component112by turning on advanced state machines and/or other processes running on the CPU component112and establishing a first state (e.g., initial state for the CPU component112).

In some examples, the device110may send (1314) a command from the CPU component112to the signal processing unit114to initialize the signal processing unit114, and may initialize (1316) the signal processing unit114in response to the command. Thus, the CPU component112may directly command the signal processing unit114to initialize. The device110may initialize the signal processing unit114by turning on simple state machines and/or other processes running on the signal processing unit114and establishing a second state (e.g., initial state for the signal processing unit114).

The disclosure is not limited thereto, however, and in other examples the signal processing unit114may initialize itself without receiving a command from the CPU component112. As illustrated inFIG. 13B, the device110may power (1310) on the CPU component112and may initialize (1312) the CPU component112, as described above with regard toFIG. 13A. However, after initializing, the CPU component112may wait to receive a signal from the signal processing unit114. For example, the device110may initialize (1320) the signal processing unit114and then may send (1322) a notification from the signal processing unit114to the CPU component112indicating that the signal processing unit114is ready. Thus, the signal processing unit114may independently come online and send an indication to the CPU component112when it is ready to begin executing the simple state machines. As described above, the device110may initialize the signal processing unit114by turning on simple state machines and/or other processes running on the signal processing unit114and establishing a second state (e.g., initial state for the signal processing unit114).

FIGS. 14A-14Bare flowcharts conceptually illustrating example methods for executing advanced state machines and simple state machines according to embodiments of the present disclosure. As illustrated inFIG. 14A, the device110may generate (1410) instruction sets (e.g., configuration file, which may be known in the art as an image) by the CPU component112. The instruction sets may include an operating system and other instructions to execute associated with the CPU component112and/or the signal processing unit114, although the disclosure is not limited thereto.

As illustrated inFIG. 14A, in some examples the device110may send (1412) a request for the instruction sets from the signal processing unit114to the CPU component112and may send (1414) the instruction sets from the CPU component112to the signal processing unit114in response to the request. Thus, in some examples the signal processing unit114may request the instruction sets from the CPU component112, although the disclosure is not limited thereto. For example, the CPU component112may wait for the signal processing unit114to initialize and send the request for the instruction sets and then may directly send the instruction sets to the signal processing unit114.

Using the instruction sets, the device110may execute (1416) the instruction sets using the CPU component112to run advanced state machines and may execute (1418) the instruction sets using the signal processing unit to run simplified state machines (e.g., simple state machines). The device110may initialize the advanced state machines by starting the advanced state machines and establishing a first state (e.g., initial state for the CPU component112). Similarly, the device110may initialize the simple state machines by starting the simple state machines and establishing a second state (e.g., initial state for the signal processing unit114).

The disclosure is not limited thereto, however, and in other examples the CPU component112may distribute the instruction sets to the signal processing unit114indirectly without departing from the disclosure. As illustrated inFIG. 14B, in other examples the device110may generate (1410) the instruction sets (e.g., image) by the CPU component112, may store (1422) the instruction sets in a common area (e.g., within a storage component) using the CPU component112, and may retrieve (1424) the instruction sets by the signal processing unit114. Thus, the CPU component114may store the instruction sets to a common area of the device110to enable the signal processing unit114to retrieve the instruction sets when it is ready to execute the simplified state machines.

Using the instruction sets, the device110may execute (1416) the instruction sets using the CPU component112to run advanced state machines and may execute (1418) the instruction sets using the signal processing unit114to run simplified state machines (e.g., simple state machines), as described above with regard toFIG. 14A.

FIGS. 15A-15Bare flowcharts conceptually illustrating example methods for controlling peripheral components using a CPU component when the CPU component is operating normally and controlling the peripheral components using a signal processor when the CPU component is not operating normally according to embodiments of the present disclosure. As illustrated inFIG. 15A, the device110may control (1510) peripheral components using the CPU component112and advanced state machines.

Periodically, the device110may set a keepalive flag indicating that the CPU component112is operating. For example, the device110may determine (1512) that a predetermined time interval has elapsed and may determine (1514) whether the CPU component112is alive. For example, the device110may determine whether the CPU component112is operating normally (e.g., operating in a first mode), although the disclosure is not limited thereto and this step may be performed implicitly when the CPU component112is operating normally. If the CPU component112is not alive (e.g., operating in a second mode), the method may end after step1514.

If the CPU component112is alive (e.g., operating in the first mode), the device110may set (1516) CPU alive flag(s) to a first value (e.g., first binary value, such as “1”), indicating that the CPU component112is alive and operating normally, and then loop to step1510and repeat steps1510-1516.

WhileFIG. 15Aillustrates the device110performing step1514as an explicit step of determining whether the CPU component112is alive, this is intended to conceptually illustrate how the device110functions when the CPU component112is operating normally or not. However, the disclosure is not limited thereto and in some examples the device110may not explicitly determine whether the CPU is alive in step1514without departing from the disclosure. For example, if the CPU component112is operating normally the device110may implicitly determine that the CPU component112is alive and set the CPU alive flag(s) in step1516. In contrast, if the CPU component112is not operating, the CPU component112may stop updating the CPU alive flag(s) without first making an explicit determination that the CPU component112is not operating without departing from the disclosure.

In some examples, the device110may set a separate CPU alive flag for each state machine of the advanced state machines. For example, the CPU component112may set a first CPU alive flag for the projector state machine (e.g., projector monitoring advanced state machine615), a second CPU alive flag for the CPU state machine (e.g., CPU monitoring advanced state machine620), a third CPU alive flag for the microphone state machine (e.g., microphone monitoring advanced state machine625), and a fourth CPU alive flag for the loudspeaker state machine (e.g., loudspeaker monitoring advanced state machine630), although the disclosure is not limited thereto.

As illustrated inFIG. 15B, the device110may determine (1530) a value of the CPU alive flag(s) and may set (1532) the CPU alive flag(s) to a second value (e.g., second binary value, such as “0”). For example, the device110may perform a clear-on-read operation so that the CPU alive flag(s) reset to the second value every time the CPU alive flag(s) are read. However, the disclosure is not limited thereto and the device110may set the CPU alive flag(s) using any techniques without departing from the disclosure. Additionally or alternatively, the first value and the second value may vary without departing from the disclosure and the first value may correspond to the second binary value and the second value may correspond to the first binary value without departing from the disclosure.

The device110may determine (1534) whether the CPU component112is alive (e.g., CPU alive flag(s) were set to the first value) and, if so, may wait (1536) a duration of time and loop to step1530to repeat steps1530-1534. Thus, while the CPU component112is operating normally in the first mode, the signal processing unit114may continually check the CPU alive flag(s) periodically but leave control of the peripheral components to the CPU component112.

If the device110determines that the CPU component112is not alive in step1534(e.g., CPU component112is operating in the second mode, indicated by the CPU alive flag(s) being set to the second value), the device110may control (1538) the peripheral components using the signal processing unit114and the simplified state machines.

To illustrate a detailed example, the projector state machine for the signal processing unit114(e.g., projector monitoring simple state machine655) may read the first CPU alive flag and determine whether the CPU component112is operating in the first mode. If the first CPU alive flag corresponds to the first value, indicating that the CPU component112is operating normally in the first mode, the projector state machine may set the first CPU alive flag to the second value and wait for the duration of time. Thus, the simple projector state machine may repeatedly reset the first CPU alive flag (e.g., set to the second value) and the advanced projector state machine may repeatedly set the first CPU alive flag (e.g., set to the first value) as long as the CPU component112is operating normally in the first mode.

If the CPU component112stops operating normally in the first mode, the advanced projector state machine will not set the first CPU alive flag to the first value and the simple projector state machine will determine that the CPU component112is not operating in the first mode and will control the peripheral components (e.g., perform action tasks associated with the projector116).

Similarly, the simple CPU state machine will read and reset the second CPU alive flag (e.g., set to the second value) and the advanced CPU state machine will set the second CPU alive flag (e.g., set to the first value), the simple microphone state machine will read and reset the third CPU alive flag (e.g., set to the second value) and the advanced microphone state machine will set the third CPU alive flag (e.g., set to the first value), and the simple loudspeaker state machine will read and reset the fourth CPU alive flag (e.g., set to the second value) and the advanced loudspeaker state machine will set the fourth CPU alive flag (e.g., set to the first value). If any of the simple state machines determine that the CPU alive flag(s) are set to the second value, the simple state machine will take control of a corresponding peripheral component(s) and/or perform action tasks associated with the corresponding peripheral component(s).

WhileFIGS. 15A-15Billustrate examples of the device110setting a CPU alive flag, the disclosure is not limited thereto. In some examples, the CPU component112may send a keepalive message or other indication that the CPU component112is operating normally to the signal processing unit114without departing from the disclosure.

FIGS. 16A-16Bare communication diagrams conceptually illustrating example methods for controlling peripheral components using a CPU component when the CPU component is operating normally and controlling the peripheral components using a signal processor when the CPU component is not operating normally according to embodiments of the present disclosure. As illustrated inFIG. 16A, the CPU component112may control (1610) peripheral components using advanced state machines and may set (1614) the CPU alive flag(s)1602to the first value. In some examples, the CPU component112may set the CPU alive flag(s) as part of a keepalive message to indicate to the signal processing unit114and/or other components that the CPU component112is operating normally. For example, the CPU component112may update a status register, flag register, and/or the like at predefined intervals during normal processing operations to indicate the current state of the processor.

In some examples, the CPU component112may set a single CPU alive flag to the first value and multiple state machines and/or components may refer to the same CPU alive flag. However, the disclosure is not limited thereto, and in other examples the CPU component112may set a separate CPU alive flag for each of the simple state machines without departing from the disclosure, as described in greater detail below.

Periodically, the signal processing unit114may retrieve (1614) values from the CPU alive flag(s)1602, may set (1616) the CPU alive flag(s)1602to a second value, and may determine (1618) that the CPU alive flag(s) indicate that the CPU component112is alive. For example, the signal processing unit114may determine that the values retrieved from the CPU alive flag(s) correspond to the first value, which indicates that the CPU component112is operating normally. Thus, the signal processing unit114may do nothing and enable the CPU component112to continue controlling the peripheral components.

During normal operation, the CPU component112may determine (1620) that a first duration of time has elapsed (e.g., since setting the CPU alive flag(s) in step1612) and may set (1622) the CPU alive flag(s) to the first value. For example, the CPU component112may set the CPU alive flag(s) to the first value at predefined intervals during normal operation.

The signal processing unit114may determine (1624) that a second duration of time has elapsed (e.g., since retrieving the CPU alive flag(s) in step1614), may retrieve (1626) the CPU alive flag(s)1602, may set (1628) the CPU alive flag(s)1602to the second value, and may determine (1630) that the CPU alive flag(s) indicate that the CPU component112is alive. For example, the signal processing unit114may determine that the values retrieved from the CPU alive flag(s) correspond to the first value, which indicates that the CPU component112is operating normally.

FIG. 16Billustrates what happens when the CPU component112stops functioning normally (e.g., operating in the first mode). As illustrated inFIG. 16B, the CPU component112and the signal processing unit114may perform steps1610-1618, as described above with regard toFIG. 16A. In this example, however, the CPU component112does not perform steps1620-1622to set the CPU alive flag(s) to the first value within the second duration of time. Thus, the CPU alive flag(s) correspond to the second value by the signal processing unit114set in step1616.

As illustrated inFIG. 16B, the signal processing unit114may determine (1650) that the second duration of time has elapsed (e.g., since retrieving the CPU alive flag(s) in step1614), may retrieve (1652) values from the CPU alive flag(s)1602, may set (1654) the CPU alive flag(s)1602to a second value, and may determine (1656) that the CPU alive flag(s) indicate that the CPU component112is not alive (e.g., the CPU component112is operating in the second mode). Thus, the signal processing unit114may control (1658) the peripheral components using the simplified state machines.

FIG. 17is a conceptual diagram illustrating an example of transitioning to simple state machines according to embodiments of the present disclosure.FIG. 17illustrates an example of simple state machines1700associated with the signal processing unit114. As illustrated inFIG. 17, the signal processing unit114may start (1705) up normal operation (e.g., initialize) and then may start (1710) a projector monitoring simple state machine, start (1715) a CPU monitoring simple state machine, start (1720) a microphone monitoring simple state machine, and start (1725) a loudspeaker monitoring simple state machine, along with additional simple state machines without departing from the disclosure.

While the signal processing unit114starts the simple state machines1700in steps1710-1725, the simple state machines1700do nothing while the CPU component112is operating normally. This is represented inFIG. 17as the CPU component112setting a CPU-alive flag at a first predefined interval so that the signal processing unit114may determine whether the CPU component112is operating normally. For example, if the CPU component112continues to set the CPU-alive flag1730, the signal processing unit114defers to the CPU component112and the advanced state machines running on the CPU component112. However, if the CPU component112fails to set the CPU-alive flag1730for a duration of time, the signal processing unit114may detect that the CPU-alive flag1730is not set and may take control using the simple state machines1700.

As illustrated inFIG. 17, the CPU component112may set (1730) a CPU-alive flag at a first predefined interval. The projector monitoring simple state machine may check (1735) the CPU-alive flag at a second predefined interval and determine (1755) whether the CPU-alive flag indicates that the CPU component112is operating normally. If the CPU-alive flag is set, indicating that the CPU component112is continuing to set the CPU-alive flag at the first predefined interval, the projector monitoring simple state machine may loop to step1735and continue to check the CPU-alive flag at the second predefined interval. If the CPU-alive flag is not set, indicating that the CPU component112did not set the CPU-alive flag for a duration of time and is therefore not operating normally, the projector monitoring simple state machine may execute (1775) projector action tasks, as described in greater detail below.

The CPU monitoring simple state machine may check (1740) the CPU-alive flag at the second predefined interval and determine (1760) whether the CPU-alive flag indicates that the CPU component112is operating normally. If the CPU-alive flag is set, indicating that the CPU component112is continuing to set the CPU-alive flag at the first predefined interval, the CPU monitoring simple state machine may loop to step1740and continue to check the CPU-alive flag at the second predefined interval. If the CPU-alive flag is not set, indicating that the CPU component112did not set the CPU-alive flag for a duration of time and is therefore not operating normally, the CPU monitoring simple state machine may execute (1780) CPU action tasks, as described in greater detail below.

The microphone monitoring simple state machine may check (1745) the CPU-alive flag at the second predefined interval and determine (1765) whether the CPU-alive flag indicates that the CPU component112is operating normally. If the CPU-alive flag is set, indicating that the CPU component112is continuing to set the CPU-alive flag at the first predefined interval, the microphone monitoring simple state machine may loop to step1745and continue to check the CPU-alive flag at the second predefined interval. If the CPU-alive flag is not set, indicating that the CPU component112did not set the CPU-alive flag for a duration of time and is therefore not operating normally, the microphone monitoring simple state machine may execute (1785) microphone action tasks, as described in greater detail below.

The loudspeaker monitoring simple state machine may check (1750) the CPU-alive flag at the second predefined interval and determine (1770) whether the CPU-alive flag indicates that the CPU component112is operating normally. If the CPU-alive flag is set, indicating that the CPU component112is continuing to set the CPU-alive flag at the first predefined interval, the loudspeaker monitoring simple state machine may loop to step1750and continue to check the CPU-alive flag at the second predefined interval. If the CPU-alive flag is not set, indicating that the CPU component112did not set the CPU-alive flag for a duration of time and is therefore not operating normally, the loudspeaker monitoring simple state machine may execute (1790) loudspeaker action tasks, as described in greater detail below.

In some examples, the CPU component112may set a single CPU-alive flag in step1730and each of the simple state machines1700may check the same CPU-alive flag. For example, one of the simple state machines, the signal processing unit114, and/or a different component may reset the CPU-alive flag at a predefined interval by setting the CPU-alive flag to a first value (e.g., 0). During normal operation, the CPU component112may set the CPU-alive flag to a second value (e.g., 1) at the first predefined interval. Thus, when each of the simple state machines1700checks the CPU-alive flag, detecting the first value indicates that the CPU component112did not set the CPU-alive flag and is not operating normally, while detecting the second value indicates that the CPU component112set the CPU-alive flag and is operating normally.

The disclosure is not limited thereto, however, and in some examples the CPU component112may set multiple CPU-alive flags without departing from the disclosure. For example, the CPU component112may set a CPU-alive flag for each of the simple state machines1700. Thus, the CPU component112may set a first CPU-alive flag for the projector monitoring simple state machine, a second CPU-alive flag for the CPU monitoring simple state machine, a third CPU-alive flag for the microphone monitoring simple state machine, and a fourth CPU-alive flag for the loudspeaker monitoring simple state machine without departing from the disclosure.

To illustrate a detailed example, the CPU component112may set the first CPU-alive flag associated with the projector monitoring simple state machine to the second value at a first time. The projector monitoring simple state machine may check the first CPU-alive flag at a second time, detect that the first CPU-alive flag is set to the second value at the second time, and then may set the first CPU-alive flag to the first value. Thus, the projector monitoring simple state machine interprets the second value as indicating that the CPU component112is operating normally and then resets the first CPU-alive flag to the first value. If the CPU component112fails to set the first CPU-alive flag to the second value within a duration of time, the projector monitoring simple state machine may check the first CPU-alive flag at a third time, detect that the first CPU-alive flag is set to the first value at the third time, interpret the first value as indicating that the CPU component112is not operating normally and then execute the projector action tasks.

FIG. 18is a conceptual diagram illustrating examples of executing simple action tasks according to embodiments of the present disclosure.FIG. 18illustrates an example in which the simple state machines1700operate as described above with regard toFIG. 17, except with additional details about the action tasks performed by the simple state machines1700. Thus,FIG. 18illustrates examples of simple action tasks1800that are performed by the simple state machines1700when the CPU component112does not set the CPU-alive flag(s) within a duration of time. As several components illustrated inFIG. 18are described above with regard toFIG. 17, a redundant description is omitted.

As illustrated inFIG. 18, if the projector monitoring simple state machine determines that the CPU is not alive (e.g., first CPU-alive flag is not set), indicating that the CPU component112is not operating normally, the projector monitoring simple state machine may execute (1875) projector action tasks. For example, the projector monitoring simple state machine may perform simple logic such as running the projector cooling fan at high speed to ensure temperature control, turning off the projector after a first amount of time Tx, and/or the like.

Similarly, if the CPU monitoring simple state machine determines that the CPU is not alive (e.g., second CPU-alive flag is not set), indicating that the CPU component112is not operating normally, the CPU monitoring simple state machine may execute (1880) CPU action tasks. For example, the CPU monitoring simple state machine may perform simple logic such as running the CPU cooling fan at high speed to ensure temperature control, turning off the CPU after a second amount of time Ty, and/or the like.

If the microphone monitoring simple state machine determines that the CPU is not alive (e.g., third CPU-alive flag is not set), indicating that the CPU component112is not operating normally, the microphone monitoring simple state machine may execute (1885) microphone action tasks. For example, the microphone monitoring simple state machine may perform simple logic such as dynamically performing mandatory calibration to ensure that an input volume level does not exceed a maximum volume level, mute the microphone (e.g., turn off the microphone) after a third amount of time Tz, and/or the like.

If the loudspeaker monitoring simple state machine determines that the CPU is not alive (e.g., fourth CPU-alive flag is not set), indicating that the CPU component112is not operating normally, the loudspeaker monitoring simple state machine may execute (1890) loudspeaker action tasks. For example, the loudspeaker monitoring simple state machine may perform simple logic such as dynamically performing mandatory calibration to ensure that an output volume level does not exceed a maximum volume level, mute the loudspeaker (e.g., turn off the loudspeaker) after a fourth amount of time Tp, and/or the like.

FIG. 19is a conceptual diagram illustrating an example of advanced state machines according to embodiments of the present disclosure.FIG. 19illustrates an example of advanced state machines1900associated with the CPU component112. As illustrated in FIG.19, the CPU component112may start (1905) normal operation (e.g., initialize) and then may start (1910) a projector monitoring advanced state machine, start (1915) a CPU monitoring advanced state machine, start (1920) a microphone monitoring advanced state machine, and start (1925) a loudspeaker monitoring advanced state machine, along with additional advanced state machines without departing from the disclosure.

After the CPU component112starts the advanced state machines1900, the advanced state machines1900may control the peripheral components of the device110as long as the CPU component112continues to operate normally. For example, the projector monitoring advanced state machine may check (1930) for aliveness at a first predefined interval and determine (1950) whether the CPU component112is alive. In some examples, steps1930and1950may be inherent steps, such that the projector monitoring advanced state machine continues to operate as long as the CPU component112continues to operate. Thus, the projector monitoring advanced state machine may not explicitly perform steps1930and1950without departing from the disclosure. However, the disclosure is not limited thereto, and in other examples the projector monitoring advanced state machine may actively check to see whether the CPU component112is alive without departing from the disclosure. For example, the projector monitoring advanced state machine may determine whether the CPU component112set a first CPU-alive flag associated with the projector monitoring simple state machine described above with regard toFIG. 17.

If the CPU component112is alive and operating normally, the projector monitoring advanced state machine may execute (1970) projector advanced action tasks, as described in greater detail below with regard toFIG. 20, and may loop to step1930and continue to check for aliveness at the first predefined interval. If the CPU component112is not alive, indicating that the CPU component112is not operating normally (e.g., did not set the first CPU-alive flag for a duration of time and/or the like), the projector monitoring advanced state machine may loop to step1910and start the projector monitoring advanced state machine again (e.g., the CPU component112may restart the projector monitoring advanced state machine).

The CPU monitoring advanced state machine may check (1935) for aliveness at the first predefined interval and determine (1955) whether the CPU component112is alive. In some examples, steps1935and1955may be inherent steps, such that the CPU monitoring advanced state machine continues to operate as long as the CPU component112continues to operate. Thus, the CPU monitoring advanced state machine may not explicitly perform steps1935and1955without departing from the disclosure. However, the disclosure is not limited thereto, and in other examples the CPU monitoring advanced state machine may actively check to see whether the CPU component112is alive without departing from the disclosure. For example, the CPU monitoring advanced state machine may determine whether the CPU component112set a second CPU-alive flag associated with the CPU monitoring simple state machine described above with regard toFIG. 17.

If the CPU component112is alive and operating normally, the CPU monitoring advanced state machine may execute (1975) CPU advanced action tasks, as described in greater detail below with regard toFIG. 20, and then may loop to step1935and continue to check for aliveness at the first predefined interval. If the CPU component112is not alive, indicating that the CPU component112is not operating normally (e.g., did not set the second CPU-alive flag for a duration of time and/or the like), the CPU monitoring advanced state machine may loop to step1915and start the CPU monitoring advanced state machine again (e.g., the CPU component112may restart the CPU monitoring advanced state machine).

The microphone monitoring advanced state machine may check (1940) for aliveness at the first predefined interval and determine (1960) whether the CPU component112is alive. In some examples, steps1940and1960may be inherent steps, such that the microphone monitoring advanced state machine continues to operate as long as the CPU component112continues to operate. Thus, the microphone monitoring advanced state machine may not explicitly perform steps1940and1960without departing from the disclosure. However, the disclosure is not limited thereto, and in other examples the microphone monitoring advanced state machine may actively check to see whether the CPU component112is alive without departing from the disclosure. For example, the microphone monitoring advanced state machine may determine whether the CPU component112set a third CPU-alive flag associated with the microphone monitoring simple state machine described above with regard toFIG. 17.

If the CPU component112is alive and operating normally, the microphone monitoring advanced state machine may execute (1980) microphone advanced action tasks, as described in greater detail below with regard toFIG. 20, and then may loop to step1940and continue to check for aliveness at the first predefined interval. If the CPU component112is not alive, indicating that the CPU component112is not operating normally (e.g., did not set the third CPU-alive flag for a duration of time and/or the like), the microphone monitoring advanced state machine may loop to step1920and start the microphone monitoring advanced state machine again (e.g., the CPU component112may restart the microphone monitoring advanced state machine).

The loudspeaker monitoring advanced state machine may check (1945) for aliveness at the first predefined interval and determine (1965) whether the CPU component112is alive. In some examples, steps1945and1965may be inherent steps, such that the loudspeaker monitoring advanced state machine continues to operate as long as the CPU component112continues to operate. Thus, the loudspeaker monitoring advanced state machine may not explicitly perform steps1945and1965without departing from the disclosure. However, the disclosure is not limited thereto, and in other examples the loudspeaker monitoring advanced state machine may actively check to see whether the CPU component112is alive without departing from the disclosure. For example, the loudspeaker monitoring advanced state machine may determine whether the CPU component112set a fourth CPU-alive flag associated with the loudspeaker monitoring simple state machine described above with regard toFIG. 17.

If the CPU component112is alive and operating normally, the loudspeaker monitoring advanced state machine may execute (1985) loudspeaker advanced action tasks, as described in greater detail below with regard toFIG. 20, and then may loop to step1945and continue to check for aliveness at the first predefined interval. If the CPU component112is not alive, indicating that the CPU component112is not operating normally (e.g., did not set the fourth CPU-alive flag for a duration of time and/or the like), the loudspeaker monitoring advanced state machine may loop to step1925and start the loudspeaker monitoring advanced state machine again (e.g., the CPU component112may restart the loudspeaker monitoring advanced state machine).

FIG. 20is a conceptual diagram illustrating examples of executing advanced action tasks according to embodiments of the present disclosure.FIG. 20illustrates an example in which the advanced state machines1900operate as described above with regard toFIG. 19, except with additional details about the advanced action tasks performed by the advanced state machines1900. Thus,FIG. 20illustrates examples of advanced action tasks2000that are performed by the advanced state machines1900when the CPU component112is operating normally. As several components illustrated inFIG. 20are described above with regard toFIG. 19, a redundant description is omitted.

As illustrated inFIG. 20, if the projector monitoring advanced state machine determines that the CPU is alive, indicating that the CPU component112is operating normally, the projector monitoring advanced state machine may execute (2070) projector advanced action tasks. For example, the projector monitoring advanced state machine may determine if a human face is detected in close range, may determine if the projected space125is detected, determine if an accelerometer detects tile, lift, or movement, and/or the like. If any of these conditions are satisfied, the projector monitoring advanced state machine may turn off the projector116.

Similarly, if the CPU monitoring advanced state machine determines that the CPU is alive, indicating that the CPU component112is operating normally, the CPU monitoring advanced state machine may execute (2075) CPU advanced action tasks, such as setting a CPU cooling fan to a particular speed based on a temperature of the CPU component112. For example,FIG. 20illustrates an example in which the CPU monitoring advanced state machine determines that the CPU temperature is X degrees (e.g., determines a first temperature associated with the CPU component112) and sets the fan speed at X % speed (e.g., sets the CPU cooling fan to a first speed corresponding to the first temperature), determines that the CPU temperature is Y degrees (e.g., determines a second temperature associated with the CPU component112) and sets the fan speed at Y % speed (e.g., sets the CPU cooling fan to a second speed corresponding to the second temperature), and so on.

If the microphone monitoring advanced state machine determines that the CPU is alive, indicating that the CPU component112is operating normally, the microphone monitoring advanced state machine may execute (2080) microphone advanced action tasks. For example, the microphone monitoring advanced state machine may run auto-calibration audio algorithms to ensure that an input is within a safe range and no unsafe gains are added to the input.

If the loudspeaker monitoring advanced state machine determines that the CPU is alive, indicating that the CPU component112is operating normally, the loudspeaker monitoring advanced state machine may execute (2085) loudspeaker advanced action tasks. For example, the loudspeaker monitoring advanced state machine may monitor output voltage and current to calculate audio output level in decibels (dB) and perform audio shutdown or clipping of the output volume for device and human ear comfort.

FIG. 21is a block diagram conceptually illustrating a device110that may be used with the system. Each device110may include one or more controllers/processors2104, which may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory2106for storing data and instructions of the respective device. The memories2106may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive memory (MRAM), and/or other types of memory. Each device110may also include a data storage component2108for storing data and controller/processor-executable instructions. Each data storage component2108may individually include one or more non-volatile storage types such as magnetic storage, optical storage, solid-state storage, etc. Each device110may also be connected to removable or external non-volatile memory and/or storage (such as a removable memory card, memory key drive, networked storage, etc.) through respective input/output device interfaces2102.

Computer instructions for operating each device110and its various components may be executed by the respective device's controller(s)/processor(s)2104, using the memory2106as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory2106, storage2108, or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software.

Each device110includes input/output device interfaces2102. A variety of components may be connected through the input/output device interfaces2102, as will be discussed further below. Additionally, each device110may include an address/data bus2124for conveying data among components of the respective device. Each component within a device110may also be directly connected to other components in addition to (or instead of) being connected to other components across the bus2124.

Referring toFIG. 21, the device110may include input/output device interfaces2102that connect to a variety of components such as an audio output component such as a loudspeaker120, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device110may also include an audio capture component. The audio capture component may be, for example, microphones118or an array of microphones, a wired headset or a wireless headset (not illustrated), etc. If an array of microphones is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array. The device110may additionally include a projector116for projecting content and/or a display2116for displaying content. The device110may further include a camera2118.

The components of the device110may include their own dedicated processors, memory, and/or storage. Alternatively, one or more of the components of the device110may utilize the I/O interfaces2102, processor(s)2104, memory2106, and/or storage2108of the device(s)110.