Patent Description:
Many consumer electronics devices, such as smartphones, media players, tablets, and personal computers, present various forms of media content to users. For example, a consumer electronics device may present visual content to a user via a display, while relying on headphones, built-in speakers, and/or external speakers (e.g., compact, portable speakers) to produce audio content for the user. In particular, headphones deliver audio directly to the ears of a user, while speakers produce surround sound that delivers audio content throughout a large area.

Using either headphones or speakers to present audio content to a user presents multiple drawbacks. In particular, although headphones are capable of providing a high-fidelity audio experience, the form factor of headphones can cause sounds from the user's environment to be blocked from entering the user's ear canal. Accordingly, wearing headphones may isolate the user from important sounds within her environment, thereby reducing the user's ability to interact with her environment effectively. On the other hand, although an external speaker system enables a user to listen to audio content without experience auditory isolated, sound emanating from such a system may be audible to persons around the user. Accordingly, the use of external speakers may be disturbing in various situations.

Furthermore, consumer electronics devices that produce visual content often require users to visually focus on a display in order to receive the visual content. For example, a consumer electronics device may download notifications relating to a user's environment from a networked database. The consumer electronics device may further generate visual content based on the notification and transmit the visual content to the user via the display.

Document <CIT> discloses sound data output and manipulation with haptic feedback. Haptic sensations are associated with sound data to assist in navigating through and editing the sound data. The sound data is loaded into computer memory and played such that sound is output from an audio device. The sound playing is controlled by user input for navigation through the sound data. Haptic commands are generated based on the sound data and are used to output haptic sensations to the user by a haptic feedback device manipulated by the user. The haptic sensations correspond to one or more characteristics of the sound data to assist the user in discerning features of the sound data during the navigation through and editing of the sound data.

Document <CIT> discloses an interface device including at least two actuator assemblies, which each include a moving inertial mass. A single control signal provided to the actuator assemblies at different magnitudes provides directional inertial sensations felt by the user. A greater magnitude waveform can be applied to one actuator to provide a sensation having a direction approximately corresponding to a position of that actuator in the housing.

One drawback of these types of consumer electronics devices is that a user who is not visually focusing on the display may be unaware that a new notification is being displayed by the consumer electronics device. Accordingly, the user may remain uninformed about the notification for extended periods of time. Alternatively, when the user is visually focusing on the display, the user may be unaware of events occurring in the local environment. Thus, the visual content presented to the user may distract the user from interacting effectively with the local environment.

One proposed solution to the above drawbacks is to provide an audio notification to a user when a visual notification is displayed. However, in addition to the above presented disadvantages of headphones and speakers, providing audio notifications via these types of devices may disrupt audio content that is concurrently being presenting by the headphones and/or speakers. Accordingly, conventional approaches for presenting visual content and audio content from a consumer electronics device to a user may distract a user from interacting effectively with the surrounding environment.

As the foregoing illustrates, techniques that present media content to a user more effectively would be useful.

The present invention sets forth a system for providing audio content and haptic sensations to a user. The system includes a device including a first set of transducers and a second set of transducers, wherein the device is configured to be integrated into a neck worn device or a shoulder worn device or is configured to be integrated into or attached to an article of clothing, and wherein the second set of transducers form an ultrasonic transducer array that is configured to generate ultrasonic waves that travel towards a particular body part of the user such that when the ultrasonic wave interacts with the user's skin, the user feels a haptic sensation. The system further includes a processor coupled to the first set of transducers and the second set of transducers, the processor being configured to determine a sound tone and a haptic sensation to deliver to the user, select a first location at which to deliver audio output corresponding to the sound tone and a second location to which to deliver haptic output corresponding to the haptic sensation, configure the first set of transducers to deliver the audio output to the first location, and configure the second set of transducers to generate the haptic output for delivery to the second location. The second location differs from a location of the second set of transducer.

The present invention provides, among other things, a method and a non-transitory computer-readable medium configured to implement the system set forth above.

At least one advantage of the disclosed techniques is that audio can be transmitted directly to the ears of a user, enabling the user to listen to audio content (e.g., music, voice conversations, notifications, etc.) without disturbing sound-sensitive objects around him or her. Additionally, because the device may be shoulder-mounted, the system does not isolate the user from sounds in his or her environment. Further, the device may interact with a user by generating haptic sensations on and/or proximate to the user. Accordingly, the device may generate haptic sensations that provide indicates to the user (e.g., indicating the location of virtual objects in space).

In the following description, numerous specific details are set forth to provide a more thorough understanding of the embodiments of the present disclosure.

<FIG> illustrates a block diagram of a device <NUM> configured to implement one or more aspects of the present invention, according to various embodiments. As shown, device <NUM> includes, without limitation, sensors <NUM>, transducers <NUM>, and computing device <NUM>. Device <NUM> is implemented as wearable device integrated into a neck and/or shoulder worn devices. Additionally or alternatively, device <NUM> may be integrated into a user's clothing (e.g., a shirt, a jacket, a sweater, etc.). Device <NUM> may also include a hook or attachment piece that facilitates device <NUM> to be attached to a user's clothing. The attachment piece may include Velcro®, an adhesive, and so forth.

Sensors <NUM> include devices capable of detecting the position of objects in an environment. Sensors <NUM> may include, without limitation, one or more of RGB imagers and cameras, depth sensors, acoustic and ultrasonic sensors, laser and thermal based sensors, infrared sensors, and structured light cameras with projection units that direct structured light to a target (e.g., a body part of a user). In some embodiments, transducers <NUM> may operate as a sensor <NUM>, as described in further detail below. In particular, in such embodiments, sensors <NUM> may include transducer devices, such as ultrasound transducers.

In operation, sensors <NUM> may generate sensor data reflective of the location of an object in the environment. The object may include various body parts of a user. In particular, the object may include an ear, a face, a chin, a temple, a neck, a shoulder, etc. The object may also include a hand, a finger, a palm, a forearm, an elbow, and arm, a leg, a foot, a calf, a torso, a back, and so forth. Sensors <NUM> may be positioned so that a user's ear is within a field-of-view of one or more of sensors <NUM>. Similarly, a second set of sensors may be positioned so that a second body part (e.g., a hand) is also within a field-of-view of sensors <NUM>. Accordingly, sensors <NUM> may be positioned to track the position of multiple objects in an environment. In particular, in some embodiments, sensors <NUM> may be positioned on a body worn object, such as a neck or shoulder worn device. Additionally or alternatively, sensors may be located on a fixed surface, such as a wall, a ceiling, a floor, and/or other fixed locations in an environment that facilitate sensors <NUM> to track the locations of objects in the environment.

In additional embodiments, sensors <NUM> may also track a position of an object by tracking the position of a companion object. For example, when tracking the position of a body part, sensors <NUM> may track the position of an object associated with the body part, such as a ring, a thimble, a glove, an effector, and so forth. Furthermore, sensors <NUM> may include devices that implement time-of-flight measurements to determine a distance to an object within the environment. In particular, sensors <NUM> may include capacitive sensors and receivers operating in a shunt mode or a transmit mode. In addition, sensors <NUM> may include generator devices that generate electromagnetic transmissions and/or receiver devices that detect electromagnetic transmissions reflecting off of objects in the environment. Receiver devices may include coils that generate an electric field and/or a magnetic field in response to receiving a reflected electromagnetic transmission. In some embodiments, two or more receiver coils may be oriented orthogonally and used in conjunction in order to provide directional information with respect to a position of a detected object. In further embodiments, acoustic devices may be implemented to detect objects located in an environment. These acoustic devices may include speaker devices that output sound and microphone devices that detect sound reflected off of objects in the environment.

In addition, device <NUM> includes transducers <NUM> which include one or more types of devices that can detect and generate one or more types of waves (e.g., mechanical waves, electromagnetic waves, etc.). For example, transducers <NUM> could include transducer devices, such as ultrasonic transducer array, that generate and detect ultrasonic waves. In operation, transducers <NUM> generate waves (e.g., continuous waves, wave pulses, pings etc.) that can be used to determine the location of objects in the environment. The generated waves travel into the environment and reflect off of objects in the environment. Transducers <NUM> detect reflected waves and transmit signals to processing unit <NUM>, via I/O interfaces <NUM>, that indicate that a wave was detected and may further indicate one or more characteristics of the wave. In addition, transducers <NUM> emit waves in order to generate a haptic sensation on an object. In one embodiment, transducers <NUM> generate continuous waves that generate a haptic sensation on a user.

In one embodiment, two or more of transducers <NUM> are attached to a sleeve or jacket of the user separately from one another. Thus, the relative position of such transducers <NUM> to each other can change dynamically. In such a case, the locations of each of the two or more transducers <NUM> in three-dimensional space are tracked separately. In one embodiment, a location tracking sensor, e.g., a sensor <NUM>, tracks the location of each transducers. In such an embodiment, the sensor may track infrared reflective markers placed on a transducer <NUM> and/or the sensor may track the location of a transducer using ultrasonic tracking. In alternative embodiments, the transducers <NUM> may include a location sensing sensor such that the transducer can self-localize with regards to a reference point, such as a user or another transducer <NUM>. In such an embodiment, the position sensing sensor may be an accelerometer that can derive location from an acceleration.

With respect to generating waves for determining the location of an object, transducers <NUM> generate a wave pulse (referred to herein as a "ping") with a particular waveform, frequency, and amplitude. In some embodiments, transducers <NUM> generate a ping with increasing or decreasing frequency. For example, an ultrasonic transducer included in transducers <NUM> could modulate the frequency of ultrasonic waves being generated in order to generate a ping with an increasing frequency. In addition, a ping could have intervals of increasing and decreasing frequency and/or amplitude.

In addition to generating pings, transducers <NUM> detect various types of waves that are incident on transducers <NUM>. Transducers <NUM> convert the detected waves into an electrical signal that is transmitted to processing unit <NUM>. The electrical signal indicates that a wave has been detected. The electrical signal may further indicate one or more characteristics of the wave, including the waveform, the frequency, the phase, and the amplitude of the wave. For example, transducers <NUM> could detect ultrasonic waves that have reflected off of a target and have traveled towards transducers <NUM>. In one embodiment, transducers <NUM> detect ping echoes of pings that have reflected off of an object in the environment.

With respect to generating a haptic sensation on a user, transducers <NUM> generate various types of haptic output, including continuous waves. For example, transducers <NUM> could include ultrasonic transducers included in ultrasonic transducer array that generate ultrasonic waves. Transducers <NUM> may be configured to generate waves that travel towards a particular part of a user, including his or her hand, leg, forearm, wrist, palm, neck, trunk, etc. When an ultrasonic wave interacts with a user's skin, the person may feel a haptic sensation. In some embodiments, transducers <NUM> generate a particular type of haptic sensation that may indicate to a user that he or she should take a particular action. For example, the haptic sensation could move in a certain direction on the user's body, thereby indicating to the user that she should move in the specified direction. Transducers <NUM> may further generate a haptic sensation with a particular size, shape, orientation, and/or intensity.

In additional embodiments, transducers <NUM> may generate audio output. In particular, transducers <NUM> may generate ultrasonic waves that demodulate to produce highly directional audio output. For example, sound may be produced by performing frequency and/or amplitude modulation of ultrasonic waves produced by transducers <NUM>. In various embodiments, transducers <NUM> can project a narrow beam of modulated ultrasonic waves that are powerful enough to substantially alter the speed of sound in the air proximate to the narrow beam. In particular, the air within the beam behaves nonlinearly and extracts the modulation signal from the ultrasound, resulting in audible sound that can be heard along the path of the beam and/or that appears to radiate from any surface that the beam strikes. This technology allows a beam of sound to be projected over a long distance and to only be audible within a small well-defined area. In particular, listeners outside the path of the beam hear very little or nothing at all. Accordingly, audio output may be directed to the ears of a user to provide a private audio experience.

Computing device <NUM> includes, without limitation, processing unit <NUM>, input/output (I/O) interfaces <NUM>, and memory <NUM>. Computing device <NUM> as a whole may be a microprocessor, an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC) and so forth. In various embodiments, computing device <NUM> receives data from sensors <NUM> and/or transducers <NUM>. Furthermore, computing device <NUM> may transmit control signals and/or data to sensors <NUM> and transducers <NUM>. In various embodiments, computing device <NUM> may be implemented as part of an external mobile computing system, such as a smart phone, a tablet, a smart watch, smart eyeglasses, and so forth. Accordingly, computing device <NUM> may be communicatively coupled with sensors <NUM> and/or transducers <NUM> via a wired or wireless connection. In particular, wireless connections may include, for example and without limitation, radio frequency (RF)-based, infrared-based, and/or network-based communication protocols. Additionally or alternatively, computing device <NUM> may be implemented as part of a cloud-based computing environment. In such embodiments, computing device <NUM> may be networked with various other elements of device <NUM>, such as sensors <NUM> and/or transducers <NUM>.

Processing unit <NUM> may include a central processing unit (CPU), a digital signal processing unit (DSP), a sensor processing unit, a controller unit, and so forth. Processing unit <NUM> may be physically embedded into device <NUM>, may be part of a cloud-based computing environment, and/or may be physically introduced into device <NUM> by a user, such as in a mobile or wearable device.

In various embodiments, processing unit <NUM> is configured to analyze data acquired by sensors <NUM> and/or transducers <NUM> to determine attributes of the user, such as the locations, distances, orientations, etc. of the user and the visual features of the user. The determined attributes of the user may be stored in a database <NUM>.

Processing unit <NUM> generate control signals to modify a frequency, an amplitude, and/or a phase of haptic output and/or audio output from transducers <NUM> in order to produce a haptic sensation and/or direct an audio signal to an ear of a user.

Processing unit <NUM> is further configured to compute a vector from a location of a transducer <NUM> to a position of an ear of the user based on the determined attributes. For example, and without limitation, processing unit <NUM> may receive data from sensor <NUM> and/or transducers <NUM> and process the data to dynamically track the movements of an object such as a head, an ear, a hand, an arm, etc. of the user. Based on changes to the position and orientation of the object, processing unit <NUM> may compute one or more vectors in order to cause transducers <NUM> to direct haptic output and/or audio output directly to the object. Processing unit <NUM> further determines, based on sensor data, various characteristics of objects in the environment of device <NUM>. Characteristics of the objects may include a position, an orientation, a size, a shape, and so forth. In addition, processing unit <NUM> may also determine a direction in which one or more objects are moving, a velocity of the moving objects, and/or an acceleration of the moving objects. In various embodiments, processing unit <NUM> is configured to execute applications, such as transducer application <NUM>, included in memory <NUM>. In addition, input/output (I/O) interfaces <NUM> may include one or more interfaces that coordinate the transfer of data and control signals between sensors <NUM>, transducers <NUM>, and processing unit <NUM>. The data and controls signals may be transmitted and received over a wired connection or wirelessly (e.g., radio frequency, infrared, network connect, etc.).

Memory <NUM> includes transducer application <NUM> configured to communicate with database <NUM>. Processing unit <NUM> executes transducer application <NUM> to implement the overall functionality of device <NUM>. Memory device <NUM> may be embedded in an external computing environment and/or may be physically part of device <NUM>. Moreover, memory device <NUM> may be included in a cloud-based computing environment.

Database <NUM> may store the types, locations, and orientations of sensors <NUM> and transducers <NUM>. In addition, for each type of transducer <NUM>, database <NUM> may store various parameters including an amplitude range and a frequency range of the type of transducer <NUM>, the waveform shapes that can be generated by the type of transducer <NUM>, and various possible device configurations of that type of transducer <NUM>. Similarly, database <NUM> may store the types of waves, waveforms, frequencies, and amplitudes that each type of device in transducers <NUM> can detect. Database <NUM> may also store instructions for configuring transducers <NUM> to produce haptic sensations, audio output, and to execute sensing functions. In addition, database <NUM> may store the positions and orientations of transducers <NUM> as well as user preferences data relating to the types of haptic sensations to generate on the user (e.g., a part of the user on which to generate haptic sensations).

In various embodiments, transducer application <NUM> coordinates the function of the device <NUM>. In particular, transducer application <NUM> configures the sensors <NUM> to generate sensor data reflective of the location of an object in the environment. The object may include a body part of a user, such as a hand, an arm, a neck, an ear, etc..

In various embodiments, transducer application <NUM> configures sensors <NUM> to generate sensor data that indicates the presence and/or position of an object in the environment. For example, transducer application <NUM> configures sensors <NUM> to detect the location of a body part of a user (e.g., an ear). In particular, transducer application <NUM> may configure an optical sensor to generate image data of a user's ear. Transducer application <NUM> may perform a calculation to determine the location of the user's ear relative to the position of the optical sensor. Additionally or alternatively, transducer application <NUM> may calculate a distance between sensors <NUM> and the detected object. For example, transducer application <NUM> may utilize time-of-flight computations to determine a distance between sensors <NUM> and the detected object.

Furthermore, transducer application <NUM> may configure sensors <NUM> to determine the location of, i.e., localize, multiple regions of the detected object. Accordingly, transducer application <NUM> may control the resolution at which sensors <NUM> image detected objects. With higher resolution imaging, transducer application <NUM> may determine a features and characteristics of the detected object, such as a pose of a hand, an orientation of a hand, and separate distances to each finger of a hand. Transducer application <NUM> may also use the relative distance to a first set of objects in order to localize a second set of objects via triangulation. For example, transducer application <NUM> may utilize relative distances to an eye, an ear, a nose, and a mouth on a face to localize a position of an ear.

In addition, transducer application <NUM> may implement one or more machine learning algorithms to generate hand pose estimates. Transducer application <NUM> may implement machine learning algorithms with one or more learning styles, such as supervised learning, unsupervised learning, and/or semi-supervised learning. Transducer application <NUM> may acquire training data stored in database <NUM>, and/or may download training data via I/O interfaces <NUM> from an external data repository, such as a cloud-based data source. In various embodiments, after selecting a machine learning algorithm, transducer application <NUM> generates a model based on an initial dataset. Transducer application <NUM> further performs a training step in which the model is trained on a variety of acquired training datasets. For example, in supervised learning, transducer application <NUM> compares the output of the model to known results. Transducer application <NUM> further modifies various parameters of the generated model until the output generated by the model matches the known results. In various embodiments, transducer application <NUM> may modify parameters until the model achieves a threshold success rate.

In various embodiments, the training datasets may include sensor data associated with various objects that is generated by various types of sensors. For example, a training data set may be generated based on image data of different types of human ears generated by a single type of sensor (e.g., an optical sensor) or multiple types of sensors (e.g., an acoustic sensor, a thermal-based sensor, and an infrared-based sensor). Transducer application <NUM> may select a machine learning algorithm and generate a model based on an initial dataset. Further, as described above, transducer application <NUM> may train the model to be able to recognize, e.g., derivative geometric features of a distal object, such as a center of mass of a hand, a skeletal structure of a hand, and/or a spherical approximation of a hand. Additionally or alternatively, transducer application <NUM> may directly calculate derivative geometric features of distal objects from sensor data generated by sensors <NUM>.

In various embodiments, transducer application <NUM> updates detected attributes of objects as sensors <NUM> generate sensor data. For example, transducer module <NUM> may update a distance, a position, an orientation, a size, etc. or a detected object. In some embodiments, transducer application <NUM> may analyze a time-delineated sequence of characteristics to generate a prediction of a next position of the detected object. Transducer application <NUM> may utilize one or more machine learning algorithms to model the motion of the detected object. In addition, transducer application <NUM> may implement a buffer and/or a real-time gesture detection and recognition algorithm to detect a gesture from a time-delineated sequence of characteristics. In various embodiments, transducer application <NUM> may image the environment proximate to transducers <NUM> by configuring transducers <NUM> to generate pings and detect ping echoes reflected from off of objects in the environment. By analyzing ping echoes, transducer application <NUM> may localize and/or image objects in the environment of device <NUM>.

In operation, transducer application <NUM> may configure transducers <NUM> to generate pings with various frequency, amplitude, and waveform characteristics. In some embodiments, transducer application <NUM> configures at least one of transducers <NUM> to emit a ping by generating wave pulses having a constant or variable frequency. Further, transducer application <NUM> configures at least one of transducers <NUM> to detect ping echoes, for example, the wave pulses that are reflected back to transducers <NUM> after a ping encounters an object in the environment. Transducer application <NUM> analyzes ping echoes detected by transducers <NUM> in order to determine various characteristics of the environment proximate to transducers <NUM>. In particular, transducer application <NUM> may utilize time-of-flight calculations to localize objects in an environment about device <NUM>. For example, transducer application <NUM> could associate a ping echo with an emitted ping and further calculate the time interval between transducers <NUM> emitting the ping and detecting the ping echo (e.g., by comparing the waveforms, frequency patterns, amplitudes, and phase of the detected ping echo to the emitted ping). Based on the calculated time-of-flight and various parameters of the particular ping, transducer application <NUM> could determine the position of an object in the environment. In particular, transducer application <NUM> could determine the distance that the ping traveled in the air before being reflected off of an object. In one embodiment, transducer application <NUM> divides the calculated time-of-flight by twice the speed of the wave associated with the ping in order to determine the distance that the ping traveled before being reflected off of an object. Transducer application <NUM> may use that distance to determine the location of the object with respect to one or more of transducers <NUM>.

Furthermore, transducer application <NUM> could identify, by comparing multiple ping-ping echo pairs, the type of object that is proximate to transducers <NUM>. In one embodiment, transducer application <NUM> utilizes a large number of pings with distinct frequencies and/or frequency patterns in order to image one or more fine structure details of an object. In such an embodiment, upon identifying the location of an object, transducer application <NUM> may configure transducers <NUM> to increase the frequency spectrum utilized when generating pings in order to increase the number of unique ping-ping echo pairs. Increasing the number of unique sets of ping-ping echo pairs facilitates transducer application <NUM> in generating a higher resolution image of the environment. Based on the image, transducer application <NUM> generates object data that includes the position and characteristics of the objects in the environment detected by analyzing ping echoes. Transducer application <NUM> may identify the type of object, the orientation of the object, and surface details of the object by analyzing ping echoes. In addition, transducer application <NUM> may determine other characteristics of the object, such as the manner in which the object is moving in the environment. For example, transducer application <NUM> could calculate the radial velocity of an object by calculating the Doppler shift between emitted pings and detected ping echoes.

In addition, transducer application <NUM> coordinates the output of haptic and audio signals from transducers <NUM>. In particular, transducer application may configure transducers <NUM> to direct haptic output and/or audio output to objects in the environment around device <NUM>. With respect to haptic output, transducer application <NUM> dynamically selects one or more of transducers <NUM> to output haptic output with a particular frequency and amplitude. The haptic output produces a haptic surface at a particular location in space. When an object contacts the haptic surface, a haptic sensation may be generated on the object. With respect to audio output, transducer application <NUM> processes haptic output produces by transducers <NUM> to produce a narrow band of audio output. The audio output may be directed to an ear of a user to provide private audio for the user.

In operation, transducer application <NUM> may select a location at which to generate a haptic sensation. Transducer application <NUM> may access the positions, orientations, and types of transducers <NUM> in device <NUM> from database <NUM>. In various embodiments, transducers <NUM> may include ultrasonic transducers. Ultrasonic transducers may be arranged in arrays (e.g., <NUM>-by-<NUM>, <NUM>-by-<NUM>, <NUM>-by-<NUM>, etc.). Each ultrasonic transducer emits ultrasonic waves of a certain frequency, phase, and intensity. Transducer application <NUM> configures the ultrasonic transducers in a manner such that haptic output generated by two or more ultrasonic transducers occupy a particular location in space at a particular time. When this occurs, the ultrasonic output of each ultrasonic transducer interferes constructively and/or destructively with the ultrasonic output of one or more other ultrasonic transducers. Transducer application <NUM> configures ultrasonic transducers such that the constructive and/or destructive interference occurs at the location at which the haptic output reaches the user, thereby generating a specific type of haptic sensation on the user. By modifying the intensity, phase, and frequency of the haptic output of each ultrasonic transducer, transducer application <NUM> shifts the location of intensity peaks, increases or decreases the number of intensity peaks, and/or adjusts the shape and/or magnitude of one or more intensity peaks. In this manner, transducer application <NUM> configures ultrasonic transducers and/or transducers <NUM> in general to generate a specific type of haptic sensation.

In various embodiments, transducer application <NUM> may configure transducers <NUM> to generate a haptic sensation proximate to various objects (e.g., a hand). The hand may be located at various positions around device <NUM>, such as in front of device <NUM>, above device <NUM>, beside device <NUM>, etc. Additionally or alternatively, transducer application <NUM> may configure transducers <NUM> to generate haptic surfaces at various locations in space, such as beside objects, above objects, below objects, and so forth. For example, device <NUM> may be integrated in an augmented reality (AR), virtual reality (VR), and/or mixed reality (MR) environment. Transducer application <NUM> may configure transducers <NUM> to generate a haptic sensation at positions in the physical environment that match the visually-perceived positions of objects. Accordingly, users may feel haptic sensations when their hands contact and/or intersect regions of the physical environment that correspond to a position of an object in the visually-perceived environment. In one embodiment, the haptic sensations may be sensed by one or more other users in proximity to device <NUM> but not necessarily in physical contact with device <NUM>.

Additionally or alternatively, transducer application <NUM> may configure transducers <NUM> to generate audio output. In operation, transducer application <NUM> generates audio output by modulating various characteristics of output produced by transducers <NUM>. In particular, transducer application <NUM> may perform frequency and/or amplitude modulation to configure the transducers <NUM> to produce modulated ultrasonic waves, which demodulate to produce audio output.

In some embodiments, the audio output may reflect off of objects in the environment, such as a hand. For example, in an AR/VR/MR environment, transducer application <NUM> may configure transducers <NUM> to reflect audio output off of a hand of a user to produce an audio event. The audio event may be produced when a user contacts and/or intersects regions of the physical environment that correspond to a position of an object in the visually-perceived environment. In particular, the user may perceive that the audio signal originates from a location of the hand. Alternatively, transducer application <NUM> may implement one or more transfer functions (e.g., head related transfer functions) to alter the sound produced by the transducers <NUM> to make sound appear to originate from a selected location in the environment. In particular, the selected location may differ from the location of the transducers <NUM>. In addition, transducer application <NUM> may configure transducers <NUM> to deliver audio output directly to a user's ears. In particular, transducer application <NUM> may analyze sensor data from sensors <NUM> to track a location of an ear of a user. Based upon the determined location of the ear, transducer application <NUM> may configure transducers <NUM> to generate a narrow beam of audio to location of the ear. In various embodiments, transducer application <NUM> configures transducers <NUM> to generate audio output that tracks the location of the ear. Accordingly, a user wearing and/or otherwise carrying the device <NUM> may receive private audio, while dynamically and continuously changing a position of his or her ear and/or a position of the device <NUM>.

In various embodiments, transducer application <NUM> configures transducers <NUM> to direct haptic output and/or audio output to various locations and/or to various objects. In particular, transducer application <NUM> may configure transducers <NUM> to generate a haptic sensation and/or direct audio output to users who are not wearing and/or carrying the device <NUM>. For example, transducer application <NUM> may configure transducers <NUM> to generate a haptic sensation on a hand of an individual who is not the user. Accordingly, device <NUM> may function as a multi-user device. Further, transducer application <NUM> may configure transducers <NUM> to provide private audio to individuals who is not the user thus creating a private communication channel with other via audio demodulating on the ears or through a shared virtual object.

According to some examples not falling under the scope of the claims, transducer application <NUM> may dynamically select a first set of transducers <NUM> to perform sensing functions, a second set of transducers <NUM> to generate a haptic sensation, and a third set of transducers <NUM> to produce audio output. Transducer application <NUM> may perform the dynamic selection based on various parameters including the number of transducers <NUM> in device <NUM>, the position and/or orientation of each of transducers <NUM>, the output strength of each of transducers <NUM>. Additionally, transducer application <NUM> may further perform dynamic selection based on the size, shape, location, and orientation of the haptic sensations for generation, and the location, volume, the frequency range, and the amplitude range of the audio signal for generation. For example, transducer application <NUM> may configure a first set of transducers <NUM> that are proximate to a user's shoulders to generate audio output to a user's ear, while a second set of transducers <NUM> that are located on a user's chest may generate a haptic sensation on a user's hand.

According to embodiments of the disclosure, transducer application <NUM> configures transducers <NUM> to operate in a dedicated mode, where each of transducers <NUM> performs a single function (e.g., sensing, haptic output generation, and/or audio output generation). Alternatively, according to examples not falling under the scope of the claims, transducer application <NUM> may pre-configure each of transducers <NUM> to switch between two or more functions at predetermined time intervals. For example, one transducer may perform sensing functions for <NUM> milliseconds and then switch to generating audio output for <NUM> milliseconds, while another transducer performs sensing for <NUM> milliseconds, generates audio output for <NUM> milliseconds, and then generates haptic output for <NUM> milliseconds.

In some examples not falling under the scope of the claims, transducer application <NUM> may configure transducers <NUM> to simultaneously perform multiple functions. For example, transducer application <NUM> may configure transducers <NUM> to detect reflected haptic output and/or audio output that reflects off of objects in the environment. Transducer application <NUM> may further localize objects based on the detected reflected haptic output and/or audio output. Similarly, in some embodiments, transducer application <NUM> may configure pairs of transducers <NUM> to produces haptic output that constructively and destructively interferes. The interference pattern may include regions of haptic output that may be audible and regions of haptic output that may generate a haptic sensation on an object. Accordingly, transducer application <NUM> may configure pairs of transducers <NUM> to simultaneously produce haptic output and audio output.

In various embodiments, computing device <NUM> may be networked with one or more external devices. Computing device <NUM> may receive wired and/or wireless signals from the external device via I/O interfaces <NUM>. Further, processing unit <NUM> may execute transducer application <NUM> to determine a modification to transducers <NUM>. For example, a user may control transducers <NUM> via a mobile phone, a remote controller, a computer, an augmented reality device, and so forth. Additionally or alternatively, a user may control transducers <NUM> by generating a gesture. In operation, transducer application <NUM> may analyze sensor data from sensors <NUM> to detect gestures performed by a user and/or any objects proximate to device <NUM>. Transducer application <NUM> may access a look-up table in database <NUM> to associate the gesture with a modification to apply to the transducers <NUM>. Modifications may include configuring transducers <NUM> to produce a sound tone, adjust a frequency and/or amplitude of the sound tone, adjust a location of the sound tone and/or a target object to which to direct the sound tone, and so forth. Modifications may further include adjusting a location, intensity, shape, size, and/or pattern of haptic output produced by transducers <NUM>. In addition, modifications may include selecting one or more of an audible sound mode, and a haptic output mode, selecting between multiple audio input streams, such as navigations systems, music, voice-agents services, and so forth. Modifications also may include play/pause functionalities, stop, skip, rewind, and fast forward commands for haptic output and/or audio output.

<FIG> illustrates device <NUM> of <FIG> in the form factor of a neck-worn device, according to various embodiments. As shown, transducers <NUM> may be arranged as a series of transducer arrays <NUM>. Each transducer array <NUM> may be located at any given position on device <NUM>. The position of a transducer array <NUM> impacts the directionality of the haptic and/or audio output emitted from that transducer array <NUM>. For example, transducer array <NUM>-<NUM> may be located at a position on the device <NUM> that is closest to a head region of a wearer of the device <NUM>. Accordingly, transducer application <NUM> may configure transducer array <NUM>-<NUM> to produce haptic output and audio output toward objects above the device <NUM>, including, but not limited to, a neck, a face, an ear, a head, and so forth. Sensor <NUM>-<NUM> is also located a position on the device <NUM> that is closest to a head region of a wearer of the device <NUM>. Accordingly, sensor <NUM>-<NUM> may track the position of objects above device <NUM>, enabling transducer application <NUM> to configure transducers in transducer array <NUM>-<NUM> to generate haptic output and/or audio output that targets a particular object and/or region in space.

Similarly, transducer array <NUM>-<NUM> may be located on the front face of device <NUM>. Accordingly, transducer application <NUM> may configure transducer array <NUM>-<NUM> to produce haptic output and audio output toward objects in front of device <NUM>, including, but not limited to, a hand, a wrist, a forearm, and so forth. Sensor <NUM>-<NUM> is also located on the front face of device <NUM>. Accordingly, sensor <NUM>-<NUM> may track the position of objects in front of device <NUM>, enabling transducer application <NUM> to configure transducers in transducer array <NUM>-<NUM> to generate haptic output and/or audio output that targets a particular object and/or region in space located in front of a user. For example, transducer application <NUM> may configure transducers in transducer array <NUM>-<NUM> to generate a haptic sensation proximate to a hand of a user. In addition, transducer application <NUM> may configure transducers in transducer array <NUM>-<NUM> to target people close by but not necessarily in physical contact with device <NUM>. For example, transducer application <NUM> may configure transducers in transducer array <NUM>-<NUM> to generate a haptic sensation proximate to the hand of another person in proximity. The location of the haptic sensation may correspond to a location of an object being viewed by the user in a VR/AR/MR environment.

<FIG> illustrates device <NUM> of <FIG> in another form factor of a neck-worn device, according to various embodiments. As discussed above, the position of a transducer array <NUM> impacts the directionality of the haptic and/or audio output emitted from that transducer array <NUM>. The haptic output of a transducer array <NUM> in the form factor illustrated in <FIG> may only point upwards. In one embodiment, the upward haptic output generates a tapping sensation on the user's head, including, for example and without limitation, on the user's temple, behind the ear, the cheeks, and the chin. Such haptic output may correspond to navigation instructions being delivered to the user or may direct the user's attention in a specific direction.

<FIG> illustrate device <NUM> of <FIG> in various other form factors of a wearable device, according to various embodiments. As discussed above, in each of the form factors illustrated in <FIG>, the position of a transducer array <NUM> impacts the directionality of the haptic and/or audio output emitted from that transducer array <NUM>. In the form factors illustrated in <FIG>, device <NUM> is asymmetric such that the transducer arrays <NUM> on one side of device <NUM> vary in number relative to another side of device <NUM>. In the form factor illustrated in <FIG>, device <NUM> is an epaulette attached to a shoulder of a user.

<FIG> illustrates the device <NUM>, in which each transducer array performs one or more of the sensing function, the haptic output function, and the audio output function, according to various embodiments. As shown, transducers <NUM> are arranged as transducer array A <NUM>-<NUM>, transducer array B <NUM>-<NUM>, and transducer array C <NUM>-<NUM>. Device <NUM>, as shown in <FIG>, may be included in any of the form factors shown in <FIG>. Further, device <NUM> as shown in <FIG>, may operate in conjunction with goggles <NUM> to provide one or more features associated with a VR, AR, or MR environment.

In some embodiments, transducers <NUM> operate in a dedicated mode. In particular, transducers <NUM> in transducer array A <NUM>-1generate audio output, transducers <NUM> in transducer array B <NUM>-2generate haptic output, and transducers <NUM> in transducer array C <NUM>-3performing sensing functions (e.g., generating pings and detecting ping echoes). Device <NUM> may also include one or more dedicated sensors <NUM> for object detection and tracking.

In another embodiment, transducers <NUM> may operate in a switching mode. Accordingly, each of transducer array A <NUM>-<NUM>, transducer array B <NUM>-<NUM>, and transducer array C <NUM>-<NUM> may switch between two or more of performing sensing, generating haptic output, and generating audio output. For example, and without limitation, during a first time interval, transducer array C <NUM>-<NUM> may generate a ping, while transducer array A <NUM>-<NUM> generates audio output and transducer array B <NUM>-<NUM> generate haptic output. During a second time interval, one or more of transducer array A <NUM>-<NUM>, transducer array B <NUM>-<NUM>, and transducer array C <NUM>-<NUM> may detect ping echoes. Additionally or alternatively, during the second time interval transducer array B <NUM>-<NUM> may generate audio output, while transducer array C <NUM>-<NUM> generates haptic output. Transducer array A <NUM>-<NUM> may continue to generate audio output and/or may detect reflected ping echoes.

In an example not falling under the scope of the claims, transducer application <NUM> may dynamically configure each transducer <NUM> to perform one or more of the sensing functionalities (e.g., generate pings, detect ping echoes), generating haptic output, and/or generating audio output. For example, transducer application <NUM> may dynamically select a first set of transducers <NUM> to perform sensing functions, a second set of transducers <NUM> to generate a haptic sensation, and a third set of transducers <NUM> to produce audio output. Transducer application <NUM> may perform the dynamic selection based on various parameters including the number of transducers <NUM> in device <NUM>, the position and/or orientation of each of transducers <NUM>, the output strength of each of transducers <NUM>. Additionally, transducer application <NUM> may further perform dynamic selection based on the size, shape, location, and orientation of the haptic sensations for generation, and the location, volume, the frequency range, and the amplitude range of the audio signal for generation. For example, as shown, transducer application <NUM> may configure a first set of transducers <NUM> that are proximate to a user's shoulders to generate audio output to a user's ear, while a second set of transducers <NUM> that are located on a user's chest may generate a haptic sensation on a user's hand.

<FIG> is a flow diagram of method steps for providing audio content and haptic sensations to a user, according to various embodiments. Although the method steps are described in conjunction with the systems of <FIG>, persons skilled in the art will understand that any system configured to perform the method steps, in any order, falls within the scope of the present disclosure.

As shown, a method <NUM> begins at step <NUM>, where transducer application <NUM> determines a sound tone and a haptic sensation to deliver to a user. Transducer application <NUM> may determine the sound tone and/or the haptic sensation based on detecting, via sensors <NUM>, that a user performed a gesture. Transducer application <NUM> may access a look-up table stored in database <NUM> to determine that a sound tone to generate and/or a haptic sensation to generate. Additionally or alternatively, transducer application <NUM> may receive control signals from an external device, such as a mobile phone, a remote controller, and/or a computer. Transducer application <NUM> may select a sound tone and/or a haptic sensation based on the control signals.

At step <NUM>, transducer application <NUM> selects a first location to which to deliver audio output and a second location to which to deliver haptic output. In various embodiments, transducer application <NUM> may configure transducers <NUM> to perform sensing functions to identify objects located around the device <NUM>. In alternative embodiments, sensors <NUM> can be used instead of or in combination with transducers <NUM> to perform sensing functions to identify objects located around the device <NUM>. Transducer application <NUM> may analyze the sensor data to identify a type of the object (e.g., a body part, a physical object, etc.). Once identified, transducer application <NUM> may further analyze the sensor data to localize the object and select whether to output audio output and/or haptic output to the object. In particular, transducer application <NUM> may select to output audio output to a first location and haptic output to a second location. The first location and the second location may correspond to a location of an object, a group of objects, and/or regions of space proximate to objects.

At step <NUM>, transducer application <NUM> configures a first set of transducers <NUM> to deliver audio output to the first location. In various embodiments, transducers <NUM> may be arranged in a transducer array, such as transducer array A <NUM>-<NUM>. In particular, transducer application <NUM> configures each of transducers <NUM> to generate ultrasonic waves that demodulate in space to produce audio output. The audio output is highly direction and providing a private audio to a user.

Further, at step <NUM>, transducer application <NUM> configures a second set of transducers <NUM> to generate haptic output that produces a haptic sensation at the second location. In various embodiments, transducer application <NUM> may configure transducers <NUM> to generate a haptic sensation at various locations in space. The location of the haptic sensation may correspond to a location of an object being viewed in a VR/AR/MR environment. Additionally or alternatively, transducer application <NUM> may configure transducers <NUM> to generate haptic output that generates a haptic sensation on a user. Transducer application <NUM> may configure one or more transducers <NUM> to generate haptic output with a particular frequency, amplitude, and phase in order to control the size, shape, location, and movement pattern of the haptic sensation produced on the object. In some embodiments, the object may be a body part of a user. In some other embodiments, the object may be a body part of a person in proximity to the user.

In sum, the device delivers audio output and generates haptic sensations on or proximate to objects in the environment via which a user can perceive haptic sensations. The device includes transducer devices that operate in various modalities, including generating audio output, generating haptic output, and sensing objects in the environment. When generating audio output, the transducers produce ultrasonic waves that demodulate to produce audio output. The transducer application may configure the transducers to deliver the audio output to a specific location (e.g., a location of an ear of a user). When generating haptic output, the transducers output directional haptic output to various locations to the environment. The haptic output may produce a haptic sensation at each location. The haptic sensation may be sensed by a user in proximity to the various locations. Finally, when performing sensing, the transducers generate pings and detect ping echoes reflected off of objects in the environment. The transducer application may perform time-of-flight calculations to localize one or more objects in the environment based on the detected ping echoes. In various not claimed embodiments, the transducer application may dynamically configure each transducer to operate in one or more of the three functionalities and/or may preconfigure each transducer device to operation in one or more of the three functionalities.

At least one advantage of the disclosed techniques is that audio can be transmitted directly to the ears of a user, enabling the user to listen to audio content (e.g., music, voice conversations, notifications, etc.) without disturbing sound-sensitive objects around him or her. Additionally, because the device may be shoulder-mounted, the system does not isolate the user from sounds in his or her environment. Further, the device may interact with a user by generating haptic sensations on and/or proximate to the user. Accordingly, the device may generate haptic sensations that provide indicates to the user or people in proximity to the user (e.g., indicating the location of virtual objects in space).

Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the appended claims.

Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable processors.

The disclosure has been described above with reference to specific embodiments. Persons of ordinary skill in the art, however, will understand that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the appended claims. For example, and without limitation, although many of the descriptions herein refer to specific types of highly-directional speakers, sensors, and audio events, persons skilled in the art will appreciate that the systems and techniques described herein are applicable to other types of highly-directional speakers, sensors, and audio events. The foregoing description and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claim 1:
A system for providing audio content and haptic sensations to a user, the system comprising:
a device comprising a first set of transducers (<NUM>) and a second set of transducers (<NUM>), wherein the device is configured to be integrated into a neck worn device or a shoulder worn device or is configured to be integrated into or attached to an article of clothing, and wherein the second set of transducers (<NUM>) form an ultrasonic transducer array that is configured to generate ultrasonic waves that travel towards a particular body part of the user such that when the ultrasonic wave interacts with the user's skin, the user feels a haptic sensation; and
a processor (<NUM>) coupled to the first set of transducers (<NUM>) and the second set of transducers (<NUM>), the processor being configured to:
determine a sound tone and a haptic sensation to deliver to the user;
select a first location at which to deliver audio output corresponding to the sound tone and a second location to which to deliver haptic output corresponding to the haptic sensation;
configure the first set of transducers (<NUM>) to deliver the audio output to the first location; and
configure the second set of transducers (<NUM>) to generate the haptic output for delivery to the second location, wherein the second location differs from a location of the second set of transducers.